2024-03-30T13:21:05.948 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-30T13:21:05.948 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} --name teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630102 --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630102/orig.config.yaml 2024-03-30T13:21:05.981 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T13:21:06.063 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630102 branch: squid description: orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} email: ceph-qa@ceph.com first_in_suite: false job_id: '7630102' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) sha1: 4e50bbcdd1630c7760b708f3a5b4d38a121e66cb ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 4e50bbcdd1630c7760b708f3a5b4d38a121e66cb selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: 5af0400050a123dc31f105425174f106e057276a owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 8777 sha1: 4e50bbcdd1630c7760b708f3a5b4d38a121e66cb sleep_before_teardown: 0 subset: 43/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_5af0400050a123dc31f105425174f106e057276a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 5af0400050a123dc31f105425174f106e057276a targets: smithi043.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM6PsHyrzJ+N5gtxEXZtkL0DOB2Yc+1hk6UM158QukrtKXAgr1PH+4Y1Hhvn9HNMZfMFYP2GXk/TAmnZ4Q4miGk= smithi113.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCu38/1DtzijFJsFOMosEFKsw6EaV5ESDEi6yWWilL+EjrhFGkFcqWMJntOUWFnepOK844Bl4KbsA8br8yrqP48= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-29_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-30T13:21:06.063 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_5af0400050a123dc31f105425174f106e057276a/qa; will attempt to use it 2024-03-30T13:21:06.064 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_5af0400050a123dc31f105425174f106e057276a/qa/tasks 2024-03-30T13:21:06.064 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-30T13:21:06.066 INFO:teuthology.task.internal:Checking packages... 2024-03-30T13:21:06.091 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb' 2024-03-30T13:21:06.091 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-30T13:21:06.091 INFO:teuthology.packaging:ref: None 2024-03-30T13:21:06.091 INFO:teuthology.packaging:tag: None 2024-03-30T13:21:06.091 INFO:teuthology.packaging:branch: squid 2024-03-30T13:21:06.091 INFO:teuthology.packaging:sha1: 4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T13:21:06.091 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-30T13:21:06.432 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1973.g5af04000 2024-03-30T13:21:06.433 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-30T13:21:06.439 INFO:teuthology.task.internal:no buildpackages task found 2024-03-30T13:21:06.439 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-30T13:21:06.468 INFO:teuthology.task.internal:Saving configuration 2024-03-30T13:21:06.480 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-30T13:21:06.485 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T13:21:06.510 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630102', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 13:15:45.196358', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM6PsHyrzJ+N5gtxEXZtkL0DOB2Yc+1hk6UM158QukrtKXAgr1PH+4Y1Hhvn9HNMZfMFYP2GXk/TAmnZ4Q4miGk='} 2024-03-30T13:21:06.531 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi113.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630102', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 13:15:45.194864', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCu38/1DtzijFJsFOMosEFKsw6EaV5ESDEi6yWWilL+EjrhFGkFcqWMJntOUWFnepOK844Bl4KbsA8br8yrqP48='} 2024-03-30T13:21:06.531 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-30T13:21:06.537 INFO:teuthology.task.internal:roles: ubuntu@smithi043.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-30T13:21:06.537 INFO:teuthology.task.internal:roles: ubuntu@smithi113.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-30T13:21:06.537 INFO:teuthology.run_tasks:Running task console_log... 2024-03-30T13:21:06.614 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f04aa7e80d0>, signals=[15]) 2024-03-30T13:21:06.615 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-30T13:21:06.620 INFO:teuthology.task.internal:Opening connections... 2024-03-30T13:21:06.620 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi043.front.sepia.ceph.com 2024-03-30T13:21:06.622 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:21:06.740 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi113.front.sepia.ceph.com 2024-03-30T13:21:06.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:21:06.831 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-30T13:21:06.837 DEBUG:teuthology.orchestra.run.smithi043:> uname -m 2024-03-30T13:21:06.855 INFO:teuthology.orchestra.run.smithi043.stdout:x86_64 2024-03-30T13:21:06.855 DEBUG:teuthology.orchestra.run.smithi043:> cat /etc/os-release 2024-03-30T13:21:06.912 INFO:teuthology.orchestra.run.smithi043.stdout:NAME="CentOS Stream" 2024-03-30T13:21:06.912 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION="9" 2024-03-30T13:21:06.912 INFO:teuthology.orchestra.run.smithi043.stdout:ID="centos" 2024-03-30T13:21:06.912 INFO:teuthology.orchestra.run.smithi043.stdout:ID_LIKE="rhel fedora" 2024-03-30T13:21:06.912 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_ID="9" 2024-03-30T13:21:06.912 INFO:teuthology.orchestra.run.smithi043.stdout:PLATFORM_ID="platform:el9" 2024-03-30T13:21:06.912 INFO:teuthology.orchestra.run.smithi043.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-30T13:21:06.912 INFO:teuthology.orchestra.run.smithi043.stdout:ANSI_COLOR="0;31" 2024-03-30T13:21:06.912 INFO:teuthology.orchestra.run.smithi043.stdout:LOGO="fedora-logo-icon" 2024-03-30T13:21:06.913 INFO:teuthology.orchestra.run.smithi043.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-30T13:21:06.913 INFO:teuthology.orchestra.run.smithi043.stdout:HOME_URL="https://centos.org/" 2024-03-30T13:21:06.913 INFO:teuthology.orchestra.run.smithi043.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-30T13:21:06.913 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-30T13:21:06.913 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-30T13:21:06.913 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server 2024-03-30T13:21:06.948 DEBUG:teuthology.orchestra.run.smithi113:> uname -m 2024-03-30T13:21:06.966 INFO:teuthology.orchestra.run.smithi113.stdout:x86_64 2024-03-30T13:21:06.967 DEBUG:teuthology.orchestra.run.smithi113:> cat /etc/os-release 2024-03-30T13:21:07.023 INFO:teuthology.orchestra.run.smithi113.stdout:NAME="CentOS Stream" 2024-03-30T13:21:07.023 INFO:teuthology.orchestra.run.smithi113.stdout:VERSION="9" 2024-03-30T13:21:07.024 INFO:teuthology.orchestra.run.smithi113.stdout:ID="centos" 2024-03-30T13:21:07.024 INFO:teuthology.orchestra.run.smithi113.stdout:ID_LIKE="rhel fedora" 2024-03-30T13:21:07.024 INFO:teuthology.orchestra.run.smithi113.stdout:VERSION_ID="9" 2024-03-30T13:21:07.024 INFO:teuthology.orchestra.run.smithi113.stdout:PLATFORM_ID="platform:el9" 2024-03-30T13:21:07.024 INFO:teuthology.orchestra.run.smithi113.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-30T13:21:07.024 INFO:teuthology.orchestra.run.smithi113.stdout:ANSI_COLOR="0;31" 2024-03-30T13:21:07.024 INFO:teuthology.orchestra.run.smithi113.stdout:LOGO="fedora-logo-icon" 2024-03-30T13:21:07.024 INFO:teuthology.orchestra.run.smithi113.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-30T13:21:07.024 INFO:teuthology.orchestra.run.smithi113.stdout:HOME_URL="https://centos.org/" 2024-03-30T13:21:07.024 INFO:teuthology.orchestra.run.smithi113.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-30T13:21:07.024 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-30T13:21:07.024 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-30T13:21:07.025 INFO:teuthology.lock.ops:Updating smithi113.front.sepia.ceph.com on lock server 2024-03-30T13:21:07.048 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-30T13:21:07.057 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-30T13:21:07.077 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-30T13:21:07.077 DEBUG:teuthology.orchestra.run.smithi043:> test '!' -e /home/ubuntu/cephtest 2024-03-30T13:21:07.079 DEBUG:teuthology.orchestra.run.smithi113:> test '!' -e /home/ubuntu/cephtest 2024-03-30T13:21:07.096 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-30T13:21:07.101 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-30T13:21:07.101 DEBUG:teuthology.orchestra.run.smithi043:> test -z $(ls -A /var/lib/ceph) 2024-03-30T13:21:07.137 DEBUG:teuthology.orchestra.run.smithi113:> test -z $(ls -A /var/lib/ceph) 2024-03-30T13:21:07.253 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-30T13:21:07.311 INFO:teuthology.run_tasks:Running task kernel... 2024-03-30T13:21:07.325 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-30T13:21:07.326 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-30T13:21:07.326 DEBUG:teuthology.orchestra.run.smithi043:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T13:21:07.326 DEBUG:teuthology.orchestra.run.smithi113:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T13:21:07.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:21:07.343 DEBUG:teuthology.orchestra.run.smithi043:> uname -r 2024-03-30T13:21:07.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:21:07.344 DEBUG:teuthology.orchestra.run.smithi113:> uname -r 2024-03-30T13:21:07.400 INFO:teuthology.orchestra.run.smithi043.stdout:5.14.0-361.el9.x86_64 2024-03-30T13:21:07.401 INFO:teuthology.task.kernel:Running kernel on smithi043: 5.14.0-361.el9.x86_64 2024-03-30T13:21:07.401 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum install -y kernel 2024-03-30T13:21:07.404 INFO:teuthology.orchestra.run.smithi113.stdout:5.14.0-361.el9.x86_64 2024-03-30T13:21:07.404 INFO:teuthology.task.kernel:Running kernel on smithi113: 5.14.0-361.el9.x86_64 2024-03-30T13:21:07.404 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum install -y kernel 2024-03-30T13:21:09.710 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS-9 - AppStream 9.0 MB/s | 6.7 MB 00:00 2024-03-30T13:21:09.959 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS-9 - AppStream 8.4 MB/s | 6.7 MB 00:00 2024-03-30T13:21:11.268 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS-9 - BaseOS 5.1 MB/s | 2.0 MB 00:00 2024-03-30T13:21:11.595 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-30T13:21:12.651 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - BaseOS 7.7 MB/s | 8.0 MB 00:01 2024-03-30T13:21:17.377 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - AppStream 5.3 MB/s | 19 MB 00:03 2024-03-30T13:21:21.977 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - CRB 7.3 MB/s | 6.1 MB 00:00 2024-03-30T13:21:24.619 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 16 kB 00:01 2024-03-30T13:21:26.123 INFO:teuthology.orchestra.run.smithi043.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-30T13:21:31.287 INFO:teuthology.orchestra.run.smithi043.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-30T13:21:33.611 INFO:teuthology.orchestra.run.smithi043.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-30T13:21:33.611 INFO:teuthology.orchestra.run.smithi043.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-30T13:21:33.697 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:21:33.706 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:21:33.707 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-03-30T13:21:33.707 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:21:33.707 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-03-30T13:21:33.707 INFO:teuthology.orchestra.run.smithi043.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-30T13:21:33.707 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-03-30T13:21:33.707 INFO:teuthology.orchestra.run.smithi043.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-30T13:21:33.707 INFO:teuthology.orchestra.run.smithi043.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-30T13:21:33.707 INFO:teuthology.orchestra.run.smithi043.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-30T13:21:33.707 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:21:33.707 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-03-30T13:21:33.707 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:21:33.707 INFO:teuthology.orchestra.run.smithi043.stdout:Install 4 Packages 2024-03-30T13:21:33.707 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:21:33.707 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 76 M 2024-03-30T13:21:33.708 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 123 M 2024-03-30T13:21:33.708 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-03-30T13:21:34.072 INFO:teuthology.orchestra.run.smithi043.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 527 kB/s | 192 kB 00:00 2024-03-30T13:21:35.205 INFO:teuthology.orchestra.run.smithi043.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 10 MB/s | 15 MB 00:01 2024-03-30T13:21:35.581 INFO:teuthology.orchestra.run.smithi043.stdout:(3/4): kernel-modules-core-5.14.0-434.el9.x86_6 18 MB/s | 28 MB 00:01 2024-03-30T13:21:36.006 INFO:teuthology.orchestra.run.smithi043.stdout:(4/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 15 MB/s | 34 MB 00:02 2024-03-30T13:21:36.009 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:21:36.009 INFO:teuthology.orchestra.run.smithi043.stdout:Total 33 MB/s | 76 MB 00:02 2024-03-30T13:21:36.009 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-03-30T13:21:36.456 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-03-30T13:21:36.456 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-03-30T13:21:37.338 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-03-30T13:21:37.339 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-03-30T13:21:39.758 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-03-30T13:21:40.372 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-30T13:21:40.465 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T13:21:43.362 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T13:21:43.511 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T13:21:48.581 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T13:21:48.618 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:21:53.726 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:22:24.719 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:22:24.719 INFO:teuthology.orchestra.run.smithi043.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-30T13:22:24.720 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:22:24.764 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:22:25.150 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:22:25.150 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-30T13:22:25.151 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T13:22:25.151 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T13:22:25.582 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:22:25.582 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:22:25.582 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-03-30T13:22:25.582 INFO:teuthology.orchestra.run.smithi043.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-30T13:22:25.582 INFO:teuthology.orchestra.run.smithi043.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-30T13:22:25.582 INFO:teuthology.orchestra.run.smithi043.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-30T13:22:25.583 INFO:teuthology.orchestra.run.smithi043.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-30T13:22:25.583 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:22:25.583 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:22:25.897 DEBUG:teuthology.orchestra.run.smithi043:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T13:22:26.003 INFO:teuthology.orchestra.run.smithi043.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T13:22:26.003 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T13:22:26.003 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-30T13:22:26.004 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-30T13:22:26.004 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-30T13:22:26.004 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-30T13:22:26.004 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi043.front.sepia.ceph.com, path=None, version=distro) 2024-03-30T13:22:26.004 DEBUG:teuthology.orchestra.run.smithi043:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T13:22:26.087 INFO:teuthology.orchestra.run.smithi043.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T13:22:26.087 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T13:22:26.087 DEBUG:teuthology.orchestra.run.smithi043:> sudo rpm -qi grub2-tools 2024-03-30T13:22:26.124 INFO:teuthology.orchestra.run.smithi043.stdout:Name : grub2-tools 2024-03-30T13:22:26.125 INFO:teuthology.orchestra.run.smithi043.stdout:Epoch : 1 2024-03-30T13:22:26.125 INFO:teuthology.orchestra.run.smithi043.stdout:Version : 2.06 2024-03-30T13:22:26.125 INFO:teuthology.orchestra.run.smithi043.stdout:Release : 68.el9 2024-03-30T13:22:26.125 INFO:teuthology.orchestra.run.smithi043.stdout:Architecture: x86_64 2024-03-30T13:22:26.125 INFO:teuthology.orchestra.run.smithi043.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-30T13:22:26.125 INFO:teuthology.orchestra.run.smithi043.stdout:Group : Unspecified 2024-03-30T13:22:26.125 INFO:teuthology.orchestra.run.smithi043.stdout:Size : 8264254 2024-03-30T13:22:26.125 INFO:teuthology.orchestra.run.smithi043.stdout:License : GPLv3+ 2024-03-30T13:22:26.125 INFO:teuthology.orchestra.run.smithi043.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-30T13:22:26.125 INFO:teuthology.orchestra.run.smithi043.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-30T13:22:26.125 INFO:teuthology.orchestra.run.smithi043.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-30T13:22:26.125 INFO:teuthology.orchestra.run.smithi043.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-30T13:22:26.126 INFO:teuthology.orchestra.run.smithi043.stdout:Packager : builder@centos.org 2024-03-30T13:22:26.126 INFO:teuthology.orchestra.run.smithi043.stdout:Vendor : CentOS 2024-03-30T13:22:26.126 INFO:teuthology.orchestra.run.smithi043.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-30T13:22:26.126 INFO:teuthology.orchestra.run.smithi043.stdout:Summary : Support tools for GRUB. 2024-03-30T13:22:26.126 INFO:teuthology.orchestra.run.smithi043.stdout:Description : 2024-03-30T13:22:26.126 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:22:26.126 INFO:teuthology.orchestra.run.smithi043.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-30T13:22:26.126 INFO:teuthology.orchestra.run.smithi043.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-30T13:22:26.126 INFO:teuthology.orchestra.run.smithi043.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-30T13:22:26.126 INFO:teuthology.orchestra.run.smithi043.stdout:hardware devices. 2024-03-30T13:22:26.126 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:22:26.126 INFO:teuthology.orchestra.run.smithi043.stdout:This subpackage provides tools for support of all platforms. 2024-03-30T13:22:26.127 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-30T13:22:26.127 INFO:teuthology.task.kernel:Updating grub on smithi043 to boot 5.14.0-434.el9.x86_64 2024-03-30T13:22:26.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-30T13:22:26.743 INFO:teuthology.orchestra.run.smithi043.stderr:Generating grub configuration file ... 2024-03-30T13:22:28.026 INFO:teuthology.orchestra.run.smithi043.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-30T13:22:28.063 INFO:teuthology.orchestra.run.smithi043.stderr:done 2024-03-30T13:22:28.066 DEBUG:teuthology.orchestra.run.smithi043:> mktemp 2024-03-30T13:22:28.081 INFO:teuthology.orchestra.run.smithi043.stdout:/tmp/tmp.NocYP8nZa9 2024-03-30T13:22:28.082 DEBUG:teuthology.orchestra.run.smithi043:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.NocYP8nZa9 2024-03-30T13:22:28.155 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0666 /tmp/tmp.NocYP8nZa9 2024-03-30T13:22:28.305 DEBUG:teuthology.orchestra.remote:smithi043:/tmp/tmp.NocYP8nZa9 is 6KB 2024-03-30T13:22:28.354 DEBUG:teuthology.orchestra.run.smithi043:> rm -fr /tmp/tmp.NocYP8nZa9 2024-03-30T13:22:28.370 DEBUG:teuthology.orchestra.run.smithi043:> sudo /bin/ls /boot/loader/entries || true 2024-03-30T13:22:28.439 INFO:teuthology.orchestra.run.smithi043.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-30T13:22:28.439 INFO:teuthology.orchestra.run.smithi043.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-30T13:22:28.439 INFO:teuthology.orchestra.run.smithi043.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-30T13:22:28.439 INFO:teuthology.orchestra.run.smithi043.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-30T13:22:28.440 DEBUG:teuthology.orchestra.run.smithi043:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-30T13:22:28.593 DEBUG:teuthology.orchestra.run.smithi043:> sudo shutdown -r now 2024-03-30T13:22:58.596 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-30T13:22:58.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-03-30T13:22:58.597 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:23:03.896 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - BaseOS 74 kB/s | 8.0 MB 01:51 2024-03-30T13:23:11.046 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - AppStream 3.1 MB/s | 19 MB 00:06 2024-03-30T13:23:17.150 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.43 2024-03-30T13:23:18.129 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - CRB 1.7 MB/s | 6.1 MB 00:03 2024-03-30T13:23:19.829 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-03-30T13:23:21.806 INFO:teuthology.orchestra.run.smithi113.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-30T13:23:26.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-03-30T13:23:26.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:23:26.714 INFO:teuthology.orchestra.run.smithi113.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-30T13:23:28.970 INFO:teuthology.orchestra.run.smithi113.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-30T13:23:28.970 INFO:teuthology.orchestra.run.smithi113.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-30T13:23:29.066 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:23:29.071 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:23:29.071 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-03-30T13:23:29.071 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:23:29.071 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-30T13:23:29.071 INFO:teuthology.orchestra.run.smithi113.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-30T13:23:29.071 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-03-30T13:23:29.071 INFO:teuthology.orchestra.run.smithi113.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-30T13:23:29.071 INFO:teuthology.orchestra.run.smithi113.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-30T13:23:29.071 INFO:teuthology.orchestra.run.smithi113.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-30T13:23:29.071 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:23:29.071 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-30T13:23:29.071 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:23:29.072 INFO:teuthology.orchestra.run.smithi113.stdout:Install 4 Packages 2024-03-30T13:23:29.072 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:23:29.072 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 76 M 2024-03-30T13:23:29.072 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 123 M 2024-03-30T13:23:29.072 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-30T13:23:29.298 INFO:teuthology.orchestra.run.smithi113.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 850 kB/s | 192 kB 00:00 2024-03-30T13:23:30.073 INFO:teuthology.orchestra.run.smithi113.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 15 MB/s | 15 MB 00:01 2024-03-30T13:23:30.940 INFO:teuthology.orchestra.run.smithi113.stdout:(3/4): kernel-modules-core-5.14.0-434.el9.x86_6 17 MB/s | 28 MB 00:01 2024-03-30T13:23:31.240 INFO:teuthology.orchestra.run.smithi113.stdout:(4/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 15 MB/s | 34 MB 00:02 2024-03-30T13:23:31.243 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:23:31.243 INFO:teuthology.orchestra.run.smithi113.stdout:Total 35 MB/s | 76 MB 00:02 2024-03-30T13:23:31.244 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-30T13:23:31.627 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-30T13:23:31.628 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-30T13:23:32.454 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-30T13:23:32.454 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-30T13:23:34.739 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-30T13:23:35.403 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-30T13:23:35.491 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T13:23:38.246 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T13:23:38.385 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T13:23:43.100 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T13:23:43.142 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:23:47.878 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:24:17.473 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:24:17.473 INFO:teuthology.orchestra.run.smithi113.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-30T13:24:17.473 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:24:17.516 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:24:17.917 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:24:17.918 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-30T13:24:17.918 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T13:24:17.918 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T13:24:18.325 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:24:18.325 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:24:18.326 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-30T13:24:18.326 INFO:teuthology.orchestra.run.smithi113.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-30T13:24:18.326 INFO:teuthology.orchestra.run.smithi113.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-30T13:24:18.326 INFO:teuthology.orchestra.run.smithi113.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-30T13:24:18.326 INFO:teuthology.orchestra.run.smithi113.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-30T13:24:18.326 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:24:18.326 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:24:18.660 DEBUG:teuthology.orchestra.run.smithi113:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T13:24:18.765 INFO:teuthology.orchestra.run.smithi113.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T13:24:18.765 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T13:24:18.765 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-30T13:24:18.765 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-30T13:24:18.766 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-30T13:24:18.766 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-30T13:24:18.766 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi113.front.sepia.ceph.com, path=None, version=distro) 2024-03-30T13:24:18.766 DEBUG:teuthology.orchestra.run.smithi113:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T13:24:18.846 INFO:teuthology.orchestra.run.smithi113.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T13:24:18.846 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T13:24:18.846 DEBUG:teuthology.orchestra.run.smithi113:> sudo rpm -qi grub2-tools 2024-03-30T13:24:18.883 INFO:teuthology.orchestra.run.smithi113.stdout:Name : grub2-tools 2024-03-30T13:24:18.883 INFO:teuthology.orchestra.run.smithi113.stdout:Epoch : 1 2024-03-30T13:24:18.883 INFO:teuthology.orchestra.run.smithi113.stdout:Version : 2.06 2024-03-30T13:24:18.883 INFO:teuthology.orchestra.run.smithi113.stdout:Release : 68.el9 2024-03-30T13:24:18.883 INFO:teuthology.orchestra.run.smithi113.stdout:Architecture: x86_64 2024-03-30T13:24:18.883 INFO:teuthology.orchestra.run.smithi113.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-30T13:24:18.883 INFO:teuthology.orchestra.run.smithi113.stdout:Group : Unspecified 2024-03-30T13:24:18.884 INFO:teuthology.orchestra.run.smithi113.stdout:Size : 8264254 2024-03-30T13:24:18.884 INFO:teuthology.orchestra.run.smithi113.stdout:License : GPLv3+ 2024-03-30T13:24:18.884 INFO:teuthology.orchestra.run.smithi113.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-30T13:24:18.884 INFO:teuthology.orchestra.run.smithi113.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-30T13:24:18.884 INFO:teuthology.orchestra.run.smithi113.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-30T13:24:18.884 INFO:teuthology.orchestra.run.smithi113.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-30T13:24:18.884 INFO:teuthology.orchestra.run.smithi113.stdout:Packager : builder@centos.org 2024-03-30T13:24:18.884 INFO:teuthology.orchestra.run.smithi113.stdout:Vendor : CentOS 2024-03-30T13:24:18.884 INFO:teuthology.orchestra.run.smithi113.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-30T13:24:18.884 INFO:teuthology.orchestra.run.smithi113.stdout:Summary : Support tools for GRUB. 2024-03-30T13:24:18.884 INFO:teuthology.orchestra.run.smithi113.stdout:Description : 2024-03-30T13:24:18.885 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:24:18.885 INFO:teuthology.orchestra.run.smithi113.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-30T13:24:18.885 INFO:teuthology.orchestra.run.smithi113.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-30T13:24:18.885 INFO:teuthology.orchestra.run.smithi113.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-30T13:24:18.885 INFO:teuthology.orchestra.run.smithi113.stdout:hardware devices. 2024-03-30T13:24:18.885 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:24:18.885 INFO:teuthology.orchestra.run.smithi113.stdout:This subpackage provides tools for support of all platforms. 2024-03-30T13:24:18.886 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-30T13:24:18.886 INFO:teuthology.task.kernel:Updating grub on smithi113 to boot 5.14.0-434.el9.x86_64 2024-03-30T13:24:18.886 DEBUG:teuthology.orchestra.run.smithi113:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-30T13:24:19.472 INFO:teuthology.orchestra.run.smithi113.stderr:Generating grub configuration file ... 2024-03-30T13:24:20.788 INFO:teuthology.orchestra.run.smithi113.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-30T13:24:20.828 INFO:teuthology.orchestra.run.smithi113.stderr:done 2024-03-30T13:24:20.830 DEBUG:teuthology.orchestra.run.smithi113:> mktemp 2024-03-30T13:24:20.845 INFO:teuthology.orchestra.run.smithi113.stdout:/tmp/tmp.G7cQbsWb1S 2024-03-30T13:24:20.845 DEBUG:teuthology.orchestra.run.smithi113:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.G7cQbsWb1S 2024-03-30T13:24:20.913 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 0666 /tmp/tmp.G7cQbsWb1S 2024-03-30T13:24:21.045 DEBUG:teuthology.orchestra.remote:smithi113:/tmp/tmp.G7cQbsWb1S is 6KB 2024-03-30T13:24:21.094 DEBUG:teuthology.orchestra.run.smithi113:> rm -fr /tmp/tmp.G7cQbsWb1S 2024-03-30T13:24:21.108 DEBUG:teuthology.orchestra.run.smithi113:> sudo /bin/ls /boot/loader/entries || true 2024-03-30T13:24:21.178 INFO:teuthology.orchestra.run.smithi113.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-30T13:24:21.178 INFO:teuthology.orchestra.run.smithi113.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-30T13:24:21.179 INFO:teuthology.orchestra.run.smithi113.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-30T13:24:21.179 INFO:teuthology.orchestra.run.smithi113.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-30T13:24:21.180 DEBUG:teuthology.orchestra.run.smithi113:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-30T13:24:21.339 DEBUG:teuthology.orchestra.run.smithi113:> sudo shutdown -r now 2024-03-30T13:24:26.158 DEBUG:teuthology.orchestra.remote:timed out 2024-03-30T13:24:38.170 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-03-30T13:24:38.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:24:38.474 DEBUG:teuthology.orchestra.run.smithi043:> true 2024-03-30T13:24:38.837 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-03-30T13:24:38.837 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-30T13:24:38.837 DEBUG:teuthology.orchestra.run.smithi043:> uname -r 2024-03-30T13:24:38.853 INFO:teuthology.orchestra.run.smithi043.stdout:5.14.0-434.el9.x86_64 2024-03-30T13:24:38.854 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-30T13:24:38.854 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-30T13:24:38.854 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-30T13:24:39.854 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-30T13:24:39.855 DEBUG:teuthology.orchestra.run.smithi043:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-30T13:24:39.937 INFO:teuthology.orchestra.run.smithi043.stdout:ttyS1 2024-03-30T13:24:39.971 DEBUG:teuthology.parallel:result is None 2024-03-30T13:24:51.343 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-30T13:24:51.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-03-30T13:24:51.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:25:09.794 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-03-30T13:25:18.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-03-30T13:25:18.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:25:21.854 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-03-30T13:25:33.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-03-30T13:25:33.856 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:25:36.930 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-03-30T13:25:51.931 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-03-30T13:25:51.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:26:23.844 DEBUG:teuthology.orchestra.run.smithi113:> true 2024-03-30T13:26:24.200 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-03-30T13:26:24.200 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-30T13:26:24.201 DEBUG:teuthology.orchestra.run.smithi113:> uname -r 2024-03-30T13:26:24.215 INFO:teuthology.orchestra.run.smithi113.stdout:5.14.0-434.el9.x86_64 2024-03-30T13:26:24.215 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-30T13:26:24.215 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-30T13:26:24.215 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-30T13:26:25.216 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-30T13:26:25.216 DEBUG:teuthology.orchestra.run.smithi113:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-30T13:26:25.289 INFO:teuthology.orchestra.run.smithi113.stdout:ttyS1 2024-03-30T13:26:25.318 DEBUG:teuthology.parallel:result is None 2024-03-30T13:26:25.318 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-30T13:26:25.325 INFO:teuthology.task.internal:Creating test directory... 2024-03-30T13:26:25.325 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T13:26:25.328 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T13:26:25.344 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-30T13:26:25.361 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-30T13:26:25.395 INFO:teuthology.task.internal:Creating archive directory... 2024-03-30T13:26:25.395 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T13:26:25.398 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T13:26:25.433 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-30T13:26:25.440 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-30T13:26:25.440 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T13:26:25.467 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T13:26:25.503 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T13:26:25.506 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T13:26:25.518 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T13:26:25.520 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T13:26:25.522 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-30T13:26:25.529 INFO:teuthology.task.internal:Configuring sudo... 2024-03-30T13:26:25.529 DEBUG:teuthology.orchestra.run.smithi043:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T13:26:25.562 DEBUG:teuthology.orchestra.run.smithi113:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T13:26:25.592 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-30T13:26:25.601 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-30T13:26:25.602 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T13:26:25.632 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T13:26:25.650 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T13:26:25.731 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T13:26:25.783 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T13:26:25.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T13:26:25.909 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-30T13:26:25.909 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T13:26:25.977 DEBUG:teuthology.orchestra.run.smithi113:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T13:26:26.020 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T13:26:26.076 DEBUG:teuthology.orchestra.run.smithi113:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T13:26:26.158 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T13:26:26.206 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-30T13:26:26.206 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T13:26:26.275 DEBUG:teuthology.orchestra.run.smithi043:> sudo service rsyslog restart 2024-03-30T13:26:26.278 DEBUG:teuthology.orchestra.run.smithi113:> sudo service rsyslog restart 2024-03-30T13:26:26.335 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T13:26:26.368 INFO:teuthology.orchestra.run.smithi113.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T13:26:26.765 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-30T13:26:26.774 INFO:teuthology.task.internal:Starting timer... 2024-03-30T13:26:26.774 INFO:teuthology.run_tasks:Running task pcp... 2024-03-30T13:26:26.823 INFO:teuthology.run_tasks:Running task selinux... 2024-03-30T13:26:26.855 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-30T13:26:26.856 DEBUG:teuthology.orchestra.run.smithi043:> sudo service auditd rotate 2024-03-30T13:26:26.912 INFO:teuthology.orchestra.run.smithi043.stdout:Rotating logs: 2024-03-30T13:26:26.914 DEBUG:teuthology.orchestra.run.smithi113:> sudo service auditd rotate 2024-03-30T13:26:26.988 INFO:teuthology.orchestra.run.smithi113.stdout:Rotating logs: 2024-03-30T13:26:26.990 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-30T13:26:26.990 DEBUG:teuthology.orchestra.run.smithi043:> /usr/sbin/getenforce 2024-03-30T13:26:27.008 INFO:teuthology.orchestra.run.smithi043.stdout:Permissive 2024-03-30T13:26:27.008 DEBUG:teuthology.orchestra.run.smithi113:> /usr/sbin/getenforce 2024-03-30T13:26:27.031 INFO:teuthology.orchestra.run.smithi113.stdout:Permissive 2024-03-30T13:26:27.031 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi043.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi113.front.sepia.ceph.com': 'permissive'} 2024-03-30T13:26:27.031 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-30T13:26:27.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:26:27.077 DEBUG:teuthology.orchestra.run.smithi113:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-30T13:26:27.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:26:27.106 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-30T13:26:27.106 DEBUG:teuthology.orchestra.run.smithi043:> sudo /usr/sbin/setenforce permissive 2024-03-30T13:26:27.152 DEBUG:teuthology.orchestra.run.smithi113:> sudo /usr/sbin/setenforce permissive 2024-03-30T13:26:27.182 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-30T13:26:27.192 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-30T13:26:27.209 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-30T13:26:27.210 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi043.front.sepia.ceph.com,smithi113.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-30T13:30:45.424 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi043.front.sepia.ceph.com'), Remote(name='ubuntu@smithi113.front.sepia.ceph.com')] 2024-03-30T13:30:45.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-03-30T13:30:45.426 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:30:45.520 DEBUG:teuthology.orchestra.run.smithi043:> true 2024-03-30T13:30:45.601 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-03-30T13:30:45.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-03-30T13:30:45.602 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:30:45.689 DEBUG:teuthology.orchestra.run.smithi113:> true 2024-03-30T13:30:45.769 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-03-30T13:30:45.769 INFO:teuthology.run_tasks:Running task clock... 2024-03-30T13:30:45.781 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-30T13:30:45.781 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T13:30:45.782 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T13:30:45.784 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T13:30:45.784 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T13:30:45.815 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-30T13:30:45.833 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-30T13:30:45.840 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-30T13:30:45.856 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-30T13:30:45.863 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: ntpd: command not found 2024-03-30T13:30:45.877 INFO:teuthology.orchestra.run.smithi043.stdout:506 Cannot talk to daemon 2024-03-30T13:30:45.886 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: ntpd: command not found 2024-03-30T13:30:45.893 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-30T13:30:45.899 INFO:teuthology.orchestra.run.smithi113.stdout:506 Cannot talk to daemon 2024-03-30T13:30:45.910 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-30T13:30:45.915 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-30T13:30:45.932 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-30T13:30:45.960 INFO:teuthology.orchestra.run.smithi043.stderr:bash: line 1: ntpq: command not found 2024-03-30T13:30:45.964 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T13:30:45.964 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2024-03-30T13:30:45.964 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T13:30:45.964 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T13:30:45.964 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T13:30:45.964 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T13:30:45.983 INFO:teuthology.orchestra.run.smithi113.stderr:bash: line 1: ntpq: command not found 2024-03-30T13:30:45.987 INFO:teuthology.orchestra.run.smithi113.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T13:30:45.987 INFO:teuthology.orchestra.run.smithi113.stdout:=============================================================================== 2024-03-30T13:30:45.987 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T13:30:45.987 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T13:30:45.987 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T13:30:45.987 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T13:30:45.988 INFO:teuthology.run_tasks:Running task pexec... 2024-03-30T13:30:45.997 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-30T13:30:45.998 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi043.front.sepia.ceph.com 2024-03-30T13:30:45.998 DEBUG:teuthology.orchestra.run.smithi043:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-30T13:30:45.998 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi113.front.sepia.ceph.com 2024-03-30T13:30:45.998 DEBUG:teuthology.orchestra.run.smithi113:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-30T13:30:46.611 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:49 ago on Sat 30 Mar 2024 01:29:57 PM UTC. 2024-03-30T13:30:46.612 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:00:52 ago on Sat 30 Mar 2024 01:29:54 PM UTC. 2024-03-30T13:30:46.761 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:30:46.762 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:30:46.762 INFO:teuthology.orchestra.run.smithi113.stdout: Package Architecture Version Repository Size 2024-03-30T13:30:46.762 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:30:46.762 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-30T13:30:46.762 INFO:teuthology.orchestra.run.smithi113.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-30T13:30:46.762 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:30:46.762 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-30T13:30:46.762 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:30:46.763 INFO:teuthology.orchestra.run.smithi113.stdout:Install 1 Package 2024-03-30T13:30:46.763 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:30:46.763 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 3.1 M 2024-03-30T13:30:46.763 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 9.8 M 2024-03-30T13:30:46.763 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-30T13:30:46.764 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:30:46.765 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:30:46.765 INFO:teuthology.orchestra.run.smithi043.stdout: Package Architecture Version Repository Size 2024-03-30T13:30:46.765 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:30:46.765 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-03-30T13:30:46.765 INFO:teuthology.orchestra.run.smithi043.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-30T13:30:46.765 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:30:46.765 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-03-30T13:30:46.765 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:30:46.765 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-03-30T13:30:46.765 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:30:46.766 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 3.1 M 2024-03-30T13:30:46.766 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 9.8 M 2024-03-30T13:30:46.766 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-03-30T13:30:47.152 INFO:teuthology.orchestra.run.smithi043.stdout:runc-1.1.12-2.el9.x86_64.rpm 12 MB/s | 3.1 MB 00:00 2024-03-30T13:30:47.153 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:30:47.153 INFO:teuthology.orchestra.run.smithi043.stdout:Total 8.1 MB/s | 3.1 MB 00:00 2024-03-30T13:30:47.190 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-03-30T13:30:47.196 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-03-30T13:30:47.196 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-03-30T13:30:47.278 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-03-30T13:30:47.279 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-03-30T13:30:47.496 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-03-30T13:30:47.582 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-30T13:30:47.831 INFO:teuthology.orchestra.run.smithi113.stdout:runc-1.1.12-2.el9.x86_64.rpm 3.1 MB/s | 3.1 MB 00:01 2024-03-30T13:30:47.831 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:30:47.831 INFO:teuthology.orchestra.run.smithi113.stdout:Total 2.9 MB/s | 3.1 MB 00:01 2024-03-30T13:30:47.867 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-30T13:30:47.873 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-30T13:30:47.873 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-30T13:30:47.954 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-30T13:30:47.954 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-30T13:30:48.157 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-30T13:30:48.160 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-30T13:30:48.258 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-30T13:30:48.453 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-30T13:30:48.453 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:30:48.453 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-03-30T13:30:48.453 INFO:teuthology.orchestra.run.smithi043.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-30T13:30:48.453 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:30:48.453 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:30:48.708 DEBUG:teuthology.parallel:result is None 2024-03-30T13:30:48.752 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-30T13:30:49.073 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-30T13:30:49.073 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:30:49.073 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-30T13:30:49.073 INFO:teuthology.orchestra.run.smithi113.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-30T13:30:49.073 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:30:49.073 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:30:49.319 DEBUG:teuthology.parallel:result is None 2024-03-30T13:30:49.319 INFO:teuthology.run_tasks:Running task install... 2024-03-30T13:30:49.328 DEBUG:teuthology.task.install:project ceph 2024-03-30T13:30:49.328 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb'}} 2024-03-30T13:30:49.328 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb'} 2024-03-30T13:30:49.328 INFO:teuthology.task.install:Using flavor: default 2024-03-30T13:30:49.335 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-30T13:30:49.335 INFO:teuthology.task.install:extra packages: [] 2024-03-30T13:30:49.335 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb', 'tag': None, 'wait_for_package': False} 2024-03-30T13:30:49.336 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T13:30:49.338 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb', 'tag': None, 'wait_for_package': False} 2024-03-30T13:30:49.338 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T13:30:49.481 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/ 2024-03-30T13:30:49.481 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1967.g4e50bbcd 2024-03-30T13:30:49.596 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-30T13:30:49.596 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-30T13:30:49.596 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-30T13:30:49.623 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-30T13:30:49.624 DEBUG:teuthology.orchestra.run.smithi113:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-30T13:30:49.692 DEBUG:teuthology.orchestra.run.smithi113:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-30T13:30:49.716 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/ 2024-03-30T13:30:49.716 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1967.g4e50bbcd 2024-03-30T13:30:49.774 DEBUG:teuthology.orchestra.run.smithi113:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-30T13:30:49.832 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-30T13:30:49.833 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-30T13:30:49.833 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-30T13:30:49.851 INFO:teuthology.orchestra.run.smithi113.stdout:check_obsoletes = 1 2024-03-30T13:30:49.853 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum clean all 2024-03-30T13:30:49.866 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-30T13:30:49.866 DEBUG:teuthology.orchestra.run.smithi043:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-30T13:30:49.937 DEBUG:teuthology.orchestra.run.smithi043:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-30T13:30:50.021 DEBUG:teuthology.orchestra.run.smithi043:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-30T13:30:50.090 INFO:teuthology.orchestra.run.smithi043.stdout:check_obsoletes = 1 2024-03-30T13:30:50.092 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum clean all 2024-03-30T13:30:50.093 INFO:teuthology.orchestra.run.smithi113.stdout:74 files removed 2024-03-30T13:30:50.120 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-radosgw 2024-03-30T13:30:50.350 INFO:teuthology.orchestra.run.smithi043.stdout:74 files removed 2024-03-30T13:30:50.378 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-radosgw 2024-03-30T13:30:50.946 INFO:teuthology.orchestra.run.smithi113.stdout:ceph packages for x86_64 159 kB/s | 83 kB 00:00 2024-03-30T13:30:51.259 INFO:teuthology.orchestra.run.smithi043.stdout:ceph packages for x86_64 144 kB/s | 83 kB 00:00 2024-03-30T13:30:51.413 INFO:teuthology.orchestra.run.smithi113.stdout:ceph noarch packages 27 kB/s | 11 kB 00:00 2024-03-30T13:30:51.634 INFO:teuthology.orchestra.run.smithi043.stdout:ceph noarch packages 35 kB/s | 11 kB 00:00 2024-03-30T13:30:51.897 INFO:teuthology.orchestra.run.smithi113.stdout:ceph source packages 4.3 kB/s | 1.9 kB 00:00 2024-03-30T13:30:52.134 INFO:teuthology.orchestra.run.smithi043.stdout:ceph source packages 4.1 kB/s | 1.9 kB 00:00 2024-03-30T13:30:52.424 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-30T13:30:52.670 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-30T13:30:54.058 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-30T13:30:54.279 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS-9 - BaseOS 4.3 MB/s | 2.0 MB 00:00 2024-03-30T13:30:55.994 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 8.0 MB 00:01 2024-03-30T13:30:58.962 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-03-30T13:31:03.021 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-30T13:31:03.555 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - BaseOS 897 kB/s | 8.0 MB 00:09 2024-03-30T13:31:04.464 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 16 kB 00:00 2024-03-30T13:31:06.490 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-03-30T13:31:10.690 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - CRB 8.9 MB/s | 6.1 MB 00:00 2024-03-30T13:31:12.583 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 16 kB 00:00 2024-03-30T13:31:13.854 INFO:teuthology.orchestra.run.smithi113.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-30T13:31:19.067 INFO:teuthology.orchestra.run.smithi113.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-03-30T13:31:21.661 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:31:21.664 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:31:21.664 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-03-30T13:31:21.664 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:31:21.664 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-30T13:31:21.664 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-radosgw x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 9.8 M 2024-03-30T13:31:21.664 INFO:teuthology.orchestra.run.smithi113.stdout:Upgrading: 2024-03-30T13:31:21.664 INFO:teuthology.orchestra.run.smithi113.stdout: librados2 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 3.4 M 2024-03-30T13:31:21.664 INFO:teuthology.orchestra.run.smithi113.stdout: librbd1 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 3.1 M 2024-03-30T13:31:21.664 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-03-30T13:31:21.664 INFO:teuthology.orchestra.run.smithi113.stdout: boost-program-options 2024-03-30T13:31:21.665 INFO:teuthology.orchestra.run.smithi113.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-30T13:31:21.665 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-base x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 5.3 M 2024-03-30T13:31:21.665 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-common x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 21 M 2024-03-30T13:31:21.665 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-selinux x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 25 k 2024-03-30T13:31:21.665 INFO:teuthology.orchestra.run.smithi113.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-30T13:31:21.665 INFO:teuthology.orchestra.run.smithi113.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-30T13:31:21.665 INFO:teuthology.orchestra.run.smithi113.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-30T13:31:21.665 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs2 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 745 k 2024-03-30T13:31:21.665 INFO:teuthology.orchestra.run.smithi113.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-30T13:31:21.665 INFO:teuthology.orchestra.run.smithi113.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-30T13:31:21.665 INFO:teuthology.orchestra.run.smithi113.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-30T13:31:21.665 INFO:teuthology.orchestra.run.smithi113.stdout: libradosstriper1 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 501 k 2024-03-30T13:31:21.665 INFO:teuthology.orchestra.run.smithi113.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-30T13:31:21.666 INFO:teuthology.orchestra.run.smithi113.stdout: librgw2 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 5.1 M 2024-03-30T13:31:21.666 INFO:teuthology.orchestra.run.smithi113.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-30T13:31:21.666 INFO:teuthology.orchestra.run.smithi113.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-30T13:31:21.666 INFO:teuthology.orchestra.run.smithi113.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-30T13:31:21.666 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-argparse 2024-03-30T13:31:21.666 INFO:teuthology.orchestra.run.smithi113.stdout: x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 45 k 2024-03-30T13:31:21.666 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-common x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 131 k 2024-03-30T13:31:21.666 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cephfs x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 161 k 2024-03-30T13:31:21.666 INFO:teuthology.orchestra.run.smithi113.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-30T13:31:21.666 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rados x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 321 k 2024-03-30T13:31:21.666 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rbd x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 298 k 2024-03-30T13:31:21.666 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rgw x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 99 k 2024-03-30T13:31:21.666 INFO:teuthology.orchestra.run.smithi113.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-30T13:31:21.666 INFO:teuthology.orchestra.run.smithi113.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-30T13:31:21.667 INFO:teuthology.orchestra.run.smithi113.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-30T13:31:21.667 INFO:teuthology.orchestra.run.smithi113.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-30T13:31:21.667 INFO:teuthology.orchestra.run.smithi113.stdout:Installing weak dependencies: 2024-03-30T13:31:21.667 INFO:teuthology.orchestra.run.smithi113.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-30T13:31:21.667 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:31:21.667 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-30T13:31:21.667 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:31:21.667 INFO:teuthology.orchestra.run.smithi113.stdout:Install 30 Packages 2024-03-30T13:31:21.667 INFO:teuthology.orchestra.run.smithi113.stdout:Upgrade 2 Packages 2024-03-30T13:31:21.667 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:31:21.668 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 59 M 2024-03-30T13:31:21.668 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-30T13:31:23.360 INFO:teuthology.orchestra.run.smithi113.stdout:(1/32): ceph-common-19.0.0-1967.g4e50bbcd.el9.x 15 MB/s | 21 MB 00:01 2024-03-30T13:31:23.394 INFO:teuthology.orchestra.run.smithi113.stdout:(2/32): ceph-selinux-19.0.0-1967.g4e50bbcd.el9. 762 kB/s | 25 kB 00:00 2024-03-30T13:31:23.469 INFO:teuthology.orchestra.run.smithi113.stdout:(3/32): libcephfs2-19.0.0-1967.g4e50bbcd.el9.x8 9.7 MB/s | 745 kB 00:00 2024-03-30T13:31:23.528 INFO:teuthology.orchestra.run.smithi113.stdout:(4/32): libradosstriper1-19.0.0-1967.g4e50bbcd. 8.4 MB/s | 501 kB 00:00 2024-03-30T13:31:23.803 INFO:teuthology.orchestra.run.smithi113.stdout:(5/32): librgw2-19.0.0-1967.g4e50bbcd.el9.x86_6 19 MB/s | 5.1 MB 00:00 2024-03-30T13:31:23.845 INFO:teuthology.orchestra.run.smithi113.stdout:(6/32): python3-ceph-argparse-19.0.0-1967.g4e50 1.1 MB/s | 45 kB 00:00 2024-03-30T13:31:23.887 INFO:teuthology.orchestra.run.smithi113.stdout:(7/32): python3-ceph-common-19.0.0-1967.g4e50bb 3.1 MB/s | 131 kB 00:00 2024-03-30T13:31:23.937 INFO:teuthology.orchestra.run.smithi113.stdout:(8/32): python3-cephfs-19.0.0-1967.g4e50bbcd.el 3.2 MB/s | 161 kB 00:00 2024-03-30T13:31:23.988 INFO:teuthology.orchestra.run.smithi113.stdout:(9/32): python3-rados-19.0.0-1967.g4e50bbcd.el9 6.3 MB/s | 321 kB 00:00 2024-03-30T13:31:24.038 INFO:teuthology.orchestra.run.smithi113.stdout:(10/32): python3-rbd-19.0.0-1967.g4e50bbcd.el9. 5.8 MB/s | 298 kB 00:00 2024-03-30T13:31:24.080 INFO:teuthology.orchestra.run.smithi113.stdout:(11/32): python3-rgw-19.0.0-1967.g4e50bbcd.el9. 2.3 MB/s | 99 kB 00:00 2024-03-30T13:31:24.338 INFO:teuthology.orchestra.run.smithi113.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 410 kB/s | 106 kB 00:00 2024-03-30T13:31:24.455 INFO:teuthology.orchestra.run.smithi113.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-30T13:31:24.522 INFO:teuthology.orchestra.run.smithi113.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 664 kB/s | 44 kB 00:00 2024-03-30T13:31:24.664 INFO:teuthology.orchestra.run.smithi113.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.6 MB/s | 661 kB 00:00 2024-03-30T13:31:24.739 INFO:teuthology.orchestra.run.smithi113.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-30T13:31:24.798 INFO:teuthology.orchestra.run.smithi113.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 706 kB/s | 41 kB 00:00 2024-03-30T13:31:24.865 INFO:teuthology.orchestra.run.smithi113.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2024-03-30T13:31:24.916 INFO:teuthology.orchestra.run.smithi113.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 697 kB/s | 35 kB 00:00 2024-03-30T13:31:25.030 INFO:teuthology.orchestra.run.smithi113.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 405 kB/s | 46 kB 00:00 2024-03-30T13:31:25.225 INFO:teuthology.orchestra.run.smithi113.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-03-30T13:31:25.381 INFO:teuthology.orchestra.run.smithi113.stdout:(22/32): ceph-base-19.0.0-1967.g4e50bbcd.el9.x8 1.6 MB/s | 5.3 MB 00:03 2024-03-30T13:31:25.542 INFO:teuthology.orchestra.run.smithi113.stdout:(23/32): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-30T13:31:25.567 INFO:teuthology.orchestra.run.smithi113.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 140 kB/s | 26 kB 00:00 2024-03-30T13:31:25.592 INFO:teuthology.orchestra.run.smithi113.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-03-30T13:31:25.659 INFO:teuthology.orchestra.run.smithi113.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-30T13:31:25.685 INFO:teuthology.orchestra.run.smithi113.stdout:(27/32): libunwind-1.6.2-1.el9.x86_64.rpm 574 kB/s | 67 kB 00:00 2024-03-30T13:31:25.710 INFO:teuthology.orchestra.run.smithi113.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-03-30T13:31:25.844 INFO:teuthology.orchestra.run.smithi113.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-03-30T13:31:26.119 INFO:teuthology.orchestra.run.smithi113.stdout:(30/32): librbd1-19.0.0-1967.g4e50bbcd.el9.x86_ 11 MB/s | 3.1 MB 00:00 2024-03-30T13:31:27.478 INFO:teuthology.orchestra.run.smithi113.stdout:(31/32): librados2-19.0.0-1967.g4e50bbcd.el9.x8 1.9 MB/s | 3.4 MB 00:01 2024-03-30T13:31:31.019 INFO:teuthology.orchestra.run.smithi113.stdout:(32/32): ceph-radosgw-19.0.0-1967.g4e50bbcd.el9 1.1 MB/s | 9.8 MB 00:08 2024-03-30T13:31:31.022 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:31:31.022 INFO:teuthology.orchestra.run.smithi113.stdout:Total 6.3 MB/s | 59 MB 00:09 2024-03-30T13:31:31.023 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-30T13:31:31.061 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-30T13:31:31.062 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-30T13:31:31.538 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-30T13:31:31.538 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-30T13:31:32.955 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-30T13:31:33.041 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-30T13:31:33.073 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-30T13:31:33.098 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-30T13:31:33.122 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-30T13:31:33.137 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-30T13:31:33.296 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-30T13:31:33.352 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-30T13:31:33.700 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-30T13:31:33.767 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 7/34 2024-03-30T13:31:33.881 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 7/34 2024-03-30T13:31:34.004 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 8/34 2024-03-30T13:31:34.027 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 9/34 2024-03-30T13:31:34.091 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 9/34 2024-03-30T13:31:34.150 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-30T13:31:34.180 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-ceph-argparse-2:19.0.0-1967.g4e50bbcd.el9. 11/34 2024-03-30T13:31:34.270 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 12/34 2024-03-30T13:31:34.305 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_6 13/34 2024-03-30T13:31:34.363 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_6 13/34 2024-03-30T13:31:34.398 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-30T13:31:34.563 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-30T13:31:34.605 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-30T13:31:34.646 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-30T13:31:34.678 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-30T13:31:34.707 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-30T13:31:34.740 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-30T13:31:35.052 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-30T13:31:35.078 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 22/34 2024-03-30T13:31:35.168 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 22/34 2024-03-30T13:31:35.199 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 23/34 2024-03-30T13:31:35.238 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-30T13:31:35.704 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-30T13:31:35.731 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 26/34 2024-03-30T13:31:35.793 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 26/34 2024-03-30T13:31:35.861 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 27/34 2024-03-30T13:31:35.892 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x8 28/34 2024-03-30T13:31:38.149 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 29/34 2024-03-30T13:31:38.188 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 29/34 2024-03-30T13:31:38.742 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 29/34 2024-03-30T13:31:38.775 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 30/34 2024-03-30T13:31:38.874 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 30/34 2024-03-30T13:31:38.874 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-30T13:31:38.874 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-30T13:31:38.874 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:31:38.918 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 31/34 2024-03-30T13:31:48.506 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 31/34 2024-03-30T13:31:48.506 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /sys 2024-03-30T13:31:48.507 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /proc 2024-03-30T13:31:48.507 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /mnt 2024-03-30T13:31:48.507 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /var/tmp 2024-03-30T13:31:48.507 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /home 2024-03-30T13:31:48.507 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /root 2024-03-30T13:31:48.507 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /tmp 2024-03-30T13:31:48.507 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:31:49.425 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/34 2024-03-30T13:31:49.449 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/34 2024-03-30T13:31:49.449 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:31:49.449 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-30T13:31:49.449 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-30T13:31:49.449 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-30T13:31:49.449 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:31:49.451 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-30T13:31:49.535 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-30T13:31:49.535 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T13:31:50.674 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T13:31:50.674 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/34 2024-03-30T13:31:50.674 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/34 2024-03-30T13:31:50.674 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 3/34 2024-03-30T13:31:50.674 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 4/34 2024-03-30T13:31:50.674 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/34 2024-03-30T13:31:50.674 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_6 6/34 2024-03-30T13:31:50.674 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 7/34 2024-03-30T13:31:50.674 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1967.g4e50bbcd.el9. 8/34 2024-03-30T13:31:50.675 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x8 9/34 2024-03-30T13:31:50.675 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 10/34 2024-03-30T13:31:50.675 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 11/34 2024-03-30T13:31:50.675 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 12/34 2024-03-30T13:31:50.675 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 13/34 2024-03-30T13:31:50.675 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-30T13:31:50.675 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-30T13:31:50.675 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-30T13:31:50.675 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-30T13:31:50.675 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-30T13:31:50.675 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-30T13:31:50.675 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-30T13:31:50.676 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-30T13:31:50.676 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-30T13:31:50.676 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-30T13:31:50.676 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-30T13:31:50.676 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-30T13:31:50.676 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-30T13:31:50.676 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-30T13:31:50.676 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-30T13:31:50.677 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-30T13:31:50.677 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-30T13:31:50.677 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 31/34 2024-03-30T13:31:50.677 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-30T13:31:50.677 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 33/34 2024-03-30T13:31:51.862 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T13:31:51.863 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:31:51.863 INFO:teuthology.orchestra.run.smithi113.stdout:Upgraded: 2024-03-30T13:31:51.863 INFO:teuthology.orchestra.run.smithi113.stdout: librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:31:51.863 INFO:teuthology.orchestra.run.smithi113.stdout: librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:31:51.863 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-30T13:31:51.863 INFO:teuthology.orchestra.run.smithi113.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-30T13:31:51.863 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:31:51.863 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:31:51.863 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:31:51.863 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:31:51.863 INFO:teuthology.orchestra.run.smithi113.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-30T13:31:51.863 INFO:teuthology.orchestra.run.smithi113.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-30T13:31:51.863 INFO:teuthology.orchestra.run.smithi113.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-30T13:31:51.864 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:31:51.864 INFO:teuthology.orchestra.run.smithi113.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-30T13:31:51.864 INFO:teuthology.orchestra.run.smithi113.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-30T13:31:51.864 INFO:teuthology.orchestra.run.smithi113.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-30T13:31:51.864 INFO:teuthology.orchestra.run.smithi113.stdout: libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:31:51.864 INFO:teuthology.orchestra.run.smithi113.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-30T13:31:51.864 INFO:teuthology.orchestra.run.smithi113.stdout: librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:31:51.864 INFO:teuthology.orchestra.run.smithi113.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-30T13:31:51.864 INFO:teuthology.orchestra.run.smithi113.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-30T13:31:51.864 INFO:teuthology.orchestra.run.smithi113.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-30T13:31:51.864 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-argparse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:31:51.864 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:31:51.864 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:31:51.864 INFO:teuthology.orchestra.run.smithi113.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-30T13:31:51.865 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:31:51.865 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:31:51.865 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:31:51.865 INFO:teuthology.orchestra.run.smithi113.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-30T13:31:51.865 INFO:teuthology.orchestra.run.smithi113.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-30T13:31:51.865 INFO:teuthology.orchestra.run.smithi113.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-30T13:31:51.865 INFO:teuthology.orchestra.run.smithi113.stdout: re2-1:20211101-3.el9.x86_64 2024-03-30T13:31:51.865 INFO:teuthology.orchestra.run.smithi113.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-30T13:31:51.865 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:31:51.865 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:31:52.179 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-test 2024-03-30T13:31:52.736 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:00:33 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:31:52.885 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:31:52.886 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:31:52.886 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-03-30T13:31:52.886 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:31:52.886 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-30T13:31:52.886 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-test x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 47 M 2024-03-30T13:31:52.886 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-03-30T13:31:52.886 INFO:teuthology.orchestra.run.smithi113.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-30T13:31:52.886 INFO:teuthology.orchestra.run.smithi113.stdout: libcephsqlite x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 163 k 2024-03-30T13:31:52.887 INFO:teuthology.orchestra.run.smithi113.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-30T13:31:52.887 INFO:teuthology.orchestra.run.smithi113.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-30T13:31:52.887 INFO:teuthology.orchestra.run.smithi113.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-30T13:31:52.887 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:31:52.887 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-30T13:31:52.887 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:31:52.887 INFO:teuthology.orchestra.run.smithi113.stdout:Install 6 Packages 2024-03-30T13:31:52.887 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:31:52.888 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 48 M 2024-03-30T13:31:52.888 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 205 M 2024-03-30T13:31:52.888 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-30T13:31:53.164 INFO:teuthology.orchestra.run.smithi113.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-30T13:31:53.214 INFO:teuthology.orchestra.run.smithi113.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-30T13:31:53.256 INFO:teuthology.orchestra.run.smithi113.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-30T13:31:53.281 INFO:teuthology.orchestra.run.smithi113.stdout:(4/6): libcephsqlite-19.0.0-1967.g4e50bbcd.el9. 587 kB/s | 163 kB 00:00 2024-03-30T13:31:53.415 INFO:teuthology.orchestra.run.smithi113.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-30T13:31:55.657 INFO:teuthology.orchestra.run.smithi113.stdout:(6/6): ceph-test-19.0.0-1967.g4e50bbcd.el9.x86_ 18 MB/s | 47 MB 00:02 2024-03-30T13:31:55.659 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:31:55.660 INFO:teuthology.orchestra.run.smithi113.stdout:Total 17 MB/s | 48 MB 00:02 2024-03-30T13:31:55.675 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-30T13:31:55.696 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-30T13:31:55.696 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-30T13:31:56.088 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-30T13:31:56.088 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-30T13:31:56.658 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-30T13:31:56.748 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-30T13:31:56.787 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-30T13:31:56.828 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-30T13:31:56.881 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-30T13:31:56.910 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/6 2024-03-30T13:32:01.370 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/6 2024-03-30T13:32:01.442 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 6/6 2024-03-30T13:32:03.842 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 6/6 2024-03-30T13:32:03.843 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/6 2024-03-30T13:32:03.843 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/6 2024-03-30T13:32:03.843 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-30T13:32:03.843 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-30T13:32:03.843 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-30T13:32:04.272 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-30T13:32:04.272 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:04.272 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-30T13:32:04.272 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:32:04.272 INFO:teuthology.orchestra.run.smithi113.stdout: jq-1.6-16.el9.x86_64 2024-03-30T13:32:04.272 INFO:teuthology.orchestra.run.smithi113.stdout: libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:32:04.272 INFO:teuthology.orchestra.run.smithi113.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-30T13:32:04.273 INFO:teuthology.orchestra.run.smithi113.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-30T13:32:04.273 INFO:teuthology.orchestra.run.smithi113.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-30T13:32:04.273 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:04.273 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:32:04.536 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph 2024-03-30T13:32:05.119 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:00:46 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:32:05.282 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:32:05.284 INFO:teuthology.orchestra.run.smithi113.stdout:========================================================================================= 2024-03-30T13:32:05.284 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-03-30T13:32:05.284 INFO:teuthology.orchestra.run.smithi113.stdout:========================================================================================= 2024-03-30T13:32:05.284 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-30T13:32:05.284 INFO:teuthology.orchestra.run.smithi113.stdout: ceph x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 6.6 k 2024-03-30T13:32:05.284 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-03-30T13:32:05.284 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mds x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 2.4 M 2024-03-30T13:32:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 1.5 M 2024-03-30T13:32:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 248 k 2024-03-30T13:32:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mon x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 4.7 M 2024-03-30T13:32:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-osd x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 17 M 2024-03-30T13:32:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-30T13:32:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-30T13:32:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-30T13:32:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-30T13:32:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-30T13:32:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-30T13:32:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-30T13:32:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-30T13:32:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-30T13:32:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-30T13:32:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-30T13:32:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-30T13:32:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-30T13:32:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-30T13:32:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-30T13:32:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-30T13:32:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-30T13:32:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-30T13:32:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-30T13:32:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-30T13:32:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-30T13:32:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-30T13:32:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-30T13:32:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-30T13:32:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-30T13:32:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-30T13:32:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-30T13:32:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-30T13:32:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-30T13:32:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-30T13:32:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:05.287 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-30T13:32:05.287 INFO:teuthology.orchestra.run.smithi113.stdout:========================================================================================= 2024-03-30T13:32:05.287 INFO:teuthology.orchestra.run.smithi113.stdout:Install 36 Packages 2024-03-30T13:32:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:05.288 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 30 M 2024-03-30T13:32:05.288 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 107 M 2024-03-30T13:32:05.288 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-30T13:32:05.918 INFO:teuthology.orchestra.run.smithi113.stdout:(1/36): ceph-19.0.0-1967.g4e50bbcd.el9.x86_64.r 28 kB/s | 6.6 kB 00:00 2024-03-30T13:32:06.068 INFO:teuthology.orchestra.run.smithi113.stdout:(2/36): ceph-mds-19.0.0-1967.g4e50bbcd.el9.x86_ 6.1 MB/s | 2.4 MB 00:00 2024-03-30T13:32:06.176 INFO:teuthology.orchestra.run.smithi113.stdout:(3/36): ceph-mgr-19.0.0-1967.g4e50bbcd.el9.x86_ 3.0 MB/s | 1.5 MB 00:00 2024-03-30T13:32:06.277 INFO:teuthology.orchestra.run.smithi113.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1967.g4e50 2.4 MB/s | 248 kB 00:00 2024-03-30T13:32:06.394 INFO:teuthology.orchestra.run.smithi113.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-30T13:32:06.444 INFO:teuthology.orchestra.run.smithi113.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-30T13:32:06.469 INFO:teuthology.orchestra.run.smithi113.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-30T13:32:06.495 INFO:teuthology.orchestra.run.smithi113.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-30T13:32:06.520 INFO:teuthology.orchestra.run.smithi113.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-30T13:32:06.570 INFO:teuthology.orchestra.run.smithi113.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-30T13:32:06.679 INFO:teuthology.orchestra.run.smithi113.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-30T13:32:06.721 INFO:teuthology.orchestra.run.smithi113.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-30T13:32:06.763 INFO:teuthology.orchestra.run.smithi113.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-30T13:32:06.930 INFO:teuthology.orchestra.run.smithi113.stdout:(14/36): ceph-osd-19.0.0-1967.g4e50bbcd.el9.x86 20 MB/s | 17 MB 00:00 2024-03-30T13:32:06.955 INFO:teuthology.orchestra.run.smithi113.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 974 kB/s | 187 kB 00:00 2024-03-30T13:32:07.080 INFO:teuthology.orchestra.run.smithi113.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 179 kB/s | 22 kB 00:00 2024-03-30T13:32:07.106 INFO:teuthology.orchestra.run.smithi113.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 714 kB/s | 125 kB 00:00 2024-03-30T13:32:07.406 INFO:teuthology.orchestra.run.smithi113.stdout:(18/36): ceph-mon-19.0.0-1967.g4e50bbcd.el9.x86 3.2 MB/s | 4.7 MB 00:01 2024-03-30T13:32:07.456 INFO:teuthology.orchestra.run.smithi113.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 124 kB/s | 43 kB 00:00 2024-03-30T13:32:07.607 INFO:teuthology.orchestra.run.smithi113.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 287 kB/s | 151 kB 00:00 2024-03-30T13:32:07.798 INFO:teuthology.orchestra.run.smithi113.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 56 kB/s | 11 kB 00:00 2024-03-30T13:32:07.832 INFO:teuthology.orchestra.run.smithi113.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 404 kB/s | 172 kB 00:00 2024-03-30T13:32:07.966 INFO:teuthology.orchestra.run.smithi113.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 704 kB/s | 358 kB 00:00 2024-03-30T13:32:08.066 INFO:teuthology.orchestra.run.smithi113.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 66 kB/s | 18 kB 00:00 2024-03-30T13:32:08.100 INFO:teuthology.orchestra.run.smithi113.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 87 kB/s | 23 kB 00:00 2024-03-30T13:32:08.250 INFO:teuthology.orchestra.run.smithi113.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 68 kB/s | 19 kB 00:00 2024-03-30T13:32:08.342 INFO:teuthology.orchestra.run.smithi113.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 71 kB/s | 20 kB 00:00 2024-03-30T13:32:08.442 INFO:teuthology.orchestra.run.smithi113.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 135 kB/s | 46 kB 00:00 2024-03-30T13:32:08.609 INFO:teuthology.orchestra.run.smithi113.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 220 kB/s | 79 kB 00:00 2024-03-30T13:32:08.709 INFO:teuthology.orchestra.run.smithi113.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 61 kB/s | 16 kB 00:00 2024-03-30T13:32:08.951 INFO:teuthology.orchestra.run.smithi113.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 447 kB/s | 272 kB 00:00 2024-03-30T13:32:08.976 INFO:teuthology.orchestra.run.smithi113.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 134 kB/s | 36 kB 00:00 2024-03-30T13:32:09.001 INFO:teuthology.orchestra.run.smithi113.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 229 kB/s | 90 kB 00:00 2024-03-30T13:32:09.269 INFO:teuthology.orchestra.run.smithi113.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 75 kB/s | 20 kB 00:00 2024-03-30T13:32:09.452 INFO:teuthology.orchestra.run.smithi113.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 459 kB/s | 230 kB 00:00 2024-03-30T13:32:09.577 INFO:teuthology.orchestra.run.smithi113.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 711 kB/s | 427 kB 00:00 2024-03-30T13:32:09.582 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:32:09.582 INFO:teuthology.orchestra.run.smithi113.stdout:Total 7.0 MB/s | 30 MB 00:04 2024-03-30T13:32:09.606 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-30T13:32:09.647 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-30T13:32:09.647 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-30T13:32:10.050 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-30T13:32:10.050 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-30T13:32:11.375 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-30T13:32:11.430 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-30T13:32:11.462 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-30T13:32:11.490 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-30T13:32:11.562 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-30T13:32:11.606 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-30T13:32:11.639 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-30T13:32:11.731 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-30T13:32:11.808 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-30T13:32:11.850 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-30T13:32:11.921 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-30T13:32:11.984 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-30T13:32:12.022 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-30T13:32:12.079 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-30T13:32:12.135 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-30T13:32:12.320 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-30T13:32:12.369 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-30T13:32:12.407 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-30T13:32:12.441 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-30T13:32:12.471 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-30T13:32:12.547 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-30T13:32:12.582 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-30T13:32:12.615 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-30T13:32:12.656 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-30T13:32:12.691 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-30T13:32:12.742 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-30T13:32:12.832 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-30T13:32:12.886 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-30T13:32:12.961 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-30T13:32:13.097 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-30T13:32:13.217 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-30T13:32:13.382 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1967.g4e50bbcd.el9. 31/36 2024-03-30T13:32:13.411 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/36 2024-03-30T13:32:13.437 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/36 2024-03-30T13:32:13.437 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:32:13.437 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-30T13:32:13.437 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-30T13:32:13.437 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-30T13:32:13.437 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:14.870 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 33/36 2024-03-30T13:32:14.896 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 33/36 2024-03-30T13:32:14.897 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:32:14.897 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-30T13:32:14.897 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-30T13:32:14.897 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-30T13:32:14.897 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:15.346 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 34/36 2024-03-30T13:32:15.369 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 34/36 2024-03-30T13:32:15.369 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:32:15.370 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-30T13:32:15.370 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-30T13:32:15.370 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-30T13:32:15.370 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:15.597 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 35/36 2024-03-30T13:32:15.620 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 35/36 2024-03-30T13:32:15.620 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:32:15.620 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-30T13:32:15.620 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-30T13:32:15.620 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-30T13:32:15.620 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:15.686 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 36/36 2024-03-30T13:32:17.460 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 36/36 2024-03-30T13:32:17.460 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/36 2024-03-30T13:32:17.461 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/36 2024-03-30T13:32:17.461 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 3/36 2024-03-30T13:32:17.461 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 4/36 2024-03-30T13:32:17.461 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/36 2024-03-30T13:32:17.461 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1967.g4e50bbcd.el9. 6/36 2024-03-30T13:32:17.461 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-30T13:32:17.461 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-30T13:32:17.461 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-30T13:32:17.461 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-30T13:32:17.461 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-30T13:32:17.461 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-30T13:32:17.461 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-30T13:32:17.461 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-30T13:32:17.461 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-30T13:32:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-30T13:32:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-30T13:32:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-30T13:32:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-30T13:32:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-30T13:32:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-30T13:32:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-30T13:32:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-30T13:32:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-30T13:32:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-30T13:32:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-30T13:32:17.462 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-30T13:32:17.463 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-30T13:32:17.463 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-30T13:32:17.463 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-30T13:32:17.463 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-30T13:32:17.463 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-30T13:32:17.463 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-30T13:32:17.463 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-30T13:32:17.463 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-30T13:32:18.575 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-30T13:32:18.576 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:18.576 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-30T13:32:18.576 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:32:18.576 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:32:18.576 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:32:18.576 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-modules-core-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:32:18.576 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:32:18.576 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:32:18.576 INFO:teuthology.orchestra.run.smithi113.stdout: lua-5.4.4-4.el9.x86_64 2024-03-30T13:32:18.576 INFO:teuthology.orchestra.run.smithi113.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-30T13:32:18.576 INFO:teuthology.orchestra.run.smithi113.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-30T13:32:18.576 INFO:teuthology.orchestra.run.smithi113.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-30T13:32:18.576 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-30T13:32:18.577 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-30T13:32:18.577 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-30T13:32:18.577 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-30T13:32:18.577 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-30T13:32:18.577 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-30T13:32:18.577 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-30T13:32:18.577 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-30T13:32:18.577 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-30T13:32:18.577 INFO:teuthology.orchestra.run.smithi113.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-30T13:32:18.577 INFO:teuthology.orchestra.run.smithi113.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-30T13:32:18.577 INFO:teuthology.orchestra.run.smithi113.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-30T13:32:18.577 INFO:teuthology.orchestra.run.smithi113.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-30T13:32:18.577 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-30T13:32:18.578 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ply-3.11-14.el9.noarch 2024-03-30T13:32:18.578 INFO:teuthology.orchestra.run.smithi113.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-30T13:32:18.578 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-30T13:32:18.578 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-30T13:32:18.578 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-30T13:32:18.578 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-30T13:32:18.578 INFO:teuthology.orchestra.run.smithi113.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-30T13:32:18.578 INFO:teuthology.orchestra.run.smithi113.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-30T13:32:18.578 INFO:teuthology.orchestra.run.smithi113.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-30T13:32:18.578 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-30T13:32:18.578 INFO:teuthology.orchestra.run.smithi113.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-30T13:32:18.578 INFO:teuthology.orchestra.run.smithi113.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-30T13:32:18.578 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:18.578 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:32:18.937 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-base 2024-03-30T13:32:19.496 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:00 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:32:19.594 INFO:teuthology.orchestra.run.smithi113.stdout:Package ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:32:19.645 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:32:19.646 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-03-30T13:32:19.646 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:32:19.714 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install cephadm 2024-03-30T13:32:20.269 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:01 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:32:20.418 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:32:20.418 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:32:20.419 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-03-30T13:32:20.419 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:32:20.419 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-30T13:32:20.419 INFO:teuthology.orchestra.run.smithi113.stdout: cephadm noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 748 k 2024-03-30T13:32:20.419 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:20.419 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-30T13:32:20.419 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:32:20.419 INFO:teuthology.orchestra.run.smithi113.stdout:Install 1 Package 2024-03-30T13:32:20.419 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:20.420 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 748 k 2024-03-30T13:32:20.420 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 755 k 2024-03-30T13:32:20.420 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-30T13:32:20.825 INFO:teuthology.orchestra.run.smithi113.stdout:cephadm-19.0.0-1967.g4e50bbcd.el9.noarch.rpm 1.8 MB/s | 748 kB 00:00 2024-03-30T13:32:20.825 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:32:20.826 INFO:teuthology.orchestra.run.smithi113.stdout:Total 1.8 MB/s | 748 kB 00:00 2024-03-30T13:32:20.826 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-30T13:32:20.831 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-30T13:32:20.831 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-30T13:32:20.836 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-30T13:32:20.837 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-30T13:32:20.914 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-30T13:32:21.349 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T13:32:21.438 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T13:32:21.819 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T13:32:22.067 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T13:32:22.067 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:22.067 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-30T13:32:22.067 INFO:teuthology.orchestra.run.smithi113.stdout: cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:32:22.067 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:22.067 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:32:22.295 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-immutable-object-cache 2024-03-30T13:32:22.863 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:03 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:32:23.018 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:32:23.018 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:32:23.018 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repo Size 2024-03-30T13:32:23.019 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:32:23.019 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-30T13:32:23.019 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 145 k 2024-03-30T13:32:23.019 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:23.019 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-30T13:32:23.019 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:32:23.019 INFO:teuthology.orchestra.run.smithi113.stdout:Install 1 Package 2024-03-30T13:32:23.019 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:23.019 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 145 k 2024-03-30T13:32:23.020 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 435 k 2024-03-30T13:32:23.020 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-30T13:32:23.431 INFO:teuthology.orchestra.run.smithi113.stdout:ceph-immutable-object-cache-19.0.0-1967.g4e50bb 352 kB/s | 145 kB 00:00 2024-03-30T13:32:23.431 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:32:23.432 INFO:teuthology.orchestra.run.smithi113.stdout:Total 351 kB/s | 145 kB 00:00 2024-03-30T13:32:23.432 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-30T13:32:23.440 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-30T13:32:23.440 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-30T13:32:23.492 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-30T13:32:23.493 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-30T13:32:23.626 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-30T13:32:23.690 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd. 1/1 2024-03-30T13:32:23.715 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd. 1/1 2024-03-30T13:32:23.715 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:32:23.715 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-30T13:32:23.715 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:24.738 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd. 1/1 2024-03-30T13:32:24.738 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:24.738 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-30T13:32:24.739 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:32:24.739 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:24.739 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:32:24.998 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-mgr 2024-03-30T13:32:25.570 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:06 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:32:25.668 INFO:teuthology.orchestra.run.smithi113.stdout:Package ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:32:25.720 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:32:25.721 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-03-30T13:32:25.722 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:32:25.814 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-mgr-dashboard 2024-03-30T13:32:26.371 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:07 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:32:26.521 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:32:26.522 INFO:teuthology.orchestra.run.smithi113.stdout:====================================================================================== 2024-03-30T13:32:26.522 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-03-30T13:32:26.522 INFO:teuthology.orchestra.run.smithi113.stdout:====================================================================================== 2024-03-30T13:32:26.522 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-30T13:32:26.522 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 3.5 M 2024-03-30T13:32:26.522 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-03-30T13:32:26.522 INFO:teuthology.orchestra.run.smithi113.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-30T13:32:26.522 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 31 k 2024-03-30T13:32:26.522 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 15 k 2024-03-30T13:32:26.522 INFO:teuthology.orchestra.run.smithi113.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-30T13:32:26.523 INFO:teuthology.orchestra.run.smithi113.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-30T13:32:26.523 INFO:teuthology.orchestra.run.smithi113.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-30T13:32:26.523 INFO:teuthology.orchestra.run.smithi113.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-30T13:32:26.523 INFO:teuthology.orchestra.run.smithi113.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-30T13:32:26.523 INFO:teuthology.orchestra.run.smithi113.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-30T13:32:26.523 INFO:teuthology.orchestra.run.smithi113.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-30T13:32:26.523 INFO:teuthology.orchestra.run.smithi113.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-30T13:32:26.523 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:26.523 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-30T13:32:26.523 INFO:teuthology.orchestra.run.smithi113.stdout:====================================================================================== 2024-03-30T13:32:26.523 INFO:teuthology.orchestra.run.smithi113.stdout:Install 12 Packages 2024-03-30T13:32:26.523 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:26.524 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 8.6 M 2024-03-30T13:32:26.524 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 103 M 2024-03-30T13:32:26.524 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-30T13:32:27.301 INFO:teuthology.orchestra.run.smithi113.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1967.g4e 117 kB/s | 31 kB 00:00 2024-03-30T13:32:27.335 INFO:teuthology.orchestra.run.smithi113.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1967.g4e5 51 kB/s | 15 kB 00:00 2024-03-30T13:32:27.519 INFO:teuthology.orchestra.run.smithi113.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-30T13:32:27.569 INFO:teuthology.orchestra.run.smithi113.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.8 MB/s | 1.0 MB 00:00 2024-03-30T13:32:27.653 INFO:teuthology.orchestra.run.smithi113.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 6.3 MB/s | 863 kB 00:00 2024-03-30T13:32:27.761 INFO:teuthology.orchestra.run.smithi113.stdout:(6/12): ceph-mgr-dashboard-19.0.0-1967.g4e50bbc 4.8 MB/s | 3.5 MB 00:00 2024-03-30T13:32:28.104 INFO:teuthology.orchestra.run.smithi113.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 43 kB/s | 19 kB 00:00 2024-03-30T13:32:28.412 INFO:teuthology.orchestra.run.smithi113.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 651 kB/s | 548 kB 00:00 2024-03-30T13:32:28.529 INFO:teuthology.orchestra.run.smithi113.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 264 kB/s | 31 kB 00:00 2024-03-30T13:32:28.587 INFO:teuthology.orchestra.run.smithi113.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 299 kB/s | 144 kB 00:00 2024-03-30T13:32:28.662 INFO:teuthology.orchestra.run.smithi113.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.4 MB/s | 188 kB 00:00 2024-03-30T13:32:29.630 INFO:teuthology.orchestra.run.smithi113.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 1.1 MB/s | 2.0 MB 00:01 2024-03-30T13:32:29.637 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:32:29.637 INFO:teuthology.orchestra.run.smithi113.stdout:Total 2.8 MB/s | 8.6 MB 00:03 2024-03-30T13:32:29.657 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-30T13:32:29.676 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-30T13:32:29.677 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-30T13:32:29.860 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-30T13:32:29.860 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-30T13:32:30.423 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-30T13:32:30.505 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-30T13:32:30.548 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-30T13:32:30.595 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-30T13:32:30.626 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-30T13:32:30.733 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-30T13:32:30.859 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-30T13:32:30.953 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-30T13:32:30.997 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-30T13:32:31.025 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-30T13:32:31.065 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1967.g4e50bbcd.el9 10/12 2024-03-30T13:32:32.114 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1967.g4e50bbcd.el 11/12 2024-03-30T13:32:32.150 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noa 12/12 2024-03-30T13:32:33.504 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noa 12/12 2024-03-30T13:32:33.504 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1967.g4e50bbcd.el 1/12 2024-03-30T13:32:33.504 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noa 2/12 2024-03-30T13:32:33.504 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1967.g4e50bbcd.el9 3/12 2024-03-30T13:32:33.505 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-30T13:32:33.505 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-30T13:32:33.505 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-30T13:32:33.505 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-30T13:32:33.505 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-30T13:32:33.505 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-30T13:32:33.505 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-30T13:32:33.505 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-30T13:32:33.835 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-30T13:32:33.836 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:33.836 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-30T13:32:33.836 INFO:teuthology.orchestra.run.smithi113.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-30T13:32:33.836 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-grafana-dashboards-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:32:33.836 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:32:33.836 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-prometheus-alerts-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:32:33.836 INFO:teuthology.orchestra.run.smithi113.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-30T13:32:33.836 INFO:teuthology.orchestra.run.smithi113.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-30T13:32:33.836 INFO:teuthology.orchestra.run.smithi113.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-30T13:32:33.836 INFO:teuthology.orchestra.run.smithi113.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-30T13:32:33.836 INFO:teuthology.orchestra.run.smithi113.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-30T13:32:33.836 INFO:teuthology.orchestra.run.smithi113.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-30T13:32:33.836 INFO:teuthology.orchestra.run.smithi113.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-30T13:32:33.837 INFO:teuthology.orchestra.run.smithi113.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-30T13:32:33.837 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:33.837 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:32:34.122 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-30T13:32:34.708 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:15 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:32:34.869 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:32:34.870 INFO:teuthology.orchestra.run.smithi113.stdout:============================================================================================ 2024-03-30T13:32:34.870 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-03-30T13:32:34.870 INFO:teuthology.orchestra.run.smithi113.stdout:============================================================================================ 2024-03-30T13:32:34.870 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-30T13:32:34.871 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 7.4 M 2024-03-30T13:32:34.871 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-03-30T13:32:34.871 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-30T13:32:34.871 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-30T13:32:34.871 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-30T13:32:34.871 INFO:teuthology.orchestra.run.smithi113.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-30T13:32:34.871 INFO:teuthology.orchestra.run.smithi113.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-30T13:32:34.871 INFO:teuthology.orchestra.run.smithi113.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-30T13:32:34.871 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-30T13:32:34.871 INFO:teuthology.orchestra.run.smithi113.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-30T13:32:34.871 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-30T13:32:34.871 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-30T13:32:34.872 INFO:teuthology.orchestra.run.smithi113.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-30T13:32:34.872 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:34.872 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-30T13:32:34.872 INFO:teuthology.orchestra.run.smithi113.stdout:============================================================================================ 2024-03-30T13:32:34.872 INFO:teuthology.orchestra.run.smithi113.stdout:Install 12 Packages 2024-03-30T13:32:34.872 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:34.872 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 37 M 2024-03-30T13:32:34.873 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 209 M 2024-03-30T13:32:34.873 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-30T13:32:35.280 INFO:teuthology.orchestra.run.smithi113.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 197 kB/s | 32 kB 00:00 2024-03-30T13:32:35.313 INFO:teuthology.orchestra.run.smithi113.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 535 kB/s | 17 kB 00:00 2024-03-30T13:32:35.355 INFO:teuthology.orchestra.run.smithi113.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 845 kB/s | 34 kB 00:00 2024-03-30T13:32:35.405 INFO:teuthology.orchestra.run.smithi113.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-30T13:32:35.623 INFO:teuthology.orchestra.run.smithi113.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-03-30T13:32:35.689 INFO:teuthology.orchestra.run.smithi113.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-03-30T13:32:35.714 INFO:teuthology.orchestra.run.smithi113.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-30T13:32:36.090 INFO:teuthology.orchestra.run.smithi113.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-30T13:32:36.215 INFO:teuthology.orchestra.run.smithi113.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-03-30T13:32:36.307 INFO:teuthology.orchestra.run.smithi113.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 902 kB/s | 195 kB 00:00 2024-03-30T13:32:36.357 INFO:teuthology.orchestra.run.smithi113.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-30T13:32:36.541 INFO:teuthology.orchestra.run.smithi113.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 5.2 MB/s | 7.4 MB 00:01 2024-03-30T13:32:36.544 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:32:36.545 INFO:teuthology.orchestra.run.smithi113.stdout:Total 22 MB/s | 37 MB 00:01 2024-03-30T13:32:36.586 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-30T13:32:36.611 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-30T13:32:36.611 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-30T13:32:36.974 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-30T13:32:36.974 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-30T13:32:37.671 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-30T13:32:37.739 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-30T13:32:37.770 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-30T13:32:37.864 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-30T13:32:37.907 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-30T13:32:38.083 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-30T13:32:38.122 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-30T13:32:38.198 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-30T13:32:38.849 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-30T13:32:38.969 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-30T13:32:40.239 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-30T13:32:41.258 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-30T13:32:41.283 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50b 12/12 2024-03-30T13:32:43.919 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50b 12/12 2024-03-30T13:32:43.920 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50b 1/12 2024-03-30T13:32:43.920 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-30T13:32:43.920 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-30T13:32:43.920 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-30T13:32:43.920 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-30T13:32:43.920 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-30T13:32:43.920 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-30T13:32:43.920 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-30T13:32:43.920 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-30T13:32:43.920 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-30T13:32:43.920 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-30T13:32:44.418 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-30T13:32:44.418 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:44.418 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-30T13:32:44.418 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:32:44.418 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-30T13:32:44.418 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-30T13:32:44.418 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-30T13:32:44.418 INFO:teuthology.orchestra.run.smithi113.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-30T13:32:44.418 INFO:teuthology.orchestra.run.smithi113.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-30T13:32:44.419 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-30T13:32:44.419 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-30T13:32:44.419 INFO:teuthology.orchestra.run.smithi113.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-30T13:32:44.419 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-30T13:32:44.419 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-30T13:32:44.419 INFO:teuthology.orchestra.run.smithi113.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-30T13:32:44.419 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:44.419 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:32:44.814 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-mgr-rook 2024-03-30T13:32:45.401 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:26 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:32:45.553 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:32:45.555 INFO:teuthology.orchestra.run.smithi113.stdout:======================================================================================== 2024-03-30T13:32:45.555 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-03-30T13:32:45.555 INFO:teuthology.orchestra.run.smithi113.stdout:======================================================================================== 2024-03-30T13:32:45.555 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-30T13:32:45.555 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-rook noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 49 k 2024-03-30T13:32:45.555 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-03-30T13:32:45.555 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-30T13:32:45.555 INFO:teuthology.orchestra.run.smithi113.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-30T13:32:45.555 INFO:teuthology.orchestra.run.smithi113.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-30T13:32:45.555 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-30T13:32:45.555 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-30T13:32:45.555 INFO:teuthology.orchestra.run.smithi113.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-30T13:32:45.556 INFO:teuthology.orchestra.run.smithi113.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-30T13:32:45.556 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-30T13:32:45.556 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-30T13:32:45.556 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-30T13:32:45.556 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-30T13:32:45.556 INFO:teuthology.orchestra.run.smithi113.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-30T13:32:45.556 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:45.556 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-30T13:32:45.556 INFO:teuthology.orchestra.run.smithi113.stdout:======================================================================================== 2024-03-30T13:32:45.556 INFO:teuthology.orchestra.run.smithi113.stdout:Install 13 Packages 2024-03-30T13:32:45.556 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:45.557 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 2.0 M 2024-03-30T13:32:45.557 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 26 M 2024-03-30T13:32:45.557 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-30T13:32:45.833 INFO:teuthology.orchestra.run.smithi113.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 207 kB/s | 24 kB 00:00 2024-03-30T13:32:45.858 INFO:teuthology.orchestra.run.smithi113.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 127 kB/s | 18 kB 00:00 2024-03-30T13:32:45.909 INFO:teuthology.orchestra.run.smithi113.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-30T13:32:45.934 INFO:teuthology.orchestra.run.smithi113.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-30T13:32:45.959 INFO:teuthology.orchestra.run.smithi113.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-30T13:32:45.985 INFO:teuthology.orchestra.run.smithi113.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2024-03-30T13:32:46.044 INFO:teuthology.orchestra.run.smithi113.stdout:(7/13): ceph-mgr-rook-19.0.0-1967.g4e50bbcd.el9 150 kB/s | 49 kB 00:00 2024-03-30T13:32:46.386 INFO:teuthology.orchestra.run.smithi113.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 37 kB/s | 15 kB 00:00 2024-03-30T13:32:46.453 INFO:teuthology.orchestra.run.smithi113.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 65 kB/s | 32 kB 00:00 2024-03-30T13:32:46.669 INFO:teuthology.orchestra.run.smithi113.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 324 kB/s | 203 kB 00:00 2024-03-30T13:32:46.694 INFO:teuthology.orchestra.run.smithi113.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 244 kB/s | 59 kB 00:00 2024-03-30T13:32:46.853 INFO:teuthology.orchestra.run.smithi113.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 489 kB/s | 90 kB 00:00 2024-03-30T13:32:47.312 INFO:teuthology.orchestra.run.smithi113.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.1 MB/s | 1.0 MB 00:00 2024-03-30T13:32:47.317 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:32:47.317 INFO:teuthology.orchestra.run.smithi113.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2024-03-30T13:32:47.317 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-30T13:32:47.332 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-30T13:32:47.332 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-30T13:32:47.431 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-30T13:32:47.432 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-30T13:32:47.964 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-30T13:32:48.125 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-30T13:32:48.179 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-30T13:32:48.234 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-30T13:32:48.272 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-30T13:32:48.313 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-30T13:32:48.406 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-30T13:32:48.507 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-30T13:32:48.556 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-30T13:32:49.293 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-30T13:32:49.337 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-30T13:32:49.370 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-30T13:32:49.419 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-30T13:32:49.441 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 13/13 2024-03-30T13:32:50.236 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 13/13 2024-03-30T13:32:50.236 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/13 2024-03-30T13:32:50.236 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-30T13:32:50.237 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-30T13:32:50.237 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-30T13:32:50.237 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-30T13:32:50.237 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-30T13:32:50.237 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-30T13:32:50.237 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-30T13:32:50.237 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-30T13:32:50.237 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-30T13:32:50.237 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-30T13:32:50.237 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-30T13:32:50.590 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-30T13:32:50.590 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:50.590 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-30T13:32:50.590 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:32:50.590 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-30T13:32:50.590 INFO:teuthology.orchestra.run.smithi113.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-30T13:32:50.590 INFO:teuthology.orchestra.run.smithi113.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-30T13:32:50.590 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-30T13:32:50.590 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-30T13:32:50.590 INFO:teuthology.orchestra.run.smithi113.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-30T13:32:50.591 INFO:teuthology.orchestra.run.smithi113.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-30T13:32:50.591 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-30T13:32:50.591 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-30T13:32:50.591 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-30T13:32:50.591 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-30T13:32:50.591 INFO:teuthology.orchestra.run.smithi113.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-30T13:32:50.591 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:50.591 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:32:50.833 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-mgr-cephadm 2024-03-30T13:32:51.392 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:32 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:32:51.543 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:32:51.544 INFO:teuthology.orchestra.run.smithi113.stdout:======================================================================================== 2024-03-30T13:32:51.544 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-03-30T13:32:51.544 INFO:teuthology.orchestra.run.smithi113.stdout:======================================================================================== 2024-03-30T13:32:51.544 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-30T13:32:51.544 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 139 k 2024-03-30T13:32:51.544 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-03-30T13:32:51.544 INFO:teuthology.orchestra.run.smithi113.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-30T13:32:51.544 INFO:teuthology.orchestra.run.smithi113.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-30T13:32:51.544 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-30T13:32:51.544 INFO:teuthology.orchestra.run.smithi113.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-30T13:32:51.545 INFO:teuthology.orchestra.run.smithi113.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-30T13:32:51.545 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:51.545 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-30T13:32:51.545 INFO:teuthology.orchestra.run.smithi113.stdout:======================================================================================== 2024-03-30T13:32:51.545 INFO:teuthology.orchestra.run.smithi113.stdout:Install 6 Packages 2024-03-30T13:32:51.545 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:51.545 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 6.8 M 2024-03-30T13:32:51.545 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 33 M 2024-03-30T13:32:51.546 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-30T13:32:52.113 INFO:teuthology.orchestra.run.smithi113.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1967.g4e50bbcd.e 546 kB/s | 139 kB 00:00 2024-03-30T13:32:52.263 INFO:teuthology.orchestra.run.smithi113.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-03-30T13:32:52.289 INFO:teuthology.orchestra.run.smithi113.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 576 kB/s | 248 kB 00:00 2024-03-30T13:32:52.397 INFO:teuthology.orchestra.run.smithi113.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 432 kB/s | 58 kB 00:00 2024-03-30T13:32:52.423 INFO:teuthology.orchestra.run.smithi113.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 382 kB/s | 51 kB 00:00 2024-03-30T13:32:52.448 INFO:teuthology.orchestra.run.smithi113.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-03-30T13:32:52.450 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:32:52.450 INFO:teuthology.orchestra.run.smithi113.stdout:Total 7.5 MB/s | 6.8 MB 00:00 2024-03-30T13:32:52.463 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-30T13:32:52.473 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-30T13:32:52.473 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-30T13:32:52.561 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-30T13:32:52.562 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-30T13:32:52.857 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-30T13:32:53.021 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-30T13:32:53.065 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-30T13:32:53.575 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-30T13:32:53.660 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-30T13:32:53.722 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-30T13:32:53.745 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 6/6 2024-03-30T13:32:54.574 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 6/6 2024-03-30T13:32:54.574 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/6 2024-03-30T13:32:54.574 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-30T13:32:54.574 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-30T13:32:54.574 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-30T13:32:54.574 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-30T13:32:54.870 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-30T13:32:54.870 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:54.870 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-30T13:32:54.870 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:32:54.870 INFO:teuthology.orchestra.run.smithi113.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-30T13:32:54.870 INFO:teuthology.orchestra.run.smithi113.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-30T13:32:54.871 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-30T13:32:54.871 INFO:teuthology.orchestra.run.smithi113.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-30T13:32:54.871 INFO:teuthology.orchestra.run.smithi113.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-30T13:32:54.871 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:54.871 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:32:55.112 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-fuse 2024-03-30T13:32:55.672 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:36 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:32:55.824 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:32:55.825 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:32:55.825 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-03-30T13:32:55.825 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:32:55.825 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-30T13:32:55.825 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-fuse x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 889 k 2024-03-30T13:32:55.825 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-03-30T13:32:55.825 INFO:teuthology.orchestra.run.smithi113.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-30T13:32:55.825 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:55.826 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-30T13:32:55.826 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:32:55.826 INFO:teuthology.orchestra.run.smithi113.stdout:Install 2 Packages 2024-03-30T13:32:55.826 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:55.826 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 968 k 2024-03-30T13:32:55.827 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 2.8 M 2024-03-30T13:32:55.827 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-30T13:32:56.049 INFO:teuthology.orchestra.run.smithi113.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 353 kB/s | 79 kB 00:00 2024-03-30T13:32:56.116 INFO:teuthology.orchestra.run.smithi113.stdout:(2/2): ceph-fuse-19.0.0-1967.g4e50bbcd.el9.x86_ 3.0 MB/s | 889 kB 00:00 2024-03-30T13:32:56.117 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:32:56.118 INFO:teuthology.orchestra.run.smithi113.stdout:Total 3.2 MB/s | 968 kB 00:00 2024-03-30T13:32:56.118 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-30T13:32:56.138 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-30T13:32:56.138 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-30T13:32:56.211 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-30T13:32:56.211 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-30T13:32:56.409 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-30T13:32:56.534 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-30T13:32:56.600 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/2 2024-03-30T13:32:57.248 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/2 2024-03-30T13:32:57.248 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/2 2024-03-30T13:32:57.563 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-30T13:32:57.563 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:57.563 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-30T13:32:57.563 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-30T13:32:57.563 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:57.563 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:32:57.837 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-volume 2024-03-30T13:32:58.406 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:39 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:32:58.559 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:32:58.559 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:32:58.560 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-03-30T13:32:58.560 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:32:58.560 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-30T13:32:58.560 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-volume noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 264 k 2024-03-30T13:32:58.560 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-03-30T13:32:58.560 INFO:teuthology.orchestra.run.smithi113.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-30T13:32:58.560 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:58.560 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-30T13:32:58.560 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:32:58.560 INFO:teuthology.orchestra.run.smithi113.stdout:Install 2 Packages 2024-03-30T13:32:58.560 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:58.561 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 332 k 2024-03-30T13:32:58.561 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 1.5 M 2024-03-30T13:32:58.561 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-30T13:32:58.713 INFO:teuthology.orchestra.run.smithi113.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 453 kB/s | 69 kB 00:00 2024-03-30T13:32:58.804 INFO:teuthology.orchestra.run.smithi113.stdout:(2/2): ceph-volume-19.0.0-1967.g4e50bbcd.el9.no 1.1 MB/s | 264 kB 00:00 2024-03-30T13:32:58.806 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:32:58.806 INFO:teuthology.orchestra.run.smithi113.stdout:Total 1.3 MB/s | 332 kB 00:00 2024-03-30T13:32:58.806 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-30T13:32:58.815 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-30T13:32:58.815 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-30T13:32:58.857 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-30T13:32:58.857 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-30T13:32:59.008 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-30T13:32:59.178 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-30T13:32:59.215 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 2/2 2024-03-30T13:32:59.232 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 2/2 2024-03-30T13:32:59.232 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:32:59.232 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-30T13:32:59.232 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:32:59.957 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/2 2024-03-30T13:33:00.290 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-30T13:33:00.290 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:00.290 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-30T13:33:00.290 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:33:00.290 INFO:teuthology.orchestra.run.smithi113.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-30T13:33:00.290 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:00.290 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:33:00.584 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install librados-devel 2024-03-30T13:33:01.157 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:42 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:33:01.307 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:33:01.308 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:33:01.308 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repo Size 2024-03-30T13:33:01.308 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:33:01.308 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-30T13:33:01.308 INFO:teuthology.orchestra.run.smithi113.stdout: librados-devel x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 127 k 2024-03-30T13:33:01.308 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:01.308 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-30T13:33:01.308 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:33:01.309 INFO:teuthology.orchestra.run.smithi113.stdout:Install 1 Package 2024-03-30T13:33:01.309 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:01.309 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 127 k 2024-03-30T13:33:01.309 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 456 k 2024-03-30T13:33:01.309 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-30T13:33:01.660 INFO:teuthology.orchestra.run.smithi113.stdout:librados-devel-19.0.0-1967.g4e50bbcd.el9.x86_64 361 kB/s | 127 kB 00:00 2024-03-30T13:33:01.661 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:33:01.661 INFO:teuthology.orchestra.run.smithi113.stdout:Total 360 kB/s | 127 kB 00:00 2024-03-30T13:33:01.661 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-30T13:33:01.667 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-30T13:33:01.667 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-30T13:33:01.723 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-30T13:33:01.723 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-30T13:33:01.921 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-30T13:33:02.026 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:33:02.479 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:33:02.752 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:33:02.752 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:02.753 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-30T13:33:02.753 INFO:teuthology.orchestra.run.smithi113.stdout: librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:33:02.753 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:02.753 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:33:02.969 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install libcephfs2 2024-03-30T13:33:03.544 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:44 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:33:03.644 INFO:teuthology.orchestra.run.smithi113.stdout:Package libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:33:03.698 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:33:03.699 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-03-30T13:33:03.699 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:33:03.764 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install libcephfs-devel 2024-03-30T13:33:04.325 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:45 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:33:04.475 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:33:04.475 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:33:04.476 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repo Size 2024-03-30T13:33:04.476 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:33:04.476 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-30T13:33:04.476 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs-devel x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 32 k 2024-03-30T13:33:04.476 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:04.476 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-30T13:33:04.476 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:33:04.476 INFO:teuthology.orchestra.run.smithi113.stdout:Install 1 Package 2024-03-30T13:33:04.476 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:04.476 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 32 k 2024-03-30T13:33:04.476 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 142 k 2024-03-30T13:33:04.476 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-30T13:33:04.639 INFO:teuthology.orchestra.run.smithi113.stdout:libcephfs-devel-19.0.0-1967.g4e50bbcd.el9.x86_6 195 kB/s | 32 kB 00:00 2024-03-30T13:33:04.640 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:33:04.640 INFO:teuthology.orchestra.run.smithi113.stdout:Total 193 kB/s | 32 kB 00:00 2024-03-30T13:33:04.640 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-30T13:33:04.643 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-30T13:33:04.644 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-30T13:33:04.672 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-30T13:33:04.673 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-30T13:33:04.752 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-30T13:33:04.854 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:33:05.229 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:33:05.484 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:33:05.484 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:05.485 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-30T13:33:05.485 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:33:05.485 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:05.485 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:33:05.665 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install librados2 2024-03-30T13:33:06.229 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:47 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:33:06.328 INFO:teuthology.orchestra.run.smithi113.stdout:Package librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:33:06.379 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:33:06.380 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-03-30T13:33:06.380 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:33:06.460 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install librbd1 2024-03-30T13:33:07.055 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:48 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:33:07.159 INFO:teuthology.orchestra.run.smithi113.stdout:Package librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:33:07.214 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:33:07.215 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-03-30T13:33:07.215 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:33:07.286 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install python3-rados 2024-03-30T13:33:07.874 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:48 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:33:07.973 INFO:teuthology.orchestra.run.smithi113.stdout:Package python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:33:08.024 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:33:08.026 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-03-30T13:33:08.026 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:33:08.095 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install python3-rgw 2024-03-30T13:33:08.660 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:49 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:33:08.757 INFO:teuthology.orchestra.run.smithi113.stdout:Package python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:33:08.810 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:33:08.811 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-03-30T13:33:08.811 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:33:08.874 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install python3-cephfs 2024-03-30T13:33:09.456 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:50 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:33:09.555 INFO:teuthology.orchestra.run.smithi113.stdout:Package python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:33:09.608 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:33:09.609 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-03-30T13:33:09.609 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:33:09.678 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install python3-rbd 2024-03-30T13:33:10.246 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:51 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:33:10.345 INFO:teuthology.orchestra.run.smithi113.stdout:Package python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:33:10.396 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:33:10.397 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-03-30T13:33:10.397 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:33:10.470 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install rbd-fuse 2024-03-30T13:33:11.034 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:52 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:33:11.184 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:33:11.185 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:33:11.185 INFO:teuthology.orchestra.run.smithi113.stdout: Package Architecture Version Repository Size 2024-03-30T13:33:11.185 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:33:11.185 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-30T13:33:11.185 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-fuse x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 85 k 2024-03-30T13:33:11.185 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:11.185 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-30T13:33:11.185 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:33:11.185 INFO:teuthology.orchestra.run.smithi113.stdout:Install 1 Package 2024-03-30T13:33:11.185 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:11.186 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 85 k 2024-03-30T13:33:11.186 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 231 k 2024-03-30T13:33:11.186 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-30T13:33:11.468 INFO:teuthology.orchestra.run.smithi113.stdout:rbd-fuse-19.0.0-1967.g4e50bbcd.el9.x86_64.rpm 303 kB/s | 85 kB 00:00 2024-03-30T13:33:11.468 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:33:11.468 INFO:teuthology.orchestra.run.smithi113.stdout:Total 301 kB/s | 85 kB 00:00 2024-03-30T13:33:11.469 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-30T13:33:11.476 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-30T13:33:11.476 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-30T13:33:11.531 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-30T13:33:11.532 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-30T13:33:11.676 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-30T13:33:11.775 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:33:12.169 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:33:12.483 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:33:12.483 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:12.484 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-30T13:33:12.484 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:33:12.484 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:12.484 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:33:12.701 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install rbd-mirror 2024-03-30T13:33:13.279 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:54 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:33:13.436 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:33:13.437 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:33:13.437 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repo Size 2024-03-30T13:33:13.437 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:33:13.437 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-30T13:33:13.437 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-mirror x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 3.1 M 2024-03-30T13:33:13.437 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:13.437 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-30T13:33:13.438 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:33:13.438 INFO:teuthology.orchestra.run.smithi113.stdout:Install 1 Package 2024-03-30T13:33:13.438 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:13.438 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 3.1 M 2024-03-30T13:33:13.438 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 13 M 2024-03-30T13:33:13.438 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-30T13:33:14.121 INFO:teuthology.orchestra.run.smithi113.stdout:rbd-mirror-19.0.0-1967.g4e50bbcd.el9.x86_64.rpm 4.5 MB/s | 3.1 MB 00:00 2024-03-30T13:33:14.122 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:33:14.122 INFO:teuthology.orchestra.run.smithi113.stdout:Total 4.5 MB/s | 3.1 MB 00:00 2024-03-30T13:33:14.122 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-30T13:33:14.131 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-30T13:33:14.131 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-30T13:33:14.205 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-30T13:33:14.205 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-30T13:33:14.630 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-30T13:33:14.688 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:33:14.713 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:33:14.713 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:33:14.713 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-30T13:33:14.713 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-30T13:33:14.713 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-30T13:33:14.713 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:15.808 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:33:15.809 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:15.809 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-30T13:33:15.809 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:33:15.809 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:15.809 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:33:16.040 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install rbd-nbd 2024-03-30T13:33:16.615 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:57 ago on Sat 30 Mar 2024 01:31:19 PM UTC. 2024-03-30T13:33:16.766 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-30T13:33:16.767 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:33:16.767 INFO:teuthology.orchestra.run.smithi113.stdout: Package Architecture Version Repository Size 2024-03-30T13:33:16.767 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:33:16.767 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-30T13:33:16.767 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-nbd x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 170 k 2024-03-30T13:33:16.767 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:16.767 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-30T13:33:16.768 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-30T13:33:16.768 INFO:teuthology.orchestra.run.smithi113.stdout:Install 1 Package 2024-03-30T13:33:16.768 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:16.768 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 170 k 2024-03-30T13:33:16.768 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 494 k 2024-03-30T13:33:16.768 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-30T13:33:17.150 INFO:teuthology.orchestra.run.smithi113.stdout:rbd-nbd-19.0.0-1967.g4e50bbcd.el9.x86_64.rpm 446 kB/s | 170 kB 00:00 2024-03-30T13:33:17.150 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:33:17.151 INFO:teuthology.orchestra.run.smithi113.stdout:Total 444 kB/s | 170 kB 00:00 2024-03-30T13:33:17.151 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-30T13:33:17.159 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-30T13:33:17.159 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-30T13:33:17.219 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-30T13:33:17.219 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-30T13:33:17.366 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-30T13:33:17.456 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:33:17.874 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:33:18.173 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:33:18.173 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:18.173 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-30T13:33:18.173 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:33:18.173 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:33:18.173 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-30T13:33:18.411 DEBUG:teuthology.parallel:result is None 2024-03-30T13:34:21.363 INFO:teuthology.orchestra.run.smithi043.stdout:Extra Packages for Enterprise Linux 108 kB/s | 21 MB 03:16 2024-03-30T13:34:26.446 INFO:teuthology.orchestra.run.smithi043.stdout:lab-extras 44 kB/s | 1.7 kB 00:00 2024-03-30T13:34:29.154 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:34:29.156 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:34:29.156 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-03-30T13:34:29.156 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:34:29.156 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-03-30T13:34:29.157 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-radosgw x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 9.8 M 2024-03-30T13:34:29.157 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrading: 2024-03-30T13:34:29.157 INFO:teuthology.orchestra.run.smithi043.stdout: librados2 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 3.4 M 2024-03-30T13:34:29.157 INFO:teuthology.orchestra.run.smithi043.stdout: librbd1 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 3.1 M 2024-03-30T13:34:29.157 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-03-30T13:34:29.157 INFO:teuthology.orchestra.run.smithi043.stdout: boost-program-options 2024-03-30T13:34:29.157 INFO:teuthology.orchestra.run.smithi043.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-30T13:34:29.157 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 5.3 M 2024-03-30T13:34:29.157 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-common x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 21 M 2024-03-30T13:34:29.157 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-selinux x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 25 k 2024-03-30T13:34:29.157 INFO:teuthology.orchestra.run.smithi043.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-30T13:34:29.157 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-30T13:34:29.157 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-30T13:34:29.157 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs2 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 745 k 2024-03-30T13:34:29.158 INFO:teuthology.orchestra.run.smithi043.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-30T13:34:29.158 INFO:teuthology.orchestra.run.smithi043.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-30T13:34:29.158 INFO:teuthology.orchestra.run.smithi043.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-30T13:34:29.158 INFO:teuthology.orchestra.run.smithi043.stdout: libradosstriper1 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 501 k 2024-03-30T13:34:29.158 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-30T13:34:29.158 INFO:teuthology.orchestra.run.smithi043.stdout: librgw2 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 5.1 M 2024-03-30T13:34:29.158 INFO:teuthology.orchestra.run.smithi043.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-30T13:34:29.158 INFO:teuthology.orchestra.run.smithi043.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-30T13:34:29.158 INFO:teuthology.orchestra.run.smithi043.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-30T13:34:29.158 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse 2024-03-30T13:34:29.158 INFO:teuthology.orchestra.run.smithi043.stdout: x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 45 k 2024-03-30T13:34:29.158 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 131 k 2024-03-30T13:34:29.159 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cephfs x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 161 k 2024-03-30T13:34:29.159 INFO:teuthology.orchestra.run.smithi043.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-30T13:34:29.159 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rados x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 321 k 2024-03-30T13:34:29.159 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rbd x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 298 k 2024-03-30T13:34:29.159 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rgw x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 99 k 2024-03-30T13:34:29.159 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-30T13:34:29.159 INFO:teuthology.orchestra.run.smithi043.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-30T13:34:29.159 INFO:teuthology.orchestra.run.smithi043.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-30T13:34:29.159 INFO:teuthology.orchestra.run.smithi043.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-30T13:34:29.159 INFO:teuthology.orchestra.run.smithi043.stdout:Installing weak dependencies: 2024-03-30T13:34:29.159 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-30T13:34:29.159 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:34:29.159 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-03-30T13:34:29.159 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:34:29.160 INFO:teuthology.orchestra.run.smithi043.stdout:Install 30 Packages 2024-03-30T13:34:29.160 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrade 2 Packages 2024-03-30T13:34:29.160 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:34:29.160 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 59 M 2024-03-30T13:34:29.160 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-03-30T13:34:29.947 INFO:teuthology.orchestra.run.smithi043.stdout:(1/32): ceph-base-19.0.0-1967.g4e50bbcd.el9.x86 7.3 MB/s | 5.3 MB 00:00 2024-03-30T13:34:30.005 INFO:teuthology.orchestra.run.smithi043.stdout:(2/32): ceph-selinux-19.0.0-1967.g4e50bbcd.el9. 430 kB/s | 25 kB 00:00 2024-03-30T13:34:30.131 INFO:teuthology.orchestra.run.smithi043.stdout:(3/32): libcephfs2-19.0.0-1967.g4e50bbcd.el9.x8 5.8 MB/s | 745 kB 00:00 2024-03-30T13:34:30.222 INFO:teuthology.orchestra.run.smithi043.stdout:(4/32): libradosstriper1-19.0.0-1967.g4e50bbcd. 5.4 MB/s | 501 kB 00:00 2024-03-30T13:34:30.948 INFO:teuthology.orchestra.run.smithi043.stdout:(5/32): librgw2-19.0.0-1967.g4e50bbcd.el9.x86_6 7.1 MB/s | 5.1 MB 00:00 2024-03-30T13:34:31.056 INFO:teuthology.orchestra.run.smithi043.stdout:(6/32): ceph-radosgw-19.0.0-1967.g4e50bbcd.el9. 5.3 MB/s | 9.8 MB 00:01 2024-03-30T13:34:31.073 INFO:teuthology.orchestra.run.smithi043.stdout:(7/32): python3-ceph-argparse-19.0.0-1967.g4e50 361 kB/s | 45 kB 00:00 2024-03-30T13:34:31.107 INFO:teuthology.orchestra.run.smithi043.stdout:(8/32): python3-ceph-common-19.0.0-1967.g4e50bb 2.6 MB/s | 131 kB 00:00 2024-03-30T13:34:31.132 INFO:teuthology.orchestra.run.smithi043.stdout:(9/32): python3-cephfs-19.0.0-1967.g4e50bbcd.el 2.7 MB/s | 161 kB 00:00 2024-03-30T13:34:31.324 INFO:teuthology.orchestra.run.smithi043.stdout:(10/32): ceph-common-19.0.0-1967.g4e50bbcd.el9. 9.8 MB/s | 21 MB 00:02 2024-03-30T13:34:31.349 INFO:teuthology.orchestra.run.smithi043.stdout:(11/32): python3-rados-19.0.0-1967.g4e50bbcd.el 1.3 MB/s | 321 kB 00:00 2024-03-30T13:34:31.374 INFO:teuthology.orchestra.run.smithi043.stdout:(12/32): python3-rbd-19.0.0-1967.g4e50bbcd.el9. 1.2 MB/s | 298 kB 00:00 2024-03-30T13:34:31.391 INFO:teuthology.orchestra.run.smithi043.stdout:(13/32): python3-rgw-19.0.0-1967.g4e50bbcd.el9. 2.3 MB/s | 99 kB 00:00 2024-03-30T13:34:31.633 INFO:teuthology.orchestra.run.smithi043.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 373 kB/s | 106 kB 00:00 2024-03-30T13:34:31.658 INFO:teuthology.orchestra.run.smithi043.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 205 kB/s | 44 kB 00:00 2024-03-30T13:34:31.692 INFO:teuthology.orchestra.run.smithi043.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 501 kB/s | 159 kB 00:00 2024-03-30T13:34:31.842 INFO:teuthology.orchestra.run.smithi043.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 661 kB 00:00 2024-03-30T13:34:31.867 INFO:teuthology.orchestra.run.smithi043.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.3 MB/s | 282 kB 00:00 2024-03-30T13:34:31.893 INFO:teuthology.orchestra.run.smithi043.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 205 kB/s | 41 kB 00:00 2024-03-30T13:34:31.918 INFO:teuthology.orchestra.run.smithi043.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2024-03-30T13:34:31.960 INFO:teuthology.orchestra.run.smithi043.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 526 kB/s | 35 kB 00:00 2024-03-30T13:34:31.985 INFO:teuthology.orchestra.run.smithi043.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 507 kB/s | 46 kB 00:00 2024-03-30T13:34:32.285 INFO:teuthology.orchestra.run.smithi043.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 87 kB/s | 26 kB 00:00 2024-03-30T13:34:32.419 INFO:teuthology.orchestra.run.smithi043.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 364 kB/s | 49 kB 00:00 2024-03-30T13:34:32.511 INFO:teuthology.orchestra.run.smithi043.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 521 kB/s | 309 kB 00:00 2024-03-30T13:34:32.536 INFO:teuthology.orchestra.run.smithi043.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 577 kB/s | 67 kB 00:00 2024-03-30T13:34:32.686 INFO:teuthology.orchestra.run.smithi043.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-30T13:34:32.879 INFO:teuthology.orchestra.run.smithi043.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.2 MB/s | 838 kB 00:00 2024-03-30T13:34:33.228 INFO:teuthology.orchestra.run.smithi043.stdout:(29/32): librados2-19.0.0-1967.g4e50bbcd.el9.x8 9.7 MB/s | 3.4 MB 00:00 2024-03-30T13:34:33.304 INFO:teuthology.orchestra.run.smithi043.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 2.6 MB/s | 1.6 MB 00:00 2024-03-30T13:34:33.504 INFO:teuthology.orchestra.run.smithi043.stdout:(31/32): librbd1-19.0.0-1967.g4e50bbcd.el9.x86_ 11 MB/s | 3.1 MB 00:00 2024-03-30T13:34:33.579 INFO:teuthology.orchestra.run.smithi043.stdout:(32/32): libarrow-9.0.0-10.el9.x86_64.rpm 2.7 MB/s | 4.4 MB 00:01 2024-03-30T13:34:33.586 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:34:33.586 INFO:teuthology.orchestra.run.smithi043.stdout:Total 13 MB/s | 59 MB 00:04 2024-03-30T13:34:33.587 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-03-30T13:34:33.629 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-03-30T13:34:33.629 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-03-30T13:34:34.123 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-03-30T13:34:34.124 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-03-30T13:34:35.676 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-03-30T13:34:35.752 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-30T13:34:35.784 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-30T13:34:35.809 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-30T13:34:35.832 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-30T13:34:35.848 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-30T13:34:35.999 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-30T13:34:36.062 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-30T13:34:36.452 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-30T13:34:36.494 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 7/34 2024-03-30T13:34:36.587 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 7/34 2024-03-30T13:34:36.676 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 8/34 2024-03-30T13:34:36.704 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 9/34 2024-03-30T13:34:36.775 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 9/34 2024-03-30T13:34:36.804 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-30T13:34:36.848 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ceph-argparse-2:19.0.0-1967.g4e50bbcd.el9. 11/34 2024-03-30T13:34:36.918 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 12/34 2024-03-30T13:34:36.940 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_6 13/34 2024-03-30T13:34:37.003 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_6 13/34 2024-03-30T13:34:37.038 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-30T13:34:37.149 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-30T13:34:37.181 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-30T13:34:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-30T13:34:37.254 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-30T13:34:37.283 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-30T13:34:37.316 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-30T13:34:37.644 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-30T13:34:37.670 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 22/34 2024-03-30T13:34:37.745 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 22/34 2024-03-30T13:34:37.775 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 23/34 2024-03-30T13:34:37.817 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-30T13:34:38.301 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-30T13:34:38.331 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 26/34 2024-03-30T13:34:38.411 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 26/34 2024-03-30T13:34:38.481 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 27/34 2024-03-30T13:34:38.508 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x8 28/34 2024-03-30T13:34:40.809 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 29/34 2024-03-30T13:34:40.846 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 29/34 2024-03-30T13:34:41.416 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 29/34 2024-03-30T13:34:41.449 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 30/34 2024-03-30T13:34:41.539 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 30/34 2024-03-30T13:34:41.539 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-30T13:34:41.539 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-30T13:34:41.539 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:34:41.576 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 31/34 2024-03-30T13:34:51.474 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 31/34 2024-03-30T13:34:51.475 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /sys 2024-03-30T13:34:51.475 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /proc 2024-03-30T13:34:51.475 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /mnt 2024-03-30T13:34:51.475 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /var/tmp 2024-03-30T13:34:51.475 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /home 2024-03-30T13:34:51.475 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /root 2024-03-30T13:34:51.475 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /tmp 2024-03-30T13:34:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:34:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/34 2024-03-30T13:34:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/34 2024-03-30T13:34:52.452 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:34:52.452 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-30T13:34:52.453 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-30T13:34:52.453 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-30T13:34:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:34:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-30T13:34:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-30T13:34:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T13:34:53.616 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T13:34:53.616 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/34 2024-03-30T13:34:53.617 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/34 2024-03-30T13:34:53.617 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 3/34 2024-03-30T13:34:53.617 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 4/34 2024-03-30T13:34:53.617 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/34 2024-03-30T13:34:53.617 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_6 6/34 2024-03-30T13:34:53.617 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 7/34 2024-03-30T13:34:53.617 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1967.g4e50bbcd.el9. 8/34 2024-03-30T13:34:53.617 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x8 9/34 2024-03-30T13:34:53.617 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 10/34 2024-03-30T13:34:53.617 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 11/34 2024-03-30T13:34:53.617 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 12/34 2024-03-30T13:34:53.617 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 13/34 2024-03-30T13:34:53.617 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-30T13:34:53.617 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-30T13:34:53.618 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-30T13:34:53.618 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-30T13:34:53.618 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-30T13:34:53.618 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-30T13:34:53.618 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-30T13:34:53.618 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-30T13:34:53.618 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-30T13:34:53.618 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-30T13:34:53.618 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-30T13:34:53.618 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-30T13:34:53.618 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-30T13:34:53.618 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-30T13:34:53.619 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-30T13:34:53.619 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-30T13:34:53.619 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-30T13:34:53.619 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 31/34 2024-03-30T13:34:53.619 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-30T13:34:53.619 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 33/34 2024-03-30T13:34:54.500 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T13:34:54.500 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:34:54.500 INFO:teuthology.orchestra.run.smithi043.stdout:Upgraded: 2024-03-30T13:34:54.500 INFO:teuthology.orchestra.run.smithi043.stdout: librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:34:54.500 INFO:teuthology.orchestra.run.smithi043.stdout: librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:34:54.500 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-03-30T13:34:54.500 INFO:teuthology.orchestra.run.smithi043.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-30T13:34:54.500 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:34:54.500 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:34:54.500 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:34:54.500 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:34:54.500 INFO:teuthology.orchestra.run.smithi043.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-30T13:34:54.501 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-30T13:34:54.501 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-30T13:34:54.501 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:34:54.501 INFO:teuthology.orchestra.run.smithi043.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-30T13:34:54.501 INFO:teuthology.orchestra.run.smithi043.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-30T13:34:54.501 INFO:teuthology.orchestra.run.smithi043.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-30T13:34:54.501 INFO:teuthology.orchestra.run.smithi043.stdout: libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:34:54.501 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-30T13:34:54.501 INFO:teuthology.orchestra.run.smithi043.stdout: librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:34:54.501 INFO:teuthology.orchestra.run.smithi043.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-30T13:34:54.501 INFO:teuthology.orchestra.run.smithi043.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-30T13:34:54.501 INFO:teuthology.orchestra.run.smithi043.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-30T13:34:54.501 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:34:54.502 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:34:54.502 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:34:54.502 INFO:teuthology.orchestra.run.smithi043.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-30T13:34:54.502 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:34:54.502 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:34:54.502 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:34:54.502 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-30T13:34:54.502 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-30T13:34:54.502 INFO:teuthology.orchestra.run.smithi043.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-30T13:34:54.502 INFO:teuthology.orchestra.run.smithi043.stdout: re2-1:20211101-3.el9.x86_64 2024-03-30T13:34:54.502 INFO:teuthology.orchestra.run.smithi043.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-30T13:34:54.502 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:34:54.502 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:34:54.794 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-test 2024-03-30T13:34:55.361 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:29 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:34:55.517 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:34:55.518 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:34:55.518 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-03-30T13:34:55.518 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:34:55.518 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-03-30T13:34:55.519 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-test x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 47 M 2024-03-30T13:34:55.519 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-03-30T13:34:55.519 INFO:teuthology.orchestra.run.smithi043.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-30T13:34:55.519 INFO:teuthology.orchestra.run.smithi043.stdout: libcephsqlite x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 163 k 2024-03-30T13:34:55.519 INFO:teuthology.orchestra.run.smithi043.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-30T13:34:55.519 INFO:teuthology.orchestra.run.smithi043.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-30T13:34:55.519 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-30T13:34:55.519 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:34:55.519 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-03-30T13:34:55.519 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:34:55.519 INFO:teuthology.orchestra.run.smithi043.stdout:Install 6 Packages 2024-03-30T13:34:55.519 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:34:55.520 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 48 M 2024-03-30T13:34:55.520 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 205 M 2024-03-30T13:34:55.520 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-03-30T13:34:55.813 INFO:teuthology.orchestra.run.smithi043.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-30T13:34:55.863 INFO:teuthology.orchestra.run.smithi043.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-30T13:34:55.896 INFO:teuthology.orchestra.run.smithi043.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-30T13:34:56.022 INFO:teuthology.orchestra.run.smithi043.stdout:(4/6): libcephsqlite-19.0.0-1967.g4e50bbcd.el9. 426 kB/s | 163 kB 00:00 2024-03-30T13:34:56.047 INFO:teuthology.orchestra.run.smithi043.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-30T13:34:58.497 INFO:teuthology.orchestra.run.smithi043.stdout:(6/6): ceph-test-19.0.0-1967.g4e50bbcd.el9.x86_ 17 MB/s | 47 MB 00:02 2024-03-30T13:34:58.500 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:34:58.500 INFO:teuthology.orchestra.run.smithi043.stdout:Total 16 MB/s | 48 MB 00:02 2024-03-30T13:34:58.516 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-03-30T13:34:58.538 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-03-30T13:34:58.538 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-03-30T13:34:58.941 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-03-30T13:34:58.942 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-03-30T13:34:59.531 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-03-30T13:34:59.606 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-30T13:34:59.650 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-30T13:34:59.690 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-30T13:34:59.738 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-30T13:34:59.758 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/6 2024-03-30T13:35:04.492 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/6 2024-03-30T13:35:04.577 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 6/6 2024-03-30T13:35:06.891 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 6/6 2024-03-30T13:35:06.891 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/6 2024-03-30T13:35:06.891 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/6 2024-03-30T13:35:06.891 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-30T13:35:06.891 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-30T13:35:06.891 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-30T13:35:07.315 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-30T13:35:07.315 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:07.315 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-03-30T13:35:07.315 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:35:07.315 INFO:teuthology.orchestra.run.smithi043.stdout: jq-1.6-16.el9.x86_64 2024-03-30T13:35:07.316 INFO:teuthology.orchestra.run.smithi043.stdout: libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:35:07.316 INFO:teuthology.orchestra.run.smithi043.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-30T13:35:07.316 INFO:teuthology.orchestra.run.smithi043.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-30T13:35:07.316 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-30T13:35:07.316 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:07.316 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:35:07.566 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph 2024-03-30T13:35:08.138 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:42 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:35:08.302 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:35:08.304 INFO:teuthology.orchestra.run.smithi043.stdout:========================================================================================= 2024-03-30T13:35:08.304 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-03-30T13:35:08.304 INFO:teuthology.orchestra.run.smithi043.stdout:========================================================================================= 2024-03-30T13:35:08.304 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-03-30T13:35:08.304 INFO:teuthology.orchestra.run.smithi043.stdout: ceph x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 6.6 k 2024-03-30T13:35:08.304 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-03-30T13:35:08.305 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mds x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 2.4 M 2024-03-30T13:35:08.305 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 1.5 M 2024-03-30T13:35:08.305 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 248 k 2024-03-30T13:35:08.305 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mon x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 4.7 M 2024-03-30T13:35:08.305 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-osd x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 17 M 2024-03-30T13:35:08.305 INFO:teuthology.orchestra.run.smithi043.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-30T13:35:08.305 INFO:teuthology.orchestra.run.smithi043.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-30T13:35:08.305 INFO:teuthology.orchestra.run.smithi043.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-30T13:35:08.305 INFO:teuthology.orchestra.run.smithi043.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-30T13:35:08.305 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-30T13:35:08.305 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-30T13:35:08.305 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-30T13:35:08.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-30T13:35:08.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-30T13:35:08.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-30T13:35:08.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-30T13:35:08.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-30T13:35:08.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-30T13:35:08.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-30T13:35:08.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-30T13:35:08.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-30T13:35:08.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-30T13:35:08.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-30T13:35:08.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-30T13:35:08.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-30T13:35:08.307 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-30T13:35:08.307 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-30T13:35:08.307 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-30T13:35:08.307 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-30T13:35:08.307 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-30T13:35:08.307 INFO:teuthology.orchestra.run.smithi043.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-30T13:35:08.307 INFO:teuthology.orchestra.run.smithi043.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-30T13:35:08.307 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-30T13:35:08.307 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-30T13:35:08.307 INFO:teuthology.orchestra.run.smithi043.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-30T13:35:08.307 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:08.307 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-03-30T13:35:08.307 INFO:teuthology.orchestra.run.smithi043.stdout:========================================================================================= 2024-03-30T13:35:08.307 INFO:teuthology.orchestra.run.smithi043.stdout:Install 36 Packages 2024-03-30T13:35:08.308 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:08.308 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 30 M 2024-03-30T13:35:08.308 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 107 M 2024-03-30T13:35:08.308 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-03-30T13:35:08.824 INFO:teuthology.orchestra.run.smithi043.stdout:(1/36): ceph-19.0.0-1967.g4e50bbcd.el9.x86_64.r 39 kB/s | 6.6 kB 00:00 2024-03-30T13:35:09.090 INFO:teuthology.orchestra.run.smithi043.stdout:(2/36): ceph-mds-19.0.0-1967.g4e50bbcd.el9.x86_ 5.4 MB/s | 2.4 MB 00:00 2024-03-30T13:35:09.416 INFO:teuthology.orchestra.run.smithi043.stdout:(3/36): ceph-mon-19.0.0-1967.g4e50bbcd.el9.x86_ 7.9 MB/s | 4.7 MB 00:00 2024-03-30T13:35:09.500 INFO:teuthology.orchestra.run.smithi043.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1967.g4e50 2.9 MB/s | 248 kB 00:00 2024-03-30T13:35:09.616 INFO:teuthology.orchestra.run.smithi043.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-30T13:35:09.658 INFO:teuthology.orchestra.run.smithi043.stdout:(6/36): ceph-mgr-19.0.0-1967.g4e50bbcd.el9.x86_ 1.5 MB/s | 1.5 MB 00:01 2024-03-30T13:35:09.692 INFO:teuthology.orchestra.run.smithi043.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-30T13:35:09.717 INFO:teuthology.orchestra.run.smithi043.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 540 kB/s | 32 kB 00:00 2024-03-30T13:35:09.743 INFO:teuthology.orchestra.run.smithi043.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 940 kB/s | 47 kB 00:00 2024-03-30T13:35:09.768 INFO:teuthology.orchestra.run.smithi043.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 768 kB/s | 39 kB 00:00 2024-03-30T13:35:09.793 INFO:teuthology.orchestra.run.smithi043.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-30T13:35:09.860 INFO:teuthology.orchestra.run.smithi043.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 13 MB/s | 1.1 MB 00:00 2024-03-30T13:35:09.885 INFO:teuthology.orchestra.run.smithi043.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-30T13:35:09.911 INFO:teuthology.orchestra.run.smithi043.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-30T13:35:09.936 INFO:teuthology.orchestra.run.smithi043.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-30T13:35:10.095 INFO:teuthology.orchestra.run.smithi043.stdout:(16/36): ceph-osd-19.0.0-1967.g4e50bbcd.el9.x86 17 MB/s | 17 MB 00:01 2024-03-30T13:35:10.195 INFO:teuthology.orchestra.run.smithi043.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 440 kB/s | 125 kB 00:00 2024-03-30T13:35:10.287 INFO:teuthology.orchestra.run.smithi043.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 64 kB/s | 22 kB 00:00 2024-03-30T13:35:10.612 INFO:teuthology.orchestra.run.smithi043.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 111 kB/s | 43 kB 00:00 2024-03-30T13:35:10.679 INFO:teuthology.orchestra.run.smithi043.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 259 kB/s | 151 kB 00:00 2024-03-30T13:35:10.779 INFO:teuthology.orchestra.run.smithi043.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-03-30T13:35:10.829 INFO:teuthology.orchestra.run.smithi043.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 317 kB/s | 172 kB 00:00 2024-03-30T13:35:10.879 INFO:teuthology.orchestra.run.smithi043.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 177 kB/s | 18 kB 00:00 2024-03-30T13:35:10.930 INFO:teuthology.orchestra.run.smithi043.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-03-30T13:35:10.963 INFO:teuthology.orchestra.run.smithi043.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.0 MB/s | 358 kB 00:00 2024-03-30T13:35:10.988 INFO:teuthology.orchestra.run.smithi043.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2024-03-30T13:35:11.030 INFO:teuthology.orchestra.run.smithi043.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-03-30T13:35:11.064 INFO:teuthology.orchestra.run.smithi043.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 461 kB/s | 46 kB 00:00 2024-03-30T13:35:11.114 INFO:teuthology.orchestra.run.smithi043.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 628 kB/s | 79 kB 00:00 2024-03-30T13:35:11.165 INFO:teuthology.orchestra.run.smithi043.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-30T13:35:11.223 INFO:teuthology.orchestra.run.smithi043.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-03-30T13:35:11.249 INFO:teuthology.orchestra.run.smithi043.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 673 kB/s | 90 kB 00:00 2024-03-30T13:35:11.274 INFO:teuthology.orchestra.run.smithi043.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 328 kB/s | 36 kB 00:00 2024-03-30T13:35:11.357 INFO:teuthology.orchestra.run.smithi043.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-30T13:35:11.382 INFO:teuthology.orchestra.run.smithi043.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 184 kB/s | 20 kB 00:00 2024-03-30T13:35:11.516 INFO:teuthology.orchestra.run.smithi043.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.6 MB/s | 427 kB 00:00 2024-03-30T13:35:11.521 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:35:11.521 INFO:teuthology.orchestra.run.smithi043.stdout:Total 9.3 MB/s | 30 MB 00:03 2024-03-30T13:35:11.547 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-03-30T13:35:11.593 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-03-30T13:35:11.593 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-03-30T13:35:12.024 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-03-30T13:35:12.025 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-03-30T13:35:13.439 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-03-30T13:35:13.502 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-30T13:35:13.544 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-30T13:35:13.572 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-30T13:35:13.647 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-30T13:35:13.704 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-30T13:35:13.737 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-30T13:35:13.832 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-30T13:35:13.918 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-30T13:35:13.965 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-30T13:35:14.043 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-30T13:35:14.100 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-30T13:35:14.145 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-30T13:35:14.195 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-30T13:35:14.260 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-30T13:35:14.454 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-30T13:35:14.499 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-30T13:35:14.546 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-30T13:35:14.581 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-30T13:35:14.611 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-30T13:35:14.691 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-30T13:35:14.729 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-30T13:35:14.770 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-30T13:35:14.811 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-30T13:35:14.846 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-30T13:35:14.889 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-30T13:35:14.981 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-30T13:35:15.028 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-30T13:35:15.102 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-30T13:35:15.242 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-30T13:35:15.366 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-30T13:35:15.527 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1967.g4e50bbcd.el9. 31/36 2024-03-30T13:35:15.549 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/36 2024-03-30T13:35:15.574 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/36 2024-03-30T13:35:15.574 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:35:15.574 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-30T13:35:15.574 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-30T13:35:15.574 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-30T13:35:15.574 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:17.100 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 33/36 2024-03-30T13:35:17.126 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 33/36 2024-03-30T13:35:17.126 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:35:17.126 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-30T13:35:17.126 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-30T13:35:17.126 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-30T13:35:17.126 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:17.601 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 34/36 2024-03-30T13:35:17.623 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 34/36 2024-03-30T13:35:17.624 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:35:17.624 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-30T13:35:17.624 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-30T13:35:17.624 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-30T13:35:17.624 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:17.860 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 35/36 2024-03-30T13:35:17.882 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 35/36 2024-03-30T13:35:17.883 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:35:17.883 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-30T13:35:17.883 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-30T13:35:17.883 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-30T13:35:17.883 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:17.961 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 36/36 2024-03-30T13:35:19.832 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 36/36 2024-03-30T13:35:19.832 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/36 2024-03-30T13:35:19.832 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/36 2024-03-30T13:35:19.832 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 3/36 2024-03-30T13:35:19.832 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 4/36 2024-03-30T13:35:19.832 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/36 2024-03-30T13:35:19.832 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1967.g4e50bbcd.el9. 6/36 2024-03-30T13:35:19.832 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-30T13:35:19.833 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-30T13:35:19.833 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-30T13:35:19.833 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-30T13:35:19.833 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-30T13:35:19.833 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-30T13:35:19.833 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-30T13:35:19.833 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-30T13:35:19.833 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-30T13:35:19.833 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-30T13:35:19.833 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-30T13:35:19.833 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-30T13:35:19.833 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-30T13:35:19.833 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-30T13:35:19.834 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-30T13:35:19.834 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-30T13:35:19.834 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-30T13:35:19.834 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-30T13:35:19.834 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-30T13:35:19.834 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-30T13:35:19.834 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-30T13:35:19.834 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-30T13:35:19.834 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-30T13:35:19.834 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-30T13:35:19.834 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-30T13:35:19.834 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-30T13:35:19.835 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-30T13:35:19.835 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-30T13:35:19.835 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-30T13:35:20.795 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-30T13:35:20.795 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:20.795 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-03-30T13:35:20.795 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:35:20.795 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:35:20.795 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:35:20.795 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:35:20.795 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:35:20.796 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:35:20.796 INFO:teuthology.orchestra.run.smithi043.stdout: lua-5.4.4-4.el9.x86_64 2024-03-30T13:35:20.796 INFO:teuthology.orchestra.run.smithi043.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-30T13:35:20.796 INFO:teuthology.orchestra.run.smithi043.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-30T13:35:20.796 INFO:teuthology.orchestra.run.smithi043.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-30T13:35:20.796 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-30T13:35:20.796 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-30T13:35:20.796 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-30T13:35:20.796 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-30T13:35:20.796 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-30T13:35:20.796 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-30T13:35:20.797 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-30T13:35:20.797 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-30T13:35:20.797 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-30T13:35:20.797 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-30T13:35:20.797 INFO:teuthology.orchestra.run.smithi043.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-30T13:35:20.797 INFO:teuthology.orchestra.run.smithi043.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-30T13:35:20.797 INFO:teuthology.orchestra.run.smithi043.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-30T13:35:20.797 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-30T13:35:20.797 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ply-3.11-14.el9.noarch 2024-03-30T13:35:20.797 INFO:teuthology.orchestra.run.smithi043.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-30T13:35:20.797 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-30T13:35:20.797 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-30T13:35:20.797 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-30T13:35:20.798 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-30T13:35:20.798 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-30T13:35:20.798 INFO:teuthology.orchestra.run.smithi043.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-30T13:35:20.798 INFO:teuthology.orchestra.run.smithi043.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-30T13:35:20.798 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-30T13:35:20.798 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-30T13:35:20.798 INFO:teuthology.orchestra.run.smithi043.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-30T13:35:20.798 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:20.798 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:35:21.172 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-base 2024-03-30T13:35:21.741 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:55 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:35:21.841 INFO:teuthology.orchestra.run.smithi043.stdout:Package ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:35:21.895 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:35:21.896 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-03-30T13:35:21.897 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:35:21.959 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install cephadm 2024-03-30T13:35:22.526 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:56 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:35:22.680 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:35:22.681 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:35:22.681 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-03-30T13:35:22.681 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:35:22.681 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-03-30T13:35:22.681 INFO:teuthology.orchestra.run.smithi043.stdout: cephadm noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 748 k 2024-03-30T13:35:22.681 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:22.682 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-03-30T13:35:22.682 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:35:22.682 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-03-30T13:35:22.682 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:22.682 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 748 k 2024-03-30T13:35:22.682 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 755 k 2024-03-30T13:35:22.682 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-03-30T13:35:23.325 INFO:teuthology.orchestra.run.smithi043.stdout:cephadm-19.0.0-1967.g4e50bbcd.el9.noarch.rpm 1.1 MB/s | 748 kB 00:00 2024-03-30T13:35:23.326 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:35:23.326 INFO:teuthology.orchestra.run.smithi043.stdout:Total 1.1 MB/s | 748 kB 00:00 2024-03-30T13:35:23.326 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-03-30T13:35:23.332 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-03-30T13:35:23.332 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-03-30T13:35:23.338 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-03-30T13:35:23.338 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-03-30T13:35:23.412 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-03-30T13:35:23.846 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T13:35:23.948 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T13:35:24.322 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T13:35:24.598 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T13:35:24.599 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:24.599 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-03-30T13:35:24.599 INFO:teuthology.orchestra.run.smithi043.stdout: cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:35:24.599 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:24.599 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:35:24.795 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-immutable-object-cache 2024-03-30T13:35:25.372 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:59 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:35:25.528 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:35:25.529 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:35:25.529 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repo Size 2024-03-30T13:35:25.529 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:35:25.529 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-03-30T13:35:25.529 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 145 k 2024-03-30T13:35:25.530 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:25.530 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-03-30T13:35:25.530 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:35:25.530 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-03-30T13:35:25.530 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:25.530 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 145 k 2024-03-30T13:35:25.530 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 435 k 2024-03-30T13:35:25.530 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-03-30T13:35:25.756 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-immutable-object-cache-19.0.0-1967.g4e50bb 641 kB/s | 145 kB 00:00 2024-03-30T13:35:25.756 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:35:25.757 INFO:teuthology.orchestra.run.smithi043.stdout:Total 638 kB/s | 145 kB 00:00 2024-03-30T13:35:25.757 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-03-30T13:35:25.765 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-03-30T13:35:25.765 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-03-30T13:35:25.819 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-03-30T13:35:25.819 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-03-30T13:35:25.962 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-03-30T13:35:26.017 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd. 1/1 2024-03-30T13:35:26.040 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd. 1/1 2024-03-30T13:35:26.040 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:35:26.040 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-30T13:35:26.040 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd. 1/1 2024-03-30T13:35:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:27.000 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-03-30T13:35:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:35:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:27.000 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:35:27.238 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-mgr 2024-03-30T13:35:27.812 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:01 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:35:27.912 INFO:teuthology.orchestra.run.smithi043.stdout:Package ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:35:27.968 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:35:27.969 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-03-30T13:35:27.969 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:35:28.034 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-mgr-dashboard 2024-03-30T13:35:28.605 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:02 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:35:28.762 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:35:28.763 INFO:teuthology.orchestra.run.smithi043.stdout:====================================================================================== 2024-03-30T13:35:28.763 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-03-30T13:35:28.763 INFO:teuthology.orchestra.run.smithi043.stdout:====================================================================================== 2024-03-30T13:35:28.763 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-03-30T13:35:28.764 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 3.5 M 2024-03-30T13:35:28.764 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-03-30T13:35:28.764 INFO:teuthology.orchestra.run.smithi043.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-30T13:35:28.764 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 31 k 2024-03-30T13:35:28.764 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 15 k 2024-03-30T13:35:28.764 INFO:teuthology.orchestra.run.smithi043.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-30T13:35:28.764 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-30T13:35:28.764 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-30T13:35:28.764 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-30T13:35:28.764 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-30T13:35:28.764 INFO:teuthology.orchestra.run.smithi043.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-30T13:35:28.764 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-30T13:35:28.765 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-30T13:35:28.765 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:28.765 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-03-30T13:35:28.765 INFO:teuthology.orchestra.run.smithi043.stdout:====================================================================================== 2024-03-30T13:35:28.765 INFO:teuthology.orchestra.run.smithi043.stdout:Install 12 Packages 2024-03-30T13:35:28.765 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:28.765 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 8.6 M 2024-03-30T13:35:28.765 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 103 M 2024-03-30T13:35:28.765 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-03-30T13:35:29.276 INFO:teuthology.orchestra.run.smithi043.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1967.g4e5 81 kB/s | 15 kB 00:00 2024-03-30T13:35:29.310 INFO:teuthology.orchestra.run.smithi043.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1967.g4e 142 kB/s | 31 kB 00:00 2024-03-30T13:35:29.469 INFO:teuthology.orchestra.run.smithi043.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-30T13:35:29.510 INFO:teuthology.orchestra.run.smithi043.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-03-30T13:35:29.569 INFO:teuthology.orchestra.run.smithi043.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1967.g4e50bbc 7.3 MB/s | 3.5 MB 00:00 2024-03-30T13:35:29.861 INFO:teuthology.orchestra.run.smithi043.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.2 MB/s | 863 kB 00:00 2024-03-30T13:35:29.928 INFO:teuthology.orchestra.run.smithi043.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 54 kB/s | 19 kB 00:00 2024-03-30T13:35:30.203 INFO:teuthology.orchestra.run.smithi043.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 792 kB/s | 548 kB 00:00 2024-03-30T13:35:30.228 INFO:teuthology.orchestra.run.smithi043.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 481 kB/s | 144 kB 00:00 2024-03-30T13:35:30.304 INFO:teuthology.orchestra.run.smithi043.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 306 kB/s | 31 kB 00:00 2024-03-30T13:35:30.429 INFO:teuthology.orchestra.run.smithi043.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 942 kB/s | 188 kB 00:00 2024-03-30T13:35:30.721 INFO:teuthology.orchestra.run.smithi043.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.4 MB/s | 2.0 MB 00:00 2024-03-30T13:35:30.727 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:35:30.728 INFO:teuthology.orchestra.run.smithi043.stdout:Total 4.4 MB/s | 8.6 MB 00:01 2024-03-30T13:35:30.747 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-03-30T13:35:30.765 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-03-30T13:35:30.765 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-03-30T13:35:30.969 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-03-30T13:35:30.969 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-03-30T13:35:31.493 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-03-30T13:35:31.572 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-30T13:35:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-30T13:35:31.654 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-30T13:35:31.684 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-30T13:35:31.792 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-30T13:35:31.924 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-30T13:35:32.017 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-30T13:35:32.064 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-30T13:35:32.091 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-30T13:35:32.123 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1967.g4e50bbcd.el9 10/12 2024-03-30T13:35:33.226 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1967.g4e50bbcd.el 11/12 2024-03-30T13:35:33.265 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noa 12/12 2024-03-30T13:35:34.532 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noa 12/12 2024-03-30T13:35:34.532 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1967.g4e50bbcd.el 1/12 2024-03-30T13:35:34.532 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noa 2/12 2024-03-30T13:35:34.532 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1967.g4e50bbcd.el9 3/12 2024-03-30T13:35:34.532 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-30T13:35:34.532 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-30T13:35:34.532 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-30T13:35:34.532 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-30T13:35:34.532 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-30T13:35:34.532 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-30T13:35:34.533 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-30T13:35:34.533 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-30T13:35:34.947 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-30T13:35:34.948 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:34.948 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-03-30T13:35:34.948 INFO:teuthology.orchestra.run.smithi043.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-30T13:35:34.948 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-grafana-dashboards-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:35:34.948 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:35:34.948 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-prometheus-alerts-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:35:34.948 INFO:teuthology.orchestra.run.smithi043.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-30T13:35:34.948 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-30T13:35:34.948 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-30T13:35:34.948 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-30T13:35:34.948 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-30T13:35:34.948 INFO:teuthology.orchestra.run.smithi043.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-30T13:35:34.949 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-30T13:35:34.949 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-30T13:35:34.949 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:34.949 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:35:35.210 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-30T13:35:35.779 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:09 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:35:35.937 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:35:35.938 INFO:teuthology.orchestra.run.smithi043.stdout:============================================================================================ 2024-03-30T13:35:35.938 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-03-30T13:35:35.939 INFO:teuthology.orchestra.run.smithi043.stdout:============================================================================================ 2024-03-30T13:35:35.939 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-03-30T13:35:35.939 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 7.4 M 2024-03-30T13:35:35.939 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-03-30T13:35:35.939 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-30T13:35:35.939 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-30T13:35:35.939 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-30T13:35:35.939 INFO:teuthology.orchestra.run.smithi043.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-30T13:35:35.939 INFO:teuthology.orchestra.run.smithi043.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-30T13:35:35.939 INFO:teuthology.orchestra.run.smithi043.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-30T13:35:35.939 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-30T13:35:35.940 INFO:teuthology.orchestra.run.smithi043.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-30T13:35:35.940 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-30T13:35:35.940 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-30T13:35:35.940 INFO:teuthology.orchestra.run.smithi043.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-30T13:35:35.940 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:35.940 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-03-30T13:35:35.940 INFO:teuthology.orchestra.run.smithi043.stdout:============================================================================================ 2024-03-30T13:35:35.940 INFO:teuthology.orchestra.run.smithi043.stdout:Install 12 Packages 2024-03-30T13:35:35.940 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:35.941 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 37 M 2024-03-30T13:35:35.941 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 209 M 2024-03-30T13:35:35.941 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-03-30T13:35:36.228 INFO:teuthology.orchestra.run.smithi043.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 202 kB/s | 32 kB 00:00 2024-03-30T13:35:36.262 INFO:teuthology.orchestra.run.smithi043.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 536 kB/s | 17 kB 00:00 2024-03-30T13:35:36.304 INFO:teuthology.orchestra.run.smithi043.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 833 kB/s | 34 kB 00:00 2024-03-30T13:35:36.371 INFO:teuthology.orchestra.run.smithi043.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-30T13:35:36.696 INFO:teuthology.orchestra.run.smithi043.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2024-03-30T13:35:36.772 INFO:teuthology.orchestra.run.smithi043.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 4.8 MB 00:00 2024-03-30T13:35:36.797 INFO:teuthology.orchestra.run.smithi043.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-30T13:35:37.339 INFO:teuthology.orchestra.run.smithi043.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-03-30T13:35:37.564 INFO:teuthology.orchestra.run.smithi043.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 870 kB/s | 195 kB 00:00 2024-03-30T13:35:37.589 INFO:teuthology.orchestra.run.smithi043.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-03-30T13:35:37.748 INFO:teuthology.orchestra.run.smithi043.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-30T13:35:38.415 INFO:teuthology.orchestra.run.smithi043.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 3.2 MB/s | 7.4 MB 00:02 2024-03-30T13:35:38.417 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:35:38.417 INFO:teuthology.orchestra.run.smithi043.stdout:Total 15 MB/s | 37 MB 00:02 2024-03-30T13:35:38.462 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-03-30T13:35:38.487 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-03-30T13:35:38.487 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-03-30T13:35:38.865 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-03-30T13:35:38.865 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-03-30T13:35:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-03-30T13:35:39.671 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-30T13:35:39.702 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-30T13:35:39.800 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-30T13:35:39.847 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-30T13:35:40.031 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-30T13:35:40.063 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-30T13:35:40.141 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-30T13:35:40.829 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-30T13:35:40.938 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-30T13:35:42.231 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-30T13:35:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-30T13:35:43.336 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50b 12/12 2024-03-30T13:35:45.875 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50b 12/12 2024-03-30T13:35:45.875 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50b 1/12 2024-03-30T13:35:45.875 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-30T13:35:45.875 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-30T13:35:45.875 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-30T13:35:45.875 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-30T13:35:45.875 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-30T13:35:45.876 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-30T13:35:45.876 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-30T13:35:45.876 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-30T13:35:45.876 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-30T13:35:45.876 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-30T13:35:46.281 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-30T13:35:46.282 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:46.282 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-03-30T13:35:46.282 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:35:46.282 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-30T13:35:46.282 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-30T13:35:46.282 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-30T13:35:46.282 INFO:teuthology.orchestra.run.smithi043.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-30T13:35:46.282 INFO:teuthology.orchestra.run.smithi043.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-30T13:35:46.282 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-30T13:35:46.282 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-30T13:35:46.283 INFO:teuthology.orchestra.run.smithi043.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-30T13:35:46.283 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-30T13:35:46.283 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-30T13:35:46.283 INFO:teuthology.orchestra.run.smithi043.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-30T13:35:46.283 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:46.283 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:35:46.626 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-mgr-rook 2024-03-30T13:35:47.202 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:21 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:35:47.361 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:35:47.362 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================== 2024-03-30T13:35:47.362 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-03-30T13:35:47.362 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================== 2024-03-30T13:35:47.362 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-03-30T13:35:47.362 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-rook noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 49 k 2024-03-30T13:35:47.362 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-03-30T13:35:47.363 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-30T13:35:47.363 INFO:teuthology.orchestra.run.smithi043.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-30T13:35:47.363 INFO:teuthology.orchestra.run.smithi043.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-30T13:35:47.363 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-30T13:35:47.363 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-30T13:35:47.363 INFO:teuthology.orchestra.run.smithi043.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-30T13:35:47.363 INFO:teuthology.orchestra.run.smithi043.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-30T13:35:47.363 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-30T13:35:47.363 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-30T13:35:47.363 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-30T13:35:47.363 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-30T13:35:47.363 INFO:teuthology.orchestra.run.smithi043.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-30T13:35:47.364 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:47.364 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-03-30T13:35:47.364 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================== 2024-03-30T13:35:47.364 INFO:teuthology.orchestra.run.smithi043.stdout:Install 13 Packages 2024-03-30T13:35:47.364 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:47.364 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 2.0 M 2024-03-30T13:35:47.364 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 26 M 2024-03-30T13:35:47.364 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-03-30T13:35:47.847 INFO:teuthology.orchestra.run.smithi043.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 208 kB/s | 24 kB 00:00 2024-03-30T13:35:47.872 INFO:teuthology.orchestra.run.smithi043.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 127 kB/s | 18 kB 00:00 2024-03-30T13:35:47.922 INFO:teuthology.orchestra.run.smithi043.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-30T13:35:47.947 INFO:teuthology.orchestra.run.smithi043.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-30T13:35:47.972 INFO:teuthology.orchestra.run.smithi043.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-30T13:35:47.998 INFO:teuthology.orchestra.run.smithi043.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-03-30T13:35:48.056 INFO:teuthology.orchestra.run.smithi043.stdout:(7/13): ceph-mgr-rook-19.0.0-1967.g4e50bbcd.el9 151 kB/s | 49 kB 00:00 2024-03-30T13:35:48.173 INFO:teuthology.orchestra.run.smithi043.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 161 kB/s | 32 kB 00:00 2024-03-30T13:35:48.198 INFO:teuthology.orchestra.run.smithi043.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 75 kB/s | 15 kB 00:00 2024-03-30T13:35:48.325 INFO:teuthology.orchestra.run.smithi043.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 473 kB/s | 59 kB 00:00 2024-03-30T13:35:48.357 INFO:teuthology.orchestra.run.smithi043.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 674 kB/s | 203 kB 00:00 2024-03-30T13:35:48.441 INFO:teuthology.orchestra.run.smithi043.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.8 MB/s | 1.0 MB 00:00 2024-03-30T13:35:48.466 INFO:teuthology.orchestra.run.smithi043.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 631 kB/s | 90 kB 00:00 2024-03-30T13:35:48.468 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:35:48.469 INFO:teuthology.orchestra.run.smithi043.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-03-30T13:35:48.469 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-03-30T13:35:48.485 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-03-30T13:35:48.485 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-03-30T13:35:48.589 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-03-30T13:35:48.590 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-03-30T13:35:49.108 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-03-30T13:35:49.277 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-30T13:35:49.326 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-30T13:35:49.399 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-30T13:35:49.435 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-30T13:35:49.477 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-30T13:35:49.574 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-30T13:35:49.692 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-30T13:35:49.746 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-30T13:35:50.503 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-30T13:35:50.550 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-30T13:35:50.592 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-30T13:35:50.641 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-30T13:35:50.670 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 13/13 2024-03-30T13:35:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 13/13 2024-03-30T13:35:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/13 2024-03-30T13:35:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-30T13:35:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-30T13:35:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-30T13:35:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-30T13:35:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-30T13:35:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-30T13:35:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-30T13:35:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-30T13:35:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-30T13:35:51.476 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-30T13:35:51.476 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-30T13:35:51.822 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-30T13:35:51.822 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:51.822 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-03-30T13:35:51.822 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:35:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-30T13:35:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-30T13:35:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-30T13:35:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-30T13:35:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-30T13:35:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-30T13:35:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-30T13:35:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-30T13:35:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-30T13:35:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-30T13:35:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-30T13:35:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-30T13:35:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:51.823 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:35:52.039 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-mgr-cephadm 2024-03-30T13:35:52.615 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:26 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:35:52.774 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:35:52.775 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================== 2024-03-30T13:35:52.775 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-03-30T13:35:52.775 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================== 2024-03-30T13:35:52.775 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-03-30T13:35:52.775 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 139 k 2024-03-30T13:35:52.775 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-03-30T13:35:52.775 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-30T13:35:52.775 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-30T13:35:52.775 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-30T13:35:52.775 INFO:teuthology.orchestra.run.smithi043.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-30T13:35:52.776 INFO:teuthology.orchestra.run.smithi043.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-30T13:35:52.776 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:52.776 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-03-30T13:35:52.776 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================== 2024-03-30T13:35:52.776 INFO:teuthology.orchestra.run.smithi043.stdout:Install 6 Packages 2024-03-30T13:35:52.776 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:52.777 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 6.8 M 2024-03-30T13:35:52.777 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 33 M 2024-03-30T13:35:52.777 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-03-30T13:35:53.301 INFO:teuthology.orchestra.run.smithi043.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 894 kB/s | 248 kB 00:00 2024-03-30T13:35:53.418 INFO:teuthology.orchestra.run.smithi043.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2024-03-30T13:35:53.485 INFO:teuthology.orchestra.run.smithi043.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1967.g4e50bbcd.e 302 kB/s | 139 kB 00:00 2024-03-30T13:35:53.702 INFO:teuthology.orchestra.run.smithi043.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 223 kB/s | 58 kB 00:00 2024-03-30T13:35:53.744 INFO:teuthology.orchestra.run.smithi043.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 219 kB/s | 51 kB 00:00 2024-03-30T13:35:53.802 INFO:teuthology.orchestra.run.smithi043.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-30T13:35:53.806 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:35:53.806 INFO:teuthology.orchestra.run.smithi043.stdout:Total 6.6 MB/s | 6.8 MB 00:01 2024-03-30T13:35:53.819 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-03-30T13:35:53.829 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-03-30T13:35:53.829 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-03-30T13:35:53.921 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-03-30T13:35:53.922 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-03-30T13:35:54.244 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-03-30T13:35:54.431 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-30T13:35:54.479 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-30T13:35:55.031 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-30T13:35:55.118 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-30T13:35:55.188 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-30T13:35:55.216 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 6/6 2024-03-30T13:35:56.000 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 6/6 2024-03-30T13:35:56.000 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/6 2024-03-30T13:35:56.000 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-30T13:35:56.000 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-30T13:35:56.001 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-30T13:35:56.001 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-30T13:35:56.295 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-30T13:35:56.295 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:56.295 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-03-30T13:35:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:35:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-30T13:35:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-30T13:35:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-30T13:35:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-30T13:35:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-30T13:35:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:56.296 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:35:56.519 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-fuse 2024-03-30T13:35:57.091 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:31 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:35:57.247 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:35:57.248 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:35:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-03-30T13:35:57.248 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:35:57.248 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-03-30T13:35:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-fuse x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 889 k 2024-03-30T13:35:57.248 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-03-30T13:35:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-30T13:35:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:57.248 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-03-30T13:35:57.248 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:35:57.248 INFO:teuthology.orchestra.run.smithi043.stdout:Install 2 Packages 2024-03-30T13:35:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:57.249 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 968 k 2024-03-30T13:35:57.249 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 2.8 M 2024-03-30T13:35:57.249 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-03-30T13:35:57.522 INFO:teuthology.orchestra.run.smithi043.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 288 kB/s | 79 kB 00:00 2024-03-30T13:35:57.805 INFO:teuthology.orchestra.run.smithi043.stdout:(2/2): ceph-fuse-19.0.0-1967.g4e50bbcd.el9.x86_ 1.6 MB/s | 889 kB 00:00 2024-03-30T13:35:57.806 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:35:57.807 INFO:teuthology.orchestra.run.smithi043.stdout:Total 1.7 MB/s | 968 kB 00:00 2024-03-30T13:35:57.807 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-03-30T13:35:57.829 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-03-30T13:35:57.830 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-03-30T13:35:57.906 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-03-30T13:35:57.906 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-03-30T13:35:58.096 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-03-30T13:35:58.236 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-30T13:35:58.317 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/2 2024-03-30T13:35:58.975 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/2 2024-03-30T13:35:58.976 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/2 2024-03-30T13:35:59.277 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-30T13:35:59.277 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:59.277 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-03-30T13:35:59.277 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-30T13:35:59.277 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:35:59.277 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:35:59.534 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-volume 2024-03-30T13:36:00.120 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:34 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:36:00.281 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:36:00.282 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:36:00.282 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-03-30T13:36:00.282 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:36:00.282 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-03-30T13:36:00.282 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-volume noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 264 k 2024-03-30T13:36:00.282 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-03-30T13:36:00.282 INFO:teuthology.orchestra.run.smithi043.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-30T13:36:00.282 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:00.282 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-03-30T13:36:00.283 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:36:00.283 INFO:teuthology.orchestra.run.smithi043.stdout:Install 2 Packages 2024-03-30T13:36:00.283 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:00.283 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 332 k 2024-03-30T13:36:00.283 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 1.5 M 2024-03-30T13:36:00.283 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-03-30T13:36:00.419 INFO:teuthology.orchestra.run.smithi043.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 508 kB/s | 69 kB 00:00 2024-03-30T13:36:00.544 INFO:teuthology.orchestra.run.smithi043.stdout:(2/2): ceph-volume-19.0.0-1967.g4e50bbcd.el9.no 1.0 MB/s | 264 kB 00:00 2024-03-30T13:36:00.545 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:36:00.546 INFO:teuthology.orchestra.run.smithi043.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-30T13:36:00.546 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-03-30T13:36:00.555 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-03-30T13:36:00.556 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-03-30T13:36:00.598 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-03-30T13:36:00.598 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-03-30T13:36:00.753 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-03-30T13:36:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-30T13:36:00.996 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 2/2 2024-03-30T13:36:01.013 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 2/2 2024-03-30T13:36:01.013 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:36:01.013 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-30T13:36:01.013 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:01.706 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/2 2024-03-30T13:36:02.058 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-30T13:36:02.058 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:02.058 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-03-30T13:36:02.058 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:36:02.058 INFO:teuthology.orchestra.run.smithi043.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-30T13:36:02.058 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:02.058 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:36:02.375 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install librados-devel 2024-03-30T13:36:02.954 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:36 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:36:03.112 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:36:03.113 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:36:03.113 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repo Size 2024-03-30T13:36:03.113 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:36:03.113 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-03-30T13:36:03.113 INFO:teuthology.orchestra.run.smithi043.stdout: librados-devel x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 127 k 2024-03-30T13:36:03.113 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:03.113 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-03-30T13:36:03.113 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:36:03.113 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-03-30T13:36:03.113 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:03.113 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 127 k 2024-03-30T13:36:03.114 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 456 k 2024-03-30T13:36:03.114 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-03-30T13:36:03.441 INFO:teuthology.orchestra.run.smithi043.stdout:librados-devel-19.0.0-1967.g4e50bbcd.el9.x86_64 388 kB/s | 127 kB 00:00 2024-03-30T13:36:03.442 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:36:03.442 INFO:teuthology.orchestra.run.smithi043.stdout:Total 386 kB/s | 127 kB 00:00 2024-03-30T13:36:03.442 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-03-30T13:36:03.448 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-03-30T13:36:03.448 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-03-30T13:36:03.506 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-03-30T13:36:03.506 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-03-30T13:36:03.771 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-03-30T13:36:03.884 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:36:04.319 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:36:04.626 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:36:04.626 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:04.626 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-03-30T13:36:04.626 INFO:teuthology.orchestra.run.smithi043.stdout: librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:36:04.626 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:04.626 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:36:04.851 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install libcephfs2 2024-03-30T13:36:05.432 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:39 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:36:05.534 INFO:teuthology.orchestra.run.smithi043.stdout:Package libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:36:05.588 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:36:05.589 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-03-30T13:36:05.589 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:36:05.647 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install libcephfs-devel 2024-03-30T13:36:06.216 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:40 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:36:06.374 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:36:06.375 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:36:06.375 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repo Size 2024-03-30T13:36:06.375 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:36:06.375 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-03-30T13:36:06.375 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-devel x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 32 k 2024-03-30T13:36:06.375 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:06.375 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-03-30T13:36:06.376 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:36:06.376 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-03-30T13:36:06.376 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:06.376 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 32 k 2024-03-30T13:36:06.376 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 142 k 2024-03-30T13:36:06.376 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-03-30T13:36:06.710 INFO:teuthology.orchestra.run.smithi043.stdout:libcephfs-devel-19.0.0-1967.g4e50bbcd.el9.x86_6 95 kB/s | 32 kB 00:00 2024-03-30T13:36:06.711 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:36:06.711 INFO:teuthology.orchestra.run.smithi043.stdout:Total 94 kB/s | 32 kB 00:00 2024-03-30T13:36:06.711 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-03-30T13:36:06.715 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-03-30T13:36:06.715 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-03-30T13:36:06.736 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-03-30T13:36:06.737 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-03-30T13:36:06.832 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-03-30T13:36:06.945 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:36:07.286 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:36:07.542 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:36:07.543 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:07.543 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-03-30T13:36:07.543 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:36:07.543 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:07.543 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:36:07.709 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install librados2 2024-03-30T13:36:08.283 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:42 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:36:08.386 INFO:teuthology.orchestra.run.smithi043.stdout:Package librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:36:08.440 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:36:08.442 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-03-30T13:36:08.442 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:36:08.505 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install librbd1 2024-03-30T13:36:09.080 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:43 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:36:09.183 INFO:teuthology.orchestra.run.smithi043.stdout:Package librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:36:09.239 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:36:09.239 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-03-30T13:36:09.239 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:36:09.301 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install python3-rados 2024-03-30T13:36:09.882 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:43 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:36:09.985 INFO:teuthology.orchestra.run.smithi043.stdout:Package python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:36:10.039 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:36:10.040 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-03-30T13:36:10.040 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:36:10.105 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install python3-rgw 2024-03-30T13:36:10.685 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:44 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:36:10.787 INFO:teuthology.orchestra.run.smithi043.stdout:Package python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:36:10.842 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:36:10.843 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-03-30T13:36:10.843 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:36:10.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install python3-cephfs 2024-03-30T13:36:11.476 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:45 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:36:11.578 INFO:teuthology.orchestra.run.smithi043.stdout:Package python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:36:11.632 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:36:11.633 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-03-30T13:36:11.634 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:36:11.704 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install python3-rbd 2024-03-30T13:36:12.280 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:46 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:36:12.382 INFO:teuthology.orchestra.run.smithi043.stdout:Package python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:36:12.436 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:36:12.437 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-03-30T13:36:12.437 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:36:12.501 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install rbd-fuse 2024-03-30T13:36:13.072 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:47 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:36:13.228 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:36:13.229 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:36:13.229 INFO:teuthology.orchestra.run.smithi043.stdout: Package Architecture Version Repository Size 2024-03-30T13:36:13.229 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:36:13.229 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-03-30T13:36:13.229 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-fuse x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 85 k 2024-03-30T13:36:13.230 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:13.230 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-03-30T13:36:13.230 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:36:13.230 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-03-30T13:36:13.230 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:13.230 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 85 k 2024-03-30T13:36:13.230 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 231 k 2024-03-30T13:36:13.230 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-03-30T13:36:13.555 INFO:teuthology.orchestra.run.smithi043.stdout:rbd-fuse-19.0.0-1967.g4e50bbcd.el9.x86_64.rpm 262 kB/s | 85 kB 00:00 2024-03-30T13:36:13.556 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:36:13.556 INFO:teuthology.orchestra.run.smithi043.stdout:Total 261 kB/s | 85 kB 00:00 2024-03-30T13:36:13.556 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-03-30T13:36:13.564 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-03-30T13:36:13.564 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-03-30T13:36:13.623 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-03-30T13:36:13.623 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-03-30T13:36:13.772 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-03-30T13:36:13.874 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:36:14.236 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:36:14.566 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:36:14.566 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:14.566 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-03-30T13:36:14.566 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:36:14.566 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:14.567 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:36:14.839 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install rbd-mirror 2024-03-30T13:36:15.428 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:49 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:36:15.588 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:36:15.588 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:36:15.588 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repo Size 2024-03-30T13:36:15.588 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:36:15.588 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-03-30T13:36:15.588 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-mirror x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 3.1 M 2024-03-30T13:36:15.589 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:15.589 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-03-30T13:36:15.589 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:36:15.589 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-03-30T13:36:15.589 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:15.589 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 3.1 M 2024-03-30T13:36:15.589 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 13 M 2024-03-30T13:36:15.590 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-03-30T13:36:16.034 INFO:teuthology.orchestra.run.smithi043.stdout:rbd-mirror-19.0.0-1967.g4e50bbcd.el9.x86_64.rpm 7.0 MB/s | 3.1 MB 00:00 2024-03-30T13:36:16.034 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:36:16.035 INFO:teuthology.orchestra.run.smithi043.stdout:Total 6.9 MB/s | 3.1 MB 00:00 2024-03-30T13:36:16.035 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-03-30T13:36:16.046 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-03-30T13:36:16.046 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-03-30T13:36:16.125 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-03-30T13:36:16.125 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-03-30T13:36:16.570 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-03-30T13:36:16.626 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:36:16.651 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:36:16.651 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:36:16.651 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-30T13:36:16.651 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-30T13:36:16.652 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-30T13:36:16.652 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:17.801 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:36:17.802 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:17.802 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-03-30T13:36:17.802 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:36:17.802 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:17.802 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:36:18.020 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install rbd-nbd 2024-03-30T13:36:18.600 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:52 ago on Sat 30 Mar 2024 01:34:26 PM UTC. 2024-03-30T13:36:18.755 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-30T13:36:18.756 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:36:18.756 INFO:teuthology.orchestra.run.smithi043.stdout: Package Architecture Version Repository Size 2024-03-30T13:36:18.756 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:36:18.756 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-03-30T13:36:18.756 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-nbd x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 170 k 2024-03-30T13:36:18.757 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:18.757 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-03-30T13:36:18.757 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-30T13:36:18.757 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-03-30T13:36:18.757 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:18.757 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 170 k 2024-03-30T13:36:18.757 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 494 k 2024-03-30T13:36:18.757 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-03-30T13:36:19.154 INFO:teuthology.orchestra.run.smithi043.stdout:rbd-nbd-19.0.0-1967.g4e50bbcd.el9.x86_64.rpm 429 kB/s | 170 kB 00:00 2024-03-30T13:36:19.155 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:36:19.155 INFO:teuthology.orchestra.run.smithi043.stdout:Total 428 kB/s | 170 kB 00:00 2024-03-30T13:36:19.155 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-03-30T13:36:19.163 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-03-30T13:36:19.163 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-03-30T13:36:19.225 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-03-30T13:36:19.226 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-03-30T13:36:19.387 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-03-30T13:36:19.496 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:36:19.889 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:36:20.148 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:36:20.149 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:20.149 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-03-30T13:36:20.149 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:36:20.149 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:36:20.149 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-30T13:36:20.378 DEBUG:teuthology.parallel:result is None 2024-03-30T13:36:20.379 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T13:36:20.718 DEBUG:teuthology.orchestra.run.smithi043:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-30T13:36:20.740 INFO:teuthology.orchestra.run.smithi043.stdout:19.0.0-1967.g4e50bbcd.el9 2024-03-30T13:36:20.740 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1967.g4e50bbcd.el9 2024-03-30T13:36:20.740 INFO:teuthology.task.install:The correct ceph version 19.0.0-1967.g4e50bbcd is installed. 2024-03-30T13:36:20.742 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T13:36:20.881 DEBUG:teuthology.orchestra.run.smithi113:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-30T13:36:20.906 INFO:teuthology.orchestra.run.smithi113.stdout:19.0.0-1967.g4e50bbcd.el9 2024-03-30T13:36:20.906 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1967.g4e50bbcd.el9 2024-03-30T13:36:20.906 INFO:teuthology.task.install:The correct ceph version 19.0.0-1967.g4e50bbcd is installed. 2024-03-30T13:36:20.908 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-30T13:36:20.908 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-30T13:36:20.908 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T13:36:20.938 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-30T13:36:20.938 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T13:36:20.978 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-30T13:36:20.978 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-30T13:36:20.978 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T13:36:21.015 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T13:36:21.083 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-30T13:36:21.083 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T13:36:21.114 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T13:36:21.183 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-30T13:36:21.184 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-30T13:36:21.184 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T13:36:21.210 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T13:36:21.278 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-30T13:36:21.278 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T13:36:21.309 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T13:36:21.378 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-30T13:36:21.378 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-30T13:36:21.378 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T13:36:21.404 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T13:36:21.471 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-30T13:36:21.471 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T13:36:21.498 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T13:36:21.566 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-30T13:36:21.664 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb', 'cephadm_mode': 'root'} 2024-03-30T13:36:21.664 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T13:36:21.665 INFO:tasks.cephadm:Cluster fsid is 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:36:21.665 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-30T13:36:21.665 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.43', 'mon.c': '[v2:172.21.15.43:3301,v1:172.21.15.43:6790]', 'mon.b': '172.21.15.113'} 2024-03-30T13:36:21.665 INFO:tasks.cephadm:First mon is mon.a on smithi043 2024-03-30T13:36:21.665 INFO:tasks.cephadm:First mgr is y 2024-03-30T13:36:21.665 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-30T13:36:21.665 DEBUG:teuthology.orchestra.run.smithi043:> sudo hostname $(hostname -s) 2024-03-30T13:36:21.693 DEBUG:teuthology.orchestra.run.smithi113:> sudo hostname $(hostname -s) 2024-03-30T13:36:21.720 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-30T13:36:21.720 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T13:36:22.105 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54746/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1967-g4e50bbcd', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1967.g4e50bbcd'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-29 16:06:09.870176', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-30T13:36:22.476 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 4e50bbcdd1630c7760b708f3a5b4d38a121e66cb from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T13:36:22.478 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/x86_64/flavors/default/cephadm 2024-03-30T13:36:22.478 DEBUG:teuthology.orchestra.run.smithi043:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-30T13:36:23.221 INFO:teuthology.orchestra.run.smithi043.stdout:-rw-r--r--. 1 ubuntu ubuntu 767434 Mar 30 13:36 /home/ubuntu/cephtest/cephadm 2024-03-30T13:36:23.221 DEBUG:teuthology.orchestra.run.smithi113:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-30T13:36:24.087 INFO:teuthology.orchestra.run.smithi113.stdout:-rw-r--r--. 1 ubuntu ubuntu 767434 Mar 30 13:36 /home/ubuntu/cephtest/cephadm 2024-03-30T13:36:24.088 DEBUG:teuthology.orchestra.run.smithi043:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-30T13:36:24.106 DEBUG:teuthology.orchestra.run.smithi113:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-30T13:36:24.137 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb on all hosts... 2024-03-30T13:36:24.137 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb pull 2024-03-30T13:36:24.149 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb pull 2024-03-30T13:36:24.997 INFO:teuthology.orchestra.run.smithi043.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb... 2024-03-30T13:36:25.172 INFO:teuthology.orchestra.run.smithi113.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb... 2024-03-30T13:36:52.999 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:36:52.999 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_version": "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)", 2024-03-30T13:36:52.999 INFO:teuthology.orchestra.run.smithi113.stdout: "image_id": "b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e", 2024-03-30T13:36:52.999 INFO:teuthology.orchestra.run.smithi113.stdout: "repo_digests": [ 2024-03-30T13:36:52.999 INFO:teuthology.orchestra.run.smithi113.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d" 2024-03-30T13:36:52.999 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:36:52.999 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:36:58.090 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:36:58.091 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_version": "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)", 2024-03-30T13:36:58.091 INFO:teuthology.orchestra.run.smithi043.stdout: "image_id": "b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e", 2024-03-30T13:36:58.091 INFO:teuthology.orchestra.run.smithi043.stdout: "repo_digests": [ 2024-03-30T13:36:58.091 INFO:teuthology.orchestra.run.smithi043.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d" 2024-03-30T13:36:58.091 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:36:58.091 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:36:58.115 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /etc/ceph 2024-03-30T13:36:58.168 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /etc/ceph 2024-03-30T13:36:58.203 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 777 /etc/ceph 2024-03-30T13:36:58.229 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 777 /etc/ceph 2024-03-30T13:36:58.274 INFO:tasks.cephadm:Writing seed config... 2024-03-30T13:36:58.275 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-30T13:36:58.275 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-30T13:36:58.275 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-03-30T13:36:58.275 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-03-30T13:36:58.275 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-03-30T13:36:58.275 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-30T13:36:58.275 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-30T13:36:58.275 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-30T13:36:58.275 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-30T13:36:58.275 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-03-30T13:36:58.276 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-03-30T13:36:58.276 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-03-30T13:36:58.276 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-03-30T13:36:58.276 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-03-30T13:36:58.276 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-30T13:36:58.276 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-30T13:36:58.276 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-03-30T13:36:58.276 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-03-30T13:36:58.276 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-03-30T13:36:58.276 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-03-30T13:36:58.276 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-03-30T13:36:58.276 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-30T13:36:58.276 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-03-30T13:36:58.277 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-30T13:36:58.277 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-30T13:36:58.291 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-03-30T13:36:58.291 DEBUG:teuthology.orchestra.run.smithi043:mon.a> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mon.a.service 2024-03-30T13:36:58.333 DEBUG:teuthology.orchestra.run.smithi043:mgr.y> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mgr.y.service 2024-03-30T13:36:58.376 INFO:tasks.cephadm:Bootstrapping... 2024-03-30T13:36:58.376 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -v bootstrap --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.43 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-30T13:36:58.571 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:36:58.571 INFO:teuthology.orchestra.run.smithi043.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb', '-v', 'bootstrap', '--fsid', '7ee932fa-ee9a-11ee-b647-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.43', '--skip-admin-label'] 2024-03-30T13:36:58.571 INFO:teuthology.orchestra.run.smithi043.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-30T13:36:58.572 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying podman|docker is present... 2024-03-30T13:36:58.596 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-30T13:36:58.596 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying lvm2 is present... 2024-03-30T13:36:58.596 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying time synchronization is in place... 2024-03-30T13:36:58.603 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-30T13:36:58.603 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-30T13:36:58.609 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-30T13:36:58.609 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-03-30T13:36:58.615 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout enabled 2024-03-30T13:36:58.621 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout active 2024-03-30T13:36:58.621 INFO:teuthology.orchestra.run.smithi043.stdout:Unit chronyd.service is enabled and running 2024-03-30T13:36:58.621 INFO:teuthology.orchestra.run.smithi043.stdout:Repeating the final host check... 2024-03-30T13:36:58.644 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-30T13:36:58.644 INFO:teuthology.orchestra.run.smithi043.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-30T13:36:58.644 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl is present 2024-03-30T13:36:58.644 INFO:teuthology.orchestra.run.smithi043.stdout:lvcreate is present 2024-03-30T13:36:58.650 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-30T13:36:58.650 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-30T13:36:58.656 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-30T13:36:58.656 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-03-30T13:36:58.662 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout enabled 2024-03-30T13:36:58.668 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout active 2024-03-30T13:36:58.668 INFO:teuthology.orchestra.run.smithi043.stdout:Unit chronyd.service is enabled and running 2024-03-30T13:36:58.668 INFO:teuthology.orchestra.run.smithi043.stdout:Host looks OK 2024-03-30T13:36:58.668 INFO:teuthology.orchestra.run.smithi043.stdout:Cluster fsid: 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:36:58.669 INFO:teuthology.orchestra.run.smithi043.stdout:Acquiring lock 139858744585280 on /run/cephadm/7ee932fa-ee9a-11ee-b647-cb9ed24678a4.lock 2024-03-30T13:36:58.669 INFO:teuthology.orchestra.run.smithi043.stdout:Lock 139858744585280 acquired on /run/cephadm/7ee932fa-ee9a-11ee-b647-cb9ed24678a4.lock 2024-03-30T13:36:58.669 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying IP 172.21.15.43 port 3300 ... 2024-03-30T13:36:58.669 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying IP 172.21.15.43 port 6789 ... 2024-03-30T13:36:58.669 INFO:teuthology.orchestra.run.smithi043.stdout:Base mon IP(s) is [172.21.15.43:3300, 172.21.15.43:6789], mon addrv is [v2:172.21.15.43:3300,v1:172.21.15.43:6789] 2024-03-30T13:36:58.672 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.43 metric 100 2024-03-30T13:36:58.672 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.43 metric 100 2024-03-30T13:36:58.674 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-30T13:36:58.675 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-30T13:36:58.675 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-30T13:36:58.677 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-30T13:36:58.677 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-30T13:36:58.678 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-30T13:36:58.678 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-30T13:36:58.678 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1358/64 scope link noprefixroute 2024-03-30T13:36:58.678 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-30T13:36:58.678 INFO:teuthology.orchestra.run.smithi043.stdout:Mon IP `172.21.15.43` is in CIDR network `172.21.0.0/20` 2024-03-30T13:36:58.678 INFO:teuthology.orchestra.run.smithi043.stdout:Mon IP `172.21.15.43` is in CIDR network `172.21.0.0/20` 2024-03-30T13:36:58.678 INFO:teuthology.orchestra.run.smithi043.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-30T13:36:58.678 INFO:teuthology.orchestra.run.smithi043.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-30T13:36:58.679 INFO:teuthology.orchestra.run.smithi043.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb... 2024-03-30T13:36:59.151 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stdout b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e 2024-03-30T13:36:59.151 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb... 2024-03-30T13:36:59.151 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Getting image source signatures 2024-03-30T13:36:59.151 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Copying blob sha256:70aeeb6837442a4655a911c520b59d95aba287a7bb7c28348ac976aba07eb7c4 2024-03-30T13:36:59.152 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Copying blob sha256:11da15500eca1f826c328eb3265caa30b0a3730510e2a32023084d6fe9cee20c 2024-03-30T13:36:59.152 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Copying config sha256:b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e 2024-03-30T13:36:59.152 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-30T13:37:00.140 INFO:teuthology.orchestra.run.smithi043.stdout:ceph: stdout ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev) 2024-03-30T13:37:00.140 INFO:teuthology.orchestra.run.smithi043.stdout:Ceph version: ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev) 2024-03-30T13:37:00.140 INFO:teuthology.orchestra.run.smithi043.stdout:Extracting ceph user uid/gid from container image... 2024-03-30T13:37:01.136 INFO:teuthology.orchestra.run.smithi043.stdout:stat: stdout 167 167 2024-03-30T13:37:01.136 INFO:teuthology.orchestra.run.smithi043.stdout:Creating initial keys... 2024-03-30T13:37:02.177 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph-authtool: stdout AQB9FQhmF+HMGxAAcBhec1jSo0E3o2G9RTUfVQ== 2024-03-30T13:37:03.191 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph-authtool: stdout AQB+FQhmJreQHxAAN0/izluImkPWFzL3nlEk1g== 2024-03-30T13:37:04.205 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph-authtool: stdout AQB/FQhmDWF4IhAA71OcJRdxqzZCrTy6eBBHdA== 2024-03-30T13:37:04.206 INFO:teuthology.orchestra.run.smithi043.stdout:Creating initial monmap... 2024-03-30T13:37:05.152 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-30T13:37:05.152 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-30T13:37:05.152 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:37:05.152 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-30T13:37:05.152 INFO:teuthology.orchestra.run.smithi043.stdout:monmaptool for a [v2:172.21.15.43:3300,v1:172.21.15.43:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-30T13:37:05.152 INFO:teuthology.orchestra.run.smithi043.stdout:setting min_mon_release = quincy 2024-03-30T13:37:05.152 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: set fsid to 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:37:05.152 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-30T13:37:05.152 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:37:05.153 INFO:teuthology.orchestra.run.smithi043.stdout:Creating mon... 2024-03-30T13:37:05.991 INFO:teuthology.orchestra.run.smithi043.stdout:create mon.a on 2024-03-30T13:37:06.198 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-30T13:37:06.338 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-30T13:37:06.507 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4.target. 2024-03-30T13:37:06.507 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4.target. 2024-03-30T13:37:06.738 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mon.a 2024-03-30T13:37:06.739 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mon.a.service: Unit ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mon.a.service not loaded. 2024-03-30T13:37:06.898 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4.target.wants/ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mon.a.service → /etc/systemd/system/ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@.service. 2024-03-30T13:37:07.366 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:07 smithi043 ceph-mon[29472]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-30T13:37:07.380 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T13:37:07.380 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout disabled 2024-03-30T13:37:07.386 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T13:37:07.386 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-03-30T13:37:07.386 INFO:teuthology.orchestra.run.smithi043.stdout:firewalld.service is not enabled 2024-03-30T13:37:07.387 INFO:teuthology.orchestra.run.smithi043.stdout:Not possible to enable service . firewalld.service is not available 2024-03-30T13:37:07.387 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mon to start... 2024-03-30T13:37:07.387 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mon... 2024-03-30T13:37:07.620 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:07 smithi043 bash[29434]: 688eecdad3c96be35fcce63bf50a22590dae17988b27786b0fab7fdb8394eddb 2024-03-30T13:37:07.620 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:07 smithi043 systemd[1]: Started Ceph mon.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T13:37:08.506 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:08 smithi043 ceph-mon[29472]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-30T13:37:08.506 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:08 smithi043 ceph-mon[29472]: monmap epoch 1 2024-03-30T13:37:08.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:08 smithi043 ceph-mon[29472]: fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:37:08.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:08 smithi043 ceph-mon[29472]: last_changed 2024-03-30T13:37:04.571563+0000 2024-03-30T13:37:08.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:08 smithi043 ceph-mon[29472]: created 2024-03-30T13:37:04.571563+0000 2024-03-30T13:37:08.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:08 smithi043 ceph-mon[29472]: min_mon_release 19 (squid) 2024-03-30T13:37:08.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:08 smithi043 ceph-mon[29472]: election_strategy: 1 2024-03-30T13:37:08.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:08 smithi043 ceph-mon[29472]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-03-30T13:37:08.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:08 smithi043 ceph-mon[29472]: fsmap 2024-03-30T13:37:08.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:08 smithi043 ceph-mon[29472]: osdmap e1: 0 total, 0 up, 0 in 2024-03-30T13:37:08.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:08 smithi043 ceph-mon[29472]: mgrmap e1: no daemons active 2024-03-30T13:37:08.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:08 smithi043 ceph-mon[29472]: from='client.? 172.21.15.43:0/4157365857' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-30T13:37:08.510 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout cluster: 2024-03-30T13:37:08.510 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout id: 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:37:08.510 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-30T13:37:08.510 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-30T13:37:08.510 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout services: 2024-03-30T13:37:08.510 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.59234s) 2024-03-30T13:37:08.510 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-30T13:37:08.510 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-30T13:37:08.511 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-30T13:37:08.511 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout data: 2024-03-30T13:37:08.511 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-30T13:37:08.511 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-30T13:37:08.511 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-30T13:37:08.511 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout pgs: 2024-03-30T13:37:08.511 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-30T13:37:08.511 INFO:teuthology.orchestra.run.smithi043.stdout:mon is available 2024-03-30T13:37:08.511 INFO:teuthology.orchestra.run.smithi043.stdout:Assimilating anything we can from ceph.conf... 2024-03-30T13:37:09.554 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:09 smithi043 ceph-mon[29472]: from='client.? 172.21.15.43:0/2246980745' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-30T13:37:09.554 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:09 smithi043 ceph-mon[29472]: from='client.? 172.21.15.43:0/2246980745' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-30T13:37:09.557 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-30T13:37:09.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [global] 2024-03-30T13:37:09.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout fsid = 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:37:09.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.43:3300,v1:172.21.15.43:6789] 2024-03-30T13:37:09.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-30T13:37:09.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-30T13:37:09.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-30T13:37:09.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-30T13:37:09.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-30T13:37:09.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-30T13:37:09.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-30T13:37:09.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-30T13:37:09.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [osd] 2024-03-30T13:37:09.560 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-30T13:37:09.560 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-30T13:37:09.560 INFO:teuthology.orchestra.run.smithi043.stdout:Generating new minimal ceph.conf... 2024-03-30T13:37:10.619 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:10 smithi043 ceph-mon[29472]: from='client.? 172.21.15.43:0/2758267531' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:37:10.621 INFO:teuthology.orchestra.run.smithi043.stdout:Restarting the monitor... 2024-03-30T13:37:10.883 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:10 smithi043 systemd[1]: Stopping Ceph mon.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T13:37:11.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:10 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a[29449]: 2024-03-30T13:37:10.880+0000 7fe78caf8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T13:37:11.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:10 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a[29449]: 2024-03-30T13:37:10.880+0000 7fe78caf8700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-30T13:37:11.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:11 smithi043 podman[29803]: 2024-03-30 13:37:11.055832014 +0000 UTC m=+0.273928323 container died 688eecdad3c96be35fcce63bf50a22590dae17988b27786b0fab7fdb8394eddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=squid-4e50bbc) 2024-03-30T13:37:11.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:11 smithi043 podman[29803]: 2024-03-30 13:37:11.41207741 +0000 UTC m=+0.630173718 container cleanup 688eecdad3c96be35fcce63bf50a22590dae17988b27786b0fab7fdb8394eddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T13:37:11.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:11 smithi043 bash[29803]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a 2024-03-30T13:37:12.006 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:11 smithi043 podman[29822]: 2024-03-30 13:37:11.654543704 +0000 UTC m=+0.589302224 container remove 688eecdad3c96be35fcce63bf50a22590dae17988b27786b0fab7fdb8394eddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0) 2024-03-30T13:37:12.006 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:11 smithi043 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mon.a.service: Deactivated successfully. 2024-03-30T13:37:12.006 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:11 smithi043 systemd[1]: Stopped Ceph mon.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T13:37:12.006 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:11 smithi043 systemd[1]: Starting Ceph mon.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T13:37:12.253 INFO:teuthology.orchestra.run.smithi043.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-30T13:37:12.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 podman[29924]: 2024-03-30 13:37:12.005448458 +0000 UTC m=+0.107726756 container create bb7d4ff2de3d44adaf589a1b09983cc543e1dca2bec1f803869a3d99a48346d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=) 2024-03-30T13:37:12.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 podman[29924]: 2024-03-30 13:37:11.915328782 +0000 UTC m=+0.017607055 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T13:37:12.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 podman[29924]: 2024-03-30 13:37:12.174136625 +0000 UTC m=+0.276414895 container init bb7d4ff2de3d44adaf589a1b09983cc543e1dca2bec1f803869a3d99a48346d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=) 2024-03-30T13:37:12.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 podman[29924]: 2024-03-30 13:37:12.179615555 +0000 UTC m=+0.281893812 container start bb7d4ff2de3d44adaf589a1b09983cc543e1dca2bec1f803869a3d99a48346d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux ) 2024-03-30T13:37:12.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: set uid:gid to 167:167 (ceph:ceph) 2024-03-30T13:37:12.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev), process ceph-mon, pid 7 2024-03-30T13:37:12.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: pidfile_write: ignore empty --pid-file 2024-03-30T13:37:12.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: load: jerasure load: lrc 2024-03-30T13:37:12.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: RocksDB version: 7.9.2 2024-03-30T13:37:12.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Git sha 0 2024-03-30T13:37:12.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Compile date 2024-03-29 15:06:16 2024-03-30T13:37:12.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: DB SUMMARY 2024-03-30T13:37:12.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: DB Session ID: DU7FN2JK4UARBEOP4N3E 2024-03-30T13:37:12.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: CURRENT file: CURRENT 2024-03-30T13:37:12.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: IDENTITY file: IDENTITY 2024-03-30T13:37:12.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-30T13:37:12.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-30T13:37:12.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 94963 ; 2024-03-30T13:37:12.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.error_if_exists: 0 2024-03-30T13:37:12.271 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.create_if_missing: 0 2024-03-30T13:37:12.271 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.paranoid_checks: 1 2024-03-30T13:37:12.271 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-30T13:37:12.271 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-30T13:37:12.271 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-30T13:37:12.271 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.env: 0x558478b5bc60 2024-03-30T13:37:12.271 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.fs: PosixFileSystem 2024-03-30T13:37:12.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.info_log: 0x558479561ae0 2024-03-30T13:37:12.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_file_opening_threads: 16 2024-03-30T13:37:12.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.statistics: (nil) 2024-03-30T13:37:12.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.use_fsync: 0 2024-03-30T13:37:12.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_log_file_size: 0 2024-03-30T13:37:12.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-30T13:37:12.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-30T13:37:12.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.keep_log_file_num: 1000 2024-03-30T13:37:12.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.recycle_log_file_num: 0 2024-03-30T13:37:12.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.allow_fallocate: 1 2024-03-30T13:37:12.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.allow_mmap_reads: 0 2024-03-30T13:37:12.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.allow_mmap_writes: 0 2024-03-30T13:37:12.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.use_direct_reads: 0 2024-03-30T13:37:12.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-30T13:37:12.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.create_missing_column_families: 0 2024-03-30T13:37:12.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.db_log_dir: 2024-03-30T13:37:12.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.wal_dir: 2024-03-30T13:37:12.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-30T13:37:12.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-30T13:37:12.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-30T13:37:12.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-30T13:37:12.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-30T13:37:12.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-30T13:37:12.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.advise_random_on_open: 1 2024-03-30T13:37:12.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.db_write_buffer_size: 0 2024-03-30T13:37:12.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.write_buffer_manager: 0x5584795341e0 2024-03-30T13:37:12.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-30T13:37:12.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-30T13:37:12.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-30T13:37:12.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.rate_limiter: (nil) 2024-03-30T13:37:12.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-30T13:37:12.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.wal_recovery_mode: 2 2024-03-30T13:37:12.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.enable_thread_tracking: 0 2024-03-30T13:37:12.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.enable_pipelined_write: 0 2024-03-30T13:37:12.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.unordered_write: 0 2024-03-30T13:37:12.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-30T13:37:12.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-30T13:37:12.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-30T13:37:12.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-30T13:37:12.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.row_cache: None 2024-03-30T13:37:12.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.wal_filter: None 2024-03-30T13:37:12.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-30T13:37:12.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.allow_ingest_behind: 0 2024-03-30T13:37:12.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.two_write_queues: 0 2024-03-30T13:37:12.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.manual_wal_flush: 0 2024-03-30T13:37:12.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.wal_compression: 0 2024-03-30T13:37:12.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.atomic_flush: 0 2024-03-30T13:37:12.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-30T13:37:12.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-30T13:37:12.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-30T13:37:12.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.log_readahead_size: 0 2024-03-30T13:37:12.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-30T13:37:12.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.best_efforts_recovery: 0 2024-03-30T13:37:12.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-30T13:37:12.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-30T13:37:12.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.allow_data_in_errors: 0 2024-03-30T13:37:12.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.db_host_id: __hostname__ 2024-03-30T13:37:12.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-30T13:37:12.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_background_jobs: 2 2024-03-30T13:37:12.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_background_compactions: -1 2024-03-30T13:37:12.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_subcompactions: 1 2024-03-30T13:37:12.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-30T13:37:12.277 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-30T13:37:12.277 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-30T13:37:12.277 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_total_wal_size: 0 2024-03-30T13:37:12.277 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-30T13:37:12.277 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-30T13:37:12.277 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-30T13:37:12.277 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-30T13:37:12.277 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_open_files: -1 2024-03-30T13:37:12.277 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.bytes_per_sync: 0 2024-03-30T13:37:12.277 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-30T13:37:12.278 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-30T13:37:12.278 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compaction_readahead_size: 0 2024-03-30T13:37:12.279 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_background_flushes: -1 2024-03-30T13:37:12.279 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Compression algorithms supported: 2024-03-30T13:37:12.279 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-30T13:37:12.279 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: kZSTD supported: 0 2024-03-30T13:37:12.279 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: kXpressCompression supported: 0 2024-03-30T13:37:12.279 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: kLZ4HCCompression supported: 1 2024-03-30T13:37:12.279 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: kZlibCompression supported: 1 2024-03-30T13:37:12.279 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: kSnappyCompression supported: 1 2024-03-30T13:37:12.279 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: kLZ4Compression supported: 1 2024-03-30T13:37:12.279 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: kBZip2Compression supported: 0 2024-03-30T13:37:12.279 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-30T13:37:12.279 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-30T13:37:12.279 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-30T13:37:12.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-30T13:37:12.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-30T13:37:12.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.merge_operator: 2024-03-30T13:37:12.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compaction_filter: None 2024-03-30T13:37:12.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compaction_filter_factory: None 2024-03-30T13:37:12.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.sst_partitioner_factory: None 2024-03-30T13:37:12.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-30T13:37:12.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-30T13:37:12.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558479561be0) 2024-03-30T13:37:12.280 INFO:journalctl@ceph.mon.a.smithi043.stdout: cache_index_and_filter_blocks: 1 2024-03-30T13:37:12.280 INFO:journalctl@ceph.mon.a.smithi043.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-30T13:37:12.280 INFO:journalctl@ceph.mon.a.smithi043.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-30T13:37:12.280 INFO:journalctl@ceph.mon.a.smithi043.stdout: pin_top_level_index_and_filter: 1 2024-03-30T13:37:12.280 INFO:journalctl@ceph.mon.a.smithi043.stdout: index_type: 0 2024-03-30T13:37:12.281 INFO:journalctl@ceph.mon.a.smithi043.stdout: data_block_index_type: 0 2024-03-30T13:37:12.281 INFO:journalctl@ceph.mon.a.smithi043.stdout: index_shortening: 1 2024-03-30T13:37:12.281 INFO:journalctl@ceph.mon.a.smithi043.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-30T13:37:12.281 INFO:journalctl@ceph.mon.a.smithi043.stdout: checksum: 4 2024-03-30T13:37:12.281 INFO:journalctl@ceph.mon.a.smithi043.stdout: no_block_cache: 0 2024-03-30T13:37:12.281 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_cache: 0x558479565090 2024-03-30T13:37:12.281 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_cache_name: BinnedLRUCache 2024-03-30T13:37:12.281 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_cache_options: 2024-03-30T13:37:12.281 INFO:journalctl@ceph.mon.a.smithi043.stdout: capacity : 536870912 2024-03-30T13:37:12.281 INFO:journalctl@ceph.mon.a.smithi043.stdout: num_shard_bits : 4 2024-03-30T13:37:12.281 INFO:journalctl@ceph.mon.a.smithi043.stdout: strict_capacity_limit : 0 2024-03-30T13:37:12.281 INFO:journalctl@ceph.mon.a.smithi043.stdout: high_pri_pool_ratio: 0.000 2024-03-30T13:37:12.281 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_cache_compressed: (nil) 2024-03-30T13:37:12.281 INFO:journalctl@ceph.mon.a.smithi043.stdout: persistent_cache: (nil) 2024-03-30T13:37:12.281 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_size: 4096 2024-03-30T13:37:12.282 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_size_deviation: 10 2024-03-30T13:37:12.282 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_restart_interval: 16 2024-03-30T13:37:12.282 INFO:journalctl@ceph.mon.a.smithi043.stdout: index_block_restart_interval: 1 2024-03-30T13:37:12.282 INFO:journalctl@ceph.mon.a.smithi043.stdout: metadata_block_size: 4096 2024-03-30T13:37:12.282 INFO:journalctl@ceph.mon.a.smithi043.stdout: partition_filters: 0 2024-03-30T13:37:12.282 INFO:journalctl@ceph.mon.a.smithi043.stdout: use_delta_encoding: 1 2024-03-30T13:37:12.282 INFO:journalctl@ceph.mon.a.smithi043.stdout: filter_policy: bloomfilter 2024-03-30T13:37:12.282 INFO:journalctl@ceph.mon.a.smithi043.stdout: whole_key_filtering: 1 2024-03-30T13:37:12.282 INFO:journalctl@ceph.mon.a.smithi043.stdout: verify_compression: 0 2024-03-30T13:37:12.282 INFO:journalctl@ceph.mon.a.smithi043.stdout: read_amp_bytes_per_bit: 0 2024-03-30T13:37:12.282 INFO:journalctl@ceph.mon.a.smithi043.stdout: format_version: 5 2024-03-30T13:37:12.282 INFO:journalctl@ceph.mon.a.smithi043.stdout: enable_index_compression: 1 2024-03-30T13:37:12.282 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_align: 0 2024-03-30T13:37:12.282 INFO:journalctl@ceph.mon.a.smithi043.stdout: max_auto_readahead_size: 262144 2024-03-30T13:37:12.283 INFO:journalctl@ceph.mon.a.smithi043.stdout: prepopulate_block_cache: 0 2024-03-30T13:37:12.283 INFO:journalctl@ceph.mon.a.smithi043.stdout: initial_auto_readahead_size: 8192 2024-03-30T13:37:12.283 INFO:journalctl@ceph.mon.a.smithi043.stdout: num_file_reads_for_auto_readahead: 2 2024-03-30T13:37:12.283 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.write_buffer_size: 33554432 2024-03-30T13:37:12.283 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_write_buffer_number: 2 2024-03-30T13:37:12.283 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compression: NoCompression 2024-03-30T13:37:12.283 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.bottommost_compression: Disabled 2024-03-30T13:37:12.283 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.prefix_extractor: nullptr 2024-03-30T13:37:12.283 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-30T13:37:12.283 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.num_levels: 7 2024-03-30T13:37:12.283 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-30T13:37:12.283 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-30T13:37:12.283 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-30T13:37:12.283 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-30T13:37:12.283 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-30T13:37:12.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-30T13:37:12.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-30T13:37:12.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-30T13:37:12.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-30T13:37:12.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-30T13:37:12.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-30T13:37:12.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-30T13:37:12.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-30T13:37:12.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compression_opts.level: 32767 2024-03-30T13:37:12.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compression_opts.strategy: 0 2024-03-30T13:37:12.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-30T13:37:12.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-30T13:37:12.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-30T13:37:12.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-30T13:37:12.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compression_opts.enabled: false 2024-03-30T13:37:12.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-30T13:37:12.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-30T13:37:12.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-30T13:37:12.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-30T13:37:12.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.target_file_size_base: 67108864 2024-03-30T13:37:12.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-30T13:37:12.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-30T13:37:12.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-30T13:37:12.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-30T13:37:12.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-30T13:37:12.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-30T13:37:12.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-30T13:37:12.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-30T13:37:12.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-30T13:37:12.286 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-30T13:37:12.286 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-30T13:37:12.286 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-30T13:37:12.286 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-30T13:37:12.286 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-30T13:37:12.286 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.arena_block_size: 1048576 2024-03-30T13:37:12.286 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-30T13:37:12.286 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-30T13:37:12.286 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.disable_auto_compactions: 0 2024-03-30T13:37:12.286 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-30T13:37:12.286 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-30T13:37:12.286 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-30T13:37:12.287 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-30T13:37:12.287 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-30T13:37:12.287 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-30T13:37:12.287 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-30T13:37:12.287 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-30T13:37:12.287 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-30T13:37:12.287 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-30T13:37:12.287 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.table_properties_collectors: 2024-03-30T13:37:12.287 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.inplace_update_support: 0 2024-03-30T13:37:12.287 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-30T13:37:12.287 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-30T13:37:12.287 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-30T13:37:12.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-30T13:37:12.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.bloom_locality: 0 2024-03-30T13:37:12.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.max_successive_merges: 0 2024-03-30T13:37:12.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-30T13:37:12.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.paranoid_file_checks: 0 2024-03-30T13:37:12.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.force_consistency_checks: 1 2024-03-30T13:37:12.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.report_bg_io_stats: 0 2024-03-30T13:37:12.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.ttl: 2592000 2024-03-30T13:37:12.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-30T13:37:12.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-30T13:37:12.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-30T13:37:12.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.enable_blob_files: false 2024-03-30T13:37:12.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.min_blob_size: 0 2024-03-30T13:37:12.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.blob_file_size: 268435456 2024-03-30T13:37:12.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-30T13:37:12.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-30T13:37:12.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-30T13:37:12.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-30T13:37:12.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-30T13:37:12.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.blob_file_starting_level: 0 2024-03-30T13:37:12.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-30T13:37:12.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-30T13:37:12.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-30T13:37:12.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 77708f8c-9941-4b9c-bd22-66ea1c8efabd 2024-03-30T13:37:12.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711805832209159, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-30T13:37:12.290 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-30T13:37:12.290 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711805832210031, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 90924, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 284, "table_properties": {"data_size": 88992, "index_size": 278, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11754, "raw_average_key_size": 48, "raw_value_size": 82511, "raw_average_value_size": 339, "num_data_blocks": 12, "num_entries": 243, "num_filter_entries": 243, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711805832, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "77708f8c-9941-4b9c-bd22-66ea1c8efabd", "db_session_id": "DU7FN2JK4UARBEOP4N3E", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-30T13:37:12.290 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711805832210121, "job": 1, "event": "recovery_finished"} 2024-03-30T13:37:12.290 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-30T13:37:12.290 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-30T13:37:12.290 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558479656000 2024-03-30T13:37:12.290 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: DB pointer 0x558479640000 2024-03-30T13:37:12.290 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-30T13:37:12.290 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-30T13:37:12.290 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** DB Stats ** 2024-03-30T13:37:12.290 INFO:journalctl@ceph.mon.a.smithi043.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T13:37:12.290 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-30T13:37:12.290 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T13:37:12.290 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T13:37:12.291 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-30T13:37:12.291 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T13:37:12.291 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T13:37:12.291 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-03-30T13:37:12.291 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** Compaction Stats [default] ** 2024-03-30T13:37:12.291 INFO:journalctl@ceph.mon.a.smithi043.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T13:37:12.291 INFO:journalctl@ceph.mon.a.smithi043.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-30T13:37:12.291 INFO:journalctl@ceph.mon.a.smithi043.stdout: L0 2/0 90.63 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T13:37:12.291 INFO:journalctl@ceph.mon.a.smithi043.stdout: Sum 2/0 90.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T13:37:12.291 INFO:journalctl@ceph.mon.a.smithi043.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T13:37:12.291 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-03-30T13:37:12.291 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** Compaction Stats [default] ** 2024-03-30T13:37:12.291 INFO:journalctl@ceph.mon.a.smithi043.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T13:37:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T13:37:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 154.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T13:37:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-03-30T13:37:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-30T13:37:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-03-30T13:37:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T13:37:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-30T13:37:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-30T13:37:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-30T13:37:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-30T13:37:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-30T13:37:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative compaction: 0.00 GB write, 29.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T13:37:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval compaction: 0.00 GB write, 29.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T13:37:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-30T13:37:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout: Block cache BinnedLRUCache@0x558479565090#7 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-30T13:37:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-30T13:37:12.293 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-03-30T13:37:12.293 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-30T13:37:12.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: starting mon.a rank 0 at public addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] at bind addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:37:12.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: mon.a@-1(???) e1 preinit fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:37:12.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: mon.a@-1(???).mds e1 new map 2024-03-30T13:37:12.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: mon.a@-1(???).mds e1 print_map 2024-03-30T13:37:12.293 INFO:journalctl@ceph.mon.a.smithi043.stdout: e1 2024-03-30T13:37:12.293 INFO:journalctl@ceph.mon.a.smithi043.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T13:37:12.293 INFO:journalctl@ceph.mon.a.smithi043.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-30T13:37:12.293 INFO:journalctl@ceph.mon.a.smithi043.stdout: legacy client fscid: -1 2024-03-30T13:37:12.293 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-03-30T13:37:12.293 INFO:journalctl@ceph.mon.a.smithi043.stdout: No filesystems configured 2024-03-30T13:37:12.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-30T13:37:12.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T13:37:12.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T13:37:12.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T13:37:12.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-30T13:37:12.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 bash[29924]: bb7d4ff2de3d44adaf589a1b09983cc543e1dca2bec1f803869a3d99a48346d9 2024-03-30T13:37:12.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 systemd[1]: Started Ceph mon.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T13:37:12.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-30T13:37:12.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: monmap epoch 1 2024-03-30T13:37:12.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:37:12.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: last_changed 2024-03-30T13:37:04.571563+0000 2024-03-30T13:37:12.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: created 2024-03-30T13:37:04.571563+0000 2024-03-30T13:37:12.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: min_mon_release 19 (squid) 2024-03-30T13:37:12.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: election_strategy: 1 2024-03-30T13:37:12.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-03-30T13:37:12.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: fsmap 2024-03-30T13:37:12.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: osdmap e1: 0 total, 0 up, 0 in 2024-03-30T13:37:12.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:12 smithi043 ceph-mon[29963]: mgrmap e1: no daemons active 2024-03-30T13:37:13.300 INFO:teuthology.orchestra.run.smithi043.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-30T13:37:13.347 INFO:teuthology.orchestra.run.smithi043.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-30T13:37:13.347 INFO:teuthology.orchestra.run.smithi043.stdout:Creating mgr... 2024-03-30T13:37:13.347 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-30T13:37:13.348 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-30T13:37:13.599 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mgr.y 2024-03-30T13:37:13.599 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mgr.y.service: Unit ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mgr.y.service not loaded. 2024-03-30T13:37:13.758 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4.target.wants/ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mgr.y.service → /etc/systemd/system/ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@.service. 2024-03-30T13:37:14.122 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:13 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3775434041' entity='client.admin' 2024-03-30T13:37:14.123 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:14 smithi043 podman[30202]: 2024-03-30 13:37:14.015134079 +0000 UTC m=+0.127169003 container create b99b930d553f0e7a5f35293971ace5fdd4c225bfbaaf3440112fb5fadfe67832 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-30T13:37:14.289 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T13:37:14.289 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout disabled 2024-03-30T13:37:14.296 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T13:37:14.296 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-03-30T13:37:14.296 INFO:teuthology.orchestra.run.smithi043.stdout:firewalld.service is not enabled 2024-03-30T13:37:14.296 INFO:teuthology.orchestra.run.smithi043.stdout:Not possible to enable service . firewalld.service is not available 2024-03-30T13:37:14.304 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T13:37:14.304 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout disabled 2024-03-30T13:37:14.311 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T13:37:14.312 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-03-30T13:37:14.312 INFO:teuthology.orchestra.run.smithi043.stdout:firewalld.service is not enabled 2024-03-30T13:37:14.312 INFO:teuthology.orchestra.run.smithi043.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-30T13:37:14.312 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mgr to start... 2024-03-30T13:37:14.312 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mgr... 2024-03-30T13:37:14.530 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:14 smithi043 podman[30202]: 2024-03-30 13:37:14.217332095 +0000 UTC m=+0.329367036 container init b99b930d553f0e7a5f35293971ace5fdd4c225bfbaaf3440112fb5fadfe67832 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-30T13:37:14.530 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:14 smithi043 podman[30202]: 2024-03-30 13:37:14.223102574 +0000 UTC m=+0.335137499 container start b99b930d553f0e7a5f35293971ace5fdd4c225bfbaaf3440112fb5fadfe67832 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc) 2024-03-30T13:37:14.530 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:14 smithi043 bash[30202]: b99b930d553f0e7a5f35293971ace5fdd4c225bfbaaf3440112fb5fadfe67832 2024-03-30T13:37:14.530 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:14 smithi043 systemd[1]: Started Ceph mgr.y for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T13:37:15.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:14 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2333413670' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T13:37:15.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:14 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:14.796+0000 7f8325155200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T13:37:15.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:14 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:14.921+0000 7f8325155200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T13:37:15.513 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-30T13:37:15.513 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-03-30T13:37:15.513 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsid": "7ee932fa-ee9a-11ee-b647-cb9ed24678a4", 2024-03-30T13:37:15.513 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T13:37:15.514 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T13:37:15.514 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T13:37:15.514 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T13:37:15.514 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:15.514 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T13:37:15.514 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T13:37:15.514 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 0 2024-03-30T13:37:15.514 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:37:15.514 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T13:37:15.514 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T13:37:15.514 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:37:15.514 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-30T13:37:15.514 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T13:37:15.515 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:37:15.515 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-30T13:37:15.515 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T13:37:15.515 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:15.515 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T13:37:15.515 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:37:15.515 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T13:37:15.515 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T13:37:15.515 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T13:37:15.515 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T13:37:15.515 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T13:37:15.515 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T13:37:15.518 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:15.519 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T13:37:15.519 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T13:37:15.519 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T13:37:15.519 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T13:37:15.519 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T13:37:15.519 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T13:37:15.519 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T13:37:15.519 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T13:37:15.519 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T13:37:15.519 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:15.519 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T13:37:15.519 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:37:15.519 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T13:37:15.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T13:37:15.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:15.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T13:37:15.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-30T13:37:15.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T13:37:15.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T13:37:15.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T13:37:15.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T13:37:15.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T13:37:15.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:37:15.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T13:37:15.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:15.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T13:37:15.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:37:15.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T13:37:07.346487+0000", 2024-03-30T13:37:15.521 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T13:37:15.521 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:15.521 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T13:37:15.521 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-03-30T13:37:15.521 INFO:teuthology.orchestra.run.smithi043.stdout:mgr not available, waiting (1/15)... 2024-03-30T13:37:15.828 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:15 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:15.589+0000 7f8325155200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T13:37:15.828 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:15 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:15.683+0000 7f8325155200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T13:37:16.148 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:15 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:15.826+0000 7f8325155200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T13:37:16.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:15 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:15.924+0000 7f8325155200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T13:37:16.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:16.008+0000 7f8325155200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T13:37:16.570 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:16.274+0000 7f8325155200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T13:37:16.571 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:16.386+0000 7f8325155200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T13:37:16.571 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:16.473+0000 7f8325155200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T13:37:16.571 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:16.569+0000 7f8325155200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T13:37:16.899 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:16.747+0000 7f8325155200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T13:37:17.888 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:17 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:17.751+0000 7f8325155200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T13:37:17.888 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:17 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:17.886+0000 7f8325155200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T13:37:18.480 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:18 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2760475287' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T13:37:18.481 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:18 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:18.278+0000 7f8325155200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T13:37:18.481 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:18 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:18.368+0000 7f8325155200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T13:37:18.755 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:18 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:18.568+0000 7f8325155200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:37:18.761 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-30T13:37:18.761 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-03-30T13:37:18.761 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsid": "7ee932fa-ee9a-11ee-b647-cb9ed24678a4", 2024-03-30T13:37:18.762 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T13:37:18.762 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T13:37:18.762 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T13:37:18.762 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T13:37:18.762 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:18.762 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T13:37:18.762 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T13:37:18.762 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 0 2024-03-30T13:37:18.762 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:37:18.762 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T13:37:18.762 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T13:37:18.762 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:37:18.763 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-30T13:37:18.763 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T13:37:18.763 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:37:18.763 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-30T13:37:18.763 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T13:37:18.763 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:18.763 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T13:37:18.763 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:37:18.763 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T13:37:18.763 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T13:37:18.763 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T13:37:18.763 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T13:37:18.763 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T13:37:18.763 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T13:37:18.763 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:18.764 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T13:37:18.764 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T13:37:18.764 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T13:37:18.766 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T13:37:18.766 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T13:37:18.766 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T13:37:18.766 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T13:37:18.767 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T13:37:18.767 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T13:37:18.767 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:18.767 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T13:37:18.767 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:37:18.767 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T13:37:18.767 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T13:37:18.767 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:18.767 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T13:37:18.767 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-30T13:37:18.767 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T13:37:18.767 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T13:37:18.767 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T13:37:18.767 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T13:37:18.767 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T13:37:18.768 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:37:18.768 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T13:37:18.768 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:18.768 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T13:37:18.768 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:37:18.768 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T13:37:07.346487+0000", 2024-03-30T13:37:18.768 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T13:37:18.768 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:18.768 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T13:37:18.768 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-03-30T13:37:18.768 INFO:teuthology.orchestra.run.smithi043.stdout:mgr not available, waiting (2/15)... 2024-03-30T13:37:19.045 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:18 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:18.756+0000 7f8325155200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T13:37:19.045 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:19 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:19.043+0000 7f8325155200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:37:19.649 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:19 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:19.306+0000 7f8325155200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T13:37:19.649 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:19 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:19.546+0000 7f8325155200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T13:37:20.650 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:20 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:20.376+0000 7f8325155200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T13:37:21.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:20 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:20.962+0000 7f8325155200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T13:37:21.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:21 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:21.065+0000 7f8325155200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T13:37:21.611 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:21 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:21.291+0000 7f8325155200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T13:37:21.611 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:21 smithi043 ceph-mon[29963]: Activating manager daemon y 2024-03-30T13:37:21.611 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:21 smithi043 ceph-mon[29963]: mgrmap e2: y(active, starting, since 0.00319355s) 2024-03-30T13:37:21.611 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:21 smithi043 ceph-mon[29963]: from='mgr.14100 172.21.15.43:0/397116501' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T13:37:21.611 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:21 smithi043 ceph-mon[29963]: from='mgr.14100 172.21.15.43:0/397116501' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T13:37:21.611 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:21 smithi043 ceph-mon[29963]: from='mgr.14100 172.21.15.43:0/397116501' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T13:37:21.611 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:21 smithi043 ceph-mon[29963]: from='mgr.14100 172.21.15.43:0/397116501' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:37:21.611 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:21 smithi043 ceph-mon[29963]: from='mgr.14100 172.21.15.43:0/397116501' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T13:37:21.611 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:21 smithi043 ceph-mon[29963]: Manager daemon y is now available 2024-03-30T13:37:21.612 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:21 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/842536603' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T13:37:21.612 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:21 smithi043 ceph-mon[29963]: from='mgr.14100 172.21.15.43:0/397116501' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T13:37:21.612 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:21 smithi043 ceph-mon[29963]: from='mgr.14100 172.21.15.43:0/397116501' entity='mgr.y' 2024-03-30T13:37:21.612 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:21 smithi043 ceph-mon[29963]: from='mgr.14100 172.21.15.43:0/397116501' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T13:37:21.612 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:21 smithi043 ceph-mon[29963]: from='mgr.14100 172.21.15.43:0/397116501' entity='mgr.y' 2024-03-30T13:37:21.612 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:21 smithi043 ceph-mon[29963]: from='mgr.14100 172.21.15.43:0/397116501' entity='mgr.y' 2024-03-30T13:37:21.900 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-30T13:37:21.900 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-03-30T13:37:21.900 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsid": "7ee932fa-ee9a-11ee-b647-cb9ed24678a4", 2024-03-30T13:37:21.900 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T13:37:21.900 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T13:37:21.900 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T13:37:21.900 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T13:37:21.900 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:21.901 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T13:37:21.901 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T13:37:21.901 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 0 2024-03-30T13:37:21.901 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:37:21.901 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T13:37:21.901 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T13:37:21.901 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:37:21.901 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-30T13:37:21.901 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T13:37:21.901 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:37:21.901 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-30T13:37:21.901 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T13:37:21.901 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:21.901 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T13:37:21.902 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:37:21.902 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T13:37:21.902 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T13:37:21.902 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T13:37:21.902 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T13:37:21.902 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T13:37:21.902 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T13:37:21.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:21.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T13:37:21.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T13:37:21.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T13:37:21.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T13:37:21.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T13:37:21.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T13:37:21.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T13:37:21.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T13:37:21.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T13:37:21.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:21.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T13:37:21.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:37:21.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T13:37:21.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T13:37:21.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:21.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T13:37:21.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-30T13:37:21.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T13:37:21.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T13:37:21.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T13:37:21.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T13:37:21.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T13:37:21.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:37:21.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T13:37:21.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:21.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T13:37:21.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:37:21.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T13:37:07.346487+0000", 2024-03-30T13:37:21.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T13:37:21.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:21.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T13:37:21.908 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-03-30T13:37:21.908 INFO:teuthology.orchestra.run.smithi043.stdout:mgr not available, waiting (3/15)... 2024-03-30T13:37:23.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:23 smithi043 ceph-mon[29963]: mgrmap e3: y(active, since 1.0068s) 2024-03-30T13:37:24.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:24 smithi043 ceph-mon[29963]: mgrmap e4: y(active, since 2s) 2024-03-30T13:37:28.324 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:28 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/621588709' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T13:37:32.690 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-30T13:37:32.691 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-03-30T13:37:32.691 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsid": "7ee932fa-ee9a-11ee-b647-cb9ed24678a4", 2024-03-30T13:37:32.691 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T13:37:32.691 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T13:37:32.691 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T13:37:32.691 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T13:37:32.691 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:32.691 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T13:37:32.691 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T13:37:32.691 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 0 2024-03-30T13:37:32.691 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:37:32.691 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T13:37:32.692 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T13:37:32.692 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:37:32.692 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-03-30T13:37:32.692 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T13:37:32.692 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:37:32.692 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-30T13:37:32.692 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T13:37:32.692 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:32.692 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T13:37:32.692 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:37:32.692 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T13:37:32.692 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T13:37:32.692 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T13:37:32.692 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T13:37:32.696 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T13:37:32.696 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T13:37:32.696 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:32.696 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T13:37:32.697 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T13:37:32.697 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T13:37:32.697 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T13:37:32.697 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T13:37:32.697 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T13:37:32.697 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T13:37:32.697 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T13:37:32.697 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T13:37:32.697 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:32.697 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T13:37:32.697 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:37:32.697 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T13:37:32.697 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T13:37:32.698 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:32.698 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T13:37:32.698 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-30T13:37:32.698 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T13:37:32.698 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T13:37:32.698 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T13:37:32.698 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T13:37:32.698 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T13:37:32.698 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:37:32.698 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T13:37:32.699 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:32.699 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T13:37:32.699 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:37:32.699 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T13:37:07.346487+0000", 2024-03-30T13:37:32.699 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T13:37:32.699 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:37:32.699 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T13:37:32.699 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-03-30T13:37:32.699 INFO:teuthology.orchestra.run.smithi043.stdout:mgr is available 2024-03-30T13:37:33.648 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:33 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/758830198' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-30T13:37:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-30T13:37:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [global] 2024-03-30T13:37:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout fsid = 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:37:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-30T13:37:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-30T13:37:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-30T13:37:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-30T13:37:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-30T13:37:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-30T13:37:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-30T13:37:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-30T13:37:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [osd] 2024-03-30T13:37:33.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-30T13:37:33.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-30T13:37:33.912 INFO:teuthology.orchestra.run.smithi043.stdout:Enabling cephadm module... 2024-03-30T13:37:35.013 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:34 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1383676289' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-30T13:37:35.013 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:34 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ignoring --setuser ceph since I am not root 2024-03-30T13:37:35.013 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:34 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ignoring --setgroup ceph since I am not root 2024-03-30T13:37:35.649 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:35 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:35.279+0000 7f783f33c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T13:37:35.649 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:35 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:35.379+0000 7f783f33c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T13:37:36.040 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:35 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1383676289' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-30T13:37:36.040 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:35 smithi043 ceph-mon[29963]: mgrmap e5: y(active, since 13s) 2024-03-30T13:37:36.316 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:36.039+0000 7f783f33c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T13:37:36.316 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:36.135+0000 7f783f33c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T13:37:36.316 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:36.288+0000 7f783f33c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T13:37:36.649 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:36.389+0000 7f783f33c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T13:37:36.649 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:36.477+0000 7f783f33c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T13:37:36.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:36 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/940450362' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-30T13:37:36.957 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:36.751+0000 7f783f33c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T13:37:36.957 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:36.864+0000 7f783f33c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T13:37:36.957 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:36.955+0000 7f783f33c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T13:37:37.234 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:37 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:37.054+0000 7f783f33c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T13:37:37.235 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:37 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:37.233+0000 7f783f33c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T13:37:37.240 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-03-30T13:37:37.240 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-30T13:37:37.241 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-30T13:37:37.241 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-30T13:37:37.241 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-30T13:37:37.241 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-03-30T13:37:37.241 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for the mgr to restart... 2024-03-30T13:37:37.241 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mgr epoch 5... 2024-03-30T13:37:38.649 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:38 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:38.238+0000 7f783f33c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T13:37:38.649 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:38 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:38.372+0000 7f783f33c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T13:37:39.009 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:38 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:38.712+0000 7f783f33c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T13:37:39.010 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:38 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:38.807+0000 7f783f33c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T13:37:39.010 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:39 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:39.008+0000 7f783f33c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:37:39.340 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:39 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:39.198+0000 7f783f33c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T13:37:39.649 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:39 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:39.494+0000 7f783f33c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:37:40.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:39 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:39.755+0000 7f783f33c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T13:37:40.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:40 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:39.999+0000 7f783f33c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T13:37:41.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:40 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:40.828+0000 7f783f33c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T13:37:41.733 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:41 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:41.410+0000 7f783f33c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T13:37:41.733 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:41 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:41.511+0000 7f783f33c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T13:37:41.733 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:41 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:37:41.731+0000 7f783f33c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T13:37:42.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:41 smithi043 ceph-mon[29963]: Active manager daemon y restarted 2024-03-30T13:37:42.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:41 smithi043 ceph-mon[29963]: Activating manager daemon y 2024-03-30T13:37:42.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:41 smithi043 ceph-mon[29963]: osdmap e2: 0 total, 0 up, 0 in 2024-03-30T13:37:42.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:41 smithi043 ceph-mon[29963]: mgrmap e6: y(active, starting, since 0.00325437s) 2024-03-30T13:37:42.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:41 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:37:42.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:41 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T13:37:42.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:41 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T13:37:42.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:41 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T13:37:42.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:41 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T13:37:42.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:41 smithi043 ceph-mon[29963]: Manager daemon y is now available 2024-03-30T13:37:42.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:41 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' 2024-03-30T13:37:42.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:41 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' 2024-03-30T13:37:42.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:41 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:37:42.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:41 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:37:42.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:41 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T13:37:42.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:41 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T13:37:43.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:43 smithi043 ceph-mon[29963]: Found migration_current of "None". Setting to last migration. 2024-03-30T13:37:43.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:43 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' 2024-03-30T13:37:43.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:43 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' 2024-03-30T13:37:43.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:43 smithi043 ceph-mon[29963]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-30T13:37:43.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:43 smithi043 ceph-mon[29963]: mgrmap e7: y(active, since 1.00566s) 2024-03-30T13:37:43.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:43 smithi043 ceph-mon[29963]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-30T13:37:43.355 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-03-30T13:37:43.355 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-30T13:37:43.355 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-30T13:37:43.355 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-03-30T13:37:43.355 INFO:teuthology.orchestra.run.smithi043.stdout:mgr epoch 5 is available 2024-03-30T13:37:43.355 INFO:teuthology.orchestra.run.smithi043.stdout:Setting orchestrator backend to cephadm... 2024-03-30T13:37:44.340 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:44 smithi043 ceph-mon[29963]: [30/Mar/2024:13:37:42] ENGINE Bus STARTING 2024-03-30T13:37:44.340 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:44 smithi043 ceph-mon[29963]: [30/Mar/2024:13:37:42] ENGINE Serving on http://172.21.15.43:8765 2024-03-30T13:37:44.340 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:44 smithi043 ceph-mon[29963]: [30/Mar/2024:13:37:43] ENGINE Serving on https://172.21.15.43:7150 2024-03-30T13:37:44.340 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:44 smithi043 ceph-mon[29963]: [30/Mar/2024:13:37:43] ENGINE Bus STARTED 2024-03-30T13:37:44.340 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:44 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:37:44.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:44 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' 2024-03-30T13:37:44.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:44 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:37:45.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:45 smithi043 ceph-mon[29963]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:37:45.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:45 smithi043 ceph-mon[29963]: mgrmap e8: y(active, since 2s) 2024-03-30T13:37:45.916 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-30T13:37:45.916 INFO:teuthology.orchestra.run.smithi043.stdout:Generating ssh key... 2024-03-30T13:37:46.165 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:46 smithi043 ceph-mon[29963]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:37:47.078 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: Generating public/private rsa key pair. 2024-03-30T13:37:47.078 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: Your identification has been saved in /tmp/tmpki3ao6pp/key. 2024-03-30T13:37:47.078 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: Your public key has been saved in /tmp/tmpki3ao6pp/key.pub. 2024-03-30T13:37:47.078 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: The key fingerprint is: 2024-03-30T13:37:47.078 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: SHA256:8gbff5s2E1kKevTqq2e4BGACQvQCVW5zmIgeuRW1HtE ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:37:47.078 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: The key's randomart image is: 2024-03-30T13:37:47.078 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: +---[RSA 3072]----+ 2024-03-30T13:37:47.078 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: |=+oooo. | 2024-03-30T13:37:47.078 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: |.oo+.ooE | 2024-03-30T13:37:47.079 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: |.+.oBo+ | 2024-03-30T13:37:47.079 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: |. =..=.. o .| 2024-03-30T13:37:47.079 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: | o .o S o o + | 2024-03-30T13:37:47.079 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: | = o. . = | 2024-03-30T13:37:47.079 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: | + oo . . | 2024-03-30T13:37:47.079 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: | . ...+ =. | 2024-03-30T13:37:47.079 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: | o*++o+ | 2024-03-30T13:37:47.079 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:37:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: +----[SHA256]-----+ 2024-03-30T13:37:47.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:47 smithi043 ceph-mon[29963]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:37:47.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:47 smithi043 ceph-mon[29963]: Generating ssh key... 2024-03-30T13:37:47.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:47 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' 2024-03-30T13:37:47.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:47 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' 2024-03-30T13:37:48.587 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/fqwGpzZTEv+KOQvSaDMV1NH4HBlXl8GyLc3UC6R2k4HhVYwxRzwaERqL5br6U5hAgGJrLXw84rSjCq7iPWP3PIwWXMfkvgmikF2lhmFsrqRxc7+Kj6E7D8zvjXvaHPSaxSk5IpsipSiBgBXN0BiWeNALk3zNlM2Rs68XY3he0myh1cbLzFgncXqykPomXFZ/yFz7wA/DyHAHsobhodN90djrM1yJ3207ldZmaYPf+tINMMorBzURqBEGMRZzVahInM6wPhQ3J/TIqykP2ozxrvtrCWAe+CVCW1o4or+U1RW71+ddOgETfNVAIHG9VuVyM8EEck701HJLFl0lt4wRMAHLMGgGKbZ5aPUdifkEwZFAIH0Q2fKwNtB63z8lo9hUP5etC3biaK/Dz8sn6uRTmyRcZvdI9OndY3TOtIORRjs/Qz1E0SH/jKqh1UghuHvx1EhHG9i+16EkkgVXu5hKgOcPx7ykf2zmOQLGHFAN23M8/fLFzUCWxr+SuLpOJhk= ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:37:48.587 INFO:teuthology.orchestra.run.smithi043.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-30T13:37:48.587 INFO:teuthology.orchestra.run.smithi043.stdout:Adding key to root@localhost authorized_keys... 2024-03-30T13:37:48.587 INFO:teuthology.orchestra.run.smithi043.stdout:Adding host smithi043... 2024-03-30T13:37:48.837 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:48 smithi043 ceph-mon[29963]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:37:49.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:49 smithi043 ceph-mon[29963]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi043", "addr": "172.21.15.43", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:37:51.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:50 smithi043 ceph-mon[29963]: Deploying cephadm binary to smithi043 2024-03-30T13:37:54.286 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout Added host 'smithi043' with addr '172.21.15.43' 2024-03-30T13:37:54.286 INFO:teuthology.orchestra.run.smithi043.stdout:Deploying unmanaged mon service... 2024-03-30T13:37:54.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:54 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' 2024-03-30T13:37:54.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:54 smithi043 ceph-mon[29963]: Added host smithi043 2024-03-30T13:37:54.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:54 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:37:55.502 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-30T13:37:55.502 INFO:teuthology.orchestra.run.smithi043.stdout:Deploying unmanaged mgr service... 2024-03-30T13:37:56.138 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:55 smithi043 ceph-mon[29963]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:37:56.138 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:55 smithi043 ceph-mon[29963]: Saving service mon spec with placement count:5 2024-03-30T13:37:56.138 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:55 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' 2024-03-30T13:37:57.606 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-30T13:37:57.872 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:57 smithi043 ceph-mon[29963]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:37:57.873 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:57 smithi043 ceph-mon[29963]: Saving service mgr spec with placement count:2 2024-03-30T13:37:57.873 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:57 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' 2024-03-30T13:37:57.873 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:57 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' 2024-03-30T13:37:57.873 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:57 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' 2024-03-30T13:37:59.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:59 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1541368871' entity='client.admin' 2024-03-30T13:37:59.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:37:59 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' 2024-03-30T13:38:00.813 INFO:teuthology.orchestra.run.smithi043.stdout:Enabling the dashboard module... 2024-03-30T13:38:01.422 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:01 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/68926213' entity='client.admin' 2024-03-30T13:38:01.422 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:01 smithi043 ceph-mon[29963]: from='mgr.14120 172.21.15.43:0/87710457' entity='mgr.y' 2024-03-30T13:38:02.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:02 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2770807277' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-30T13:38:02.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:02 smithi043 ceph-mon[29963]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:02.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:02 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ignoring --setuser ceph since I am not root 2024-03-30T13:38:02.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:02 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ignoring --setgroup ceph since I am not root 2024-03-30T13:38:03.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:02 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:02.839+0000 7f3bdf0a9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T13:38:03.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:02 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:02.935+0000 7f3bdf0a9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T13:38:03.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:03 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2770807277' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-30T13:38:03.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:03 smithi043 ceph-mon[29963]: mgrmap e9: y(active, since 20s) 2024-03-30T13:38:03.865 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:03 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:03.604+0000 7f3bdf0a9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T13:38:03.866 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:03 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:03.702+0000 7f3bdf0a9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T13:38:03.866 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:03 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:03.864+0000 7f3bdf0a9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T13:38:04.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:03 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:03.963+0000 7f3bdf0a9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T13:38:04.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:04 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:04.053+0000 7f3bdf0a9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T13:38:04.529 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:04 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2884299389' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-30T13:38:04.529 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:04 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:04.325+0000 7f3bdf0a9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T13:38:04.529 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:04 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:04.439+0000 7f3bdf0a9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T13:38:04.529 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:04 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:04.527+0000 7f3bdf0a9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T13:38:04.666 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-03-30T13:38:04.666 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-30T13:38:04.666 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-30T13:38:04.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-30T13:38:04.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-30T13:38:04.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-03-30T13:38:04.667 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for the mgr to restart... 2024-03-30T13:38:04.667 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mgr epoch 9... 2024-03-30T13:38:04.791 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:04 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:04.625+0000 7f3bdf0a9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T13:38:05.123 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:04 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:04.808+0000 7f3bdf0a9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T13:38:06.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:05 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:05.811+0000 7f3bdf0a9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T13:38:06.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:05 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:05.946+0000 7f3bdf0a9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T13:38:06.583 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:06.286+0000 7f3bdf0a9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T13:38:06.583 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:06.379+0000 7f3bdf0a9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T13:38:06.583 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:06.581+0000 7f3bdf0a9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:38:06.898 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:06.774+0000 7f3bdf0a9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T13:38:07.332 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:07 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:07.067+0000 7f3bdf0a9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:38:07.649 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:07 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:07.330+0000 7f3bdf0a9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T13:38:07.649 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:07 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:07.575+0000 7f3bdf0a9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T13:38:08.898 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:08 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:08.408+0000 7f3bdf0a9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T13:38:09.321 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:08 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:08.990+0000 7f3bdf0a9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T13:38:09.321 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:09 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:09.095+0000 7f3bdf0a9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T13:38:09.321 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:38:09 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:38:09.319+0000 7f3bdf0a9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T13:38:09.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:09 smithi043 ceph-mon[29963]: Active manager daemon y restarted 2024-03-30T13:38:09.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:09 smithi043 ceph-mon[29963]: Activating manager daemon y 2024-03-30T13:38:09.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:09 smithi043 ceph-mon[29963]: osdmap e3: 0 total, 0 up, 0 in 2024-03-30T13:38:09.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:09 smithi043 ceph-mon[29963]: mgrmap e10: y(active, starting, since 0.00337634s) 2024-03-30T13:38:09.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:38:09.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T13:38:09.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T13:38:09.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T13:38:09.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T13:38:09.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:09 smithi043 ceph-mon[29963]: Manager daemon y is now available 2024-03-30T13:38:09.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:38:09.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T13:38:10.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:10 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T13:38:10.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:10 smithi043 ceph-mon[29963]: mgrmap e11: y(active, since 1.00612s) 2024-03-30T13:38:11.300 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-03-30T13:38:11.300 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-30T13:38:11.300 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-30T13:38:11.300 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-03-30T13:38:11.301 INFO:teuthology.orchestra.run.smithi043.stdout:mgr epoch 9 is available 2024-03-30T13:38:11.301 INFO:teuthology.orchestra.run.smithi043.stdout:Generating a dashboard self-signed certificate... 2024-03-30T13:38:12.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:11 smithi043 ceph-mon[29963]: [30/Mar/2024:13:38:10] ENGINE Bus STARTING 2024-03-30T13:38:12.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:11 smithi043 ceph-mon[29963]: [30/Mar/2024:13:38:10] ENGINE Serving on https://172.21.15.43:7150 2024-03-30T13:38:12.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:11 smithi043 ceph-mon[29963]: [30/Mar/2024:13:38:10] ENGINE Serving on http://172.21.15.43:8765 2024-03-30T13:38:12.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:11 smithi043 ceph-mon[29963]: [30/Mar/2024:13:38:10] ENGINE Bus STARTED 2024-03-30T13:38:12.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:11 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:12.755 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-30T13:38:12.755 INFO:teuthology.orchestra.run.smithi043.stdout:Creating initial admin user... 2024-03-30T13:38:13.122 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:12 smithi043 ceph-mon[29963]: mgrmap e12: y(active, since 2s) 2024-03-30T13:38:13.122 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:12 smithi043 ceph-mon[29963]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:38:13.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:12 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:13.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:12 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:13.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:12 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:14.465 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$TildJHCVrcyXfnZoan2f/OJhqG/o6U91WBkU.UjGSHjC7nVefywbK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711805893, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-30T13:38:14.465 INFO:teuthology.orchestra.run.smithi043.stdout:Fetching dashboard port number... 2024-03-30T13:38:14.724 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:14 smithi043 ceph-mon[29963]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:38:14.724 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:14 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:15.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:15 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1332381386' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-30T13:38:16.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 8443 2024-03-30T13:38:16.211 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T13:38:16.211 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout disabled 2024-03-30T13:38:16.218 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T13:38:16.219 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-03-30T13:38:16.219 INFO:teuthology.orchestra.run.smithi043.stdout:firewalld.service is not enabled 2024-03-30T13:38:16.219 INFO:teuthology.orchestra.run.smithi043.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-30T13:38:16.221 INFO:teuthology.orchestra.run.smithi043.stdout:Ceph Dashboard is now available at: 2024-03-30T13:38:16.221 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:38:16.221 INFO:teuthology.orchestra.run.smithi043.stdout: URL: https://smithi043.front.sepia.ceph.com:8443/ 2024-03-30T13:38:16.221 INFO:teuthology.orchestra.run.smithi043.stdout: User: admin 2024-03-30T13:38:16.221 INFO:teuthology.orchestra.run.smithi043.stdout: Password: yaa1bzqq9k 2024-03-30T13:38:16.221 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:38:16.221 INFO:teuthology.orchestra.run.smithi043.stdout:Saving cluster configuration to /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config directory 2024-03-30T13:38:16.222 INFO:teuthology.orchestra.run.smithi043.stdout:Enabling autotune for osd_memory_target 2024-03-30T13:38:16.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:16 smithi043 ceph-mon[29963]: mgrmap e13: y(active, since 6s) 2024-03-30T13:38:18.963 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-30T13:38:18.963 INFO:teuthology.orchestra.run.smithi043.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-30T13:38:18.963 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:38:18.963 INFO:teuthology.orchestra.run.smithi043.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-30T13:38:18.963 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:38:18.963 INFO:teuthology.orchestra.run.smithi043.stdout:Or, if you are only running a single cluster on this host: 2024-03-30T13:38:18.963 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:38:18.963 INFO:teuthology.orchestra.run.smithi043.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-30T13:38:18.963 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:38:18.963 INFO:teuthology.orchestra.run.smithi043.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-30T13:38:18.964 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:38:18.964 INFO:teuthology.orchestra.run.smithi043.stdout: ceph telemetry on 2024-03-30T13:38:18.964 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:38:18.964 INFO:teuthology.orchestra.run.smithi043.stdout:For more information see: 2024-03-30T13:38:18.964 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:38:18.964 INFO:teuthology.orchestra.run.smithi043.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-30T13:38:18.964 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:38:18.964 INFO:teuthology.orchestra.run.smithi043.stdout:Bootstrap complete. 2024-03-30T13:38:19.008 INFO:tasks.cephadm:Fetching config... 2024-03-30T13:38:19.009 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-30T13:38:19.009 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-30T13:38:19.024 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-30T13:38:19.024 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-30T13:38:19.024 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-30T13:38:19.079 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-30T13:38:19.079 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-30T13:38:19.079 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/keyring of=/dev/stdout 2024-03-30T13:38:19.146 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-30T13:38:19.146 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-30T13:38:19.146 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-30T13:38:19.201 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-30T13:38:19.201 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/fqwGpzZTEv+KOQvSaDMV1NH4HBlXl8GyLc3UC6R2k4HhVYwxRzwaERqL5br6U5hAgGJrLXw84rSjCq7iPWP3PIwWXMfkvgmikF2lhmFsrqRxc7+Kj6E7D8zvjXvaHPSaxSk5IpsipSiBgBXN0BiWeNALk3zNlM2Rs68XY3he0myh1cbLzFgncXqykPomXFZ/yFz7wA/DyHAHsobhodN90djrM1yJ3207ldZmaYPf+tINMMorBzURqBEGMRZzVahInM6wPhQ3J/TIqykP2ozxrvtrCWAe+CVCW1o4or+U1RW71+ddOgETfNVAIHG9VuVyM8EEck701HJLFl0lt4wRMAHLMGgGKbZ5aPUdifkEwZFAIH0Q2fKwNtB63z8lo9hUP5etC3biaK/Dz8sn6uRTmyRcZvdI9OndY3TOtIORRjs/Qz1E0SH/jKqh1UghuHvx1EhHG9i+16EkkgVXu5hKgOcPx7ykf2zmOQLGHFAN23M8/fLFzUCWxr+SuLpOJhk= ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-30T13:38:19.291 INFO:teuthology.orchestra.run.smithi043.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/fqwGpzZTEv+KOQvSaDMV1NH4HBlXl8GyLc3UC6R2k4HhVYwxRzwaERqL5br6U5hAgGJrLXw84rSjCq7iPWP3PIwWXMfkvgmikF2lhmFsrqRxc7+Kj6E7D8zvjXvaHPSaxSk5IpsipSiBgBXN0BiWeNALk3zNlM2Rs68XY3he0myh1cbLzFgncXqykPomXFZ/yFz7wA/DyHAHsobhodN90djrM1yJ3207ldZmaYPf+tINMMorBzURqBEGMRZzVahInM6wPhQ3J/TIqykP2ozxrvtrCWAe+CVCW1o4or+U1RW71+ddOgETfNVAIHG9VuVyM8EEck701HJLFl0lt4wRMAHLMGgGKbZ5aPUdifkEwZFAIH0Q2fKwNtB63z8lo9hUP5etC3biaK/Dz8sn6uRTmyRcZvdI9OndY3TOtIORRjs/Qz1E0SH/jKqh1UghuHvx1EhHG9i+16EkkgVXu5hKgOcPx7ykf2zmOQLGHFAN23M8/fLFzUCWxr+SuLpOJhk= ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:38:19.317 DEBUG:teuthology.orchestra.run.smithi113:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/fqwGpzZTEv+KOQvSaDMV1NH4HBlXl8GyLc3UC6R2k4HhVYwxRzwaERqL5br6U5hAgGJrLXw84rSjCq7iPWP3PIwWXMfkvgmikF2lhmFsrqRxc7+Kj6E7D8zvjXvaHPSaxSk5IpsipSiBgBXN0BiWeNALk3zNlM2Rs68XY3he0myh1cbLzFgncXqykPomXFZ/yFz7wA/DyHAHsobhodN90djrM1yJ3207ldZmaYPf+tINMMorBzURqBEGMRZzVahInM6wPhQ3J/TIqykP2ozxrvtrCWAe+CVCW1o4or+U1RW71+ddOgETfNVAIHG9VuVyM8EEck701HJLFl0lt4wRMAHLMGgGKbZ5aPUdifkEwZFAIH0Q2fKwNtB63z8lo9hUP5etC3biaK/Dz8sn6uRTmyRcZvdI9OndY3TOtIORRjs/Qz1E0SH/jKqh1UghuHvx1EhHG9i+16EkkgVXu5hKgOcPx7ykf2zmOQLGHFAN23M8/fLFzUCWxr+SuLpOJhk= ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-30T13:38:19.362 INFO:teuthology.orchestra.run.smithi113.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/fqwGpzZTEv+KOQvSaDMV1NH4HBlXl8GyLc3UC6R2k4HhVYwxRzwaERqL5br6U5hAgGJrLXw84rSjCq7iPWP3PIwWXMfkvgmikF2lhmFsrqRxc7+Kj6E7D8zvjXvaHPSaxSk5IpsipSiBgBXN0BiWeNALk3zNlM2Rs68XY3he0myh1cbLzFgncXqykPomXFZ/yFz7wA/DyHAHsobhodN90djrM1yJ3207ldZmaYPf+tINMMorBzURqBEGMRZzVahInM6wPhQ3J/TIqykP2ozxrvtrCWAe+CVCW1o4or+U1RW71+ddOgETfNVAIHG9VuVyM8EEck701HJLFl0lt4wRMAHLMGgGKbZ5aPUdifkEwZFAIH0Q2fKwNtB63z8lo9hUP5etC3biaK/Dz8sn6uRTmyRcZvdI9OndY3TOtIORRjs/Qz1E0SH/jKqh1UghuHvx1EhHG9i+16EkkgVXu5hKgOcPx7ykf2zmOQLGHFAN23M8/fLFzUCWxr+SuLpOJhk= ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:38:19.377 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-30T13:38:19.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:19 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4091568689' entity='client.admin' 2024-03-30T13:38:19.618 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:38:23.578 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:23 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4026400082' entity='client.admin' 2024-03-30T13:38:23.578 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:23 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:23.578 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:23 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:23.578 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:23 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi043", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:38:23.578 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:23 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:23.578 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:23 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:38:23.578 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:23 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:23.604 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-30T13:38:23.604 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-30T13:38:23.843 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:38:26.381 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi113 2024-03-30T13:38:26.381 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-30T13:38:26.381 DEBUG:teuthology.orchestra.run.smithi113:> dd of=/etc/ceph/ceph.conf 2024-03-30T13:38:26.399 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-30T13:38:26.399 DEBUG:teuthology.orchestra.run.smithi113:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:38:26.456 INFO:tasks.cephadm:Adding host smithi113 to orchestrator... 2024-03-30T13:38:26.456 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph orch host add smithi113 2024-03-30T13:38:26.695 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:38:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:26 smithi043 ceph-mon[29963]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:38:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:26 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:26 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:38:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:26 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:38:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:26 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:38:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:26 smithi043 ceph-mon[29963]: Updating smithi043:/etc/ceph/ceph.conf 2024-03-30T13:38:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:26 smithi043 ceph-mon[29963]: Updating smithi043:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:38:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:26 smithi043 ceph-mon[29963]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:38:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:26 smithi043 ceph-mon[29963]: Updating smithi043:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T13:38:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:26 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:26 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:26.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:26 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:29.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:29 smithi043 ceph-mon[29963]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi113", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:38:29.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:29 smithi043 ceph-mon[29963]: Deploying cephadm binary to smithi113 2024-03-30T13:38:30.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:30 smithi043 ceph-mon[29963]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:32.648 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:32 smithi043 ceph-mon[29963]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:32.879 INFO:teuthology.orchestra.run.smithi043.stdout:Added host 'smithi113' with addr '172.21.15.113' 2024-03-30T13:38:33.464 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph orch host ls --format=json 2024-03-30T13:38:33.712 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:38:34.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:33 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:34.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:33 smithi043 ceph-mon[29963]: Added host smithi113 2024-03-30T13:38:34.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:33 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:38:34.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:33 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:35.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:34 smithi043 ceph-mon[29963]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:35.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:34 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:35.673 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:38:35.673 INFO:teuthology.orchestra.run.smithi043.stdout:[{"addr": "172.21.15.43", "hostname": "smithi043", "labels": [], "status": ""}, {"addr": "172.21.15.113", "hostname": "smithi113", "labels": [], "status": ""}] 2024-03-30T13:38:36.324 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-30T13:38:36.324 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd crush tunables default 2024-03-30T13:38:36.565 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:38:36.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:36 smithi043 ceph-mon[29963]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:36.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:36 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:36.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:36 smithi043 ceph-mon[29963]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:38:38.453 INFO:teuthology.orchestra.run.smithi043.stderr:adjusted tunables profile to default 2024-03-30T13:38:38.747 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:38 smithi043 ceph-mon[29963]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:38.747 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:38 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3289933880' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-30T13:38:39.090 INFO:tasks.cephadm:Adding mon.a on smithi043 2024-03-30T13:38:39.090 INFO:tasks.cephadm:Adding mon.c on smithi043 2024-03-30T13:38:39.090 INFO:tasks.cephadm:Adding mon.b on smithi113 2024-03-30T13:38:39.091 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph orch apply mon '3;smithi043:172.21.15.43=a;smithi043:[v2:172.21.15.43:3301,v1:172.21.15.43:6790]=c;smithi113:172.21.15.113=b' 2024-03-30T13:38:39.324 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:38:39.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:39 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3289933880' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-30T13:38:39.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:39 smithi043 ceph-mon[29963]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T13:38:39.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:39 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:39.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:39 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:39.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:39 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:39.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:39 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:39.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:39 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi113", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:38:39.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:39 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:38:39.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:39 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:38:39.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:39 smithi043 ceph-mon[29963]: Updating smithi113:/etc/ceph/ceph.conf 2024-03-30T13:38:39.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:39 smithi043 ceph-mon[29963]: Updating smithi113:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:38:39.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:39 smithi043 ceph-mon[29963]: Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:38:39.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:39 smithi043 ceph-mon[29963]: Updating smithi113:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T13:38:39.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:39 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:39.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:39 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:39.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:39 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:40.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:40 smithi043 ceph-mon[29963]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:40.998 INFO:teuthology.orchestra.run.smithi113.stdout:Scheduled mon update... 2024-03-30T13:38:41.596 DEBUG:teuthology.orchestra.run.smithi043:mon.c> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mon.c.service 2024-03-30T13:38:41.599 DEBUG:teuthology.orchestra.run.smithi113:mon.b> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mon.b.service 2024-03-30T13:38:41.601 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-30T13:38:41.601 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T13:38:42.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:41 smithi043 ceph-mon[29963]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi043:172.21.15.43=a;smithi043:[v2:172.21.15.43:3301,v1:172.21.15.43:6790]=c;smithi113:172.21.15.113=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:38:42.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:41 smithi043 ceph-mon[29963]: Saving service mon spec with placement smithi043:172.21.15.43=a;smithi043:[v2:172.21.15.43:3301,v1:172.21.15.43:6790]=c;smithi113:172.21.15.113=b;count:3 2024-03-30T13:38:42.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:41 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:42.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:41 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:38:42.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:41 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:38:42.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:41 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:38:42.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:41 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:42.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:41 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:38:42.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:41 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:38:42.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:41 smithi043 ceph-mon[29963]: Deploying daemon mon.b on smithi113 2024-03-30T13:38:42.659 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:38:43.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:43 smithi043 ceph-mon[29963]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:44.946 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:44 smithi113 systemd[1]: Starting Ceph mon.b for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T13:38:45.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:45 smithi043 ceph-mon[29963]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:45.540 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 podman[34294]: 2024-03-30 13:38:45.444690283 +0000 UTC m=+0.092221485 container create 90cc320c3d3242a1d38ceb76f70f123f4ca7b904538080266f3d9b5f0dbb0190 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-b, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-30T13:38:45.540 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 podman[34294]: 2024-03-30 13:38:45.370289456 +0000 UTC m=+0.017820670 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T13:38:45.652 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:38:45.652 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":1,"fsid":"7ee932fa-ee9a-11ee-b647-cb9ed24678a4","modified":"2024-03-30T13:37:04.571563Z","created":"2024-03-30T13:37:04.571563Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T13:38:45.654 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 1 2024-03-30T13:38:45.909 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 podman[34294]: 2024-03-30 13:38:45.605289422 +0000 UTC m=+0.252820621 container init 90cc320c3d3242a1d38ceb76f70f123f4ca7b904538080266f3d9b5f0dbb0190 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-30T13:38:45.909 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 podman[34294]: 2024-03-30 13:38:45.610471968 +0000 UTC m=+0.258003172 container start 90cc320c3d3242a1d38ceb76f70f123f4ca7b904538080266f3d9b5f0dbb0190 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-b, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-30T13:38:45.909 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: set uid:gid to 167:167 (ceph:ceph) 2024-03-30T13:38:45.909 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev), process ceph-mon, pid 7 2024-03-30T13:38:45.910 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: pidfile_write: ignore empty --pid-file 2024-03-30T13:38:45.910 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: load: jerasure load: lrc 2024-03-30T13:38:45.910 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: RocksDB version: 7.9.2 2024-03-30T13:38:45.910 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Git sha 0 2024-03-30T13:38:45.910 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Compile date 2024-03-29 15:06:16 2024-03-30T13:38:45.910 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: DB SUMMARY 2024-03-30T13:38:45.910 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: DB Session ID: 7EUDLFI0D6RK56ZYHLOV 2024-03-30T13:38:45.910 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: CURRENT file: CURRENT 2024-03-30T13:38:45.910 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: IDENTITY file: IDENTITY 2024-03-30T13:38:45.910 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-30T13:38:45.911 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-30T13:38:45.911 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-03-30T13:38:45.911 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.error_if_exists: 0 2024-03-30T13:38:45.911 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.create_if_missing: 0 2024-03-30T13:38:45.911 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.paranoid_checks: 1 2024-03-30T13:38:45.911 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-30T13:38:45.911 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-30T13:38:45.911 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-30T13:38:45.911 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.env: 0x55d910e9ec60 2024-03-30T13:38:45.911 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.fs: PosixFileSystem 2024-03-30T13:38:45.911 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.info_log: 0x55d912607ae0 2024-03-30T13:38:45.912 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_file_opening_threads: 16 2024-03-30T13:38:45.912 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.statistics: (nil) 2024-03-30T13:38:45.912 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.use_fsync: 0 2024-03-30T13:38:45.912 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_log_file_size: 0 2024-03-30T13:38:45.912 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-30T13:38:45.912 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-30T13:38:45.912 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.keep_log_file_num: 1000 2024-03-30T13:38:45.912 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.recycle_log_file_num: 0 2024-03-30T13:38:45.912 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.allow_fallocate: 1 2024-03-30T13:38:45.912 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.allow_mmap_reads: 0 2024-03-30T13:38:45.912 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.allow_mmap_writes: 0 2024-03-30T13:38:45.912 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.use_direct_reads: 0 2024-03-30T13:38:45.913 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-30T13:38:45.913 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.create_missing_column_families: 0 2024-03-30T13:38:45.913 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.db_log_dir: 2024-03-30T13:38:45.913 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.wal_dir: 2024-03-30T13:38:45.913 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-30T13:38:45.913 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-30T13:38:45.913 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-30T13:38:45.913 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-30T13:38:45.913 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-30T13:38:45.913 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-30T13:38:45.913 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.advise_random_on_open: 1 2024-03-30T13:38:45.913 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.db_write_buffer_size: 0 2024-03-30T13:38:45.913 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.write_buffer_manager: 0x55d9125da1e0 2024-03-30T13:38:45.913 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-30T13:38:45.914 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-30T13:38:45.914 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-30T13:38:45.914 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.rate_limiter: (nil) 2024-03-30T13:38:45.914 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-30T13:38:45.914 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.wal_recovery_mode: 2 2024-03-30T13:38:45.914 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.enable_thread_tracking: 0 2024-03-30T13:38:45.914 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.enable_pipelined_write: 0 2024-03-30T13:38:45.914 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.unordered_write: 0 2024-03-30T13:38:45.914 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-30T13:38:45.914 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-30T13:38:45.914 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-30T13:38:45.914 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-30T13:38:45.914 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.row_cache: None 2024-03-30T13:38:45.915 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.wal_filter: None 2024-03-30T13:38:45.916 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-30T13:38:45.916 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.allow_ingest_behind: 0 2024-03-30T13:38:45.916 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.two_write_queues: 0 2024-03-30T13:38:45.916 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.manual_wal_flush: 0 2024-03-30T13:38:45.916 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.wal_compression: 0 2024-03-30T13:38:45.916 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.atomic_flush: 0 2024-03-30T13:38:45.916 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-30T13:38:45.916 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-30T13:38:45.916 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-30T13:38:45.916 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.log_readahead_size: 0 2024-03-30T13:38:45.917 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-30T13:38:45.917 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.best_efforts_recovery: 0 2024-03-30T13:38:45.917 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-30T13:38:45.917 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-30T13:38:45.917 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.allow_data_in_errors: 0 2024-03-30T13:38:45.917 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.db_host_id: __hostname__ 2024-03-30T13:38:45.917 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-30T13:38:45.917 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_background_jobs: 2 2024-03-30T13:38:45.917 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_background_compactions: -1 2024-03-30T13:38:45.917 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_subcompactions: 1 2024-03-30T13:38:45.917 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-30T13:38:45.917 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-30T13:38:45.917 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-30T13:38:45.917 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_total_wal_size: 0 2024-03-30T13:38:45.918 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-30T13:38:45.918 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-30T13:38:45.918 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-30T13:38:45.918 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-30T13:38:45.918 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_open_files: -1 2024-03-30T13:38:45.918 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.bytes_per_sync: 0 2024-03-30T13:38:45.918 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-30T13:38:45.918 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-30T13:38:45.918 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compaction_readahead_size: 0 2024-03-30T13:38:45.918 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_background_flushes: -1 2024-03-30T13:38:45.918 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Compression algorithms supported: 2024-03-30T13:38:45.918 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-30T13:38:45.918 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: kZSTD supported: 0 2024-03-30T13:38:45.918 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: kXpressCompression supported: 0 2024-03-30T13:38:45.919 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: kLZ4HCCompression supported: 1 2024-03-30T13:38:45.919 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: kZlibCompression supported: 1 2024-03-30T13:38:45.919 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: kSnappyCompression supported: 1 2024-03-30T13:38:45.919 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: kLZ4Compression supported: 1 2024-03-30T13:38:45.919 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: kBZip2Compression supported: 0 2024-03-30T13:38:45.919 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-30T13:38:45.919 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-30T13:38:45.919 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-30T13:38:45.919 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-30T13:38:45.919 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-30T13:38:45.919 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.merge_operator: 2024-03-30T13:38:45.919 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compaction_filter: None 2024-03-30T13:38:45.919 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compaction_filter_factory: None 2024-03-30T13:38:45.920 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.sst_partitioner_factory: None 2024-03-30T13:38:45.920 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-30T13:38:45.920 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-30T13:38:45.920 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d912607be0) 2024-03-30T13:38:45.920 INFO:journalctl@ceph.mon.b.smithi113.stdout: cache_index_and_filter_blocks: 1 2024-03-30T13:38:45.920 INFO:journalctl@ceph.mon.b.smithi113.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-30T13:38:45.920 INFO:journalctl@ceph.mon.b.smithi113.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-30T13:38:45.920 INFO:journalctl@ceph.mon.b.smithi113.stdout: pin_top_level_index_and_filter: 1 2024-03-30T13:38:45.920 INFO:journalctl@ceph.mon.b.smithi113.stdout: index_type: 0 2024-03-30T13:38:45.920 INFO:journalctl@ceph.mon.b.smithi113.stdout: data_block_index_type: 0 2024-03-30T13:38:45.920 INFO:journalctl@ceph.mon.b.smithi113.stdout: index_shortening: 1 2024-03-30T13:38:45.920 INFO:journalctl@ceph.mon.b.smithi113.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-30T13:38:45.920 INFO:journalctl@ceph.mon.b.smithi113.stdout: checksum: 4 2024-03-30T13:38:45.920 INFO:journalctl@ceph.mon.b.smithi113.stdout: no_block_cache: 0 2024-03-30T13:38:45.920 INFO:journalctl@ceph.mon.b.smithi113.stdout: block_cache: 0x55d91260b090 2024-03-30T13:38:45.921 INFO:journalctl@ceph.mon.b.smithi113.stdout: block_cache_name: BinnedLRUCache 2024-03-30T13:38:45.921 INFO:journalctl@ceph.mon.b.smithi113.stdout: block_cache_options: 2024-03-30T13:38:45.921 INFO:journalctl@ceph.mon.b.smithi113.stdout: capacity : 536870912 2024-03-30T13:38:45.921 INFO:journalctl@ceph.mon.b.smithi113.stdout: num_shard_bits : 4 2024-03-30T13:38:45.921 INFO:journalctl@ceph.mon.b.smithi113.stdout: strict_capacity_limit : 0 2024-03-30T13:38:45.921 INFO:journalctl@ceph.mon.b.smithi113.stdout: high_pri_pool_ratio: 0.000 2024-03-30T13:38:45.921 INFO:journalctl@ceph.mon.b.smithi113.stdout: block_cache_compressed: (nil) 2024-03-30T13:38:45.921 INFO:journalctl@ceph.mon.b.smithi113.stdout: persistent_cache: (nil) 2024-03-30T13:38:45.921 INFO:journalctl@ceph.mon.b.smithi113.stdout: block_size: 4096 2024-03-30T13:38:45.921 INFO:journalctl@ceph.mon.b.smithi113.stdout: block_size_deviation: 10 2024-03-30T13:38:45.921 INFO:journalctl@ceph.mon.b.smithi113.stdout: block_restart_interval: 16 2024-03-30T13:38:45.921 INFO:journalctl@ceph.mon.b.smithi113.stdout: index_block_restart_interval: 1 2024-03-30T13:38:45.921 INFO:journalctl@ceph.mon.b.smithi113.stdout: metadata_block_size: 4096 2024-03-30T13:38:45.921 INFO:journalctl@ceph.mon.b.smithi113.stdout: partition_filters: 0 2024-03-30T13:38:45.921 INFO:journalctl@ceph.mon.b.smithi113.stdout: use_delta_encoding: 1 2024-03-30T13:38:45.922 INFO:journalctl@ceph.mon.b.smithi113.stdout: filter_policy: bloomfilter 2024-03-30T13:38:45.922 INFO:journalctl@ceph.mon.b.smithi113.stdout: whole_key_filtering: 1 2024-03-30T13:38:45.922 INFO:journalctl@ceph.mon.b.smithi113.stdout: verify_compression: 0 2024-03-30T13:38:45.922 INFO:journalctl@ceph.mon.b.smithi113.stdout: read_amp_bytes_per_bit: 0 2024-03-30T13:38:45.922 INFO:journalctl@ceph.mon.b.smithi113.stdout: format_version: 5 2024-03-30T13:38:45.922 INFO:journalctl@ceph.mon.b.smithi113.stdout: enable_index_compression: 1 2024-03-30T13:38:45.922 INFO:journalctl@ceph.mon.b.smithi113.stdout: block_align: 0 2024-03-30T13:38:45.922 INFO:journalctl@ceph.mon.b.smithi113.stdout: max_auto_readahead_size: 262144 2024-03-30T13:38:45.922 INFO:journalctl@ceph.mon.b.smithi113.stdout: prepopulate_block_cache: 0 2024-03-30T13:38:45.922 INFO:journalctl@ceph.mon.b.smithi113.stdout: initial_auto_readahead_size: 8192 2024-03-30T13:38:45.922 INFO:journalctl@ceph.mon.b.smithi113.stdout: num_file_reads_for_auto_readahead: 2 2024-03-30T13:38:45.922 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.write_buffer_size: 33554432 2024-03-30T13:38:45.922 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_write_buffer_number: 2 2024-03-30T13:38:45.922 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compression: NoCompression 2024-03-30T13:38:45.922 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.bottommost_compression: Disabled 2024-03-30T13:38:45.923 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.prefix_extractor: nullptr 2024-03-30T13:38:45.923 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-30T13:38:45.923 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.num_levels: 7 2024-03-30T13:38:45.923 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-30T13:38:45.923 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-30T13:38:45.923 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-30T13:38:45.923 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-30T13:38:45.923 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-30T13:38:45.923 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-30T13:38:45.923 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-30T13:38:45.923 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-30T13:38:45.923 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-30T13:38:45.923 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-30T13:38:45.923 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-30T13:38:45.923 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-30T13:38:45.924 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-30T13:38:45.924 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compression_opts.level: 32767 2024-03-30T13:38:45.924 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compression_opts.strategy: 0 2024-03-30T13:38:45.924 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-30T13:38:45.924 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-30T13:38:45.924 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-30T13:38:45.924 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-30T13:38:45.924 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compression_opts.enabled: false 2024-03-30T13:38:45.924 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-30T13:38:45.924 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-30T13:38:45.924 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-30T13:38:45.924 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-30T13:38:45.924 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.target_file_size_base: 67108864 2024-03-30T13:38:45.924 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-30T13:38:45.925 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-30T13:38:45.925 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-30T13:38:45.925 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-30T13:38:45.925 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-30T13:38:45.925 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-30T13:38:45.925 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-30T13:38:45.925 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-30T13:38:45.925 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-30T13:38:45.925 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-30T13:38:45.925 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-30T13:38:45.925 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-30T13:38:45.925 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-30T13:38:45.925 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-30T13:38:45.925 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.arena_block_size: 1048576 2024-03-30T13:38:45.925 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-30T13:38:45.926 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-30T13:38:45.926 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.disable_auto_compactions: 0 2024-03-30T13:38:45.926 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-30T13:38:45.926 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-30T13:38:45.926 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-30T13:38:45.926 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-30T13:38:45.926 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-30T13:38:45.926 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-30T13:38:45.926 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-30T13:38:45.926 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-30T13:38:45.926 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-30T13:38:45.926 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-30T13:38:45.926 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.table_properties_collectors: 2024-03-30T13:38:45.926 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.inplace_update_support: 0 2024-03-30T13:38:45.927 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-30T13:38:45.927 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-30T13:38:45.927 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-30T13:38:45.927 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-30T13:38:45.927 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.bloom_locality: 0 2024-03-30T13:38:45.927 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.max_successive_merges: 0 2024-03-30T13:38:45.927 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-30T13:38:45.927 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.paranoid_file_checks: 0 2024-03-30T13:38:45.927 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.force_consistency_checks: 1 2024-03-30T13:38:45.927 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.report_bg_io_stats: 0 2024-03-30T13:38:45.927 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.ttl: 2592000 2024-03-30T13:38:45.927 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-30T13:38:45.927 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-30T13:38:45.927 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-30T13:38:45.928 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.enable_blob_files: false 2024-03-30T13:38:45.928 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.min_blob_size: 0 2024-03-30T13:38:45.928 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.blob_file_size: 268435456 2024-03-30T13:38:45.928 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-30T13:38:45.928 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-30T13:38:45.928 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-30T13:38:45.928 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-30T13:38:45.928 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-30T13:38:45.928 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.blob_file_starting_level: 0 2024-03-30T13:38:45.928 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-30T13:38:45.928 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-30T13:38:45.928 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-30T13:38:45.928 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e8abf38e-fbdc-4bb5-94fa-9d1e3f116048 2024-03-30T13:38:45.929 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711805925651279, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-30T13:38:45.929 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-30T13:38:45.929 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711805925651818, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711805925, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e8abf38e-fbdc-4bb5-94fa-9d1e3f116048", "db_session_id": "7EUDLFI0D6RK56ZYHLOV", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-30T13:38:45.929 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711805925651919, "job": 1, "event": "recovery_finished"} 2024-03-30T13:38:45.929 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-30T13:38:45.929 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-30T13:38:45.929 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d9126fc000 2024-03-30T13:38:45.929 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: DB pointer 0x55d9126e6000 2024-03-30T13:38:45.929 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-30T13:38:45.929 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-30T13:38:45.929 INFO:journalctl@ceph.mon.b.smithi113.stdout: ** DB Stats ** 2024-03-30T13:38:45.929 INFO:journalctl@ceph.mon.b.smithi113.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T13:38:45.929 INFO:journalctl@ceph.mon.b.smithi113.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-30T13:38:45.929 INFO:journalctl@ceph.mon.b.smithi113.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T13:38:45.930 INFO:journalctl@ceph.mon.b.smithi113.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T13:38:45.930 INFO:journalctl@ceph.mon.b.smithi113.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-30T13:38:45.930 INFO:journalctl@ceph.mon.b.smithi113.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T13:38:45.930 INFO:journalctl@ceph.mon.b.smithi113.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T13:38:45.930 INFO:journalctl@ceph.mon.b.smithi113.stdout: 2024-03-30T13:38:45.930 INFO:journalctl@ceph.mon.b.smithi113.stdout: ** Compaction Stats [default] ** 2024-03-30T13:38:45.930 INFO:journalctl@ceph.mon.b.smithi113.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T13:38:45.930 INFO:journalctl@ceph.mon.b.smithi113.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-30T13:38:45.930 INFO:journalctl@ceph.mon.b.smithi113.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T13:38:45.930 INFO:journalctl@ceph.mon.b.smithi113.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T13:38:45.930 INFO:journalctl@ceph.mon.b.smithi113.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T13:38:45.930 INFO:journalctl@ceph.mon.b.smithi113.stdout: 2024-03-30T13:38:45.931 INFO:journalctl@ceph.mon.b.smithi113.stdout: ** Compaction Stats [default] ** 2024-03-30T13:38:45.931 INFO:journalctl@ceph.mon.b.smithi113.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T13:38:45.931 INFO:journalctl@ceph.mon.b.smithi113.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T13:38:45.931 INFO:journalctl@ceph.mon.b.smithi113.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T13:38:45.931 INFO:journalctl@ceph.mon.b.smithi113.stdout: 2024-03-30T13:38:45.931 INFO:journalctl@ceph.mon.b.smithi113.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-30T13:38:45.931 INFO:journalctl@ceph.mon.b.smithi113.stdout: 2024-03-30T13:38:45.931 INFO:journalctl@ceph.mon.b.smithi113.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T13:38:45.931 INFO:journalctl@ceph.mon.b.smithi113.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-30T13:38:45.931 INFO:journalctl@ceph.mon.b.smithi113.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-30T13:38:45.931 INFO:journalctl@ceph.mon.b.smithi113.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-30T13:38:45.931 INFO:journalctl@ceph.mon.b.smithi113.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-30T13:38:45.931 INFO:journalctl@ceph.mon.b.smithi113.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-30T13:38:45.931 INFO:journalctl@ceph.mon.b.smithi113.stdout: Cumulative compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T13:38:45.931 INFO:journalctl@ceph.mon.b.smithi113.stdout: Interval compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T13:38:45.932 INFO:journalctl@ceph.mon.b.smithi113.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-30T13:38:45.932 INFO:journalctl@ceph.mon.b.smithi113.stdout: Block cache BinnedLRUCache@0x55d91260b090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-03-30T13:38:45.932 INFO:journalctl@ceph.mon.b.smithi113.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-30T13:38:45.932 INFO:journalctl@ceph.mon.b.smithi113.stdout: 2024-03-30T13:38:45.932 INFO:journalctl@ceph.mon.b.smithi113.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-30T13:38:45.932 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-30T13:38:45.932 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: using public_addr v2:172.21.15.113:0/0 -> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] 2024-03-30T13:38:45.932 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: starting mon.b rank -1 at public addrs [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] at bind addrs [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:38:45.932 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: mon.b@-1(???) e0 preinit fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:38:45.932 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: mon.b@-1(synchronizing).mds e1 new map 2024-03-30T13:38:45.932 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-30T13:38:45.932 INFO:journalctl@ceph.mon.b.smithi113.stdout: e1 2024-03-30T13:38:45.932 INFO:journalctl@ceph.mon.b.smithi113.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T13:38:45.932 INFO:journalctl@ceph.mon.b.smithi113.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-30T13:38:45.932 INFO:journalctl@ceph.mon.b.smithi113.stdout: legacy client fscid: -1 2024-03-30T13:38:45.933 INFO:journalctl@ceph.mon.b.smithi113.stdout: 2024-03-30T13:38:45.933 INFO:journalctl@ceph.mon.b.smithi113.stdout: No filesystems configured 2024-03-30T13:38:45.933 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-30T13:38:45.933 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-30T13:38:45.933 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-30T13:38:45.933 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-30T13:38:45.933 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-30T13:38:45.933 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-30T13:38:45.933 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-30T13:38:45.933 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T13:38:45.933 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T13:38:45.933 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T13:38:45.933 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3289933880' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-30T13:38:45.933 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T13:38:45.933 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:45.933 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:45.934 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:45.934 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:45.934 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi113", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:38:45.934 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:38:45.934 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:38:45.934 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: Updating smithi113:/etc/ceph/ceph.conf 2024-03-30T13:38:45.934 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: Updating smithi113:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:38:45.934 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:38:45.934 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: Updating smithi113:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T13:38:45.934 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:45.934 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:45.934 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:45.934 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:45.935 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi043:172.21.15.43=a;smithi043:[v2:172.21.15.43:3301,v1:172.21.15.43:6790]=c;smithi113:172.21.15.113=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:38:45.935 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: Saving service mon spec with placement smithi043:172.21.15.43=a;smithi043:[v2:172.21.15.43:3301,v1:172.21.15.43:6790]=c;smithi113:172.21.15.113=b;count:3 2024-03-30T13:38:45.935 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:45.935 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:38:45.935 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:38:45.935 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:38:45.935 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:45.935 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:38:45.935 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:38:45.935 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: Deploying daemon mon.b on smithi113 2024-03-30T13:38:45.935 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:45.935 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:45.935 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 ceph-mon[34341]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-30T13:38:45.935 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 bash[34294]: 90cc320c3d3242a1d38ceb76f70f123f4ca7b904538080266f3d9b5f0dbb0190 2024-03-30T13:38:45.936 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:45 smithi113 systemd[1]: Started Ceph mon.b for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T13:38:47.338 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-30T13:38:47.338 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T13:38:47.589 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:38:51.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:51.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:38:51.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: mon.a calling monitor election 2024-03-30T13:38:51.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:51.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:51.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:51.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:51.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: mon.b calling monitor election 2024-03-30T13:38:51.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:51.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:51.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:51.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:51.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-30T13:38:51.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: monmap epoch 2 2024-03-30T13:38:51.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:38:51.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: last_changed 2024-03-30T13:38:45.680430+0000 2024-03-30T13:38:51.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: created 2024-03-30T13:37:04.571563+0000 2024-03-30T13:38:51.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: min_mon_release 19 (squid) 2024-03-30T13:38:51.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: election_strategy: 1 2024-03-30T13:38:51.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-03-30T13:38:51.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: 1: [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon.b 2024-03-30T13:38:51.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: fsmap 2024-03-30T13:38:51.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T13:38:51.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: mgrmap e13: y(active, since 41s) 2024-03-30T13:38:51.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: overall HEALTH_OK 2024-03-30T13:38:51.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:51.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:51.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:51.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:38:51.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:50 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:38:51.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:51.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:38:51.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: mon.a calling monitor election 2024-03-30T13:38:51.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:51.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:51.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:51.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:51.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: mon.b calling monitor election 2024-03-30T13:38:51.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:51.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:51.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:51.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:51.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-30T13:38:51.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: monmap epoch 2 2024-03-30T13:38:51.224 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:38:51.224 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: last_changed 2024-03-30T13:38:45.680430+0000 2024-03-30T13:38:51.224 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: created 2024-03-30T13:37:04.571563+0000 2024-03-30T13:38:51.224 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: min_mon_release 19 (squid) 2024-03-30T13:38:51.224 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: election_strategy: 1 2024-03-30T13:38:51.224 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-03-30T13:38:51.224 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: 1: [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon.b 2024-03-30T13:38:51.224 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: fsmap 2024-03-30T13:38:51.224 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T13:38:51.224 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: mgrmap e13: y(active, since 41s) 2024-03-30T13:38:51.224 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: overall HEALTH_OK 2024-03-30T13:38:51.224 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:51.224 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:51.224 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:51.225 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:38:51.225 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:50 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:38:52.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:51 smithi043 ceph-mon[29963]: Deploying daemon mon.c on smithi043 2024-03-30T13:38:52.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:51 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:52.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:51 smithi113 ceph-mon[34341]: Deploying daemon mon.c on smithi043 2024-03-30T13:38:52.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:51 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:52.284 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:38:52.285 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":2,"fsid":"7ee932fa-ee9a-11ee-b647-cb9ed24678a4","modified":"2024-03-30T13:38:45.680430Z","created":"2024-03-30T13:37:04.571563Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-30T13:38:52.286 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 2 2024-03-30T13:38:52.864 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:52 smithi113 ceph-mon[34341]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:52.864 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:52 smithi113 ceph-mon[34341]: from='client.? 172.21.15.113:0/3606896439' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T13:38:53.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:52 smithi043 ceph-mon[29963]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:53.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:52 smithi043 ceph-mon[29963]: from='client.? 172.21.15.113:0/3606896439' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T13:38:53.892 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-30T13:38:53.892 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T13:38:53.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:53 smithi043 systemd[1]: Starting Ceph mon.c for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T13:38:54.138 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:38:54.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:53 smithi043 podman[42062]: 2024-03-30 13:38:53.794595236 +0000 UTC m=+0.017569893 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T13:38:54.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:53 smithi043 podman[42062]: 2024-03-30 13:38:53.902829269 +0000 UTC m=+0.125803927 container create d117d57e32aba55f3c0fd3b50867b8c5c53b703dbe19a6d1ab3ee41a91039bf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T13:38:54.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 podman[42062]: 2024-03-30 13:38:54.122056347 +0000 UTC m=+0.345031010 container init d117d57e32aba55f3c0fd3b50867b8c5c53b703dbe19a6d1ab3ee41a91039bf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326) 2024-03-30T13:38:54.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 podman[42062]: 2024-03-30 13:38:54.127849631 +0000 UTC m=+0.350824298 container start d117d57e32aba55f3c0fd3b50867b8c5c53b703dbe19a6d1ab3ee41a91039bf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T13:38:54.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: set uid:gid to 167:167 (ceph:ceph) 2024-03-30T13:38:54.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev), process ceph-mon, pid 7 2024-03-30T13:38:54.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: pidfile_write: ignore empty --pid-file 2024-03-30T13:38:54.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: load: jerasure load: lrc 2024-03-30T13:38:54.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: RocksDB version: 7.9.2 2024-03-30T13:38:54.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Git sha 0 2024-03-30T13:38:54.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Compile date 2024-03-29 15:06:16 2024-03-30T13:38:54.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: DB SUMMARY 2024-03-30T13:38:54.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: DB Session ID: IBQME89O8LBK8W86MJLI 2024-03-30T13:38:54.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: CURRENT file: CURRENT 2024-03-30T13:38:54.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: IDENTITY file: IDENTITY 2024-03-30T13:38:54.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-30T13:38:54.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-30T13:38:54.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-03-30T13:38:54.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.error_if_exists: 0 2024-03-30T13:38:54.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.create_if_missing: 0 2024-03-30T13:38:54.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.paranoid_checks: 1 2024-03-30T13:38:54.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-30T13:38:54.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-30T13:38:54.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-30T13:38:54.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.env: 0x55ec94500c60 2024-03-30T13:38:54.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.fs: PosixFileSystem 2024-03-30T13:38:54.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.info_log: 0x55ec95905ae0 2024-03-30T13:38:54.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_file_opening_threads: 16 2024-03-30T13:38:54.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.statistics: (nil) 2024-03-30T13:38:54.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.use_fsync: 0 2024-03-30T13:38:54.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_log_file_size: 0 2024-03-30T13:38:54.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-30T13:38:54.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-30T13:38:54.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.keep_log_file_num: 1000 2024-03-30T13:38:54.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.recycle_log_file_num: 0 2024-03-30T13:38:54.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.allow_fallocate: 1 2024-03-30T13:38:54.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.allow_mmap_reads: 0 2024-03-30T13:38:54.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.allow_mmap_writes: 0 2024-03-30T13:38:54.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.use_direct_reads: 0 2024-03-30T13:38:54.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-30T13:38:54.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.create_missing_column_families: 0 2024-03-30T13:38:54.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.db_log_dir: 2024-03-30T13:38:54.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.wal_dir: 2024-03-30T13:38:54.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-30T13:38:54.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-30T13:38:54.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-30T13:38:54.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-30T13:38:54.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-30T13:38:54.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-30T13:38:54.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.advise_random_on_open: 1 2024-03-30T13:38:54.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.db_write_buffer_size: 0 2024-03-30T13:38:54.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.write_buffer_manager: 0x55ec958d81e0 2024-03-30T13:38:54.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-30T13:38:54.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-30T13:38:54.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-30T13:38:54.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.rate_limiter: (nil) 2024-03-30T13:38:54.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-30T13:38:54.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.wal_recovery_mode: 2 2024-03-30T13:38:54.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.enable_thread_tracking: 0 2024-03-30T13:38:54.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.enable_pipelined_write: 0 2024-03-30T13:38:54.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.unordered_write: 0 2024-03-30T13:38:54.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-30T13:38:54.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-30T13:38:54.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-30T13:38:54.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-30T13:38:54.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.row_cache: None 2024-03-30T13:38:54.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.wal_filter: None 2024-03-30T13:38:54.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-30T13:38:54.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.allow_ingest_behind: 0 2024-03-30T13:38:54.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.two_write_queues: 0 2024-03-30T13:38:54.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.manual_wal_flush: 0 2024-03-30T13:38:54.655 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.wal_compression: 0 2024-03-30T13:38:54.655 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.atomic_flush: 0 2024-03-30T13:38:54.655 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-30T13:38:54.655 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-30T13:38:54.655 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-30T13:38:54.655 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.log_readahead_size: 0 2024-03-30T13:38:54.655 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-30T13:38:54.655 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.best_efforts_recovery: 0 2024-03-30T13:38:54.655 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-30T13:38:54.655 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-30T13:38:54.655 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.allow_data_in_errors: 0 2024-03-30T13:38:54.655 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.db_host_id: __hostname__ 2024-03-30T13:38:54.655 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-30T13:38:54.657 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_background_jobs: 2 2024-03-30T13:38:54.657 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_background_compactions: -1 2024-03-30T13:38:54.657 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_subcompactions: 1 2024-03-30T13:38:54.657 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-30T13:38:54.657 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-30T13:38:54.657 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-30T13:38:54.657 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_total_wal_size: 0 2024-03-30T13:38:54.657 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-30T13:38:54.657 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-30T13:38:54.657 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-30T13:38:54.657 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-30T13:38:54.657 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_open_files: -1 2024-03-30T13:38:54.658 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.bytes_per_sync: 0 2024-03-30T13:38:54.658 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-30T13:38:54.658 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-30T13:38:54.658 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compaction_readahead_size: 0 2024-03-30T13:38:54.658 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_background_flushes: -1 2024-03-30T13:38:54.658 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Compression algorithms supported: 2024-03-30T13:38:54.658 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-30T13:38:54.658 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: kZSTD supported: 0 2024-03-30T13:38:54.658 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: kXpressCompression supported: 0 2024-03-30T13:38:54.658 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: kLZ4HCCompression supported: 1 2024-03-30T13:38:54.658 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: kZlibCompression supported: 1 2024-03-30T13:38:54.658 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: kSnappyCompression supported: 1 2024-03-30T13:38:54.658 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: kLZ4Compression supported: 1 2024-03-30T13:38:54.659 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: kBZip2Compression supported: 0 2024-03-30T13:38:54.659 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-30T13:38:54.659 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-30T13:38:54.659 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-30T13:38:54.659 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-30T13:38:54.659 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-30T13:38:54.659 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.merge_operator: 2024-03-30T13:38:54.659 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compaction_filter: None 2024-03-30T13:38:54.659 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compaction_filter_factory: None 2024-03-30T13:38:54.659 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.sst_partitioner_factory: None 2024-03-30T13:38:54.659 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-30T13:38:54.659 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-30T13:38:54.659 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ec95905be0) 2024-03-30T13:38:54.660 INFO:journalctl@ceph.mon.c.smithi043.stdout: cache_index_and_filter_blocks: 1 2024-03-30T13:38:54.660 INFO:journalctl@ceph.mon.c.smithi043.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-30T13:38:54.660 INFO:journalctl@ceph.mon.c.smithi043.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-30T13:38:54.660 INFO:journalctl@ceph.mon.c.smithi043.stdout: pin_top_level_index_and_filter: 1 2024-03-30T13:38:54.660 INFO:journalctl@ceph.mon.c.smithi043.stdout: index_type: 0 2024-03-30T13:38:54.660 INFO:journalctl@ceph.mon.c.smithi043.stdout: data_block_index_type: 0 2024-03-30T13:38:54.660 INFO:journalctl@ceph.mon.c.smithi043.stdout: index_shortening: 1 2024-03-30T13:38:54.660 INFO:journalctl@ceph.mon.c.smithi043.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-30T13:38:54.660 INFO:journalctl@ceph.mon.c.smithi043.stdout: checksum: 4 2024-03-30T13:38:54.660 INFO:journalctl@ceph.mon.c.smithi043.stdout: no_block_cache: 0 2024-03-30T13:38:54.660 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_cache: 0x55ec95909090 2024-03-30T13:38:54.660 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_cache_name: BinnedLRUCache 2024-03-30T13:38:54.660 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_cache_options: 2024-03-30T13:38:54.660 INFO:journalctl@ceph.mon.c.smithi043.stdout: capacity : 536870912 2024-03-30T13:38:54.661 INFO:journalctl@ceph.mon.c.smithi043.stdout: num_shard_bits : 4 2024-03-30T13:38:54.661 INFO:journalctl@ceph.mon.c.smithi043.stdout: strict_capacity_limit : 0 2024-03-30T13:38:54.661 INFO:journalctl@ceph.mon.c.smithi043.stdout: high_pri_pool_ratio: 0.000 2024-03-30T13:38:54.661 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_cache_compressed: (nil) 2024-03-30T13:38:54.661 INFO:journalctl@ceph.mon.c.smithi043.stdout: persistent_cache: (nil) 2024-03-30T13:38:54.661 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_size: 4096 2024-03-30T13:38:54.661 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_size_deviation: 10 2024-03-30T13:38:54.661 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_restart_interval: 16 2024-03-30T13:38:54.661 INFO:journalctl@ceph.mon.c.smithi043.stdout: index_block_restart_interval: 1 2024-03-30T13:38:54.661 INFO:journalctl@ceph.mon.c.smithi043.stdout: metadata_block_size: 4096 2024-03-30T13:38:54.661 INFO:journalctl@ceph.mon.c.smithi043.stdout: partition_filters: 0 2024-03-30T13:38:54.661 INFO:journalctl@ceph.mon.c.smithi043.stdout: use_delta_encoding: 1 2024-03-30T13:38:54.661 INFO:journalctl@ceph.mon.c.smithi043.stdout: filter_policy: bloomfilter 2024-03-30T13:38:54.661 INFO:journalctl@ceph.mon.c.smithi043.stdout: whole_key_filtering: 1 2024-03-30T13:38:54.661 INFO:journalctl@ceph.mon.c.smithi043.stdout: verify_compression: 0 2024-03-30T13:38:54.662 INFO:journalctl@ceph.mon.c.smithi043.stdout: read_amp_bytes_per_bit: 0 2024-03-30T13:38:54.662 INFO:journalctl@ceph.mon.c.smithi043.stdout: format_version: 5 2024-03-30T13:38:54.662 INFO:journalctl@ceph.mon.c.smithi043.stdout: enable_index_compression: 1 2024-03-30T13:38:54.662 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_align: 0 2024-03-30T13:38:54.662 INFO:journalctl@ceph.mon.c.smithi043.stdout: max_auto_readahead_size: 262144 2024-03-30T13:38:54.662 INFO:journalctl@ceph.mon.c.smithi043.stdout: prepopulate_block_cache: 0 2024-03-30T13:38:54.662 INFO:journalctl@ceph.mon.c.smithi043.stdout: initial_auto_readahead_size: 8192 2024-03-30T13:38:54.662 INFO:journalctl@ceph.mon.c.smithi043.stdout: num_file_reads_for_auto_readahead: 2 2024-03-30T13:38:54.662 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.write_buffer_size: 33554432 2024-03-30T13:38:54.662 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_write_buffer_number: 2 2024-03-30T13:38:54.662 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compression: NoCompression 2024-03-30T13:38:54.662 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.bottommost_compression: Disabled 2024-03-30T13:38:54.662 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.prefix_extractor: nullptr 2024-03-30T13:38:54.663 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-30T13:38:54.663 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.num_levels: 7 2024-03-30T13:38:54.663 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-30T13:38:54.663 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-30T13:38:54.663 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-30T13:38:54.663 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-30T13:38:54.663 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-30T13:38:54.663 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-30T13:38:54.663 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-30T13:38:54.663 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-30T13:38:54.663 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-30T13:38:54.663 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-30T13:38:54.663 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-30T13:38:54.663 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-30T13:38:54.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-30T13:38:54.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compression_opts.level: 32767 2024-03-30T13:38:54.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compression_opts.strategy: 0 2024-03-30T13:38:54.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-30T13:38:54.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-30T13:38:54.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-30T13:38:54.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-30T13:38:54.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compression_opts.enabled: false 2024-03-30T13:38:54.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-30T13:38:54.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-30T13:38:54.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-30T13:38:54.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-30T13:38:54.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.target_file_size_base: 67108864 2024-03-30T13:38:54.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-30T13:38:54.665 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-30T13:38:54.665 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-30T13:38:54.665 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-30T13:38:54.665 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-30T13:38:54.665 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-30T13:38:54.665 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-30T13:38:54.665 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-30T13:38:54.665 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-30T13:38:54.665 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-30T13:38:54.665 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-30T13:38:54.665 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-30T13:38:54.665 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-30T13:38:54.665 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-30T13:38:54.665 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.arena_block_size: 1048576 2024-03-30T13:38:54.666 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-30T13:38:54.666 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-30T13:38:54.666 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.disable_auto_compactions: 0 2024-03-30T13:38:54.666 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-30T13:38:54.666 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-30T13:38:54.666 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-30T13:38:54.666 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-30T13:38:54.666 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-30T13:38:54.666 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-30T13:38:54.666 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-30T13:38:54.666 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-30T13:38:54.666 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-30T13:38:54.666 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-30T13:38:54.667 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.table_properties_collectors: 2024-03-30T13:38:54.667 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.inplace_update_support: 0 2024-03-30T13:38:54.667 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-30T13:38:54.667 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-30T13:38:54.667 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-30T13:38:54.667 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-30T13:38:54.667 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.bloom_locality: 0 2024-03-30T13:38:54.667 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.max_successive_merges: 0 2024-03-30T13:38:54.667 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-30T13:38:54.667 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.paranoid_file_checks: 0 2024-03-30T13:38:54.667 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.force_consistency_checks: 1 2024-03-30T13:38:54.667 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.report_bg_io_stats: 0 2024-03-30T13:38:54.667 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.ttl: 2592000 2024-03-30T13:38:54.667 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-30T13:38:54.667 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-30T13:38:54.668 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-30T13:38:54.668 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.enable_blob_files: false 2024-03-30T13:38:54.668 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.min_blob_size: 0 2024-03-30T13:38:54.668 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.blob_file_size: 268435456 2024-03-30T13:38:54.668 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-30T13:38:54.668 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-30T13:38:54.668 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-30T13:38:54.668 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-30T13:38:54.668 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-30T13:38:54.668 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.blob_file_starting_level: 0 2024-03-30T13:38:54.668 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-30T13:38:54.668 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-30T13:38:54.668 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-30T13:38:54.668 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f8b34974-c2b6-4fe6-978b-313e91ce6a32 2024-03-30T13:38:54.668 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711805934158268, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-30T13:38:54.669 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-30T13:38:54.669 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711805934158662, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711805934, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f8b34974-c2b6-4fe6-978b-313e91ce6a32", "db_session_id": "IBQME89O8LBK8W86MJLI", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-30T13:38:54.669 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711805934158745, "job": 1, "event": "recovery_finished"} 2024-03-30T13:38:54.669 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-30T13:38:54.669 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-30T13:38:54.669 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ec959fa000 2024-03-30T13:38:54.669 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: DB pointer 0x55ec959e4000 2024-03-30T13:38:54.669 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-30T13:38:54.669 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-30T13:38:54.669 INFO:journalctl@ceph.mon.c.smithi043.stdout: ** DB Stats ** 2024-03-30T13:38:54.669 INFO:journalctl@ceph.mon.c.smithi043.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T13:38:54.669 INFO:journalctl@ceph.mon.c.smithi043.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-30T13:38:54.669 INFO:journalctl@ceph.mon.c.smithi043.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T13:38:54.669 INFO:journalctl@ceph.mon.c.smithi043.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T13:38:54.670 INFO:journalctl@ceph.mon.c.smithi043.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-30T13:38:54.670 INFO:journalctl@ceph.mon.c.smithi043.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T13:38:54.670 INFO:journalctl@ceph.mon.c.smithi043.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T13:38:54.670 INFO:journalctl@ceph.mon.c.smithi043.stdout: 2024-03-30T13:38:54.670 INFO:journalctl@ceph.mon.c.smithi043.stdout: ** Compaction Stats [default] ** 2024-03-30T13:38:54.670 INFO:journalctl@ceph.mon.c.smithi043.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T13:38:54.670 INFO:journalctl@ceph.mon.c.smithi043.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-30T13:38:54.670 INFO:journalctl@ceph.mon.c.smithi043.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-30T13:38:54.670 INFO:journalctl@ceph.mon.c.smithi043.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-30T13:38:54.670 INFO:journalctl@ceph.mon.c.smithi043.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-30T13:38:54.670 INFO:journalctl@ceph.mon.c.smithi043.stdout: 2024-03-30T13:38:54.670 INFO:journalctl@ceph.mon.c.smithi043.stdout: ** Compaction Stats [default] ** 2024-03-30T13:38:54.670 INFO:journalctl@ceph.mon.c.smithi043.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T13:38:54.670 INFO:journalctl@ceph.mon.c.smithi043.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T13:38:54.671 INFO:journalctl@ceph.mon.c.smithi043.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-30T13:38:54.671 INFO:journalctl@ceph.mon.c.smithi043.stdout: 2024-03-30T13:38:54.671 INFO:journalctl@ceph.mon.c.smithi043.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-30T13:38:54.671 INFO:journalctl@ceph.mon.c.smithi043.stdout: 2024-03-30T13:38:54.671 INFO:journalctl@ceph.mon.c.smithi043.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T13:38:54.671 INFO:journalctl@ceph.mon.c.smithi043.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-30T13:38:54.671 INFO:journalctl@ceph.mon.c.smithi043.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-30T13:38:54.671 INFO:journalctl@ceph.mon.c.smithi043.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-30T13:38:54.671 INFO:journalctl@ceph.mon.c.smithi043.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-30T13:38:54.671 INFO:journalctl@ceph.mon.c.smithi043.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-30T13:38:54.671 INFO:journalctl@ceph.mon.c.smithi043.stdout: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T13:38:54.671 INFO:journalctl@ceph.mon.c.smithi043.stdout: Interval compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T13:38:54.671 INFO:journalctl@ceph.mon.c.smithi043.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-30T13:38:54.671 INFO:journalctl@ceph.mon.c.smithi043.stdout: Block cache BinnedLRUCache@0x55ec95909090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-30T13:38:54.671 INFO:journalctl@ceph.mon.c.smithi043.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-30T13:38:54.672 INFO:journalctl@ceph.mon.c.smithi043.stdout: 2024-03-30T13:38:54.672 INFO:journalctl@ceph.mon.c.smithi043.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-30T13:38:54.672 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-30T13:38:54.672 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: using public_addrv [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-03-30T13:38:54.672 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: starting mon.c rank -1 at public addrs [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] at bind addrs [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:38:54.672 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: mon.c@-1(???) e0 preinit fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:38:54.672 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: mon.c@-1(synchronizing).mds e1 new map 2024-03-30T13:38:54.672 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-30T13:38:54.672 INFO:journalctl@ceph.mon.c.smithi043.stdout: e1 2024-03-30T13:38:54.672 INFO:journalctl@ceph.mon.c.smithi043.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T13:38:54.672 INFO:journalctl@ceph.mon.c.smithi043.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-30T13:38:54.672 INFO:journalctl@ceph.mon.c.smithi043.stdout: legacy client fscid: -1 2024-03-30T13:38:54.673 INFO:journalctl@ceph.mon.c.smithi043.stdout: 2024-03-30T13:38:54.673 INFO:journalctl@ceph.mon.c.smithi043.stdout: No filesystems configured 2024-03-30T13:38:54.673 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-30T13:38:54.673 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-30T13:38:54.673 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-30T13:38:54.673 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-30T13:38:54.673 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-30T13:38:54.673 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-30T13:38:54.673 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-30T13:38:54.673 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T13:38:54.673 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T13:38:54.673 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T13:38:54.673 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3289933880' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-30T13:38:54.673 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T13:38:54.673 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:54.674 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:54.674 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:54.674 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:54.674 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi113", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:38:54.674 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:38:54.674 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:38:54.674 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: Updating smithi113:/etc/ceph/ceph.conf 2024-03-30T13:38:54.674 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: Updating smithi113:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:38:54.674 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:38:54.674 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: Updating smithi113:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T13:38:54.674 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:54.674 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:54.674 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:54.675 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:54.675 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi043:172.21.15.43=a;smithi043:[v2:172.21.15.43:3301,v1:172.21.15.43:6790]=c;smithi113:172.21.15.113=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:38:54.675 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: Saving service mon spec with placement smithi043:172.21.15.43=a;smithi043:[v2:172.21.15.43:3301,v1:172.21.15.43:6790]=c;smithi113:172.21.15.113=b;count:3 2024-03-30T13:38:54.675 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:54.675 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:38:54.675 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:38:54.675 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:38:54.675 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:54.675 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:38:54.675 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:38:54.675 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: Deploying daemon mon.b on smithi113 2024-03-30T13:38:54.675 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:54.675 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:54.675 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:54.675 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:38:54.676 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: mon.a calling monitor election 2024-03-30T13:38:54.676 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:54.676 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:54.676 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:54.676 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:54.676 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: mon.b calling monitor election 2024-03-30T13:38:54.676 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:54.676 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:54.676 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:54.676 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:54.676 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-30T13:38:54.676 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: monmap epoch 2 2024-03-30T13:38:54.676 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:38:54.676 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: last_changed 2024-03-30T13:38:45.680430+0000 2024-03-30T13:38:54.676 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: created 2024-03-30T13:37:04.571563+0000 2024-03-30T13:38:54.677 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: min_mon_release 19 (squid) 2024-03-30T13:38:54.677 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: election_strategy: 1 2024-03-30T13:38:54.677 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-03-30T13:38:54.677 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: 1: [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon.b 2024-03-30T13:38:54.677 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: fsmap 2024-03-30T13:38:54.677 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T13:38:54.677 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: mgrmap e13: y(active, since 41s) 2024-03-30T13:38:54.677 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: overall HEALTH_OK 2024-03-30T13:38:54.677 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:54.677 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:54.677 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:54.677 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:38:54.677 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:38:54.677 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: Deploying daemon mon.c on smithi043 2024-03-30T13:38:54.678 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:54.678 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:54.678 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: from='client.? 172.21.15.113:0/3606896439' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T13:38:54.678 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 ceph-mon[42099]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-30T13:38:54.678 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 bash[42062]: d117d57e32aba55f3c0fd3b50867b8c5c53b703dbe19a6d1ab3ee41a91039bf6 2024-03-30T13:38:54.678 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:54 smithi043 systemd[1]: Started Ceph mon.c for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T13:38:59.573 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-30T13:38:59.574 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":3,"fsid":"7ee932fa-ee9a-11ee-b647-cb9ed24678a4","modified":"2024-03-30T13:38:54.181369Z","created":"2024-03-30T13:37:04.571563Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-30T13:38:59.574 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:59.574 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:38:59.574 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: mon.a calling monitor election 2024-03-30T13:38:59.574 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:59.574 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:38:59.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: mon.b calling monitor election 2024-03-30T13:38:59.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:38:59.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:59.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:38:59.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: mon.c calling monitor election 2024-03-30T13:38:59.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:38:59.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:59.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:38:59.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:38:59.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-30T13:38:59.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: monmap epoch 3 2024-03-30T13:38:59.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:38:59.576 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: last_changed 2024-03-30T13:38:54.181369+0000 2024-03-30T13:38:59.576 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: created 2024-03-30T13:37:04.571563+0000 2024-03-30T13:38:59.576 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: min_mon_release 19 (squid) 2024-03-30T13:38:59.576 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: election_strategy: 1 2024-03-30T13:38:59.576 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-03-30T13:38:59.576 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: 1: [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon.b 2024-03-30T13:38:59.576 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: 2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.c 2024-03-30T13:38:59.576 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: fsmap 2024-03-30T13:38:59.576 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T13:38:59.576 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: mgrmap e13: y(active, since 49s) 2024-03-30T13:38:59.576 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: overall HEALTH_OK 2024-03-30T13:38:59.576 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:59.577 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:59.577 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:59.577 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:38:59 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:59.577 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 3 2024-03-30T13:38:59.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:59.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:38:59.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: mon.a calling monitor election 2024-03-30T13:38:59.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:59.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:38:59.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: mon.b calling monitor election 2024-03-30T13:38:59.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:38:59.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:59.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:38:59.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: mon.c calling monitor election 2024-03-30T13:38:59.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:38:59.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:59.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:38:59.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:38:59.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-30T13:38:59.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: monmap epoch 3 2024-03-30T13:38:59.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:38:59.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: last_changed 2024-03-30T13:38:54.181369+0000 2024-03-30T13:38:59.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: created 2024-03-30T13:37:04.571563+0000 2024-03-30T13:38:59.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: min_mon_release 19 (squid) 2024-03-30T13:38:59.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: election_strategy: 1 2024-03-30T13:38:59.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-03-30T13:38:59.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: 1: [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon.b 2024-03-30T13:38:59.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: 2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.c 2024-03-30T13:38:59.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: fsmap 2024-03-30T13:38:59.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T13:38:59.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: mgrmap e13: y(active, since 49s) 2024-03-30T13:38:59.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: overall HEALTH_OK 2024-03-30T13:38:59.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:59.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:59.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:59.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:59.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:59.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:38:59.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: mon.a calling monitor election 2024-03-30T13:38:59.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:38:59.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:38:59.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: mon.b calling monitor election 2024-03-30T13:38:59.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:38:59.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:59.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:38:59.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: mon.c calling monitor election 2024-03-30T13:38:59.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:38:59.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:38:59.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:38:59.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:38:59.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-30T13:38:59.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: monmap epoch 3 2024-03-30T13:38:59.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:38:59.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: last_changed 2024-03-30T13:38:54.181369+0000 2024-03-30T13:38:59.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: created 2024-03-30T13:37:04.571563+0000 2024-03-30T13:38:59.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: min_mon_release 19 (squid) 2024-03-30T13:38:59.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: election_strategy: 1 2024-03-30T13:38:59.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-03-30T13:38:59.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: 1: [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon.b 2024-03-30T13:38:59.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: 2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.c 2024-03-30T13:38:59.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: fsmap 2024-03-30T13:38:59.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T13:38:59.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: mgrmap e13: y(active, since 49s) 2024-03-30T13:38:59.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: overall HEALTH_OK 2024-03-30T13:38:59.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:59.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:59.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:38:59.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:38:59 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:00.312 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-30T13:39:00.313 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-03-30T13:39:00.425 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:00 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:00.426 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:00 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:00.426 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:00 smithi043 ceph-mon[29963]: from='client.? 172.21.15.113:0/2338764670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T13:39:00.426 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:00 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:39:00.426 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:00 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:00.426 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:00 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:00.426 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:00 smithi043 ceph-mon[42099]: from='client.? 172.21.15.113:0/2338764670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T13:39:00.426 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:00 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:39:00.569 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:00 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:00.569 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:00 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:00.569 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:00 smithi113 ceph-mon[34341]: from='client.? 172.21.15.113:0/2338764670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T13:39:00.569 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:00 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:39:00.696 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:39:01.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:01 smithi043 ceph-mon[42099]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:01.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:01 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:01.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:01 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:01.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:01 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:01.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:01 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:01.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:01 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:01.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:01 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:01.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:01 smithi043 ceph-mon[29963]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:01.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:01 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:01.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:01 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:01.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:01 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:01.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:01 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:01.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:01 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:01.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:01 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:01.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:01 smithi113 ceph-mon[34341]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:01.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:01 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:01.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:01 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:01.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:01 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:01.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:01 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:01.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:01 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:01.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:01 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:02.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[29963]: Updating smithi043:/etc/ceph/ceph.conf 2024-03-30T13:39:02.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[29963]: Updating smithi113:/etc/ceph/ceph.conf 2024-03-30T13:39:02.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[29963]: Updating smithi113:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:39:02.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[29963]: Updating smithi043:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:39:02.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:39:02.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T13:39:02.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:02.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[42099]: Updating smithi043:/etc/ceph/ceph.conf 2024-03-30T13:39:02.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[42099]: Updating smithi113:/etc/ceph/ceph.conf 2024-03-30T13:39:02.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[42099]: Updating smithi113:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:39:02.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[42099]: Updating smithi043:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:39:02.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.402 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.402 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:39:02.402 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T13:39:02.402 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:02 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:02.590 INFO:teuthology.orchestra.run.smithi043.stdout:# minimal ceph.conf for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:39:02.591 INFO:teuthology.orchestra.run.smithi043.stdout:[global] 2024-03-30T13:39:02.591 INFO:teuthology.orchestra.run.smithi043.stdout: fsid = 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:39:02.591 INFO:teuthology.orchestra.run.smithi043.stdout: mon_host = [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-03-30T13:39:02.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:02 smithi113 ceph-mon[34341]: Updating smithi043:/etc/ceph/ceph.conf 2024-03-30T13:39:02.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:02 smithi113 ceph-mon[34341]: Updating smithi113:/etc/ceph/ceph.conf 2024-03-30T13:39:02.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:02 smithi113 ceph-mon[34341]: Updating smithi113:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:39:02.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:02 smithi113 ceph-mon[34341]: Updating smithi043:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:39:02.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:02 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:02 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:02 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:02 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:02 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:02 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:02 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:02 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:02 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:02.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:02 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:39:02.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:02 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T13:39:02.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:02 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:03.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:03 smithi043 ceph-mon[29963]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:03.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:03 smithi043 ceph-mon[29963]: Reconfiguring mon.a (unknown last config time)... 2024-03-30T13:39:03.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:03 smithi043 ceph-mon[29963]: Reconfiguring daemon mon.a on smithi043 2024-03-30T13:39:03.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:03 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3183650904' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:03.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:03 smithi043 ceph-mon[42099]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:03.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:03 smithi043 ceph-mon[42099]: Reconfiguring mon.a (unknown last config time)... 2024-03-30T13:39:03.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:03 smithi043 ceph-mon[42099]: Reconfiguring daemon mon.a on smithi043 2024-03-30T13:39:03.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:03 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3183650904' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:03.682 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-30T13:39:03.682 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-30T13:39:03.683 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.conf 2024-03-30T13:39:03.714 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-30T13:39:03.714 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:39:03.721 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:03 smithi113 ceph-mon[34341]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:03.721 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:03 smithi113 ceph-mon[34341]: Reconfiguring mon.a (unknown last config time)... 2024-03-30T13:39:03.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:03 smithi113 ceph-mon[34341]: Reconfiguring daemon mon.a on smithi043 2024-03-30T13:39:03.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:03 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3183650904' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:03.784 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-30T13:39:03.785 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/ceph/ceph.conf 2024-03-30T13:39:03.820 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-30T13:39:03.820 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:39:03.891 INFO:tasks.cephadm:Adding mgr.y on smithi043 2024-03-30T13:39:03.891 INFO:tasks.cephadm:Adding mgr.x on smithi113 2024-03-30T13:39:03.891 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph orch apply mgr '2;smithi043=y;smithi113=x' 2024-03-30T13:39:04.163 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:39:05.345 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:05 smithi113 ceph-mon[34341]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:05.345 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:05 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:05.345 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:05 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:05.345 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:05 smithi113 ceph-mon[34341]: Reconfiguring mon.c (monmap changed)... 2024-03-30T13:39:05.345 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:05 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:39:05.345 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:05 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T13:39:05.346 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:05 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:05.346 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:05 smithi113 ceph-mon[34341]: Reconfiguring daemon mon.c on smithi043 2024-03-30T13:39:05.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:05 smithi043 ceph-mon[29963]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:05.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:05 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:05.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:05 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:05.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:05 smithi043 ceph-mon[29963]: Reconfiguring mon.c (monmap changed)... 2024-03-30T13:39:05.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:05 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:39:05.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:05 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T13:39:05.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:05 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:05.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:05 smithi043 ceph-mon[29963]: Reconfiguring daemon mon.c on smithi043 2024-03-30T13:39:05.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:05 smithi043 ceph-mon[42099]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:05.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:05 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:05.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:05 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:05.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:05 smithi043 ceph-mon[42099]: Reconfiguring mon.c (monmap changed)... 2024-03-30T13:39:05.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:05 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:39:05.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:05 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T13:39:05.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:05 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:05.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:05 smithi043 ceph-mon[42099]: Reconfiguring daemon mon.c on smithi043 2024-03-30T13:39:05.895 INFO:teuthology.orchestra.run.smithi113.stdout:Scheduled mgr update... 2024-03-30T13:39:06.510 DEBUG:teuthology.orchestra.run.smithi113:mgr.x> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mgr.x.service 2024-03-30T13:39:06.513 INFO:tasks.cephadm:Deploying OSDs... 2024-03-30T13:39:06.513 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-30T13:39:06.513 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T13:39:06.531 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T13:39:06.532 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_1 2024-03-30T13:39:06.589 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T13:39:06.589 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T13:39:06.589 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-30T13:39:06.589 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T13:39:06.589 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T13:39:06.589 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-03-30 13:38:53.619554060 +0000 2024-03-30T13:39:06.590 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-03-30 13:30:22.757453273 +0000 2024-03-30T13:39:06.590 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-03-30 13:30:22.757453273 +0000 2024-03-30T13:39:06.590 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-03-30 13:30:22.757453273 +0000 2024-03-30T13:39:06.590 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T13:39:06.659 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-03-30T13:39:06.659 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-03-30T13:39:06.659 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000123941 s, 4.1 MB/s 2024-03-30T13:39:06.660 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T13:39:06.718 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_2 2024-03-30T13:39:06.774 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T13:39:06.775 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T13:39:06.775 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-30T13:39:06.775 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T13:39:06.775 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T13:39:06.775 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-03-30 13:38:53.619554060 +0000 2024-03-30T13:39:06.775 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-03-30 13:30:23.088448881 +0000 2024-03-30T13:39:06.775 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-03-30 13:30:23.088448881 +0000 2024-03-30T13:39:06.775 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-03-30 13:30:23.088448881 +0000 2024-03-30T13:39:06.775 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T13:39:06.844 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-03-30T13:39:06.844 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-03-30T13:39:06.844 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000118272 s, 4.3 MB/s 2024-03-30T13:39:06.845 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T13:39:06.902 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_3 2024-03-30T13:39:06.958 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T13:39:06.958 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T13:39:06.958 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-30T13:39:06.958 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T13:39:06.958 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T13:39:06.958 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-03-30 13:38:53.620554046 +0000 2024-03-30T13:39:06.958 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-03-30 13:30:23.404444688 +0000 2024-03-30T13:39:06.958 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-03-30 13:30:23.404444688 +0000 2024-03-30T13:39:06.958 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-03-30 13:30:23.404444688 +0000 2024-03-30T13:39:06.959 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T13:39:07.023 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[29963]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:07.023 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:07.023 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:07.023 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[29963]: Reconfiguring mon.b (monmap changed)... 2024-03-30T13:39:07.023 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:39:07.023 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T13:39:07.023 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:07.023 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[29963]: Reconfiguring daemon mon.b on smithi113 2024-03-30T13:39:07.023 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[29963]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi043=y;smithi113=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:39:07.023 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[29963]: Saving service mgr spec with placement smithi043=y;smithi113=x;count:2 2024-03-30T13:39:07.023 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:07.024 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[42099]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:07.024 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:07.024 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:07.024 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[42099]: Reconfiguring mon.b (monmap changed)... 2024-03-30T13:39:07.024 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:39:07.024 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T13:39:07.024 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:07.024 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[42099]: Reconfiguring daemon mon.b on smithi113 2024-03-30T13:39:07.024 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[42099]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi043=y;smithi113=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:39:07.024 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[42099]: Saving service mgr spec with placement smithi043=y;smithi113=x;count:2 2024-03-30T13:39:07.024 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:06 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:07.025 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-03-30T13:39:07.025 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-03-30T13:39:07.025 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000111744 s, 4.6 MB/s 2024-03-30T13:39:07.026 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T13:39:07.082 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_4 2024-03-30T13:39:07.137 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T13:39:07.138 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T13:39:07.138 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-30T13:39:07.138 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T13:39:07.138 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T13:39:07.138 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-03-30 13:38:53.620554046 +0000 2024-03-30T13:39:07.138 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-03-30 13:30:23.737440269 +0000 2024-03-30T13:39:07.138 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-03-30 13:30:23.737440269 +0000 2024-03-30T13:39:07.138 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-03-30 13:30:23.737440269 +0000 2024-03-30T13:39:07.138 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T13:39:07.204 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-03-30T13:39:07.205 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-03-30T13:39:07.205 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000111296 s, 4.6 MB/s 2024-03-30T13:39:07.206 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T13:39:07.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:06 smithi113 ceph-mon[34341]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:07.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:06 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:07.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:06 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:07.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:06 smithi113 ceph-mon[34341]: Reconfiguring mon.b (monmap changed)... 2024-03-30T13:39:07.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:06 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:39:07.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:06 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T13:39:07.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:06 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:07.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:06 smithi113 ceph-mon[34341]: Reconfiguring daemon mon.b on smithi113 2024-03-30T13:39:07.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:06 smithi113 ceph-mon[34341]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi043=y;smithi113=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:39:07.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:06 smithi113 ceph-mon[34341]: Saving service mgr spec with placement smithi043=y;smithi113=x;count:2 2024-03-30T13:39:07.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:06 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:07.262 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-30T13:39:07.263 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T13:39:07.279 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T13:39:07.279 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_1 2024-03-30T13:39:07.334 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T13:39:07.334 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T13:39:07.335 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-30T13:39:07.335 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T13:39:07.335 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T13:39:07.335 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-03-30 13:38:44.801099496 +0000 2024-03-30T13:39:07.335 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-03-30 13:30:22.774689070 +0000 2024-03-30T13:39:07.335 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-03-30 13:30:22.774689070 +0000 2024-03-30T13:39:07.335 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2024-03-30 13:30:22.774689070 +0000 2024-03-30T13:39:07.335 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T13:39:07.402 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-03-30T13:39:07.402 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-03-30T13:39:07.402 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000216926 s, 2.4 MB/s 2024-03-30T13:39:07.403 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T13:39:07.459 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_2 2024-03-30T13:39:07.514 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T13:39:07.514 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T13:39:07.514 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-30T13:39:07.515 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T13:39:07.515 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T13:39:07.515 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-03-30 13:38:44.802099475 +0000 2024-03-30T13:39:07.515 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-03-30 13:30:23.133681645 +0000 2024-03-30T13:39:07.515 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-03-30 13:30:23.133681645 +0000 2024-03-30T13:39:07.515 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2024-03-30 13:30:23.133681645 +0000 2024-03-30T13:39:07.515 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T13:39:07.580 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-03-30T13:39:07.580 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-03-30T13:39:07.580 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000211863 s, 2.4 MB/s 2024-03-30T13:39:07.581 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T13:39:07.637 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_3 2024-03-30T13:39:07.694 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T13:39:07.694 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T13:39:07.694 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-30T13:39:07.694 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T13:39:07.694 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T13:39:07.694 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-03-30 13:38:44.802099475 +0000 2024-03-30T13:39:07.694 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-03-30 13:30:23.485674365 +0000 2024-03-30T13:39:07.694 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-03-30 13:30:23.485674365 +0000 2024-03-30T13:39:07.694 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2024-03-30 13:30:23.485674365 +0000 2024-03-30T13:39:07.695 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T13:39:07.762 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-03-30T13:39:07.763 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-03-30T13:39:07.763 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.00024197 s, 2.1 MB/s 2024-03-30T13:39:07.763 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T13:39:07.817 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_4 2024-03-30T13:39:07.872 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T13:39:07.872 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T13:39:07.872 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-30T13:39:07.872 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T13:39:07.872 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T13:39:07.873 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-03-30 13:38:44.803099454 +0000 2024-03-30T13:39:07.873 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-03-30 13:30:23.824667354 +0000 2024-03-30T13:39:07.873 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-03-30 13:30:23.824667354 +0000 2024-03-30T13:39:07.873 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2024-03-30 13:30:23.824667354 +0000 2024-03-30T13:39:07.873 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T13:39:07.938 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-03-30T13:39:07.938 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-03-30T13:39:07.938 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000203009 s, 2.5 MB/s 2024-03-30T13:39:07.939 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T13:39:07.995 INFO:tasks.cephadm:Deploying osd.0 on smithi043 with /dev/vg_nvme/lv_4... 2024-03-30T13:39:07.995 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-30T13:39:08.243 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:39:08.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[29963]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:08.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:08.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:08.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:08.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:08.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:08.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:08.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T13:39:08.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T13:39:08.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T13:39:08.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:08.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[29963]: Deploying daemon mgr.x on smithi113 2024-03-30T13:39:08.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[42099]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:08.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:08.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:08.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:08.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:08.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:08.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:08.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T13:39:08.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T13:39:08.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T13:39:08.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:08.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:08 smithi043 ceph-mon[42099]: Deploying daemon mgr.x on smithi113 2024-03-30T13:39:08.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:08 smithi113 ceph-mon[34341]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:08.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:08 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:08.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:08 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:08.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:08 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:08.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:08 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:08.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:08 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:08.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:08 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:08.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:08 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T13:39:08.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:08 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T13:39:08.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:08 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T13:39:08.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:08 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:08.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:08 smithi113 ceph-mon[34341]: Deploying daemon mgr.x on smithi113 2024-03-30T13:39:09.722 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:09 smithi113 systemd[1]: Starting Ceph mgr.x for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T13:39:10.085 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:09 smithi113 podman[36433]: 2024-03-30 13:39:09.714089786 +0000 UTC m=+0.017487959 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T13:39:10.086 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:09 smithi113 podman[36433]: 2024-03-30 13:39:09.81518653 +0000 UTC m=+0.118584728 container create e8327be872dc9e591a9d6af20a44a2eec47aee1d5470a056b0fe78c3ce0332a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-30T13:39:10.086 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:10 smithi113 podman[36433]: 2024-03-30 13:39:10.017847277 +0000 UTC m=+0.321245445 container init e8327be872dc9e591a9d6af20a44a2eec47aee1d5470a056b0fe78c3ce0332a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-30T13:39:10.086 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:10 smithi113 podman[36433]: 2024-03-30 13:39:10.023436907 +0000 UTC m=+0.326835064 container start e8327be872dc9e591a9d6af20a44a2eec47aee1d5470a056b0fe78c3ce0332a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326) 2024-03-30T13:39:10.086 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:10 smithi113 ceph-mgr[36472]: -- 172.21.15.113:0/2923158824 <== mon.1 v2:172.21.15.113:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561b3074a5a0 con 0x561b3071f000 2024-03-30T13:39:10.086 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:10 smithi113 bash[36433]: e8327be872dc9e591a9d6af20a44a2eec47aee1d5470a056b0fe78c3ce0332a3 2024-03-30T13:39:10.233 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-30T13:39:10.234 INFO:teuthology.orchestra.run.smithi043.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-30T13:39:10.234 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10+0 records in 2024-03-30T13:39:10.234 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2024-03-30T13:39:10.234 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00850154 s, 1.2 GB/s 2024-03-30T13:39:10.234 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping successful for: 2024-03-30T13:39:10.471 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:10 smithi113 systemd[1]: Started Ceph mgr.x for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T13:39:10.972 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:10 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:10.575+0000 7fd0443a0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T13:39:10.972 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:10 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:10.670+0000 7fd0443a0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T13:39:10.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi043:vg_nvme/lv_4 2024-03-30T13:39:11.243 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:39:11.352 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:11 smithi113 ceph-mon[34341]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:11.352 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:11 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:11.353 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:11 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:11.353 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:11 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:11.353 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:11 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:11.353 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:11 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:11.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:11 smithi043 ceph-mon[42099]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:11.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:11 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:11.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:11 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:11.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:11 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:11.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:11 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:11.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:11 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:11.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:11 smithi043 ceph-mon[29963]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:11.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:11 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:11.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:11 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:11.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:11 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:11.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:11 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:11.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:11 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:11.689 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:11 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:11.350+0000 7fd0443a0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T13:39:11.689 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:11 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:11.443+0000 7fd0443a0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T13:39:11.689 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:11 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:11.590+0000 7fd0443a0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T13:39:11.689 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:11 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:11.687+0000 7fd0443a0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T13:39:11.972 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:11 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:11.772+0000 7fd0443a0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T13:39:12.329 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:12 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:12.038+0000 7fd0443a0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T13:39:12.329 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:12 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:12.147+0000 7fd0443a0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T13:39:12.330 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:12 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:12.232+0000 7fd0443a0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T13:39:12.330 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:12 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:12.327+0000 7fd0443a0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T13:39:12.603 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:12 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:12.500+0000 7fd0443a0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T13:39:12.876 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[29963]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:12.876 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:12.876 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:12.876 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:12.876 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:12.876 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:12.876 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[29963]: Reconfiguring mgr.y (unknown last config time)... 2024-03-30T13:39:12.876 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T13:39:12.876 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T13:39:12.877 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:12.877 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[29963]: Reconfiguring daemon mgr.y on smithi043 2024-03-30T13:39:12.877 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[42099]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:12.877 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:12.877 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:12.877 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:12.877 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:12.877 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:12.877 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[42099]: Reconfiguring mgr.y (unknown last config time)... 2024-03-30T13:39:12.877 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T13:39:12.877 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T13:39:12.877 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:12.877 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:12 smithi043 ceph-mon[42099]: Reconfiguring daemon mgr.y on smithi043 2024-03-30T13:39:12.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:12 smithi113 ceph-mon[34341]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:12.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:12 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:12.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:12 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:12.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:12 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:12.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:12 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:12.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:12 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:12.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:12 smithi113 ceph-mon[34341]: Reconfiguring mgr.y (unknown last config time)... 2024-03-30T13:39:12.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:12 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T13:39:12.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:12 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T13:39:12.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:12 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:12.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:12 smithi113 ceph-mon[34341]: Reconfiguring daemon mgr.y on smithi043 2024-03-30T13:39:13.722 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:13 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:13.457+0000 7fd0443a0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T13:39:13.722 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:13 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:13.586+0000 7fd0443a0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T13:39:13.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:13 smithi113 ceph-mon[34341]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:39:13.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:13 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:39:13.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:13 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:39:13.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:13 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:13.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:13 smithi043 ceph-mon[29963]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:39:13.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:13 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:39:13.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:13 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:39:13.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:13 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:13.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:13 smithi043 ceph-mon[42099]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:39:13.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:13 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:39:13.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:13 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:39:13.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:13 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:14.199 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:13 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:13.915+0000 7fd0443a0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T13:39:14.199 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:14 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:14.004+0000 7fd0443a0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T13:39:14.199 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:14 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:14.197+0000 7fd0443a0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:39:14.471 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:14 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:14.380+0000 7fd0443a0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T13:39:14.920 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:14 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:14.659+0000 7fd0443a0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:39:14.920 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:14 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:14.918+0000 7fd0443a0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T13:39:15.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:14 smithi043 ceph-mon[29963]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:15.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:14 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:15.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:14 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:15.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:14 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:15.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:14 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:15.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:14 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:15.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:14 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:15.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:14 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:15.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:14 smithi043 ceph-mon[42099]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:15.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:14 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:15.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:14 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:15.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:14 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:15.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:14 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:15.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:14 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:15.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:14 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:15.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:14 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:15.222 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:15 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:15.153+0000 7fd0443a0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T13:39:15.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:14 smithi113 ceph-mon[34341]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:15.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:14 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:15.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:14 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:15.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:14 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:15.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:14 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:15.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:14 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:15.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:14 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:15.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:14 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:16.221 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:15 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:15.942+0000 7fd0443a0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T13:39:16.818 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:16 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:16.504+0000 7fd0443a0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T13:39:16.818 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:16 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:16.600+0000 7fd0443a0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T13:39:16.818 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:39:16 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:39:16.816+0000 7fd0443a0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T13:39:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:16 smithi113 ceph-mon[34341]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:16 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3871249032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94ea083e-6a27-41db-83af-2f7d3cae1956"}]: dispatch 2024-03-30T13:39:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:16 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3871249032' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94ea083e-6a27-41db-83af-2f7d3cae1956"}]': finished 2024-03-30T13:39:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:16 smithi113 ceph-mon[34341]: osdmap e5: 1 total, 0 up, 1 in 2024-03-30T13:39:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:16 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:39:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:16 smithi113 ceph-mon[34341]: Standby manager daemon x started 2024-03-30T13:39:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:16 smithi113 ceph-mon[34341]: from='mgr.? 172.21.15.113:0/1248082428' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-30T13:39:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:16 smithi113 ceph-mon[34341]: from='mgr.? 172.21.15.113:0/1248082428' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T13:39:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:16 smithi113 ceph-mon[34341]: from='mgr.? 172.21.15.113:0/1248082428' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-30T13:39:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:16 smithi113 ceph-mon[34341]: from='mgr.? 172.21.15.113:0/1248082428' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T13:39:17.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:16 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1969268684' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:39:17.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[29963]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:17.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3871249032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94ea083e-6a27-41db-83af-2f7d3cae1956"}]: dispatch 2024-03-30T13:39:17.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3871249032' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94ea083e-6a27-41db-83af-2f7d3cae1956"}]': finished 2024-03-30T13:39:17.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[29963]: osdmap e5: 1 total, 0 up, 1 in 2024-03-30T13:39:17.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:39:17.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[29963]: Standby manager daemon x started 2024-03-30T13:39:17.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[29963]: from='mgr.? 172.21.15.113:0/1248082428' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-30T13:39:17.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[29963]: from='mgr.? 172.21.15.113:0/1248082428' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T13:39:17.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[29963]: from='mgr.? 172.21.15.113:0/1248082428' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-30T13:39:17.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[29963]: from='mgr.? 172.21.15.113:0/1248082428' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T13:39:17.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1969268684' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:39:17.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[42099]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:17.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3871249032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94ea083e-6a27-41db-83af-2f7d3cae1956"}]: dispatch 2024-03-30T13:39:17.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3871249032' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94ea083e-6a27-41db-83af-2f7d3cae1956"}]': finished 2024-03-30T13:39:17.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[42099]: osdmap e5: 1 total, 0 up, 1 in 2024-03-30T13:39:17.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:39:17.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[42099]: Standby manager daemon x started 2024-03-30T13:39:17.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[42099]: from='mgr.? 172.21.15.113:0/1248082428' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-30T13:39:17.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[42099]: from='mgr.? 172.21.15.113:0/1248082428' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T13:39:17.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[42099]: from='mgr.? 172.21.15.113:0/1248082428' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-30T13:39:17.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[42099]: from='mgr.? 172.21.15.113:0/1248082428' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T13:39:17.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:16 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1969268684' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:39:18.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:18 smithi043 ceph-mon[29963]: mgrmap e14: y(active, since 67s), standbys: x 2024-03-30T13:39:18.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:18 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T13:39:18.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:18 smithi043 ceph-mon[42099]: mgrmap e14: y(active, since 67s), standbys: x 2024-03-30T13:39:18.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:18 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T13:39:18.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:18 smithi113 ceph-mon[34341]: mgrmap e14: y(active, since 67s), standbys: x 2024-03-30T13:39:18.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:18 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T13:39:19.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:19 smithi043 ceph-mon[29963]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:19.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:19 smithi043 ceph-mon[42099]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:19.623 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:19 smithi113 ceph-mon[34341]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:21.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:21 smithi043 ceph-mon[29963]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:21.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:21 smithi043 ceph-mon[42099]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:21.721 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:21 smithi113 ceph-mon[34341]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:23.621 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:23 smithi043 ceph-mon[29963]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:23.621 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:23 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T13:39:23.621 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:23 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:23.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:23 smithi043 ceph-mon[42099]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:23.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:23 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T13:39:23.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:23 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:23.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:23 smithi113 ceph-mon[34341]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:23.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:23 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T13:39:23.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:23 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:24.366 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:24 smithi043 ceph-mon[29963]: Deploying daemon osd.0 on smithi043 2024-03-30T13:39:24.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:24 smithi043 ceph-mon[42099]: Deploying daemon osd.0 on smithi043 2024-03-30T13:39:24.721 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:24 smithi113 ceph-mon[34341]: Deploying daemon osd.0 on smithi043 2024-03-30T13:39:25.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:25 smithi043 ceph-mon[29963]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:25.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:25 smithi043 ceph-mon[42099]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:25.721 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:25 smithi113 ceph-mon[34341]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:27.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:27 smithi043 ceph-mon[29963]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:27.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:27 smithi043 ceph-mon[42099]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:27.721 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:27 smithi113 ceph-mon[34341]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:29.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:29 smithi043 ceph-mon[42099]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:29.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:29 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:29.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:29 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:29.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:29 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:29.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:29 smithi043 ceph-mon[29963]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:29.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:29 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:29.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:29 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:29.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:29 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:29.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:29 smithi113 ceph-mon[34341]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:29.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:29 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:29.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:29 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:29.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:29 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:31.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:31 smithi113 ceph-mon[34341]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:31.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:31 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:31.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:31 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:31.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:31 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:31.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:31 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:31.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:31 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:31.617 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:31 smithi043 ceph-mon[29963]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:31.617 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:31 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:31.617 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:31 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:31.617 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:31 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:31.618 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:31 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:31.618 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:31 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:31.618 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:31 smithi043 ceph-mon[42099]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:31.618 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:31 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:31.618 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:31 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:31.618 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:31 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:31.618 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:31 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:31.618 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:31 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:31.920 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 0 on host 'smithi043' 2024-03-30T13:39:32.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:32 smithi113 ceph-mon[34341]: from='osd.0 [v2:172.21.15.43:6802/1144592572,v1:172.21.15.43:6803/1144592572]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T13:39:32.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:32 smithi113 ceph-mon[34341]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T13:39:32.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:32.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:32.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:32.590 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:32 smithi043 ceph-mon[42099]: from='osd.0 [v2:172.21.15.43:6802/1144592572,v1:172.21.15.43:6803/1144592572]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T13:39:32.590 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:32 smithi043 ceph-mon[42099]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T13:39:32.590 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:32.590 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:32.590 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:32.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:32 smithi043 ceph-mon[29963]: from='osd.0 [v2:172.21.15.43:6802/1144592572,v1:172.21.15.43:6803/1144592572]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T13:39:32.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:32 smithi043 ceph-mon[29963]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T13:39:32.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:32.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:32.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:32.616 DEBUG:teuthology.orchestra.run.smithi043:osd.0> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.0.service 2024-03-30T13:39:32.618 INFO:tasks.cephadm:Deploying osd.1 on smithi043 with /dev/vg_nvme/lv_3... 2024-03-30T13:39:32.618 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-30T13:39:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:39:33.467 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:33 smithi043 ceph-mon[29963]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:33.467 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:33 smithi043 ceph-mon[29963]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T13:39:33.467 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:33 smithi043 ceph-mon[29963]: from='osd.0 [v2:172.21.15.43:6802/1144592572,v1:172.21.15.43:6803/1144592572]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T13:39:33.467 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:33 smithi043 ceph-mon[29963]: osdmap e6: 1 total, 0 up, 1 in 2024-03-30T13:39:33.467 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:33 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:39:33.467 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:33 smithi043 ceph-mon[29963]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T13:39:33.467 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:33 smithi043 ceph-mon[42099]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:33.467 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:33 smithi043 ceph-mon[42099]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T13:39:33.467 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:33 smithi043 ceph-mon[42099]: from='osd.0 [v2:172.21.15.43:6802/1144592572,v1:172.21.15.43:6803/1144592572]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T13:39:33.467 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:33 smithi043 ceph-mon[42099]: osdmap e6: 1 total, 0 up, 1 in 2024-03-30T13:39:33.468 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:33 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:39:33.468 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:33 smithi043 ceph-mon[42099]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T13:39:33.471 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:33 smithi113 ceph-mon[34341]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:33.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:33 smithi113 ceph-mon[34341]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T13:39:33.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:33 smithi113 ceph-mon[34341]: from='osd.0 [v2:172.21.15.43:6802/1144592572,v1:172.21.15.43:6803/1144592572]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T13:39:33.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:33 smithi113 ceph-mon[34341]: osdmap e6: 1 total, 0 up, 1 in 2024-03-30T13:39:33.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:33 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:39:33.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:33 smithi113 ceph-mon[34341]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T13:39:33.744 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 13:39:33 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-0[46512]: 2024-03-30T13:39:33.464+0000 7f993d6ce700 -1 osd.0 0 waiting for initial osdmap 2024-03-30T13:39:33.744 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 13:39:33 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-0[46512]: 2024-03-30T13:39:33.466+0000 7f99354b9700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:39:34.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:34 smithi043 ceph-mon[29963]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-30T13:39:34.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:34 smithi043 ceph-mon[29963]: osdmap e7: 1 total, 0 up, 1 in 2024-03-30T13:39:34.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:34 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:39:34.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:34 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:39:34.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:34 smithi043 ceph-mon[29963]: from='osd.0 ' entity='osd.0' 2024-03-30T13:39:34.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:34 smithi043 ceph-mon[42099]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-30T13:39:34.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:34 smithi043 ceph-mon[42099]: osdmap e7: 1 total, 0 up, 1 in 2024-03-30T13:39:34.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:34 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:39:34.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:34 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:39:34.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:34 smithi043 ceph-mon[42099]: from='osd.0 ' entity='osd.0' 2024-03-30T13:39:34.471 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:34 smithi113 ceph-mon[34341]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-30T13:39:34.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:34 smithi113 ceph-mon[34341]: osdmap e7: 1 total, 0 up, 1 in 2024-03-30T13:39:34.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:34 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:39:34.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:34 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:39:34.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:34 smithi113 ceph-mon[34341]: from='osd.0 ' entity='osd.0' 2024-03-30T13:39:35.466 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:35 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:39:35.466 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:35 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:39:35.466 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:35 smithi043 ceph-mon[42099]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:35.466 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:35 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:39:35.466 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:35 smithi043 ceph-mon[42099]: osd.0 [v2:172.21.15.43:6802/1144592572,v1:172.21.15.43:6803/1144592572] boot 2024-03-30T13:39:35.466 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:35 smithi043 ceph-mon[42099]: osdmap e8: 1 total, 1 up, 1 in 2024-03-30T13:39:35.466 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:35 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:39:35.466 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:35 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:39:35.467 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:35 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:39:35.467 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:35 smithi043 ceph-mon[29963]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:35.467 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:35 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:39:35.467 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:35 smithi043 ceph-mon[29963]: osd.0 [v2:172.21.15.43:6802/1144592572,v1:172.21.15.43:6803/1144592572] boot 2024-03-30T13:39:35.467 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:35 smithi043 ceph-mon[29963]: osdmap e8: 1 total, 1 up, 1 in 2024-03-30T13:39:35.467 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:35 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:39:35.471 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:35 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:39:35.471 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:35 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:39:35.471 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:35 smithi113 ceph-mon[34341]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:39:35.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:35 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:39:35.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:35 smithi113 ceph-mon[34341]: osd.0 [v2:172.21.15.43:6802/1144592572,v1:172.21.15.43:6803/1144592572] boot 2024-03-30T13:39:35.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:35 smithi113 ceph-mon[34341]: osdmap e8: 1 total, 1 up, 1 in 2024-03-30T13:39:35.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:35 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:39:36.300 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-30T13:39:36.300 INFO:teuthology.orchestra.run.smithi043.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-30T13:39:36.300 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10+0 records in 2024-03-30T13:39:36.300 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2024-03-30T13:39:36.300 INFO:teuthology.orchestra.run.smithi043.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0096543 s, 1.1 GB/s 2024-03-30T13:39:36.300 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping successful for: 2024-03-30T13:39:37.002 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi043:vg_nvme/lv_3 2024-03-30T13:39:37.244 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:39:37.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:37 smithi043 ceph-mon[29963]: pgmap v41: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:37.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:37 smithi043 ceph-mon[29963]: osdmap e9: 1 total, 1 up, 1 in 2024-03-30T13:39:37.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:37 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:37.366 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:37 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:37.366 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:37 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:39:37.366 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:37 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:37.366 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:37 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:37.366 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:37 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:37.366 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:37 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:37.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:37 smithi043 ceph-mon[42099]: pgmap v41: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:37.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:37 smithi043 ceph-mon[42099]: osdmap e9: 1 total, 1 up, 1 in 2024-03-30T13:39:37.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:37 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:37.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:37 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:37.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:37 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:39:37.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:37 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:37.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:37 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:37.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:37 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:37.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:37 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:37.471 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:37 smithi113 ceph-mon[34341]: pgmap v41: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:37.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:37 smithi113 ceph-mon[34341]: osdmap e9: 1 total, 1 up, 1 in 2024-03-30T13:39:37.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:37 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:37.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:37 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:37.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:37 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:39:37.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:37 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:37.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:37 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:37.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:37 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:37.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:37 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:38.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:38 smithi043 ceph-mon[29963]: Detected new or changed devices on smithi043 2024-03-30T13:39:38.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:38 smithi043 ceph-mon[29963]: Adjusting osd_memory_target on smithi043 to 16069M 2024-03-30T13:39:38.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:38 smithi043 ceph-mon[42099]: Detected new or changed devices on smithi043 2024-03-30T13:39:38.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:38 smithi043 ceph-mon[42099]: Adjusting osd_memory_target on smithi043 to 16069M 2024-03-30T13:39:38.471 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:38 smithi113 ceph-mon[34341]: Detected new or changed devices on smithi043 2024-03-30T13:39:38.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:38 smithi113 ceph-mon[34341]: Adjusting osd_memory_target on smithi043 to 16069M 2024-03-30T13:39:39.471 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:39 smithi113 ceph-mon[34341]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:39.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:39 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:39:39.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:39 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:39:39.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:39 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:39.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:39 smithi043 ceph-mon[42099]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:39.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:39 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:39:39.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:39 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:39:39.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:39 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:39.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:39 smithi043 ceph-mon[29963]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:39.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:39 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:39:39.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:39 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:39:39.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:39 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:40.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:40 smithi043 ceph-mon[42099]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:39:40.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:40 smithi043 ceph-mon[29963]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:39:40.471 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:40 smithi113 ceph-mon[34341]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:39:41.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:41 smithi113 ceph-mon[34341]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:41.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:41 smithi043 ceph-mon[29963]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:41.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:41 smithi043 ceph-mon[42099]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:42.471 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:42 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1603655186' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a"}]: dispatch 2024-03-30T13:39:42.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:42 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1603655186' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a"}]': finished 2024-03-30T13:39:42.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:42 smithi113 ceph-mon[34341]: osdmap e10: 2 total, 1 up, 2 in 2024-03-30T13:39:42.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:42 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:39:42.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:42 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1603655186' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a"}]: dispatch 2024-03-30T13:39:42.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:42 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1603655186' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a"}]': finished 2024-03-30T13:39:42.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:42 smithi043 ceph-mon[29963]: osdmap e10: 2 total, 1 up, 2 in 2024-03-30T13:39:42.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:42 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:39:42.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:42 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1603655186' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a"}]: dispatch 2024-03-30T13:39:42.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:42 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1603655186' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a"}]': finished 2024-03-30T13:39:42.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:42 smithi043 ceph-mon[42099]: osdmap e10: 2 total, 1 up, 2 in 2024-03-30T13:39:42.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:42 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:39:43.471 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:43 smithi113 ceph-mon[34341]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:43.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:43 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1376659225' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:39:43.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:43 smithi043 ceph-mon[29963]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:43.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:43 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1376659225' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:39:43.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:43 smithi043 ceph-mon[42099]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:43.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:43 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1376659225' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:39:45.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:45 smithi043 ceph-mon[29963]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:45.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:45 smithi043 ceph-mon[42099]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:45.721 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:45 smithi113 ceph-mon[34341]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:47.531 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:47 smithi043 ceph-mon[29963]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:47.532 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:47 smithi043 ceph-mon[42099]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:47.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:47 smithi113 ceph-mon[34341]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:49.524 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:49 smithi113 ceph-mon[34341]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:49.524 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:49 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T13:39:49.524 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:49 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:49.524 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:49 smithi113 ceph-mon[34341]: Deploying daemon osd.1 on smithi043 2024-03-30T13:39:49.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:49 smithi043 ceph-mon[29963]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:49.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:49 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T13:39:49.626 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:49 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:49.626 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:49 smithi043 ceph-mon[29963]: Deploying daemon osd.1 on smithi043 2024-03-30T13:39:49.626 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:49 smithi043 ceph-mon[42099]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:49.626 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:49 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T13:39:49.626 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:49 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:49.626 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:49 smithi043 ceph-mon[42099]: Deploying daemon osd.1 on smithi043 2024-03-30T13:39:51.510 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:51 smithi043 ceph-mon[29963]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:51.511 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:51 smithi043 ceph-mon[42099]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:51.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:51 smithi113 ceph-mon[34341]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:53.315 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:53 smithi043 ceph-mon[29963]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:53.315 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:53 smithi043 ceph-mon[42099]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:53.721 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:53 smithi113 ceph-mon[34341]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:55.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:54 smithi043 ceph-mon[29963]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:55.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:54 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:55.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:54 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:55.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:54 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:55.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:54 smithi043 ceph-mon[42099]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:55.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:55.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:55.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:55.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:54 smithi113 ceph-mon[34341]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:55.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:54 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:55.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:54 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:55.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:54 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:56.983 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:56 smithi043 ceph-mon[29963]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:56.983 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:56 smithi043 ceph-mon[42099]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:57.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:56 smithi113 ceph-mon[34341]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:57.695 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 1 on host 'smithi043' 2024-03-30T13:39:58.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:57 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:58.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:57 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:58.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:57 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:58.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:57 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:58.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:57 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:58.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:57 smithi113 ceph-mon[34341]: from='osd.1 [v2:172.21.15.43:6810/3901165928,v1:172.21.15.43:6811/3901165928]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T13:39:58.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:57 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:58.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:57 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:58.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:57 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:58.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:57 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:58.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:57 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:58.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:57 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:58.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:57 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:58.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:57 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:58.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:57 smithi043 ceph-mon[29963]: from='osd.1 [v2:172.21.15.43:6810/3901165928,v1:172.21.15.43:6811/3901165928]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T13:39:58.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:57 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:58.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:57 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:58.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:57 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:58.281 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:57 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:58.281 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:57 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:58.281 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:57 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:39:58.281 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:57 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:39:58.281 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:57 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:58.281 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:57 smithi043 ceph-mon[42099]: from='osd.1 [v2:172.21.15.43:6810/3901165928,v1:172.21.15.43:6811/3901165928]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T13:39:58.281 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:57 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:58.281 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:57 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:39:58.281 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:57 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:39:58.304 DEBUG:teuthology.orchestra.run.smithi043:osd.1> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.1.service 2024-03-30T13:39:58.307 INFO:tasks.cephadm:Deploying osd.2 on smithi043 with /dev/vg_nvme/lv_2... 2024-03-30T13:39:58.307 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-30T13:39:59.033 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:39:59.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:58 smithi113 ceph-mon[34341]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:59.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:58 smithi113 ceph-mon[34341]: from='osd.1 [v2:172.21.15.43:6810/3901165928,v1:172.21.15.43:6811/3901165928]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T13:39:59.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:58 smithi113 ceph-mon[34341]: osdmap e11: 2 total, 1 up, 2 in 2024-03-30T13:39:59.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:58 smithi113 ceph-mon[34341]: from='osd.1 [v2:172.21.15.43:6810/3901165928,v1:172.21.15.43:6811/3901165928]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T13:39:59.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:58 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:39:59.232 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:58 smithi043 ceph-mon[29963]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:59.233 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:58 smithi043 ceph-mon[29963]: from='osd.1 [v2:172.21.15.43:6810/3901165928,v1:172.21.15.43:6811/3901165928]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T13:39:59.233 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:58 smithi043 ceph-mon[29963]: osdmap e11: 2 total, 1 up, 2 in 2024-03-30T13:39:59.233 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:58 smithi043 ceph-mon[29963]: from='osd.1 [v2:172.21.15.43:6810/3901165928,v1:172.21.15.43:6811/3901165928]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T13:39:59.233 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:58 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:39:59.233 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:58 smithi043 ceph-mon[42099]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:39:59.233 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:58 smithi043 ceph-mon[42099]: from='osd.1 [v2:172.21.15.43:6810/3901165928,v1:172.21.15.43:6811/3901165928]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T13:39:59.233 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:58 smithi043 ceph-mon[42099]: osdmap e11: 2 total, 1 up, 2 in 2024-03-30T13:39:59.233 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:58 smithi043 ceph-mon[42099]: from='osd.1 [v2:172.21.15.43:6810/3901165928,v1:172.21.15.43:6811/3901165928]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T13:39:59.234 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:58 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:39:59.234 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 13:39:58 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-1[52902]: 2024-03-30T13:39:58.951+0000 7f0be9603700 -1 osd.1 0 waiting for initial osdmap 2024-03-30T13:39:59.234 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 13:39:58 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-1[52902]: 2024-03-30T13:39:58.955+0000 7f0be53f6700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:40:00.074 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:59 smithi043 ceph-mon[29963]: from='osd.1 [v2:172.21.15.43:6810/3901165928,v1:172.21.15.43:6811/3901165928]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-30T13:40:00.074 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:59 smithi043 ceph-mon[29963]: osdmap e12: 2 total, 1 up, 2 in 2024-03-30T13:40:00.074 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:59 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:40:00.074 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:59 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:40:00.074 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:39:59 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:40:00.074 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:59 smithi043 ceph-mon[42099]: from='osd.1 [v2:172.21.15.43:6810/3901165928,v1:172.21.15.43:6811/3901165928]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-30T13:40:00.074 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:59 smithi043 ceph-mon[42099]: osdmap e12: 2 total, 1 up, 2 in 2024-03-30T13:40:00.074 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:59 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:40:00.075 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:59 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:40:00.075 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:39:59 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:40:00.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:59 smithi113 ceph-mon[34341]: from='osd.1 [v2:172.21.15.43:6810/3901165928,v1:172.21.15.43:6811/3901165928]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-30T13:40:00.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:59 smithi113 ceph-mon[34341]: osdmap e12: 2 total, 1 up, 2 in 2024-03-30T13:40:00.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:59 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:40:00.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:59 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:40:00.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:39:59 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:40:01.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:00 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:40:01.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:00 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:40:01.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:00 smithi113 ceph-mon[34341]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:40:01.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:00 smithi113 ceph-mon[34341]: osd.1 [v2:172.21.15.43:6810/3901165928,v1:172.21.15.43:6811/3901165928] boot 2024-03-30T13:40:01.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:00 smithi113 ceph-mon[34341]: osdmap e13: 2 total, 2 up, 2 in 2024-03-30T13:40:01.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:00 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:40:01.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:00 smithi113 ceph-mon[34341]: overall HEALTH_OK 2024-03-30T13:40:01.290 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:00 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:40:01.290 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:00 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:40:01.290 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:00 smithi043 ceph-mon[29963]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:40:01.290 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:00 smithi043 ceph-mon[29963]: osd.1 [v2:172.21.15.43:6810/3901165928,v1:172.21.15.43:6811/3901165928] boot 2024-03-30T13:40:01.290 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:00 smithi043 ceph-mon[29963]: osdmap e13: 2 total, 2 up, 2 in 2024-03-30T13:40:01.290 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:00 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:40:01.290 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:00 smithi043 ceph-mon[29963]: overall HEALTH_OK 2024-03-30T13:40:01.291 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:00 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:40:01.291 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:00 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:40:01.291 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:00 smithi043 ceph-mon[42099]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:40:01.291 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:00 smithi043 ceph-mon[42099]: osd.1 [v2:172.21.15.43:6810/3901165928,v1:172.21.15.43:6811/3901165928] boot 2024-03-30T13:40:01.291 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:00 smithi043 ceph-mon[42099]: osdmap e13: 2 total, 2 up, 2 in 2024-03-30T13:40:01.291 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:00 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:40:01.291 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:00 smithi043 ceph-mon[42099]: overall HEALTH_OK 2024-03-30T13:40:01.883 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-30T13:40:01.883 INFO:teuthology.orchestra.run.smithi043.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-30T13:40:01.883 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10+0 records in 2024-03-30T13:40:01.883 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2024-03-30T13:40:01.883 INFO:teuthology.orchestra.run.smithi043.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00926928 s, 1.1 GB/s 2024-03-30T13:40:01.883 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping successful for: 2024-03-30T13:40:02.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:01 smithi043 ceph-mon[42099]: osdmap e14: 2 total, 2 up, 2 in 2024-03-30T13:40:02.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:01 smithi043 ceph-mon[29963]: osdmap e14: 2 total, 2 up, 2 in 2024-03-30T13:40:02.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:01 smithi113 ceph-mon[34341]: osdmap e14: 2 total, 2 up, 2 in 2024-03-30T13:40:02.585 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi043:vg_nvme/lv_2 2024-03-30T13:40:03.165 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:40:03.286 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:02 smithi043 ceph-mon[29963]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:03.287 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:02 smithi043 ceph-mon[42099]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:03.471 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:02 smithi113 ceph-mon[34341]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:04.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:04 smithi043 ceph-mon[29963]: Detected new or changed devices on smithi043 2024-03-30T13:40:04.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:04 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:04.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:04 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:04.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:04 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:04.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:04 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:04.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:04 smithi043 ceph-mon[29963]: Adjusting osd_memory_target on smithi043 to 8034M 2024-03-30T13:40:04.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:04 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:04.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:04 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:04.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:04 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:40:04.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:04 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:04.347 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:04 smithi043 ceph-mon[42099]: Detected new or changed devices on smithi043 2024-03-30T13:40:04.347 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:04 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:04.347 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:04 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:04.347 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:04 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:04.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:04 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:04.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:04 smithi043 ceph-mon[42099]: Adjusting osd_memory_target on smithi043 to 8034M 2024-03-30T13:40:04.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:04 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:04.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:04 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:04.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:04 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:40:04.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:04 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:04.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:04 smithi113 ceph-mon[34341]: Detected new or changed devices on smithi043 2024-03-30T13:40:04.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:04 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:04.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:04 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:04.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:04 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:04.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:04 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:04.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:04 smithi113 ceph-mon[34341]: Adjusting osd_memory_target on smithi043 to 8034M 2024-03-30T13:40:04.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:04 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:04.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:04 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:04.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:04 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:40:04.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:04 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:05.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:05 smithi043 ceph-mon[29963]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:05.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:05 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:40:05.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:05 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:40:05.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:05 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:05.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:05 smithi043 ceph-mon[42099]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:05.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:05 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:40:05.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:05 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:40:05.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:05 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:05.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:05 smithi113 ceph-mon[34341]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:05.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:05 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:40:05.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:05 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:40:05.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:05 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:06.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:06 smithi043 ceph-mon[29963]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:40:06.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:06 smithi043 ceph-mon[42099]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:40:06.471 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:06 smithi113 ceph-mon[34341]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:40:07.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:07 smithi043 ceph-mon[29963]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:07.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:07 smithi043 ceph-mon[42099]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:07.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:07 smithi113 ceph-mon[34341]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:08.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:08 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3977146698' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6db6782e-5889-443e-bc6c-df47a1f93bc8"}]: dispatch 2024-03-30T13:40:08.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:08 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3977146698' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6db6782e-5889-443e-bc6c-df47a1f93bc8"}]': finished 2024-03-30T13:40:08.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:08 smithi043 ceph-mon[29963]: osdmap e15: 3 total, 2 up, 3 in 2024-03-30T13:40:08.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:08 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:40:08.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:08 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3977146698' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6db6782e-5889-443e-bc6c-df47a1f93bc8"}]: dispatch 2024-03-30T13:40:08.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:08 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3977146698' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6db6782e-5889-443e-bc6c-df47a1f93bc8"}]': finished 2024-03-30T13:40:08.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:08 smithi043 ceph-mon[42099]: osdmap e15: 3 total, 2 up, 3 in 2024-03-30T13:40:08.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:08 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:40:08.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:08 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3977146698' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6db6782e-5889-443e-bc6c-df47a1f93bc8"}]: dispatch 2024-03-30T13:40:08.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:08 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3977146698' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6db6782e-5889-443e-bc6c-df47a1f93bc8"}]': finished 2024-03-30T13:40:08.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:08 smithi113 ceph-mon[34341]: osdmap e15: 3 total, 2 up, 3 in 2024-03-30T13:40:08.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:08 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:40:09.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:09 smithi043 ceph-mon[42099]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:09.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:09 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1605461726' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:40:09.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:09 smithi043 ceph-mon[29963]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:09.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:09 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1605461726' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:40:09.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:09 smithi113 ceph-mon[34341]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:09.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:09 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1605461726' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:40:11.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:11 smithi043 ceph-mon[29963]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:11.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:11 smithi043 ceph-mon[42099]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:11.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:11 smithi113 ceph-mon[34341]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:13.566 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:13 smithi043 ceph-mon[29963]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:13.566 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:13 smithi043 ceph-mon[42099]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:13.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:13 smithi113 ceph-mon[34341]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:15.431 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:15 smithi043 ceph-mon[29963]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:15.431 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:15 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T13:40:15.431 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:15 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:15.431 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:15 smithi043 ceph-mon[42099]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:15.431 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:15 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T13:40:15.431 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:15 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:15.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:15 smithi113 ceph-mon[34341]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:15.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:15 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T13:40:15.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:15 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:16.538 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:16 smithi043 ceph-mon[29963]: Deploying daemon osd.2 on smithi043 2024-03-30T13:40:16.538 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:16 smithi043 ceph-mon[42099]: Deploying daemon osd.2 on smithi043 2024-03-30T13:40:16.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:16 smithi113 ceph-mon[34341]: Deploying daemon osd.2 on smithi043 2024-03-30T13:40:17.541 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:17 smithi043 ceph-mon[29963]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:17.541 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:17 smithi043 ceph-mon[42099]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:17.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:17 smithi113 ceph-mon[34341]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:19.524 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:19 smithi113 ceph-mon[34341]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:19.599 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:19 smithi043 ceph-mon[29963]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:19.599 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:19 smithi043 ceph-mon[42099]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:20.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:20 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:40:20.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:20 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:20.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:20 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:20.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:20 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:40:20.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:20 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:20.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:20 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:20.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:20 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:40:20.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:20 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:20.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:20 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:21.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:21 smithi043 ceph-mon[29963]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:21.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:21 smithi043 ceph-mon[42099]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:21.721 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:21 smithi113 ceph-mon[34341]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:23.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:23 smithi043 ceph-mon[29963]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:23.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:23 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:23.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:23 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:23.594 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:23 smithi043 ceph-mon[42099]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:23.594 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:23 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:23.594 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:23 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:23.721 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:23 smithi113 ceph-mon[34341]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:23.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:23 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:23.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:23 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:23.948 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 2 on host 'smithi043' 2024-03-30T13:40:24.522 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:24 smithi043 ceph-mon[29963]: from='osd.2 [v2:172.21.15.43:6818/3415037523,v1:172.21.15.43:6819/3415037523]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T13:40:24.522 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:24 smithi043 ceph-mon[29963]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T13:40:24.522 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:24 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:24.522 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:24 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:40:24.522 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:24 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:24.522 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:24 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:24.523 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:24 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:24.523 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:24 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:40:24.523 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:24 smithi043 ceph-mon[42099]: from='osd.2 [v2:172.21.15.43:6818/3415037523,v1:172.21.15.43:6819/3415037523]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T13:40:24.523 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:24 smithi043 ceph-mon[42099]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T13:40:24.523 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:24 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:24.523 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:24 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:40:24.523 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:24 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:24.523 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:24 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:24.523 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:24 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:24.523 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:24 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:40:24.547 DEBUG:teuthology.orchestra.run.smithi043:osd.2> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.2.service 2024-03-30T13:40:24.548 INFO:tasks.cephadm:Deploying osd.3 on smithi043 with /dev/vg_nvme/lv_1... 2024-03-30T13:40:24.549 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-30T13:40:24.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:24 smithi113 ceph-mon[34341]: from='osd.2 [v2:172.21.15.43:6818/3415037523,v1:172.21.15.43:6819/3415037523]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T13:40:24.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:24 smithi113 ceph-mon[34341]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T13:40:24.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:24 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:24.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:24 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:40:24.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:24 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:24.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:24 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:24.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:24 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:24.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:24 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:40:25.319 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:40:25.547 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:25 smithi043 ceph-mon[29963]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:25.548 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:25 smithi043 ceph-mon[29963]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T13:40:25.548 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:25 smithi043 ceph-mon[29963]: osdmap e16: 3 total, 2 up, 3 in 2024-03-30T13:40:25.548 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:25 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:40:25.548 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:25 smithi043 ceph-mon[29963]: from='osd.2 [v2:172.21.15.43:6818/3415037523,v1:172.21.15.43:6819/3415037523]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T13:40:25.548 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:25 smithi043 ceph-mon[29963]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T13:40:25.548 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:25 smithi043 ceph-mon[42099]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:25.548 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:25 smithi043 ceph-mon[42099]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T13:40:25.548 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:25 smithi043 ceph-mon[42099]: osdmap e16: 3 total, 2 up, 3 in 2024-03-30T13:40:25.548 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:25 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:40:25.548 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:25 smithi043 ceph-mon[42099]: from='osd.2 [v2:172.21.15.43:6818/3415037523,v1:172.21.15.43:6819/3415037523]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T13:40:25.549 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:25 smithi043 ceph-mon[42099]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T13:40:25.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:25 smithi113 ceph-mon[34341]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:25.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:25 smithi113 ceph-mon[34341]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T13:40:25.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:25 smithi113 ceph-mon[34341]: osdmap e16: 3 total, 2 up, 3 in 2024-03-30T13:40:25.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:25 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:40:25.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:25 smithi113 ceph-mon[34341]: from='osd.2 [v2:172.21.15.43:6818/3415037523,v1:172.21.15.43:6819/3415037523]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T13:40:25.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:25 smithi113 ceph-mon[34341]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T13:40:25.899 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:40:25 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2[59302]: 2024-03-30T13:40:25.545+0000 7f59e8b59700 -1 osd.2 0 waiting for initial osdmap 2024-03-30T13:40:25.899 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:40:25 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2[59302]: 2024-03-30T13:40:25.548+0000 7f59e0143700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:40:26.539 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:26 smithi043 ceph-mon[29963]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-30T13:40:26.539 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:26 smithi043 ceph-mon[29963]: osdmap e17: 3 total, 2 up, 3 in 2024-03-30T13:40:26.539 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:26 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:40:26.539 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:26 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:40:26.539 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:26 smithi043 ceph-mon[29963]: from='osd.2 ' entity='osd.2' 2024-03-30T13:40:26.540 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:26 smithi043 ceph-mon[42099]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-30T13:40:26.540 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:26 smithi043 ceph-mon[42099]: osdmap e17: 3 total, 2 up, 3 in 2024-03-30T13:40:26.540 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:26 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:40:26.540 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:26 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:40:26.540 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:26 smithi043 ceph-mon[42099]: from='osd.2 ' entity='osd.2' 2024-03-30T13:40:26.721 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:26 smithi113 ceph-mon[34341]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-30T13:40:26.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:26 smithi113 ceph-mon[34341]: osdmap e17: 3 total, 2 up, 3 in 2024-03-30T13:40:26.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:26 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:40:26.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:26 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:40:26.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:26 smithi113 ceph-mon[34341]: from='osd.2 ' entity='osd.2' 2024-03-30T13:40:27.495 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:27 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:40:27.495 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:27 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:40:27.495 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:27 smithi043 ceph-mon[42099]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:27.495 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:27 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:40:27.495 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:27 smithi043 ceph-mon[42099]: osd.2 [v2:172.21.15.43:6818/3415037523,v1:172.21.15.43:6819/3415037523] boot 2024-03-30T13:40:27.495 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:27 smithi043 ceph-mon[42099]: osdmap e18: 3 total, 3 up, 3 in 2024-03-30T13:40:27.495 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:27 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:40:27.495 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:27 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:40:27.496 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:27 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:40:27.496 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:27 smithi043 ceph-mon[29963]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:27.496 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:27 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:40:27.496 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:27 smithi043 ceph-mon[29963]: osd.2 [v2:172.21.15.43:6818/3415037523,v1:172.21.15.43:6819/3415037523] boot 2024-03-30T13:40:27.496 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:27 smithi043 ceph-mon[29963]: osdmap e18: 3 total, 3 up, 3 in 2024-03-30T13:40:27.496 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:27 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:40:27.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:27 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:40:27.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:27 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:40:27.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:27 smithi113 ceph-mon[34341]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:40:27.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:27 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:40:27.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:27 smithi113 ceph-mon[34341]: osd.2 [v2:172.21.15.43:6818/3415037523,v1:172.21.15.43:6819/3415037523] boot 2024-03-30T13:40:27.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:27 smithi113 ceph-mon[34341]: osdmap e18: 3 total, 3 up, 3 in 2024-03-30T13:40:27.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:27 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:40:28.246 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-30T13:40:28.247 INFO:teuthology.orchestra.run.smithi043.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-30T13:40:28.247 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10+0 records in 2024-03-30T13:40:28.247 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2024-03-30T13:40:28.247 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00926189 s, 1.1 GB/s 2024-03-30T13:40:28.247 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping successful for: 2024-03-30T13:40:28.619 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:28 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:40:28.619 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:28 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-30T13:40:28.619 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:28 smithi043 ceph-mon[29963]: osdmap e19: 3 total, 3 up, 3 in 2024-03-30T13:40:28.619 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:28 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:40:28.619 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:28 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:40:28.619 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:28 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-30T13:40:28.619 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:28 smithi043 ceph-mon[42099]: osdmap e19: 3 total, 3 up, 3 in 2024-03-30T13:40:28.619 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:28 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:40:28.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:28 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:40:28.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:28 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-30T13:40:28.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:28 smithi113 ceph-mon[34341]: osdmap e19: 3 total, 3 up, 3 in 2024-03-30T13:40:28.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:28 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:40:28.971 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi043:vg_nvme/lv_1 2024-03-30T13:40:29.214 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:40:29.616 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[29963]: pgmap v77: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:29.616 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:29.616 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:29.616 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:29.616 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:29.616 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:29.616 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:29.616 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:29.617 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:40:29.617 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:29.617 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-30T13:40:29.617 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[29963]: osdmap e20: 3 total, 3 up, 3 in 2024-03-30T13:40:29.617 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[42099]: pgmap v77: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:29.617 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:29.617 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:29.617 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:29.617 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:29.617 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:29.617 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:29.617 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:29.618 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:40:29.618 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:29.618 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-30T13:40:29.618 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:29 smithi043 ceph-mon[42099]: osdmap e20: 3 total, 3 up, 3 in 2024-03-30T13:40:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:29 smithi113 ceph-mon[34341]: pgmap v77: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:29 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:29 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:29 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:29 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:29 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:29 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:29 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:29 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:40:29.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:29 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:29.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:29 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-30T13:40:29.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:29 smithi113 ceph-mon[34341]: osdmap e20: 3 total, 3 up, 3 in 2024-03-30T13:40:29.886 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 13:40:29 smithi043 sudo[63592]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T13:40:29.886 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 13:40:29 smithi043 sudo[63592]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:40:29.886 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 13:40:29 smithi043 sudo[63592]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:40:29.886 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 13:40:29 smithi043 sudo[63592]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:40:30.149 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 13:40:29 smithi043 sudo[63613]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T13:40:30.149 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 13:40:29 smithi043 sudo[63613]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:40:30.149 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 13:40:29 smithi043 sudo[63613]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:40:30.149 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 13:40:29 smithi043 sudo[63613]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:40:30.433 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:30 smithi043 ceph-mon[29963]: Detected new or changed devices on smithi043 2024-03-30T13:40:30.433 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:30 smithi043 ceph-mon[29963]: Adjusting osd_memory_target on smithi043 to 5356M 2024-03-30T13:40:30.433 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:30 smithi043 ceph-mon[29963]: osdmap e21: 3 total, 3 up, 3 in 2024-03-30T13:40:30.433 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:30 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:30.434 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:30 smithi043 ceph-mon[42099]: Detected new or changed devices on smithi043 2024-03-30T13:40:30.434 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:30 smithi043 ceph-mon[42099]: Adjusting osd_memory_target on smithi043 to 5356M 2024-03-30T13:40:30.434 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:30 smithi043 ceph-mon[42099]: osdmap e21: 3 total, 3 up, 3 in 2024-03-30T13:40:30.434 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:30 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:30.434 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 13:40:30 smithi043 sudo[63624]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T13:40:30.434 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 13:40:30 smithi043 sudo[63624]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:40:30.434 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 13:40:30 smithi043 sudo[63624]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:40:30.434 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 13:40:30 smithi043 sudo[63624]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:40:30.693 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 13:40:30 smithi043 sudo[63630]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T13:40:30.693 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 13:40:30 smithi043 sudo[63630]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:40:30.693 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 13:40:30 smithi043 sudo[63630]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:40:30.693 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 13:40:30 smithi043 sudo[63630]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:40:30.693 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:40:30 smithi043 sudo[63658]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T13:40:30.721 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:30 smithi113 ceph-mon[34341]: Detected new or changed devices on smithi043 2024-03-30T13:40:30.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:30 smithi113 ceph-mon[34341]: Adjusting osd_memory_target on smithi043 to 5356M 2024-03-30T13:40:30.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:30 smithi113 ceph-mon[34341]: osdmap e21: 3 total, 3 up, 3 in 2024-03-30T13:40:30.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:30 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:30.963 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:40:30 smithi043 sudo[63658]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:40:30.963 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:40:30 smithi043 sudo[63658]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:40:30.963 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:40:30 smithi043 sudo[63658]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:40:31.222 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:40:30 smithi043 sudo[63692]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T13:40:31.223 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:40:30 smithi043 sudo[63692]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:40:31.223 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:40:30 smithi043 sudo[63692]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:40:31.223 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:40:30 smithi043 sudo[63692]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:40:31.491 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:31 smithi043 sudo[63711]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T13:40:31.491 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:31 smithi043 sudo[63711]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:40:31.491 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:31 smithi043 sudo[63711]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:40:31.491 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:31 smithi043 sudo[63711]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:40:31.754 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:31 smithi113 ceph-mon[34341]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:31.754 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:31 smithi113 ceph-mon[34341]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-30T13:40:31.754 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:31 smithi113 ceph-mon[34341]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T13:40:31.755 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:31 smithi113 sudo[37199]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T13:40:31.755 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:31 smithi113 sudo[37199]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:40:31.755 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:31 smithi113 sudo[37199]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:40:31.755 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:31 smithi113 sudo[37199]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:40:31.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:31 smithi043 ceph-mon[42099]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:31.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:31 smithi043 ceph-mon[42099]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-30T13:40:31.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:31 smithi043 ceph-mon[42099]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T13:40:31.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:31 smithi043 sudo[63715]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T13:40:31.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:31 smithi043 sudo[63715]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:40:31.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:31 smithi043 sudo[63715]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:40:31.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:31 smithi043 sudo[63715]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:40:31.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:31 smithi043 ceph-mon[29963]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:31.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:31 smithi043 ceph-mon[29963]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-30T13:40:31.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:31 smithi043 ceph-mon[29963]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T13:40:32.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:31 smithi113 sudo[37202]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T13:40:32.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:31 smithi113 sudo[37202]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:40:32.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:31 smithi113 sudo[37202]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:40:32.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:31 smithi113 sudo[37202]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:40:32.532 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 sudo[63843]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T13:40:32.533 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 sudo[63843]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:40:32.533 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 sudo[63843]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:40:32.533 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 sudo[63843]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:40:32.784 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[29963]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:32.784 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[29963]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T13:40:32.784 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:40:32.784 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:40:32.784 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:40:32.784 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[29963]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T13:40:32.784 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:40:32.784 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:40:32.785 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:40:32.785 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[29963]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:40:32.785 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:40:32.785 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:40:32.785 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:32.785 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[29963]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T13:40:32.785 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:32.785 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:40:32.785 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:40:32.785 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:40:32.785 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 sudo[63865]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T13:40:32.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 sudo[63865]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:40:32.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 sudo[63865]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:40:32.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 sudo[63865]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:40:32.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[42099]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:32.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[42099]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T13:40:32.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:40:32.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:40:32.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:40:32.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[42099]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T13:40:32.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:40:32.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:40:32.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:40:32.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[42099]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:40:32.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:40:32.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:40:32.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:32.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[42099]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T13:40:32.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:32.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:40:32.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:40:32.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:40:32.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:32 smithi113 ceph-mon[34341]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:32.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:32 smithi113 ceph-mon[34341]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T13:40:32.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:40:32.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:40:32.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:40:32.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:32 smithi113 ceph-mon[34341]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T13:40:32.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:40:32.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:40:32.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:40:32.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:32 smithi113 ceph-mon[34341]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:40:32.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:40:32.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:40:32.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:32.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:32 smithi113 ceph-mon[34341]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T13:40:32.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:32.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:40:32.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:40:32.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:40:33.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:33 smithi043 ceph-mon[42099]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T13:40:33.630 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:33 smithi043 ceph-mon[42099]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T13:40:33.630 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:33 smithi043 ceph-mon[29963]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T13:40:33.630 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:33 smithi043 ceph-mon[29963]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T13:40:33.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:33 smithi113 ceph-mon[34341]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T13:40:33.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:33 smithi113 ceph-mon[34341]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T13:40:34.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:34 smithi043 ceph-mon[29963]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:34.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:34 smithi043 ceph-mon[42099]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:34.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:34 smithi113 ceph-mon[34341]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:35.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:35 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3843287716' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7048ff46-2cba-4d18-9835-95ecbddb2bb6"}]: dispatch 2024-03-30T13:40:35.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:35 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3843287716' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7048ff46-2cba-4d18-9835-95ecbddb2bb6"}]': finished 2024-03-30T13:40:35.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:35 smithi043 ceph-mon[29963]: osdmap e22: 4 total, 3 up, 4 in 2024-03-30T13:40:35.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:35 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:40:35.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:35 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1191429334' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:40:35.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:35 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3843287716' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7048ff46-2cba-4d18-9835-95ecbddb2bb6"}]: dispatch 2024-03-30T13:40:35.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:35 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3843287716' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7048ff46-2cba-4d18-9835-95ecbddb2bb6"}]': finished 2024-03-30T13:40:35.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:35 smithi043 ceph-mon[42099]: osdmap e22: 4 total, 3 up, 4 in 2024-03-30T13:40:35.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:35 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:40:35.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:35 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1191429334' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:40:35.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:35 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3843287716' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7048ff46-2cba-4d18-9835-95ecbddb2bb6"}]: dispatch 2024-03-30T13:40:35.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:35 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3843287716' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7048ff46-2cba-4d18-9835-95ecbddb2bb6"}]': finished 2024-03-30T13:40:35.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:35 smithi113 ceph-mon[34341]: osdmap e22: 4 total, 3 up, 4 in 2024-03-30T13:40:35.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:35 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:40:35.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:35 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1191429334' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:40:36.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:36 smithi043 ceph-mon[29963]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:36.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:36 smithi043 ceph-mon[42099]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:36.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:36 smithi113 ceph-mon[34341]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:38.865 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:38 smithi043 ceph-mon[29963]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-30T13:40:38.866 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:38 smithi043 ceph-mon[42099]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-30T13:40:38.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:38 smithi113 ceph-mon[34341]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-30T13:40:40.883 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:40 smithi043 ceph-mon[29963]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-30T13:40:40.883 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:40 smithi043 ceph-mon[42099]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-30T13:40:40.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:40 smithi113 ceph-mon[34341]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-30T13:40:41.883 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:41 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T13:40:41.883 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:41 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:41.883 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:41 smithi043 ceph-mon[29963]: Deploying daemon osd.3 on smithi043 2024-03-30T13:40:41.884 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:41 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T13:40:41.884 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:41 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:41.884 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:41 smithi043 ceph-mon[42099]: Deploying daemon osd.3 on smithi043 2024-03-30T13:40:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:41 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T13:40:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:41 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:41 smithi113 ceph-mon[34341]: Deploying daemon osd.3 on smithi043 2024-03-30T13:40:42.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:42 smithi043 ceph-mon[29963]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-30T13:40:42.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:42 smithi043 ceph-mon[42099]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-30T13:40:42.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:42 smithi113 ceph-mon[34341]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-30T13:40:44.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:44 smithi043 ceph-mon[29963]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-30T13:40:44.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:44 smithi043 ceph-mon[42099]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-30T13:40:44.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:44 smithi113 ceph-mon[34341]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-30T13:40:46.871 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:46 smithi043 ceph-mon[29963]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 73 KiB/s wr, 5 op/s 2024-03-30T13:40:46.871 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:46 smithi043 ceph-mon[42099]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 73 KiB/s wr, 5 op/s 2024-03-30T13:40:46.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:46 smithi113 ceph-mon[34341]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 73 KiB/s wr, 5 op/s 2024-03-30T13:40:48.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:48 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:48.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:48 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:48.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:48 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:40:48.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:48 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:48.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:48 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:48.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:48 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:40:48.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:48 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:48.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:48 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:48.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:48 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:40:49.359 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:49 smithi043 ceph-mon[29963]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-30T13:40:49.359 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:49 smithi043 ceph-mon[42099]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-30T13:40:49.721 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:49 smithi113 ceph-mon[34341]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-30T13:40:50.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:50 smithi043 ceph-mon[29963]: from='osd.3 [v2:172.21.15.43:6826/2933589376,v1:172.21.15.43:6827/2933589376]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T13:40:50.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:50 smithi043 ceph-mon[42099]: from='osd.3 [v2:172.21.15.43:6826/2933589376,v1:172.21.15.43:6827/2933589376]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T13:40:50.721 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:50 smithi113 ceph-mon[34341]: from='osd.3 [v2:172.21.15.43:6826/2933589376,v1:172.21.15.43:6827/2933589376]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T13:40:50.724 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 3 on host 'smithi043' 2024-03-30T13:40:51.386 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:51 smithi043 ceph-mon[29963]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:51.386 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:51 smithi043 ceph-mon[29963]: from='osd.3 [v2:172.21.15.43:6826/2933589376,v1:172.21.15.43:6827/2933589376]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T13:40:51.386 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:51 smithi043 ceph-mon[29963]: osdmap e23: 4 total, 3 up, 4 in 2024-03-30T13:40:51.386 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:51 smithi043 ceph-mon[29963]: from='osd.3 [v2:172.21.15.43:6826/2933589376,v1:172.21.15.43:6827/2933589376]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T13:40:51.386 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:51 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:40:51.386 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:51 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:51.386 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:51 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:51.387 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:51 smithi043 ceph-mon[42099]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:51.387 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:51 smithi043 ceph-mon[42099]: from='osd.3 [v2:172.21.15.43:6826/2933589376,v1:172.21.15.43:6827/2933589376]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T13:40:51.387 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:51 smithi043 ceph-mon[42099]: osdmap e23: 4 total, 3 up, 4 in 2024-03-30T13:40:51.387 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:51 smithi043 ceph-mon[42099]: from='osd.3 [v2:172.21.15.43:6826/2933589376,v1:172.21.15.43:6827/2933589376]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T13:40:51.387 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:51 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:40:51.387 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:51 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:51.387 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:51 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:51.410 DEBUG:teuthology.orchestra.run.smithi043:osd.3> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.3.service 2024-03-30T13:40:51.413 INFO:tasks.cephadm:Deploying osd.4 on smithi113 with /dev/vg_nvme/lv_4... 2024-03-30T13:40:51.413 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-30T13:40:51.658 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:40:51.721 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:51 smithi113 ceph-mon[34341]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:51.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:51 smithi113 ceph-mon[34341]: from='osd.3 [v2:172.21.15.43:6826/2933589376,v1:172.21.15.43:6827/2933589376]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T13:40:51.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:51 smithi113 ceph-mon[34341]: osdmap e23: 4 total, 3 up, 4 in 2024-03-30T13:40:51.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:51 smithi113 ceph-mon[34341]: from='osd.3 [v2:172.21.15.43:6826/2933589376,v1:172.21.15.43:6827/2933589376]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T13:40:51.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:51 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:40:51.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:51 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:51.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:51 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:52.394 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:52 smithi113 ceph-mon[34341]: from='osd.3 [v2:172.21.15.43:6826/2933589376,v1:172.21.15.43:6827/2933589376]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-30T13:40:52.394 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:52 smithi113 ceph-mon[34341]: osdmap e24: 4 total, 3 up, 4 in 2024-03-30T13:40:52.394 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:52 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:40:52.394 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:52 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:40:52.394 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:52 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:52.394 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:52 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:52.539 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:52 smithi043 ceph-mon[29963]: from='osd.3 [v2:172.21.15.43:6826/2933589376,v1:172.21.15.43:6827/2933589376]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-30T13:40:52.539 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:52 smithi043 ceph-mon[29963]: osdmap e24: 4 total, 3 up, 4 in 2024-03-30T13:40:52.539 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:52 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:40:52.539 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:52 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:40:52.540 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:52 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:52.540 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:52 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:52.540 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:52 smithi043 ceph-mon[42099]: from='osd.3 [v2:172.21.15.43:6826/2933589376,v1:172.21.15.43:6827/2933589376]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-30T13:40:52.540 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:52 smithi043 ceph-mon[42099]: osdmap e24: 4 total, 3 up, 4 in 2024-03-30T13:40:52.540 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:52 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:40:52.540 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:52 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:40:52.540 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:52 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:52.540 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:52 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:53.397 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:53 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:40:53.397 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:53 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:40:53.397 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:53 smithi113 ceph-mon[34341]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:53.397 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:53 smithi113 ceph-mon[34341]: osd.3 [v2:172.21.15.43:6826/2933589376,v1:172.21.15.43:6827/2933589376] boot 2024-03-30T13:40:53.397 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:53 smithi113 ceph-mon[34341]: osdmap e25: 4 total, 4 up, 4 in 2024-03-30T13:40:53.397 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:53 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:40:53.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:53 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:40:53.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:53 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:40:53.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:53 smithi043 ceph-mon[42099]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:53.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:53 smithi043 ceph-mon[42099]: osd.3 [v2:172.21.15.43:6826/2933589376,v1:172.21.15.43:6827/2933589376] boot 2024-03-30T13:40:53.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:53 smithi043 ceph-mon[42099]: osdmap e25: 4 total, 4 up, 4 in 2024-03-30T13:40:53.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:53 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:40:53.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:53 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:40:53.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:53 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:40:53.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:53 smithi043 ceph-mon[29963]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:40:53.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:53 smithi043 ceph-mon[29963]: osd.3 [v2:172.21.15.43:6826/2933589376,v1:172.21.15.43:6827/2933589376] boot 2024-03-30T13:40:53.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:53 smithi043 ceph-mon[29963]: osdmap e25: 4 total, 4 up, 4 in 2024-03-30T13:40:53.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:53 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:40:53.970 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-30T13:40:53.971 INFO:teuthology.orchestra.run.smithi113.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-30T13:40:53.971 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10+0 records in 2024-03-30T13:40:53.971 INFO:teuthology.orchestra.run.smithi113.stderr:10+0 records out 2024-03-30T13:40:53.971 INFO:teuthology.orchestra.run.smithi113.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00855119 s, 1.2 GB/s 2024-03-30T13:40:53.971 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 2024-03-30T13:40:53.971 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping successful for: 2024-03-30T13:40:54.562 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:54 smithi113 ceph-mon[34341]: osdmap e26: 4 total, 4 up, 4 in 2024-03-30T13:40:54.589 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi113:vg_nvme/lv_4 2024-03-30T13:40:54.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:54 smithi043 ceph-mon[29963]: osdmap e26: 4 total, 4 up, 4 in 2024-03-30T13:40:54.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:54 smithi043 ceph-mon[42099]: osdmap e26: 4 total, 4 up, 4 in 2024-03-30T13:40:54.822 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:40:55.568 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:55 smithi113 ceph-mon[34341]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:40:55.568 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:55 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:55.569 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:55 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:55.569 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:55 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:55.569 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:55 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:55.569 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:55 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:55.569 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:55 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:55.569 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:55 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:55.569 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:55 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:55.569 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:55 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:40:55.569 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:55 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:55.569 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:55 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:40:55.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[29963]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:40:55.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:55.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:55.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:55.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:55.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:55.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:55.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:55.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:55.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:40:55.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:55.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:40:55.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[42099]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:40:55.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:55.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:55.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:55.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:55.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:55.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:40:55.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:55.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:55.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:40:55.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:55.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:55 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:40:56.471 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:56 smithi113 ceph-mon[34341]: Detected new or changed devices on smithi043 2024-03-30T13:40:56.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:56 smithi113 ceph-mon[34341]: Adjusting osd_memory_target on smithi043 to 4017M 2024-03-30T13:40:56.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:56 smithi043 ceph-mon[29963]: Detected new or changed devices on smithi043 2024-03-30T13:40:56.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:56 smithi043 ceph-mon[29963]: Adjusting osd_memory_target on smithi043 to 4017M 2024-03-30T13:40:56.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:56 smithi043 ceph-mon[42099]: Detected new or changed devices on smithi043 2024-03-30T13:40:56.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:56 smithi043 ceph-mon[42099]: Adjusting osd_memory_target on smithi043 to 4017M 2024-03-30T13:40:57.382 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:57 smithi113 ceph-mon[34341]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:40:57.382 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:57 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:57.382 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:57 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:40:57.382 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:57 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:57.382 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:57 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:40:57.382 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:57 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:40:57.383 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:57 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:57.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:57 smithi043 ceph-mon[29963]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:40:57.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:57 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:57.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:57 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:40:57.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:57 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:57.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:57 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:40:57.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:57 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:40:57.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:57 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:57.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:57 smithi043 ceph-mon[42099]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:40:57.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:57 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:57.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:57 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:40:57.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:57 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:40:57.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:57 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:40:57.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:57 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:40:57.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:57 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:40:58.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:58 smithi043 ceph-mon[29963]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:40:58.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:58 smithi043 ceph-mon[42099]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:40:58.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:58 smithi113 ceph-mon[34341]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:40:59.550 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:59 smithi043 ceph-mon[29963]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:40:59.551 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:59 smithi043 ceph-mon[29963]: from='client.? 172.21.15.113:0/993817305' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "42da30e6-9eec-4ed5-8c73-9ce78aa8ff63"}]: dispatch 2024-03-30T13:40:59.551 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:59 smithi043 ceph-mon[29963]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "42da30e6-9eec-4ed5-8c73-9ce78aa8ff63"}]: dispatch 2024-03-30T13:40:59.551 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:59 smithi043 ceph-mon[29963]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "42da30e6-9eec-4ed5-8c73-9ce78aa8ff63"}]': finished 2024-03-30T13:40:59.551 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:59 smithi043 ceph-mon[29963]: osdmap e27: 5 total, 4 up, 5 in 2024-03-30T13:40:59.551 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:40:59 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:40:59.551 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:59 smithi043 ceph-mon[42099]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:40:59.551 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:59 smithi043 ceph-mon[42099]: from='client.? 172.21.15.113:0/993817305' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "42da30e6-9eec-4ed5-8c73-9ce78aa8ff63"}]: dispatch 2024-03-30T13:40:59.551 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:59 smithi043 ceph-mon[42099]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "42da30e6-9eec-4ed5-8c73-9ce78aa8ff63"}]: dispatch 2024-03-30T13:40:59.551 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:59 smithi043 ceph-mon[42099]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "42da30e6-9eec-4ed5-8c73-9ce78aa8ff63"}]': finished 2024-03-30T13:40:59.551 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:59 smithi043 ceph-mon[42099]: osdmap e27: 5 total, 4 up, 5 in 2024-03-30T13:40:59.552 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:40:59 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:40:59.724 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:59 smithi113 ceph-mon[34341]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:40:59.724 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:59 smithi113 ceph-mon[34341]: from='client.? 172.21.15.113:0/993817305' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "42da30e6-9eec-4ed5-8c73-9ce78aa8ff63"}]: dispatch 2024-03-30T13:40:59.724 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:59 smithi113 ceph-mon[34341]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "42da30e6-9eec-4ed5-8c73-9ce78aa8ff63"}]: dispatch 2024-03-30T13:40:59.724 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:59 smithi113 ceph-mon[34341]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "42da30e6-9eec-4ed5-8c73-9ce78aa8ff63"}]': finished 2024-03-30T13:40:59.724 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:59 smithi113 ceph-mon[34341]: osdmap e27: 5 total, 4 up, 5 in 2024-03-30T13:40:59.724 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:40:59 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:41:00.566 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:00 smithi043 ceph-mon[29963]: from='client.? 172.21.15.113:0/2104154057' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:41:00.566 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:00 smithi043 ceph-mon[42099]: from='client.? 172.21.15.113:0/2104154057' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:41:00.721 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:00 smithi113 ceph-mon[34341]: from='client.? 172.21.15.113:0/2104154057' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:41:01.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:01 smithi043 ceph-mon[29963]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:01.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:01 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:41:01.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:01 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:41:01.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:01 smithi043 ceph-mon[29963]: osdmap e28: 5 total, 4 up, 5 in 2024-03-30T13:41:01.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:01 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:41:01.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:01 smithi043 ceph-mon[42099]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:01.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:01 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:41:01.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:01 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:41:01.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:01 smithi043 ceph-mon[42099]: osdmap e28: 5 total, 4 up, 5 in 2024-03-30T13:41:01.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:01 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:41:01.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:01 smithi113 ceph-mon[34341]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:01.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:01 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:41:01.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:01 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:41:01.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:01 smithi113 ceph-mon[34341]: osdmap e28: 5 total, 4 up, 5 in 2024-03-30T13:41:01.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:01 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:41:03.406 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:03 smithi113 ceph-mon[34341]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:03.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:03 smithi043 ceph-mon[29963]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:03.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:03 smithi043 ceph-mon[42099]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:05.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:05 smithi043 ceph-mon[29963]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:05.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:05 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T13:41:05.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:05 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:05.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:05 smithi043 ceph-mon[42099]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:05.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:05 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T13:41:05.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:05 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:05.721 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:05 smithi113 ceph-mon[34341]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:05.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:05 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T13:41:05.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:05 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:06.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:06 smithi113 ceph-mon[34341]: Deploying daemon osd.4 on smithi113 2024-03-30T13:41:06.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:06 smithi043 ceph-mon[29963]: Deploying daemon osd.4 on smithi113 2024-03-30T13:41:06.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:06 smithi043 ceph-mon[42099]: Deploying daemon osd.4 on smithi113 2024-03-30T13:41:07.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:07 smithi043 ceph-mon[29963]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:07.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:07 smithi043 ceph-mon[42099]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:07.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:07 smithi113 ceph-mon[34341]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:08.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:08 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:41:08.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:08 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:41:08.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:08 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:41:08.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:08 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:41:08.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:08 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:41:08.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:08 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:41:09.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:09 smithi043 ceph-mon[29963]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:09.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:09 smithi043 ceph-mon[29963]: osdmap e29: 5 total, 4 up, 5 in 2024-03-30T13:41:09.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:41:09.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:09 smithi043 ceph-mon[42099]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:09.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:09 smithi043 ceph-mon[42099]: osdmap e29: 5 total, 4 up, 5 in 2024-03-30T13:41:09.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:09 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:41:09.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:09 smithi113 ceph-mon[34341]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:09.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:09 smithi113 ceph-mon[34341]: osdmap e29: 5 total, 4 up, 5 in 2024-03-30T13:41:09.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:09 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:41:11.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:11 smithi043 ceph-mon[29963]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:11.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:11 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:41:11.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:11 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:11.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:11 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:11.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:11 smithi043 ceph-mon[42099]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:11.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:11 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:41:11.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:11 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:11.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:11 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:11.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:11 smithi113 ceph-mon[34341]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:11.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:11 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:41:11.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:11 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:11.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:11 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:13.425 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:13 smithi113 ceph-mon[34341]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:13.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:13 smithi043 ceph-mon[29963]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:13.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:13 smithi043 ceph-mon[42099]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:13.759 INFO:teuthology.orchestra.run.smithi113.stdout:Created osd(s) 4 on host 'smithi113' 2024-03-30T13:41:14.387 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:14 smithi113 ceph-mon[34341]: from='osd.4 [v2:172.21.15.113:6800/1945833081,v1:172.21.15.113:6801/1945833081]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T13:41:14.388 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:14 smithi113 ceph-mon[34341]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T13:41:14.388 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:14 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:14.388 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:14 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:14.388 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:14 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:14.388 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:14 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:14.418 DEBUG:teuthology.orchestra.run.smithi113:osd.4> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.4.service 2024-03-30T13:41:14.421 INFO:tasks.cephadm:Deploying osd.5 on smithi113 with /dev/vg_nvme/lv_3... 2024-03-30T13:41:14.421 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-30T13:41:14.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:14 smithi043 ceph-mon[29963]: from='osd.4 [v2:172.21.15.113:6800/1945833081,v1:172.21.15.113:6801/1945833081]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T13:41:14.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:14 smithi043 ceph-mon[29963]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T13:41:14.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:14 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:14.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:14 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:14.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:14 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:14.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:14 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:14.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:14 smithi043 ceph-mon[42099]: from='osd.4 [v2:172.21.15.113:6800/1945833081,v1:172.21.15.113:6801/1945833081]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T13:41:14.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:14 smithi043 ceph-mon[42099]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T13:41:14.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:14 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:14.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:14 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:14.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:14 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:14.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:14 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:15.113 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:41:15.536 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:15 smithi113 ceph-mon[34341]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:15.536 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:15 smithi113 ceph-mon[34341]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T13:41:15.536 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:15 smithi113 ceph-mon[34341]: osdmap e30: 5 total, 4 up, 5 in 2024-03-30T13:41:15.536 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:15 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:41:15.536 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:15 smithi113 ceph-mon[34341]: from='osd.4 [v2:172.21.15.113:6800/1945833081,v1:172.21.15.113:6801/1945833081]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:41:15.536 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:15 smithi113 ceph-mon[34341]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:41:15.536 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 13:41:15 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4[39900]: 2024-03-30T13:41:15.334+0000 7eff1ae65700 -1 osd.4 0 waiting for initial osdmap 2024-03-30T13:41:15.537 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 13:41:15 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4[39900]: 2024-03-30T13:41:15.338+0000 7eff16c58700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:41:15.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:15 smithi043 ceph-mon[29963]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:15.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:15 smithi043 ceph-mon[29963]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T13:41:15.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:15 smithi043 ceph-mon[29963]: osdmap e30: 5 total, 4 up, 5 in 2024-03-30T13:41:15.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:15 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:41:15.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:15 smithi043 ceph-mon[29963]: from='osd.4 [v2:172.21.15.113:6800/1945833081,v1:172.21.15.113:6801/1945833081]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:41:15.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:15 smithi043 ceph-mon[29963]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:41:15.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:15 smithi043 ceph-mon[42099]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:15.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:15 smithi043 ceph-mon[42099]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T13:41:15.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:15 smithi043 ceph-mon[42099]: osdmap e30: 5 total, 4 up, 5 in 2024-03-30T13:41:15.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:15 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:41:15.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:15 smithi043 ceph-mon[42099]: from='osd.4 [v2:172.21.15.113:6800/1945833081,v1:172.21.15.113:6801/1945833081]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:41:15.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:15 smithi043 ceph-mon[42099]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:41:16.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:16 smithi113 ceph-mon[34341]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-30T13:41:16.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:16 smithi113 ceph-mon[34341]: osdmap e31: 5 total, 4 up, 5 in 2024-03-30T13:41:16.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:16 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:41:16.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:16 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:41:16.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:16 smithi113 ceph-mon[34341]: osd.4 [v2:172.21.15.113:6800/1945833081,v1:172.21.15.113:6801/1945833081] boot 2024-03-30T13:41:16.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:16 smithi113 ceph-mon[34341]: osdmap e32: 5 total, 5 up, 5 in 2024-03-30T13:41:16.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:16 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:41:16.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:16 smithi043 ceph-mon[29963]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-30T13:41:16.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:16 smithi043 ceph-mon[29963]: osdmap e31: 5 total, 4 up, 5 in 2024-03-30T13:41:16.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:16 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:41:16.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:16 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:41:16.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:16 smithi043 ceph-mon[29963]: osd.4 [v2:172.21.15.113:6800/1945833081,v1:172.21.15.113:6801/1945833081] boot 2024-03-30T13:41:16.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:16 smithi043 ceph-mon[29963]: osdmap e32: 5 total, 5 up, 5 in 2024-03-30T13:41:16.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:16 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:41:16.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:16 smithi043 ceph-mon[42099]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-30T13:41:16.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:16 smithi043 ceph-mon[42099]: osdmap e31: 5 total, 4 up, 5 in 2024-03-30T13:41:16.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:16 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:41:16.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:16 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:41:16.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:16 smithi043 ceph-mon[42099]: osd.4 [v2:172.21.15.113:6800/1945833081,v1:172.21.15.113:6801/1945833081] boot 2024-03-30T13:41:16.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:16 smithi043 ceph-mon[42099]: osdmap e32: 5 total, 5 up, 5 in 2024-03-30T13:41:16.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:16 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:41:17.431 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:17 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:41:17.431 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:17 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:41:17.431 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:17 smithi113 ceph-mon[34341]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:17.431 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:17 smithi113 ceph-mon[34341]: osdmap e33: 5 total, 5 up, 5 in 2024-03-30T13:41:17.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:17 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:41:17.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:17 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:41:17.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:17 smithi043 ceph-mon[29963]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:17.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:17 smithi043 ceph-mon[29963]: osdmap e33: 5 total, 5 up, 5 in 2024-03-30T13:41:17.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:17 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:41:17.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:17 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:41:17.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:17 smithi043 ceph-mon[42099]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-30T13:41:17.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:17 smithi043 ceph-mon[42099]: osdmap e33: 5 total, 5 up, 5 in 2024-03-30T13:41:17.716 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-30T13:41:17.716 INFO:teuthology.orchestra.run.smithi113.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-30T13:41:17.716 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10+0 records in 2024-03-30T13:41:17.716 INFO:teuthology.orchestra.run.smithi113.stderr:10+0 records out 2024-03-30T13:41:17.716 INFO:teuthology.orchestra.run.smithi113.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0100459 s, 1.0 GB/s 2024-03-30T13:41:17.716 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping successful for: 2024-03-30T13:41:18.353 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi113:vg_nvme/lv_3 2024-03-30T13:41:19.021 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:41:19.424 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:19 smithi113 ceph-mon[34341]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:19.424 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:19 smithi113 ceph-mon[34341]: osdmap e34: 5 total, 5 up, 5 in 2024-03-30T13:41:19.424 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:19 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:19.424 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:19 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:19.424 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:19 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:41:19.424 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:19 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:19.424 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:19 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:19.424 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:19 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:41:19.425 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:19 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:19.425 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:19 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:41:19.425 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:19 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:19.425 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:19 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:41:19.425 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:19 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:19.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[29963]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:19.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[29963]: osdmap e34: 5 total, 5 up, 5 in 2024-03-30T13:41:19.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:19.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:19.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:41:19.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:19.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:19.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:41:19.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:19.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:41:19.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:19.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:41:19.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:19.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[42099]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:19.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[42099]: osdmap e34: 5 total, 5 up, 5 in 2024-03-30T13:41:19.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:19.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:19.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:41:19.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:19.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:19.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:41:19.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:19.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:41:19.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:19.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:41:19.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:19 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:20.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:20 smithi113 ceph-mon[34341]: Detected new or changed devices on smithi113 2024-03-30T13:41:20.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:20 smithi113 ceph-mon[34341]: Adjusting osd_memory_target on smithi113 to 17088M 2024-03-30T13:41:20.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:20 smithi113 ceph-mon[34341]: osdmap e35: 5 total, 5 up, 5 in 2024-03-30T13:41:20.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:20 smithi043 ceph-mon[29963]: Detected new or changed devices on smithi113 2024-03-30T13:41:20.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:20 smithi043 ceph-mon[29963]: Adjusting osd_memory_target on smithi113 to 17088M 2024-03-30T13:41:20.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:20 smithi043 ceph-mon[29963]: osdmap e35: 5 total, 5 up, 5 in 2024-03-30T13:41:20.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:20 smithi043 ceph-mon[42099]: Detected new or changed devices on smithi113 2024-03-30T13:41:20.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:20 smithi043 ceph-mon[42099]: Adjusting osd_memory_target on smithi113 to 17088M 2024-03-30T13:41:20.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:20 smithi043 ceph-mon[42099]: osdmap e35: 5 total, 5 up, 5 in 2024-03-30T13:41:21.571 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:21 smithi113 ceph-mon[34341]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:21.571 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:21 smithi113 ceph-mon[34341]: osdmap e36: 5 total, 5 up, 5 in 2024-03-30T13:41:21.571 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:21 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:41:21.571 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:21 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:41:21.571 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:21 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:21.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:21 smithi043 ceph-mon[29963]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:21.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:21 smithi043 ceph-mon[29963]: osdmap e36: 5 total, 5 up, 5 in 2024-03-30T13:41:21.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:21 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:41:21.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:21 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:41:21.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:21 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:21.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:21 smithi043 ceph-mon[42099]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:21.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:21 smithi043 ceph-mon[42099]: osdmap e36: 5 total, 5 up, 5 in 2024-03-30T13:41:21.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:21 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:41:21.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:21 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:41:21.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:21 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:22.537 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:22 smithi113 ceph-mon[34341]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:41:22.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:22 smithi043 ceph-mon[29963]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:41:22.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:22 smithi043 ceph-mon[42099]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:41:23.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:23 smithi043 ceph-mon[29963]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-30T13:41:23.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:23 smithi043 ceph-mon[42099]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-30T13:41:23.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:23 smithi113 ceph-mon[34341]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-30T13:41:24.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:24 smithi043 ceph-mon[29963]: from='client.? 172.21.15.113:0/366337763' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99a11b81-0df7-4b1d-946a-4297ba62e263"}]: dispatch 2024-03-30T13:41:24.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:24 smithi043 ceph-mon[29963]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99a11b81-0df7-4b1d-946a-4297ba62e263"}]: dispatch 2024-03-30T13:41:24.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:24 smithi043 ceph-mon[29963]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "99a11b81-0df7-4b1d-946a-4297ba62e263"}]': finished 2024-03-30T13:41:24.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:24 smithi043 ceph-mon[29963]: osdmap e37: 6 total, 5 up, 6 in 2024-03-30T13:41:24.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:24 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:41:24.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:24 smithi043 ceph-mon[29963]: from='client.? 172.21.15.113:0/2917110334' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:41:24.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:24 smithi043 ceph-mon[42099]: from='client.? 172.21.15.113:0/366337763' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99a11b81-0df7-4b1d-946a-4297ba62e263"}]: dispatch 2024-03-30T13:41:24.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:24 smithi043 ceph-mon[42099]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99a11b81-0df7-4b1d-946a-4297ba62e263"}]: dispatch 2024-03-30T13:41:24.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:24 smithi043 ceph-mon[42099]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "99a11b81-0df7-4b1d-946a-4297ba62e263"}]': finished 2024-03-30T13:41:24.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:24 smithi043 ceph-mon[42099]: osdmap e37: 6 total, 5 up, 6 in 2024-03-30T13:41:24.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:24 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:41:24.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:24 smithi043 ceph-mon[42099]: from='client.? 172.21.15.113:0/2917110334' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:41:24.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:24 smithi113 ceph-mon[34341]: from='client.? 172.21.15.113:0/366337763' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99a11b81-0df7-4b1d-946a-4297ba62e263"}]: dispatch 2024-03-30T13:41:24.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:24 smithi113 ceph-mon[34341]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99a11b81-0df7-4b1d-946a-4297ba62e263"}]: dispatch 2024-03-30T13:41:24.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:24 smithi113 ceph-mon[34341]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "99a11b81-0df7-4b1d-946a-4297ba62e263"}]': finished 2024-03-30T13:41:24.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:24 smithi113 ceph-mon[34341]: osdmap e37: 6 total, 5 up, 6 in 2024-03-30T13:41:24.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:24 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:41:24.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:24 smithi113 ceph-mon[34341]: from='client.? 172.21.15.113:0/2917110334' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:41:25.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:25 smithi043 ceph-mon[29963]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-30T13:41:25.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:25 smithi043 ceph-mon[42099]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-30T13:41:25.721 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:25 smithi113 ceph-mon[34341]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-30T13:41:27.643 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:27 smithi113 ceph-mon[34341]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-30T13:41:27.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:27 smithi043 ceph-mon[29963]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-30T13:41:27.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:27 smithi043 ceph-mon[42099]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-30T13:41:29.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:29 smithi113 ceph-mon[34341]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T13:41:29.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:29 smithi043 ceph-mon[29963]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T13:41:29.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:29 smithi043 ceph-mon[42099]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T13:41:30.538 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:30 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T13:41:30.538 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:30 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:30.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:30 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T13:41:30.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:30 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:30.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:30 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T13:41:30.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:30 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:31.565 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:31 smithi113 ceph-mon[34341]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-30T13:41:31.565 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:31 smithi113 ceph-mon[34341]: Deploying daemon osd.5 on smithi113 2024-03-30T13:41:31.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:31 smithi043 ceph-mon[29963]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-30T13:41:31.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:31 smithi043 ceph-mon[29963]: Deploying daemon osd.5 on smithi113 2024-03-30T13:41:31.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:31 smithi043 ceph-mon[42099]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-30T13:41:31.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:31 smithi043 ceph-mon[42099]: Deploying daemon osd.5 on smithi113 2024-03-30T13:41:33.549 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:33 smithi113 ceph-mon[34341]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:33.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:33 smithi043 ceph-mon[29963]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:33.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:33 smithi043 ceph-mon[42099]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:35.565 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:35 smithi113 ceph-mon[34341]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:35.565 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:35 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:35.565 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:35 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:35.565 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:35 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:41:35.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:35 smithi043 ceph-mon[29963]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:35.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:35 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:35.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:35 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:35.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:35 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:41:35.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:35 smithi043 ceph-mon[42099]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:35.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:35 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:35.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:35 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:35.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:35 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:41:37.551 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:37 smithi113 ceph-mon[34341]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:37.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:37 smithi043 ceph-mon[29963]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:37.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:37 smithi043 ceph-mon[42099]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:38.374 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:38 smithi113 ceph-mon[34341]: from='osd.5 [v2:172.21.15.113:6808/3426360325,v1:172.21.15.113:6809/3426360325]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T13:41:38.374 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:38 smithi113 ceph-mon[34341]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T13:41:38.444 INFO:teuthology.orchestra.run.smithi113.stdout:Created osd(s) 5 on host 'smithi113' 2024-03-30T13:41:38.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:38 smithi043 ceph-mon[29963]: from='osd.5 [v2:172.21.15.113:6808/3426360325,v1:172.21.15.113:6809/3426360325]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T13:41:38.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:38 smithi043 ceph-mon[29963]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T13:41:38.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:38 smithi043 ceph-mon[42099]: from='osd.5 [v2:172.21.15.113:6808/3426360325,v1:172.21.15.113:6809/3426360325]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T13:41:38.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:38 smithi043 ceph-mon[42099]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T13:41:39.019 DEBUG:teuthology.orchestra.run.smithi113:osd.5> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.5.service 2024-03-30T13:41:39.021 INFO:tasks.cephadm:Deploying osd.6 on smithi113 with /dev/vg_nvme/lv_2... 2024-03-30T13:41:39.021 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-30T13:41:39.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:39 smithi043 ceph-mon[29963]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:39.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:39 smithi043 ceph-mon[29963]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T13:41:39.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:39 smithi043 ceph-mon[29963]: from='osd.5 [v2:172.21.15.113:6808/3426360325,v1:172.21.15.113:6809/3426360325]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:41:39.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:39 smithi043 ceph-mon[29963]: osdmap e38: 6 total, 5 up, 6 in 2024-03-30T13:41:39.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:39 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:41:39.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:39 smithi043 ceph-mon[29963]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:41:39.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:39 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:39.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:39 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:39.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:39 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:39.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:39 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:39.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:39 smithi043 ceph-mon[42099]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:39.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:39 smithi043 ceph-mon[42099]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T13:41:39.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:39 smithi043 ceph-mon[42099]: from='osd.5 [v2:172.21.15.113:6808/3426360325,v1:172.21.15.113:6809/3426360325]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:41:39.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:39 smithi043 ceph-mon[42099]: osdmap e38: 6 total, 5 up, 6 in 2024-03-30T13:41:39.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:39 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:41:39.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:39 smithi043 ceph-mon[42099]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:41:39.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:39 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:39.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:39 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:39.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:39 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:39.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:39 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:39.722 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 13:41:39 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T13:41:39.313+0000 7f626f511700 -1 osd.5 0 waiting for initial osdmap 2024-03-30T13:41:39.722 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 13:41:39 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T13:41:39.317+0000 7f626b304700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:41:39.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:39 smithi113 ceph-mon[34341]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:39.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:39 smithi113 ceph-mon[34341]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T13:41:39.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:39 smithi113 ceph-mon[34341]: from='osd.5 [v2:172.21.15.113:6808/3426360325,v1:172.21.15.113:6809/3426360325]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:41:39.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:39 smithi113 ceph-mon[34341]: osdmap e38: 6 total, 5 up, 6 in 2024-03-30T13:41:39.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:39 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:41:39.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:39 smithi113 ceph-mon[34341]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:41:39.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:39 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:39.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:39 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:39.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:39 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:39.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:39 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:39.726 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:41:40.402 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:40 smithi113 ceph-mon[34341]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-30T13:41:40.402 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:40 smithi113 ceph-mon[34341]: osdmap e39: 6 total, 5 up, 6 in 2024-03-30T13:41:40.402 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:40 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:41:40.403 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:40 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:41:40.403 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:40 smithi113 ceph-mon[34341]: osd.5 [v2:172.21.15.113:6808/3426360325,v1:172.21.15.113:6809/3426360325] boot 2024-03-30T13:41:40.403 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:40 smithi113 ceph-mon[34341]: osdmap e40: 6 total, 6 up, 6 in 2024-03-30T13:41:40.403 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:40 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:41:40.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:40 smithi043 ceph-mon[29963]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-30T13:41:40.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:40 smithi043 ceph-mon[29963]: osdmap e39: 6 total, 5 up, 6 in 2024-03-30T13:41:40.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:40 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:41:40.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:40 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:41:40.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:40 smithi043 ceph-mon[29963]: osd.5 [v2:172.21.15.113:6808/3426360325,v1:172.21.15.113:6809/3426360325] boot 2024-03-30T13:41:40.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:40 smithi043 ceph-mon[29963]: osdmap e40: 6 total, 6 up, 6 in 2024-03-30T13:41:40.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:40 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:41:40.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:40 smithi043 ceph-mon[42099]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-30T13:41:40.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:40 smithi043 ceph-mon[42099]: osdmap e39: 6 total, 5 up, 6 in 2024-03-30T13:41:40.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:40 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:41:40.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:40 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:41:40.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:40 smithi043 ceph-mon[42099]: osd.5 [v2:172.21.15.113:6808/3426360325,v1:172.21.15.113:6809/3426360325] boot 2024-03-30T13:41:40.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:40 smithi043 ceph-mon[42099]: osdmap e40: 6 total, 6 up, 6 in 2024-03-30T13:41:40.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:40 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:41:41.569 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:41 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:41:41.569 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:41 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:41:41.569 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:41 smithi113 ceph-mon[34341]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:41.569 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:41 smithi113 ceph-mon[34341]: osdmap e41: 6 total, 6 up, 6 in 2024-03-30T13:41:41.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:41 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:41:41.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:41 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:41:41.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:41 smithi043 ceph-mon[42099]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:41.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:41 smithi043 ceph-mon[42099]: osdmap e41: 6 total, 6 up, 6 in 2024-03-30T13:41:41.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:41 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:41:41.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:41 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:41:41.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:41 smithi043 ceph-mon[29963]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:41:41.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:41 smithi043 ceph-mon[29963]: osdmap e41: 6 total, 6 up, 6 in 2024-03-30T13:41:42.278 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-30T13:41:42.278 INFO:teuthology.orchestra.run.smithi113.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-30T13:41:42.278 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10+0 records in 2024-03-30T13:41:42.278 INFO:teuthology.orchestra.run.smithi113.stderr:10+0 records out 2024-03-30T13:41:42.279 INFO:teuthology.orchestra.run.smithi113.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0103338 s, 1.0 GB/s 2024-03-30T13:41:42.279 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping successful for: 2024-03-30T13:41:42.899 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi113:vg_nvme/lv_2 2024-03-30T13:41:43.418 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:43 smithi113 ceph-mon[34341]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:41:43.419 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:43 smithi113 ceph-mon[34341]: osdmap e42: 6 total, 6 up, 6 in 2024-03-30T13:41:43.571 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:41:43.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:43 smithi043 ceph-mon[29963]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:41:43.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:43 smithi043 ceph-mon[29963]: osdmap e42: 6 total, 6 up, 6 in 2024-03-30T13:41:43.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:43 smithi043 ceph-mon[42099]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:41:43.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:43 smithi043 ceph-mon[42099]: osdmap e42: 6 total, 6 up, 6 in 2024-03-30T13:41:44.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:44 smithi113 ceph-mon[34341]: osdmap e43: 6 total, 6 up, 6 in 2024-03-30T13:41:44.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:44 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:44.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:44 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:44.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:44 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:41:44.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:44 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:41:44.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:44 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:44.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:44 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:44.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:44 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:41:44.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:44 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:44.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:44 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:41:44.575 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:44 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:44.576 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:44 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:41:44.576 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:44 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:44.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[29963]: osdmap e43: 6 total, 6 up, 6 in 2024-03-30T13:41:44.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:44.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:44.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:41:44.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:41:44.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:44.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:44.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:41:44.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:44.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:41:44.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:44.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:41:44.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:44.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[42099]: osdmap e43: 6 total, 6 up, 6 in 2024-03-30T13:41:44.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:44.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:44.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:41:44.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:41:44.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:44.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:44.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:41:44.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:44.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:41:44.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:44.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:41:44.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:44 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:41:45.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:45 smithi043 ceph-mon[29963]: pgmap v140: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:41:45.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:45 smithi043 ceph-mon[29963]: Detected new or changed devices on smithi113 2024-03-30T13:41:45.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:45 smithi043 ceph-mon[29963]: Adjusting osd_memory_target on smithi113 to 8544M 2024-03-30T13:41:45.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:45 smithi043 ceph-mon[29963]: osdmap e44: 6 total, 6 up, 6 in 2024-03-30T13:41:45.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:45 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:41:45.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:45 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:41:45.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:45 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:41:45.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:45 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:41:45.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:45 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:45.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:45 smithi043 ceph-mon[42099]: pgmap v140: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:41:45.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:45 smithi043 ceph-mon[42099]: Detected new or changed devices on smithi113 2024-03-30T13:41:45.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:45 smithi043 ceph-mon[42099]: Adjusting osd_memory_target on smithi113 to 8544M 2024-03-30T13:41:45.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:45 smithi043 ceph-mon[42099]: osdmap e44: 6 total, 6 up, 6 in 2024-03-30T13:41:45.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:45 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:41:45.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:45 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:41:45.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:45 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:41:45.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:45 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:41:45.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:45 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:45.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:45 smithi113 ceph-mon[34341]: pgmap v140: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:41:45.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:45 smithi113 ceph-mon[34341]: Detected new or changed devices on smithi113 2024-03-30T13:41:45.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:45 smithi113 ceph-mon[34341]: Adjusting osd_memory_target on smithi113 to 8544M 2024-03-30T13:41:45.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:45 smithi113 ceph-mon[34341]: osdmap e44: 6 total, 6 up, 6 in 2024-03-30T13:41:45.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:45 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:41:45.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:45 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:41:45.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:41:45.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:41:45.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:46.471 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:46 smithi113 ceph-mon[34341]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:41:46.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:46 smithi113 ceph-mon[34341]: osdmap e45: 6 total, 6 up, 6 in 2024-03-30T13:41:46.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:46 smithi043 ceph-mon[29963]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:41:46.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:46 smithi043 ceph-mon[29963]: osdmap e45: 6 total, 6 up, 6 in 2024-03-30T13:41:46.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:46 smithi043 ceph-mon[42099]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:41:46.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:46 smithi043 ceph-mon[42099]: osdmap e45: 6 total, 6 up, 6 in 2024-03-30T13:41:47.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:47 smithi113 ceph-mon[34341]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-30T13:41:47.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:47 smithi043 ceph-mon[29963]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-30T13:41:47.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:47 smithi043 ceph-mon[42099]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-30T13:41:48.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:48 smithi043 ceph-mon[42099]: from='client.? 172.21.15.113:0/3898943213' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cafd0e26-78e9-4164-a43a-96ca8382fefe"}]: dispatch 2024-03-30T13:41:48.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:48 smithi043 ceph-mon[42099]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cafd0e26-78e9-4164-a43a-96ca8382fefe"}]: dispatch 2024-03-30T13:41:48.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:48 smithi043 ceph-mon[42099]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cafd0e26-78e9-4164-a43a-96ca8382fefe"}]': finished 2024-03-30T13:41:48.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:48 smithi043 ceph-mon[42099]: osdmap e46: 7 total, 6 up, 7 in 2024-03-30T13:41:48.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:48 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:41:48.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:48 smithi043 ceph-mon[29963]: from='client.? 172.21.15.113:0/3898943213' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cafd0e26-78e9-4164-a43a-96ca8382fefe"}]: dispatch 2024-03-30T13:41:48.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:48 smithi043 ceph-mon[29963]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cafd0e26-78e9-4164-a43a-96ca8382fefe"}]: dispatch 2024-03-30T13:41:48.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:48 smithi043 ceph-mon[29963]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cafd0e26-78e9-4164-a43a-96ca8382fefe"}]': finished 2024-03-30T13:41:48.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:48 smithi043 ceph-mon[29963]: osdmap e46: 7 total, 6 up, 7 in 2024-03-30T13:41:48.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:48 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:41:48.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:48 smithi113 ceph-mon[34341]: from='client.? 172.21.15.113:0/3898943213' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cafd0e26-78e9-4164-a43a-96ca8382fefe"}]: dispatch 2024-03-30T13:41:48.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:48 smithi113 ceph-mon[34341]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cafd0e26-78e9-4164-a43a-96ca8382fefe"}]: dispatch 2024-03-30T13:41:48.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:48 smithi113 ceph-mon[34341]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cafd0e26-78e9-4164-a43a-96ca8382fefe"}]': finished 2024-03-30T13:41:48.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:48 smithi113 ceph-mon[34341]: osdmap e46: 7 total, 6 up, 7 in 2024-03-30T13:41:48.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:48 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:41:49.623 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:49 smithi113 ceph-mon[34341]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-30T13:41:49.623 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:49 smithi113 ceph-mon[34341]: from='client.? 172.21.15.113:0/3601243819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:41:49.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:49 smithi043 ceph-mon[29963]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-30T13:41:49.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:49 smithi043 ceph-mon[29963]: from='client.? 172.21.15.113:0/3601243819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:41:49.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:49 smithi043 ceph-mon[42099]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-30T13:41:49.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:49 smithi043 ceph-mon[42099]: from='client.? 172.21.15.113:0/3601243819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:41:51.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:51 smithi043 ceph-mon[29963]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T13:41:51.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:51 smithi043 ceph-mon[42099]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T13:41:51.721 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:51 smithi113 ceph-mon[34341]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T13:41:53.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:53 smithi043 ceph-mon[29963]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-30T13:41:53.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:53 smithi043 ceph-mon[42099]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-30T13:41:53.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:53 smithi113 ceph-mon[34341]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-30T13:41:55.645 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:55 smithi113 ceph-mon[34341]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:41:55.645 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:55 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T13:41:55.646 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:55 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:55.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:55 smithi043 ceph-mon[42099]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:41:55.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:55 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T13:41:55.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:55 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:55.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:55 smithi043 ceph-mon[29963]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:41:55.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:55 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T13:41:55.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:55 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:41:56.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:56 smithi113 ceph-mon[34341]: Deploying daemon osd.6 on smithi113 2024-03-30T13:41:56.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:56 smithi043 ceph-mon[42099]: Deploying daemon osd.6 on smithi113 2024-03-30T13:41:56.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:56 smithi043 ceph-mon[29963]: Deploying daemon osd.6 on smithi113 2024-03-30T13:41:57.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:57 smithi043 ceph-mon[42099]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:41:57.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:57 smithi043 ceph-mon[42099]: 1.0 scrub starts 2024-03-30T13:41:57.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:57 smithi043 ceph-mon[42099]: 1.0 scrub ok 2024-03-30T13:41:57.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:57 smithi043 ceph-mon[29963]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:41:57.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:57 smithi043 ceph-mon[29963]: 1.0 scrub starts 2024-03-30T13:41:57.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:57 smithi043 ceph-mon[29963]: 1.0 scrub ok 2024-03-30T13:41:57.651 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:57 smithi113 ceph-mon[34341]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:41:57.651 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:57 smithi113 ceph-mon[34341]: 1.0 scrub starts 2024-03-30T13:41:57.651 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:57 smithi113 ceph-mon[34341]: 1.0 scrub ok 2024-03-30T13:41:59.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:41:59 smithi043 ceph-mon[42099]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:41:59.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:41:59 smithi043 ceph-mon[29963]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:41:59.672 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:41:59 smithi113 ceph-mon[34341]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:42:01.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:00 smithi043 ceph-mon[42099]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:42:01.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:00 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:01.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:00 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:01.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:00 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:01.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:00 smithi043 ceph-mon[29963]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:42:01.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:00 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:01.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:00 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:01.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:00 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:01.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:00 smithi113 ceph-mon[34341]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:42:01.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:00 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:01.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:00 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:01.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:00 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:03.188 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:02 smithi113 ceph-mon[34341]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:42:03.233 INFO:teuthology.orchestra.run.smithi113.stdout:Created osd(s) 6 on host 'smithi113' 2024-03-30T13:42:03.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:02 smithi043 ceph-mon[29963]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:42:03.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:02 smithi043 ceph-mon[42099]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:42:03.752 DEBUG:teuthology.orchestra.run.smithi113:osd.6> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.6.service 2024-03-30T13:42:03.755 INFO:tasks.cephadm:Deploying osd.7 on smithi113 with /dev/vg_nvme/lv_1... 2024-03-30T13:42:03.755 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-30T13:42:04.040 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:42:04.151 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:03 smithi113 ceph-mon[34341]: from='osd.6 [v2:172.21.15.113:6816/1527017698,v1:172.21.15.113:6817/1527017698]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T13:42:04.151 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:03 smithi113 ceph-mon[34341]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T13:42:04.151 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:03 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:04.151 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:03 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:04.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:03 smithi043 ceph-mon[29963]: from='osd.6 [v2:172.21.15.113:6816/1527017698,v1:172.21.15.113:6817/1527017698]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T13:42:04.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:03 smithi043 ceph-mon[29963]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T13:42:04.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:03 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:04.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:03 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:04.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:03 smithi043 ceph-mon[42099]: from='osd.6 [v2:172.21.15.113:6816/1527017698,v1:172.21.15.113:6817/1527017698]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T13:42:04.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:03 smithi043 ceph-mon[42099]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T13:42:04.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:03 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:04.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:03 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:05.156 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:04 smithi113 ceph-mon[34341]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:42:05.156 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:04 smithi113 ceph-mon[34341]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-30T13:42:05.156 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:04 smithi113 ceph-mon[34341]: osdmap e47: 7 total, 6 up, 7 in 2024-03-30T13:42:05.156 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:04 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:42:05.156 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:04 smithi113 ceph-mon[34341]: from='osd.6 [v2:172.21.15.113:6816/1527017698,v1:172.21.15.113:6817/1527017698]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:42:05.157 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:04 smithi113 ceph-mon[34341]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:42:05.157 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:04 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:05.157 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:04 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:05.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:04 smithi043 ceph-mon[29963]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:42:05.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:04 smithi043 ceph-mon[29963]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-30T13:42:05.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:04 smithi043 ceph-mon[29963]: osdmap e47: 7 total, 6 up, 7 in 2024-03-30T13:42:05.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:04 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:42:05.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:04 smithi043 ceph-mon[29963]: from='osd.6 [v2:172.21.15.113:6816/1527017698,v1:172.21.15.113:6817/1527017698]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:42:05.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:04 smithi043 ceph-mon[29963]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:42:05.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:04 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:05.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:04 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:05.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:04 smithi043 ceph-mon[42099]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:42:05.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:04 smithi043 ceph-mon[42099]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-30T13:42:05.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:04 smithi043 ceph-mon[42099]: osdmap e47: 7 total, 6 up, 7 in 2024-03-30T13:42:05.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:04 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:42:05.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:04 smithi043 ceph-mon[42099]: from='osd.6 [v2:172.21.15.113:6816/1527017698,v1:172.21.15.113:6817/1527017698]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:42:05.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:04 smithi043 ceph-mon[42099]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:42:05.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:04 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:05.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:04 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:05.453 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 13:42:05 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T13:42:05.197+0000 7f5887ff0700 -1 osd.6 0 waiting for initial osdmap 2024-03-30T13:42:05.453 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 13:42:05 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T13:42:05.203+0000 7f587fddb700 -1 osd.6 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:42:06.065 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-30T13:42:06.065 INFO:teuthology.orchestra.run.smithi113.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-30T13:42:06.065 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10+0 records in 2024-03-30T13:42:06.066 INFO:teuthology.orchestra.run.smithi113.stderr:10+0 records out 2024-03-30T13:42:06.066 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0103734 s, 1.0 GB/s 2024-03-30T13:42:06.066 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping successful for: 2024-03-30T13:42:06.090 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:05 smithi113 ceph-mon[34341]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-30T13:42:06.090 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:05 smithi113 ceph-mon[34341]: osdmap e48: 7 total, 6 up, 7 in 2024-03-30T13:42:06.090 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:05 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:42:06.090 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:05 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:42:06.090 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:05 smithi113 ceph-mon[34341]: from='osd.6 ' entity='osd.6' 2024-03-30T13:42:06.090 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:05 smithi113 ceph-mon[34341]: osd.6 [v2:172.21.15.113:6816/1527017698,v1:172.21.15.113:6817/1527017698] boot 2024-03-30T13:42:06.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:05 smithi043 ceph-mon[29963]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-30T13:42:06.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:05 smithi043 ceph-mon[29963]: osdmap e48: 7 total, 6 up, 7 in 2024-03-30T13:42:06.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:05 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:42:06.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:05 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:42:06.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:05 smithi043 ceph-mon[29963]: from='osd.6 ' entity='osd.6' 2024-03-30T13:42:06.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:05 smithi043 ceph-mon[29963]: osd.6 [v2:172.21.15.113:6816/1527017698,v1:172.21.15.113:6817/1527017698] boot 2024-03-30T13:42:06.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:05 smithi043 ceph-mon[42099]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-30T13:42:06.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:05 smithi043 ceph-mon[42099]: osdmap e48: 7 total, 6 up, 7 in 2024-03-30T13:42:06.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:05 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:42:06.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:05 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:42:06.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:05 smithi043 ceph-mon[42099]: from='osd.6 ' entity='osd.6' 2024-03-30T13:42:06.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:05 smithi043 ceph-mon[42099]: osd.6 [v2:172.21.15.113:6816/1527017698,v1:172.21.15.113:6817/1527017698] boot 2024-03-30T13:42:06.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:06 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:42:06.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:06 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:42:06.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:06 smithi113 ceph-mon[34341]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:42:06.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:06 smithi113 ceph-mon[34341]: osdmap e49: 7 total, 7 up, 7 in 2024-03-30T13:42:06.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:06 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:42:06.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:06 smithi113 ceph-mon[34341]: osdmap e50: 7 total, 7 up, 7 in 2024-03-30T13:42:07.311 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi113:vg_nvme/lv_1 2024-03-30T13:42:07.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:06 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:42:07.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:06 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:42:07.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:06 smithi043 ceph-mon[29963]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:42:07.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:06 smithi043 ceph-mon[29963]: osdmap e49: 7 total, 7 up, 7 in 2024-03-30T13:42:07.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:06 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:42:07.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:06 smithi043 ceph-mon[29963]: osdmap e50: 7 total, 7 up, 7 in 2024-03-30T13:42:07.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:06 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:42:07.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:06 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:42:07.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:06 smithi043 ceph-mon[42099]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:42:07.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:06 smithi043 ceph-mon[42099]: osdmap e49: 7 total, 7 up, 7 in 2024-03-30T13:42:07.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:06 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:42:07.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:06 smithi043 ceph-mon[42099]: osdmap e50: 7 total, 7 up, 7 in 2024-03-30T13:42:07.728 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:42:08.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:08 smithi113 ceph-mon[34341]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 429 MiB used, 625 GiB / 626 GiB avail 2024-03-30T13:42:08.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:08 smithi113 ceph-mon[34341]: osdmap e51: 7 total, 7 up, 7 in 2024-03-30T13:42:09.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:08 smithi043 ceph-mon[29963]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 429 MiB used, 625 GiB / 626 GiB avail 2024-03-30T13:42:09.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:08 smithi043 ceph-mon[29963]: osdmap e51: 7 total, 7 up, 7 in 2024-03-30T13:42:09.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:08 smithi043 ceph-mon[42099]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 429 MiB used, 625 GiB / 626 GiB avail 2024-03-30T13:42:09.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:08 smithi043 ceph-mon[42099]: osdmap e51: 7 total, 7 up, 7 in 2024-03-30T13:42:09.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:09 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:42:09.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:09 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:42:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:09 smithi113 ceph-mon[34341]: osdmap e52: 7 total, 7 up, 7 in 2024-03-30T13:42:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:09 smithi113 ceph-mon[34341]: Detected new or changed devices on smithi113 2024-03-30T13:42:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:09 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:09 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:09 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:09 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:09 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:09 smithi113 ceph-mon[34341]: Adjusting osd_memory_target on smithi113 to 5696M 2024-03-30T13:42:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:09 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:09 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:09 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:42:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:09 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:09 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:09.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:09 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:09.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:09 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:42:09.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:09 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:10.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:42:10.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:42:10.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[29963]: osdmap e52: 7 total, 7 up, 7 in 2024-03-30T13:42:10.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[29963]: Detected new or changed devices on smithi113 2024-03-30T13:42:10.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:10.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:10.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:10.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:10.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:10.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[29963]: Adjusting osd_memory_target on smithi113 to 5696M 2024-03-30T13:42:10.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:10.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:10.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:42:10.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:10.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:10.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:10.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:42:10.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:10.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:42:10.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:42:10.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[42099]: osdmap e52: 7 total, 7 up, 7 in 2024-03-30T13:42:10.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[42099]: Detected new or changed devices on smithi113 2024-03-30T13:42:10.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:10.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:10.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:10.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:10.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:10.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[42099]: Adjusting osd_memory_target on smithi113 to 5696M 2024-03-30T13:42:10.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:10.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:10.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:42:10.402 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:10.402 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:10.402 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:10.402 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:42:10.402 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:09 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:11.024 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:10 smithi113 ceph-mon[34341]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 429 MiB used, 625 GiB / 626 GiB avail 2024-03-30T13:42:11.024 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:10 smithi113 ceph-mon[34341]: osdmap e53: 7 total, 7 up, 7 in 2024-03-30T13:42:11.024 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:10 smithi113 ceph-mon[34341]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:42:11.024 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:10 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:42:11.025 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:10 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:42:11.025 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:10 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:11.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:10 smithi043 ceph-mon[29963]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 429 MiB used, 625 GiB / 626 GiB avail 2024-03-30T13:42:11.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:10 smithi043 ceph-mon[29963]: osdmap e53: 7 total, 7 up, 7 in 2024-03-30T13:42:11.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:10 smithi043 ceph-mon[29963]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:42:11.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:10 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:42:11.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:10 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:42:11.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:10 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:11.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:10 smithi043 ceph-mon[42099]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 429 MiB used, 625 GiB / 626 GiB avail 2024-03-30T13:42:11.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:10 smithi043 ceph-mon[42099]: osdmap e53: 7 total, 7 up, 7 in 2024-03-30T13:42:11.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:10 smithi043 ceph-mon[42099]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:42:11.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:10 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:42:11.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:10 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:42:11.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:10 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:13.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:12 smithi113 ceph-mon[34341]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-03-30T13:42:13.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:12 smithi113 ceph-mon[34341]: from='client.? 172.21.15.113:0/2876096732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9d59662-9132-48f4-9e95-842a3cfbd79f"}]: dispatch 2024-03-30T13:42:13.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:12 smithi113 ceph-mon[34341]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9d59662-9132-48f4-9e95-842a3cfbd79f"}]: dispatch 2024-03-30T13:42:13.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:12 smithi113 ceph-mon[34341]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9d59662-9132-48f4-9e95-842a3cfbd79f"}]': finished 2024-03-30T13:42:13.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:12 smithi113 ceph-mon[34341]: osdmap e54: 8 total, 7 up, 8 in 2024-03-30T13:42:13.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:12 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:13.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:12 smithi043 ceph-mon[29963]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-03-30T13:42:13.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:12 smithi043 ceph-mon[29963]: from='client.? 172.21.15.113:0/2876096732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9d59662-9132-48f4-9e95-842a3cfbd79f"}]: dispatch 2024-03-30T13:42:13.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:12 smithi043 ceph-mon[29963]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9d59662-9132-48f4-9e95-842a3cfbd79f"}]: dispatch 2024-03-30T13:42:13.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:12 smithi043 ceph-mon[29963]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9d59662-9132-48f4-9e95-842a3cfbd79f"}]': finished 2024-03-30T13:42:13.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:12 smithi043 ceph-mon[29963]: osdmap e54: 8 total, 7 up, 8 in 2024-03-30T13:42:13.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:12 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:13.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:12 smithi043 ceph-mon[42099]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-03-30T13:42:13.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:12 smithi043 ceph-mon[42099]: from='client.? 172.21.15.113:0/2876096732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9d59662-9132-48f4-9e95-842a3cfbd79f"}]: dispatch 2024-03-30T13:42:13.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:12 smithi043 ceph-mon[42099]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9d59662-9132-48f4-9e95-842a3cfbd79f"}]: dispatch 2024-03-30T13:42:13.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:12 smithi043 ceph-mon[42099]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9d59662-9132-48f4-9e95-842a3cfbd79f"}]': finished 2024-03-30T13:42:13.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:12 smithi043 ceph-mon[42099]: osdmap e54: 8 total, 7 up, 8 in 2024-03-30T13:42:13.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:12 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:14.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:13 smithi113 ceph-mon[34341]: from='client.? 172.21.15.113:0/1354775568' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:42:14.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:13 smithi043 ceph-mon[29963]: from='client.? 172.21.15.113:0/1354775568' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:42:14.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:13 smithi043 ceph-mon[42099]: from='client.? 172.21.15.113:0/1354775568' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:42:15.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:14 smithi113 ceph-mon[34341]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-30T13:42:15.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:14 smithi043 ceph-mon[29963]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-30T13:42:15.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:14 smithi043 ceph-mon[42099]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-30T13:42:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:16 smithi113 ceph-mon[34341]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-30T13:42:17.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:16 smithi043 ceph-mon[29963]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-30T13:42:17.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:16 smithi043 ceph-mon[42099]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-30T13:42:18.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:18 smithi113 ceph-mon[34341]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T13:42:19.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:18 smithi043 ceph-mon[29963]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T13:42:19.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:18 smithi043 ceph-mon[42099]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T13:42:20.208 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:19 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T13:42:20.209 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:19 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:20.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:19 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T13:42:20.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:19 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:20.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:19 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T13:42:20.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:19 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:21.164 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:20 smithi113 ceph-mon[34341]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-30T13:42:21.164 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:20 smithi113 ceph-mon[34341]: Deploying daemon osd.7 on smithi113 2024-03-30T13:42:21.164 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:20 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:42:21.164 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:20 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:42:21.164 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:20 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:42:21.164 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:20 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:42:21.164 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:20 smithi113 ceph-mon[34341]: osdmap e55: 8 total, 7 up, 8 in 2024-03-30T13:42:21.165 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:20 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:21.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:20 smithi043 ceph-mon[29963]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-30T13:42:21.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:20 smithi043 ceph-mon[29963]: Deploying daemon osd.7 on smithi113 2024-03-30T13:42:21.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:20 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:42:21.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:20 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:42:21.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:20 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:42:21.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:20 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:42:21.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:20 smithi043 ceph-mon[29963]: osdmap e55: 8 total, 7 up, 8 in 2024-03-30T13:42:21.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:20 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:21.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:20 smithi043 ceph-mon[42099]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-30T13:42:21.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:20 smithi043 ceph-mon[42099]: Deploying daemon osd.7 on smithi113 2024-03-30T13:42:21.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:20 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:42:21.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:20 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:42:21.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:20 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:42:21.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:20 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:42:21.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:20 smithi043 ceph-mon[42099]: osdmap e55: 8 total, 7 up, 8 in 2024-03-30T13:42:21.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:20 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:23.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:22 smithi113 ceph-mon[34341]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:42:23.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:22 smithi043 ceph-mon[29963]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:42:23.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:22 smithi043 ceph-mon[42099]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:42:25.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:24 smithi113 ceph-mon[34341]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:42:25.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:24 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:25.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:24 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:25.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:24 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:25.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:24 smithi043 ceph-mon[29963]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:42:25.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:24 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:25.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:24 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:25.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:24 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:25.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:24 smithi043 ceph-mon[42099]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:42:25.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:24 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:25.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:24 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:25.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:24 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:27.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:26 smithi113 ceph-mon[34341]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:42:27.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:26 smithi043 ceph-mon[29963]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:42:27.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:26 smithi043 ceph-mon[42099]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:42:28.517 INFO:teuthology.orchestra.run.smithi113.stdout:Created osd(s) 7 on host 'smithi113' 2024-03-30T13:42:28.869 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:28 smithi113 ceph-mon[34341]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:42:28.870 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:28 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:28.870 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:28 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:28.870 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:28 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:28.870 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:28 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:42:28.870 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:28 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:28.870 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:28 smithi113 ceph-mon[34341]: from='osd.7 [v2:172.21.15.113:6824/3182476015,v1:172.21.15.113:6825/3182476015]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T13:42:28.870 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:28 smithi113 ceph-mon[34341]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T13:42:28.870 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:28 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:28.870 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:28 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:28.870 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:28 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:29.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[29963]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:42:29.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:29.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:29.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:29.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:42:29.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:29.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[29963]: from='osd.7 [v2:172.21.15.113:6824/3182476015,v1:172.21.15.113:6825/3182476015]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T13:42:29.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[29963]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T13:42:29.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:29.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:29.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:29.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[42099]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:42:29.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:29.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:29.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:29.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:42:29.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:29.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[42099]: from='osd.7 [v2:172.21.15.113:6824/3182476015,v1:172.21.15.113:6825/3182476015]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T13:42:29.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[42099]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T13:42:29.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:29.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:29.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:28 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:29.163 DEBUG:teuthology.orchestra.run.smithi113:osd.7> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.7.service 2024-03-30T13:42:29.165 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-30T13:42:29.165 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd stat -f json 2024-03-30T13:42:29.420 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:42:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:29 smithi113 ceph-mon[34341]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-30T13:42:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:29 smithi113 ceph-mon[34341]: from='osd.7 [v2:172.21.15.113:6824/3182476015,v1:172.21.15.113:6825/3182476015]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:42:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:29 smithi113 ceph-mon[34341]: osdmap e56: 8 total, 7 up, 8 in 2024-03-30T13:42:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:29 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:29 smithi113 ceph-mon[34341]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:42:29.814 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:29 smithi043 ceph-mon[29963]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-30T13:42:29.814 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:29 smithi043 ceph-mon[29963]: from='osd.7 [v2:172.21.15.113:6824/3182476015,v1:172.21.15.113:6825/3182476015]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:42:29.814 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:29 smithi043 ceph-mon[29963]: osdmap e56: 8 total, 7 up, 8 in 2024-03-30T13:42:29.814 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:29 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:29.814 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:29 smithi043 ceph-mon[29963]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:42:29.814 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:29 smithi043 ceph-mon[42099]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-30T13:42:29.814 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:29 smithi043 ceph-mon[42099]: from='osd.7 [v2:172.21.15.113:6824/3182476015,v1:172.21.15.113:6825/3182476015]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:42:29.814 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:29 smithi043 ceph-mon[42099]: osdmap e56: 8 total, 7 up, 8 in 2024-03-30T13:42:29.815 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:29 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:29.815 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:29 smithi043 ceph-mon[42099]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-30T13:42:30.222 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 13:42:29 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T13:42:29.964+0000 7ffb7ef39700 -1 osd.7 0 waiting for initial osdmap 2024-03-30T13:42:30.222 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 13:42:29 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T13:42:29.971+0000 7ffb76523700 -1 osd.7 57 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:42:30.788 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:30 smithi113 ceph-mon[34341]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:42:30.788 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:30 smithi113 ceph-mon[34341]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-30T13:42:30.788 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:30 smithi113 ceph-mon[34341]: osdmap e57: 8 total, 7 up, 8 in 2024-03-30T13:42:30.788 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:30 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:30.788 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:30 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:30.789 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:30 smithi113 ceph-mon[34341]: from='osd.7 ' entity='osd.7' 2024-03-30T13:42:30.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:30 smithi043 ceph-mon[29963]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:42:30.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:30 smithi043 ceph-mon[29963]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-30T13:42:30.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:30 smithi043 ceph-mon[29963]: osdmap e57: 8 total, 7 up, 8 in 2024-03-30T13:42:30.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:30 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:30.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:30 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:30.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:30 smithi043 ceph-mon[29963]: from='osd.7 ' entity='osd.7' 2024-03-30T13:42:30.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:30 smithi043 ceph-mon[42099]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:42:30.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:30 smithi043 ceph-mon[42099]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-30T13:42:30.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:30 smithi043 ceph-mon[42099]: osdmap e57: 8 total, 7 up, 8 in 2024-03-30T13:42:30.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:30 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:30.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:30 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:30.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:30 smithi043 ceph-mon[42099]: from='osd.7 ' entity='osd.7' 2024-03-30T13:42:31.407 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:42:31.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:31 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:42:31.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:31 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:42:31.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:31 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:31.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:31 smithi113 ceph-mon[34341]: osd.7 [v2:172.21.15.113:6824/3182476015,v1:172.21.15.113:6825/3182476015] boot 2024-03-30T13:42:31.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:31 smithi113 ceph-mon[34341]: osdmap e58: 8 total, 8 up, 8 in 2024-03-30T13:42:31.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:31 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:31.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:31 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3365395715' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T13:42:32.049 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:31 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:42:32.049 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:31 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:42:32.049 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:31 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:32.049 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:31 smithi043 ceph-mon[29963]: osd.7 [v2:172.21.15.113:6824/3182476015,v1:172.21.15.113:6825/3182476015] boot 2024-03-30T13:42:32.049 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:31 smithi043 ceph-mon[29963]: osdmap e58: 8 total, 8 up, 8 in 2024-03-30T13:42:32.050 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:31 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:32.050 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:31 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3365395715' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T13:42:32.050 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:31 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:42:32.050 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:31 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:42:32.050 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:31 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:32.050 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:31 smithi043 ceph-mon[42099]: osd.7 [v2:172.21.15.113:6824/3182476015,v1:172.21.15.113:6825/3182476015] boot 2024-03-30T13:42:32.050 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:31 smithi043 ceph-mon[42099]: osdmap e58: 8 total, 8 up, 8 in 2024-03-30T13:42:32.050 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:31 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:42:32.050 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:31 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3365395715' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T13:42:32.077 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":58,"num_osds":8,"num_up_osds":8,"osd_up_since":1711806150,"num_in_osds":8,"osd_in_since":1711806132,"num_remapped_pgs":1} 2024-03-30T13:42:32.078 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T13:42:32.321 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:42:33.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[29963]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:42:33.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:42:33.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:42:33.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[29963]: osdmap e59: 8 total, 8 up, 8 in 2024-03-30T13:42:33.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[29963]: Detected new or changed devices on smithi113 2024-03-30T13:42:33.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:33.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:33.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:33.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:33.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:33.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:33.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[29963]: Adjusting osd_memory_target on smithi113 to 4272M 2024-03-30T13:42:33.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:33.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:33.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:42:33.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:33.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[42099]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:42:33.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:42:33.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:42:33.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[42099]: osdmap e59: 8 total, 8 up, 8 in 2024-03-30T13:42:33.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[42099]: Detected new or changed devices on smithi113 2024-03-30T13:42:33.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:33.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:33.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:33.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:33.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:33.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:33.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[42099]: Adjusting osd_memory_target on smithi113 to 4272M 2024-03-30T13:42:33.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:33.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:33.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:42:33.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:32 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:33.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:32 smithi113 ceph-mon[34341]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:42:33.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:32 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:42:33.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:32 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:42:33.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:32 smithi113 ceph-mon[34341]: osdmap e59: 8 total, 8 up, 8 in 2024-03-30T13:42:33.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:32 smithi113 ceph-mon[34341]: Detected new or changed devices on smithi113 2024-03-30T13:42:33.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:33.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:33.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:33.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:33.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:33.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:42:33.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:32 smithi113 ceph-mon[34341]: Adjusting osd_memory_target on smithi113 to 4272M 2024-03-30T13:42:33.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:33.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:33.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:42:33.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:32 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:34.076 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:33 smithi043 ceph-mon[29963]: osdmap e60: 8 total, 8 up, 8 in 2024-03-30T13:42:34.077 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:33 smithi043 ceph-mon[42099]: osdmap e60: 8 total, 8 up, 8 in 2024-03-30T13:42:34.292 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:42:34.293 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":61,"fsid":"7ee932fa-ee9a-11ee-b647-cb9ed24678a4","created":"2024-03-30T13:37:07.346020+0000","modified":"2024-03-30T13:42:33.978358+0000","last_up_change":"2024-03-30T13:42:30.966931+0000","last_in_change":"2024-03-30T13:42:12.831084+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T13:40:27.658979+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94ea083e-6a27-41db-83af-2f7d3cae1956","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1144592572},{"type":"v1","addr":"172.21.15.43:6803","nonce":1144592572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1144592572},{"type":"v1","addr":"172.21.15.43:6805","nonce":1144592572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":1144592572},{"type":"v1","addr":"172.21.15.43:6809","nonce":1144592572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1144592572},{"type":"v1","addr":"172.21.15.43:6807","nonce":1144592572}]},"public_addr":"172.21.15.43:6803/1144592572","cluster_addr":"172.21.15.43:6805/1144592572","heartbeat_back_addr":"172.21.15.43:6809/1144592572","heartbeat_front_addr":"172.21.15.43:6807/1144592572","state":["exists","up"]},{"osd":1,"uuid":"8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":3901165928},{"type":"v1","addr":"172.21.15.43:6811","nonce":3901165928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":3901165928},{"type":"v1","addr":"172.21.15.43:6813","nonce":3901165928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":3901165928},{"type":"v1","addr":"172.21.15.43:6817","nonce":3901165928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":3901165928},{"type":"v1","addr":"172.21.15.43:6815","nonce":3901165928}]},"public_addr":"172.21.15.43:6811/3901165928","cluster_addr":"172.21.15.43:6813/3901165928","heartbeat_back_addr":"172.21.15.43:6817/3901165928","heartbeat_front_addr":"172.21.15.43:6815/3901165928","state":["exists","up"]},{"osd":2,"uuid":"6db6782e-5889-443e-bc6c-df47a1f93bc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":3415037523},{"type":"v1","addr":"172.21.15.43:6819","nonce":3415037523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":3415037523},{"type":"v1","addr":"172.21.15.43:6821","nonce":3415037523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":3415037523},{"type":"v1","addr":"172.21.15.43:6825","nonce":3415037523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":3415037523},{"type":"v1","addr":"172.21.15.43:6823","nonce":3415037523}]},"public_addr":"172.21.15.43:6819/3415037523","cluster_addr":"172.21.15.43:6821/3415037523","heartbeat_back_addr":"172.21.15.43:6825/3415037523","heartbeat_front_addr":"172.21.15.43:6823/3415037523","state":["exists","up"]},{"osd":3,"uuid":"7048ff46-2cba-4d18-9835-95ecbddb2bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2933589376},{"type":"v1","addr":"172.21.15.43:6827","nonce":2933589376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2933589376},{"type":"v1","addr":"172.21.15.43:6829","nonce":2933589376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2933589376},{"type":"v1","addr":"172.21.15.43:6833","nonce":2933589376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2933589376},{"type":"v1","addr":"172.21.15.43:6831","nonce":2933589376}]},"public_addr":"172.21.15.43:6827/2933589376","cluster_addr":"172.21.15.43:6829/2933589376","heartbeat_back_addr":"172.21.15.43:6833/2933589376","heartbeat_front_addr":"172.21.15.43:6831/2933589376","state":["exists","up"]},{"osd":4,"uuid":"42da30e6-9eec-4ed5-8c73-9ce78aa8ff63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1945833081},{"type":"v1","addr":"172.21.15.113:6801","nonce":1945833081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1945833081},{"type":"v1","addr":"172.21.15.113:6803","nonce":1945833081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1945833081},{"type":"v1","addr":"172.21.15.113:6807","nonce":1945833081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1945833081},{"type":"v1","addr":"172.21.15.113:6805","nonce":1945833081}]},"public_addr":"172.21.15.113:6801/1945833081","cluster_addr":"172.21.15.113:6803/1945833081","heartbeat_back_addr":"172.21.15.113:6807/1945833081","heartbeat_front_addr":"172.21.15.113:6805/1945833081","state":["exists","up"]},{"osd":5,"uuid":"99a11b81-0df7-4b1d-946a-4297ba62e263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3426360325},{"type":"v1","addr":"172.21.15.113:6809","nonce":3426360325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3426360325},{"type":"v1","addr":"172.21.15.113:6811","nonce":3426360325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3426360325},{"type":"v1","addr":"172.21.15.113:6815","nonce":3426360325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3426360325},{"type":"v1","addr":"172.21.15.113:6813","nonce":3426360325}]},"public_addr":"172.21.15.113:6809/3426360325","cluster_addr":"172.21.15.113:6811/3426360325","heartbeat_back_addr":"172.21.15.113:6815/3426360325","heartbeat_front_addr":"172.21.15.113:6813/3426360325","state":["exists","up"]},{"osd":6,"uuid":"cafd0e26-78e9-4164-a43a-96ca8382fefe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1527017698},{"type":"v1","addr":"172.21.15.113:6817","nonce":1527017698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1527017698},{"type":"v1","addr":"172.21.15.113:6819","nonce":1527017698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1527017698},{"type":"v1","addr":"172.21.15.113:6823","nonce":1527017698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1527017698},{"type":"v1","addr":"172.21.15.113:6821","nonce":1527017698}]},"public_addr":"172.21.15.113:6817/1527017698","cluster_addr":"172.21.15.113:6819/1527017698","heartbeat_back_addr":"172.21.15.113:6823/1527017698","heartbeat_front_addr":"172.21.15.113:6821/1527017698","state":["exists","up"]},{"osd":7,"uuid":"e9d59662-9132-48f4-9e95-842a3cfbd79f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":3182476015},{"type":"v1","addr":"172.21.15.113:6825","nonce":3182476015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":3182476015},{"type":"v1","addr":"172.21.15.113:6827","nonce":3182476015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":3182476015},{"type":"v1","addr":"172.21.15.113:6831","nonce":3182476015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":3182476015},{"type":"v1","addr":"172.21.15.113:6829","nonce":3182476015}]},"public_addr":"172.21.15.113:6825/3182476015","cluster_addr":"172.21.15.113:6827/3182476015","heartbeat_back_addr":"172.21.15.113:6831/3182476015","heartbeat_front_addr":"172.21.15.113:6829/3182476015","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:42:19.457967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:42:31.948677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:41:45.189417+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:42:08.482028+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:42:19.937695+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:41:39.228520+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:42:04.066936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:42:28.849417+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/564868778":"2024-03-31T13:38:09.321740+0000","172.21.15.43:6801/1339921781":"2024-03-31T13:38:09.321740+0000","172.21.15.43:6800/1339921781":"2024-03-31T13:38:09.321740+0000","172.21.15.43:0/886631486":"2024-03-31T13:37:41.733515+0000","172.21.15.43:0/1206218784":"2024-03-31T13:37:41.733515+0000","172.21.15.43:6800/3411723337":"2024-03-31T13:37:41.733515+0000","172.21.15.43:0/2618817647":"2024-03-31T13:38:09.321740+0000","172.21.15.43:0/3656823679":"2024-03-31T13:37:41.733515+0000","172.21.15.43:0/2794494528":"2024-03-31T13:38:09.321740+0000","172.21.15.43:6801/3411723337":"2024-03-31T13:37:41.733515+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:42:34.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:33 smithi113 ceph-mon[34341]: osdmap e60: 8 total, 8 up, 8 in 2024-03-30T13:42:35.045 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T13:40:27.658979+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-30T13:42:35.046 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-03-30T13:42:35.290 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:42:35.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:34 smithi043 ceph-mon[42099]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:42:35.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:34 smithi043 ceph-mon[42099]: osdmap e61: 8 total, 8 up, 8 in 2024-03-30T13:42:35.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:34 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/663218831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T13:42:35.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:34 smithi043 ceph-mon[29963]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:42:35.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:34 smithi043 ceph-mon[29963]: osdmap e61: 8 total, 8 up, 8 in 2024-03-30T13:42:35.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:34 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/663218831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T13:42:35.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:34 smithi113 ceph-mon[34341]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:42:35.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:34 smithi113 ceph-mon[34341]: osdmap e61: 8 total, 8 up, 8 in 2024-03-30T13:42:35.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:34 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/663218831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T13:42:36.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:35 smithi043 ceph-mon[29963]: osdmap e62: 8 total, 8 up, 8 in 2024-03-30T13:42:36.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:35 smithi043 ceph-mon[42099]: osdmap e62: 8 total, 8 up, 8 in 2024-03-30T13:42:36.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:35 smithi113 ceph-mon[34341]: osdmap e62: 8 total, 8 up, 8 in 2024-03-30T13:42:37.075 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:36 smithi043 ceph-mon[29963]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:42:37.076 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:36 smithi043 ceph-mon[42099]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:42:37.405 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-03-30T13:42:37.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:36 smithi113 ceph-mon[34341]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:42:38.105 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi043 2024-03-30T13:42:38.105 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph orch apply rgw foo.a --placement '1;smithi043=foo.a' 2024-03-30T13:42:38.354 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:42:38.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:38 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1109815659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T13:42:38.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:38 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1109815659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T13:42:38.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:38 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1109815659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T13:42:39.398 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:39 smithi113 ceph-mon[34341]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-30T13:42:39.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:39 smithi043 ceph-mon[29963]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-30T13:42:39.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:39 smithi043 ceph-mon[42099]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-30T13:42:40.279 INFO:teuthology.orchestra.run.smithi113.stdout:Scheduled rgw.foo.a update... 2024-03-30T13:42:40.880 DEBUG:teuthology.orchestra.run.smithi043:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@rgw.foo.a.service 2024-03-30T13:42:40.882 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi113 2024-03-30T13:42:40.883 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool create datapool 3 3 replicated 2024-03-30T13:42:41.118 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:42:41.491 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:41 smithi113 ceph-mon[34341]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-30T13:42:41.491 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:41 smithi113 ceph-mon[34341]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi043=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:42:41.491 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:41 smithi113 ceph-mon[34341]: Saving service rgw.foo.a spec with placement smithi043=foo.a;count:1 2024-03-30T13:42:41.491 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:41 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:41.491 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:41 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:41.491 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:41 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:41.492 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:41 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:42:41.492 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:41 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:41.492 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:41 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-30T13:42:41.492 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:41 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-30T13:42:41.492 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:41 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:41.492 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:41 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:41.534 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[29963]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-30T13:42:41.535 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[29963]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi043=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:42:41.535 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[29963]: Saving service rgw.foo.a spec with placement smithi043=foo.a;count:1 2024-03-30T13:42:41.535 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:41.535 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:41.535 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:41.535 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:42:41.535 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:41.535 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-30T13:42:41.535 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-30T13:42:41.535 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:41.535 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:41.536 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[42099]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-30T13:42:41.536 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[42099]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi043=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:42:41.536 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[42099]: Saving service rgw.foo.a spec with placement smithi043=foo.a;count:1 2024-03-30T13:42:41.536 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:41.536 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:41.536 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:41.536 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:42:41.536 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:41.536 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-30T13:42:41.536 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-30T13:42:41.536 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:41.537 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:41 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:42.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:42 smithi113 ceph-mon[34341]: Deploying daemon rgw.foo.a on smithi043 2024-03-30T13:42:42.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:42 smithi043 ceph-mon[42099]: Deploying daemon rgw.foo.a on smithi043 2024-03-30T13:42:42.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:42 smithi043 ceph-mon[29963]: Deploying daemon rgw.foo.a on smithi043 2024-03-30T13:42:43.283 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Mar 30 13:42:43 smithi043 systemd[1]: Starting Ceph rgw.foo.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T13:42:43.293 INFO:teuthology.orchestra.run.smithi113.stderr:pool 'datapool' created 2024-03-30T13:42:43.592 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:43 smithi113 ceph-mon[34341]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T13:42:43.592 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:43 smithi113 ceph-mon[34341]: from='client.? 172.21.15.113:0/3808347549' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-30T13:42:43.592 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:43 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-30T13:42:43.626 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:43 smithi043 ceph-mon[29963]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T13:42:43.626 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:43 smithi043 ceph-mon[29963]: from='client.? 172.21.15.113:0/3808347549' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-30T13:42:43.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:43 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-30T13:42:43.627 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:43 smithi043 ceph-mon[42099]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T13:42:43.627 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:43 smithi043 ceph-mon[42099]: from='client.? 172.21.15.113:0/3808347549' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-30T13:42:43.627 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:43 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-30T13:42:43.627 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Mar 30 13:42:43 smithi043 podman[71178]: 2024-03-30 13:42:43.286329749 +0000 UTC m=+0.018625073 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T13:42:43.627 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Mar 30 13:42:43 smithi043 podman[71178]: 2024-03-30 13:42:43.388999135 +0000 UTC m=+0.121294435 container create b9e161922958d2fa1dcc15021343f10f8044bcb349ba4cf4bd8f379685f27afa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-30T13:42:43.627 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Mar 30 13:42:43 smithi043 podman[71178]: 2024-03-30 13:42:43.625902271 +0000 UTC m=+0.358197581 container init b9e161922958d2fa1dcc15021343f10f8044bcb349ba4cf4bd8f379685f27afa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-rgw-foo-a, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-30T13:42:43.882 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- rbd pool init datapool 2024-03-30T13:42:43.899 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Mar 30 13:42:43 smithi043 podman[71178]: 2024-03-30 13:42:43.631701374 +0000 UTC m=+0.363996672 container start b9e161922958d2fa1dcc15021343f10f8044bcb349ba4cf4bd8f379685f27afa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-rgw-foo-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T13:42:43.899 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Mar 30 13:42:43 smithi043 bash[71178]: b9e161922958d2fa1dcc15021343f10f8044bcb349ba4cf4bd8f379685f27afa 2024-03-30T13:42:43.899 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Mar 30 13:42:43 smithi043 systemd[1]: Started Ceph rgw.foo.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T13:42:44.124 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:42:44.561 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:44 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-30T13:42:44.561 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:44 smithi113 ceph-mon[34341]: osdmap e63: 8 total, 8 up, 8 in 2024-03-30T13:42:44.561 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:44 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:44.562 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:44 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:44.562 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:44 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:44.562 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:44 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:44.562 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:44 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:44.562 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:44 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:44.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:44 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-30T13:42:44.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:44 smithi043 ceph-mon[29963]: osdmap e63: 8 total, 8 up, 8 in 2024-03-30T13:42:44.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:44 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:44.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:44 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:44.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:44 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:44.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:44 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:44.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:44 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:44.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:44 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:44.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:44 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-30T13:42:44.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:44 smithi043 ceph-mon[42099]: osdmap e63: 8 total, 8 up, 8 in 2024-03-30T13:42:44.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:44 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:44.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:44 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:44.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:44 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:44.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:44 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:44.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:44 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:44.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:44 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:45.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:45 smithi043 ceph-mon[29963]: pgmap v190: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-30T13:42:45.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:45 smithi043 ceph-mon[29963]: Saving service rgw.foo.a spec with placement smithi043=foo.a;count:1 2024-03-30T13:42:45.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:45 smithi043 ceph-mon[29963]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:42:45.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:45 smithi043 ceph-mon[29963]: osdmap e64: 8 total, 8 up, 8 in 2024-03-30T13:42:45.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:45 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3935377197' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-30T13:42:45.401 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:45 smithi043 ceph-mon[29963]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-30T13:42:45.401 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:45 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:45.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:45 smithi043 ceph-mon[42099]: pgmap v190: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-30T13:42:45.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:45 smithi043 ceph-mon[42099]: Saving service rgw.foo.a spec with placement smithi043=foo.a;count:1 2024-03-30T13:42:45.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:45 smithi043 ceph-mon[42099]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:42:45.402 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:45 smithi043 ceph-mon[42099]: osdmap e64: 8 total, 8 up, 8 in 2024-03-30T13:42:45.402 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:45 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3935377197' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-30T13:42:45.402 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:45 smithi043 ceph-mon[42099]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-30T13:42:45.402 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:45 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:45.571 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:45 smithi113 ceph-mon[34341]: pgmap v190: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-30T13:42:45.571 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:45 smithi113 ceph-mon[34341]: Saving service rgw.foo.a spec with placement smithi043=foo.a;count:1 2024-03-30T13:42:45.571 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:45 smithi113 ceph-mon[34341]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:42:45.571 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:45 smithi113 ceph-mon[34341]: osdmap e64: 8 total, 8 up, 8 in 2024-03-30T13:42:45.571 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:45 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3935377197' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-30T13:42:45.571 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:45 smithi113 ceph-mon[34341]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-30T13:42:45.571 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:45 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:46.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:46 smithi043 ceph-mon[42099]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-30T13:42:46.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:46 smithi043 ceph-mon[42099]: osdmap e65: 8 total, 8 up, 8 in 2024-03-30T13:42:46.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:46 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:46.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:46 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:46.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:46 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:46.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:46 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:42:46.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:46 smithi043 ceph-mon[42099]: from='client.? 172.21.15.113:0/2088912227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-30T13:42:46.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:46 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-30T13:42:46.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:46 smithi043 ceph-mon[29963]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-30T13:42:46.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:46 smithi043 ceph-mon[29963]: osdmap e65: 8 total, 8 up, 8 in 2024-03-30T13:42:46.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:46 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:46.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:46 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:46.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:46 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:46.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:46 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:42:46.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:46 smithi043 ceph-mon[29963]: from='client.? 172.21.15.113:0/2088912227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-30T13:42:46.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:46 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-30T13:42:46.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:46 smithi113 ceph-mon[34341]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-30T13:42:46.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:46 smithi113 ceph-mon[34341]: osdmap e65: 8 total, 8 up, 8 in 2024-03-30T13:42:46.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:46 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:46.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:46 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:46.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:46 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:46.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:46 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:42:46.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:46 smithi113 ceph-mon[34341]: from='client.? 172.21.15.113:0/2088912227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-30T13:42:46.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:46 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-30T13:42:47.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:47 smithi043 ceph-mon[29963]: pgmap v193: 36 pgs: 6 creating+peering, 7 active+clean, 23 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:42:47.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:47 smithi043 ceph-mon[29963]: Checking dashboard <-> RGW credentials 2024-03-30T13:42:47.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:47 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-30T13:42:47.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:47 smithi043 ceph-mon[29963]: osdmap e66: 8 total, 8 up, 8 in 2024-03-30T13:42:47.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:47 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:42:47.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:47 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3517294948' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:42:47.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:47 smithi043 ceph-mon[29963]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:42:47.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:47 smithi043 ceph-mon[42099]: pgmap v193: 36 pgs: 6 creating+peering, 7 active+clean, 23 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:42:47.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:47 smithi043 ceph-mon[42099]: Checking dashboard <-> RGW credentials 2024-03-30T13:42:47.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:47 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-30T13:42:47.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:47 smithi043 ceph-mon[42099]: osdmap e66: 8 total, 8 up, 8 in 2024-03-30T13:42:47.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:47 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:42:47.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:47 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3517294948' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:42:47.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:47 smithi043 ceph-mon[42099]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:42:47.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:47 smithi113 ceph-mon[34341]: pgmap v193: 36 pgs: 6 creating+peering, 7 active+clean, 23 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:42:47.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:47 smithi113 ceph-mon[34341]: Checking dashboard <-> RGW credentials 2024-03-30T13:42:47.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:47 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-30T13:42:47.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:47 smithi113 ceph-mon[34341]: osdmap e66: 8 total, 8 up, 8 in 2024-03-30T13:42:47.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:47 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:42:47.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:47 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3517294948' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:42:47.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:47 smithi113 ceph-mon[34341]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:42:48.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:48 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-30T13:42:48.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:48 smithi043 ceph-mon[29963]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-30T13:42:48.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:48 smithi043 ceph-mon[29963]: osdmap e67: 8 total, 8 up, 8 in 2024-03-30T13:42:48.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:48 smithi043 ceph-mon[29963]: osdmap e68: 8 total, 8 up, 8 in 2024-03-30T13:42:48.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:48 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:42:48.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:48 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-30T13:42:48.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:48 smithi043 ceph-mon[42099]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-30T13:42:48.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:48 smithi043 ceph-mon[42099]: osdmap e67: 8 total, 8 up, 8 in 2024-03-30T13:42:48.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:48 smithi043 ceph-mon[42099]: osdmap e68: 8 total, 8 up, 8 in 2024-03-30T13:42:48.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:48 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:42:48.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:48 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-30T13:42:48.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:48 smithi113 ceph-mon[34341]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-30T13:42:48.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:48 smithi113 ceph-mon[34341]: osdmap e67: 8 total, 8 up, 8 in 2024-03-30T13:42:48.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:48 smithi113 ceph-mon[34341]: osdmap e68: 8 total, 8 up, 8 in 2024-03-30T13:42:48.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:48 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:42:49.106 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.113 --placement '1;smithi113=iscsi.a' 2024-03-30T13:42:49.327 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:49 smithi113 ceph-mon[34341]: pgmap v196: 68 pgs: 16 creating+peering, 24 active+clean, 28 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T13:42:49.327 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:49 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3517294948' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:42:49.327 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:49 smithi113 ceph-mon[34341]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T13:42:49.327 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:49 smithi113 ceph-mon[34341]: Cluster is now healthy 2024-03-30T13:42:49.327 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:49 smithi113 ceph-mon[34341]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:42:49.327 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:49 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-30T13:42:49.328 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:49 smithi113 ceph-mon[34341]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-30T13:42:49.328 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:49 smithi113 ceph-mon[34341]: osdmap e69: 8 total, 8 up, 8 in 2024-03-30T13:42:49.364 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:42:49.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:49 smithi043 ceph-mon[29963]: pgmap v196: 68 pgs: 16 creating+peering, 24 active+clean, 28 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T13:42:49.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:49 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3517294948' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:42:49.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:49 smithi043 ceph-mon[29963]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T13:42:49.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:49 smithi043 ceph-mon[29963]: Cluster is now healthy 2024-03-30T13:42:49.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:49 smithi043 ceph-mon[29963]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:42:49.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:49 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-30T13:42:49.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:49 smithi043 ceph-mon[29963]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-30T13:42:49.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:49 smithi043 ceph-mon[29963]: osdmap e69: 8 total, 8 up, 8 in 2024-03-30T13:42:49.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:49 smithi043 ceph-mon[42099]: pgmap v196: 68 pgs: 16 creating+peering, 24 active+clean, 28 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T13:42:49.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:49 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3517294948' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:42:49.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:49 smithi043 ceph-mon[42099]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T13:42:49.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:49 smithi043 ceph-mon[42099]: Cluster is now healthy 2024-03-30T13:42:49.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:49 smithi043 ceph-mon[42099]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:42:49.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:49 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-30T13:42:49.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:49 smithi043 ceph-mon[42099]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-30T13:42:49.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:49 smithi043 ceph-mon[42099]: osdmap e69: 8 total, 8 up, 8 in 2024-03-30T13:42:51.159 INFO:teuthology.orchestra.run.smithi113.stdout:Scheduled iscsi.datapool update... 2024-03-30T13:42:51.460 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:51 smithi113 ceph-mon[34341]: pgmap v199: 100 pgs: 16 creating+peering, 37 active+clean, 47 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T13:42:51.460 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:51 smithi113 ceph-mon[34341]: osdmap e70: 8 total, 8 up, 8 in 2024-03-30T13:42:51.460 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:51 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:42:51.460 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:51 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3517294948' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:42:51.460 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:51 smithi113 ceph-mon[34341]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:42:51.460 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:51 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:51.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:51 smithi043 ceph-mon[29963]: pgmap v199: 100 pgs: 16 creating+peering, 37 active+clean, 47 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T13:42:51.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:51 smithi043 ceph-mon[29963]: osdmap e70: 8 total, 8 up, 8 in 2024-03-30T13:42:51.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:51 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:42:51.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:51 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3517294948' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:42:51.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:51 smithi043 ceph-mon[29963]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:42:51.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:51 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:51.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:51 smithi043 ceph-mon[42099]: pgmap v199: 100 pgs: 16 creating+peering, 37 active+clean, 47 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T13:42:51.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:51 smithi043 ceph-mon[42099]: osdmap e70: 8 total, 8 up, 8 in 2024-03-30T13:42:51.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:51 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:42:51.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:51 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3517294948' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:42:51.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:51 smithi043 ceph-mon[42099]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:42:51.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:51 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:51.738 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-30T13:42:51.738 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-30T13:42:51.738 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-30T13:42:51.775 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-30T13:42:51.775 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-30T13:42:51.804 DEBUG:teuthology.orchestra.run.smithi113:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@iscsi.iscsi.a.service 2024-03-30T13:42:51.847 INFO:tasks.cephadm:Adding prometheus.a on smithi113 2024-03-30T13:42:51.847 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph orch apply prometheus '1;smithi113=a' 2024-03-30T13:42:52.124 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:42:52.505 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:52 smithi113 ceph-mon[34341]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.113", "placement": "1;smithi113=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:42:52.505 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:52 smithi113 ceph-mon[34341]: Saving service iscsi.datapool spec with placement smithi113=iscsi.a;count:1 2024-03-30T13:42:52.505 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:52 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-30T13:42:52.505 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:52 smithi113 ceph-mon[34341]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-30T13:42:52.506 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:52 smithi113 ceph-mon[34341]: osdmap e71: 8 total, 8 up, 8 in 2024-03-30T13:42:52.506 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:52 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:42:52.506 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:52 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3517294948' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:42:52.506 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:52 smithi113 ceph-mon[34341]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:42:52.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:52 smithi043 ceph-mon[29963]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.113", "placement": "1;smithi113=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:42:52.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:52 smithi043 ceph-mon[29963]: Saving service iscsi.datapool spec with placement smithi113=iscsi.a;count:1 2024-03-30T13:42:52.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:52 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-30T13:42:52.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:52 smithi043 ceph-mon[29963]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-30T13:42:52.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:52 smithi043 ceph-mon[29963]: osdmap e71: 8 total, 8 up, 8 in 2024-03-30T13:42:52.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:52 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:42:52.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:52 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3517294948' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:42:52.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:52 smithi043 ceph-mon[29963]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:42:52.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:52 smithi043 ceph-mon[42099]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.113", "placement": "1;smithi113=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:42:52.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:52 smithi043 ceph-mon[42099]: Saving service iscsi.datapool spec with placement smithi113=iscsi.a;count:1 2024-03-30T13:42:52.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:52 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-30T13:42:52.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:52 smithi043 ceph-mon[42099]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-30T13:42:52.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:52 smithi043 ceph-mon[42099]: osdmap e71: 8 total, 8 up, 8 in 2024-03-30T13:42:52.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:52 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:42:52.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:52 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3517294948' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:42:52.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:52 smithi043 ceph-mon[42099]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:42:53.542 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:53 smithi113 ceph-mon[34341]: pgmap v202: 132 pgs: 1 creating+activating, 8 creating+peering, 90 active+clean, 33 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T13:42:53.543 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:53 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-30T13:42:53.543 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:53 smithi113 ceph-mon[34341]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-30T13:42:53.543 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:53 smithi113 ceph-mon[34341]: osdmap e72: 8 total, 8 up, 8 in 2024-03-30T13:42:53.543 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:53 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:53.543 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:53 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:53.543 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:53 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:53.543 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:53 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-30T13:42:53.543 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:53 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-30T13:42:53.543 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:53 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:53.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:53 smithi043 ceph-mon[29963]: pgmap v202: 132 pgs: 1 creating+activating, 8 creating+peering, 90 active+clean, 33 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T13:42:53.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:53 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-30T13:42:53.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:53 smithi043 ceph-mon[29963]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-30T13:42:53.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:53 smithi043 ceph-mon[29963]: osdmap e72: 8 total, 8 up, 8 in 2024-03-30T13:42:53.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:53 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:53.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:53 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:53.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:53 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:53.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:53 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-30T13:42:53.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:53 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-30T13:42:53.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:53 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:53.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:53 smithi043 ceph-mon[42099]: pgmap v202: 132 pgs: 1 creating+activating, 8 creating+peering, 90 active+clean, 33 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T13:42:53.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:53 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3211716515' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-30T13:42:53.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:53 smithi043 ceph-mon[42099]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-30T13:42:53.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:53 smithi043 ceph-mon[42099]: osdmap e72: 8 total, 8 up, 8 in 2024-03-30T13:42:53.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:53 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:53.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:53 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:53.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:53 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:53.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:53 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-30T13:42:53.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:53 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-30T13:42:53.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:53 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:42:54.060 INFO:teuthology.orchestra.run.smithi113.stdout:Scheduled prometheus update... 2024-03-30T13:42:54.618 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:54 smithi113 ceph-mon[34341]: Deploying daemon iscsi.iscsi.a on smithi113 2024-03-30T13:42:54.618 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:54 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:54.618 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:54 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:42:54.618 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:54 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:42:54.645 DEBUG:teuthology.orchestra.run.smithi113:prometheus.a> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@prometheus.a.service 2024-03-30T13:42:54.648 INFO:tasks.cephadm:Adding node-exporter.a on smithi043 2024-03-30T13:42:54.648 INFO:tasks.cephadm:Adding node-exporter.b on smithi113 2024-03-30T13:42:54.648 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph orch apply node-exporter '2;smithi043=a;smithi113=b' 2024-03-30T13:42:54.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:54 smithi043 ceph-mon[29963]: Deploying daemon iscsi.iscsi.a on smithi113 2024-03-30T13:42:54.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:54 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:54.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:54 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:42:54.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:54 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:42:54.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:54 smithi043 ceph-mon[42099]: Deploying daemon iscsi.iscsi.a on smithi113 2024-03-30T13:42:54.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:54 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:54.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:54 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:42:54.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:54 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:42:55.557 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:55 smithi113 ceph-mon[34341]: pgmap v204: 132 pgs: 1 creating+activating, 7 creating+peering, 119 active+clean, 5 unknown; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 3.2 KiB/s wr, 60 op/s 2024-03-30T13:42:55.557 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:55 smithi113 ceph-mon[34341]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi113=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:42:55.557 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:55 smithi113 ceph-mon[34341]: Saving service prometheus spec with placement smithi113=a;count:1 2024-03-30T13:42:55.558 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:55 smithi113 ceph-mon[34341]: osdmap e73: 8 total, 8 up, 8 in 2024-03-30T13:42:55.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:55 smithi043 ceph-mon[29963]: pgmap v204: 132 pgs: 1 creating+activating, 7 creating+peering, 119 active+clean, 5 unknown; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 3.2 KiB/s wr, 60 op/s 2024-03-30T13:42:55.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:55 smithi043 ceph-mon[29963]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi113=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:42:55.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:55 smithi043 ceph-mon[29963]: Saving service prometheus spec with placement smithi113=a;count:1 2024-03-30T13:42:55.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:55 smithi043 ceph-mon[29963]: osdmap e73: 8 total, 8 up, 8 in 2024-03-30T13:42:55.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:55 smithi043 ceph-mon[42099]: pgmap v204: 132 pgs: 1 creating+activating, 7 creating+peering, 119 active+clean, 5 unknown; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 3.2 KiB/s wr, 60 op/s 2024-03-30T13:42:55.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:55 smithi043 ceph-mon[42099]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi113=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:42:55.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:55 smithi043 ceph-mon[42099]: Saving service prometheus spec with placement smithi113=a;count:1 2024-03-30T13:42:55.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:55 smithi043 ceph-mon[42099]: osdmap e73: 8 total, 8 up, 8 in 2024-03-30T13:42:55.662 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:42:57.216 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:42:56 smithi113 podman[64634]: 2024-03-30 13:42:56.973177369 +0000 UTC m=+0.143628653 container create 2f903e8571e343740df1900336a26310e1f46a03d9b79056308c65d0f8266d53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-30T13:42:57.474 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:57 smithi113 ceph-mon[34341]: pgmap v206: 132 pgs: 1 creating+activating, 3 creating+peering, 128 active+clean; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 3.6 KiB/s wr, 95 op/s 2024-03-30T13:42:57.474 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:57 smithi113 ceph-mon[34341]: osdmap e74: 8 total, 8 up, 8 in 2024-03-30T13:42:57.474 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:42:57 smithi113 podman[64634]: 2024-03-30 13:42:57.420757895 +0000 UTC m=+0.591209137 container init 2f903e8571e343740df1900336a26310e1f46a03d9b79056308c65d0f8266d53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-30T13:42:57.474 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:42:57 smithi113 podman[64634]: 2024-03-30 13:42:57.426402613 +0000 UTC m=+0.596853856 container start 2f903e8571e343740df1900336a26310e1f46a03d9b79056308c65d0f8266d53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T13:42:57.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:57 smithi043 ceph-mon[29963]: pgmap v206: 132 pgs: 1 creating+activating, 3 creating+peering, 128 active+clean; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 3.6 KiB/s wr, 95 op/s 2024-03-30T13:42:57.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:57 smithi043 ceph-mon[29963]: osdmap e74: 8 total, 8 up, 8 in 2024-03-30T13:42:57.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:57 smithi043 ceph-mon[42099]: pgmap v206: 132 pgs: 1 creating+activating, 3 creating+peering, 128 active+clean; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 3.6 KiB/s wr, 95 op/s 2024-03-30T13:42:57.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:57 smithi043 ceph-mon[42099]: osdmap e74: 8 total, 8 up, 8 in 2024-03-30T13:42:57.870 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:42:57 smithi113 bash[64634]: 2f903e8571e343740df1900336a26310e1f46a03d9b79056308c65d0f8266d53 2024-03-30T13:42:57.870 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:42:57 smithi113 systemd[1]: Started Ceph iscsi.iscsi.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T13:42:57.870 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:42:57 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug Started the configuration object watcher 2024-03-30T13:42:57.870 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:42:57 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug Checking for config object changes every 1s 2024-03-30T13:42:57.870 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:42:57 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug Processing osd blocklist entries for this node 2024-03-30T13:42:58.010 INFO:teuthology.orchestra.run.smithi113.stdout:Scheduled node-exporter update... 2024-03-30T13:42:58.222 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:42:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug Reading the configuration object to update local LIO configuration 2024-03-30T13:42:58.222 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:42:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug Configuration does not have an entry for this host(smithi113.front.sepia.ceph.com) - nothing to define to LIO 2024-03-30T13:42:58.222 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:42:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-30T13:42:58.679 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:58 smithi113 ceph-mon[34341]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 185 op/s 2024-03-30T13:42:58.679 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:58 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:58.679 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:58 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:58.679 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:58 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:58.679 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:58 smithi113 ceph-mon[34341]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-30T13:42:58.679 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:58 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:58.679 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:58 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:58.679 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:58 smithi113 ceph-mon[34341]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi043=a;smithi113=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:42:58.680 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:58 smithi113 ceph-mon[34341]: Saving service node-exporter spec with placement smithi043=a;smithi113=b;count:2 2024-03-30T13:42:58.680 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:58 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:58.680 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:42:58 smithi113 ceph-mon[34341]: from='client.? 172.21.15.113:0/678004711' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T13:42:58.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[29963]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 185 op/s 2024-03-30T13:42:58.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:58.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:58.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:58.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[29963]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-30T13:42:58.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:58.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:58.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[29963]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi043=a;smithi113=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:42:58.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[29963]: Saving service node-exporter spec with placement smithi043=a;smithi113=b;count:2 2024-03-30T13:42:58.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:58.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[29963]: from='client.? 172.21.15.113:0/678004711' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T13:42:58.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[42099]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 185 op/s 2024-03-30T13:42:58.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:58.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:58.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:58.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[42099]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-30T13:42:58.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:58.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:42:58.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[42099]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi043=a;smithi113=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:42:58.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[42099]: Saving service node-exporter spec with placement smithi043=a;smithi113=b;count:2 2024-03-30T13:42:58.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:42:58.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:42:58 smithi043 ceph-mon[42099]: from='client.? 172.21.15.113:0/678004711' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T13:42:59.005 DEBUG:teuthology.orchestra.run.smithi043:node-exporter.a> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@node-exporter.a.service 2024-03-30T13:42:59.008 DEBUG:teuthology.orchestra.run.smithi113:node-exporter.b> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@node-exporter.b.service 2024-03-30T13:42:59.011 INFO:tasks.cephadm:Adding alertmanager.a on smithi043 2024-03-30T13:42:59.012 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph orch apply alertmanager '1;smithi043=a' 2024-03-30T13:42:59.704 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:43:00.658 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:00 smithi043 ceph-mon[29963]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 4.8 KiB/s wr, 157 op/s 2024-03-30T13:43:00.658 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:00 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:00.658 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:00 smithi043 ceph-mon[42099]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 4.8 KiB/s wr, 157 op/s 2024-03-30T13:43:00.658 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:00 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:00.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:00 smithi113 ceph-mon[34341]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 4.8 KiB/s wr, 157 op/s 2024-03-30T13:43:00.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:00 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:01.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:01 smithi113 ceph-mon[34341]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-30T13:43:01.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:01 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:43:01.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:01 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:43:01.628 INFO:teuthology.orchestra.run.smithi113.stdout:Scheduled alertmanager update... 2024-03-30T13:43:01.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:01 smithi043 ceph-mon[29963]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-30T13:43:01.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:01 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:43:01.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:01 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:43:01.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:01 smithi043 ceph-mon[42099]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-30T13:43:01.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:01 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:43:01.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:01 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:43:02.684 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:02 smithi113 ceph-mon[34341]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-03-30T13:43:02.684 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:02 smithi113 ceph-mon[34341]: osdmap e75: 8 total, 8 up, 8 in 2024-03-30T13:43:02.684 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:02 smithi113 ceph-mon[34341]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi043=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:43:02.684 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:02 smithi113 ceph-mon[34341]: Saving service alertmanager spec with placement smithi043=a;count:1 2024-03-30T13:43:02.684 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:02 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:02.684 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:02 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:02.684 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:02 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:02.684 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:02 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:43:02.684 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:02 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:43:02.684 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:02 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:02.712 DEBUG:teuthology.orchestra.run.smithi043:alertmanager.a> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@alertmanager.a.service 2024-03-30T13:43:02.715 INFO:tasks.cephadm:Adding grafana.a on smithi113 2024-03-30T13:43:02.715 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph orch apply grafana '1;smithi113=a' 2024-03-30T13:43:02.738 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:02 smithi043 ceph-mon[29963]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-03-30T13:43:02.738 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:02 smithi043 ceph-mon[29963]: osdmap e75: 8 total, 8 up, 8 in 2024-03-30T13:43:02.738 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:02 smithi043 ceph-mon[29963]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi043=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:43:02.738 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:02 smithi043 ceph-mon[29963]: Saving service alertmanager spec with placement smithi043=a;count:1 2024-03-30T13:43:02.738 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:02 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:02.739 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:02 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:02.739 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:02 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:02.739 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:02 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:43:02.739 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:02 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:43:02.739 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:02 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:02.739 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:02 smithi043 ceph-mon[42099]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-03-30T13:43:02.739 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:02 smithi043 ceph-mon[42099]: osdmap e75: 8 total, 8 up, 8 in 2024-03-30T13:43:02.739 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:02 smithi043 ceph-mon[42099]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi043=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:43:02.739 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:02 smithi043 ceph-mon[42099]: Saving service alertmanager spec with placement smithi043=a;count:1 2024-03-30T13:43:02.739 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:02 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:02.739 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:02 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:02.740 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:02 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:02.740 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:02 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:43:02.740 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:02 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:43:02.740 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:02 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:02.962 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:43:03.707 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:03 smithi113 ceph-mon[34341]: Deploying daemon prometheus.a on smithi113 2024-03-30T13:43:03.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:03 smithi043 ceph-mon[29963]: Deploying daemon prometheus.a on smithi113 2024-03-30T13:43:03.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:03 smithi043 ceph-mon[42099]: Deploying daemon prometheus.a on smithi113 2024-03-30T13:43:04.698 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:04 smithi113 ceph-mon[34341]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 2.7 KiB/s wr, 83 op/s 2024-03-30T13:43:04.699 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:04 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:04.699 INFO:teuthology.orchestra.run.smithi113.stdout:Scheduled grafana update... 2024-03-30T13:43:04.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:04 smithi043 ceph-mon[29963]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 2.7 KiB/s wr, 83 op/s 2024-03-30T13:43:04.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:04 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:04.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:04 smithi043 ceph-mon[42099]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 2.7 KiB/s wr, 83 op/s 2024-03-30T13:43:04.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:04 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:05.697 DEBUG:teuthology.orchestra.run.smithi113:grafana.a> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@grafana.a.service 2024-03-30T13:43:05.699 INFO:tasks.cephadm:Setting up client nodes... 2024-03-30T13:43:05.700 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-30T13:43:05.953 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:43:05.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:05 smithi113 ceph-mon[34341]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi113=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:43:05.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:05 smithi113 ceph-mon[34341]: Saving service grafana spec with placement smithi113=a;count:1 2024-03-30T13:43:05.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:05 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:06.075 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:05 smithi043 ceph-mon[29963]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi113=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:43:06.075 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:05 smithi043 ceph-mon[29963]: Saving service grafana spec with placement smithi113=a;count:1 2024-03-30T13:43:06.075 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:05 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:06.076 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:05 smithi043 ceph-mon[42099]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi113=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:43:06.076 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:05 smithi043 ceph-mon[42099]: Saving service grafana spec with placement smithi113=a;count:1 2024-03-30T13:43:06.076 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:05 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:06.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:06 smithi113 ceph-mon[34341]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 252 B/s wr, 6 op/s 2024-03-30T13:43:07.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:06 smithi043 ceph-mon[29963]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 252 B/s wr, 6 op/s 2024-03-30T13:43:07.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:06 smithi043 ceph-mon[42099]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 252 B/s wr, 6 op/s 2024-03-30T13:43:07.972 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:43:07 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:43:08.220 INFO:teuthology.orchestra.run.smithi043.stdout:[client.0] 2024-03-30T13:43:08.221 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQDsFghmxh30DBAA3O+2iclozxP3yI3O69Jhcw== 2024-03-30T13:43:08.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:08 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3841571431' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T13:43:08.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:08 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3841571431' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T13:43:08.513 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:08 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3841571431' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T13:43:08.513 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:08 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3841571431' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T13:43:08.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:08 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3841571431' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T13:43:08.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:08 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3841571431' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T13:43:08.814 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-30T13:43:08.814 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-30T13:43:08.814 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-30T13:43:08.857 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-30T13:43:09.210 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:43:09.619 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:09 smithi113 ceph-mon[34341]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 102 B/s wr, 1 op/s 2024-03-30T13:43:09.619 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:09 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:43:09.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:09 smithi043 ceph-mon[29963]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 102 B/s wr, 1 op/s 2024-03-30T13:43:09.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:09 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:43:09.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:09 smithi043 ceph-mon[42099]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 102 B/s wr, 1 op/s 2024-03-30T13:43:09.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:09 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:43:10.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:10 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-03-30T13:43:10.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:10 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-03-30T13:43:10.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:10 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-30T13:43:10.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:10 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:43:10.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:10 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:43:10.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:10 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-03-30T13:43:10.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:10 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-03-30T13:43:10.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:10 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-30T13:43:10.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:10 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:43:10.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:10 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:43:10.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:10 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-03-30T13:43:10.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:10 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-03-30T13:43:10.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:10 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-30T13:43:10.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:10 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:43:10.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:10 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:43:11.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:11 smithi043 ceph-mon[29963]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 102 B/s wr, 1 op/s 2024-03-30T13:43:11.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:11 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-03-30T13:43:11.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:11 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-03-30T13:43:11.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:11 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-30T13:43:11.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:11 smithi043 ceph-mon[29963]: osdmap e76: 8 total, 8 up, 8 in 2024-03-30T13:43:11.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:11 smithi043 ceph-mon[42099]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 102 B/s wr, 1 op/s 2024-03-30T13:43:11.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:11 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-03-30T13:43:11.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:11 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-03-30T13:43:11.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:11 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-30T13:43:11.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:11 smithi043 ceph-mon[42099]: osdmap e76: 8 total, 8 up, 8 in 2024-03-30T13:43:11.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:11 smithi113 ceph-mon[34341]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 102 B/s wr, 1 op/s 2024-03-30T13:43:11.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:11 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-03-30T13:43:11.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:11 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-03-30T13:43:11.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:11 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-30T13:43:11.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:11 smithi113 ceph-mon[34341]: osdmap e76: 8 total, 8 up, 8 in 2024-03-30T13:43:12.496 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:12 smithi113 ceph-mon[34341]: osdmap e77: 8 total, 8 up, 8 in 2024-03-30T13:43:12.496 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:12 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:43:12.496 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:12 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:43:12.496 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:12 smithi113 ceph-mon[34341]: osdmap e78: 8 total, 8 up, 8 in 2024-03-30T13:43:12.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:12 smithi043 ceph-mon[29963]: osdmap e77: 8 total, 8 up, 8 in 2024-03-30T13:43:12.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:12 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:43:12.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:12 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:43:12.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:12 smithi043 ceph-mon[29963]: osdmap e78: 8 total, 8 up, 8 in 2024-03-30T13:43:12.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:12 smithi043 ceph-mon[42099]: osdmap e77: 8 total, 8 up, 8 in 2024-03-30T13:43:12.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:12 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:43:12.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:12 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:43:12.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:12 smithi043 ceph-mon[42099]: osdmap e78: 8 total, 8 up, 8 in 2024-03-30T13:43:13.100 INFO:teuthology.orchestra.run.smithi113.stdout:[client.1] 2024-03-30T13:43:13.100 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDxFghm0P6pBRAALdbHGR29+h0w2MvbApZbUA== 2024-03-30T13:43:13.412 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:13 smithi113 ceph-mon[34341]: pgmap v218: 132 pgs: 1 activating+degraded, 131 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 5/594 objects degraded (0.842%) 2024-03-30T13:43:13.412 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:13 smithi113 ceph-mon[34341]: Health check failed: Degraded data redundancy: 5/594 objects degraded (0.842%), 1 pg degraded (PG_DEGRADED) 2024-03-30T13:43:13.412 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:13 smithi113 ceph-mon[34341]: from='client.? 172.21.15.113:0/785750988' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T13:43:13.412 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:13 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T13:43:13.412 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:13 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T13:43:13.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:13 smithi043 ceph-mon[29963]: pgmap v218: 132 pgs: 1 activating+degraded, 131 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 5/594 objects degraded (0.842%) 2024-03-30T13:43:13.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:13 smithi043 ceph-mon[29963]: Health check failed: Degraded data redundancy: 5/594 objects degraded (0.842%), 1 pg degraded (PG_DEGRADED) 2024-03-30T13:43:13.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:13 smithi043 ceph-mon[29963]: from='client.? 172.21.15.113:0/785750988' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T13:43:13.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:13 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T13:43:13.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:13 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T13:43:13.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:13 smithi043 ceph-mon[42099]: pgmap v218: 132 pgs: 1 activating+degraded, 131 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 5/594 objects degraded (0.842%) 2024-03-30T13:43:13.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:13 smithi043 ceph-mon[42099]: Health check failed: Degraded data redundancy: 5/594 objects degraded (0.842%), 1 pg degraded (PG_DEGRADED) 2024-03-30T13:43:13.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:13 smithi043 ceph-mon[42099]: from='client.? 172.21.15.113:0/785750988' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T13:43:13.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:13 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T13:43:13.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:13 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T13:43:14.530 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-30T13:43:14.530 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-30T13:43:14.530 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-30T13:43:14.576 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-30T13:43:14.576 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-30T13:43:14.577 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph mgr dump --format=json 2024-03-30T13:43:14.830 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:43:15.576 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:15 smithi113 ceph-mon[34341]: pgmap v220: 132 pgs: 1 activating+degraded, 131 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 5/594 objects degraded (0.842%); 0 B/s, 0 objects/s recovering 2024-03-30T13:43:15.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:15 smithi043 ceph-mon[29963]: pgmap v220: 132 pgs: 1 activating+degraded, 131 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 5/594 objects degraded (0.842%); 0 B/s, 0 objects/s recovering 2024-03-30T13:43:15.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:15 smithi043 ceph-mon[42099]: pgmap v220: 132 pgs: 1 activating+degraded, 131 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 5/594 objects degraded (0.842%); 0 B/s, 0 objects/s recovering 2024-03-30T13:43:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:43:17.222 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:17 smithi113 systemd[1]: Starting Ceph prometheus.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T13:43:17.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:17 smithi043 ceph-mon[29963]: pgmap v221: 132 pgs: 1 activating+degraded, 131 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 5/594 objects degraded (0.842%); 18 B/s, 1 objects/s recovering 2024-03-30T13:43:17.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:17 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3031243297' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-30T13:43:17.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:17 smithi043 ceph-mon[29963]: osdmap e79: 8 total, 8 up, 8 in 2024-03-30T13:43:17.585 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:17 smithi043 ceph-mon[42099]: pgmap v221: 132 pgs: 1 activating+degraded, 131 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 5/594 objects degraded (0.842%); 18 B/s, 1 objects/s recovering 2024-03-30T13:43:17.585 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:17 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3031243297' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-30T13:43:17.585 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:17 smithi043 ceph-mon[42099]: osdmap e79: 8 total, 8 up, 8 in 2024-03-30T13:43:17.615 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":3033707884},{"type":"v1","addr":"172.21.15.43:6801","nonce":3033707884}]},"active_addr":"172.21.15.43:6801/3033707884","active_change":"2024-03-30T13:38:09.321856+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.43:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":2524269526}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":538304234}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":3539398453}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":3923597429}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":2414364740}]}]} 2024-03-30T13:43:17.619 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-30T13:43:17.620 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-30T13:43:17.620 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T13:43:17.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:17 smithi113 ceph-mon[34341]: pgmap v221: 132 pgs: 1 activating+degraded, 131 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 5/594 objects degraded (0.842%); 18 B/s, 1 objects/s recovering 2024-03-30T13:43:17.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:17 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3031243297' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-30T13:43:17.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:17 smithi113 ceph-mon[34341]: osdmap e79: 8 total, 8 up, 8 in 2024-03-30T13:43:17.722 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:17 smithi113 podman[66446]: 2024-03-30 13:43:17.196350505 +0000 UTC m=+0.019269788 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-30T13:43:17.866 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:43:18.222 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:43:17 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:43:18.517 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 podman[66446]: 2024-03-30 13:43:18.25747058 +0000 UTC m=+1.080389847 container create 37bd40d9ec3389dcb95a2720e99830cecc7b090a5595e04774105eda2f10531d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T13:43:18.518 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-mon[34341]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/594 objects degraded (0.842%), 1 pg degraded) 2024-03-30T13:43:18.518 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-mon[34341]: Cluster is now healthy 2024-03-30T13:43:18.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:18 smithi043 ceph-mon[29963]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/594 objects degraded (0.842%), 1 pg degraded) 2024-03-30T13:43:18.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:18 smithi043 ceph-mon[29963]: Cluster is now healthy 2024-03-30T13:43:18.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:18 smithi043 ceph-mon[42099]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/594 objects degraded (0.842%), 1 pg degraded) 2024-03-30T13:43:18.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:18 smithi043 ceph-mon[42099]: Cluster is now healthy 2024-03-30T13:43:18.972 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 podman[66446]: 2024-03-30 13:43:18.618204914 +0000 UTC m=+1.441124191 container init 37bd40d9ec3389dcb95a2720e99830cecc7b090a5595e04774105eda2f10531d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T13:43:18.973 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 podman[66446]: 2024-03-30 13:43:18.624198151 +0000 UTC m=+1.447117419 container start 37bd40d9ec3389dcb95a2720e99830cecc7b090a5595e04774105eda2f10531d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T13:43:18.973 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:43:18.648Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-30T13:43:18.973 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:43:18.648Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-30T13:43:18.973 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:43:18.648Z caller=main.go:570 level=info host_details="(Linux 5.14.0-434.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Mar 26 19:20:47 UTC 2024 x86_64 smithi113 (none))" 2024-03-30T13:43:18.973 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:43:18.648Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-30T13:43:18.973 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:43:18.648Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-30T13:43:18.973 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:43:18.649Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-30T13:43:18.973 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:43:18.650Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-30T13:43:18.973 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:43:18.650Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-30T13:43:18.973 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:43:18.650Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-30T13:43:18.973 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:43:18.653Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-30T13:43:18.973 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:43:18.653Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.93µs 2024-03-30T13:43:18.974 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:43:18.653Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-30T13:43:18.974 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:43:18.653Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-30T13:43:18.974 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:43:18.653Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.604µs wal_replay_duration=229.339µs wbl_replay_duration=116ns total_replay_duration=266.443µs 2024-03-30T13:43:18.974 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:43:18.654Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-30T13:43:18.974 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:43:18.654Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-30T13:43:18.974 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:43:18.654Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-30T13:43:18.974 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:43:18.672Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.747474ms db_storage=890ns remote_storage=1.088µs web_handler=367ns query_engine=780ns scrape=402.723µs scrape_sd=67.096µs notify=1.03µs notify_sd=2.245µs rules=17.095865ms tracing=11.231µs 2024-03-30T13:43:18.974 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:43:18.672Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-30T13:43:18.974 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:43:18.672Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-30T13:43:18.974 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 bash[66446]: 37bd40d9ec3389dcb95a2720e99830cecc7b090a5595e04774105eda2f10531d 2024-03-30T13:43:18.974 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:43:18 smithi113 systemd[1]: Started Ceph prometheus.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T13:43:19.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:19 smithi043 ceph-mon[29963]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 36 B/s, 2 objects/s recovering 2024-03-30T13:43:19.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:19 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:43:19.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:19 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:19.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:19 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:19.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:19 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:19.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:19 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-30T13:43:19.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:19 smithi043 ceph-mon[42099]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 36 B/s, 2 objects/s recovering 2024-03-30T13:43:19.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:19 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:43:19.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:19 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:19.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:19 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:19.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:19 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:19.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:19 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-30T13:43:19.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:19 smithi113 ceph-mon[34341]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 36 B/s, 2 objects/s recovering 2024-03-30T13:43:19.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:19 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:43:19.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:19 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:19.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:19 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:19.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:19 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:19.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:19 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-30T13:43:19.861 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:43:19.861 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":79,"fsid":"7ee932fa-ee9a-11ee-b647-cb9ed24678a4","created":"2024-03-30T13:37:07.346020+0000","modified":"2024-03-30T13:43:17.280239+0000","last_up_change":"2024-03-30T13:42:30.966931+0000","last_in_change":"2024-03-30T13:42:12.831084+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T13:40:27.658979+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-30T13:42:42.989975+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":68,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-30T13:42:43.910306+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-30T13:42:45.633132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-30T13:42:47.570084+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-30T13:42:49.637189+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94ea083e-6a27-41db-83af-2f7d3cae1956","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1144592572},{"type":"v1","addr":"172.21.15.43:6803","nonce":1144592572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1144592572},{"type":"v1","addr":"172.21.15.43:6805","nonce":1144592572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":1144592572},{"type":"v1","addr":"172.21.15.43:6809","nonce":1144592572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1144592572},{"type":"v1","addr":"172.21.15.43:6807","nonce":1144592572}]},"public_addr":"172.21.15.43:6803/1144592572","cluster_addr":"172.21.15.43:6805/1144592572","heartbeat_back_addr":"172.21.15.43:6809/1144592572","heartbeat_front_addr":"172.21.15.43:6807/1144592572","state":["exists","up"]},{"osd":1,"uuid":"8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":3901165928},{"type":"v1","addr":"172.21.15.43:6811","nonce":3901165928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":3901165928},{"type":"v1","addr":"172.21.15.43:6813","nonce":3901165928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":3901165928},{"type":"v1","addr":"172.21.15.43:6817","nonce":3901165928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":3901165928},{"type":"v1","addr":"172.21.15.43:6815","nonce":3901165928}]},"public_addr":"172.21.15.43:6811/3901165928","cluster_addr":"172.21.15.43:6813/3901165928","heartbeat_back_addr":"172.21.15.43:6817/3901165928","heartbeat_front_addr":"172.21.15.43:6815/3901165928","state":["exists","up"]},{"osd":2,"uuid":"6db6782e-5889-443e-bc6c-df47a1f93bc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":3415037523},{"type":"v1","addr":"172.21.15.43:6819","nonce":3415037523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":3415037523},{"type":"v1","addr":"172.21.15.43:6821","nonce":3415037523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":3415037523},{"type":"v1","addr":"172.21.15.43:6825","nonce":3415037523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":3415037523},{"type":"v1","addr":"172.21.15.43:6823","nonce":3415037523}]},"public_addr":"172.21.15.43:6819/3415037523","cluster_addr":"172.21.15.43:6821/3415037523","heartbeat_back_addr":"172.21.15.43:6825/3415037523","heartbeat_front_addr":"172.21.15.43:6823/3415037523","state":["exists","up"]},{"osd":3,"uuid":"7048ff46-2cba-4d18-9835-95ecbddb2bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2933589376},{"type":"v1","addr":"172.21.15.43:6827","nonce":2933589376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2933589376},{"type":"v1","addr":"172.21.15.43:6829","nonce":2933589376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2933589376},{"type":"v1","addr":"172.21.15.43:6833","nonce":2933589376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2933589376},{"type":"v1","addr":"172.21.15.43:6831","nonce":2933589376}]},"public_addr":"172.21.15.43:6827/2933589376","cluster_addr":"172.21.15.43:6829/2933589376","heartbeat_back_addr":"172.21.15.43:6833/2933589376","heartbeat_front_addr":"172.21.15.43:6831/2933589376","state":["exists","up"]},{"osd":4,"uuid":"42da30e6-9eec-4ed5-8c73-9ce78aa8ff63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1945833081},{"type":"v1","addr":"172.21.15.113:6801","nonce":1945833081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1945833081},{"type":"v1","addr":"172.21.15.113:6803","nonce":1945833081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1945833081},{"type":"v1","addr":"172.21.15.113:6807","nonce":1945833081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1945833081},{"type":"v1","addr":"172.21.15.113:6805","nonce":1945833081}]},"public_addr":"172.21.15.113:6801/1945833081","cluster_addr":"172.21.15.113:6803/1945833081","heartbeat_back_addr":"172.21.15.113:6807/1945833081","heartbeat_front_addr":"172.21.15.113:6805/1945833081","state":["exists","up"]},{"osd":5,"uuid":"99a11b81-0df7-4b1d-946a-4297ba62e263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3426360325},{"type":"v1","addr":"172.21.15.113:6809","nonce":3426360325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3426360325},{"type":"v1","addr":"172.21.15.113:6811","nonce":3426360325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3426360325},{"type":"v1","addr":"172.21.15.113:6815","nonce":3426360325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3426360325},{"type":"v1","addr":"172.21.15.113:6813","nonce":3426360325}]},"public_addr":"172.21.15.113:6809/3426360325","cluster_addr":"172.21.15.113:6811/3426360325","heartbeat_back_addr":"172.21.15.113:6815/3426360325","heartbeat_front_addr":"172.21.15.113:6813/3426360325","state":["exists","up"]},{"osd":6,"uuid":"cafd0e26-78e9-4164-a43a-96ca8382fefe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1527017698},{"type":"v1","addr":"172.21.15.113:6817","nonce":1527017698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1527017698},{"type":"v1","addr":"172.21.15.113:6819","nonce":1527017698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1527017698},{"type":"v1","addr":"172.21.15.113:6823","nonce":1527017698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1527017698},{"type":"v1","addr":"172.21.15.113:6821","nonce":1527017698}]},"public_addr":"172.21.15.113:6817/1527017698","cluster_addr":"172.21.15.113:6819/1527017698","heartbeat_back_addr":"172.21.15.113:6823/1527017698","heartbeat_front_addr":"172.21.15.113:6821/1527017698","state":["exists","up"]},{"osd":7,"uuid":"e9d59662-9132-48f4-9e95-842a3cfbd79f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":3182476015},{"type":"v1","addr":"172.21.15.113:6825","nonce":3182476015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":3182476015},{"type":"v1","addr":"172.21.15.113:6827","nonce":3182476015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":3182476015},{"type":"v1","addr":"172.21.15.113:6831","nonce":3182476015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":3182476015},{"type":"v1","addr":"172.21.15.113:6829","nonce":3182476015}]},"public_addr":"172.21.15.113:6825/3182476015","cluster_addr":"172.21.15.113:6827/3182476015","heartbeat_back_addr":"172.21.15.113:6831/3182476015","heartbeat_front_addr":"172.21.15.113:6829/3182476015","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:42:19.457967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:42:31.948677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:43:01.254280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:43:11.968334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:42:19.937695+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:42:54.094975+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:43:09.805291+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:42:28.849417+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.b","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/564868778":"2024-03-31T13:38:09.321740+0000","172.21.15.43:6801/1339921781":"2024-03-31T13:38:09.321740+0000","172.21.15.43:6800/1339921781":"2024-03-31T13:38:09.321740+0000","172.21.15.43:0/886631486":"2024-03-31T13:37:41.733515+0000","172.21.15.43:0/1206218784":"2024-03-31T13:37:41.733515+0000","172.21.15.43:6800/3411723337":"2024-03-31T13:37:41.733515+0000","172.21.15.43:0/2618817647":"2024-03-31T13:38:09.321740+0000","172.21.15.43:0/3656823679":"2024-03-31T13:37:41.733515+0000","172.21.15.43:0/2794494528":"2024-03-31T13:38:09.321740+0000","172.21.15.43:6801/3411723337":"2024-03-31T13:37:41.733515+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:43:20.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:19 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ignoring --setuser ceph since I am not root 2024-03-30T13:43:20.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:19 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ignoring --setgroup ceph since I am not root 2024-03-30T13:43:20.222 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:19 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: ignoring --setuser ceph since I am not root 2024-03-30T13:43:20.222 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:19 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: ignoring --setgroup ceph since I am not root 2024-03-30T13:43:20.222 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:19 smithi113 ceph-mgr[36472]: -- 172.21.15.113:0/2393690797 <== mon.1 v2:172.21.15.113:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55972ef285a0 con 0x55972eefd000 2024-03-30T13:43:20.409 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:20 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:20.376+0000 7fc80f023200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T13:43:20.661 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-30T13:43:20.661 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T13:43:20.686 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:20 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:20.686 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:20 smithi043 ceph-mon[29963]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-30T13:43:20.686 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:20 smithi043 ceph-mon[29963]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-30T13:43:20.686 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:20 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1552114139' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T13:43:20.687 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:20 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:20.475+0000 7fc80f023200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T13:43:20.687 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:20 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:20.687 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:20 smithi043 ceph-mon[42099]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-30T13:43:20.687 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:20 smithi043 ceph-mon[42099]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-30T13:43:20.687 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:20 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1552114139' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T13:43:20.722 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:20.347+0000 7f38e5b2c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T13:43:20.722 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:20.442+0000 7f38e5b2c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T13:43:20.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:20 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' 2024-03-30T13:43:20.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:20 smithi113 ceph-mon[34341]: from='mgr.14152 172.21.15.43:0/1332198408' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-30T13:43:20.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:20 smithi113 ceph-mon[34341]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-30T13:43:20.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:20 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1552114139' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T13:43:20.904 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:43:21.095 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:21.086+0000 7f38e5b2c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T13:43:21.312 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:21 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:21.157+0000 7fc80f023200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T13:43:21.312 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:21 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:21.255+0000 7fc80f023200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T13:43:21.434 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:21.182+0000 7f38e5b2c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T13:43:21.435 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:21.334+0000 7f38e5b2c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T13:43:21.435 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:21.433+0000 7f38e5b2c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T13:43:21.599 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:21 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:21.411+0000 7fc80f023200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T13:43:21.599 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:21 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:21.510+0000 7fc80f023200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T13:43:21.599 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:21 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:21.597+0000 7fc80f023200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T13:43:21.722 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:21.520+0000 7f38e5b2c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T13:43:22.065 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:21.791+0000 7f38e5b2c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T13:43:22.065 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:21.904+0000 7f38e5b2c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T13:43:22.066 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:21.992+0000 7f38e5b2c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T13:43:22.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:21 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:21.870+0000 7fc80f023200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T13:43:22.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:21 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:21.984+0000 7fc80f023200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T13:43:22.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:22 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:22.073+0000 7fc80f023200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T13:43:22.472 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:22.091+0000 7f38e5b2c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T13:43:22.472 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:22.265+0000 7f38e5b2c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T13:43:22.486 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:22 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:22.175+0000 7fc80f023200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T13:43:22.487 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:22 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:22.362+0000 7fc80f023200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T13:43:22.842 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:43:22.842 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":80,"fsid":"7ee932fa-ee9a-11ee-b647-cb9ed24678a4","created":"2024-03-30T13:37:07.346020+0000","modified":"2024-03-30T13:43:22.281623+0000","last_up_change":"2024-03-30T13:42:30.966931+0000","last_in_change":"2024-03-30T13:42:12.831084+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T13:40:27.658979+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-30T13:42:42.989975+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":68,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-30T13:42:43.910306+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-30T13:42:45.633132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-30T13:42:47.570084+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-30T13:42:49.637189+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94ea083e-6a27-41db-83af-2f7d3cae1956","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1144592572},{"type":"v1","addr":"172.21.15.43:6803","nonce":1144592572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1144592572},{"type":"v1","addr":"172.21.15.43:6805","nonce":1144592572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":1144592572},{"type":"v1","addr":"172.21.15.43:6809","nonce":1144592572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1144592572},{"type":"v1","addr":"172.21.15.43:6807","nonce":1144592572}]},"public_addr":"172.21.15.43:6803/1144592572","cluster_addr":"172.21.15.43:6805/1144592572","heartbeat_back_addr":"172.21.15.43:6809/1144592572","heartbeat_front_addr":"172.21.15.43:6807/1144592572","state":["exists","up"]},{"osd":1,"uuid":"8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":3901165928},{"type":"v1","addr":"172.21.15.43:6811","nonce":3901165928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":3901165928},{"type":"v1","addr":"172.21.15.43:6813","nonce":3901165928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":3901165928},{"type":"v1","addr":"172.21.15.43:6817","nonce":3901165928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":3901165928},{"type":"v1","addr":"172.21.15.43:6815","nonce":3901165928}]},"public_addr":"172.21.15.43:6811/3901165928","cluster_addr":"172.21.15.43:6813/3901165928","heartbeat_back_addr":"172.21.15.43:6817/3901165928","heartbeat_front_addr":"172.21.15.43:6815/3901165928","state":["exists","up"]},{"osd":2,"uuid":"6db6782e-5889-443e-bc6c-df47a1f93bc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":3415037523},{"type":"v1","addr":"172.21.15.43:6819","nonce":3415037523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":3415037523},{"type":"v1","addr":"172.21.15.43:6821","nonce":3415037523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":3415037523},{"type":"v1","addr":"172.21.15.43:6825","nonce":3415037523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":3415037523},{"type":"v1","addr":"172.21.15.43:6823","nonce":3415037523}]},"public_addr":"172.21.15.43:6819/3415037523","cluster_addr":"172.21.15.43:6821/3415037523","heartbeat_back_addr":"172.21.15.43:6825/3415037523","heartbeat_front_addr":"172.21.15.43:6823/3415037523","state":["exists","up"]},{"osd":3,"uuid":"7048ff46-2cba-4d18-9835-95ecbddb2bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2933589376},{"type":"v1","addr":"172.21.15.43:6827","nonce":2933589376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2933589376},{"type":"v1","addr":"172.21.15.43:6829","nonce":2933589376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2933589376},{"type":"v1","addr":"172.21.15.43:6833","nonce":2933589376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2933589376},{"type":"v1","addr":"172.21.15.43:6831","nonce":2933589376}]},"public_addr":"172.21.15.43:6827/2933589376","cluster_addr":"172.21.15.43:6829/2933589376","heartbeat_back_addr":"172.21.15.43:6833/2933589376","heartbeat_front_addr":"172.21.15.43:6831/2933589376","state":["exists","up"]},{"osd":4,"uuid":"42da30e6-9eec-4ed5-8c73-9ce78aa8ff63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1945833081},{"type":"v1","addr":"172.21.15.113:6801","nonce":1945833081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1945833081},{"type":"v1","addr":"172.21.15.113:6803","nonce":1945833081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1945833081},{"type":"v1","addr":"172.21.15.113:6807","nonce":1945833081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1945833081},{"type":"v1","addr":"172.21.15.113:6805","nonce":1945833081}]},"public_addr":"172.21.15.113:6801/1945833081","cluster_addr":"172.21.15.113:6803/1945833081","heartbeat_back_addr":"172.21.15.113:6807/1945833081","heartbeat_front_addr":"172.21.15.113:6805/1945833081","state":["exists","up"]},{"osd":5,"uuid":"99a11b81-0df7-4b1d-946a-4297ba62e263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3426360325},{"type":"v1","addr":"172.21.15.113:6809","nonce":3426360325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3426360325},{"type":"v1","addr":"172.21.15.113:6811","nonce":3426360325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3426360325},{"type":"v1","addr":"172.21.15.113:6815","nonce":3426360325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3426360325},{"type":"v1","addr":"172.21.15.113:6813","nonce":3426360325}]},"public_addr":"172.21.15.113:6809/3426360325","cluster_addr":"172.21.15.113:6811/3426360325","heartbeat_back_addr":"172.21.15.113:6815/3426360325","heartbeat_front_addr":"172.21.15.113:6813/3426360325","state":["exists","up"]},{"osd":6,"uuid":"cafd0e26-78e9-4164-a43a-96ca8382fefe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1527017698},{"type":"v1","addr":"172.21.15.113:6817","nonce":1527017698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1527017698},{"type":"v1","addr":"172.21.15.113:6819","nonce":1527017698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1527017698},{"type":"v1","addr":"172.21.15.113:6823","nonce":1527017698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1527017698},{"type":"v1","addr":"172.21.15.113:6821","nonce":1527017698}]},"public_addr":"172.21.15.113:6817/1527017698","cluster_addr":"172.21.15.113:6819/1527017698","heartbeat_back_addr":"172.21.15.113:6823/1527017698","heartbeat_front_addr":"172.21.15.113:6821/1527017698","state":["exists","up"]},{"osd":7,"uuid":"e9d59662-9132-48f4-9e95-842a3cfbd79f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":3182476015},{"type":"v1","addr":"172.21.15.113:6825","nonce":3182476015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":3182476015},{"type":"v1","addr":"172.21.15.113:6827","nonce":3182476015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":3182476015},{"type":"v1","addr":"172.21.15.113:6831","nonce":3182476015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":3182476015},{"type":"v1","addr":"172.21.15.113:6829","nonce":3182476015}]},"public_addr":"172.21.15.113:6825/3182476015","cluster_addr":"172.21.15.113:6827/3182476015","heartbeat_back_addr":"172.21.15.113:6831/3182476015","heartbeat_front_addr":"172.21.15.113:6829/3182476015","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:42:19.457967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:42:31.948677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:43:01.254280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:43:11.968334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:42:19.937695+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:42:54.094975+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:43:09.805291+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:42:28.849417+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.b","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/564868778":"2024-03-31T13:38:09.321740+0000","172.21.15.43:6801/1339921781":"2024-03-31T13:38:09.321740+0000","172.21.15.43:6800/1339921781":"2024-03-31T13:38:09.321740+0000","172.21.15.43:0/886631486":"2024-03-31T13:37:41.733515+0000","172.21.15.43:0/1206218784":"2024-03-31T13:37:41.733515+0000","172.21.15.43:6800/3411723337":"2024-03-31T13:37:41.733515+0000","172.21.15.43:0/2618817647":"2024-03-31T13:38:09.321740+0000","172.21.15.43:0/3656823679":"2024-03-31T13:37:41.733515+0000","172.21.15.43:0/2794494528":"2024-03-31T13:38:09.321740+0000","172.21.15.43:6801/3411723337":"2024-03-31T13:37:41.733515+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:43:23.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:23 smithi043 ceph-mon[29963]: osdmap e80: 8 total, 8 up, 8 in 2024-03-30T13:43:23.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:23 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3425046719' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T13:43:23.544 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:23 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:23.404+0000 7fc80f023200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T13:43:23.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:23 smithi043 ceph-mon[42099]: osdmap e80: 8 total, 8 up, 8 in 2024-03-30T13:43:23.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:23 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3425046719' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T13:43:23.687 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:23 smithi113 ceph-mon[34341]: osdmap e80: 8 total, 8 up, 8 in 2024-03-30T13:43:23.687 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:23 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3425046719' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T13:43:23.687 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:23 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:23.222+0000 7f38e5b2c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T13:43:23.687 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:23 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:23.352+0000 7f38e5b2c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T13:43:23.688 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:23 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:23.686+0000 7f38e5b2c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T13:43:23.702 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-03-30T13:43:23.703 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-03-30T13:43:23.703 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-03-30T13:43:23.703 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-03-30T13:43:23.704 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-03-30T13:43:23.704 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-03-30T13:43:23.704 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-03-30T13:43:23.705 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-03-30T13:43:23.899 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:23 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:23.541+0000 7fc80f023200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T13:43:23.972 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:23 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:23.776+0000 7f38e5b2c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T13:43:24.285 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:23 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:23.972+0000 7f38e5b2c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:43:24.286 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:24 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:24.157+0000 7f38e5b2c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T13:43:24.286 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:23 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:23.972+0000 7fc80f023200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T13:43:24.286 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:24 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:24.143+0000 7fc80f023200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T13:43:24.304 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:43:24.307 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:43:24.308 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:43:24.313 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:43:24.313 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:43:24.318 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:43:24.323 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:43:24.327 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:43:24.601 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:24 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:24.433+0000 7fc80f023200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:43:24.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:24 smithi043 ceph-mon[29963]: 1.0 scrub starts 2024-03-30T13:43:24.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:24 smithi043 ceph-mon[29963]: 1.0 scrub ok 2024-03-30T13:43:24.602 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:24 smithi043 ceph-mon[42099]: 1.0 scrub starts 2024-03-30T13:43:24.602 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:24 smithi043 ceph-mon[42099]: 1.0 scrub ok 2024-03-30T13:43:24.699 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:24 smithi113 ceph-mon[34341]: 1.0 scrub starts 2024-03-30T13:43:24.699 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:24 smithi113 ceph-mon[34341]: 1.0 scrub ok 2024-03-30T13:43:24.699 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:24 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:24.442+0000 7f38e5b2c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:43:24.700 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:24 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:24.698+0000 7f38e5b2c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T13:43:24.879 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:24 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:24.634+0000 7fc80f023200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T13:43:24.972 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:24 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:24.935+0000 7f38e5b2c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T13:43:25.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:24 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:24.936+0000 7fc80f023200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:43:25.460 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:25 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:25.201+0000 7fc80f023200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T13:43:25.726 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:25 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:25.457+0000 7fc80f023200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T13:43:26.222 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:25.730+0000 7f38e5b2c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T13:43:26.591 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:26.319+0000 7fc80f023200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T13:43:26.606 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:26 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:26.295+0000 7f38e5b2c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T13:43:26.606 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:26 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:26.391+0000 7f38e5b2c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T13:43:26.606 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:26 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:26.605+0000 7f38e5b2c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T13:43:26.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:26 smithi043 ceph-mon[29963]: Standby manager daemon x restarted 2024-03-30T13:43:26.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:26 smithi043 ceph-mon[29963]: Standby manager daemon x started 2024-03-30T13:43:26.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:26 smithi043 ceph-mon[29963]: from='mgr.? 172.21.15.113:0/3464392819' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-30T13:43:26.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:26 smithi043 ceph-mon[29963]: from='mgr.? 172.21.15.113:0/3464392819' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T13:43:26.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:26 smithi043 ceph-mon[29963]: from='mgr.? 172.21.15.113:0/3464392819' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-30T13:43:26.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:26 smithi043 ceph-mon[29963]: from='mgr.? 172.21.15.113:0/3464392819' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T13:43:26.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:26 smithi043 ceph-mon[42099]: Standby manager daemon x restarted 2024-03-30T13:43:26.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:26 smithi043 ceph-mon[42099]: Standby manager daemon x started 2024-03-30T13:43:26.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:26 smithi043 ceph-mon[42099]: from='mgr.? 172.21.15.113:0/3464392819' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-30T13:43:26.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:26 smithi043 ceph-mon[42099]: from='mgr.? 172.21.15.113:0/3464392819' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T13:43:26.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:26 smithi043 ceph-mon[42099]: from='mgr.? 172.21.15.113:0/3464392819' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-30T13:43:26.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:26 smithi043 ceph-mon[42099]: from='mgr.? 172.21.15.113:0/3464392819' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T13:43:26.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:26 smithi113 ceph-mon[34341]: Standby manager daemon x restarted 2024-03-30T13:43:26.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:26 smithi113 ceph-mon[34341]: Standby manager daemon x started 2024-03-30T13:43:26.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:26 smithi113 ceph-mon[34341]: from='mgr.? 172.21.15.113:0/3464392819' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-30T13:43:26.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:26 smithi113 ceph-mon[34341]: from='mgr.? 172.21.15.113:0/3464392819' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T13:43:26.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:26 smithi113 ceph-mon[34341]: from='mgr.? 172.21.15.113:0/3464392819' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-30T13:43:26.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:26 smithi113 ceph-mon[34341]: from='mgr.? 172.21.15.113:0/3464392819' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T13:43:26.972 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:26 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: [30/Mar/2024:13:43:26] ENGINE Bus STARTING 2024-03-30T13:43:26.973 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:26 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: CherryPy Checker: 2024-03-30T13:43:26.973 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:26 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: The Application mounted at '' has an empty config. 2024-03-30T13:43:26.973 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:26 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: 2024-03-30T13:43:26.973 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:26 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: [30/Mar/2024:13:43:26] ENGINE Serving on http://:::9283 2024-03-30T13:43:26.973 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 13:43:26 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x[36449]: [30/Mar/2024:13:43:26] ENGINE Bus STARTED 2024-03-30T13:43:27.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:26.922+0000 7fc80f023200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T13:43:27.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:27.024+0000 7fc80f023200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T13:43:27.649 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:27.250+0000 7fc80f023200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T13:43:27.650 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:43:27] ENGINE Bus STARTING 2024-03-30T13:43:27.650 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: CherryPy Checker: 2024-03-30T13:43:27.650 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: The Application mounted at '' has an empty config. 2024-03-30T13:43:27.650 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: 2024-03-30T13:43:27.650 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:43:27] ENGINE Serving on http://:::9283 2024-03-30T13:43:27.650 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:43:27] ENGINE Bus STARTED 2024-03-30T13:43:27.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-30T13:43:27.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: Active manager daemon y restarted 2024-03-30T13:43:27.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: Activating manager daemon y 2024-03-30T13:43:27.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: osdmap e81: 8 total, 8 up, 8 in 2024-03-30T13:43:27.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: mgrmap e19: y(active, starting, since 0.00899369s), standbys: x 2024-03-30T13:43:27.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:43:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:43:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:43:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T13:43:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T13:43:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:43:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:43:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:43:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:43:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:43:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:43:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:43:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:43:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T13:43:27.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T13:43:27.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T13:43:27.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: Manager daemon y is now available 2024-03-30T13:43:27.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: osdmap e82: 8 total, 8 up, 8 in 2024-03-30T13:43:27.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:43:27.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:27.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:43:27.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T13:43:27.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T13:43:28.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-30T13:43:28.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: Active manager daemon y restarted 2024-03-30T13:43:28.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: Activating manager daemon y 2024-03-30T13:43:28.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: osdmap e81: 8 total, 8 up, 8 in 2024-03-30T13:43:28.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: mgrmap e19: y(active, starting, since 0.00899369s), standbys: x 2024-03-30T13:43:28.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:43:28.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:43:28.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:43:28.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T13:43:28.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T13:43:28.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:43:28.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:43:28.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:43:28.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:43:28.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:43:28.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:43:28.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:43:28.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:43:28.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T13:43:28.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T13:43:28.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T13:43:28.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: Manager daemon y is now available 2024-03-30T13:43:28.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: osdmap e82: 8 total, 8 up, 8 in 2024-03-30T13:43:28.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:43:28.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:28.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:43:28.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T13:43:28.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T13:43:28.152 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-30T13:43:28.152 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: Active manager daemon y restarted 2024-03-30T13:43:28.152 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: Activating manager daemon y 2024-03-30T13:43:28.152 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: osdmap e81: 8 total, 8 up, 8 in 2024-03-30T13:43:28.152 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: mgrmap e19: y(active, starting, since 0.00899369s), standbys: x 2024-03-30T13:43:28.152 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:43:28.152 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:43:28.152 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:43:28.152 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T13:43:28.153 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T13:43:28.153 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:43:28.153 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:43:28.153 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:43:28.153 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:43:28.153 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:43:28.153 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:43:28.153 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:43:28.153 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:43:28.153 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T13:43:28.153 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T13:43:28.153 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T13:43:28.153 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: Manager daemon y is now available 2024-03-30T13:43:28.153 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: osdmap e82: 8 total, 8 up, 8 in 2024-03-30T13:43:28.154 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:43:28.154 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:28.154 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:43:28.154 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T13:43:28.154 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T13:43:28.561 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:43:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:43:29.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:29 smithi043 ceph-mon[29963]: [30/Mar/2024:13:43:28] ENGINE Bus STARTING 2024-03-30T13:43:29.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:29 smithi043 ceph-mon[29963]: [30/Mar/2024:13:43:28] ENGINE Serving on http://172.21.15.43:8765 2024-03-30T13:43:29.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:29 smithi043 ceph-mon[29963]: mgrmap e20: y(active, since 1.01239s), standbys: x 2024-03-30T13:43:29.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:29 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:43:29.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:29 smithi043 ceph-mon[29963]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:43:29.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:29 smithi043 ceph-mon[29963]: [30/Mar/2024:13:43:28] ENGINE Serving on https://172.21.15.43:7150 2024-03-30T13:43:29.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:29 smithi043 ceph-mon[29963]: [30/Mar/2024:13:43:28] ENGINE Bus STARTED 2024-03-30T13:43:29.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:29 smithi043 ceph-mon[42099]: [30/Mar/2024:13:43:28] ENGINE Bus STARTING 2024-03-30T13:43:29.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:29 smithi043 ceph-mon[42099]: [30/Mar/2024:13:43:28] ENGINE Serving on http://172.21.15.43:8765 2024-03-30T13:43:29.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:29 smithi043 ceph-mon[42099]: mgrmap e20: y(active, since 1.01239s), standbys: x 2024-03-30T13:43:29.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:29 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:43:29.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:29 smithi043 ceph-mon[42099]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:43:29.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:29 smithi043 ceph-mon[42099]: [30/Mar/2024:13:43:28] ENGINE Serving on https://172.21.15.43:7150 2024-03-30T13:43:29.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:29 smithi043 ceph-mon[42099]: [30/Mar/2024:13:43:28] ENGINE Bus STARTED 2024-03-30T13:43:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:29 smithi113 ceph-mon[34341]: [30/Mar/2024:13:43:28] ENGINE Bus STARTING 2024-03-30T13:43:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:29 smithi113 ceph-mon[34341]: [30/Mar/2024:13:43:28] ENGINE Serving on http://172.21.15.43:8765 2024-03-30T13:43:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:29 smithi113 ceph-mon[34341]: mgrmap e20: y(active, since 1.01239s), standbys: x 2024-03-30T13:43:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:29 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:43:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:29 smithi113 ceph-mon[34341]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:43:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:29 smithi113 ceph-mon[34341]: [30/Mar/2024:13:43:28] ENGINE Serving on https://172.21.15.43:7150 2024-03-30T13:43:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:29 smithi113 ceph-mon[34341]: [30/Mar/2024:13:43:28] ENGINE Bus STARTED 2024-03-30T13:43:30.389 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:30 smithi043 ceph-mon[29963]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:43:30.389 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:30 smithi043 ceph-mon[29963]: mgrmap e21: y(active, since 2s), standbys: x 2024-03-30T13:43:30.390 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:30 smithi043 ceph-mon[42099]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:43:30.390 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:30 smithi043 ceph-mon[42099]: mgrmap e21: y(active, since 2s), standbys: x 2024-03-30T13:43:30.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:30 smithi113 ceph-mon[34341]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:43:30.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:30 smithi113 ceph-mon[34341]: mgrmap e21: y(active, since 2s), standbys: x 2024-03-30T13:43:31.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:31 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:31.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:31 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:31.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:31 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:31.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:31 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:31.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:31 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:31.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:31 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:32.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[29963]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:43:32.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[29963]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-30T13:43:32.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:32.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:32.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:32.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:32.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:32.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:32.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[29963]: Adjusting osd_memory_target on smithi113 to 3760M 2024-03-30T13:43:32.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:32.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[29963]: osdmap e83: 8 total, 8 up, 8 in 2024-03-30T13:43:32.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[42099]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:43:32.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[42099]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-30T13:43:32.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:32.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:32.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:32.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:32.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:32.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:32.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[42099]: Adjusting osd_memory_target on smithi113 to 3760M 2024-03-30T13:43:32.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:32.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:32 smithi043 ceph-mon[42099]: osdmap e83: 8 total, 8 up, 8 in 2024-03-30T13:43:32.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:32 smithi113 ceph-mon[34341]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:43:32.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:32 smithi113 ceph-mon[34341]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-30T13:43:32.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:32 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:32.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:32 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:32.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:32 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:32.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:32 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:32.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:32 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:32.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:32 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:32.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:32 smithi113 ceph-mon[34341]: Adjusting osd_memory_target on smithi113 to 3760M 2024-03-30T13:43:32.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:32 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:32.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:32 smithi113 ceph-mon[34341]: osdmap e83: 8 total, 8 up, 8 in 2024-03-30T13:43:33.649 INFO:teuthology.orchestra.run.smithi043.stdout:34359738417 2024-03-30T13:43:33.649 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-03-30T13:43:34.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:34 smithi043 ceph-mon[29963]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-30T13:43:34.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:34 smithi043 ceph-mon[42099]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-30T13:43:34.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:34 smithi113 ceph-mon[34341]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-30T13:43:36.349 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:43:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:43:36.384 INFO:teuthology.orchestra.run.smithi043.stdout:107374182434 2024-03-30T13:43:36.384 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-03-30T13:43:36.599 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:36 smithi043 ceph-mon[29963]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-30T13:43:36.599 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:36 smithi043 ceph-mon[42099]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-30T13:43:36.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:36 smithi113 ceph-mon[34341]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-30T13:43:37.096 INFO:teuthology.orchestra.run.smithi043.stdout:249108103183 2024-03-30T13:43:37.096 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-03-30T13:43:37.790 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:37 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:43:37.791 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:37 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:43:37.791 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:37 smithi043 ceph-mon[29963]: osdmap e84: 8 total, 8 up, 8 in 2024-03-30T13:43:37.791 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:37 smithi043 ceph-mon[29963]: osdmap e85: 8 total, 8 up, 8 in 2024-03-30T13:43:37.791 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:37 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:43:37.791 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:37 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:43:37.791 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:37 smithi043 ceph-mon[42099]: osdmap e84: 8 total, 8 up, 8 in 2024-03-30T13:43:37.791 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:37 smithi043 ceph-mon[42099]: osdmap e85: 8 total, 8 up, 8 in 2024-03-30T13:43:37.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:37 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:43:37.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:37 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:43:37.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:37 smithi113 ceph-mon[34341]: osdmap e84: 8 total, 8 up, 8 in 2024-03-30T13:43:37.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:37 smithi113 ceph-mon[34341]: osdmap e85: 8 total, 8 up, 8 in 2024-03-30T13:43:38.603 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:43:38 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:43:38.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:38 smithi043 ceph-mon[42099]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-30T13:43:38.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:38 smithi043 ceph-mon[29963]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-30T13:43:38.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:38 smithi113 ceph-mon[34341]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-30T13:43:39.669 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:43:39.759 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:39 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:43:39.760 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:39 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:43:39.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:39 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:43:40.164 INFO:teuthology.orchestra.run.smithi043.stdout:55834574894 2024-03-30T13:43:40.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-03-30T13:43:40.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:40 smithi043 ceph-mon[29963]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 18 op/s 2024-03-30T13:43:40.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:40 smithi043 ceph-mon[42099]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 18 op/s 2024-03-30T13:43:40.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:40 smithi113 ceph-mon[34341]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 18 op/s 2024-03-30T13:43:41.720 INFO:teuthology.orchestra.run.smithi043.stdout:77309411369 2024-03-30T13:43:41.720 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-03-30T13:43:42.350 INFO:teuthology.orchestra.run.smithi043.stdout:171798691864 2024-03-30T13:43:42.350 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-03-30T13:43:42.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:42 smithi043 ceph-mon[29963]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:43:42.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:42 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:43:42.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:42 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:43:42.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:42 smithi043 ceph-mon[29963]: osdmap e86: 8 total, 8 up, 8 in 2024-03-30T13:43:42.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:42 smithi043 ceph-mon[29963]: osdmap e87: 8 total, 8 up, 8 in 2024-03-30T13:43:42.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:42 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:43:42.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:42 smithi043 ceph-mon[42099]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:43:42.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:42 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:43:42.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:42 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:43:42.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:42 smithi043 ceph-mon[42099]: osdmap e86: 8 total, 8 up, 8 in 2024-03-30T13:43:42.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:42 smithi043 ceph-mon[42099]: osdmap e87: 8 total, 8 up, 8 in 2024-03-30T13:43:42.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:42 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:43:42.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:42 smithi113 ceph-mon[34341]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:43:42.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:42 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:43:42.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:42 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:43:42.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:42 smithi113 ceph-mon[34341]: osdmap e86: 8 total, 8 up, 8 in 2024-03-30T13:43:42.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:42 smithi113 ceph-mon[34341]: osdmap e87: 8 total, 8 up, 8 in 2024-03-30T13:43:42.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:42 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:43:43.244 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:43:43.244 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:43:43.282 INFO:teuthology.orchestra.run.smithi043.stdout:210453397523 2024-03-30T13:43:43.283 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-03-30T13:43:43.918 INFO:teuthology.orchestra.run.smithi043.stdout:137438953502 2024-03-30T13:43:43.918 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-03-30T13:43:44.662 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:43:44.872 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:44 smithi043 ceph-mon[42099]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T13:43:44.873 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:44 smithi043 ceph-mon[29963]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T13:43:44.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:44 smithi113 ceph-mon[34341]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T13:43:46.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:43:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:43:46.450 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:43:46.849 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:46 smithi043 ceph-mon[29963]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T13:43:46.849 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:46 smithi043 ceph-mon[29963]: 3.f scrub starts 2024-03-30T13:43:46.849 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:46 smithi043 ceph-mon[29963]: 3.f scrub ok 2024-03-30T13:43:46.849 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:46 smithi043 ceph-mon[42099]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T13:43:46.849 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:46 smithi043 ceph-mon[42099]: 3.f scrub starts 2024-03-30T13:43:46.849 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:46 smithi043 ceph-mon[42099]: 3.f scrub ok 2024-03-30T13:43:46.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:46 smithi113 ceph-mon[34341]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T13:43:46.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:46 smithi113 ceph-mon[34341]: 3.f scrub starts 2024-03-30T13:43:46.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:46 smithi113 ceph-mon[34341]: 3.f scrub ok 2024-03-30T13:43:48.621 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:43:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:43:48.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:48 smithi043 ceph-mon[29963]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:43:48.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:48 smithi043 ceph-mon[29963]: 3.19 scrub starts 2024-03-30T13:43:48.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:48 smithi043 ceph-mon[29963]: 3.19 scrub ok 2024-03-30T13:43:48.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:48 smithi043 ceph-mon[42099]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:43:48.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:48 smithi043 ceph-mon[42099]: 3.19 scrub starts 2024-03-30T13:43:48.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:48 smithi043 ceph-mon[42099]: 3.19 scrub ok 2024-03-30T13:43:48.692 INFO:teuthology.orchestra.run.smithi043.stdout:107374182437 2024-03-30T13:43:48.915 INFO:teuthology.orchestra.run.smithi043.stdout:249108103185 2024-03-30T13:43:48.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:48 smithi113 ceph-mon[34341]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:43:48.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:48 smithi113 ceph-mon[34341]: 3.19 scrub starts 2024-03-30T13:43:48.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:48 smithi113 ceph-mon[34341]: 3.19 scrub ok 2024-03-30T13:43:49.035 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:43:49.475 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:43:49.776 INFO:teuthology.orchestra.run.smithi043.stdout:34359738420 2024-03-30T13:43:49.873 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:49 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:43:49.874 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:49 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2140277534' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T13:43:49.874 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:49 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/608956383' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T13:43:49.874 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:49 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:49.874 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:49 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:49.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:49 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:43:49.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:49 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2140277534' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T13:43:49.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:49 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/608956383' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T13:43:49.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:49 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:49.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:49 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:49.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:49 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:43:49.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:49 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2140277534' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T13:43:49.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:49 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/608956383' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T13:43:49.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:49 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:49.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:49 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:50.530 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738417 got 34359738420 for osd.0 2024-03-30T13:43:50.530 DEBUG:teuthology.parallel:result is None 2024-03-30T13:43:50.760 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:50 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:43:50.760 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:50 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:43:50.760 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:50 smithi043 ceph-mon[29963]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:43:50.760 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:50 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3457076674' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T13:43:50.760 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:50 smithi043 ceph-mon[29963]: 3.b scrub starts 2024-03-30T13:43:50.760 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:50 smithi043 ceph-mon[29963]: 3.b scrub ok 2024-03-30T13:43:50.760 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:50 smithi043 ceph-mon[29963]: osdmap e88: 8 total, 8 up, 8 in 2024-03-30T13:43:50.761 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:50 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:43:50.761 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:50 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:43:50.761 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:50 smithi043 ceph-mon[42099]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:43:50.761 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:50 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3457076674' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T13:43:50.761 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:50 smithi043 ceph-mon[42099]: 3.b scrub starts 2024-03-30T13:43:50.761 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:50 smithi043 ceph-mon[42099]: 3.b scrub ok 2024-03-30T13:43:50.761 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:50 smithi043 ceph-mon[42099]: osdmap e88: 8 total, 8 up, 8 in 2024-03-30T13:43:50.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:50 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:43:50.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:50 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:43:50.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:50 smithi113 ceph-mon[34341]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:43:50.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:50 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3457076674' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T13:43:50.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:50 smithi113 ceph-mon[34341]: 3.b scrub starts 2024-03-30T13:43:50.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:50 smithi113 ceph-mon[34341]: 3.b scrub ok 2024-03-30T13:43:50.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:50 smithi113 ceph-mon[34341]: osdmap e88: 8 total, 8 up, 8 in 2024-03-30T13:43:51.523 INFO:teuthology.orchestra.run.smithi043.stdout:55834574896 2024-03-30T13:43:51.719 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[29963]: 3.9 scrub starts 2024-03-30T13:43:51.719 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[29963]: 3.9 scrub ok 2024-03-30T13:43:51.719 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:51.719 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:51.719 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:51.719 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:51.719 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:51.719 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:51.719 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[29963]: Adjusting osd_memory_target on smithi043 to 3761M 2024-03-30T13:43:51.719 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:51.720 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:43:51.720 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:43:51.720 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[29963]: Updating smithi043:/etc/ceph/ceph.conf 2024-03-30T13:43:51.720 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[29963]: Updating smithi113:/etc/ceph/ceph.conf 2024-03-30T13:43:51.720 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[29963]: 5.d scrub starts 2024-03-30T13:43:51.720 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[29963]: 5.d scrub ok 2024-03-30T13:43:51.720 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/449714118' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T13:43:51.720 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[42099]: 3.9 scrub starts 2024-03-30T13:43:51.720 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[42099]: 3.9 scrub ok 2024-03-30T13:43:51.720 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:51.720 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:51.721 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:51.721 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:51.721 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:51.721 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:51.721 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[42099]: Adjusting osd_memory_target on smithi043 to 3761M 2024-03-30T13:43:51.721 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:51.721 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:43:51.721 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:43:51.721 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[42099]: Updating smithi043:/etc/ceph/ceph.conf 2024-03-30T13:43:51.721 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[42099]: Updating smithi113:/etc/ceph/ceph.conf 2024-03-30T13:43:51.721 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[42099]: 5.d scrub starts 2024-03-30T13:43:51.721 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[42099]: 5.d scrub ok 2024-03-30T13:43:51.721 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:51 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/449714118' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T13:43:51.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:51 smithi113 ceph-mon[34341]: 3.9 scrub starts 2024-03-30T13:43:51.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:51 smithi113 ceph-mon[34341]: 3.9 scrub ok 2024-03-30T13:43:51.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:51 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:51.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:51 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:51.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:51 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:51.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:51 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:51.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:51 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:51.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:51 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:43:51.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:51 smithi113 ceph-mon[34341]: Adjusting osd_memory_target on smithi043 to 3761M 2024-03-30T13:43:51.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:51 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:51.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:51 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:43:51.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:51 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:43:51.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:51 smithi113 ceph-mon[34341]: Updating smithi043:/etc/ceph/ceph.conf 2024-03-30T13:43:51.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:51 smithi113 ceph-mon[34341]: Updating smithi113:/etc/ceph/ceph.conf 2024-03-30T13:43:51.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:51 smithi113 ceph-mon[34341]: 5.d scrub starts 2024-03-30T13:43:51.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:51 smithi113 ceph-mon[34341]: 5.d scrub ok 2024-03-30T13:43:51.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:51 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/449714118' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T13:43:52.812 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[29963]: 4.1b deep-scrub starts 2024-03-30T13:43:52.812 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[29963]: 4.1b deep-scrub ok 2024-03-30T13:43:52.812 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[29963]: Updating smithi043:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:43:52.812 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[29963]: Updating smithi113:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:43:52.813 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[29963]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T13:43:52.813 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[29963]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:43:52.813 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[29963]: Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:43:52.813 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[29963]: Updating smithi043:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T13:43:52.813 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[29963]: Updating smithi113:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T13:43:52.813 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:52.813 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:52.813 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[29963]: 5.a scrub starts 2024-03-30T13:43:52.813 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[29963]: 5.a scrub ok 2024-03-30T13:43:52.813 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:52.813 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:52.813 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:52.813 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[29963]: 4.1d deep-scrub starts 2024-03-30T13:43:52.813 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[29963]: 4.1d deep-scrub ok 2024-03-30T13:43:52.814 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[29963]: osdmap e89: 8 total, 8 up, 8 in 2024-03-30T13:43:52.814 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[42099]: 4.1b deep-scrub starts 2024-03-30T13:43:52.814 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[42099]: 4.1b deep-scrub ok 2024-03-30T13:43:52.814 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[42099]: Updating smithi043:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:43:52.814 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[42099]: Updating smithi113:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:43:52.814 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[42099]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T13:43:52.814 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[42099]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:43:52.814 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[42099]: Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:43:52.814 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[42099]: Updating smithi043:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T13:43:52.814 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[42099]: Updating smithi113:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T13:43:52.814 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:52.815 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:52.815 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[42099]: 5.a scrub starts 2024-03-30T13:43:52.815 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[42099]: 5.a scrub ok 2024-03-30T13:43:52.815 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:52.815 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:52.815 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:52.815 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[42099]: 4.1d deep-scrub starts 2024-03-30T13:43:52.815 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[42099]: 4.1d deep-scrub ok 2024-03-30T13:43:52.815 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:52 smithi043 ceph-mon[42099]: osdmap e89: 8 total, 8 up, 8 in 2024-03-30T13:43:52.839 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574894 got 55834574896 for osd.1 2024-03-30T13:43:52.840 DEBUG:teuthology.parallel:result is None 2024-03-30T13:43:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:52 smithi113 ceph-mon[34341]: 4.1b deep-scrub starts 2024-03-30T13:43:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:52 smithi113 ceph-mon[34341]: 4.1b deep-scrub ok 2024-03-30T13:43:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:52 smithi113 ceph-mon[34341]: Updating smithi043:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:43:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:52 smithi113 ceph-mon[34341]: Updating smithi113:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:43:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:52 smithi113 ceph-mon[34341]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T13:43:52.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:52 smithi113 ceph-mon[34341]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:43:52.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:52 smithi113 ceph-mon[34341]: Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:43:52.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:52 smithi113 ceph-mon[34341]: Updating smithi043:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T13:43:52.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:52 smithi113 ceph-mon[34341]: Updating smithi113:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T13:43:52.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:52 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:52.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:52 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:52.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:52 smithi113 ceph-mon[34341]: 5.a scrub starts 2024-03-30T13:43:52.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:52 smithi113 ceph-mon[34341]: 5.a scrub ok 2024-03-30T13:43:52.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:52 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:52.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:52 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:52.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:52 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:43:52.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:52 smithi113 ceph-mon[34341]: 4.1d deep-scrub starts 2024-03-30T13:43:52.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:52 smithi113 ceph-mon[34341]: 4.1d deep-scrub ok 2024-03-30T13:43:52.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:52 smithi113 ceph-mon[34341]: osdmap e89: 8 total, 8 up, 8 in 2024-03-30T13:43:53.217 INFO:tasks.cephadm.ceph_manager.ceph:need seq 249108103183 got 249108103185 for osd.7 2024-03-30T13:43:53.218 DEBUG:teuthology.parallel:result is None 2024-03-30T13:43:53.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:53 smithi043 ceph-mon[29963]: Deploying daemon alertmanager.a on smithi043 2024-03-30T13:43:53.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:53 smithi043 ceph-mon[29963]: 4.9 scrub starts 2024-03-30T13:43:53.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:53 smithi043 ceph-mon[29963]: 4.9 scrub ok 2024-03-30T13:43:53.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:53 smithi043 ceph-mon[29963]: 3.15 deep-scrub starts 2024-03-30T13:43:53.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:53 smithi043 ceph-mon[29963]: 3.15 deep-scrub ok 2024-03-30T13:43:53.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:53 smithi043 ceph-mon[29963]: 6.14 scrub starts 2024-03-30T13:43:53.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:53 smithi043 ceph-mon[29963]: 6.14 scrub ok 2024-03-30T13:43:53.697 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:53 smithi043 ceph-mon[29963]: 4.15 scrub starts 2024-03-30T13:43:53.697 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:53 smithi043 ceph-mon[29963]: 4.15 scrub ok 2024-03-30T13:43:53.697 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:53 smithi043 ceph-mon[42099]: Deploying daemon alertmanager.a on smithi043 2024-03-30T13:43:53.697 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:53 smithi043 ceph-mon[42099]: 4.9 scrub starts 2024-03-30T13:43:53.697 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:53 smithi043 ceph-mon[42099]: 4.9 scrub ok 2024-03-30T13:43:53.697 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:53 smithi043 ceph-mon[42099]: 3.15 deep-scrub starts 2024-03-30T13:43:53.697 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:53 smithi043 ceph-mon[42099]: 3.15 deep-scrub ok 2024-03-30T13:43:53.697 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:53 smithi043 ceph-mon[42099]: 6.14 scrub starts 2024-03-30T13:43:53.697 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:53 smithi043 ceph-mon[42099]: 6.14 scrub ok 2024-03-30T13:43:53.697 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:53 smithi043 ceph-mon[42099]: 4.15 scrub starts 2024-03-30T13:43:53.697 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:53 smithi043 ceph-mon[42099]: 4.15 scrub ok 2024-03-30T13:43:53.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:53 smithi113 ceph-mon[34341]: Deploying daemon alertmanager.a on smithi043 2024-03-30T13:43:53.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:53 smithi113 ceph-mon[34341]: 4.9 scrub starts 2024-03-30T13:43:53.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:53 smithi113 ceph-mon[34341]: 4.9 scrub ok 2024-03-30T13:43:53.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:53 smithi113 ceph-mon[34341]: 3.15 deep-scrub starts 2024-03-30T13:43:53.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:53 smithi113 ceph-mon[34341]: 3.15 deep-scrub ok 2024-03-30T13:43:53.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:53 smithi113 ceph-mon[34341]: 6.14 scrub starts 2024-03-30T13:43:53.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:53 smithi113 ceph-mon[34341]: 6.14 scrub ok 2024-03-30T13:43:53.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:53 smithi113 ceph-mon[34341]: 4.15 scrub starts 2024-03-30T13:43:53.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:53 smithi113 ceph-mon[34341]: 4.15 scrub ok 2024-03-30T13:43:54.864 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:43:54.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:54 smithi043 ceph-mon[42099]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:43:54.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:54 smithi043 ceph-mon[42099]: 5.11 scrub starts 2024-03-30T13:43:54.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:54 smithi043 ceph-mon[42099]: 5.11 scrub ok 2024-03-30T13:43:54.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:54 smithi043 ceph-mon[42099]: 4.12 scrub starts 2024-03-30T13:43:54.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:54 smithi043 ceph-mon[42099]: 4.12 scrub ok 2024-03-30T13:43:54.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:54 smithi043 ceph-mon[29963]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:43:54.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:54 smithi043 ceph-mon[29963]: 5.11 scrub starts 2024-03-30T13:43:54.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:54 smithi043 ceph-mon[29963]: 5.11 scrub ok 2024-03-30T13:43:54.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:54 smithi043 ceph-mon[29963]: 4.12 scrub starts 2024-03-30T13:43:54.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:54 smithi043 ceph-mon[29963]: 4.12 scrub ok 2024-03-30T13:43:54.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:54 smithi113 ceph-mon[34341]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:43:54.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:54 smithi113 ceph-mon[34341]: 5.11 scrub starts 2024-03-30T13:43:54.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:54 smithi113 ceph-mon[34341]: 5.11 scrub ok 2024-03-30T13:43:54.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:54 smithi113 ceph-mon[34341]: 4.12 scrub starts 2024-03-30T13:43:54.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:54 smithi113 ceph-mon[34341]: 4.12 scrub ok 2024-03-30T13:43:55.574 INFO:teuthology.orchestra.run.smithi043.stdout:77309411372 2024-03-30T13:43:55.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:55 smithi043 ceph-mon[42099]: 5.5 scrub starts 2024-03-30T13:43:55.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:55 smithi043 ceph-mon[42099]: 5.5 scrub ok 2024-03-30T13:43:55.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:55 smithi043 ceph-mon[42099]: 4.1f scrub starts 2024-03-30T13:43:55.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:55 smithi043 ceph-mon[42099]: 4.1f scrub ok 2024-03-30T13:43:55.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:55 smithi043 ceph-mon[42099]: 4.11 scrub starts 2024-03-30T13:43:55.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:55 smithi043 ceph-mon[42099]: 4.11 scrub ok 2024-03-30T13:43:55.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:55 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2086736689' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T13:43:55.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:55 smithi043 ceph-mon[29963]: 5.5 scrub starts 2024-03-30T13:43:55.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:55 smithi043 ceph-mon[29963]: 5.5 scrub ok 2024-03-30T13:43:55.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:55 smithi043 ceph-mon[29963]: 4.1f scrub starts 2024-03-30T13:43:55.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:55 smithi043 ceph-mon[29963]: 4.1f scrub ok 2024-03-30T13:43:55.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:55 smithi043 ceph-mon[29963]: 4.11 scrub starts 2024-03-30T13:43:55.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:55 smithi043 ceph-mon[29963]: 4.11 scrub ok 2024-03-30T13:43:55.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:55 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2086736689' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T13:43:55.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:55 smithi113 ceph-mon[34341]: 5.5 scrub starts 2024-03-30T13:43:55.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:55 smithi113 ceph-mon[34341]: 5.5 scrub ok 2024-03-30T13:43:55.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:55 smithi113 ceph-mon[34341]: 4.1f scrub starts 2024-03-30T13:43:55.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:55 smithi113 ceph-mon[34341]: 4.1f scrub ok 2024-03-30T13:43:55.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:55 smithi113 ceph-mon[34341]: 4.11 scrub starts 2024-03-30T13:43:55.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:55 smithi113 ceph-mon[34341]: 4.11 scrub ok 2024-03-30T13:43:55.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:55 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2086736689' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T13:43:56.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:43:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:43:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:43:56.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:56 smithi043 ceph-mon[29963]: 4.13 scrub starts 2024-03-30T13:43:56.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:56 smithi043 ceph-mon[29963]: 4.13 scrub ok 2024-03-30T13:43:56.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:56 smithi043 ceph-mon[29963]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:43:56.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:56 smithi043 ceph-mon[29963]: 4.10 scrub starts 2024-03-30T13:43:56.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:56 smithi043 ceph-mon[29963]: 4.10 scrub ok 2024-03-30T13:43:56.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:56 smithi043 ceph-mon[42099]: 4.13 scrub starts 2024-03-30T13:43:56.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:56 smithi043 ceph-mon[42099]: 4.13 scrub ok 2024-03-30T13:43:56.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:56 smithi043 ceph-mon[42099]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:43:56.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:56 smithi043 ceph-mon[42099]: 4.10 scrub starts 2024-03-30T13:43:56.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:56 smithi043 ceph-mon[42099]: 4.10 scrub ok 2024-03-30T13:43:56.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:56 smithi113 ceph-mon[34341]: 4.13 scrub starts 2024-03-30T13:43:56.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:56 smithi113 ceph-mon[34341]: 4.13 scrub ok 2024-03-30T13:43:56.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:56 smithi113 ceph-mon[34341]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:43:56.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:56 smithi113 ceph-mon[34341]: 4.10 scrub starts 2024-03-30T13:43:56.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:56 smithi113 ceph-mon[34341]: 4.10 scrub ok 2024-03-30T13:43:57.282 INFO:teuthology.orchestra.run.smithi043.stdout:210453397527 2024-03-30T13:43:57.851 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411369 got 77309411372 for osd.2 2024-03-30T13:43:57.851 DEBUG:teuthology.parallel:result is None 2024-03-30T13:43:57.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:57 smithi043 ceph-mon[29963]: 5.9 scrub starts 2024-03-30T13:43:57.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:57 smithi043 ceph-mon[29963]: 5.9 scrub ok 2024-03-30T13:43:57.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:57 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/188816603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T13:43:57.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:57 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:43:57.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:57 smithi043 ceph-mon[42099]: 5.9 scrub starts 2024-03-30T13:43:57.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:57 smithi043 ceph-mon[42099]: 5.9 scrub ok 2024-03-30T13:43:57.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:57 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/188816603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T13:43:57.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:57 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:43:57.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:57 smithi113 ceph-mon[34341]: 5.9 scrub starts 2024-03-30T13:43:57.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:57 smithi113 ceph-mon[34341]: 5.9 scrub ok 2024-03-30T13:43:57.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:57 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/188816603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T13:43:57.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:57 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:43:58.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:43:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:43:58.657 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397523 got 210453397527 for osd.6 2024-03-30T13:43:58.657 DEBUG:teuthology.parallel:result is None 2024-03-30T13:43:58.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:58 smithi043 ceph-mon[29963]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:43:58.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:58 smithi043 ceph-mon[29963]: 3.5 deep-scrub starts 2024-03-30T13:43:58.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:58 smithi043 ceph-mon[29963]: 3.5 deep-scrub ok 2024-03-30T13:43:58.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:58 smithi043 ceph-mon[29963]: 4.b scrub starts 2024-03-30T13:43:58.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:58 smithi043 ceph-mon[29963]: 4.b scrub ok 2024-03-30T13:43:58.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:58 smithi043 ceph-mon[29963]: 2.1 scrub starts 2024-03-30T13:43:58.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:58 smithi043 ceph-mon[29963]: 2.1 scrub ok 2024-03-30T13:43:58.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:58 smithi043 ceph-mon[42099]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:43:58.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:58 smithi043 ceph-mon[42099]: 3.5 deep-scrub starts 2024-03-30T13:43:58.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:58 smithi043 ceph-mon[42099]: 3.5 deep-scrub ok 2024-03-30T13:43:58.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:58 smithi043 ceph-mon[42099]: 4.b scrub starts 2024-03-30T13:43:58.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:58 smithi043 ceph-mon[42099]: 4.b scrub ok 2024-03-30T13:43:58.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:58 smithi043 ceph-mon[42099]: 2.1 scrub starts 2024-03-30T13:43:58.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:58 smithi043 ceph-mon[42099]: 2.1 scrub ok 2024-03-30T13:43:58.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:58 smithi113 ceph-mon[34341]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:43:58.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:58 smithi113 ceph-mon[34341]: 3.5 deep-scrub starts 2024-03-30T13:43:58.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:58 smithi113 ceph-mon[34341]: 3.5 deep-scrub ok 2024-03-30T13:43:58.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:58 smithi113 ceph-mon[34341]: 4.b scrub starts 2024-03-30T13:43:58.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:58 smithi113 ceph-mon[34341]: 4.b scrub ok 2024-03-30T13:43:58.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:58 smithi113 ceph-mon[34341]: 2.1 scrub starts 2024-03-30T13:43:58.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:58 smithi113 ceph-mon[34341]: 2.1 scrub ok 2024-03-30T13:43:59.403 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182434 got 107374182437 for osd.3 2024-03-30T13:43:59.403 DEBUG:teuthology.parallel:result is None 2024-03-30T13:43:59.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:59 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:43:59.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:59 smithi043 ceph-mon[42099]: 3.17 scrub starts 2024-03-30T13:43:59.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:59 smithi043 ceph-mon[42099]: 3.17 scrub ok 2024-03-30T13:43:59.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:59 smithi043 ceph-mon[42099]: 5.1b scrub starts 2024-03-30T13:43:59.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:43:59 smithi043 ceph-mon[42099]: 5.1b scrub ok 2024-03-30T13:43:59.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:59 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:43:59.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:59 smithi043 ceph-mon[29963]: 3.17 scrub starts 2024-03-30T13:43:59.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:59 smithi043 ceph-mon[29963]: 3.17 scrub ok 2024-03-30T13:43:59.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:59 smithi043 ceph-mon[29963]: 5.1b scrub starts 2024-03-30T13:43:59.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:43:59 smithi043 ceph-mon[29963]: 5.1b scrub ok 2024-03-30T13:43:59.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:59 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:43:59.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:59 smithi113 ceph-mon[34341]: 3.17 scrub starts 2024-03-30T13:43:59.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:59 smithi113 ceph-mon[34341]: 3.17 scrub ok 2024-03-30T13:43:59.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:59 smithi113 ceph-mon[34341]: 5.1b scrub starts 2024-03-30T13:43:59.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:43:59 smithi113 ceph-mon[34341]: 5.1b scrub ok 2024-03-30T13:44:00.306 INFO:teuthology.orchestra.run.smithi043.stdout:171798691869 2024-03-30T13:44:00.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:00 smithi043 ceph-mon[29963]: 5.e scrub starts 2024-03-30T13:44:00.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:00 smithi043 ceph-mon[29963]: 5.e scrub ok 2024-03-30T13:44:00.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:00 smithi043 ceph-mon[29963]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T13:44:00.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:00 smithi043 ceph-mon[29963]: 6.12 scrub starts 2024-03-30T13:44:00.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:00 smithi043 ceph-mon[29963]: 6.12 scrub ok 2024-03-30T13:44:00.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:00 smithi043 ceph-mon[29963]: 3.0 deep-scrub starts 2024-03-30T13:44:00.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:00 smithi043 ceph-mon[29963]: 3.0 deep-scrub ok 2024-03-30T13:44:00.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:00 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2084915892' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T13:44:00.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:00 smithi043 ceph-mon[42099]: 5.e scrub starts 2024-03-30T13:44:00.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:00 smithi043 ceph-mon[42099]: 5.e scrub ok 2024-03-30T13:44:00.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:00 smithi043 ceph-mon[42099]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T13:44:00.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:00 smithi043 ceph-mon[42099]: 6.12 scrub starts 2024-03-30T13:44:00.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:00 smithi043 ceph-mon[42099]: 6.12 scrub ok 2024-03-30T13:44:00.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:00 smithi043 ceph-mon[42099]: 3.0 deep-scrub starts 2024-03-30T13:44:00.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:00 smithi043 ceph-mon[42099]: 3.0 deep-scrub ok 2024-03-30T13:44:00.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:00 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2084915892' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T13:44:00.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:00 smithi113 ceph-mon[34341]: 5.e scrub starts 2024-03-30T13:44:00.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:00 smithi113 ceph-mon[34341]: 5.e scrub ok 2024-03-30T13:44:00.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:00 smithi113 ceph-mon[34341]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T13:44:00.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:00 smithi113 ceph-mon[34341]: 6.12 scrub starts 2024-03-30T13:44:00.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:00 smithi113 ceph-mon[34341]: 6.12 scrub ok 2024-03-30T13:44:00.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:00 smithi113 ceph-mon[34341]: 3.0 deep-scrub starts 2024-03-30T13:44:00.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:00 smithi113 ceph-mon[34341]: 3.0 deep-scrub ok 2024-03-30T13:44:00.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:00 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2084915892' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T13:44:01.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:01 smithi113 ceph-mon[34341]: 4.16 scrub starts 2024-03-30T13:44:01.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:01 smithi113 ceph-mon[34341]: 4.16 scrub ok 2024-03-30T13:44:02.039 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:01 smithi043 ceph-mon[29963]: 4.16 scrub starts 2024-03-30T13:44:02.039 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:01 smithi043 ceph-mon[29963]: 4.16 scrub ok 2024-03-30T13:44:02.040 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:01 smithi043 ceph-mon[42099]: 4.16 scrub starts 2024-03-30T13:44:02.040 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:01 smithi043 ceph-mon[42099]: 4.16 scrub ok 2024-03-30T13:44:02.118 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691864 got 171798691869 for osd.5 2024-03-30T13:44:02.119 DEBUG:teuthology.parallel:result is None 2024-03-30T13:44:02.848 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:02 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:44:02.848 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:02 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:44:02.848 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:02 smithi043 ceph-mon[29963]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:02.848 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:02 smithi043 ceph-mon[29963]: osdmap e90: 8 total, 8 up, 8 in 2024-03-30T13:44:02.849 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:02 smithi043 ceph-mon[29963]: 5.b scrub starts 2024-03-30T13:44:02.849 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:02 smithi043 ceph-mon[29963]: 5.b scrub ok 2024-03-30T13:44:02.849 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:02 smithi043 ceph-mon[29963]: osdmap e91: 8 total, 8 up, 8 in 2024-03-30T13:44:02.849 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:02 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:44:02.849 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:02 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:44:02.849 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:02 smithi043 ceph-mon[42099]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:02.849 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:02 smithi043 ceph-mon[42099]: osdmap e90: 8 total, 8 up, 8 in 2024-03-30T13:44:02.849 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:02 smithi043 ceph-mon[42099]: 5.b scrub starts 2024-03-30T13:44:02.849 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:02 smithi043 ceph-mon[42099]: 5.b scrub ok 2024-03-30T13:44:02.849 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:02 smithi043 ceph-mon[42099]: osdmap e91: 8 total, 8 up, 8 in 2024-03-30T13:44:02.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:02 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:44:02.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:02 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:44:02.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:02 smithi113 ceph-mon[34341]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:02.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:02 smithi113 ceph-mon[34341]: osdmap e90: 8 total, 8 up, 8 in 2024-03-30T13:44:02.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:02 smithi113 ceph-mon[34341]: 5.b scrub starts 2024-03-30T13:44:02.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:02 smithi113 ceph-mon[34341]: 5.b scrub ok 2024-03-30T13:44:02.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:02 smithi113 ceph-mon[34341]: osdmap e91: 8 total, 8 up, 8 in 2024-03-30T13:44:03.597 INFO:teuthology.orchestra.run.smithi043.stdout:137438953506 2024-03-30T13:44:03.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:03 smithi043 ceph-mon[29963]: 5.1e scrub starts 2024-03-30T13:44:03.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:03 smithi043 ceph-mon[29963]: 5.1e scrub ok 2024-03-30T13:44:03.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:03 smithi043 ceph-mon[29963]: 5.c scrub starts 2024-03-30T13:44:03.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:03 smithi043 ceph-mon[29963]: 5.c scrub ok 2024-03-30T13:44:03.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:03 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2119825219' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T13:44:03.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:03 smithi043 ceph-mon[42099]: 5.1e scrub starts 2024-03-30T13:44:03.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:03 smithi043 ceph-mon[42099]: 5.1e scrub ok 2024-03-30T13:44:03.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:03 smithi043 ceph-mon[42099]: 5.c scrub starts 2024-03-30T13:44:03.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:03 smithi043 ceph-mon[42099]: 5.c scrub ok 2024-03-30T13:44:03.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:03 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2119825219' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T13:44:03.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:03 smithi113 ceph-mon[34341]: 5.1e scrub starts 2024-03-30T13:44:03.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:03 smithi113 ceph-mon[34341]: 5.1e scrub ok 2024-03-30T13:44:03.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:03 smithi113 ceph-mon[34341]: 5.c scrub starts 2024-03-30T13:44:03.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:03 smithi113 ceph-mon[34341]: 5.c scrub ok 2024-03-30T13:44:03.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:03 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2119825219' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T13:44:04.844 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:04 smithi043 ceph-mon[29963]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:04.844 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:04 smithi043 ceph-mon[29963]: 3.1e scrub starts 2024-03-30T13:44:04.844 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:04 smithi043 ceph-mon[29963]: 3.1e scrub ok 2024-03-30T13:44:04.844 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:04 smithi043 ceph-mon[42099]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:04.844 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:04 smithi043 ceph-mon[42099]: 3.1e scrub starts 2024-03-30T13:44:04.845 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:04 smithi043 ceph-mon[42099]: 3.1e scrub ok 2024-03-30T13:44:04.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:04 smithi113 ceph-mon[34341]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:04.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:04 smithi113 ceph-mon[34341]: 3.1e scrub starts 2024-03-30T13:44:04.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:04 smithi113 ceph-mon[34341]: 3.1e scrub ok 2024-03-30T13:44:05.723 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953502 got 137438953506 for osd.4 2024-03-30T13:44:05.723 DEBUG:teuthology.parallel:result is None 2024-03-30T13:44:05.723 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-30T13:44:05.724 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T13:44:05.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:05 smithi113 ceph-mon[34341]: 6.2 scrub starts 2024-03-30T13:44:05.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:05 smithi113 ceph-mon[34341]: 6.2 scrub ok 2024-03-30T13:44:05.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:05 smithi113 ceph-mon[34341]: 3.1c scrub starts 2024-03-30T13:44:05.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:05 smithi113 ceph-mon[34341]: 3.1c scrub ok 2024-03-30T13:44:05.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:05 smithi113 ceph-mon[34341]: 6.0 deep-scrub starts 2024-03-30T13:44:05.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:05 smithi113 ceph-mon[34341]: 6.0 deep-scrub ok 2024-03-30T13:44:05.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:05 smithi113 ceph-mon[34341]: 5.19 scrub starts 2024-03-30T13:44:05.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:05 smithi113 ceph-mon[34341]: 5.19 scrub ok 2024-03-30T13:44:06.096 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:05 smithi043 ceph-mon[42099]: 6.2 scrub starts 2024-03-30T13:44:06.096 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:05 smithi043 ceph-mon[42099]: 6.2 scrub ok 2024-03-30T13:44:06.096 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:05 smithi043 ceph-mon[42099]: 3.1c scrub starts 2024-03-30T13:44:06.096 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:05 smithi043 ceph-mon[42099]: 3.1c scrub ok 2024-03-30T13:44:06.096 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:05 smithi043 ceph-mon[42099]: 6.0 deep-scrub starts 2024-03-30T13:44:06.096 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:05 smithi043 ceph-mon[42099]: 6.0 deep-scrub ok 2024-03-30T13:44:06.096 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:05 smithi043 ceph-mon[42099]: 5.19 scrub starts 2024-03-30T13:44:06.096 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:05 smithi043 ceph-mon[42099]: 5.19 scrub ok 2024-03-30T13:44:06.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:05 smithi043 ceph-mon[29963]: 6.2 scrub starts 2024-03-30T13:44:06.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:05 smithi043 ceph-mon[29963]: 6.2 scrub ok 2024-03-30T13:44:06.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:05 smithi043 ceph-mon[29963]: 3.1c scrub starts 2024-03-30T13:44:06.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:05 smithi043 ceph-mon[29963]: 3.1c scrub ok 2024-03-30T13:44:06.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:05 smithi043 ceph-mon[29963]: 6.0 deep-scrub starts 2024-03-30T13:44:06.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:05 smithi043 ceph-mon[29963]: 6.0 deep-scrub ok 2024-03-30T13:44:06.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:05 smithi043 ceph-mon[29963]: 5.19 scrub starts 2024-03-30T13:44:06.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:05 smithi043 ceph-mon[29963]: 5.19 scrub ok 2024-03-30T13:44:06.310 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:44:06.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:44:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:44:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:44:06.730 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:06 smithi043 ceph-mon[29963]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:06.730 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:06 smithi043 ceph-mon[29963]: 3.11 scrub starts 2024-03-30T13:44:06.730 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:06 smithi043 ceph-mon[29963]: 3.11 scrub ok 2024-03-30T13:44:06.730 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:06 smithi043 ceph-mon[29963]: 6.1 scrub starts 2024-03-30T13:44:06.730 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:06 smithi043 ceph-mon[29963]: 6.1 scrub ok 2024-03-30T13:44:06.730 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:06 smithi043 ceph-mon[29963]: 4.1c scrub starts 2024-03-30T13:44:06.730 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:06 smithi043 ceph-mon[29963]: 4.1c scrub ok 2024-03-30T13:44:06.730 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:06 smithi043 ceph-mon[29963]: 4.0 scrub starts 2024-03-30T13:44:06.731 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:06 smithi043 ceph-mon[29963]: 4.0 scrub ok 2024-03-30T13:44:06.731 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:06 smithi043 ceph-mon[42099]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:06.731 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:06 smithi043 ceph-mon[42099]: 3.11 scrub starts 2024-03-30T13:44:06.731 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:06 smithi043 ceph-mon[42099]: 3.11 scrub ok 2024-03-30T13:44:06.731 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:06 smithi043 ceph-mon[42099]: 6.1 scrub starts 2024-03-30T13:44:06.731 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:06 smithi043 ceph-mon[42099]: 6.1 scrub ok 2024-03-30T13:44:06.731 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:06 smithi043 ceph-mon[42099]: 4.1c scrub starts 2024-03-30T13:44:06.731 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:06 smithi043 ceph-mon[42099]: 4.1c scrub ok 2024-03-30T13:44:06.731 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:06 smithi043 ceph-mon[42099]: 4.0 scrub starts 2024-03-30T13:44:06.731 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:06 smithi043 ceph-mon[42099]: 4.0 scrub ok 2024-03-30T13:44:06.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:06 smithi113 ceph-mon[34341]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:06.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:06 smithi113 ceph-mon[34341]: 3.11 scrub starts 2024-03-30T13:44:06.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:06 smithi113 ceph-mon[34341]: 3.11 scrub ok 2024-03-30T13:44:06.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:06 smithi113 ceph-mon[34341]: 6.1 scrub starts 2024-03-30T13:44:06.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:06 smithi113 ceph-mon[34341]: 6.1 scrub ok 2024-03-30T13:44:06.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:06 smithi113 ceph-mon[34341]: 4.1c scrub starts 2024-03-30T13:44:06.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:06 smithi113 ceph-mon[34341]: 4.1c scrub ok 2024-03-30T13:44:06.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:06 smithi113 ceph-mon[34341]: 4.0 scrub starts 2024-03-30T13:44:06.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:06 smithi113 ceph-mon[34341]: 4.0 scrub ok 2024-03-30T13:44:07.822 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:07 smithi043 ceph-mon[29963]: 4.1 deep-scrub starts 2024-03-30T13:44:07.822 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:07 smithi043 ceph-mon[29963]: 4.1 deep-scrub ok 2024-03-30T13:44:07.822 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:07 smithi043 ceph-mon[29963]: 3.e scrub starts 2024-03-30T13:44:07.822 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:07 smithi043 ceph-mon[29963]: 3.e scrub ok 2024-03-30T13:44:07.823 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:07 smithi043 ceph-mon[29963]: 3.4 deep-scrub starts 2024-03-30T13:44:07.823 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:07 smithi043 ceph-mon[29963]: 3.4 deep-scrub ok 2024-03-30T13:44:07.823 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:07 smithi043 ceph-mon[29963]: 6.6 scrub starts 2024-03-30T13:44:07.823 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:07 smithi043 ceph-mon[29963]: 6.6 scrub ok 2024-03-30T13:44:07.823 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:07 smithi043 ceph-mon[42099]: 4.1 deep-scrub starts 2024-03-30T13:44:07.823 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:07 smithi043 ceph-mon[42099]: 4.1 deep-scrub ok 2024-03-30T13:44:07.823 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:07 smithi043 ceph-mon[42099]: 3.e scrub starts 2024-03-30T13:44:07.823 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:07 smithi043 ceph-mon[42099]: 3.e scrub ok 2024-03-30T13:44:07.823 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:07 smithi043 ceph-mon[42099]: 3.4 deep-scrub starts 2024-03-30T13:44:07.823 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:07 smithi043 ceph-mon[42099]: 3.4 deep-scrub ok 2024-03-30T13:44:07.823 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:07 smithi043 ceph-mon[42099]: 6.6 scrub starts 2024-03-30T13:44:07.824 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:07 smithi043 ceph-mon[42099]: 6.6 scrub ok 2024-03-30T13:44:07.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:07 smithi113 ceph-mon[34341]: 4.1 deep-scrub starts 2024-03-30T13:44:07.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:07 smithi113 ceph-mon[34341]: 4.1 deep-scrub ok 2024-03-30T13:44:07.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:07 smithi113 ceph-mon[34341]: 3.e scrub starts 2024-03-30T13:44:07.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:07 smithi113 ceph-mon[34341]: 3.e scrub ok 2024-03-30T13:44:07.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:07 smithi113 ceph-mon[34341]: 3.4 deep-scrub starts 2024-03-30T13:44:07.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:07 smithi113 ceph-mon[34341]: 3.4 deep-scrub ok 2024-03-30T13:44:07.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:07 smithi113 ceph-mon[34341]: 6.6 scrub starts 2024-03-30T13:44:07.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:07 smithi113 ceph-mon[34341]: 6.6 scrub ok 2024-03-30T13:44:08.301 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:44:08.304 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-30T13:44:08.399 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:44:08 smithi043 podman[80388]: 2024-03-30 13:44:08.314122435 +0000 UTC m=+0.225452728 container create 441cce32a2c4fb219ac2a1cdd09bb2e8973032df479b569c3833c6efe7d6f945 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T13:44:08.674 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:44:08 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:44:08.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:08 smithi113 ceph-mon[34341]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:44:08.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:08 smithi113 ceph-mon[34341]: 3.6 scrub starts 2024-03-30T13:44:08.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:08 smithi113 ceph-mon[34341]: 3.6 scrub ok 2024-03-30T13:44:08.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:08 smithi113 ceph-mon[34341]: 5.10 scrub starts 2024-03-30T13:44:08.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:08 smithi113 ceph-mon[34341]: 5.10 scrub ok 2024-03-30T13:44:08.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:08 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:44:08.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:08 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:44:09.113 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-mon[29963]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:44:09.113 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-mon[29963]: 3.6 scrub starts 2024-03-30T13:44:09.113 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-mon[29963]: 3.6 scrub ok 2024-03-30T13:44:09.113 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-mon[29963]: 5.10 scrub starts 2024-03-30T13:44:09.114 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-mon[29963]: 5.10 scrub ok 2024-03-30T13:44:09.114 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:44:09.114 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:44:09.114 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:44:08 smithi043 podman[80388]: 2024-03-30 13:44:08.801184528 +0000 UTC m=+0.712514817 container init 441cce32a2c4fb219ac2a1cdd09bb2e8973032df479b569c3833c6efe7d6f945 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T13:44:09.114 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:44:08 smithi043 podman[80388]: 2024-03-30 13:44:08.806997818 +0000 UTC m=+0.718328113 container start 441cce32a2c4fb219ac2a1cdd09bb2e8973032df479b569c3833c6efe7d6f945 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T13:44:09.114 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[80418]: ts=2024-03-30T13:44:08.819Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-30T13:44:09.114 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[80418]: ts=2024-03-30T13:44:08.819Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-30T13:44:09.115 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[80418]: ts=2024-03-30T13:44:08.820Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.43 port=9094 2024-03-30T13:44:09.115 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[80418]: ts=2024-03-30T13:44:08.821Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-30T13:44:09.115 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[80418]: ts=2024-03-30T13:44:08.845Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-30T13:44:09.115 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[80418]: ts=2024-03-30T13:44:08.845Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-30T13:44:09.115 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[80418]: ts=2024-03-30T13:44:08.848Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-30T13:44:09.115 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[80418]: ts=2024-03-30T13:44:08.848Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-30T13:44:09.115 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:44:08 smithi043 bash[80388]: 441cce32a2c4fb219ac2a1cdd09bb2e8973032df479b569c3833c6efe7d6f945 2024-03-30T13:44:09.115 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:44:08 smithi043 systemd[1]: Started Ceph alertmanager.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T13:44:09.115 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-mon[42099]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:44:09.115 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-mon[42099]: 3.6 scrub starts 2024-03-30T13:44:09.115 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-mon[42099]: 3.6 scrub ok 2024-03-30T13:44:09.115 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-mon[42099]: 5.10 scrub starts 2024-03-30T13:44:09.116 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-mon[42099]: 5.10 scrub ok 2024-03-30T13:44:09.116 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:44:09.116 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:08 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:44:09.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:44:09] ENGINE Bus STOPPING 2024-03-30T13:44:09.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:44:09] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-30T13:44:09.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:44:09] ENGINE Bus STOPPED 2024-03-30T13:44:09.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:44:09] ENGINE Bus STARTING 2024-03-30T13:44:09.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:44:09] ENGINE Serving on http://:::9283 2024-03-30T13:44:09.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:44:09] ENGINE Bus STARTED 2024-03-30T13:44:09.566 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-03-30T13:44:07.278330+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1005,"num_read_kb":851,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":607,"ondisk_log_size":607,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":241936,"kb_used_data":9124,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749719280,"statfs":{"total":767960285184,"available":767712542720,"internally_reserved":0,"allocated":9342976,"data_stored":5352258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001917"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297664+0000","last_change":"2024-03-30T13:42:51.330132+0000","last_active":"2024-03-30T13:44:02.297664+0000","last_peered":"2024-03-30T13:44:02.297664+0000","last_clean":"2024-03-30T13:44:02.297664+0000","last_became_active":"2024-03-30T13:42:51.329797+0000","last_became_peered":"2024-03-30T13:42:51.329797+0000","last_unstale":"2024-03-30T13:44:02.297664+0000","last_undegraded":"2024-03-30T13:44:02.297664+0000","last_fullsized":"2024-03-30T13:44:02.297664+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:08.354306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.677830+0000","last_change":"2024-03-30T13:42:49.323586+0000","last_active":"2024-03-30T13:44:02.677830+0000","last_peered":"2024-03-30T13:44:02.677830+0000","last_clean":"2024-03-30T13:44:02.677830+0000","last_became_active":"2024-03-30T13:42:49.323268+0000","last_became_peered":"2024-03-30T13:42:49.323268+0000","last_unstale":"2024-03-30T13:44:02.677830+0000","last_undegraded":"2024-03-30T13:44:02.677830+0000","last_fullsized":"2024-03-30T13:44:02.677830+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"65'1","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405493+0000","last_change":"2024-03-30T13:42:45.314709+0000","last_active":"2024-03-30T13:44:02.405493+0000","last_peered":"2024-03-30T13:44:02.405493+0000","last_clean":"2024-03-30T13:44:02.405493+0000","last_became_active":"2024-03-30T13:42:45.313936+0000","last_became_peered":"2024-03-30T13:42:45.313936+0000","last_unstale":"2024-03-30T13:44:02.405493+0000","last_undegraded":"2024-03-30T13:44:02.405493+0000","last_fullsized":"2024-03-30T13:44:02.405493+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:07.481265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":75,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297524+0000","last_change":"2024-03-30T13:42:47.320128+0000","last_active":"2024-03-30T13:44:02.297524+0000","last_peered":"2024-03-30T13:44:02.297524+0000","last_clean":"2024-03-30T13:44:02.297524+0000","last_became_active":"2024-03-30T13:42:47.319638+0000","last_became_peered":"2024-03-30T13:42:47.319638+0000","last_unstale":"2024-03-30T13:44:02.297524+0000","last_undegraded":"2024-03-30T13:44:02.297524+0000","last_fullsized":"2024-03-30T13:44:02.297524+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:05.845454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295495+0000","last_change":"2024-03-30T13:42:51.374776+0000","last_active":"2024-03-30T13:44:02.295495+0000","last_peered":"2024-03-30T13:44:02.295495+0000","last_clean":"2024-03-30T13:44:02.295495+0000","last_became_active":"2024-03-30T13:42:51.374403+0000","last_became_peered":"2024-03-30T13:42:51.374403+0000","last_unstale":"2024-03-30T13:44:02.295495+0000","last_undegraded":"2024-03-30T13:44:02.295495+0000","last_fullsized":"2024-03-30T13:44:02.295495+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:13.438854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297531+0000","last_change":"2024-03-30T13:42:49.327437+0000","last_active":"2024-03-30T13:44:02.297531+0000","last_peered":"2024-03-30T13:44:02.297531+0000","last_clean":"2024-03-30T13:44:02.297531+0000","last_became_active":"2024-03-30T13:42:49.327124+0000","last_became_peered":"2024-03-30T13:42:49.327124+0000","last_unstale":"2024-03-30T13:44:02.297531+0000","last_undegraded":"2024-03-30T13:44:02.297531+0000","last_fullsized":"2024-03-30T13:44:02.297531+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:17.902978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.296465+0000","last_change":"2024-03-30T13:42:45.311144+0000","last_active":"2024-03-30T13:44:02.296465+0000","last_peered":"2024-03-30T13:44:02.296465+0000","last_clean":"2024-03-30T13:44:02.296465+0000","last_became_active":"2024-03-30T13:42:45.310771+0000","last_became_peered":"2024-03-30T13:42:45.310771+0000","last_unstale":"2024-03-30T13:44:02.296465+0000","last_undegraded":"2024-03-30T13:44:02.296465+0000","last_fullsized":"2024-03-30T13:44:02.296465+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"72'15","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295286+0000","last_change":"2024-03-30T13:42:47.317438+0000","last_active":"2024-03-30T13:44:02.295286+0000","last_peered":"2024-03-30T13:44:02.295286+0000","last_clean":"2024-03-30T13:44:02.295286+0000","last_became_active":"2024-03-30T13:42:47.317102+0000","last_became_peered":"2024-03-30T13:42:47.317102+0000","last_unstale":"2024-03-30T13:44:02.295286+0000","last_undegraded":"2024-03-30T13:44:02.295286+0000","last_fullsized":"2024-03-30T13:44:02.295286+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:06.434090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405070+0000","last_change":"2024-03-30T13:42:51.331563+0000","last_active":"2024-03-30T13:44:02.405070+0000","last_peered":"2024-03-30T13:44:02.405070+0000","last_clean":"2024-03-30T13:44:02.405070+0000","last_became_active":"2024-03-30T13:42:51.331196+0000","last_became_peered":"2024-03-30T13:42:51.331196+0000","last_unstale":"2024-03-30T13:44:02.405070+0000","last_undegraded":"2024-03-30T13:44:02.405070+0000","last_fullsized":"2024-03-30T13:44:02.405070+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:09.803811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679009+0000","last_change":"2024-03-30T13:43:59.262080+0000","last_active":"2024-03-30T13:44:02.679009+0000","last_peered":"2024-03-30T13:44:02.679009+0000","last_clean":"2024-03-30T13:44:02.679009+0000","last_became_active":"2024-03-30T13:42:49.321811+0000","last_became_peered":"2024-03-30T13:42:49.321811+0000","last_unstale":"2024-03-30T13:44:02.679009+0000","last_undegraded":"2024-03-30T13:44:02.679009+0000","last_fullsized":"2024-03-30T13:44:02.679009+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:59.261981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:59.261981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:20.497993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678880+0000","last_change":"2024-03-30T13:42:45.320122+0000","last_active":"2024-03-30T13:44:02.678880+0000","last_peered":"2024-03-30T13:44:02.678880+0000","last_clean":"2024-03-30T13:44:02.678880+0000","last_became_active":"2024-03-30T13:42:45.319827+0000","last_became_peered":"2024-03-30T13:42:45.319827+0000","last_unstale":"2024-03-30T13:44:02.678880+0000","last_undegraded":"2024-03-30T13:44:02.678880+0000","last_fullsized":"2024-03-30T13:44:02.678880+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":75,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297832+0000","last_change":"2024-03-30T13:42:47.310779+0000","last_active":"2024-03-30T13:44:02.297832+0000","last_peered":"2024-03-30T13:44:02.297832+0000","last_clean":"2024-03-30T13:44:02.297832+0000","last_became_active":"2024-03-30T13:42:47.310289+0000","last_became_peered":"2024-03-30T13:42:47.310289+0000","last_unstale":"2024-03-30T13:44:02.297832+0000","last_undegraded":"2024-03-30T13:44:02.297832+0000","last_fullsized":"2024-03-30T13:44:02.297832+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.677924+0000","last_change":"2024-03-30T13:42:51.326065+0000","last_active":"2024-03-30T13:44:02.677924+0000","last_peered":"2024-03-30T13:44:02.677924+0000","last_clean":"2024-03-30T13:44:02.677924+0000","last_became_active":"2024-03-30T13:42:51.325748+0000","last_became_peered":"2024-03-30T13:42:51.325748+0000","last_unstale":"2024-03-30T13:44:02.677924+0000","last_undegraded":"2024-03-30T13:44:02.677924+0000","last_fullsized":"2024-03-30T13:44:02.677924+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406271+0000","last_change":"2024-03-30T13:42:49.326311+0000","last_active":"2024-03-30T13:44:02.406271+0000","last_peered":"2024-03-30T13:44:02.406271+0000","last_clean":"2024-03-30T13:44:02.406271+0000","last_became_active":"2024-03-30T13:42:49.326027+0000","last_became_peered":"2024-03-30T13:42:49.326027+0000","last_unstale":"2024-03-30T13:44:02.406271+0000","last_undegraded":"2024-03-30T13:44:02.406271+0000","last_fullsized":"2024-03-30T13:44:02.406271+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:17.535137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"65'1","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678529+0000","last_change":"2024-03-30T13:42:45.320319+0000","last_active":"2024-03-30T13:44:02.678529+0000","last_peered":"2024-03-30T13:44:02.678529+0000","last_clean":"2024-03-30T13:44:02.678529+0000","last_became_active":"2024-03-30T13:42:45.319975+0000","last_became_peered":"2024-03-30T13:42:45.319975+0000","last_unstale":"2024-03-30T13:44:02.678529+0000","last_undegraded":"2024-03-30T13:44:02.678529+0000","last_fullsized":"2024-03-30T13:44:02.678529+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297997+0000","last_change":"2024-03-30T13:43:50.961052+0000","last_active":"2024-03-30T13:44:02.297997+0000","last_peered":"2024-03-30T13:44:02.297997+0000","last_clean":"2024-03-30T13:44:02.297997+0000","last_became_active":"2024-03-30T13:42:47.317352+0000","last_became_peered":"2024-03-30T13:42:47.317352+0000","last_unstale":"2024-03-30T13:44:02.297997+0000","last_undegraded":"2024-03-30T13:44:02.297997+0000","last_fullsized":"2024-03-30T13:44:02.297997+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-30T13:43:50.960980+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-30T13:43:50.960980+0000","last_clean_scrub_stamp":"2024-03-30T13:43:50.960980+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:15.585035+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.298580+0000","last_change":"2024-03-30T13:42:51.372350+0000","last_active":"2024-03-30T13:44:02.298580+0000","last_peered":"2024-03-30T13:44:02.298580+0000","last_clean":"2024-03-30T13:44:02.298580+0000","last_became_active":"2024-03-30T13:42:51.372011+0000","last_became_peered":"2024-03-30T13:42:51.372011+0000","last_unstale":"2024-03-30T13:44:02.298580+0000","last_undegraded":"2024-03-30T13:44:02.298580+0000","last_fullsized":"2024-03-30T13:44:02.298580+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678385+0000","last_change":"2024-03-30T13:42:49.323017+0000","last_active":"2024-03-30T13:44:02.678385+0000","last_peered":"2024-03-30T13:44:02.678385+0000","last_clean":"2024-03-30T13:44:02.678385+0000","last_became_active":"2024-03-30T13:42:49.322665+0000","last_became_peered":"2024-03-30T13:42:49.322665+0000","last_unstale":"2024-03-30T13:44:02.678385+0000","last_undegraded":"2024-03-30T13:44:02.678385+0000","last_fullsized":"2024-03-30T13:44:02.678385+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:04.198076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405473+0000","last_change":"2024-03-30T13:42:45.317711+0000","last_active":"2024-03-30T13:44:02.405473+0000","last_peered":"2024-03-30T13:44:02.405473+0000","last_clean":"2024-03-30T13:44:02.405473+0000","last_became_active":"2024-03-30T13:42:45.317378+0000","last_became_peered":"2024-03-30T13:42:45.317378+0000","last_unstale":"2024-03-30T13:44:02.405473+0000","last_undegraded":"2024-03-30T13:44:02.405473+0000","last_fullsized":"2024-03-30T13:44:02.405473+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:14.011698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":84,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.296323+0000","last_change":"2024-03-30T13:42:47.320507+0000","last_active":"2024-03-30T13:44:02.296323+0000","last_peered":"2024-03-30T13:44:02.296323+0000","last_clean":"2024-03-30T13:44:02.296323+0000","last_became_active":"2024-03-30T13:42:47.320168+0000","last_became_peered":"2024-03-30T13:42:47.320168+0000","last_unstale":"2024-03-30T13:44:02.296323+0000","last_undegraded":"2024-03-30T13:44:02.296323+0000","last_fullsized":"2024-03-30T13:44:02.296323+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:03.054150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.296636+0000","last_change":"2024-03-30T13:42:51.371902+0000","last_active":"2024-03-30T13:44:02.296636+0000","last_peered":"2024-03-30T13:44:02.296636+0000","last_clean":"2024-03-30T13:44:02.296636+0000","last_became_active":"2024-03-30T13:42:51.371378+0000","last_became_peered":"2024-03-30T13:42:51.371378+0000","last_unstale":"2024-03-30T13:44:02.296636+0000","last_undegraded":"2024-03-30T13:44:02.296636+0000","last_fullsized":"2024-03-30T13:44:02.296636+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:20.136855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.298078+0000","last_change":"2024-03-30T13:42:49.323001+0000","last_active":"2024-03-30T13:44:02.298078+0000","last_peered":"2024-03-30T13:44:02.298078+0000","last_clean":"2024-03-30T13:44:02.298078+0000","last_became_active":"2024-03-30T13:42:49.322561+0000","last_became_peered":"2024-03-30T13:42:49.322561+0000","last_unstale":"2024-03-30T13:44:02.298078+0000","last_undegraded":"2024-03-30T13:44:02.298078+0000","last_fullsized":"2024-03-30T13:44:02.298078+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297912+0000","last_change":"2024-03-30T13:42:45.308820+0000","last_active":"2024-03-30T13:44:02.297912+0000","last_peered":"2024-03-30T13:44:02.297912+0000","last_clean":"2024-03-30T13:44:02.297912+0000","last_became_active":"2024-03-30T13:42:45.308468+0000","last_became_peered":"2024-03-30T13:42:45.308468+0000","last_unstale":"2024-03-30T13:44:02.297912+0000","last_undegraded":"2024-03-30T13:44:02.297912+0000","last_fullsized":"2024-03-30T13:44:02.297912+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:10.369269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"88'17","reported_seq":94,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.296024+0000","last_change":"2024-03-30T13:43:52.044529+0000","last_active":"2024-03-30T13:44:02.296024+0000","last_peered":"2024-03-30T13:44:02.296024+0000","last_clean":"2024-03-30T13:44:02.296024+0000","last_became_active":"2024-03-30T13:42:47.317881+0000","last_became_peered":"2024-03-30T13:42:47.317881+0000","last_unstale":"2024-03-30T13:44:02.296024+0000","last_undegraded":"2024-03-30T13:44:02.296024+0000","last_fullsized":"2024-03-30T13:44:02.296024+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'17","last_scrub_stamp":"2024-03-30T13:43:52.044427+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-03-30T13:43:52.044427+0000","last_clean_scrub_stamp":"2024-03-30T13:43:52.044427+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:11.077720+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406721+0000","last_change":"2024-03-30T13:42:51.328035+0000","last_active":"2024-03-30T13:44:02.406721+0000","last_peered":"2024-03-30T13:44:02.406721+0000","last_clean":"2024-03-30T13:44:02.406721+0000","last_became_active":"2024-03-30T13:42:51.327749+0000","last_became_peered":"2024-03-30T13:42:51.327749+0000","last_unstale":"2024-03-30T13:44:02.406721+0000","last_undegraded":"2024-03-30T13:44:02.406721+0000","last_fullsized":"2024-03-30T13:44:02.406721+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679659+0000","last_change":"2024-03-30T13:42:49.323699+0000","last_active":"2024-03-30T13:44:02.679659+0000","last_peered":"2024-03-30T13:44:02.679659+0000","last_clean":"2024-03-30T13:44:02.679659+0000","last_became_active":"2024-03-30T13:42:49.323362+0000","last_became_peered":"2024-03-30T13:42:49.323362+0000","last_unstale":"2024-03-30T13:44:02.679659+0000","last_undegraded":"2024-03-30T13:44:02.679659+0000","last_fullsized":"2024-03-30T13:44:02.679659+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"65'1","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.677944+0000","last_change":"2024-03-30T13:43:47.807153+0000","last_active":"2024-03-30T13:44:02.677944+0000","last_peered":"2024-03-30T13:44:02.677944+0000","last_clean":"2024-03-30T13:44:02.677944+0000","last_became_active":"2024-03-30T13:42:45.307832+0000","last_became_peered":"2024-03-30T13:42:45.307832+0000","last_unstale":"2024-03-30T13:44:02.677944+0000","last_undegraded":"2024-03-30T13:44:02.677944+0000","last_fullsized":"2024-03-30T13:44:02.677944+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:43:47.807055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:43:47.807055+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:10.931779+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405317+0000","last_change":"2024-03-30T13:42:47.501206+0000","last_active":"2024-03-30T13:44:02.405317+0000","last_peered":"2024-03-30T13:44:02.405317+0000","last_clean":"2024-03-30T13:44:02.405317+0000","last_became_active":"2024-03-30T13:42:47.500763+0000","last_became_peered":"2024-03-30T13:42:47.500763+0000","last_unstale":"2024-03-30T13:44:02.405317+0000","last_undegraded":"2024-03-30T13:44:02.405317+0000","last_fullsized":"2024-03-30T13:44:02.405317+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:04.060530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678240+0000","last_change":"2024-03-30T13:42:51.325851+0000","last_active":"2024-03-30T13:44:02.678240+0000","last_peered":"2024-03-30T13:44:02.678240+0000","last_clean":"2024-03-30T13:44:02.678240+0000","last_became_active":"2024-03-30T13:42:51.325513+0000","last_became_peered":"2024-03-30T13:42:51.325513+0000","last_unstale":"2024-03-30T13:44:02.678240+0000","last_undegraded":"2024-03-30T13:44:02.678240+0000","last_fullsized":"2024-03-30T13:44:02.678240+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:09.388177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.408681+0000","last_change":"2024-03-30T13:44:02.408701+0000","last_active":"2024-03-30T13:44:02.408681+0000","last_peered":"2024-03-30T13:44:02.408681+0000","last_clean":"2024-03-30T13:44:02.408681+0000","last_became_active":"2024-03-30T13:42:49.330335+0000","last_became_peered":"2024-03-30T13:42:49.330335+0000","last_unstale":"2024-03-30T13:44:02.408681+0000","last_undegraded":"2024-03-30T13:44:02.408681+0000","last_fullsized":"2024-03-30T13:44:02.408681+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:02.408612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:02.408612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:23.673286+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.294962+0000","last_change":"2024-03-30T13:42:45.310476+0000","last_active":"2024-03-30T13:44:02.294962+0000","last_peered":"2024-03-30T13:44:02.294962+0000","last_clean":"2024-03-30T13:44:02.294962+0000","last_became_active":"2024-03-30T13:42:45.310165+0000","last_became_peered":"2024-03-30T13:42:45.310165+0000","last_unstale":"2024-03-30T13:44:02.294962+0000","last_undegraded":"2024-03-30T13:44:02.294962+0000","last_fullsized":"2024-03-30T13:44:02.294962+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:03.398892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679665+0000","last_change":"2024-03-30T13:43:54.758046+0000","last_active":"2024-03-30T13:44:02.679665+0000","last_peered":"2024-03-30T13:44:02.679665+0000","last_clean":"2024-03-30T13:44:02.679665+0000","last_became_active":"2024-03-30T13:42:47.324399+0000","last_became_peered":"2024-03-30T13:42:47.324399+0000","last_unstale":"2024-03-30T13:44:02.679665+0000","last_undegraded":"2024-03-30T13:44:02.679665+0000","last_fullsized":"2024-03-30T13:44:02.679665+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:43:54.757869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:54.757869+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:15.753990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406263+0000","last_change":"2024-03-30T13:43:59.682402+0000","last_active":"2024-03-30T13:44:02.406263+0000","last_peered":"2024-03-30T13:44:02.406263+0000","last_clean":"2024-03-30T13:44:02.406263+0000","last_became_active":"2024-03-30T13:42:51.327000+0000","last_became_peered":"2024-03-30T13:42:51.327000+0000","last_unstale":"2024-03-30T13:44:02.406263+0000","last_undegraded":"2024-03-30T13:44:02.406263+0000","last_fullsized":"2024-03-30T13:44:02.406263+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:59.682298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:43:59.682298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:25.906592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678245+0000","last_change":"2024-03-30T13:43:53.750352+0000","last_active":"2024-03-30T13:44:02.678245+0000","last_peered":"2024-03-30T13:44:02.678245+0000","last_clean":"2024-03-30T13:44:02.678245+0000","last_became_active":"2024-03-30T13:42:49.322799+0000","last_became_peered":"2024-03-30T13:42:49.322799+0000","last_unstale":"2024-03-30T13:44:02.678245+0000","last_undegraded":"2024-03-30T13:44:02.678245+0000","last_fullsized":"2024-03-30T13:44:02.678245+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:53.750254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:53.750254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:59.399300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":68,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405432+0000","last_change":"2024-03-30T13:43:58.386027+0000","last_active":"2024-03-30T13:44:02.405432+0000","last_peered":"2024-03-30T13:44:02.405432+0000","last_clean":"2024-03-30T13:44:02.405432+0000","last_became_active":"2024-03-30T13:42:45.314302+0000","last_became_peered":"2024-03-30T13:42:45.314302+0000","last_unstale":"2024-03-30T13:44:02.405432+0000","last_undegraded":"2024-03-30T13:44:02.405432+0000","last_fullsized":"2024-03-30T13:44:02.405432+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:58.385891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:43:58.385891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:03.698225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295060+0000","last_change":"2024-03-30T13:43:55.894589+0000","last_active":"2024-03-30T13:44:02.295060+0000","last_peered":"2024-03-30T13:44:02.295060+0000","last_clean":"2024-03-30T13:44:02.295060+0000","last_became_active":"2024-03-30T13:42:47.321428+0000","last_became_peered":"2024-03-30T13:42:47.321428+0000","last_unstale":"2024-03-30T13:44:02.295060+0000","last_undegraded":"2024-03-30T13:44:02.295060+0000","last_fullsized":"2024-03-30T13:44:02.295060+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-30T13:43:55.894502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:55.894502+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:56.441104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295334+0000","last_change":"2024-03-30T13:42:51.374755+0000","last_active":"2024-03-30T13:44:02.295334+0000","last_peered":"2024-03-30T13:44:02.295334+0000","last_clean":"2024-03-30T13:44:02.295334+0000","last_became_active":"2024-03-30T13:42:51.374304+0000","last_became_peered":"2024-03-30T13:42:51.374304+0000","last_unstale":"2024-03-30T13:44:02.295334+0000","last_undegraded":"2024-03-30T13:44:02.295334+0000","last_fullsized":"2024-03-30T13:44:02.295334+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:18.990969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406924+0000","last_change":"2024-03-30T13:42:49.324543+0000","last_active":"2024-03-30T13:44:02.406924+0000","last_peered":"2024-03-30T13:44:02.406924+0000","last_clean":"2024-03-30T13:44:02.406924+0000","last_became_active":"2024-03-30T13:42:49.324206+0000","last_became_peered":"2024-03-30T13:42:49.324206+0000","last_unstale":"2024-03-30T13:44:02.406924+0000","last_undegraded":"2024-03-30T13:44:02.406924+0000","last_fullsized":"2024-03-30T13:44:02.406924+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679264+0000","last_change":"2024-03-30T13:42:45.319833+0000","last_active":"2024-03-30T13:44:02.679264+0000","last_peered":"2024-03-30T13:44:02.679264+0000","last_clean":"2024-03-30T13:44:02.679264+0000","last_became_active":"2024-03-30T13:42:45.319482+0000","last_became_peered":"2024-03-30T13:42:45.319482+0000","last_unstale":"2024-03-30T13:44:02.679264+0000","last_undegraded":"2024-03-30T13:44:02.679264+0000","last_fullsized":"2024-03-30T13:44:02.679264+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":85,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295184+0000","last_change":"2024-03-30T13:43:54.926339+0000","last_active":"2024-03-30T13:44:02.295184+0000","last_peered":"2024-03-30T13:44:02.295184+0000","last_clean":"2024-03-30T13:44:02.295184+0000","last_became_active":"2024-03-30T13:42:47.500712+0000","last_became_peered":"2024-03-30T13:42:47.500712+0000","last_unstale":"2024-03-30T13:44:02.295184+0000","last_undegraded":"2024-03-30T13:44:02.295184+0000","last_fullsized":"2024-03-30T13:44:02.295184+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:43:54.926242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:54.926242+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:59.626117+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.404782+0000","last_change":"2024-03-30T13:42:51.334062+0000","last_active":"2024-03-30T13:44:02.404782+0000","last_peered":"2024-03-30T13:44:02.404782+0000","last_clean":"2024-03-30T13:44:02.404782+0000","last_became_active":"2024-03-30T13:42:51.333603+0000","last_became_peered":"2024-03-30T13:42:51.333603+0000","last_unstale":"2024-03-30T13:44:02.404782+0000","last_undegraded":"2024-03-30T13:44:02.404782+0000","last_fullsized":"2024-03-30T13:44:02.404782+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:09.821391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295522+0000","last_change":"2024-03-30T13:42:49.325524+0000","last_active":"2024-03-30T13:44:02.295522+0000","last_peered":"2024-03-30T13:44:02.295522+0000","last_clean":"2024-03-30T13:44:02.295522+0000","last_became_active":"2024-03-30T13:42:49.325219+0000","last_became_peered":"2024-03-30T13:42:49.325219+0000","last_unstale":"2024-03-30T13:44:02.295522+0000","last_undegraded":"2024-03-30T13:44:02.295522+0000","last_fullsized":"2024-03-30T13:44:02.295522+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":68,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.407046+0000","last_change":"2024-03-30T13:43:52.764537+0000","last_active":"2024-03-30T13:44:02.407046+0000","last_peered":"2024-03-30T13:44:02.407046+0000","last_clean":"2024-03-30T13:44:02.407046+0000","last_became_active":"2024-03-30T13:42:45.311071+0000","last_became_peered":"2024-03-30T13:42:45.311071+0000","last_unstale":"2024-03-30T13:44:02.407046+0000","last_undegraded":"2024-03-30T13:44:02.407046+0000","last_fullsized":"2024-03-30T13:44:02.407046+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:52.764446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:52.764446+0000","last_clean_scrub_stamp":"2024-03-30T13:43:52.764446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:14.951666+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678151+0000","last_change":"2024-03-30T13:43:53.806905+0000","last_active":"2024-03-30T13:44:02.678151+0000","last_peered":"2024-03-30T13:44:02.678151+0000","last_clean":"2024-03-30T13:44:02.678151+0000","last_became_active":"2024-03-30T13:42:47.324421+0000","last_became_peered":"2024-03-30T13:42:47.324421+0000","last_unstale":"2024-03-30T13:44:02.678151+0000","last_undegraded":"2024-03-30T13:44:02.678151+0000","last_fullsized":"2024-03-30T13:44:02.678151+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:43:53.806768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:53.806768+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:11.831276+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.294765+0000","last_change":"2024-03-30T13:42:51.335069+0000","last_active":"2024-03-30T13:44:02.294765+0000","last_peered":"2024-03-30T13:44:02.294765+0000","last_clean":"2024-03-30T13:44:02.294765+0000","last_became_active":"2024-03-30T13:42:51.334597+0000","last_became_peered":"2024-03-30T13:42:51.334597+0000","last_unstale":"2024-03-30T13:44:02.294765+0000","last_undegraded":"2024-03-30T13:44:02.294765+0000","last_fullsized":"2024-03-30T13:44:02.294765+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678049+0000","last_change":"2024-03-30T13:42:49.323180+0000","last_active":"2024-03-30T13:44:02.678049+0000","last_peered":"2024-03-30T13:44:02.678049+0000","last_clean":"2024-03-30T13:44:02.678049+0000","last_became_active":"2024-03-30T13:42:49.322843+0000","last_became_peered":"2024-03-30T13:42:49.322843+0000","last_unstale":"2024-03-30T13:44:02.678049+0000","last_undegraded":"2024-03-30T13:44:02.678049+0000","last_fullsized":"2024-03-30T13:44:02.678049+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:12.351693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.298253+0000","last_change":"2024-03-30T13:42:45.316182+0000","last_active":"2024-03-30T13:44:02.298253+0000","last_peered":"2024-03-30T13:44:02.298253+0000","last_clean":"2024-03-30T13:44:02.298253+0000","last_became_active":"2024-03-30T13:42:45.315864+0000","last_became_peered":"2024-03-30T13:42:45.315864+0000","last_unstale":"2024-03-30T13:44:02.298253+0000","last_undegraded":"2024-03-30T13:44:02.298253+0000","last_fullsized":"2024-03-30T13:44:02.298253+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:05.199162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":104,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.298038+0000","last_change":"2024-03-30T13:43:54.985253+0000","last_active":"2024-03-30T13:44:02.298038+0000","last_peered":"2024-03-30T13:44:02.298038+0000","last_clean":"2024-03-30T13:44:02.298038+0000","last_became_active":"2024-03-30T13:43:12.037286+0000","last_became_peered":"2024-03-30T13:43:12.037286+0000","last_unstale":"2024-03-30T13:44:02.298038+0000","last_undegraded":"2024-03-30T13:44:02.298038+0000","last_fullsized":"2024-03-30T13:44:02.298038+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:43:54.985130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:54.985130+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:00.861139+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405182+0000","last_change":"2024-03-30T13:42:51.336473+0000","last_active":"2024-03-30T13:44:02.405182+0000","last_peered":"2024-03-30T13:44:02.405182+0000","last_clean":"2024-03-30T13:44:02.405182+0000","last_became_active":"2024-03-30T13:42:51.336162+0000","last_became_peered":"2024-03-30T13:42:51.336162+0000","last_unstale":"2024-03-30T13:44:02.405182+0000","last_undegraded":"2024-03-30T13:44:02.405182+0000","last_fullsized":"2024-03-30T13:44:02.405182+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:06.520978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678969+0000","last_change":"2024-03-30T13:42:49.321501+0000","last_active":"2024-03-30T13:44:02.678969+0000","last_peered":"2024-03-30T13:44:02.678969+0000","last_clean":"2024-03-30T13:44:02.678969+0000","last_became_active":"2024-03-30T13:42:49.321203+0000","last_became_peered":"2024-03-30T13:42:49.321203+0000","last_unstale":"2024-03-30T13:44:02.678969+0000","last_undegraded":"2024-03-30T13:44:02.678969+0000","last_fullsized":"2024-03-30T13:44:02.678969+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406386+0000","last_change":"2024-03-30T13:42:45.318320+0000","last_active":"2024-03-30T13:44:02.406386+0000","last_peered":"2024-03-30T13:44:02.406386+0000","last_clean":"2024-03-30T13:44:02.406386+0000","last_became_active":"2024-03-30T13:42:45.318016+0000","last_became_peered":"2024-03-30T13:42:45.318016+0000","last_unstale":"2024-03-30T13:44:02.406386+0000","last_undegraded":"2024-03-30T13:44:02.406386+0000","last_fullsized":"2024-03-30T13:44:02.406386+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":91,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295710+0000","last_change":"2024-03-30T13:43:12.043856+0000","last_active":"2024-03-30T13:44:02.295710+0000","last_peered":"2024-03-30T13:44:02.295710+0000","last_clean":"2024-03-30T13:44:02.295710+0000","last_became_active":"2024-03-30T13:43:12.037240+0000","last_became_peered":"2024-03-30T13:43:12.037240+0000","last_unstale":"2024-03-30T13:44:02.295710+0000","last_undegraded":"2024-03-30T13:44:02.295710+0000","last_fullsized":"2024-03-30T13:44:02.295710+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:15.454436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.298736+0000","last_change":"2024-03-30T13:42:51.325383+0000","last_active":"2024-03-30T13:44:02.298736+0000","last_peered":"2024-03-30T13:44:02.298736+0000","last_clean":"2024-03-30T13:44:02.298736+0000","last_became_active":"2024-03-30T13:42:51.325064+0000","last_became_peered":"2024-03-30T13:42:51.325064+0000","last_unstale":"2024-03-30T13:44:02.298736+0000","last_undegraded":"2024-03-30T13:44:02.298736+0000","last_fullsized":"2024-03-30T13:44:02.298736+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:08.485150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"72'8","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295716+0000","last_change":"2024-03-30T13:42:49.328971+0000","last_active":"2024-03-30T13:44:02.295716+0000","last_peered":"2024-03-30T13:44:02.295716+0000","last_clean":"2024-03-30T13:44:02.295716+0000","last_became_active":"2024-03-30T13:42:49.328616+0000","last_became_peered":"2024-03-30T13:42:49.328616+0000","last_unstale":"2024-03-30T13:44:02.295716+0000","last_undegraded":"2024-03-30T13:44:02.295716+0000","last_fullsized":"2024-03-30T13:44:02.295716+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405053+0000","last_change":"2024-03-30T13:42:45.315502+0000","last_active":"2024-03-30T13:44:02.405053+0000","last_peered":"2024-03-30T13:44:02.405053+0000","last_clean":"2024-03-30T13:44:02.405053+0000","last_became_active":"2024-03-30T13:42:45.314585+0000","last_became_peered":"2024-03-30T13:42:45.314585+0000","last_unstale":"2024-03-30T13:44:02.405053+0000","last_undegraded":"2024-03-30T13:44:02.405053+0000","last_fullsized":"2024-03-30T13:44:02.405053+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:14.270223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679100+0000","last_change":"2024-03-30T13:43:53.236751+0000","last_active":"2024-03-30T13:44:02.679100+0000","last_peered":"2024-03-30T13:44:02.679100+0000","last_clean":"2024-03-30T13:44:02.679100+0000","last_became_active":"2024-03-30T13:42:47.501559+0000","last_became_peered":"2024-03-30T13:42:47.501559+0000","last_unstale":"2024-03-30T13:44:02.679100+0000","last_undegraded":"2024-03-30T13:44:02.679100+0000","last_fullsized":"2024-03-30T13:44:02.679100+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:43:53.236633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:53.236633+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:05.517754+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.296647+0000","last_change":"2024-03-30T13:43:52.834487+0000","last_active":"2024-03-30T13:44:02.296647+0000","last_peered":"2024-03-30T13:44:02.296647+0000","last_clean":"2024-03-30T13:44:02.296647+0000","last_became_active":"2024-03-30T13:42:51.325522+0000","last_became_peered":"2024-03-30T13:42:51.325522+0000","last_unstale":"2024-03-30T13:44:02.296647+0000","last_undegraded":"2024-03-30T13:44:02.296647+0000","last_fullsized":"2024-03-30T13:44:02.296647+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:52.834381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:43:52.834381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:56.638872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.296886+0000","last_change":"2024-03-30T13:42:49.323616+0000","last_active":"2024-03-30T13:44:02.296886+0000","last_peered":"2024-03-30T13:44:02.296886+0000","last_clean":"2024-03-30T13:44:02.296886+0000","last_became_active":"2024-03-30T13:42:49.323222+0000","last_became_peered":"2024-03-30T13:42:49.323222+0000","last_unstale":"2024-03-30T13:44:02.296886+0000","last_undegraded":"2024-03-30T13:44:02.296886+0000","last_fullsized":"2024-03-30T13:44:02.296886+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:17.439407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406433+0000","last_change":"2024-03-30T13:42:45.313946+0000","last_active":"2024-03-30T13:44:02.406433+0000","last_peered":"2024-03-30T13:44:02.406433+0000","last_clean":"2024-03-30T13:44:02.406433+0000","last_became_active":"2024-03-30T13:42:45.313617+0000","last_became_peered":"2024-03-30T13:42:45.313617+0000","last_unstale":"2024-03-30T13:44:02.406433+0000","last_undegraded":"2024-03-30T13:44:02.406433+0000","last_fullsized":"2024-03-30T13:44:02.406433+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":80,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405395+0000","last_change":"2024-03-30T13:44:00.374645+0000","last_active":"2024-03-30T13:44:02.405395+0000","last_peered":"2024-03-30T13:44:02.405395+0000","last_clean":"2024-03-30T13:44:02.405395+0000","last_became_active":"2024-03-30T13:42:47.501925+0000","last_became_peered":"2024-03-30T13:42:47.501925+0000","last_unstale":"2024-03-30T13:44:02.405395+0000","last_undegraded":"2024-03-30T13:44:02.405395+0000","last_fullsized":"2024-03-30T13:44:02.405395+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:44:00.374534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:00.374534+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:25.183504+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"72'1","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406820+0000","last_change":"2024-03-30T13:42:51.372361+0000","last_active":"2024-03-30T13:44:02.406820+0000","last_peered":"2024-03-30T13:44:02.406820+0000","last_clean":"2024-03-30T13:44:02.406820+0000","last_became_active":"2024-03-30T13:42:51.372023+0000","last_became_peered":"2024-03-30T13:42:51.372023+0000","last_unstale":"2024-03-30T13:44:02.406820+0000","last_undegraded":"2024-03-30T13:44:02.406820+0000","last_fullsized":"2024-03-30T13:44:02.406820+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679356+0000","last_change":"2024-03-30T13:42:49.324555+0000","last_active":"2024-03-30T13:44:02.679356+0000","last_peered":"2024-03-30T13:44:02.679356+0000","last_clean":"2024-03-30T13:44:02.679356+0000","last_became_active":"2024-03-30T13:42:49.323924+0000","last_became_peered":"2024-03-30T13:42:49.323924+0000","last_unstale":"2024-03-30T13:44:02.679356+0000","last_undegraded":"2024-03-30T13:44:02.679356+0000","last_fullsized":"2024-03-30T13:44:02.679356+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678095+0000","last_change":"2024-03-30T13:42:45.319317+0000","last_active":"2024-03-30T13:44:02.678095+0000","last_peered":"2024-03-30T13:44:02.678095+0000","last_clean":"2024-03-30T13:44:02.678095+0000","last_became_active":"2024-03-30T13:42:45.318956+0000","last_became_peered":"2024-03-30T13:42:45.318956+0000","last_unstale":"2024-03-30T13:44:02.678095+0000","last_undegraded":"2024-03-30T13:44:02.678095+0000","last_fullsized":"2024-03-30T13:44:02.678095+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:08.614119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.296836+0000","last_change":"2024-03-30T13:42:47.502035+0000","last_active":"2024-03-30T13:44:02.296836+0000","last_peered":"2024-03-30T13:44:02.296836+0000","last_clean":"2024-03-30T13:44:02.296836+0000","last_became_active":"2024-03-30T13:42:47.501426+0000","last_became_peered":"2024-03-30T13:42:47.501426+0000","last_unstale":"2024-03-30T13:44:02.296836+0000","last_undegraded":"2024-03-30T13:44:02.296836+0000","last_fullsized":"2024-03-30T13:44:02.296836+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678461+0000","last_change":"2024-03-30T13:42:51.371670+0000","last_active":"2024-03-30T13:44:02.678461+0000","last_peered":"2024-03-30T13:44:02.678461+0000","last_clean":"2024-03-30T13:44:02.678461+0000","last_became_active":"2024-03-30T13:42:51.371360+0000","last_became_peered":"2024-03-30T13:42:51.371360+0000","last_unstale":"2024-03-30T13:44:02.678461+0000","last_undegraded":"2024-03-30T13:44:02.678461+0000","last_fullsized":"2024-03-30T13:44:02.678461+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":80,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.407361+0000","last_change":"2024-03-30T13:43:45.701210+0000","last_active":"2024-03-30T13:44:02.407361+0000","last_peered":"2024-03-30T13:44:02.407361+0000","last_clean":"2024-03-30T13:44:02.407361+0000","last_became_active":"2024-03-30T13:42:45.318963+0000","last_became_peered":"2024-03-30T13:42:45.318963+0000","last_unstale":"2024-03-30T13:44:02.407361+0000","last_undegraded":"2024-03-30T13:44:02.407361+0000","last_fullsized":"2024-03-30T13:44:02.407361+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-30T13:43:45.701065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:43:45.701065+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:04.819362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.677969+0000","last_change":"2024-03-30T13:42:47.502290+0000","last_active":"2024-03-30T13:44:02.677969+0000","last_peered":"2024-03-30T13:44:02.677969+0000","last_clean":"2024-03-30T13:44:02.677969+0000","last_became_active":"2024-03-30T13:42:47.501937+0000","last_became_peered":"2024-03-30T13:42:47.501937+0000","last_unstale":"2024-03-30T13:44:02.677969+0000","last_undegraded":"2024-03-30T13:44:02.677969+0000","last_fullsized":"2024-03-30T13:44:02.677969+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:03.534390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"72'8","reported_seq":84,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406597+0000","last_change":"2024-03-30T13:43:56.742250+0000","last_active":"2024-03-30T13:44:02.406597+0000","last_peered":"2024-03-30T13:44:02.406597+0000","last_clean":"2024-03-30T13:44:02.406597+0000","last_became_active":"2024-03-30T13:42:49.322675+0000","last_became_peered":"2024-03-30T13:42:49.322675+0000","last_unstale":"2024-03-30T13:44:02.406597+0000","last_undegraded":"2024-03-30T13:44:02.406597+0000","last_fullsized":"2024-03-30T13:44:02.406597+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:43:56.742129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:56.742129+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:17.131326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295433+0000","last_change":"2024-03-30T13:42:51.335656+0000","last_active":"2024-03-30T13:44:02.295433+0000","last_peered":"2024-03-30T13:44:02.295433+0000","last_clean":"2024-03-30T13:44:02.295433+0000","last_became_active":"2024-03-30T13:42:51.334672+0000","last_became_peered":"2024-03-30T13:42:51.334672+0000","last_unstale":"2024-03-30T13:44:02.295433+0000","last_undegraded":"2024-03-30T13:44:02.295433+0000","last_fullsized":"2024-03-30T13:44:02.295433+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:10.851044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.407306+0000","last_change":"2024-03-30T13:42:45.313614+0000","last_active":"2024-03-30T13:44:02.407306+0000","last_peered":"2024-03-30T13:44:02.407306+0000","last_clean":"2024-03-30T13:44:02.407306+0000","last_became_active":"2024-03-30T13:42:45.312328+0000","last_became_peered":"2024-03-30T13:42:45.312328+0000","last_unstale":"2024-03-30T13:44:02.407306+0000","last_undegraded":"2024-03-30T13:44:02.407306+0000","last_fullsized":"2024-03-30T13:44:02.407306+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":90,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297909+0000","last_change":"2024-03-30T13:43:51.992642+0000","last_active":"2024-03-30T13:44:02.297909+0000","last_peered":"2024-03-30T13:44:02.297909+0000","last_clean":"2024-03-30T13:44:02.297909+0000","last_became_active":"2024-03-30T13:42:47.316187+0000","last_became_peered":"2024-03-30T13:42:47.316187+0000","last_unstale":"2024-03-30T13:44:02.297909+0000","last_undegraded":"2024-03-30T13:44:02.297909+0000","last_fullsized":"2024-03-30T13:44:02.297909+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-30T13:43:51.992552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:51.992552+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:09.690586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.294696+0000","last_change":"2024-03-30T13:42:49.322920+0000","last_active":"2024-03-30T13:44:02.294696+0000","last_peered":"2024-03-30T13:44:02.294696+0000","last_clean":"2024-03-30T13:44:02.294696+0000","last_became_active":"2024-03-30T13:42:49.322478+0000","last_became_peered":"2024-03-30T13:42:49.322478+0000","last_unstale":"2024-03-30T13:44:02.294696+0000","last_undegraded":"2024-03-30T13:44:02.294696+0000","last_fullsized":"2024-03-30T13:44:02.294696+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:05.443637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406209+0000","last_change":"2024-03-30T13:42:51.334086+0000","last_active":"2024-03-30T13:44:02.406209+0000","last_peered":"2024-03-30T13:44:02.406209+0000","last_clean":"2024-03-30T13:44:02.406209+0000","last_became_active":"2024-03-30T13:42:51.333773+0000","last_became_peered":"2024-03-30T13:42:51.333773+0000","last_unstale":"2024-03-30T13:44:02.406209+0000","last_undegraded":"2024-03-30T13:44:02.406209+0000","last_fullsized":"2024-03-30T13:44:02.406209+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.296758+0000","last_change":"2024-03-30T13:44:01.850315+0000","last_active":"2024-03-30T13:44:02.296758+0000","last_peered":"2024-03-30T13:44:02.296758+0000","last_clean":"2024-03-30T13:44:02.296758+0000","last_became_active":"2024-03-30T13:42:49.320798+0000","last_became_peered":"2024-03-30T13:42:49.320798+0000","last_unstale":"2024-03-30T13:44:02.296758+0000","last_undegraded":"2024-03-30T13:44:02.296758+0000","last_fullsized":"2024-03-30T13:44:02.296758+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:01.850229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:01.850229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:12.718080+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.407156+0000","last_change":"2024-03-30T13:42:45.316023+0000","last_active":"2024-03-30T13:44:02.407156+0000","last_peered":"2024-03-30T13:44:02.407156+0000","last_clean":"2024-03-30T13:44:02.407156+0000","last_became_active":"2024-03-30T13:42:45.315687+0000","last_became_peered":"2024-03-30T13:42:45.315687+0000","last_unstale":"2024-03-30T13:44:02.407156+0000","last_undegraded":"2024-03-30T13:44:02.407156+0000","last_fullsized":"2024-03-30T13:44:02.407156+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"72'19","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679002+0000","last_change":"2024-03-30T13:42:47.501349+0000","last_active":"2024-03-30T13:44:02.679002+0000","last_peered":"2024-03-30T13:44:02.679002+0000","last_clean":"2024-03-30T13:44:02.679002+0000","last_became_active":"2024-03-30T13:42:47.500898+0000","last_became_peered":"2024-03-30T13:42:47.500898+0000","last_unstale":"2024-03-30T13:44:02.679002+0000","last_undegraded":"2024-03-30T13:44:02.679002+0000","last_fullsized":"2024-03-30T13:44:02.679002+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:10.044571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.404868+0000","last_change":"2024-03-30T13:42:51.330739+0000","last_active":"2024-03-30T13:44:02.404868+0000","last_peered":"2024-03-30T13:44:02.404868+0000","last_clean":"2024-03-30T13:44:02.404868+0000","last_became_active":"2024-03-30T13:42:51.330403+0000","last_became_peered":"2024-03-30T13:42:51.330403+0000","last_unstale":"2024-03-30T13:44:02.404868+0000","last_undegraded":"2024-03-30T13:44:02.404868+0000","last_fullsized":"2024-03-30T13:44:02.404868+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:19.083830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295324+0000","last_change":"2024-03-30T13:43:51.840857+0000","last_active":"2024-03-30T13:44:02.295324+0000","last_peered":"2024-03-30T13:44:02.295324+0000","last_clean":"2024-03-30T13:44:02.295324+0000","last_became_active":"2024-03-30T13:42:49.324022+0000","last_became_peered":"2024-03-30T13:42:49.324022+0000","last_unstale":"2024-03-30T13:44:02.295324+0000","last_undegraded":"2024-03-30T13:44:02.295324+0000","last_fullsized":"2024-03-30T13:44:02.295324+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:51.840740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:51.840740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:00.887228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678645+0000","last_change":"2024-03-30T13:42:45.315058+0000","last_active":"2024-03-30T13:44:02.678645+0000","last_peered":"2024-03-30T13:44:02.678645+0000","last_clean":"2024-03-30T13:44:02.678645+0000","last_became_active":"2024-03-30T13:42:45.314767+0000","last_became_peered":"2024-03-30T13:42:45.314767+0000","last_unstale":"2024-03-30T13:44:02.678645+0000","last_undegraded":"2024-03-30T13:44:02.678645+0000","last_fullsized":"2024-03-30T13:44:02.678645+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":96,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405384+0000","last_change":"2024-03-30T13:43:57.361388+0000","last_active":"2024-03-30T13:44:02.405384+0000","last_peered":"2024-03-30T13:44:02.405384+0000","last_clean":"2024-03-30T13:44:02.405384+0000","last_became_active":"2024-03-30T13:43:11.446379+0000","last_became_peered":"2024-03-30T13:43:11.446379+0000","last_unstale":"2024-03-30T13:44:02.405384+0000","last_undegraded":"2024-03-30T13:44:02.405384+0000","last_fullsized":"2024-03-30T13:44:02.405384+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:43:57.361300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:57.361300+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:13.159418+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.296205+0000","last_change":"2024-03-30T13:42:51.332588+0000","last_active":"2024-03-30T13:44:02.296205+0000","last_peered":"2024-03-30T13:44:02.296205+0000","last_clean":"2024-03-30T13:44:02.296205+0000","last_became_active":"2024-03-30T13:42:51.331054+0000","last_became_peered":"2024-03-30T13:42:51.331054+0000","last_unstale":"2024-03-30T13:44:02.296205+0000","last_undegraded":"2024-03-30T13:44:02.296205+0000","last_fullsized":"2024-03-30T13:44:02.296205+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678183+0000","last_change":"2024-03-30T13:42:48.314967+0000","last_active":"2024-03-30T13:44:02.678183+0000","last_peered":"2024-03-30T13:44:02.678183+0000","last_clean":"2024-03-30T13:44:02.678183+0000","last_became_active":"2024-03-30T13:42:44.300789+0000","last_became_peered":"2024-03-30T13:42:44.300789+0000","last_unstale":"2024-03-30T13:44:02.678183+0000","last_undegraded":"2024-03-30T13:44:02.678183+0000","last_fullsized":"2024-03-30T13:44:02.678183+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:42:43.281063+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021837800000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297788+0000","last_change":"2024-03-30T13:42:45.310512+0000","last_active":"2024-03-30T13:44:02.297788+0000","last_peered":"2024-03-30T13:44:02.297788+0000","last_clean":"2024-03-30T13:44:02.297788+0000","last_became_active":"2024-03-30T13:42:45.310209+0000","last_became_peered":"2024-03-30T13:42:45.310209+0000","last_unstale":"2024-03-30T13:44:02.297788+0000","last_undegraded":"2024-03-30T13:44:02.297788+0000","last_fullsized":"2024-03-30T13:44:02.297788+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:04.787006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"72'28","reported_seq":122,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678261+0000","last_change":"2024-03-30T13:42:47.324184+0000","last_active":"2024-03-30T13:44:02.678261+0000","last_peered":"2024-03-30T13:44:02.678261+0000","last_clean":"2024-03-30T13:44:02.678261+0000","last_became_active":"2024-03-30T13:42:47.323845+0000","last_became_peered":"2024-03-30T13:42:47.323845+0000","last_unstale":"2024-03-30T13:44:02.678261+0000","last_undegraded":"2024-03-30T13:44:02.678261+0000","last_fullsized":"2024-03-30T13:44:02.678261+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:07.498588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405223+0000","last_change":"2024-03-30T13:43:54.326967+0000","last_active":"2024-03-30T13:44:02.405223+0000","last_peered":"2024-03-30T13:44:02.405223+0000","last_clean":"2024-03-30T13:44:02.405223+0000","last_became_active":"2024-03-30T13:42:49.322154+0000","last_became_peered":"2024-03-30T13:42:49.322154+0000","last_unstale":"2024-03-30T13:44:02.405223+0000","last_undegraded":"2024-03-30T13:44:02.405223+0000","last_fullsized":"2024-03-30T13:44:02.405223+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:54.326829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:54.326829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:09.771943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.407339+0000","last_change":"2024-03-30T13:42:51.372579+0000","last_active":"2024-03-30T13:44:02.407339+0000","last_peered":"2024-03-30T13:44:02.407339+0000","last_clean":"2024-03-30T13:44:02.407339+0000","last_became_active":"2024-03-30T13:42:51.372320+0000","last_became_peered":"2024-03-30T13:42:51.372320+0000","last_unstale":"2024-03-30T13:44:02.407339+0000","last_undegraded":"2024-03-30T13:44:02.407339+0000","last_fullsized":"2024-03-30T13:44:02.407339+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:10.769625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678026+0000","last_change":"2024-03-30T13:43:59.810299+0000","last_active":"2024-03-30T13:44:02.678026+0000","last_peered":"2024-03-30T13:44:02.678026+0000","last_clean":"2024-03-30T13:44:02.678026+0000","last_became_active":"2024-03-30T13:42:45.314771+0000","last_became_peered":"2024-03-30T13:42:45.314771+0000","last_unstale":"2024-03-30T13:44:02.678026+0000","last_undegraded":"2024-03-30T13:44:02.678026+0000","last_fullsized":"2024-03-30T13:44:02.678026+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:59.810189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:59.810189+0000","last_clean_scrub_stamp":"2024-03-30T13:43:59.810189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:06.406345+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":80,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295022+0000","last_change":"2024-03-30T13:43:57.866600+0000","last_active":"2024-03-30T13:44:02.295022+0000","last_peered":"2024-03-30T13:44:02.295022+0000","last_clean":"2024-03-30T13:44:02.295022+0000","last_became_active":"2024-03-30T13:42:44.302344+0000","last_became_peered":"2024-03-30T13:42:44.302344+0000","last_unstale":"2024-03-30T13:44:02.295022+0000","last_undegraded":"2024-03-30T13:44:02.295022+0000","last_fullsized":"2024-03-30T13:44:02.295022+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:57.866519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:43:57.866519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:59.119000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001401518,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"72'13","reported_seq":85,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678489+0000","last_change":"2024-03-30T13:42:47.321918+0000","last_active":"2024-03-30T13:44:02.678489+0000","last_peered":"2024-03-30T13:44:02.678489+0000","last_clean":"2024-03-30T13:44:02.678489+0000","last_became_active":"2024-03-30T13:42:47.320541+0000","last_became_peered":"2024-03-30T13:42:47.320541+0000","last_unstale":"2024-03-30T13:44:02.678489+0000","last_undegraded":"2024-03-30T13:44:02.678489+0000","last_fullsized":"2024-03-30T13:44:02.678489+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:05.609369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295695+0000","last_change":"2024-03-30T13:42:49.320258+0000","last_active":"2024-03-30T13:44:02.295695+0000","last_peered":"2024-03-30T13:44:02.295695+0000","last_clean":"2024-03-30T13:44:02.295695+0000","last_became_active":"2024-03-30T13:42:49.319826+0000","last_became_peered":"2024-03-30T13:42:49.319826+0000","last_unstale":"2024-03-30T13:44:02.295695+0000","last_undegraded":"2024-03-30T13:44:02.295695+0000","last_fullsized":"2024-03-30T13:44:02.295695+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:13.062008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679597+0000","last_change":"2024-03-30T13:42:51.331412+0000","last_active":"2024-03-30T13:44:02.679597+0000","last_peered":"2024-03-30T13:44:02.679597+0000","last_clean":"2024-03-30T13:44:02.679597+0000","last_became_active":"2024-03-30T13:42:51.331081+0000","last_became_peered":"2024-03-30T13:42:51.331081+0000","last_unstale":"2024-03-30T13:44:02.679597+0000","last_undegraded":"2024-03-30T13:44:02.679597+0000","last_fullsized":"2024-03-30T13:44:02.679597+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":313,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406577+0000","last_change":"2024-03-30T13:43:23.836532+0000","last_active":"2024-03-30T13:44:02.406577+0000","last_peered":"2024-03-30T13:44:02.406577+0000","last_clean":"2024-03-30T13:44:02.406577+0000","last_became_active":"2024-03-30T13:42:34.988246+0000","last_became_peered":"2024-03-30T13:42:34.988246+0000","last_unstale":"2024-03-30T13:44:02.406577+0000","last_undegraded":"2024-03-30T13:44:02.406577+0000","last_fullsized":"2024-03-30T13:44:02.406577+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-03-30T13:43:23.836417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:43:23.836417+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:30.177755+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295171+0000","last_change":"2024-03-30T13:42:45.310322+0000","last_active":"2024-03-30T13:44:02.295171+0000","last_peered":"2024-03-30T13:44:02.295171+0000","last_clean":"2024-03-30T13:44:02.295171+0000","last_became_active":"2024-03-30T13:42:45.309954+0000","last_became_peered":"2024-03-30T13:42:45.309954+0000","last_unstale":"2024-03-30T13:44:02.295171+0000","last_undegraded":"2024-03-30T13:44:02.295171+0000","last_fullsized":"2024-03-30T13:44:02.295171+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":94,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679282+0000","last_change":"2024-03-30T13:42:47.502798+0000","last_active":"2024-03-30T13:44:02.679282+0000","last_peered":"2024-03-30T13:44:02.679282+0000","last_clean":"2024-03-30T13:44:02.679282+0000","last_became_active":"2024-03-30T13:42:47.502338+0000","last_became_peered":"2024-03-30T13:42:47.502338+0000","last_unstale":"2024-03-30T13:44:02.679282+0000","last_undegraded":"2024-03-30T13:44:02.679282+0000","last_fullsized":"2024-03-30T13:44:02.679282+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:03.382508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406924+0000","last_change":"2024-03-30T13:42:49.329181+0000","last_active":"2024-03-30T13:44:02.406924+0000","last_peered":"2024-03-30T13:44:02.406924+0000","last_clean":"2024-03-30T13:44:02.406924+0000","last_became_active":"2024-03-30T13:42:49.328861+0000","last_became_peered":"2024-03-30T13:42:49.328861+0000","last_unstale":"2024-03-30T13:44:02.406924+0000","last_undegraded":"2024-03-30T13:44:02.406924+0000","last_fullsized":"2024-03-30T13:44:02.406924+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:11.705785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678034+0000","last_change":"2024-03-30T13:42:51.335522+0000","last_active":"2024-03-30T13:44:02.678034+0000","last_peered":"2024-03-30T13:44:02.678034+0000","last_clean":"2024-03-30T13:44:02.678034+0000","last_became_active":"2024-03-30T13:42:51.335188+0000","last_became_peered":"2024-03-30T13:42:51.335188+0000","last_unstale":"2024-03-30T13:44:02.678034+0000","last_undegraded":"2024-03-30T13:44:02.678034+0000","last_fullsized":"2024-03-30T13:44:02.678034+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:09.504888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405003+0000","last_change":"2024-03-30T13:42:45.308893+0000","last_active":"2024-03-30T13:44:02.405003+0000","last_peered":"2024-03-30T13:44:02.405003+0000","last_clean":"2024-03-30T13:44:02.405003+0000","last_became_active":"2024-03-30T13:42:45.307905+0000","last_became_peered":"2024-03-30T13:42:45.307905+0000","last_unstale":"2024-03-30T13:44:02.405003+0000","last_undegraded":"2024-03-30T13:44:02.405003+0000","last_fullsized":"2024-03-30T13:44:02.405003+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:03.325741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":143,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406574+0000","last_change":"2024-03-30T13:42:48.313590+0000","last_active":"2024-03-30T13:44:02.406574+0000","last_peered":"2024-03-30T13:44:02.406574+0000","last_clean":"2024-03-30T13:44:02.406574+0000","last_became_active":"2024-03-30T13:42:44.302343+0000","last_became_peered":"2024-03-30T13:42:44.302343+0000","last_unstale":"2024-03-30T13:44:02.406574+0000","last_undegraded":"2024-03-30T13:44:02.406574+0000","last_fullsized":"2024-03-30T13:44:02.406574+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:42:43.281063+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:10.824760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00126391,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":71,"num_read_kb":66,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":74,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405204+0000","last_change":"2024-03-30T13:42:47.319576+0000","last_active":"2024-03-30T13:44:02.405204+0000","last_peered":"2024-03-30T13:44:02.405204+0000","last_clean":"2024-03-30T13:44:02.405204+0000","last_became_active":"2024-03-30T13:42:47.319155+0000","last_became_peered":"2024-03-30T13:42:47.319155+0000","last_unstale":"2024-03-30T13:44:02.405204+0000","last_undegraded":"2024-03-30T13:44:02.405204+0000","last_fullsized":"2024-03-30T13:44:02.405204+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:08.029848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679775+0000","last_change":"2024-03-30T13:42:49.322962+0000","last_active":"2024-03-30T13:44:02.679775+0000","last_peered":"2024-03-30T13:44:02.679775+0000","last_clean":"2024-03-30T13:44:02.679775+0000","last_became_active":"2024-03-30T13:42:49.322700+0000","last_became_peered":"2024-03-30T13:42:49.322700+0000","last_unstale":"2024-03-30T13:44:02.679775+0000","last_undegraded":"2024-03-30T13:44:02.679775+0000","last_fullsized":"2024-03-30T13:44:02.679775+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:08.787689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.677794+0000","last_change":"2024-03-30T13:42:51.371691+0000","last_active":"2024-03-30T13:44:02.677794+0000","last_peered":"2024-03-30T13:44:02.677794+0000","last_clean":"2024-03-30T13:44:02.677794+0000","last_became_active":"2024-03-30T13:42:51.371202+0000","last_became_peered":"2024-03-30T13:42:51.371202+0000","last_unstale":"2024-03-30T13:44:02.677794+0000","last_undegraded":"2024-03-30T13:44:02.677794+0000","last_fullsized":"2024-03-30T13:44:02.677794+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678266+0000","last_change":"2024-03-30T13:42:45.310437+0000","last_active":"2024-03-30T13:44:02.678266+0000","last_peered":"2024-03-30T13:44:02.678266+0000","last_clean":"2024-03-30T13:44:02.678266+0000","last_became_active":"2024-03-30T13:42:45.310134+0000","last_became_peered":"2024-03-30T13:42:45.310134+0000","last_unstale":"2024-03-30T13:44:02.678266+0000","last_undegraded":"2024-03-30T13:44:02.678266+0000","last_fullsized":"2024-03-30T13:44:02.678266+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":92,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405502+0000","last_change":"2024-03-30T13:42:47.501567+0000","last_active":"2024-03-30T13:44:02.405502+0000","last_peered":"2024-03-30T13:44:02.405502+0000","last_clean":"2024-03-30T13:44:02.405502+0000","last_became_active":"2024-03-30T13:42:47.501272+0000","last_became_peered":"2024-03-30T13:42:47.501272+0000","last_unstale":"2024-03-30T13:44:02.405502+0000","last_undegraded":"2024-03-30T13:44:02.405502+0000","last_fullsized":"2024-03-30T13:44:02.405502+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:03.584110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679171+0000","last_change":"2024-03-30T13:42:49.321802+0000","last_active":"2024-03-30T13:44:02.679171+0000","last_peered":"2024-03-30T13:44:02.679171+0000","last_clean":"2024-03-30T13:44:02.679171+0000","last_became_active":"2024-03-30T13:42:49.321520+0000","last_became_peered":"2024-03-30T13:42:49.321520+0000","last_unstale":"2024-03-30T13:44:02.679171+0000","last_undegraded":"2024-03-30T13:44:02.679171+0000","last_fullsized":"2024-03-30T13:44:02.679171+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:06.176165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.404794+0000","last_change":"2024-03-30T13:42:51.337231+0000","last_active":"2024-03-30T13:44:02.404794+0000","last_peered":"2024-03-30T13:44:02.404794+0000","last_clean":"2024-03-30T13:44:02.404794+0000","last_became_active":"2024-03-30T13:42:51.336892+0000","last_became_peered":"2024-03-30T13:42:51.336892+0000","last_unstale":"2024-03-30T13:44:02.404794+0000","last_undegraded":"2024-03-30T13:44:02.404794+0000","last_fullsized":"2024-03-30T13:44:02.404794+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678999+0000","last_change":"2024-03-30T13:43:57.299723+0000","last_active":"2024-03-30T13:44:02.678999+0000","last_peered":"2024-03-30T13:44:02.678999+0000","last_clean":"2024-03-30T13:44:02.678999+0000","last_became_active":"2024-03-30T13:42:45.317136+0000","last_became_peered":"2024-03-30T13:42:45.317136+0000","last_unstale":"2024-03-30T13:44:02.678999+0000","last_undegraded":"2024-03-30T13:44:02.678999+0000","last_fullsized":"2024-03-30T13:44:02.678999+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:57.299618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:57.299618+0000","last_clean_scrub_stamp":"2024-03-30T13:43:57.299618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:09.120401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678084+0000","last_change":"2024-03-30T13:42:47.322034+0000","last_active":"2024-03-30T13:44:02.678084+0000","last_peered":"2024-03-30T13:44:02.678084+0000","last_clean":"2024-03-30T13:44:02.678084+0000","last_became_active":"2024-03-30T13:42:47.321433+0000","last_became_peered":"2024-03-30T13:42:47.321433+0000","last_unstale":"2024-03-30T13:44:02.678084+0000","last_undegraded":"2024-03-30T13:44:02.678084+0000","last_fullsized":"2024-03-30T13:44:02.678084+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:04.775648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405571+0000","last_change":"2024-03-30T13:42:49.326702+0000","last_active":"2024-03-30T13:44:02.405571+0000","last_peered":"2024-03-30T13:44:02.405571+0000","last_clean":"2024-03-30T13:44:02.405571+0000","last_became_active":"2024-03-30T13:42:49.325533+0000","last_became_peered":"2024-03-30T13:42:49.325533+0000","last_unstale":"2024-03-30T13:44:02.405571+0000","last_undegraded":"2024-03-30T13:44:02.405571+0000","last_fullsized":"2024-03-30T13:44:02.405571+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:16.967785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406634+0000","last_change":"2024-03-30T13:42:51.374719+0000","last_active":"2024-03-30T13:44:02.406634+0000","last_peered":"2024-03-30T13:44:02.406634+0000","last_clean":"2024-03-30T13:44:02.406634+0000","last_became_active":"2024-03-30T13:42:51.374437+0000","last_became_peered":"2024-03-30T13:42:51.374437+0000","last_unstale":"2024-03-30T13:44:02.406634+0000","last_undegraded":"2024-03-30T13:44:02.406634+0000","last_fullsized":"2024-03-30T13:44:02.406634+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:09.130892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405290+0000","last_change":"2024-03-30T13:42:45.310288+0000","last_active":"2024-03-30T13:44:02.405290+0000","last_peered":"2024-03-30T13:44:02.405290+0000","last_clean":"2024-03-30T13:44:02.405290+0000","last_became_active":"2024-03-30T13:42:45.309886+0000","last_became_peered":"2024-03-30T13:42:45.309886+0000","last_unstale":"2024-03-30T13:44:02.405290+0000","last_undegraded":"2024-03-30T13:44:02.405290+0000","last_fullsized":"2024-03-30T13:44:02.405290+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":80,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297776+0000","last_change":"2024-03-30T13:42:47.314561+0000","last_active":"2024-03-30T13:44:02.297776+0000","last_peered":"2024-03-30T13:44:02.297776+0000","last_clean":"2024-03-30T13:44:02.297776+0000","last_became_active":"2024-03-30T13:42:47.314184+0000","last_became_peered":"2024-03-30T13:42:47.314184+0000","last_unstale":"2024-03-30T13:44:02.297776+0000","last_undegraded":"2024-03-30T13:44:02.297776+0000","last_fullsized":"2024-03-30T13:44:02.297776+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295361+0000","last_change":"2024-03-30T13:42:49.321877+0000","last_active":"2024-03-30T13:44:02.295361+0000","last_peered":"2024-03-30T13:44:02.295361+0000","last_clean":"2024-03-30T13:44:02.295361+0000","last_became_active":"2024-03-30T13:42:49.321423+0000","last_became_peered":"2024-03-30T13:42:49.321423+0000","last_unstale":"2024-03-30T13:44:02.295361+0000","last_undegraded":"2024-03-30T13:44:02.295361+0000","last_fullsized":"2024-03-30T13:44:02.295361+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297679+0000","last_change":"2024-03-30T13:42:51.327321+0000","last_active":"2024-03-30T13:44:02.297679+0000","last_peered":"2024-03-30T13:44:02.297679+0000","last_clean":"2024-03-30T13:44:02.297679+0000","last_became_active":"2024-03-30T13:42:51.327003+0000","last_became_peered":"2024-03-30T13:42:51.327003+0000","last_unstale":"2024-03-30T13:44:02.297679+0000","last_undegraded":"2024-03-30T13:44:02.297679+0000","last_fullsized":"2024-03-30T13:44:02.297679+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.296752+0000","last_change":"2024-03-30T13:42:45.314830+0000","last_active":"2024-03-30T13:44:02.296752+0000","last_peered":"2024-03-30T13:44:02.296752+0000","last_clean":"2024-03-30T13:44:02.296752+0000","last_became_active":"2024-03-30T13:42:45.314533+0000","last_became_peered":"2024-03-30T13:42:45.314533+0000","last_unstale":"2024-03-30T13:44:02.296752+0000","last_undegraded":"2024-03-30T13:44:02.296752+0000","last_fullsized":"2024-03-30T13:44:02.296752+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:03.121327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":90,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295258+0000","last_change":"2024-03-30T13:42:47.503315+0000","last_active":"2024-03-30T13:44:02.295258+0000","last_peered":"2024-03-30T13:44:02.295258+0000","last_clean":"2024-03-30T13:44:02.295258+0000","last_became_active":"2024-03-30T13:42:47.502708+0000","last_became_peered":"2024-03-30T13:42:47.502708+0000","last_unstale":"2024-03-30T13:44:02.295258+0000","last_undegraded":"2024-03-30T13:44:02.295258+0000","last_fullsized":"2024-03-30T13:44:02.295258+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297579+0000","last_change":"2024-03-30T13:42:49.324517+0000","last_active":"2024-03-30T13:44:02.297579+0000","last_peered":"2024-03-30T13:44:02.297579+0000","last_clean":"2024-03-30T13:44:02.297579+0000","last_became_active":"2024-03-30T13:42:49.324022+0000","last_became_peered":"2024-03-30T13:42:49.324022+0000","last_unstale":"2024-03-30T13:44:02.297579+0000","last_undegraded":"2024-03-30T13:44:02.297579+0000","last_fullsized":"2024-03-30T13:44:02.297579+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:08.917548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678368+0000","last_change":"2024-03-30T13:42:51.329317+0000","last_active":"2024-03-30T13:44:02.678368+0000","last_peered":"2024-03-30T13:44:02.678368+0000","last_clean":"2024-03-30T13:44:02.678368+0000","last_became_active":"2024-03-30T13:42:51.329026+0000","last_became_peered":"2024-03-30T13:42:51.329026+0000","last_unstale":"2024-03-30T13:44:02.678368+0000","last_undegraded":"2024-03-30T13:44:02.678368+0000","last_fullsized":"2024-03-30T13:44:02.678368+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:04.561073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.298639+0000","last_change":"2024-03-30T13:43:58.942488+0000","last_active":"2024-03-30T13:44:02.298639+0000","last_peered":"2024-03-30T13:44:02.298639+0000","last_clean":"2024-03-30T13:44:02.298639+0000","last_became_active":"2024-03-30T13:42:49.321790+0000","last_became_peered":"2024-03-30T13:42:49.321790+0000","last_unstale":"2024-03-30T13:44:02.298639+0000","last_undegraded":"2024-03-30T13:44:02.298639+0000","last_fullsized":"2024-03-30T13:44:02.298639+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:43:58.942311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:58.942311+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:17.003904+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295160+0000","last_change":"2024-03-30T13:42:45.315666+0000","last_active":"2024-03-30T13:44:02.295160+0000","last_peered":"2024-03-30T13:44:02.295160+0000","last_clean":"2024-03-30T13:44:02.295160+0000","last_became_active":"2024-03-30T13:42:45.315319+0000","last_became_peered":"2024-03-30T13:42:45.315319+0000","last_unstale":"2024-03-30T13:44:02.295160+0000","last_undegraded":"2024-03-30T13:44:02.295160+0000","last_fullsized":"2024-03-30T13:44:02.295160+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:11.352775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678538+0000","last_change":"2024-03-30T13:42:47.324692+0000","last_active":"2024-03-30T13:44:02.678538+0000","last_peered":"2024-03-30T13:44:02.678538+0000","last_clean":"2024-03-30T13:44:02.678538+0000","last_became_active":"2024-03-30T13:42:47.324350+0000","last_became_peered":"2024-03-30T13:42:47.324350+0000","last_unstale":"2024-03-30T13:44:02.678538+0000","last_undegraded":"2024-03-30T13:44:02.678538+0000","last_fullsized":"2024-03-30T13:44:02.678538+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:10.515261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295562+0000","last_change":"2024-03-30T13:42:51.371742+0000","last_active":"2024-03-30T13:44:02.295562+0000","last_peered":"2024-03-30T13:44:02.295562+0000","last_clean":"2024-03-30T13:44:02.295562+0000","last_became_active":"2024-03-30T13:42:51.371369+0000","last_became_peered":"2024-03-30T13:42:51.371369+0000","last_unstale":"2024-03-30T13:44:02.295562+0000","last_undegraded":"2024-03-30T13:44:02.295562+0000","last_fullsized":"2024-03-30T13:44:02.295562+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:19.941761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679775+0000","last_change":"2024-03-30T13:42:49.323595+0000","last_active":"2024-03-30T13:44:02.679775+0000","last_peered":"2024-03-30T13:44:02.679775+0000","last_clean":"2024-03-30T13:44:02.679775+0000","last_became_active":"2024-03-30T13:42:49.323319+0000","last_became_peered":"2024-03-30T13:42:49.323319+0000","last_unstale":"2024-03-30T13:44:02.679775+0000","last_undegraded":"2024-03-30T13:44:02.679775+0000","last_fullsized":"2024-03-30T13:44:02.679775+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:03.310676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297628+0000","last_change":"2024-03-30T13:43:49.990189+0000","last_active":"2024-03-30T13:44:02.297628+0000","last_peered":"2024-03-30T13:44:02.297628+0000","last_clean":"2024-03-30T13:44:02.297628+0000","last_became_active":"2024-03-30T13:42:45.315018+0000","last_became_peered":"2024-03-30T13:42:45.315018+0000","last_unstale":"2024-03-30T13:44:02.297628+0000","last_undegraded":"2024-03-30T13:44:02.297628+0000","last_fullsized":"2024-03-30T13:44:02.297628+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:49.990065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:43:49.990065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:58.482640+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"72'11","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.298500+0000","last_change":"2024-03-30T13:42:47.315984+0000","last_active":"2024-03-30T13:44:02.298500+0000","last_peered":"2024-03-30T13:44:02.298500+0000","last_clean":"2024-03-30T13:44:02.298500+0000","last_became_active":"2024-03-30T13:42:47.315667+0000","last_became_peered":"2024-03-30T13:42:47.315667+0000","last_unstale":"2024-03-30T13:44:02.298500+0000","last_undegraded":"2024-03-30T13:44:02.298500+0000","last_fullsized":"2024-03-30T13:44:02.298500+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.296059+0000","last_change":"2024-03-30T13:42:51.331741+0000","last_active":"2024-03-30T13:44:02.296059+0000","last_peered":"2024-03-30T13:44:02.296059+0000","last_clean":"2024-03-30T13:44:02.296059+0000","last_became_active":"2024-03-30T13:42:51.331372+0000","last_became_peered":"2024-03-30T13:42:51.331372+0000","last_unstale":"2024-03-30T13:44:02.296059+0000","last_undegraded":"2024-03-30T13:44:02.296059+0000","last_fullsized":"2024-03-30T13:44:02.296059+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:17.417830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.833967+0000","last_change":"2024-03-30T13:44:02.833988+0000","last_active":"2024-03-30T13:44:02.833967+0000","last_peered":"2024-03-30T13:44:02.833967+0000","last_clean":"2024-03-30T13:44:02.833967+0000","last_became_active":"2024-03-30T13:42:49.321360+0000","last_became_peered":"2024-03-30T13:42:49.321360+0000","last_unstale":"2024-03-30T13:44:02.833967+0000","last_undegraded":"2024-03-30T13:44:02.833967+0000","last_fullsized":"2024-03-30T13:44:02.833967+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:02.833909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:02.833909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:31.595404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678731+0000","last_change":"2024-03-30T13:42:45.315544+0000","last_active":"2024-03-30T13:44:02.678731+0000","last_peered":"2024-03-30T13:44:02.678731+0000","last_clean":"2024-03-30T13:44:02.678731+0000","last_became_active":"2024-03-30T13:42:45.315251+0000","last_became_peered":"2024-03-30T13:42:45.315251+0000","last_unstale":"2024-03-30T13:44:02.678731+0000","last_undegraded":"2024-03-30T13:44:02.678731+0000","last_fullsized":"2024-03-30T13:44:02.678731+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.298490+0000","last_change":"2024-03-30T13:42:47.316447+0000","last_active":"2024-03-30T13:44:02.298490+0000","last_peered":"2024-03-30T13:44:02.298490+0000","last_clean":"2024-03-30T13:44:02.298490+0000","last_became_active":"2024-03-30T13:42:47.315495+0000","last_became_peered":"2024-03-30T13:42:47.315495+0000","last_unstale":"2024-03-30T13:44:02.298490+0000","last_undegraded":"2024-03-30T13:44:02.298490+0000","last_fullsized":"2024-03-30T13:44:02.298490+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.298700+0000","last_change":"2024-03-30T13:42:51.325672+0000","last_active":"2024-03-30T13:44:02.298700+0000","last_peered":"2024-03-30T13:44:02.298700+0000","last_clean":"2024-03-30T13:44:02.298700+0000","last_became_active":"2024-03-30T13:42:51.325389+0000","last_became_peered":"2024-03-30T13:42:51.325389+0000","last_unstale":"2024-03-30T13:44:02.298700+0000","last_undegraded":"2024-03-30T13:44:02.298700+0000","last_fullsized":"2024-03-30T13:44:02.298700+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"72'8","reported_seq":90,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295458+0000","last_change":"2024-03-30T13:43:50.850857+0000","last_active":"2024-03-30T13:44:02.295458+0000","last_peered":"2024-03-30T13:44:02.295458+0000","last_clean":"2024-03-30T13:44:02.295458+0000","last_became_active":"2024-03-30T13:42:49.323371+0000","last_became_peered":"2024-03-30T13:42:49.323371+0000","last_unstale":"2024-03-30T13:44:02.295458+0000","last_undegraded":"2024-03-30T13:44:02.295458+0000","last_fullsized":"2024-03-30T13:44:02.295458+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:43:50.850759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:50.850759+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:01.654480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295588+0000","last_change":"2024-03-30T13:43:50.050389+0000","last_active":"2024-03-30T13:44:02.295588+0000","last_peered":"2024-03-30T13:44:02.295588+0000","last_clean":"2024-03-30T13:44:02.295588+0000","last_became_active":"2024-03-30T13:42:45.309265+0000","last_became_peered":"2024-03-30T13:42:45.309265+0000","last_unstale":"2024-03-30T13:44:02.295588+0000","last_undegraded":"2024-03-30T13:44:02.295588+0000","last_fullsized":"2024-03-30T13:44:02.295588+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:50.050303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:43:50.050303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:05.509528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":74,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.298126+0000","last_change":"2024-03-30T13:42:47.313353+0000","last_active":"2024-03-30T13:44:02.298126+0000","last_peered":"2024-03-30T13:44:02.298126+0000","last_clean":"2024-03-30T13:44:02.298126+0000","last_became_active":"2024-03-30T13:42:47.313026+0000","last_became_peered":"2024-03-30T13:42:47.313026+0000","last_unstale":"2024-03-30T13:44:02.298126+0000","last_undegraded":"2024-03-30T13:44:02.298126+0000","last_fullsized":"2024-03-30T13:44:02.298126+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:03.773018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":400,"ondisk_log_size":400,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":66,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103189,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32684,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712468,"statfs":{"total":95995035648,"available":95961567232,"internally_reserved":0,"allocated":1540096,"data_stored":1039826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397529,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32664,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712488,"statfs":{"total":95995035648,"available":95961587712,"internally_reserved":0,"allocated":1519616,"data_stored":1038774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":450356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32676,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712476,"statfs":{"total":95995035648,"available":95961575424,"internally_reserved":0,"allocated":1531904,"data_stored":1032722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":925696,"data_stored":449493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182441,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":449284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953507,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":449409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691870,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":925696,"data_stored":442394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:44:09.573 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T13:44:09.819 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:44:09.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[42099]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:44:09.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:44:09.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[42099]: 3.1 scrub starts 2024-03-30T13:44:09.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[42099]: 3.1 scrub ok 2024-03-30T13:44:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[42099]: osdmap e92: 8 total, 8 up, 8 in 2024-03-30T13:44:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[42099]: 3.a scrub starts 2024-03-30T13:44:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[42099]: 3.a scrub ok 2024-03-30T13:44:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[42099]: 5.1d scrub starts 2024-03-30T13:44:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[42099]: 5.1d scrub ok 2024-03-30T13:44:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[42099]: 6.11 scrub starts 2024-03-30T13:44:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[42099]: 6.11 scrub ok 2024-03-30T13:44:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T13:44:09.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:09.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[29963]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:44:09.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:44:09.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[29963]: 3.1 scrub starts 2024-03-30T13:44:09.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[29963]: 3.1 scrub ok 2024-03-30T13:44:09.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[29963]: osdmap e92: 8 total, 8 up, 8 in 2024-03-30T13:44:09.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[29963]: 3.a scrub starts 2024-03-30T13:44:09.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[29963]: 3.a scrub ok 2024-03-30T13:44:09.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[29963]: 5.1d scrub starts 2024-03-30T13:44:09.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[29963]: 5.1d scrub ok 2024-03-30T13:44:09.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:09.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:09.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:09.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:09.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[29963]: 6.11 scrub starts 2024-03-30T13:44:09.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[29963]: 6.11 scrub ok 2024-03-30T13:44:09.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:09.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:09.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T13:44:09.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:09 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:09.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:09 smithi113 ceph-mon[34341]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:44:09.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:09 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:44:09.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:09 smithi113 ceph-mon[34341]: 3.1 scrub starts 2024-03-30T13:44:09.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:09 smithi113 ceph-mon[34341]: 3.1 scrub ok 2024-03-30T13:44:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:09 smithi113 ceph-mon[34341]: osdmap e92: 8 total, 8 up, 8 in 2024-03-30T13:44:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:09 smithi113 ceph-mon[34341]: 3.a scrub starts 2024-03-30T13:44:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:09 smithi113 ceph-mon[34341]: 3.a scrub ok 2024-03-30T13:44:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:09 smithi113 ceph-mon[34341]: 5.1d scrub starts 2024-03-30T13:44:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:09 smithi113 ceph-mon[34341]: 5.1d scrub ok 2024-03-30T13:44:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:09 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:09 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:09 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:09 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:09 smithi113 ceph-mon[34341]: 6.11 scrub starts 2024-03-30T13:44:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:09 smithi113 ceph-mon[34341]: 6.11 scrub ok 2024-03-30T13:44:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:09 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:09 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:09.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:09 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T13:44:09.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:09 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:10.899 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:44:10 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[80418]: ts=2024-03-30T13:44:10.822Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000202174s 2024-03-30T13:44:10.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:10 smithi043 ceph-mon[29963]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-30T13:44:10.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:10 smithi043 ceph-mon[29963]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T13:44:10.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:10 smithi043 ceph-mon[29963]: Deploying daemon grafana.a on smithi113 2024-03-30T13:44:10.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:10 smithi043 ceph-mon[29963]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T13:44:10.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:10 smithi043 ceph-mon[29963]: 3.2 scrub starts 2024-03-30T13:44:10.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:10 smithi043 ceph-mon[29963]: 3.2 scrub ok 2024-03-30T13:44:10.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:10 smithi043 ceph-mon[42099]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-30T13:44:10.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:10 smithi043 ceph-mon[42099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T13:44:10.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:10 smithi043 ceph-mon[42099]: Deploying daemon grafana.a on smithi113 2024-03-30T13:44:10.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:10 smithi043 ceph-mon[42099]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T13:44:10.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:10 smithi043 ceph-mon[42099]: 3.2 scrub starts 2024-03-30T13:44:10.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:10 smithi043 ceph-mon[42099]: 3.2 scrub ok 2024-03-30T13:44:10.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:10 smithi113 ceph-mon[34341]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-30T13:44:10.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:10 smithi113 ceph-mon[34341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T13:44:10.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:10 smithi113 ceph-mon[34341]: Deploying daemon grafana.a on smithi113 2024-03-30T13:44:10.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:10 smithi113 ceph-mon[34341]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T13:44:10.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:10 smithi113 ceph-mon[34341]: 3.2 scrub starts 2024-03-30T13:44:10.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:10 smithi113 ceph-mon[34341]: 3.2 scrub ok 2024-03-30T13:44:11.750 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:44:11.752 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-30T13:44:11.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:11 smithi113 ceph-mon[34341]: 5.1f scrub starts 2024-03-30T13:44:11.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:11 smithi113 ceph-mon[34341]: 5.1f scrub ok 2024-03-30T13:44:11.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:11 smithi113 ceph-mon[34341]: 5.8 scrub starts 2024-03-30T13:44:11.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:11 smithi113 ceph-mon[34341]: 5.8 scrub ok 2024-03-30T13:44:12.027 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:11 smithi043 ceph-mon[29963]: 5.1f scrub starts 2024-03-30T13:44:12.027 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:11 smithi043 ceph-mon[29963]: 5.1f scrub ok 2024-03-30T13:44:12.027 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:11 smithi043 ceph-mon[29963]: 5.8 scrub starts 2024-03-30T13:44:12.027 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:11 smithi043 ceph-mon[29963]: 5.8 scrub ok 2024-03-30T13:44:12.028 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:11 smithi043 ceph-mon[42099]: 5.1f scrub starts 2024-03-30T13:44:12.028 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:11 smithi043 ceph-mon[42099]: 5.1f scrub ok 2024-03-30T13:44:12.028 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:11 smithi043 ceph-mon[42099]: 5.8 scrub starts 2024-03-30T13:44:12.028 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:11 smithi043 ceph-mon[42099]: 5.8 scrub ok 2024-03-30T13:44:12.463 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-03-30T13:44:11.279068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1010,"num_read_kb":856,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":614,"ondisk_log_size":614,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":241916,"kb_used_data":9148,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749719300,"statfs":{"total":767960285184,"available":767712563200,"internally_reserved":0,"allocated":9367552,"data_stored":5367674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.985531"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297664+0000","last_change":"2024-03-30T13:42:51.330132+0000","last_active":"2024-03-30T13:44:02.297664+0000","last_peered":"2024-03-30T13:44:02.297664+0000","last_clean":"2024-03-30T13:44:02.297664+0000","last_became_active":"2024-03-30T13:42:51.329797+0000","last_became_peered":"2024-03-30T13:42:51.329797+0000","last_unstale":"2024-03-30T13:44:02.297664+0000","last_undegraded":"2024-03-30T13:44:02.297664+0000","last_fullsized":"2024-03-30T13:44:02.297664+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:08.354306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:04.797656+0000","last_change":"2024-03-30T13:44:04.797677+0000","last_active":"2024-03-30T13:44:04.797656+0000","last_peered":"2024-03-30T13:44:04.797656+0000","last_clean":"2024-03-30T13:44:04.797656+0000","last_became_active":"2024-03-30T13:42:49.323268+0000","last_became_peered":"2024-03-30T13:42:49.323268+0000","last_unstale":"2024-03-30T13:44:04.797656+0000","last_undegraded":"2024-03-30T13:44:04.797656+0000","last_fullsized":"2024-03-30T13:44:04.797656+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:04.797592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:04.797592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:20.695069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"65'1","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405493+0000","last_change":"2024-03-30T13:42:45.314709+0000","last_active":"2024-03-30T13:44:02.405493+0000","last_peered":"2024-03-30T13:44:02.405493+0000","last_clean":"2024-03-30T13:44:02.405493+0000","last_became_active":"2024-03-30T13:42:45.313936+0000","last_became_peered":"2024-03-30T13:42:45.313936+0000","last_unstale":"2024-03-30T13:44:02.405493+0000","last_undegraded":"2024-03-30T13:44:02.405493+0000","last_fullsized":"2024-03-30T13:44:02.405493+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:07.481265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":75,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297524+0000","last_change":"2024-03-30T13:42:47.320128+0000","last_active":"2024-03-30T13:44:02.297524+0000","last_peered":"2024-03-30T13:44:02.297524+0000","last_clean":"2024-03-30T13:44:02.297524+0000","last_became_active":"2024-03-30T13:42:47.319638+0000","last_became_peered":"2024-03-30T13:42:47.319638+0000","last_unstale":"2024-03-30T13:44:02.297524+0000","last_undegraded":"2024-03-30T13:44:02.297524+0000","last_fullsized":"2024-03-30T13:44:02.297524+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:05.845454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295495+0000","last_change":"2024-03-30T13:42:51.374776+0000","last_active":"2024-03-30T13:44:02.295495+0000","last_peered":"2024-03-30T13:44:02.295495+0000","last_clean":"2024-03-30T13:44:02.295495+0000","last_became_active":"2024-03-30T13:42:51.374403+0000","last_became_peered":"2024-03-30T13:42:51.374403+0000","last_unstale":"2024-03-30T13:44:02.295495+0000","last_undegraded":"2024-03-30T13:44:02.295495+0000","last_fullsized":"2024-03-30T13:44:02.295495+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:13.438854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297531+0000","last_change":"2024-03-30T13:42:49.327437+0000","last_active":"2024-03-30T13:44:02.297531+0000","last_peered":"2024-03-30T13:44:02.297531+0000","last_clean":"2024-03-30T13:44:02.297531+0000","last_became_active":"2024-03-30T13:42:49.327124+0000","last_became_peered":"2024-03-30T13:42:49.327124+0000","last_unstale":"2024-03-30T13:44:02.297531+0000","last_undegraded":"2024-03-30T13:44:02.297531+0000","last_fullsized":"2024-03-30T13:44:02.297531+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:17.902978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:04.008953+0000","last_change":"2024-03-30T13:44:04.008976+0000","last_active":"2024-03-30T13:44:04.008953+0000","last_peered":"2024-03-30T13:44:04.008953+0000","last_clean":"2024-03-30T13:44:04.008953+0000","last_became_active":"2024-03-30T13:42:45.310771+0000","last_became_peered":"2024-03-30T13:42:45.310771+0000","last_unstale":"2024-03-30T13:44:04.008953+0000","last_undegraded":"2024-03-30T13:44:04.008953+0000","last_fullsized":"2024-03-30T13:44:04.008953+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:04.008875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:04.008875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:26.922817+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"72'15","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295286+0000","last_change":"2024-03-30T13:42:47.317438+0000","last_active":"2024-03-30T13:44:02.295286+0000","last_peered":"2024-03-30T13:44:02.295286+0000","last_clean":"2024-03-30T13:44:02.295286+0000","last_became_active":"2024-03-30T13:42:47.317102+0000","last_became_peered":"2024-03-30T13:42:47.317102+0000","last_unstale":"2024-03-30T13:44:02.295286+0000","last_undegraded":"2024-03-30T13:44:02.295286+0000","last_fullsized":"2024-03-30T13:44:02.295286+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:06.434090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405070+0000","last_change":"2024-03-30T13:42:51.331563+0000","last_active":"2024-03-30T13:44:02.405070+0000","last_peered":"2024-03-30T13:44:02.405070+0000","last_clean":"2024-03-30T13:44:02.405070+0000","last_became_active":"2024-03-30T13:42:51.331196+0000","last_became_peered":"2024-03-30T13:42:51.331196+0000","last_unstale":"2024-03-30T13:44:02.405070+0000","last_undegraded":"2024-03-30T13:44:02.405070+0000","last_fullsized":"2024-03-30T13:44:02.405070+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:09.803811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679009+0000","last_change":"2024-03-30T13:43:59.262080+0000","last_active":"2024-03-30T13:44:02.679009+0000","last_peered":"2024-03-30T13:44:02.679009+0000","last_clean":"2024-03-30T13:44:02.679009+0000","last_became_active":"2024-03-30T13:42:49.321811+0000","last_became_peered":"2024-03-30T13:42:49.321811+0000","last_unstale":"2024-03-30T13:44:02.679009+0000","last_undegraded":"2024-03-30T13:44:02.679009+0000","last_fullsized":"2024-03-30T13:44:02.679009+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:59.261981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:59.261981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:20.497993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678880+0000","last_change":"2024-03-30T13:42:45.320122+0000","last_active":"2024-03-30T13:44:02.678880+0000","last_peered":"2024-03-30T13:44:02.678880+0000","last_clean":"2024-03-30T13:44:02.678880+0000","last_became_active":"2024-03-30T13:42:45.319827+0000","last_became_peered":"2024-03-30T13:42:45.319827+0000","last_unstale":"2024-03-30T13:44:02.678880+0000","last_undegraded":"2024-03-30T13:44:02.678880+0000","last_fullsized":"2024-03-30T13:44:02.678880+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":75,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297832+0000","last_change":"2024-03-30T13:42:47.310779+0000","last_active":"2024-03-30T13:44:02.297832+0000","last_peered":"2024-03-30T13:44:02.297832+0000","last_clean":"2024-03-30T13:44:02.297832+0000","last_became_active":"2024-03-30T13:42:47.310289+0000","last_became_peered":"2024-03-30T13:42:47.310289+0000","last_unstale":"2024-03-30T13:44:02.297832+0000","last_undegraded":"2024-03-30T13:44:02.297832+0000","last_fullsized":"2024-03-30T13:44:02.297832+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.677924+0000","last_change":"2024-03-30T13:42:51.326065+0000","last_active":"2024-03-30T13:44:02.677924+0000","last_peered":"2024-03-30T13:44:02.677924+0000","last_clean":"2024-03-30T13:44:02.677924+0000","last_became_active":"2024-03-30T13:42:51.325748+0000","last_became_peered":"2024-03-30T13:42:51.325748+0000","last_unstale":"2024-03-30T13:44:02.677924+0000","last_undegraded":"2024-03-30T13:44:02.677924+0000","last_fullsized":"2024-03-30T13:44:02.677924+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406271+0000","last_change":"2024-03-30T13:42:49.326311+0000","last_active":"2024-03-30T13:44:02.406271+0000","last_peered":"2024-03-30T13:44:02.406271+0000","last_clean":"2024-03-30T13:44:02.406271+0000","last_became_active":"2024-03-30T13:42:49.326027+0000","last_became_peered":"2024-03-30T13:42:49.326027+0000","last_unstale":"2024-03-30T13:44:02.406271+0000","last_undegraded":"2024-03-30T13:44:02.406271+0000","last_fullsized":"2024-03-30T13:44:02.406271+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:17.535137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"65'1","reported_seq":75,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:04.348556+0000","last_change":"2024-03-30T13:44:04.348591+0000","last_active":"2024-03-30T13:44:04.348556+0000","last_peered":"2024-03-30T13:44:04.348556+0000","last_clean":"2024-03-30T13:44:04.348556+0000","last_became_active":"2024-03-30T13:42:45.319975+0000","last_became_peered":"2024-03-30T13:42:45.319975+0000","last_unstale":"2024-03-30T13:44:04.348556+0000","last_undegraded":"2024-03-30T13:44:04.348556+0000","last_fullsized":"2024-03-30T13:44:04.348556+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:44:04.348486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:04.348486+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:34.284640+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297997+0000","last_change":"2024-03-30T13:43:50.961052+0000","last_active":"2024-03-30T13:44:02.297997+0000","last_peered":"2024-03-30T13:44:02.297997+0000","last_clean":"2024-03-30T13:44:02.297997+0000","last_became_active":"2024-03-30T13:42:47.317352+0000","last_became_peered":"2024-03-30T13:42:47.317352+0000","last_unstale":"2024-03-30T13:44:02.297997+0000","last_undegraded":"2024-03-30T13:44:02.297997+0000","last_fullsized":"2024-03-30T13:44:02.297997+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-30T13:43:50.960980+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-30T13:43:50.960980+0000","last_clean_scrub_stamp":"2024-03-30T13:43:50.960980+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:15.585035+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.298580+0000","last_change":"2024-03-30T13:42:51.372350+0000","last_active":"2024-03-30T13:44:02.298580+0000","last_peered":"2024-03-30T13:44:02.298580+0000","last_clean":"2024-03-30T13:44:02.298580+0000","last_became_active":"2024-03-30T13:42:51.372011+0000","last_became_peered":"2024-03-30T13:42:51.372011+0000","last_unstale":"2024-03-30T13:44:02.298580+0000","last_undegraded":"2024-03-30T13:44:02.298580+0000","last_fullsized":"2024-03-30T13:44:02.298580+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678385+0000","last_change":"2024-03-30T13:42:49.323017+0000","last_active":"2024-03-30T13:44:02.678385+0000","last_peered":"2024-03-30T13:44:02.678385+0000","last_clean":"2024-03-30T13:44:02.678385+0000","last_became_active":"2024-03-30T13:42:49.322665+0000","last_became_peered":"2024-03-30T13:42:49.322665+0000","last_unstale":"2024-03-30T13:44:02.678385+0000","last_undegraded":"2024-03-30T13:44:02.678385+0000","last_fullsized":"2024-03-30T13:44:02.678385+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:04.198076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405473+0000","last_change":"2024-03-30T13:42:45.317711+0000","last_active":"2024-03-30T13:44:02.405473+0000","last_peered":"2024-03-30T13:44:02.405473+0000","last_clean":"2024-03-30T13:44:02.405473+0000","last_became_active":"2024-03-30T13:42:45.317378+0000","last_became_peered":"2024-03-30T13:42:45.317378+0000","last_unstale":"2024-03-30T13:44:02.405473+0000","last_undegraded":"2024-03-30T13:44:02.405473+0000","last_fullsized":"2024-03-30T13:44:02.405473+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:14.011698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":92,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:05.870619+0000","last_change":"2024-03-30T13:44:05.870649+0000","last_active":"2024-03-30T13:44:05.870619+0000","last_peered":"2024-03-30T13:44:05.870619+0000","last_clean":"2024-03-30T13:44:05.870619+0000","last_became_active":"2024-03-30T13:42:47.320168+0000","last_became_peered":"2024-03-30T13:42:47.320168+0000","last_unstale":"2024-03-30T13:44:05.870619+0000","last_undegraded":"2024-03-30T13:44:05.870619+0000","last_fullsized":"2024-03-30T13:44:05.870619+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-30T13:44:05.870550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:05.870550+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:10.637627+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.296636+0000","last_change":"2024-03-30T13:42:51.371902+0000","last_active":"2024-03-30T13:44:02.296636+0000","last_peered":"2024-03-30T13:44:02.296636+0000","last_clean":"2024-03-30T13:44:02.296636+0000","last_became_active":"2024-03-30T13:42:51.371378+0000","last_became_peered":"2024-03-30T13:42:51.371378+0000","last_unstale":"2024-03-30T13:44:02.296636+0000","last_undegraded":"2024-03-30T13:44:02.296636+0000","last_fullsized":"2024-03-30T13:44:02.296636+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:20.136855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.298078+0000","last_change":"2024-03-30T13:42:49.323001+0000","last_active":"2024-03-30T13:44:02.298078+0000","last_peered":"2024-03-30T13:44:02.298078+0000","last_clean":"2024-03-30T13:44:02.298078+0000","last_became_active":"2024-03-30T13:42:49.322561+0000","last_became_peered":"2024-03-30T13:42:49.322561+0000","last_unstale":"2024-03-30T13:44:02.298078+0000","last_undegraded":"2024-03-30T13:44:02.298078+0000","last_fullsized":"2024-03-30T13:44:02.298078+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297912+0000","last_change":"2024-03-30T13:42:45.308820+0000","last_active":"2024-03-30T13:44:02.297912+0000","last_peered":"2024-03-30T13:44:02.297912+0000","last_clean":"2024-03-30T13:44:02.297912+0000","last_became_active":"2024-03-30T13:42:45.308468+0000","last_became_peered":"2024-03-30T13:42:45.308468+0000","last_unstale":"2024-03-30T13:44:02.297912+0000","last_undegraded":"2024-03-30T13:44:02.297912+0000","last_fullsized":"2024-03-30T13:44:02.297912+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:10.369269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"88'17","reported_seq":94,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.296024+0000","last_change":"2024-03-30T13:43:52.044529+0000","last_active":"2024-03-30T13:44:02.296024+0000","last_peered":"2024-03-30T13:44:02.296024+0000","last_clean":"2024-03-30T13:44:02.296024+0000","last_became_active":"2024-03-30T13:42:47.317881+0000","last_became_peered":"2024-03-30T13:42:47.317881+0000","last_unstale":"2024-03-30T13:44:02.296024+0000","last_undegraded":"2024-03-30T13:44:02.296024+0000","last_fullsized":"2024-03-30T13:44:02.296024+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'17","last_scrub_stamp":"2024-03-30T13:43:52.044427+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-03-30T13:43:52.044427+0000","last_clean_scrub_stamp":"2024-03-30T13:43:52.044427+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:11.077720+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406721+0000","last_change":"2024-03-30T13:42:51.328035+0000","last_active":"2024-03-30T13:44:02.406721+0000","last_peered":"2024-03-30T13:44:02.406721+0000","last_clean":"2024-03-30T13:44:02.406721+0000","last_became_active":"2024-03-30T13:42:51.327749+0000","last_became_peered":"2024-03-30T13:42:51.327749+0000","last_unstale":"2024-03-30T13:44:02.406721+0000","last_undegraded":"2024-03-30T13:44:02.406721+0000","last_fullsized":"2024-03-30T13:44:02.406721+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":80,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679704+0000","last_change":"2024-03-30T13:42:49.323699+0000","last_active":"2024-03-30T13:44:02.679704+0000","last_peered":"2024-03-30T13:44:02.679704+0000","last_clean":"2024-03-30T13:44:02.679704+0000","last_became_active":"2024-03-30T13:42:49.323362+0000","last_became_peered":"2024-03-30T13:42:49.323362+0000","last_unstale":"2024-03-30T13:44:02.679704+0000","last_undegraded":"2024-03-30T13:44:02.679704+0000","last_fullsized":"2024-03-30T13:44:02.679704+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"65'1","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.677944+0000","last_change":"2024-03-30T13:43:47.807153+0000","last_active":"2024-03-30T13:44:02.677944+0000","last_peered":"2024-03-30T13:44:02.677944+0000","last_clean":"2024-03-30T13:44:02.677944+0000","last_became_active":"2024-03-30T13:42:45.307832+0000","last_became_peered":"2024-03-30T13:42:45.307832+0000","last_unstale":"2024-03-30T13:44:02.677944+0000","last_undegraded":"2024-03-30T13:44:02.677944+0000","last_fullsized":"2024-03-30T13:44:02.677944+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:43:47.807055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:43:47.807055+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:10.931779+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405317+0000","last_change":"2024-03-30T13:42:47.501206+0000","last_active":"2024-03-30T13:44:02.405317+0000","last_peered":"2024-03-30T13:44:02.405317+0000","last_clean":"2024-03-30T13:44:02.405317+0000","last_became_active":"2024-03-30T13:42:47.500763+0000","last_became_peered":"2024-03-30T13:42:47.500763+0000","last_unstale":"2024-03-30T13:44:02.405317+0000","last_undegraded":"2024-03-30T13:44:02.405317+0000","last_fullsized":"2024-03-30T13:44:02.405317+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:04.060530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678240+0000","last_change":"2024-03-30T13:42:51.325851+0000","last_active":"2024-03-30T13:44:02.678240+0000","last_peered":"2024-03-30T13:44:02.678240+0000","last_clean":"2024-03-30T13:44:02.678240+0000","last_became_active":"2024-03-30T13:42:51.325513+0000","last_became_peered":"2024-03-30T13:42:51.325513+0000","last_unstale":"2024-03-30T13:44:02.678240+0000","last_undegraded":"2024-03-30T13:44:02.678240+0000","last_fullsized":"2024-03-30T13:44:02.678240+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:09.388177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.408681+0000","last_change":"2024-03-30T13:44:02.408701+0000","last_active":"2024-03-30T13:44:02.408681+0000","last_peered":"2024-03-30T13:44:02.408681+0000","last_clean":"2024-03-30T13:44:02.408681+0000","last_became_active":"2024-03-30T13:42:49.330335+0000","last_became_peered":"2024-03-30T13:42:49.330335+0000","last_unstale":"2024-03-30T13:44:02.408681+0000","last_undegraded":"2024-03-30T13:44:02.408681+0000","last_fullsized":"2024-03-30T13:44:02.408681+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:02.408612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:02.408612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:23.673286+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.294962+0000","last_change":"2024-03-30T13:42:45.310476+0000","last_active":"2024-03-30T13:44:02.294962+0000","last_peered":"2024-03-30T13:44:02.294962+0000","last_clean":"2024-03-30T13:44:02.294962+0000","last_became_active":"2024-03-30T13:42:45.310165+0000","last_became_peered":"2024-03-30T13:42:45.310165+0000","last_unstale":"2024-03-30T13:44:02.294962+0000","last_undegraded":"2024-03-30T13:44:02.294962+0000","last_fullsized":"2024-03-30T13:44:02.294962+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:03.398892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679665+0000","last_change":"2024-03-30T13:43:54.758046+0000","last_active":"2024-03-30T13:44:02.679665+0000","last_peered":"2024-03-30T13:44:02.679665+0000","last_clean":"2024-03-30T13:44:02.679665+0000","last_became_active":"2024-03-30T13:42:47.324399+0000","last_became_peered":"2024-03-30T13:42:47.324399+0000","last_unstale":"2024-03-30T13:44:02.679665+0000","last_undegraded":"2024-03-30T13:44:02.679665+0000","last_fullsized":"2024-03-30T13:44:02.679665+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:43:54.757869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:54.757869+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:15.753990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406263+0000","last_change":"2024-03-30T13:43:59.682402+0000","last_active":"2024-03-30T13:44:02.406263+0000","last_peered":"2024-03-30T13:44:02.406263+0000","last_clean":"2024-03-30T13:44:02.406263+0000","last_became_active":"2024-03-30T13:42:51.327000+0000","last_became_peered":"2024-03-30T13:42:51.327000+0000","last_unstale":"2024-03-30T13:44:02.406263+0000","last_undegraded":"2024-03-30T13:44:02.406263+0000","last_fullsized":"2024-03-30T13:44:02.406263+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:59.682298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:43:59.682298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:25.906592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678245+0000","last_change":"2024-03-30T13:43:53.750352+0000","last_active":"2024-03-30T13:44:02.678245+0000","last_peered":"2024-03-30T13:44:02.678245+0000","last_clean":"2024-03-30T13:44:02.678245+0000","last_became_active":"2024-03-30T13:42:49.322799+0000","last_became_peered":"2024-03-30T13:42:49.322799+0000","last_unstale":"2024-03-30T13:44:02.678245+0000","last_undegraded":"2024-03-30T13:44:02.678245+0000","last_fullsized":"2024-03-30T13:44:02.678245+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:53.750254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:53.750254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:59.399300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":68,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405432+0000","last_change":"2024-03-30T13:43:58.386027+0000","last_active":"2024-03-30T13:44:02.405432+0000","last_peered":"2024-03-30T13:44:02.405432+0000","last_clean":"2024-03-30T13:44:02.405432+0000","last_became_active":"2024-03-30T13:42:45.314302+0000","last_became_peered":"2024-03-30T13:42:45.314302+0000","last_unstale":"2024-03-30T13:44:02.405432+0000","last_undegraded":"2024-03-30T13:44:02.405432+0000","last_fullsized":"2024-03-30T13:44:02.405432+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:58.385891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:43:58.385891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:03.698225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295060+0000","last_change":"2024-03-30T13:43:55.894589+0000","last_active":"2024-03-30T13:44:02.295060+0000","last_peered":"2024-03-30T13:44:02.295060+0000","last_clean":"2024-03-30T13:44:02.295060+0000","last_became_active":"2024-03-30T13:42:47.321428+0000","last_became_peered":"2024-03-30T13:42:47.321428+0000","last_unstale":"2024-03-30T13:44:02.295060+0000","last_undegraded":"2024-03-30T13:44:02.295060+0000","last_fullsized":"2024-03-30T13:44:02.295060+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-30T13:43:55.894502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:55.894502+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:56.441104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295334+0000","last_change":"2024-03-30T13:42:51.374755+0000","last_active":"2024-03-30T13:44:02.295334+0000","last_peered":"2024-03-30T13:44:02.295334+0000","last_clean":"2024-03-30T13:44:02.295334+0000","last_became_active":"2024-03-30T13:42:51.374304+0000","last_became_peered":"2024-03-30T13:42:51.374304+0000","last_unstale":"2024-03-30T13:44:02.295334+0000","last_undegraded":"2024-03-30T13:44:02.295334+0000","last_fullsized":"2024-03-30T13:44:02.295334+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:18.990969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:07.744706+0000","last_change":"2024-03-30T13:44:07.744732+0000","last_active":"2024-03-30T13:44:07.744706+0000","last_peered":"2024-03-30T13:44:07.744706+0000","last_clean":"2024-03-30T13:44:07.744706+0000","last_became_active":"2024-03-30T13:42:49.324206+0000","last_became_peered":"2024-03-30T13:42:49.324206+0000","last_unstale":"2024-03-30T13:44:07.744706+0000","last_undegraded":"2024-03-30T13:44:07.744706+0000","last_fullsized":"2024-03-30T13:44:07.744706+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:07.744620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:07.744620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:33.978349+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679264+0000","last_change":"2024-03-30T13:42:45.319833+0000","last_active":"2024-03-30T13:44:02.679264+0000","last_peered":"2024-03-30T13:44:02.679264+0000","last_clean":"2024-03-30T13:44:02.679264+0000","last_became_active":"2024-03-30T13:42:45.319482+0000","last_became_peered":"2024-03-30T13:42:45.319482+0000","last_unstale":"2024-03-30T13:44:02.679264+0000","last_undegraded":"2024-03-30T13:44:02.679264+0000","last_fullsized":"2024-03-30T13:44:02.679264+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":85,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295184+0000","last_change":"2024-03-30T13:43:54.926339+0000","last_active":"2024-03-30T13:44:02.295184+0000","last_peered":"2024-03-30T13:44:02.295184+0000","last_clean":"2024-03-30T13:44:02.295184+0000","last_became_active":"2024-03-30T13:42:47.500712+0000","last_became_peered":"2024-03-30T13:42:47.500712+0000","last_unstale":"2024-03-30T13:44:02.295184+0000","last_undegraded":"2024-03-30T13:44:02.295184+0000","last_fullsized":"2024-03-30T13:44:02.295184+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:43:54.926242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:54.926242+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:59.626117+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.404782+0000","last_change":"2024-03-30T13:42:51.334062+0000","last_active":"2024-03-30T13:44:02.404782+0000","last_peered":"2024-03-30T13:44:02.404782+0000","last_clean":"2024-03-30T13:44:02.404782+0000","last_became_active":"2024-03-30T13:42:51.333603+0000","last_became_peered":"2024-03-30T13:42:51.333603+0000","last_unstale":"2024-03-30T13:44:02.404782+0000","last_undegraded":"2024-03-30T13:44:02.404782+0000","last_fullsized":"2024-03-30T13:44:02.404782+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:09.821391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295522+0000","last_change":"2024-03-30T13:42:49.325524+0000","last_active":"2024-03-30T13:44:02.295522+0000","last_peered":"2024-03-30T13:44:02.295522+0000","last_clean":"2024-03-30T13:44:02.295522+0000","last_became_active":"2024-03-30T13:42:49.325219+0000","last_became_peered":"2024-03-30T13:42:49.325219+0000","last_unstale":"2024-03-30T13:44:02.295522+0000","last_undegraded":"2024-03-30T13:44:02.295522+0000","last_fullsized":"2024-03-30T13:44:02.295522+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":68,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.407046+0000","last_change":"2024-03-30T13:43:52.764537+0000","last_active":"2024-03-30T13:44:02.407046+0000","last_peered":"2024-03-30T13:44:02.407046+0000","last_clean":"2024-03-30T13:44:02.407046+0000","last_became_active":"2024-03-30T13:42:45.311071+0000","last_became_peered":"2024-03-30T13:42:45.311071+0000","last_unstale":"2024-03-30T13:44:02.407046+0000","last_undegraded":"2024-03-30T13:44:02.407046+0000","last_fullsized":"2024-03-30T13:44:02.407046+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:52.764446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:52.764446+0000","last_clean_scrub_stamp":"2024-03-30T13:43:52.764446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:14.951666+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678151+0000","last_change":"2024-03-30T13:43:53.806905+0000","last_active":"2024-03-30T13:44:02.678151+0000","last_peered":"2024-03-30T13:44:02.678151+0000","last_clean":"2024-03-30T13:44:02.678151+0000","last_became_active":"2024-03-30T13:42:47.324421+0000","last_became_peered":"2024-03-30T13:42:47.324421+0000","last_unstale":"2024-03-30T13:44:02.678151+0000","last_undegraded":"2024-03-30T13:44:02.678151+0000","last_fullsized":"2024-03-30T13:44:02.678151+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:43:53.806768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:53.806768+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:11.831276+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.294765+0000","last_change":"2024-03-30T13:42:51.335069+0000","last_active":"2024-03-30T13:44:02.294765+0000","last_peered":"2024-03-30T13:44:02.294765+0000","last_clean":"2024-03-30T13:44:02.294765+0000","last_became_active":"2024-03-30T13:42:51.334597+0000","last_became_peered":"2024-03-30T13:42:51.334597+0000","last_unstale":"2024-03-30T13:44:02.294765+0000","last_undegraded":"2024-03-30T13:44:02.294765+0000","last_fullsized":"2024-03-30T13:44:02.294765+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678049+0000","last_change":"2024-03-30T13:42:49.323180+0000","last_active":"2024-03-30T13:44:02.678049+0000","last_peered":"2024-03-30T13:44:02.678049+0000","last_clean":"2024-03-30T13:44:02.678049+0000","last_became_active":"2024-03-30T13:42:49.322843+0000","last_became_peered":"2024-03-30T13:42:49.322843+0000","last_unstale":"2024-03-30T13:44:02.678049+0000","last_undegraded":"2024-03-30T13:44:02.678049+0000","last_fullsized":"2024-03-30T13:44:02.678049+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:12.351693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.298253+0000","last_change":"2024-03-30T13:42:45.316182+0000","last_active":"2024-03-30T13:44:02.298253+0000","last_peered":"2024-03-30T13:44:02.298253+0000","last_clean":"2024-03-30T13:44:02.298253+0000","last_became_active":"2024-03-30T13:42:45.315864+0000","last_became_peered":"2024-03-30T13:42:45.315864+0000","last_unstale":"2024-03-30T13:44:02.298253+0000","last_undegraded":"2024-03-30T13:44:02.298253+0000","last_fullsized":"2024-03-30T13:44:02.298253+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:05.199162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":104,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.298038+0000","last_change":"2024-03-30T13:43:54.985253+0000","last_active":"2024-03-30T13:44:02.298038+0000","last_peered":"2024-03-30T13:44:02.298038+0000","last_clean":"2024-03-30T13:44:02.298038+0000","last_became_active":"2024-03-30T13:43:12.037286+0000","last_became_peered":"2024-03-30T13:43:12.037286+0000","last_unstale":"2024-03-30T13:44:02.298038+0000","last_undegraded":"2024-03-30T13:44:02.298038+0000","last_fullsized":"2024-03-30T13:44:02.298038+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:43:54.985130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:54.985130+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:00.861139+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405182+0000","last_change":"2024-03-30T13:42:51.336473+0000","last_active":"2024-03-30T13:44:02.405182+0000","last_peered":"2024-03-30T13:44:02.405182+0000","last_clean":"2024-03-30T13:44:02.405182+0000","last_became_active":"2024-03-30T13:42:51.336162+0000","last_became_peered":"2024-03-30T13:42:51.336162+0000","last_unstale":"2024-03-30T13:44:02.405182+0000","last_undegraded":"2024-03-30T13:44:02.405182+0000","last_fullsized":"2024-03-30T13:44:02.405182+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:06.520978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679013+0000","last_change":"2024-03-30T13:42:49.321501+0000","last_active":"2024-03-30T13:44:02.679013+0000","last_peered":"2024-03-30T13:44:02.679013+0000","last_clean":"2024-03-30T13:44:02.679013+0000","last_became_active":"2024-03-30T13:42:49.321203+0000","last_became_peered":"2024-03-30T13:42:49.321203+0000","last_unstale":"2024-03-30T13:44:02.679013+0000","last_undegraded":"2024-03-30T13:44:02.679013+0000","last_fullsized":"2024-03-30T13:44:02.679013+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406386+0000","last_change":"2024-03-30T13:42:45.318320+0000","last_active":"2024-03-30T13:44:02.406386+0000","last_peered":"2024-03-30T13:44:02.406386+0000","last_clean":"2024-03-30T13:44:02.406386+0000","last_became_active":"2024-03-30T13:42:45.318016+0000","last_became_peered":"2024-03-30T13:42:45.318016+0000","last_unstale":"2024-03-30T13:44:02.406386+0000","last_undegraded":"2024-03-30T13:44:02.406386+0000","last_fullsized":"2024-03-30T13:44:02.406386+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":91,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295710+0000","last_change":"2024-03-30T13:43:12.043856+0000","last_active":"2024-03-30T13:44:02.295710+0000","last_peered":"2024-03-30T13:44:02.295710+0000","last_clean":"2024-03-30T13:44:02.295710+0000","last_became_active":"2024-03-30T13:43:12.037240+0000","last_became_peered":"2024-03-30T13:43:12.037240+0000","last_unstale":"2024-03-30T13:44:02.295710+0000","last_undegraded":"2024-03-30T13:44:02.295710+0000","last_fullsized":"2024-03-30T13:44:02.295710+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:15.454436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.298736+0000","last_change":"2024-03-30T13:42:51.325383+0000","last_active":"2024-03-30T13:44:02.298736+0000","last_peered":"2024-03-30T13:44:02.298736+0000","last_clean":"2024-03-30T13:44:02.298736+0000","last_became_active":"2024-03-30T13:42:51.325064+0000","last_became_peered":"2024-03-30T13:42:51.325064+0000","last_unstale":"2024-03-30T13:44:02.298736+0000","last_undegraded":"2024-03-30T13:44:02.298736+0000","last_fullsized":"2024-03-30T13:44:02.298736+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:08.485150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"72'8","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295759+0000","last_change":"2024-03-30T13:42:49.328971+0000","last_active":"2024-03-30T13:44:02.295759+0000","last_peered":"2024-03-30T13:44:02.295759+0000","last_clean":"2024-03-30T13:44:02.295759+0000","last_became_active":"2024-03-30T13:42:49.328616+0000","last_became_peered":"2024-03-30T13:42:49.328616+0000","last_unstale":"2024-03-30T13:44:02.295759+0000","last_undegraded":"2024-03-30T13:44:02.295759+0000","last_fullsized":"2024-03-30T13:44:02.295759+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405053+0000","last_change":"2024-03-30T13:42:45.315502+0000","last_active":"2024-03-30T13:44:02.405053+0000","last_peered":"2024-03-30T13:44:02.405053+0000","last_clean":"2024-03-30T13:44:02.405053+0000","last_became_active":"2024-03-30T13:42:45.314585+0000","last_became_peered":"2024-03-30T13:42:45.314585+0000","last_unstale":"2024-03-30T13:44:02.405053+0000","last_undegraded":"2024-03-30T13:44:02.405053+0000","last_fullsized":"2024-03-30T13:44:02.405053+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:14.270223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679100+0000","last_change":"2024-03-30T13:43:53.236751+0000","last_active":"2024-03-30T13:44:02.679100+0000","last_peered":"2024-03-30T13:44:02.679100+0000","last_clean":"2024-03-30T13:44:02.679100+0000","last_became_active":"2024-03-30T13:42:47.501559+0000","last_became_peered":"2024-03-30T13:42:47.501559+0000","last_unstale":"2024-03-30T13:44:02.679100+0000","last_undegraded":"2024-03-30T13:44:02.679100+0000","last_fullsized":"2024-03-30T13:44:02.679100+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:43:53.236633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:53.236633+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:05.517754+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.296647+0000","last_change":"2024-03-30T13:43:52.834487+0000","last_active":"2024-03-30T13:44:02.296647+0000","last_peered":"2024-03-30T13:44:02.296647+0000","last_clean":"2024-03-30T13:44:02.296647+0000","last_became_active":"2024-03-30T13:42:51.325522+0000","last_became_peered":"2024-03-30T13:42:51.325522+0000","last_unstale":"2024-03-30T13:44:02.296647+0000","last_undegraded":"2024-03-30T13:44:02.296647+0000","last_fullsized":"2024-03-30T13:44:02.296647+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:52.834381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:43:52.834381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:56.638872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.296886+0000","last_change":"2024-03-30T13:42:49.323616+0000","last_active":"2024-03-30T13:44:02.296886+0000","last_peered":"2024-03-30T13:44:02.296886+0000","last_clean":"2024-03-30T13:44:02.296886+0000","last_became_active":"2024-03-30T13:42:49.323222+0000","last_became_peered":"2024-03-30T13:42:49.323222+0000","last_unstale":"2024-03-30T13:44:02.296886+0000","last_undegraded":"2024-03-30T13:44:02.296886+0000","last_fullsized":"2024-03-30T13:44:02.296886+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:17.439407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":68,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:05.783558+0000","last_change":"2024-03-30T13:44:05.783586+0000","last_active":"2024-03-30T13:44:05.783558+0000","last_peered":"2024-03-30T13:44:05.783558+0000","last_clean":"2024-03-30T13:44:05.783558+0000","last_became_active":"2024-03-30T13:42:45.313617+0000","last_became_peered":"2024-03-30T13:42:45.313617+0000","last_unstale":"2024-03-30T13:44:05.783558+0000","last_undegraded":"2024-03-30T13:44:05.783558+0000","last_fullsized":"2024-03-30T13:44:05.783558+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:05.783467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:05.783467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:22.470211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":80,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405395+0000","last_change":"2024-03-30T13:44:00.374645+0000","last_active":"2024-03-30T13:44:02.405395+0000","last_peered":"2024-03-30T13:44:02.405395+0000","last_clean":"2024-03-30T13:44:02.405395+0000","last_became_active":"2024-03-30T13:42:47.501925+0000","last_became_peered":"2024-03-30T13:42:47.501925+0000","last_unstale":"2024-03-30T13:44:02.405395+0000","last_undegraded":"2024-03-30T13:44:02.405395+0000","last_fullsized":"2024-03-30T13:44:02.405395+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:44:00.374534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:00.374534+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:25.183504+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"72'1","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406820+0000","last_change":"2024-03-30T13:42:51.372361+0000","last_active":"2024-03-30T13:44:02.406820+0000","last_peered":"2024-03-30T13:44:02.406820+0000","last_clean":"2024-03-30T13:44:02.406820+0000","last_became_active":"2024-03-30T13:42:51.372023+0000","last_became_peered":"2024-03-30T13:42:51.372023+0000","last_unstale":"2024-03-30T13:44:02.406820+0000","last_undegraded":"2024-03-30T13:44:02.406820+0000","last_fullsized":"2024-03-30T13:44:02.406820+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679356+0000","last_change":"2024-03-30T13:42:49.324555+0000","last_active":"2024-03-30T13:44:02.679356+0000","last_peered":"2024-03-30T13:44:02.679356+0000","last_clean":"2024-03-30T13:44:02.679356+0000","last_became_active":"2024-03-30T13:42:49.323924+0000","last_became_peered":"2024-03-30T13:42:49.323924+0000","last_unstale":"2024-03-30T13:44:02.679356+0000","last_undegraded":"2024-03-30T13:44:02.679356+0000","last_fullsized":"2024-03-30T13:44:02.679356+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678095+0000","last_change":"2024-03-30T13:42:45.319317+0000","last_active":"2024-03-30T13:44:02.678095+0000","last_peered":"2024-03-30T13:44:02.678095+0000","last_clean":"2024-03-30T13:44:02.678095+0000","last_became_active":"2024-03-30T13:42:45.318956+0000","last_became_peered":"2024-03-30T13:42:45.318956+0000","last_unstale":"2024-03-30T13:44:02.678095+0000","last_undegraded":"2024-03-30T13:44:02.678095+0000","last_fullsized":"2024-03-30T13:44:02.678095+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:08.614119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.296836+0000","last_change":"2024-03-30T13:42:47.502035+0000","last_active":"2024-03-30T13:44:02.296836+0000","last_peered":"2024-03-30T13:44:02.296836+0000","last_clean":"2024-03-30T13:44:02.296836+0000","last_became_active":"2024-03-30T13:42:47.501426+0000","last_became_peered":"2024-03-30T13:42:47.501426+0000","last_unstale":"2024-03-30T13:44:02.296836+0000","last_undegraded":"2024-03-30T13:44:02.296836+0000","last_fullsized":"2024-03-30T13:44:02.296836+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678461+0000","last_change":"2024-03-30T13:42:51.371670+0000","last_active":"2024-03-30T13:44:02.678461+0000","last_peered":"2024-03-30T13:44:02.678461+0000","last_clean":"2024-03-30T13:44:02.678461+0000","last_became_active":"2024-03-30T13:42:51.371360+0000","last_became_peered":"2024-03-30T13:42:51.371360+0000","last_unstale":"2024-03-30T13:44:02.678461+0000","last_undegraded":"2024-03-30T13:44:02.678461+0000","last_fullsized":"2024-03-30T13:44:02.678461+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":80,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.407361+0000","last_change":"2024-03-30T13:43:45.701210+0000","last_active":"2024-03-30T13:44:02.407361+0000","last_peered":"2024-03-30T13:44:02.407361+0000","last_clean":"2024-03-30T13:44:02.407361+0000","last_became_active":"2024-03-30T13:42:45.318963+0000","last_became_peered":"2024-03-30T13:42:45.318963+0000","last_unstale":"2024-03-30T13:44:02.407361+0000","last_undegraded":"2024-03-30T13:44:02.407361+0000","last_fullsized":"2024-03-30T13:44:02.407361+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-30T13:43:45.701065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:43:45.701065+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:04.819362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.677969+0000","last_change":"2024-03-30T13:42:47.502290+0000","last_active":"2024-03-30T13:44:02.677969+0000","last_peered":"2024-03-30T13:44:02.677969+0000","last_clean":"2024-03-30T13:44:02.677969+0000","last_became_active":"2024-03-30T13:42:47.501937+0000","last_became_peered":"2024-03-30T13:42:47.501937+0000","last_unstale":"2024-03-30T13:44:02.677969+0000","last_undegraded":"2024-03-30T13:44:02.677969+0000","last_fullsized":"2024-03-30T13:44:02.677969+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:03.534390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"72'8","reported_seq":85,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406659+0000","last_change":"2024-03-30T13:43:56.742250+0000","last_active":"2024-03-30T13:44:02.406659+0000","last_peered":"2024-03-30T13:44:02.406659+0000","last_clean":"2024-03-30T13:44:02.406659+0000","last_became_active":"2024-03-30T13:42:49.322675+0000","last_became_peered":"2024-03-30T13:42:49.322675+0000","last_unstale":"2024-03-30T13:44:02.406659+0000","last_undegraded":"2024-03-30T13:44:02.406659+0000","last_fullsized":"2024-03-30T13:44:02.406659+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:43:56.742129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:56.742129+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:17.131326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295433+0000","last_change":"2024-03-30T13:42:51.335656+0000","last_active":"2024-03-30T13:44:02.295433+0000","last_peered":"2024-03-30T13:44:02.295433+0000","last_clean":"2024-03-30T13:44:02.295433+0000","last_became_active":"2024-03-30T13:42:51.334672+0000","last_became_peered":"2024-03-30T13:42:51.334672+0000","last_unstale":"2024-03-30T13:44:02.295433+0000","last_undegraded":"2024-03-30T13:44:02.295433+0000","last_fullsized":"2024-03-30T13:44:02.295433+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:10.851044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":68,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:06.790863+0000","last_change":"2024-03-30T13:44:06.790905+0000","last_active":"2024-03-30T13:44:06.790863+0000","last_peered":"2024-03-30T13:44:06.790863+0000","last_clean":"2024-03-30T13:44:06.790863+0000","last_became_active":"2024-03-30T13:42:45.312328+0000","last_became_peered":"2024-03-30T13:42:45.312328+0000","last_unstale":"2024-03-30T13:44:06.790863+0000","last_undegraded":"2024-03-30T13:44:06.790863+0000","last_fullsized":"2024-03-30T13:44:06.790863+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:06.790755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:06.790755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:10.517615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":90,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297909+0000","last_change":"2024-03-30T13:43:51.992642+0000","last_active":"2024-03-30T13:44:02.297909+0000","last_peered":"2024-03-30T13:44:02.297909+0000","last_clean":"2024-03-30T13:44:02.297909+0000","last_became_active":"2024-03-30T13:42:47.316187+0000","last_became_peered":"2024-03-30T13:42:47.316187+0000","last_unstale":"2024-03-30T13:44:02.297909+0000","last_undegraded":"2024-03-30T13:44:02.297909+0000","last_fullsized":"2024-03-30T13:44:02.297909+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-30T13:43:51.992552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:51.992552+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:09.690586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.294696+0000","last_change":"2024-03-30T13:42:49.322920+0000","last_active":"2024-03-30T13:44:02.294696+0000","last_peered":"2024-03-30T13:44:02.294696+0000","last_clean":"2024-03-30T13:44:02.294696+0000","last_became_active":"2024-03-30T13:42:49.322478+0000","last_became_peered":"2024-03-30T13:42:49.322478+0000","last_unstale":"2024-03-30T13:44:02.294696+0000","last_undegraded":"2024-03-30T13:44:02.294696+0000","last_fullsized":"2024-03-30T13:44:02.294696+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:05.443637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406209+0000","last_change":"2024-03-30T13:42:51.334086+0000","last_active":"2024-03-30T13:44:02.406209+0000","last_peered":"2024-03-30T13:44:02.406209+0000","last_clean":"2024-03-30T13:44:02.406209+0000","last_became_active":"2024-03-30T13:42:51.333773+0000","last_became_peered":"2024-03-30T13:42:51.333773+0000","last_unstale":"2024-03-30T13:44:02.406209+0000","last_undegraded":"2024-03-30T13:44:02.406209+0000","last_fullsized":"2024-03-30T13:44:02.406209+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.296758+0000","last_change":"2024-03-30T13:44:01.850315+0000","last_active":"2024-03-30T13:44:02.296758+0000","last_peered":"2024-03-30T13:44:02.296758+0000","last_clean":"2024-03-30T13:44:02.296758+0000","last_became_active":"2024-03-30T13:42:49.320798+0000","last_became_peered":"2024-03-30T13:42:49.320798+0000","last_unstale":"2024-03-30T13:44:02.296758+0000","last_undegraded":"2024-03-30T13:44:02.296758+0000","last_fullsized":"2024-03-30T13:44:02.296758+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:01.850229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:01.850229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:12.718080+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.407156+0000","last_change":"2024-03-30T13:42:45.316023+0000","last_active":"2024-03-30T13:44:02.407156+0000","last_peered":"2024-03-30T13:44:02.407156+0000","last_clean":"2024-03-30T13:44:02.407156+0000","last_became_active":"2024-03-30T13:42:45.315687+0000","last_became_peered":"2024-03-30T13:42:45.315687+0000","last_unstale":"2024-03-30T13:44:02.407156+0000","last_undegraded":"2024-03-30T13:44:02.407156+0000","last_fullsized":"2024-03-30T13:44:02.407156+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"72'19","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679002+0000","last_change":"2024-03-30T13:42:47.501349+0000","last_active":"2024-03-30T13:44:02.679002+0000","last_peered":"2024-03-30T13:44:02.679002+0000","last_clean":"2024-03-30T13:44:02.679002+0000","last_became_active":"2024-03-30T13:42:47.500898+0000","last_became_peered":"2024-03-30T13:42:47.500898+0000","last_unstale":"2024-03-30T13:44:02.679002+0000","last_undegraded":"2024-03-30T13:44:02.679002+0000","last_fullsized":"2024-03-30T13:44:02.679002+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:10.044571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.404868+0000","last_change":"2024-03-30T13:42:51.330739+0000","last_active":"2024-03-30T13:44:02.404868+0000","last_peered":"2024-03-30T13:44:02.404868+0000","last_clean":"2024-03-30T13:44:02.404868+0000","last_became_active":"2024-03-30T13:42:51.330403+0000","last_became_peered":"2024-03-30T13:42:51.330403+0000","last_unstale":"2024-03-30T13:44:02.404868+0000","last_undegraded":"2024-03-30T13:44:02.404868+0000","last_fullsized":"2024-03-30T13:44:02.404868+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:19.083830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295324+0000","last_change":"2024-03-30T13:43:51.840857+0000","last_active":"2024-03-30T13:44:02.295324+0000","last_peered":"2024-03-30T13:44:02.295324+0000","last_clean":"2024-03-30T13:44:02.295324+0000","last_became_active":"2024-03-30T13:42:49.324022+0000","last_became_peered":"2024-03-30T13:42:49.324022+0000","last_unstale":"2024-03-30T13:44:02.295324+0000","last_undegraded":"2024-03-30T13:44:02.295324+0000","last_fullsized":"2024-03-30T13:44:02.295324+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:51.840740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:51.840740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:00.887228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678645+0000","last_change":"2024-03-30T13:42:45.315058+0000","last_active":"2024-03-30T13:44:02.678645+0000","last_peered":"2024-03-30T13:44:02.678645+0000","last_clean":"2024-03-30T13:44:02.678645+0000","last_became_active":"2024-03-30T13:42:45.314767+0000","last_became_peered":"2024-03-30T13:42:45.314767+0000","last_unstale":"2024-03-30T13:44:02.678645+0000","last_undegraded":"2024-03-30T13:44:02.678645+0000","last_fullsized":"2024-03-30T13:44:02.678645+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":96,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405384+0000","last_change":"2024-03-30T13:43:57.361388+0000","last_active":"2024-03-30T13:44:02.405384+0000","last_peered":"2024-03-30T13:44:02.405384+0000","last_clean":"2024-03-30T13:44:02.405384+0000","last_became_active":"2024-03-30T13:43:11.446379+0000","last_became_peered":"2024-03-30T13:43:11.446379+0000","last_unstale":"2024-03-30T13:44:02.405384+0000","last_undegraded":"2024-03-30T13:44:02.405384+0000","last_fullsized":"2024-03-30T13:44:02.405384+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:43:57.361300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:57.361300+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:13.159418+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:06.994156+0000","last_change":"2024-03-30T13:44:06.994178+0000","last_active":"2024-03-30T13:44:06.994156+0000","last_peered":"2024-03-30T13:44:06.994156+0000","last_clean":"2024-03-30T13:44:06.994156+0000","last_became_active":"2024-03-30T13:42:51.331054+0000","last_became_peered":"2024-03-30T13:42:51.331054+0000","last_unstale":"2024-03-30T13:44:06.994156+0000","last_undegraded":"2024-03-30T13:44:06.994156+0000","last_fullsized":"2024-03-30T13:44:06.994156+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:06.994097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:06.994097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:17.075287+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678183+0000","last_change":"2024-03-30T13:42:48.314967+0000","last_active":"2024-03-30T13:44:02.678183+0000","last_peered":"2024-03-30T13:44:02.678183+0000","last_clean":"2024-03-30T13:44:02.678183+0000","last_became_active":"2024-03-30T13:42:44.300789+0000","last_became_peered":"2024-03-30T13:42:44.300789+0000","last_unstale":"2024-03-30T13:44:02.678183+0000","last_undegraded":"2024-03-30T13:44:02.678183+0000","last_fullsized":"2024-03-30T13:44:02.678183+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:42:43.281063+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021837800000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297788+0000","last_change":"2024-03-30T13:42:45.310512+0000","last_active":"2024-03-30T13:44:02.297788+0000","last_peered":"2024-03-30T13:44:02.297788+0000","last_clean":"2024-03-30T13:44:02.297788+0000","last_became_active":"2024-03-30T13:42:45.310209+0000","last_became_peered":"2024-03-30T13:42:45.310209+0000","last_unstale":"2024-03-30T13:44:02.297788+0000","last_undegraded":"2024-03-30T13:44:02.297788+0000","last_fullsized":"2024-03-30T13:44:02.297788+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:04.787006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"72'28","reported_seq":123,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678328+0000","last_change":"2024-03-30T13:42:47.324184+0000","last_active":"2024-03-30T13:44:02.678328+0000","last_peered":"2024-03-30T13:44:02.678328+0000","last_clean":"2024-03-30T13:44:02.678328+0000","last_became_active":"2024-03-30T13:42:47.323845+0000","last_became_peered":"2024-03-30T13:42:47.323845+0000","last_unstale":"2024-03-30T13:44:02.678328+0000","last_undegraded":"2024-03-30T13:44:02.678328+0000","last_fullsized":"2024-03-30T13:44:02.678328+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:07.498588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405223+0000","last_change":"2024-03-30T13:43:54.326967+0000","last_active":"2024-03-30T13:44:02.405223+0000","last_peered":"2024-03-30T13:44:02.405223+0000","last_clean":"2024-03-30T13:44:02.405223+0000","last_became_active":"2024-03-30T13:42:49.322154+0000","last_became_peered":"2024-03-30T13:42:49.322154+0000","last_unstale":"2024-03-30T13:44:02.405223+0000","last_undegraded":"2024-03-30T13:44:02.405223+0000","last_fullsized":"2024-03-30T13:44:02.405223+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:54.326829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:54.326829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:09.771943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.407339+0000","last_change":"2024-03-30T13:42:51.372579+0000","last_active":"2024-03-30T13:44:02.407339+0000","last_peered":"2024-03-30T13:44:02.407339+0000","last_clean":"2024-03-30T13:44:02.407339+0000","last_became_active":"2024-03-30T13:42:51.372320+0000","last_became_peered":"2024-03-30T13:42:51.372320+0000","last_unstale":"2024-03-30T13:44:02.407339+0000","last_undegraded":"2024-03-30T13:44:02.407339+0000","last_fullsized":"2024-03-30T13:44:02.407339+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:10.769625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678026+0000","last_change":"2024-03-30T13:43:59.810299+0000","last_active":"2024-03-30T13:44:02.678026+0000","last_peered":"2024-03-30T13:44:02.678026+0000","last_clean":"2024-03-30T13:44:02.678026+0000","last_became_active":"2024-03-30T13:42:45.314771+0000","last_became_peered":"2024-03-30T13:42:45.314771+0000","last_unstale":"2024-03-30T13:44:02.678026+0000","last_undegraded":"2024-03-30T13:44:02.678026+0000","last_fullsized":"2024-03-30T13:44:02.678026+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:59.810189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:59.810189+0000","last_clean_scrub_stamp":"2024-03-30T13:43:59.810189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:06.406345+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":80,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295022+0000","last_change":"2024-03-30T13:43:57.866600+0000","last_active":"2024-03-30T13:44:02.295022+0000","last_peered":"2024-03-30T13:44:02.295022+0000","last_clean":"2024-03-30T13:44:02.295022+0000","last_became_active":"2024-03-30T13:42:44.302344+0000","last_became_peered":"2024-03-30T13:42:44.302344+0000","last_unstale":"2024-03-30T13:44:02.295022+0000","last_undegraded":"2024-03-30T13:44:02.295022+0000","last_fullsized":"2024-03-30T13:44:02.295022+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:57.866519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:43:57.866519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:59.119000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001401518,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"72'13","reported_seq":85,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678489+0000","last_change":"2024-03-30T13:42:47.321918+0000","last_active":"2024-03-30T13:44:02.678489+0000","last_peered":"2024-03-30T13:44:02.678489+0000","last_clean":"2024-03-30T13:44:02.678489+0000","last_became_active":"2024-03-30T13:42:47.320541+0000","last_became_peered":"2024-03-30T13:42:47.320541+0000","last_unstale":"2024-03-30T13:44:02.678489+0000","last_undegraded":"2024-03-30T13:44:02.678489+0000","last_fullsized":"2024-03-30T13:44:02.678489+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:05.609369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295695+0000","last_change":"2024-03-30T13:42:49.320258+0000","last_active":"2024-03-30T13:44:02.295695+0000","last_peered":"2024-03-30T13:44:02.295695+0000","last_clean":"2024-03-30T13:44:02.295695+0000","last_became_active":"2024-03-30T13:42:49.319826+0000","last_became_peered":"2024-03-30T13:42:49.319826+0000","last_unstale":"2024-03-30T13:44:02.295695+0000","last_undegraded":"2024-03-30T13:44:02.295695+0000","last_fullsized":"2024-03-30T13:44:02.295695+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:13.062008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679597+0000","last_change":"2024-03-30T13:42:51.331412+0000","last_active":"2024-03-30T13:44:02.679597+0000","last_peered":"2024-03-30T13:44:02.679597+0000","last_clean":"2024-03-30T13:44:02.679597+0000","last_became_active":"2024-03-30T13:42:51.331081+0000","last_became_peered":"2024-03-30T13:42:51.331081+0000","last_unstale":"2024-03-30T13:44:02.679597+0000","last_undegraded":"2024-03-30T13:44:02.679597+0000","last_fullsized":"2024-03-30T13:44:02.679597+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":313,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406577+0000","last_change":"2024-03-30T13:43:23.836532+0000","last_active":"2024-03-30T13:44:02.406577+0000","last_peered":"2024-03-30T13:44:02.406577+0000","last_clean":"2024-03-30T13:44:02.406577+0000","last_became_active":"2024-03-30T13:42:34.988246+0000","last_became_peered":"2024-03-30T13:42:34.988246+0000","last_unstale":"2024-03-30T13:44:02.406577+0000","last_undegraded":"2024-03-30T13:44:02.406577+0000","last_fullsized":"2024-03-30T13:44:02.406577+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-03-30T13:43:23.836417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:43:23.836417+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:30.177755+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295171+0000","last_change":"2024-03-30T13:42:45.310322+0000","last_active":"2024-03-30T13:44:02.295171+0000","last_peered":"2024-03-30T13:44:02.295171+0000","last_clean":"2024-03-30T13:44:02.295171+0000","last_became_active":"2024-03-30T13:42:45.309954+0000","last_became_peered":"2024-03-30T13:42:45.309954+0000","last_unstale":"2024-03-30T13:44:02.295171+0000","last_undegraded":"2024-03-30T13:44:02.295171+0000","last_fullsized":"2024-03-30T13:44:02.295171+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":95,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679327+0000","last_change":"2024-03-30T13:42:47.502798+0000","last_active":"2024-03-30T13:44:02.679327+0000","last_peered":"2024-03-30T13:44:02.679327+0000","last_clean":"2024-03-30T13:44:02.679327+0000","last_became_active":"2024-03-30T13:42:47.502338+0000","last_became_peered":"2024-03-30T13:42:47.502338+0000","last_unstale":"2024-03-30T13:44:02.679327+0000","last_undegraded":"2024-03-30T13:44:02.679327+0000","last_fullsized":"2024-03-30T13:44:02.679327+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406924+0000","last_change":"2024-03-30T13:42:49.329181+0000","last_active":"2024-03-30T13:44:02.406924+0000","last_peered":"2024-03-30T13:44:02.406924+0000","last_clean":"2024-03-30T13:44:02.406924+0000","last_became_active":"2024-03-30T13:42:49.328861+0000","last_became_peered":"2024-03-30T13:42:49.328861+0000","last_unstale":"2024-03-30T13:44:02.406924+0000","last_undegraded":"2024-03-30T13:44:02.406924+0000","last_fullsized":"2024-03-30T13:44:02.406924+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:11.705785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678034+0000","last_change":"2024-03-30T13:42:51.335522+0000","last_active":"2024-03-30T13:44:02.678034+0000","last_peered":"2024-03-30T13:44:02.678034+0000","last_clean":"2024-03-30T13:44:02.678034+0000","last_became_active":"2024-03-30T13:42:51.335188+0000","last_became_peered":"2024-03-30T13:42:51.335188+0000","last_unstale":"2024-03-30T13:44:02.678034+0000","last_undegraded":"2024-03-30T13:44:02.678034+0000","last_fullsized":"2024-03-30T13:44:02.678034+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:09.504888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405003+0000","last_change":"2024-03-30T13:42:45.308893+0000","last_active":"2024-03-30T13:44:02.405003+0000","last_peered":"2024-03-30T13:44:02.405003+0000","last_clean":"2024-03-30T13:44:02.405003+0000","last_became_active":"2024-03-30T13:42:45.307905+0000","last_became_peered":"2024-03-30T13:42:45.307905+0000","last_unstale":"2024-03-30T13:44:02.405003+0000","last_undegraded":"2024-03-30T13:44:02.405003+0000","last_fullsized":"2024-03-30T13:44:02.405003+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:03.325741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":148,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:06.810323+0000","last_change":"2024-03-30T13:42:48.313590+0000","last_active":"2024-03-30T13:44:06.810323+0000","last_peered":"2024-03-30T13:44:06.810323+0000","last_clean":"2024-03-30T13:44:06.810323+0000","last_became_active":"2024-03-30T13:42:44.302343+0000","last_became_peered":"2024-03-30T13:42:44.302343+0000","last_unstale":"2024-03-30T13:44:06.810323+0000","last_undegraded":"2024-03-30T13:44:06.810323+0000","last_fullsized":"2024-03-30T13:44:06.810323+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:42:43.281063+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:10.824760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00126391,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":74,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405204+0000","last_change":"2024-03-30T13:42:47.319576+0000","last_active":"2024-03-30T13:44:02.405204+0000","last_peered":"2024-03-30T13:44:02.405204+0000","last_clean":"2024-03-30T13:44:02.405204+0000","last_became_active":"2024-03-30T13:42:47.319155+0000","last_became_peered":"2024-03-30T13:42:47.319155+0000","last_unstale":"2024-03-30T13:44:02.405204+0000","last_undegraded":"2024-03-30T13:44:02.405204+0000","last_fullsized":"2024-03-30T13:44:02.405204+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:08.029848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679775+0000","last_change":"2024-03-30T13:42:49.322962+0000","last_active":"2024-03-30T13:44:02.679775+0000","last_peered":"2024-03-30T13:44:02.679775+0000","last_clean":"2024-03-30T13:44:02.679775+0000","last_became_active":"2024-03-30T13:42:49.322700+0000","last_became_peered":"2024-03-30T13:42:49.322700+0000","last_unstale":"2024-03-30T13:44:02.679775+0000","last_undegraded":"2024-03-30T13:44:02.679775+0000","last_fullsized":"2024-03-30T13:44:02.679775+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:08.787689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:05.809623+0000","last_change":"2024-03-30T13:44:05.809643+0000","last_active":"2024-03-30T13:44:05.809623+0000","last_peered":"2024-03-30T13:44:05.809623+0000","last_clean":"2024-03-30T13:44:05.809623+0000","last_became_active":"2024-03-30T13:42:51.371202+0000","last_became_peered":"2024-03-30T13:42:51.371202+0000","last_unstale":"2024-03-30T13:44:05.809623+0000","last_undegraded":"2024-03-30T13:44:05.809623+0000","last_fullsized":"2024-03-30T13:44:05.809623+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:05.809548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:05.809548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:18.651258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:06.800343+0000","last_change":"2024-03-30T13:44:06.800362+0000","last_active":"2024-03-30T13:44:06.800343+0000","last_peered":"2024-03-30T13:44:06.800343+0000","last_clean":"2024-03-30T13:44:06.800343+0000","last_became_active":"2024-03-30T13:42:45.310134+0000","last_became_peered":"2024-03-30T13:42:45.310134+0000","last_unstale":"2024-03-30T13:44:06.800343+0000","last_undegraded":"2024-03-30T13:44:06.800343+0000","last_fullsized":"2024-03-30T13:44:06.800343+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:06.800290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:06.800290+0000","last_clean_scrub_stamp":"2024-03-30T13:44:06.800290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:30.870285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":92,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405502+0000","last_change":"2024-03-30T13:42:47.501567+0000","last_active":"2024-03-30T13:44:02.405502+0000","last_peered":"2024-03-30T13:44:02.405502+0000","last_clean":"2024-03-30T13:44:02.405502+0000","last_became_active":"2024-03-30T13:42:47.501272+0000","last_became_peered":"2024-03-30T13:42:47.501272+0000","last_unstale":"2024-03-30T13:44:02.405502+0000","last_undegraded":"2024-03-30T13:44:02.405502+0000","last_fullsized":"2024-03-30T13:44:02.405502+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:03.584110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679171+0000","last_change":"2024-03-30T13:42:49.321802+0000","last_active":"2024-03-30T13:44:02.679171+0000","last_peered":"2024-03-30T13:44:02.679171+0000","last_clean":"2024-03-30T13:44:02.679171+0000","last_became_active":"2024-03-30T13:42:49.321520+0000","last_became_peered":"2024-03-30T13:42:49.321520+0000","last_unstale":"2024-03-30T13:44:02.679171+0000","last_undegraded":"2024-03-30T13:44:02.679171+0000","last_fullsized":"2024-03-30T13:44:02.679171+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:06.176165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:04.440717+0000","last_change":"2024-03-30T13:44:04.440757+0000","last_active":"2024-03-30T13:44:04.440717+0000","last_peered":"2024-03-30T13:44:04.440717+0000","last_clean":"2024-03-30T13:44:04.440717+0000","last_became_active":"2024-03-30T13:42:51.336892+0000","last_became_peered":"2024-03-30T13:42:51.336892+0000","last_unstale":"2024-03-30T13:44:04.440717+0000","last_undegraded":"2024-03-30T13:44:04.440717+0000","last_fullsized":"2024-03-30T13:44:04.440717+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:04.440623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:04.440623+0000","last_clean_scrub_stamp":"2024-03-30T13:44:04.440623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:20.104438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678999+0000","last_change":"2024-03-30T13:43:57.299723+0000","last_active":"2024-03-30T13:44:02.678999+0000","last_peered":"2024-03-30T13:44:02.678999+0000","last_clean":"2024-03-30T13:44:02.678999+0000","last_became_active":"2024-03-30T13:42:45.317136+0000","last_became_peered":"2024-03-30T13:42:45.317136+0000","last_unstale":"2024-03-30T13:44:02.678999+0000","last_undegraded":"2024-03-30T13:44:02.678999+0000","last_fullsized":"2024-03-30T13:44:02.678999+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:57.299618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:57.299618+0000","last_clean_scrub_stamp":"2024-03-30T13:43:57.299618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:09.120401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678084+0000","last_change":"2024-03-30T13:42:47.322034+0000","last_active":"2024-03-30T13:44:02.678084+0000","last_peered":"2024-03-30T13:44:02.678084+0000","last_clean":"2024-03-30T13:44:02.678084+0000","last_became_active":"2024-03-30T13:42:47.321433+0000","last_became_peered":"2024-03-30T13:42:47.321433+0000","last_unstale":"2024-03-30T13:44:02.678084+0000","last_undegraded":"2024-03-30T13:44:02.678084+0000","last_fullsized":"2024-03-30T13:44:02.678084+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:04.775648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.405615+0000","last_change":"2024-03-30T13:42:49.326702+0000","last_active":"2024-03-30T13:44:02.405615+0000","last_peered":"2024-03-30T13:44:02.405615+0000","last_clean":"2024-03-30T13:44:02.405615+0000","last_became_active":"2024-03-30T13:42:49.325533+0000","last_became_peered":"2024-03-30T13:42:49.325533+0000","last_unstale":"2024-03-30T13:44:02.405615+0000","last_undegraded":"2024-03-30T13:44:02.405615+0000","last_fullsized":"2024-03-30T13:44:02.405615+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:16.967785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.406634+0000","last_change":"2024-03-30T13:42:51.374719+0000","last_active":"2024-03-30T13:44:02.406634+0000","last_peered":"2024-03-30T13:44:02.406634+0000","last_clean":"2024-03-30T13:44:02.406634+0000","last_became_active":"2024-03-30T13:42:51.374437+0000","last_became_peered":"2024-03-30T13:42:51.374437+0000","last_unstale":"2024-03-30T13:44:02.406634+0000","last_undegraded":"2024-03-30T13:44:02.406634+0000","last_fullsized":"2024-03-30T13:44:02.406634+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:09.130892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:07.397479+0000","last_change":"2024-03-30T13:44:07.397507+0000","last_active":"2024-03-30T13:44:07.397479+0000","last_peered":"2024-03-30T13:44:07.397479+0000","last_clean":"2024-03-30T13:44:07.397479+0000","last_became_active":"2024-03-30T13:42:45.309886+0000","last_became_peered":"2024-03-30T13:42:45.309886+0000","last_unstale":"2024-03-30T13:44:07.397479+0000","last_undegraded":"2024-03-30T13:44:07.397479+0000","last_fullsized":"2024-03-30T13:44:07.397479+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:44:07.397409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:07.397409+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:07.807875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"91'21","reported_seq":95,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:05.917134+0000","last_change":"2024-03-30T13:44:05.917185+0000","last_active":"2024-03-30T13:44:05.917134+0000","last_peered":"2024-03-30T13:44:05.917134+0000","last_clean":"2024-03-30T13:44:05.917134+0000","last_became_active":"2024-03-30T13:42:47.314184+0000","last_became_peered":"2024-03-30T13:42:47.314184+0000","last_unstale":"2024-03-30T13:44:05.917134+0000","last_undegraded":"2024-03-30T13:44:05.917134+0000","last_fullsized":"2024-03-30T13:44:05.917134+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'21","last_scrub_stamp":"2024-03-30T13:44:05.917064+0000","last_deep_scrub":"91'21","last_deep_scrub_stamp":"2024-03-30T13:44:05.917064+0000","last_clean_scrub_stamp":"2024-03-30T13:44:05.917064+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:07.613278+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295408+0000","last_change":"2024-03-30T13:42:49.321877+0000","last_active":"2024-03-30T13:44:02.295408+0000","last_peered":"2024-03-30T13:44:02.295408+0000","last_clean":"2024-03-30T13:44:02.295408+0000","last_became_active":"2024-03-30T13:42:49.321423+0000","last_became_peered":"2024-03-30T13:42:49.321423+0000","last_unstale":"2024-03-30T13:44:02.295408+0000","last_undegraded":"2024-03-30T13:44:02.295408+0000","last_fullsized":"2024-03-30T13:44:02.295408+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:03.968507+0000","last_change":"2024-03-30T13:44:03.968532+0000","last_active":"2024-03-30T13:44:03.968507+0000","last_peered":"2024-03-30T13:44:03.968507+0000","last_clean":"2024-03-30T13:44:03.968507+0000","last_became_active":"2024-03-30T13:42:51.327003+0000","last_became_peered":"2024-03-30T13:42:51.327003+0000","last_unstale":"2024-03-30T13:44:03.968507+0000","last_undegraded":"2024-03-30T13:44:03.968507+0000","last_fullsized":"2024-03-30T13:44:03.968507+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:03.968435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:03.968435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:12.556459+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.296752+0000","last_change":"2024-03-30T13:42:45.314830+0000","last_active":"2024-03-30T13:44:02.296752+0000","last_peered":"2024-03-30T13:44:02.296752+0000","last_clean":"2024-03-30T13:44:02.296752+0000","last_became_active":"2024-03-30T13:42:45.314533+0000","last_became_peered":"2024-03-30T13:42:45.314533+0000","last_unstale":"2024-03-30T13:44:02.296752+0000","last_undegraded":"2024-03-30T13:44:02.296752+0000","last_fullsized":"2024-03-30T13:44:02.296752+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:03.121327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":98,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:06.016601+0000","last_change":"2024-03-30T13:44:06.016635+0000","last_active":"2024-03-30T13:44:06.016601+0000","last_peered":"2024-03-30T13:44:06.016601+0000","last_clean":"2024-03-30T13:44:06.016601+0000","last_became_active":"2024-03-30T13:42:47.502708+0000","last_became_peered":"2024-03-30T13:42:47.502708+0000","last_unstale":"2024-03-30T13:44:06.016601+0000","last_undegraded":"2024-03-30T13:44:06.016601+0000","last_fullsized":"2024-03-30T13:44:06.016601+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-03-30T13:44:06.016537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:06.016537+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:09.822245+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297579+0000","last_change":"2024-03-30T13:42:49.324517+0000","last_active":"2024-03-30T13:44:02.297579+0000","last_peered":"2024-03-30T13:44:02.297579+0000","last_clean":"2024-03-30T13:44:02.297579+0000","last_became_active":"2024-03-30T13:42:49.324022+0000","last_became_peered":"2024-03-30T13:42:49.324022+0000","last_unstale":"2024-03-30T13:44:02.297579+0000","last_undegraded":"2024-03-30T13:44:02.297579+0000","last_fullsized":"2024-03-30T13:44:02.297579+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:08.917548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678368+0000","last_change":"2024-03-30T13:42:51.329317+0000","last_active":"2024-03-30T13:44:02.678368+0000","last_peered":"2024-03-30T13:44:02.678368+0000","last_clean":"2024-03-30T13:44:02.678368+0000","last_became_active":"2024-03-30T13:42:51.329026+0000","last_became_peered":"2024-03-30T13:42:51.329026+0000","last_unstale":"2024-03-30T13:44:02.678368+0000","last_undegraded":"2024-03-30T13:44:02.678368+0000","last_fullsized":"2024-03-30T13:44:02.678368+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:04.561073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.298694+0000","last_change":"2024-03-30T13:43:58.942488+0000","last_active":"2024-03-30T13:44:02.298694+0000","last_peered":"2024-03-30T13:44:02.298694+0000","last_clean":"2024-03-30T13:44:02.298694+0000","last_became_active":"2024-03-30T13:42:49.321790+0000","last_became_peered":"2024-03-30T13:42:49.321790+0000","last_unstale":"2024-03-30T13:44:02.298694+0000","last_undegraded":"2024-03-30T13:44:02.298694+0000","last_fullsized":"2024-03-30T13:44:02.298694+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:43:58.942311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:58.942311+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:17.003904+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295160+0000","last_change":"2024-03-30T13:42:45.315666+0000","last_active":"2024-03-30T13:44:02.295160+0000","last_peered":"2024-03-30T13:44:02.295160+0000","last_clean":"2024-03-30T13:44:02.295160+0000","last_became_active":"2024-03-30T13:42:45.315319+0000","last_became_peered":"2024-03-30T13:42:45.315319+0000","last_unstale":"2024-03-30T13:44:02.295160+0000","last_undegraded":"2024-03-30T13:44:02.295160+0000","last_fullsized":"2024-03-30T13:44:02.295160+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:11.352775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678538+0000","last_change":"2024-03-30T13:42:47.324692+0000","last_active":"2024-03-30T13:44:02.678538+0000","last_peered":"2024-03-30T13:44:02.678538+0000","last_clean":"2024-03-30T13:44:02.678538+0000","last_became_active":"2024-03-30T13:42:47.324350+0000","last_became_peered":"2024-03-30T13:42:47.324350+0000","last_unstale":"2024-03-30T13:44:02.678538+0000","last_undegraded":"2024-03-30T13:44:02.678538+0000","last_fullsized":"2024-03-30T13:44:02.678538+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:10.515261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295562+0000","last_change":"2024-03-30T13:42:51.371742+0000","last_active":"2024-03-30T13:44:02.295562+0000","last_peered":"2024-03-30T13:44:02.295562+0000","last_clean":"2024-03-30T13:44:02.295562+0000","last_became_active":"2024-03-30T13:42:51.371369+0000","last_became_peered":"2024-03-30T13:42:51.371369+0000","last_unstale":"2024-03-30T13:44:02.295562+0000","last_undegraded":"2024-03-30T13:44:02.295562+0000","last_fullsized":"2024-03-30T13:44:02.295562+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:19.941761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.679775+0000","last_change":"2024-03-30T13:42:49.323595+0000","last_active":"2024-03-30T13:44:02.679775+0000","last_peered":"2024-03-30T13:44:02.679775+0000","last_clean":"2024-03-30T13:44:02.679775+0000","last_became_active":"2024-03-30T13:42:49.323319+0000","last_became_peered":"2024-03-30T13:42:49.323319+0000","last_unstale":"2024-03-30T13:44:02.679775+0000","last_undegraded":"2024-03-30T13:44:02.679775+0000","last_fullsized":"2024-03-30T13:44:02.679775+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:03.310676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.297628+0000","last_change":"2024-03-30T13:43:49.990189+0000","last_active":"2024-03-30T13:44:02.297628+0000","last_peered":"2024-03-30T13:44:02.297628+0000","last_clean":"2024-03-30T13:44:02.297628+0000","last_became_active":"2024-03-30T13:42:45.315018+0000","last_became_peered":"2024-03-30T13:42:45.315018+0000","last_unstale":"2024-03-30T13:44:02.297628+0000","last_undegraded":"2024-03-30T13:44:02.297628+0000","last_fullsized":"2024-03-30T13:44:02.297628+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:49.990065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:43:49.990065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:58.482640+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"72'11","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.298500+0000","last_change":"2024-03-30T13:42:47.315984+0000","last_active":"2024-03-30T13:44:02.298500+0000","last_peered":"2024-03-30T13:44:02.298500+0000","last_clean":"2024-03-30T13:44:02.298500+0000","last_became_active":"2024-03-30T13:42:47.315667+0000","last_became_peered":"2024-03-30T13:42:47.315667+0000","last_unstale":"2024-03-30T13:44:02.298500+0000","last_undegraded":"2024-03-30T13:44:02.298500+0000","last_fullsized":"2024-03-30T13:44:02.298500+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.296059+0000","last_change":"2024-03-30T13:42:51.331741+0000","last_active":"2024-03-30T13:44:02.296059+0000","last_peered":"2024-03-30T13:44:02.296059+0000","last_clean":"2024-03-30T13:44:02.296059+0000","last_became_active":"2024-03-30T13:42:51.331372+0000","last_became_peered":"2024-03-30T13:42:51.331372+0000","last_unstale":"2024-03-30T13:44:02.296059+0000","last_undegraded":"2024-03-30T13:44:02.296059+0000","last_fullsized":"2024-03-30T13:44:02.296059+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:17.417830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.833967+0000","last_change":"2024-03-30T13:44:02.833988+0000","last_active":"2024-03-30T13:44:02.833967+0000","last_peered":"2024-03-30T13:44:02.833967+0000","last_clean":"2024-03-30T13:44:02.833967+0000","last_became_active":"2024-03-30T13:42:49.321360+0000","last_became_peered":"2024-03-30T13:42:49.321360+0000","last_unstale":"2024-03-30T13:44:02.833967+0000","last_undegraded":"2024-03-30T13:44:02.833967+0000","last_fullsized":"2024-03-30T13:44:02.833967+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:02.833909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:02.833909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:31.595404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.678731+0000","last_change":"2024-03-30T13:42:45.315544+0000","last_active":"2024-03-30T13:44:02.678731+0000","last_peered":"2024-03-30T13:44:02.678731+0000","last_clean":"2024-03-30T13:44:02.678731+0000","last_became_active":"2024-03-30T13:42:45.315251+0000","last_became_peered":"2024-03-30T13:42:45.315251+0000","last_unstale":"2024-03-30T13:44:02.678731+0000","last_undegraded":"2024-03-30T13:44:02.678731+0000","last_fullsized":"2024-03-30T13:44:02.678731+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.298490+0000","last_change":"2024-03-30T13:42:47.316447+0000","last_active":"2024-03-30T13:44:02.298490+0000","last_peered":"2024-03-30T13:44:02.298490+0000","last_clean":"2024-03-30T13:44:02.298490+0000","last_became_active":"2024-03-30T13:42:47.315495+0000","last_became_peered":"2024-03-30T13:42:47.315495+0000","last_unstale":"2024-03-30T13:44:02.298490+0000","last_undegraded":"2024-03-30T13:44:02.298490+0000","last_fullsized":"2024-03-30T13:44:02.298490+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.298700+0000","last_change":"2024-03-30T13:42:51.325672+0000","last_active":"2024-03-30T13:44:02.298700+0000","last_peered":"2024-03-30T13:44:02.298700+0000","last_clean":"2024-03-30T13:44:02.298700+0000","last_became_active":"2024-03-30T13:42:51.325389+0000","last_became_peered":"2024-03-30T13:42:51.325389+0000","last_unstale":"2024-03-30T13:44:02.298700+0000","last_undegraded":"2024-03-30T13:44:02.298700+0000","last_fullsized":"2024-03-30T13:44:02.298700+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"72'8","reported_seq":91,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295495+0000","last_change":"2024-03-30T13:43:50.850857+0000","last_active":"2024-03-30T13:44:02.295495+0000","last_peered":"2024-03-30T13:44:02.295495+0000","last_clean":"2024-03-30T13:44:02.295495+0000","last_became_active":"2024-03-30T13:42:49.323371+0000","last_became_peered":"2024-03-30T13:42:49.323371+0000","last_unstale":"2024-03-30T13:44:02.295495+0000","last_undegraded":"2024-03-30T13:44:02.295495+0000","last_fullsized":"2024-03-30T13:44:02.295495+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:43:50.850759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:50.850759+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:01.654480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.295588+0000","last_change":"2024-03-30T13:43:50.050389+0000","last_active":"2024-03-30T13:44:02.295588+0000","last_peered":"2024-03-30T13:44:02.295588+0000","last_clean":"2024-03-30T13:44:02.295588+0000","last_became_active":"2024-03-30T13:42:45.309265+0000","last_became_peered":"2024-03-30T13:42:45.309265+0000","last_unstale":"2024-03-30T13:44:02.295588+0000","last_undegraded":"2024-03-30T13:44:02.295588+0000","last_fullsized":"2024-03-30T13:44:02.295588+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:50.050303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:43:50.050303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:05.509528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":74,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-30T13:44:02.298126+0000","last_change":"2024-03-30T13:42:47.313353+0000","last_active":"2024-03-30T13:44:02.298126+0000","last_peered":"2024-03-30T13:44:02.298126+0000","last_clean":"2024-03-30T13:44:02.298126+0000","last_became_active":"2024-03-30T13:42:47.313026+0000","last_became_peered":"2024-03-30T13:42:47.313026+0000","last_unstale":"2024-03-30T13:44:02.298126+0000","last_undegraded":"2024-03-30T13:44:02.298126+0000","last_fullsized":"2024-03-30T13:44:02.298126+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:03.773018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":407,"ondisk_log_size":407,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103190,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32680,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712472,"statfs":{"total":95995035648,"available":95961571328,"internally_reserved":0,"allocated":1540096,"data_stored":1039826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397530,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32660,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712492,"statfs":{"total":95995035648,"available":95961591808,"internally_reserved":0,"allocated":1519616,"data_stored":1038774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":450356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32684,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712468,"statfs":{"total":95995035648,"available":95961567232,"internally_reserved":0,"allocated":1544192,"data_stored":1040430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":449493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":449284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953508,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":449409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691871,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":450102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:44:12.469 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-30T13:44:12.469 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-30T13:44:12.469 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-30T13:44:12.469 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph health --format=json 2024-03-30T13:44:12.717 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:44:12.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:12 smithi043 ceph-mon[29963]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T13:44:12.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:12 smithi043 ceph-mon[29963]: 4.3 scrub starts 2024-03-30T13:44:12.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:12 smithi043 ceph-mon[29963]: 4.3 scrub ok 2024-03-30T13:44:12.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:12 smithi043 ceph-mon[29963]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:44:12.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:12 smithi043 ceph-mon[29963]: 6.15 scrub starts 2024-03-30T13:44:12.839 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:12 smithi043 ceph-mon[29963]: 6.15 scrub ok 2024-03-30T13:44:12.839 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:12 smithi043 ceph-mon[29963]: osdmap e93: 8 total, 8 up, 8 in 2024-03-30T13:44:12.839 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:12 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:12.839 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:12 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:44:12.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:12 smithi043 ceph-mon[42099]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T13:44:12.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:12 smithi043 ceph-mon[42099]: 4.3 scrub starts 2024-03-30T13:44:12.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:12 smithi043 ceph-mon[42099]: 4.3 scrub ok 2024-03-30T13:44:12.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:12 smithi043 ceph-mon[42099]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:44:12.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:12 smithi043 ceph-mon[42099]: 6.15 scrub starts 2024-03-30T13:44:12.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:12 smithi043 ceph-mon[42099]: 6.15 scrub ok 2024-03-30T13:44:12.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:12 smithi043 ceph-mon[42099]: osdmap e93: 8 total, 8 up, 8 in 2024-03-30T13:44:12.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:12 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:12.840 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:12 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:44:12.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:12 smithi113 ceph-mon[34341]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T13:44:12.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:12 smithi113 ceph-mon[34341]: 4.3 scrub starts 2024-03-30T13:44:12.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:12 smithi113 ceph-mon[34341]: 4.3 scrub ok 2024-03-30T13:44:12.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:12 smithi113 ceph-mon[34341]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:44:12.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:12 smithi113 ceph-mon[34341]: 6.15 scrub starts 2024-03-30T13:44:12.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:12 smithi113 ceph-mon[34341]: 6.15 scrub ok 2024-03-30T13:44:12.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:12 smithi113 ceph-mon[34341]: osdmap e93: 8 total, 8 up, 8 in 2024-03-30T13:44:12.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:12 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:12.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:12 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:44:13.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:13 smithi043 ceph-mon[29963]: 6.7 deep-scrub starts 2024-03-30T13:44:13.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:13 smithi043 ceph-mon[29963]: 6.7 deep-scrub ok 2024-03-30T13:44:13.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:13 smithi043 ceph-mon[29963]: 4.2 scrub starts 2024-03-30T13:44:13.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:13 smithi043 ceph-mon[29963]: 4.2 scrub ok 2024-03-30T13:44:13.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:13 smithi043 ceph-mon[42099]: 6.7 deep-scrub starts 2024-03-30T13:44:13.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:13 smithi043 ceph-mon[42099]: 6.7 deep-scrub ok 2024-03-30T13:44:13.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:13 smithi043 ceph-mon[42099]: 4.2 scrub starts 2024-03-30T13:44:13.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:13 smithi043 ceph-mon[42099]: 4.2 scrub ok 2024-03-30T13:44:13.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:13 smithi113 ceph-mon[34341]: 6.7 deep-scrub starts 2024-03-30T13:44:13.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:13 smithi113 ceph-mon[34341]: 6.7 deep-scrub ok 2024-03-30T13:44:13.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:13 smithi113 ceph-mon[34341]: 4.2 scrub starts 2024-03-30T13:44:13.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:13 smithi113 ceph-mon[34341]: 4.2 scrub ok 2024-03-30T13:44:14.777 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:44:14.778 INFO:teuthology.orchestra.run.smithi043.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-30T13:44:14.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:14 smithi113 ceph-mon[34341]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:14.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:14 smithi113 ceph-mon[34341]: 5.6 scrub starts 2024-03-30T13:44:14.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:14 smithi113 ceph-mon[34341]: 5.6 scrub ok 2024-03-30T13:44:14.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:14 smithi113 ceph-mon[34341]: 4.5 scrub starts 2024-03-30T13:44:14.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:14 smithi113 ceph-mon[34341]: 4.5 scrub ok 2024-03-30T13:44:14.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:14 smithi113 ceph-mon[34341]: 6.8 scrub starts 2024-03-30T13:44:14.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:14 smithi113 ceph-mon[34341]: 6.8 scrub ok 2024-03-30T13:44:14.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:14 smithi113 ceph-mon[34341]: 5.14 scrub starts 2024-03-30T13:44:14.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:14 smithi113 ceph-mon[34341]: 5.14 scrub ok 2024-03-30T13:44:15.126 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:14 smithi043 ceph-mon[29963]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:15.126 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:14 smithi043 ceph-mon[29963]: 5.6 scrub starts 2024-03-30T13:44:15.126 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:14 smithi043 ceph-mon[29963]: 5.6 scrub ok 2024-03-30T13:44:15.127 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:14 smithi043 ceph-mon[29963]: 4.5 scrub starts 2024-03-30T13:44:15.127 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:14 smithi043 ceph-mon[29963]: 4.5 scrub ok 2024-03-30T13:44:15.127 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:14 smithi043 ceph-mon[29963]: 6.8 scrub starts 2024-03-30T13:44:15.127 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:14 smithi043 ceph-mon[29963]: 6.8 scrub ok 2024-03-30T13:44:15.127 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:14 smithi043 ceph-mon[29963]: 5.14 scrub starts 2024-03-30T13:44:15.128 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:14 smithi043 ceph-mon[29963]: 5.14 scrub ok 2024-03-30T13:44:15.128 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:14 smithi043 ceph-mon[42099]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:15.128 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:14 smithi043 ceph-mon[42099]: 5.6 scrub starts 2024-03-30T13:44:15.128 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:14 smithi043 ceph-mon[42099]: 5.6 scrub ok 2024-03-30T13:44:15.128 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:14 smithi043 ceph-mon[42099]: 4.5 scrub starts 2024-03-30T13:44:15.129 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:14 smithi043 ceph-mon[42099]: 4.5 scrub ok 2024-03-30T13:44:15.129 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:14 smithi043 ceph-mon[42099]: 6.8 scrub starts 2024-03-30T13:44:15.129 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:14 smithi043 ceph-mon[42099]: 6.8 scrub ok 2024-03-30T13:44:15.129 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:14 smithi043 ceph-mon[42099]: 5.14 scrub starts 2024-03-30T13:44:15.129 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:14 smithi043 ceph-mon[42099]: 5.14 scrub ok 2024-03-30T13:44:15.507 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-30T13:44:15.507 INFO:tasks.cephadm:Setup complete, yielding 2024-03-30T13:44:15.507 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-30T13:44:15.522 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-30T13:44:15.522 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'squid', 'sha1': '5af0400050a123dc31f105425174f106e057276a'}} 2024-03-30T13:44:15.522 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-30T13:44:15.522 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-30T13:44:15.768 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:44:15.890 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:15 smithi043 ceph-mon[29963]: 4.1a scrub starts 2024-03-30T13:44:15.890 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:15 smithi043 ceph-mon[29963]: 4.1a scrub ok 2024-03-30T13:44:15.890 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:15 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3723467018' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-30T13:44:15.890 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:15 smithi043 ceph-mon[29963]: 4.7 scrub starts 2024-03-30T13:44:15.890 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:15 smithi043 ceph-mon[29963]: 4.7 scrub ok 2024-03-30T13:44:15.890 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:15 smithi043 ceph-mon[29963]: 6.1c scrub starts 2024-03-30T13:44:15.890 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:15 smithi043 ceph-mon[29963]: 6.1c scrub ok 2024-03-30T13:44:15.890 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:15 smithi043 ceph-mon[42099]: 4.1a scrub starts 2024-03-30T13:44:15.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:15 smithi043 ceph-mon[42099]: 4.1a scrub ok 2024-03-30T13:44:15.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:15 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3723467018' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-30T13:44:15.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:15 smithi043 ceph-mon[42099]: 4.7 scrub starts 2024-03-30T13:44:15.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:15 smithi043 ceph-mon[42099]: 4.7 scrub ok 2024-03-30T13:44:15.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:15 smithi043 ceph-mon[42099]: 6.1c scrub starts 2024-03-30T13:44:15.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:15 smithi043 ceph-mon[42099]: 6.1c scrub ok 2024-03-30T13:44:15.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:15 smithi113 ceph-mon[34341]: 4.1a scrub starts 2024-03-30T13:44:15.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:15 smithi113 ceph-mon[34341]: 4.1a scrub ok 2024-03-30T13:44:15.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:15 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3723467018' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-30T13:44:15.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:15 smithi113 ceph-mon[34341]: 4.7 scrub starts 2024-03-30T13:44:15.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:15 smithi113 ceph-mon[34341]: 4.7 scrub ok 2024-03-30T13:44:15.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:15 smithi113 ceph-mon[34341]: 6.1c scrub starts 2024-03-30T13:44:15.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:15 smithi113 ceph-mon[34341]: 6.1c scrub ok 2024-03-30T13:44:16.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:44:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:44:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:44:16.884 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:16 smithi043 ceph-mon[29963]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:16.884 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:16 smithi043 ceph-mon[29963]: 4.1e scrub starts 2024-03-30T13:44:16.884 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:16 smithi043 ceph-mon[29963]: 4.1e scrub ok 2024-03-30T13:44:16.884 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:16 smithi043 ceph-mon[29963]: 5.2 scrub starts 2024-03-30T13:44:16.884 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:16 smithi043 ceph-mon[29963]: 5.2 scrub ok 2024-03-30T13:44:16.884 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:16 smithi043 ceph-mon[42099]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:16.884 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:16 smithi043 ceph-mon[42099]: 4.1e scrub starts 2024-03-30T13:44:16.884 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:16 smithi043 ceph-mon[42099]: 4.1e scrub ok 2024-03-30T13:44:16.885 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:16 smithi043 ceph-mon[42099]: 5.2 scrub starts 2024-03-30T13:44:16.885 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:16 smithi043 ceph-mon[42099]: 5.2 scrub ok 2024-03-30T13:44:16.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:16 smithi113 ceph-mon[34341]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:17.156 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:16 smithi113 ceph-mon[34341]: 4.1e scrub starts 2024-03-30T13:44:17.156 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:16 smithi113 ceph-mon[34341]: 4.1e scrub ok 2024-03-30T13:44:17.156 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:16 smithi113 ceph-mon[34341]: 5.2 scrub starts 2024-03-30T13:44:17.156 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:16 smithi113 ceph-mon[34341]: 5.2 scrub ok 2024-03-30T13:44:17.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:17 smithi043 ceph-mon[29963]: 4.e scrub starts 2024-03-30T13:44:17.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:17 smithi043 ceph-mon[29963]: 4.e scrub ok 2024-03-30T13:44:17.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:17 smithi043 ceph-mon[29963]: 6.16 scrub starts 2024-03-30T13:44:17.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:17 smithi043 ceph-mon[29963]: 6.16 scrub ok 2024-03-30T13:44:17.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:17 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2772187123' entity='client.admin' 2024-03-30T13:44:17.900 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:44:17 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:44:17] ENGINE Bus STOPPING 2024-03-30T13:44:17.900 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:44:17 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:44:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-30T13:44:17.900 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:44:17 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:44:17] ENGINE Bus STOPPED 2024-03-30T13:44:17.900 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:44:17 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:44:17] ENGINE Bus STARTING 2024-03-30T13:44:17.900 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:44:17 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:44:17] ENGINE Serving on http://:::9283 2024-03-30T13:44:17.901 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:44:17 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:44:17] ENGINE Bus STARTED 2024-03-30T13:44:17.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:17 smithi043 ceph-mon[42099]: 4.e scrub starts 2024-03-30T13:44:17.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:17 smithi043 ceph-mon[42099]: 4.e scrub ok 2024-03-30T13:44:17.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:17 smithi043 ceph-mon[42099]: 6.16 scrub starts 2024-03-30T13:44:17.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:17 smithi043 ceph-mon[42099]: 6.16 scrub ok 2024-03-30T13:44:17.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:17 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2772187123' entity='client.admin' 2024-03-30T13:44:17.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:17 smithi113 ceph-mon[34341]: 4.e scrub starts 2024-03-30T13:44:17.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:17 smithi113 ceph-mon[34341]: 4.e scrub ok 2024-03-30T13:44:17.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:17 smithi113 ceph-mon[34341]: 6.16 scrub starts 2024-03-30T13:44:17.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:17 smithi113 ceph-mon[34341]: 6.16 scrub ok 2024-03-30T13:44:17.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:17 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2772187123' entity='client.admin' 2024-03-30T13:44:18.246 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-30T13:44:18.247 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-30T13:44:18.247 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T13:44:18.458 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:44:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:44:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:44:18.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:18 smithi113 ceph-mon[34341]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 0 op/s 2024-03-30T13:44:18.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:18 smithi043 ceph-mon[29963]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 0 op/s 2024-03-30T13:44:18.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:18 smithi043 ceph-mon[42099]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 0 op/s 2024-03-30T13:44:18.894 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:44:18 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[80418]: ts=2024-03-30T13:44:18.825Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003648159s 2024-03-30T13:44:19.960 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:19 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:44:19.960 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:19 smithi113 ceph-mon[34341]: 6.19 scrub starts 2024-03-30T13:44:19.960 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:19 smithi113 ceph-mon[34341]: 6.19 scrub ok 2024-03-30T13:44:19.960 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:19 smithi113 ceph-mon[34341]: 4.4 scrub starts 2024-03-30T13:44:19.960 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:19 smithi113 ceph-mon[34341]: 4.4 scrub ok 2024-03-30T13:44:19.960 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:19 smithi113 ceph-mon[34341]: 3.13 scrub starts 2024-03-30T13:44:19.960 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:19 smithi113 ceph-mon[34341]: 3.13 scrub ok 2024-03-30T13:44:20.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:19 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:44:20.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:19 smithi043 ceph-mon[29963]: 6.19 scrub starts 2024-03-30T13:44:20.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:19 smithi043 ceph-mon[29963]: 6.19 scrub ok 2024-03-30T13:44:20.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:19 smithi043 ceph-mon[29963]: 4.4 scrub starts 2024-03-30T13:44:20.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:19 smithi043 ceph-mon[29963]: 4.4 scrub ok 2024-03-30T13:44:20.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:19 smithi043 ceph-mon[29963]: 3.13 scrub starts 2024-03-30T13:44:20.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:19 smithi043 ceph-mon[29963]: 3.13 scrub ok 2024-03-30T13:44:20.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:19 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:44:20.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:19 smithi043 ceph-mon[42099]: 6.19 scrub starts 2024-03-30T13:44:20.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:19 smithi043 ceph-mon[42099]: 6.19 scrub ok 2024-03-30T13:44:20.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:19 smithi043 ceph-mon[42099]: 4.4 scrub starts 2024-03-30T13:44:20.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:19 smithi043 ceph-mon[42099]: 4.4 scrub ok 2024-03-30T13:44:20.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:19 smithi043 ceph-mon[42099]: 3.13 scrub starts 2024-03-30T13:44:20.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:19 smithi043 ceph-mon[42099]: 3.13 scrub ok 2024-03-30T13:44:20.222 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 systemd[1]: Starting Ceph grafana.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T13:44:20.707 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 podman[68553]: 2024-03-30 13:44:20.257947544 +0000 UTC m=+0.019142206 image pull 2bacad6d85d802cd176c146528adcc0e47a8a83e94bc4a8b7ba1a9b9ef2c65e7 quay.io/ceph/grafana:9.4.12 2024-03-30T13:44:20.708 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 podman[68553]: 2024-03-30 13:44:20.359084605 +0000 UTC m=+0.120279248 container create d71d74017d3f4c67fd3101752e2225b11d8b0f5a3d4f7c47b9d50035e0f07b69 (image=quay.io/ceph/grafana:9.4.12, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a, maintainer=Grafana Labs ) 2024-03-30T13:44:20.708 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 podman[68553]: 2024-03-30 13:44:20.595667602 +0000 UTC m=+0.356862260 container init d71d74017d3f4c67fd3101752e2225b11d8b0f5a3d4f7c47b9d50035e0f07b69 (image=quay.io/ceph/grafana:9.4.12, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a, maintainer=Grafana Labs ) 2024-03-30T13:44:20.708 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 podman[68553]: 2024-03-30 13:44:20.601146724 +0000 UTC m=+0.362341369 container start d71d74017d3f4c67fd3101752e2225b11d8b0f5a3d4f7c47b9d50035e0f07b69 (image=quay.io/ceph/grafana:9.4.12, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a, maintainer=Grafana Labs ) 2024-03-30T13:44:20.717 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:44:20.719 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-30T13:44:20.958 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-mon[34341]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:20.958 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-mon[34341]: 6.1d scrub starts 2024-03-30T13:44:20.958 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-mon[34341]: 6.1d scrub ok 2024-03-30T13:44:20.958 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-mon[34341]: 3.d scrub starts 2024-03-30T13:44:20.958 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-mon[34341]: 3.d scrub ok 2024-03-30T13:44:20.958 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-mon[34341]: 5.0 scrub starts 2024-03-30T13:44:20.959 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-mon[34341]: 5.0 scrub ok 2024-03-30T13:44:20.959 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=settings t=2024-03-30T13:44:20.748612494Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-30T13:44:20.959 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=settings t=2024-03-30T13:44:20.7487817Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-30T13:44:20.960 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=settings t=2024-03-30T13:44:20.748792503Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-30T13:44:20.960 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=settings t=2024-03-30T13:44:20.748795934Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-30T13:44:20.960 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=settings t=2024-03-30T13:44:20.748798972Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-03-30T13:44:20.960 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=settings t=2024-03-30T13:44:20.74880179Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-03-30T13:44:20.960 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=settings t=2024-03-30T13:44:20.748804471Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-03-30T13:44:20.960 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=settings t=2024-03-30T13:44:20.748807011Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-03-30T13:44:20.960 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=settings t=2024-03-30T13:44:20.748809552Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-03-30T13:44:20.960 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=settings t=2024-03-30T13:44:20.74881245Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-30T13:44:20.960 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=settings t=2024-03-30T13:44:20.748815221Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-30T13:44:20.960 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=settings t=2024-03-30T13:44:20.748817709Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-30T13:44:20.961 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=settings t=2024-03-30T13:44:20.74882026Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-30T13:44:20.961 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=settings t=2024-03-30T13:44:20.748822849Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-30T13:44:20.961 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=settings t=2024-03-30T13:44:20.748825579Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-30T13:44:20.961 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=settings t=2024-03-30T13:44:20.748828131Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-30T13:44:20.961 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=settings t=2024-03-30T13:44:20.74883055Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-30T13:44:20.961 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=settings t=2024-03-30T13:44:20.748833132Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-30T13:44:20.961 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=settings t=2024-03-30T13:44:20.748836069Z level=info msg="App mode production" 2024-03-30T13:44:20.961 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=sqlstore t=2024-03-30T13:44:20.748875268Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-30T13:44:20.961 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=sqlstore t=2024-03-30T13:44:20.74888586Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-30T13:44:20.961 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.749736952Z level=info msg="Starting DB migrations" 2024-03-30T13:44:20.961 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.749797004Z level=info msg="Executing migration" id="create migration_log table" 2024-03-30T13:44:20.961 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.750355153Z level=info msg="Executing migration" id="create user table" 2024-03-30T13:44:20.962 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.75079728Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-30T13:44:20.962 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.753266291Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-30T13:44:20.962 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.755959578Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-30T13:44:20.962 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.758612913Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-30T13:44:20.962 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.759018118Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-30T13:44:20.962 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.761800857Z level=info msg="Executing migration" id="create user table v2" 2024-03-30T13:44:20.962 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 bash[68553]: d71d74017d3f4c67fd3101752e2225b11d8b0f5a3d4f7c47b9d50035e0f07b69 2024-03-30T13:44:20.962 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.764384106Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-30T13:44:20.962 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 systemd[1]: Started Ceph grafana.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T13:44:20.962 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.766895229Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-30T13:44:20.962 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.767507786Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-30T13:44:20.962 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.770072796Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-30T13:44:20.963 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.772678798Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-30T13:44:20.963 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.775212228Z level=info msg="Executing migration" id="Update user table charset" 2024-03-30T13:44:20.963 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.775543251Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-30T13:44:20.963 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.778181286Z level=info msg="Executing migration" id="Add missing user data" 2024-03-30T13:44:20.963 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.778720361Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-30T13:44:20.963 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.780945244Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-30T13:44:20.963 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.78152997Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-30T13:44:20.963 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.783347623Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-30T13:44:20.963 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.785366664Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-30T13:44:20.963 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.787859901Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-30T13:44:20.963 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.790357045Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-30T13:44:20.964 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.792729625Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-30T13:44:20.964 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.795163295Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-30T13:44:20.964 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.797567846Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-30T13:44:20.964 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.7978879Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-30T13:44:20.964 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.800296325Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-30T13:44:20.964 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.800765271Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-30T13:44:20.964 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.802640865Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-30T13:44:20.964 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.803051436Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-30T13:44:20.964 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.805126583Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-30T13:44:20.964 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.805589983Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-30T13:44:20.964 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.805967071Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-30T13:44:20.964 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.806389929Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-30T13:44:20.965 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.809061882Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-30T13:44:20.965 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.809621053Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-30T13:44:20.965 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.810080607Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-30T13:44:20.965 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.810551066Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-30T13:44:20.965 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.810984185Z level=info msg="Executing migration" id="create star table" 2024-03-30T13:44:20.965 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.811480039Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-30T13:44:20.965 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.812037937Z level=info msg="Executing migration" id="create org table v1" 2024-03-30T13:44:20.965 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.812453239Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-30T13:44:20.965 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.812853542Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-30T13:44:20.965 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.813239713Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-30T13:44:20.965 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.813767159Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-30T13:44:20.965 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.814145019Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-30T13:44:20.965 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.814520633Z level=info msg="Executing migration" id="Update org table charset" 2024-03-30T13:44:20.966 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.817129994Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-30T13:44:20.966 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.819817116Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-30T13:44:20.966 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.82252201Z level=info msg="Executing migration" id="create dashboard table" 2024-03-30T13:44:20.966 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.825298054Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-30T13:44:20.966 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.825774111Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-30T13:44:20.966 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.828468272Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-30T13:44:20.967 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.831093394Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-30T13:44:20.967 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.833680715Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-30T13:44:20.967 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.83423904Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-30T13:44:20.967 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.836950384Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-30T13:44:20.967 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.839647969Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-30T13:44:20.967 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.842215726Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-30T13:44:20.967 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.842797953Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-30T13:44:20.967 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.845404148Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-30T13:44:20.968 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.846061768Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-30T13:44:20.968 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.847977502Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-30T13:44:20.968 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.850675286Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-30T13:44:20.968 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.85137586Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-30T13:44:20.968 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.854108982Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-30T13:44:20.968 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.856581758Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-30T13:44:20.968 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.859382019Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-30T13:44:20.968 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.859954539Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-30T13:44:20.968 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.86233011Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-30T13:44:20.968 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.864530573Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-30T13:44:20.968 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.86693613Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-30T13:44:20.969 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.867531462Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-30T13:44:20.969 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.868063663Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-30T13:44:20.969 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.870414028Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-30T13:44:20.969 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.872911209Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-30T13:44:20.969 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.875426197Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-30T13:44:20.969 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.877881477Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-30T13:44:20.969 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.880314023Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-30T13:44:20.969 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.882620399Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-30T13:44:20.969 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.884996775Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-30T13:44:20.969 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.887455886Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-30T13:44:20.969 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.890046801Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-30T13:44:20.969 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.892678378Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-30T13:44:20.970 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.895088866Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-30T13:44:20.970 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.897388904Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-30T13:44:20.970 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.899760907Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-30T13:44:20.970 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.902153544Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-30T13:44:20.970 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.904675593Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-30T13:44:20.970 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.907167803Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-30T13:44:20.970 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.909654104Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-30T13:44:20.970 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.912080186Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-30T13:44:20.970 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.914469264Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-30T13:44:20.970 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.91698046Z level=info msg="Executing migration" id="create data_source table" 2024-03-30T13:44:20.970 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.919327411Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-30T13:44:20.971 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.919877779Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-30T13:44:20.971 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.92169045Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-30T13:44:20.971 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.924076323Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-30T13:44:20.971 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.926484101Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-30T13:44:20.971 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.928956934Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-30T13:44:20.971 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.931249942Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-30T13:44:20.971 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.933690534Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-30T13:44:20.971 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.936097563Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-30T13:44:20.971 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.938399166Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-30T13:44:20.971 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.940869275Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-30T13:44:20.971 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.943227393Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-30T13:44:20.971 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.944245202Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-30T13:44:20.972 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.945205447Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-30T13:44:20.972 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.946203582Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-30T13:44:20.972 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.947220575Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-30T13:44:20.972 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.948274566Z level=info msg="Executing migration" id="Add uid column" 2024-03-30T13:44:20.972 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.949245631Z level=info msg="Executing migration" id="Update uid value" 2024-03-30T13:44:20.972 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.950203865Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-30T13:44:20.972 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.951273191Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-30T13:44:20.972 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.952302295Z level=info msg="Executing migration" id="create api_key table" 2024-03-30T13:44:20.972 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.953311364Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-30T13:44:20.972 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.954365623Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-30T13:44:20.972 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.955286411Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-30T13:44:20.972 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.956299002Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-30T13:44:20.973 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.957244247Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-30T13:44:20.973 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.958198372Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-30T13:44:20.973 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.959184261Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-30T13:44:21.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:20 smithi043 ceph-mon[29963]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:21.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:20 smithi043 ceph-mon[29963]: 6.1d scrub starts 2024-03-30T13:44:21.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:20 smithi043 ceph-mon[29963]: 6.1d scrub ok 2024-03-30T13:44:21.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:20 smithi043 ceph-mon[29963]: 3.d scrub starts 2024-03-30T13:44:21.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:20 smithi043 ceph-mon[29963]: 3.d scrub ok 2024-03-30T13:44:21.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:20 smithi043 ceph-mon[29963]: 5.0 scrub starts 2024-03-30T13:44:21.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:20 smithi043 ceph-mon[29963]: 5.0 scrub ok 2024-03-30T13:44:21.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:20 smithi043 ceph-mon[42099]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:21.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:20 smithi043 ceph-mon[42099]: 6.1d scrub starts 2024-03-30T13:44:21.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:20 smithi043 ceph-mon[42099]: 6.1d scrub ok 2024-03-30T13:44:21.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:20 smithi043 ceph-mon[42099]: 3.d scrub starts 2024-03-30T13:44:21.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:20 smithi043 ceph-mon[42099]: 3.d scrub ok 2024-03-30T13:44:21.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:20 smithi043 ceph-mon[42099]: 5.0 scrub starts 2024-03-30T13:44:21.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:20 smithi043 ceph-mon[42099]: 5.0 scrub ok 2024-03-30T13:44:21.209 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.961076889Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-30T13:44:21.209 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.961537305Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-30T13:44:21.209 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.961926209Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-30T13:44:21.209 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.962327851Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-30T13:44:21.209 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.962725812Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-30T13:44:21.209 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.963006538Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-30T13:44:21.209 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.963380327Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-30T13:44:21.210 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.963657345Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-30T13:44:21.210 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.964204141Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-30T13:44:21.210 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.964803371Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-30T13:44:21.210 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.965111475Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-30T13:44:21.210 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.965801873Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-30T13:44:21.210 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.966449235Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-30T13:44:21.210 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.966858996Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-30T13:44:21.210 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.96724357Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-30T13:44:21.210 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.967677441Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-30T13:44:21.210 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.968065193Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-30T13:44:21.210 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.968526495Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-30T13:44:21.210 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.968910438Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-30T13:44:21.210 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.969155144Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-30T13:44:21.211 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.969436004Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-30T13:44:21.211 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.970006552Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-30T13:44:21.211 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.970591337Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-30T13:44:21.211 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.97087318Z level=info msg="Executing migration" id="create quota table v1" 2024-03-30T13:44:21.211 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.971253029Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-30T13:44:21.211 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.971654922Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-30T13:44:21.211 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.971902184Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-30T13:44:21.212 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.972278751Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-30T13:44:21.212 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.9726838Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-30T13:44:21.212 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.973246003Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-30T13:44:21.212 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.973532643Z level=info msg="Executing migration" id="create session table" 2024-03-30T13:44:21.212 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.973934848Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-30T13:44:21.212 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.97418937Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-30T13:44:21.212 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.974487026Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-30T13:44:21.212 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.97486771Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-30T13:44:21.213 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.975229117Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-30T13:44:21.213 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.975502217Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-30T13:44:21.213 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.975741561Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-30T13:44:21.214 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.975996578Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-30T13:44:21.214 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.976262008Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-30T13:44:21.215 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.976640759Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-30T13:44:21.215 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.978185277Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-30T13:44:21.215 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.978862563Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-30T13:44:21.215 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.98063999Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-30T13:44:21.215 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.981254131Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-30T13:44:21.215 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.983180908Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-30T13:44:21.215 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.983551125Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-30T13:44:21.215 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.985856493Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-30T13:44:21.215 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.988301727Z level=info msg="Executing migration" id="create alert table v1" 2024-03-30T13:44:21.215 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.990624671Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-30T13:44:21.215 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.991083588Z level=info msg="Executing migration" id="add index alert state" 2024-03-30T13:44:21.215 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.993375675Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-30T13:44:21.215 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.995747028Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-30T13:44:21.215 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.998062361Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-30T13:44:21.216 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:20 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:20.998506521Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-30T13:44:21.216 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.000883454Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-30T13:44:21.216 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.003278943Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-30T13:44:21.216 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.005719858Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-30T13:44:21.216 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.006156289Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-30T13:44:21.216 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.008470339Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-30T13:44:21.216 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.010888693Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-30T13:44:21.216 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.01333259Z level=info msg="Executing migration" id="Add column is_default" 2024-03-30T13:44:21.217 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.014066179Z level=info msg="Executing migration" id="Add column frequency" 2024-03-30T13:44:21.217 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.01635582Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-30T13:44:21.217 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.018767151Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-30T13:44:21.217 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.021118804Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-30T13:44:21.217 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.021566183Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-30T13:44:21.217 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.023999396Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-30T13:44:21.217 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.026374451Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-30T13:44:21.218 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.028740809Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-30T13:44:21.218 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.029178918Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-30T13:44:21.218 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.03025173Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-30T13:44:21.218 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.031265251Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-30T13:44:21.218 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.032211901Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-30T13:44:21.218 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.032978794Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-30T13:44:21.219 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.033928255Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-30T13:44:21.219 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.034968457Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-30T13:44:21.219 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.035993546Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-30T13:44:21.219 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.03647644Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-30T13:44:21.219 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.037473341Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-30T13:44:21.219 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.038436552Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-30T13:44:21.219 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.039457916Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-30T13:44:21.219 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.039893857Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-30T13:44:21.219 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.040820115Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-30T13:44:21.220 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.041368121Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-30T13:44:21.220 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.041850243Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-30T13:44:21.220 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.042832072Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-30T13:44:21.220 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.043279304Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-30T13:44:21.220 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.044298655Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-30T13:44:21.220 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.045319051Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-30T13:44:21.220 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.046322754Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-30T13:44:21.220 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.047109651Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-30T13:44:21.220 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.048091397Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-30T13:44:21.220 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.049146306Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-30T13:44:21.221 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.05014543Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-30T13:44:21.221 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.050596145Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-30T13:44:21.221 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.051620398Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-30T13:44:21.221 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.053609517Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-30T13:44:21.221 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.054525906Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-30T13:44:21.221 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.054967866Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-30T13:44:21.221 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.055956312Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-30T13:44:21.221 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.056395513Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-30T13:44:21.221 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.056962571Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-30T13:44:21.221 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.058001416Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-30T13:44:21.221 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.058786245Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-30T13:44:21.221 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.059708917Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-30T13:44:21.221 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.060758626Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-30T13:44:21.221 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.06171473Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-30T13:44:21.222 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.062539583Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-30T13:44:21.222 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.063491199Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-30T13:44:21.222 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.064444979Z level=info msg="Executing migration" id="Move region to single row" 2024-03-30T13:44:21.222 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.065401977Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-30T13:44:21.222 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.065859497Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-30T13:44:21.222 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.066792417Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-30T13:44:21.222 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.067256871Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-30T13:44:21.222 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.067768891Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-30T13:44:21.222 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.068706329Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-30T13:44:21.222 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.06919284Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-30T13:44:21.222 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.069524882Z level=info msg="Executing migration" id="create test_data table" 2024-03-30T13:44:21.223 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.070481378Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-30T13:44:21.223 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.07143232Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-30T13:44:21.223 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.072388228Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-30T13:44:21.223 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.072834135Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-30T13:44:21.223 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.07375442Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-30T13:44:21.223 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.074095093Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-30T13:44:21.223 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.074737918Z level=info msg="Executing migration" id="create team table" 2024-03-30T13:44:21.223 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.075723191Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-30T13:44:21.223 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.076186499Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-30T13:44:21.223 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.077195602Z level=info msg="Executing migration" id="create team member table" 2024-03-30T13:44:21.223 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.078181738Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-30T13:44:21.224 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.079103895Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-30T13:44:21.224 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.079560544Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-30T13:44:21.224 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.080510426Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-30T13:44:21.224 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.081464848Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-30T13:44:21.224 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.082454765Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-30T13:44:21.224 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.083279898Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-30T13:44:21.224 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.084219463Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-30T13:44:21.224 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.085210499Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-30T13:44:21.224 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.08617544Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-30T13:44:21.224 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.086625252Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-30T13:44:21.224 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.087562973Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-30T13:44:21.224 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.08848285Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-30T13:44:21.224 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.091257578Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-30T13:44:21.224 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.093936769Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-30T13:44:21.225 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.096501283Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-30T13:44:21.225 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.099109757Z level=info msg="Executing migration" id="create tag table" 2024-03-30T13:44:21.225 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.101808125Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-30T13:44:21.225 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.104474352Z level=info msg="Executing migration" id="create login attempt table" 2024-03-30T13:44:21.225 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.107282063Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-30T13:44:21.225 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.11007062Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-30T13:44:21.225 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.112751042Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-30T13:44:21.225 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.11541899Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-30T13:44:21.225 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.11800886Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-30T13:44:21.225 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.120714724Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-30T13:44:21.225 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.12331491Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-30T13:44:21.225 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.125984565Z level=info msg="Executing migration" id="create user auth table" 2024-03-30T13:44:21.225 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.128674614Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-30T13:44:21.225 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.131344951Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-30T13:44:21.226 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.1341665Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-30T13:44:21.226 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.136859603Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-30T13:44:21.226 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.139631661Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-30T13:44:21.226 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.142320657Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-30T13:44:21.226 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.145114299Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-30T13:44:21.226 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.147962342Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-30T13:44:21.226 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.15054886Z level=info msg="Executing migration" id="create server_lock table" 2024-03-30T13:44:21.226 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.153272477Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-30T13:44:21.226 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.153722024Z level=info msg="Executing migration" id="create user auth token table" 2024-03-30T13:44:21.226 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.156504518Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-30T13:44:21.226 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.159184445Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-30T13:44:21.226 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.161822713Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-30T13:44:21.226 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.162296751Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-30T13:44:21.226 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.164943751Z level=info msg="Executing migration" id="create cache_data table" 2024-03-30T13:44:21.227 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.16769291Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-30T13:44:21.227 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.170315059Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-30T13:44:21.227 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.170772222Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-30T13:44:21.227 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.173330128Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-30T13:44:21.227 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.173665891Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-30T13:44:21.227 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.173960529Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-30T13:44:21.227 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.174453345Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-30T13:44:21.227 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.174915396Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-30T13:44:21.227 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.175365908Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-30T13:44:21.227 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.175695294Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-30T13:44:21.227 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.176126466Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-30T13:44:21.227 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.176566757Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-30T13:44:21.227 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.177072764Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-30T13:44:21.227 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.177562821Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-30T13:44:21.228 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.178488698Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-30T13:44:21.228 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.178958624Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-30T13:44:21.228 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.179259426Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-30T13:44:21.228 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.179706412Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-30T13:44:21.228 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.180121822Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-30T13:44:21.228 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.18059465Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-30T13:44:21.228 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.180922064Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-30T13:44:21.228 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.181438397Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-30T13:44:21.228 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.181903999Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-30T13:44:21.229 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.182333978Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-30T13:44:21.229 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.18279826Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-30T13:44:21.229 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.183691562Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-30T13:44:21.229 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.184142205Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-30T13:44:21.229 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.184586045Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-30T13:44:21.229 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.189101392Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-30T13:44:21.229 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.193561168Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-30T13:44:21.229 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.194037399Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-30T13:44:21.229 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.194508422Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-30T13:44:21.229 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.195484953Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-30T13:44:21.229 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.195940091Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-30T13:44:21.229 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.198631328Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-30T13:44:21.229 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.19911972Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-30T13:44:21.229 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.201807023Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-30T13:44:21.229 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.20451657Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-30T13:44:21.230 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.207137185Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-30T13:44:21.230 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.208057928Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-30T13:44:21.230 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.210713523Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-30T13:44:21.472 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.213277021Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-30T13:44:21.472 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.215945268Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-30T13:44:21.472 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.216885112Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-30T13:44:21.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.219500227Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-30T13:44:21.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.222236372Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-30T13:44:21.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.224906581Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-30T13:44:21.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.22588915Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-30T13:44:21.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.228557287Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-30T13:44:21.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.230964079Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-30T13:44:21.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.233487494Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-30T13:44:21.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.233941834Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-30T13:44:21.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.234239748Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-30T13:44:21.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.235238888Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-30T13:44:21.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.236804669Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-30T13:44:21.474 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.239216141Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-30T13:44:21.474 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.240173021Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-30T13:44:21.474 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.242636839Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-30T13:44:21.474 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.245039618Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-30T13:44:21.474 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.247364386Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-30T13:44:21.474 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.248310518Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-30T13:44:21.474 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.250919134Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-30T13:44:21.474 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.25337975Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-30T13:44:21.474 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.256038919Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-30T13:44:21.474 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.257007411Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-30T13:44:21.474 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.259380239Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-30T13:44:21.474 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.261732867Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-30T13:44:21.474 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.264137477Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-30T13:44:21.475 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.264582576Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-30T13:44:21.475 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.266890524Z level=info msg="Executing migration" id="create alert_image table" 2024-03-30T13:44:21.475 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.269379623Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-30T13:44:21.475 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.271759252Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-30T13:44:21.475 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.272082979Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-30T13:44:21.475 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.274495359Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-30T13:44:21.475 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.274960507Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-30T13:44:21.475 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.275047445Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-30T13:44:21.476 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.277288675Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-30T13:44:21.476 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.277685455Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-30T13:44:21.476 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.278116811Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-30T13:44:21.477 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.278255749Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-30T13:44:21.477 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.278384476Z level=info msg="alerts found to migrate" alerts=0 2024-03-30T13:44:21.477 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.278822216Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-30T13:44:21.477 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.280510788Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-30T13:44:21.477 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.282953866Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-30T13:44:21.477 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.285402164Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-30T13:44:21.477 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.285888897Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-30T13:44:21.477 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.288187913Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-30T13:44:21.477 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.29054428Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-30T13:44:21.477 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.292809119Z level=info msg="Executing migration" id="create data_keys table" 2024-03-30T13:44:21.477 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.293271137Z level=info msg="Executing migration" id="create secrets table" 2024-03-30T13:44:21.477 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.294217506Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-30T13:44:21.478 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.299798029Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-30T13:44:21.478 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.300883172Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-30T13:44:21.478 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.301211005Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-30T13:44:21.478 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.306963869Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-30T13:44:21.478 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.31302477Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-30T13:44:21.478 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.314036632Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-30T13:44:21.478 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.31453969Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-30T13:44:21.478 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.315561349Z level=info msg="Executing migration" id="create permission table" 2024-03-30T13:44:21.478 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.316579008Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-30T13:44:21.478 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.317804855Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-30T13:44:21.478 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.318443447Z level=info msg="Executing migration" id="create role table" 2024-03-30T13:44:21.478 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.319411951Z level=info msg="Executing migration" id="add column display_name" 2024-03-30T13:44:21.478 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.320608546Z level=info msg="Executing migration" id="add column group_name" 2024-03-30T13:44:21.479 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.321921928Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-30T13:44:21.479 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.322417342Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-30T13:44:21.479 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.323441268Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-30T13:44:21.479 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.32447047Z level=info msg="Executing migration" id="create team role table" 2024-03-30T13:44:21.479 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.325453805Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-30T13:44:21.479 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.325877245Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-30T13:44:21.479 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.326835206Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-30T13:44:21.479 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.327892897Z level=info msg="Executing migration" id="create user role table" 2024-03-30T13:44:21.479 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.328940969Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-30T13:44:21.479 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.329372336Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-30T13:44:21.479 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.330361929Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-30T13:44:21.479 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.331329919Z level=info msg="Executing migration" id="create builtin role table" 2024-03-30T13:44:21.479 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.3323442Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-30T13:44:21.479 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.332775259Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-30T13:44:21.480 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.333817249Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-30T13:44:21.480 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.335010986Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-30T13:44:21.480 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.335502735Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-30T13:44:21.480 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.336396634Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-30T13:44:21.480 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.336847075Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-30T13:44:21.480 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.337274112Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-30T13:44:21.480 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.337682778Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-30T13:44:21.480 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.33813588Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-30T13:44:21.480 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.339288839Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-30T13:44:21.480 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.340277608Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-30T13:44:21.480 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.340739849Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-30T13:44:21.480 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.341021151Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-30T13:44:21.480 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.341463151Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-30T13:44:21.481 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.341935106Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-30T13:44:21.481 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.342290889Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-30T13:44:21.481 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.342615053Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-30T13:44:21.481 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.342957081Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-30T13:44:21.481 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.343378629Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-30T13:44:21.481 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.343836583Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-30T13:44:21.481 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.344995476Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-30T13:44:21.481 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.345315497Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-30T13:44:21.481 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.345749017Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-30T13:44:21.481 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.346192715Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-30T13:44:21.481 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.346642941Z level=info msg="Executing migration" id="add correlation config column" 2024-03-30T13:44:21.481 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.347808143Z level=info msg="Executing migration" id="create entity_events table" 2024-03-30T13:44:21.481 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.348256496Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-30T13:44:21.481 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.348680248Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-30T13:44:21.482 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.348778356Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-30T13:44:21.482 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.349062957Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-30T13:44:21.482 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.349159001Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-30T13:44:21.482 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.351707586Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-30T13:44:21.482 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.354325182Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-30T13:44:21.482 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.356913492Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-30T13:44:21.482 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.357415101Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-30T13:44:21.482 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.360151158Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-30T13:44:21.482 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.362624153Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-30T13:44:21.482 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.365009164Z level=info msg="Executing migration" id="Drop public config table" 2024-03-30T13:44:21.482 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.365473299Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-30T13:44:21.482 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.36802819Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-30T13:44:21.483 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.37076378Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-30T13:44:21.483 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.373121917Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-30T13:44:21.483 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.373600897Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-30T13:44:21.483 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.377399861Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-30T13:44:21.483 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.380001473Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-30T13:44:21.483 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.382365107Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-30T13:44:21.483 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.382706879Z level=info msg="Executing migration" id="add share column" 2024-03-30T13:44:21.483 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.385110254Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-30T13:44:21.483 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.387441762Z level=info msg="Executing migration" id="create file table" 2024-03-30T13:44:21.483 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.389787557Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-30T13:44:21.483 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.390295711Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-30T13:44:21.483 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.392632554Z level=info msg="Executing migration" id="create file_meta table" 2024-03-30T13:44:21.483 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.395093971Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-30T13:44:21.484 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.397445332Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-30T13:44:21.484 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.397769068Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-30T13:44:21.484 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.400100708Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-30T13:44:21.484 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.400453544Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-30T13:44:21.484 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.402648468Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-30T13:44:21.484 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.404989311Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-30T13:44:21.484 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.405337494Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-30T13:44:21.484 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.407751713Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-30T13:44:21.484 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.410158309Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-30T13:44:21.484 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.412533186Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-30T13:44:21.484 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.41293333Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-30T13:44:21.484 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.415333542Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-30T13:44:21.484 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.417765832Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-30T13:44:21.484 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.420149897Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-30T13:44:21.485 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.433059018Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-30T13:44:21.485 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.435558544Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-30T13:44:21.485 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.437863138Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-30T13:44:21.485 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.442087276Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-30T13:44:21.485 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.442437004Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-30T13:44:21.485 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.443635568Z level=info msg="Executing migration" id="create folder table" 2024-03-30T13:44:21.485 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.444423478Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-30T13:44:21.485 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.445436554Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-30T13:44:21.485 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.445928284Z level=info msg="Executing migration" id="Update folder title length" 2024-03-30T13:44:21.485 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.446976774Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-30T13:44:21.485 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=migrator t=2024-03-30T13:44:21.447711941Z level=info msg="migrations completed" performed=485 skipped=0 duration=697.932478ms 2024-03-30T13:44:21.485 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=sqlstore t=2024-03-30T13:44:21.448152127Z level=info msg="Created default organization" 2024-03-30T13:44:21.561 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-03-30T13:44:19.280902+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1020,"num_read_kb":866,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":614,"ondisk_log_size":614,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":242108,"kb_used_data":9340,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749719108,"statfs":{"total":767960285184,"available":767712366592,"internally_reserved":0,"allocated":9564160,"data_stored":5491402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002198"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301158+0000","last_change":"2024-03-30T13:42:51.330132+0000","last_active":"2024-03-30T13:44:12.301158+0000","last_peered":"2024-03-30T13:44:12.301158+0000","last_clean":"2024-03-30T13:44:12.301158+0000","last_became_active":"2024-03-30T13:42:51.329797+0000","last_became_peered":"2024-03-30T13:42:51.329797+0000","last_unstale":"2024-03-30T13:44:12.301158+0000","last_undegraded":"2024-03-30T13:44:12.301158+0000","last_fullsized":"2024-03-30T13:44:12.301158+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.296683+0000","last_change":"2024-03-30T13:44:04.797677+0000","last_active":"2024-03-30T13:44:12.296683+0000","last_peered":"2024-03-30T13:44:12.296683+0000","last_clean":"2024-03-30T13:44:12.296683+0000","last_became_active":"2024-03-30T13:42:49.323268+0000","last_became_peered":"2024-03-30T13:42:49.323268+0000","last_unstale":"2024-03-30T13:44:12.296683+0000","last_undegraded":"2024-03-30T13:44:12.296683+0000","last_fullsized":"2024-03-30T13:44:12.296683+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:04.797592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:04.797592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:20.695069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"65'1","reported_seq":70,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297582+0000","last_change":"2024-03-30T13:42:45.314709+0000","last_active":"2024-03-30T13:44:12.297582+0000","last_peered":"2024-03-30T13:44:12.297582+0000","last_clean":"2024-03-30T13:44:12.297582+0000","last_became_active":"2024-03-30T13:42:45.313936+0000","last_became_peered":"2024-03-30T13:42:45.313936+0000","last_unstale":"2024-03-30T13:44:12.297582+0000","last_undegraded":"2024-03-30T13:44:12.297582+0000","last_fullsized":"2024-03-30T13:44:12.297582+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":79,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301002+0000","last_change":"2024-03-30T13:42:47.320128+0000","last_active":"2024-03-30T13:44:12.301002+0000","last_peered":"2024-03-30T13:44:12.301002+0000","last_clean":"2024-03-30T13:44:12.301002+0000","last_became_active":"2024-03-30T13:42:47.319638+0000","last_became_peered":"2024-03-30T13:42:47.319638+0000","last_unstale":"2024-03-30T13:44:12.301002+0000","last_undegraded":"2024-03-30T13:44:12.301002+0000","last_fullsized":"2024-03-30T13:44:12.301002+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433693+0000","last_change":"2024-03-30T13:42:51.374776+0000","last_active":"2024-03-30T13:44:12.433693+0000","last_peered":"2024-03-30T13:44:12.433693+0000","last_clean":"2024-03-30T13:44:12.433693+0000","last_became_active":"2024-03-30T13:42:51.374403+0000","last_became_peered":"2024-03-30T13:42:51.374403+0000","last_unstale":"2024-03-30T13:44:12.433693+0000","last_undegraded":"2024-03-30T13:44:12.433693+0000","last_fullsized":"2024-03-30T13:44:12.433693+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:13.438854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301117+0000","last_change":"2024-03-30T13:42:49.327437+0000","last_active":"2024-03-30T13:44:12.301117+0000","last_peered":"2024-03-30T13:44:12.301117+0000","last_clean":"2024-03-30T13:44:12.301117+0000","last_became_active":"2024-03-30T13:42:49.327124+0000","last_became_peered":"2024-03-30T13:42:49.327124+0000","last_unstale":"2024-03-30T13:44:12.301117+0000","last_undegraded":"2024-03-30T13:44:12.301117+0000","last_fullsized":"2024-03-30T13:44:12.301117+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:17.902978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":74,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.434120+0000","last_change":"2024-03-30T13:44:04.008976+0000","last_active":"2024-03-30T13:44:12.434120+0000","last_peered":"2024-03-30T13:44:12.434120+0000","last_clean":"2024-03-30T13:44:12.434120+0000","last_became_active":"2024-03-30T13:42:45.310771+0000","last_became_peered":"2024-03-30T13:42:45.310771+0000","last_unstale":"2024-03-30T13:44:12.434120+0000","last_undegraded":"2024-03-30T13:44:12.434120+0000","last_fullsized":"2024-03-30T13:44:12.434120+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:04.008875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:04.008875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:26.922817+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"72'15","reported_seq":87,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433501+0000","last_change":"2024-03-30T13:42:47.317438+0000","last_active":"2024-03-30T13:44:12.433501+0000","last_peered":"2024-03-30T13:44:12.433501+0000","last_clean":"2024-03-30T13:44:12.433501+0000","last_became_active":"2024-03-30T13:42:47.317102+0000","last_became_peered":"2024-03-30T13:42:47.317102+0000","last_unstale":"2024-03-30T13:44:12.433501+0000","last_undegraded":"2024-03-30T13:44:12.433501+0000","last_fullsized":"2024-03-30T13:44:12.433501+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.296929+0000","last_change":"2024-03-30T13:42:51.331563+0000","last_active":"2024-03-30T13:44:12.296929+0000","last_peered":"2024-03-30T13:44:12.296929+0000","last_clean":"2024-03-30T13:44:12.296929+0000","last_became_active":"2024-03-30T13:42:51.331196+0000","last_became_peered":"2024-03-30T13:42:51.331196+0000","last_unstale":"2024-03-30T13:44:12.296929+0000","last_undegraded":"2024-03-30T13:44:12.296929+0000","last_fullsized":"2024-03-30T13:44:12.296929+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.431941+0000","last_change":"2024-03-30T13:43:59.262080+0000","last_active":"2024-03-30T13:44:12.431941+0000","last_peered":"2024-03-30T13:44:12.431941+0000","last_clean":"2024-03-30T13:44:12.431941+0000","last_became_active":"2024-03-30T13:42:49.321811+0000","last_became_peered":"2024-03-30T13:42:49.321811+0000","last_unstale":"2024-03-30T13:44:12.431941+0000","last_undegraded":"2024-03-30T13:44:12.431941+0000","last_fullsized":"2024-03-30T13:44:12.431941+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:59.261981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:59.261981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:20.497993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.429347+0000","last_change":"2024-03-30T13:42:45.320122+0000","last_active":"2024-03-30T13:44:12.429347+0000","last_peered":"2024-03-30T13:44:12.429347+0000","last_clean":"2024-03-30T13:44:12.429347+0000","last_became_active":"2024-03-30T13:42:45.319827+0000","last_became_peered":"2024-03-30T13:42:45.319827+0000","last_unstale":"2024-03-30T13:44:12.429347+0000","last_undegraded":"2024-03-30T13:44:12.429347+0000","last_fullsized":"2024-03-30T13:44:12.429347+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":87,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:13.831914+0000","last_change":"2024-03-30T13:44:13.831942+0000","last_active":"2024-03-30T13:44:13.831914+0000","last_peered":"2024-03-30T13:44:13.831914+0000","last_clean":"2024-03-30T13:44:13.831914+0000","last_became_active":"2024-03-30T13:42:47.310289+0000","last_became_peered":"2024-03-30T13:42:47.310289+0000","last_unstale":"2024-03-30T13:44:13.831914+0000","last_undegraded":"2024-03-30T13:44:13.831914+0000","last_fullsized":"2024-03-30T13:44:13.831914+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:44:13.831850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:13.831850+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:37.858895+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.428830+0000","last_change":"2024-03-30T13:42:51.326065+0000","last_active":"2024-03-30T13:44:12.428830+0000","last_peered":"2024-03-30T13:44:12.428830+0000","last_clean":"2024-03-30T13:44:12.428830+0000","last_became_active":"2024-03-30T13:42:51.325748+0000","last_became_peered":"2024-03-30T13:42:51.325748+0000","last_unstale":"2024-03-30T13:44:12.428830+0000","last_undegraded":"2024-03-30T13:44:12.428830+0000","last_fullsized":"2024-03-30T13:44:12.428830+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":56,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516276+0000","last_change":"2024-03-30T13:42:49.326311+0000","last_active":"2024-03-30T13:44:12.516276+0000","last_peered":"2024-03-30T13:44:12.516276+0000","last_clean":"2024-03-30T13:44:12.516276+0000","last_became_active":"2024-03-30T13:42:49.326027+0000","last_became_peered":"2024-03-30T13:42:49.326027+0000","last_unstale":"2024-03-30T13:44:12.516276+0000","last_undegraded":"2024-03-30T13:44:12.516276+0000","last_fullsized":"2024-03-30T13:44:12.516276+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:17.535137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"65'1","reported_seq":79,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.429662+0000","last_change":"2024-03-30T13:44:04.348591+0000","last_active":"2024-03-30T13:44:12.429662+0000","last_peered":"2024-03-30T13:44:12.429662+0000","last_clean":"2024-03-30T13:44:12.429662+0000","last_became_active":"2024-03-30T13:42:45.319975+0000","last_became_peered":"2024-03-30T13:42:45.319975+0000","last_unstale":"2024-03-30T13:44:12.429662+0000","last_undegraded":"2024-03-30T13:44:12.429662+0000","last_fullsized":"2024-03-30T13:44:12.429662+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:44:04.348486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:04.348486+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:34.284640+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":83,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301518+0000","last_change":"2024-03-30T13:43:50.961052+0000","last_active":"2024-03-30T13:44:12.301518+0000","last_peered":"2024-03-30T13:44:12.301518+0000","last_clean":"2024-03-30T13:44:12.301518+0000","last_became_active":"2024-03-30T13:42:47.317352+0000","last_became_peered":"2024-03-30T13:42:47.317352+0000","last_unstale":"2024-03-30T13:44:12.301518+0000","last_undegraded":"2024-03-30T13:44:12.301518+0000","last_fullsized":"2024-03-30T13:44:12.301518+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-30T13:43:50.960980+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-30T13:43:50.960980+0000","last_clean_scrub_stamp":"2024-03-30T13:43:50.960980+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:15.585035+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301970+0000","last_change":"2024-03-30T13:42:51.372350+0000","last_active":"2024-03-30T13:44:12.301970+0000","last_peered":"2024-03-30T13:44:12.301970+0000","last_clean":"2024-03-30T13:44:12.301970+0000","last_became_active":"2024-03-30T13:42:51.372011+0000","last_became_peered":"2024-03-30T13:42:51.372011+0000","last_unstale":"2024-03-30T13:44:12.301970+0000","last_undegraded":"2024-03-30T13:44:12.301970+0000","last_fullsized":"2024-03-30T13:44:12.301970+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.298523+0000","last_change":"2024-03-30T13:44:08.821913+0000","last_active":"2024-03-30T13:44:12.298523+0000","last_peered":"2024-03-30T13:44:12.298523+0000","last_clean":"2024-03-30T13:44:12.298523+0000","last_became_active":"2024-03-30T13:42:49.322665+0000","last_became_peered":"2024-03-30T13:42:49.322665+0000","last_unstale":"2024-03-30T13:44:12.298523+0000","last_undegraded":"2024-03-30T13:44:12.298523+0000","last_fullsized":"2024-03-30T13:44:12.298523+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:08.821811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:08.821811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:10.515727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297849+0000","last_change":"2024-03-30T13:42:45.317711+0000","last_active":"2024-03-30T13:44:12.297849+0000","last_peered":"2024-03-30T13:44:12.297849+0000","last_clean":"2024-03-30T13:44:12.297849+0000","last_became_active":"2024-03-30T13:42:45.317378+0000","last_became_peered":"2024-03-30T13:42:45.317378+0000","last_unstale":"2024-03-30T13:44:12.297849+0000","last_undegraded":"2024-03-30T13:44:12.297849+0000","last_fullsized":"2024-03-30T13:44:12.297849+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:14.011698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":96,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.515927+0000","last_change":"2024-03-30T13:44:05.870649+0000","last_active":"2024-03-30T13:44:12.515927+0000","last_peered":"2024-03-30T13:44:12.515927+0000","last_clean":"2024-03-30T13:44:12.515927+0000","last_became_active":"2024-03-30T13:42:47.320168+0000","last_became_peered":"2024-03-30T13:42:47.320168+0000","last_unstale":"2024-03-30T13:44:12.515927+0000","last_undegraded":"2024-03-30T13:44:12.515927+0000","last_fullsized":"2024-03-30T13:44:12.515927+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-30T13:44:05.870550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:05.870550+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:10.637627+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.434088+0000","last_change":"2024-03-30T13:42:51.371902+0000","last_active":"2024-03-30T13:44:12.434088+0000","last_peered":"2024-03-30T13:44:12.434088+0000","last_clean":"2024-03-30T13:44:12.434088+0000","last_became_active":"2024-03-30T13:42:51.371378+0000","last_became_peered":"2024-03-30T13:42:51.371378+0000","last_unstale":"2024-03-30T13:44:12.434088+0000","last_undegraded":"2024-03-30T13:44:12.434088+0000","last_fullsized":"2024-03-30T13:44:12.434088+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:20.136855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301714+0000","last_change":"2024-03-30T13:42:49.323001+0000","last_active":"2024-03-30T13:44:12.301714+0000","last_peered":"2024-03-30T13:44:12.301714+0000","last_clean":"2024-03-30T13:44:12.301714+0000","last_became_active":"2024-03-30T13:42:49.322561+0000","last_became_peered":"2024-03-30T13:42:49.322561+0000","last_unstale":"2024-03-30T13:44:12.301714+0000","last_undegraded":"2024-03-30T13:44:12.301714+0000","last_fullsized":"2024-03-30T13:44:12.301714+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301525+0000","last_change":"2024-03-30T13:42:45.308820+0000","last_active":"2024-03-30T13:44:12.301525+0000","last_peered":"2024-03-30T13:44:12.301525+0000","last_clean":"2024-03-30T13:44:12.301525+0000","last_became_active":"2024-03-30T13:42:45.308468+0000","last_became_peered":"2024-03-30T13:42:45.308468+0000","last_unstale":"2024-03-30T13:44:12.301525+0000","last_undegraded":"2024-03-30T13:44:12.301525+0000","last_fullsized":"2024-03-30T13:44:12.301525+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"88'17","reported_seq":98,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433874+0000","last_change":"2024-03-30T13:43:52.044529+0000","last_active":"2024-03-30T13:44:12.433874+0000","last_peered":"2024-03-30T13:44:12.433874+0000","last_clean":"2024-03-30T13:44:12.433874+0000","last_became_active":"2024-03-30T13:42:47.317881+0000","last_became_peered":"2024-03-30T13:42:47.317881+0000","last_unstale":"2024-03-30T13:44:12.433874+0000","last_undegraded":"2024-03-30T13:44:12.433874+0000","last_fullsized":"2024-03-30T13:44:12.433874+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'17","last_scrub_stamp":"2024-03-30T13:43:52.044427+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-03-30T13:43:52.044427+0000","last_clean_scrub_stamp":"2024-03-30T13:43:52.044427+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:11.077720+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:14.833893+0000","last_change":"2024-03-30T13:44:14.833942+0000","last_active":"2024-03-30T13:44:14.833893+0000","last_peered":"2024-03-30T13:44:14.833893+0000","last_clean":"2024-03-30T13:44:14.833893+0000","last_became_active":"2024-03-30T13:42:51.327749+0000","last_became_peered":"2024-03-30T13:42:51.327749+0000","last_unstale":"2024-03-30T13:44:14.833893+0000","last_undegraded":"2024-03-30T13:44:14.833893+0000","last_fullsized":"2024-03-30T13:44:14.833893+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T13:44:14.833803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:14.833803+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:27.114310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":94,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.517062+0000","last_change":"2024-03-30T13:44:10.804691+0000","last_active":"2024-03-30T13:44:12.517062+0000","last_peered":"2024-03-30T13:44:12.517062+0000","last_clean":"2024-03-30T13:44:12.517062+0000","last_became_active":"2024-03-30T13:42:49.323362+0000","last_became_peered":"2024-03-30T13:42:49.323362+0000","last_unstale":"2024-03-30T13:44:12.517062+0000","last_undegraded":"2024-03-30T13:44:12.517062+0000","last_fullsized":"2024-03-30T13:44:12.517062+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:44:10.804550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:10.804550+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:38.462339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"65'1","reported_seq":75,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.296777+0000","last_change":"2024-03-30T13:43:47.807153+0000","last_active":"2024-03-30T13:44:12.296777+0000","last_peered":"2024-03-30T13:44:12.296777+0000","last_clean":"2024-03-30T13:44:12.296777+0000","last_became_active":"2024-03-30T13:42:45.307832+0000","last_became_peered":"2024-03-30T13:42:45.307832+0000","last_unstale":"2024-03-30T13:44:12.296777+0000","last_undegraded":"2024-03-30T13:44:12.296777+0000","last_fullsized":"2024-03-30T13:44:12.296777+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:43:47.807055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:43:47.807055+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:10.931779+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":83,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:15.356830+0000","last_change":"2024-03-30T13:44:15.356873+0000","last_active":"2024-03-30T13:44:15.356830+0000","last_peered":"2024-03-30T13:44:15.356830+0000","last_clean":"2024-03-30T13:44:15.356830+0000","last_became_active":"2024-03-30T13:42:47.500763+0000","last_became_peered":"2024-03-30T13:42:47.500763+0000","last_unstale":"2024-03-30T13:44:15.356830+0000","last_undegraded":"2024-03-30T13:44:15.356830+0000","last_fullsized":"2024-03-30T13:44:15.356830+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-30T13:44:15.356762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:15.356762+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:42.015410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.298277+0000","last_change":"2024-03-30T13:42:51.325851+0000","last_active":"2024-03-30T13:44:12.298277+0000","last_peered":"2024-03-30T13:44:12.298277+0000","last_clean":"2024-03-30T13:44:12.298277+0000","last_became_active":"2024-03-30T13:42:51.325513+0000","last_became_peered":"2024-03-30T13:42:51.325513+0000","last_unstale":"2024-03-30T13:44:12.298277+0000","last_undegraded":"2024-03-30T13:44:12.298277+0000","last_fullsized":"2024-03-30T13:44:12.298277+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.298372+0000","last_change":"2024-03-30T13:44:02.408701+0000","last_active":"2024-03-30T13:44:12.298372+0000","last_peered":"2024-03-30T13:44:12.298372+0000","last_clean":"2024-03-30T13:44:12.298372+0000","last_became_active":"2024-03-30T13:42:49.330335+0000","last_became_peered":"2024-03-30T13:42:49.330335+0000","last_unstale":"2024-03-30T13:44:12.298372+0000","last_undegraded":"2024-03-30T13:44:12.298372+0000","last_fullsized":"2024-03-30T13:44:12.298372+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:02.408612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:02.408612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:23.673286+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433246+0000","last_change":"2024-03-30T13:42:45.310476+0000","last_active":"2024-03-30T13:44:12.433246+0000","last_peered":"2024-03-30T13:44:12.433246+0000","last_clean":"2024-03-30T13:44:12.433246+0000","last_became_active":"2024-03-30T13:42:45.310165+0000","last_became_peered":"2024-03-30T13:42:45.310165+0000","last_unstale":"2024-03-30T13:44:12.433246+0000","last_undegraded":"2024-03-30T13:44:12.433246+0000","last_fullsized":"2024-03-30T13:44:12.433246+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":87,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516927+0000","last_change":"2024-03-30T13:43:54.758046+0000","last_active":"2024-03-30T13:44:12.516927+0000","last_peered":"2024-03-30T13:44:12.516927+0000","last_clean":"2024-03-30T13:44:12.516927+0000","last_became_active":"2024-03-30T13:42:47.324399+0000","last_became_peered":"2024-03-30T13:42:47.324399+0000","last_unstale":"2024-03-30T13:44:12.516927+0000","last_undegraded":"2024-03-30T13:44:12.516927+0000","last_fullsized":"2024-03-30T13:44:12.516927+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:43:54.757869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:54.757869+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:15.753990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":60,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516169+0000","last_change":"2024-03-30T13:43:59.682402+0000","last_active":"2024-03-30T13:44:12.516169+0000","last_peered":"2024-03-30T13:44:12.516169+0000","last_clean":"2024-03-30T13:44:12.516169+0000","last_became_active":"2024-03-30T13:42:51.327000+0000","last_became_peered":"2024-03-30T13:42:51.327000+0000","last_unstale":"2024-03-30T13:44:12.516169+0000","last_undegraded":"2024-03-30T13:44:12.516169+0000","last_fullsized":"2024-03-30T13:44:12.516169+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:59.682298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:43:59.682298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:25.906592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516174+0000","last_change":"2024-03-30T13:43:53.750352+0000","last_active":"2024-03-30T13:44:12.516174+0000","last_peered":"2024-03-30T13:44:12.516174+0000","last_clean":"2024-03-30T13:44:12.516174+0000","last_became_active":"2024-03-30T13:42:49.322799+0000","last_became_peered":"2024-03-30T13:42:49.322799+0000","last_unstale":"2024-03-30T13:44:12.516174+0000","last_undegraded":"2024-03-30T13:44:12.516174+0000","last_fullsized":"2024-03-30T13:44:12.516174+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:53.750254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:53.750254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:59.399300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297487+0000","last_change":"2024-03-30T13:43:58.386027+0000","last_active":"2024-03-30T13:44:12.297487+0000","last_peered":"2024-03-30T13:44:12.297487+0000","last_clean":"2024-03-30T13:44:12.297487+0000","last_became_active":"2024-03-30T13:42:45.314302+0000","last_became_peered":"2024-03-30T13:42:45.314302+0000","last_unstale":"2024-03-30T13:44:12.297487+0000","last_undegraded":"2024-03-30T13:44:12.297487+0000","last_fullsized":"2024-03-30T13:44:12.297487+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:58.385891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:43:58.385891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:03.698225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":76,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433247+0000","last_change":"2024-03-30T13:43:55.894589+0000","last_active":"2024-03-30T13:44:12.433247+0000","last_peered":"2024-03-30T13:44:12.433247+0000","last_clean":"2024-03-30T13:44:12.433247+0000","last_became_active":"2024-03-30T13:42:47.321428+0000","last_became_peered":"2024-03-30T13:42:47.321428+0000","last_unstale":"2024-03-30T13:44:12.433247+0000","last_undegraded":"2024-03-30T13:44:12.433247+0000","last_fullsized":"2024-03-30T13:44:12.433247+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-30T13:43:55.894502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:55.894502+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:56.441104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433493+0000","last_change":"2024-03-30T13:42:51.374755+0000","last_active":"2024-03-30T13:44:12.433493+0000","last_peered":"2024-03-30T13:44:12.433493+0000","last_clean":"2024-03-30T13:44:12.433493+0000","last_became_active":"2024-03-30T13:42:51.374304+0000","last_became_peered":"2024-03-30T13:42:51.374304+0000","last_unstale":"2024-03-30T13:44:12.433493+0000","last_undegraded":"2024-03-30T13:44:12.433493+0000","last_fullsized":"2024-03-30T13:44:12.433493+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:18.990969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.517441+0000","last_change":"2024-03-30T13:44:07.744732+0000","last_active":"2024-03-30T13:44:12.517441+0000","last_peered":"2024-03-30T13:44:12.517441+0000","last_clean":"2024-03-30T13:44:12.517441+0000","last_became_active":"2024-03-30T13:42:49.324206+0000","last_became_peered":"2024-03-30T13:42:49.324206+0000","last_unstale":"2024-03-30T13:44:12.517441+0000","last_undegraded":"2024-03-30T13:44:12.517441+0000","last_fullsized":"2024-03-30T13:44:12.517441+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:07.744620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:07.744620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:33.978349+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.429523+0000","last_change":"2024-03-30T13:42:45.319833+0000","last_active":"2024-03-30T13:44:12.429523+0000","last_peered":"2024-03-30T13:44:12.429523+0000","last_clean":"2024-03-30T13:44:12.429523+0000","last_became_active":"2024-03-30T13:42:45.319482+0000","last_became_peered":"2024-03-30T13:42:45.319482+0000","last_unstale":"2024-03-30T13:44:12.429523+0000","last_undegraded":"2024-03-30T13:44:12.429523+0000","last_fullsized":"2024-03-30T13:44:12.429523+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":89,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433574+0000","last_change":"2024-03-30T13:43:54.926339+0000","last_active":"2024-03-30T13:44:12.433574+0000","last_peered":"2024-03-30T13:44:12.433574+0000","last_clean":"2024-03-30T13:44:12.433574+0000","last_became_active":"2024-03-30T13:42:47.500712+0000","last_became_peered":"2024-03-30T13:42:47.500712+0000","last_unstale":"2024-03-30T13:44:12.433574+0000","last_undegraded":"2024-03-30T13:44:12.433574+0000","last_fullsized":"2024-03-30T13:44:12.433574+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:43:54.926242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:54.926242+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:59.626117+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.296661+0000","last_change":"2024-03-30T13:42:51.334062+0000","last_active":"2024-03-30T13:44:12.296661+0000","last_peered":"2024-03-30T13:44:12.296661+0000","last_clean":"2024-03-30T13:44:12.296661+0000","last_became_active":"2024-03-30T13:42:51.333603+0000","last_became_peered":"2024-03-30T13:42:51.333603+0000","last_unstale":"2024-03-30T13:44:12.296661+0000","last_undegraded":"2024-03-30T13:44:12.296661+0000","last_fullsized":"2024-03-30T13:44:12.296661+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433890+0000","last_change":"2024-03-30T13:42:49.325524+0000","last_active":"2024-03-30T13:44:12.433890+0000","last_peered":"2024-03-30T13:44:12.433890+0000","last_clean":"2024-03-30T13:44:12.433890+0000","last_became_active":"2024-03-30T13:42:49.325219+0000","last_became_peered":"2024-03-30T13:42:49.325219+0000","last_unstale":"2024-03-30T13:44:12.433890+0000","last_undegraded":"2024-03-30T13:44:12.433890+0000","last_fullsized":"2024-03-30T13:44:12.433890+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.517441+0000","last_change":"2024-03-30T13:43:52.764537+0000","last_active":"2024-03-30T13:44:12.517441+0000","last_peered":"2024-03-30T13:44:12.517441+0000","last_clean":"2024-03-30T13:44:12.517441+0000","last_became_active":"2024-03-30T13:42:45.311071+0000","last_became_peered":"2024-03-30T13:42:45.311071+0000","last_unstale":"2024-03-30T13:44:12.517441+0000","last_undegraded":"2024-03-30T13:44:12.517441+0000","last_fullsized":"2024-03-30T13:44:12.517441+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:52.764446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:52.764446+0000","last_clean_scrub_stamp":"2024-03-30T13:43:52.764446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:14.951666+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":86,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297305+0000","last_change":"2024-03-30T13:43:53.806905+0000","last_active":"2024-03-30T13:44:12.297305+0000","last_peered":"2024-03-30T13:44:12.297305+0000","last_clean":"2024-03-30T13:44:12.297305+0000","last_became_active":"2024-03-30T13:42:47.324421+0000","last_became_peered":"2024-03-30T13:42:47.324421+0000","last_unstale":"2024-03-30T13:44:12.297305+0000","last_undegraded":"2024-03-30T13:44:12.297305+0000","last_fullsized":"2024-03-30T13:44:12.297305+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:43:53.806768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:53.806768+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:11.831276+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":62,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433247+0000","last_change":"2024-03-30T13:44:09.040324+0000","last_active":"2024-03-30T13:44:12.433247+0000","last_peered":"2024-03-30T13:44:12.433247+0000","last_clean":"2024-03-30T13:44:12.433247+0000","last_became_active":"2024-03-30T13:42:51.334597+0000","last_became_peered":"2024-03-30T13:42:51.334597+0000","last_unstale":"2024-03-30T13:44:12.433247+0000","last_undegraded":"2024-03-30T13:44:12.433247+0000","last_fullsized":"2024-03-30T13:44:12.433247+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:09.040207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:09.040207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:25.970222+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297128+0000","last_change":"2024-03-30T13:42:49.323180+0000","last_active":"2024-03-30T13:44:12.297128+0000","last_peered":"2024-03-30T13:44:12.297128+0000","last_clean":"2024-03-30T13:44:12.297128+0000","last_became_active":"2024-03-30T13:42:49.322843+0000","last_became_peered":"2024-03-30T13:42:49.322843+0000","last_unstale":"2024-03-30T13:44:12.297128+0000","last_undegraded":"2024-03-30T13:44:12.297128+0000","last_fullsized":"2024-03-30T13:44:12.297128+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:12.351693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301890+0000","last_change":"2024-03-30T13:42:45.316182+0000","last_active":"2024-03-30T13:44:12.301890+0000","last_peered":"2024-03-30T13:44:12.301890+0000","last_clean":"2024-03-30T13:44:12.301890+0000","last_became_active":"2024-03-30T13:42:45.315864+0000","last_became_peered":"2024-03-30T13:42:45.315864+0000","last_unstale":"2024-03-30T13:44:12.301890+0000","last_undegraded":"2024-03-30T13:44:12.301890+0000","last_fullsized":"2024-03-30T13:44:12.301890+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":108,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301639+0000","last_change":"2024-03-30T13:43:54.985253+0000","last_active":"2024-03-30T13:44:12.301639+0000","last_peered":"2024-03-30T13:44:12.301639+0000","last_clean":"2024-03-30T13:44:12.301639+0000","last_became_active":"2024-03-30T13:43:12.037286+0000","last_became_peered":"2024-03-30T13:43:12.037286+0000","last_unstale":"2024-03-30T13:44:12.301639+0000","last_undegraded":"2024-03-30T13:44:12.301639+0000","last_fullsized":"2024-03-30T13:44:12.301639+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:43:54.985130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:54.985130+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:00.861139+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":60,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:16.339060+0000","last_change":"2024-03-30T13:44:16.339082+0000","last_active":"2024-03-30T13:44:16.339060+0000","last_peered":"2024-03-30T13:44:16.339060+0000","last_clean":"2024-03-30T13:44:16.339060+0000","last_became_active":"2024-03-30T13:42:51.336162+0000","last_became_peered":"2024-03-30T13:42:51.336162+0000","last_unstale":"2024-03-30T13:44:16.339060+0000","last_undegraded":"2024-03-30T13:44:16.339060+0000","last_fullsized":"2024-03-30T13:44:16.339060+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:16.339001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:16.339001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:45.272290+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":84,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.432211+0000","last_change":"2024-03-30T13:42:49.321501+0000","last_active":"2024-03-30T13:44:12.432211+0000","last_peered":"2024-03-30T13:44:12.432211+0000","last_clean":"2024-03-30T13:44:12.432211+0000","last_became_active":"2024-03-30T13:42:49.321203+0000","last_became_peered":"2024-03-30T13:42:49.321203+0000","last_unstale":"2024-03-30T13:44:12.432211+0000","last_undegraded":"2024-03-30T13:44:12.432211+0000","last_fullsized":"2024-03-30T13:44:12.432211+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.517018+0000","last_change":"2024-03-30T13:42:45.318320+0000","last_active":"2024-03-30T13:44:12.517018+0000","last_peered":"2024-03-30T13:44:12.517018+0000","last_clean":"2024-03-30T13:44:12.517018+0000","last_became_active":"2024-03-30T13:42:45.318016+0000","last_became_peered":"2024-03-30T13:42:45.318016+0000","last_unstale":"2024-03-30T13:44:12.517018+0000","last_undegraded":"2024-03-30T13:44:12.517018+0000","last_fullsized":"2024-03-30T13:44:12.517018+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":95,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433881+0000","last_change":"2024-03-30T13:43:12.043856+0000","last_active":"2024-03-30T13:44:12.433881+0000","last_peered":"2024-03-30T13:44:12.433881+0000","last_clean":"2024-03-30T13:44:12.433881+0000","last_became_active":"2024-03-30T13:43:12.037240+0000","last_became_peered":"2024-03-30T13:43:12.037240+0000","last_unstale":"2024-03-30T13:44:12.433881+0000","last_undegraded":"2024-03-30T13:44:12.433881+0000","last_fullsized":"2024-03-30T13:44:12.433881+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:15.454436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301984+0000","last_change":"2024-03-30T13:42:51.325383+0000","last_active":"2024-03-30T13:44:12.301984+0000","last_peered":"2024-03-30T13:44:12.301984+0000","last_clean":"2024-03-30T13:44:12.301984+0000","last_became_active":"2024-03-30T13:42:51.325064+0000","last_became_peered":"2024-03-30T13:42:51.325064+0000","last_unstale":"2024-03-30T13:44:12.301984+0000","last_undegraded":"2024-03-30T13:44:12.301984+0000","last_fullsized":"2024-03-30T13:44:12.301984+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"72'8","reported_seq":93,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:14.078896+0000","last_change":"2024-03-30T13:44:14.078896+0000","last_active":"2024-03-30T13:44:14.078896+0000","last_peered":"2024-03-30T13:44:14.078896+0000","last_clean":"2024-03-30T13:44:14.078896+0000","last_became_active":"2024-03-30T13:42:49.328616+0000","last_became_peered":"2024-03-30T13:42:49.328616+0000","last_unstale":"2024-03-30T13:44:14.078896+0000","last_undegraded":"2024-03-30T13:44:14.078896+0000","last_fullsized":"2024-03-30T13:44:14.078896+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:44:14.078795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:14.078795+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:23.519694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297246+0000","last_change":"2024-03-30T13:42:45.315502+0000","last_active":"2024-03-30T13:44:12.297246+0000","last_peered":"2024-03-30T13:44:12.297246+0000","last_clean":"2024-03-30T13:44:12.297246+0000","last_became_active":"2024-03-30T13:42:45.314585+0000","last_became_peered":"2024-03-30T13:42:45.314585+0000","last_unstale":"2024-03-30T13:44:12.297246+0000","last_undegraded":"2024-03-30T13:44:12.297246+0000","last_fullsized":"2024-03-30T13:44:12.297246+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:14.270223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.429620+0000","last_change":"2024-03-30T13:43:53.236751+0000","last_active":"2024-03-30T13:44:12.429620+0000","last_peered":"2024-03-30T13:44:12.429620+0000","last_clean":"2024-03-30T13:44:12.429620+0000","last_became_active":"2024-03-30T13:42:47.501559+0000","last_became_peered":"2024-03-30T13:42:47.501559+0000","last_unstale":"2024-03-30T13:44:12.429620+0000","last_undegraded":"2024-03-30T13:44:12.429620+0000","last_fullsized":"2024-03-30T13:44:12.429620+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:43:53.236633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:53.236633+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:05.517754+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516168+0000","last_change":"2024-03-30T13:43:52.834487+0000","last_active":"2024-03-30T13:44:12.516168+0000","last_peered":"2024-03-30T13:44:12.516168+0000","last_clean":"2024-03-30T13:44:12.516168+0000","last_became_active":"2024-03-30T13:42:51.325522+0000","last_became_peered":"2024-03-30T13:42:51.325522+0000","last_unstale":"2024-03-30T13:44:12.516168+0000","last_undegraded":"2024-03-30T13:44:12.516168+0000","last_fullsized":"2024-03-30T13:44:12.516168+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:52.834381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:43:52.834381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:56.638872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.434210+0000","last_change":"2024-03-30T13:42:49.323616+0000","last_active":"2024-03-30T13:44:12.434210+0000","last_peered":"2024-03-30T13:44:12.434210+0000","last_clean":"2024-03-30T13:44:12.434210+0000","last_became_active":"2024-03-30T13:42:49.323222+0000","last_became_peered":"2024-03-30T13:42:49.323222+0000","last_unstale":"2024-03-30T13:44:12.434210+0000","last_undegraded":"2024-03-30T13:44:12.434210+0000","last_fullsized":"2024-03-30T13:44:12.434210+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:17.439407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516650+0000","last_change":"2024-03-30T13:44:05.783586+0000","last_active":"2024-03-30T13:44:12.516650+0000","last_peered":"2024-03-30T13:44:12.516650+0000","last_clean":"2024-03-30T13:44:12.516650+0000","last_became_active":"2024-03-30T13:42:45.313617+0000","last_became_peered":"2024-03-30T13:42:45.313617+0000","last_unstale":"2024-03-30T13:44:12.516650+0000","last_undegraded":"2024-03-30T13:44:12.516650+0000","last_fullsized":"2024-03-30T13:44:12.516650+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:05.783467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:05.783467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:22.470211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":84,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.298216+0000","last_change":"2024-03-30T13:44:00.374645+0000","last_active":"2024-03-30T13:44:12.298216+0000","last_peered":"2024-03-30T13:44:12.298216+0000","last_clean":"2024-03-30T13:44:12.298216+0000","last_became_active":"2024-03-30T13:42:47.501925+0000","last_became_peered":"2024-03-30T13:42:47.501925+0000","last_unstale":"2024-03-30T13:44:12.298216+0000","last_undegraded":"2024-03-30T13:44:12.298216+0000","last_fullsized":"2024-03-30T13:44:12.298216+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:44:00.374534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:00.374534+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:25.183504+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"72'1","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.517341+0000","last_change":"2024-03-30T13:44:11.831875+0000","last_active":"2024-03-30T13:44:12.517341+0000","last_peered":"2024-03-30T13:44:12.517341+0000","last_clean":"2024-03-30T13:44:12.517341+0000","last_became_active":"2024-03-30T13:42:51.372023+0000","last_became_peered":"2024-03-30T13:42:51.372023+0000","last_unstale":"2024-03-30T13:44:12.517341+0000","last_undegraded":"2024-03-30T13:44:12.517341+0000","last_fullsized":"2024-03-30T13:44:12.517341+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T13:44:11.831787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:11.831787+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:29.823530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.432336+0000","last_change":"2024-03-30T13:42:49.324555+0000","last_active":"2024-03-30T13:44:12.432336+0000","last_peered":"2024-03-30T13:44:12.432336+0000","last_clean":"2024-03-30T13:44:12.432336+0000","last_became_active":"2024-03-30T13:42:49.323924+0000","last_became_peered":"2024-03-30T13:42:49.323924+0000","last_unstale":"2024-03-30T13:44:12.432336+0000","last_undegraded":"2024-03-30T13:44:12.432336+0000","last_fullsized":"2024-03-30T13:44:12.432336+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.515860+0000","last_change":"2024-03-30T13:42:45.319317+0000","last_active":"2024-03-30T13:44:12.515860+0000","last_peered":"2024-03-30T13:44:12.515860+0000","last_clean":"2024-03-30T13:44:12.515860+0000","last_became_active":"2024-03-30T13:42:45.318956+0000","last_became_peered":"2024-03-30T13:42:45.318956+0000","last_unstale":"2024-03-30T13:44:12.515860+0000","last_undegraded":"2024-03-30T13:44:12.515860+0000","last_fullsized":"2024-03-30T13:44:12.515860+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":71,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.434196+0000","last_change":"2024-03-30T13:42:47.502035+0000","last_active":"2024-03-30T13:44:12.434196+0000","last_peered":"2024-03-30T13:44:12.434196+0000","last_clean":"2024-03-30T13:44:12.434196+0000","last_became_active":"2024-03-30T13:42:47.501426+0000","last_became_peered":"2024-03-30T13:42:47.501426+0000","last_unstale":"2024-03-30T13:44:12.434196+0000","last_undegraded":"2024-03-30T13:44:12.434196+0000","last_fullsized":"2024-03-30T13:44:12.434196+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.429119+0000","last_change":"2024-03-30T13:42:51.371670+0000","last_active":"2024-03-30T13:44:12.429119+0000","last_peered":"2024-03-30T13:44:12.429119+0000","last_clean":"2024-03-30T13:44:12.429119+0000","last_became_active":"2024-03-30T13:42:51.371360+0000","last_became_peered":"2024-03-30T13:42:51.371360+0000","last_unstale":"2024-03-30T13:44:12.429119+0000","last_undegraded":"2024-03-30T13:44:12.429119+0000","last_fullsized":"2024-03-30T13:44:12.429119+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":84,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.517366+0000","last_change":"2024-03-30T13:43:45.701210+0000","last_active":"2024-03-30T13:44:12.517366+0000","last_peered":"2024-03-30T13:44:12.517366+0000","last_clean":"2024-03-30T13:44:12.517366+0000","last_became_active":"2024-03-30T13:42:45.318963+0000","last_became_peered":"2024-03-30T13:42:45.318963+0000","last_unstale":"2024-03-30T13:44:12.517366+0000","last_undegraded":"2024-03-30T13:44:12.517366+0000","last_fullsized":"2024-03-30T13:44:12.517366+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-30T13:43:45.701065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:43:45.701065+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:04.819362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":86,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.428751+0000","last_change":"2024-03-30T13:42:47.502290+0000","last_active":"2024-03-30T13:44:12.428751+0000","last_peered":"2024-03-30T13:44:12.428751+0000","last_clean":"2024-03-30T13:44:12.428751+0000","last_became_active":"2024-03-30T13:42:47.501937+0000","last_became_peered":"2024-03-30T13:42:47.501937+0000","last_unstale":"2024-03-30T13:44:12.428751+0000","last_undegraded":"2024-03-30T13:44:12.428751+0000","last_fullsized":"2024-03-30T13:44:12.428751+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"72'8","reported_seq":91,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516878+0000","last_change":"2024-03-30T13:43:56.742250+0000","last_active":"2024-03-30T13:44:12.516878+0000","last_peered":"2024-03-30T13:44:12.516878+0000","last_clean":"2024-03-30T13:44:12.516878+0000","last_became_active":"2024-03-30T13:42:49.322675+0000","last_became_peered":"2024-03-30T13:42:49.322675+0000","last_unstale":"2024-03-30T13:44:12.516878+0000","last_undegraded":"2024-03-30T13:44:12.516878+0000","last_fullsized":"2024-03-30T13:44:12.516878+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:43:56.742129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:56.742129+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:17.131326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433601+0000","last_change":"2024-03-30T13:42:51.335656+0000","last_active":"2024-03-30T13:44:12.433601+0000","last_peered":"2024-03-30T13:44:12.433601+0000","last_clean":"2024-03-30T13:44:12.433601+0000","last_became_active":"2024-03-30T13:42:51.334672+0000","last_became_peered":"2024-03-30T13:42:51.334672+0000","last_unstale":"2024-03-30T13:44:12.433601+0000","last_undegraded":"2024-03-30T13:44:12.433601+0000","last_fullsized":"2024-03-30T13:44:12.433601+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516846+0000","last_change":"2024-03-30T13:44:06.790905+0000","last_active":"2024-03-30T13:44:12.516846+0000","last_peered":"2024-03-30T13:44:12.516846+0000","last_clean":"2024-03-30T13:44:12.516846+0000","last_became_active":"2024-03-30T13:42:45.312328+0000","last_became_peered":"2024-03-30T13:42:45.312328+0000","last_unstale":"2024-03-30T13:44:12.516846+0000","last_undegraded":"2024-03-30T13:44:12.516846+0000","last_fullsized":"2024-03-30T13:44:12.516846+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:06.790755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:06.790755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:10.517615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":94,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301382+0000","last_change":"2024-03-30T13:43:51.992642+0000","last_active":"2024-03-30T13:44:12.301382+0000","last_peered":"2024-03-30T13:44:12.301382+0000","last_clean":"2024-03-30T13:44:12.301382+0000","last_became_active":"2024-03-30T13:42:47.316187+0000","last_became_peered":"2024-03-30T13:42:47.316187+0000","last_unstale":"2024-03-30T13:44:12.301382+0000","last_undegraded":"2024-03-30T13:44:12.301382+0000","last_fullsized":"2024-03-30T13:44:12.301382+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-30T13:43:51.992552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:51.992552+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:09.690586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.515661+0000","last_change":"2024-03-30T13:44:10.825465+0000","last_active":"2024-03-30T13:44:12.515661+0000","last_peered":"2024-03-30T13:44:12.515661+0000","last_clean":"2024-03-30T13:44:12.515661+0000","last_became_active":"2024-03-30T13:42:49.322478+0000","last_became_peered":"2024-03-30T13:42:49.322478+0000","last_unstale":"2024-03-30T13:44:12.515661+0000","last_undegraded":"2024-03-30T13:44:12.515661+0000","last_fullsized":"2024-03-30T13:44:12.515661+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:10.825365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:10.825365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:37.651091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":60,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:13.838232+0000","last_change":"2024-03-30T13:44:13.838271+0000","last_active":"2024-03-30T13:44:13.838232+0000","last_peered":"2024-03-30T13:44:13.838232+0000","last_clean":"2024-03-30T13:44:13.838232+0000","last_became_active":"2024-03-30T13:42:51.333773+0000","last_became_peered":"2024-03-30T13:42:51.333773+0000","last_unstale":"2024-03-30T13:44:13.838232+0000","last_undegraded":"2024-03-30T13:44:13.838232+0000","last_fullsized":"2024-03-30T13:44:13.838232+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:13.838155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:13.838155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:31.168338+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.515911+0000","last_change":"2024-03-30T13:44:01.850315+0000","last_active":"2024-03-30T13:44:12.515911+0000","last_peered":"2024-03-30T13:44:12.515911+0000","last_clean":"2024-03-30T13:44:12.515911+0000","last_became_active":"2024-03-30T13:42:49.320798+0000","last_became_peered":"2024-03-30T13:42:49.320798+0000","last_unstale":"2024-03-30T13:44:12.515911+0000","last_undegraded":"2024-03-30T13:44:12.515911+0000","last_fullsized":"2024-03-30T13:44:12.515911+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:01.850229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:01.850229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:12.718080+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.517604+0000","last_change":"2024-03-30T13:42:45.316023+0000","last_active":"2024-03-30T13:44:12.517604+0000","last_peered":"2024-03-30T13:44:12.517604+0000","last_clean":"2024-03-30T13:44:12.517604+0000","last_became_active":"2024-03-30T13:42:45.315687+0000","last_became_peered":"2024-03-30T13:42:45.315687+0000","last_unstale":"2024-03-30T13:44:12.517604+0000","last_undegraded":"2024-03-30T13:44:12.517604+0000","last_fullsized":"2024-03-30T13:44:12.517604+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"72'19","reported_seq":91,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516763+0000","last_change":"2024-03-30T13:42:47.501349+0000","last_active":"2024-03-30T13:44:12.516763+0000","last_peered":"2024-03-30T13:44:12.516763+0000","last_clean":"2024-03-30T13:44:12.516763+0000","last_became_active":"2024-03-30T13:42:47.500898+0000","last_became_peered":"2024-03-30T13:42:47.500898+0000","last_unstale":"2024-03-30T13:44:12.516763+0000","last_undegraded":"2024-03-30T13:44:12.516763+0000","last_fullsized":"2024-03-30T13:44:12.516763+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.296659+0000","last_change":"2024-03-30T13:42:51.330739+0000","last_active":"2024-03-30T13:44:12.296659+0000","last_peered":"2024-03-30T13:44:12.296659+0000","last_clean":"2024-03-30T13:44:12.296659+0000","last_became_active":"2024-03-30T13:42:51.330403+0000","last_became_peered":"2024-03-30T13:42:51.330403+0000","last_unstale":"2024-03-30T13:44:12.296659+0000","last_undegraded":"2024-03-30T13:44:12.296659+0000","last_fullsized":"2024-03-30T13:44:12.296659+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:19.083830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.515774+0000","last_change":"2024-03-30T13:43:51.840857+0000","last_active":"2024-03-30T13:44:12.515774+0000","last_peered":"2024-03-30T13:44:12.515774+0000","last_clean":"2024-03-30T13:44:12.515774+0000","last_became_active":"2024-03-30T13:42:49.324022+0000","last_became_peered":"2024-03-30T13:42:49.324022+0000","last_unstale":"2024-03-30T13:44:12.515774+0000","last_undegraded":"2024-03-30T13:44:12.515774+0000","last_fullsized":"2024-03-30T13:44:12.515774+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:51.840740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:51.840740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:00.887228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.429762+0000","last_change":"2024-03-30T13:42:45.315058+0000","last_active":"2024-03-30T13:44:12.429762+0000","last_peered":"2024-03-30T13:44:12.429762+0000","last_clean":"2024-03-30T13:44:12.429762+0000","last_became_active":"2024-03-30T13:42:45.314767+0000","last_became_peered":"2024-03-30T13:42:45.314767+0000","last_unstale":"2024-03-30T13:44:12.429762+0000","last_undegraded":"2024-03-30T13:44:12.429762+0000","last_fullsized":"2024-03-30T13:44:12.429762+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":100,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297694+0000","last_change":"2024-03-30T13:43:57.361388+0000","last_active":"2024-03-30T13:44:12.297694+0000","last_peered":"2024-03-30T13:44:12.297694+0000","last_clean":"2024-03-30T13:44:12.297694+0000","last_became_active":"2024-03-30T13:43:11.446379+0000","last_became_peered":"2024-03-30T13:43:11.446379+0000","last_unstale":"2024-03-30T13:44:12.297694+0000","last_undegraded":"2024-03-30T13:44:12.297694+0000","last_fullsized":"2024-03-30T13:44:12.297694+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:43:57.361300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:57.361300+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:13.159418+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":62,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.434014+0000","last_change":"2024-03-30T13:44:06.994178+0000","last_active":"2024-03-30T13:44:12.434014+0000","last_peered":"2024-03-30T13:44:12.434014+0000","last_clean":"2024-03-30T13:44:12.434014+0000","last_became_active":"2024-03-30T13:42:51.331054+0000","last_became_peered":"2024-03-30T13:42:51.331054+0000","last_unstale":"2024-03-30T13:44:12.434014+0000","last_undegraded":"2024-03-30T13:44:12.434014+0000","last_fullsized":"2024-03-30T13:44:12.434014+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:06.994097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:06.994097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:17.075287+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":76,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.428584+0000","last_change":"2024-03-30T13:42:48.314967+0000","last_active":"2024-03-30T13:44:12.428584+0000","last_peered":"2024-03-30T13:44:12.428584+0000","last_clean":"2024-03-30T13:44:12.428584+0000","last_became_active":"2024-03-30T13:42:44.300789+0000","last_became_peered":"2024-03-30T13:42:44.300789+0000","last_unstale":"2024-03-30T13:44:12.428584+0000","last_undegraded":"2024-03-30T13:44:12.428584+0000","last_fullsized":"2024-03-30T13:44:12.428584+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:42:43.281063+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021837800000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301336+0000","last_change":"2024-03-30T13:42:45.310512+0000","last_active":"2024-03-30T13:44:12.301336+0000","last_peered":"2024-03-30T13:44:12.301336+0000","last_clean":"2024-03-30T13:44:12.301336+0000","last_became_active":"2024-03-30T13:42:45.310209+0000","last_became_peered":"2024-03-30T13:42:45.310209+0000","last_unstale":"2024-03-30T13:44:12.301336+0000","last_undegraded":"2024-03-30T13:44:12.301336+0000","last_fullsized":"2024-03-30T13:44:12.301336+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"72'28","reported_seq":129,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297730+0000","last_change":"2024-03-30T13:42:47.324184+0000","last_active":"2024-03-30T13:44:12.297730+0000","last_peered":"2024-03-30T13:44:12.297730+0000","last_clean":"2024-03-30T13:44:12.297730+0000","last_became_active":"2024-03-30T13:42:47.323845+0000","last_became_peered":"2024-03-30T13:42:47.323845+0000","last_unstale":"2024-03-30T13:44:12.297730+0000","last_undegraded":"2024-03-30T13:44:12.297730+0000","last_fullsized":"2024-03-30T13:44:12.297730+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297721+0000","last_change":"2024-03-30T13:43:54.326967+0000","last_active":"2024-03-30T13:44:12.297721+0000","last_peered":"2024-03-30T13:44:12.297721+0000","last_clean":"2024-03-30T13:44:12.297721+0000","last_became_active":"2024-03-30T13:42:49.322154+0000","last_became_peered":"2024-03-30T13:42:49.322154+0000","last_unstale":"2024-03-30T13:44:12.297721+0000","last_undegraded":"2024-03-30T13:44:12.297721+0000","last_fullsized":"2024-03-30T13:44:12.297721+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:54.326829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:54.326829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:09.771943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.517701+0000","last_change":"2024-03-30T13:42:51.372579+0000","last_active":"2024-03-30T13:44:12.517701+0000","last_peered":"2024-03-30T13:44:12.517701+0000","last_clean":"2024-03-30T13:44:12.517701+0000","last_became_active":"2024-03-30T13:42:51.372320+0000","last_became_peered":"2024-03-30T13:42:51.372320+0000","last_unstale":"2024-03-30T13:44:12.517701+0000","last_undegraded":"2024-03-30T13:44:12.517701+0000","last_fullsized":"2024-03-30T13:44:12.517701+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":74,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.296508+0000","last_change":"2024-03-30T13:43:59.810299+0000","last_active":"2024-03-30T13:44:12.296508+0000","last_peered":"2024-03-30T13:44:12.296508+0000","last_clean":"2024-03-30T13:44:12.296508+0000","last_became_active":"2024-03-30T13:42:45.314771+0000","last_became_peered":"2024-03-30T13:42:45.314771+0000","last_unstale":"2024-03-30T13:44:12.296508+0000","last_undegraded":"2024-03-30T13:44:12.296508+0000","last_fullsized":"2024-03-30T13:44:12.296508+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:59.810189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:59.810189+0000","last_clean_scrub_stamp":"2024-03-30T13:43:59.810189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:06.406345+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":84,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.515538+0000","last_change":"2024-03-30T13:43:57.866600+0000","last_active":"2024-03-30T13:44:12.515538+0000","last_peered":"2024-03-30T13:44:12.515538+0000","last_clean":"2024-03-30T13:44:12.515538+0000","last_became_active":"2024-03-30T13:42:44.302344+0000","last_became_peered":"2024-03-30T13:42:44.302344+0000","last_unstale":"2024-03-30T13:44:12.515538+0000","last_undegraded":"2024-03-30T13:44:12.515538+0000","last_fullsized":"2024-03-30T13:44:12.515538+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:57.866519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:43:57.866519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:59.119000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001401518,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"72'13","reported_seq":97,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:14.782493+0000","last_change":"2024-03-30T13:44:14.782524+0000","last_active":"2024-03-30T13:44:14.782493+0000","last_peered":"2024-03-30T13:44:14.782493+0000","last_clean":"2024-03-30T13:44:14.782493+0000","last_became_active":"2024-03-30T13:42:47.320541+0000","last_became_peered":"2024-03-30T13:42:47.320541+0000","last_unstale":"2024-03-30T13:44:14.782493+0000","last_undegraded":"2024-03-30T13:44:14.782493+0000","last_fullsized":"2024-03-30T13:44:14.782493+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'13","last_scrub_stamp":"2024-03-30T13:44:14.782432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:14.782432+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:22.236946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:13.762418+0000","last_change":"2024-03-30T13:44:13.762445+0000","last_active":"2024-03-30T13:44:13.762418+0000","last_peered":"2024-03-30T13:44:13.762418+0000","last_clean":"2024-03-30T13:44:13.762418+0000","last_became_active":"2024-03-30T13:42:49.319826+0000","last_became_peered":"2024-03-30T13:42:49.319826+0000","last_unstale":"2024-03-30T13:44:13.762418+0000","last_undegraded":"2024-03-30T13:44:13.762418+0000","last_fullsized":"2024-03-30T13:44:13.762418+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:13.762345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:13.762345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:42.481193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433250+0000","last_change":"2024-03-30T13:44:12.433273+0000","last_active":"2024-03-30T13:44:12.433250+0000","last_peered":"2024-03-30T13:44:12.433250+0000","last_clean":"2024-03-30T13:44:12.433250+0000","last_became_active":"2024-03-30T13:42:51.331081+0000","last_became_peered":"2024-03-30T13:42:51.331081+0000","last_unstale":"2024-03-30T13:44:12.433250+0000","last_undegraded":"2024-03-30T13:44:12.433250+0000","last_fullsized":"2024-03-30T13:44:12.433250+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:12.433166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:12.433166+0000","last_clean_scrub_stamp":"2024-03-30T13:44:12.433166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:23.843912+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":317,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.517136+0000","last_change":"2024-03-30T13:43:23.836532+0000","last_active":"2024-03-30T13:44:12.517136+0000","last_peered":"2024-03-30T13:44:12.517136+0000","last_clean":"2024-03-30T13:44:12.517136+0000","last_became_active":"2024-03-30T13:42:34.988246+0000","last_became_peered":"2024-03-30T13:42:34.988246+0000","last_unstale":"2024-03-30T13:44:12.517136+0000","last_undegraded":"2024-03-30T13:44:12.517136+0000","last_fullsized":"2024-03-30T13:44:12.517136+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-03-30T13:43:23.836417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:43:23.836417+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:30.177755+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":74,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433566+0000","last_change":"2024-03-30T13:44:09.995041+0000","last_active":"2024-03-30T13:44:12.433566+0000","last_peered":"2024-03-30T13:44:12.433566+0000","last_clean":"2024-03-30T13:44:12.433566+0000","last_became_active":"2024-03-30T13:42:45.309954+0000","last_became_peered":"2024-03-30T13:42:45.309954+0000","last_unstale":"2024-03-30T13:44:12.433566+0000","last_undegraded":"2024-03-30T13:44:12.433566+0000","last_fullsized":"2024-03-30T13:44:12.433566+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:09.994923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:09.994923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:23.612678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":109,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:13.820800+0000","last_change":"2024-03-30T13:44:13.820800+0000","last_active":"2024-03-30T13:44:13.820800+0000","last_peered":"2024-03-30T13:44:13.820800+0000","last_clean":"2024-03-30T13:44:13.820800+0000","last_became_active":"2024-03-30T13:42:47.502338+0000","last_became_peered":"2024-03-30T13:42:47.502338+0000","last_unstale":"2024-03-30T13:44:13.820800+0000","last_undegraded":"2024-03-30T13:44:13.820800+0000","last_fullsized":"2024-03-30T13:44:13.820800+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'16","last_scrub_stamp":"2024-03-30T13:44:13.820689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:13.820689+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:18.638625+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":56,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516436+0000","last_change":"2024-03-30T13:42:49.329181+0000","last_active":"2024-03-30T13:44:12.516436+0000","last_peered":"2024-03-30T13:44:12.516436+0000","last_clean":"2024-03-30T13:44:12.516436+0000","last_became_active":"2024-03-30T13:42:49.328861+0000","last_became_peered":"2024-03-30T13:42:49.328861+0000","last_unstale":"2024-03-30T13:44:12.516436+0000","last_undegraded":"2024-03-30T13:44:12.516436+0000","last_fullsized":"2024-03-30T13:44:12.516436+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297335+0000","last_change":"2024-03-30T13:42:51.335522+0000","last_active":"2024-03-30T13:44:12.297335+0000","last_peered":"2024-03-30T13:44:12.297335+0000","last_clean":"2024-03-30T13:44:12.297335+0000","last_became_active":"2024-03-30T13:42:51.335188+0000","last_became_peered":"2024-03-30T13:42:51.335188+0000","last_unstale":"2024-03-30T13:44:12.297335+0000","last_undegraded":"2024-03-30T13:44:12.297335+0000","last_fullsized":"2024-03-30T13:44:12.297335+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.296890+0000","last_change":"2024-03-30T13:44:08.397886+0000","last_active":"2024-03-30T13:44:12.296890+0000","last_peered":"2024-03-30T13:44:12.296890+0000","last_clean":"2024-03-30T13:44:12.296890+0000","last_became_active":"2024-03-30T13:42:45.307905+0000","last_became_peered":"2024-03-30T13:42:45.307905+0000","last_unstale":"2024-03-30T13:44:12.296890+0000","last_undegraded":"2024-03-30T13:44:12.296890+0000","last_fullsized":"2024-03-30T13:44:12.296890+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:08.397798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:08.397798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:35.437418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":162,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:16.822346+0000","last_change":"2024-03-30T13:42:48.313590+0000","last_active":"2024-03-30T13:44:16.822346+0000","last_peered":"2024-03-30T13:44:16.822346+0000","last_clean":"2024-03-30T13:44:16.822346+0000","last_became_active":"2024-03-30T13:42:44.302343+0000","last_became_peered":"2024-03-30T13:42:44.302343+0000","last_unstale":"2024-03-30T13:44:16.822346+0000","last_undegraded":"2024-03-30T13:44:16.822346+0000","last_fullsized":"2024-03-30T13:44:16.822346+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:42:43.281063+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00126391,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":86,"num_read_kb":81,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":78,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297843+0000","last_change":"2024-03-30T13:42:47.319576+0000","last_active":"2024-03-30T13:44:12.297843+0000","last_peered":"2024-03-30T13:44:12.297843+0000","last_clean":"2024-03-30T13:44:12.297843+0000","last_became_active":"2024-03-30T13:42:47.319155+0000","last_became_peered":"2024-03-30T13:42:47.319155+0000","last_unstale":"2024-03-30T13:44:12.297843+0000","last_undegraded":"2024-03-30T13:44:12.297843+0000","last_fullsized":"2024-03-30T13:44:12.297843+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.430011+0000","last_change":"2024-03-30T13:42:49.322962+0000","last_active":"2024-03-30T13:44:12.430011+0000","last_peered":"2024-03-30T13:44:12.430011+0000","last_clean":"2024-03-30T13:44:12.430011+0000","last_became_active":"2024-03-30T13:42:49.322700+0000","last_became_peered":"2024-03-30T13:42:49.322700+0000","last_unstale":"2024-03-30T13:44:12.430011+0000","last_undegraded":"2024-03-30T13:44:12.430011+0000","last_fullsized":"2024-03-30T13:44:12.430011+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":62,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.296495+0000","last_change":"2024-03-30T13:44:05.809643+0000","last_active":"2024-03-30T13:44:12.296495+0000","last_peered":"2024-03-30T13:44:12.296495+0000","last_clean":"2024-03-30T13:44:12.296495+0000","last_became_active":"2024-03-30T13:42:51.371202+0000","last_became_peered":"2024-03-30T13:42:51.371202+0000","last_unstale":"2024-03-30T13:44:12.296495+0000","last_undegraded":"2024-03-30T13:44:12.296495+0000","last_fullsized":"2024-03-30T13:44:12.296495+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:05.809548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:05.809548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:18.651258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":74,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297582+0000","last_change":"2024-03-30T13:44:06.800362+0000","last_active":"2024-03-30T13:44:12.297582+0000","last_peered":"2024-03-30T13:44:12.297582+0000","last_clean":"2024-03-30T13:44:12.297582+0000","last_became_active":"2024-03-30T13:42:45.310134+0000","last_became_peered":"2024-03-30T13:42:45.310134+0000","last_unstale":"2024-03-30T13:44:12.297582+0000","last_undegraded":"2024-03-30T13:44:12.297582+0000","last_fullsized":"2024-03-30T13:44:12.297582+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:06.800290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:06.800290+0000","last_clean_scrub_stamp":"2024-03-30T13:44:06.800290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:30.870285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":104,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297955+0000","last_change":"2024-03-30T13:44:11.366827+0000","last_active":"2024-03-30T13:44:12.297955+0000","last_peered":"2024-03-30T13:44:12.297955+0000","last_clean":"2024-03-30T13:44:12.297955+0000","last_became_active":"2024-03-30T13:42:47.501272+0000","last_became_peered":"2024-03-30T13:42:47.501272+0000","last_unstale":"2024-03-30T13:44:12.297955+0000","last_undegraded":"2024-03-30T13:44:12.297955+0000","last_fullsized":"2024-03-30T13:44:12.297955+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-30T13:44:11.366726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:11.366726+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:18.908561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:15.794516+0000","last_change":"2024-03-30T13:44:15.794564+0000","last_active":"2024-03-30T13:44:15.794516+0000","last_peered":"2024-03-30T13:44:15.794516+0000","last_clean":"2024-03-30T13:44:15.794516+0000","last_became_active":"2024-03-30T13:42:49.321520+0000","last_became_peered":"2024-03-30T13:42:49.321520+0000","last_unstale":"2024-03-30T13:44:15.794516+0000","last_undegraded":"2024-03-30T13:44:15.794516+0000","last_fullsized":"2024-03-30T13:44:15.794516+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:15.794436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:15.794436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:33.056162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":60,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.296928+0000","last_change":"2024-03-30T13:44:04.440757+0000","last_active":"2024-03-30T13:44:12.296928+0000","last_peered":"2024-03-30T13:44:12.296928+0000","last_clean":"2024-03-30T13:44:12.296928+0000","last_became_active":"2024-03-30T13:42:51.336892+0000","last_became_peered":"2024-03-30T13:42:51.336892+0000","last_unstale":"2024-03-30T13:44:12.296928+0000","last_undegraded":"2024-03-30T13:44:12.296928+0000","last_fullsized":"2024-03-30T13:44:12.296928+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:04.440623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:04.440623+0000","last_clean_scrub_stamp":"2024-03-30T13:44:04.440623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:20.104438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.429473+0000","last_change":"2024-03-30T13:43:57.299723+0000","last_active":"2024-03-30T13:44:12.429473+0000","last_peered":"2024-03-30T13:44:12.429473+0000","last_clean":"2024-03-30T13:44:12.429473+0000","last_became_active":"2024-03-30T13:42:45.317136+0000","last_became_peered":"2024-03-30T13:42:45.317136+0000","last_unstale":"2024-03-30T13:44:12.429473+0000","last_undegraded":"2024-03-30T13:44:12.429473+0000","last_fullsized":"2024-03-30T13:44:12.429473+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:57.299618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:57.299618+0000","last_clean_scrub_stamp":"2024-03-30T13:43:57.299618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:09.120401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.765004+0000","last_change":"2024-03-30T13:44:12.765034+0000","last_active":"2024-03-30T13:44:12.765004+0000","last_peered":"2024-03-30T13:44:12.765004+0000","last_clean":"2024-03-30T13:44:12.765004+0000","last_became_active":"2024-03-30T13:42:47.321433+0000","last_became_peered":"2024-03-30T13:42:47.321433+0000","last_unstale":"2024-03-30T13:44:12.765004+0000","last_undegraded":"2024-03-30T13:44:12.765004+0000","last_fullsized":"2024-03-30T13:44:12.765004+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-30T13:44:12.764944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:12.764944+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:32.186703+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":83,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.298101+0000","last_change":"2024-03-30T13:42:49.326702+0000","last_active":"2024-03-30T13:44:12.298101+0000","last_peered":"2024-03-30T13:44:12.298101+0000","last_clean":"2024-03-30T13:44:12.298101+0000","last_became_active":"2024-03-30T13:42:49.325533+0000","last_became_peered":"2024-03-30T13:42:49.325533+0000","last_unstale":"2024-03-30T13:44:12.298101+0000","last_undegraded":"2024-03-30T13:44:12.298101+0000","last_fullsized":"2024-03-30T13:44:12.298101+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:16.967785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.517235+0000","last_change":"2024-03-30T13:42:51.374719+0000","last_active":"2024-03-30T13:44:12.517235+0000","last_peered":"2024-03-30T13:44:12.517235+0000","last_clean":"2024-03-30T13:44:12.517235+0000","last_became_active":"2024-03-30T13:42:51.374437+0000","last_became_peered":"2024-03-30T13:42:51.374437+0000","last_unstale":"2024-03-30T13:44:12.517235+0000","last_undegraded":"2024-03-30T13:44:12.517235+0000","last_fullsized":"2024-03-30T13:44:12.517235+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297968+0000","last_change":"2024-03-30T13:44:07.397507+0000","last_active":"2024-03-30T13:44:12.297968+0000","last_peered":"2024-03-30T13:44:12.297968+0000","last_clean":"2024-03-30T13:44:12.297968+0000","last_became_active":"2024-03-30T13:42:45.309886+0000","last_became_peered":"2024-03-30T13:42:45.309886+0000","last_unstale":"2024-03-30T13:44:12.297968+0000","last_undegraded":"2024-03-30T13:44:12.297968+0000","last_fullsized":"2024-03-30T13:44:12.297968+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:44:07.397409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:07.397409+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:07.807875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"91'21","reported_seq":99,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301240+0000","last_change":"2024-03-30T13:44:05.917185+0000","last_active":"2024-03-30T13:44:12.301240+0000","last_peered":"2024-03-30T13:44:12.301240+0000","last_clean":"2024-03-30T13:44:12.301240+0000","last_became_active":"2024-03-30T13:42:47.314184+0000","last_became_peered":"2024-03-30T13:42:47.314184+0000","last_unstale":"2024-03-30T13:44:12.301240+0000","last_undegraded":"2024-03-30T13:44:12.301240+0000","last_fullsized":"2024-03-30T13:44:12.301240+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'21","last_scrub_stamp":"2024-03-30T13:44:05.917064+0000","last_deep_scrub":"91'21","last_deep_scrub_stamp":"2024-03-30T13:44:05.917064+0000","last_clean_scrub_stamp":"2024-03-30T13:44:05.917064+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:07.613278+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433631+0000","last_change":"2024-03-30T13:42:49.321877+0000","last_active":"2024-03-30T13:44:12.433631+0000","last_peered":"2024-03-30T13:44:12.433631+0000","last_clean":"2024-03-30T13:44:12.433631+0000","last_became_active":"2024-03-30T13:42:49.321423+0000","last_became_peered":"2024-03-30T13:42:49.321423+0000","last_unstale":"2024-03-30T13:44:12.433631+0000","last_undegraded":"2024-03-30T13:44:12.433631+0000","last_fullsized":"2024-03-30T13:44:12.433631+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301273+0000","last_change":"2024-03-30T13:44:03.968532+0000","last_active":"2024-03-30T13:44:12.301273+0000","last_peered":"2024-03-30T13:44:12.301273+0000","last_clean":"2024-03-30T13:44:12.301273+0000","last_became_active":"2024-03-30T13:42:51.327003+0000","last_became_peered":"2024-03-30T13:42:51.327003+0000","last_unstale":"2024-03-30T13:44:12.301273+0000","last_undegraded":"2024-03-30T13:44:12.301273+0000","last_fullsized":"2024-03-30T13:44:12.301273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:03.968435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:03.968435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:12.556459+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.434157+0000","last_change":"2024-03-30T13:42:45.314830+0000","last_active":"2024-03-30T13:44:12.434157+0000","last_peered":"2024-03-30T13:44:12.434157+0000","last_clean":"2024-03-30T13:44:12.434157+0000","last_became_active":"2024-03-30T13:42:45.314533+0000","last_became_peered":"2024-03-30T13:42:45.314533+0000","last_unstale":"2024-03-30T13:44:12.434157+0000","last_undegraded":"2024-03-30T13:44:12.434157+0000","last_fullsized":"2024-03-30T13:44:12.434157+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":102,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433495+0000","last_change":"2024-03-30T13:44:06.016635+0000","last_active":"2024-03-30T13:44:12.433495+0000","last_peered":"2024-03-30T13:44:12.433495+0000","last_clean":"2024-03-30T13:44:12.433495+0000","last_became_active":"2024-03-30T13:42:47.502708+0000","last_became_peered":"2024-03-30T13:42:47.502708+0000","last_unstale":"2024-03-30T13:44:12.433495+0000","last_undegraded":"2024-03-30T13:44:12.433495+0000","last_fullsized":"2024-03-30T13:44:12.433495+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-03-30T13:44:06.016537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:06.016537+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:09.822245+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301452+0000","last_change":"2024-03-30T13:42:49.324517+0000","last_active":"2024-03-30T13:44:12.301452+0000","last_peered":"2024-03-30T13:44:12.301452+0000","last_clean":"2024-03-30T13:44:12.301452+0000","last_became_active":"2024-03-30T13:42:49.324022+0000","last_became_peered":"2024-03-30T13:42:49.324022+0000","last_unstale":"2024-03-30T13:44:12.301452+0000","last_undegraded":"2024-03-30T13:44:12.301452+0000","last_fullsized":"2024-03-30T13:44:12.301452+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.429244+0000","last_change":"2024-03-30T13:42:51.329317+0000","last_active":"2024-03-30T13:44:12.429244+0000","last_peered":"2024-03-30T13:44:12.429244+0000","last_clean":"2024-03-30T13:44:12.429244+0000","last_became_active":"2024-03-30T13:42:51.329026+0000","last_became_peered":"2024-03-30T13:42:51.329026+0000","last_unstale":"2024-03-30T13:44:12.429244+0000","last_undegraded":"2024-03-30T13:44:12.429244+0000","last_fullsized":"2024-03-30T13:44:12.429244+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":94,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.302106+0000","last_change":"2024-03-30T13:43:58.942488+0000","last_active":"2024-03-30T13:44:12.302106+0000","last_peered":"2024-03-30T13:44:12.302106+0000","last_clean":"2024-03-30T13:44:12.302106+0000","last_became_active":"2024-03-30T13:42:49.321790+0000","last_became_peered":"2024-03-30T13:42:49.321790+0000","last_unstale":"2024-03-30T13:44:12.302106+0000","last_undegraded":"2024-03-30T13:44:12.302106+0000","last_fullsized":"2024-03-30T13:44:12.302106+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:43:58.942311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:58.942311+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:17.003904+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433398+0000","last_change":"2024-03-30T13:42:45.315666+0000","last_active":"2024-03-30T13:44:12.433398+0000","last_peered":"2024-03-30T13:44:12.433398+0000","last_clean":"2024-03-30T13:44:12.433398+0000","last_became_active":"2024-03-30T13:42:45.315319+0000","last_became_peered":"2024-03-30T13:42:45.315319+0000","last_unstale":"2024-03-30T13:44:12.433398+0000","last_undegraded":"2024-03-30T13:44:12.433398+0000","last_fullsized":"2024-03-30T13:44:12.433398+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":87,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.298878+0000","last_change":"2024-03-30T13:42:47.324692+0000","last_active":"2024-03-30T13:44:12.298878+0000","last_peered":"2024-03-30T13:44:12.298878+0000","last_clean":"2024-03-30T13:44:12.298878+0000","last_became_active":"2024-03-30T13:42:47.324350+0000","last_became_peered":"2024-03-30T13:42:47.324350+0000","last_unstale":"2024-03-30T13:44:12.298878+0000","last_undegraded":"2024-03-30T13:44:12.298878+0000","last_fullsized":"2024-03-30T13:44:12.298878+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433753+0000","last_change":"2024-03-30T13:42:51.371742+0000","last_active":"2024-03-30T13:44:12.433753+0000","last_peered":"2024-03-30T13:44:12.433753+0000","last_clean":"2024-03-30T13:44:12.433753+0000","last_became_active":"2024-03-30T13:42:51.371369+0000","last_became_peered":"2024-03-30T13:42:51.371369+0000","last_unstale":"2024-03-30T13:44:12.433753+0000","last_undegraded":"2024-03-30T13:44:12.433753+0000","last_fullsized":"2024-03-30T13:44:12.433753+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:19.941761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.429932+0000","last_change":"2024-03-30T13:42:49.323595+0000","last_active":"2024-03-30T13:44:12.429932+0000","last_peered":"2024-03-30T13:44:12.429932+0000","last_clean":"2024-03-30T13:44:12.429932+0000","last_became_active":"2024-03-30T13:42:49.323319+0000","last_became_peered":"2024-03-30T13:42:49.323319+0000","last_unstale":"2024-03-30T13:44:12.429932+0000","last_undegraded":"2024-03-30T13:44:12.429932+0000","last_fullsized":"2024-03-30T13:44:12.429932+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":75,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301067+0000","last_change":"2024-03-30T13:43:49.990189+0000","last_active":"2024-03-30T13:44:12.301067+0000","last_peered":"2024-03-30T13:44:12.301067+0000","last_clean":"2024-03-30T13:44:12.301067+0000","last_became_active":"2024-03-30T13:42:45.315018+0000","last_became_peered":"2024-03-30T13:42:45.315018+0000","last_unstale":"2024-03-30T13:44:12.301067+0000","last_undegraded":"2024-03-30T13:44:12.301067+0000","last_fullsized":"2024-03-30T13:44:12.301067+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:49.990065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:43:49.990065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:58.482640+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"72'11","reported_seq":90,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:15.841689+0000","last_change":"2024-03-30T13:44:15.841741+0000","last_active":"2024-03-30T13:44:15.841689+0000","last_peered":"2024-03-30T13:44:15.841689+0000","last_clean":"2024-03-30T13:44:15.841689+0000","last_became_active":"2024-03-30T13:42:47.315667+0000","last_became_peered":"2024-03-30T13:42:47.315667+0000","last_unstale":"2024-03-30T13:44:15.841689+0000","last_undegraded":"2024-03-30T13:44:15.841689+0000","last_fullsized":"2024-03-30T13:44:15.841689+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:44:15.841626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:15.841626+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:23.388522+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516177+0000","last_change":"2024-03-30T13:42:51.331741+0000","last_active":"2024-03-30T13:44:12.516177+0000","last_peered":"2024-03-30T13:44:12.516177+0000","last_clean":"2024-03-30T13:44:12.516177+0000","last_became_active":"2024-03-30T13:42:51.331372+0000","last_became_peered":"2024-03-30T13:42:51.331372+0000","last_unstale":"2024-03-30T13:44:12.516177+0000","last_undegraded":"2024-03-30T13:44:12.516177+0000","last_fullsized":"2024-03-30T13:44:12.516177+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:17.417830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297784+0000","last_change":"2024-03-30T13:44:02.833988+0000","last_active":"2024-03-30T13:44:12.297784+0000","last_peered":"2024-03-30T13:44:12.297784+0000","last_clean":"2024-03-30T13:44:12.297784+0000","last_became_active":"2024-03-30T13:42:49.321360+0000","last_became_peered":"2024-03-30T13:42:49.321360+0000","last_unstale":"2024-03-30T13:44:12.297784+0000","last_undegraded":"2024-03-30T13:44:12.297784+0000","last_fullsized":"2024-03-30T13:44:12.297784+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:02.833909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:02.833909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:31.595404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516625+0000","last_change":"2024-03-30T13:44:08.809025+0000","last_active":"2024-03-30T13:44:12.516625+0000","last_peered":"2024-03-30T13:44:12.516625+0000","last_clean":"2024-03-30T13:44:12.516625+0000","last_became_active":"2024-03-30T13:42:45.315251+0000","last_became_peered":"2024-03-30T13:42:45.315251+0000","last_unstale":"2024-03-30T13:44:12.516625+0000","last_undegraded":"2024-03-30T13:44:12.516625+0000","last_fullsized":"2024-03-30T13:44:12.516625+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:08.808863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:08.808863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:23.892469+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":91,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301852+0000","last_change":"2024-03-30T13:42:47.316447+0000","last_active":"2024-03-30T13:44:12.301852+0000","last_peered":"2024-03-30T13:44:12.301852+0000","last_clean":"2024-03-30T13:44:12.301852+0000","last_became_active":"2024-03-30T13:42:47.315495+0000","last_became_peered":"2024-03-30T13:42:47.315495+0000","last_unstale":"2024-03-30T13:44:12.301852+0000","last_undegraded":"2024-03-30T13:44:12.301852+0000","last_fullsized":"2024-03-30T13:44:12.301852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.302147+0000","last_change":"2024-03-30T13:42:51.325672+0000","last_active":"2024-03-30T13:44:12.302147+0000","last_peered":"2024-03-30T13:44:12.302147+0000","last_clean":"2024-03-30T13:44:12.302147+0000","last_became_active":"2024-03-30T13:42:51.325389+0000","last_became_peered":"2024-03-30T13:42:51.325389+0000","last_unstale":"2024-03-30T13:44:12.302147+0000","last_undegraded":"2024-03-30T13:44:12.302147+0000","last_fullsized":"2024-03-30T13:44:12.302147+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"72'8","reported_seq":97,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.515976+0000","last_change":"2024-03-30T13:43:50.850857+0000","last_active":"2024-03-30T13:44:12.515976+0000","last_peered":"2024-03-30T13:44:12.515976+0000","last_clean":"2024-03-30T13:44:12.515976+0000","last_became_active":"2024-03-30T13:42:49.323371+0000","last_became_peered":"2024-03-30T13:42:49.323371+0000","last_unstale":"2024-03-30T13:44:12.515976+0000","last_undegraded":"2024-03-30T13:44:12.515976+0000","last_fullsized":"2024-03-30T13:44:12.515976+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:43:50.850759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:50.850759+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:01.654480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":74,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433897+0000","last_change":"2024-03-30T13:43:50.050389+0000","last_active":"2024-03-30T13:44:12.433897+0000","last_peered":"2024-03-30T13:44:12.433897+0000","last_clean":"2024-03-30T13:44:12.433897+0000","last_became_active":"2024-03-30T13:42:45.309265+0000","last_became_peered":"2024-03-30T13:42:45.309265+0000","last_unstale":"2024-03-30T13:44:12.433897+0000","last_undegraded":"2024-03-30T13:44:12.433897+0000","last_fullsized":"2024-03-30T13:44:12.433897+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:50.050303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:43:50.050303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:05.509528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":78,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301816+0000","last_change":"2024-03-30T13:42:47.313353+0000","last_active":"2024-03-30T13:44:12.301816+0000","last_peered":"2024-03-30T13:44:12.301816+0000","last_clean":"2024-03-30T13:44:12.301816+0000","last_became_active":"2024-03-30T13:42:47.313026+0000","last_became_peered":"2024-03-30T13:42:47.313026+0000","last_unstale":"2024-03-30T13:44:12.301816+0000","last_undegraded":"2024-03-30T13:44:12.301816+0000","last_fullsized":"2024-03-30T13:44:12.301816+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":407,"ondisk_log_size":407,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":81,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103192,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32704,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712448,"statfs":{"total":95995035648,"available":95961546752,"internally_reserved":0,"allocated":1564672,"data_stored":1055292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397532,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32684,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712468,"statfs":{"total":95995035648,"available":95961567232,"internally_reserved":0,"allocated":1544192,"data_stored":1054240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":465822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32708,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712444,"statfs":{"total":95995035648,"available":95961542656,"internally_reserved":0,"allocated":1568768,"data_stored":1055896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":464959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182444,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":464750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953510,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":464875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691873,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":465568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:44:21.567 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T13:44:21.820 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=plugin.finder t=2024-03-30T13:44:21.486086797Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-03-30T13:44:21.820 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=secrets t=2024-03-30T13:44:21.486191972Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-30T13:44:21.820 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=query_data t=2024-03-30T13:44:21.489926917Z level=info msg="Query Service initialization" 2024-03-30T13:44:21.820 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=live.push_http t=2024-03-30T13:44:21.49230896Z level=info msg="Live Push Gateway initialization" 2024-03-30T13:44:21.842 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:44:22.096 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-mon[34341]: 3.1f scrub starts 2024-03-30T13:44:22.096 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-mon[34341]: 3.1f scrub ok 2024-03-30T13:44:22.096 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-mon[34341]: from='client.24620 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:44:22.096 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-mon[34341]: 6.4 scrub starts 2024-03-30T13:44:22.096 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-mon[34341]: 6.4 scrub ok 2024-03-30T13:44:22.096 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:22.096 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:22.096 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:22.096 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:22.096 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-mon[34341]: 3.10 deep-scrub starts 2024-03-30T13:44:22.096 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-mon[34341]: 3.10 deep-scrub ok 2024-03-30T13:44:22.096 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-mon[34341]: 4.17 scrub starts 2024-03-30T13:44:22.097 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:21 smithi113 ceph-mon[34341]: 4.17 scrub ok 2024-03-30T13:44:22.132 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[42099]: 3.1f scrub starts 2024-03-30T13:44:22.132 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[42099]: 3.1f scrub ok 2024-03-30T13:44:22.132 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[42099]: from='client.24620 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:44:22.132 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[42099]: 6.4 scrub starts 2024-03-30T13:44:22.132 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[42099]: 6.4 scrub ok 2024-03-30T13:44:22.132 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:22.132 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:22.132 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:22.132 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:22.133 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[42099]: 3.10 deep-scrub starts 2024-03-30T13:44:22.133 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[42099]: 3.10 deep-scrub ok 2024-03-30T13:44:22.133 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[42099]: 4.17 scrub starts 2024-03-30T13:44:22.133 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[42099]: 4.17 scrub ok 2024-03-30T13:44:22.133 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[29963]: 3.1f scrub starts 2024-03-30T13:44:22.133 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[29963]: 3.1f scrub ok 2024-03-30T13:44:22.133 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[29963]: from='client.24620 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:44:22.133 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[29963]: 6.4 scrub starts 2024-03-30T13:44:22.133 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[29963]: 6.4 scrub ok 2024-03-30T13:44:22.133 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:22.133 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:22.133 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:22.134 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:22.134 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[29963]: 3.10 deep-scrub starts 2024-03-30T13:44:22.134 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[29963]: 3.10 deep-scrub ok 2024-03-30T13:44:22.134 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[29963]: 4.17 scrub starts 2024-03-30T13:44:22.134 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:21 smithi043 ceph-mon[29963]: 4.17 scrub ok 2024-03-30T13:44:22.472 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=infra.usagestats.collector t=2024-03-30T13:44:22.094900184Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-30T13:44:22.472 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=provisioning.datasources t=2024-03-30T13:44:22.096019506Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-30T13:44:22.472 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=provisioning.datasources t=2024-03-30T13:44:22.106934505Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-30T13:44:22.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=provisioning.alerting t=2024-03-30T13:44:22.117458443Z level=info msg="starting to provision alerting" 2024-03-30T13:44:22.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=provisioning.alerting t=2024-03-30T13:44:22.11746831Z level=info msg="finished to provision alerting" 2024-03-30T13:44:22.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=grafanaStorageLogger t=2024-03-30T13:44:22.117658596Z level=info msg="storage starting" 2024-03-30T13:44:22.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=ngalert.state.manager t=2024-03-30T13:44:22.117725136Z level=info msg="Warming state cache for startup" 2024-03-30T13:44:22.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=http.server t=2024-03-30T13:44:22.12008484Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-30T13:44:22.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=ngalert.state.manager t=2024-03-30T13:44:22.12338374Z level=info msg="State cache has been initialized" states=0 duration=5.65507ms 2024-03-30T13:44:22.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=ticker t=2024-03-30T13:44:22.123467196Z level=info msg=starting first_tick=2024-03-30T13:44:30Z 2024-03-30T13:44:22.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=ngalert.multiorg.alertmanager t=2024-03-30T13:44:22.123455586Z level=info msg="starting MultiOrg Alertmanager" 2024-03-30T13:44:22.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=sqlstore.transactions t=2024-03-30T13:44:22.12836548Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-30T13:44:22.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=sqlstore.transactions t=2024-03-30T13:44:22.139479Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-30T13:44:22.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=sqlstore.transactions t=2024-03-30T13:44:22.149807023Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-03-30T13:44:22.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=sqlstore.transactions t=2024-03-30T13:44:22.160461442Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-03-30T13:44:22.473 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=sqlstore.transactions t=2024-03-30T13:44:22.170804529Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=4 code="database is locked" 2024-03-30T13:44:23.082 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:22 smithi043 ceph-mon[29963]: Deploying daemon node-exporter.a on smithi043 2024-03-30T13:44:23.082 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:22 smithi043 ceph-mon[29963]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:23.082 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:22 smithi043 ceph-mon[29963]: 4.6 deep-scrub starts 2024-03-30T13:44:23.082 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:22 smithi043 ceph-mon[29963]: 4.6 deep-scrub ok 2024-03-30T13:44:23.082 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:22 smithi043 ceph-mon[29963]: 3.1d scrub starts 2024-03-30T13:44:23.083 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:22 smithi043 ceph-mon[29963]: 3.1d scrub ok 2024-03-30T13:44:23.083 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:22 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:23.083 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:22 smithi043 ceph-mon[42099]: Deploying daemon node-exporter.a on smithi043 2024-03-30T13:44:23.083 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:22 smithi043 ceph-mon[42099]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:23.083 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:22 smithi043 ceph-mon[42099]: 4.6 deep-scrub starts 2024-03-30T13:44:23.083 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:22 smithi043 ceph-mon[42099]: 4.6 deep-scrub ok 2024-03-30T13:44:23.083 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:22 smithi043 ceph-mon[42099]: 3.1d scrub starts 2024-03-30T13:44:23.083 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:22 smithi043 ceph-mon[42099]: 3.1d scrub ok 2024-03-30T13:44:23.083 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:22 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:23.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-mon[34341]: Deploying daemon node-exporter.a on smithi043 2024-03-30T13:44:23.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-mon[34341]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:23.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-mon[34341]: 4.6 deep-scrub starts 2024-03-30T13:44:23.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-mon[34341]: 4.6 deep-scrub ok 2024-03-30T13:44:23.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-mon[34341]: 3.1d scrub starts 2024-03-30T13:44:23.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-mon[34341]: 3.1d scrub ok 2024-03-30T13:44:23.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:22 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:23.825 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:44:23.827 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-30T13:44:23.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[42099]: 6.f scrub starts 2024-03-30T13:44:23.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[42099]: 6.f scrub ok 2024-03-30T13:44:23.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:44:23.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:44:23.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[42099]: 6.3 scrub starts 2024-03-30T13:44:23.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[42099]: 6.3 scrub ok 2024-03-30T13:44:23.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[42099]: 6.1e scrub starts 2024-03-30T13:44:23.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[42099]: 6.1e scrub ok 2024-03-30T13:44:23.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:44:23.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:44:23.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[42099]: 5.13 scrub starts 2024-03-30T13:44:23.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[42099]: 5.13 scrub ok 2024-03-30T13:44:23.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[42099]: osdmap e94: 8 total, 8 up, 8 in 2024-03-30T13:44:23.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[29963]: 6.f scrub starts 2024-03-30T13:44:23.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[29963]: 6.f scrub ok 2024-03-30T13:44:23.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:44:23.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:44:23.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[29963]: 6.3 scrub starts 2024-03-30T13:44:23.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[29963]: 6.3 scrub ok 2024-03-30T13:44:23.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[29963]: 6.1e scrub starts 2024-03-30T13:44:23.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[29963]: 6.1e scrub ok 2024-03-30T13:44:23.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:44:23.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:44:23.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[29963]: 5.13 scrub starts 2024-03-30T13:44:23.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[29963]: 5.13 scrub ok 2024-03-30T13:44:23.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:23 smithi043 ceph-mon[29963]: osdmap e94: 8 total, 8 up, 8 in 2024-03-30T13:44:24.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:23 smithi113 ceph-mon[34341]: 6.f scrub starts 2024-03-30T13:44:24.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:23 smithi113 ceph-mon[34341]: 6.f scrub ok 2024-03-30T13:44:24.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:23 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:44:24.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:23 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:44:24.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:23 smithi113 ceph-mon[34341]: 6.3 scrub starts 2024-03-30T13:44:24.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:23 smithi113 ceph-mon[34341]: 6.3 scrub ok 2024-03-30T13:44:24.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:23 smithi113 ceph-mon[34341]: 6.1e scrub starts 2024-03-30T13:44:24.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:23 smithi113 ceph-mon[34341]: 6.1e scrub ok 2024-03-30T13:44:24.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:23 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:44:24.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:23 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:44:24.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:23 smithi113 ceph-mon[34341]: 5.13 scrub starts 2024-03-30T13:44:24.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:23 smithi113 ceph-mon[34341]: 5.13 scrub ok 2024-03-30T13:44:24.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:23 smithi113 ceph-mon[34341]: osdmap e94: 8 total, 8 up, 8 in 2024-03-30T13:44:24.373 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:24 smithi043 bash[81708]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-30T13:44:25.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:24 smithi043 ceph-mon[29963]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-30T13:44:25.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:24 smithi043 ceph-mon[29963]: 5.1c scrub starts 2024-03-30T13:44:25.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:24 smithi043 ceph-mon[29963]: 5.1c scrub ok 2024-03-30T13:44:25.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:24 smithi043 ceph-mon[29963]: from='client.24626 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:44:25.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:24 smithi043 ceph-mon[29963]: 3.7 deep-scrub starts 2024-03-30T13:44:25.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:24 smithi043 ceph-mon[29963]: 3.7 deep-scrub ok 2024-03-30T13:44:25.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:24 smithi043 ceph-mon[42099]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-30T13:44:25.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:24 smithi043 ceph-mon[42099]: 5.1c scrub starts 2024-03-30T13:44:25.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:24 smithi043 ceph-mon[42099]: 5.1c scrub ok 2024-03-30T13:44:25.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:24 smithi043 ceph-mon[42099]: from='client.24626 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:44:25.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:24 smithi043 ceph-mon[42099]: 3.7 deep-scrub starts 2024-03-30T13:44:25.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:24 smithi043 ceph-mon[42099]: 3.7 deep-scrub ok 2024-03-30T13:44:25.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:24 smithi113 ceph-mon[34341]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-30T13:44:25.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:24 smithi113 ceph-mon[34341]: 5.1c scrub starts 2024-03-30T13:44:25.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:24 smithi113 ceph-mon[34341]: 5.1c scrub ok 2024-03-30T13:44:25.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:24 smithi113 ceph-mon[34341]: from='client.24626 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:44:25.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:24 smithi113 ceph-mon[34341]: 3.7 deep-scrub starts 2024-03-30T13:44:25.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:24 smithi113 ceph-mon[34341]: 3.7 deep-scrub ok 2024-03-30T13:44:25.872 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-03-30T13:44:23.342268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1025,"num_read_kb":871,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":620,"ondisk_log_size":620,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":242108,"kb_used_data":9340,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749719108,"statfs":{"total":767960285184,"available":767712366592,"internally_reserved":0,"allocated":9564160,"data_stored":5491402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.062740"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301158+0000","last_change":"2024-03-30T13:42:51.330132+0000","last_active":"2024-03-30T13:44:12.301158+0000","last_peered":"2024-03-30T13:44:12.301158+0000","last_clean":"2024-03-30T13:44:12.301158+0000","last_became_active":"2024-03-30T13:42:51.329797+0000","last_became_peered":"2024-03-30T13:42:51.329797+0000","last_unstale":"2024-03-30T13:44:12.301158+0000","last_undegraded":"2024-03-30T13:44:12.301158+0000","last_fullsized":"2024-03-30T13:44:12.301158+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.296683+0000","last_change":"2024-03-30T13:44:04.797677+0000","last_active":"2024-03-30T13:44:12.296683+0000","last_peered":"2024-03-30T13:44:12.296683+0000","last_clean":"2024-03-30T13:44:12.296683+0000","last_became_active":"2024-03-30T13:42:49.323268+0000","last_became_peered":"2024-03-30T13:42:49.323268+0000","last_unstale":"2024-03-30T13:44:12.296683+0000","last_undegraded":"2024-03-30T13:44:12.296683+0000","last_fullsized":"2024-03-30T13:44:12.296683+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:04.797592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:04.797592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:20.695069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"65'1","reported_seq":78,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:20.372195+0000","last_change":"2024-03-30T13:44:20.372253+0000","last_active":"2024-03-30T13:44:20.372195+0000","last_peered":"2024-03-30T13:44:20.372195+0000","last_clean":"2024-03-30T13:44:20.372195+0000","last_became_active":"2024-03-30T13:42:45.313936+0000","last_became_peered":"2024-03-30T13:42:45.313936+0000","last_unstale":"2024-03-30T13:44:20.372195+0000","last_undegraded":"2024-03-30T13:44:20.372195+0000","last_fullsized":"2024-03-30T13:44:20.372195+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:44:20.372089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:20.372089+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:47.174676+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":79,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301002+0000","last_change":"2024-03-30T13:42:47.320128+0000","last_active":"2024-03-30T13:44:12.301002+0000","last_peered":"2024-03-30T13:44:12.301002+0000","last_clean":"2024-03-30T13:44:12.301002+0000","last_became_active":"2024-03-30T13:42:47.319638+0000","last_became_peered":"2024-03-30T13:42:47.319638+0000","last_unstale":"2024-03-30T13:44:12.301002+0000","last_undegraded":"2024-03-30T13:44:12.301002+0000","last_fullsized":"2024-03-30T13:44:12.301002+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433693+0000","last_change":"2024-03-30T13:42:51.374776+0000","last_active":"2024-03-30T13:44:12.433693+0000","last_peered":"2024-03-30T13:44:12.433693+0000","last_clean":"2024-03-30T13:44:12.433693+0000","last_became_active":"2024-03-30T13:42:51.374403+0000","last_became_peered":"2024-03-30T13:42:51.374403+0000","last_unstale":"2024-03-30T13:44:12.433693+0000","last_undegraded":"2024-03-30T13:44:12.433693+0000","last_fullsized":"2024-03-30T13:44:12.433693+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:13.438854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301117+0000","last_change":"2024-03-30T13:42:49.327437+0000","last_active":"2024-03-30T13:44:12.301117+0000","last_peered":"2024-03-30T13:44:12.301117+0000","last_clean":"2024-03-30T13:44:12.301117+0000","last_became_active":"2024-03-30T13:42:49.327124+0000","last_became_peered":"2024-03-30T13:42:49.327124+0000","last_unstale":"2024-03-30T13:44:12.301117+0000","last_undegraded":"2024-03-30T13:44:12.301117+0000","last_fullsized":"2024-03-30T13:44:12.301117+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:17.902978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":74,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.434120+0000","last_change":"2024-03-30T13:44:04.008976+0000","last_active":"2024-03-30T13:44:12.434120+0000","last_peered":"2024-03-30T13:44:12.434120+0000","last_clean":"2024-03-30T13:44:12.434120+0000","last_became_active":"2024-03-30T13:42:45.310771+0000","last_became_peered":"2024-03-30T13:42:45.310771+0000","last_unstale":"2024-03-30T13:44:12.434120+0000","last_undegraded":"2024-03-30T13:44:12.434120+0000","last_fullsized":"2024-03-30T13:44:12.434120+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:04.008875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:04.008875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:26.922817+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"72'15","reported_seq":87,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433501+0000","last_change":"2024-03-30T13:42:47.317438+0000","last_active":"2024-03-30T13:44:12.433501+0000","last_peered":"2024-03-30T13:44:12.433501+0000","last_clean":"2024-03-30T13:44:12.433501+0000","last_became_active":"2024-03-30T13:42:47.317102+0000","last_became_peered":"2024-03-30T13:42:47.317102+0000","last_unstale":"2024-03-30T13:44:12.433501+0000","last_undegraded":"2024-03-30T13:44:12.433501+0000","last_fullsized":"2024-03-30T13:44:12.433501+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.296929+0000","last_change":"2024-03-30T13:42:51.331563+0000","last_active":"2024-03-30T13:44:12.296929+0000","last_peered":"2024-03-30T13:44:12.296929+0000","last_clean":"2024-03-30T13:44:12.296929+0000","last_became_active":"2024-03-30T13:42:51.331196+0000","last_became_peered":"2024-03-30T13:42:51.331196+0000","last_unstale":"2024-03-30T13:44:12.296929+0000","last_undegraded":"2024-03-30T13:44:12.296929+0000","last_fullsized":"2024-03-30T13:44:12.296929+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.431941+0000","last_change":"2024-03-30T13:43:59.262080+0000","last_active":"2024-03-30T13:44:12.431941+0000","last_peered":"2024-03-30T13:44:12.431941+0000","last_clean":"2024-03-30T13:44:12.431941+0000","last_became_active":"2024-03-30T13:42:49.321811+0000","last_became_peered":"2024-03-30T13:42:49.321811+0000","last_unstale":"2024-03-30T13:44:12.431941+0000","last_undegraded":"2024-03-30T13:44:12.431941+0000","last_fullsized":"2024-03-30T13:44:12.431941+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:59.261981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:59.261981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:20.497993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:21.474344+0000","last_change":"2024-03-30T13:44:21.474364+0000","last_active":"2024-03-30T13:44:21.474344+0000","last_peered":"2024-03-30T13:44:21.474344+0000","last_clean":"2024-03-30T13:44:21.474344+0000","last_became_active":"2024-03-30T13:42:45.319827+0000","last_became_peered":"2024-03-30T13:42:45.319827+0000","last_unstale":"2024-03-30T13:44:21.474344+0000","last_undegraded":"2024-03-30T13:44:21.474344+0000","last_fullsized":"2024-03-30T13:44:21.474344+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:21.474272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:21.474272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:45.758125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":87,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:13.831914+0000","last_change":"2024-03-30T13:44:13.831942+0000","last_active":"2024-03-30T13:44:13.831914+0000","last_peered":"2024-03-30T13:44:13.831914+0000","last_clean":"2024-03-30T13:44:13.831914+0000","last_became_active":"2024-03-30T13:42:47.310289+0000","last_became_peered":"2024-03-30T13:42:47.310289+0000","last_unstale":"2024-03-30T13:44:13.831914+0000","last_undegraded":"2024-03-30T13:44:13.831914+0000","last_fullsized":"2024-03-30T13:44:13.831914+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:44:13.831850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:13.831850+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:37.858895+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:18.478400+0000","last_change":"2024-03-30T13:44:18.478424+0000","last_active":"2024-03-30T13:44:18.478400+0000","last_peered":"2024-03-30T13:44:18.478400+0000","last_clean":"2024-03-30T13:44:18.478400+0000","last_became_active":"2024-03-30T13:42:51.325748+0000","last_became_peered":"2024-03-30T13:42:51.325748+0000","last_unstale":"2024-03-30T13:44:18.478400+0000","last_undegraded":"2024-03-30T13:44:18.478400+0000","last_fullsized":"2024-03-30T13:44:18.478400+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:18.478333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:18.478333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:29.038242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":56,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516276+0000","last_change":"2024-03-30T13:42:49.326311+0000","last_active":"2024-03-30T13:44:12.516276+0000","last_peered":"2024-03-30T13:44:12.516276+0000","last_clean":"2024-03-30T13:44:12.516276+0000","last_became_active":"2024-03-30T13:42:49.326027+0000","last_became_peered":"2024-03-30T13:42:49.326027+0000","last_unstale":"2024-03-30T13:44:12.516276+0000","last_undegraded":"2024-03-30T13:44:12.516276+0000","last_fullsized":"2024-03-30T13:44:12.516276+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:17.535137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"65'1","reported_seq":79,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.429662+0000","last_change":"2024-03-30T13:44:04.348591+0000","last_active":"2024-03-30T13:44:12.429662+0000","last_peered":"2024-03-30T13:44:12.429662+0000","last_clean":"2024-03-30T13:44:12.429662+0000","last_became_active":"2024-03-30T13:42:45.319975+0000","last_became_peered":"2024-03-30T13:42:45.319975+0000","last_unstale":"2024-03-30T13:44:12.429662+0000","last_undegraded":"2024-03-30T13:44:12.429662+0000","last_fullsized":"2024-03-30T13:44:12.429662+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:44:04.348486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:04.348486+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:34.284640+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":83,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301518+0000","last_change":"2024-03-30T13:43:50.961052+0000","last_active":"2024-03-30T13:44:12.301518+0000","last_peered":"2024-03-30T13:44:12.301518+0000","last_clean":"2024-03-30T13:44:12.301518+0000","last_became_active":"2024-03-30T13:42:47.317352+0000","last_became_peered":"2024-03-30T13:42:47.317352+0000","last_unstale":"2024-03-30T13:44:12.301518+0000","last_undegraded":"2024-03-30T13:44:12.301518+0000","last_fullsized":"2024-03-30T13:44:12.301518+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-30T13:43:50.960980+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-30T13:43:50.960980+0000","last_clean_scrub_stamp":"2024-03-30T13:43:50.960980+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:15.585035+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:22.815233+0000","last_change":"2024-03-30T13:44:22.815266+0000","last_active":"2024-03-30T13:44:22.815233+0000","last_peered":"2024-03-30T13:44:22.815233+0000","last_clean":"2024-03-30T13:44:22.815233+0000","last_became_active":"2024-03-30T13:42:51.372011+0000","last_became_peered":"2024-03-30T13:42:51.372011+0000","last_unstale":"2024-03-30T13:44:22.815233+0000","last_undegraded":"2024-03-30T13:44:22.815233+0000","last_fullsized":"2024-03-30T13:44:22.815233+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:22.815168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:22.815168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:41.178238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.298523+0000","last_change":"2024-03-30T13:44:08.821913+0000","last_active":"2024-03-30T13:44:12.298523+0000","last_peered":"2024-03-30T13:44:12.298523+0000","last_clean":"2024-03-30T13:44:12.298523+0000","last_became_active":"2024-03-30T13:42:49.322665+0000","last_became_peered":"2024-03-30T13:42:49.322665+0000","last_unstale":"2024-03-30T13:44:12.298523+0000","last_undegraded":"2024-03-30T13:44:12.298523+0000","last_fullsized":"2024-03-30T13:44:12.298523+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:08.821811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:08.821811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:10.515727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297849+0000","last_change":"2024-03-30T13:42:45.317711+0000","last_active":"2024-03-30T13:44:12.297849+0000","last_peered":"2024-03-30T13:44:12.297849+0000","last_clean":"2024-03-30T13:44:12.297849+0000","last_became_active":"2024-03-30T13:42:45.317378+0000","last_became_peered":"2024-03-30T13:42:45.317378+0000","last_unstale":"2024-03-30T13:44:12.297849+0000","last_undegraded":"2024-03-30T13:44:12.297849+0000","last_fullsized":"2024-03-30T13:44:12.297849+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:14.011698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":96,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.515927+0000","last_change":"2024-03-30T13:44:05.870649+0000","last_active":"2024-03-30T13:44:12.515927+0000","last_peered":"2024-03-30T13:44:12.515927+0000","last_clean":"2024-03-30T13:44:12.515927+0000","last_became_active":"2024-03-30T13:42:47.320168+0000","last_became_peered":"2024-03-30T13:42:47.320168+0000","last_unstale":"2024-03-30T13:44:12.515927+0000","last_undegraded":"2024-03-30T13:44:12.515927+0000","last_fullsized":"2024-03-30T13:44:12.515927+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-30T13:44:05.870550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:05.870550+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:10.637627+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.434088+0000","last_change":"2024-03-30T13:42:51.371902+0000","last_active":"2024-03-30T13:44:12.434088+0000","last_peered":"2024-03-30T13:44:12.434088+0000","last_clean":"2024-03-30T13:44:12.434088+0000","last_became_active":"2024-03-30T13:42:51.371378+0000","last_became_peered":"2024-03-30T13:42:51.371378+0000","last_unstale":"2024-03-30T13:44:12.434088+0000","last_undegraded":"2024-03-30T13:44:12.434088+0000","last_fullsized":"2024-03-30T13:44:12.434088+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:20.136855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301714+0000","last_change":"2024-03-30T13:42:49.323001+0000","last_active":"2024-03-30T13:44:12.301714+0000","last_peered":"2024-03-30T13:44:12.301714+0000","last_clean":"2024-03-30T13:44:12.301714+0000","last_became_active":"2024-03-30T13:42:49.322561+0000","last_became_peered":"2024-03-30T13:42:49.322561+0000","last_unstale":"2024-03-30T13:44:12.301714+0000","last_undegraded":"2024-03-30T13:44:12.301714+0000","last_fullsized":"2024-03-30T13:44:12.301714+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301525+0000","last_change":"2024-03-30T13:42:45.308820+0000","last_active":"2024-03-30T13:44:12.301525+0000","last_peered":"2024-03-30T13:44:12.301525+0000","last_clean":"2024-03-30T13:44:12.301525+0000","last_became_active":"2024-03-30T13:42:45.308468+0000","last_became_peered":"2024-03-30T13:42:45.308468+0000","last_unstale":"2024-03-30T13:44:12.301525+0000","last_undegraded":"2024-03-30T13:44:12.301525+0000","last_fullsized":"2024-03-30T13:44:12.301525+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"88'17","reported_seq":98,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433874+0000","last_change":"2024-03-30T13:43:52.044529+0000","last_active":"2024-03-30T13:44:12.433874+0000","last_peered":"2024-03-30T13:44:12.433874+0000","last_clean":"2024-03-30T13:44:12.433874+0000","last_became_active":"2024-03-30T13:42:47.317881+0000","last_became_peered":"2024-03-30T13:42:47.317881+0000","last_unstale":"2024-03-30T13:44:12.433874+0000","last_undegraded":"2024-03-30T13:44:12.433874+0000","last_fullsized":"2024-03-30T13:44:12.433874+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'17","last_scrub_stamp":"2024-03-30T13:43:52.044427+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-03-30T13:43:52.044427+0000","last_clean_scrub_stamp":"2024-03-30T13:43:52.044427+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:11.077720+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:14.833893+0000","last_change":"2024-03-30T13:44:14.833942+0000","last_active":"2024-03-30T13:44:14.833893+0000","last_peered":"2024-03-30T13:44:14.833893+0000","last_clean":"2024-03-30T13:44:14.833893+0000","last_became_active":"2024-03-30T13:42:51.327749+0000","last_became_peered":"2024-03-30T13:42:51.327749+0000","last_unstale":"2024-03-30T13:44:14.833893+0000","last_undegraded":"2024-03-30T13:44:14.833893+0000","last_fullsized":"2024-03-30T13:44:14.833893+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T13:44:14.833803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:14.833803+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:27.114310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":95,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:15.378474+0000","last_change":"2024-03-30T13:44:10.804691+0000","last_active":"2024-03-30T13:44:15.378474+0000","last_peered":"2024-03-30T13:44:15.378474+0000","last_clean":"2024-03-30T13:44:15.378474+0000","last_became_active":"2024-03-30T13:42:49.323362+0000","last_became_peered":"2024-03-30T13:42:49.323362+0000","last_unstale":"2024-03-30T13:44:15.378474+0000","last_undegraded":"2024-03-30T13:44:15.378474+0000","last_fullsized":"2024-03-30T13:44:15.378474+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:44:10.804550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:10.804550+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:38.462339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"65'1","reported_seq":75,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.296777+0000","last_change":"2024-03-30T13:43:47.807153+0000","last_active":"2024-03-30T13:44:12.296777+0000","last_peered":"2024-03-30T13:44:12.296777+0000","last_clean":"2024-03-30T13:44:12.296777+0000","last_became_active":"2024-03-30T13:42:45.307832+0000","last_became_peered":"2024-03-30T13:42:45.307832+0000","last_unstale":"2024-03-30T13:44:12.296777+0000","last_undegraded":"2024-03-30T13:44:12.296777+0000","last_fullsized":"2024-03-30T13:44:12.296777+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:43:47.807055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:43:47.807055+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:10.931779+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":83,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:15.356830+0000","last_change":"2024-03-30T13:44:15.356873+0000","last_active":"2024-03-30T13:44:15.356830+0000","last_peered":"2024-03-30T13:44:15.356830+0000","last_clean":"2024-03-30T13:44:15.356830+0000","last_became_active":"2024-03-30T13:42:47.500763+0000","last_became_peered":"2024-03-30T13:42:47.500763+0000","last_unstale":"2024-03-30T13:44:15.356830+0000","last_undegraded":"2024-03-30T13:44:15.356830+0000","last_fullsized":"2024-03-30T13:44:15.356830+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-30T13:44:15.356762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:15.356762+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:42.015410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":62,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:19.787192+0000","last_change":"2024-03-30T13:44:19.787217+0000","last_active":"2024-03-30T13:44:19.787192+0000","last_peered":"2024-03-30T13:44:19.787192+0000","last_clean":"2024-03-30T13:44:19.787192+0000","last_became_active":"2024-03-30T13:42:51.325513+0000","last_became_peered":"2024-03-30T13:42:51.325513+0000","last_unstale":"2024-03-30T13:44:19.787192+0000","last_undegraded":"2024-03-30T13:44:19.787192+0000","last_fullsized":"2024-03-30T13:44:19.787192+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:19.787125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:19.787125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:32.974543+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.298372+0000","last_change":"2024-03-30T13:44:02.408701+0000","last_active":"2024-03-30T13:44:12.298372+0000","last_peered":"2024-03-30T13:44:12.298372+0000","last_clean":"2024-03-30T13:44:12.298372+0000","last_became_active":"2024-03-30T13:42:49.330335+0000","last_became_peered":"2024-03-30T13:42:49.330335+0000","last_unstale":"2024-03-30T13:44:12.298372+0000","last_undegraded":"2024-03-30T13:44:12.298372+0000","last_fullsized":"2024-03-30T13:44:12.298372+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:02.408612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:02.408612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:23.673286+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433246+0000","last_change":"2024-03-30T13:42:45.310476+0000","last_active":"2024-03-30T13:44:12.433246+0000","last_peered":"2024-03-30T13:44:12.433246+0000","last_clean":"2024-03-30T13:44:12.433246+0000","last_became_active":"2024-03-30T13:42:45.310165+0000","last_became_peered":"2024-03-30T13:42:45.310165+0000","last_unstale":"2024-03-30T13:44:12.433246+0000","last_undegraded":"2024-03-30T13:44:12.433246+0000","last_fullsized":"2024-03-30T13:44:12.433246+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":87,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516927+0000","last_change":"2024-03-30T13:43:54.758046+0000","last_active":"2024-03-30T13:44:12.516927+0000","last_peered":"2024-03-30T13:44:12.516927+0000","last_clean":"2024-03-30T13:44:12.516927+0000","last_became_active":"2024-03-30T13:42:47.324399+0000","last_became_peered":"2024-03-30T13:42:47.324399+0000","last_unstale":"2024-03-30T13:44:12.516927+0000","last_undegraded":"2024-03-30T13:44:12.516927+0000","last_fullsized":"2024-03-30T13:44:12.516927+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:43:54.757869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:54.757869+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:15.753990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":60,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516169+0000","last_change":"2024-03-30T13:43:59.682402+0000","last_active":"2024-03-30T13:44:12.516169+0000","last_peered":"2024-03-30T13:44:12.516169+0000","last_clean":"2024-03-30T13:44:12.516169+0000","last_became_active":"2024-03-30T13:42:51.327000+0000","last_became_peered":"2024-03-30T13:42:51.327000+0000","last_unstale":"2024-03-30T13:44:12.516169+0000","last_undegraded":"2024-03-30T13:44:12.516169+0000","last_fullsized":"2024-03-30T13:44:12.516169+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:59.682298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:43:59.682298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:25.906592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516174+0000","last_change":"2024-03-30T13:43:53.750352+0000","last_active":"2024-03-30T13:44:12.516174+0000","last_peered":"2024-03-30T13:44:12.516174+0000","last_clean":"2024-03-30T13:44:12.516174+0000","last_became_active":"2024-03-30T13:42:49.322799+0000","last_became_peered":"2024-03-30T13:42:49.322799+0000","last_unstale":"2024-03-30T13:44:12.516174+0000","last_undegraded":"2024-03-30T13:44:12.516174+0000","last_fullsized":"2024-03-30T13:44:12.516174+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:53.750254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:53.750254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:59.399300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297487+0000","last_change":"2024-03-30T13:43:58.386027+0000","last_active":"2024-03-30T13:44:12.297487+0000","last_peered":"2024-03-30T13:44:12.297487+0000","last_clean":"2024-03-30T13:44:12.297487+0000","last_became_active":"2024-03-30T13:42:45.314302+0000","last_became_peered":"2024-03-30T13:42:45.314302+0000","last_unstale":"2024-03-30T13:44:12.297487+0000","last_undegraded":"2024-03-30T13:44:12.297487+0000","last_fullsized":"2024-03-30T13:44:12.297487+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:58.385891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:43:58.385891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:03.698225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":76,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433247+0000","last_change":"2024-03-30T13:43:55.894589+0000","last_active":"2024-03-30T13:44:12.433247+0000","last_peered":"2024-03-30T13:44:12.433247+0000","last_clean":"2024-03-30T13:44:12.433247+0000","last_became_active":"2024-03-30T13:42:47.321428+0000","last_became_peered":"2024-03-30T13:42:47.321428+0000","last_unstale":"2024-03-30T13:44:12.433247+0000","last_undegraded":"2024-03-30T13:44:12.433247+0000","last_fullsized":"2024-03-30T13:44:12.433247+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-30T13:43:55.894502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:55.894502+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:56.441104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433493+0000","last_change":"2024-03-30T13:42:51.374755+0000","last_active":"2024-03-30T13:44:12.433493+0000","last_peered":"2024-03-30T13:44:12.433493+0000","last_clean":"2024-03-30T13:44:12.433493+0000","last_became_active":"2024-03-30T13:42:51.374304+0000","last_became_peered":"2024-03-30T13:42:51.374304+0000","last_unstale":"2024-03-30T13:44:12.433493+0000","last_undegraded":"2024-03-30T13:44:12.433493+0000","last_fullsized":"2024-03-30T13:44:12.433493+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:18.990969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.517441+0000","last_change":"2024-03-30T13:44:07.744732+0000","last_active":"2024-03-30T13:44:12.517441+0000","last_peered":"2024-03-30T13:44:12.517441+0000","last_clean":"2024-03-30T13:44:12.517441+0000","last_became_active":"2024-03-30T13:42:49.324206+0000","last_became_peered":"2024-03-30T13:42:49.324206+0000","last_unstale":"2024-03-30T13:44:12.517441+0000","last_undegraded":"2024-03-30T13:44:12.517441+0000","last_fullsized":"2024-03-30T13:44:12.517441+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:07.744620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:07.744620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:33.978349+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.429523+0000","last_change":"2024-03-30T13:42:45.319833+0000","last_active":"2024-03-30T13:44:12.429523+0000","last_peered":"2024-03-30T13:44:12.429523+0000","last_clean":"2024-03-30T13:44:12.429523+0000","last_became_active":"2024-03-30T13:42:45.319482+0000","last_became_peered":"2024-03-30T13:42:45.319482+0000","last_unstale":"2024-03-30T13:44:12.429523+0000","last_undegraded":"2024-03-30T13:44:12.429523+0000","last_fullsized":"2024-03-30T13:44:12.429523+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":89,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433574+0000","last_change":"2024-03-30T13:43:54.926339+0000","last_active":"2024-03-30T13:44:12.433574+0000","last_peered":"2024-03-30T13:44:12.433574+0000","last_clean":"2024-03-30T13:44:12.433574+0000","last_became_active":"2024-03-30T13:42:47.500712+0000","last_became_peered":"2024-03-30T13:42:47.500712+0000","last_unstale":"2024-03-30T13:44:12.433574+0000","last_undegraded":"2024-03-30T13:44:12.433574+0000","last_fullsized":"2024-03-30T13:44:12.433574+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:43:54.926242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:54.926242+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:59.626117+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.296661+0000","last_change":"2024-03-30T13:42:51.334062+0000","last_active":"2024-03-30T13:44:12.296661+0000","last_peered":"2024-03-30T13:44:12.296661+0000","last_clean":"2024-03-30T13:44:12.296661+0000","last_became_active":"2024-03-30T13:42:51.333603+0000","last_became_peered":"2024-03-30T13:42:51.333603+0000","last_unstale":"2024-03-30T13:44:12.296661+0000","last_undegraded":"2024-03-30T13:44:12.296661+0000","last_fullsized":"2024-03-30T13:44:12.296661+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:23.149727+0000","last_change":"2024-03-30T13:44:23.149767+0000","last_active":"2024-03-30T13:44:23.149727+0000","last_peered":"2024-03-30T13:44:23.149727+0000","last_clean":"2024-03-30T13:44:23.149727+0000","last_became_active":"2024-03-30T13:42:49.325219+0000","last_became_peered":"2024-03-30T13:42:49.325219+0000","last_unstale":"2024-03-30T13:44:23.149727+0000","last_undegraded":"2024-03-30T13:44:23.149727+0000","last_fullsized":"2024-03-30T13:44:23.149727+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:23.149653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:23.149653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:42.145206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.517441+0000","last_change":"2024-03-30T13:43:52.764537+0000","last_active":"2024-03-30T13:44:12.517441+0000","last_peered":"2024-03-30T13:44:12.517441+0000","last_clean":"2024-03-30T13:44:12.517441+0000","last_became_active":"2024-03-30T13:42:45.311071+0000","last_became_peered":"2024-03-30T13:42:45.311071+0000","last_unstale":"2024-03-30T13:44:12.517441+0000","last_undegraded":"2024-03-30T13:44:12.517441+0000","last_fullsized":"2024-03-30T13:44:12.517441+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:52.764446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:52.764446+0000","last_clean_scrub_stamp":"2024-03-30T13:43:52.764446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:14.951666+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":86,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297305+0000","last_change":"2024-03-30T13:43:53.806905+0000","last_active":"2024-03-30T13:44:12.297305+0000","last_peered":"2024-03-30T13:44:12.297305+0000","last_clean":"2024-03-30T13:44:12.297305+0000","last_became_active":"2024-03-30T13:42:47.324421+0000","last_became_peered":"2024-03-30T13:42:47.324421+0000","last_unstale":"2024-03-30T13:44:12.297305+0000","last_undegraded":"2024-03-30T13:44:12.297305+0000","last_fullsized":"2024-03-30T13:44:12.297305+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:43:53.806768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:53.806768+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:11.831276+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":62,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433247+0000","last_change":"2024-03-30T13:44:09.040324+0000","last_active":"2024-03-30T13:44:12.433247+0000","last_peered":"2024-03-30T13:44:12.433247+0000","last_clean":"2024-03-30T13:44:12.433247+0000","last_became_active":"2024-03-30T13:42:51.334597+0000","last_became_peered":"2024-03-30T13:42:51.334597+0000","last_unstale":"2024-03-30T13:44:12.433247+0000","last_undegraded":"2024-03-30T13:44:12.433247+0000","last_fullsized":"2024-03-30T13:44:12.433247+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:09.040207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:09.040207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:25.970222+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297128+0000","last_change":"2024-03-30T13:42:49.323180+0000","last_active":"2024-03-30T13:44:12.297128+0000","last_peered":"2024-03-30T13:44:12.297128+0000","last_clean":"2024-03-30T13:44:12.297128+0000","last_became_active":"2024-03-30T13:42:49.322843+0000","last_became_peered":"2024-03-30T13:42:49.322843+0000","last_unstale":"2024-03-30T13:44:12.297128+0000","last_undegraded":"2024-03-30T13:44:12.297128+0000","last_fullsized":"2024-03-30T13:44:12.297128+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:12.351693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301890+0000","last_change":"2024-03-30T13:42:45.316182+0000","last_active":"2024-03-30T13:44:12.301890+0000","last_peered":"2024-03-30T13:44:12.301890+0000","last_clean":"2024-03-30T13:44:12.301890+0000","last_became_active":"2024-03-30T13:42:45.315864+0000","last_became_peered":"2024-03-30T13:42:45.315864+0000","last_unstale":"2024-03-30T13:44:12.301890+0000","last_undegraded":"2024-03-30T13:44:12.301890+0000","last_fullsized":"2024-03-30T13:44:12.301890+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":108,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301639+0000","last_change":"2024-03-30T13:43:54.985253+0000","last_active":"2024-03-30T13:44:12.301639+0000","last_peered":"2024-03-30T13:44:12.301639+0000","last_clean":"2024-03-30T13:44:12.301639+0000","last_became_active":"2024-03-30T13:43:12.037286+0000","last_became_peered":"2024-03-30T13:43:12.037286+0000","last_unstale":"2024-03-30T13:44:12.301639+0000","last_undegraded":"2024-03-30T13:44:12.301639+0000","last_fullsized":"2024-03-30T13:44:12.301639+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:43:54.985130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:54.985130+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:00.861139+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":60,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:16.339060+0000","last_change":"2024-03-30T13:44:16.339082+0000","last_active":"2024-03-30T13:44:16.339060+0000","last_peered":"2024-03-30T13:44:16.339060+0000","last_clean":"2024-03-30T13:44:16.339060+0000","last_became_active":"2024-03-30T13:42:51.336162+0000","last_became_peered":"2024-03-30T13:42:51.336162+0000","last_unstale":"2024-03-30T13:44:16.339060+0000","last_undegraded":"2024-03-30T13:44:16.339060+0000","last_fullsized":"2024-03-30T13:44:16.339060+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:16.339001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:16.339001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:45.272290+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:15.378483+0000","last_change":"2024-03-30T13:42:49.321501+0000","last_active":"2024-03-30T13:44:15.378483+0000","last_peered":"2024-03-30T13:44:15.378483+0000","last_clean":"2024-03-30T13:44:15.378483+0000","last_became_active":"2024-03-30T13:42:49.321203+0000","last_became_peered":"2024-03-30T13:42:49.321203+0000","last_unstale":"2024-03-30T13:44:15.378483+0000","last_undegraded":"2024-03-30T13:44:15.378483+0000","last_fullsized":"2024-03-30T13:44:15.378483+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:18.782978+0000","last_change":"2024-03-30T13:44:18.783015+0000","last_active":"2024-03-30T13:44:18.782978+0000","last_peered":"2024-03-30T13:44:18.782978+0000","last_clean":"2024-03-30T13:44:18.782978+0000","last_became_active":"2024-03-30T13:42:45.318016+0000","last_became_peered":"2024-03-30T13:42:45.318016+0000","last_unstale":"2024-03-30T13:44:18.782978+0000","last_undegraded":"2024-03-30T13:44:18.782978+0000","last_fullsized":"2024-03-30T13:44:18.782978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:18.782866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:18.782866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:32.766957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":95,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433881+0000","last_change":"2024-03-30T13:43:12.043856+0000","last_active":"2024-03-30T13:44:12.433881+0000","last_peered":"2024-03-30T13:44:12.433881+0000","last_clean":"2024-03-30T13:44:12.433881+0000","last_became_active":"2024-03-30T13:43:12.037240+0000","last_became_peered":"2024-03-30T13:43:12.037240+0000","last_unstale":"2024-03-30T13:44:12.433881+0000","last_undegraded":"2024-03-30T13:44:12.433881+0000","last_fullsized":"2024-03-30T13:44:12.433881+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:15.454436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301984+0000","last_change":"2024-03-30T13:42:51.325383+0000","last_active":"2024-03-30T13:44:12.301984+0000","last_peered":"2024-03-30T13:44:12.301984+0000","last_clean":"2024-03-30T13:44:12.301984+0000","last_became_active":"2024-03-30T13:42:51.325064+0000","last_became_peered":"2024-03-30T13:42:51.325064+0000","last_unstale":"2024-03-30T13:44:12.301984+0000","last_undegraded":"2024-03-30T13:44:12.301984+0000","last_fullsized":"2024-03-30T13:44:12.301984+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"72'8","reported_seq":94,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:15.378610+0000","last_change":"2024-03-30T13:44:14.078896+0000","last_active":"2024-03-30T13:44:15.378610+0000","last_peered":"2024-03-30T13:44:15.378610+0000","last_clean":"2024-03-30T13:44:15.378610+0000","last_became_active":"2024-03-30T13:42:49.328616+0000","last_became_peered":"2024-03-30T13:42:49.328616+0000","last_unstale":"2024-03-30T13:44:15.378610+0000","last_undegraded":"2024-03-30T13:44:15.378610+0000","last_fullsized":"2024-03-30T13:44:15.378610+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:44:14.078795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:14.078795+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:23.519694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297246+0000","last_change":"2024-03-30T13:42:45.315502+0000","last_active":"2024-03-30T13:44:12.297246+0000","last_peered":"2024-03-30T13:44:12.297246+0000","last_clean":"2024-03-30T13:44:12.297246+0000","last_became_active":"2024-03-30T13:42:45.314585+0000","last_became_peered":"2024-03-30T13:42:45.314585+0000","last_unstale":"2024-03-30T13:44:12.297246+0000","last_undegraded":"2024-03-30T13:44:12.297246+0000","last_fullsized":"2024-03-30T13:44:12.297246+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:14.270223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.429620+0000","last_change":"2024-03-30T13:43:53.236751+0000","last_active":"2024-03-30T13:44:12.429620+0000","last_peered":"2024-03-30T13:44:12.429620+0000","last_clean":"2024-03-30T13:44:12.429620+0000","last_became_active":"2024-03-30T13:42:47.501559+0000","last_became_peered":"2024-03-30T13:42:47.501559+0000","last_unstale":"2024-03-30T13:44:12.429620+0000","last_undegraded":"2024-03-30T13:44:12.429620+0000","last_fullsized":"2024-03-30T13:44:12.429620+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:43:53.236633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:53.236633+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:05.517754+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516168+0000","last_change":"2024-03-30T13:43:52.834487+0000","last_active":"2024-03-30T13:44:12.516168+0000","last_peered":"2024-03-30T13:44:12.516168+0000","last_clean":"2024-03-30T13:44:12.516168+0000","last_became_active":"2024-03-30T13:42:51.325522+0000","last_became_peered":"2024-03-30T13:42:51.325522+0000","last_unstale":"2024-03-30T13:44:12.516168+0000","last_undegraded":"2024-03-30T13:44:12.516168+0000","last_fullsized":"2024-03-30T13:44:12.516168+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:52.834381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:43:52.834381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:56.638872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.434210+0000","last_change":"2024-03-30T13:42:49.323616+0000","last_active":"2024-03-30T13:44:12.434210+0000","last_peered":"2024-03-30T13:44:12.434210+0000","last_clean":"2024-03-30T13:44:12.434210+0000","last_became_active":"2024-03-30T13:42:49.323222+0000","last_became_peered":"2024-03-30T13:42:49.323222+0000","last_unstale":"2024-03-30T13:44:12.434210+0000","last_undegraded":"2024-03-30T13:44:12.434210+0000","last_fullsized":"2024-03-30T13:44:12.434210+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:17.439407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516650+0000","last_change":"2024-03-30T13:44:05.783586+0000","last_active":"2024-03-30T13:44:12.516650+0000","last_peered":"2024-03-30T13:44:12.516650+0000","last_clean":"2024-03-30T13:44:12.516650+0000","last_became_active":"2024-03-30T13:42:45.313617+0000","last_became_peered":"2024-03-30T13:42:45.313617+0000","last_unstale":"2024-03-30T13:44:12.516650+0000","last_undegraded":"2024-03-30T13:44:12.516650+0000","last_fullsized":"2024-03-30T13:44:12.516650+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:05.783467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:05.783467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:22.470211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":84,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.298216+0000","last_change":"2024-03-30T13:44:00.374645+0000","last_active":"2024-03-30T13:44:12.298216+0000","last_peered":"2024-03-30T13:44:12.298216+0000","last_clean":"2024-03-30T13:44:12.298216+0000","last_became_active":"2024-03-30T13:42:47.501925+0000","last_became_peered":"2024-03-30T13:42:47.501925+0000","last_unstale":"2024-03-30T13:44:12.298216+0000","last_undegraded":"2024-03-30T13:44:12.298216+0000","last_fullsized":"2024-03-30T13:44:12.298216+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:44:00.374534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:00.374534+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:25.183504+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"72'1","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.517341+0000","last_change":"2024-03-30T13:44:11.831875+0000","last_active":"2024-03-30T13:44:12.517341+0000","last_peered":"2024-03-30T13:44:12.517341+0000","last_clean":"2024-03-30T13:44:12.517341+0000","last_became_active":"2024-03-30T13:42:51.372023+0000","last_became_peered":"2024-03-30T13:42:51.372023+0000","last_unstale":"2024-03-30T13:44:12.517341+0000","last_undegraded":"2024-03-30T13:44:12.517341+0000","last_fullsized":"2024-03-30T13:44:12.517341+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T13:44:11.831787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:11.831787+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:29.823530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.432336+0000","last_change":"2024-03-30T13:42:49.324555+0000","last_active":"2024-03-30T13:44:12.432336+0000","last_peered":"2024-03-30T13:44:12.432336+0000","last_clean":"2024-03-30T13:44:12.432336+0000","last_became_active":"2024-03-30T13:42:49.323924+0000","last_became_peered":"2024-03-30T13:42:49.323924+0000","last_unstale":"2024-03-30T13:44:12.432336+0000","last_undegraded":"2024-03-30T13:44:12.432336+0000","last_fullsized":"2024-03-30T13:44:12.432336+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:20.912172+0000","last_change":"2024-03-30T13:44:20.912193+0000","last_active":"2024-03-30T13:44:20.912172+0000","last_peered":"2024-03-30T13:44:20.912172+0000","last_clean":"2024-03-30T13:44:20.912172+0000","last_became_active":"2024-03-30T13:42:45.318956+0000","last_became_peered":"2024-03-30T13:42:45.318956+0000","last_unstale":"2024-03-30T13:44:20.912172+0000","last_undegraded":"2024-03-30T13:44:20.912172+0000","last_fullsized":"2024-03-30T13:44:20.912172+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:20.912113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:20.912113+0000","last_clean_scrub_stamp":"2024-03-30T13:44:20.912113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:34.648685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":79,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:21.118064+0000","last_change":"2024-03-30T13:44:21.118094+0000","last_active":"2024-03-30T13:44:21.118064+0000","last_peered":"2024-03-30T13:44:21.118064+0000","last_clean":"2024-03-30T13:44:21.118064+0000","last_became_active":"2024-03-30T13:42:47.501426+0000","last_became_peered":"2024-03-30T13:42:47.501426+0000","last_unstale":"2024-03-30T13:44:21.118064+0000","last_undegraded":"2024-03-30T13:44:21.118064+0000","last_fullsized":"2024-03-30T13:44:21.118064+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-03-30T13:44:21.117990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:21.117990+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:46.736793+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.429119+0000","last_change":"2024-03-30T13:42:51.371670+0000","last_active":"2024-03-30T13:44:12.429119+0000","last_peered":"2024-03-30T13:44:12.429119+0000","last_clean":"2024-03-30T13:44:12.429119+0000","last_became_active":"2024-03-30T13:42:51.371360+0000","last_became_peered":"2024-03-30T13:42:51.371360+0000","last_unstale":"2024-03-30T13:44:12.429119+0000","last_undegraded":"2024-03-30T13:44:12.429119+0000","last_fullsized":"2024-03-30T13:44:12.429119+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":84,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.517366+0000","last_change":"2024-03-30T13:43:45.701210+0000","last_active":"2024-03-30T13:44:12.517366+0000","last_peered":"2024-03-30T13:44:12.517366+0000","last_clean":"2024-03-30T13:44:12.517366+0000","last_became_active":"2024-03-30T13:42:45.318963+0000","last_became_peered":"2024-03-30T13:42:45.318963+0000","last_unstale":"2024-03-30T13:44:12.517366+0000","last_undegraded":"2024-03-30T13:44:12.517366+0000","last_fullsized":"2024-03-30T13:44:12.517366+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-30T13:43:45.701065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:43:45.701065+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:04.819362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":86,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.428751+0000","last_change":"2024-03-30T13:42:47.502290+0000","last_active":"2024-03-30T13:44:12.428751+0000","last_peered":"2024-03-30T13:44:12.428751+0000","last_clean":"2024-03-30T13:44:12.428751+0000","last_became_active":"2024-03-30T13:42:47.501937+0000","last_became_peered":"2024-03-30T13:42:47.501937+0000","last_unstale":"2024-03-30T13:44:12.428751+0000","last_undegraded":"2024-03-30T13:44:12.428751+0000","last_fullsized":"2024-03-30T13:44:12.428751+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"72'8","reported_seq":92,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:15.378484+0000","last_change":"2024-03-30T13:43:56.742250+0000","last_active":"2024-03-30T13:44:15.378484+0000","last_peered":"2024-03-30T13:44:15.378484+0000","last_clean":"2024-03-30T13:44:15.378484+0000","last_became_active":"2024-03-30T13:42:49.322675+0000","last_became_peered":"2024-03-30T13:42:49.322675+0000","last_unstale":"2024-03-30T13:44:15.378484+0000","last_undegraded":"2024-03-30T13:44:15.378484+0000","last_fullsized":"2024-03-30T13:44:15.378484+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:43:56.742129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:56.742129+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:17.131326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433601+0000","last_change":"2024-03-30T13:42:51.335656+0000","last_active":"2024-03-30T13:44:12.433601+0000","last_peered":"2024-03-30T13:44:12.433601+0000","last_clean":"2024-03-30T13:44:12.433601+0000","last_became_active":"2024-03-30T13:42:51.334672+0000","last_became_peered":"2024-03-30T13:42:51.334672+0000","last_unstale":"2024-03-30T13:44:12.433601+0000","last_undegraded":"2024-03-30T13:44:12.433601+0000","last_fullsized":"2024-03-30T13:44:12.433601+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516846+0000","last_change":"2024-03-30T13:44:06.790905+0000","last_active":"2024-03-30T13:44:12.516846+0000","last_peered":"2024-03-30T13:44:12.516846+0000","last_clean":"2024-03-30T13:44:12.516846+0000","last_became_active":"2024-03-30T13:42:45.312328+0000","last_became_peered":"2024-03-30T13:42:45.312328+0000","last_unstale":"2024-03-30T13:44:12.516846+0000","last_undegraded":"2024-03-30T13:44:12.516846+0000","last_fullsized":"2024-03-30T13:44:12.516846+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:06.790755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:06.790755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:10.517615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":94,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301382+0000","last_change":"2024-03-30T13:43:51.992642+0000","last_active":"2024-03-30T13:44:12.301382+0000","last_peered":"2024-03-30T13:44:12.301382+0000","last_clean":"2024-03-30T13:44:12.301382+0000","last_became_active":"2024-03-30T13:42:47.316187+0000","last_became_peered":"2024-03-30T13:42:47.316187+0000","last_unstale":"2024-03-30T13:44:12.301382+0000","last_undegraded":"2024-03-30T13:44:12.301382+0000","last_fullsized":"2024-03-30T13:44:12.301382+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-30T13:43:51.992552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:51.992552+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:09.690586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.515661+0000","last_change":"2024-03-30T13:44:10.825465+0000","last_active":"2024-03-30T13:44:12.515661+0000","last_peered":"2024-03-30T13:44:12.515661+0000","last_clean":"2024-03-30T13:44:12.515661+0000","last_became_active":"2024-03-30T13:42:49.322478+0000","last_became_peered":"2024-03-30T13:42:49.322478+0000","last_unstale":"2024-03-30T13:44:12.515661+0000","last_undegraded":"2024-03-30T13:44:12.515661+0000","last_fullsized":"2024-03-30T13:44:12.515661+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:10.825365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:10.825365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:37.651091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":60,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:13.838232+0000","last_change":"2024-03-30T13:44:13.838271+0000","last_active":"2024-03-30T13:44:13.838232+0000","last_peered":"2024-03-30T13:44:13.838232+0000","last_clean":"2024-03-30T13:44:13.838232+0000","last_became_active":"2024-03-30T13:42:51.333773+0000","last_became_peered":"2024-03-30T13:42:51.333773+0000","last_unstale":"2024-03-30T13:44:13.838232+0000","last_undegraded":"2024-03-30T13:44:13.838232+0000","last_fullsized":"2024-03-30T13:44:13.838232+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:13.838155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:13.838155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:31.168338+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.515911+0000","last_change":"2024-03-30T13:44:01.850315+0000","last_active":"2024-03-30T13:44:12.515911+0000","last_peered":"2024-03-30T13:44:12.515911+0000","last_clean":"2024-03-30T13:44:12.515911+0000","last_became_active":"2024-03-30T13:42:49.320798+0000","last_became_peered":"2024-03-30T13:42:49.320798+0000","last_unstale":"2024-03-30T13:44:12.515911+0000","last_undegraded":"2024-03-30T13:44:12.515911+0000","last_fullsized":"2024-03-30T13:44:12.515911+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:01.850229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:01.850229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:12.718080+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:19.824291+0000","last_change":"2024-03-30T13:44:19.824312+0000","last_active":"2024-03-30T13:44:19.824291+0000","last_peered":"2024-03-30T13:44:19.824291+0000","last_clean":"2024-03-30T13:44:19.824291+0000","last_became_active":"2024-03-30T13:42:45.315687+0000","last_became_peered":"2024-03-30T13:42:45.315687+0000","last_unstale":"2024-03-30T13:44:19.824291+0000","last_undegraded":"2024-03-30T13:44:19.824291+0000","last_fullsized":"2024-03-30T13:44:19.824291+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:19.824218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:19.824218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:31.303512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"72'19","reported_seq":91,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516763+0000","last_change":"2024-03-30T13:42:47.501349+0000","last_active":"2024-03-30T13:44:12.516763+0000","last_peered":"2024-03-30T13:44:12.516763+0000","last_clean":"2024-03-30T13:44:12.516763+0000","last_became_active":"2024-03-30T13:42:47.500898+0000","last_became_peered":"2024-03-30T13:42:47.500898+0000","last_unstale":"2024-03-30T13:44:12.516763+0000","last_undegraded":"2024-03-30T13:44:12.516763+0000","last_fullsized":"2024-03-30T13:44:12.516763+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.296659+0000","last_change":"2024-03-30T13:42:51.330739+0000","last_active":"2024-03-30T13:44:12.296659+0000","last_peered":"2024-03-30T13:44:12.296659+0000","last_clean":"2024-03-30T13:44:12.296659+0000","last_became_active":"2024-03-30T13:42:51.330403+0000","last_became_peered":"2024-03-30T13:42:51.330403+0000","last_unstale":"2024-03-30T13:44:12.296659+0000","last_undegraded":"2024-03-30T13:44:12.296659+0000","last_fullsized":"2024-03-30T13:44:12.296659+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:19.083830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.515774+0000","last_change":"2024-03-30T13:43:51.840857+0000","last_active":"2024-03-30T13:44:12.515774+0000","last_peered":"2024-03-30T13:44:12.515774+0000","last_clean":"2024-03-30T13:44:12.515774+0000","last_became_active":"2024-03-30T13:42:49.324022+0000","last_became_peered":"2024-03-30T13:42:49.324022+0000","last_unstale":"2024-03-30T13:44:12.515774+0000","last_undegraded":"2024-03-30T13:44:12.515774+0000","last_fullsized":"2024-03-30T13:44:12.515774+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:51.840740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:51.840740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:00.887228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.429762+0000","last_change":"2024-03-30T13:42:45.315058+0000","last_active":"2024-03-30T13:44:12.429762+0000","last_peered":"2024-03-30T13:44:12.429762+0000","last_clean":"2024-03-30T13:44:12.429762+0000","last_became_active":"2024-03-30T13:42:45.314767+0000","last_became_peered":"2024-03-30T13:42:45.314767+0000","last_unstale":"2024-03-30T13:44:12.429762+0000","last_undegraded":"2024-03-30T13:44:12.429762+0000","last_fullsized":"2024-03-30T13:44:12.429762+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":100,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297694+0000","last_change":"2024-03-30T13:43:57.361388+0000","last_active":"2024-03-30T13:44:12.297694+0000","last_peered":"2024-03-30T13:44:12.297694+0000","last_clean":"2024-03-30T13:44:12.297694+0000","last_became_active":"2024-03-30T13:43:11.446379+0000","last_became_peered":"2024-03-30T13:43:11.446379+0000","last_unstale":"2024-03-30T13:44:12.297694+0000","last_undegraded":"2024-03-30T13:44:12.297694+0000","last_fullsized":"2024-03-30T13:44:12.297694+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:43:57.361300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:43:57.361300+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:13.159418+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":62,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.434014+0000","last_change":"2024-03-30T13:44:06.994178+0000","last_active":"2024-03-30T13:44:12.434014+0000","last_peered":"2024-03-30T13:44:12.434014+0000","last_clean":"2024-03-30T13:44:12.434014+0000","last_became_active":"2024-03-30T13:42:51.331054+0000","last_became_peered":"2024-03-30T13:42:51.331054+0000","last_unstale":"2024-03-30T13:44:12.434014+0000","last_undegraded":"2024-03-30T13:44:12.434014+0000","last_fullsized":"2024-03-30T13:44:12.434014+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:06.994097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:06.994097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:17.075287+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":76,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.428584+0000","last_change":"2024-03-30T13:42:48.314967+0000","last_active":"2024-03-30T13:44:12.428584+0000","last_peered":"2024-03-30T13:44:12.428584+0000","last_clean":"2024-03-30T13:44:12.428584+0000","last_became_active":"2024-03-30T13:42:44.300789+0000","last_became_peered":"2024-03-30T13:42:44.300789+0000","last_unstale":"2024-03-30T13:44:12.428584+0000","last_undegraded":"2024-03-30T13:44:12.428584+0000","last_fullsized":"2024-03-30T13:44:12.428584+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:42:43.281063+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021837800000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301336+0000","last_change":"2024-03-30T13:42:45.310512+0000","last_active":"2024-03-30T13:44:12.301336+0000","last_peered":"2024-03-30T13:44:12.301336+0000","last_clean":"2024-03-30T13:44:12.301336+0000","last_became_active":"2024-03-30T13:42:45.310209+0000","last_became_peered":"2024-03-30T13:42:45.310209+0000","last_unstale":"2024-03-30T13:44:12.301336+0000","last_undegraded":"2024-03-30T13:44:12.301336+0000","last_fullsized":"2024-03-30T13:44:12.301336+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"72'28","reported_seq":138,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:18.748464+0000","last_change":"2024-03-30T13:44:18.748464+0000","last_active":"2024-03-30T13:44:18.748464+0000","last_peered":"2024-03-30T13:44:18.748464+0000","last_clean":"2024-03-30T13:44:18.748464+0000","last_became_active":"2024-03-30T13:42:47.323845+0000","last_became_peered":"2024-03-30T13:42:47.323845+0000","last_unstale":"2024-03-30T13:44:18.748464+0000","last_undegraded":"2024-03-30T13:44:18.748464+0000","last_fullsized":"2024-03-30T13:44:18.748464+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-03-30T13:44:18.748330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:18.748330+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:47.821731+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297721+0000","last_change":"2024-03-30T13:43:54.326967+0000","last_active":"2024-03-30T13:44:12.297721+0000","last_peered":"2024-03-30T13:44:12.297721+0000","last_clean":"2024-03-30T13:44:12.297721+0000","last_became_active":"2024-03-30T13:42:49.322154+0000","last_became_peered":"2024-03-30T13:42:49.322154+0000","last_unstale":"2024-03-30T13:44:12.297721+0000","last_undegraded":"2024-03-30T13:44:12.297721+0000","last_fullsized":"2024-03-30T13:44:12.297721+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:54.326829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:54.326829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:09.771943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.517701+0000","last_change":"2024-03-30T13:42:51.372579+0000","last_active":"2024-03-30T13:44:12.517701+0000","last_peered":"2024-03-30T13:44:12.517701+0000","last_clean":"2024-03-30T13:44:12.517701+0000","last_became_active":"2024-03-30T13:42:51.372320+0000","last_became_peered":"2024-03-30T13:42:51.372320+0000","last_unstale":"2024-03-30T13:44:12.517701+0000","last_undegraded":"2024-03-30T13:44:12.517701+0000","last_fullsized":"2024-03-30T13:44:12.517701+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":74,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.296508+0000","last_change":"2024-03-30T13:43:59.810299+0000","last_active":"2024-03-30T13:44:12.296508+0000","last_peered":"2024-03-30T13:44:12.296508+0000","last_clean":"2024-03-30T13:44:12.296508+0000","last_became_active":"2024-03-30T13:42:45.314771+0000","last_became_peered":"2024-03-30T13:42:45.314771+0000","last_unstale":"2024-03-30T13:44:12.296508+0000","last_undegraded":"2024-03-30T13:44:12.296508+0000","last_fullsized":"2024-03-30T13:44:12.296508+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:59.810189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:59.810189+0000","last_clean_scrub_stamp":"2024-03-30T13:43:59.810189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:06.406345+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":84,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.515538+0000","last_change":"2024-03-30T13:43:57.866600+0000","last_active":"2024-03-30T13:44:12.515538+0000","last_peered":"2024-03-30T13:44:12.515538+0000","last_clean":"2024-03-30T13:44:12.515538+0000","last_became_active":"2024-03-30T13:42:44.302344+0000","last_became_peered":"2024-03-30T13:42:44.302344+0000","last_unstale":"2024-03-30T13:44:12.515538+0000","last_undegraded":"2024-03-30T13:44:12.515538+0000","last_fullsized":"2024-03-30T13:44:12.515538+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:57.866519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:43:57.866519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:59.119000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001401518,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"72'13","reported_seq":97,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:14.782493+0000","last_change":"2024-03-30T13:44:14.782524+0000","last_active":"2024-03-30T13:44:14.782493+0000","last_peered":"2024-03-30T13:44:14.782493+0000","last_clean":"2024-03-30T13:44:14.782493+0000","last_became_active":"2024-03-30T13:42:47.320541+0000","last_became_peered":"2024-03-30T13:42:47.320541+0000","last_unstale":"2024-03-30T13:44:14.782493+0000","last_undegraded":"2024-03-30T13:44:14.782493+0000","last_fullsized":"2024-03-30T13:44:14.782493+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'13","last_scrub_stamp":"2024-03-30T13:44:14.782432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:14.782432+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:22.236946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:13.762418+0000","last_change":"2024-03-30T13:44:13.762445+0000","last_active":"2024-03-30T13:44:13.762418+0000","last_peered":"2024-03-30T13:44:13.762418+0000","last_clean":"2024-03-30T13:44:13.762418+0000","last_became_active":"2024-03-30T13:42:49.319826+0000","last_became_peered":"2024-03-30T13:42:49.319826+0000","last_unstale":"2024-03-30T13:44:13.762418+0000","last_undegraded":"2024-03-30T13:44:13.762418+0000","last_fullsized":"2024-03-30T13:44:13.762418+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:13.762345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:13.762345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:42.481193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433250+0000","last_change":"2024-03-30T13:44:12.433273+0000","last_active":"2024-03-30T13:44:12.433250+0000","last_peered":"2024-03-30T13:44:12.433250+0000","last_clean":"2024-03-30T13:44:12.433250+0000","last_became_active":"2024-03-30T13:42:51.331081+0000","last_became_peered":"2024-03-30T13:42:51.331081+0000","last_unstale":"2024-03-30T13:44:12.433250+0000","last_undegraded":"2024-03-30T13:44:12.433250+0000","last_fullsized":"2024-03-30T13:44:12.433250+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:12.433166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:12.433166+0000","last_clean_scrub_stamp":"2024-03-30T13:44:12.433166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:23.843912+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":317,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.517136+0000","last_change":"2024-03-30T13:43:23.836532+0000","last_active":"2024-03-30T13:44:12.517136+0000","last_peered":"2024-03-30T13:44:12.517136+0000","last_clean":"2024-03-30T13:44:12.517136+0000","last_became_active":"2024-03-30T13:42:34.988246+0000","last_became_peered":"2024-03-30T13:42:34.988246+0000","last_unstale":"2024-03-30T13:44:12.517136+0000","last_undegraded":"2024-03-30T13:44:12.517136+0000","last_fullsized":"2024-03-30T13:44:12.517136+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-03-30T13:43:23.836417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:43:23.836417+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:30.177755+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":74,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433566+0000","last_change":"2024-03-30T13:44:09.995041+0000","last_active":"2024-03-30T13:44:12.433566+0000","last_peered":"2024-03-30T13:44:12.433566+0000","last_clean":"2024-03-30T13:44:12.433566+0000","last_became_active":"2024-03-30T13:42:45.309954+0000","last_became_peered":"2024-03-30T13:42:45.309954+0000","last_unstale":"2024-03-30T13:44:12.433566+0000","last_undegraded":"2024-03-30T13:44:12.433566+0000","last_fullsized":"2024-03-30T13:44:12.433566+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:09.994923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:09.994923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:23.612678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":110,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:15.378810+0000","last_change":"2024-03-30T13:44:13.820800+0000","last_active":"2024-03-30T13:44:15.378810+0000","last_peered":"2024-03-30T13:44:15.378810+0000","last_clean":"2024-03-30T13:44:15.378810+0000","last_became_active":"2024-03-30T13:42:47.502338+0000","last_became_peered":"2024-03-30T13:42:47.502338+0000","last_unstale":"2024-03-30T13:44:15.378810+0000","last_undegraded":"2024-03-30T13:44:15.378810+0000","last_fullsized":"2024-03-30T13:44:15.378810+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'16","last_scrub_stamp":"2024-03-30T13:44:13.820689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:13.820689+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:18.638625+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":56,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516436+0000","last_change":"2024-03-30T13:42:49.329181+0000","last_active":"2024-03-30T13:44:12.516436+0000","last_peered":"2024-03-30T13:44:12.516436+0000","last_clean":"2024-03-30T13:44:12.516436+0000","last_became_active":"2024-03-30T13:42:49.328861+0000","last_became_peered":"2024-03-30T13:42:49.328861+0000","last_unstale":"2024-03-30T13:44:12.516436+0000","last_undegraded":"2024-03-30T13:44:12.516436+0000","last_fullsized":"2024-03-30T13:44:12.516436+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":62,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:20.789584+0000","last_change":"2024-03-30T13:44:20.789622+0000","last_active":"2024-03-30T13:44:20.789584+0000","last_peered":"2024-03-30T13:44:20.789584+0000","last_clean":"2024-03-30T13:44:20.789584+0000","last_became_active":"2024-03-30T13:42:51.335188+0000","last_became_peered":"2024-03-30T13:42:51.335188+0000","last_unstale":"2024-03-30T13:44:20.789584+0000","last_undegraded":"2024-03-30T13:44:20.789584+0000","last_fullsized":"2024-03-30T13:44:20.789584+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:20.789512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:20.789512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:25.402381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.296890+0000","last_change":"2024-03-30T13:44:08.397886+0000","last_active":"2024-03-30T13:44:12.296890+0000","last_peered":"2024-03-30T13:44:12.296890+0000","last_clean":"2024-03-30T13:44:12.296890+0000","last_became_active":"2024-03-30T13:42:45.307905+0000","last_became_peered":"2024-03-30T13:42:45.307905+0000","last_unstale":"2024-03-30T13:44:12.296890+0000","last_undegraded":"2024-03-30T13:44:12.296890+0000","last_fullsized":"2024-03-30T13:44:12.296890+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:08.397798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:08.397798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:35.437418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":167,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:21.827591+0000","last_change":"2024-03-30T13:42:48.313590+0000","last_active":"2024-03-30T13:44:21.827591+0000","last_peered":"2024-03-30T13:44:21.827591+0000","last_clean":"2024-03-30T13:44:21.827591+0000","last_became_active":"2024-03-30T13:42:44.302343+0000","last_became_peered":"2024-03-30T13:42:44.302343+0000","last_unstale":"2024-03-30T13:44:21.827591+0000","last_undegraded":"2024-03-30T13:44:21.827591+0000","last_fullsized":"2024-03-30T13:44:21.827591+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:42:43.281063+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00126391,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":91,"num_read_kb":86,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"93'18","reported_seq":92,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:21.409717+0000","last_change":"2024-03-30T13:44:21.409752+0000","last_active":"2024-03-30T13:44:21.409717+0000","last_peered":"2024-03-30T13:44:21.409717+0000","last_clean":"2024-03-30T13:44:21.409717+0000","last_became_active":"2024-03-30T13:42:47.319155+0000","last_became_peered":"2024-03-30T13:42:47.319155+0000","last_unstale":"2024-03-30T13:44:21.409717+0000","last_undegraded":"2024-03-30T13:44:21.409717+0000","last_fullsized":"2024-03-30T13:44:21.409717+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'18","last_scrub_stamp":"2024-03-30T13:44:21.409658+0000","last_deep_scrub":"93'18","last_deep_scrub_stamp":"2024-03-30T13:44:21.409658+0000","last_clean_scrub_stamp":"2024-03-30T13:44:21.409658+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:34.093907+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.430011+0000","last_change":"2024-03-30T13:42:49.322962+0000","last_active":"2024-03-30T13:44:12.430011+0000","last_peered":"2024-03-30T13:44:12.430011+0000","last_clean":"2024-03-30T13:44:12.430011+0000","last_became_active":"2024-03-30T13:42:49.322700+0000","last_became_peered":"2024-03-30T13:42:49.322700+0000","last_unstale":"2024-03-30T13:44:12.430011+0000","last_undegraded":"2024-03-30T13:44:12.430011+0000","last_fullsized":"2024-03-30T13:44:12.430011+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":62,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.296495+0000","last_change":"2024-03-30T13:44:05.809643+0000","last_active":"2024-03-30T13:44:12.296495+0000","last_peered":"2024-03-30T13:44:12.296495+0000","last_clean":"2024-03-30T13:44:12.296495+0000","last_became_active":"2024-03-30T13:42:51.371202+0000","last_became_peered":"2024-03-30T13:42:51.371202+0000","last_unstale":"2024-03-30T13:44:12.296495+0000","last_undegraded":"2024-03-30T13:44:12.296495+0000","last_fullsized":"2024-03-30T13:44:12.296495+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:05.809548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:05.809548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:18.651258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":74,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297582+0000","last_change":"2024-03-30T13:44:06.800362+0000","last_active":"2024-03-30T13:44:12.297582+0000","last_peered":"2024-03-30T13:44:12.297582+0000","last_clean":"2024-03-30T13:44:12.297582+0000","last_became_active":"2024-03-30T13:42:45.310134+0000","last_became_peered":"2024-03-30T13:42:45.310134+0000","last_unstale":"2024-03-30T13:44:12.297582+0000","last_undegraded":"2024-03-30T13:44:12.297582+0000","last_fullsized":"2024-03-30T13:44:12.297582+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:06.800290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:06.800290+0000","last_clean_scrub_stamp":"2024-03-30T13:44:06.800290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:30.870285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":104,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297955+0000","last_change":"2024-03-30T13:44:11.366827+0000","last_active":"2024-03-30T13:44:12.297955+0000","last_peered":"2024-03-30T13:44:12.297955+0000","last_clean":"2024-03-30T13:44:12.297955+0000","last_became_active":"2024-03-30T13:42:47.501272+0000","last_became_peered":"2024-03-30T13:42:47.501272+0000","last_unstale":"2024-03-30T13:44:12.297955+0000","last_undegraded":"2024-03-30T13:44:12.297955+0000","last_fullsized":"2024-03-30T13:44:12.297955+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-30T13:44:11.366726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:11.366726+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:18.908561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:15.794516+0000","last_change":"2024-03-30T13:44:15.794564+0000","last_active":"2024-03-30T13:44:15.794516+0000","last_peered":"2024-03-30T13:44:15.794516+0000","last_clean":"2024-03-30T13:44:15.794516+0000","last_became_active":"2024-03-30T13:42:49.321520+0000","last_became_peered":"2024-03-30T13:42:49.321520+0000","last_unstale":"2024-03-30T13:44:15.794516+0000","last_undegraded":"2024-03-30T13:44:15.794516+0000","last_fullsized":"2024-03-30T13:44:15.794516+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:15.794436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:15.794436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:33.056162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":60,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.296928+0000","last_change":"2024-03-30T13:44:04.440757+0000","last_active":"2024-03-30T13:44:12.296928+0000","last_peered":"2024-03-30T13:44:12.296928+0000","last_clean":"2024-03-30T13:44:12.296928+0000","last_became_active":"2024-03-30T13:42:51.336892+0000","last_became_peered":"2024-03-30T13:42:51.336892+0000","last_unstale":"2024-03-30T13:44:12.296928+0000","last_undegraded":"2024-03-30T13:44:12.296928+0000","last_fullsized":"2024-03-30T13:44:12.296928+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:04.440623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:04.440623+0000","last_clean_scrub_stamp":"2024-03-30T13:44:04.440623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:20.104438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.429473+0000","last_change":"2024-03-30T13:43:57.299723+0000","last_active":"2024-03-30T13:44:12.429473+0000","last_peered":"2024-03-30T13:44:12.429473+0000","last_clean":"2024-03-30T13:44:12.429473+0000","last_became_active":"2024-03-30T13:42:45.317136+0000","last_became_peered":"2024-03-30T13:42:45.317136+0000","last_unstale":"2024-03-30T13:44:12.429473+0000","last_undegraded":"2024-03-30T13:44:12.429473+0000","last_fullsized":"2024-03-30T13:44:12.429473+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:57.299618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:57.299618+0000","last_clean_scrub_stamp":"2024-03-30T13:43:57.299618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:09.120401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.765004+0000","last_change":"2024-03-30T13:44:12.765034+0000","last_active":"2024-03-30T13:44:12.765004+0000","last_peered":"2024-03-30T13:44:12.765004+0000","last_clean":"2024-03-30T13:44:12.765004+0000","last_became_active":"2024-03-30T13:42:47.321433+0000","last_became_peered":"2024-03-30T13:42:47.321433+0000","last_unstale":"2024-03-30T13:44:12.765004+0000","last_undegraded":"2024-03-30T13:44:12.765004+0000","last_fullsized":"2024-03-30T13:44:12.765004+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-30T13:44:12.764944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:12.764944+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:32.186703+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":84,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:15.378369+0000","last_change":"2024-03-30T13:42:49.326702+0000","last_active":"2024-03-30T13:44:15.378369+0000","last_peered":"2024-03-30T13:44:15.378369+0000","last_clean":"2024-03-30T13:44:15.378369+0000","last_became_active":"2024-03-30T13:42:49.325533+0000","last_became_peered":"2024-03-30T13:42:49.325533+0000","last_unstale":"2024-03-30T13:44:15.378369+0000","last_undegraded":"2024-03-30T13:44:15.378369+0000","last_fullsized":"2024-03-30T13:44:15.378369+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":60,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:22.774319+0000","last_change":"2024-03-30T13:44:22.774340+0000","last_active":"2024-03-30T13:44:22.774319+0000","last_peered":"2024-03-30T13:44:22.774319+0000","last_clean":"2024-03-30T13:44:22.774319+0000","last_became_active":"2024-03-30T13:42:51.374437+0000","last_became_peered":"2024-03-30T13:42:51.374437+0000","last_unstale":"2024-03-30T13:44:22.774319+0000","last_undegraded":"2024-03-30T13:44:22.774319+0000","last_fullsized":"2024-03-30T13:44:22.774319+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:22.774262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:22.774262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:52.251774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297968+0000","last_change":"2024-03-30T13:44:07.397507+0000","last_active":"2024-03-30T13:44:12.297968+0000","last_peered":"2024-03-30T13:44:12.297968+0000","last_clean":"2024-03-30T13:44:12.297968+0000","last_became_active":"2024-03-30T13:42:45.309886+0000","last_became_peered":"2024-03-30T13:42:45.309886+0000","last_unstale":"2024-03-30T13:44:12.297968+0000","last_undegraded":"2024-03-30T13:44:12.297968+0000","last_fullsized":"2024-03-30T13:44:12.297968+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:44:07.397409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:07.397409+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:07.807875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"91'21","reported_seq":99,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301240+0000","last_change":"2024-03-30T13:44:05.917185+0000","last_active":"2024-03-30T13:44:12.301240+0000","last_peered":"2024-03-30T13:44:12.301240+0000","last_clean":"2024-03-30T13:44:12.301240+0000","last_became_active":"2024-03-30T13:42:47.314184+0000","last_became_peered":"2024-03-30T13:42:47.314184+0000","last_unstale":"2024-03-30T13:44:12.301240+0000","last_undegraded":"2024-03-30T13:44:12.301240+0000","last_fullsized":"2024-03-30T13:44:12.301240+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'21","last_scrub_stamp":"2024-03-30T13:44:05.917064+0000","last_deep_scrub":"91'21","last_deep_scrub_stamp":"2024-03-30T13:44:05.917064+0000","last_clean_scrub_stamp":"2024-03-30T13:44:05.917064+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:07.613278+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":94,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:20.069564+0000","last_change":"2024-03-30T13:44:20.069564+0000","last_active":"2024-03-30T13:44:20.069564+0000","last_peered":"2024-03-30T13:44:20.069564+0000","last_clean":"2024-03-30T13:44:20.069564+0000","last_became_active":"2024-03-30T13:42:49.321423+0000","last_became_peered":"2024-03-30T13:42:49.321423+0000","last_unstale":"2024-03-30T13:44:20.069564+0000","last_undegraded":"2024-03-30T13:44:20.069564+0000","last_fullsized":"2024-03-30T13:44:20.069564+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:44:20.069451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:20.069451+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:38.391906+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301273+0000","last_change":"2024-03-30T13:44:03.968532+0000","last_active":"2024-03-30T13:44:12.301273+0000","last_peered":"2024-03-30T13:44:12.301273+0000","last_clean":"2024-03-30T13:44:12.301273+0000","last_became_active":"2024-03-30T13:42:51.327003+0000","last_became_peered":"2024-03-30T13:42:51.327003+0000","last_unstale":"2024-03-30T13:44:12.301273+0000","last_undegraded":"2024-03-30T13:44:12.301273+0000","last_fullsized":"2024-03-30T13:44:12.301273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:03.968435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:03.968435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:12.556459+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.434157+0000","last_change":"2024-03-30T13:42:45.314830+0000","last_active":"2024-03-30T13:44:12.434157+0000","last_peered":"2024-03-30T13:44:12.434157+0000","last_clean":"2024-03-30T13:44:12.434157+0000","last_became_active":"2024-03-30T13:42:45.314533+0000","last_became_peered":"2024-03-30T13:42:45.314533+0000","last_unstale":"2024-03-30T13:44:12.434157+0000","last_undegraded":"2024-03-30T13:44:12.434157+0000","last_fullsized":"2024-03-30T13:44:12.434157+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":102,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433495+0000","last_change":"2024-03-30T13:44:06.016635+0000","last_active":"2024-03-30T13:44:12.433495+0000","last_peered":"2024-03-30T13:44:12.433495+0000","last_clean":"2024-03-30T13:44:12.433495+0000","last_became_active":"2024-03-30T13:42:47.502708+0000","last_became_peered":"2024-03-30T13:42:47.502708+0000","last_unstale":"2024-03-30T13:44:12.433495+0000","last_undegraded":"2024-03-30T13:44:12.433495+0000","last_fullsized":"2024-03-30T13:44:12.433495+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-03-30T13:44:06.016537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:06.016537+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:09.822245+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301452+0000","last_change":"2024-03-30T13:42:49.324517+0000","last_active":"2024-03-30T13:44:12.301452+0000","last_peered":"2024-03-30T13:44:12.301452+0000","last_clean":"2024-03-30T13:44:12.301452+0000","last_became_active":"2024-03-30T13:42:49.324022+0000","last_became_peered":"2024-03-30T13:42:49.324022+0000","last_unstale":"2024-03-30T13:44:12.301452+0000","last_undegraded":"2024-03-30T13:44:12.301452+0000","last_fullsized":"2024-03-30T13:44:12.301452+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.429244+0000","last_change":"2024-03-30T13:42:51.329317+0000","last_active":"2024-03-30T13:44:12.429244+0000","last_peered":"2024-03-30T13:44:12.429244+0000","last_clean":"2024-03-30T13:44:12.429244+0000","last_became_active":"2024-03-30T13:42:51.329026+0000","last_became_peered":"2024-03-30T13:42:51.329026+0000","last_unstale":"2024-03-30T13:44:12.429244+0000","last_undegraded":"2024-03-30T13:44:12.429244+0000","last_fullsized":"2024-03-30T13:44:12.429244+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":95,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:15.378495+0000","last_change":"2024-03-30T13:43:58.942488+0000","last_active":"2024-03-30T13:44:15.378495+0000","last_peered":"2024-03-30T13:44:15.378495+0000","last_clean":"2024-03-30T13:44:15.378495+0000","last_became_active":"2024-03-30T13:42:49.321790+0000","last_became_peered":"2024-03-30T13:42:49.321790+0000","last_unstale":"2024-03-30T13:44:15.378495+0000","last_undegraded":"2024-03-30T13:44:15.378495+0000","last_fullsized":"2024-03-30T13:44:15.378495+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:43:58.942311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:58.942311+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:17.003904+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433398+0000","last_change":"2024-03-30T13:42:45.315666+0000","last_active":"2024-03-30T13:44:12.433398+0000","last_peered":"2024-03-30T13:44:12.433398+0000","last_clean":"2024-03-30T13:44:12.433398+0000","last_became_active":"2024-03-30T13:42:45.315319+0000","last_became_peered":"2024-03-30T13:42:45.315319+0000","last_unstale":"2024-03-30T13:44:12.433398+0000","last_undegraded":"2024-03-30T13:44:12.433398+0000","last_fullsized":"2024-03-30T13:44:12.433398+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:42:44.287804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":87,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.298878+0000","last_change":"2024-03-30T13:42:47.324692+0000","last_active":"2024-03-30T13:44:12.298878+0000","last_peered":"2024-03-30T13:44:12.298878+0000","last_clean":"2024-03-30T13:44:12.298878+0000","last_became_active":"2024-03-30T13:42:47.324350+0000","last_became_peered":"2024-03-30T13:42:47.324350+0000","last_unstale":"2024-03-30T13:44:12.298878+0000","last_undegraded":"2024-03-30T13:44:12.298878+0000","last_fullsized":"2024-03-30T13:44:12.298878+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433753+0000","last_change":"2024-03-30T13:42:51.371742+0000","last_active":"2024-03-30T13:44:12.433753+0000","last_peered":"2024-03-30T13:44:12.433753+0000","last_clean":"2024-03-30T13:44:12.433753+0000","last_became_active":"2024-03-30T13:42:51.371369+0000","last_became_peered":"2024-03-30T13:42:51.371369+0000","last_unstale":"2024-03-30T13:44:12.433753+0000","last_undegraded":"2024-03-30T13:44:12.433753+0000","last_fullsized":"2024-03-30T13:44:12.433753+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:19.941761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.429932+0000","last_change":"2024-03-30T13:42:49.323595+0000","last_active":"2024-03-30T13:44:12.429932+0000","last_peered":"2024-03-30T13:44:12.429932+0000","last_clean":"2024-03-30T13:44:12.429932+0000","last_became_active":"2024-03-30T13:42:49.323319+0000","last_became_peered":"2024-03-30T13:42:49.323319+0000","last_unstale":"2024-03-30T13:44:12.429932+0000","last_undegraded":"2024-03-30T13:44:12.429932+0000","last_fullsized":"2024-03-30T13:44:12.429932+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:42:48.304004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":75,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301067+0000","last_change":"2024-03-30T13:43:49.990189+0000","last_active":"2024-03-30T13:44:12.301067+0000","last_peered":"2024-03-30T13:44:12.301067+0000","last_clean":"2024-03-30T13:44:12.301067+0000","last_became_active":"2024-03-30T13:42:45.315018+0000","last_became_peered":"2024-03-30T13:42:45.315018+0000","last_unstale":"2024-03-30T13:44:12.301067+0000","last_undegraded":"2024-03-30T13:44:12.301067+0000","last_fullsized":"2024-03-30T13:44:12.301067+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:49.990065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:43:49.990065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:44:58.482640+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"72'11","reported_seq":90,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:15.841689+0000","last_change":"2024-03-30T13:44:15.841741+0000","last_active":"2024-03-30T13:44:15.841689+0000","last_peered":"2024-03-30T13:44:15.841689+0000","last_clean":"2024-03-30T13:44:15.841689+0000","last_became_active":"2024-03-30T13:42:47.315667+0000","last_became_peered":"2024-03-30T13:42:47.315667+0000","last_unstale":"2024-03-30T13:44:15.841689+0000","last_undegraded":"2024-03-30T13:44:15.841689+0000","last_fullsized":"2024-03-30T13:44:15.841689+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:44:15.841626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:44:15.841626+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:23.388522+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:22.672801+0000","last_change":"2024-03-30T13:44:22.672823+0000","last_active":"2024-03-30T13:44:22.672801+0000","last_peered":"2024-03-30T13:44:22.672801+0000","last_clean":"2024-03-30T13:44:22.672801+0000","last_became_active":"2024-03-30T13:42:51.331372+0000","last_became_peered":"2024-03-30T13:42:51.331372+0000","last_unstale":"2024-03-30T13:44:22.672801+0000","last_undegraded":"2024-03-30T13:44:22.672801+0000","last_fullsized":"2024-03-30T13:44:22.672801+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:22.672741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:44:22.672741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:47.507729+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.297784+0000","last_change":"2024-03-30T13:44:02.833988+0000","last_active":"2024-03-30T13:44:12.297784+0000","last_peered":"2024-03-30T13:44:12.297784+0000","last_clean":"2024-03-30T13:44:12.297784+0000","last_became_active":"2024-03-30T13:42:49.321360+0000","last_became_peered":"2024-03-30T13:42:49.321360+0000","last_unstale":"2024-03-30T13:44:12.297784+0000","last_undegraded":"2024-03-30T13:44:12.297784+0000","last_fullsized":"2024-03-30T13:44:12.297784+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:02.833909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:44:02.833909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:31.595404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.516625+0000","last_change":"2024-03-30T13:44:08.809025+0000","last_active":"2024-03-30T13:44:12.516625+0000","last_peered":"2024-03-30T13:44:12.516625+0000","last_clean":"2024-03-30T13:44:12.516625+0000","last_became_active":"2024-03-30T13:42:45.315251+0000","last_became_peered":"2024-03-30T13:42:45.315251+0000","last_unstale":"2024-03-30T13:44:12.516625+0000","last_undegraded":"2024-03-30T13:44:12.516625+0000","last_fullsized":"2024-03-30T13:44:12.516625+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:44:08.808863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:44:08.808863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:23.892469+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":91,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301852+0000","last_change":"2024-03-30T13:42:47.316447+0000","last_active":"2024-03-30T13:44:12.301852+0000","last_peered":"2024-03-30T13:44:12.301852+0000","last_clean":"2024-03-30T13:44:12.301852+0000","last_became_active":"2024-03-30T13:42:47.315495+0000","last_became_peered":"2024-03-30T13:42:47.315495+0000","last_unstale":"2024-03-30T13:44:12.301852+0000","last_undegraded":"2024-03-30T13:44:12.301852+0000","last_fullsized":"2024-03-30T13:44:12.301852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.302147+0000","last_change":"2024-03-30T13:42:51.325672+0000","last_active":"2024-03-30T13:44:12.302147+0000","last_peered":"2024-03-30T13:44:12.302147+0000","last_clean":"2024-03-30T13:44:12.302147+0000","last_became_active":"2024-03-30T13:42:51.325389+0000","last_became_peered":"2024-03-30T13:42:51.325389+0000","last_unstale":"2024-03-30T13:44:12.302147+0000","last_undegraded":"2024-03-30T13:44:12.302147+0000","last_fullsized":"2024-03-30T13:44:12.302147+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:42:50.307175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"72'8","reported_seq":98,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:15.378419+0000","last_change":"2024-03-30T13:43:50.850857+0000","last_active":"2024-03-30T13:44:15.378419+0000","last_peered":"2024-03-30T13:44:15.378419+0000","last_clean":"2024-03-30T13:44:15.378419+0000","last_became_active":"2024-03-30T13:42:49.323371+0000","last_became_peered":"2024-03-30T13:42:49.323371+0000","last_unstale":"2024-03-30T13:44:15.378419+0000","last_undegraded":"2024-03-30T13:44:15.378419+0000","last_fullsized":"2024-03-30T13:44:15.378419+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:43:50.850759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:43:50.850759+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:01.654480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":74,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.433897+0000","last_change":"2024-03-30T13:43:50.050389+0000","last_active":"2024-03-30T13:44:12.433897+0000","last_peered":"2024-03-30T13:44:12.433897+0000","last_clean":"2024-03-30T13:44:12.433897+0000","last_became_active":"2024-03-30T13:42:45.309265+0000","last_became_peered":"2024-03-30T13:42:45.309265+0000","last_unstale":"2024-03-30T13:44:12.433897+0000","last_undegraded":"2024-03-30T13:44:12.433897+0000","last_fullsized":"2024-03-30T13:44:12.433897+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:43:50.050303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:43:50.050303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:45:05.509528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":78,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T13:44:12.301816+0000","last_change":"2024-03-30T13:42:47.313353+0000","last_active":"2024-03-30T13:44:12.301816+0000","last_peered":"2024-03-30T13:44:12.301816+0000","last_clean":"2024-03-30T13:44:12.301816+0000","last_became_active":"2024-03-30T13:42:47.313026+0000","last_became_peered":"2024-03-30T13:42:47.313026+0000","last_unstale":"2024-03-30T13:44:12.301816+0000","last_undegraded":"2024-03-30T13:44:12.301816+0000","last_fullsized":"2024-03-30T13:44:12.301816+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:42:46.298397+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":413,"ondisk_log_size":413,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":86,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103193,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32704,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712448,"statfs":{"total":95995035648,"available":95961546752,"internally_reserved":0,"allocated":1564672,"data_stored":1055292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397533,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32684,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712468,"statfs":{"total":95995035648,"available":95961567232,"internally_reserved":0,"allocated":1544192,"data_stored":1054240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":465822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32708,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712444,"statfs":{"total":95995035648,"available":95961542656,"internally_reserved":0,"allocated":1568768,"data_stored":1055896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":464959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182445,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":464750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953511,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":464875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691874,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":465568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:44:25.883 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-30T13:44:25.883 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd dump 2024-03-30T13:44:26.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:25 smithi043 ceph-mon[29963]: 4.a scrub starts 2024-03-30T13:44:26.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:25 smithi043 ceph-mon[29963]: 4.a scrub ok 2024-03-30T13:44:26.096 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:25 smithi043 ceph-mon[42099]: 4.a scrub starts 2024-03-30T13:44:26.096 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:25 smithi043 ceph-mon[42099]: 4.a scrub ok 2024-03-30T13:44:26.097 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:25 smithi043 bash[81708]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-30T13:44:26.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:25 smithi113 ceph-mon[34341]: 4.a scrub starts 2024-03-30T13:44:26.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:25 smithi113 ceph-mon[34341]: 4.a scrub ok 2024-03-30T13:44:26.345 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:44:26.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:44:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:44:26.399 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 bash[81708]: Writing manifest to image destination 2024-03-30T13:44:26.745 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 podman[81708]: 2024-03-30 13:44:26.369046971 +0000 UTC m=+2.988163391 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-30T13:44:26.745 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 podman[81708]: 2024-03-30 13:44:26.500907332 +0000 UTC m=+3.120023747 container create 4ef41d68b90e4be012ea6f03bc40a8c8d27ce6f87a9ce58f173d40e9d255ad48 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-30T13:44:27.006 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-mon[29963]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:27.006 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-mon[29963]: 6.18 scrub starts 2024-03-30T13:44:27.006 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-mon[29963]: 6.18 scrub ok 2024-03-30T13:44:27.006 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-mon[29963]: 3.c scrub starts 2024-03-30T13:44:27.006 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-mon[29963]: 3.c scrub ok 2024-03-30T13:44:27.007 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-mon[42099]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:27.007 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-mon[42099]: 6.18 scrub starts 2024-03-30T13:44:27.007 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-mon[42099]: 6.18 scrub ok 2024-03-30T13:44:27.007 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-mon[42099]: 3.c scrub starts 2024-03-30T13:44:27.007 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-mon[42099]: 3.c scrub ok 2024-03-30T13:44:27.007 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 podman[81708]: 2024-03-30 13:44:26.921910343 +0000 UTC m=+3.541026771 container init 4ef41d68b90e4be012ea6f03bc40a8c8d27ce6f87a9ce58f173d40e9d255ad48 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-30T13:44:27.007 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 podman[81708]: 2024-03-30 13:44:26.927290033 +0000 UTC m=+3.546406448 container start 4ef41d68b90e4be012ea6f03bc40a8c8d27ce6f87a9ce58f173d40e9d255ad48 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-30T13:44:27.008 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.931Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-30T13:44:27.008 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.931Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-30T13:44:27.008 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.931Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-30T13:44:27.008 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.931Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-30T13:44:27.008 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.931Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-30T13:44:27.008 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.931Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-30T13:44:27.008 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-30T13:44:27.008 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=arp 2024-03-30T13:44:27.009 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-30T13:44:27.009 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-30T13:44:27.009 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-30T13:44:27.009 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-30T13:44:27.009 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-30T13:44:27.009 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-30T13:44:27.009 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-30T13:44:27.009 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-30T13:44:27.009 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=edac 2024-03-30T13:44:27.009 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-30T13:44:27.009 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-30T13:44:27.009 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-30T13:44:27.009 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-30T13:44:27.010 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-30T13:44:27.010 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-30T13:44:27.010 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-30T13:44:27.010 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-30T13:44:27.010 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-30T13:44:27.010 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-30T13:44:27.011 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-30T13:44:27.012 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-30T13:44:27.012 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-30T13:44:27.012 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-30T13:44:27.012 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-30T13:44:27.012 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-30T13:44:27.012 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=os 2024-03-30T13:44:27.012 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-30T13:44:27.013 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-30T13:44:27.013 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-30T13:44:27.013 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-30T13:44:27.013 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-30T13:44:27.013 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-30T13:44:27.013 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-30T13:44:27.013 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=stat 2024-03-30T13:44:27.013 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-30T13:44:27.013 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-30T13:44:27.013 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-30T13:44:27.014 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=time 2024-03-30T13:44:27.014 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-30T13:44:27.014 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=uname 2024-03-30T13:44:27.014 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-30T13:44:27.014 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-30T13:44:27.014 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-30T13:44:27.014 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-30T13:44:27.014 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a[81880]: ts=2024-03-30T13:44:26.932Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-30T13:44:27.015 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:27 smithi043 bash[81708]: 4ef41d68b90e4be012ea6f03bc40a8c8d27ce6f87a9ce58f173d40e9d255ad48 2024-03-30T13:44:27.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:26 smithi113 ceph-mon[34341]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:27.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:26 smithi113 ceph-mon[34341]: 6.18 scrub starts 2024-03-30T13:44:27.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:26 smithi113 ceph-mon[34341]: 6.18 scrub ok 2024-03-30T13:44:27.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:26 smithi113 ceph-mon[34341]: 3.c scrub starts 2024-03-30T13:44:27.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:26 smithi113 ceph-mon[34341]: 3.c scrub ok 2024-03-30T13:44:27.399 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 13:44:27 smithi043 systemd[1]: Started Ceph node-exporter.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T13:44:28.318 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:28 smithi113 ceph-mon[34341]: 4.d scrub starts 2024-03-30T13:44:28.318 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:28 smithi113 ceph-mon[34341]: 4.d scrub ok 2024-03-30T13:44:28.318 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:28.319 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:28.319 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:28.319 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:28 smithi113 ceph-mon[34341]: Deploying daemon node-exporter.b on smithi113 2024-03-30T13:44:28.319 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:28 smithi113 ceph-mon[34341]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:28.319 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:28 smithi113 ceph-mon[34341]: osdmap e95: 8 total, 8 up, 8 in 2024-03-30T13:44:28.319 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:44:28.319 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:28 smithi113 ceph-mon[34341]: 6.5 scrub starts 2024-03-30T13:44:28.319 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:28 smithi113 ceph-mon[34341]: 6.5 scrub ok 2024-03-30T13:44:28.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[29963]: 4.d scrub starts 2024-03-30T13:44:28.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[29963]: 4.d scrub ok 2024-03-30T13:44:28.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:28.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:28.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:28.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[29963]: Deploying daemon node-exporter.b on smithi113 2024-03-30T13:44:28.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[29963]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:28.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[29963]: osdmap e95: 8 total, 8 up, 8 in 2024-03-30T13:44:28.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:44:28.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[29963]: 6.5 scrub starts 2024-03-30T13:44:28.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[29963]: 6.5 scrub ok 2024-03-30T13:44:28.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[42099]: 4.d scrub starts 2024-03-30T13:44:28.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[42099]: 4.d scrub ok 2024-03-30T13:44:28.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:28.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:28.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:28.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[42099]: Deploying daemon node-exporter.b on smithi113 2024-03-30T13:44:28.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[42099]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:28.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[42099]: osdmap e95: 8 total, 8 up, 8 in 2024-03-30T13:44:28.329 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:44:28.329 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[42099]: 6.5 scrub starts 2024-03-30T13:44:28.329 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:28 smithi043 ceph-mon[42099]: 6.5 scrub ok 2024-03-30T13:44:28.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:44:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:44:28.751 INFO:teuthology.orchestra.run.smithi043.stdout:epoch 95 2024-03-30T13:44:28.751 INFO:teuthology.orchestra.run.smithi043.stdout:fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T13:44:28.751 INFO:teuthology.orchestra.run.smithi043.stdout:created 2024-03-30T13:37:07.346020+0000 2024-03-30T13:44:28.751 INFO:teuthology.orchestra.run.smithi043.stdout:modified 2024-03-30T13:44:27.297169+0000 2024-03-30T13:44:28.751 INFO:teuthology.orchestra.run.smithi043.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-30T13:44:28.751 INFO:teuthology.orchestra.run.smithi043.stdout:crush_version 18 2024-03-30T13:44:28.751 INFO:teuthology.orchestra.run.smithi043.stdout:full_ratio 0.95 2024-03-30T13:44:28.751 INFO:teuthology.orchestra.run.smithi043.stdout:backfillfull_ratio 0.9 2024-03-30T13:44:28.752 INFO:teuthology.orchestra.run.smithi043.stdout:nearfull_ratio 0.85 2024-03-30T13:44:28.752 INFO:teuthology.orchestra.run.smithi043.stdout:require_min_compat_client luminous 2024-03-30T13:44:28.752 INFO:teuthology.orchestra.run.smithi043.stdout:min_compat_client luminous 2024-03-30T13:44:28.752 INFO:teuthology.orchestra.run.smithi043.stdout:require_osd_release squid 2024-03-30T13:44:28.752 INFO:teuthology.orchestra.run.smithi043.stdout:stretch_mode_enabled false 2024-03-30T13:44:28.752 INFO:teuthology.orchestra.run.smithi043.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-03-30T13:44:28.752 INFO:teuthology.orchestra.run.smithi043.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 68 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-03-30T13:44:28.752 INFO:teuthology.orchestra.run.smithi043.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 66 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-03-30T13:44:28.752 INFO:teuthology.orchestra.run.smithi043.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 68 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-03-30T13:44:28.752 INFO:teuthology.orchestra.run.smithi043.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 70 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-03-30T13:44:28.752 INFO:teuthology.orchestra.run.smithi043.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 72 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-03-30T13:44:28.752 INFO:teuthology.orchestra.run.smithi043.stdout:max_osd 8 2024-03-30T13:44:28.752 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0 up in weight 1 up_from 8 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.43:6802/1144592572,v1:172.21.15.43:6803/1144592572] [v2:172.21.15.43:6804/1144592572,v1:172.21.15.43:6805/1144592572] exists,up 94ea083e-6a27-41db-83af-2f7d3cae1956 2024-03-30T13:44:28.753 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1 up in weight 1 up_from 13 up_thru 70 down_at 0 last_clean_interval [0,0) [v2:172.21.15.43:6810/3901165928,v1:172.21.15.43:6811/3901165928] [v2:172.21.15.43:6812/3901165928,v1:172.21.15.43:6813/3901165928] exists,up 8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a 2024-03-30T13:44:28.753 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2 up in weight 1 up_from 18 up_thru 70 down_at 0 last_clean_interval [0,0) [v2:172.21.15.43:6818/3415037523,v1:172.21.15.43:6819/3415037523] [v2:172.21.15.43:6820/3415037523,v1:172.21.15.43:6821/3415037523] exists,up 6db6782e-5889-443e-bc6c-df47a1f93bc8 2024-03-30T13:44:28.753 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3 up in weight 1 up_from 25 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.43:6826/2933589376,v1:172.21.15.43:6827/2933589376] [v2:172.21.15.43:6828/2933589376,v1:172.21.15.43:6829/2933589376] exists,up 7048ff46-2cba-4d18-9835-95ecbddb2bb6 2024-03-30T13:44:28.753 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4 up in weight 1 up_from 32 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.113:6800/1945833081,v1:172.21.15.113:6801/1945833081] [v2:172.21.15.113:6802/1945833081,v1:172.21.15.113:6803/1945833081] exists,up 42da30e6-9eec-4ed5-8c73-9ce78aa8ff63 2024-03-30T13:44:28.753 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5 up in weight 1 up_from 40 up_thru 70 down_at 0 last_clean_interval [0,0) [v2:172.21.15.113:6808/3426360325,v1:172.21.15.113:6809/3426360325] [v2:172.21.15.113:6810/3426360325,v1:172.21.15.113:6811/3426360325] exists,up 99a11b81-0df7-4b1d-946a-4297ba62e263 2024-03-30T13:44:28.753 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6 up in weight 1 up_from 49 up_thru 68 down_at 0 last_clean_interval [0,0) [v2:172.21.15.113:6816/1527017698,v1:172.21.15.113:6817/1527017698] [v2:172.21.15.113:6818/1527017698,v1:172.21.15.113:6819/1527017698] exists,up cafd0e26-78e9-4164-a43a-96ca8382fefe 2024-03-30T13:44:28.753 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7 up in weight 1 up_from 58 up_thru 70 down_at 0 last_clean_interval [0,0) [v2:172.21.15.113:6824/3182476015,v1:172.21.15.113:6825/3182476015] [v2:172.21.15.113:6826/3182476015,v1:172.21.15.113:6827/3182476015] exists,up e9d59662-9132-48f4-9e95-842a3cfbd79f 2024-03-30T13:44:28.753 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 4.b [1,5] 2024-03-30T13:44:28.753 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 4.13 [1,2] 2024-03-30T13:44:28.753 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 4.14 [1,2] 2024-03-30T13:44:28.753 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/538304234 expires 2024-03-31T13:43:27.253365+0000 2024-03-30T13:44:28.753 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:6801/3411723337 expires 2024-03-31T13:37:41.733515+0000 2024-03-30T13:44:28.753 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/3656823679 expires 2024-03-31T13:37:41.733515+0000 2024-03-30T13:44:28.753 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/2618817647 expires 2024-03-31T13:38:09.321740+0000 2024-03-30T13:44:28.753 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:6800/3411723337 expires 2024-03-31T13:37:41.733515+0000 2024-03-30T13:44:28.754 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/1206218784 expires 2024-03-31T13:37:41.733515+0000 2024-03-30T13:44:28.754 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/886631486 expires 2024-03-31T13:37:41.733515+0000 2024-03-30T13:44:28.754 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:6800/1339921781 expires 2024-03-31T13:38:09.321740+0000 2024-03-30T13:44:28.754 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:6801/1339921781 expires 2024-03-31T13:38:09.321740+0000 2024-03-30T13:44:28.754 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/2794494528 expires 2024-03-31T13:38:09.321740+0000 2024-03-30T13:44:28.754 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/564868778 expires 2024-03-31T13:38:09.321740+0000 2024-03-30T13:44:28.754 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:6800/3033707884 expires 2024-03-31T13:43:27.253365+0000 2024-03-30T13:44:28.754 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/3539398453 expires 2024-03-31T13:43:27.253365+0000 2024-03-30T13:44:28.754 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/2414364740 expires 2024-03-31T13:43:27.253365+0000 2024-03-30T13:44:28.754 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:6801/3033707884 expires 2024-03-31T13:43:27.253365+0000 2024-03-30T13:44:28.754 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/3923597429 expires 2024-03-31T13:43:27.253365+0000 2024-03-30T13:44:28.754 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/2524269526 expires 2024-03-31T13:43:27.253365+0000 2024-03-30T13:44:29.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:29 smithi043 ceph-mon[42099]: 3.18 scrub starts 2024-03-30T13:44:29.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:29 smithi043 ceph-mon[42099]: 3.18 scrub ok 2024-03-30T13:44:29.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:29 smithi043 ceph-mon[42099]: 6.10 deep-scrub starts 2024-03-30T13:44:29.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:29 smithi043 ceph-mon[42099]: 6.10 deep-scrub ok 2024-03-30T13:44:29.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:29 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:44:29.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:29 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/210944650' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-30T13:44:29.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:29 smithi043 ceph-mon[42099]: 4.f deep-scrub starts 2024-03-30T13:44:29.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:29 smithi043 ceph-mon[42099]: 4.f deep-scrub ok 2024-03-30T13:44:29.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:29 smithi043 ceph-mon[29963]: 3.18 scrub starts 2024-03-30T13:44:29.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:29 smithi043 ceph-mon[29963]: 3.18 scrub ok 2024-03-30T13:44:29.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:29 smithi043 ceph-mon[29963]: 6.10 deep-scrub starts 2024-03-30T13:44:29.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:29 smithi043 ceph-mon[29963]: 6.10 deep-scrub ok 2024-03-30T13:44:29.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:29 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:44:29.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:29 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/210944650' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-30T13:44:29.401 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:29 smithi043 ceph-mon[29963]: 4.f deep-scrub starts 2024-03-30T13:44:29.401 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:29 smithi043 ceph-mon[29963]: 4.f deep-scrub ok 2024-03-30T13:44:29.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:29 smithi113 ceph-mon[34341]: 3.18 scrub starts 2024-03-30T13:44:29.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:29 smithi113 ceph-mon[34341]: 3.18 scrub ok 2024-03-30T13:44:29.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:29 smithi113 ceph-mon[34341]: 6.10 deep-scrub starts 2024-03-30T13:44:29.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:29 smithi113 ceph-mon[34341]: 6.10 deep-scrub ok 2024-03-30T13:44:29.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:29 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:44:29.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:29 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/210944650' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-30T13:44:29.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:29 smithi113 ceph-mon[34341]: 4.f deep-scrub starts 2024-03-30T13:44:29.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:29 smithi113 ceph-mon[34341]: 4.f deep-scrub ok 2024-03-30T13:44:29.473 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:29 smithi113 bash[68862]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-30T13:44:29.721 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.43:6802/1144592572,v1:172.21.15.43:6803/1144592572] [v2:172.21.15.43:6804/1144592572,v1:172.21.15.43:6805/1144592572] exists,up 94ea083e-6a27-41db-83af-2f7d3cae1956', 'osd.1 up in weight 1 up_from 13 up_thru 70 down_at 0 last_clean_interval [0,0) [v2:172.21.15.43:6810/3901165928,v1:172.21.15.43:6811/3901165928] [v2:172.21.15.43:6812/3901165928,v1:172.21.15.43:6813/3901165928] exists,up 8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a', 'osd.2 up in weight 1 up_from 18 up_thru 70 down_at 0 last_clean_interval [0,0) [v2:172.21.15.43:6818/3415037523,v1:172.21.15.43:6819/3415037523] [v2:172.21.15.43:6820/3415037523,v1:172.21.15.43:6821/3415037523] exists,up 6db6782e-5889-443e-bc6c-df47a1f93bc8', 'osd.3 up in weight 1 up_from 25 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.43:6826/2933589376,v1:172.21.15.43:6827/2933589376] [v2:172.21.15.43:6828/2933589376,v1:172.21.15.43:6829/2933589376] exists,up 7048ff46-2cba-4d18-9835-95ecbddb2bb6', 'osd.4 up in weight 1 up_from 32 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.113:6800/1945833081,v1:172.21.15.113:6801/1945833081] [v2:172.21.15.113:6802/1945833081,v1:172.21.15.113:6803/1945833081] exists,up 42da30e6-9eec-4ed5-8c73-9ce78aa8ff63', 'osd.5 up in weight 1 up_from 40 up_thru 70 down_at 0 last_clean_interval [0,0) [v2:172.21.15.113:6808/3426360325,v1:172.21.15.113:6809/3426360325] [v2:172.21.15.113:6810/3426360325,v1:172.21.15.113:6811/3426360325] exists,up 99a11b81-0df7-4b1d-946a-4297ba62e263', 'osd.6 up in weight 1 up_from 49 up_thru 68 down_at 0 last_clean_interval [0,0) [v2:172.21.15.113:6816/1527017698,v1:172.21.15.113:6817/1527017698] [v2:172.21.15.113:6818/1527017698,v1:172.21.15.113:6819/1527017698] exists,up cafd0e26-78e9-4164-a43a-96ca8382fefe', 'osd.7 up in weight 1 up_from 58 up_thru 70 down_at 0 last_clean_interval [0,0) [v2:172.21.15.113:6824/3182476015,v1:172.21.15.113:6825/3182476015] [v2:172.21.15.113:6826/3182476015,v1:172.21.15.113:6827/3182476015] exists,up e9d59662-9132-48f4-9e95-842a3cfbd79f'] 2024-03-30T13:44:29.722 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon mon.a config show 2024-03-30T13:44:29.972 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:29 smithi113 bash[68862]: Getting image source signatures 2024-03-30T13:44:29.972 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:29 smithi113 bash[68862]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-30T13:44:29.972 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:29 smithi113 bash[68862]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-30T13:44:29.972 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:29 smithi113 bash[68862]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-30T13:44:29.974 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:44:30.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:30 smithi043 ceph-mon[29963]: 5.16 scrub starts 2024-03-30T13:44:30.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:30 smithi043 ceph-mon[29963]: 5.16 scrub ok 2024-03-30T13:44:30.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:30 smithi043 ceph-mon[29963]: 4.19 deep-scrub starts 2024-03-30T13:44:30.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:30 smithi043 ceph-mon[29963]: 4.19 deep-scrub ok 2024-03-30T13:44:30.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:30 smithi043 ceph-mon[29963]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:30.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:30 smithi043 ceph-mon[42099]: 5.16 scrub starts 2024-03-30T13:44:30.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:30 smithi043 ceph-mon[42099]: 5.16 scrub ok 2024-03-30T13:44:30.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:30 smithi043 ceph-mon[42099]: 4.19 deep-scrub starts 2024-03-30T13:44:30.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:30 smithi043 ceph-mon[42099]: 4.19 deep-scrub ok 2024-03-30T13:44:30.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:30 smithi043 ceph-mon[42099]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:30.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:30 smithi113 ceph-mon[34341]: 5.16 scrub starts 2024-03-30T13:44:30.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:30 smithi113 ceph-mon[34341]: 5.16 scrub ok 2024-03-30T13:44:30.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:30 smithi113 ceph-mon[34341]: 4.19 deep-scrub starts 2024-03-30T13:44:30.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:30 smithi113 ceph-mon[34341]: 4.19 deep-scrub ok 2024-03-30T13:44:30.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:30 smithi113 ceph-mon[34341]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:31.554 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:44:31.554 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mon.a", 2024-03-30T13:44:31.554 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster": "ceph", 2024-03-30T13:44:31.554 INFO:teuthology.orchestra.run.smithi043.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-30T13:44:31.554 INFO:teuthology.orchestra.run.smithi043.stdout: "admin_socket_mode": "", 2024-03-30T13:44:31.554 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-03-30T13:44:31.554 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_client_required": "cephx, none", 2024-03-30T13:44:31.554 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_cluster_required": "cephx", 2024-03-30T13:44:31.554 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_debug": "true", 2024-03-30T13:44:31.554 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-30T13:44:31.555 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-30T13:44:31.555 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_service_required": "cephx", 2024-03-30T13:44:31.555 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-30T13:44:31.555 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_supported": "", 2024-03-30T13:44:31.555 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_aio": "true", 2024-03-30T13:44:31.555 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-30T13:44:31.555 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_aio_poll_ms": "250", 2024-03-30T13:44:31.555 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_aio_reap_max": "16", 2024-03-30T13:44:31.555 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_async_discard": "false", 2024-03-30T13:44:31.555 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_block_size": "4096", 2024-03-30T13:44:31.555 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_debug_aio": "false", 2024-03-30T13:44:31.555 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-30T13:44:31.556 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-30T13:44:31.556 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_debug_inflight_ios": "false", 2024-03-30T13:44:31.556 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_enable_discard": "false", 2024-03-30T13:44:31.556 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_flock_retry": "3", 2024-03-30T13:44:31.556 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-30T13:44:31.556 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_inject_crash": "0", 2024-03-30T13:44:31.556 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-30T13:44:31.556 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_ioring": "false", 2024-03-30T13:44:31.556 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_ioring_hipri": "false", 2024-03-30T13:44:31.556 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-30T13:44:31.556 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-30T13:44:31.556 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-30T13:44:31.557 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-30T13:44:31.557 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_type": "", 2024-03-30T13:44:31.557 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_alloc_size": "1048576", 2024-03-30T13:44:31.557 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_allocator": "hybrid", 2024-03-30T13:44:31.557 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_buffered_io": "true", 2024-03-30T13:44:31.557 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_check_for_zeros": "false", 2024-03-30T13:44:31.557 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-30T13:44:31.557 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-30T13:44:31.557 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_compact_log_sync": "false", 2024-03-30T13:44:31.557 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-30T13:44:31.557 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-30T13:44:31.558 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-30T13:44:31.558 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-30T13:44:31.558 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_max_log_runway": "4194304", 2024-03-30T13:44:31.558 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_max_prefetch": "1048576", 2024-03-30T13:44:31.558 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_min_flush_size": "524288", 2024-03-30T13:44:31.558 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_min_log_runway": "1048576", 2024-03-30T13:44:31.558 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_replay_recovery": "false", 2024-03-30T13:44:31.558 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-30T13:44:31.558 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-30T13:44:31.558 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_sync_write": "false", 2024-03-30T13:44:31.558 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-30T13:44:31.558 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-30T13:44:31.558 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-30T13:44:31.559 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_allocation_from_file": "true", 2024-03-30T13:44:31.559 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_allocator": "hybrid", 2024-03-30T13:44:31.559 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-30T13:44:31.559 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-30T13:44:31.559 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-30T13:44:31.559 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-30T13:44:31.559 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-30T13:44:31.559 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-30T13:44:31.559 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-30T13:44:31.559 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_block_create": "true", 2024-03-30T13:44:31.559 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_block_db_create": "false", 2024-03-30T13:44:31.559 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_block_db_path": "", 2024-03-30T13:44:31.559 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_block_db_size": "0", 2024-03-30T13:44:31.559 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_block_path": "", 2024-03-30T13:44:31.560 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_block_preallocate_file": "false", 2024-03-30T13:44:31.560 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_block_size": "107374182400", 2024-03-30T13:44:31.560 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_block_wal_create": "false", 2024-03-30T13:44:31.560 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_block_wal_path": "", 2024-03-30T13:44:31.560 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_block_wal_size": "100663296", 2024-03-30T13:44:31.560 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_bluefs": "true", 2024-03-30T13:44:31.560 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-30T13:44:31.560 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-30T13:44:31.560 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-30T13:44:31.560 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-30T13:44:31.560 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-30T13:44:31.560 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-30T13:44:31.560 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-30T13:44:31.560 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-30T13:44:31.561 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_autotune": "true", 2024-03-30T13:44:31.561 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-30T13:44:31.561 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-30T13:44:31.561 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-30T13:44:31.561 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-30T13:44:31.561 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_size": "0", 2024-03-30T13:44:31.561 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-30T13:44:31.561 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-30T13:44:31.561 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-30T13:44:31.561 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-30T13:44:31.561 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_type": "2q", 2024-03-30T13:44:31.561 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-30T13:44:31.561 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_clone_cow": "true", 2024-03-30T13:44:31.562 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-30T13:44:31.562 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-30T13:44:31.562 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-30T13:44:31.562 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-30T13:44:31.562 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-30T13:44:31.562 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-30T13:44:31.562 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-30T13:44:31.562 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_compression_mode": "none", 2024-03-30T13:44:31.562 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-30T13:44:31.562 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_csum_type": "crc32c", 2024-03-30T13:44:31.562 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-30T13:44:31.562 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_freelist": "false", 2024-03-30T13:44:31.562 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-30T13:44:31.563 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-30T13:44:31.563 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-30T13:44:31.563 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-30T13:44:31.563 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-30T13:44:31.563 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-30T13:44:31.563 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-30T13:44:31.563 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-30T13:44:31.563 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-30T13:44:31.563 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-30T13:44:31.563 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-30T13:44:31.563 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-30T13:44:31.564 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-30T13:44:31.564 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_small_allocations": "0", 2024-03-30T13:44:31.564 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-30T13:44:31.564 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_default_buffered_read": "true", 2024-03-30T13:44:31.564 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_default_buffered_write": "false", 2024-03-30T13:44:31.564 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-30T13:44:31.564 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-30T13:44:31.564 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-30T13:44:31.564 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-30T13:44:31.564 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-30T13:44:31.564 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-30T13:44:31.564 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-30T13:44:31.564 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-30T13:44:31.564 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-30T13:44:31.565 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fail_eio": "false", 2024-03-30T13:44:31.565 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-30T13:44:31.565 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-30T13:44:31.565 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-30T13:44:31.565 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-30T13:44:31.565 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-30T13:44:31.565 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-30T13:44:31.565 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_on_mount": "false", 2024-03-30T13:44:31.565 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-30T13:44:31.565 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_on_umount": "false", 2024-03-30T13:44:31.565 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-30T13:44:31.565 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-30T13:44:31.565 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-30T13:44:31.565 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-30T13:44:31.566 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-30T13:44:31.566 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-30T13:44:31.566 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-30T13:44:31.566 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-30T13:44:31.566 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_ignore_data_csum": "false", 2024-03-30T13:44:31.566 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-30T13:44:31.566 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-30T13:44:31.566 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-30T13:44:31.566 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-30T13:44:31.566 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_log_op_age": "5.000000", 2024-03-30T13:44:31.566 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_max_alloc_size": "0", 2024-03-30T13:44:31.566 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_max_blob_size": "0", 2024-03-30T13:44:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-30T13:44:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-30T13:44:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-30T13:44:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_max_deferred_txc": "32", 2024-03-30T13:44:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_min_alloc_size": "0", 2024-03-30T13:44:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-30T13:44:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-30T13:44:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_nid_prealloc": "1024", 2024-03-30T13:44:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-30T13:44:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-30T13:44:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-30T13:44:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-30T13:44:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_retry_disk_reads": "3", 2024-03-30T13:44:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_rocksdb_cf": "true", 2024-03-30T13:44:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-30T13:44:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-30T13:44:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-30T13:44:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-30T13:44:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-30T13:44:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-30T13:44:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_spdk_mem": "512", 2024-03-30T13:44:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-30T13:44:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-30T13:44:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-30T13:44:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-30T13:44:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-30T13:44:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-30T13:44:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-30T13:44:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_tracing": "false", 2024-03-30T13:44:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-30T13:44:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-30T13:44:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-30T13:44:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-30T13:44:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-30T13:44:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-30T13:44:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-30T13:44:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-30T13:44:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-30T13:44:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_zero_block_detection": "false", 2024-03-30T13:44:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-30T13:44:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-30T13:44:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-30T13:44:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-30T13:44:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-30T13:44:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-30T13:44:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-30T13:44:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-03-30T13:44:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-30T13:44:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-30T13:44:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-30T13:44:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-30T13:44:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-30T13:44:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-30T13:44:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: "cephx_cluster_require_signatures": "false", 2024-03-30T13:44:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: "cephx_cluster_require_version": "2", 2024-03-30T13:44:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: "cephx_require_signatures": "false", 2024-03-30T13:44:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: "cephx_require_version": "2", 2024-03-30T13:44:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: "cephx_service_require_signatures": "false", 2024-03-30T13:44:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: "cephx_service_require_version": "2", 2024-03-30T13:44:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: "cephx_sign_messages": "true", 2024-03-30T13:44:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: "chdir": "", 2024-03-30T13:44:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: "client_acl_type": "", 2024-03-30T13:44:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: "client_asio_thread_count": "2", 2024-03-30T13:44:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: "client_cache_mid": "0.750000", 2024-03-30T13:44:31.572 INFO:teuthology.orchestra.run.smithi043.stdout: "client_cache_size": "16384", 2024-03-30T13:44:31.572 INFO:teuthology.orchestra.run.smithi043.stdout: "client_caps_release_delay": "5", 2024-03-30T13:44:31.572 INFO:teuthology.orchestra.run.smithi043.stdout: "client_check_pool_perm": "true", 2024-03-30T13:44:31.572 INFO:teuthology.orchestra.run.smithi043.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-30T13:44:31.572 INFO:teuthology.orchestra.run.smithi043.stdout: "client_debug_force_sync_read": "false", 2024-03-30T13:44:31.572 INFO:teuthology.orchestra.run.smithi043.stdout: "client_debug_getattr_caps": "false", 2024-03-30T13:44:31.572 INFO:teuthology.orchestra.run.smithi043.stdout: "client_debug_inject_tick_delay": "0", 2024-03-30T13:44:31.572 INFO:teuthology.orchestra.run.smithi043.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-30T13:44:31.572 INFO:teuthology.orchestra.run.smithi043.stdout: "client_die_on_failed_remount": "false", 2024-03-30T13:44:31.572 INFO:teuthology.orchestra.run.smithi043.stdout: "client_dirsize_rbytes": "true", 2024-03-30T13:44:31.572 INFO:teuthology.orchestra.run.smithi043.stdout: "client_force_lazyio": "false", 2024-03-30T13:44:31.572 INFO:teuthology.orchestra.run.smithi043.stdout: "client_fs": "", 2024-03-30T13:44:31.572 INFO:teuthology.orchestra.run.smithi043.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-30T13:44:31.572 INFO:teuthology.orchestra.run.smithi043.stdout: "client_inject_release_failure": "false", 2024-03-30T13:44:31.573 INFO:teuthology.orchestra.run.smithi043.stdout: "client_max_inline_size": "4096", 2024-03-30T13:44:31.573 INFO:teuthology.orchestra.run.smithi043.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-30T13:44:31.573 INFO:teuthology.orchestra.run.smithi043.stdout: "client_mds_namespace": "", 2024-03-30T13:44:31.573 INFO:teuthology.orchestra.run.smithi043.stdout: "client_metadata": "", 2024-03-30T13:44:31.573 INFO:teuthology.orchestra.run.smithi043.stdout: "client_mount_gid": "-1", 2024-03-30T13:44:31.573 INFO:teuthology.orchestra.run.smithi043.stdout: "client_mount_timeout": "300", 2024-03-30T13:44:31.573 INFO:teuthology.orchestra.run.smithi043.stdout: "client_mount_uid": "-1", 2024-03-30T13:44:31.573 INFO:teuthology.orchestra.run.smithi043.stdout: "client_mountpoint": "/", 2024-03-30T13:44:31.573 INFO:teuthology.orchestra.run.smithi043.stdout: "client_notify_timeout": "10", 2024-03-30T13:44:31.573 INFO:teuthology.orchestra.run.smithi043.stdout: "client_oc": "true", 2024-03-30T13:44:31.573 INFO:teuthology.orchestra.run.smithi043.stdout: "client_oc_max_dirty": "104857600", 2024-03-30T13:44:31.573 INFO:teuthology.orchestra.run.smithi043.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-30T13:44:31.573 INFO:teuthology.orchestra.run.smithi043.stdout: "client_oc_max_objects": "1000", 2024-03-30T13:44:31.573 INFO:teuthology.orchestra.run.smithi043.stdout: "client_oc_size": "209715200", 2024-03-30T13:44:31.574 INFO:teuthology.orchestra.run.smithi043.stdout: "client_oc_target_dirty": "8388608", 2024-03-30T13:44:31.574 INFO:teuthology.orchestra.run.smithi043.stdout: "client_permissions": "true", 2024-03-30T13:44:31.574 INFO:teuthology.orchestra.run.smithi043.stdout: "client_quota": "true", 2024-03-30T13:44:31.574 INFO:teuthology.orchestra.run.smithi043.stdout: "client_quota_df": "true", 2024-03-30T13:44:31.574 INFO:teuthology.orchestra.run.smithi043.stdout: "client_readahead_max_bytes": "0", 2024-03-30T13:44:31.574 INFO:teuthology.orchestra.run.smithi043.stdout: "client_readahead_max_periods": "4", 2024-03-30T13:44:31.574 INFO:teuthology.orchestra.run.smithi043.stdout: "client_readahead_min": "131072", 2024-03-30T13:44:31.574 INFO:teuthology.orchestra.run.smithi043.stdout: "client_reconnect_stale": "false", 2024-03-30T13:44:31.574 INFO:teuthology.orchestra.run.smithi043.stdout: "client_shutdown_timeout": "30", 2024-03-30T13:44:31.574 INFO:teuthology.orchestra.run.smithi043.stdout: "client_snapdir": ".snap", 2024-03-30T13:44:31.574 INFO:teuthology.orchestra.run.smithi043.stdout: "client_tick_interval": "1", 2024-03-30T13:44:31.574 INFO:teuthology.orchestra.run.smithi043.stdout: "client_trace": "", 2024-03-30T13:44:31.575 INFO:teuthology.orchestra.run.smithi043.stdout: "client_try_dentry_invalidate": "false", 2024-03-30T13:44:31.575 INFO:teuthology.orchestra.run.smithi043.stdout: "client_use_faked_inos": "false", 2024-03-30T13:44:31.575 INFO:teuthology.orchestra.run.smithi043.stdout: "client_use_random_mds": "false", 2024-03-30T13:44:31.575 INFO:teuthology.orchestra.run.smithi043.stdout: "clog_to_graylog": "false", 2024-03-30T13:44:31.575 INFO:teuthology.orchestra.run.smithi043.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-30T13:44:31.575 INFO:teuthology.orchestra.run.smithi043.stdout: "clog_to_graylog_port": "12201", 2024-03-30T13:44:31.575 INFO:teuthology.orchestra.run.smithi043.stdout: "clog_to_monitors": "default=true", 2024-03-30T13:44:31.575 INFO:teuthology.orchestra.run.smithi043.stdout: "clog_to_syslog": "false", 2024-03-30T13:44:31.575 INFO:teuthology.orchestra.run.smithi043.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-30T13:44:31.575 INFO:teuthology.orchestra.run.smithi043.stdout: "clog_to_syslog_level": "info", 2024-03-30T13:44:31.575 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "-", 2024-03-30T13:44:31.575 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_network": "", 2024-03-30T13:44:31.575 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_network_interface": "", 2024-03-30T13:44:31.575 INFO:teuthology.orchestra.run.smithi043.stdout: "compressor_zlib_isal": "false", 2024-03-30T13:44:31.576 INFO:teuthology.orchestra.run.smithi043.stdout: "compressor_zlib_level": "5", 2024-03-30T13:44:31.576 INFO:teuthology.orchestra.run.smithi043.stdout: "compressor_zlib_winsize": "-15", 2024-03-30T13:44:31.576 INFO:teuthology.orchestra.run.smithi043.stdout: "compressor_zstd_level": "1", 2024-03-30T13:44:31.576 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T13:44:31.576 INFO:teuthology.orchestra.run.smithi043.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-30T13:44:31.576 INFO:teuthology.orchestra.run.smithi043.stdout: "crimson_alien_op_num_threads": "6", 2024-03-30T13:44:31.576 INFO:teuthology.orchestra.run.smithi043.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-30T13:44:31.576 INFO:teuthology.orchestra.run.smithi043.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-30T13:44:31.576 INFO:teuthology.orchestra.run.smithi043.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-30T13:44:31.576 INFO:teuthology.orchestra.run.smithi043.stdout: "crimson_seastar_cpu_cores": "", 2024-03-30T13:44:31.576 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "", 2024-03-30T13:44:31.576 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location_hook": "", 2024-03-30T13:44:31.576 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location_hook_timeout": "10", 2024-03-30T13:44:31.576 INFO:teuthology.orchestra.run.smithi043.stdout: "daemonize": "false", 2024-03-30T13:44:31.577 INFO:teuthology.orchestra.run.smithi043.stdout: "daos_pool": "tank", 2024-03-30T13:44:31.577 INFO:teuthology.orchestra.run.smithi043.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-30T13:44:31.577 INFO:teuthology.orchestra.run.smithi043.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-30T13:44:31.577 INFO:teuthology.orchestra.run.smithi043.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-30T13:44:31.577 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_alienstore": "0/5", 2024-03-30T13:44:31.577 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_allow_any_pool_priority": "false", 2024-03-30T13:44:31.577 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_asok": "1/5", 2024-03-30T13:44:31.577 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_asok_assert_abort": "false", 2024-03-30T13:44:31.577 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_asserts_on_shutdown": "true", 2024-03-30T13:44:31.577 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_auth": "1/5", 2024-03-30T13:44:31.577 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_bdev": "1/3", 2024-03-30T13:44:31.577 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_bluefs": "1/5", 2024-03-30T13:44:31.577 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_bluestore": "1/5", 2024-03-30T13:44:31.577 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_buffer": "0/1", 2024-03-30T13:44:31.578 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_ceph_exporter": "1/5", 2024-03-30T13:44:31.578 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_cephfs_mirror": "0/5", 2024-03-30T13:44:31.578 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_cephsqlite": "0/5", 2024-03-30T13:44:31.578 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_client": "0/5", 2024-03-30T13:44:31.578 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_compressor": "1/5", 2024-03-30T13:44:31.578 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_context": "0/1", 2024-03-30T13:44:31.578 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_crimson_interrupt": "0/5", 2024-03-30T13:44:31.578 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_crush": "1/1", 2024-03-30T13:44:31.578 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_crypto": "1/5", 2024-03-30T13:44:31.578 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_cyanstore": "0/5", 2024-03-30T13:44:31.579 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_deliberately_leak_memory": "false", 2024-03-30T13:44:31.579 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_disable_randomized_ping": "false", 2024-03-30T13:44:31.579 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_dpdk": "1/5", 2024-03-30T13:44:31.579 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_eventtrace": "1/5", 2024-03-30T13:44:31.579 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_filer": "0/1", 2024-03-30T13:44:31.579 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_filestore": "1/3", 2024-03-30T13:44:31.579 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_finisher": "1/1", 2024-03-30T13:44:31.579 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_fuse": "1/5", 2024-03-30T13:44:31.579 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_heartbeat_testing_span": "0", 2024-03-30T13:44:31.579 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_heartbeatmap": "1/5", 2024-03-30T13:44:31.579 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-30T13:44:31.579 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_javaclient": "1/5", 2024-03-30T13:44:31.579 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_journal": "1/3", 2024-03-30T13:44:31.580 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_journaler": "0/5", 2024-03-30T13:44:31.580 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_kstore": "1/5", 2024-03-30T13:44:31.580 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_lockdep": "0/1", 2024-03-30T13:44:31.580 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mclock": "1/5", 2024-03-30T13:44:31.580 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mds": "1/5", 2024-03-30T13:44:31.580 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mds_balancer": "1/5", 2024-03-30T13:44:31.580 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mds_locker": "1/5", 2024-03-30T13:44:31.580 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mds_log": "1/5", 2024-03-30T13:44:31.580 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mds_log_expire": "1/5", 2024-03-30T13:44:31.580 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mds_migrator": "1/5", 2024-03-30T13:44:31.580 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mds_quiesce": "3/5", 2024-03-30T13:44:31.580 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_memstore": "1/5", 2024-03-30T13:44:31.580 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mgr": "2/5", 2024-03-30T13:44:31.580 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mgrc": "1/5", 2024-03-30T13:44:31.580 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mon": "20/20", 2024-03-30T13:44:31.581 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_monc": "0/10", 2024-03-30T13:44:31.581 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_ms": "1/1", 2024-03-30T13:44:31.581 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_none": "0/5", 2024-03-30T13:44:31.581 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_objclass": "0/5", 2024-03-30T13:44:31.581 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_objectcacher": "0/5", 2024-03-30T13:44:31.581 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_objecter": "0/1", 2024-03-30T13:44:31.581 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_optracker": "0/5", 2024-03-30T13:44:31.581 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_osd": "1/5", 2024-03-30T13:44:31.581 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_paxos": "20/20", 2024-03-30T13:44:31.581 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_perfcounter": "1/5", 2024-03-30T13:44:31.581 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_prioritycache": "1/5", 2024-03-30T13:44:31.581 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rados": "0/5", 2024-03-30T13:44:31.581 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rbd": "0/5", 2024-03-30T13:44:31.581 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rbd_mirror": "0/5", 2024-03-30T13:44:31.582 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rbd_pwl": "0/5", 2024-03-30T13:44:31.582 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rbd_replay": "0/5", 2024-03-30T13:44:31.582 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_refs": "0/0", 2024-03-30T13:44:31.582 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_reserver": "1/1", 2024-03-30T13:44:31.582 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rgw": "1/5", 2024-03-30T13:44:31.582 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rgw_access": "1/5", 2024-03-30T13:44:31.582 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rgw_datacache": "1/5", 2024-03-30T13:44:31.582 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rgw_dbstore": "1/5", 2024-03-30T13:44:31.582 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rgw_flight": "1/5", 2024-03-30T13:44:31.582 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-30T13:44:31.582 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rgw_sync": "1/5", 2024-03-30T13:44:31.582 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rocksdb": "4/5", 2024-03-30T13:44:31.582 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore": "0/5", 2024-03-30T13:44:31.582 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_backref": "0/5", 2024-03-30T13:44:31.582 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_cache": "0/5", 2024-03-30T13:44:31.583 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_cleaner": "0/5", 2024-03-30T13:44:31.583 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_device": "0/5", 2024-03-30T13:44:31.583 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_epm": "0/5", 2024-03-30T13:44:31.583 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-30T13:44:31.583 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_journal": "0/5", 2024-03-30T13:44:31.583 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_lba": "0/5", 2024-03-30T13:44:31.583 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_odata": "0/5", 2024-03-30T13:44:31.583 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_omap": "0/5", 2024-03-30T13:44:31.583 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_onode": "0/5", 2024-03-30T13:44:31.583 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_t": "0/5", 2024-03-30T13:44:31.583 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_tm": "0/5", 2024-03-30T13:44:31.583 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_striper": "0/1", 2024-03-30T13:44:31.584 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_test": "0/5", 2024-03-30T13:44:31.584 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_throttle": "1/1", 2024-03-30T13:44:31.584 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_timer": "0/1", 2024-03-30T13:44:31.584 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_tp": "0/5", 2024-03-30T13:44:31.584 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_trace": "1/5", 2024-03-30T13:44:31.585 INFO:teuthology.orchestra.run.smithi043.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-03-30T13:44:31.585 INFO:teuthology.orchestra.run.smithi043.stdout: "device_failure_prediction_mode": "none", 2024-03-30T13:44:31.585 INFO:teuthology.orchestra.run.smithi043.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-30T13:44:31.585 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-30T13:44:31.585 INFO:teuthology.orchestra.run.smithi043.stdout: "err_to_graylog": "false", 2024-03-30T13:44:31.585 INFO:teuthology.orchestra.run.smithi043.stdout: "err_to_journald": "false", 2024-03-30T13:44:31.585 INFO:teuthology.orchestra.run.smithi043.stdout: "err_to_stderr": "true", 2024-03-30T13:44:31.585 INFO:teuthology.orchestra.run.smithi043.stdout: "err_to_syslog": "false", 2024-03-30T13:44:31.585 INFO:teuthology.orchestra.run.smithi043.stdout: "event_tracing": "false", 2024-03-30T13:44:31.585 INFO:teuthology.orchestra.run.smithi043.stdout: "exporter_addr": "0.0.0.0", 2024-03-30T13:44:31.585 INFO:teuthology.orchestra.run.smithi043.stdout: "exporter_http_port": "9926", 2024-03-30T13:44:31.585 INFO:teuthology.orchestra.run.smithi043.stdout: "exporter_prio_limit": "5", 2024-03-30T13:44:31.586 INFO:teuthology.orchestra.run.smithi043.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-30T13:44:31.586 INFO:teuthology.orchestra.run.smithi043.stdout: "exporter_sort_metrics": "true", 2024-03-30T13:44:31.586 INFO:teuthology.orchestra.run.smithi043.stdout: "exporter_stats_period": "5", 2024-03-30T13:44:31.586 INFO:teuthology.orchestra.run.smithi043.stdout: "fake_statfs_for_testing": "0", 2024-03-30T13:44:31.586 INFO:teuthology.orchestra.run.smithi043.stdout: "fatal_signal_handlers": "true", 2024-03-30T13:44:31.586 INFO:teuthology.orchestra.run.smithi043.stdout: "filer_max_purge_ops": "10", 2024-03-30T13:44:31.586 INFO:teuthology.orchestra.run.smithi043.stdout: "filer_max_truncate_ops": "128", 2024-03-30T13:44:31.586 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_apply_finisher_threads": "1", 2024-03-30T13:44:31.587 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_blackhole": "false", 2024-03-30T13:44:31.587 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_btrfs_clone_range": "true", 2024-03-30T13:44:31.587 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_btrfs_snap": "true", 2024-03-30T13:44:31.587 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_caller_concurrency": "10", 2024-03-30T13:44:31.587 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_collect_device_partition_information": "true", 2024-03-30T13:44:31.587 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_commit_timeout": "600.000000", 2024-03-30T13:44:31.587 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_debug_inject_read_err": "false", 2024-03-30T13:44:31.587 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_debug_omap_check": "false", 2024-03-30T13:44:31.587 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_debug_random_read_err": "0.000000", 2024-03-30T13:44:31.587 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_debug_verify_split": "false", 2024-03-30T13:44:31.587 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_dump_file": "", 2024-03-30T13:44:31.587 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-30T13:44:31.587 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-30T13:44:31.588 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_fadvise": "true", 2024-03-30T13:44:31.588 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_fail_eio": "true", 2024-03-30T13:44:31.588 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_fd_cache_shards": "16", 2024-03-30T13:44:31.588 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_fd_cache_size": "128", 2024-03-30T13:44:31.588 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_fiemap": "false", 2024-03-30T13:44:31.588 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_fiemap_threshold": "4096", 2024-03-30T13:44:31.588 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-30T13:44:31.588 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-30T13:44:31.588 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_inject_stall": "0", 2024-03-30T13:44:31.588 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_journal_parallel": "false", 2024-03-30T13:44:31.588 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_journal_trailing": "false", 2024-03-30T13:44:31.589 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_journal_writeahead": "false", 2024-03-30T13:44:31.589 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_kill_at": "0", 2024-03-30T13:44:31.589 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-30T13:44:31.589 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-30T13:44:31.589 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-30T13:44:31.589 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-30T13:44:31.589 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-30T13:44:31.589 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_inline_xattrs": "0", 2024-03-30T13:44:31.589 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-30T13:44:31.589 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-30T13:44:31.589 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-30T13:44:31.589 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-30T13:44:31.589 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_xattr_value_size": "0", 2024-03-30T13:44:31.589 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-30T13:44:31.589 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-30T13:44:31.590 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-30T13:44:31.590 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_merge_threshold": "-10", 2024-03-30T13:44:31.590 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-30T13:44:31.590 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_odsync_write": "false", 2024-03-30T13:44:31.590 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_omap_backend": "rocksdb", 2024-03-30T13:44:31.590 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_omap_backend_path": "", 2024-03-30T13:44:31.590 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-30T13:44:31.590 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-30T13:44:31.590 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-30T13:44:31.590 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_op_thread_timeout": "60", 2024-03-30T13:44:31.590 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_op_threads": "2", 2024-03-30T13:44:31.591 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_punch_hole": "false", 2024-03-30T13:44:31.591 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-30T13:44:31.591 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-30T13:44:31.591 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-30T13:44:31.591 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-30T13:44:31.591 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-30T13:44:31.591 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-30T13:44:31.591 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-30T13:44:31.591 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-30T13:44:31.591 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-30T13:44:31.591 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_queue_max_ops": "50", 2024-03-30T13:44:31.591 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-30T13:44:31.591 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_seek_data_hole": "false", 2024-03-30T13:44:31.591 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_sloppy_crc": "false", 2024-03-30T13:44:31.592 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-30T13:44:31.592 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_splice": "false", 2024-03-30T13:44:31.592 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_split_multiple": "2", 2024-03-30T13:44:31.592 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_split_rand_factor": "20", 2024-03-30T13:44:31.592 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_update_to": "1000", 2024-03-30T13:44:31.592 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-30T13:44:31.592 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-30T13:44:31.592 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-30T13:44:31.592 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-30T13:44:31.592 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-30T13:44:31.592 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-30T13:44:31.592 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_enable": "true", 2024-03-30T13:44:31.592 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-30T13:44:31.592 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-30T13:44:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-30T13:44:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-30T13:44:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-30T13:44:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-30T13:44:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_xfs_extsize": "false", 2024-03-30T13:44:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_zfs_snap": "false", 2024-03-30T13:44:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "fio_dir": "/tmp/fio", 2024-03-30T13:44:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "7ee932fa-ee9a-11ee-b647-cb9ed24678a4", 2024-03-30T13:44:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_allow_other": "true", 2024-03-30T13:44:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_atomic_o_trunc": "true", 2024-03-30T13:44:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_big_writes": "true", 2024-03-30T13:44:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_debug": "false", 2024-03-30T13:44:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_default_permissions": "false", 2024-03-30T13:44:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_disable_pagecache": "false", 2024-03-30T13:44:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_max_write": "0", 2024-03-30T13:44:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_multithreaded": "true", 2024-03-30T13:44:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_require_active_mds": "true", 2024-03-30T13:44:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_set_user_groups": "true", 2024-03-30T13:44:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_splice_move": "true", 2024-03-30T13:44:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_splice_read": "true", 2024-03-30T13:44:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_splice_write": "true", 2024-03-30T13:44:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-30T13:44:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_use_invalidate_cb": "true", 2024-03-30T13:44:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-30T13:44:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "gss_target_name": "ceph", 2024-03-30T13:44:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_file": "", 2024-03-30T13:44:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_inject_failure": "0", 2024-03-30T13:44:31.595 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_interval": "5", 2024-03-30T13:44:31.595 INFO:teuthology.orchestra.run.smithi043.stdout: "host": "smithi043", 2024-03-30T13:44:31.595 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-30T13:44:31.595 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-30T13:44:31.595 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-30T13:44:31.595 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-30T13:44:31.595 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-30T13:44:31.595 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-30T13:44:31.595 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-30T13:44:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-30T13:44:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-30T13:44:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-30T13:44:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-30T13:44:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-30T13:44:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-30T13:44:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: "inject_early_sigterm": "false", 2024-03-30T13:44:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: "jaeger_agent_port": "6799", 2024-03-30T13:44:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: "jaeger_tracing_enable": "false", 2024-03-30T13:44:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_aio": "true", 2024-03-30T13:44:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_align_min_size": "65536", 2024-03-30T13:44:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_block_align": "true", 2024-03-30T13:44:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_block_size": "4096", 2024-03-30T13:44:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_dio": "true", 2024-03-30T13:44:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_discard": "false", 2024-03-30T13:44:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_force_aio": "false", 2024-03-30T13:44:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_ignore_corruption": "false", 2024-03-30T13:44:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_max_write_bytes": "10485760", 2024-03-30T13:44:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_max_write_entries": "100", 2024-03-30T13:44:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_replay_from": "0", 2024-03-30T13:44:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-30T13:44:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-30T13:44:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-30T13:44:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-30T13:44:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_write_header_frequency": "0", 2024-03-30T13:44:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_zero_on_create": "false", 2024-03-30T13:44:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "journaler_prefetch_periods": "10", 2024-03-30T13:44:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: "journaler_prezero_periods": "5", 2024-03-30T13:44:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: "journaler_write_head_interval": "15", 2024-03-30T13:44:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "", 2024-03-30T13:44:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: "keyfile": "", 2024-03-30T13:44:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-30T13:44:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_backend": "rocksdb", 2024-03-30T13:44:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_default_stripe_size": "65536", 2024-03-30T13:44:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_fsck_on_mount": "false", 2024-03-30T13:44:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-30T13:44:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_max_bytes": "67108864", 2024-03-30T13:44:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_max_ops": "512", 2024-03-30T13:44:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_nid_prealloc": "1024", 2024-03-30T13:44:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_onode_map_size": "1024", 2024-03-30T13:44:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-30T13:44:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_sync_submit_transaction": "false", 2024-03-30T13:44:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_sync_transaction": "false", 2024-03-30T13:44:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "librados_thread_count": "2", 2024-03-30T13:44:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "lockdep": "false", 2024-03-30T13:44:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "lockdep_force_backtrace": "false", 2024-03-30T13:44:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "log_coarse_timestamps": "true", 2024-03-30T13:44:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-30T13:44:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "log_flush_on_exit": "false", 2024-03-30T13:44:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "log_graylog_host": "127.0.0.1", 2024-03-30T13:44:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "log_graylog_port": "12201", 2024-03-30T13:44:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "log_max_new": "1000", 2024-03-30T13:44:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "log_max_recent": "10000", 2024-03-30T13:44:31.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:31 smithi043 ceph-mon[29963]: 5.15 scrub starts 2024-03-30T13:44:31.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:31 smithi043 ceph-mon[29963]: 5.15 scrub ok 2024-03-30T13:44:31.600 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:31 smithi043 ceph-mon[42099]: 5.15 scrub starts 2024-03-30T13:44:31.600 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:31 smithi043 ceph-mon[42099]: 5.15 scrub ok 2024-03-30T13:44:31.601 INFO:teuthology.orchestra.run.smithi043.stdout: "log_stderr_prefix": "", 2024-03-30T13:44:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: "log_stop_at_utilization": "0.970000", 2024-03-30T13:44:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": "true", 2024-03-30T13:44:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_graylog": "false", 2024-03-30T13:44:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_journald": "false", 2024-03-30T13:44:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_stderr": "false", 2024-03-30T13:44:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_syslog": "false", 2024-03-30T13:44:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: "max_rotating_auth_attempts": "10", 2024-03-30T13:44:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-30T13:44:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_action_on_write_error": "1", 2024-03-30T13:44:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_alternate_name_max": "8192", 2024-03-30T13:44:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_asio_thread_count": "2", 2024-03-30T13:44:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_export_pin": "true", 2024-03-30T13:44:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_fragment_dirs": "true", 2024-03-30T13:44:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-30T13:44:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_fragment_interval": "5", 2024-03-30T13:44:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-30T13:44:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-30T13:44:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_interval": "10", 2024-03-30T13:44:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_max": "-1", 2024-03-30T13:44:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_max_until": "-1", 2024-03-30T13:44:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_merge_size": "50", 2024-03-30T13:44:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_midchunk": "0.300000", 2024-03-30T13:44:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-30T13:44:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_min_start": "0.200000", 2024-03-30T13:44:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_minchunk": "0.001000", 2024-03-30T13:44:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_mode": "0", 2024-03-30T13:44:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_need_max": "1.200000", 2024-03-30T13:44:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_need_min": "0.800000", 2024-03-30T13:44:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_overload_epochs": "2", 2024-03-30T13:44:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-30T13:44:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-30T13:44:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_split_bits": "3", 2024-03-30T13:44:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-30T13:44:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_split_size": "10000", 2024-03-30T13:44:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-30T13:44:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_target_decay": "10.000000", 2024-03-30T13:44:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-30T13:44:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_beacon_grace": "15.000000", 2024-03-30T13:44:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_beacon_interval": "4.000000", 2024-03-30T13:44:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-30T13:44:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-30T13:44:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_mid": "0.700000", 2024-03-30T13:44:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-03-30T13:44:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_quiesce_delay": "0", 2024-03-30T13:44:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_quiesce_sleep": "200", 2024-03-30T13:44:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_quiesce_splitauth": "true", 2024-03-30T13:44:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_quiesce_threshold": "524288", 2024-03-30T13:44:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_release_free_interval": "10", 2024-03-30T13:44:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_reservation": "0.050000", 2024-03-30T13:44:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-30T13:44:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_trim_interval": "1", 2024-03-30T13:44:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_trim_threshold": "262144", 2024-03-30T13:44:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-30T13:44:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-30T13:44:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-30T13:44:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_client_prealloc_inos": "1000", 2024-03-30T13:44:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-30T13:44:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_connect_bootstrapping": "false", 2024-03-30T13:44:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_damage_table_max_entries": "10000", 2024-03-30T13:44:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-30T13:44:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_debug_auth_pins": "false", 2024-03-30T13:44:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_debug_frag": "false", 2024-03-30T13:44:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_debug_scatterstat": "false", 2024-03-30T13:44:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_debug_subtrees": "false", 2024-03-30T13:44:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_decay_halflife": "5.000000", 2024-03-30T13:44:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_default_dir_hash": "2", 2024-03-30T13:44:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_defer_session_stale": "true", 2024-03-30T13:44:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_deny_all_reconnect": "false", 2024-03-30T13:44:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_dir_keys_per_op": "16384", 2024-03-30T13:44:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_dir_max_commit_size": "10", 2024-03-30T13:44:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_dir_max_entries": "0", 2024-03-30T13:44:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_dir_prefetch": "true", 2024-03-30T13:44:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-30T13:44:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-30T13:44:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_dump_cache_on_map": "false", 2024-03-30T13:44:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-30T13:44:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-30T13:44:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_early_reply": "true", 2024-03-30T13:44:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_enable_op_tracker": "true", 2024-03-30T13:44:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_enforce_unique_name": "true", 2024-03-30T13:44:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-30T13:44:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-30T13:44:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_export_ephemeral_random": "true", 2024-03-30T13:44:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-30T13:44:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-30T13:44:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-30T13:44:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-30T13:44:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-30T13:44:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-30T13:44:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-30T13:44:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-30T13:44:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_health_summarize_threshold": "10", 2024-03-30T13:44:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-30T13:44:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-30T13:44:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_inject_health_dummy": "false", 2024-03-30T13:44:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-30T13:44:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_inject_migrator_session_race": "false", 2024-03-30T13:44:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-30T13:44:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-30T13:44:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-30T13:44:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_join_fs": "", 2024-03-30T13:44:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_journal_format": "1", 2024-03-30T13:44:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_create_at": "0", 2024-03-30T13:44:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_export_at": "0", 2024-03-30T13:44:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_import_at": "0", 2024-03-30T13:44:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_journal_at": "0", 2024-03-30T13:44:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_journal_expire_at": "0", 2024-03-30T13:44:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_journal_replay_at": "0", 2024-03-30T13:44:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_link_at": "0", 2024-03-30T13:44:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_mdstable_at": "0", 2024-03-30T13:44:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_openc_at": "0", 2024-03-30T13:44:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_rename_at": "0", 2024-03-30T13:44:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_shutdown_at": "0", 2024-03-30T13:44:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-30T13:44:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_event_large_threshold": "524288", 2024-03-30T13:44:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_events_per_segment": "1024", 2024-03-30T13:44:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-30T13:44:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_max_events": "-1", 2024-03-30T13:44:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_max_segments": "128", 2024-03-30T13:44:31.611 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_pause": "false", 2024-03-30T13:44:31.611 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_segment_size": "0", 2024-03-30T13:44:31.611 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-30T13:44:31.611 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-30T13:44:31.611 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-30T13:44:31.611 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_trim_threshold": "128", 2024-03-30T13:44:31.611 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-30T13:44:31.611 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_warn_factor": "2.000000", 2024-03-30T13:44:31.611 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_max_caps_per_client": "1048576", 2024-03-30T13:44:31.611 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_max_completed_flushes": "100000", 2024-03-30T13:44:31.612 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_max_completed_requests": "100000", 2024-03-30T13:44:31.612 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_max_export_size": "20971520", 2024-03-30T13:44:31.612 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_max_file_recover": "32", 2024-03-30T13:44:31.612 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_max_purge_files": "64", 2024-03-30T13:44:31.612 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_max_purge_ops": "8192", 2024-03-30T13:44:31.612 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-30T13:44:31.612 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-30T13:44:31.612 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_max_snaps_per_dir": "100", 2024-03-30T13:44:31.612 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_metrics_update_interval": "2", 2024-03-30T13:44:31.612 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_min_caps_per_client": "100", 2024-03-30T13:44:31.612 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_min_caps_working_set": "10000", 2024-03-30T13:44:31.612 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-30T13:44:31.612 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_numa_node": "-1", 2024-03-30T13:44:31.612 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-30T13:44:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_op_complaint_time": "30.000000", 2024-03-30T13:44:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_op_history_duration": "600", 2024-03-30T13:44:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_op_history_size": "20", 2024-03-30T13:44:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_op_history_slow_op_size": "20", 2024-03-30T13:44:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-30T13:44:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_op_log_threshold": "5", 2024-03-30T13:44:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_ping_grace": "15", 2024-03-30T13:44:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_ping_interval": "5", 2024-03-30T13:44:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-30T13:44:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-30T13:44:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_recall_max_caps": "30000", 2024-03-30T13:44:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-30T13:44:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-30T13:44:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-30T13:44:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_recall_warning_threshold": "262144", 2024-03-30T13:44:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-30T13:44:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_replay_interval": "1.000000", 2024-03-30T13:44:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-30T13:44:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-30T13:44:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_root_ino_gid": "0", 2024-03-30T13:44:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_root_ino_uid": "0", 2024-03-30T13:44:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-30T13:44:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-30T13:44:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-30T13:44:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-30T13:44:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-30T13:44:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-30T13:44:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-30T13:44:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-30T13:44:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-30T13:44:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-30T13:44:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_shutdown_check": "0", 2024-03-30T13:44:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_skip_ino": "0", 2024-03-30T13:44:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-30T13:44:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_snap_max_uid": "4294967294", 2024-03-30T13:44:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_snap_min_uid": "0", 2024-03-30T13:44:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_snap_rstat": "false", 2024-03-30T13:44:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_standby_replay_damaged": "false", 2024-03-30T13:44:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_symlink_recovery": "true", 2024-03-30T13:44:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-30T13:44:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_thrash_exports": "0", 2024-03-30T13:44:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_thrash_fragments": "0", 2024-03-30T13:44:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_tick_interval": "5.000000", 2024-03-30T13:44:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_valgrind_exit": "false", 2024-03-30T13:44:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_verify_backtrace": "1", 2024-03-30T13:44:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_verify_scatter": "false", 2024-03-30T13:44:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-30T13:44:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_wipe_sessions": "false", 2024-03-30T13:44:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "mempool_debug": "false", 2024-03-30T13:44:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-30T13:44:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "memstore_device_bytes": "1073741824", 2024-03-30T13:44:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "memstore_page_set": "false", 2024-03-30T13:44:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "memstore_page_size": "65536", 2024-03-30T13:44:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_client_bytes": "134217728", 2024-03-30T13:44:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_client_messages": "512", 2024-03-30T13:44:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-30T13:44:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-30T13:44:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-30T13:44:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-30T13:44:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_disabled_modules": "", 2024-03-30T13:44:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-30T13:44:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_max_pg_num_change": "128", 2024-03-30T13:44:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_mds_bytes": "134217728", 2024-03-30T13:44:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_mds_messages": "128", 2024-03-30T13:44:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-30T13:44:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_mon_bytes": "134217728", 2024-03-30T13:44:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_mon_messages": "128", 2024-03-30T13:44:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_osd_bytes": "536870912", 2024-03-30T13:44:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_osd_messages": "8192", 2024-03-30T13:44:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_pool": "true", 2024-03-30T13:44:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-30T13:44:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_standby_modules": "true", 2024-03-30T13:44:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_stats_period": "5", 2024-03-30T13:44:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_stats_threshold": "5", 2024-03-30T13:44:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_tick_period": "2", 2024-03-30T13:44:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-30T13:44:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-30T13:44:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_allow_pool_delete": "true", 2024-03-30T13:44:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_allow_pool_size_one": "false", 2024-03-30T13:44:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_auth_validate_all_caps": "true", 2024-03-30T13:44:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-30T13:44:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-30T13:44:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_bytes": "104857600", 2024-03-30T13:44:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_directed_command_retry": "2", 2024-03-30T13:44:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-30T13:44:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-30T13:44:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-30T13:44:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-30T13:44:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_hunt_parallel": "3", 2024-03-30T13:44:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_log_interval": "1.000000", 2024-03-30T13:44:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-30T13:44:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_ping_interval": "10.000000", 2024-03-30T13:44:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-30T13:44:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_target_rank": "-1", 2024-03-30T13:44:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-30T13:44:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-30T13:44:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-30T13:44:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_file_level": "debug", 2024-03-30T13:44:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_to_file": "true", 2024-03-30T13:44:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-30T13:44:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-30T13:44:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-30T13:44:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_to_journald": "true", 2024-03-30T13:44:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-30T13:44:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-30T13:44:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-30T13:44:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-30T13:44:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_compact_on_bootstrap": "false", 2024-03-30T13:44:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_compact_on_start": "false", 2024-03-30T13:44:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_compact_on_trim": "true", 2024-03-30T13:44:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-30T13:44:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-30T13:44:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-30T13:44:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cpu_threads": "4", 2024-03-30T13:44:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_crush_min_required_version": "hammer", 2024-03-30T13:44:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_daemon_bytes": "419430400", 2024-03-30T13:44:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-30T13:44:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_data_avail_crit": "5", 2024-03-30T13:44:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_data_avail_warn": "5", 2024-03-30T13:44:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_data_size_warn": "16106127360", 2024-03-30T13:44:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-30T13:44:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-30T13:44:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_dump_json": "false", 2024-03-30T13:44:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-30T13:44:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_dump_transactions": "false", 2024-03-30T13:44:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_extra_checks": "false", 2024-03-30T13:44:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-30T13:44:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-30T13:44:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_no_require_reef": "false", 2024-03-30T13:44:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_no_require_squid": "false", 2024-03-30T13:44:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-30T13:44:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-30T13:44:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-30T13:44:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_down_mkfs_grace": "60", 2024-03-30T13:44:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_down_uptime_grace": "60", 2024-03-30T13:44:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_election_default_strategy": "1", 2024-03-30T13:44:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_election_timeout": "5.000000", 2024-03-30T13:44:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-30T13:44:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_elector_ping_divisor": "2", 2024-03-30T13:44:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-30T13:44:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_enable_op_tracker": "true", 2024-03-30T13:44:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_fake_pool_delete": "false", 2024-03-30T13:44:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_force_quorum_join": "false", 2024-03-30T13:44:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_globalid_prealloc": "10000", 2024-03-30T13:44:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_health_detail_to_clog": "true", 2024-03-30T13:44:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_health_log_update_period": "5", 2024-03-30T13:44:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_health_max_detail": "50", 2024-03-30T13:44:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_health_to_clog": "true", 2024-03-30T13:44:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_health_to_clog_interval": "600", 2024-03-30T13:44:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-30T13:44:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_host": "[v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0]", 2024-03-30T13:44:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_host_override": "", 2024-03-30T13:44:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_initial_members": "", 2024-03-30T13:44:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-30T13:44:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-30T13:44:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-30T13:44:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-30T13:44:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-30T13:44:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_lease": "5.000000", 2024-03-30T13:44:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-30T13:44:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-30T13:44:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_log_full_interval": "50", 2024-03-30T13:44:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_log_max": "10000", 2024-03-30T13:44:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_log_max_summary": "50", 2024-03-30T13:44:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-30T13:44:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_max_log_epochs": "500", 2024-03-30T13:44:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-30T13:44:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-30T13:44:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_max_osd": "10000", 2024-03-30T13:44:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_max_pg_per_osd": "10000", 2024-03-30T13:44:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_max_pool_pg_num": "65536", 2024-03-30T13:44:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-30T13:44:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-30T13:44:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_mds_force_trim_to": "0", 2024-03-30T13:44:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_mds_skip_sanity": "false", 2024-03-30T13:44:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_memory_autotune": "true", 2024-03-30T13:44:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_memory_target": "2147483648", 2024-03-30T13:44:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_mgr_beacon_grace": "30", 2024-03-30T13:44:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-30T13:44:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_mgr_digest_period": "5", 2024-03-30T13:44:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_mgr_inactive_grace": "60", 2024-03-30T13:44:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-30T13:44:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-30T13:44:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_min_osdmap_epochs": "50", 2024-03-30T13:44:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_op_complaint_time": "30", 2024-03-30T13:44:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_op_history_duration": "600", 2024-03-30T13:44:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_op_history_size": "20", 2024-03-30T13:44:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_op_history_slow_op_size": "20", 2024-03-30T13:44:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-30T13:44:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_op_log_threshold": "5", 2024-03-30T13:44:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-30T13:44:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-30T13:44:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-30T13:44:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_auto_mark_in": "false", 2024-03-30T13:44:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-30T13:44:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-30T13:44:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-30T13:44:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_cache_size": "500", 2024-03-30T13:44:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-30T13:44:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-30T13:44:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-30T13:44:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_down_out_interval": "600", 2024-03-30T13:44:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-30T13:44:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-30T13:44:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_force_trim_to": "0", 2024-03-30T13:44:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-30T13:44:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-30T13:44:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-30T13:44:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-30T13:44:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-30T13:44:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-30T13:44:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-30T13:44:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-30T13:44:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_min_down_reporters": "2", 2024-03-30T13:44:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-30T13:44:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-30T13:44:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-30T13:44:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-30T13:44:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-30T13:44:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-30T13:44:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_report_timeout": "900", 2024-03-30T13:44:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-30T13:44:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-30T13:44:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-30T13:44:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-30T13:44:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-30T13:44:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-30T13:44:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-30T13:44:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-30T13:44:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-30T13:44:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_pg_stuck_threshold": "60", 2024-03-30T13:44:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-30T13:44:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-30T13:44:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-30T13:44:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-30T13:44:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-30T13:44:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-30T13:44:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_probe_timeout": "2.000000", 2024-03-30T13:44:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_reweight_max_change": "0.050000", 2024-03-30T13:44:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_reweight_max_osds": "4", 2024-03-30T13:44:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-30T13:44:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-30T13:44:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-30T13:44:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-30T13:44:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-30T13:44:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_scrub_interval": "86400", 2024-03-30T13:44:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_scrub_max_keys": "100", 2024-03-30T13:44:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_scrub_timeout": "300", 2024-03-30T13:44:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_session_timeout": "300", 2024-03-30T13:44:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_smart_report_timeout": "5", 2024-03-30T13:44:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_stat_smooth_intervals": "6", 2024-03-30T13:44:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-30T13:44:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_stretch_pool_min_size": "2", 2024-03-30T13:44:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_stretch_pool_size": "4", 2024-03-30T13:44:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-30T13:44:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-30T13:44:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_sync_debug": "false", 2024-03-30T13:44:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-30T13:44:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-30T13:44:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_sync_provider_kill_at": "0", 2024-03-30T13:44:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_sync_requester_kill_at": "0", 2024-03-30T13:44:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_sync_timeout": "60.000000", 2024-03-30T13:44:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_target_pg_per_osd": "100", 2024-03-30T13:44:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_tick_interval": "5", 2024-03-30T13:44:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_timecheck_interval": "300.000000", 2024-03-30T13:44:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-30T13:44:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_use_min_delay_socket": "false", 2024-03-30T13:44:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_older_version_delay": "604800", 2024-03-30T13:44:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-30T13:44:31.633 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-30T13:44:31.633 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-30T13:44:31.633 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_filestore_osds": "true", 2024-03-30T13:44:31.633 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-30T13:44:31.633 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-30T13:44:31.633 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-30T13:44:31.633 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_misplaced": "false", 2024-03-30T13:44:31.633 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-30T13:44:31.633 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_older_version": "true", 2024-03-30T13:44:31.633 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-30T13:44:31.633 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-30T13:44:31.633 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-03-30T13:44:31.633 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-30T13:44:31.633 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-30T13:44:31.634 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-30T13:44:31.634 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-30T13:44:31.634 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-30T13:44:31.634 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-30T13:44:31.634 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": "", 2024-03-30T13:44:31.634 INFO:teuthology.orchestra.run.smithi043.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-30T13:44:31.634 INFO:teuthology.orchestra.run.smithi043.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-30T13:44:31.634 INFO:teuthology.orchestra.run.smithi043.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-30T13:44:31.634 INFO:teuthology.orchestra.run.smithi043.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-30T13:44:31.634 INFO:teuthology.orchestra.run.smithi043.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-30T13:44:31.634 INFO:teuthology.orchestra.run.smithi043.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-30T13:44:31.634 INFO:teuthology.orchestra.run.smithi043.stdout: "motr_tracing_enabled": "false", 2024-03-30T13:44:31.634 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_op_threads": "3", 2024-03-30T13:44:31.634 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-30T13:44:31.635 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_cm": "false", 2024-03-30T13:44:31.635 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_device_name": "", 2024-03-30T13:44:31.635 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_dscp": "96", 2024-03-30T13:44:31.635 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-30T13:44:31.635 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-30T13:44:31.635 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_local_gid": "", 2024-03-30T13:44:31.635 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-30T13:44:31.635 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_port_num": "1", 2024-03-30T13:44:31.635 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-30T13:44:31.635 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-30T13:44:31.635 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-30T13:44:31.635 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-30T13:44:31.635 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_sl": "3", 2024-03-30T13:44:31.635 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_support_srq": "true", 2024-03-30T13:44:31.636 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_type": "ib", 2024-03-30T13:44:31.636 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_reap_threshold": "5", 2024-03-30T13:44:31.636 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_bind_before_connect": "false", 2024-03-30T13:44:31.636 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_bind_ipv4": "true", 2024-03-30T13:44:31.636 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_bind_ipv6": "false", 2024-03-30T13:44:31.636 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_bind_msgr1": "true", 2024-03-30T13:44:31.636 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_bind_msgr2": "true", 2024-03-30T13:44:31.636 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_bind_port_max": "7568", 2024-03-30T13:44:31.636 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_bind_port_min": "6800", 2024-03-30T13:44:31.636 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-30T13:44:31.636 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_bind_retry_count": "3", 2024-03-30T13:44:31.636 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_bind_retry_delay": "5", 2024-03-30T13:44:31.636 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_blackhole_client": "false", 2024-03-30T13:44:31.636 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_blackhole_mds": "false", 2024-03-30T13:44:31.637 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_blackhole_mgr": "false", 2024-03-30T13:44:31.637 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_blackhole_mon": "false", 2024-03-30T13:44:31.637 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_blackhole_osd": "false", 2024-03-30T13:44:31.637 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_client_mode": "crc secure", 2024-03-30T13:44:31.637 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_cluster_mode": "crc secure", 2024-03-30T13:44:31.637 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_cluster_type": "", 2024-03-30T13:44:31.637 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_compress_secure": "false", 2024-03-30T13:44:31.637 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_connection_idle_timeout": "900", 2024-03-30T13:44:31.637 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_connection_ready_timeout": "10", 2024-03-30T13:44:31.637 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_crc_data": "true", 2024-03-30T13:44:31.637 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_crc_header": "true", 2024-03-30T13:44:31.637 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_die_on_bad_msg": "false", 2024-03-30T13:44:31.637 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_die_on_bug": "true", 2024-03-30T13:44:31.637 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_die_on_old_message": "true", 2024-03-30T13:44:31.638 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_die_on_skipped_message": "false", 2024-03-30T13:44:31.638 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-30T13:44:31.638 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-30T13:44:31.638 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_coremask": "0xF", 2024-03-30T13:44:31.638 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-30T13:44:31.638 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-30T13:44:31.638 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_enable_tso": "true", 2024-03-30T13:44:31.638 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-30T13:44:31.638 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-30T13:44:31.638 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_hugepages": "", 2024-03-30T13:44:31.638 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-30T13:44:31.638 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-30T13:44:31.638 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_lro": "true", 2024-03-30T13:44:31.639 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_memory_channel": "4", 2024-03-30T13:44:31.639 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-30T13:44:31.639 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_pmd": "", 2024-03-30T13:44:31.639 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_port_id": "0", 2024-03-30T13:44:31.639 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-30T13:44:31.639 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-30T13:44:31.639 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dump_on_send": "false", 2024-03-30T13:44:31.639 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_initial_backoff": "0.200000", 2024-03-30T13:44:31.639 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_inject_delay_max": "1.000000", 2024-03-30T13:44:31.639 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-30T13:44:31.639 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_inject_delay_type": "", 2024-03-30T13:44:31.639 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-30T13:44:31.639 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_inject_network_congestion": "0", 2024-03-30T13:44:31.639 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_inject_socket_failures": "0", 2024-03-30T13:44:31.640 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_learn_addr_from_peer": "true", 2024-03-30T13:44:31.640 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_max_accept_failures": "4", 2024-03-30T13:44:31.640 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_max_backoff": "15.000000", 2024-03-30T13:44:31.640 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_mon_client_mode": "secure crc", 2024-03-30T13:44:31.640 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-30T13:44:31.640 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_mon_service_mode": "secure crc", 2024-03-30T13:44:31.640 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_osd_compress_min_size": "1024", 2024-03-30T13:44:31.640 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_osd_compress_mode": "none", 2024-03-30T13:44:31.640 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-30T13:44:31.640 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-30T13:44:31.640 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_pq_min_cost": "65536", 2024-03-30T13:44:31.640 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_public_type": "", 2024-03-30T13:44:31.640 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_service_mode": "crc secure", 2024-03-30T13:44:31.640 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_tcp_listen_backlog": "512", 2024-03-30T13:44:31.641 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_tcp_nodelay": "true", 2024-03-30T13:44:31.641 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-30T13:44:31.641 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_tcp_rcvbuf": "0", 2024-03-30T13:44:31.641 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_type": "async+posix", 2024-03-30T13:44:31.641 INFO:teuthology.orchestra.run.smithi043.stdout: "no_config_file": "false", 2024-03-30T13:44:31.641 INFO:teuthology.orchestra.run.smithi043.stdout: "objecter_completion_locks_per_session": "32", 2024-03-30T13:44:31.641 INFO:teuthology.orchestra.run.smithi043.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-30T13:44:31.641 INFO:teuthology.orchestra.run.smithi043.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-30T13:44:31.641 INFO:teuthology.orchestra.run.smithi043.stdout: "objecter_inflight_ops": "1024", 2024-03-30T13:44:31.641 INFO:teuthology.orchestra.run.smithi043.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-30T13:44:31.641 INFO:teuthology.orchestra.run.smithi043.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-30T13:44:31.641 INFO:teuthology.orchestra.run.smithi043.stdout: "objecter_tick_interval": "5.000000", 2024-03-30T13:44:31.641 INFO:teuthology.orchestra.run.smithi043.stdout: "objecter_timeout": "10.000000", 2024-03-30T13:44:31.641 INFO:teuthology.orchestra.run.smithi043.stdout: "objectstore_blackhole": "false", 2024-03-30T13:44:31.642 INFO:teuthology.orchestra.run.smithi043.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-30T13:44:31.642 INFO:teuthology.orchestra.run.smithi043.stdout: "openssl_engine_opts": "", 2024-03-30T13:44:31.642 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_agent_delay_time": "5.000000", 2024-03-30T13:44:31.642 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_agent_hist_halflife": "1000", 2024-03-30T13:44:31.642 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_agent_max_low_ops": "2", 2024-03-30T13:44:31.642 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_agent_max_ops": "4", 2024-03-30T13:44:31.642 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-30T13:44:31.642 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-30T13:44:31.642 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_agent_slop": "0.020000", 2024-03-30T13:44:31.642 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-30T13:44:31.642 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-30T13:44:31.642 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_asio_thread_count": "2", 2024-03-30T13:44:31.642 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_async_recovery_min_cost": "100", 2024-03-30T13:44:31.642 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-30T13:44:31.643 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-30T13:44:31.643 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_backfill_scan_max": "512", 2024-03-30T13:44:31.643 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_backfill_scan_min": "64", 2024-03-30T13:44:31.643 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_backoff_on_degraded": "false", 2024-03-30T13:44:31.643 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_backoff_on_peering": "false", 2024-03-30T13:44:31.643 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_backoff_on_unfound": "true", 2024-03-30T13:44:31.643 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_beacon_report_interval": "300", 2024-03-30T13:44:31.643 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_bench_duration": "30", 2024-03-30T13:44:31.643 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-30T13:44:31.643 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_bench_max_block_size": "67108864", 2024-03-30T13:44:31.643 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-30T13:44:31.643 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_blkin_trace_all": "false", 2024-03-30T13:44:31.643 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-30T13:44:31.643 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-30T13:44:31.644 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-30T13:44:31.644 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-30T13:44:31.644 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_check_for_log_corruption": "false", 2024-03-30T13:44:31.644 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-30T13:44:31.644 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-30T13:44:31.644 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-30T13:44:31.644 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-30T13:44:31.644 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_class_update_on_start": "true", 2024-03-30T13:44:31.644 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_client_message_cap": "256", 2024-03-30T13:44:31.644 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_client_message_size_cap": "524288000", 2024-03-30T13:44:31.644 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_client_op_priority": "63", 2024-03-30T13:44:31.644 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_client_watch_timeout": "30", 2024-03-30T13:44:31.644 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_command_max_records": "256", 2024-03-30T13:44:31.644 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_compact_on_start": "false", 2024-03-30T13:44:31.645 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-30T13:44:31.645 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_crush_chooseleaf_type": "1", 2024-03-30T13:44:31.645 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-30T13:44:31.645 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_crush_update_on_start": "true", 2024-03-30T13:44:31.645 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_crush_update_weight_set": "true", 2024-03-30T13:44:31.645 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-30T13:44:31.645 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-30T13:44:31.645 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-30T13:44:31.645 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-30T13:44:31.645 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-30T13:44:31.645 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_feed_pullee": "-1", 2024-03-30T13:44:31.645 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-30T13:44:31.645 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-30T13:44:31.645 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-30T13:44:31.646 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_misdirected_ops": "false", 2024-03-30T13:44:31.646 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_no_acting_change": "false", 2024-03-30T13:44:31.646 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_no_purge_strays": "false", 2024-03-30T13:44:31.646 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_op_order": "false", 2024-03-30T13:44:31.646 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-30T13:44:31.646 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-30T13:44:31.646 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-30T13:44:31.646 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-30T13:44:31.646 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_shutdown": "false", 2024-03-30T13:44:31.646 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-30T13:44:31.646 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-30T13:44:31.646 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_trim_objects": "false", 2024-03-30T13:44:31.647 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-30T13:44:31.647 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-30T13:44:31.647 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_verify_snaps": "false", 2024-03-30T13:44:31.647 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-30T13:44:31.647 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-30T13:44:31.647 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_deep_scrub_keys": "1024", 2024-03-30T13:44:31.647 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-30T13:44:31.647 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-30T13:44:31.647 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-30T13:44:31.647 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_deep_scrub_stride": "524288", 2024-03-30T13:44:31.647 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-30T13:44:31.647 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_default_data_pool_replay_window": "45", 2024-03-30T13:44:31.647 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_default_notify_timeout": "30", 2024-03-30T13:44:31.647 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_delete_sleep": "0.000000", 2024-03-30T13:44:31.648 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-30T13:44:31.648 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-30T13:44:31.648 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-30T13:44:31.648 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_discard_disconnected_ops": "true", 2024-03-30T13:44:31.648 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_enable_op_tracker": "true", 2024-03-30T13:44:31.648 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-30T13:44:31.648 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-30T13:44:31.648 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-30T13:44:31.648 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-30T13:44:31.648 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_fast_info": "true", 2024-03-30T13:44:31.648 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_fast_shutdown": "true", 2024-03-30T13:44:31.648 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-30T13:44:31.648 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-30T13:44:31.649 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-30T13:44:31.649 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-30T13:44:31.649 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-30T13:44:31.649 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_function_tracing": "false", 2024-03-30T13:44:31.649 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_heartbeat_grace": "20", 2024-03-30T13:44:31.649 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_heartbeat_interval": "6", 2024-03-30T13:44:31.649 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-30T13:44:31.649 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_heartbeat_min_peers": "10", 2024-03-30T13:44:31.649 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_heartbeat_min_size": "2000", 2024-03-30T13:44:31.649 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_heartbeat_stale": "600", 2024-03-30T13:44:31.649 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-30T13:44:31.649 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_hit_set_max_size": "100000", 2024-03-30T13:44:31.649 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_hit_set_min_size": "1000", 2024-03-30T13:44:31.650 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-30T13:44:31.650 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-30T13:44:31.650 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-30T13:44:31.650 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-30T13:44:31.650 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-30T13:44:31.650 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-30T13:44:31.650 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_journal_size": "5120", 2024-03-30T13:44:31.650 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_kill_backfill_at": "0", 2024-03-30T13:44:31.650 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-30T13:44:31.650 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_map_cache_size": "50", 2024-03-30T13:44:31.650 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_map_dedup": "true", 2024-03-30T13:44:31.650 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_map_message_max": "40", 2024-03-30T13:44:31.650 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-30T13:44:31.651 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_map_share_max_epochs": "40", 2024-03-30T13:44:31.651 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_attr_name_len": "100", 2024-03-30T13:44:31.651 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_attr_size": "0", 2024-03-30T13:44:31.651 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_backfills": "1", 2024-03-30T13:44:31.651 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_markdown_count": "5", 2024-03-30T13:44:31.651 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_markdown_period": "600", 2024-03-30T13:44:31.651 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_object_name_len": "2048", 2024-03-30T13:44:31.651 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_object_namespace_len": "256", 2024-03-30T13:44:31.651 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_object_size": "134217728", 2024-03-30T13:44:31.651 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-30T13:44:31.651 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-30T13:44:31.651 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_pg_blocked_by": "16", 2024-03-30T13:44:31.651 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_pg_log_entries": "10000", 2024-03-30T13:44:31.652 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-30T13:44:31.652 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_pgls": "1024", 2024-03-30T13:44:31.652 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_push_cost": "8388608", 2024-03-30T13:44:31.652 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_push_objects": "10", 2024-03-30T13:44:31.652 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_scrubs": "3", 2024-03-30T13:44:31.652 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-30T13:44:31.652 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_trimming_pgs": "2", 2024-03-30T13:44:31.652 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_write_op_reply_len": "64", 2024-03-30T13:44:31.652 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_write_size": "90", 2024-03-30T13:44:31.652 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-30T13:44:31.652 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-30T13:44:31.652 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-30T13:44:31.652 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-30T13:44:31.652 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-30T13:44:31.653 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-30T13:44:31.653 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-30T13:44:31.653 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-30T13:44:31.653 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_profile": "balanced", 2024-03-30T13:44:31.653 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-30T13:44:31.653 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-30T13:44:31.653 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-30T13:44:31.653 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-30T13:44:31.653 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-30T13:44:31.653 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-30T13:44:31.653 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-30T13:44:31.653 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-30T13:44:31.653 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-30T13:44:31.654 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-30T13:44:31.654 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-30T13:44:31.654 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_memory_base": "805306368", 2024-03-30T13:44:31.654 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_memory_cache_min": "134217728", 2024-03-30T13:44:31.654 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-30T13:44:31.654 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-30T13:44:31.654 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_memory_target": "4294967296", 2024-03-30T13:44:31.654 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_memory_target_autotune": "false", 2024-03-30T13:44:31.654 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-30T13:44:31.654 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_min_pg_log_entries": "250", 2024-03-30T13:44:31.654 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_min_recovery_priority": "0", 2024-03-30T13:44:31.654 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-30T13:44:31.654 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-30T13:44:31.654 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mon_report_interval": "5", 2024-03-30T13:44:31.655 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-30T13:44:31.655 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-30T13:44:31.655 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_num_cache_shards": "32", 2024-03-30T13:44:31.655 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_num_op_tracker_shard": "32", 2024-03-30T13:44:31.655 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_numa_auto_affinity": "true", 2024-03-30T13:44:31.655 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_numa_node": "-1", 2024-03-30T13:44:31.655 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_numa_prefer_iface": "true", 2024-03-30T13:44:31.655 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-30T13:44:31.655 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_objecter_finishers": "1", 2024-03-30T13:44:31.655 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_objectstore": "bluestore", 2024-03-30T13:44:31.655 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_objectstore_fuse": "false", 2024-03-30T13:44:31.655 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_objectstore_tracing": "false", 2024-03-30T13:44:31.655 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_complaint_time": "30.000000", 2024-03-30T13:44:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_history_duration": "600", 2024-03-30T13:44:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_history_size": "20", 2024-03-30T13:44:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_history_slow_op_size": "20", 2024-03-30T13:44:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-30T13:44:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_log_threshold": "5", 2024-03-30T13:44:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_num_shards": "0", 2024-03-30T13:44:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_num_shards_hdd": "5", 2024-03-30T13:44:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_num_shards_ssd": "8", 2024-03-30T13:44:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-30T13:44:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-30T13:44:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-30T13:44:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-30T13:44:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_pq_min_cost": "65536", 2024-03-30T13:44:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-30T13:44:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_queue_cut_off": "high", 2024-03-30T13:44:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-30T13:44:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_thread_timeout": "15", 2024-03-30T13:44:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_open_classes_on_start": "true", 2024-03-30T13:44:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_os_flags": "0", 2024-03-30T13:44:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_peering_op_priority": "255", 2024-03-30T13:44:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_delete_cost": "1048576", 2024-03-30T13:44:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_delete_priority": "5", 2024-03-30T13:44:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-30T13:44:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-30T13:44:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-30T13:44:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_log_trim_max": "10000", 2024-03-30T13:44:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_log_trim_min": "100", 2024-03-30T13:44:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-30T13:44:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-30T13:44:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-30T13:44:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-30T13:44:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-30T13:44:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-30T13:44:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-30T13:44:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-30T13:44:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-30T13:44:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-30T13:44:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_crimson": "false", 2024-03-30T13:44:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-30T13:44:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-30T13:44:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-30T13:44:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-30T13:44:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-30T13:44:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-30T13:44:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-30T13:44:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-30T13:44:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_flags": "0", 2024-03-30T13:44:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-30T13:44:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_min_size": "0", 2024-03-30T13:44:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-30T13:44:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_pg_num": "32", 2024-03-30T13:44:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_pgp_num": "0", 2024-03-30T13:44:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-30T13:44:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_read_ratio": "70", 2024-03-30T13:44:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_size": "3", 2024-03-30T13:44:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_type": "replicated", 2024-03-30T13:44:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-30T13:44:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-30T13:44:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_push_per_object_cost": "1000", 2024-03-30T13:44:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-30T13:44:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recover_clone_overlap": "true", 2024-03-30T13:44:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-30T13:44:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_cost": "20971520", 2024-03-30T13:44:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-30T13:44:31.661 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_max_active": "0", 2024-03-30T13:44:31.661 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-30T13:44:31.661 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-30T13:44:31.661 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-30T13:44:31.661 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-30T13:44:31.661 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_max_single_start": "1", 2024-03-30T13:44:31.661 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_op_priority": "3", 2024-03-30T13:44:31.661 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-30T13:44:31.661 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_priority": "5", 2024-03-30T13:44:31.661 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-30T13:44:31.661 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_sleep": "0.000000", 2024-03-30T13:44:31.661 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-30T13:44:31.661 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-30T13:44:31.661 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-30T13:44:31.661 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_repair_during_recovery": "false", 2024-03-30T13:44:31.662 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_requested_scrub_priority": "120", 2024-03-30T13:44:31.662 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-30T13:44:31.662 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-30T13:44:31.662 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_auto_repair": "false", 2024-03-30T13:44:31.662 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-30T13:44:31.662 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-30T13:44:31.662 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_begin_hour": "0", 2024-03-30T13:44:31.662 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_begin_week_day": "0", 2024-03-30T13:44:31.662 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_chunk_max": "25", 2024-03-30T13:44:31.662 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_chunk_min": "5", 2024-03-30T13:44:31.662 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_cost": "52428800", 2024-03-30T13:44:31.662 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-30T13:44:31.662 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_during_recovery": "false", 2024-03-30T13:44:31.663 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_end_hour": "0", 2024-03-30T13:44:31.663 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_end_week_day": "0", 2024-03-30T13:44:31.663 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_event_cost": "4096", 2024-03-30T13:44:31.663 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-30T13:44:31.663 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-30T13:44:31.663 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_invalid_stats": "true", 2024-03-30T13:44:31.663 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-30T13:44:31.663 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-30T13:44:31.663 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_max_preemptions": "5", 2024-03-30T13:44:31.663 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-30T13:44:31.663 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_priority": "5", 2024-03-30T13:44:31.663 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-30T13:44:31.663 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_sleep": "0.000000", 2024-03-30T13:44:31.664 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-03-30T13:44:31.664 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-30T13:44:31.664 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-30T13:44:31.664 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-30T13:44:31.664 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_skip_data_digest": "false", 2024-03-30T13:44:31.664 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_smart_report_timeout": "5", 2024-03-30T13:44:31.664 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_snap_trim_cost": "1048576", 2024-03-30T13:44:31.664 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_snap_trim_priority": "5", 2024-03-30T13:44:31.664 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-30T13:44:31.664 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-30T13:44:31.664 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-30T13:44:31.664 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-30T13:44:31.664 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-30T13:44:31.664 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-30T13:44:31.665 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-30T13:44:31.665 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_target_transaction_size": "30", 2024-03-30T13:44:31.665 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-30T13:44:31.665 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-30T13:44:31.665 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-30T13:44:31.665 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-30T13:44:31.665 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-30T13:44:31.665 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-30T13:44:31.665 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-30T13:44:31.665 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-30T13:44:31.665 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-30T13:44:31.665 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-30T13:44:31.665 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tracing": "false", 2024-03-30T13:44:31.665 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_use_stale_snap": "false", 2024-03-30T13:44:31.666 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-30T13:44:31.666 INFO:teuthology.orchestra.run.smithi043.stdout: "osdc_blkin_trace_all": "false", 2024-03-30T13:44:31.666 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_kill_at": "0", 2024-03-30T13:44:31.666 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_max_join_drift": "10", 2024-03-30T13:44:31.666 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_min": "500", 2024-03-30T13:44:31.666 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_min_wait": "0.050000", 2024-03-30T13:44:31.666 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_propose_interval": "1.000000", 2024-03-30T13:44:31.666 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_service_trim_max": "500", 2024-03-30T13:44:31.666 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-30T13:44:31.666 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_service_trim_min": "10", 2024-03-30T13:44:31.666 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_stash_full_interval": "25", 2024-03-30T13:44:31.666 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_trim_max": "500", 2024-03-30T13:44:31.666 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_trim_min": "250", 2024-03-30T13:44:31.667 INFO:teuthology.orchestra.run.smithi043.stdout: "perf": "true", 2024-03-30T13:44:31.667 INFO:teuthology.orchestra.run.smithi043.stdout: "pid_file": "", 2024-03-30T13:44:31.667 INFO:teuthology.orchestra.run.smithi043.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-30T13:44:31.667 INFO:teuthology.orchestra.run.smithi043.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-30T13:44:31.667 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "-", 2024-03-30T13:44:31.667 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrv": "", 2024-03-30T13:44:31.667 INFO:teuthology.orchestra.run.smithi043.stdout: "public_bind_addr": "-", 2024-03-30T13:44:31.667 INFO:teuthology.orchestra.run.smithi043.stdout: "public_network": "", 2024-03-30T13:44:31.667 INFO:teuthology.orchestra.run.smithi043.stdout: "public_network_interface": "", 2024-03-30T13:44:31.667 INFO:teuthology.orchestra.run.smithi043.stdout: "qat_compressor_enabled": "false", 2024-03-30T13:44:31.667 INFO:teuthology.orchestra.run.smithi043.stdout: "qat_compressor_session_max_number": "256", 2024-03-30T13:44:31.667 INFO:teuthology.orchestra.run.smithi043.stdout: "rados_mon_op_timeout": "0", 2024-03-30T13:44:31.667 INFO:teuthology.orchestra.run.smithi043.stdout: "rados_osd_op_timeout": "0", 2024-03-30T13:44:31.668 INFO:teuthology.orchestra.run.smithi043.stdout: "rados_tracing": "false", 2024-03-30T13:44:31.668 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_atime_update_interval": "60", 2024-03-30T13:44:31.668 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-30T13:44:31.668 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_balance_parent_reads": "false", 2024-03-30T13:44:31.668 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_balance_snap_reads": "false", 2024-03-30T13:44:31.668 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_blkin_trace_all": "false", 2024-03-30T13:44:31.668 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-30T13:44:31.668 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-30T13:44:31.668 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_cache": "true", 2024-03-30T13:44:31.668 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-30T13:44:31.668 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-30T13:44:31.668 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-30T13:44:31.668 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-30T13:44:31.668 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_cache_policy": "writearound", 2024-03-30T13:44:31.669 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_cache_size": "33554432", 2024-03-30T13:44:31.669 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-30T13:44:31.669 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-30T13:44:31.669 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_clone_copy_on_read": "false", 2024-03-30T13:44:31.669 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_compression_hint": "none", 2024-03-30T13:44:31.669 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_concurrent_management_ops": "10", 2024-03-30T13:44:31.669 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-30T13:44:31.669 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_default_clone_format": "auto", 2024-03-30T13:44:31.669 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_default_data_pool": "", 2024-03-30T13:44:31.669 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_default_features": "61", 2024-03-30T13:44:31.669 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_default_format": "2", 2024-03-30T13:44:31.669 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_default_map_options": "", 2024-03-30T13:44:31.669 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_default_order": "22", 2024-03-30T13:44:31.670 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_default_pool": "rbd", 2024-03-30T13:44:31.670 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-30T13:44:31.670 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_default_stripe_count": "0", 2024-03-30T13:44:31.670 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_default_stripe_unit": "0", 2024-03-30T13:44:31.670 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-30T13:44:31.670 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-30T13:44:31.670 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-30T13:44:31.670 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_enable_alloc_hint": "true", 2024-03-30T13:44:31.670 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-30T13:44:31.670 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_io_scheduler": "simple", 2024-03-30T13:44:31.670 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-30T13:44:31.670 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-30T13:44:31.670 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-30T13:44:31.671 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-30T13:44:31.671 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-30T13:44:31.671 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-30T13:44:31.671 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-30T13:44:31.671 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-30T13:44:31.671 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-30T13:44:31.671 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_order": "24", 2024-03-30T13:44:31.671 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_pool": "", 2024-03-30T13:44:31.671 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_splay_width": "4", 2024-03-30T13:44:31.671 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_localize_parent_reads": "false", 2024-03-30T13:44:31.671 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_localize_snap_reads": "false", 2024-03-30T13:44:31.671 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-30T13:44:31.672 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-30T13:44:31.672 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-30T13:44:31.672 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-30T13:44:31.672 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-30T13:44:31.672 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-30T13:44:31.672 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-30T13:44:31.672 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-30T13:44:31.672 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-30T13:44:31.672 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-30T13:44:31.672 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-30T13:44:31.672 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-30T13:44:31.672 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-30T13:44:31.672 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-30T13:44:31.672 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-30T13:44:31.673 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-30T13:44:31.673 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-30T13:44:31.673 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-30T13:44:31.673 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-30T13:44:31.673 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-30T13:44:31.673 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-30T13:44:31.673 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-30T13:44:31.673 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-30T13:44:31.673 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-30T13:44:31.673 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-30T13:44:31.673 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-30T13:44:31.673 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-30T13:44:31.673 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-30T13:44:31.673 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-30T13:44:31.674 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-30T13:44:31.674 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-30T13:44:31.674 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mtime_update_interval": "60", 2024-03-30T13:44:31.674 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_non_blocking_aio": "true", 2024-03-30T13:44:31.674 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_op_thread_timeout": "60", 2024-03-30T13:44:31.674 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_op_threads": "1", 2024-03-30T13:44:31.674 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_parent_cache_enabled": "false", 2024-03-30T13:44:31.674 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-30T13:44:31.674 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-30T13:44:31.674 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-30T13:44:31.674 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_plugins": "", 2024-03-30T13:44:31.674 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_bps_burst": "0", 2024-03-30T13:44:31.674 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-30T13:44:31.675 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_bps_limit": "0", 2024-03-30T13:44:31.675 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_exclude_ops": "0", 2024-03-30T13:44:31.675 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_iops_burst": "0", 2024-03-30T13:44:31.675 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-30T13:44:31.675 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_iops_limit": "0", 2024-03-30T13:44:31.675 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-30T13:44:31.675 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-30T13:44:31.675 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-30T13:44:31.675 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-30T13:44:31.675 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-30T13:44:31.675 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-30T13:44:31.675 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-30T13:44:31.675 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-30T13:44:31.675 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-30T13:44:31.676 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-30T13:44:31.676 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-30T13:44:31.676 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-30T13:44:31.676 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-30T13:44:31.676 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-30T13:44:31.676 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_read_from_replica_policy": "default", 2024-03-30T13:44:31.676 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-30T13:44:31.676 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-30T13:44:31.676 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-30T13:44:31.676 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-30T13:44:31.676 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_skip_partial_discard": "true", 2024-03-30T13:44:31.676 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-30T13:44:31.676 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_tracing": "false", 2024-03-30T13:44:31.676 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_validate_names": "true", 2024-03-30T13:44:31.677 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_validate_pool": "true", 2024-03-30T13:44:31.677 INFO:teuthology.orchestra.run.smithi043.stdout: "restapi_base_url": "", 2024-03-30T13:44:31.677 INFO:teuthology.orchestra.run.smithi043.stdout: "restapi_log_level": "", 2024-03-30T13:44:31.677 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_acl_grants_max_num": "100", 2024-03-30T13:44:31.677 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_admin_entry": "admin", 2024-03-30T13:44:31.677 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-30T13:44:31.677 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_backend_store": "rados", 2024-03-30T13:44:31.677 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_barbican_url": "", 2024-03-30T13:44:31.677 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_beast_enable_async": "true", 2024-03-30T13:44:31.677 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_bucket_counters_cache": "false", 2024-03-30T13:44:31.677 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-30T13:44:31.677 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-30T13:44:31.677 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-30T13:44:31.677 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-30T13:44:31.678 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-30T13:44:31.678 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-30T13:44:31.678 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-30T13:44:31.678 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-30T13:44:31.678 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_cache_enabled": "true", 2024-03-30T13:44:31.678 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_cache_expiry_interval": "900", 2024-03-30T13:44:31.678 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_cache_lru_size": "10000", 2024-03-30T13:44:31.678 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_config_store": "rados", 2024-03-30T13:44:31.678 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_content_length_compat": "false", 2024-03-30T13:44:31.678 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_copy_obj_progress": "true", 2024-03-30T13:44:31.678 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-30T13:44:31.678 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_cors_rules_max_num": "100", 2024-03-30T13:44:31.678 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_cross_domain_policy": "", 2024-03-30T13:44:31.679 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-30T13:44:31.679 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_kmip_addr": "", 2024-03-30T13:44:31.679 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-30T13:44:31.679 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-30T13:44:31.679 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-30T13:44:31.679 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-30T13:44:31.679 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_kmip_password": "", 2024-03-30T13:44:31.679 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-30T13:44:31.679 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_kmip_username": "", 2024-03-30T13:44:31.679 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_require_ssl": "true", 2024-03-30T13:44:31.679 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-30T13:44:31.679 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-30T13:44:31.679 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-30T13:44:31.679 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-30T13:44:31.680 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-30T13:44:31.680 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-30T13:44:31.680 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-30T13:44:31.680 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-30T13:44:31.680 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-30T13:44:31.680 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-30T13:44:31.680 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-30T13:44:31.680 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-30T13:44:31.680 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-30T13:44:31.680 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-30T13:44:31.680 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-30T13:44:31.680 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_vault_addr": "", 2024-03-30T13:44:31.680 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_vault_auth": "token", 2024-03-30T13:44:31.681 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_vault_namespace": "", 2024-03-30T13:44:31.681 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_vault_prefix": "", 2024-03-30T13:44:31.681 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-30T13:44:31.681 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-30T13:44:31.681 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-30T13:44:31.681 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-30T13:44:31.681 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_vault_token_file": "", 2024-03-30T13:44:31.681 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-30T13:44:31.681 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_curl_buffersize": "524288", 2024-03-30T13:44:31.681 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-30T13:44:31.681 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_curl_low_speed_time": "300", 2024-03-30T13:44:31.681 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-30T13:44:31.681 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-30T13:44:31.681 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-30T13:44:31.682 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-30T13:44:31.682 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-30T13:44:31.682 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-30T13:44:31.682 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-30T13:44:31.682 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-30T13:44:31.682 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-30T13:44:31.682 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-30T13:44:31.682 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-30T13:44:31.682 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d4n_port": "6379", 2024-03-30T13:44:31.682 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-30T13:44:31.682 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_data_log_changes_size": "1000", 2024-03-30T13:44:31.682 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_data_log_num_shards": "128", 2024-03-30T13:44:31.682 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-30T13:44:31.682 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_data_log_window": "30", 2024-03-30T13:44:31.683 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-30T13:44:31.683 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-30T13:44:31.683 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-30T13:44:31.683 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-30T13:44:31.683 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-30T13:44:31.683 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-30T13:44:31.683 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-30T13:44:31.683 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-30T13:44:31.683 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-30T13:44:31.683 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-30T13:44:31.683 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-30T13:44:31.683 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-30T13:44:31.683 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-30T13:44:31.684 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-30T13:44:31.684 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-30T13:44:31.684 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-30T13:44:31.684 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-30T13:44:31.684 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-30T13:44:31.684 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-30T13:44:31.684 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-30T13:44:31.684 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-30T13:44:31.684 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-30T13:44:31.684 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-30T13:44:31.684 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-30T13:44:31.684 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-30T13:44:31.684 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dns_name": "", 2024-03-30T13:44:31.684 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dns_s3website_name": "", 2024-03-30T13:44:31.684 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dynamic_resharding": "true", 2024-03-30T13:44:31.685 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-30T13:44:31.685 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_enable_gc_threads": "true", 2024-03-30T13:44:31.685 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_enable_lc_threads": "true", 2024-03-30T13:44:31.685 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_enable_ops_log": "false", 2024-03-30T13:44:31.685 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_enable_quota_threads": "true", 2024-03-30T13:44:31.685 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_enable_static_website": "false", 2024-03-30T13:44:31.685 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_enable_usage_log": "false", 2024-03-30T13:44:31.685 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_enforce_swift_acls": "true", 2024-03-30T13:44:31.685 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_exit_timeout_secs": "120", 2024-03-30T13:44:31.685 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_expose_bucket": "false", 2024-03-30T13:44:31.685 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_extended_http_attrs": "", 2024-03-30T13:44:31.685 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_filter": "none", 2024-03-30T13:44:31.685 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-30T13:44:31.685 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_frontends": "beast port=7480", 2024-03-30T13:44:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-30T13:44:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_gc_max_deferred": "50", 2024-03-30T13:44:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-30T13:44:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_gc_max_objs": "32", 2024-03-30T13:44:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-30T13:44:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-30T13:44:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-30T13:44:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-30T13:44:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_gc_processor_period": "3600", 2024-03-30T13:44:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-30T13:44:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-30T13:44:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-30T13:44:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-30T13:44:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_init_timeout": "300", 2024-03-30T13:44:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-30T13:44:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-30T13:44:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_kafka_connection_idle": "300", 2024-03-30T13:44:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_kafka_message_timeout": "5000", 2024-03-30T13:44:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-30T13:44:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-30T13:44:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-30T13:44:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-30T13:44:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_admin_domain": "", 2024-03-30T13:44:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_admin_password": "", 2024-03-30T13:44:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_admin_password_path": "", 2024-03-30T13:44:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_admin_project": "", 2024-03-30T13:44:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_admin_tenant": "", 2024-03-30T13:44:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_admin_token": "", 2024-03-30T13:44:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_admin_token_path": "", 2024-03-30T13:44:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_admin_user": "", 2024-03-30T13:44:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_api_version": "2", 2024-03-30T13:44:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_barbican_domain": "", 2024-03-30T13:44:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_barbican_password": "", 2024-03-30T13:44:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_barbican_project": "", 2024-03-30T13:44:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-30T13:44:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_barbican_user": "", 2024-03-30T13:44:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-30T13:44:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-30T13:44:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-30T13:44:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-30T13:44:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-30T13:44:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_url": "", 2024-03-30T13:44:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-30T13:44:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_lc_debug_interval": "-1", 2024-03-30T13:44:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_lc_lock_max_time": "90", 2024-03-30T13:44:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_lc_max_objs": "32", 2024-03-30T13:44:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_lc_max_rules": "1000", 2024-03-30T13:44:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_lc_max_worker": "3", 2024-03-30T13:44:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-30T13:44:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_lc_thread_delay": "0", 2024-03-30T13:44:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-30T13:44:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ldap_dnattr": "uid", 2024-03-30T13:44:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-30T13:44:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ldap_searchfilter": "", 2024-03-30T13:44:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-30T13:44:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-30T13:44:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-30T13:44:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-30T13:44:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-30T13:44:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_log_http_headers": "", 2024-03-30T13:44:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-30T13:44:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-30T13:44:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_log_object_name_utc": "false", 2024-03-30T13:44:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-30T13:44:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-30T13:44:31.691 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_attr_name_len": "0", 2024-03-30T13:44:31.691 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_attr_size": "0", 2024-03-30T13:44:31.691 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-30T13:44:31.691 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_chunk_size": "4194304", 2024-03-30T13:44:31.691 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-30T13:44:31.691 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-30T13:44:31.691 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-30T13:44:31.691 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_listing_results": "1000", 2024-03-30T13:44:31.691 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_notify_retries": "10", 2024-03-30T13:44:31.691 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-30T13:44:31.691 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_put_param_size": "1048576", 2024-03-30T13:44:31.691 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_put_size": "5368709120", 2024-03-30T13:44:31.691 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_slo_entries": "1000", 2024-03-30T13:44:31.692 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_md_log_max_shards": "64", 2024-03-30T13:44:31.692 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-30T13:44:31.692 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-30T13:44:31.692 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-30T13:44:31.692 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-30T13:44:31.692 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_mp_lock_max_time": "600", 2024-03-30T13:44:31.692 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-30T13:44:31.692 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-30T13:44:31.692 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-30T13:44:31.692 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-30T13:44:31.692 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-30T13:44:31.692 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-30T13:44:31.692 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-30T13:44:31.692 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-30T13:44:31.693 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_max_gc": "300", 2024-03-30T13:44:31.693 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-30T13:44:31.693 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-30T13:44:31.693 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-30T13:44:31.693 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-30T13:44:31.693 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-30T13:44:31.693 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-30T13:44:31.693 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-30T13:44:31.693 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_num_async_rados_threads": "32", 2024-03-30T13:44:31.693 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_num_control_oids": "8", 2024-03-30T13:44:31.693 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_numa_node": "-1", 2024-03-30T13:44:31.693 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-30T13:44:31.693 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-30T13:44:31.693 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_objexp_chunk_size": "100", 2024-03-30T13:44:31.693 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_objexp_gc_interval": "600", 2024-03-30T13:44:31.694 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-30T13:44:31.694 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-30T13:44:31.694 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-30T13:44:31.694 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_op_thread_timeout": "600", 2024-03-30T13:44:31.694 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_op_tracing": "false", 2024-03-30T13:44:31.694 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_opa_token": "", 2024-03-30T13:44:31.694 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_opa_url": "", 2024-03-30T13:44:31.694 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_opa_verify_ssl": "true", 2024-03-30T13:44:31.694 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-30T13:44:31.694 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-30T13:44:31.694 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ops_log_rados": "false", 2024-03-30T13:44:31.694 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ops_log_socket_path": "", 2024-03-30T13:44:31.694 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-30T13:44:31.695 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-30T13:44:31.695 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-30T13:44:31.695 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_period_push_interval": "2.000000", 2024-03-30T13:44:31.695 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-30T13:44:31.695 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-30T13:44:31.695 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-30T13:44:31.695 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-30T13:44:31.695 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_posix_cache_lanes": "3", 2024-03-30T13:44:31.695 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-30T13:44:31.695 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-30T13:44:31.695 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_posix_cache_partitions": "3", 2024-03-30T13:44:31.695 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-30T13:44:31.695 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_print_continue": "true", 2024-03-30T13:44:31.695 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-30T13:44:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-30T13:44:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-30T13:44:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-30T13:44:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-30T13:44:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_rados_tracing": "false", 2024-03-30T13:44:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_realm": "", 2024-03-30T13:44:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_realm_id": "", 2024-03-30T13:44:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-30T13:44:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_region": "", 2024-03-30T13:44:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-30T13:44:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-30T13:44:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-30T13:44:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-30T13:44:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_request_uri": "", 2024-03-30T13:44:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_reshard_batch_size": "64", 2024-03-30T13:44:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-30T13:44:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_reshard_max_aio": "128", 2024-03-30T13:44:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_reshard_num_logs": "16", 2024-03-30T13:44:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_reshard_thread_interval": "600", 2024-03-30T13:44:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_resolve_cname": "false", 2024-03-30T13:44:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-30T13:44:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_run_sync_thread": "true", 2024-03-30T13:44:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-30T13:44:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-30T13:44:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-30T13:44:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-30T13:44:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-30T13:44:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-30T13:44:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-30T13:44:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-30T13:44:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_scheduler_type": "throttler", 2024-03-30T13:44:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_script_uri": "", 2024-03-30T13:44:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_service_provider_name": "", 2024-03-30T13:44:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-30T13:44:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sts_client_id": "", 2024-03-30T13:44:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sts_client_secret": "", 2024-03-30T13:44:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sts_entry": "sts", 2024-03-30T13:44:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sts_key": "sts", 2024-03-30T13:44:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-30T13:44:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sts_min_session_duration": "900", 2024-03-30T13:44:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sts_token_introspection_url": "", 2024-03-30T13:44:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_account_in_url": "false", 2024-03-30T13:44:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_auth_entry": "auth", 2024-03-30T13:44:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_auth_url": "", 2024-03-30T13:44:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_custom_header": "", 2024-03-30T13:44:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-30T13:44:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_need_stats": "true", 2024-03-30T13:44:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_tenant_name": "", 2024-03-30T13:44:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_token_expiration": "86400", 2024-03-30T13:44:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_url": "", 2024-03-30T13:44:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_url_prefix": "swift", 2024-03-30T13:44:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-30T13:44:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-30T13:44:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-30T13:44:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_lease_period": "120", 2024-03-30T13:44:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-30T13:44:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-30T13:44:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-30T13:44:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-30T13:44:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-30T13:44:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-30T13:44:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-30T13:44:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-30T13:44:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-30T13:44:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_thread_pool_size": "512", 2024-03-30T13:44:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-30T13:44:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-30T13:44:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-30T13:44:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-30T13:44:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_torrent_comment": "", 2024-03-30T13:44:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_torrent_createby": "", 2024-03-30T13:44:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_torrent_encoding": "", 2024-03-30T13:44:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_torrent_flag": "false", 2024-03-30T13:44:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-30T13:44:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_torrent_origin": "", 2024-03-30T13:44:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-30T13:44:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_torrent_tracker": "", 2024-03-30T13:44:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_trust_forwarded_https": "false", 2024-03-30T13:44:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-30T13:44:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-30T13:44:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_usage_max_shards": "32", 2024-03-30T13:44:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_usage_max_user_shards": "1", 2024-03-30T13:44:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_use_opa_authz": "false", 2024-03-30T13:44:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_counters_cache": "false", 2024-03-30T13:44:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-30T13:44:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-30T13:44:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-30T13:44:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_max_buckets": "1000", 2024-03-30T13:44:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_policies_max_num": "100", 2024-03-30T13:44:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-30T13:44:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-30T13:44:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-30T13:44:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-30T13:44:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_unique_email": "true", 2024-03-30T13:44:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_verify_ssl": "true", 2024-03-30T13:44:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-30T13:44:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_zone": "", 2024-03-30T13:44:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_zone_id": "", 2024-03-30T13:44:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-30T13:44:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_zonegroup": "", 2024-03-30T13:44:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_zonegroup_id": "", 2024-03-30T13:44:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-30T13:44:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "rgwlc_auto_session_clear": "true", 2024-03-30T13:44:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-30T13:44:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_block_size": "4096", 2024-03-30T13:44:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-30T13:44:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-30T13:44:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-30T13:44:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-30T13:44:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-30T13:44:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_cache_size": "536870912", 2024-03-30T13:44:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-30T13:44:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-30T13:44:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-30T13:44:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-30T13:44:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-30T13:44:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-30T13:44:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-30T13:44:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-30T13:44:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_index_type": "binary_search", 2024-03-30T13:44:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-30T13:44:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-30T13:44:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_partition_filters": "false", 2024-03-30T13:44:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_perf": "false", 2024-03-30T13:44:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-30T13:44:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-30T13:44:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-30T13:44:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "run_dir": "/var/run/ceph", 2024-03-30T13:44:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_block_create": "true", 2024-03-30T13:44:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_cache_lru_size": "67108864", 2024-03-30T13:44:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-30T13:44:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-30T13:44:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_default_max_object_size": "16777216", 2024-03-30T13:44:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-30T13:44:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_device_size": "53687091200", 2024-03-30T13:44:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_journal_batch_capacity": "16", 2024-03-30T13:44:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-30T13:44:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-30T13:44:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-30T13:44:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_main_device_type": "SSD", 2024-03-30T13:44:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-30T13:44:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-30T13:44:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-30T13:44:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-30T13:44:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-30T13:44:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_segment_size": "67108864", 2024-03-30T13:44:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "set_keepcaps": "false", 2024-03-30T13:44:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "setgroup": "ceph", 2024-03-30T13:44:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "setuser": "ceph", 2024-03-30T13:44:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "setuser_match_path": "", 2024-03-30T13:44:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-30T13:44:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "thp": "false", 2024-03-30T13:44:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "threadpool_default_timeout": "60", 2024-03-30T13:44:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-30T13:44:31.707 INFO:teuthology.orchestra.run.smithi043.stdout: "throttler_perf_counter": "true" 2024-03-30T13:44:31.707 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:44:31.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:31 smithi113 ceph-mon[34341]: 5.15 scrub starts 2024-03-30T13:44:31.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:31 smithi113 ceph-mon[34341]: 5.15 scrub ok 2024-03-30T13:44:32.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:32 smithi043 ceph-mon[29963]: 6.e scrub starts 2024-03-30T13:44:32.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:32 smithi043 ceph-mon[29963]: 6.e scrub ok 2024-03-30T13:44:32.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:32 smithi043 ceph-mon[29963]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:44:32.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:32 smithi043 ceph-mon[29963]: 2.0 scrub starts 2024-03-30T13:44:32.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:32 smithi043 ceph-mon[29963]: 2.0 scrub ok 2024-03-30T13:44:32.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:32 smithi043 ceph-mon[29963]: 3.1b scrub starts 2024-03-30T13:44:32.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:32 smithi043 ceph-mon[29963]: 3.1b scrub ok 2024-03-30T13:44:32.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:32 smithi043 ceph-mon[42099]: 6.e scrub starts 2024-03-30T13:44:32.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:32 smithi043 ceph-mon[42099]: 6.e scrub ok 2024-03-30T13:44:32.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:32 smithi043 ceph-mon[42099]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:44:32.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:32 smithi043 ceph-mon[42099]: 2.0 scrub starts 2024-03-30T13:44:32.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:32 smithi043 ceph-mon[42099]: 2.0 scrub ok 2024-03-30T13:44:32.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:32 smithi043 ceph-mon[42099]: 3.1b scrub starts 2024-03-30T13:44:32.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:32 smithi043 ceph-mon[42099]: 3.1b scrub ok 2024-03-30T13:44:32.609 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-30T13:44:32.655 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:32 smithi113 ceph-mon[34341]: 6.e scrub starts 2024-03-30T13:44:32.656 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:32 smithi113 ceph-mon[34341]: 6.e scrub ok 2024-03-30T13:44:32.656 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:32 smithi113 ceph-mon[34341]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:44:32.656 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:32 smithi113 ceph-mon[34341]: 2.0 scrub starts 2024-03-30T13:44:32.656 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:32 smithi113 ceph-mon[34341]: 2.0 scrub ok 2024-03-30T13:44:32.656 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:32 smithi113 ceph-mon[34341]: 3.1b scrub starts 2024-03-30T13:44:32.656 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:32 smithi113 ceph-mon[34341]: 3.1b scrub ok 2024-03-30T13:44:32.855 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:44:32.972 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:32 smithi113 bash[68862]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-30T13:44:32.972 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:32 smithi113 bash[68862]: Writing manifest to image destination 2024-03-30T13:44:33.345 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 podman[68862]: 2024-03-30 13:44:33.069270333 +0000 UTC m=+3.918012680 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-30T13:44:33.345 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 podman[68862]: 2024-03-30 13:44:33.168379995 +0000 UTC m=+4.017122388 container create 5e993f09406b6f5614f70bf613ff8cad6e6910c5f23f532fd6512a2867f78b2e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-30T13:44:33.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:33 smithi043 ceph-mon[29963]: 6.a scrub starts 2024-03-30T13:44:33.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:33 smithi043 ceph-mon[29963]: 6.a scrub ok 2024-03-30T13:44:33.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:33 smithi043 ceph-mon[42099]: 6.a scrub starts 2024-03-30T13:44:33.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:33 smithi043 ceph-mon[42099]: 6.a scrub ok 2024-03-30T13:44:33.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-mon[34341]: 6.a scrub starts 2024-03-30T13:44:33.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-mon[34341]: 6.a scrub ok 2024-03-30T13:44:33.723 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 podman[68862]: 2024-03-30 13:44:33.412749697 +0000 UTC m=+4.261492048 container init 5e993f09406b6f5614f70bf613ff8cad6e6910c5f23f532fd6512a2867f78b2e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-30T13:44:33.723 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 podman[68862]: 2024-03-30 13:44:33.418774654 +0000 UTC m=+4.267517004 container start 5e993f09406b6f5614f70bf613ff8cad6e6910c5f23f532fd6512a2867f78b2e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-30T13:44:33.723 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.422Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-30T13:44:33.723 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.422Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-30T13:44:33.723 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.423Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-30T13:44:33.723 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.423Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-30T13:44:33.723 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.423Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-30T13:44:33.723 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.423Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-30T13:44:33.723 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.423Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-30T13:44:33.723 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=arp 2024-03-30T13:44:33.723 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-30T13:44:33.724 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-30T13:44:33.724 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-30T13:44:33.724 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-30T13:44:33.724 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-30T13:44:33.724 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-30T13:44:33.724 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-30T13:44:33.724 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-30T13:44:33.724 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=edac 2024-03-30T13:44:33.724 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-30T13:44:33.724 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-30T13:44:33.724 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-30T13:44:33.724 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-30T13:44:33.724 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-30T13:44:33.725 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-30T13:44:33.725 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-30T13:44:33.725 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-30T13:44:33.725 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-30T13:44:33.725 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-30T13:44:33.725 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-30T13:44:33.725 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-30T13:44:33.725 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-30T13:44:33.725 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-30T13:44:33.725 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-30T13:44:33.725 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-30T13:44:33.725 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=os 2024-03-30T13:44:33.725 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-30T13:44:33.725 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-30T13:44:33.726 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-30T13:44:33.726 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-30T13:44:33.726 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-30T13:44:33.726 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-30T13:44:33.726 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-30T13:44:33.726 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=stat 2024-03-30T13:44:33.726 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-30T13:44:33.726 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-30T13:44:33.726 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-30T13:44:33.726 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=time 2024-03-30T13:44:33.726 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-30T13:44:33.726 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=uname 2024-03-30T13:44:33.726 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-30T13:44:33.726 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-30T13:44:33.727 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-30T13:44:33.727 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-30T13:44:33.727 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b[68914]: ts=2024-03-30T13:44:33.424Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-30T13:44:33.727 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 bash[68862]: 5e993f09406b6f5614f70bf613ff8cad6e6910c5f23f532fd6512a2867f78b2e 2024-03-30T13:44:33.727 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 13:44:33 smithi113 systemd[1]: Started Ceph node-exporter.b for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T13:44:34.569 INFO:teuthology.orchestra.run.smithi043.stdout:mon.a: {} 2024-03-30T13:44:34.581 INFO:teuthology.orchestra.run.smithi043.stdout:mon.b: {} 2024-03-30T13:44:34.591 INFO:teuthology.orchestra.run.smithi043.stdout:mon.c: {} 2024-03-30T13:44:34.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[29963]: 4.c scrub starts 2024-03-30T13:44:34.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[29963]: 4.c scrub ok 2024-03-30T13:44:34.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[29963]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T13:44:34.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:34.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:34.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:34.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:34.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:34.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:44:34.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[29963]: 3.12 deep-scrub starts 2024-03-30T13:44:34.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[29963]: 3.12 deep-scrub ok 2024-03-30T13:44:34.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/713939502' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T13:44:34.593 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[42099]: 4.c scrub starts 2024-03-30T13:44:34.593 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[42099]: 4.c scrub ok 2024-03-30T13:44:34.593 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[42099]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T13:44:34.593 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:34.593 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:34.593 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:34.593 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:34.593 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:34.593 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:44:34.594 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[42099]: 3.12 deep-scrub starts 2024-03-30T13:44:34.594 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[42099]: 3.12 deep-scrub ok 2024-03-30T13:44:34.594 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:34 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/713939502' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T13:44:34.594 INFO:teuthology.orchestra.run.smithi043.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-30T13:44:34.594 INFO:teuthology.orchestra.run.smithi043.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-30T13:44:34.594 INFO:teuthology.orchestra.run.smithi043.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-30T13:44:34.942 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:34 smithi113 ceph-mon[34341]: 4.c scrub starts 2024-03-30T13:44:34.943 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:34 smithi113 ceph-mon[34341]: 4.c scrub ok 2024-03-30T13:44:34.943 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:34 smithi113 ceph-mon[34341]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T13:44:34.943 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:34 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:34.943 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:34 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:34.943 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:34 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:34.943 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:34 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:34.943 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:34 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:34.943 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:34 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:44:34.943 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:34 smithi113 ceph-mon[34341]: 3.12 deep-scrub starts 2024-03-30T13:44:34.943 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:34 smithi113 ceph-mon[34341]: 3.12 deep-scrub ok 2024-03-30T13:44:34.943 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:34 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/713939502' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T13:44:35.270 INFO:teuthology.run_tasks:Running task full_sequential... 2024-03-30T13:44:35.283 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-30T13:44:35.295 INFO:tasks.radosbench:Beginning radosbench... 2024-03-30T13:44:35.295 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-03-30T13:44:35.295 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool create unique_pool_0 16 2024-03-30T13:44:35.296 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-30T13:44:35.296 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T13:44:35.296 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:44:35.296 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-03-30T13:44:35.296 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-30T13:44:35.297 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-30T13:44:35.297 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-03-30T13:44:35.297 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-30T13:44:35.297 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:35.298 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-30T13:44:35.298 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:44:35.298 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-30T13:44:35.298 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-30T13:44:35.299 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-30T13:44:35.299 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T13:44:35.333 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:35.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:35.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:35.435 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:35.459 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:35.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:35.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:35.561 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:44:35.585 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:35.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:35.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:44:35.687 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:35.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:35.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:35.819 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:35.844 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:35.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:35.872 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:35 smithi113 ceph-mon[34341]: 5.4 scrub starts 2024-03-30T13:44:35.872 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:35 smithi113 ceph-mon[34341]: 5.4 scrub ok 2024-03-30T13:44:35.872 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:35 smithi113 ceph-mon[34341]: 5.12 deep-scrub starts 2024-03-30T13:44:35.872 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:35 smithi113 ceph-mon[34341]: 5.12 deep-scrub ok 2024-03-30T13:44:35.872 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:35 smithi113 ceph-mon[34341]: 6.b scrub starts 2024-03-30T13:44:35.872 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:35 smithi113 ceph-mon[34341]: 6.b scrub ok 2024-03-30T13:44:35.872 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:35 smithi113 ceph-mon[34341]: 5.3 scrub starts 2024-03-30T13:44:35.872 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:35 smithi113 ceph-mon[34341]: 5.3 scrub ok 2024-03-30T13:44:35.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:35 smithi043 ceph-mon[29963]: 5.4 scrub starts 2024-03-30T13:44:35.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:35 smithi043 ceph-mon[29963]: 5.4 scrub ok 2024-03-30T13:44:35.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:35 smithi043 ceph-mon[29963]: 5.12 deep-scrub starts 2024-03-30T13:44:35.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:35 smithi043 ceph-mon[29963]: 5.12 deep-scrub ok 2024-03-30T13:44:35.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:35 smithi043 ceph-mon[29963]: 6.b scrub starts 2024-03-30T13:44:35.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:35 smithi043 ceph-mon[29963]: 6.b scrub ok 2024-03-30T13:44:35.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:35 smithi043 ceph-mon[29963]: 5.3 scrub starts 2024-03-30T13:44:35.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:35 smithi043 ceph-mon[29963]: 5.3 scrub ok 2024-03-30T13:44:35.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:35 smithi043 ceph-mon[42099]: 5.4 scrub starts 2024-03-30T13:44:35.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:35 smithi043 ceph-mon[42099]: 5.4 scrub ok 2024-03-30T13:44:35.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:35 smithi043 ceph-mon[42099]: 5.12 deep-scrub starts 2024-03-30T13:44:35.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:35 smithi043 ceph-mon[42099]: 5.12 deep-scrub ok 2024-03-30T13:44:35.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:35 smithi043 ceph-mon[42099]: 6.b scrub starts 2024-03-30T13:44:35.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:35 smithi043 ceph-mon[42099]: 6.b scrub ok 2024-03-30T13:44:35.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:35 smithi043 ceph-mon[42099]: 5.3 scrub starts 2024-03-30T13:44:35.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:35 smithi043 ceph-mon[42099]: 5.3 scrub ok 2024-03-30T13:44:35.947 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:35.968 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:44:35.968 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:44:35.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:35.977 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:35.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:36.078 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:36.102 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:36.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:36.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:36.203 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:36.212 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:44:36.227 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:36.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:36.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:36.329 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:36.352 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:36.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:36.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:36.394 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:44:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:44:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:44:36.454 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:36.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:36.478 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:36.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:36.579 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:36.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:36.604 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:36.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:36.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:36 smithi043 ceph-mon[29963]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:36.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:36 smithi043 ceph-mon[29963]: 2.2 scrub starts 2024-03-30T13:44:36.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:36 smithi043 ceph-mon[29963]: 2.2 scrub ok 2024-03-30T13:44:36.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:36 smithi043 ceph-mon[42099]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:36.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:36 smithi043 ceph-mon[42099]: 2.2 scrub starts 2024-03-30T13:44:36.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:36 smithi043 ceph-mon[42099]: 2.2 scrub ok 2024-03-30T13:44:36.705 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:36.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:36 smithi113 ceph-mon[34341]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:36.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:36 smithi113 ceph-mon[34341]: 2.2 scrub starts 2024-03-30T13:44:36.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:36 smithi113 ceph-mon[34341]: 2.2 scrub ok 2024-03-30T13:44:36.729 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:36.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:36.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:36.831 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:36.855 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:36.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:36.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:36.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:36.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:36.981 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:36.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:37.083 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:37.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:37.107 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:37.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:37.208 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:37.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:37.233 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:37.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:37.334 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:37.358 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:37.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:37.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:37.460 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:37.483 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:37.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:37.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:37.585 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:37.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:37.609 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:37.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:37.710 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:37.734 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:37.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:37.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:37.789 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:37 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:44:37.789 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:37 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:44:37.789 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:37 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:37.836 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:37.860 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:37.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:37.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:37.862 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:37 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:44:37.862 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:37 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:44:37.862 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:37 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:37.862 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:37 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:44:37.863 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:37 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:44:37.863 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:37 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:37.897 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:44:37.897 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:44:37.961 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:37.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:37.986 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:37.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:38.087 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:38.110 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:38.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:38.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:38.212 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:44:38.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:38.237 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:38.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:44:38.338 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:38.363 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:38.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:38.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:38.389 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:44:38.389 INFO:teuthology.orchestra.run.smithi113.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-30T13:44:38.389 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:44:38.465 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:38.490 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:38.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:38.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:38.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:44:38 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:44:38.592 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:38.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:38.616 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:38.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:38.718 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:38.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:38.742 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:38.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:38.843 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:38.866 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:38.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:38.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:38.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:38 smithi043 ceph-mon[29963]: 3.3 scrub starts 2024-03-30T13:44:38.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:38 smithi043 ceph-mon[29963]: 3.3 scrub ok 2024-03-30T13:44:38.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:38 smithi043 ceph-mon[29963]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:38.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:38 smithi043 ceph-mon[29963]: 6.9 deep-scrub starts 2024-03-30T13:44:38.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:38 smithi043 ceph-mon[29963]: 6.9 deep-scrub ok 2024-03-30T13:44:38.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:38 smithi043 ceph-mon[29963]: osdmap e96: 8 total, 8 up, 8 in 2024-03-30T13:44:38.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:38 smithi043 ceph-mon[42099]: 3.3 scrub starts 2024-03-30T13:44:38.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:38 smithi043 ceph-mon[42099]: 3.3 scrub ok 2024-03-30T13:44:38.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:38 smithi043 ceph-mon[42099]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:38.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:38 smithi043 ceph-mon[42099]: 6.9 deep-scrub starts 2024-03-30T13:44:38.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:38 smithi043 ceph-mon[42099]: 6.9 deep-scrub ok 2024-03-30T13:44:38.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:38 smithi043 ceph-mon[42099]: osdmap e96: 8 total, 8 up, 8 in 2024-03-30T13:44:38.968 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:38.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:38 smithi113 ceph-mon[34341]: 3.3 scrub starts 2024-03-30T13:44:38.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:38 smithi113 ceph-mon[34341]: 3.3 scrub ok 2024-03-30T13:44:38.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:38 smithi113 ceph-mon[34341]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:38.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:38 smithi113 ceph-mon[34341]: 6.9 deep-scrub starts 2024-03-30T13:44:38.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:38 smithi113 ceph-mon[34341]: 6.9 deep-scrub ok 2024-03-30T13:44:38.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:38 smithi113 ceph-mon[34341]: osdmap e96: 8 total, 8 up, 8 in 2024-03-30T13:44:38.994 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:38.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:38.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:39.095 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:39.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:39.119 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:39.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:39.220 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:39.243 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:39.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:39.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:39.345 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:39.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:39.370 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:39.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:39.471 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:39.496 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:39.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:39.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:39.598 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:39.622 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:39.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:39.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:39.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:39 smithi113 ceph-mon[34341]: 3.14 scrub starts 2024-03-30T13:44:39.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:39 smithi113 ceph-mon[34341]: 3.14 scrub ok 2024-03-30T13:44:39.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:39 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:44:39.724 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:39.748 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:39.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:39.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:39.849 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:39.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:39.875 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:39.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:39.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:39 smithi043 ceph-mon[29963]: 3.14 scrub starts 2024-03-30T13:44:39.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:39 smithi043 ceph-mon[29963]: 3.14 scrub ok 2024-03-30T13:44:39.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:39 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:44:39.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:39 smithi043 ceph-mon[42099]: 3.14 scrub starts 2024-03-30T13:44:39.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:39 smithi043 ceph-mon[42099]: 3.14 scrub ok 2024-03-30T13:44:39.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:39 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:44:39.976 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:40.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:40.000 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:40.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:40.101 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:40.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:40.125 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:40.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:40.226 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:40.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:40.255 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:40.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:40.357 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:40.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:40.381 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:40.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:40.482 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:40.513 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:40.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:40.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:40.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:40 smithi043 ceph-mon[29963]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:40.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:40 smithi043 ceph-mon[29963]: 3.16 scrub starts 2024-03-30T13:44:40.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:40 smithi043 ceph-mon[29963]: 3.16 scrub ok 2024-03-30T13:44:40.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:40 smithi043 ceph-mon[42099]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:40.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:40 smithi043 ceph-mon[42099]: 3.16 scrub starts 2024-03-30T13:44:40.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:40 smithi043 ceph-mon[42099]: 3.16 scrub ok 2024-03-30T13:44:40.615 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:40.639 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:40.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:40.740 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:40.744 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:44:40.744 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T13:44:40.744 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 0 2024-03-30T13:44:40.744 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:44:40.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:40.765 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:40.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:40.866 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:40.890 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:40.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:40.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:40.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:40 smithi113 ceph-mon[34341]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:40.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:40 smithi113 ceph-mon[34341]: 3.16 scrub starts 2024-03-30T13:44:40.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:40 smithi113 ceph-mon[34341]: 3.16 scrub ok 2024-03-30T13:44:40.991 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:41.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:41.017 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:41.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:41.119 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:41.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:41.146 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:41.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:41.247 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:41.271 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:41.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:41.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:41.373 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:44:41.396 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:41.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:41.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:44:41.498 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:41.519 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:44:41.521 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:41.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:41.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:41.539 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:44:41.560 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:44:41.580 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:44:41.600 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:44:41.622 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:44:41.622 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:41.643 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:44:41.647 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:41.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:41.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:41.663 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:44:41.664 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:44:41.665 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:44:41.665 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:44:41.665 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:44:41.665 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:44:41.665 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:44:41.665 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:44:41.665 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:44:41.749 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:44:41.773 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:41.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:41.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:44:41.774 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:41 smithi043 ceph-mon[29963]: 5.f scrub starts 2024-03-30T13:44:41.774 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:41 smithi043 ceph-mon[29963]: 5.f scrub ok 2024-03-30T13:44:41.774 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:41 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:41.774 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:41 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:41.774 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:41 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3583263435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:44:41.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:41 smithi043 ceph-mon[42099]: 5.f scrub starts 2024-03-30T13:44:41.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:41 smithi043 ceph-mon[42099]: 5.f scrub ok 2024-03-30T13:44:41.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:41 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:41.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:41 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:41.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:41 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3583263435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:44:41.874 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:41.898 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:41.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:41.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:41 smithi113 ceph-mon[34341]: 5.f scrub starts 2024-03-30T13:44:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:41 smithi113 ceph-mon[34341]: 5.f scrub ok 2024-03-30T13:44:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:41 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:41 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:41 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3583263435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:44:42.000 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:42.023 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:42.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:42.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:42.125 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:42.149 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:42.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:42.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:42.251 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:42.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:42.276 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:42.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:42.377 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:42.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:42.403 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:42.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:42.505 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:42.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:42.627 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:42.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:42.729 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:42.752 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:42.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:42.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:42.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:42 smithi043 ceph-mon[29963]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:42.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:42 smithi043 ceph-mon[29963]: 5.1a deep-scrub starts 2024-03-30T13:44:42.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:42 smithi043 ceph-mon[29963]: 5.1a deep-scrub ok 2024-03-30T13:44:42.795 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:42 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:44:42.795 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:42 smithi043 ceph-mon[42099]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:42.795 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:42 smithi043 ceph-mon[42099]: 5.1a deep-scrub starts 2024-03-30T13:44:42.795 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:42 smithi043 ceph-mon[42099]: 5.1a deep-scrub ok 2024-03-30T13:44:42.795 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:42 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:44:42.820 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-03-30T13:44:42.854 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:42.886 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:42.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:42.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:42.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:42 smithi113 ceph-mon[34341]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:42.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:42 smithi113 ceph-mon[34341]: 5.1a deep-scrub starts 2024-03-30T13:44:42.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:42 smithi113 ceph-mon[34341]: 5.1a deep-scrub ok 2024-03-30T13:44:42.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:42 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:44:42.989 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:44:43.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:43.016 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:43.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:44:43.117 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:43.141 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:43.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:43.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:43.243 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:43.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:43.267 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:43.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:43.368 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:43.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:43.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:43.494 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:43.518 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:43.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:43.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:43.620 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:43.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:43 smithi043 ceph-mon[29963]: 4.18 scrub starts 2024-03-30T13:44:43.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:43 smithi043 ceph-mon[29963]: 4.18 scrub ok 2024-03-30T13:44:43.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:43 smithi043 ceph-mon[42099]: 4.18 scrub starts 2024-03-30T13:44:43.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:43 smithi043 ceph-mon[42099]: 4.18 scrub ok 2024-03-30T13:44:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:43.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:43.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:43.748 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:43.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:43.772 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:43.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:43.873 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:43.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:43.899 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:43.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:43.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:43 smithi113 ceph-mon[34341]: 4.18 scrub starts 2024-03-30T13:44:43.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:43 smithi113 ceph-mon[34341]: 4.18 scrub ok 2024-03-30T13:44:44.001 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:44.029 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:44.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:44.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:44.091 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T13:44:44.091 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:44:44.091 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-03-30T13:44:44.091 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 reset_purged_snaps_last 2024-03-30T13:44:44.131 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:44:44.159 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:44.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:44.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:44:44.261 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:44.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:44.285 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:44.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:44.386 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:44.412 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:44.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:44.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:44.514 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:44.539 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:44.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:44.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:44.621 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-03-30T13:44:44.641 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:44.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:44.666 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:44.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:44.716 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:44 smithi043 ceph-mon[29963]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:44.716 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:44 smithi043 ceph-mon[29963]: 4.8 scrub starts 2024-03-30T13:44:44.716 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:44 smithi043 ceph-mon[29963]: 4.8 scrub ok 2024-03-30T13:44:44.716 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:44 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3266318176' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:44:44.717 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:44 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:44:44.717 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:44 smithi043 ceph-mon[42099]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:44.717 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:44 smithi043 ceph-mon[42099]: 4.8 scrub starts 2024-03-30T13:44:44.717 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:44 smithi043 ceph-mon[42099]: 4.8 scrub ok 2024-03-30T13:44:44.717 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:44 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3266318176' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:44:44.717 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:44 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:44:44.767 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:44.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:44.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:44.893 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:44.917 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:44.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:44.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:44 smithi113 ceph-mon[34341]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:44:44.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:44 smithi113 ceph-mon[34341]: 4.8 scrub starts 2024-03-30T13:44:44.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:44 smithi113 ceph-mon[34341]: 4.8 scrub ok 2024-03-30T13:44:44.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:44 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3266318176' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:44:44.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:44 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:44:45.018 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:45.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:45.043 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:45.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:45.144 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:45.172 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:45.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:45.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:45.274 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:45.302 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:45.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:45.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:45.372 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:44:45.373 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:44:45.393 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:44:45.404 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:45.460 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:45.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:45.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:45.562 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:45.586 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:45.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:45.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:45.631 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'unique_pool_0' created 2024-03-30T13:44:45.688 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:45.715 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:45.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:45.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:45.817 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:45.829 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:45 smithi043 ceph-mon[29963]: 6.d scrub starts 2024-03-30T13:44:45.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:45 smithi043 ceph-mon[29963]: 6.d scrub ok 2024-03-30T13:44:45.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:45 smithi043 ceph-mon[29963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:44:45.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:45 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:44:45.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:45 smithi043 ceph-mon[29963]: osdmap e97: 8 total, 8 up, 8 in 2024-03-30T13:44:45.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:45 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2596658268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-30T13:44:45.830 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:45 smithi043 ceph-mon[42099]: 6.d scrub starts 2024-03-30T13:44:45.830 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:45 smithi043 ceph-mon[42099]: 6.d scrub ok 2024-03-30T13:44:45.830 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:45 smithi043 ceph-mon[42099]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:44:45.830 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:45 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:44:45.830 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:45 smithi043 ceph-mon[42099]: osdmap e97: 8 total, 8 up, 8 in 2024-03-30T13:44:45.831 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:45 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2596658268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-30T13:44:45.842 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:45.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:45.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:45.945 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:45.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:45 smithi113 ceph-mon[34341]: 6.d scrub starts 2024-03-30T13:44:45.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:45 smithi113 ceph-mon[34341]: 6.d scrub ok 2024-03-30T13:44:45.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:45 smithi113 ceph-mon[34341]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:44:45.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:45 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:44:45.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:45 smithi113 ceph-mon[34341]: osdmap e97: 8 total, 8 up, 8 in 2024-03-30T13:44:45.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:45 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2596658268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-30T13:44:45.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:45.973 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:45.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:46.074 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:46.100 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:46.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:46.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:46.202 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:46.230 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:46.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:46.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:46.332 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:46.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:46.360 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:46.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:46.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:44:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:44:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:44:46.461 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:44:46.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:46.491 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:46.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:44:46.592 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:44:46.619 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:46.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:46.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:44:46.668 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-03-30T13:44:46.721 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:46.748 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:46.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:46.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:46.850 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:46.875 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:46 smithi043 ceph-mon[29963]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:46.876 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:46 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2596658268' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-30T13:44:46.876 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:46 smithi043 ceph-mon[29963]: osdmap e98: 8 total, 8 up, 8 in 2024-03-30T13:44:46.876 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:46 smithi043 ceph-mon[42099]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:46.876 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:46 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2596658268' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-30T13:44:46.876 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:46 smithi043 ceph-mon[42099]: osdmap e98: 8 total, 8 up, 8 in 2024-03-30T13:44:46.876 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:46.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:46.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:46.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:46 smithi113 ceph-mon[34341]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:46.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:46 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2596658268' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-30T13:44:46.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:46 smithi113 ceph-mon[34341]: osdmap e98: 8 total, 8 up, 8 in 2024-03-30T13:44:46.978 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:44:47.005 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:47.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:47.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:44:47.106 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:47.134 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:47.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:47.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:47.236 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:47.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:47.259 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:47.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:47.360 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:47.384 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:47.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:47.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:47.485 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:47.514 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:47.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:47.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:47.616 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:47.644 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:47.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:47.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:47.746 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:47.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:47.775 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:47.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:47.858 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T13:44:47.876 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:47.923 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:47 smithi043 ceph-mon[29963]: osdmap e99: 8 total, 8 up, 8 in 2024-03-30T13:44:47.923 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:47 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:47.923 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:47 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:47.923 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:47 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:44:47.923 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:47 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:44:47.923 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:47 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:47.923 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:47 smithi043 ceph-mon[29963]: osdmap e100: 8 total, 8 up, 8 in 2024-03-30T13:44:47.923 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:47 smithi043 ceph-mon[42099]: osdmap e99: 8 total, 8 up, 8 in 2024-03-30T13:44:47.923 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:47 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:47.924 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:47 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:47.924 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:47 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:44:47.924 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:47 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:44:47.924 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:47 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:47.924 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:47 smithi043 ceph-mon[42099]: osdmap e100: 8 total, 8 up, 8 in 2024-03-30T13:44:47.931 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:47.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:47.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:47.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:47 smithi113 ceph-mon[34341]: osdmap e99: 8 total, 8 up, 8 in 2024-03-30T13:44:47.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:47 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:47.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:47 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:47.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:47 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:44:47.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:47 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:44:47.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:47 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:47.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:47 smithi113 ceph-mon[34341]: osdmap e100: 8 total, 8 up, 8 in 2024-03-30T13:44:48.033 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:48.061 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:48.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:48.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:48.163 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:48.188 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:48.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:48.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:48.290 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:48.318 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:48.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:48.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:48.420 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:48.445 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:48.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:48.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:48.547 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:48.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:48.572 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:48.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:48.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:44:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:44:48.673 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:48.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:48.701 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:48.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:48.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:48.825 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:48.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:48.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:48.926 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:48.950 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:48.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:48.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:48.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:48 smithi113 ceph-mon[34341]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-30T13:44:48.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:48 smithi113 ceph-mon[34341]: Reconfiguring daemon alertmanager.a on smithi043 2024-03-30T13:44:48.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:48 smithi113 ceph-mon[34341]: pgmap v61: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:44:48.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:48 smithi113 ceph-mon[34341]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:44:49.008 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:48 smithi043 ceph-mon[29963]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-30T13:44:49.008 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:48 smithi043 ceph-mon[29963]: Reconfiguring daemon alertmanager.a on smithi043 2024-03-30T13:44:49.008 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:48 smithi043 ceph-mon[29963]: pgmap v61: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:44:49.008 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:48 smithi043 ceph-mon[29963]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:44:49.009 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:48 smithi043 ceph-mon[42099]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-30T13:44:49.009 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:48 smithi043 ceph-mon[42099]: Reconfiguring daemon alertmanager.a on smithi043 2024-03-30T13:44:49.009 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:48 smithi043 ceph-mon[42099]: pgmap v61: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:44:49.009 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:48 smithi043 ceph-mon[42099]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:44:49.052 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:49.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:49.077 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:49.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:49.178 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:49.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:49.206 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:49.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:49.306 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:44:49.308 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:49.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:49.335 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:49.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:49.437 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:49.459 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:49.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:49.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:49.561 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:49.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:49.585 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:49.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:49.686 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:49.714 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:49.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:49.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:49.729 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:49 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:44:49.729 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:49 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:44:49.816 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:49.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:49.841 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:49.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:49.927 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:44:49.942 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:49.968 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:49.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:49.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:49.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:49 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:44:50.070 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:50.095 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:50.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:50.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:50.197 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:50.220 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:50.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:50.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:50.322 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:50.345 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:50.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:50.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:50.447 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:50.471 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:50.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:50.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:50.572 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:50.596 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:50.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:50.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:50.626 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:44:50.626 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:44:50.698 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:50.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:50.722 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:50.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:50.823 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:44:50.846 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:50.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:50.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:44:50.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:50 smithi043 ceph-mon[29963]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T13:44:50.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:50 smithi043 ceph-mon[42099]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T13:44:50.948 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:50.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:50 smithi113 ceph-mon[34341]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T13:44:50.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:50.972 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:50.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:51.074 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:51.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:51.098 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:51.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:51.199 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:51.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:51.223 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:51.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:51.324 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:51.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:51.350 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:51.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:51.451 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:51.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:51.474 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:51.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:51.575 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:51.599 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:51.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:51.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:51.701 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:51.725 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:51.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:51.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:51.826 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:51.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:51.855 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:51.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:51.956 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:51.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:51.983 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:51.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:52.084 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:52.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:52.108 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:52.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:52.209 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:52.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:52.234 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:52.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:52.335 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:52.361 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:52.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:52.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:52.463 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:52.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:52.490 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:52.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:52.591 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:52.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:52.616 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:52.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:52.717 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:52.740 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:52.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:52.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:52.842 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:52.870 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:52.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:52.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:52.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:52 smithi043 ceph-mon[29963]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:44:52.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:52 smithi043 ceph-mon[42099]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:44:52.972 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:52 smithi113 ceph-mon[34341]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:44:52.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:52.999 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:52.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:53.100 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:53.126 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:53.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:53.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:53.228 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:53.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:53.253 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:53.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:53.354 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:53.381 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:53.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:53.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:53.483 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:53.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:53.510 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:53.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:53.611 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:44:53.638 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:53.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:53.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:44:53.740 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:53.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:53.765 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:53.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:53.866 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:53.894 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:53.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:53.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:53.896 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-03-30T13:44:53.996 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:54.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:54.020 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:54.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:54.121 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:54.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:54.145 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:54.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:54.246 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:54.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:54.271 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:54.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:54.372 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:54.401 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:54.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:54.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:54.502 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:54.530 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:54.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:54.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:54.634 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:44:54.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:54.663 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:54.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:44:54.764 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:44:54.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:54.792 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:54.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:44:54.893 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:54.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:54.918 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:54.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:54.946 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:54 smithi043 ceph-mon[29963]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T13:44:54.946 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:54 smithi043 ceph-mon[42099]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T13:44:54.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:54 smithi113 ceph-mon[34341]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T13:44:55.019 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:55.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:55.044 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:55.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:55.145 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:55.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:55.170 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:55.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:55.271 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:55.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:55.295 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:55.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:55.396 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:55.425 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:55.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:55.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:44:55.527 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:55.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:55.552 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:55.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:55.653 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:55.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:55.677 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:55.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:55.778 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:55.806 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:55.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:55.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:55.908 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:55.936 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:55.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:55.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:55.954 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:44:55.973 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:44:55.992 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:44:56.012 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:44:56.033 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:44:56.039 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:56.053 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:44:56.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:56.067 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:56.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:56.074 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:44:56.094 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:44:56.096 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:44:56.096 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:44:56.096 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:44:56.096 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:44:56.096 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:44:56.097 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:44:56.097 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:44:56.097 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:44:56.168 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:56.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:56.194 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:56.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:56.195 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:44:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:44:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:44:56.295 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:56.323 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:56.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:56.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:56.425 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:56.452 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:56.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:56.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:56.554 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:56.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:56.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:56.670 INFO:teuthology.orchestra.run.smithi043.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-03-30T13:44:56.680 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:56.706 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:56.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:56.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:56.791 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:56 smithi043 ceph-mon[29963]: pgmap v66: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:56.791 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:56 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/318881797' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:44:56.791 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:56 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3515873912' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:44:56.791 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:56 smithi043 ceph-mon[42099]: pgmap v66: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:56.791 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:56 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/318881797' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:44:56.791 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:56 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3515873912' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:44:56.808 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:56.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:56.835 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:56.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:56.936 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:56.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:56.960 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:56.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:56.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:56 smithi113 ceph-mon[34341]: pgmap v66: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:44:56.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:56 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/318881797' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:44:56.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:56 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3515873912' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:44:57.061 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:44:57.085 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:57.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:57.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:44:57.187 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:57.212 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:57.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:57.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:57.313 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:44:57.335 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:44:57.343 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:57.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:57.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:44:57.445 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:57.472 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:57.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:57.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:57.574 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:57.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:57.599 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:57.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:57.700 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:57.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:57.730 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:57.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:57.830 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:57.858 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:57.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:57.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:57.960 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:57.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:57 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3515873912' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-30T13:44:57.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:57 smithi113 ceph-mon[34341]: osdmap e101: 8 total, 8 up, 8 in 2024-03-30T13:44:57.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:57 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:57.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:57 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:44:57.984 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:57.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:57.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:58.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:57 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3515873912' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-30T13:44:58.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:57 smithi043 ceph-mon[29963]: osdmap e101: 8 total, 8 up, 8 in 2024-03-30T13:44:58.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:57 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:58.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:57 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:44:58.057 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:57 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3515873912' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-30T13:44:58.057 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:57 smithi043 ceph-mon[42099]: osdmap e101: 8 total, 8 up, 8 in 2024-03-30T13:44:58.057 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:57 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:44:58.057 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:57 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:44:58.082 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 reset_purged_snaps_last 2024-03-30T13:44:58.086 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:58.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:58.150 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:58.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:58.251 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:44:58.276 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:58.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:58.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:44:58.378 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:58.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:58.402 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:58.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:58.503 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:58.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:58.532 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:58.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:58.633 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:44:58.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:58.661 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:58.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:44:58.676 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:44:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:44:58.763 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:58.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:58.788 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:58.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:58.816 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:44:58.890 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:58.919 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:58.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:58.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:58.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:58 smithi113 ceph-mon[34341]: pgmap v68: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T13:44:58.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:58 smithi113 ceph-mon[34341]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T13:44:59.020 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:59.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:59.045 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:59.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:59.146 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:44:59.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:58 smithi043 ceph-mon[29963]: pgmap v68: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T13:44:59.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:58 smithi043 ceph-mon[29963]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T13:44:59.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:58 smithi043 ceph-mon[42099]: pgmap v68: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T13:44:59.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:58 smithi043 ceph-mon[42099]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T13:44:59.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:59.175 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:59.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:44:59.276 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:44:59.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:59.306 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:59.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:44:59.407 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:44:59.434 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:59.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:59.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:44:59.538 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:44:59.566 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:44:59.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:59.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:44:59.667 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:59.692 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:59.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:59.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:59.794 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:44:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:59.819 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:59.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:44:59.920 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:44:59.943 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:44:59.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:44:59.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:44:59.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:44:59 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:00.044 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:00.072 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:00.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:00.086 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:44:59 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:00.086 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:44:59 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:00.174 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:00.202 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:00.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:00.304 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:00.313 DEBUG:teuthology.orchestra.run.smithi043:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-03-30T13:45:00.316 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-30T13:45:00.316 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-30T13:45:00.332 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:00.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:00.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:00.391 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:hints = 1 2024-03-30T13:45:00.391 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-30T13:45:00.392 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Object prefix: benchmark_data_smithi043_86989 2024-03-30T13:45:00.392 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T13:45:00.392 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 0 0 0 0 0 0 - 0 2024-03-30T13:45:00.399 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:00 smithi043 systemd[1]: Stopping Ceph alertmanager.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T13:45:00.434 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:00.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:00.480 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:00.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:00.581 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:00.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:00.616 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:00.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:00.717 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:00.767 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:00.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:00.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:00.869 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:00.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:00 smithi043 ceph-mon[29963]: pgmap v69: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:45:00.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:00 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:45:00.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:00 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:45:00.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:00 smithi043 ceph-mon[42099]: pgmap v69: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:45:00.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:00 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:45:00.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:00 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:45:00.901 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:00.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:00.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:00.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:00 smithi113 ceph-mon[34341]: pgmap v69: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:45:00.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:00 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:45:00.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:00 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:45:01.003 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:01.035 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:01.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:01.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:01.057 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:45:01.137 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:01.172 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:01.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:01.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:01.274 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:01.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:01.320 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:01.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:01.391 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 1 16 2824 2808 175.49 175.5 0.00518991 0.00567992 2024-03-30T13:45:01.422 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:01.453 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:01.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:01.555 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:01.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:01.596 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:01.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:01.697 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:01.726 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:01.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:01.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:01.828 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:01.856 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:01.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:01.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:01.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:01 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:45:01.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:01 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:45:01.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:01 smithi043 ceph-mon[29963]: osdmap e102: 8 total, 8 up, 8 in 2024-03-30T13:45:01.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:01 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:45:01.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:01 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:45:01.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:01 smithi043 ceph-mon[42099]: osdmap e102: 8 total, 8 up, 8 in 2024-03-30T13:45:01.958 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:01.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:01 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:45:01.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:01 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:45:01.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:01 smithi113 ceph-mon[34341]: osdmap e102: 8 total, 8 up, 8 in 2024-03-30T13:45:01.989 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:01.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:01.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:02.012 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:45:02.090 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:02.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:02.135 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:02.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:02.236 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:02.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:02.264 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:02.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:02.365 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:02.391 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 2 15 5435 5420 169.358 163.25 0.0068235 0.0058857 2024-03-30T13:45:02.409 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:02.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:02.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:02.511 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:02.555 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:02.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:02.657 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:02.700 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:02.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:02.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:02.802 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:02.832 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:02.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:02.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:02.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:02 smithi043 ceph-mon[29963]: pgmap v71: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:45:02.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:02 smithi043 ceph-mon[42099]: pgmap v71: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:45:02.935 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:02.967 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:02.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:02.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:02.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:02 smithi113 ceph-mon[34341]: pgmap v71: 148 pgs: 148 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:45:03.070 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:03.102 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:03.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:03.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:03.204 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:03.236 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:03.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:03.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:03.338 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:03.368 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:03.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:03.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:03.391 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 3 16 8188 8172 170.233 172 0.0023759 0.00586307 2024-03-30T13:45:03.470 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:03.502 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:03.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:03.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:03.604 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:03.634 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:03.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:03.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:03.732 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-03-30T13:45:03.736 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:03.766 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:03.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:03.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:03.868 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:03.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:03.892 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:03.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:03.946 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:03 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1489868757' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:45:03.947 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:03 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1489868757' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:45:03.947 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:03 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[80418]: ts=2024-03-30T13:45:03.665Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-30T13:45:03.993 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:04.026 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:04.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:04.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:04.128 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:04.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:04.170 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:04.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:04.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:03 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1489868757' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:45:04.271 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:04.313 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:04.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:04.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:04.391 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 4 16 10959 10943 170.968 173.188 0.00322541 0.00584056 2024-03-30T13:45:04.415 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:04.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:04.447 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:04.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:04.549 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:04.582 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:04 smithi043 podman[87181]: 2024-03-30 13:45:04.329098663 +0000 UTC m=+2.489323010 container died 441cce32a2c4fb219ac2a1cdd09bb2e8973032df479b569c3833c6efe7d6f945 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T13:45:04.588 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:04.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:04.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:04.690 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:04.723 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:04.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:04.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:04.825 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:04.856 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:04.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:04.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:04.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:04 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:45:04.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:04 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:45:04.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:04 smithi043 ceph-mon[29963]: pgmap v72: 148 pgs: 148 active+clean; 490 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 980 op/s 2024-03-30T13:45:04.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:04 smithi043 ceph-mon[29963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:45:04.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:04 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1489868757' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:45:04.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:04 smithi043 ceph-mon[29963]: osdmap e103: 8 total, 8 up, 8 in 2024-03-30T13:45:04.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:04 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:45:04.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:04 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:45:04.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:04 smithi043 ceph-mon[42099]: pgmap v72: 148 pgs: 148 active+clean; 490 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 980 op/s 2024-03-30T13:45:04.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:04 smithi043 ceph-mon[42099]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:45:04.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:04 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1489868757' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:45:04.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:04 smithi043 ceph-mon[42099]: osdmap e103: 8 total, 8 up, 8 in 2024-03-30T13:45:04.958 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:04.999 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:05.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:05.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:05.101 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:05.143 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:05.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:05.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:05.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:04 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:45:05.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:04 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:45:05.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:04 smithi113 ceph-mon[34341]: pgmap v72: 148 pgs: 148 active+clean; 490 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 980 op/s 2024-03-30T13:45:05.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:04 smithi113 ceph-mon[34341]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:45:05.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:04 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1489868757' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:45:05.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:04 smithi113 ceph-mon[34341]: osdmap e103: 8 total, 8 up, 8 in 2024-03-30T13:45:05.245 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:05.290 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:05.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:05.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:05.391 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 5 16 13761 13745 171.796 175.125 0.00309874 0.00581228 2024-03-30T13:45:05.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:05.431 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:05.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:05.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:05.477 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:05 smithi043 podman[87181]: 2024-03-30 13:45:05.201238528 +0000 UTC m=+3.361462871 container cleanup 441cce32a2c4fb219ac2a1cdd09bb2e8973032df479b569c3833c6efe7d6f945 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T13:45:05.477 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:05 smithi043 bash[87181]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a 2024-03-30T13:45:05.534 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:05.573 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:05.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:05.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:05.675 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:05.708 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:05.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:05.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:05.810 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:05.850 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:05.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:05.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:05.899 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:05 smithi043 podman[87395]: 2024-03-30 13:45:05.571459077 +0000 UTC m=+1.861501134 container remove 441cce32a2c4fb219ac2a1cdd09bb2e8973032df479b569c3833c6efe7d6f945 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T13:45:05.899 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:05 smithi043 podman[87395]: 2024-03-30 13:45:05.68125086 +0000 UTC m=+1.971292927 volume remove 199154504b0b62d78333ef958a8fd433939dcbfcb68a4af37b17bdcb60321766 2024-03-30T13:45:05.952 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:05.984 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:05.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:05.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:06.087 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:06.118 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:06.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:06.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:06.220 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:06.270 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:06.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:06.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:06.372 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:06.391 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 6 16 16372 16356 170.355 163.188 0.00570493 0.00585967 2024-03-30T13:45:06.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:45:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:45:06.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:06.420 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:06.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:06.521 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:06.551 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:06.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:06.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:06.653 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:06.688 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:06.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:06.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:06.789 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:06.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:06.834 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:06.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:06.935 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:06.982 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:06.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:06.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:07.084 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:07.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:07.127 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:07.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:07.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:06 smithi043 ceph-mon[29963]: pgmap v74: 148 pgs: 148 active+clean; 490 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 980 op/s 2024-03-30T13:45:07.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:06 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:45:07.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:06 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:45:07.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:06 smithi043 ceph-mon[42099]: pgmap v74: 148 pgs: 148 active+clean; 490 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 980 op/s 2024-03-30T13:45:07.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:06 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:45:07.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:06 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:45:07.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:06 smithi113 ceph-mon[34341]: pgmap v74: 148 pgs: 148 active+clean; 490 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 980 op/s 2024-03-30T13:45:07.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:06 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:45:07.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:06 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:45:07.228 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:07.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:07.252 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:07.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:07.353 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:07.391 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:07.392 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 7 16 18603 18587 165.937 139.438 0.0118341 0.00601918 2024-03-30T13:45:07.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:07.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:07.494 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:07.526 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:07.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:07.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:07.628 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:07.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:07.653 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:07.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:07.754 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:07.795 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:07.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:07.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:07.897 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:07.928 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:07.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:07.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:08.030 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:08.064 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:08.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:08.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:07 smithi043 ceph-mon[29963]: osdmap e104: 8 total, 8 up, 8 in 2024-03-30T13:45:08.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:07 smithi043 ceph-mon[42099]: osdmap e104: 8 total, 8 up, 8 in 2024-03-30T13:45:08.166 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:08.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:08.189 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:08.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:08.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:07 smithi113 ceph-mon[34341]: osdmap e104: 8 total, 8 up, 8 in 2024-03-30T13:45:08.291 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:08.324 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:08.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:08.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:08.392 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 8 16 20632 20616 161.045 126.812 0.00599314 0.00620308 2024-03-30T13:45:08.425 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:08.456 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:08.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:08.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:08.558 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:08.590 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:08.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:08.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:08.692 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:08.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:45:08 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:45:08.723 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:08.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:08.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:08.825 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:08.855 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:08.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:08.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:08.934 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T13:45:08.957 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:08.981 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:08 smithi043 ceph-mon[29963]: pgmap v76: 148 pgs: 148 active+clean; 490 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 774 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-30T13:45:08.982 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:08 smithi043 ceph-mon[42099]: pgmap v76: 148 pgs: 148 active+clean; 490 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 774 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-30T13:45:08.986 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:08.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:08.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:09.088 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:09.118 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:09.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:09.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:09.220 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:09.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:08 smithi113 ceph-mon[34341]: pgmap v76: 148 pgs: 148 active+clean; 490 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 774 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-30T13:45:09.268 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:09.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:09.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:09.370 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:09.392 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 9 16 22724 22708 157.677 130.75 0.00479029 0.00632514 2024-03-30T13:45:09.395 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:09.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:09.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:09.497 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:09.536 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:09.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:09.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:09.638 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:09.677 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:09.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:09.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:09.779 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:09.819 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:09.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:09.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:09.834 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:45:09.834 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-03-30T13:45:09.834 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-03-30T13:45:09.834 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-03-30T13:45:09.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.834 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:207 7.6 7:6a488315:::benchmark_data_smithi043_86989_object206:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-30T13:45:09.835 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:00.458630+0000", 2024-03-30T13:45:09.835 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 6.0076783550000004, 2024-03-30T13:45:09.835 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.016944221999999998, 2024-03-30T13:45:09.835 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:45:09.835 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:45:09.835 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:45:09.835 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:45:09.835 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:45:09.835 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:45:09.835 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 207 2024-03-30T13:45:09.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.835 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:45:09.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.836 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:45:09.836 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.458630+0000", 2024-03-30T13:45:09.836 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.836 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:45:09.836 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.458630+0000", 2024-03-30T13:45:09.836 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.836 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:45:09.836 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.458631+0000", 2024-03-30T13:45:09.836 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0100000000000001e-06 2024-03-30T13:45:09.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.837 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:45:09.837 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.458655+0000", 2024-03-30T13:45:09.837 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4511e-05 2024-03-30T13:45:09.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.837 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:45:09.837 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.458668+0000", 2024-03-30T13:45:09.837 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2531000000000001e-05 2024-03-30T13:45:09.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.837 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:45:09.838 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.458676+0000", 2024-03-30T13:45:09.838 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2309999999999996e-06 2024-03-30T13:45:09.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.838 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.838 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.461311+0000", 2024-03-30T13:45:09.838 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026351059999999999 2024-03-30T13:45:09.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.838 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:45:09.838 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.461563+0000", 2024-03-30T13:45:09.838 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00025209800000000002 2024-03-30T13:45:09.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.839 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:45:09.839 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.461974+0000", 2024-03-30T13:45:09.839 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00041099799999999998 2024-03-30T13:45:09.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.839 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:45:09.839 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.462457+0000", 2024-03-30T13:45:09.839 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00048288299999999998 2024-03-30T13:45:09.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.840 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.840 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.467534+0000", 2024-03-30T13:45:09.840 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0050769309999999998 2024-03-30T13:45:09.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.840 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.840 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.475489+0000", 2024-03-30T13:45:09.840 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0079551499999999994 2024-03-30T13:45:09.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.840 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:45:09.840 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.475536+0000", 2024-03-30T13:45:09.840 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7448999999999999e-05 2024-03-30T13:45:09.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.841 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:45:09.841 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.475574+0000", 2024-03-30T13:45:09.841 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7323999999999997e-05 2024-03-30T13:45:09.841 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.841 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.841 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.841 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:863 7.6 7:6c01b6d6:::benchmark_data_smithi043_86989_object862:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-30T13:45:09.841 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:00.674425+0000", 2024-03-30T13:45:09.841 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 5.7918828199999997, 2024-03-30T13:45:09.842 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.016977979000000001, 2024-03-30T13:45:09.842 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:45:09.842 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:45:09.842 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:45:09.842 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:45:09.842 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:45:09.842 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:45:09.842 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 863 2024-03-30T13:45:09.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.842 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:45:09.842 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.842 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:45:09.842 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.674425+0000", 2024-03-30T13:45:09.843 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.843 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:45:09.843 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.674425+0000", 2024-03-30T13:45:09.843 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.843 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:45:09.843 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.674426+0000", 2024-03-30T13:45:09.843 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7e-07 2024-03-30T13:45:09.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.843 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:45:09.844 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.674473+0000", 2024-03-30T13:45:09.844 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7596000000000003e-05 2024-03-30T13:45:09.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.844 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:45:09.844 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.674482+0000", 2024-03-30T13:45:09.844 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.551e-06 2024-03-30T13:45:09.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.844 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:45:09.844 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.674488+0000", 2024-03-30T13:45:09.844 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8409999999999996e-06 2024-03-30T13:45:09.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.845 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.845 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.681677+0000", 2024-03-30T13:45:09.845 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0071890399999999998 2024-03-30T13:45:09.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.845 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:45:09.845 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.681805+0000", 2024-03-30T13:45:09.845 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012819799999999999 2024-03-30T13:45:09.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.845 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:45:09.845 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.681946+0000", 2024-03-30T13:45:09.846 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014102799999999999 2024-03-30T13:45:09.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.846 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.846 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:45:09.846 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.687450+0000", 2024-03-30T13:45:09.846 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0055039299999999998 2024-03-30T13:45:09.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.846 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.846 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.846 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.687564+0000", 2024-03-30T13:45:09.846 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000113837 2024-03-30T13:45:09.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.846 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.847 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.847 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.691316+0000", 2024-03-30T13:45:09.847 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0037525729999999999 2024-03-30T13:45:09.847 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.847 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.847 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:45:09.847 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.691366+0000", 2024-03-30T13:45:09.847 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0179999999999997e-05 2024-03-30T13:45:09.847 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.847 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.847 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:45:09.847 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.691403+0000", 2024-03-30T13:45:09.847 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6735e-05 2024-03-30T13:45:09.847 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.848 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.848 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.848 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:881 7.9 7:9f9d5657:::benchmark_data_smithi043_86989_object880:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-30T13:45:09.848 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:00.682521+0000", 2024-03-30T13:45:09.848 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 5.7837871170000001, 2024-03-30T13:45:09.848 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.020615573000000002, 2024-03-30T13:45:09.848 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:45:09.848 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:45:09.848 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:45:09.848 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:45:09.848 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:45:09.849 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:45:09.849 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 881 2024-03-30T13:45:09.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.849 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:45:09.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.849 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:45:09.849 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.682521+0000", 2024-03-30T13:45:09.849 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.849 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:45:09.849 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.682521+0000", 2024-03-30T13:45:09.849 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.850 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.850 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:45:09.850 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.682522+0000", 2024-03-30T13:45:09.850 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5300000000000002e-07 2024-03-30T13:45:09.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.850 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.850 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:45:09.850 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.682542+0000", 2024-03-30T13:45:09.850 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0741999999999999e-05 2024-03-30T13:45:09.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.850 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.850 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:45:09.851 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.682555+0000", 2024-03-30T13:45:09.851 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2089000000000001e-05 2024-03-30T13:45:09.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.851 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:45:09.851 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.682561+0000", 2024-03-30T13:45:09.851 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1070000000000001e-06 2024-03-30T13:45:09.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.851 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.851 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.682607+0000", 2024-03-30T13:45:09.851 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6655e-05 2024-03-30T13:45:09.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.852 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:45:09.852 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.682766+0000", 2024-03-30T13:45:09.852 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015858199999999999 2024-03-30T13:45:09.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.852 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:45:09.852 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.682914+0000", 2024-03-30T13:45:09.852 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014812899999999999 2024-03-30T13:45:09.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.852 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.852 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.686287+0000", 2024-03-30T13:45:09.852 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0033728510000000001 2024-03-30T13:45:09.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.853 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.853 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.853 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.686334+0000", 2024-03-30T13:45:09.853 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7493000000000001e-05 2024-03-30T13:45:09.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.853 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.853 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:45:09.853 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.703043+0000", 2024-03-30T13:45:09.853 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.016708766 2024-03-30T13:45:09.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.853 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.853 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:45:09.853 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.703094+0000", 2024-03-30T13:45:09.853 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0714999999999998e-05 2024-03-30T13:45:09.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.854 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:45:09.854 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.703136+0000", 2024-03-30T13:45:09.854 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2490999999999997e-05 2024-03-30T13:45:09.854 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.854 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.854 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.854 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:883 7.9 7:99936dca:::benchmark_data_smithi043_86989_object882:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-30T13:45:09.854 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:00.686158+0000", 2024-03-30T13:45:09.855 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 5.780149464, 2024-03-30T13:45:09.855 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.017066988000000002, 2024-03-30T13:45:09.855 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:45:09.855 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:45:09.855 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:45:09.855 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:45:09.855 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:45:09.855 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:45:09.855 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 883 2024-03-30T13:45:09.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.855 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:45:09.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.855 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:45:09.855 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.686158+0000", 2024-03-30T13:45:09.856 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.856 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.856 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:45:09.856 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.686158+0000", 2024-03-30T13:45:09.856 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.856 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.856 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:45:09.856 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.686160+0000", 2024-03-30T13:45:09.856 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1489999999999999e-06 2024-03-30T13:45:09.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.856 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.857 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:45:09.857 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.686183+0000", 2024-03-30T13:45:09.857 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3033000000000001e-05 2024-03-30T13:45:09.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.857 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.857 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:45:09.857 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.686193+0000", 2024-03-30T13:45:09.857 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0856000000000001e-05 2024-03-30T13:45:09.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.857 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.857 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:45:09.857 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.686201+0000", 2024-03-30T13:45:09.857 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3019999999999997e-06 2024-03-30T13:45:09.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.858 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.858 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.858 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.686367+0000", 2024-03-30T13:45:09.858 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016672 2024-03-30T13:45:09.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.858 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.858 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:45:09.858 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.686524+0000", 2024-03-30T13:45:09.859 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000156226 2024-03-30T13:45:09.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.859 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:45:09.859 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.686686+0000", 2024-03-30T13:45:09.859 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016193800000000001 2024-03-30T13:45:09.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.859 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.859 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.687518+0000", 2024-03-30T13:45:09.859 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00083264299999999999 2024-03-30T13:45:09.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.859 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.860 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.688483+0000", 2024-03-30T13:45:09.860 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00096516500000000003 2024-03-30T13:45:09.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.860 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:45:09.860 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.703146+0000", 2024-03-30T13:45:09.860 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.014662585000000001 2024-03-30T13:45:09.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.860 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:45:09.860 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.703190+0000", 2024-03-30T13:45:09.860 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3781e-05 2024-03-30T13:45:09.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.861 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:45:09.861 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.703225+0000", 2024-03-30T13:45:09.861 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5590000000000003e-05 2024-03-30T13:45:09.861 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.861 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.861 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.861 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:888 7.f 7:f21e47d3:::benchmark_data_smithi043_86989_object887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-30T13:45:09.861 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:00.690420+0000", 2024-03-30T13:45:09.861 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 5.7758874330000003, 2024-03-30T13:45:09.861 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.018145555000000001, 2024-03-30T13:45:09.862 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:45:09.862 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:45:09.862 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:45:09.862 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:45:09.862 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:45:09.862 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:45:09.862 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 888 2024-03-30T13:45:09.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.862 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:45:09.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.862 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:45:09.862 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690420+0000", 2024-03-30T13:45:09.863 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.863 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:45:09.863 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690420+0000", 2024-03-30T13:45:09.863 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.863 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:45:09.863 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690422+0000", 2024-03-30T13:45:09.863 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1119999999999999e-06 2024-03-30T13:45:09.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.863 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:45:09.864 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690445+0000", 2024-03-30T13:45:09.864 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3637000000000001e-05 2024-03-30T13:45:09.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.864 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:45:09.864 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690456+0000", 2024-03-30T13:45:09.864 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0902e-05 2024-03-30T13:45:09.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.864 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:45:09.864 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690465+0000", 2024-03-30T13:45:09.864 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5189999999999992e-06 2024-03-30T13:45:09.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.865 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.865 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690534+0000", 2024-03-30T13:45:09.865 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9747999999999997e-05 2024-03-30T13:45:09.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.865 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.865 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op must wait for map", 2024-03-30T13:45:09.865 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690542+0000", 2024-03-30T13:45:09.865 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3080000000000002e-06 2024-03-30T13:45:09.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.865 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.865 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.865 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.700332+0000", 2024-03-30T13:45:09.866 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0097906499999999997 2024-03-30T13:45:09.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.866 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.866 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:45:09.866 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.700470+0000", 2024-03-30T13:45:09.866 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013744500000000001 2024-03-30T13:45:09.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.866 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.866 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:45:09.866 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.700620+0000", 2024-03-30T13:45:09.866 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014997799999999999 2024-03-30T13:45:09.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.867 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:45:09.867 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.705211+0000", 2024-03-30T13:45:09.867 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0045911520000000003 2024-03-30T13:45:09.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.867 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.867 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.705256+0000", 2024-03-30T13:45:09.867 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5401000000000001e-05 2024-03-30T13:45:09.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.867 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.867 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.708467+0000", 2024-03-30T13:45:09.868 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0032111349999999999 2024-03-30T13:45:09.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.868 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.868 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:45:09.868 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.708524+0000", 2024-03-30T13:45:09.868 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6848000000000002e-05 2024-03-30T13:45:09.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.868 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.868 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:45:09.868 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.708566+0000", 2024-03-30T13:45:09.868 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.172e-05 2024-03-30T13:45:09.868 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.869 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.869 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.869 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.869 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.869 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:890 7.9 7:95377f4e:::benchmark_data_smithi043_86989_object889:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-30T13:45:09.869 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:00.690684+0000", 2024-03-30T13:45:09.869 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 5.7756234869999998, 2024-03-30T13:45:09.869 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.019871818999999999, 2024-03-30T13:45:09.869 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:45:09.869 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:45:09.869 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:45:09.869 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:45:09.869 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:45:09.869 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:45:09.870 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 890 2024-03-30T13:45:09.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.870 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:45:09.870 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.870 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:45:09.870 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690684+0000", 2024-03-30T13:45:09.870 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.870 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.870 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:45:09.871 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690684+0000", 2024-03-30T13:45:09.871 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.871 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.871 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:45:09.871 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690685+0000", 2024-03-30T13:45:09.871 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7299999999999996e-07 2024-03-30T13:45:09.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.871 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.871 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:45:09.871 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690715+0000", 2024-03-30T13:45:09.871 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.0312e-05 2024-03-30T13:45:09.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.871 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.872 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:45:09.872 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690726+0000", 2024-03-30T13:45:09.872 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0647e-05 2024-03-30T13:45:09.872 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.872 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.872 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:45:09.872 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690733+0000", 2024-03-30T13:45:09.872 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1999999999999997e-06 2024-03-30T13:45:09.872 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.873 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.873 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.873 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690812+0000", 2024-03-30T13:45:09.873 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9136999999999998e-05 2024-03-30T13:45:09.873 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.873 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.873 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op must wait for map", 2024-03-30T13:45:09.873 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690819+0000", 2024-03-30T13:45:09.873 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6699999999999997e-06 2024-03-30T13:45:09.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.874 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.874 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.874 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.703255+0000", 2024-03-30T13:45:09.874 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.012435873 2024-03-30T13:45:09.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.874 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.874 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:45:09.874 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.703366+0000", 2024-03-30T13:45:09.875 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000111137 2024-03-30T13:45:09.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.875 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.875 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:45:09.875 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.703495+0000", 2024-03-30T13:45:09.875 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000129076 2024-03-30T13:45:09.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.875 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.875 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.875 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.710420+0000", 2024-03-30T13:45:09.875 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0069253049999999997 2024-03-30T13:45:09.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.875 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.875 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.876 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.710464+0000", 2024-03-30T13:45:09.876 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4122000000000002e-05 2024-03-30T13:45:09.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.876 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.876 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:45:09.876 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.710476+0000", 2024-03-30T13:45:09.876 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1174e-05 2024-03-30T13:45:09.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.877 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:45:09.877 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.710520+0000", 2024-03-30T13:45:09.877 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4261999999999997e-05 2024-03-30T13:45:09.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.878 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:45:09.878 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.710556+0000", 2024-03-30T13:45:09.878 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6331000000000002e-05 2024-03-30T13:45:09.878 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.878 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.878 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.879 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.879 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:891 7.f 7:f83dfaff:::benchmark_data_smithi043_86989_object890:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-30T13:45:09.879 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:00.690831+0000", 2024-03-30T13:45:09.879 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 5.7754769870000002, 2024-03-30T13:45:09.879 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.017866822000000001, 2024-03-30T13:45:09.879 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:45:09.879 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:45:09.879 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:45:09.879 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:45:09.879 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:45:09.879 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:45:09.879 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 891 2024-03-30T13:45:09.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.880 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:45:09.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.880 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:45:09.880 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690831+0000", 2024-03-30T13:45:09.880 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.880 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:45:09.880 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690831+0000", 2024-03-30T13:45:09.880 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.880 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:45:09.880 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690832+0000", 2024-03-30T13:45:09.881 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0699999999999996e-07 2024-03-30T13:45:09.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.881 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:45:09.881 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690855+0000", 2024-03-30T13:45:09.881 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3499e-05 2024-03-30T13:45:09.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.881 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:45:09.881 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690874+0000", 2024-03-30T13:45:09.881 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8465999999999999e-05 2024-03-30T13:45:09.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.881 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:45:09.882 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690881+0000", 2024-03-30T13:45:09.882 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5650000000000004e-06 2024-03-30T13:45:09.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.882 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.882 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.882 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690931+0000", 2024-03-30T13:45:09.882 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.003e-05 2024-03-30T13:45:09.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.882 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.882 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting_for_map not empty", 2024-03-30T13:45:09.883 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.690937+0000", 2024-03-30T13:45:09.883 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0549999999999996e-06 2024-03-30T13:45:09.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.883 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.883 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.701028+0000", 2024-03-30T13:45:09.883 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010091091 2024-03-30T13:45:09.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.884 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:45:09.884 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.701140+0000", 2024-03-30T13:45:09.884 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011213999999999999 2024-03-30T13:45:09.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.884 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:45:09.884 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.701283+0000", 2024-03-30T13:45:09.884 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014218600000000001 2024-03-30T13:45:09.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.885 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:45:09.885 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.705216+0000", 2024-03-30T13:45:09.885 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003933616 2024-03-30T13:45:09.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.885 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.885 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.705423+0000", 2024-03-30T13:45:09.885 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000207068 2024-03-30T13:45:09.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.886 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.886 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.708608+0000", 2024-03-30T13:45:09.886 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0031841899999999999 2024-03-30T13:45:09.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.886 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:45:09.886 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.708658+0000", 2024-03-30T13:45:09.886 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0368999999999998e-05 2024-03-30T13:45:09.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.887 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:45:09.887 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:00.708698+0000", 2024-03-30T13:45:09.887 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9839999999999998e-05 2024-03-30T13:45:09.887 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.887 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.887 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.888 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:1827 7.6 7:69a5ea2e:::benchmark_data_smithi043_86989_object1826:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-30T13:45:09.888 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:01.038647+0000", 2024-03-30T13:45:09.888 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 5.4276612350000004, 2024-03-30T13:45:09.888 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.01704083, 2024-03-30T13:45:09.888 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:45:09.888 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:45:09.888 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:45:09.888 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:45:09.888 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:45:09.888 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:45:09.889 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 1827 2024-03-30T13:45:09.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.889 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:45:09.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.889 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:45:09.889 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.038647+0000", 2024-03-30T13:45:09.889 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.889 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:45:09.889 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.038647+0000", 2024-03-30T13:45:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:45:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.038647+0000", 2024-03-30T13:45:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.75e-07 2024-03-30T13:45:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:45:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.038668+0000", 2024-03-30T13:45:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0380000000000001e-05 2024-03-30T13:45:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:45:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.038676+0000", 2024-03-30T13:45:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2349999999999999e-06 2024-03-30T13:45:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:45:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.038682+0000", 2024-03-30T13:45:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4810000000000003e-06 2024-03-30T13:45:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.041870+0000", 2024-03-30T13:45:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0031874210000000002 2024-03-30T13:45:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.892 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:45:09.892 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.041979+0000", 2024-03-30T13:45:09.892 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000109206 2024-03-30T13:45:09.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.892 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:45:09.892 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.042716+0000", 2024-03-30T13:45:09.892 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00073711299999999996 2024-03-30T13:45:09.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.892 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:45:09.893 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.047852+0000", 2024-03-30T13:45:09.893 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0051362070000000003 2024-03-30T13:45:09.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.893 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.893 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.048005+0000", 2024-03-30T13:45:09.893 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000152829 2024-03-30T13:45:09.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.893 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.894 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.055600+0000", 2024-03-30T13:45:09.894 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0075950130000000003 2024-03-30T13:45:09.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.894 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:45:09.894 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.055651+0000", 2024-03-30T13:45:09.894 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1028e-05 2024-03-30T13:45:09.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.895 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:45:09.895 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.055687+0000", 2024-03-30T13:45:09.895 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6341999999999999e-05 2024-03-30T13:45:09.895 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.895 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.895 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:3448 7.6 7:6572ef0c:::benchmark_data_smithi043_86989_object3447:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-30T13:45:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:01.611938+0000", 2024-03-30T13:45:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 4.8543694259999999, 2024-03-30T13:45:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.018305265000000001, 2024-03-30T13:45:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:45:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:45:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:45:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:45:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:45:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:45:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 3448 2024-03-30T13:45:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:45:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:45:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.611938+0000", 2024-03-30T13:45:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:45:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.611938+0000", 2024-03-30T13:45:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:45:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.611940+0000", 2024-03-30T13:45:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.229e-06 2024-03-30T13:45:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:45:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.611965+0000", 2024-03-30T13:45:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4873000000000001e-05 2024-03-30T13:45:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:45:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.611974+0000", 2024-03-30T13:45:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5610000000000004e-06 2024-03-30T13:45:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:45:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.611982+0000", 2024-03-30T13:45:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0660000000000004e-06 2024-03-30T13:45:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.617777+0000", 2024-03-30T13:45:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0057944440000000002 2024-03-30T13:45:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:45:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.617924+0000", 2024-03-30T13:45:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014703200000000001 2024-03-30T13:45:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:45:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.618078+0000", 2024-03-30T13:45:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015393500000000001 2024-03-30T13:45:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:45:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.618652+0000", 2024-03-30T13:45:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000574057 2024-03-30T13:45:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.620496+0000", 2024-03-30T13:45:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0018439179999999999 2024-03-30T13:45:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.630152+0000", 2024-03-30T13:45:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0096565439999999995 2024-03-30T13:45:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:45:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.630198+0000", 2024-03-30T13:45:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5667999999999997e-05 2024-03-30T13:45:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:45:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:01.630244+0000", 2024-03-30T13:45:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5938e-05 2024-03-30T13:45:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:5085 7.6 7:6f60758c:::benchmark_data_smithi043_86989_object5084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-30T13:45:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:02.247421+0000", 2024-03-30T13:45:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 4.2188868719999997, 2024-03-30T13:45:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.024232383999999999, 2024-03-30T13:45:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:45:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:45:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:45:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:45:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:45:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:45:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 5085 2024-03-30T13:45:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:45:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:45:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.247421+0000", 2024-03-30T13:45:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:45:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.247421+0000", 2024-03-30T13:45:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:45:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.247422+0000", 2024-03-30T13:45:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4649999999999999e-06 2024-03-30T13:45:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:45:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.247448+0000", 2024-03-30T13:45:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5250999999999999e-05 2024-03-30T13:45:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:45:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.247465+0000", 2024-03-30T13:45:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6875e-05 2024-03-30T13:45:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:45:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.247473+0000", 2024-03-30T13:45:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8699999999999998e-06 2024-03-30T13:45:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.248036+0000", 2024-03-30T13:45:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00056294500000000003 2024-03-30T13:45:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:45:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.248175+0000", 2024-03-30T13:45:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013847800000000001 2024-03-30T13:45:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:45:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.248321+0000", 2024-03-30T13:45:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014641900000000001 2024-03-30T13:45:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:45:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.249899+0000", 2024-03-30T13:45:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0015779539999999999 2024-03-30T13:45:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.252247+0000", 2024-03-30T13:45:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0023480929999999999 2024-03-30T13:45:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.271539+0000", 2024-03-30T13:45:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.019291543000000001 2024-03-30T13:45:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:45:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.271604+0000", 2024-03-30T13:45:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5568000000000006e-05 2024-03-30T13:45:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:45:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.271653+0000", 2024-03-30T13:45:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8922999999999999e-05 2024-03-30T13:45:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:5086 7.f 7:f3a9b4d5:::benchmark_data_smithi043_86989_object5085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-30T13:45:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:02.248595+0000", 2024-03-30T13:45:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 4.2177125990000004, 2024-03-30T13:45:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.02355351, 2024-03-30T13:45:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:45:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:45:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:45:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:45:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:45:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:45:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 5086 2024-03-30T13:45:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:45:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:45:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.248595+0000", 2024-03-30T13:45:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:45:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.248595+0000", 2024-03-30T13:45:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:45:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.248596+0000", 2024-03-30T13:45:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.8899999999999998e-07 2024-03-30T13:45:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:45:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.248617+0000", 2024-03-30T13:45:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0846999999999999e-05 2024-03-30T13:45:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:45:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.248629+0000", 2024-03-30T13:45:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1372999999999999e-05 2024-03-30T13:45:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:45:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.248636+0000", 2024-03-30T13:45:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9920000000000007e-06 2024-03-30T13:45:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.248748+0000", 2024-03-30T13:45:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000111597 2024-03-30T13:45:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:45:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.248831+0000", 2024-03-30T13:45:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3176000000000005e-05 2024-03-30T13:45:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:45:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.248975+0000", 2024-03-30T13:45:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000143418 2024-03-30T13:45:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:45:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.249875+0000", 2024-03-30T13:45:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00090028999999999999 2024-03-30T13:45:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.251326+0000", 2024-03-30T13:45:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0014514930000000001 2024-03-30T13:45:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.272051+0000", 2024-03-30T13:45:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.020724622000000002 2024-03-30T13:45:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:45:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.272107+0000", 2024-03-30T13:45:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6057999999999998e-05 2024-03-30T13:45:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:45:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.272149+0000", 2024-03-30T13:45:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1655e-05 2024-03-30T13:45:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:5096 7.f 7:f925a9e2:::benchmark_data_smithi043_86989_object5095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-30T13:45:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:02.252366+0000", 2024-03-30T13:45:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 4.2139414540000004, 2024-03-30T13:45:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.019900490999999999, 2024-03-30T13:45:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:45:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:45:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:45:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:45:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:45:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:45:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 5096 2024-03-30T13:45:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:45:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:45:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.252366+0000", 2024-03-30T13:45:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:45:09.920 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.252366+0000", 2024-03-30T13:45:09.920 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.920 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.920 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:45:09.920 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.252368+0000", 2024-03-30T13:45:09.920 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9400000000000001e-06 2024-03-30T13:45:09.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.920 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.920 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:45:09.920 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.252392+0000", 2024-03-30T13:45:09.920 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3312e-05 2024-03-30T13:45:09.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.921 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.921 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:45:09.921 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.252406+0000", 2024-03-30T13:45:09.921 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4721e-05 2024-03-30T13:45:09.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.921 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.921 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:45:09.921 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.252414+0000", 2024-03-30T13:45:09.921 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.588e-06 2024-03-30T13:45:09.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.921 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.921 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.921 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.252463+0000", 2024-03-30T13:45:09.921 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9342000000000003e-05 2024-03-30T13:45:09.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.922 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.922 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:45:09.922 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.252592+0000", 2024-03-30T13:45:09.922 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012874300000000001 2024-03-30T13:45:09.922 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.922 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.922 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:45:09.922 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.252757+0000", 2024-03-30T13:45:09.922 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016443299999999999 2024-03-30T13:45:09.922 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.922 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.922 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:45:09.922 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.253453+0000", 2024-03-30T13:45:09.922 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00069658100000000004 2024-03-30T13:45:09.923 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.923 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.923 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.923 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.255666+0000", 2024-03-30T13:45:09.923 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022127510000000002 2024-03-30T13:45:09.923 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.923 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.923 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.923 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.272186+0000", 2024-03-30T13:45:09.923 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.016520284999999999 2024-03-30T13:45:09.923 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.923 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.923 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:45:09.923 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.272234+0000", 2024-03-30T13:45:09.924 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8148e-05 2024-03-30T13:45:09.924 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.924 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.924 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:45:09.924 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.272267+0000", 2024-03-30T13:45:09.924 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.2647000000000001e-05 2024-03-30T13:45:09.924 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.924 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.924 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.924 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.924 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.924 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:5102 7.6 7:6f45b90e:::benchmark_data_smithi043_86989_object5101:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-30T13:45:09.924 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:02.254143+0000", 2024-03-30T13:45:09.924 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 4.2121648220000001, 2024-03-30T13:45:09.924 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.018299701000000002, 2024-03-30T13:45:09.925 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:45:09.925 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:45:09.925 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:45:09.925 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:45:09.925 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:45:09.925 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:45:09.925 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 5102 2024-03-30T13:45:09.925 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.925 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:45:09.925 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.925 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:45:09.925 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.254143+0000", 2024-03-30T13:45:09.925 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.925 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.925 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.926 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:45:09.926 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.254143+0000", 2024-03-30T13:45:09.926 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.926 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.926 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.926 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:45:09.926 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.254145+0000", 2024-03-30T13:45:09.926 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.598e-06 2024-03-30T13:45:09.926 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.926 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.926 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:45:09.926 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.254169+0000", 2024-03-30T13:45:09.927 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3859e-05 2024-03-30T13:45:09.927 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.927 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.927 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:45:09.927 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.254181+0000", 2024-03-30T13:45:09.927 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2044e-05 2024-03-30T13:45:09.927 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.927 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.927 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:45:09.927 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.254286+0000", 2024-03-30T13:45:09.927 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000105704 2024-03-30T13:45:09.927 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.927 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.927 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.928 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.254405+0000", 2024-03-30T13:45:09.928 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011835 2024-03-30T13:45:09.928 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.928 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.928 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:45:09.928 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.254586+0000", 2024-03-30T13:45:09.928 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018093100000000001 2024-03-30T13:45:09.928 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.928 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.928 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:45:09.928 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.254719+0000", 2024-03-30T13:45:09.928 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013383700000000001 2024-03-30T13:45:09.928 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.928 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.929 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:45:09.929 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.255694+0000", 2024-03-30T13:45:09.929 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00097490000000000001 2024-03-30T13:45:09.929 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.929 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.929 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.929 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.256061+0000", 2024-03-30T13:45:09.929 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00036651199999999999 2024-03-30T13:45:09.929 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.929 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.929 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.929 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.272260+0000", 2024-03-30T13:45:09.929 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.016199667000000001 2024-03-30T13:45:09.929 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.930 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.930 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:45:09.930 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.272371+0000", 2024-03-30T13:45:09.930 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000110091 2024-03-30T13:45:09.930 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.930 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.930 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:45:09.930 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.272443+0000", 2024-03-30T13:45:09.930 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2207999999999997e-05 2024-03-30T13:45:09.930 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.931 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.931 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.931 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.931 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.931 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:5689 7.f 7:f8a17bb8:::benchmark_data_smithi043_86989_object5688:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-30T13:45:09.931 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:02.494456+0000", 2024-03-30T13:45:09.931 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 3.9718515139999999, 2024-03-30T13:45:09.931 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0174647, 2024-03-30T13:45:09.931 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:45:09.931 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:45:09.931 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:45:09.931 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:45:09.931 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:45:09.932 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:45:09.932 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 5689 2024-03-30T13:45:09.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.932 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:45:09.932 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.932 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:45:09.932 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.494456+0000", 2024-03-30T13:45:09.932 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.932 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.932 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:45:09.932 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.494456+0000", 2024-03-30T13:45:09.932 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.932 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.933 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:45:09.933 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.494457+0000", 2024-03-30T13:45:09.933 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.6500000000000008e-07 2024-03-30T13:45:09.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.933 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:45:09.933 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.494480+0000", 2024-03-30T13:45:09.933 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2568000000000001e-05 2024-03-30T13:45:09.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.933 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:45:09.933 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.494490+0000", 2024-03-30T13:45:09.933 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0423e-05 2024-03-30T13:45:09.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.934 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.934 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:45:09.934 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.494498+0000", 2024-03-30T13:45:09.934 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6599999999999995e-06 2024-03-30T13:45:09.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.934 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.934 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.934 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.497191+0000", 2024-03-30T13:45:09.934 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026931189999999999 2024-03-30T13:45:09.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.934 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.934 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:45:09.934 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.497337+0000", 2024-03-30T13:45:09.934 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000146281 2024-03-30T13:45:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:45:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.498377+0000", 2024-03-30T13:45:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0010393799999999999 2024-03-30T13:45:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.500687+0000", 2024-03-30T13:45:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0023104699999999998 2024-03-30T13:45:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:45:09.935 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.501560+0000", 2024-03-30T13:45:09.936 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00087242099999999996 2024-03-30T13:45:09.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.936 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.936 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.511827+0000", 2024-03-30T13:45:09.936 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010267322000000001 2024-03-30T13:45:09.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.936 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:45:09.936 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.511883+0000", 2024-03-30T13:45:09.936 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6499000000000002e-05 2024-03-30T13:45:09.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.936 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:45:09.937 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:02.511921+0000", 2024-03-30T13:45:09.937 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7592000000000002e-05 2024-03-30T13:45:09.937 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.937 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.937 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.937 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:9473 7.4 7:22f71512:::benchmark_data_smithi043_86989_object9472:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-30T13:45:09.937 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:03.839615+0000", 2024-03-30T13:45:09.937 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 2.626692485, 2024-03-30T13:45:09.937 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.055444245000000003, 2024-03-30T13:45:09.937 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:45:09.937 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:45:09.937 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:45:09.937 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:45:09.938 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:45:09.938 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:45:09.938 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 9473 2024-03-30T13:45:09.938 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.938 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:45:09.938 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.938 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:45:09.938 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.839615+0000", 2024-03-30T13:45:09.938 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.938 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.938 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.938 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:45:09.938 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.839615+0000", 2024-03-30T13:45:09.939 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.939 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:45:09.939 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.839616+0000", 2024-03-30T13:45:09.939 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6000000000000002e-07 2024-03-30T13:45:09.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.939 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:45:09.939 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.839638+0000", 2024-03-30T13:45:09.939 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1639000000000001e-05 2024-03-30T13:45:09.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.939 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:45:09.939 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.839650+0000", 2024-03-30T13:45:09.940 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1790000000000001e-05 2024-03-30T13:45:09.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.940 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:45:09.940 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.839657+0000", 2024-03-30T13:45:09.940 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6580000000000002e-06 2024-03-30T13:45:09.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.940 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.940 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.842506+0000", 2024-03-30T13:45:09.940 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0028489079999999998 2024-03-30T13:45:09.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.940 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:45:09.940 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.842620+0000", 2024-03-30T13:45:09.941 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011366099999999999 2024-03-30T13:45:09.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.941 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:45:09.941 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.842754+0000", 2024-03-30T13:45:09.941 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013364400000000001 2024-03-30T13:45:09.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.941 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:45:09.941 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.844823+0000", 2024-03-30T13:45:09.941 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002069605 2024-03-30T13:45:09.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.941 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.942 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.845425+0000", 2024-03-30T13:45:09.942 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00060174999999999996 2024-03-30T13:45:09.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.942 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.942 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.894948+0000", 2024-03-30T13:45:09.942 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.049522869999999997 2024-03-30T13:45:09.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.942 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:45:09.942 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.895006+0000", 2024-03-30T13:45:09.942 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8270999999999998e-05 2024-03-30T13:45:09.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.943 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:45:09.943 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.895060+0000", 2024-03-30T13:45:09.943 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3588999999999998e-05 2024-03-30T13:45:09.943 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.943 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.943 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.943 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:9494 7.4 7:26d30a56:::benchmark_data_smithi043_86989_object9493:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-30T13:45:09.943 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:03.846933+0000", 2024-03-30T13:45:09.943 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 2.619374654, 2024-03-30T13:45:09.943 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.049736015000000001, 2024-03-30T13:45:09.943 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:45:09.943 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:45:09.944 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:45:09.944 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:45:09.944 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:45:09.944 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:45:09.944 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 9494 2024-03-30T13:45:09.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.944 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:45:09.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.944 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:45:09.944 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.846933+0000", 2024-03-30T13:45:09.944 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.944 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:45:09.945 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.846933+0000", 2024-03-30T13:45:09.945 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.945 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:45:09.945 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.846935+0000", 2024-03-30T13:45:09.945 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2839999999999999e-06 2024-03-30T13:45:09.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.945 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:45:09.945 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.846957+0000", 2024-03-30T13:45:09.945 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2311999999999999e-05 2024-03-30T13:45:09.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.946 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:45:09.946 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.846970+0000", 2024-03-30T13:45:09.946 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3193999999999999e-05 2024-03-30T13:45:09.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.946 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:45:09.946 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.846978+0000", 2024-03-30T13:45:09.946 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4549999999999995e-06 2024-03-30T13:45:09.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.946 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.946 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.847027+0000", 2024-03-30T13:45:09.946 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8445000000000001e-05 2024-03-30T13:45:09.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.947 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:45:09.947 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.847114+0000", 2024-03-30T13:45:09.947 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6601999999999995e-05 2024-03-30T13:45:09.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.947 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:45:09.947 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.847229+0000", 2024-03-30T13:45:09.947 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000115151 2024-03-30T13:45:09.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.947 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:45:09.947 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.848721+0000", 2024-03-30T13:45:09.947 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0014920689999999999 2024-03-30T13:45:09.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.948 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.948 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.849075+0000", 2024-03-30T13:45:09.948 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00035407700000000001 2024-03-30T13:45:09.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.948 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.948 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.896566+0000", 2024-03-30T13:45:09.948 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.047491431000000001 2024-03-30T13:45:09.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.949 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:45:09.949 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.896624+0000", 2024-03-30T13:45:09.949 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7788000000000003e-05 2024-03-30T13:45:09.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.949 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:45:09.949 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.896669+0000", 2024-03-30T13:45:09.949 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5207000000000003e-05 2024-03-30T13:45:09.949 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.949 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.949 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.950 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:9505 7.9 7:944f94ed:::benchmark_data_smithi043_86989_object9504:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-30T13:45:09.950 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:03.850724+0000", 2024-03-30T13:45:09.950 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 2.6155835229999997, 2024-03-30T13:45:09.950 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.046399620000000003, 2024-03-30T13:45:09.950 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:45:09.950 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:45:09.950 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:45:09.950 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:45:09.950 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:45:09.950 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:45:09.950 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 9505 2024-03-30T13:45:09.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.950 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.850724+0000", 2024-03-30T13:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.850724+0000", 2024-03-30T13:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.850725+0000", 2024-03-30T13:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3699999999999999e-07 2024-03-30T13:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.850777+0000", 2024-03-30T13:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.134e-05 2024-03-30T13:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.850787+0000", 2024-03-30T13:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0414000000000001e-05 2024-03-30T13:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.850795+0000", 2024-03-30T13:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9249999999999995e-06 2024-03-30T13:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.851006+0000", 2024-03-30T13:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00021116799999999999 2024-03-30T13:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.851125+0000", 2024-03-30T13:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000118359 2024-03-30T13:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.851239+0000", 2024-03-30T13:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000114221 2024-03-30T13:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.852040+0000", 2024-03-30T13:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00080098300000000001 2024-03-30T13:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.853727+0000", 2024-03-30T13:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001686807 2024-03-30T13:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.897030+0000", 2024-03-30T13:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.043303884000000001 2024-03-30T13:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.897086+0000", 2024-03-30T13:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5405999999999998e-05 2024-03-30T13:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.897124+0000", 2024-03-30T13:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8176000000000002e-05 2024-03-30T13:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:9508 7.9 7:9b9f3810:::benchmark_data_smithi043_86989_object9507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-30T13:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:03.852633+0000", 2024-03-30T13:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 2.6136745650000002, 2024-03-30T13:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.04464601, 2024-03-30T13:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 9508 2024-03-30T13:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.852633+0000", 2024-03-30T13:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.852633+0000", 2024-03-30T13:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.852634+0000", 2024-03-30T13:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.37e-07 2024-03-30T13:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.852798+0000", 2024-03-30T13:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016390799999999999 2024-03-30T13:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.852820+0000", 2024-03-30T13:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1946e-05 2024-03-30T13:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.852833+0000", 2024-03-30T13:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3511e-05 2024-03-30T13:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.852902+0000", 2024-03-30T13:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8856000000000006e-05 2024-03-30T13:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.853002+0000", 2024-03-30T13:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000100059 2024-03-30T13:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.853155+0000", 2024-03-30T13:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000152745 2024-03-30T13:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.853909+0000", 2024-03-30T13:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00075383500000000001 2024-03-30T13:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.854151+0000", 2024-03-30T13:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00024241900000000001 2024-03-30T13:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.897202+0000", 2024-03-30T13:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.043050815999999999 2024-03-30T13:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.897246+0000", 2024-03-30T13:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4236000000000001e-05 2024-03-30T13:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.897279+0000", 2024-03-30T13:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3142000000000002e-05 2024-03-30T13:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:9519 7.4 7:21f55d7f:::benchmark_data_smithi043_86989_object9518:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-30T13:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:03.857449+0000", 2024-03-30T13:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 2.6088590260000002, 2024-03-30T13:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.039650083000000003, 2024-03-30T13:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 9519 2024-03-30T13:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.857449+0000", 2024-03-30T13:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.857449+0000", 2024-03-30T13:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.857450+0000", 2024-03-30T13:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5379999999999999e-06 2024-03-30T13:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.857481+0000", 2024-03-30T13:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.0190000000000001e-05 2024-03-30T13:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.857492+0000", 2024-03-30T13:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1419e-05 2024-03-30T13:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.857500+0000", 2024-03-30T13:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6559999999999992e-06 2024-03-30T13:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.857558+0000", 2024-03-30T13:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8681000000000003e-05 2024-03-30T13:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.857681+0000", 2024-03-30T13:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012236199999999999 2024-03-30T13:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.857835+0000", 2024-03-30T13:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000154439 2024-03-30T13:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.858637+0000", 2024-03-30T13:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00080220200000000001 2024-03-30T13:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.858799+0000", 2024-03-30T13:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016183199999999999 2024-03-30T13:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.897001+0000", 2024-03-30T13:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.038201653000000002 2024-03-30T13:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.897059+0000", 2024-03-30T13:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7979999999999997e-05 2024-03-30T13:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.897099+0000", 2024-03-30T13:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0130999999999999e-05 2024-03-30T13:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:9530 7.4 7:2642d351:::benchmark_data_smithi043_86989_object9529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-30T13:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:03.864781+0000", 2024-03-30T13:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 2.601527242, 2024-03-30T13:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.032446595000000002, 2024-03-30T13:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 9530 2024-03-30T13:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.864781+0000", 2024-03-30T13:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.864781+0000", 2024-03-30T13:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.864782+0000", 2024-03-30T13:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5239999999999999e-06 2024-03-30T13:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.864829+0000", 2024-03-30T13:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7225999999999998e-05 2024-03-30T13:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.864842+0000", 2024-03-30T13:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2363e-05 2024-03-30T13:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.864848+0000", 2024-03-30T13:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9970000000000004e-06 2024-03-30T13:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.864914+0000", 2024-03-30T13:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6410999999999998e-05 2024-03-30T13:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.865043+0000", 2024-03-30T13:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012921499999999999 2024-03-30T13:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.865179+0000", 2024-03-30T13:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000135554 2024-03-30T13:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.865750+0000", 2024-03-30T13:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00057093200000000002 2024-03-30T13:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.866172+0000", 2024-03-30T13:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00042170700000000001 2024-03-30T13:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.897138+0000", 2024-03-30T13:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.030966457999999999 2024-03-30T13:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.897190+0000", 2024-03-30T13:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2209000000000001e-05 2024-03-30T13:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:03.897227+0000", 2024-03-30T13:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6999000000000003e-05 2024-03-30T13:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:45:09.974 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:10.014 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:10.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:10.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:10.116 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:10.148 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:10.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:09 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:10.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:09 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:10.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:10.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:10.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:09 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:10.250 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:10.288 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:10.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:10.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:10.390 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:10.392 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 10 15 24919 24904 155.633 137.25 0.00741201 0.00641925 2024-03-30T13:45:10.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:10.424 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:10.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:10.525 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:10.577 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:10.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:10.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:10.679 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:10.723 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:10.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:10.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:10.825 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:10.858 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:10.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:10.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:10.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:10 smithi043 ceph-mon[29963]: pgmap v77: 148 pgs: 148 active+clean; 1.2 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 159 MiB/s wr, 2.54k op/s 2024-03-30T13:45:10.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:10 smithi043 ceph-mon[42099]: pgmap v77: 148 pgs: 148 active+clean; 1.2 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 159 MiB/s wr, 2.54k op/s 2024-03-30T13:45:10.960 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:10.991 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:10.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:10.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:11.093 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:11.122 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:11.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:11.124 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-30T13:45:11.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:11.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:10 smithi113 ceph-mon[34341]: pgmap v77: 148 pgs: 148 active+clean; 1.2 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 159 MiB/s wr, 2.54k op/s 2024-03-30T13:45:11.225 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:11.253 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:11.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:11.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:11.355 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:11.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:11.387 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:11.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:11.392 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 11 16 26994 26978 153.267 129.625 0.0069749 0.00651771 2024-03-30T13:45:11.488 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:11.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:11.537 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:11.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:11.638 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:11.669 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:11.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:11.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:11.771 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:11.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:11.803 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:11.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:11.904 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:11.937 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:11.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:11.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:12.039 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:12.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:12.075 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:12.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:12.176 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:12.204 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:45:12.206 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:12.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:12.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:12.248 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:45:12.274 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:45:12.294 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:45:12.307 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:12.325 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:45:12.334 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:12.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:12.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:12.350 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:45:12.391 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:45:12.393 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 12 16 28813 28797 149.956 113.688 0.00506407 0.00666093 2024-03-30T13:45:12.431 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:45:12.436 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:12.467 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:12.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:12.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:12.478 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:45:12.517 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:45:12.517 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:12 smithi043 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@alertmanager.a.service: Deactivated successfully. 2024-03-30T13:45:12.517 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:12 smithi043 systemd[1]: Stopped Ceph alertmanager.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T13:45:12.517 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:12 smithi043 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@alertmanager.a.service: Consumed 1.306s CPU time. 2024-03-30T13:45:12.517 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:12 smithi043 systemd[1]: Starting Ceph alertmanager.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T13:45:12.522 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:45:12.522 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:45:12.522 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:45:12.522 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:45:12.522 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:45:12.522 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:45:12.522 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:45:12.523 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:45:12.568 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:12.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:12.592 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:12.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:12.694 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:12.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:12.725 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:12.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:12.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:12.850 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:12.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:12.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:12.952 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:12.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:12.991 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:12.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:13.092 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:13.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:13.131 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:13.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:13.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:12 smithi043 ceph-mon[29963]: pgmap v78: 148 pgs: 148 active+clean; 1.2 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 98 MiB/s wr, 1.56k op/s 2024-03-30T13:45:13.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:12 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1700552660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:45:13.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:12 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:13.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:12 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:45:13.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:12 smithi043 ceph-mon[42099]: pgmap v78: 148 pgs: 148 active+clean; 1.2 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 98 MiB/s wr, 1.56k op/s 2024-03-30T13:45:13.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:12 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1700552660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:45:13.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:12 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:13.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:12 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:45:13.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:12 smithi113 ceph-mon[34341]: pgmap v78: 148 pgs: 148 active+clean; 1.2 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 98 MiB/s wr, 1.56k op/s 2024-03-30T13:45:13.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:12 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1700552660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:45:13.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:12 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:13.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:12 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:45:13.232 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:13.262 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:13.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:13.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:13.393 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 13 16 30315 30299 145.642 93.875 0.00162188 0.00684335 2024-03-30T13:45:13.479 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:13.509 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:13.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:13.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:13.611 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:13.641 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:13.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:13.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:13.744 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:13.774 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:13.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:13.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:13.877 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:13.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:13.906 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:13.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:14.009 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:14.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:14.054 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:14.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:14.156 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:14.186 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:14.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:14.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:14.288 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:14.320 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:14.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:14.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:14.393 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 14 16 32293 32277 144.069 123.625 0.0112985 0.00693512 2024-03-30T13:45:14.423 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:14.452 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:14.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:14.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:14.554 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:14.585 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:14.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:14.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:14.687 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:14.734 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:14.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:14.835 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:14.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:14.867 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:14.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:14.968 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:15.007 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:15.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:15.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:15.109 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:15.147 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:15.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:15.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:15.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:14 smithi043 ceph-mon[29963]: pgmap v79: 148 pgs: 148 active+clean; 1.8 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 146 MiB/s wr, 2.33k op/s 2024-03-30T13:45:15.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:14 smithi043 ceph-mon[42099]: pgmap v79: 148 pgs: 148 active+clean; 1.8 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 146 MiB/s wr, 2.33k op/s 2024-03-30T13:45:15.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:14 smithi113 ceph-mon[34341]: pgmap v79: 148 pgs: 148 active+clean; 1.8 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 146 MiB/s wr, 2.33k op/s 2024-03-30T13:45:15.249 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:15.279 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:15.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:15.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:15.381 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:15.393 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 15 15 34308 34293 142.861 126 0.00409463 0.00699377 2024-03-30T13:45:15.423 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:15.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:15.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:15.525 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:15.556 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:15.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:15.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:15.658 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:15.697 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:15.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:15.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:15.799 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:15.831 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:15.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:15.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:15.933 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:15.973 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:15.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:15.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:16.075 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:16.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:16.107 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:16.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:16.208 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:16.246 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:16.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:16.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:16.348 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:16.389 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:16.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:16.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:16.393 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 16 16 36190 36174 141.28 117.562 0.00734113 0.00707229 2024-03-30T13:45:16.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:45:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:45:16.491 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:16.522 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:16.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:16.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:16.624 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:16.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:16.671 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:16.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:16.772 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:16.818 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:16.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:16.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:16.920 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:16.958 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:16.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:16.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:17.060 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:17.093 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:17.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:17.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:17.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:16 smithi043 ceph-mon[29963]: pgmap v80: 148 pgs: 148 active+clean; 1.8 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 139 MiB/s wr, 2.23k op/s 2024-03-30T13:45:17.122 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:16 smithi043 ceph-mon[42099]: pgmap v80: 148 pgs: 148 active+clean; 1.8 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 139 MiB/s wr, 2.23k op/s 2024-03-30T13:45:17.177 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 reset_purged_snaps_last 2024-03-30T13:45:17.194 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:17.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:16 smithi113 ceph-mon[34341]: pgmap v80: 148 pgs: 148 active+clean; 1.8 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 139 MiB/s wr, 2.23k op/s 2024-03-30T13:45:17.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:17.271 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:17.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:17.372 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:17.378 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:45:17.394 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 17 16 38020 38004 139.697 114.375 0.00917542 0.00715241 2024-03-30T13:45:17.401 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:17.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:17.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:17.504 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:17.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:17.536 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:17.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:17.638 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:17.667 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:17.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:17.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:17.769 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:17.825 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:17.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:17.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:17.927 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:17.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:17.970 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:17.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:18.071 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:18.114 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:18.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:18.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:18.149 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:17 smithi043 podman[88797]: 2024-03-30 13:45:17.716972675 +0000 UTC m=+0.040224291 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-30T13:45:18.149 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:17 smithi043 podman[88797]: 2024-03-30 13:45:17.844262102 +0000 UTC m=+0.167513699 volume create 4126d65f6f44ee6af51dbff3faac7c1022b7e372929d7f9f1968fa9bdd9558c8 2024-03-30T13:45:18.149 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:17 smithi043 podman[88797]: 2024-03-30 13:45:17.961321203 +0000 UTC m=+0.284572800 container create 165ce8ff7c932b48df9973805204b5a141e551da0f3cd187ae76ff18cf4f019b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T13:45:18.216 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:18.264 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:18.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:18.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:18.366 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:18.394 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 18 16 39704 39688 137.782 105.25 0.00920261 0.00725237 2024-03-30T13:45:18.397 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:18.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:18.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:18.499 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:18.529 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:18.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:18.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:18.631 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:18.661 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:18.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:18.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:18.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:45:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:45:18.763 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:18.793 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:18 smithi043 podman[88797]: 2024-03-30 13:45:18.560907274 +0000 UTC m=+0.884158886 container init 165ce8ff7c932b48df9973805204b5a141e551da0f3cd187ae76ff18cf4f019b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T13:45:18.793 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:18 smithi043 podman[88797]: 2024-03-30 13:45:18.571703942 +0000 UTC m=+0.894955536 container start 165ce8ff7c932b48df9973805204b5a141e551da0f3cd187ae76ff18cf4f019b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T13:45:18.793 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:18 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[88910]: ts=2024-03-30T13:45:18.594Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-30T13:45:18.793 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:18 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[88910]: ts=2024-03-30T13:45:18.594Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-30T13:45:18.793 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:18 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[88910]: ts=2024-03-30T13:45:18.596Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.43 port=9094 2024-03-30T13:45:18.793 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:18 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[88910]: ts=2024-03-30T13:45:18.601Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-30T13:45:18.794 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:18 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[88910]: ts=2024-03-30T13:45:18.664Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-30T13:45:18.794 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:18 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[88910]: ts=2024-03-30T13:45:18.665Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-30T13:45:18.794 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:18 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[88910]: ts=2024-03-30T13:45:18.669Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-30T13:45:18.794 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:18 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[88910]: ts=2024-03-30T13:45:18.669Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-30T13:45:18.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:18.795 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:18.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:18.896 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:18.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:18.930 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:18.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:19.031 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:19.069 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:19.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:19.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:19.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:18 smithi043 ceph-mon[29963]: pgmap v81: 148 pgs: 148 active+clean; 1.8 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 969 B/s rd, 132 MiB/s wr, 2.11k op/s 2024-03-30T13:45:19.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:18 smithi043 ceph-mon[42099]: pgmap v81: 148 pgs: 148 active+clean; 1.8 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 969 B/s rd, 132 MiB/s wr, 2.11k op/s 2024-03-30T13:45:19.171 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:19.212 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:19.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:19.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:19.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:18 smithi113 ceph-mon[34341]: pgmap v81: 148 pgs: 148 active+clean; 1.8 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 969 B/s rd, 132 MiB/s wr, 2.11k op/s 2024-03-30T13:45:19.314 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:19.339 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:19.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:19.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:19.394 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 19 16 41390 41374 136.076 105.375 0.00704524 0.00734404 2024-03-30T13:45:19.441 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:19.472 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:19.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:19.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:19.574 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:19.603 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:19.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:19.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:19.705 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:19.735 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:19.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:19.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:19.777 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:45:19.837 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:19.869 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:19.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:19.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:19.971 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:20.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:20.016 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:20.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:20.117 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:20.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:19 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:20.124 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:19 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:20.124 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:19 smithi043 bash[88797]: 165ce8ff7c932b48df9973805204b5a141e551da0f3cd187ae76ff18cf4f019b 2024-03-30T13:45:20.124 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:19 smithi043 systemd[1]: Started Ceph alertmanager.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T13:45:20.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:20.159 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:20.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:20.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:19 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:20.260 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:20.299 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:20.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:20.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:20.394 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-03-30T13:45:20.393892+0000 min lat: 0.00133902 max lat: 0.0940746 avg lat: 0.00745113 2024-03-30T13:45:20.394 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T13:45:20.394 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 20 16 42846 42830 133.822 91 0.00525621 0.00745113 2024-03-30T13:45:20.402 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:20.425 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:20.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:20.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:20.527 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:20.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:20.563 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:20.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:20.608 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:45:20.609 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:45:20.664 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:20.701 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:20.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:20.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:20.803 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:20.850 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:20.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:20.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:20 smithi043 ceph-mon[42099]: pgmap v82: 148 pgs: 148 active+clean; 2.4 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 165 MiB/s wr, 2.64k op/s 2024-03-30T13:45:20.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:20 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:20.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:20 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:20.900 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:20 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[88910]: ts=2024-03-30T13:45:20.602Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000698765s 2024-03-30T13:45:20.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:20 smithi043 ceph-mon[29963]: pgmap v82: 148 pgs: 148 active+clean; 2.4 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 165 MiB/s wr, 2.64k op/s 2024-03-30T13:45:20.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:20 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:20.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:20 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:20.952 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:20.996 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:20.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:20.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:21.098 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:21.101 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:45:21.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:21.155 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:21.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:21.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:20 smithi113 ceph-mon[34341]: pgmap v82: 148 pgs: 148 active+clean; 2.4 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 165 MiB/s wr, 2.64k op/s 2024-03-30T13:45:21.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:20 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:21.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:20 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:21.255 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:21.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:21.303 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:21.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:21.394 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 21 16 44391 44375 132.047 96.5625 0.00775488 0.00756676 2024-03-30T13:45:21.404 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:21.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:21.436 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:21.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:21.538 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:21.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:21.572 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:21.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:21.674 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:21.706 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:21.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:21.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:21.807 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:21.840 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:21.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:21.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:21.873 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T13:45:21.938 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-03-30T13:45:21.942 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:22.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:22.022 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:22.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:22.123 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:22.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:21 smithi043 ceph-mon[29963]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-30T13:45:22.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:21 smithi043 ceph-mon[29963]: Reconfiguring daemon prometheus.a on smithi113 2024-03-30T13:45:22.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:21 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4153017852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:45:22.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:21 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:45:22.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:21 smithi043 ceph-mon[42099]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-30T13:45:22.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:21 smithi043 ceph-mon[42099]: Reconfiguring daemon prometheus.a on smithi113 2024-03-30T13:45:22.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:21 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/4153017852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:45:22.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:21 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:45:22.154 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:22.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:22.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:22.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:21 smithi113 ceph-mon[34341]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-30T13:45:22.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:21 smithi113 ceph-mon[34341]: Reconfiguring daemon prometheus.a on smithi113 2024-03-30T13:45:22.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:21 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/4153017852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:45:22.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:21 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:45:22.256 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:22.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:22.288 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:22.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:22.390 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:22.395 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 22 16 46014 45998 130.654 101.438 0.015984 0.0076487 2024-03-30T13:45:22.431 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:22.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:22.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:22.533 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:22.574 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:22.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:22.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:22.676 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:22.708 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:22.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:22.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:22.810 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:22.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:22.844 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:22.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:22.945 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:22.985 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:22.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:22.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:23.087 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:23.114 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:22 smithi113 ceph-mon[34341]: pgmap v83: 148 pgs: 148 active+clean; 2.4 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-03-30T13:45:23.114 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:22 smithi113 ceph-mon[34341]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:45:23.114 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:22 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:45:23.114 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:22 smithi113 ceph-mon[34341]: osdmap e105: 8 total, 8 up, 8 in 2024-03-30T13:45:23.114 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:22 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:45:23.114 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:22 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:45:23.128 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:23.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:23.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:23.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:22 smithi043 ceph-mon[29963]: pgmap v83: 148 pgs: 148 active+clean; 2.4 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-03-30T13:45:23.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:22 smithi043 ceph-mon[29963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:45:23.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:22 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:45:23.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:22 smithi043 ceph-mon[29963]: osdmap e105: 8 total, 8 up, 8 in 2024-03-30T13:45:23.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:22 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:45:23.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:22 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:45:23.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:22 smithi043 ceph-mon[42099]: pgmap v83: 148 pgs: 148 active+clean; 2.4 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-03-30T13:45:23.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:22 smithi043 ceph-mon[42099]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:45:23.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:22 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:45:23.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:22 smithi043 ceph-mon[42099]: osdmap e105: 8 total, 8 up, 8 in 2024-03-30T13:45:23.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:22 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:45:23.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:22 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:45:23.230 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:23.265 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:23.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:23.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:23.367 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:23.395 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 23 16 47660 47644 129.446 102.875 0.0145602 0.00771951 2024-03-30T13:45:23.398 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:23.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:23.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:23.500 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:23.532 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:23.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:23.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:23.633 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:23.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:23.672 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:23.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:23.773 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:23.816 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:23.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:23.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:23.918 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:23.960 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:23.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:23.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:24.062 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:24.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:24.123 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:24.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:24.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:23 smithi043 ceph-mon[42099]: 6.14 scrub starts 2024-03-30T13:45:24.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:23 smithi043 ceph-mon[42099]: 6.14 scrub ok 2024-03-30T13:45:24.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:23 smithi043 ceph-mon[42099]: osdmap e106: 8 total, 8 up, 8 in 2024-03-30T13:45:24.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:23 smithi043 ceph-mon[42099]: 5.7 scrub starts 2024-03-30T13:45:24.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:23 smithi043 ceph-mon[42099]: 5.7 scrub ok 2024-03-30T13:45:24.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:23 smithi043 ceph-mon[42099]: 4.b scrub starts 2024-03-30T13:45:24.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:23 smithi043 ceph-mon[42099]: 4.b scrub ok 2024-03-30T13:45:24.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:23 smithi043 ceph-mon[42099]: pgmap v86: 148 pgs: 148 active+clean; 2.7 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-30T13:45:24.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:23 smithi043 ceph-mon[29963]: 6.14 scrub starts 2024-03-30T13:45:24.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:23 smithi043 ceph-mon[29963]: 6.14 scrub ok 2024-03-30T13:45:24.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:23 smithi043 ceph-mon[29963]: osdmap e106: 8 total, 8 up, 8 in 2024-03-30T13:45:24.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:23 smithi043 ceph-mon[29963]: 5.7 scrub starts 2024-03-30T13:45:24.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:23 smithi043 ceph-mon[29963]: 5.7 scrub ok 2024-03-30T13:45:24.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:23 smithi043 ceph-mon[29963]: 4.b scrub starts 2024-03-30T13:45:24.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:23 smithi043 ceph-mon[29963]: 4.b scrub ok 2024-03-30T13:45:24.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:23 smithi043 ceph-mon[29963]: pgmap v86: 148 pgs: 148 active+clean; 2.7 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-30T13:45:24.222 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:23 smithi113 systemd[1]: Stopping Ceph prometheus.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T13:45:24.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:23 smithi113 ceph-mon[34341]: 6.14 scrub starts 2024-03-30T13:45:24.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:23 smithi113 ceph-mon[34341]: 6.14 scrub ok 2024-03-30T13:45:24.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:23 smithi113 ceph-mon[34341]: osdmap e106: 8 total, 8 up, 8 in 2024-03-30T13:45:24.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:23 smithi113 ceph-mon[34341]: 5.7 scrub starts 2024-03-30T13:45:24.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:23 smithi113 ceph-mon[34341]: 5.7 scrub ok 2024-03-30T13:45:24.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:23 smithi113 ceph-mon[34341]: 4.b scrub starts 2024-03-30T13:45:24.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:23 smithi113 ceph-mon[34341]: 4.b scrub ok 2024-03-30T13:45:24.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:23 smithi113 ceph-mon[34341]: pgmap v86: 148 pgs: 148 active+clean; 2.7 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-30T13:45:24.224 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:24.256 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:24.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:24.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:24.358 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:24.395 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 24 16 49305 49289 128.336 102.812 0.00719122 0.00778692 2024-03-30T13:45:24.403 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:24.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:24.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:24.505 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:24.553 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:24.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:24.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:24.655 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:24.670 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:24 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:45:24.353Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-30T13:45:24.670 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:24 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:45:24.353Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-30T13:45:24.670 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:24 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:45:24.353Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-30T13:45:24.671 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:24 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:45:24.353Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-30T13:45:24.671 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:24 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:45:24.353Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-30T13:45:24.671 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:24 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:45:24.353Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-30T13:45:24.671 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:24 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:45:24.353Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-30T13:45:24.671 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:24 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:45:24.353Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-30T13:45:24.671 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:24 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:45:24.353Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-30T13:45:24.671 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:24 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:45:24.353Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-30T13:45:24.671 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:24 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:45:24.353Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-30T13:45:24.671 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:24 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[66461]: ts=2024-03-30T13:45:24.353Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-30T13:45:24.671 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:24 smithi113 podman[73262]: 2024-03-30 13:45:24.364562442 +0000 UTC m=+0.181315086 container died 37bd40d9ec3389dcb95a2720e99830cecc7b090a5595e04774105eda2f10531d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T13:45:24.702 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:24.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:24.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:24.804 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:24.842 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:24.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:24.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:24.944 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:24.972 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:24 smithi113 podman[73262]: 2024-03-30 13:45:24.749614702 +0000 UTC m=+0.566367345 container cleanup 37bd40d9ec3389dcb95a2720e99830cecc7b090a5595e04774105eda2f10531d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T13:45:24.972 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:24 smithi113 bash[73262]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a 2024-03-30T13:45:24.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:24 smithi113 ceph-mon[34341]: 1.0 scrub starts 2024-03-30T13:45:24.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:24 smithi113 ceph-mon[34341]: 1.0 scrub ok 2024-03-30T13:45:24.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:24 smithi113 ceph-mon[34341]: 5.11 scrub starts 2024-03-30T13:45:24.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:24 smithi113 ceph-mon[34341]: 5.11 scrub ok 2024-03-30T13:45:24.988 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:45:24.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:24.991 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:24.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:25.092 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:25.138 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:24 smithi043 ceph-mon[29963]: 1.0 scrub starts 2024-03-30T13:45:25.138 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:24 smithi043 ceph-mon[29963]: 1.0 scrub ok 2024-03-30T13:45:25.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:24 smithi043 ceph-mon[29963]: 5.11 scrub starts 2024-03-30T13:45:25.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:24 smithi043 ceph-mon[29963]: 5.11 scrub ok 2024-03-30T13:45:25.139 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:24 smithi043 ceph-mon[42099]: 1.0 scrub starts 2024-03-30T13:45:25.139 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:24 smithi043 ceph-mon[42099]: 1.0 scrub ok 2024-03-30T13:45:25.139 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:24 smithi043 ceph-mon[42099]: 5.11 scrub starts 2024-03-30T13:45:25.140 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:24 smithi043 ceph-mon[42099]: 5.11 scrub ok 2024-03-30T13:45:25.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:25.141 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:25.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:25.242 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:25.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:25.277 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:25.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:25.378 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:25.395 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 25 16 50853 50837 127.072 96.75 0.00520171 0.00786411 2024-03-30T13:45:25.412 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:25.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:25.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:25.472 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 podman[73296]: 2024-03-30 13:45:25.026598144 +0000 UTC m=+0.664001082 container remove 37bd40d9ec3389dcb95a2720e99830cecc7b090a5595e04774105eda2f10531d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T13:45:25.472 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@prometheus.a.service: Deactivated successfully. 2024-03-30T13:45:25.472 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 systemd[1]: Stopped Ceph prometheus.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T13:45:25.472 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@prometheus.a.service: Consumed 1.335s CPU time. 2024-03-30T13:45:25.472 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 systemd[1]: Starting Ceph prometheus.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T13:45:25.514 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:25.540 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:25.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:25.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:25.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T13:45:25.642 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:25.675 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:25.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:25.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:25.777 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:25.798 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 podman[73423]: 2024-03-30 13:45:25.390006759 +0000 UTC m=+0.028480616 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-30T13:45:25.798 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 podman[73423]: 2024-03-30 13:45:25.520470814 +0000 UTC m=+0.158944646 container create 396a2e4a311209eb5833acf6b53cc7510955e3cdc18c60eecc0b7bc6a5dbe488 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T13:45:25.806 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:25.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:25.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:25.907 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:25.943 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:25.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:25.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:26.045 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:26.076 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:26.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:26.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:26.157 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[29963]: 3.0 scrub starts 2024-03-30T13:45:26.157 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[29963]: 3.0 scrub ok 2024-03-30T13:45:26.157 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[29963]: 3.f scrub starts 2024-03-30T13:45:26.157 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[29963]: 3.f scrub ok 2024-03-30T13:45:26.157 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[29963]: 4.15 scrub starts 2024-03-30T13:45:26.157 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[29963]: 4.15 scrub ok 2024-03-30T13:45:26.157 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[29963]: 4.e scrub starts 2024-03-30T13:45:26.157 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[29963]: 4.e scrub ok 2024-03-30T13:45:26.157 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:45:26.157 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:45:26.157 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[29963]: pgmap v87: 148 pgs: 148 active+clean; 2.9 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.17k op/s 2024-03-30T13:45:26.158 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:45:25] ENGINE Bus STOPPING 2024-03-30T13:45:26.158 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:45:26] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-30T13:45:26.158 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:45:26] ENGINE Bus STOPPED 2024-03-30T13:45:26.158 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:45:26] ENGINE Bus STARTING 2024-03-30T13:45:26.158 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:45:26] ENGINE Serving on http://:::9283 2024-03-30T13:45:26.158 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:45:26] ENGINE Bus STARTED 2024-03-30T13:45:26.158 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:45:26] ENGINE Bus STOPPING 2024-03-30T13:45:26.158 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:45:26] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-30T13:45:26.158 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:45:26] ENGINE Bus STOPPED 2024-03-30T13:45:26.159 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[42099]: 3.0 scrub starts 2024-03-30T13:45:26.159 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[42099]: 3.0 scrub ok 2024-03-30T13:45:26.159 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[42099]: 3.f scrub starts 2024-03-30T13:45:26.159 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[42099]: 3.f scrub ok 2024-03-30T13:45:26.159 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[42099]: 4.15 scrub starts 2024-03-30T13:45:26.159 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[42099]: 4.15 scrub ok 2024-03-30T13:45:26.159 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[42099]: 4.e scrub starts 2024-03-30T13:45:26.159 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[42099]: 4.e scrub ok 2024-03-30T13:45:26.159 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:45:26.159 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:45:26.159 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:25 smithi043 ceph-mon[42099]: pgmap v87: 148 pgs: 148 active+clean; 2.9 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.17k op/s 2024-03-30T13:45:26.178 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:26.215 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:26.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:26.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:26.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-mon[34341]: 3.0 scrub starts 2024-03-30T13:45:26.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-mon[34341]: 3.0 scrub ok 2024-03-30T13:45:26.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-mon[34341]: 3.f scrub starts 2024-03-30T13:45:26.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-mon[34341]: 3.f scrub ok 2024-03-30T13:45:26.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-mon[34341]: 4.15 scrub starts 2024-03-30T13:45:26.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-mon[34341]: 4.15 scrub ok 2024-03-30T13:45:26.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-mon[34341]: 4.e scrub starts 2024-03-30T13:45:26.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-mon[34341]: 4.e scrub ok 2024-03-30T13:45:26.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:45:26.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:45:26.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-mon[34341]: pgmap v87: 148 pgs: 148 active+clean; 2.9 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.17k op/s 2024-03-30T13:45:26.223 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 podman[73423]: 2024-03-30 13:45:25.797748587 +0000 UTC m=+0.436222433 container init 396a2e4a311209eb5833acf6b53cc7510955e3cdc18c60eecc0b7bc6a5dbe488 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T13:45:26.223 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 podman[73423]: 2024-03-30 13:45:25.804874951 +0000 UTC m=+0.443348789 container start 396a2e4a311209eb5833acf6b53cc7510955e3cdc18c60eecc0b7bc6a5dbe488 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T13:45:26.223 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.840Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-30T13:45:26.223 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.840Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-30T13:45:26.223 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.840Z caller=main.go:570 level=info host_details="(Linux 5.14.0-434.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Mar 26 19:20:47 UTC 2024 x86_64 smithi113 (none))" 2024-03-30T13:45:26.223 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.840Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-30T13:45:26.224 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.840Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-30T13:45:26.224 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.842Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-30T13:45:26.224 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.843Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-30T13:45:26.224 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.844Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-30T13:45:26.224 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.844Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-30T13:45:26.224 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.846Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-30T13:45:26.224 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.846Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.266µs 2024-03-30T13:45:26.224 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.847Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-30T13:45:26.224 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.848Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-30T13:45:26.224 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.849Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-30T13:45:26.224 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.849Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=30.721µs wal_replay_duration=2.051058ms wbl_replay_duration=126ns total_replay_duration=2.106158ms 2024-03-30T13:45:26.224 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.849Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-30T13:45:26.225 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.849Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-30T13:45:26.225 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.849Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-30T13:45:26.225 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.870Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.982898ms db_storage=963ns remote_storage=1.412µs web_handler=458ns query_engine=720ns scrape=603.977µs scrape_sd=122.641µs notify=17.653µs notify_sd=16.866µs rules=19.931035ms tracing=8.135µs 2024-03-30T13:45:26.225 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.871Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-30T13:45:26.225 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T13:45:25.871Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-30T13:45:26.225 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 bash[73423]: 396a2e4a311209eb5833acf6b53cc7510955e3cdc18c60eecc0b7bc6a5dbe488 2024-03-30T13:45:26.225 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 13:45:25 smithi113 systemd[1]: Started Ceph prometheus.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T13:45:26.318 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:26.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:26.350 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:26.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:26.395 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 26 15 52117 52102 125.226 79.0625 0.024456 0.00797959 2024-03-30T13:45:26.452 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:26.484 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:26.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:26.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:26.586 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:26.634 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:26.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:26.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:26.650 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:45:26] ENGINE Bus STARTING 2024-03-30T13:45:26.650 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:45:26] ENGINE Serving on http://:::9283 2024-03-30T13:45:26.650 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:45:26] ENGINE Bus STARTED 2024-03-30T13:45:26.650 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:45:26] ENGINE Bus STOPPING 2024-03-30T13:45:26.650 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:45:26] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-30T13:45:26.650 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:45:26] ENGINE Bus STOPPED 2024-03-30T13:45:26.650 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:45:26] ENGINE Bus STARTING 2024-03-30T13:45:26.650 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:45:26] ENGINE Serving on http://:::9283 2024-03-30T13:45:26.650 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:13:45:26] ENGINE Bus STARTED 2024-03-30T13:45:26.736 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:26.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:26.791 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:26.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:26.892 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:26.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:26.929 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:26.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:27.030 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:27.075 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:27.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:27.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: osdmap e107: 8 total, 8 up, 8 in 2024-03-30T13:45:27.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:27.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:27.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T13:45:27.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T13:45:27.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi043.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T13:45:27.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi043.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T13:45:27.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:27.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T13:45:27.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T13:45:27.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi113.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T13:45:27.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi113.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T13:45:27.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:27.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T13:45:27.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T13:45:27.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi113.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T13:45:27.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi113.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T13:45:27.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:27.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: 4.1f scrub starts 2024-03-30T13:45:27.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: 4.1f scrub ok 2024-03-30T13:45:27.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:45:27.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: osdmap e107: 8 total, 8 up, 8 in 2024-03-30T13:45:27.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:27.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:27.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T13:45:27.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T13:45:27.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi043.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T13:45:27.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi043.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T13:45:27.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:27.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T13:45:27.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T13:45:27.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi113.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T13:45:27.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi113.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T13:45:27.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:27.153 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T13:45:27.153 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T13:45:27.153 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi113.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T13:45:27.153 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi113.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T13:45:27.153 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:27.153 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: 4.1f scrub starts 2024-03-30T13:45:27.153 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: 4.1f scrub ok 2024-03-30T13:45:27.153 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:26 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:45:27.153 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: osdmap e107: 8 total, 8 up, 8 in 2024-03-30T13:45:27.154 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:27.154 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:27.154 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T13:45:27.154 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T13:45:27.154 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi043.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T13:45:27.154 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi043.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T13:45:27.154 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:27.154 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T13:45:27.154 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T13:45:27.154 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi113.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T13:45:27.154 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi113.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T13:45:27.154 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:27.154 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T13:45:27.155 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T13:45:27.155 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi113.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T13:45:27.155 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi113.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T13:45:27.155 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:27.155 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: 4.1f scrub starts 2024-03-30T13:45:27.155 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: 4.1f scrub ok 2024-03-30T13:45:27.155 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:26 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:45:27.177 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:27.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:27.205 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:27.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:27.306 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:27.338 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:27.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:27.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:27.395 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 27 16 53401 53385 123.557 80.1875 0.00273767 0.00808326 2024-03-30T13:45:27.440 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:27.468 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:27.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:27.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:27.570 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:27.609 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:27.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:27.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:27.712 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:27.748 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:27.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:27.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:27.850 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:27.882 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:27.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:27.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:27.984 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:28.028 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:28.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:28.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:28.130 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:28.169 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:28.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:28.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:28.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:27 smithi113 ceph-mon[34341]: 5.1d scrub starts 2024-03-30T13:45:28.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:27 smithi113 ceph-mon[34341]: 5.1d scrub ok 2024-03-30T13:45:28.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:27 smithi113 ceph-mon[34341]: 3.5 scrub starts 2024-03-30T13:45:28.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:27 smithi113 ceph-mon[34341]: 3.5 scrub ok 2024-03-30T13:45:28.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:27 smithi113 ceph-mon[34341]: 4.5 scrub starts 2024-03-30T13:45:28.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:27 smithi113 ceph-mon[34341]: 4.5 scrub ok 2024-03-30T13:45:28.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:27 smithi113 ceph-mon[34341]: 3.17 scrub starts 2024-03-30T13:45:28.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:27 smithi113 ceph-mon[34341]: 3.17 scrub ok 2024-03-30T13:45:28.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:27 smithi113 ceph-mon[34341]: pgmap v89: 148 pgs: 148 active+clean; 2.9 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-30T13:45:28.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:45:28.270 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:28.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:28.311 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:28.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:28.395 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 28 16 54838 54822 122.352 89.8125 0.0126858 0.00816738 2024-03-30T13:45:28.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:27 smithi043 ceph-mon[29963]: 5.1d scrub starts 2024-03-30T13:45:28.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:27 smithi043 ceph-mon[29963]: 5.1d scrub ok 2024-03-30T13:45:28.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:27 smithi043 ceph-mon[29963]: 3.5 scrub starts 2024-03-30T13:45:28.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:27 smithi043 ceph-mon[29963]: 3.5 scrub ok 2024-03-30T13:45:28.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:27 smithi043 ceph-mon[29963]: 4.5 scrub starts 2024-03-30T13:45:28.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:27 smithi043 ceph-mon[29963]: 4.5 scrub ok 2024-03-30T13:45:28.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:27 smithi043 ceph-mon[29963]: 3.17 scrub starts 2024-03-30T13:45:28.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:27 smithi043 ceph-mon[29963]: 3.17 scrub ok 2024-03-30T13:45:28.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:27 smithi043 ceph-mon[29963]: pgmap v89: 148 pgs: 148 active+clean; 2.9 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-30T13:45:28.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:45:28.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:27 smithi043 ceph-mon[42099]: 5.1d scrub starts 2024-03-30T13:45:28.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:27 smithi043 ceph-mon[42099]: 5.1d scrub ok 2024-03-30T13:45:28.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:27 smithi043 ceph-mon[42099]: 3.5 scrub starts 2024-03-30T13:45:28.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:27 smithi043 ceph-mon[42099]: 3.5 scrub ok 2024-03-30T13:45:28.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:27 smithi043 ceph-mon[42099]: 4.5 scrub starts 2024-03-30T13:45:28.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:27 smithi043 ceph-mon[42099]: 4.5 scrub ok 2024-03-30T13:45:28.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:27 smithi043 ceph-mon[42099]: 3.17 scrub starts 2024-03-30T13:45:28.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:27 smithi043 ceph-mon[42099]: 3.17 scrub ok 2024-03-30T13:45:28.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:27 smithi043 ceph-mon[42099]: pgmap v89: 148 pgs: 148 active+clean; 2.9 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-30T13:45:28.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:45:28.412 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:28.455 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:28.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:28.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:28.557 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:28.601 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:28.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:28.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:28.704 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:28.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:45:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:45:28.733 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:28.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:28.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:28.735 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 13:45:28 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[88910]: ts=2024-03-30T13:45:28.605Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003806438s 2024-03-30T13:45:28.835 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:28.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:28.865 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:28.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:28.966 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:28.999 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:29.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:29.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:29.101 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:29.144 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:29.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:29.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:28 smithi043 ceph-mon[29963]: 6.1a scrub starts 2024-03-30T13:45:29.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:28 smithi043 ceph-mon[29963]: 6.1a scrub ok 2024-03-30T13:45:29.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:28 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:29.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:28 smithi043 ceph-mon[29963]: 3.e scrub starts 2024-03-30T13:45:29.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:28 smithi043 ceph-mon[29963]: 3.e scrub ok 2024-03-30T13:45:29.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:28 smithi043 ceph-mon[42099]: 6.1a scrub starts 2024-03-30T13:45:29.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:28 smithi043 ceph-mon[42099]: 6.1a scrub ok 2024-03-30T13:45:29.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:28 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:29.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:28 smithi043 ceph-mon[42099]: 3.e scrub starts 2024-03-30T13:45:29.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:28 smithi043 ceph-mon[42099]: 3.e scrub ok 2024-03-30T13:45:29.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:28 smithi113 ceph-mon[34341]: 6.1a scrub starts 2024-03-30T13:45:29.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:28 smithi113 ceph-mon[34341]: 6.1a scrub ok 2024-03-30T13:45:29.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:28 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:29.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:28 smithi113 ceph-mon[34341]: 3.e scrub starts 2024-03-30T13:45:29.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:28 smithi113 ceph-mon[34341]: 3.e scrub ok 2024-03-30T13:45:29.245 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:29.274 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:29.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:29.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:29.376 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:29.395 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 29 16 56285 56269 121.251 90.4375 0.0173298 0.00824145 2024-03-30T13:45:29.405 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:29.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:29.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:29.507 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:29.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:29.555 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:29.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:29.616 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:45:29.656 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:29.657 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:45:29.700 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:29.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:29.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:29.706 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:45:29.766 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:45:29.803 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:29.807 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:45:29.832 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:29.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:29.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:29.848 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:45:29.891 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:45:29.934 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:29.944 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:45:29.946 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:45:29.946 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:45:29.946 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:45:29.946 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:45:29.947 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:45:29.947 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:45:29.947 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:45:29.947 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:45:29.963 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:29.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:29.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:30.065 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:30.080 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:29 smithi043 ceph-mon[29963]: 3.19 scrub starts 2024-03-30T13:45:30.080 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:29 smithi043 ceph-mon[29963]: 3.19 scrub ok 2024-03-30T13:45:30.080 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:29 smithi043 ceph-mon[29963]: 2.1 scrub starts 2024-03-30T13:45:30.080 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:29 smithi043 ceph-mon[29963]: 2.1 scrub ok 2024-03-30T13:45:30.081 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:29 smithi043 ceph-mon[29963]: 5.1b scrub starts 2024-03-30T13:45:30.081 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:29 smithi043 ceph-mon[29963]: 5.1b scrub ok 2024-03-30T13:45:30.081 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:29 smithi043 ceph-mon[29963]: 3.6 scrub starts 2024-03-30T13:45:30.081 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:29 smithi043 ceph-mon[29963]: 3.6 scrub ok 2024-03-30T13:45:30.081 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:29 smithi043 ceph-mon[29963]: pgmap v90: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 127 MiB/s wr, 2.04k op/s 2024-03-30T13:45:30.081 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:29 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3621579947' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:45:30.081 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:29 smithi043 ceph-mon[42099]: 3.19 scrub starts 2024-03-30T13:45:30.081 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:29 smithi043 ceph-mon[42099]: 3.19 scrub ok 2024-03-30T13:45:30.081 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:29 smithi043 ceph-mon[42099]: 2.1 scrub starts 2024-03-30T13:45:30.081 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:29 smithi043 ceph-mon[42099]: 2.1 scrub ok 2024-03-30T13:45:30.082 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:29 smithi043 ceph-mon[42099]: 5.1b scrub starts 2024-03-30T13:45:30.082 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:29 smithi043 ceph-mon[42099]: 5.1b scrub ok 2024-03-30T13:45:30.082 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:29 smithi043 ceph-mon[42099]: 3.6 scrub starts 2024-03-30T13:45:30.082 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:29 smithi043 ceph-mon[42099]: 3.6 scrub ok 2024-03-30T13:45:30.082 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:29 smithi043 ceph-mon[42099]: pgmap v90: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 127 MiB/s wr, 2.04k op/s 2024-03-30T13:45:30.082 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:29 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3621579947' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:45:30.098 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:30.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:30.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:30.125 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 reset_purged_snaps_last 2024-03-30T13:45:30.200 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:30.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:29 smithi113 ceph-mon[34341]: 3.19 scrub starts 2024-03-30T13:45:30.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:29 smithi113 ceph-mon[34341]: 3.19 scrub ok 2024-03-30T13:45:30.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:29 smithi113 ceph-mon[34341]: 2.1 scrub starts 2024-03-30T13:45:30.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:29 smithi113 ceph-mon[34341]: 2.1 scrub ok 2024-03-30T13:45:30.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:29 smithi113 ceph-mon[34341]: 5.1b scrub starts 2024-03-30T13:45:30.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:29 smithi113 ceph-mon[34341]: 5.1b scrub ok 2024-03-30T13:45:30.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:29 smithi113 ceph-mon[34341]: 3.6 scrub starts 2024-03-30T13:45:30.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:29 smithi113 ceph-mon[34341]: 3.6 scrub ok 2024-03-30T13:45:30.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:29 smithi113 ceph-mon[34341]: pgmap v90: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 127 MiB/s wr, 2.04k op/s 2024-03-30T13:45:30.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:29 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3621579947' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:45:30.227 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:30.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:30.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:30.329 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:30.355 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:30.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:30.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:30.395 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 30 16 57526 57510 119.795 77.5625 0.0175214 0.00834151 2024-03-30T13:45:30.457 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:30.485 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:30.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:30.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:30.587 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:30.617 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:30.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:30.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:30.719 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:30.764 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:30.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:30.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:30.866 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:30.907 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:30.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:30.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:31.009 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:31.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:31.051 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:31.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:31.081 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:30 smithi113 ceph-mon[34341]: 6.7 scrub starts 2024-03-30T13:45:31.081 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:30 smithi113 ceph-mon[34341]: 6.7 scrub ok 2024-03-30T13:45:31.081 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:30 smithi113 ceph-mon[34341]: 6.17 scrub starts 2024-03-30T13:45:31.081 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:30 smithi113 ceph-mon[34341]: 6.17 scrub ok 2024-03-30T13:45:31.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:30 smithi043 ceph-mon[29963]: 6.7 scrub starts 2024-03-30T13:45:31.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:30 smithi043 ceph-mon[29963]: 6.7 scrub ok 2024-03-30T13:45:31.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:30 smithi043 ceph-mon[29963]: 6.17 scrub starts 2024-03-30T13:45:31.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:30 smithi043 ceph-mon[29963]: 6.17 scrub ok 2024-03-30T13:45:31.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:30 smithi043 ceph-mon[42099]: 6.7 scrub starts 2024-03-30T13:45:31.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:30 smithi043 ceph-mon[42099]: 6.7 scrub ok 2024-03-30T13:45:31.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:30 smithi043 ceph-mon[42099]: 6.17 scrub starts 2024-03-30T13:45:31.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:30 smithi043 ceph-mon[42099]: 6.17 scrub ok 2024-03-30T13:45:31.152 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:31.181 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:31.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:31.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:31.283 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:31.309 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:31.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:31.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:31.315 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:45:31.398 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 31 16 58665 58649 118.224 71.1875 0.027643 0.00845296 2024-03-30T13:45:31.411 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:31.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:31.442 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:31.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:31.543 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:31.574 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:31.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:31.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:31.676 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:31.723 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:31.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:31.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:31.825 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:31.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:31.864 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:31.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:31.965 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:32.008 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:32.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:32.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:32.030 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:31 smithi043 ceph-mon[29963]: 5.a scrub starts 2024-03-30T13:45:32.030 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:31 smithi043 ceph-mon[29963]: 5.a scrub ok 2024-03-30T13:45:32.030 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:31 smithi043 ceph-mon[29963]: 5.5 scrub starts 2024-03-30T13:45:32.030 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:31 smithi043 ceph-mon[29963]: 5.5 scrub ok 2024-03-30T13:45:32.030 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:31 smithi043 ceph-mon[29963]: pgmap v91: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.80k op/s 2024-03-30T13:45:32.031 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:31 smithi043 ceph-mon[42099]: 5.a scrub starts 2024-03-30T13:45:32.031 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:31 smithi043 ceph-mon[42099]: 5.a scrub ok 2024-03-30T13:45:32.031 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:31 smithi043 ceph-mon[42099]: 5.5 scrub starts 2024-03-30T13:45:32.031 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:31 smithi043 ceph-mon[42099]: 5.5 scrub ok 2024-03-30T13:45:32.031 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:31 smithi043 ceph-mon[42099]: pgmap v91: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.80k op/s 2024-03-30T13:45:32.110 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:32.140 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:32.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:32.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:32.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:31 smithi113 ceph-mon[34341]: 5.a scrub starts 2024-03-30T13:45:32.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:31 smithi113 ceph-mon[34341]: 5.a scrub ok 2024-03-30T13:45:32.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:31 smithi113 ceph-mon[34341]: 5.5 scrub starts 2024-03-30T13:45:32.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:31 smithi113 ceph-mon[34341]: 5.5 scrub ok 2024-03-30T13:45:32.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:31 smithi113 ceph-mon[34341]: pgmap v91: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.80k op/s 2024-03-30T13:45:32.243 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:32.290 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:32.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:32.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:32.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:32.396 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 32 16 59886 59870 116.914 76.3125 0.00659137 0.00854782 2024-03-30T13:45:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:32.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:32.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:32.530 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:32.562 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:32.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:32.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:32.574 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:45:32.574 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T13:45:32.575 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-03-30T13:45:32.575 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-03-30T13:45:32.575 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:45:32.664 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:32.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:32.706 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:32.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:32.798 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:45:32.808 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:32.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:45:32.837 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:32.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:32.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:32.940 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:33.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:33.006 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:33.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:33.107 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:33.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[29963]: 3.a scrub starts 2024-03-30T13:45:33.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[29963]: 3.a scrub ok 2024-03-30T13:45:33.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[29963]: 3.8 scrub starts 2024-03-30T13:45:33.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[29963]: 3.8 scrub ok 2024-03-30T13:45:33.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[29963]: 5.d scrub starts 2024-03-30T13:45:33.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[29963]: 5.d scrub ok 2024-03-30T13:45:33.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[29963]: 6.19 scrub starts 2024-03-30T13:45:33.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[29963]: 6.19 scrub ok 2024-03-30T13:45:33.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[29963]: 5.1 scrub starts 2024-03-30T13:45:33.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[29963]: 5.1 scrub ok 2024-03-30T13:45:33.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[29963]: 4.12 scrub starts 2024-03-30T13:45:33.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[29963]: 4.12 scrub ok 2024-03-30T13:45:33.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[42099]: 3.a scrub starts 2024-03-30T13:45:33.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[42099]: 3.a scrub ok 2024-03-30T13:45:33.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[42099]: 3.8 scrub starts 2024-03-30T13:45:33.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[42099]: 3.8 scrub ok 2024-03-30T13:45:33.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[42099]: 5.d scrub starts 2024-03-30T13:45:33.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[42099]: 5.d scrub ok 2024-03-30T13:45:33.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[42099]: 6.19 scrub starts 2024-03-30T13:45:33.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[42099]: 6.19 scrub ok 2024-03-30T13:45:33.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[42099]: 5.1 scrub starts 2024-03-30T13:45:33.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[42099]: 5.1 scrub ok 2024-03-30T13:45:33.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[42099]: 4.12 scrub starts 2024-03-30T13:45:33.152 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:32 smithi043 ceph-mon[42099]: 4.12 scrub ok 2024-03-30T13:45:33.157 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:33.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:33.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:33.194 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:32 smithi113 ceph-mon[34341]: 3.a scrub starts 2024-03-30T13:45:33.194 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:32 smithi113 ceph-mon[34341]: 3.a scrub ok 2024-03-30T13:45:33.194 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:32 smithi113 ceph-mon[34341]: 3.8 scrub starts 2024-03-30T13:45:33.194 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:32 smithi113 ceph-mon[34341]: 3.8 scrub ok 2024-03-30T13:45:33.194 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:32 smithi113 ceph-mon[34341]: 5.d scrub starts 2024-03-30T13:45:33.194 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:32 smithi113 ceph-mon[34341]: 5.d scrub ok 2024-03-30T13:45:33.194 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:32 smithi113 ceph-mon[34341]: 6.19 scrub starts 2024-03-30T13:45:33.194 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:32 smithi113 ceph-mon[34341]: 6.19 scrub ok 2024-03-30T13:45:33.194 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:32 smithi113 ceph-mon[34341]: 5.1 scrub starts 2024-03-30T13:45:33.194 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:32 smithi113 ceph-mon[34341]: 5.1 scrub ok 2024-03-30T13:45:33.195 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:32 smithi113 ceph-mon[34341]: 4.12 scrub starts 2024-03-30T13:45:33.195 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:32 smithi113 ceph-mon[34341]: 4.12 scrub ok 2024-03-30T13:45:33.259 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:33.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:33.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:33.396 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 33 15 61178 61163 115.82 80.8125 0.0171575 0.00862945 2024-03-30T13:45:33.399 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:33.431 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:33.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:33.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:33.533 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:33.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:33.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:33.675 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:33.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:33.709 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:33.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:33.810 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:33.852 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:33.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:33.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:33.954 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:33.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:33.980 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:33.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:34.081 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:34.120 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:34.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:34.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:34.222 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:34.253 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:34.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:34.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:34.355 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:34.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:34.387 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:34.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:34.397 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 34 16 62432 62416 114.717 78.3125 0.00553298 0.00870712 2024-03-30T13:45:34.488 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:34.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:34.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:34.631 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:34.663 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:34.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:34.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:34.765 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:34.806 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:34.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:34.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:34.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:34 smithi043 ceph-mon[29963]: 6.1 scrub starts 2024-03-30T13:45:34.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:34 smithi043 ceph-mon[29963]: 6.1 scrub ok 2024-03-30T13:45:34.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:34 smithi043 ceph-mon[29963]: 5.2 scrub starts 2024-03-30T13:45:34.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:34 smithi043 ceph-mon[29963]: 5.2 scrub ok 2024-03-30T13:45:34.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:34 smithi043 ceph-mon[29963]: pgmap v92: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-30T13:45:34.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:34 smithi043 ceph-mon[29963]: 6.1b scrub starts 2024-03-30T13:45:34.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:34 smithi043 ceph-mon[29963]: 6.1b scrub ok 2024-03-30T13:45:34.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:34 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:34.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:34 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:34.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:34 smithi043 ceph-mon[42099]: 6.1 scrub starts 2024-03-30T13:45:34.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:34 smithi043 ceph-mon[42099]: 6.1 scrub ok 2024-03-30T13:45:34.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:34 smithi043 ceph-mon[42099]: 5.2 scrub starts 2024-03-30T13:45:34.902 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:34 smithi043 ceph-mon[42099]: 5.2 scrub ok 2024-03-30T13:45:34.902 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:34 smithi043 ceph-mon[42099]: pgmap v92: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-30T13:45:34.902 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:34 smithi043 ceph-mon[42099]: 6.1b scrub starts 2024-03-30T13:45:34.902 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:34 smithi043 ceph-mon[42099]: 6.1b scrub ok 2024-03-30T13:45:34.902 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:34 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:34.902 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:34 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:34.908 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:34.939 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:34.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:34.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:34.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:34 smithi113 ceph-mon[34341]: 6.1 scrub starts 2024-03-30T13:45:34.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:34 smithi113 ceph-mon[34341]: 6.1 scrub ok 2024-03-30T13:45:34.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:34 smithi113 ceph-mon[34341]: 5.2 scrub starts 2024-03-30T13:45:34.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:34 smithi113 ceph-mon[34341]: 5.2 scrub ok 2024-03-30T13:45:34.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:34 smithi113 ceph-mon[34341]: pgmap v92: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-30T13:45:34.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:34 smithi113 ceph-mon[34341]: 6.1b scrub starts 2024-03-30T13:45:34.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:34 smithi113 ceph-mon[34341]: 6.1b scrub ok 2024-03-30T13:45:34.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:34 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:34.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:34 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:35.041 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:35.069 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:35.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:35.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:35.171 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:35.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:35.215 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:35.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:35.316 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:35.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:35.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:35.398 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 35 16 63699 63683 113.697 79.1875 0.0142904 0.00878843 2024-03-30T13:45:35.405 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-03-30T13:45:35.458 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:35.554 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:35.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:35.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:35.622 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:35 smithi043 ceph-mon[29963]: 3.1a scrub starts 2024-03-30T13:45:35.622 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:35 smithi043 ceph-mon[29963]: 3.1a scrub ok 2024-03-30T13:45:35.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:35 smithi043 ceph-mon[42099]: 3.1a scrub starts 2024-03-30T13:45:35.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:35 smithi043 ceph-mon[42099]: 3.1a scrub ok 2024-03-30T13:45:35.657 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:35.686 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:35.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:35.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:35.788 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:35.845 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:35.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:35.947 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:35.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:35 smithi113 ceph-mon[34341]: 3.1a scrub starts 2024-03-30T13:45:35.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:35 smithi113 ceph-mon[34341]: 3.1a scrub ok 2024-03-30T13:45:35.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:35.995 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:35.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:36.096 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:36.126 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:36.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:36.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:36.229 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:36.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:36.259 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:36.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:36.361 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:36.390 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:36.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:36.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:36.400 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:45:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:45:36.400 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 36 16 64962 64946 112.729 78.9375 0.00855497 0.00886526 2024-03-30T13:45:36.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:36.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:36.553 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:36.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:36.654 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:36.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:36.685 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:36.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:36.786 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:36.817 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:36.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:36.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:36.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:36 smithi043 ceph-mon[29963]: 5.18 scrub starts 2024-03-30T13:45:36.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:36 smithi043 ceph-mon[29963]: 5.18 scrub ok 2024-03-30T13:45:36.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:36 smithi043 ceph-mon[29963]: 3.10 scrub starts 2024-03-30T13:45:36.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:36 smithi043 ceph-mon[29963]: 3.10 scrub ok 2024-03-30T13:45:36.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:36 smithi043 ceph-mon[29963]: pgmap v93: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.35k op/s 2024-03-30T13:45:36.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:36 smithi043 ceph-mon[29963]: 4.14 scrub starts 2024-03-30T13:45:36.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:36 smithi043 ceph-mon[29963]: 4.14 scrub ok 2024-03-30T13:45:36.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:36 smithi043 ceph-mon[42099]: 5.18 scrub starts 2024-03-30T13:45:36.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:36 smithi043 ceph-mon[42099]: 5.18 scrub ok 2024-03-30T13:45:36.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:36 smithi043 ceph-mon[42099]: 3.10 scrub starts 2024-03-30T13:45:36.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:36 smithi043 ceph-mon[42099]: 3.10 scrub ok 2024-03-30T13:45:36.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:36 smithi043 ceph-mon[42099]: pgmap v93: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.35k op/s 2024-03-30T13:45:36.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:36 smithi043 ceph-mon[42099]: 4.14 scrub starts 2024-03-30T13:45:36.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:36 smithi043 ceph-mon[42099]: 4.14 scrub ok 2024-03-30T13:45:36.919 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:36.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:36.948 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:36.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:36.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:36 smithi113 ceph-mon[34341]: 5.18 scrub starts 2024-03-30T13:45:36.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:36 smithi113 ceph-mon[34341]: 5.18 scrub ok 2024-03-30T13:45:36.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:36 smithi113 ceph-mon[34341]: 3.10 scrub starts 2024-03-30T13:45:36.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:36 smithi113 ceph-mon[34341]: 3.10 scrub ok 2024-03-30T13:45:36.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:36 smithi113 ceph-mon[34341]: pgmap v93: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.35k op/s 2024-03-30T13:45:36.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:36 smithi113 ceph-mon[34341]: 4.14 scrub starts 2024-03-30T13:45:36.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:36 smithi113 ceph-mon[34341]: 4.14 scrub ok 2024-03-30T13:45:37.049 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:37.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:37.096 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:37.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:37.198 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:37.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:37.230 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:37.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:37.331 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:37.363 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:37.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:37.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:37.399 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 37 16 66303 66287 111.947 83.8125 0.0109781 0.00892777 2024-03-30T13:45:37.465 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:37.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:37.525 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:37.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:37.626 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:37.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:37.656 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:37.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:37.757 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:37.814 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:37 smithi043 ceph-mon[29963]: 3.15 scrub starts 2024-03-30T13:45:37.815 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:37 smithi043 ceph-mon[29963]: 3.15 scrub ok 2024-03-30T13:45:37.815 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:37 smithi043 ceph-mon[42099]: 3.15 scrub starts 2024-03-30T13:45:37.815 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:37 smithi043 ceph-mon[42099]: 3.15 scrub ok 2024-03-30T13:45:37.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:37.825 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:37.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:37.926 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:37.970 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:37.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:37.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:37.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:37 smithi113 ceph-mon[34341]: 3.15 scrub starts 2024-03-30T13:45:37.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:37 smithi113 ceph-mon[34341]: 3.15 scrub ok 2024-03-30T13:45:38.072 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:38.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:38.103 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:38.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:38.205 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:38.235 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:38.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:38.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:38.337 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:38.389 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:38.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:38.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:38.399 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 38 16 67595 67579 111.125 80.75 0.00863187 0.00899313 2024-03-30T13:45:38.492 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:38.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:38.541 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:38.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:38.584 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:38 smithi043 ceph-mon[29963]: pgmap v94: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 897 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-30T13:45:38.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:38 smithi043 ceph-mon[29963]: 5.9 scrub starts 2024-03-30T13:45:38.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:38 smithi043 ceph-mon[29963]: 5.9 scrub ok 2024-03-30T13:45:38.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:38 smithi043 ceph-mon[29963]: 4.1c scrub starts 2024-03-30T13:45:38.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:38 smithi043 ceph-mon[29963]: 4.1c scrub ok 2024-03-30T13:45:38.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:38 smithi043 ceph-mon[29963]: 5.19 scrub starts 2024-03-30T13:45:38.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:38 smithi043 ceph-mon[29963]: 5.19 scrub ok 2024-03-30T13:45:38.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:38 smithi043 ceph-mon[29963]: 4.3 scrub starts 2024-03-30T13:45:38.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:38 smithi043 ceph-mon[29963]: 4.3 scrub ok 2024-03-30T13:45:38.585 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:38 smithi043 ceph-mon[42099]: pgmap v94: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 897 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-30T13:45:38.585 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:38 smithi043 ceph-mon[42099]: 5.9 scrub starts 2024-03-30T13:45:38.585 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:38 smithi043 ceph-mon[42099]: 5.9 scrub ok 2024-03-30T13:45:38.585 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:38 smithi043 ceph-mon[42099]: 4.1c scrub starts 2024-03-30T13:45:38.586 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:38 smithi043 ceph-mon[42099]: 4.1c scrub ok 2024-03-30T13:45:38.586 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:38 smithi043 ceph-mon[42099]: 5.19 scrub starts 2024-03-30T13:45:38.586 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:38 smithi043 ceph-mon[42099]: 5.19 scrub ok 2024-03-30T13:45:38.586 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:38 smithi043 ceph-mon[42099]: 4.3 scrub starts 2024-03-30T13:45:38.586 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:38 smithi043 ceph-mon[42099]: 4.3 scrub ok 2024-03-30T13:45:38.643 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:38.671 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:38.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:38.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:38.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:45:38 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:45:38.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:38 smithi113 ceph-mon[34341]: pgmap v94: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 897 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-30T13:45:38.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:38 smithi113 ceph-mon[34341]: 5.9 scrub starts 2024-03-30T13:45:38.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:38 smithi113 ceph-mon[34341]: 5.9 scrub ok 2024-03-30T13:45:38.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:38 smithi113 ceph-mon[34341]: 4.1c scrub starts 2024-03-30T13:45:38.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:38 smithi113 ceph-mon[34341]: 4.1c scrub ok 2024-03-30T13:45:38.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:38 smithi113 ceph-mon[34341]: 5.19 scrub starts 2024-03-30T13:45:38.724 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:38 smithi113 ceph-mon[34341]: 5.19 scrub ok 2024-03-30T13:45:38.724 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:38 smithi113 ceph-mon[34341]: 4.3 scrub starts 2024-03-30T13:45:38.724 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:38 smithi113 ceph-mon[34341]: 4.3 scrub ok 2024-03-30T13:45:38.738 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:45:38.774 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:38.800 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:38.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:38.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:38.903 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:38.930 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:38.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:38.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:39.033 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:39.060 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:39.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:39.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:39.162 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:39.191 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:39.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:39.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:39.293 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:39.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:39.340 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:39.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:39.399 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 39 16 68750 68734 110.126 72.1875 0.01147 0.00907427 2024-03-30T13:45:39.442 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:39.486 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:39.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:39.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-03-30T13:45:39.589 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:39.636 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:39.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:39.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:39.738 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:39.777 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:39.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:39.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:39.863 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:39 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:39.864 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:39 smithi043 ceph-mon[29963]: 3.1c scrub starts 2024-03-30T13:45:39.864 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:39 smithi043 ceph-mon[29963]: 3.1c scrub ok 2024-03-30T13:45:39.864 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:39 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1940011784' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:45:39.864 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:39 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:45:39.864 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:39 smithi043 ceph-mon[29963]: 5.17 scrub starts 2024-03-30T13:45:39.864 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:39 smithi043 ceph-mon[29963]: 5.17 scrub ok 2024-03-30T13:45:39.864 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:39 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:39.864 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:39 smithi043 ceph-mon[42099]: 3.1c scrub starts 2024-03-30T13:45:39.864 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:39 smithi043 ceph-mon[42099]: 3.1c scrub ok 2024-03-30T13:45:39.864 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:39 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1940011784' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:45:39.865 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:39 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:45:39.865 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:39 smithi043 ceph-mon[42099]: 5.17 scrub starts 2024-03-30T13:45:39.865 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:39 smithi043 ceph-mon[42099]: 5.17 scrub ok 2024-03-30T13:45:39.878 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:39.910 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:39.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:39.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:39.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:39 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:39.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:39 smithi113 ceph-mon[34341]: 3.1c scrub starts 2024-03-30T13:45:39.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:39 smithi113 ceph-mon[34341]: 3.1c scrub ok 2024-03-30T13:45:39.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:39 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1940011784' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:45:39.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:39 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:45:39.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:39 smithi113 ceph-mon[34341]: 5.17 scrub starts 2024-03-30T13:45:39.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:39 smithi113 ceph-mon[34341]: 5.17 scrub ok 2024-03-30T13:45:40.012 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:40.057 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:40.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:40.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:40.086 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:45:40.159 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:40.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:40.217 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:40.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:40.318 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:40.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:40.351 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:40.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:40.400 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-03-30T13:45:40.399406+0000 min lat: 0.00133902 max lat: 0.102529 avg lat: 0.00914582 2024-03-30T13:45:40.400 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T13:45:40.400 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 40 15 69967 69952 109.276 76.125 0.00788397 0.00914582 2024-03-30T13:45:40.453 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:40.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:40.497 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:40.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:40.563 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:40 smithi043 ceph-mon[29963]: pgmap v95: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-30T13:45:40.563 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:40 smithi043 ceph-mon[29963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:45:40.563 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:40 smithi043 ceph-mon[29963]: Cluster is now healthy 2024-03-30T13:45:40.563 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:40 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:45:40.599 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:40.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:40.641 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:40.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:40.742 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:40.786 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:40.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:40.887 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:40.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:40 smithi043 ceph-mon[29963]: osdmap e108: 8 total, 8 up, 8 in 2024-03-30T13:45:40.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:40 smithi043 ceph-mon[42099]: pgmap v95: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-30T13:45:40.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:40 smithi043 ceph-mon[42099]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:45:40.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:40 smithi043 ceph-mon[42099]: Cluster is now healthy 2024-03-30T13:45:40.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:40 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:45:40.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:40 smithi043 ceph-mon[42099]: osdmap e108: 8 total, 8 up, 8 in 2024-03-30T13:45:40.919 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:40.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:40.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:40 smithi113 ceph-mon[34341]: pgmap v95: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-30T13:45:40.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:40 smithi113 ceph-mon[34341]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:45:40.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:40 smithi113 ceph-mon[34341]: Cluster is now healthy 2024-03-30T13:45:40.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:40 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:45:40.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:40 smithi113 ceph-mon[34341]: osdmap e108: 8 total, 8 up, 8 in 2024-03-30T13:45:41.021 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:41.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:41.074 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:41.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:41.175 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:41.205 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:41.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:41.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:41.307 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:41.338 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:41.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:41.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:41.400 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 41 16 71256 71240 108.574 80.5 0.0117433 0.00920475 2024-03-30T13:45:41.440 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:41.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:41.473 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:41.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:41.574 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:41.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:41.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:41.726 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:41.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:41.768 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:41.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:41.869 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:41.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:41.895 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:41.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:41.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:41 smithi043 ceph-mon[29963]: 3.c scrub starts 2024-03-30T13:45:41.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:41 smithi043 ceph-mon[29963]: 3.c scrub ok 2024-03-30T13:45:41.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:41 smithi043 ceph-mon[29963]: 4.7 scrub starts 2024-03-30T13:45:41.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:41 smithi043 ceph-mon[29963]: 4.7 scrub ok 2024-03-30T13:45:41.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:41 smithi043 ceph-mon[29963]: 5.1f scrub starts 2024-03-30T13:45:41.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:41 smithi043 ceph-mon[29963]: 5.1f scrub ok 2024-03-30T13:45:41.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:41 smithi043 ceph-mon[29963]: 6.0 deep-scrub starts 2024-03-30T13:45:41.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:41 smithi043 ceph-mon[29963]: 6.0 deep-scrub ok 2024-03-30T13:45:41.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:41 smithi043 ceph-mon[42099]: 3.c scrub starts 2024-03-30T13:45:41.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:41 smithi043 ceph-mon[42099]: 3.c scrub ok 2024-03-30T13:45:41.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:41 smithi043 ceph-mon[42099]: 4.7 scrub starts 2024-03-30T13:45:41.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:41 smithi043 ceph-mon[42099]: 4.7 scrub ok 2024-03-30T13:45:41.902 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:41 smithi043 ceph-mon[42099]: 5.1f scrub starts 2024-03-30T13:45:41.902 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:41 smithi043 ceph-mon[42099]: 5.1f scrub ok 2024-03-30T13:45:41.902 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:41 smithi043 ceph-mon[42099]: 6.0 deep-scrub starts 2024-03-30T13:45:41.902 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:41 smithi043 ceph-mon[42099]: 6.0 deep-scrub ok 2024-03-30T13:45:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:41 smithi113 ceph-mon[34341]: 3.c scrub starts 2024-03-30T13:45:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:41 smithi113 ceph-mon[34341]: 3.c scrub ok 2024-03-30T13:45:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:41 smithi113 ceph-mon[34341]: 4.7 scrub starts 2024-03-30T13:45:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:41 smithi113 ceph-mon[34341]: 4.7 scrub ok 2024-03-30T13:45:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:41 smithi113 ceph-mon[34341]: 5.1f scrub starts 2024-03-30T13:45:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:41 smithi113 ceph-mon[34341]: 5.1f scrub ok 2024-03-30T13:45:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:41 smithi113 ceph-mon[34341]: 6.0 deep-scrub starts 2024-03-30T13:45:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:41 smithi113 ceph-mon[34341]: 6.0 deep-scrub ok 2024-03-30T13:45:41.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:42.040 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:42.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:42.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:42.141 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:42.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:42.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:42.276 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:42.319 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:42.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:42.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:42.400 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 42 16 72436 72420 107.745 73.75 0.012942 0.00927629 2024-03-30T13:45:42.420 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:42.460 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:42.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:42.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:42.563 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:42.591 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:42.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:42.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:42.694 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:42.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:42.723 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:42.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:42.824 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:42.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:42.861 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:42.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:42.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:42 smithi043 ceph-mon[29963]: pgmap v97: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-30T13:45:42.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:42 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:42.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:42 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:45:42.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:42 smithi043 ceph-mon[42099]: pgmap v97: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-30T13:45:42.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:42 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:42.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:42 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:45:42.962 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:42.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:42 smithi113 ceph-mon[34341]: pgmap v97: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-30T13:45:42.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:42 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:42.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:42 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:45:43.007 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:43.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:43.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:43.109 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:43.152 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:43.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:43.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:43.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:43.297 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:43.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:43.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:43.398 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:43.400 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 43 16 73663 73647 107.022 76.6875 0.00933899 0.00933854 2024-03-30T13:45:43.442 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:43.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:43.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:43.525 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T13:45:43.544 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:43.626 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:43.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:43.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:43.728 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:43.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:43.761 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:43.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:43.862 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:43.886 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:43.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:43.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:43.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:43 smithi043 ceph-mon[42099]: 3.11 scrub starts 2024-03-30T13:45:43.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:43 smithi043 ceph-mon[42099]: 3.11 scrub ok 2024-03-30T13:45:43.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:43 smithi043 ceph-mon[29963]: 3.11 scrub starts 2024-03-30T13:45:43.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:43 smithi043 ceph-mon[29963]: 3.11 scrub ok 2024-03-30T13:45:43.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:43 smithi113 ceph-mon[34341]: 3.11 scrub starts 2024-03-30T13:45:43.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:43 smithi113 ceph-mon[34341]: 3.11 scrub ok 2024-03-30T13:45:43.987 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:44.017 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:44.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:44.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:44.119 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:44.161 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:44.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:44.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:44.263 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:44.295 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:44.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:44.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:44.398 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:44.401 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 44 16 74767 74751 106.158 69 0.0099329 0.00941512 2024-03-30T13:45:44.442 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:44.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:44.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:44.545 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:44.574 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:44.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:44.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:44.676 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:44.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:44.705 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:44.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:44.806 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:44.835 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:44.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:44.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:44.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:44 smithi043 ceph-mon[29963]: 3.9 scrub starts 2024-03-30T13:45:44.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:44 smithi043 ceph-mon[29963]: 3.9 scrub ok 2024-03-30T13:45:44.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:44 smithi043 ceph-mon[29963]: pgmap v98: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-30T13:45:44.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:44 smithi043 ceph-mon[29963]: 6.13 scrub starts 2024-03-30T13:45:44.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:44 smithi043 ceph-mon[29963]: 6.13 scrub ok 2024-03-30T13:45:44.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:44 smithi043 ceph-mon[29963]: 2.2 scrub starts 2024-03-30T13:45:44.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:44 smithi043 ceph-mon[29963]: 2.2 scrub ok 2024-03-30T13:45:44.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:44 smithi043 ceph-mon[42099]: 3.9 scrub starts 2024-03-30T13:45:44.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:44 smithi043 ceph-mon[42099]: 3.9 scrub ok 2024-03-30T13:45:44.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:44 smithi043 ceph-mon[42099]: pgmap v98: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-30T13:45:44.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:44 smithi043 ceph-mon[42099]: 6.13 scrub starts 2024-03-30T13:45:44.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:44 smithi043 ceph-mon[42099]: 6.13 scrub ok 2024-03-30T13:45:44.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:44 smithi043 ceph-mon[42099]: 2.2 scrub starts 2024-03-30T13:45:44.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:44 smithi043 ceph-mon[42099]: 2.2 scrub ok 2024-03-30T13:45:44.937 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:44.967 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:44.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:44.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:44.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:44 smithi113 ceph-mon[34341]: 3.9 scrub starts 2024-03-30T13:45:44.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:44 smithi113 ceph-mon[34341]: 3.9 scrub ok 2024-03-30T13:45:44.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:44 smithi113 ceph-mon[34341]: pgmap v98: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-30T13:45:44.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:44 smithi113 ceph-mon[34341]: 6.13 scrub starts 2024-03-30T13:45:44.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:44 smithi113 ceph-mon[34341]: 6.13 scrub ok 2024-03-30T13:45:44.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:44 smithi113 ceph-mon[34341]: 2.2 scrub starts 2024-03-30T13:45:44.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:44 smithi113 ceph-mon[34341]: 2.2 scrub ok 2024-03-30T13:45:44.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 reset_purged_snaps_last 2024-03-30T13:45:45.069 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:45.096 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:45.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:45.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:45.198 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:45.224 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:45.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:45.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:45.323 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:45:45.328 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:45.369 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:45:45.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:45.383 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:45.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:45.400 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 45 16 75763 75747 105.182 62.25 0.0166897 0.00950252 2024-03-30T13:45:45.416 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:45:45.465 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:45:45.484 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:45.506 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:45:45.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:45.524 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:45.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:45.547 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:45:45.593 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:45:45.625 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:45.629 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:45:45.642 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:45:45.643 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:45:45.643 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:45:45.643 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:45:45.643 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:45:45.643 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:45:45.643 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:45:45.643 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:45:45.676 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:45.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:45.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:45.778 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:45.806 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:45.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:45.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:45.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:45 smithi043 ceph-mon[29963]: 4.13 scrub starts 2024-03-30T13:45:45.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:45 smithi043 ceph-mon[29963]: 4.13 scrub ok 2024-03-30T13:45:45.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:45 smithi043 ceph-mon[29963]: 6.c scrub starts 2024-03-30T13:45:45.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:45 smithi043 ceph-mon[29963]: 6.c scrub ok 2024-03-30T13:45:45.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:45 smithi043 ceph-mon[29963]: 6.4 scrub starts 2024-03-30T13:45:45.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:45 smithi043 ceph-mon[29963]: 6.4 scrub ok 2024-03-30T13:45:45.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:45 smithi043 ceph-mon[29963]: 5.1e scrub starts 2024-03-30T13:45:45.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:45 smithi043 ceph-mon[29963]: 5.1e scrub ok 2024-03-30T13:45:45.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:45 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/236537275' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:45:45.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:45 smithi043 ceph-mon[42099]: 4.13 scrub starts 2024-03-30T13:45:45.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:45 smithi043 ceph-mon[42099]: 4.13 scrub ok 2024-03-30T13:45:45.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:45 smithi043 ceph-mon[42099]: 6.c scrub starts 2024-03-30T13:45:45.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:45 smithi043 ceph-mon[42099]: 6.c scrub ok 2024-03-30T13:45:45.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:45 smithi043 ceph-mon[42099]: 6.4 scrub starts 2024-03-30T13:45:45.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:45 smithi043 ceph-mon[42099]: 6.4 scrub ok 2024-03-30T13:45:45.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:45 smithi043 ceph-mon[42099]: 5.1e scrub starts 2024-03-30T13:45:45.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:45 smithi043 ceph-mon[42099]: 5.1e scrub ok 2024-03-30T13:45:45.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:45 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/236537275' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:45:45.907 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:45.937 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:45.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:45.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:45.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:45 smithi113 ceph-mon[34341]: 4.13 scrub starts 2024-03-30T13:45:45.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:45 smithi113 ceph-mon[34341]: 4.13 scrub ok 2024-03-30T13:45:45.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:45 smithi113 ceph-mon[34341]: 6.c scrub starts 2024-03-30T13:45:45.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:45 smithi113 ceph-mon[34341]: 6.c scrub ok 2024-03-30T13:45:45.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:45 smithi113 ceph-mon[34341]: 6.4 scrub starts 2024-03-30T13:45:45.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:45 smithi113 ceph-mon[34341]: 6.4 scrub ok 2024-03-30T13:45:45.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:45 smithi113 ceph-mon[34341]: 5.1e scrub starts 2024-03-30T13:45:45.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:45 smithi113 ceph-mon[34341]: 5.1e scrub ok 2024-03-30T13:45:45.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:45 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/236537275' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:45:46.038 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:46.067 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:46.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:46.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:46.169 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:46.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:46.198 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:46.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:46.299 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:46.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:46.343 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:46.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:46.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:45:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:45:46.400 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 46 15 76966 76951 104.532 75.25 0.0199812 0.00956182 2024-03-30T13:45:46.445 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:46.494 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:46.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:46.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:46.596 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:46.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:46.644 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:46.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:46.745 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:46.776 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:46.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:46.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:46.813 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:46 smithi043 ceph-mon[29963]: 4.1 scrub starts 2024-03-30T13:45:46.813 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:46 smithi043 ceph-mon[29963]: 4.1 scrub ok 2024-03-30T13:45:46.814 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:46 smithi043 ceph-mon[29963]: pgmap v99: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-30T13:45:46.814 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:46 smithi043 ceph-mon[29963]: 6.12 scrub starts 2024-03-30T13:45:46.814 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:46 smithi043 ceph-mon[29963]: 6.12 scrub ok 2024-03-30T13:45:46.814 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:46 smithi043 ceph-mon[29963]: 3.4 deep-scrub starts 2024-03-30T13:45:46.814 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:46 smithi043 ceph-mon[29963]: 3.4 deep-scrub ok 2024-03-30T13:45:46.816 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:46 smithi043 ceph-mon[42099]: 4.1 scrub starts 2024-03-30T13:45:46.817 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:46 smithi043 ceph-mon[42099]: 4.1 scrub ok 2024-03-30T13:45:46.817 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:46 smithi043 ceph-mon[42099]: pgmap v99: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-30T13:45:46.817 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:46 smithi043 ceph-mon[42099]: 6.12 scrub starts 2024-03-30T13:45:46.817 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:46 smithi043 ceph-mon[42099]: 6.12 scrub ok 2024-03-30T13:45:46.817 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:46 smithi043 ceph-mon[42099]: 3.4 deep-scrub starts 2024-03-30T13:45:46.817 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:46 smithi043 ceph-mon[42099]: 3.4 deep-scrub ok 2024-03-30T13:45:46.878 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:46.893 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:45:46.904 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:46.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:46.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:46.920 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:45:46.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:46 smithi113 ceph-mon[34341]: 4.1 scrub starts 2024-03-30T13:45:46.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:46 smithi113 ceph-mon[34341]: 4.1 scrub ok 2024-03-30T13:45:46.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:46 smithi113 ceph-mon[34341]: pgmap v99: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-30T13:45:46.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:46 smithi113 ceph-mon[34341]: 6.12 scrub starts 2024-03-30T13:45:46.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:46 smithi113 ceph-mon[34341]: 6.12 scrub ok 2024-03-30T13:45:46.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:46 smithi113 ceph-mon[34341]: 3.4 deep-scrub starts 2024-03-30T13:45:46.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:46 smithi113 ceph-mon[34341]: 3.4 deep-scrub ok 2024-03-30T13:45:47.006 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:47.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:47.039 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:47.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:47.140 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:47.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:47.197 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:47.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:47.298 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:47.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:47.329 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:47.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:47.400 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 47 16 78121 78105 103.842 72.125 0.00998108 0.00962496 2024-03-30T13:45:47.430 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:47.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:47.481 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:47.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:47.582 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:47.626 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:47.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:47.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:47.728 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:47.765 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:47.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:47.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:47.867 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:47.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:47.900 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:47.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:47.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:47 smithi043 ceph-mon[29963]: 4.9 scrub starts 2024-03-30T13:45:47.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:47 smithi043 ceph-mon[29963]: 4.9 scrub ok 2024-03-30T13:45:47.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:47 smithi043 ceph-mon[29963]: 3.1d deep-scrub starts 2024-03-30T13:45:47.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:47 smithi043 ceph-mon[29963]: 3.1d deep-scrub ok 2024-03-30T13:45:47.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:47 smithi043 ceph-mon[29963]: 4.16 scrub starts 2024-03-30T13:45:47.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:47 smithi043 ceph-mon[29963]: 4.16 scrub ok 2024-03-30T13:45:47.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:47 smithi043 ceph-mon[42099]: 4.9 scrub starts 2024-03-30T13:45:47.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:47 smithi043 ceph-mon[42099]: 4.9 scrub ok 2024-03-30T13:45:47.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:47 smithi043 ceph-mon[42099]: 3.1d deep-scrub starts 2024-03-30T13:45:47.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:47 smithi043 ceph-mon[42099]: 3.1d deep-scrub ok 2024-03-30T13:45:47.902 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:47 smithi043 ceph-mon[42099]: 4.16 scrub starts 2024-03-30T13:45:47.902 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:47 smithi043 ceph-mon[42099]: 4.16 scrub ok 2024-03-30T13:45:47.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:47 smithi113 ceph-mon[34341]: 4.9 scrub starts 2024-03-30T13:45:47.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:47 smithi113 ceph-mon[34341]: 4.9 scrub ok 2024-03-30T13:45:47.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:47 smithi113 ceph-mon[34341]: 3.1d deep-scrub starts 2024-03-30T13:45:47.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:47 smithi113 ceph-mon[34341]: 3.1d deep-scrub ok 2024-03-30T13:45:47.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:47 smithi113 ceph-mon[34341]: 4.16 scrub starts 2024-03-30T13:45:47.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:47 smithi113 ceph-mon[34341]: 4.16 scrub ok 2024-03-30T13:45:48.001 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:48.027 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:48.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:48.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:48.129 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:48.159 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:48.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:48.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:48.261 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:48.289 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:48.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:48.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:48.391 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:48.400 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 48 16 79340 79324 103.266 76.1875 0.0101853 0.00967885 2024-03-30T13:45:48.424 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:48.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:48.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:48.482 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:45:48.525 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:48.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:48.576 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:48.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:48.677 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:48.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:48 smithi113 ceph-mon[34341]: 6.2 scrub starts 2024-03-30T13:45:48.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:48 smithi113 ceph-mon[34341]: 6.2 scrub ok 2024-03-30T13:45:48.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:48 smithi113 ceph-mon[34341]: pgmap v100: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-30T13:45:48.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:45:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:45:48.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:48.742 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:48.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:48.841 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:48 smithi043 ceph-mon[29963]: 6.2 scrub starts 2024-03-30T13:45:48.841 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:48 smithi043 ceph-mon[29963]: 6.2 scrub ok 2024-03-30T13:45:48.841 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:48 smithi043 ceph-mon[29963]: pgmap v100: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-30T13:45:48.842 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:48 smithi043 ceph-mon[42099]: 6.2 scrub starts 2024-03-30T13:45:48.842 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:48 smithi043 ceph-mon[42099]: 6.2 scrub ok 2024-03-30T13:45:48.842 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:48 smithi043 ceph-mon[42099]: pgmap v100: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-30T13:45:48.843 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:48.875 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:48.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:48.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:48.977 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:49.004 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:49.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:49.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:49.106 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:49.137 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:49.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:49.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:49.239 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:49.271 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:49.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:49.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:49.373 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:49.400 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 49 16 80476 80460 102.607 71 0.0179216 0.00974106 2024-03-30T13:45:49.403 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:49.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:49.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:49.504 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:49.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:49.556 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:49.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:49.657 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:49.711 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:49.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:49.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:49.813 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:49.834 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:49 smithi043 ceph-mon[29963]: 4.1b scrub starts 2024-03-30T13:45:49.834 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:49 smithi043 ceph-mon[29963]: 4.1b scrub ok 2024-03-30T13:45:49.834 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:49 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:49.834 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:49 smithi043 ceph-mon[29963]: 4.a deep-scrub starts 2024-03-30T13:45:49.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:49 smithi043 ceph-mon[42099]: 4.1b scrub starts 2024-03-30T13:45:49.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:49 smithi043 ceph-mon[42099]: 4.1b scrub ok 2024-03-30T13:45:49.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:49 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:49.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:49 smithi043 ceph-mon[42099]: 4.a deep-scrub starts 2024-03-30T13:45:49.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:49.872 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:49.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:49.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:49 smithi113 ceph-mon[34341]: 4.1b scrub starts 2024-03-30T13:45:49.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:49 smithi113 ceph-mon[34341]: 4.1b scrub ok 2024-03-30T13:45:49.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:49 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:49.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:49 smithi113 ceph-mon[34341]: 4.a deep-scrub starts 2024-03-30T13:45:49.974 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:50.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:50.015 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:50.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:50.117 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:50.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:50.179 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:50.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:50.281 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:50.328 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:50.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:50.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:50.401 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 50 16 81573 81557 101.926 68.5625 0.0145161 0.00980581 2024-03-30T13:45:50.429 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:50.472 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:50.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:50.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:50.574 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:50.611 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:50.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:50.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:50.713 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:50.752 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:50.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:50.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:50.854 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:50.888 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:50.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:50.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:50.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:50 smithi113 ceph-mon[34341]: 4.a deep-scrub ok 2024-03-30T13:45:50.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:50 smithi113 ceph-mon[34341]: pgmap v101: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-30T13:45:50.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:50 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:45:50.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:50 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:45:50.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:50 smithi113 ceph-mon[34341]: osdmap e109: 8 total, 8 up, 8 in 2024-03-30T13:45:50.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:50 smithi113 ceph-mon[34341]: 6.1c scrub starts 2024-03-30T13:45:50.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:50 smithi113 ceph-mon[34341]: 6.1c scrub ok 2024-03-30T13:45:50.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:50 smithi113 ceph-mon[34341]: 5.c scrub starts 2024-03-30T13:45:50.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:50 smithi113 ceph-mon[34341]: 5.c scrub ok 2024-03-30T13:45:50.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:50 smithi113 ceph-mon[34341]: 4.6 scrub starts 2024-03-30T13:45:50.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:50 smithi113 ceph-mon[34341]: 4.6 scrub ok 2024-03-30T13:45:50.989 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:51.024 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:51.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:51.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:51.126 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:51.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[42099]: 4.a deep-scrub ok 2024-03-30T13:45:51.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[42099]: pgmap v101: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-30T13:45:51.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:45:51.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:45:51.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[42099]: osdmap e109: 8 total, 8 up, 8 in 2024-03-30T13:45:51.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[42099]: 6.1c scrub starts 2024-03-30T13:45:51.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[42099]: 6.1c scrub ok 2024-03-30T13:45:51.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[42099]: 5.c scrub starts 2024-03-30T13:45:51.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[42099]: 5.c scrub ok 2024-03-30T13:45:51.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[42099]: 4.6 scrub starts 2024-03-30T13:45:51.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[42099]: 4.6 scrub ok 2024-03-30T13:45:51.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[29963]: 4.a deep-scrub ok 2024-03-30T13:45:51.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[29963]: pgmap v101: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-30T13:45:51.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:45:51.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:45:51.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[29963]: osdmap e109: 8 total, 8 up, 8 in 2024-03-30T13:45:51.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[29963]: 6.1c scrub starts 2024-03-30T13:45:51.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[29963]: 6.1c scrub ok 2024-03-30T13:45:51.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[29963]: 5.c scrub starts 2024-03-30T13:45:51.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[29963]: 5.c scrub ok 2024-03-30T13:45:51.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[29963]: 4.6 scrub starts 2024-03-30T13:45:51.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:50 smithi043 ceph-mon[29963]: 4.6 scrub ok 2024-03-30T13:45:51.169 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:51.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:51.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:51.259 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:45:51.260 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:45:51.271 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:51.318 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:51.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:51.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:51.401 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 51 16 82721 82705 101.335 71.75 0.010088 0.00986319 2024-03-30T13:45:51.420 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:51.452 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:51.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:51.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:51.553 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:51.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:51.603 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:51.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:51.704 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:51.739 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:51.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:51.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:51.841 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:51.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:51.875 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:51.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:51.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:51 smithi043 ceph-mon[29963]: 5.e scrub starts 2024-03-30T13:45:51.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:51 smithi043 ceph-mon[29963]: 5.e scrub ok 2024-03-30T13:45:51.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:51 smithi043 ceph-mon[29963]: 4.2 deep-scrub starts 2024-03-30T13:45:51.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:51 smithi043 ceph-mon[29963]: 4.2 deep-scrub ok 2024-03-30T13:45:51.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:51 smithi043 ceph-mon[29963]: 3.1 scrub starts 2024-03-30T13:45:51.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:51 smithi043 ceph-mon[29963]: 3.1 scrub ok 2024-03-30T13:45:51.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:51 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:51.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:51 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:51.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:51 smithi043 ceph-mon[42099]: 5.e scrub starts 2024-03-30T13:45:51.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:51 smithi043 ceph-mon[42099]: 5.e scrub ok 2024-03-30T13:45:51.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:51 smithi043 ceph-mon[42099]: 4.2 deep-scrub starts 2024-03-30T13:45:51.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:51 smithi043 ceph-mon[42099]: 4.2 deep-scrub ok 2024-03-30T13:45:51.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:51 smithi043 ceph-mon[42099]: 3.1 scrub starts 2024-03-30T13:45:51.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:51 smithi043 ceph-mon[42099]: 3.1 scrub ok 2024-03-30T13:45:51.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:51 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:51.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:51 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:51.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:51 smithi113 ceph-mon[34341]: 5.e scrub starts 2024-03-30T13:45:51.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:51 smithi113 ceph-mon[34341]: 5.e scrub ok 2024-03-30T13:45:51.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:51 smithi113 ceph-mon[34341]: 4.2 deep-scrub starts 2024-03-30T13:45:51.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:51 smithi113 ceph-mon[34341]: 4.2 deep-scrub ok 2024-03-30T13:45:51.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:51 smithi113 ceph-mon[34341]: 3.1 scrub starts 2024-03-30T13:45:51.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:51 smithi113 ceph-mon[34341]: 3.1 scrub ok 2024-03-30T13:45:51.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:51 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:51.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:51 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:51.976 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:52.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:52.016 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:52.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:52.073 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-30T13:45:52.117 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:52.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:52.182 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:52.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:52.283 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:52.311 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:52.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:52.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:52.401 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 52 16 83626 83610 100.474 56.5625 0.0140259 0.00994672 2024-03-30T13:45:52.413 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:52.442 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:52.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:52.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:52.544 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:52.575 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:52.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:52.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:52.677 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:52.715 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:52.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:52.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:52.816 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:52.851 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:52.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:52.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:52.926 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:45:52.952 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:52 smithi113 ceph-mon[34341]: 5.1c scrub starts 2024-03-30T13:45:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:52 smithi113 ceph-mon[34341]: 5.1c scrub ok 2024-03-30T13:45:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:52 smithi113 ceph-mon[34341]: 7.7 scrub starts 2024-03-30T13:45:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:52 smithi113 ceph-mon[34341]: pgmap v103: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-30T13:45:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:52 smithi113 ceph-mon[34341]: 6.1f scrub starts 2024-03-30T13:45:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:52 smithi113 ceph-mon[34341]: 6.1f scrub ok 2024-03-30T13:45:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:52 smithi113 ceph-mon[34341]: 6.15 scrub starts 2024-03-30T13:45:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:52 smithi113 ceph-mon[34341]: 6.15 scrub ok 2024-03-30T13:45:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:52 smithi113 ceph-mon[34341]: 4.1e scrub starts 2024-03-30T13:45:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:52 smithi113 ceph-mon[34341]: 4.1e scrub ok 2024-03-30T13:45:52.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:52 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:45:52.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:52 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:45:52.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:52 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:53.014 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:53.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:53.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:53.067 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[42099]: 5.1c scrub starts 2024-03-30T13:45:53.067 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[42099]: 5.1c scrub ok 2024-03-30T13:45:53.067 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[42099]: 7.7 scrub starts 2024-03-30T13:45:53.067 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[42099]: pgmap v103: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-30T13:45:53.067 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[42099]: 6.1f scrub starts 2024-03-30T13:45:53.067 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[42099]: 6.1f scrub ok 2024-03-30T13:45:53.067 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[42099]: 6.15 scrub starts 2024-03-30T13:45:53.067 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[42099]: 6.15 scrub ok 2024-03-30T13:45:53.067 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[42099]: 4.1e scrub starts 2024-03-30T13:45:53.068 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[42099]: 4.1e scrub ok 2024-03-30T13:45:53.068 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:45:53.068 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:45:53.068 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:53.068 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[29963]: 5.1c scrub starts 2024-03-30T13:45:53.068 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[29963]: 5.1c scrub ok 2024-03-30T13:45:53.068 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[29963]: 7.7 scrub starts 2024-03-30T13:45:53.068 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[29963]: pgmap v103: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-30T13:45:53.068 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[29963]: 6.1f scrub starts 2024-03-30T13:45:53.068 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[29963]: 6.1f scrub ok 2024-03-30T13:45:53.068 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[29963]: 6.15 scrub starts 2024-03-30T13:45:53.069 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[29963]: 6.15 scrub ok 2024-03-30T13:45:53.069 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[29963]: 4.1e scrub starts 2024-03-30T13:45:53.069 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[29963]: 4.1e scrub ok 2024-03-30T13:45:53.069 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:45:53.069 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:45:53.069 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:52 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:53.116 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:53.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:53.147 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:53.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:53.248 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:53.275 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:53.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:53.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:53.377 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:53.401 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 53 16 84660 84644 99.7972 64.625 0.0106927 0.0100149 2024-03-30T13:45:53.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:53.419 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:53.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:53.520 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:53.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:53.549 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:53.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:53.650 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:53.678 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:53.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:53.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:53.681 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-03-30T13:45:53.779 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:53.824 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:53.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:53.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:53.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[29963]: 5.f deep-scrub starts 2024-03-30T13:45:53.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[29963]: 5.f deep-scrub ok 2024-03-30T13:45:53.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[29963]: 6.8 scrub starts 2024-03-30T13:45:53.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[29963]: 6.8 scrub ok 2024-03-30T13:45:53.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/36701985' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:45:53.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:45:53.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[29963]: 7.7 scrub ok 2024-03-30T13:45:53.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[29963]: 6.1d scrub starts 2024-03-30T13:45:53.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[29963]: 6.1d scrub ok 2024-03-30T13:45:53.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[29963]: 5.3 scrub starts 2024-03-30T13:45:53.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[29963]: 5.3 scrub ok 2024-03-30T13:45:53.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[42099]: 5.f deep-scrub starts 2024-03-30T13:45:53.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[42099]: 5.f deep-scrub ok 2024-03-30T13:45:53.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[42099]: 6.8 scrub starts 2024-03-30T13:45:53.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[42099]: 6.8 scrub ok 2024-03-30T13:45:53.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/36701985' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:45:53.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:45:53.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[42099]: 7.7 scrub ok 2024-03-30T13:45:53.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[42099]: 6.1d scrub starts 2024-03-30T13:45:53.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[42099]: 6.1d scrub ok 2024-03-30T13:45:53.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[42099]: 5.3 scrub starts 2024-03-30T13:45:53.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:53 smithi043 ceph-mon[42099]: 5.3 scrub ok 2024-03-30T13:45:53.925 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:53.953 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:53.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:53.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:53.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:53 smithi113 ceph-mon[34341]: 5.f deep-scrub starts 2024-03-30T13:45:53.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:53 smithi113 ceph-mon[34341]: 5.f deep-scrub ok 2024-03-30T13:45:53.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:53 smithi113 ceph-mon[34341]: 6.8 scrub starts 2024-03-30T13:45:53.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:53 smithi113 ceph-mon[34341]: 6.8 scrub ok 2024-03-30T13:45:53.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:53 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/36701985' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:45:53.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:53 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:45:53.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:53 smithi113 ceph-mon[34341]: 7.7 scrub ok 2024-03-30T13:45:53.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:53 smithi113 ceph-mon[34341]: 6.1d scrub starts 2024-03-30T13:45:53.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:53 smithi113 ceph-mon[34341]: 6.1d scrub ok 2024-03-30T13:45:53.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:53 smithi113 ceph-mon[34341]: 5.3 scrub starts 2024-03-30T13:45:53.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:53 smithi113 ceph-mon[34341]: 5.3 scrub ok 2024-03-30T13:45:54.055 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:54.094 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:54.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:54.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:54.195 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:54.223 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:54.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:54.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:54.325 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:54.357 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:54.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:54.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:54.401 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 54 16 85771 85755 99.2349 69.4375 0.0141032 0.0100717 2024-03-30T13:45:54.460 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:54.491 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:54.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:54.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:54.593 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:54.623 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:54.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:54.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:54.725 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:54.752 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:54.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:54.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:54.853 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:54.880 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:54.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:54.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:54.943 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:54 smithi043 ceph-mon[29963]: pgmap v104: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-30T13:45:54.944 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:54 smithi043 ceph-mon[29963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:45:54.944 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:54 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:45:54.944 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:54 smithi043 ceph-mon[29963]: osdmap e110: 8 total, 8 up, 8 in 2024-03-30T13:45:54.944 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:54 smithi043 ceph-mon[42099]: pgmap v104: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-30T13:45:54.944 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:54 smithi043 ceph-mon[42099]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:45:54.944 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:54 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:45:54.944 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:54 smithi043 ceph-mon[42099]: osdmap e110: 8 total, 8 up, 8 in 2024-03-30T13:45:54.956 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:45:54.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:54 smithi113 ceph-mon[34341]: pgmap v104: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-30T13:45:54.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:54 smithi113 ceph-mon[34341]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:45:54.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:54 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:45:54.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:54 smithi113 ceph-mon[34341]: osdmap e110: 8 total, 8 up, 8 in 2024-03-30T13:45:54.983 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:55.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:55.011 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:55.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:55.112 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:55.142 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:55.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:55.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:55.243 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:55.284 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:55.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:55.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:55.386 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:55.401 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 55 16 86886 86870 98.6976 69.6875 0.00606806 0.0101256 2024-03-30T13:45:55.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:55.418 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:55.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:55.519 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:55.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:55.549 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:55.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:55.650 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:55.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:55.676 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:55.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:55.777 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:55.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:55.820 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:55.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:55.921 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:55.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:55.951 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:55.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:56.052 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:56.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:56.090 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:56.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:56.191 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:56.219 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:56.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:56.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:56.321 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:56.362 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:56.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:56.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:56.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:45:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:45:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:45:56.401 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 56 16 87998 87982 98.1762 69.5 0.00798802 0.0101811 2024-03-30T13:45:56.465 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:56.495 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:56.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:56.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:56.596 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:56.627 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:56.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:56.729 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:56.757 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:56.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:56.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:56.858 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:56.896 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:56.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:56.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:56.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:56 smithi043 ceph-mon[29963]: pgmap v106: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-30T13:45:56.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:56 smithi043 ceph-mon[42099]: pgmap v106: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-30T13:45:56.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:56 smithi113 ceph-mon[34341]: pgmap v106: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-30T13:45:56.999 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:57.029 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:57.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:57.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:57.131 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:57.161 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:57.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:57.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:57.263 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:57.290 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:57.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:57.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:57.392 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:57.401 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 57 16 89061 89045 97.6193 66.4375 0.0167928 0.0102384 2024-03-30T13:45:57.422 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:57.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:57.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:57.524 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:57.554 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:57.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:57.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:57.656 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:57.683 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:57.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:57.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:57.785 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:57.828 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:57.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:57.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:57.930 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:57.974 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:57.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:57.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:58.019 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:45:58.067 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:45:58.077 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:58.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:58.105 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:58.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:58.108 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:45:58.156 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:45:58.196 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:45:58.206 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:45:58.234 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:58.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:58.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:45:58.238 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:45:58.289 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:45:58.333 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:45:58.336 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:58.340 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:45:58.340 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:45:58.340 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:45:58.340 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:45:58.340 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:45:58.340 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:45:58.340 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:45:58.340 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:45:58.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:58.367 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:58.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:58.401 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 58 16 90105 90089 97.0612 65.25 0.00430145 0.0102973 2024-03-30T13:45:58.468 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:45:58.499 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:58.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:45:58.601 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:58.632 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:58.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:58.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:58.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:58 smithi043 ceph-mon[29963]: pgmap v107: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 668 op/s 2024-03-30T13:45:58.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:58 smithi043 ceph-mon[29963]: osdmap e111: 8 total, 8 up, 8 in 2024-03-30T13:45:58.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:58 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:58.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:58 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:45:58.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:58 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2376377394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:45:58.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:58 smithi043 ceph-mon[42099]: pgmap v107: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 668 op/s 2024-03-30T13:45:58.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:58 smithi043 ceph-mon[42099]: osdmap e111: 8 total, 8 up, 8 in 2024-03-30T13:45:58.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:58 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:58.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:58 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:45:58.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:58 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2376377394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:45:58.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:58 smithi113 ceph-mon[34341]: pgmap v107: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 668 op/s 2024-03-30T13:45:58.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:58 smithi113 ceph-mon[34341]: osdmap e111: 8 total, 8 up, 8 in 2024-03-30T13:45:58.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:58 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:45:58.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:58 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:45:58.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:58 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2376377394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:45:58.723 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:45:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:45:58.734 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:45:58.778 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:58.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:58.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:45:58.880 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:45:58.911 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:58.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:58.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:45:59.013 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:45:59.054 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:59.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:59.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:45:59.156 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:59.184 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:59.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:59.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:59.288 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:45:59.317 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:45:59.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:59.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:45:59.401 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 59 16 91186 91170 96.5611 67.5625 0.008049 0.0103514 2024-03-30T13:45:59.419 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:59.467 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:59.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:59.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:59.568 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:59.610 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:59.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:59.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:45:59.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:45:59 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:59.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:45:59 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:59.706 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T13:45:59.712 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:45:59.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:45:59 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:45:59.798 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:59.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:59.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:45:59.899 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:45:59.958 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:45:59.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:45:59.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:00.060 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:00.090 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:00.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:00.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:00.192 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:00.221 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:00.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:00.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:00.323 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:00.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:00.364 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:00.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:00.402 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-03-30T13:46:00.401304+0000 min lat: 0.00133902 max lat: 0.124905 avg lat: 0.0104033 2024-03-30T13:46:00.402 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T13:46:00.402 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 60 16 92262 92246 96.0726 67.25 0.0125547 0.0104033 2024-03-30T13:46:00.465 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:00.511 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:00.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:00.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:00.613 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:00.645 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:00.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:00.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:00.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:00 smithi043 ceph-mon[29963]: pgmap v109: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-30T13:46:00.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:00 smithi043 ceph-mon[42099]: pgmap v109: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-30T13:46:00.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:00 smithi113 ceph-mon[34341]: pgmap v109: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-30T13:46:00.746 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:00.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:00.796 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:00.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:00.897 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:00.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:00.946 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:00.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:01.047 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:01.087 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:01.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:01.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:01.189 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:01.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:01.233 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:01.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:01.333 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:01.360 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:01.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:01.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:01.373 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 reset_purged_snaps_last 2024-03-30T13:46:01.402 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 61 16 93223 93207 95.4811 60.0625 0.0103158 0.0104678 2024-03-30T13:46:01.462 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:01.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:01.507 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:01.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:01.608 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:01.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:01.659 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:01.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:01.760 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:01.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:01.798 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:01.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:01.899 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:01.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:01.989 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:01.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:02.008 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:46:02.023 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:46:02.023 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-03-30T13:46:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:93704 7.e 7:75bdc705:::benchmark_data_smithi043_86989_object93703:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-30T13:46:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:46:01.996048+0000", 2024-03-30T13:46:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.024836852, 2024-03-30T13:46:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.024886243999999998, 2024-03-30T13:46:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:46:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:46:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-03-30T13:46:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:46:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:46:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:46:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 93704 2024-03-30T13:46:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:46:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:46:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:01.996048+0000", 2024-03-30T13:46:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:46:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:46:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:01.996048+0000", 2024-03-30T13:46:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:46:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:46:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:01.996049+0000", 2024-03-30T13:46:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.669e-06 2024-03-30T13:46:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:46:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:01.996083+0000", 2024-03-30T13:46:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3837000000000001e-05 2024-03-30T13:46:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:46:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:01.996125+0000", 2024-03-30T13:46:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1755999999999998e-05 2024-03-30T13:46:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:46:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:01.996139+0000", 2024-03-30T13:46:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4195999999999999e-05 2024-03-30T13:46:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:46:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:01.996208+0000", 2024-03-30T13:46:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9091000000000006e-05 2024-03-30T13:46:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:46:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:01.996364+0000", 2024-03-30T13:46:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000155845 2024-03-30T13:46:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,5", 2024-03-30T13:46:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:01.996500+0000", 2024-03-30T13:46:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013616400000000001 2024-03-30T13:46:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:46:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:02.000888+0000", 2024-03-30T13:46:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0043876790000000002 2024-03-30T13:46:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:46:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:02.002955+0000", 2024-03-30T13:46:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0020667540000000001 2024-03-30T13:46:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:46:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:46:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:46:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:93707 7.8 7:1885bb63:::benchmark_data_smithi043_86989_object93706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-30T13:46:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:46:02.013979+0000", 2024-03-30T13:46:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0069058269999999998, 2024-03-30T13:46:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0070639049999999997, 2024-03-30T13:46:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:46:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:46:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-03-30T13:46:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:46:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:46:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:46:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 93707 2024-03-30T13:46:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:46:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:46:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:02.013979+0000", 2024-03-30T13:46:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:46:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:46:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:02.013979+0000", 2024-03-30T13:46:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:46:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:46:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:02.013980+0000", 2024-03-30T13:46:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.674e-06 2024-03-30T13:46:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:46:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:02.014009+0000", 2024-03-30T13:46:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9051e-05 2024-03-30T13:46:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:46:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:02.014027+0000", 2024-03-30T13:46:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7470999999999999e-05 2024-03-30T13:46:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:46:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:02.014038+0000", 2024-03-30T13:46:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0995000000000001e-05 2024-03-30T13:46:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:46:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:02.014114+0000", 2024-03-30T13:46:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6091e-05 2024-03-30T13:46:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:46:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:02.014267+0000", 2024-03-30T13:46:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000153151 2024-03-30T13:46:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,7", 2024-03-30T13:46:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:02.015396+0000", 2024-03-30T13:46:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0011293830000000001 2024-03-30T13:46:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:46:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:46:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:46:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:46:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:46:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 2 2024-03-30T13:46:02.034 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:46:02.043 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:46:02.090 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:02.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:02.141 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:02.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:02.242 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:02.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:02.291 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:02.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:02.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:02.404 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 62 16 94056 94040 94.7795 52.0625 0.0113512 0.0105463 2024-03-30T13:46:02.435 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:02.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:02.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:02.537 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:02.570 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:02.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:02.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:02.623 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:02 smithi113 ceph-mon[34341]: pgmap v110: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-30T13:46:02.623 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:02 smithi113 ceph-mon[34341]: osdmap e112: 8 total, 8 up, 8 in 2024-03-30T13:46:02.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:02 smithi043 ceph-mon[29963]: pgmap v110: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-30T13:46:02.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:02 smithi043 ceph-mon[29963]: osdmap e112: 8 total, 8 up, 8 in 2024-03-30T13:46:02.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:02 smithi043 ceph-mon[42099]: pgmap v110: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-30T13:46:02.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:02 smithi043 ceph-mon[42099]: osdmap e112: 8 total, 8 up, 8 in 2024-03-30T13:46:02.673 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:02.713 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:02.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:02.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:02.801 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-03-30T13:46:02.816 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:02.849 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:02.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:02.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:02.951 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:03.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:03.008 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:03.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:03.052 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:46:03.109 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:03.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:03.210 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:03.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:03.311 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:03.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:03.389 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:03.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:03.408 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 63 16 95069 95053 94.278 63.3125 0.0337334 0.0106022 2024-03-30T13:46:03.490 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:03.537 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:03.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:03.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:03.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:03 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:46:03.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:03 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:46:03.593 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:03 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:46:03.593 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:03 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:46:03.639 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:03.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:03.681 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:03.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:03.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:03 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:46:03.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:03 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:46:03.781 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:03.825 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:03.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:03.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:03.926 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:03.967 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:03.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:03.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:04.069 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:04.099 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:04.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:04.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:04.201 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:04.247 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:04.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:04.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:04.349 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:04.389 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:04.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:04.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:04.405 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 64 16 96173 96157 93.883 69 0.0157016 0.0106469 2024-03-30T13:46:04.492 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:04.538 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:04.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:04.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:04.575 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:04 smithi043 ceph-mon[29963]: pgmap v112: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 958 op/s 2024-03-30T13:46:04.575 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:04 smithi043 ceph-mon[42099]: pgmap v112: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 958 op/s 2024-03-30T13:46:04.640 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:04.678 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:04.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:04.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:04.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:04 smithi113 ceph-mon[34341]: pgmap v112: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 958 op/s 2024-03-30T13:46:04.780 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:04.808 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:04.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:04.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:04.910 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:04.948 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:04.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:04.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:05.050 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:05.082 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:05.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:05.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:05.184 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:05.212 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:05.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:05.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:05.314 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:05.346 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:05.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:05.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:05.405 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 65 15 97297 97282 93.5203 70.3125 0.0141922 0.0106876 2024-03-30T13:46:05.448 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:05.479 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:05.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:05.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:05.581 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:05.633 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:05.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:05.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:05.734 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:05.766 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:05.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:05.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:05.800 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:46:05.868 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:05.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:05.929 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:05.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:06.030 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:06.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:06.064 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:06.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:06.165 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:06.202 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:06.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:06.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:06.304 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:06.345 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:06.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:06.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:06.370 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:46:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:46:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:46:06.405 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 66 16 98247 98231 93.0019 59.3125 0.00510898 0.0107461 2024-03-30T13:46:06.448 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:06.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:06.494 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:06.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:06.595 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:06.618 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:06.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:06.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:06.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:06 smithi043 ceph-mon[29963]: pgmap v113: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-30T13:46:06.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:06 smithi043 ceph-mon[42099]: pgmap v113: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-30T13:46:06.720 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:06.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:06 smithi113 ceph-mon[34341]: pgmap v113: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-30T13:46:06.749 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:06.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:06.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:06.851 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:06.884 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:06.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:06.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:06.986 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:07.016 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:07.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:07.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:07.118 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:07.164 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:07.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:07.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:07.265 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:07.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:07.307 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:07.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:07.405 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 67 15 99261 99246 92.5606 63.4375 0.03085 0.0107988 2024-03-30T13:46:07.409 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:07.455 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:07.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:07.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:07.557 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:07.598 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:07.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:07.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:07.700 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:07.746 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:07.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:07.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:07.848 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:07.896 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:07.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:07.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:08.000 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:08.029 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:08.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:08.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:08.131 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:08.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:08.175 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:08.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:08.269 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:46:08.276 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:08.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:08.346 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:08.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:08.406 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 68 16 100277 100261 92.1318 63.4375 0.00947585 0.0108493 2024-03-30T13:46:08.446 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:08.475 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:08.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:08.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:08.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:08.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:08.615 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:08.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:08.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:08 smithi043 ceph-mon[29963]: pgmap v114: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-30T13:46:08.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:08 smithi043 ceph-mon[29963]: osdmap e113: 8 total, 8 up, 8 in 2024-03-30T13:46:08.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:08 smithi043 ceph-mon[42099]: pgmap v114: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-30T13:46:08.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:08 smithi043 ceph-mon[42099]: osdmap e113: 8 total, 8 up, 8 in 2024-03-30T13:46:08.716 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:08.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:08 smithi113 ceph-mon[34341]: pgmap v114: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-30T13:46:08.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:08 smithi113 ceph-mon[34341]: osdmap e113: 8 total, 8 up, 8 in 2024-03-30T13:46:08.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:46:08 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:46:08.739 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 reset_purged_snaps_last 2024-03-30T13:46:08.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:08.775 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:08.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:08.875 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:08.906 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:08.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:08.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:09.008 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:09.036 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:09.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:09.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:09.138 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:09.194 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:09.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:09.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:09.296 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:09.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:09.326 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:09.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:09.406 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 69 16 101259 101243 91.686 61.375 0.017561 0.0109019 2024-03-30T13:46:09.427 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:09.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:09.478 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:09.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:09.489 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:09 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:46:09.489 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:09 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:46:09.580 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:09.606 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:09.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:09.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:09.708 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:09.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:09 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:46:09.737 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:09.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:09.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:09.839 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:09.871 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:09.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:09.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:09.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:10.017 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:10.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:10.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:10.118 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:10.162 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:10.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:10.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:10.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:10.306 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:10.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:10.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:10.385 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-03-30T13:46:10.406 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 70 16 102293 102277 91.2994 64.625 0.00947466 0.0109482 2024-03-30T13:46:10.409 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:10.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:10.467 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:10.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:10.568 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:10.597 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:10.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:10.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:10.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:10 smithi043 ceph-mon[29963]: pgmap v116: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-30T13:46:10.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:10 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1942699845' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:46:10.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:10 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:46:10.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:10 smithi043 ceph-mon[42099]: pgmap v116: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-30T13:46:10.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:10 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1942699845' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:46:10.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:10 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:46:10.699 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:10.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:10 smithi113 ceph-mon[34341]: pgmap v116: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-30T13:46:10.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:10 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1942699845' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:46:10.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:10 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:46:10.728 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:10.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:10.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:10.830 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:10.871 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:10.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:10.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:10.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:11.018 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:11.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:11.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:11.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:11.163 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:11.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:11.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:11.265 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:11.305 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:11.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:11.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:11.406 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 71 16 103386 103370 90.9756 68.3125 0.0164001 0.0109869 2024-03-30T13:46:11.407 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:11.435 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:11.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:11.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:11.537 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:11.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:11.583 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:11.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:11.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:11 smithi043 ceph-mon[29963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:46:11.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:11 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:46:11.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:11 smithi043 ceph-mon[29963]: osdmap e114: 8 total, 8 up, 8 in 2024-03-30T13:46:11.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:11 smithi043 ceph-mon[42099]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:46:11.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:11 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:46:11.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:11 smithi043 ceph-mon[42099]: osdmap e114: 8 total, 8 up, 8 in 2024-03-30T13:46:11.684 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:11.714 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:11.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:11.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:11.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:11 smithi113 ceph-mon[34341]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:46:11.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:11 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:46:11.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:11 smithi113 ceph-mon[34341]: osdmap e114: 8 total, 8 up, 8 in 2024-03-30T13:46:11.816 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:11.883 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:11.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:11.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:11.985 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:12.028 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:12.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:12.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:12.130 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:12.179 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:12.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:12.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:12.281 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:12.333 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:12.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:12.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:12.406 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 72 16 104420 104404 90.6095 64.625 0.0138889 0.0110306 2024-03-30T13:46:12.435 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:12.462 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:12.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:12.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:12.563 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:12.591 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:12.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:12.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:12.632 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:12 smithi043 ceph-mon[29963]: pgmap v118: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 997 op/s 2024-03-30T13:46:12.633 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:12 smithi043 ceph-mon[29963]: osdmap e115: 8 total, 8 up, 8 in 2024-03-30T13:46:12.633 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:12 smithi043 ceph-mon[42099]: pgmap v118: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 997 op/s 2024-03-30T13:46:12.633 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:12 smithi043 ceph-mon[42099]: osdmap e115: 8 total, 8 up, 8 in 2024-03-30T13:46:12.693 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:12.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:12 smithi113 ceph-mon[34341]: pgmap v118: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 997 op/s 2024-03-30T13:46:12.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:12 smithi113 ceph-mon[34341]: osdmap e115: 8 total, 8 up, 8 in 2024-03-30T13:46:12.734 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:12.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:12.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:12.836 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:12.884 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:12.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:12.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:12.943 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:46:12.963 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:46:12.963 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T13:46:12.963 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-03-30T13:46:12.963 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-03-30T13:46:12.963 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:46:12.986 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:12.987 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:46:13.011 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:13.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:13.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:13.026 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:46:13.066 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:46:13.107 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:46:13.113 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:13.147 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:46:13.155 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:13.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:13.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:13.190 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:46:13.237 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:46:13.240 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:46:13.240 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:46:13.240 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:46:13.240 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:46:13.240 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:46:13.240 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:46:13.240 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:46:13.241 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:46:13.258 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:13.288 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:13.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:13.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:13.390 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:13.406 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 73 16 105375 105359 90.1852 59.6875 0.0143056 0.0110824 2024-03-30T13:46:13.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:13.435 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:13.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:13.537 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:13.566 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:13.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:13.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:13.636 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:13 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:46:13.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:13 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/892186706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:46:13.637 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:13 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:46:13.637 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:13 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/892186706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:46:13.667 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:13.698 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:13.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:13.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:13.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:13 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:46:13.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:13 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/892186706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:46:13.800 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:13.843 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:13.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:13.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:13.944 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:13.980 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:13.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:13.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:14.082 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:14.123 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:14.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:14.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:14.225 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:14.270 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:14.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:14.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:14.323 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:46:14.367 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-03-30T13:46:14.375 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:14.409 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 74 16 106471 106455 89.8892 68.5 0.00605015 0.0111191 2024-03-30T13:46:14.456 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:14.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:14.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:14.557 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:14.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:14.609 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:14.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:14.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:14 smithi043 ceph-mon[29963]: pgmap v120: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-30T13:46:14.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:14 smithi043 ceph-mon[42099]: pgmap v120: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-30T13:46:14.712 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:14.724 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:14 smithi113 ceph-mon[34341]: pgmap v120: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-30T13:46:14.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:14.746 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:14.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:14.847 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:14.911 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:14.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:14.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:15.013 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:15.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:15.045 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:15.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:15.146 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:15.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:15.194 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:15.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:15.295 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:15.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:15.333 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:15.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:15.409 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 75 16 107475 107459 89.5273 62.75 0.00962639 0.0111647 2024-03-30T13:46:15.435 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:15.463 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:15.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:15.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:15.565 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:15.592 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:15.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:15.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:15.687 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T13:46:15.693 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:15.730 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:46:15.799 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:15.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:15.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:15.901 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:15.932 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:15.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:15.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:16.034 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:16.079 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:16.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:16.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:16.181 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:16.261 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:16.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:16.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:16.363 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:16.368 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:46:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:46:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:46:16.410 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 76 16 108491 108475 89.1849 63.5 0.0250271 0.011207 2024-03-30T13:46:16.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:16.417 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:16.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:16.518 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:16.546 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:16.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:16.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:16.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:16 smithi043 ceph-mon[29963]: pgmap v121: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-30T13:46:16.637 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:16 smithi043 ceph-mon[42099]: pgmap v121: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-30T13:46:16.649 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:16.688 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:16.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:16.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:16.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:16 smithi113 ceph-mon[34341]: pgmap v121: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-30T13:46:16.789 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:16.832 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:16.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:16.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:16.869 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:46:16.934 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:16.978 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:16.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:17.082 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:17.111 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:17.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:17.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:17.213 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:17.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:17.256 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:17.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:17.357 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:17.383 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:17.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:17.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:17.409 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 77 16 109427 109411 88.7863 58.5 0.0104545 0.0112576 2024-03-30T13:46:17.485 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:17.525 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:17.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:17.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:17.627 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:17.650 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:17.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:17.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:17.752 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:17.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:17.795 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:17.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:17.896 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:17.936 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:17.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:17.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:18.038 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:18.065 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:18.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:18.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:18.167 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:18.196 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:18.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:18.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:18.298 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:18.337 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:18.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:18.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:18.409 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 78 16 110253 110237 88.3099 51.625 0.0201503 0.0113188 2024-03-30T13:46:18.439 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:18.467 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:18.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:18.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:18.568 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:18.599 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:18.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:18.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:18.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:18 smithi043 ceph-mon[29963]: pgmap v122: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 40 MiB/s wr, 639 op/s 2024-03-30T13:46:18.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:18 smithi043 ceph-mon[29963]: osdmap e116: 8 total, 8 up, 8 in 2024-03-30T13:46:18.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:18 smithi043 ceph-mon[42099]: pgmap v122: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 40 MiB/s wr, 639 op/s 2024-03-30T13:46:18.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:18 smithi043 ceph-mon[42099]: osdmap e116: 8 total, 8 up, 8 in 2024-03-30T13:46:18.701 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:18.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:18 smithi113 ceph-mon[34341]: pgmap v122: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 40 MiB/s wr, 639 op/s 2024-03-30T13:46:18.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:18 smithi113 ceph-mon[34341]: osdmap e116: 8 total, 8 up, 8 in 2024-03-30T13:46:18.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:46:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:46:18.746 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:18.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:18.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:18.848 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:18.886 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:18.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:18.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:18.989 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:19.021 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:19.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:19.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:19.123 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:19.161 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:19.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:19.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:19.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:19.308 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:19.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:19.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:19.410 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 79 16 111097 111081 87.8598 52.75 0.0159718 0.0113772 2024-03-30T13:46:19.410 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:19.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:19.440 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:19.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:19.541 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:19.570 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:19.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:19.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:19.623 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:19 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:46:19.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:19 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:46:19.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:19 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:46:19.672 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:19.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:19.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:19.817 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:19.855 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:19.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:19.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:19.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:19.996 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:19.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:20.099 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:20.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:20.169 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:20.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:20.271 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:20.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:20.303 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:20.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:20.404 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:20.412 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-03-30T13:46:20.411443+0000 min lat: 0.00133902 max lat: 0.166691 avg lat: 0.0114217 2024-03-30T13:46:20.412 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T13:46:20.412 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 80 16 112061 112045 87.5124 60.25 0.0178977 0.0114217 2024-03-30T13:46:20.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:20.456 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:20.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:20.557 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:20.575 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:20 smithi043 ceph-mon[29963]: pgmap v124: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-30T13:46:20.576 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:20 smithi043 ceph-mon[42099]: pgmap v124: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-30T13:46:20.586 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:20.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:20.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:20.657 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:20 smithi113 ceph-mon[34341]: pgmap v124: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-30T13:46:20.688 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:20.718 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:20.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:20.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:20.820 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:20.853 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:20.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:20.954 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:20.974 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:46:20.974 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:46:20.987 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:20.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:20.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:21.089 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:21.131 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:21.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:21.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:21.145 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:46:21.234 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:21.265 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:21.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:21.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:21.367 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:21.412 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 81 15 113071 113056 87.2121 63.1875 0.0129109 0.0114607 2024-03-30T13:46:21.414 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:21.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:21.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:21.516 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:21.549 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:21.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:21.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:21.651 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:21.684 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:21.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:21.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:21.786 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:21.825 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:21.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:21.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:21.876 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 reset_purged_snaps_last 2024-03-30T13:46:21.926 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:21.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:21.961 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:21.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:22.062 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:22.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:22.113 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:22.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:22.214 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:22.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:22.268 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:22.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:22.370 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:22.409 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:22.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:22.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:22.412 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 82 16 114126 114110 86.9519 65.875 0.0190358 0.0114956 2024-03-30T13:46:22.467 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:22 smithi043 ceph-mon[29963]: pgmap v125: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-30T13:46:22.467 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:22 smithi043 ceph-mon[42099]: pgmap v125: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-30T13:46:22.482 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:46:22.511 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:22.531 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-30T13:46:22.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:22.563 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:22.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:22.665 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:22.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:22.720 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:22.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:22.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:22 smithi113 ceph-mon[34341]: pgmap v125: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-30T13:46:22.820 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:22.849 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:22.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:22.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:22.951 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:22.983 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:22.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:22.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:23.085 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:23.139 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:23.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:23.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:23.241 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:23.270 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:23.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:23.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:23.372 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:23.412 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:23.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:23.413 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 83 16 115135 115119 86.664 63.0625 0.0142871 0.0115342 2024-03-30T13:46:23.512 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:23.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:23.567 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:23.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:23.668 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:23.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:23.712 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:23.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:23.813 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:23.844 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:23.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:23.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:23.947 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:23.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:23.976 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:23.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:24.077 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:24.106 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:24.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:24.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:24.209 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:24.250 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:24.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:24.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:24.352 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:24.413 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 84 16 116182 116166 86.4113 65.4375 0.0252589 0.0115679 2024-03-30T13:46:24.423 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:24.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:24.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:24.525 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:24.546 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:24 smithi043 ceph-mon[29963]: pgmap v126: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-03-30T13:46:24.546 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:24 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:46:24.546 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:24 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:46:24.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:24 smithi043 ceph-mon[42099]: pgmap v126: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-03-30T13:46:24.547 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:24 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:46:24.547 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:24 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:46:24.555 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:24.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:24.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:24.657 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:24.685 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:24.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:24.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:24.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:24 smithi113 ceph-mon[34341]: pgmap v126: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-03-30T13:46:24.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:24 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:46:24.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:24 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:46:24.781 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:46:24.787 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:24.833 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:24.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:24.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:24.937 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:24.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:24.971 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:24.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:25.073 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:25.073 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:46:25.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:25.124 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:25.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:25.225 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:25.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:25.274 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:25.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:25.375 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:25.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:25.403 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:25.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:25.412 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 85 16 117238 117222 86.1711 66 0.0244152 0.0115993 2024-03-30T13:46:25.505 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:25.535 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:25.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:25.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:25.637 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:25.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:25 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:46:25.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:25 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:46:25.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:25 smithi043 ceph-mon[29963]: osdmap e117: 8 total, 8 up, 8 in 2024-03-30T13:46:25.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:25 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:46:25.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:25 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:46:25.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:25 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:46:25.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:25 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:46:25.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:25 smithi043 ceph-mon[42099]: osdmap e117: 8 total, 8 up, 8 in 2024-03-30T13:46:25.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:25 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:46:25.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:25 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:46:25.665 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:25.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:25.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:25.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:25 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:46:25.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:25 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:46:25.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:25 smithi113 ceph-mon[34341]: osdmap e117: 8 total, 8 up, 8 in 2024-03-30T13:46:25.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:25 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:46:25.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:25 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:46:25.767 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:25.796 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:25.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:25.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:25.898 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:25.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:25.947 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:25.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:26.048 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:26.078 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:26.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:26.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:26.180 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:26.209 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:26.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:26.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:26.311 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:26.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:26.341 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:26.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:26.369 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:46:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:46:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:46:26.412 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 86 16 118259 118243 85.9111 63.8125 0.00595396 0.011635 2024-03-30T13:46:26.443 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:26.472 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:26.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:26.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:26.574 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:26.620 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:26.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:26.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:26.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:26 smithi043 ceph-mon[29963]: pgmap v128: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-30T13:46:26.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:26 smithi043 ceph-mon[29963]: osdmap e118: 8 total, 8 up, 8 in 2024-03-30T13:46:26.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:26 smithi043 ceph-mon[42099]: pgmap v128: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-30T13:46:26.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:26 smithi043 ceph-mon[42099]: osdmap e118: 8 total, 8 up, 8 in 2024-03-30T13:46:26.722 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:26.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:26 smithi113 ceph-mon[34341]: pgmap v128: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-30T13:46:26.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:26 smithi113 ceph-mon[34341]: osdmap e118: 8 total, 8 up, 8 in 2024-03-30T13:46:26.762 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:26.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:26.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:26.864 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:26.896 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:26.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:26.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:26.998 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:27.029 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:27.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:27.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:27.131 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:27.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:27.168 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:27.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:27.269 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:27.299 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:27.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:27.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:27.401 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:27.412 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 87 16 119307 119291 85.6764 65.5 0.010511 0.0116665 2024-03-30T13:46:27.432 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:27.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:27.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:27.534 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:27.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:27.580 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:27.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:27.629 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T13:46:27.630 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:27 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:46:27.630 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:27 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:46:27.630 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:27 smithi043 ceph-mon[29963]: osdmap e119: 8 total, 8 up, 8 in 2024-03-30T13:46:27.630 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:27 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4154193247' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:46:27.630 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:27 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:46:27.630 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:27 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:46:27.630 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:27 smithi043 ceph-mon[42099]: osdmap e119: 8 total, 8 up, 8 in 2024-03-30T13:46:27.630 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:27 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/4154193247' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:46:27.681 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:27.711 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:27.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:27.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:27.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:27 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:46:27.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:27 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:46:27.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:27 smithi113 ceph-mon[34341]: osdmap e119: 8 total, 8 up, 8 in 2024-03-30T13:46:27.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:27 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/4154193247' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:46:27.814 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:27.852 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:27.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:27.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:27.954 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:27.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:27.984 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:27.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:28.086 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:28.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:28.136 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:28.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:28.237 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:28.267 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:28.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:28.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:28.369 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:28.396 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:28.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:28.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:28.412 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 88 16 120309 120293 85.4144 62.625 0.0308404 0.0117016 2024-03-30T13:46:28.498 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:28.539 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:28.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:28.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:28.641 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:28.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:28 smithi043 ceph-mon[29963]: pgmap v131: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 820 op/s 2024-03-30T13:46:28.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:28 smithi043 ceph-mon[29963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:46:28.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:28 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4154193247' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:46:28.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:28 smithi043 ceph-mon[29963]: osdmap e120: 8 total, 8 up, 8 in 2024-03-30T13:46:28.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:46:28.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:28 smithi043 ceph-mon[29963]: 5.b scrub starts 2024-03-30T13:46:28.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:28 smithi043 ceph-mon[29963]: 5.b scrub ok 2024-03-30T13:46:28.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:28 smithi043 ceph-mon[42099]: pgmap v131: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 820 op/s 2024-03-30T13:46:28.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:28 smithi043 ceph-mon[42099]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:46:28.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:28 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/4154193247' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:46:28.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:28 smithi043 ceph-mon[42099]: osdmap e120: 8 total, 8 up, 8 in 2024-03-30T13:46:28.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:46:28.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:28 smithi043 ceph-mon[42099]: 5.b scrub starts 2024-03-30T13:46:28.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:28 smithi043 ceph-mon[42099]: 5.b scrub ok 2024-03-30T13:46:28.672 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:28.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:28.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:28.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:46:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:46:28.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:28 smithi113 ceph-mon[34341]: pgmap v131: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 820 op/s 2024-03-30T13:46:28.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:28 smithi113 ceph-mon[34341]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:46:28.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:28 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/4154193247' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:46:28.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:28 smithi113 ceph-mon[34341]: osdmap e120: 8 total, 8 up, 8 in 2024-03-30T13:46:28.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:46:28.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:28 smithi113 ceph-mon[34341]: 5.b scrub starts 2024-03-30T13:46:28.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:28 smithi113 ceph-mon[34341]: 5.b scrub ok 2024-03-30T13:46:28.774 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:28.816 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:28.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:28.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:28.918 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:28.964 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:28.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:28.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:29.066 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:29.109 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:29.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:29.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:29.211 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:29.239 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:29.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:29.341 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:29.367 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:29.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:29.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:29.413 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 89 16 121399 121383 85.2201 68.125 0.0199572 0.0117292 2024-03-30T13:46:29.469 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:29.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:29.498 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:29.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:29.599 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:29.640 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:29.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:29.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:29.694 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:29 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:46:29.694 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:29 smithi043 ceph-mon[29963]: 4.10 scrub starts 2024-03-30T13:46:29.694 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:29 smithi043 ceph-mon[29963]: 4.10 scrub ok 2024-03-30T13:46:29.694 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:29 smithi043 ceph-mon[29963]: 4.d scrub starts 2024-03-30T13:46:29.694 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:29 smithi043 ceph-mon[29963]: 4.d scrub ok 2024-03-30T13:46:29.694 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:29 smithi043 ceph-mon[29963]: 5.8 scrub starts 2024-03-30T13:46:29.694 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:29 smithi043 ceph-mon[29963]: 5.8 scrub ok 2024-03-30T13:46:29.695 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:29 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:46:29.695 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:29 smithi043 ceph-mon[42099]: 4.10 scrub starts 2024-03-30T13:46:29.695 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:29 smithi043 ceph-mon[42099]: 4.10 scrub ok 2024-03-30T13:46:29.695 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:29 smithi043 ceph-mon[42099]: 4.d scrub starts 2024-03-30T13:46:29.695 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:29 smithi043 ceph-mon[42099]: 4.d scrub ok 2024-03-30T13:46:29.695 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:29 smithi043 ceph-mon[42099]: 5.8 scrub starts 2024-03-30T13:46:29.695 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:29 smithi043 ceph-mon[42099]: 5.8 scrub ok 2024-03-30T13:46:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:29 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:46:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:29 smithi113 ceph-mon[34341]: 4.10 scrub starts 2024-03-30T13:46:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:29 smithi113 ceph-mon[34341]: 4.10 scrub ok 2024-03-30T13:46:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:29 smithi113 ceph-mon[34341]: 4.d scrub starts 2024-03-30T13:46:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:29 smithi113 ceph-mon[34341]: 4.d scrub ok 2024-03-30T13:46:29.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:29 smithi113 ceph-mon[34341]: 5.8 scrub starts 2024-03-30T13:46:29.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:29 smithi113 ceph-mon[34341]: 5.8 scrub ok 2024-03-30T13:46:29.742 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:29.786 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:29.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:29.887 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:29.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:29.916 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:29.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:29.933 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:46:29.933 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:46:29.934 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:46:29.934 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:46:29.934 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:46:29.934 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:46:29.934 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:46:29.934 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:46:29.934 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:46:29.934 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:46:29.934 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:46:29.934 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:46:29.935 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:46:29.935 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:46:29.935 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:46:29.935 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:46:30.017 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:30.062 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:30.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:30.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:30.164 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:30.195 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:30.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:30.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:30.297 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:30.326 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:30.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:30.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:30.413 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Total time run: 90.0157 2024-03-30T13:46:30.413 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Total writes made: 122335 2024-03-30T13:46:30.413 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Write size: 65536 2024-03-30T13:46:30.413 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Object size: 65536 2024-03-30T13:46:30.413 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Bandwidth (MB/sec): 84.94 2024-03-30T13:46:30.413 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev Bandwidth: 30.9823 2024-03-30T13:46:30.413 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max bandwidth (MB/sec): 175.5 2024-03-30T13:46:30.413 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min bandwidth (MB/sec): 51.625 2024-03-30T13:46:30.414 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Average IOPS: 1359 2024-03-30T13:46:30.414 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev IOPS: 495.718 2024-03-30T13:46:30.414 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max IOPS: 2808 2024-03-30T13:46:30.414 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min IOPS: 826 2024-03-30T13:46:30.414 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Average Latency(s): 0.0117688 2024-03-30T13:46:30.414 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev Latency(s): 0.00834792 2024-03-30T13:46:30.414 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max latency(s): 0.166691 2024-03-30T13:46:30.414 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min latency(s): 0.00133902 2024-03-30T13:46:30.419 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Cleaning up (deleting benchmark objects) 2024-03-30T13:46:30.428 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:30.456 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:30.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:30.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:30.558 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:30.584 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:30.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:30.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:30.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:30 smithi043 ceph-mon[29963]: pgmap v133: 148 pgs: 148 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-30T13:46:30.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:30 smithi043 ceph-mon[29963]: 4.11 scrub starts 2024-03-30T13:46:30.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:30 smithi043 ceph-mon[29963]: 4.11 scrub ok 2024-03-30T13:46:30.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:30 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2744540732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:46:30.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:30 smithi043 ceph-mon[29963]: 5.16 scrub starts 2024-03-30T13:46:30.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:30 smithi043 ceph-mon[29963]: 5.16 scrub ok 2024-03-30T13:46:30.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:30 smithi043 ceph-mon[42099]: pgmap v133: 148 pgs: 148 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-30T13:46:30.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:30 smithi043 ceph-mon[42099]: 4.11 scrub starts 2024-03-30T13:46:30.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:30 smithi043 ceph-mon[42099]: 4.11 scrub ok 2024-03-30T13:46:30.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:30 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2744540732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:46:30.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:30 smithi043 ceph-mon[42099]: 5.16 scrub starts 2024-03-30T13:46:30.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:30 smithi043 ceph-mon[42099]: 5.16 scrub ok 2024-03-30T13:46:30.686 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:30.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:30.718 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:30.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:30.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:30 smithi113 ceph-mon[34341]: pgmap v133: 148 pgs: 148 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-30T13:46:30.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:30 smithi113 ceph-mon[34341]: 4.11 scrub starts 2024-03-30T13:46:30.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:30 smithi113 ceph-mon[34341]: 4.11 scrub ok 2024-03-30T13:46:30.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:30 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2744540732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:46:30.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:30 smithi113 ceph-mon[34341]: 5.16 scrub starts 2024-03-30T13:46:30.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:30 smithi113 ceph-mon[34341]: 5.16 scrub ok 2024-03-30T13:46:30.818 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:30.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:30.885 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:30.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:30.986 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:31.031 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:31.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:31.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:31.132 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:31.161 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:31.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:31.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:31.211 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T13:46:31.262 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:31.287 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:31.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:31.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:31.389 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:31.441 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:31.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:31.495 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:31 smithi043 ceph-mon[29963]: 6.16 scrub starts 2024-03-30T13:46:31.495 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:31 smithi043 ceph-mon[29963]: 6.16 scrub ok 2024-03-30T13:46:31.496 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:31 smithi043 ceph-mon[42099]: 6.16 scrub starts 2024-03-30T13:46:31.497 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:31 smithi043 ceph-mon[42099]: 6.16 scrub ok 2024-03-30T13:46:31.542 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:31.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:31.599 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:31.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:31.614 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:46:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-03-30T13:46:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:123364 7.3 7:c3f3d3f7:::benchmark_data_smithi043_86989_object1026:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T13:46:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:46:31.593845+0000", 2024-03-30T13:46:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.011775601, 2024-03-30T13:46:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011822954, 2024-03-30T13:46:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:46:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:46:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-03-30T13:46:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:46:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:46:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:46:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 123364 2024-03-30T13:46:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:46:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:46:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.593845+0000", 2024-03-30T13:46:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:46:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:46:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.593845+0000", 2024-03-30T13:46:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:46:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:46:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.593846+0000", 2024-03-30T13:46:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4799999999999998e-07 2024-03-30T13:46:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:46:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.593849+0000", 2024-03-30T13:46:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5190000000000001e-06 2024-03-30T13:46:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:46:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.593857+0000", 2024-03-30T13:46:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1100000000000003e-06 2024-03-30T13:46:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:46:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.593873+0000", 2024-03-30T13:46:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6028999999999998e-05 2024-03-30T13:46:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:46:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.598138+0000", 2024-03-30T13:46:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0042644960000000004 2024-03-30T13:46:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:46:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.598266+0000", 2024-03-30T13:46:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000128467 2024-03-30T13:46:31.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,4", 2024-03-30T13:46:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.598387+0000", 2024-03-30T13:46:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012129300000000001 2024-03-30T13:46:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:46:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.604229+0000", 2024-03-30T13:46:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0058411920000000003 2024-03-30T13:46:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:46:31.619 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.604396+0000", 2024-03-30T13:46:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016714400000000001 2024-03-30T13:46:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:46:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:46:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:46:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:123372 7.3 7:c8358a14:::benchmark_data_smithi043_86989_object1034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T13:46:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:46:31.605397+0000", 2024-03-30T13:46:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.00022389299999999999, 2024-03-30T13:46:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00038598499999999998, 2024-03-30T13:46:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:46:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:46:31.620 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-03-30T13:46:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:46:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:46:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:46:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 123372 2024-03-30T13:46:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:46:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:46:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605397+0000", 2024-03-30T13:46:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:46:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.621 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:46:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605397+0000", 2024-03-30T13:46:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:46:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:46:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605398+0000", 2024-03-30T13:46:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3790000000000001e-06 2024-03-30T13:46:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:46:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605404+0000", 2024-03-30T13:46:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7130000000000002e-06 2024-03-30T13:46:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:46:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605419+0000", 2024-03-30T13:46:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4919000000000001e-05 2024-03-30T13:46:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.623 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:46:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605427+0000", 2024-03-30T13:46:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5019999999999992e-06 2024-03-30T13:46:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:46:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605488+0000", 2024-03-30T13:46:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0381999999999999e-05 2024-03-30T13:46:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:46:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605621+0000", 2024-03-30T13:46:31.624 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013350600000000001 2024-03-30T13:46:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,4", 2024-03-30T13:46:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605739+0000", 2024-03-30T13:46:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000117557 2024-03-30T13:46:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:46:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:46:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:46:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:123374 7.3 7.92f860d3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T13:46:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:46:31.605696+0000", 2024-03-30T13:46:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 4294967295.9999242, 2024-03-30T13:46:31.625 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000169261, 2024-03-30T13:46:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:46:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:46:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "queued for pg", 2024-03-30T13:46:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:46:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:46:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:46:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 123374 2024-03-30T13:46:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:46:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:46:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605696+0000", 2024-03-30T13:46:31.626 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:46:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:46:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605696+0000", 2024-03-30T13:46:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:46:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:46:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605698+0000", 2024-03-30T13:46:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1349999999999999e-06 2024-03-30T13:46:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:46:31.627 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605703+0000", 2024-03-30T13:46:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7599999999999999e-06 2024-03-30T13:46:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:46:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605716+0000", 2024-03-30T13:46:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.261e-05 2024-03-30T13:46:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:46:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605725+0000", 2024-03-30T13:46:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5860000000000005e-06 2024-03-30T13:46:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:46:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:46:31.628 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:46:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:123377 7.a 7:584d7768:::benchmark_data_smithi043_86989_object1039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T13:46:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:46:31.605738+0000", 2024-03-30T13:46:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 4294967295.9998832, 2024-03-30T13:46:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018302800000000001, 2024-03-30T13:46:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:46:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:46:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-03-30T13:46:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:46:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:46:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:46:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 123377 2024-03-30T13:46:31.629 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:46:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:46:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605738+0000", 2024-03-30T13:46:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:46:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:46:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605738+0000", 2024-03-30T13:46:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:46:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.630 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:46:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605738+0000", 2024-03-30T13:46:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2899999999999999e-07 2024-03-30T13:46:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:46:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605741+0000", 2024-03-30T13:46:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.8590000000000001e-06 2024-03-30T13:46:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:46:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605748+0000", 2024-03-30T13:46:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3379999999999999e-06 2024-03-30T13:46:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.631 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:46:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605767+0000", 2024-03-30T13:46:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8576e-05 2024-03-30T13:46:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:46:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:46:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:46:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:31.605818+0000", 2024-03-30T13:46:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1279000000000002e-05 2024-03-30T13:46:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:46:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:46:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:46:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:46:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:46:31.632 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 4 2024-03-30T13:46:31.633 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:46:31.700 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:31.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:31 smithi113 ceph-mon[34341]: 6.16 scrub starts 2024-03-30T13:46:31.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:31 smithi113 ceph-mon[34341]: 6.16 scrub ok 2024-03-30T13:46:31.728 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:31.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:31.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:31.830 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:31.860 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:31.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:31.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:31.962 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:31.989 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:31.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:32.091 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:32.120 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:32.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:32.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:32.222 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:32.251 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:32.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:32.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:32.353 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:32.385 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:32.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:32.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:32.487 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:32.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:32.517 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:32.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:32.543 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-03-30T13:46:32.618 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:32.645 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:32.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:32.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:32.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:32 smithi113 ceph-mon[34341]: pgmap v134: 148 pgs: 148 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 54 MiB/s wr, 864 op/s 2024-03-30T13:46:32.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:32 smithi113 ceph-mon[34341]: 3.b scrub starts 2024-03-30T13:46:32.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:32 smithi113 ceph-mon[34341]: 3.b scrub ok 2024-03-30T13:46:32.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:32 smithi113 ceph-mon[34341]: 6.a scrub starts 2024-03-30T13:46:32.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:32 smithi113 ceph-mon[34341]: 6.a scrub ok 2024-03-30T13:46:32.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:32 smithi113 ceph-mon[34341]: 3.1f scrub starts 2024-03-30T13:46:32.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:32 smithi113 ceph-mon[34341]: 3.1f scrub ok 2024-03-30T13:46:32.747 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:32.776 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:32 smithi043 ceph-mon[29963]: pgmap v134: 148 pgs: 148 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 54 MiB/s wr, 864 op/s 2024-03-30T13:46:32.776 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:32 smithi043 ceph-mon[29963]: 3.b scrub starts 2024-03-30T13:46:32.777 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:32 smithi043 ceph-mon[29963]: 3.b scrub ok 2024-03-30T13:46:32.777 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:32 smithi043 ceph-mon[29963]: 6.a scrub starts 2024-03-30T13:46:32.777 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:32 smithi043 ceph-mon[29963]: 6.a scrub ok 2024-03-30T13:46:32.777 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:32 smithi043 ceph-mon[29963]: 3.1f scrub starts 2024-03-30T13:46:32.777 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:32 smithi043 ceph-mon[29963]: 3.1f scrub ok 2024-03-30T13:46:32.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:32 smithi043 ceph-mon[42099]: pgmap v134: 148 pgs: 148 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 54 MiB/s wr, 864 op/s 2024-03-30T13:46:32.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:32 smithi043 ceph-mon[42099]: 3.b scrub starts 2024-03-30T13:46:32.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:32 smithi043 ceph-mon[42099]: 3.b scrub ok 2024-03-30T13:46:32.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:32 smithi043 ceph-mon[42099]: 6.a scrub starts 2024-03-30T13:46:32.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:32 smithi043 ceph-mon[42099]: 6.a scrub ok 2024-03-30T13:46:32.778 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:32 smithi043 ceph-mon[42099]: 3.1f scrub starts 2024-03-30T13:46:32.778 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:32 smithi043 ceph-mon[42099]: 3.1f scrub ok 2024-03-30T13:46:32.778 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:32.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:32.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:32.861 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:46:32.878 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:46:32.880 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:32.905 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:32.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:32.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:33.007 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:33.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:33.064 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:33.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:33.165 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:33.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:33.221 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:33.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:33.322 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:33.377 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:33.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:33.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:33.479 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:33.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:33.509 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:33.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:33.610 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:33.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:33 smithi043 ceph-mon[42099]: 7.0 scrub starts 2024-03-30T13:46:33.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:33 smithi043 ceph-mon[29963]: 7.0 scrub starts 2024-03-30T13:46:33.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:33.652 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:33.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:33.753 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:33.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:33.783 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:33.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:33.884 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:33.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:33.912 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:33.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:33.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:33 smithi113 ceph-mon[34341]: 7.0 scrub starts 2024-03-30T13:46:34.013 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:34.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:34.061 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:34.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:34.162 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:34.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:34.200 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:34.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:34.301 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:34.343 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:34.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:34.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:34.445 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:34.483 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:34.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:34.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:34.585 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:34.615 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:34.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:34.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:34.717 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:34.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:34.766 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:34.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:34.800 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:34 smithi043 ceph-mon[29963]: pgmap v135: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 47 MiB/s wr, 885 op/s 2024-03-30T13:46:34.800 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:34 smithi043 ceph-mon[29963]: 3.d scrub starts 2024-03-30T13:46:34.800 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:34 smithi043 ceph-mon[29963]: 3.d scrub ok 2024-03-30T13:46:34.800 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:34 smithi043 ceph-mon[29963]: 4.8 scrub starts 2024-03-30T13:46:34.800 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:34 smithi043 ceph-mon[29963]: 4.8 scrub ok 2024-03-30T13:46:34.800 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:34 smithi043 ceph-mon[42099]: pgmap v135: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 47 MiB/s wr, 885 op/s 2024-03-30T13:46:34.800 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:34 smithi043 ceph-mon[42099]: 3.d scrub starts 2024-03-30T13:46:34.800 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:34 smithi043 ceph-mon[42099]: 3.d scrub ok 2024-03-30T13:46:34.801 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:34 smithi043 ceph-mon[42099]: 4.8 scrub starts 2024-03-30T13:46:34.801 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:34 smithi043 ceph-mon[42099]: 4.8 scrub ok 2024-03-30T13:46:34.866 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:34.895 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:34.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:34.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:34.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:34 smithi113 ceph-mon[34341]: pgmap v135: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 47 MiB/s wr, 885 op/s 2024-03-30T13:46:34.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:34 smithi113 ceph-mon[34341]: 3.d scrub starts 2024-03-30T13:46:34.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:34 smithi113 ceph-mon[34341]: 3.d scrub ok 2024-03-30T13:46:34.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:34 smithi113 ceph-mon[34341]: 4.8 scrub starts 2024-03-30T13:46:34.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:34 smithi113 ceph-mon[34341]: 4.8 scrub ok 2024-03-30T13:46:34.997 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:35.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:35.031 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:35.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:35.133 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:35.160 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:35.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:35.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:35.182 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:46:35.216 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:46:35.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:35.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:35.316 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:35.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:35.417 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:35.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:35.457 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:35.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:35.558 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:35.588 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:35.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:35.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:35.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:35 smithi043 ceph-mon[29963]: 7.0 scrub ok 2024-03-30T13:46:35.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:35 smithi043 ceph-mon[29963]: 7.d scrub starts 2024-03-30T13:46:35.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:35 smithi043 ceph-mon[42099]: 7.0 scrub ok 2024-03-30T13:46:35.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:35 smithi043 ceph-mon[42099]: 7.d scrub starts 2024-03-30T13:46:35.690 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:35.728 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:35.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:35.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:35.830 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:35.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:35.871 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:35.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:35.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:35 smithi113 ceph-mon[34341]: 7.0 scrub ok 2024-03-30T13:46:35.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:35 smithi113 ceph-mon[34341]: 7.d scrub starts 2024-03-30T13:46:35.972 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:36.015 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:36.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:36.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:36.117 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:36.147 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:36.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:36.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:36.249 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:36.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:36.280 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:36.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:36.334 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:46:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:46:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:46:36.381 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:36.410 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:36.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:36.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:36.512 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:36.550 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:36.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:36.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:36.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:36 smithi043 ceph-mon[29963]: pgmap v136: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 1.13k op/s 2024-03-30T13:46:36.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:36 smithi043 ceph-mon[29963]: 4.0 scrub starts 2024-03-30T13:46:36.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:36 smithi043 ceph-mon[29963]: 4.0 scrub ok 2024-03-30T13:46:36.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:36 smithi043 ceph-mon[29963]: 3.13 scrub starts 2024-03-30T13:46:36.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:36 smithi043 ceph-mon[29963]: 3.13 scrub ok 2024-03-30T13:46:36.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:36 smithi043 ceph-mon[29963]: 4.c scrub starts 2024-03-30T13:46:36.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:36 smithi043 ceph-mon[29963]: 4.c scrub ok 2024-03-30T13:46:36.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:36 smithi043 ceph-mon[42099]: pgmap v136: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 1.13k op/s 2024-03-30T13:46:36.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:36 smithi043 ceph-mon[42099]: 4.0 scrub starts 2024-03-30T13:46:36.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:36 smithi043 ceph-mon[42099]: 4.0 scrub ok 2024-03-30T13:46:36.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:36 smithi043 ceph-mon[42099]: 3.13 scrub starts 2024-03-30T13:46:36.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:36 smithi043 ceph-mon[42099]: 3.13 scrub ok 2024-03-30T13:46:36.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:36 smithi043 ceph-mon[42099]: 4.c scrub starts 2024-03-30T13:46:36.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:36 smithi043 ceph-mon[42099]: 4.c scrub ok 2024-03-30T13:46:36.653 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:36.686 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:36.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:36.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:36.788 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:36.827 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:36.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:36.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:36.929 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:36.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:36.961 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:36.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:36.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:36 smithi113 ceph-mon[34341]: pgmap v136: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 1.13k op/s 2024-03-30T13:46:36.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:36 smithi113 ceph-mon[34341]: 4.0 scrub starts 2024-03-30T13:46:36.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:36 smithi113 ceph-mon[34341]: 4.0 scrub ok 2024-03-30T13:46:36.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:36 smithi113 ceph-mon[34341]: 3.13 scrub starts 2024-03-30T13:46:36.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:36 smithi113 ceph-mon[34341]: 3.13 scrub ok 2024-03-30T13:46:36.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:36 smithi113 ceph-mon[34341]: 4.c scrub starts 2024-03-30T13:46:36.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:36 smithi113 ceph-mon[34341]: 4.c scrub ok 2024-03-30T13:46:37.062 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:37.094 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:37.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:37.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:37.196 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:37.227 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:37.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:37.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:37.329 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:37.383 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:37.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:37.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:37.485 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:37.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:37.556 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:37.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:37.634 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:37 smithi043 ceph-mon[29963]: 4.1d scrub starts 2024-03-30T13:46:37.634 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:37 smithi043 ceph-mon[29963]: 4.1d scrub ok 2024-03-30T13:46:37.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:37 smithi043 ceph-mon[29963]: 5.6 scrub starts 2024-03-30T13:46:37.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:37 smithi043 ceph-mon[29963]: 5.6 scrub ok 2024-03-30T13:46:37.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:37 smithi043 ceph-mon[29963]: 5.15 scrub starts 2024-03-30T13:46:37.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:37 smithi043 ceph-mon[29963]: 5.15 scrub ok 2024-03-30T13:46:37.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:37 smithi043 ceph-mon[29963]: 7.c scrub starts 2024-03-30T13:46:37.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:37 smithi043 ceph-mon[29963]: 7.d scrub ok 2024-03-30T13:46:37.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:37 smithi043 ceph-mon[29963]: 6.10 scrub starts 2024-03-30T13:46:37.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:37 smithi043 ceph-mon[29963]: 6.10 scrub ok 2024-03-30T13:46:37.635 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:37 smithi043 ceph-mon[42099]: 4.1d scrub starts 2024-03-30T13:46:37.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:37 smithi043 ceph-mon[42099]: 4.1d scrub ok 2024-03-30T13:46:37.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:37 smithi043 ceph-mon[42099]: 5.6 scrub starts 2024-03-30T13:46:37.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:37 smithi043 ceph-mon[42099]: 5.6 scrub ok 2024-03-30T13:46:37.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:37 smithi043 ceph-mon[42099]: 5.15 scrub starts 2024-03-30T13:46:37.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:37 smithi043 ceph-mon[42099]: 5.15 scrub ok 2024-03-30T13:46:37.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:37 smithi043 ceph-mon[42099]: 7.c scrub starts 2024-03-30T13:46:37.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:37 smithi043 ceph-mon[42099]: 7.d scrub ok 2024-03-30T13:46:37.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:37 smithi043 ceph-mon[42099]: 6.10 scrub starts 2024-03-30T13:46:37.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:37 smithi043 ceph-mon[42099]: 6.10 scrub ok 2024-03-30T13:46:37.658 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:37.691 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:37.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:37.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:37.793 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:37.831 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:37.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:37.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:37.933 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:37.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:37 smithi113 ceph-mon[34341]: 4.1d scrub starts 2024-03-30T13:46:37.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:37 smithi113 ceph-mon[34341]: 4.1d scrub ok 2024-03-30T13:46:37.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:37 smithi113 ceph-mon[34341]: 5.6 scrub starts 2024-03-30T13:46:37.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:37 smithi113 ceph-mon[34341]: 5.6 scrub ok 2024-03-30T13:46:37.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:37 smithi113 ceph-mon[34341]: 5.15 scrub starts 2024-03-30T13:46:37.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:37 smithi113 ceph-mon[34341]: 5.15 scrub ok 2024-03-30T13:46:37.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:37 smithi113 ceph-mon[34341]: 7.c scrub starts 2024-03-30T13:46:37.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:37 smithi113 ceph-mon[34341]: 7.d scrub ok 2024-03-30T13:46:37.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:37 smithi113 ceph-mon[34341]: 6.10 scrub starts 2024-03-30T13:46:37.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:37 smithi113 ceph-mon[34341]: 6.10 scrub ok 2024-03-30T13:46:37.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:37.974 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:37.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:38.075 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:38.102 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:38.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:38.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:38.204 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:38.233 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:38.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:38.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:38.338 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:38.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:38.380 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:38.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:38.481 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:38.513 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:38.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:38.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:38.615 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:38.642 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:38.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:38.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:38.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:46:38 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:46:38.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:38 smithi113 ceph-mon[34341]: pgmap v137: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 45 MiB/s wr, 1.01k op/s 2024-03-30T13:46:38.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:38 smithi113 ceph-mon[34341]: 4.4 scrub starts 2024-03-30T13:46:38.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:38 smithi113 ceph-mon[34341]: 4.4 scrub ok 2024-03-30T13:46:38.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:38 smithi113 ceph-mon[34341]: 6.18 scrub starts 2024-03-30T13:46:38.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:38 smithi113 ceph-mon[34341]: 6.18 scrub ok 2024-03-30T13:46:38.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:38 smithi113 ceph-mon[34341]: 6.6 scrub starts 2024-03-30T13:46:38.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:38 smithi113 ceph-mon[34341]: 6.6 scrub ok 2024-03-30T13:46:38.746 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:38.783 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:38 smithi043 ceph-mon[29963]: pgmap v137: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 45 MiB/s wr, 1.01k op/s 2024-03-30T13:46:38.783 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:38 smithi043 ceph-mon[29963]: 4.4 scrub starts 2024-03-30T13:46:38.783 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:38 smithi043 ceph-mon[29963]: 4.4 scrub ok 2024-03-30T13:46:38.784 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:38 smithi043 ceph-mon[29963]: 6.18 scrub starts 2024-03-30T13:46:38.784 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:38 smithi043 ceph-mon[29963]: 6.18 scrub ok 2024-03-30T13:46:38.784 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:38 smithi043 ceph-mon[29963]: 6.6 scrub starts 2024-03-30T13:46:38.784 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:38 smithi043 ceph-mon[29963]: 6.6 scrub ok 2024-03-30T13:46:38.784 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:38 smithi043 ceph-mon[42099]: pgmap v137: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 45 MiB/s wr, 1.01k op/s 2024-03-30T13:46:38.784 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:38 smithi043 ceph-mon[42099]: 4.4 scrub starts 2024-03-30T13:46:38.784 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:38 smithi043 ceph-mon[42099]: 4.4 scrub ok 2024-03-30T13:46:38.784 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:38 smithi043 ceph-mon[42099]: 6.18 scrub starts 2024-03-30T13:46:38.784 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:38 smithi043 ceph-mon[42099]: 6.18 scrub ok 2024-03-30T13:46:38.784 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:38 smithi043 ceph-mon[42099]: 6.6 scrub starts 2024-03-30T13:46:38.784 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:38 smithi043 ceph-mon[42099]: 6.6 scrub ok 2024-03-30T13:46:38.794 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:38.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:38.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:38.896 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:38.925 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:38.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:38.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:38.929 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T13:46:38.929 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:46:38.929 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-30T13:46:38.930 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 'trim stale osdmaps' 2024-03-30T13:46:39.027 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:39.052 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:39.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:39.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:39.153 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:39.187 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:39.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:39.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:39.289 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:39.318 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:39.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:39.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:39.420 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:39.450 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:39.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:39.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:39.544 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-03-30T13:46:39.553 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:39.575 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:39.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:39.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:39.677 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:39.706 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:39.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:39.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:39.808 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:39.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:39.837 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:39.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:39.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:39 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:46:39.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:39 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2770155659' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:46:39.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:39 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:46:39.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:39 smithi043 ceph-mon[29963]: 7.c scrub ok 2024-03-30T13:46:39.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:39 smithi043 ceph-mon[29963]: 6.f scrub starts 2024-03-30T13:46:39.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:39 smithi043 ceph-mon[29963]: 6.f scrub ok 2024-03-30T13:46:39.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:39 smithi043 ceph-mon[29963]: 4.5 scrub starts 2024-03-30T13:46:39.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:39 smithi043 ceph-mon[29963]: 4.5 scrub ok 2024-03-30T13:46:39.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:39 smithi043 ceph-mon[29963]: 5.14 scrub starts 2024-03-30T13:46:39.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:39 smithi043 ceph-mon[29963]: 5.14 scrub ok 2024-03-30T13:46:39.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:39 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:46:39.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:39 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2770155659' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:46:39.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:39 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:46:39.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:39 smithi043 ceph-mon[42099]: 7.c scrub ok 2024-03-30T13:46:39.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:39 smithi043 ceph-mon[42099]: 6.f scrub starts 2024-03-30T13:46:39.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:39 smithi043 ceph-mon[42099]: 6.f scrub ok 2024-03-30T13:46:39.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:39 smithi043 ceph-mon[42099]: 4.5 scrub starts 2024-03-30T13:46:39.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:39 smithi043 ceph-mon[42099]: 4.5 scrub ok 2024-03-30T13:46:39.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:39 smithi043 ceph-mon[42099]: 5.14 scrub starts 2024-03-30T13:46:39.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:39 smithi043 ceph-mon[42099]: 5.14 scrub ok 2024-03-30T13:46:39.938 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:39.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:39 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:46:39.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:39 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2770155659' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:46:39.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:39 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:46:39.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:39 smithi113 ceph-mon[34341]: 7.c scrub ok 2024-03-30T13:46:39.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:39 smithi113 ceph-mon[34341]: 6.f scrub starts 2024-03-30T13:46:39.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:39 smithi113 ceph-mon[34341]: 6.f scrub ok 2024-03-30T13:46:39.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:39 smithi113 ceph-mon[34341]: 4.5 scrub starts 2024-03-30T13:46:39.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:39 smithi113 ceph-mon[34341]: 4.5 scrub ok 2024-03-30T13:46:39.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:39 smithi113 ceph-mon[34341]: 5.14 scrub starts 2024-03-30T13:46:39.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:39 smithi113 ceph-mon[34341]: 5.14 scrub ok 2024-03-30T13:46:39.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:39.980 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:39.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:40.081 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:40.123 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:40.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:40.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:40.225 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:40.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:40.269 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:40.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:40.370 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:40.413 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:40.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:40.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:40.514 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:40.532 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:40 smithi043 ceph-mon[29963]: pgmap v138: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 38 MiB/s wr, 1.29k op/s 2024-03-30T13:46:40.532 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:40 smithi043 ceph-mon[29963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:46:40.532 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:40 smithi043 ceph-mon[29963]: Cluster is now healthy 2024-03-30T13:46:40.532 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:40 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:46:40.532 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:40 smithi043 ceph-mon[29963]: osdmap e121: 8 total, 8 up, 8 in 2024-03-30T13:46:40.532 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:40 smithi043 ceph-mon[29963]: 4.1a scrub starts 2024-03-30T13:46:40.533 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:40 smithi043 ceph-mon[29963]: 4.1a scrub ok 2024-03-30T13:46:40.533 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:40 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2972788082' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:46:40.533 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:40 smithi043 ceph-mon[42099]: pgmap v138: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 38 MiB/s wr, 1.29k op/s 2024-03-30T13:46:40.533 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:40 smithi043 ceph-mon[42099]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:46:40.533 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:40 smithi043 ceph-mon[42099]: Cluster is now healthy 2024-03-30T13:46:40.533 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:40 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:46:40.533 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:40 smithi043 ceph-mon[42099]: osdmap e121: 8 total, 8 up, 8 in 2024-03-30T13:46:40.533 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:40 smithi043 ceph-mon[42099]: 4.1a scrub starts 2024-03-30T13:46:40.533 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:40 smithi043 ceph-mon[42099]: 4.1a scrub ok 2024-03-30T13:46:40.533 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:40 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2972788082' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:46:40.540 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:40.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:40.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:40.642 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:40.685 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:40.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:40.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:40.710 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:46:40.710 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:46:40.710 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:46:40.710 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:46:40.711 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:46:40.711 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:46:40.711 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:46:40.711 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:46:40.711 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:46:40.711 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:46:40.711 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:46:40.711 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:46:40.711 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:46:40.711 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:46:40.712 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:46:40.712 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:46:40.787 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:40.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:40.823 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:40.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:40.924 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:40.952 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:40.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:40.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:40.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:40 smithi113 ceph-mon[34341]: pgmap v138: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 38 MiB/s wr, 1.29k op/s 2024-03-30T13:46:40.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:40 smithi113 ceph-mon[34341]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:46:40.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:40 smithi113 ceph-mon[34341]: Cluster is now healthy 2024-03-30T13:46:40.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:40 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:46:40.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:40 smithi113 ceph-mon[34341]: osdmap e121: 8 total, 8 up, 8 in 2024-03-30T13:46:40.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:40 smithi113 ceph-mon[34341]: 4.1a scrub starts 2024-03-30T13:46:40.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:40 smithi113 ceph-mon[34341]: 4.1a scrub ok 2024-03-30T13:46:40.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:40 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2972788082' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:46:41.054 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:41.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:41.101 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:41.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:41.163 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:46:41.163 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T13:46:41.163 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-03-30T13:46:41.163 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-03-30T13:46:41.163 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:46:41.202 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:41.229 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:41.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:41.331 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:41.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:41.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:41.477 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:41.505 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:41.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:41.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:41.608 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:41.636 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:41.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:41.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:41.738 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:41.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:41.777 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:41.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:41.786 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:41 smithi043 ceph-mon[29963]: 6.1e scrub starts 2024-03-30T13:46:41.786 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:41 smithi043 ceph-mon[29963]: 6.1e scrub ok 2024-03-30T13:46:41.786 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:41 smithi043 ceph-mon[29963]: 3.a scrub starts 2024-03-30T13:46:41.786 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:41 smithi043 ceph-mon[29963]: 3.a scrub ok 2024-03-30T13:46:41.786 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:41 smithi043 ceph-mon[29963]: 3.2 scrub starts 2024-03-30T13:46:41.786 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:41 smithi043 ceph-mon[29963]: 3.2 scrub ok 2024-03-30T13:46:41.786 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:41 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:46:41.786 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:41 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:46:41.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:41 smithi043 ceph-mon[42099]: 6.1e scrub starts 2024-03-30T13:46:41.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:41 smithi043 ceph-mon[42099]: 6.1e scrub ok 2024-03-30T13:46:41.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:41 smithi043 ceph-mon[42099]: 3.a scrub starts 2024-03-30T13:46:41.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:41 smithi043 ceph-mon[42099]: 3.a scrub ok 2024-03-30T13:46:41.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:41 smithi043 ceph-mon[42099]: 3.2 scrub starts 2024-03-30T13:46:41.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:41 smithi043 ceph-mon[42099]: 3.2 scrub ok 2024-03-30T13:46:41.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:41 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:46:41.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:41 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:46:41.832 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-03-30T13:46:41.878 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:41.959 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:41.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:41.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:41 smithi113 ceph-mon[34341]: 6.1e scrub starts 2024-03-30T13:46:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:41 smithi113 ceph-mon[34341]: 6.1e scrub ok 2024-03-30T13:46:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:41 smithi113 ceph-mon[34341]: 3.a scrub starts 2024-03-30T13:46:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:41 smithi113 ceph-mon[34341]: 3.a scrub ok 2024-03-30T13:46:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:41 smithi113 ceph-mon[34341]: 3.2 scrub starts 2024-03-30T13:46:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:41 smithi113 ceph-mon[34341]: 3.2 scrub ok 2024-03-30T13:46:41.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:41 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:46:41.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:41 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:46:42.060 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:42.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:42.088 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:42.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:42.189 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:42.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:42.234 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:42.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:42.335 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:42.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:42.373 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:42.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:42.474 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:42.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:42.524 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:42.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:42.625 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:42.653 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:42.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:42.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:42.755 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:42.782 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:42.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:42.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:42.872 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:42 smithi043 ceph-mon[29963]: pgmap v140: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 MiB/s wr, 1.02k op/s 2024-03-30T13:46:42.873 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:42 smithi043 ceph-mon[29963]: osdmap e122: 8 total, 8 up, 8 in 2024-03-30T13:46:42.873 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:42 smithi043 ceph-mon[42099]: pgmap v140: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 MiB/s wr, 1.02k op/s 2024-03-30T13:46:42.873 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:42 smithi043 ceph-mon[42099]: osdmap e122: 8 total, 8 up, 8 in 2024-03-30T13:46:42.884 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:42.941 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:42.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:42.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:42.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:42 smithi113 ceph-mon[34341]: pgmap v140: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 MiB/s wr, 1.02k op/s 2024-03-30T13:46:42.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:42 smithi113 ceph-mon[34341]: osdmap e122: 8 total, 8 up, 8 in 2024-03-30T13:46:43.043 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:43.068 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:46:43.079 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:46:43.107 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:43.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:43.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:43.208 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:43.238 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:43.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:43.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:43.340 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:43.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:43.387 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:43.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:43.488 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:43.528 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:43.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:43.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:43.630 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:43.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:43.670 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:43.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:43.773 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:43.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:43.805 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:43.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:43.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:43 smithi043 ceph-mon[29963]: 7.b scrub starts 2024-03-30T13:46:43.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:43 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:46:43.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:43 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:46:43.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:43 smithi043 ceph-mon[29963]: 5.10 deep-scrub starts 2024-03-30T13:46:43.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:43 smithi043 ceph-mon[29963]: 5.10 deep-scrub ok 2024-03-30T13:46:43.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:43 smithi043 ceph-mon[29963]: 3.1b deep-scrub starts 2024-03-30T13:46:43.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:43 smithi043 ceph-mon[29963]: 3.1b deep-scrub ok 2024-03-30T13:46:43.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:43 smithi043 ceph-mon[42099]: 7.b scrub starts 2024-03-30T13:46:43.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:43 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:46:43.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:43 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:46:43.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:43 smithi043 ceph-mon[42099]: 5.10 deep-scrub starts 2024-03-30T13:46:43.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:43 smithi043 ceph-mon[42099]: 5.10 deep-scrub ok 2024-03-30T13:46:43.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:43 smithi043 ceph-mon[42099]: 3.1b deep-scrub starts 2024-03-30T13:46:43.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:43 smithi043 ceph-mon[42099]: 3.1b deep-scrub ok 2024-03-30T13:46:43.906 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:43.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:43.930 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:43.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:43.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:43 smithi113 ceph-mon[34341]: 7.b scrub starts 2024-03-30T13:46:43.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:43 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:46:43.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:43 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:46:43.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:43 smithi113 ceph-mon[34341]: 5.10 deep-scrub starts 2024-03-30T13:46:43.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:43 smithi113 ceph-mon[34341]: 5.10 deep-scrub ok 2024-03-30T13:46:43.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:43 smithi113 ceph-mon[34341]: 3.1b deep-scrub starts 2024-03-30T13:46:43.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:43 smithi113 ceph-mon[34341]: 3.1b deep-scrub ok 2024-03-30T13:46:44.031 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:44.061 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:44.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:44.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:44.164 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:44.195 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:44.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:44.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:44.296 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:44.340 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:44.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:44.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:44.442 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:44.469 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:44.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:44.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:44.489 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:46:44.570 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:44.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:44.646 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:44.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:44.747 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:44.813 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:44.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:44.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:44.878 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:44 smithi043 ceph-mon[29963]: pgmap v142: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 903 op/s 2024-03-30T13:46:44.879 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:44 smithi043 ceph-mon[29963]: 7.e scrub starts 2024-03-30T13:46:44.879 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:44 smithi043 ceph-mon[29963]: 6.9 deep-scrub starts 2024-03-30T13:46:44.879 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:44 smithi043 ceph-mon[29963]: 6.9 deep-scrub ok 2024-03-30T13:46:44.879 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:44 smithi043 ceph-mon[29963]: 6.11 scrub starts 2024-03-30T13:46:44.879 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:44 smithi043 ceph-mon[29963]: 6.11 scrub ok 2024-03-30T13:46:44.879 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:44 smithi043 ceph-mon[42099]: pgmap v142: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 903 op/s 2024-03-30T13:46:44.879 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:44 smithi043 ceph-mon[42099]: 7.e scrub starts 2024-03-30T13:46:44.879 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:44 smithi043 ceph-mon[42099]: 6.9 deep-scrub starts 2024-03-30T13:46:44.879 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:44 smithi043 ceph-mon[42099]: 6.9 deep-scrub ok 2024-03-30T13:46:44.880 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:44 smithi043 ceph-mon[42099]: 6.11 scrub starts 2024-03-30T13:46:44.880 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:44 smithi043 ceph-mon[42099]: 6.11 scrub ok 2024-03-30T13:46:44.915 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:44.930 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T13:46:44.943 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:44.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:44.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:44.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:44 smithi113 ceph-mon[34341]: pgmap v142: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 903 op/s 2024-03-30T13:46:44.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:44 smithi113 ceph-mon[34341]: 7.e scrub starts 2024-03-30T13:46:44.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:44 smithi113 ceph-mon[34341]: 6.9 deep-scrub starts 2024-03-30T13:46:44.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:44 smithi113 ceph-mon[34341]: 6.9 deep-scrub ok 2024-03-30T13:46:44.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:44 smithi113 ceph-mon[34341]: 6.11 scrub starts 2024-03-30T13:46:44.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:44 smithi113 ceph-mon[34341]: 6.11 scrub ok 2024-03-30T13:46:45.045 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:45.070 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:45.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:45.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:45.172 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:45.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:45.229 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:45.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:45.330 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:45.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:45.361 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:45.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:45.462 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:45.509 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:45.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:45.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:45.611 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:45.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:45 smithi043 ceph-mon[29963]: 7.b scrub ok 2024-03-30T13:46:45.636 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:45 smithi043 ceph-mon[29963]: 5.4 scrub starts 2024-03-30T13:46:45.636 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:45 smithi043 ceph-mon[29963]: 5.4 scrub ok 2024-03-30T13:46:45.636 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:45 smithi043 ceph-mon[29963]: 7.8 scrub starts 2024-03-30T13:46:45.636 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:45 smithi043 ceph-mon[29963]: 3.12 scrub starts 2024-03-30T13:46:45.636 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:45 smithi043 ceph-mon[29963]: 3.12 scrub ok 2024-03-30T13:46:45.636 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:45 smithi043 ceph-mon[29963]: 3.1e scrub starts 2024-03-30T13:46:45.636 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:45 smithi043 ceph-mon[29963]: 3.1e scrub ok 2024-03-30T13:46:45.639 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:45.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:45.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:45.741 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:45.768 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:45.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:45.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:45.870 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:45.898 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:45.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:45 smithi043 ceph-mon[42099]: 7.b scrub ok 2024-03-30T13:46:45.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:45 smithi043 ceph-mon[42099]: 5.4 scrub starts 2024-03-30T13:46:45.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:45 smithi043 ceph-mon[42099]: 5.4 scrub ok 2024-03-30T13:46:45.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:45 smithi043 ceph-mon[42099]: 7.8 scrub starts 2024-03-30T13:46:45.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:45 smithi043 ceph-mon[42099]: 3.12 scrub starts 2024-03-30T13:46:45.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:45 smithi043 ceph-mon[42099]: 3.12 scrub ok 2024-03-30T13:46:45.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:45 smithi043 ceph-mon[42099]: 3.1e scrub starts 2024-03-30T13:46:45.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:45 smithi043 ceph-mon[42099]: 3.1e scrub ok 2024-03-30T13:46:45.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:45.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:45.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:45 smithi113 ceph-mon[34341]: 7.b scrub ok 2024-03-30T13:46:45.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:45 smithi113 ceph-mon[34341]: 5.4 scrub starts 2024-03-30T13:46:45.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:45 smithi113 ceph-mon[34341]: 5.4 scrub ok 2024-03-30T13:46:45.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:45 smithi113 ceph-mon[34341]: 7.8 scrub starts 2024-03-30T13:46:45.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:45 smithi113 ceph-mon[34341]: 3.12 scrub starts 2024-03-30T13:46:45.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:45 smithi113 ceph-mon[34341]: 3.12 scrub ok 2024-03-30T13:46:45.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:45 smithi113 ceph-mon[34341]: 3.1e scrub starts 2024-03-30T13:46:45.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:45 smithi113 ceph-mon[34341]: 3.1e scrub ok 2024-03-30T13:46:46.002 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:46.030 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:46.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:46.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:46.132 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:46.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:46.164 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:46.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:46.264 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:46.285 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:46:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:46:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:46:46.300 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:46.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:46.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:46.402 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:46.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:46.431 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:46.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:46.532 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:46.572 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:46.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:46.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:46.674 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:46.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:46.706 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:46.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:46.807 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:46.836 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:46.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:46.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:46 smithi043 ceph-mon[29963]: 6.e scrub starts 2024-03-30T13:46:46.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:46 smithi043 ceph-mon[29963]: 6.e scrub ok 2024-03-30T13:46:46.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:46 smithi043 ceph-mon[29963]: 4.f scrub starts 2024-03-30T13:46:46.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:46 smithi043 ceph-mon[29963]: 4.f scrub ok 2024-03-30T13:46:46.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:46 smithi043 ceph-mon[29963]: pgmap v143: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-30T13:46:46.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:46 smithi043 ceph-mon[29963]: 6.14 scrub starts 2024-03-30T13:46:46.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:46 smithi043 ceph-mon[29963]: 6.14 scrub ok 2024-03-30T13:46:46.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:46 smithi043 ceph-mon[29963]: 7.4 scrub starts 2024-03-30T13:46:46.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:46 smithi043 ceph-mon[29963]: 7.e scrub ok 2024-03-30T13:46:46.902 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:46 smithi043 ceph-mon[42099]: 6.e scrub starts 2024-03-30T13:46:46.902 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:46 smithi043 ceph-mon[42099]: 6.e scrub ok 2024-03-30T13:46:46.902 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:46 smithi043 ceph-mon[42099]: 4.f scrub starts 2024-03-30T13:46:46.902 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:46 smithi043 ceph-mon[42099]: 4.f scrub ok 2024-03-30T13:46:46.902 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:46 smithi043 ceph-mon[42099]: pgmap v143: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-30T13:46:46.903 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:46 smithi043 ceph-mon[42099]: 6.14 scrub starts 2024-03-30T13:46:46.903 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:46 smithi043 ceph-mon[42099]: 6.14 scrub ok 2024-03-30T13:46:46.903 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:46 smithi043 ceph-mon[42099]: 7.4 scrub starts 2024-03-30T13:46:46.903 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:46 smithi043 ceph-mon[42099]: 7.e scrub ok 2024-03-30T13:46:46.938 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:46.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:46 smithi113 ceph-mon[34341]: 6.e scrub starts 2024-03-30T13:46:46.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:46 smithi113 ceph-mon[34341]: 6.e scrub ok 2024-03-30T13:46:46.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:46 smithi113 ceph-mon[34341]: 4.f scrub starts 2024-03-30T13:46:46.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:46 smithi113 ceph-mon[34341]: 4.f scrub ok 2024-03-30T13:46:46.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:46 smithi113 ceph-mon[34341]: pgmap v143: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-30T13:46:46.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:46 smithi113 ceph-mon[34341]: 6.14 scrub starts 2024-03-30T13:46:46.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:46 smithi113 ceph-mon[34341]: 6.14 scrub ok 2024-03-30T13:46:46.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:46 smithi113 ceph-mon[34341]: 7.4 scrub starts 2024-03-30T13:46:46.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:46 smithi113 ceph-mon[34341]: 7.e scrub ok 2024-03-30T13:46:47.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:47.002 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:47.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:47.103 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:47.139 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:47.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:47.241 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:47.266 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:47.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:47.368 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:47.424 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:47.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:47.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:47.472 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:46:47.527 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:47.556 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:47.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:47.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:47.580 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:46:47.580 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:46:47.658 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:47.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:47.715 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:47.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:47.816 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:47.841 INFO:teuthology.orchestra.run.smithi043.stderr:no valid command found; 10 closest matches: 2024-03-30T13:46:47.842 INFO:teuthology.orchestra.run.smithi043.stderr:0 2024-03-30T13:46:47.842 INFO:teuthology.orchestra.run.smithi043.stderr:1 2024-03-30T13:46:47.842 INFO:teuthology.orchestra.run.smithi043.stderr:2 2024-03-30T13:46:47.842 INFO:teuthology.orchestra.run.smithi043.stderr:abort 2024-03-30T13:46:47.842 INFO:teuthology.orchestra.run.smithi043.stderr:assert 2024-03-30T13:46:47.842 INFO:teuthology.orchestra.run.smithi043.stderr:bench [] [] [] [] 2024-03-30T13:46:47.842 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs debug_inject_read_zeros 2024-03-30T13:46:47.842 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs files list 2024-03-30T13:46:47.842 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs stats 2024-03-30T13:46:47.842 INFO:teuthology.orchestra.run.smithi043.stderr:bluestore allocator dump block 2024-03-30T13:46:47.842 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: invalid command 2024-03-30T13:46:47.843 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:47 smithi043 ceph-mon[29963]: 7.1 deep-scrub starts 2024-03-30T13:46:47.843 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:47 smithi043 ceph-mon[29963]: 5.d scrub starts 2024-03-30T13:46:47.843 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:47 smithi043 ceph-mon[29963]: 5.d scrub ok 2024-03-30T13:46:47.843 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:47 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:46:47.843 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:47 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:46:47.843 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:47 smithi043 ceph-mon[29963]: 7.8 scrub ok 2024-03-30T13:46:47.843 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:47 smithi043 ceph-mon[29963]: 7.9 scrub starts 2024-03-30T13:46:47.843 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:47 smithi043 ceph-mon[42099]: 7.1 deep-scrub starts 2024-03-30T13:46:47.843 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:47 smithi043 ceph-mon[42099]: 5.d scrub starts 2024-03-30T13:46:47.843 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:47 smithi043 ceph-mon[42099]: 5.d scrub ok 2024-03-30T13:46:47.844 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:47 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:46:47.844 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:47 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:46:47.844 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:47 smithi043 ceph-mon[42099]: 7.8 scrub ok 2024-03-30T13:46:47.844 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:47 smithi043 ceph-mon[42099]: 7.9 scrub starts 2024-03-30T13:46:47.860 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:47.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:47.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:47.962 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:47.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:47 smithi113 ceph-mon[34341]: 7.1 deep-scrub starts 2024-03-30T13:46:47.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:47 smithi113 ceph-mon[34341]: 5.d scrub starts 2024-03-30T13:46:47.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:47 smithi113 ceph-mon[34341]: 5.d scrub ok 2024-03-30T13:46:47.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:47 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:46:47.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:47 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:46:47.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:47 smithi113 ceph-mon[34341]: 7.8 scrub ok 2024-03-30T13:46:47.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:47 smithi113 ceph-mon[34341]: 7.9 scrub starts 2024-03-30T13:46:47.999 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:47.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:48.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:48.101 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:48.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:48.145 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:48.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:48.246 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:48.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:48.276 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:48.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:48.377 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:48.428 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:48.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:48.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:48.530 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:48.563 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:48.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:48.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:48.665 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:48.693 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:48.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:48.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:48.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:46:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:46:48.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:48 smithi113 ceph-mon[34341]: pgmap v144: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 637 op/s 2024-03-30T13:46:48.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:48 smithi113 ceph-mon[34341]: 5.1a scrub starts 2024-03-30T13:46:48.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:48 smithi113 ceph-mon[34341]: 5.1a scrub ok 2024-03-30T13:46:48.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:48 smithi113 ceph-mon[34341]: 3.16 deep-scrub starts 2024-03-30T13:46:48.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:48 smithi113 ceph-mon[34341]: 3.16 deep-scrub ok 2024-03-30T13:46:48.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:48 smithi113 ceph-mon[34341]: osdmap e123: 8 total, 8 up, 8 in 2024-03-30T13:46:48.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:48 smithi113 ceph-mon[34341]: 3.7 scrub starts 2024-03-30T13:46:48.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:48 smithi113 ceph-mon[34341]: 3.7 scrub ok 2024-03-30T13:46:48.795 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:48.822 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:48.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:48.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:48.834 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:46:48.835 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-30T13:46:48.835 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 'trim stale osdmaps' 2024-03-30T13:46:48.925 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:48.954 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:48.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:48.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:49.023 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:48 smithi043 ceph-mon[29963]: pgmap v144: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 637 op/s 2024-03-30T13:46:49.024 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:48 smithi043 ceph-mon[29963]: 5.1a scrub starts 2024-03-30T13:46:49.024 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:48 smithi043 ceph-mon[29963]: 5.1a scrub ok 2024-03-30T13:46:49.024 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:48 smithi043 ceph-mon[29963]: 3.16 deep-scrub starts 2024-03-30T13:46:49.024 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:48 smithi043 ceph-mon[29963]: 3.16 deep-scrub ok 2024-03-30T13:46:49.024 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:48 smithi043 ceph-mon[29963]: osdmap e123: 8 total, 8 up, 8 in 2024-03-30T13:46:49.024 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:48 smithi043 ceph-mon[29963]: 3.7 scrub starts 2024-03-30T13:46:49.024 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:48 smithi043 ceph-mon[29963]: 3.7 scrub ok 2024-03-30T13:46:49.024 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:48 smithi043 ceph-mon[42099]: pgmap v144: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 637 op/s 2024-03-30T13:46:49.024 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:48 smithi043 ceph-mon[42099]: 5.1a scrub starts 2024-03-30T13:46:49.025 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:48 smithi043 ceph-mon[42099]: 5.1a scrub ok 2024-03-30T13:46:49.025 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:48 smithi043 ceph-mon[42099]: 3.16 deep-scrub starts 2024-03-30T13:46:49.025 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:48 smithi043 ceph-mon[42099]: 3.16 deep-scrub ok 2024-03-30T13:46:49.025 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:48 smithi043 ceph-mon[42099]: osdmap e123: 8 total, 8 up, 8 in 2024-03-30T13:46:49.025 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:48 smithi043 ceph-mon[42099]: 3.7 scrub starts 2024-03-30T13:46:49.025 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:48 smithi043 ceph-mon[42099]: 3.7 scrub ok 2024-03-30T13:46:49.055 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:49.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:49.114 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:49.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:49.217 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:49.262 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:49.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:49.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:49.365 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:49.405 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:49.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:49.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:49.507 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:49.538 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:49.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:49.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:49.640 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:49.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:49.670 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:49.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:49.771 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:49.780 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:49 smithi043 ceph-mon[29963]: 4.18 scrub starts 2024-03-30T13:46:49.780 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:49 smithi043 ceph-mon[29963]: 4.18 scrub ok 2024-03-30T13:46:49.780 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:49 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:46:49.780 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:49 smithi043 ceph-mon[29963]: 7.4 scrub ok 2024-03-30T13:46:49.780 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:49 smithi043 ceph-mon[29963]: 6.d scrub starts 2024-03-30T13:46:49.780 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:49 smithi043 ceph-mon[29963]: 6.d scrub ok 2024-03-30T13:46:49.780 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:49 smithi043 ceph-mon[29963]: 3.10 scrub starts 2024-03-30T13:46:49.780 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:49 smithi043 ceph-mon[29963]: 3.10 scrub ok 2024-03-30T13:46:49.780 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:49 smithi043 ceph-mon[29963]: 4.19 scrub starts 2024-03-30T13:46:49.780 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:49 smithi043 ceph-mon[29963]: 4.19 scrub ok 2024-03-30T13:46:49.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:49 smithi043 ceph-mon[42099]: 4.18 scrub starts 2024-03-30T13:46:49.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:49 smithi043 ceph-mon[42099]: 4.18 scrub ok 2024-03-30T13:46:49.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:49 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:46:49.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:49 smithi043 ceph-mon[42099]: 7.4 scrub ok 2024-03-30T13:46:49.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:49 smithi043 ceph-mon[42099]: 6.d scrub starts 2024-03-30T13:46:49.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:49 smithi043 ceph-mon[42099]: 6.d scrub ok 2024-03-30T13:46:49.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:49 smithi043 ceph-mon[42099]: 3.10 scrub starts 2024-03-30T13:46:49.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:49 smithi043 ceph-mon[42099]: 3.10 scrub ok 2024-03-30T13:46:49.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:49 smithi043 ceph-mon[42099]: 4.19 scrub starts 2024-03-30T13:46:49.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:49 smithi043 ceph-mon[42099]: 4.19 scrub ok 2024-03-30T13:46:49.807 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:49.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:49.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:49.827 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:46:49.839 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-30T13:46:49.909 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:49.944 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:49.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:49.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:49 smithi113 ceph-mon[34341]: 4.18 scrub starts 2024-03-30T13:46:49.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:49 smithi113 ceph-mon[34341]: 4.18 scrub ok 2024-03-30T13:46:49.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:49 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:46:49.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:49 smithi113 ceph-mon[34341]: 7.4 scrub ok 2024-03-30T13:46:49.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:49 smithi113 ceph-mon[34341]: 6.d scrub starts 2024-03-30T13:46:49.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:49 smithi113 ceph-mon[34341]: 6.d scrub ok 2024-03-30T13:46:49.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:49 smithi113 ceph-mon[34341]: 3.10 scrub starts 2024-03-30T13:46:49.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:49 smithi113 ceph-mon[34341]: 3.10 scrub ok 2024-03-30T13:46:49.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:49 smithi113 ceph-mon[34341]: 4.19 scrub starts 2024-03-30T13:46:49.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:49 smithi113 ceph-mon[34341]: 4.19 scrub ok 2024-03-30T13:46:50.047 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:50.085 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:50.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:50.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:50.164 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:46:50.187 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:50.225 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:50.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:50.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:50.327 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:50.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:50.361 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:50.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:50.462 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:50.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:50.504 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:50.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:50.605 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:50.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:50.638 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:50.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:50.739 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:50.802 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:50.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:50.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:50.904 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:50.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:50.936 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:50.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:50.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:50 smithi113 ceph-mon[34341]: 3.14 scrub starts 2024-03-30T13:46:50.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:50 smithi113 ceph-mon[34341]: 3.14 scrub ok 2024-03-30T13:46:50.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:50 smithi113 ceph-mon[34341]: pgmap v146: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-30T13:46:50.997 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:50 smithi043 ceph-mon[29963]: 3.14 scrub starts 2024-03-30T13:46:50.997 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:50 smithi043 ceph-mon[29963]: 3.14 scrub ok 2024-03-30T13:46:50.997 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:50 smithi043 ceph-mon[29963]: pgmap v146: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-30T13:46:50.997 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:50 smithi043 ceph-mon[42099]: 3.14 scrub starts 2024-03-30T13:46:50.998 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:50 smithi043 ceph-mon[42099]: 3.14 scrub ok 2024-03-30T13:46:50.998 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:50 smithi043 ceph-mon[42099]: pgmap v146: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-30T13:46:51.037 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:51.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:51.090 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:51.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:51.191 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:51.221 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:51.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:51.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:51.323 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:51.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:51.372 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:51.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:51.474 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:51.500 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:51.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:51.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:51.602 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:51.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:51.639 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:51.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:51.740 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:51.781 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:51.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:51.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:51.883 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:51.884 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:51 smithi113 ceph-mon[34341]: 5.1b scrub starts 2024-03-30T13:46:51.884 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:51 smithi113 ceph-mon[34341]: 5.1b scrub ok 2024-03-30T13:46:51.884 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:51 smithi113 ceph-mon[34341]: 7.9 scrub ok 2024-03-30T13:46:51.912 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:51.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:51.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:52.013 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:52.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:52.049 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:52.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:52.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:51 smithi043 ceph-mon[29963]: 5.1b scrub starts 2024-03-30T13:46:52.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:51 smithi043 ceph-mon[29963]: 5.1b scrub ok 2024-03-30T13:46:52.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:51 smithi043 ceph-mon[29963]: 7.9 scrub ok 2024-03-30T13:46:52.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:51 smithi043 ceph-mon[42099]: 5.1b scrub starts 2024-03-30T13:46:52.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:51 smithi043 ceph-mon[42099]: 5.1b scrub ok 2024-03-30T13:46:52.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:51 smithi043 ceph-mon[42099]: 7.9 scrub ok 2024-03-30T13:46:52.150 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:52.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:52.185 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:52.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:52.196 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:46:52.196 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:46:52.286 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:52.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:52.316 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:52.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:52.417 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:52.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:52.460 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:52.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:52.561 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:52.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:52.596 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:52.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:52.697 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:52.728 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:52.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:52.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:52.830 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:52.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:52.897 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:52.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:52 smithi113 ceph-mon[34341]: pgmap v147: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 957 op/s 2024-03-30T13:46:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:52 smithi113 ceph-mon[34341]: 6.7 scrub starts 2024-03-30T13:46:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:52 smithi113 ceph-mon[34341]: 6.7 scrub ok 2024-03-30T13:46:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:52 smithi113 ceph-mon[34341]: 5.2 deep-scrub starts 2024-03-30T13:46:52.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:52 smithi113 ceph-mon[34341]: 5.2 deep-scrub ok 2024-03-30T13:46:52.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:52 smithi113 ceph-mon[34341]: 2.1 deep-scrub starts 2024-03-30T13:46:52.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:52 smithi113 ceph-mon[34341]: 2.1 deep-scrub ok 2024-03-30T13:46:52.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:52 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:46:52.998 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:53.021 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:53.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:53.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:53.123 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:53.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:52 smithi043 ceph-mon[29963]: pgmap v147: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 957 op/s 2024-03-30T13:46:53.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:52 smithi043 ceph-mon[29963]: 6.7 scrub starts 2024-03-30T13:46:53.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:52 smithi043 ceph-mon[29963]: 6.7 scrub ok 2024-03-30T13:46:53.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:52 smithi043 ceph-mon[29963]: 5.2 deep-scrub starts 2024-03-30T13:46:53.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:52 smithi043 ceph-mon[29963]: 5.2 deep-scrub ok 2024-03-30T13:46:53.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:52 smithi043 ceph-mon[29963]: 2.1 deep-scrub starts 2024-03-30T13:46:53.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:52 smithi043 ceph-mon[29963]: 2.1 deep-scrub ok 2024-03-30T13:46:53.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:52 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:46:53.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:52 smithi043 ceph-mon[42099]: pgmap v147: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 957 op/s 2024-03-30T13:46:53.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:52 smithi043 ceph-mon[42099]: 6.7 scrub starts 2024-03-30T13:46:53.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:52 smithi043 ceph-mon[42099]: 6.7 scrub ok 2024-03-30T13:46:53.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:52 smithi043 ceph-mon[42099]: 5.2 deep-scrub starts 2024-03-30T13:46:53.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:52 smithi043 ceph-mon[42099]: 5.2 deep-scrub ok 2024-03-30T13:46:53.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:52 smithi043 ceph-mon[42099]: 2.1 deep-scrub starts 2024-03-30T13:46:53.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:52 smithi043 ceph-mon[42099]: 2.1 deep-scrub ok 2024-03-30T13:46:53.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:52 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:46:53.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:53.157 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:53.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:53.258 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:53.303 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:53.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:53.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:53.405 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:53.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:53.437 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:53.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:53.538 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:53.559 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:46:53.564 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:53.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:53.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:53.665 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:53.671 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:46:53.706 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:53.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:53.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:53.714 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:46:53.757 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:46:53.796 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:46:53.808 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:53.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:53 smithi043 ceph-mon[29963]: 6.19 scrub starts 2024-03-30T13:46:53.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:53 smithi043 ceph-mon[29963]: 6.19 scrub ok 2024-03-30T13:46:53.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:53 smithi043 ceph-mon[29963]: 7.f scrub starts 2024-03-30T13:46:53.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:53 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3531806350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:46:53.831 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:53 smithi043 ceph-mon[42099]: 6.19 scrub starts 2024-03-30T13:46:53.831 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:53 smithi043 ceph-mon[42099]: 6.19 scrub ok 2024-03-30T13:46:53.831 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:53 smithi043 ceph-mon[42099]: 7.f scrub starts 2024-03-30T13:46:53.831 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:53 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3531806350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:46:53.835 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:46:53.836 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:53.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:53.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:53.882 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:46:53.924 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:46:53.937 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:53.964 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:46:53.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:53 smithi113 ceph-mon[34341]: 6.19 scrub starts 2024-03-30T13:46:53.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:53 smithi113 ceph-mon[34341]: 6.19 scrub ok 2024-03-30T13:46:53.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:53 smithi113 ceph-mon[34341]: 7.f scrub starts 2024-03-30T13:46:53.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:53 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3531806350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:46:53.974 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:46:53.975 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:46:53.975 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:46:53.975 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:46:53.975 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:46:53.975 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:46:53.975 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:46:53.975 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:46:53.990 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:53.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:53.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:54.092 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:54.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:54.128 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:54.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:54.230 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:54.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:54.269 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:54.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:54.370 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:54.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:54.402 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:54.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:54.503 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:54.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:54.549 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:54.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:54.650 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:54.675 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-03-30T13:46:54.696 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:54.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:54.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:54.798 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:54.838 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:54.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:54.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:54.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:54 smithi043 ceph-mon[29963]: 3.3 scrub starts 2024-03-30T13:46:54.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:54 smithi043 ceph-mon[29963]: 3.3 scrub ok 2024-03-30T13:46:54.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:54 smithi043 ceph-mon[29963]: pgmap v148: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 885 op/s 2024-03-30T13:46:54.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:54 smithi043 ceph-mon[29963]: 7.6 scrub starts 2024-03-30T13:46:54.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:54 smithi043 ceph-mon[42099]: 3.3 scrub starts 2024-03-30T13:46:54.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:54 smithi043 ceph-mon[42099]: 3.3 scrub ok 2024-03-30T13:46:54.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:54 smithi043 ceph-mon[42099]: pgmap v148: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 885 op/s 2024-03-30T13:46:54.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:54 smithi043 ceph-mon[42099]: 7.6 scrub starts 2024-03-30T13:46:54.940 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:54.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:54 smithi113 ceph-mon[34341]: 3.3 scrub starts 2024-03-30T13:46:54.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:54 smithi113 ceph-mon[34341]: 3.3 scrub ok 2024-03-30T13:46:54.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:54 smithi113 ceph-mon[34341]: pgmap v148: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 885 op/s 2024-03-30T13:46:54.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:54 smithi113 ceph-mon[34341]: 7.6 scrub starts 2024-03-30T13:46:54.976 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:54.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:54.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:55.078 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:55.109 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:55.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:55.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:55.211 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:55.241 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:55.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:55.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:55.343 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:55.373 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:55.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:55.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:55.475 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:55.507 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:55.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:55.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:55.609 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:55.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:55.633 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:55.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:55.680 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:46:55.732 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-03-30T13:46:55.734 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:55.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:55.771 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:55.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:55.872 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:55.897 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:55.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:55.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:55.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:55 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:46:55.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:55 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:46:55.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:55 smithi113 ceph-mon[34341]: 5.11 deep-scrub starts 2024-03-30T13:46:55.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:55 smithi113 ceph-mon[34341]: 5.11 deep-scrub ok 2024-03-30T13:46:55.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:55 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/27637014' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:46:55.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:55 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:46:55.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:55 smithi113 ceph-mon[34341]: 7.f scrub ok 2024-03-30T13:46:55.999 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:56.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:56.039 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:56.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:56.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:55 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:46:56.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:55 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:46:56.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:55 smithi043 ceph-mon[29963]: 5.11 deep-scrub starts 2024-03-30T13:46:56.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:55 smithi043 ceph-mon[29963]: 5.11 deep-scrub ok 2024-03-30T13:46:56.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:55 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/27637014' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:46:56.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:55 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:46:56.098 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:55 smithi043 ceph-mon[29963]: 7.f scrub ok 2024-03-30T13:46:56.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:55 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:46:56.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:55 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:46:56.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:55 smithi043 ceph-mon[42099]: 5.11 deep-scrub starts 2024-03-30T13:46:56.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:55 smithi043 ceph-mon[42099]: 5.11 deep-scrub ok 2024-03-30T13:46:56.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:55 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/27637014' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:46:56.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:55 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:46:56.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:55 smithi043 ceph-mon[42099]: 7.f scrub ok 2024-03-30T13:46:56.140 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:56.176 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:56.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:56.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:56.278 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:56.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:56.314 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:56.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:56.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:46:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:46:56.415 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:56.449 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:56.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:56.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:56.551 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:56.625 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:56.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:56.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:56.727 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:56.759 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:56.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:56.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:56.861 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:56.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-mon[29963]: pgmap v149: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 865 op/s 2024-03-30T13:46:56.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-mon[29963]: 5.12 scrub starts 2024-03-30T13:46:56.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-mon[29963]: 5.12 scrub ok 2024-03-30T13:46:56.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-mon[29963]: 3.5 deep-scrub starts 2024-03-30T13:46:56.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-mon[29963]: 3.5 deep-scrub ok 2024-03-30T13:46:56.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-mon[29963]: 6.5 scrub starts 2024-03-30T13:46:56.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-mon[29963]: 6.5 scrub ok 2024-03-30T13:46:56.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-mon[29963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:46:56.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:46:56.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-mon[29963]: osdmap e124: 8 total, 8 up, 8 in 2024-03-30T13:46:56.903 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-mon[42099]: pgmap v149: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 865 op/s 2024-03-30T13:46:56.903 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-mon[42099]: 5.12 scrub starts 2024-03-30T13:46:56.903 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-mon[42099]: 5.12 scrub ok 2024-03-30T13:46:56.903 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-mon[42099]: 3.5 deep-scrub starts 2024-03-30T13:46:56.903 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-mon[42099]: 3.5 deep-scrub ok 2024-03-30T13:46:56.903 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-mon[42099]: 6.5 scrub starts 2024-03-30T13:46:56.903 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-mon[42099]: 6.5 scrub ok 2024-03-30T13:46:56.903 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-mon[42099]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:46:56.903 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:46:56.903 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:56 smithi043 ceph-mon[42099]: osdmap e124: 8 total, 8 up, 8 in 2024-03-30T13:46:56.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:56.912 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:56.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:56.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:56 smithi113 ceph-mon[34341]: pgmap v149: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 865 op/s 2024-03-30T13:46:56.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:56 smithi113 ceph-mon[34341]: 5.12 scrub starts 2024-03-30T13:46:56.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:56 smithi113 ceph-mon[34341]: 5.12 scrub ok 2024-03-30T13:46:56.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:56 smithi113 ceph-mon[34341]: 3.5 deep-scrub starts 2024-03-30T13:46:56.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:56 smithi113 ceph-mon[34341]: 3.5 deep-scrub ok 2024-03-30T13:46:56.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:56 smithi113 ceph-mon[34341]: 6.5 scrub starts 2024-03-30T13:46:56.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:56 smithi113 ceph-mon[34341]: 6.5 scrub ok 2024-03-30T13:46:56.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:56 smithi113 ceph-mon[34341]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:46:56.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:56 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:46:56.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:56 smithi113 ceph-mon[34341]: osdmap e124: 8 total, 8 up, 8 in 2024-03-30T13:46:57.013 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:57.044 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:57.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:57.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:57.146 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:57.182 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:57.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:57.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:57.284 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:57.314 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:57.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:57.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:57.416 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:57.452 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:57.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:57.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:57.554 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:57.591 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:57.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:57.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:57.693 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:46:57.729 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:57.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:57.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:46:57.831 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:57.863 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:57.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:57.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:57.965 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:57.976 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:46:57.976 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [], 2024-03-30T13:46:57.976 INFO:teuthology.orchestra.run.smithi113.stdout: "complaint_time": 30, 2024-03-30T13:46:57.977 INFO:teuthology.orchestra.run.smithi113.stdout: "num_blocked_ops": 0 2024-03-30T13:46:57.977 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:46:57.998 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:57.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:58.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:58.027 INFO:teuthology.orchestra.run.smithi043.stderr:no valid command found; 10 closest matches: 2024-03-30T13:46:58.027 INFO:teuthology.orchestra.run.smithi043.stderr:0 2024-03-30T13:46:58.028 INFO:teuthology.orchestra.run.smithi043.stderr:1 2024-03-30T13:46:58.028 INFO:teuthology.orchestra.run.smithi043.stderr:2 2024-03-30T13:46:58.028 INFO:teuthology.orchestra.run.smithi043.stderr:abort 2024-03-30T13:46:58.028 INFO:teuthology.orchestra.run.smithi043.stderr:assert 2024-03-30T13:46:58.028 INFO:teuthology.orchestra.run.smithi043.stderr:bench [] [] [] [] 2024-03-30T13:46:58.028 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs debug_inject_read_zeros 2024-03-30T13:46:58.028 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs files list 2024-03-30T13:46:58.028 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs stats 2024-03-30T13:46:58.028 INFO:teuthology.orchestra.run.smithi043.stderr:bluestore allocator dump block 2024-03-30T13:46:58.028 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: invalid command 2024-03-30T13:46:58.101 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:58.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:58.136 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:58.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:58.238 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:58.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:58.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:58.380 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:58.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:58.420 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:58.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:58.521 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:46:58.549 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:58.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:58.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:46:58.608 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:46:58.651 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:58.687 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:58.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:58.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:58.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:46:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:46:58.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:58 smithi113 ceph-mon[34341]: pgmap v151: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 892 op/s 2024-03-30T13:46:58.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:58 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:46:58.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:58 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:46:58.789 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:46:58.836 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:58.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:58.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:46:58.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:58 smithi043 ceph-mon[29963]: pgmap v151: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 892 op/s 2024-03-30T13:46:58.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:58 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:46:58.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:58 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:46:58.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:58 smithi043 ceph-mon[42099]: pgmap v151: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 892 op/s 2024-03-30T13:46:58.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:58 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:46:58.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:58 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:46:58.939 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:58.969 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:58.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:58.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:59.071 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:46:59.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:59.110 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:59.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:46:59.211 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:59.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:59.241 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:59.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:59.342 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:46:59.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:59.399 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:59.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:46:59.500 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:46:59.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:59.526 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:59.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:46:59.627 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:59.628 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T13:46:59.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:59.655 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:59.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:59.756 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:46:59.783 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:46:59.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:59.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:46:59.885 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:46:59.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:46:59 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:46:59.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:46:59 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:46:59.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:46:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:46:59.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:46:59.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:46:59 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:47:00.034 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:00.067 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:00.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:00.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:00.169 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:00.215 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:00.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:00.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:00.317 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:00.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:00.370 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:00.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:00.471 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:00.517 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:00.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:00.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:00.619 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:00.658 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:00.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:00.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:00.760 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:00.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:00.814 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:00.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:00.875 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:00 smithi043 ceph-mon[29963]: pgmap v152: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 864 op/s 2024-03-30T13:47:00.875 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:00 smithi043 ceph-mon[42099]: pgmap v152: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 864 op/s 2024-03-30T13:47:00.915 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:00.916 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:00 smithi113 ceph-mon[34341]: pgmap v152: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 864 op/s 2024-03-30T13:47:00.955 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-03-30T13:47:00.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:00.961 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:00.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:01.061 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:01.104 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:01.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:01.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:01.206 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:01.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:01.248 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:01.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:01.350 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:01.388 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:01.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:01.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:01.490 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:01.519 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:01.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:01.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:01.621 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:01.654 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:01.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:01.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:01.702 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:47:01.756 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:01.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:01.789 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:01.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:01.890 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:01.916 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:01.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:01.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:02.018 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:02.057 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:02.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:02.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:02.159 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:02.198 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:02.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:02.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:02.303 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:02.333 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:02.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:02.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:02.435 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:02.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:02.469 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:02.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:02.571 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:02.604 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:02.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:02.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:02.706 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:02.743 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:02.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:02.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:02.791 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:02 smithi043 ceph-mon[29963]: pgmap v153: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 864 op/s 2024-03-30T13:47:02.791 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:02 smithi043 ceph-mon[42099]: pgmap v153: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 864 op/s 2024-03-30T13:47:02.841 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:47:02.842 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-30T13:47:02.842 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 'trim stale osdmaps' 2024-03-30T13:47:02.845 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:02.881 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:02.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:02.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:02.893 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:47:02.947 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:47:02.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:02 smithi113 ceph-mon[34341]: pgmap v153: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 864 op/s 2024-03-30T13:47:02.983 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:03.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:03.047 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:03.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:03.148 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:03.176 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:03.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:03.278 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:03.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:03.305 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:03.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:03.406 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:03.445 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:03.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:03.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:03.548 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:03.609 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:03.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:03.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:03.711 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:03.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:03.757 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:03.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:03.858 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:03.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:03.899 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:03.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:04.001 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:04.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:04.041 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:04.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:04.142 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:04.173 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:04.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:04.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:04.275 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:04.309 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:04.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:04.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:04.411 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:04.438 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:04.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:04.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:04.540 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:04.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:04.574 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:04.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:04.675 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:04.704 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:04.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:04.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:04.798 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:04 smithi043 ceph-mon[29963]: pgmap v154: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 778 op/s 2024-03-30T13:47:04.798 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:04 smithi043 ceph-mon[42099]: pgmap v154: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 778 op/s 2024-03-30T13:47:04.806 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:04.833 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:04.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:04.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:04.935 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:04.968 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:04.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:04.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:04.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:04 smithi113 ceph-mon[34341]: pgmap v154: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 778 op/s 2024-03-30T13:47:05.071 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:05.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:05.111 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:05.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:05.212 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:05.237 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:47:05.237 INFO:teuthology.orchestra.run.smithi113.stdout: "size": 20, 2024-03-30T13:47:05.237 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 600, 2024-03-30T13:47:05.237 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [ 2024-03-30T13:47:05.237 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.237 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:145830 7.1 7:878d945b:::benchmark_data_smithi043_86989_object23492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-30T13:47:05.237 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:55.519173+0000", 2024-03-30T13:47:05.238 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 9.5587243050000001, 2024-03-30T13:47:05.238 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.208673674, 2024-03-30T13:47:05.238 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:05.238 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:05.238 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:05.238 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:05.238 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:05.238 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:05.238 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 145830 2024-03-30T13:47:05.238 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.238 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:05.238 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.238 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:05.239 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.519173+0000", 2024-03-30T13:47:05.239 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.239 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.239 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.239 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:05.239 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.519173+0000", 2024-03-30T13:47:05.239 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.239 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.239 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.239 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:05.239 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.519174+0000", 2024-03-30T13:47:05.240 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.347e-06 2024-03-30T13:47:05.240 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.240 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.240 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:05.240 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.519180+0000", 2024-03-30T13:47:05.240 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6160000000000001e-06 2024-03-30T13:47:05.240 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.240 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.240 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:05.240 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.519193+0000", 2024-03-30T13:47:05.240 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3038e-05 2024-03-30T13:47:05.240 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.240 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.241 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:05.241 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.519200+0000", 2024-03-30T13:47:05.241 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.6369999999999999e-06 2024-03-30T13:47:05.241 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.241 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.241 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:05.241 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.548091+0000", 2024-03-30T13:47:05.241 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.028891212999999999 2024-03-30T13:47:05.241 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.241 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.241 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:05.241 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.548219+0000", 2024-03-30T13:47:05.241 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000127086 2024-03-30T13:47:05.241 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.242 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.242 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,6", 2024-03-30T13:47:05.242 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.548359+0000", 2024-03-30T13:47:05.242 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00014056999999999999 2024-03-30T13:47:05.242 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.242 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.242 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:05.242 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.549528+0000", 2024-03-30T13:47:05.242 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0011689560000000001 2024-03-30T13:47:05.242 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.242 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.242 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.243 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.549647+0000", 2024-03-30T13:47:05.243 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011864800000000001 2024-03-30T13:47:05.243 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.243 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.243 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.243 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.727650+0000", 2024-03-30T13:47:05.243 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.17800371300000001 2024-03-30T13:47:05.243 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.243 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.243 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:05.243 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.727771+0000", 2024-03-30T13:47:05.243 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000120241 2024-03-30T13:47:05.243 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.243 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.244 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:05.244 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.727846+0000", 2024-03-30T13:47:05.244 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.5609000000000001e-05 2024-03-30T13:47:05.244 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.244 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.244 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.244 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.244 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.244 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:149144 7.1 7:8af65c65:::benchmark_data_smithi043_86989_object26806:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-30T13:47:05.244 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:59.185466+0000", 2024-03-30T13:47:05.244 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 5.8924308669999999, 2024-03-30T13:47:05.244 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.12234450299999999, 2024-03-30T13:47:05.244 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:05.244 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:05.245 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:05.245 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:05.245 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:05.245 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:05.245 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 149144 2024-03-30T13:47:05.245 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.245 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:05.245 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.245 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:05.245 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.185466+0000", 2024-03-30T13:47:05.245 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.245 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.245 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.245 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:05.245 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.185466+0000", 2024-03-30T13:47:05.246 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.246 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.246 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.246 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:05.246 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.185467+0000", 2024-03-30T13:47:05.246 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2500000000000001e-06 2024-03-30T13:47:05.246 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.246 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.246 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:05.246 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.185472+0000", 2024-03-30T13:47:05.246 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7809999999999996e-06 2024-03-30T13:47:05.246 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.246 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.247 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:05.247 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.185482+0000", 2024-03-30T13:47:05.247 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0278000000000001e-05 2024-03-30T13:47:05.247 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.247 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.247 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:05.247 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.185488+0000", 2024-03-30T13:47:05.247 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9479999999999996e-06 2024-03-30T13:47:05.247 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.247 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.247 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:05.247 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.234786+0000", 2024-03-30T13:47:05.247 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.049297464999999999 2024-03-30T13:47:05.247 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.248 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.248 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:05.248 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.234863+0000", 2024-03-30T13:47:05.248 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.7142000000000001e-05 2024-03-30T13:47:05.248 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.248 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.248 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,6", 2024-03-30T13:47:05.249 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.234940+0000", 2024-03-30T13:47:05.249 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.7148000000000001e-05 2024-03-30T13:47:05.249 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.249 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.249 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:05.249 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.240535+0000", 2024-03-30T13:47:05.249 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0055947269999999999 2024-03-30T13:47:05.249 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.249 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.250 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.250 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.240586+0000", 2024-03-30T13:47:05.250 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1051000000000003e-05 2024-03-30T13:47:05.250 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.250 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.250 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.250 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.307710+0000", 2024-03-30T13:47:05.250 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.067123882999999995 2024-03-30T13:47:05.250 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.250 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.250 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:05.250 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.307746+0000", 2024-03-30T13:47:05.250 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.6026999999999999e-05 2024-03-30T13:47:05.250 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.251 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.251 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:05.251 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.307811+0000", 2024-03-30T13:47:05.251 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.4802999999999996e-05 2024-03-30T13:47:05.251 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.251 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.251 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.251 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.251 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.251 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:149270 7.1 7:81f22dab:::benchmark_data_smithi043_86989_object26932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-30T13:47:05.251 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:59.361756+0000", 2024-03-30T13:47:05.251 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 5.7161409389999998, 2024-03-30T13:47:05.251 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.22454207000000001, 2024-03-30T13:47:05.251 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:05.252 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:05.252 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:05.252 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:05.252 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:05.252 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:05.252 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 149270 2024-03-30T13:47:05.252 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.252 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:05.252 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.252 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:05.252 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.361756+0000", 2024-03-30T13:47:05.252 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.252 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.252 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.253 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:05.253 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.361756+0000", 2024-03-30T13:47:05.253 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.253 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.253 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.253 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:05.253 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.361757+0000", 2024-03-30T13:47:05.253 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2950000000000001e-06 2024-03-30T13:47:05.253 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.253 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.253 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:05.253 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.361762+0000", 2024-03-30T13:47:05.253 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0980000000000001e-06 2024-03-30T13:47:05.253 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.253 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.254 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:05.254 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.361772+0000", 2024-03-30T13:47:05.254 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0061000000000001e-05 2024-03-30T13:47:05.254 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.254 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.254 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:05.254 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.361780+0000", 2024-03-30T13:47:05.254 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.0949999999999998e-06 2024-03-30T13:47:05.254 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.254 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.254 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:05.254 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.375380+0000", 2024-03-30T13:47:05.254 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.013600111999999999 2024-03-30T13:47:05.255 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.255 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.255 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:05.255 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.375496+0000", 2024-03-30T13:47:05.255 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000116358 2024-03-30T13:47:05.255 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.255 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.255 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,6", 2024-03-30T13:47:05.255 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.375578+0000", 2024-03-30T13:47:05.255 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.1959999999999995e-05 2024-03-30T13:47:05.255 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.255 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.255 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:05.256 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.377992+0000", 2024-03-30T13:47:05.256 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0024139309999999998 2024-03-30T13:47:05.256 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.256 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.256 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.256 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.378212+0000", 2024-03-30T13:47:05.256 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00022033400000000001 2024-03-30T13:47:05.256 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.256 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.256 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.256 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586161+0000", 2024-03-30T13:47:05.256 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.20794913300000001 2024-03-30T13:47:05.256 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.256 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.257 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:05.257 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586220+0000", 2024-03-30T13:47:05.257 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9134e-05 2024-03-30T13:47:05.257 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.257 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.257 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:05.257 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586298+0000", 2024-03-30T13:47:05.257 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.7558999999999994e-05 2024-03-30T13:47:05.257 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.257 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.257 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.257 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.257 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.257 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:149292 7.1 7:8a2438a2:::benchmark_data_smithi043_86989_object26954:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-30T13:47:05.258 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:59.384771+0000", 2024-03-30T13:47:05.258 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 5.6931260750000003, 2024-03-30T13:47:05.258 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.20168697599999999, 2024-03-30T13:47:05.258 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:05.258 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:05.258 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:05.258 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:05.258 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:05.258 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:05.258 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 149292 2024-03-30T13:47:05.258 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.258 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:05.262 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.262 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:05.263 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.384771+0000", 2024-03-30T13:47:05.263 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.263 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.264 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.264 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:05.264 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.384771+0000", 2024-03-30T13:47:05.264 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.264 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.264 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.264 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:05.264 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.384772+0000", 2024-03-30T13:47:05.265 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.469e-06 2024-03-30T13:47:05.265 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.265 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.265 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:05.265 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.384777+0000", 2024-03-30T13:47:05.265 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1869999999999996e-06 2024-03-30T13:47:05.265 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.265 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.266 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:05.266 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.384796+0000", 2024-03-30T13:47:05.266 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.8142999999999999e-05 2024-03-30T13:47:05.266 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.266 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.266 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:05.266 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.384803+0000", 2024-03-30T13:47:05.266 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8280000000000003e-06 2024-03-30T13:47:05.267 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.267 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.267 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:05.267 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.384874+0000", 2024-03-30T13:47:05.267 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.0726e-05 2024-03-30T13:47:05.267 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.267 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.267 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:05.268 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.384996+0000", 2024-03-30T13:47:05.268 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000121827 2024-03-30T13:47:05.268 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.268 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.268 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,6", 2024-03-30T13:47:05.268 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.385106+0000", 2024-03-30T13:47:05.268 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000110433 2024-03-30T13:47:05.268 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.269 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.269 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.269 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.388485+0000", 2024-03-30T13:47:05.269 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.003378079 2024-03-30T13:47:05.269 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.269 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.269 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:05.269 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.388598+0000", 2024-03-30T13:47:05.269 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011331 2024-03-30T13:47:05.269 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.269 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.270 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.270 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586346+0000", 2024-03-30T13:47:05.270 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.19774828699999999 2024-03-30T13:47:05.270 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.270 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.270 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:05.270 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586397+0000", 2024-03-30T13:47:05.270 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1258999999999998e-05 2024-03-30T13:47:05.270 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.270 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.270 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:05.270 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586458+0000", 2024-03-30T13:47:05.271 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0427999999999998e-05 2024-03-30T13:47:05.271 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.271 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.271 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.271 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.271 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.271 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:149307 7.1 7:879cb2e1:::benchmark_data_smithi043_86989_object26969:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-30T13:47:05.271 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:59.404139+0000", 2024-03-30T13:47:05.271 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 5.6737574930000001, 2024-03-30T13:47:05.271 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.18264171400000001, 2024-03-30T13:47:05.271 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:05.271 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:05.271 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:05.272 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:05.272 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:05.272 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:05.272 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 149307 2024-03-30T13:47:05.272 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.272 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:05.272 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.272 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:05.272 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.404139+0000", 2024-03-30T13:47:05.272 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.272 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.272 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.273 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:05.273 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.404139+0000", 2024-03-30T13:47:05.273 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.273 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.273 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.273 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:05.273 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.404141+0000", 2024-03-30T13:47:05.274 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.212e-06 2024-03-30T13:47:05.274 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.274 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.274 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:05.274 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.404146+0000", 2024-03-30T13:47:05.274 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8930000000000004e-06 2024-03-30T13:47:05.274 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.274 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.274 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:05.275 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.404155+0000", 2024-03-30T13:47:05.275 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.3109999999999995e-06 2024-03-30T13:47:05.275 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.275 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.275 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:05.275 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.404161+0000", 2024-03-30T13:47:05.275 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5089999999999999e-06 2024-03-30T13:47:05.276 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.276 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.276 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:05.276 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.406697+0000", 2024-03-30T13:47:05.276 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0025355909999999998 2024-03-30T13:47:05.276 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.276 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.276 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:05.276 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.406784+0000", 2024-03-30T13:47:05.277 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.7532000000000002e-05 2024-03-30T13:47:05.277 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.277 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.277 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,6", 2024-03-30T13:47:05.277 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.406904+0000", 2024-03-30T13:47:05.277 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000120006 2024-03-30T13:47:05.277 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.277 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.277 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:05.277 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.408987+0000", 2024-03-30T13:47:05.277 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0020828370000000001 2024-03-30T13:47:05.277 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.277 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.277 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.278 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.409575+0000", 2024-03-30T13:47:05.278 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00058733699999999995 2024-03-30T13:47:05.278 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.278 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.278 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.278 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586675+0000", 2024-03-30T13:47:05.278 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.177100282 2024-03-30T13:47:05.278 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.278 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.278 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:05.278 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586733+0000", 2024-03-30T13:47:05.278 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8150999999999999e-05 2024-03-30T13:47:05.279 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.279 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.279 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:05.279 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586781+0000", 2024-03-30T13:47:05.279 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8053000000000002e-05 2024-03-30T13:47:05.279 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.279 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.279 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.279 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.279 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.279 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:149314 7.1 7:8fb66399:::benchmark_data_smithi043_86989_object26976:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-30T13:47:05.279 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:59.410425+0000", 2024-03-30T13:47:05.279 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 5.6674714819999998, 2024-03-30T13:47:05.279 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.176554767, 2024-03-30T13:47:05.280 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:05.280 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:05.280 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:05.280 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:05.280 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:05.280 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:05.280 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 149314 2024-03-30T13:47:05.280 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.280 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:05.280 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.280 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:05.280 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.410425+0000", 2024-03-30T13:47:05.280 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.280 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.281 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.281 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:05.281 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.410425+0000", 2024-03-30T13:47:05.281 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.281 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.281 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.281 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:05.281 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.410427+0000", 2024-03-30T13:47:05.281 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.144e-06 2024-03-30T13:47:05.281 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.281 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.281 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:05.281 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.410431+0000", 2024-03-30T13:47:05.281 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3769999999999998e-06 2024-03-30T13:47:05.281 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.282 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.282 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:05.282 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.410440+0000", 2024-03-30T13:47:05.282 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.3309999999999993e-06 2024-03-30T13:47:05.282 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.282 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.282 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:05.282 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.410446+0000", 2024-03-30T13:47:05.282 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.648e-06 2024-03-30T13:47:05.282 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.282 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.282 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:05.283 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.410483+0000", 2024-03-30T13:47:05.283 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.7449e-05 2024-03-30T13:47:05.283 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.283 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.283 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:05.283 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.410562+0000", 2024-03-30T13:47:05.283 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8532e-05 2024-03-30T13:47:05.283 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.283 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.283 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,6", 2024-03-30T13:47:05.283 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.410633+0000", 2024-03-30T13:47:05.283 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1116000000000001e-05 2024-03-30T13:47:05.283 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.284 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.284 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:05.284 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.412605+0000", 2024-03-30T13:47:05.284 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0019715900000000001 2024-03-30T13:47:05.284 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.284 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.284 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.284 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.413316+0000", 2024-03-30T13:47:05.284 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00071127300000000001 2024-03-30T13:47:05.284 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.284 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.284 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.284 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586873+0000", 2024-03-30T13:47:05.284 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.17355693699999999 2024-03-30T13:47:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586928+0000", 2024-03-30T13:47:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5597000000000003e-05 2024-03-30T13:47:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586980+0000", 2024-03-30T13:47:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1773e-05 2024-03-30T13:47:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.285 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop_reply(client.14817.0:149270 7.1 e124/98 ondisk, result = 0)", 2024-03-30T13:47:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:59.426031+0000", 2024-03-30T13:47:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 5.6518654829999999, 2024-03-30T13:47:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.16027376300000001, 2024-03-30T13:47:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:47:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.426031+0000", 2024-03-30T13:47:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.286 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.426031+0000", 2024-03-30T13:47:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.426032+0000", 2024-03-30T13:47:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1200000000000003e-07 2024-03-30T13:47:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.426036+0000", 2024-03-30T13:47:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.9160000000000003e-06 2024-03-30T13:47:05.287 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.288 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.288 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:05.288 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.426044+0000", 2024-03-30T13:47:05.288 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6510000000000007e-06 2024-03-30T13:47:05.288 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.288 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.288 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:05.288 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.426050+0000", 2024-03-30T13:47:05.288 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7440000000000004e-06 2024-03-30T13:47:05.288 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.288 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.288 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:05.288 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586145+0000", 2024-03-30T13:47:05.288 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.160094344 2024-03-30T13:47:05.289 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.289 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.289 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:05.289 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586154+0000", 2024-03-30T13:47:05.289 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.8069999999999994e-06 2024-03-30T13:47:05.289 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.289 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.289 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:05.289 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586305+0000", 2024-03-30T13:47:05.289 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000150789 2024-03-30T13:47:05.289 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.289 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.289 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.289 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.289 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.290 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop(client.14817.0:149328 7.4 e124/98 7:20b7bbb0:::benchmark_data_smithi043_86989_object26990:head v 124'9576, mlcod=124'9574)", 2024-03-30T13:47:05.290 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:59.427109+0000", 2024-03-30T13:47:05.290 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 5.6507877249999998, 2024-03-30T13:47:05.290 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.115076609, 2024-03-30T13:47:05.290 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:05.290 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:05.290 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:05.290 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:05.290 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.290 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:05.290 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.427109+0000", 2024-03-30T13:47:05.290 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.291 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.291 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.291 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:05.291 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.427109+0000", 2024-03-30T13:47:05.291 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.291 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.291 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.291 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:05.291 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.427110+0000", 2024-03-30T13:47:05.291 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.9100000000000003e-07 2024-03-30T13:47:05.291 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.291 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.291 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:05.291 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.427116+0000", 2024-03-30T13:47:05.291 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7230000000000001e-06 2024-03-30T13:47:05.292 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.292 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.292 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:05.292 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.427133+0000", 2024-03-30T13:47:05.292 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.7254000000000001e-05 2024-03-30T13:47:05.292 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.292 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.292 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:05.292 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.427140+0000", 2024-03-30T13:47:05.292 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8600000000000004e-06 2024-03-30T13:47:05.292 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.292 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.292 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:05.292 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.427188+0000", 2024-03-30T13:47:05.293 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8137999999999999e-05 2024-03-30T13:47:05.293 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.293 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.293 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:05.293 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.427201+0000", 2024-03-30T13:47:05.293 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3379e-05 2024-03-30T13:47:05.293 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.293 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.293 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:05.293 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.542141+0000", 2024-03-30T13:47:05.293 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.11493935500000001 2024-03-30T13:47:05.293 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.294 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.294 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:05.294 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.542186+0000", 2024-03-30T13:47:05.294 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.5108999999999998e-05 2024-03-30T13:47:05.294 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.294 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.294 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.295 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.295 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.295 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop_reply(client.14817.0:149292 7.1 e124/98 ondisk, result = 0)", 2024-03-30T13:47:05.295 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:59.428187+0000", 2024-03-30T13:47:05.295 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 5.649709595, 2024-03-30T13:47:05.295 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15837279700000001, 2024-03-30T13:47:05.295 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:05.296 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:05.296 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:47:05.296 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:05.296 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.296 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:05.296 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428187+0000", 2024-03-30T13:47:05.296 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.296 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.297 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.297 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:05.297 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428187+0000", 2024-03-30T13:47:05.297 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.297 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.297 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.297 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:05.298 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428188+0000", 2024-03-30T13:47:05.298 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0999999999999998e-07 2024-03-30T13:47:05.298 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.298 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.298 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:05.298 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428193+0000", 2024-03-30T13:47:05.299 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6399999999999996e-06 2024-03-30T13:47:05.299 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.299 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.299 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:05.299 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428201+0000", 2024-03-30T13:47:05.299 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.4719999999999995e-06 2024-03-30T13:47:05.300 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.300 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.300 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:05.300 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428207+0000", 2024-03-30T13:47:05.300 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.2149999999999998e-06 2024-03-30T13:47:05.300 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.300 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.301 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:05.301 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586336+0000", 2024-03-30T13:47:05.301 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15812916299999999 2024-03-30T13:47:05.301 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.301 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.301 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:05.301 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586342+0000", 2024-03-30T13:47:05.302 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.113e-06 2024-03-30T13:47:05.302 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.302 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.302 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:05.302 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586560+0000", 2024-03-30T13:47:05.302 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00021858399999999999 2024-03-30T13:47:05.302 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.302 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.303 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.303 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.303 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.303 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop_reply(client.14817.0:149307 7.1 e124/98 ondisk, result = 0)", 2024-03-30T13:47:05.303 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:59.428233+0000", 2024-03-30T13:47:05.303 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 5.6496636340000004, 2024-03-30T13:47:05.303 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15855548999999999, 2024-03-30T13:47:05.304 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:05.304 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:05.304 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:47:05.304 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:05.304 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.304 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:05.305 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428233+0000", 2024-03-30T13:47:05.305 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.305 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.305 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.305 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:05.305 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428233+0000", 2024-03-30T13:47:05.306 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.306 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.306 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.306 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:05.306 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428234+0000", 2024-03-30T13:47:05.306 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.1399999999999998e-07 2024-03-30T13:47:05.306 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.306 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.306 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:05.306 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428236+0000", 2024-03-30T13:47:05.307 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.7360000000000001e-06 2024-03-30T13:47:05.307 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.307 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.307 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:05.307 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428242+0000", 2024-03-30T13:47:05.307 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8690000000000001e-06 2024-03-30T13:47:05.307 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.307 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.307 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:05.307 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428247+0000", 2024-03-30T13:47:05.307 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.5040000000000004e-06 2024-03-30T13:47:05.307 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.307 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.307 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:05.308 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586658+0000", 2024-03-30T13:47:05.308 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.158411093 2024-03-30T13:47:05.308 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.308 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.308 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:05.308 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586667+0000", 2024-03-30T13:47:05.308 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.6889999999999999e-06 2024-03-30T13:47:05.308 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.308 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.308 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:05.308 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586789+0000", 2024-03-30T13:47:05.308 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000121285 2024-03-30T13:47:05.308 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.308 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.308 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.309 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.309 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.309 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop_reply(client.14817.0:149314 7.1 e124/98 ondisk, result = 0)", 2024-03-30T13:47:05.309 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:59.428259+0000", 2024-03-30T13:47:05.309 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 5.6496383479999999, 2024-03-30T13:47:05.309 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15872702899999999, 2024-03-30T13:47:05.309 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:05.309 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:05.309 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:47:05.309 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:05.309 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.309 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:05.309 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428259+0000", 2024-03-30T13:47:05.309 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.310 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.310 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.310 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:05.310 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428259+0000", 2024-03-30T13:47:05.310 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.310 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.310 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.310 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:05.310 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428259+0000", 2024-03-30T13:47:05.310 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.6199999999999999e-07 2024-03-30T13:47:05.310 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.310 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.311 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:05.311 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428261+0000", 2024-03-30T13:47:05.311 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.0150000000000002e-06 2024-03-30T13:47:05.311 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.311 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.311 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:05.311 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428266+0000", 2024-03-30T13:47:05.311 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8729999999999998e-06 2024-03-30T13:47:05.311 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.311 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.311 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:05.311 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428270+0000", 2024-03-30T13:47:05.311 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.0999999999999997e-06 2024-03-30T13:47:05.312 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.312 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.312 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:05.312 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586859+0000", 2024-03-30T13:47:05.312 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.158588751 2024-03-30T13:47:05.312 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.312 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.312 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:05.312 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586866+0000", 2024-03-30T13:47:05.312 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.6920000000000002e-06 2024-03-30T13:47:05.312 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.312 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.312 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:05.313 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586986+0000", 2024-03-30T13:47:05.313 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000119336 2024-03-30T13:47:05.313 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.313 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.313 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.313 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.313 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.313 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop_reply(osd.4.0:12289 7.1 e124/98 ondisk, result = 0)", 2024-03-30T13:47:05.313 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:59.428490+0000", 2024-03-30T13:47:05.313 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 5.6494070750000001, 2024-03-30T13:47:05.313 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.158128245, 2024-03-30T13:47:05.313 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:05.313 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:05.313 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:47:05.314 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:05.314 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.314 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:05.314 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428490+0000", 2024-03-30T13:47:05.314 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.314 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.314 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.314 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:05.314 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428490+0000", 2024-03-30T13:47:05.314 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.314 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.314 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.314 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:05.315 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428490+0000", 2024-03-30T13:47:05.315 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.2500000000000005e-07 2024-03-30T13:47:05.315 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.315 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.315 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:05.315 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.428495+0000", 2024-03-30T13:47:05.315 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3810000000000001e-06 2024-03-30T13:47:05.315 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.315 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.315 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:05.316 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.538023+0000", 2024-03-30T13:47:05.316 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.10952796099999999 2024-03-30T13:47:05.316 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.316 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.316 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:05.316 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.538037+0000", 2024-03-30T13:47:05.316 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4381e-05 2024-03-30T13:47:05.316 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.316 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.316 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:05.316 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586591+0000", 2024-03-30T13:47:05.317 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.048553698999999999 2024-03-30T13:47:05.317 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.317 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.317 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:05.317 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586599+0000", 2024-03-30T13:47:05.317 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.7929999999999998e-06 2024-03-30T13:47:05.317 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.317 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.317 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:05.317 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.586618+0000", 2024-03-30T13:47:05.317 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.9405000000000001e-05 2024-03-30T13:47:05.317 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.317 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.317 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.318 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.318 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.318 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:149443 7.1 7:80ee8e36:::benchmark_data_smithi043_86989_object27105:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-30T13:47:05.318 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:59.623206+0000", 2024-03-30T13:47:05.318 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 5.4546910689999999, 2024-03-30T13:47:05.318 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.11529853599999999, 2024-03-30T13:47:05.318 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:05.318 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:05.318 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:05.319 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:05.319 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:05.319 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:05.319 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 149443 2024-03-30T13:47:05.319 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.319 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:05.319 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.319 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:05.319 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.623206+0000", 2024-03-30T13:47:05.319 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.319 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.319 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.319 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:05.319 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.623206+0000", 2024-03-30T13:47:05.320 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.320 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.320 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.320 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:05.320 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.623206+0000", 2024-03-30T13:47:05.320 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8999999999999996e-07 2024-03-30T13:47:05.320 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.320 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.320 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:05.320 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.623210+0000", 2024-03-30T13:47:05.320 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.4089999999999999e-06 2024-03-30T13:47:05.320 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.320 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.320 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:05.321 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.623216+0000", 2024-03-30T13:47:05.321 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6409999999999996e-06 2024-03-30T13:47:05.321 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.321 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.321 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:05.321 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.623228+0000", 2024-03-30T13:47:05.321 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1497e-05 2024-03-30T13:47:05.321 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.321 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.321 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:05.321 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.631813+0000", 2024-03-30T13:47:05.321 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0085850690000000007 2024-03-30T13:47:05.322 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.322 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.322 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:05.322 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.631933+0000", 2024-03-30T13:47:05.322 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011985399999999999 2024-03-30T13:47:05.322 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.322 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.322 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,6", 2024-03-30T13:47:05.322 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.632010+0000", 2024-03-30T13:47:05.322 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.6790000000000002e-05 2024-03-30T13:47:05.322 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.322 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.322 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.323 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.636700+0000", 2024-03-30T13:47:05.323 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0046907490000000001 2024-03-30T13:47:05.323 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.323 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.323 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:05.323 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.647325+0000", 2024-03-30T13:47:05.323 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.010624197 2024-03-30T13:47:05.323 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.323 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.323 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.323 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.738391+0000", 2024-03-30T13:47:05.323 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.091066616000000003 2024-03-30T13:47:05.323 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.324 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.324 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:05.324 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.738444+0000", 2024-03-30T13:47:05.324 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3186999999999997e-05 2024-03-30T13:47:05.324 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.324 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.324 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:05.324 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.738504+0000", 2024-03-30T13:47:05.324 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9936999999999999e-05 2024-03-30T13:47:05.324 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.324 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.324 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.325 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.325 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.325 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:150064 7.1 7:81161842:::benchmark_data_smithi043_86989_object27726:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-30T13:47:05.325 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:00.330187+0000", 2024-03-30T13:47:05.325 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 4.7477094549999999, 2024-03-30T13:47:05.325 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.13588318499999999, 2024-03-30T13:47:05.325 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:05.325 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:05.325 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:05.325 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:05.325 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:05.325 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:05.326 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 150064 2024-03-30T13:47:05.326 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.326 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:05.326 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.326 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:05.326 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.330187+0000", 2024-03-30T13:47:05.326 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.326 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.326 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.326 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:05.326 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.330187+0000", 2024-03-30T13:47:05.326 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.327 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.327 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.327 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:05.327 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.330189+0000", 2024-03-30T13:47:05.327 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.48e-06 2024-03-30T13:47:05.327 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.327 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.327 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:05.327 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.330195+0000", 2024-03-30T13:47:05.327 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6189999999999999e-06 2024-03-30T13:47:05.327 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.327 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.327 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:05.328 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.330211+0000", 2024-03-30T13:47:05.328 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.6067e-05 2024-03-30T13:47:05.328 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.328 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.328 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:05.328 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.330218+0000", 2024-03-30T13:47:05.328 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.7409999999999992e-06 2024-03-30T13:47:05.328 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.328 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.328 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:05.328 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.391449+0000", 2024-03-30T13:47:05.328 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.061230672 2024-03-30T13:47:05.328 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.328 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.329 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:05.329 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.391556+0000", 2024-03-30T13:47:05.329 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000107227 2024-03-30T13:47:05.329 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.329 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.329 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,6", 2024-03-30T13:47:05.329 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.391681+0000", 2024-03-30T13:47:05.329 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012499200000000001 2024-03-30T13:47:05.329 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.329 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.329 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.329 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.408613+0000", 2024-03-30T13:47:05.329 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.016931616 2024-03-30T13:47:05.329 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.330 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.330 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:05.330 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.408667+0000", 2024-03-30T13:47:05.330 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4214e-05 2024-03-30T13:47:05.330 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.330 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.330 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.330 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.465944+0000", 2024-03-30T13:47:05.330 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.057276729999999998 2024-03-30T13:47:05.330 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.330 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.330 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:05.330 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.466005+0000", 2024-03-30T13:47:05.330 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1680000000000006e-05 2024-03-30T13:47:05.331 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.331 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.331 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:05.331 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.466071+0000", 2024-03-30T13:47:05.331 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5147000000000005e-05 2024-03-30T13:47:05.331 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.331 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.331 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.331 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.331 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.331 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:150233 7.1 7:8a2801ed:::benchmark_data_smithi043_86989_object27895:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-30T13:47:05.331 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:00.536258+0000", 2024-03-30T13:47:05.331 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 4.5416389200000005, 2024-03-30T13:47:05.331 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.12240862399999999, 2024-03-30T13:47:05.331 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:05.332 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:05.332 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:05.332 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:05.332 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:05.332 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:05.332 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 150233 2024-03-30T13:47:05.332 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.332 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:05.332 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.332 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:05.332 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.536258+0000", 2024-03-30T13:47:05.332 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.332 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.332 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.333 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:05.333 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.536258+0000", 2024-03-30T13:47:05.333 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.333 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.333 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.333 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:05.333 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.536259+0000", 2024-03-30T13:47:05.333 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5030000000000001e-06 2024-03-30T13:47:05.333 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.333 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.333 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:05.333 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.536266+0000", 2024-03-30T13:47:05.333 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3829999999999997e-06 2024-03-30T13:47:05.334 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.334 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.334 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:05.334 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.536278+0000", 2024-03-30T13:47:05.334 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2614e-05 2024-03-30T13:47:05.334 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.334 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.334 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:05.334 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.536288+0000", 2024-03-30T13:47:05.334 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.7389999999999994e-06 2024-03-30T13:47:05.334 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.335 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.335 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:05.335 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.583106+0000", 2024-03-30T13:47:05.335 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.046818143 2024-03-30T13:47:05.335 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.335 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.335 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:05.335 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.583192+0000", 2024-03-30T13:47:05.335 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5185999999999994e-05 2024-03-30T13:47:05.335 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.335 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.335 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,6", 2024-03-30T13:47:05.335 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.583281+0000", 2024-03-30T13:47:05.335 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.9806e-05 2024-03-30T13:47:05.336 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.336 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.336 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:05.336 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.585200+0000", 2024-03-30T13:47:05.336 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0019187620000000001 2024-03-30T13:47:05.336 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.336 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.336 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.336 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.586893+0000", 2024-03-30T13:47:05.337 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0016924780000000001 2024-03-30T13:47:05.337 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.337 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.337 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.337 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.658554+0000", 2024-03-30T13:47:05.337 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.071661013999999995 2024-03-30T13:47:05.337 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.337 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.338 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:05.338 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.658610+0000", 2024-03-30T13:47:05.338 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6569e-05 2024-03-30T13:47:05.338 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.338 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.338 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:05.338 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:00.658667+0000", 2024-03-30T13:47:05.338 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6427e-05 2024-03-30T13:47:05.338 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.338 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.338 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.338 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.339 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.339 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:150965 7.1 7:88c85f82:::benchmark_data_smithi043_86989_object28627:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-30T13:47:05.339 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:01.416497+0000", 2024-03-30T13:47:05.339 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 3.661399651, 2024-03-30T13:47:05.339 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.134958509, 2024-03-30T13:47:05.339 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:05.339 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:05.339 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:05.339 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:05.339 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:05.339 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:05.339 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 150965 2024-03-30T13:47:05.339 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.339 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:05.340 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.340 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:05.340 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.416497+0000", 2024-03-30T13:47:05.340 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.340 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.340 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.340 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:05.340 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.416497+0000", 2024-03-30T13:47:05.340 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.340 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.340 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.341 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:05.341 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.416499+0000", 2024-03-30T13:47:05.341 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3820000000000001e-06 2024-03-30T13:47:05.341 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.341 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.341 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:05.341 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.416504+0000", 2024-03-30T13:47:05.341 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0490000000000003e-06 2024-03-30T13:47:05.341 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.341 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.341 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:05.341 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.416517+0000", 2024-03-30T13:47:05.341 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3294e-05 2024-03-30T13:47:05.342 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.342 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.342 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:05.342 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.416525+0000", 2024-03-30T13:47:05.342 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.9179999999999994e-06 2024-03-30T13:47:05.342 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.342 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.342 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:05.342 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.444269+0000", 2024-03-30T13:47:05.342 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.02774424 2024-03-30T13:47:05.342 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.342 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.342 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:05.343 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.444378+0000", 2024-03-30T13:47:05.343 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000109103 2024-03-30T13:47:05.343 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.343 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.343 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,6", 2024-03-30T13:47:05.343 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.444518+0000", 2024-03-30T13:47:05.343 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00014011899999999999 2024-03-30T13:47:05.343 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.343 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.343 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.343 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.451909+0000", 2024-03-30T13:47:05.343 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.007390761 2024-03-30T13:47:05.344 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.344 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.344 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:05.344 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.451977+0000", 2024-03-30T13:47:05.344 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8255000000000003e-05 2024-03-30T13:47:05.344 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.344 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.344 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.344 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.551375+0000", 2024-03-30T13:47:05.344 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.099397760000000002 2024-03-30T13:47:05.344 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.344 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.344 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:05.344 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.551413+0000", 2024-03-30T13:47:05.344 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.7591e-05 2024-03-30T13:47:05.345 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.345 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.345 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:05.345 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.551456+0000", 2024-03-30T13:47:05.345 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3037000000000001e-05 2024-03-30T13:47:05.345 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.345 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.345 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.345 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.345 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.345 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:150977 7.1 7:80d942eb:::benchmark_data_smithi043_86989_object28639:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-30T13:47:05.345 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:01.429798+0000", 2024-03-30T13:47:05.345 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 3.6480985439999998, 2024-03-30T13:47:05.345 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.12182976, 2024-03-30T13:47:05.346 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:05.346 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:05.346 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:05.346 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:05.346 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:05.346 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:05.346 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 150977 2024-03-30T13:47:05.346 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.346 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:05.346 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.346 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:05.346 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.429798+0000", 2024-03-30T13:47:05.347 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.347 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.347 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.347 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:05.347 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.429798+0000", 2024-03-30T13:47:05.347 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.347 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.347 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.347 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:05.347 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.429800+0000", 2024-03-30T13:47:05.347 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4079999999999999e-06 2024-03-30T13:47:05.347 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.347 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.347 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:05.348 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.429805+0000", 2024-03-30T13:47:05.348 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.271e-06 2024-03-30T13:47:05.348 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.348 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.348 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:05.348 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.429819+0000", 2024-03-30T13:47:05.348 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4287000000000001e-05 2024-03-30T13:47:05.348 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.348 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.348 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:05.348 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.429828+0000", 2024-03-30T13:47:05.348 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6929999999999996e-06 2024-03-30T13:47:05.348 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.348 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.348 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:05.349 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.447043+0000", 2024-03-30T13:47:05.349 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.017214742000000002 2024-03-30T13:47:05.349 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.349 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.349 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:05.349 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.447163+0000", 2024-03-30T13:47:05.349 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012063899999999999 2024-03-30T13:47:05.349 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.349 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.349 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,6", 2024-03-30T13:47:05.349 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.447298+0000", 2024-03-30T13:47:05.349 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013412999999999999 2024-03-30T13:47:05.349 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.350 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.350 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.350 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.452031+0000", 2024-03-30T13:47:05.350 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0047339310000000002 2024-03-30T13:47:05.350 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.350 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.350 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:05.350 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.454394+0000", 2024-03-30T13:47:05.350 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.002362126 2024-03-30T13:47:05.350 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.350 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.350 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.351 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.551511+0000", 2024-03-30T13:47:05.351 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.097117531000000007 2024-03-30T13:47:05.351 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.351 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.351 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:05.351 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.551570+0000", 2024-03-30T13:47:05.351 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8622999999999998e-05 2024-03-30T13:47:05.351 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.351 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.351 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:05.351 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.551628+0000", 2024-03-30T13:47:05.351 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8378999999999998e-05 2024-03-30T13:47:05.351 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.351 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.351 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.352 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.352 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.352 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:151987 7.1 7:8479dd0b:::benchmark_data_smithi043_86989_object29649:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-30T13:47:05.352 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:02.580445+0000", 2024-03-30T13:47:05.352 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 2.4974517939999998, 2024-03-30T13:47:05.352 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.12508966799999999, 2024-03-30T13:47:05.352 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:05.352 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:05.352 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:05.352 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:05.352 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:05.352 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:05.352 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 151987 2024-03-30T13:47:05.352 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.353 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:05.353 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.353 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:05.353 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:02.580445+0000", 2024-03-30T13:47:05.353 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.353 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.353 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.353 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:05.353 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:02.580445+0000", 2024-03-30T13:47:05.353 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.353 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.353 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.353 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:05.353 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:02.580447+0000", 2024-03-30T13:47:05.353 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.8390000000000001e-06 2024-03-30T13:47:05.353 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.354 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.354 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:05.354 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:02.580454+0000", 2024-03-30T13:47:05.354 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8820000000000003e-06 2024-03-30T13:47:05.354 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.354 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.354 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:05.354 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:02.580473+0000", 2024-03-30T13:47:05.354 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.9079000000000001e-05 2024-03-30T13:47:05.354 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.354 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.354 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:05.354 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:02.580483+0000", 2024-03-30T13:47:05.355 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.7380000000000006e-06 2024-03-30T13:47:05.355 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.355 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.355 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:05.355 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:02.611318+0000", 2024-03-30T13:47:05.355 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.030834980000000001 2024-03-30T13:47:05.355 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.355 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.355 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:05.355 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:02.611434+0000", 2024-03-30T13:47:05.355 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000116372 2024-03-30T13:47:05.355 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.355 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.355 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,6", 2024-03-30T13:47:05.356 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:02.611518+0000", 2024-03-30T13:47:05.356 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.3825999999999993e-05 2024-03-30T13:47:05.356 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.356 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.356 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.356 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:02.616107+0000", 2024-03-30T13:47:05.356 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0045892789999999999 2024-03-30T13:47:05.356 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.356 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.356 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:05.356 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:02.616122+0000", 2024-03-30T13:47:05.356 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4951e-05 2024-03-30T13:47:05.356 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.356 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.356 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.357 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:02.705403+0000", 2024-03-30T13:47:05.357 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.089280944000000001 2024-03-30T13:47:05.357 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.357 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.357 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:05.357 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:02.705471+0000", 2024-03-30T13:47:05.357 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8021999999999993e-05 2024-03-30T13:47:05.357 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.357 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.357 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:05.357 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:02.705535+0000", 2024-03-30T13:47:05.357 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3756000000000004e-05 2024-03-30T13:47:05.357 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.357 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.357 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.358 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.358 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.358 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:153436 7.1 7:8b4bed4a:::benchmark_data_smithi043_86989_object31098:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-30T13:47:05.358 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:04.148986+0000", 2024-03-30T13:47:05.358 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.92891065100000003, 2024-03-30T13:47:05.358 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.117334858, 2024-03-30T13:47:05.358 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:05.358 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:05.358 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:05.358 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:05.358 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:05.358 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:05.358 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 153436 2024-03-30T13:47:05.359 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.359 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:05.359 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.359 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:05.359 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.148986+0000", 2024-03-30T13:47:05.359 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.359 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.359 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.359 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:05.359 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.148986+0000", 2024-03-30T13:47:05.359 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.359 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.359 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.359 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:05.359 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.148988+0000", 2024-03-30T13:47:05.360 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.702e-06 2024-03-30T13:47:05.360 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.360 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.360 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:05.360 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.148994+0000", 2024-03-30T13:47:05.360 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1970000000000001e-06 2024-03-30T13:47:05.360 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.360 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.360 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:05.360 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.149012+0000", 2024-03-30T13:47:05.360 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.7737e-05 2024-03-30T13:47:05.361 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.361 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.361 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:05.361 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.149021+0000", 2024-03-30T13:47:05.361 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.9239999999999996e-06 2024-03-30T13:47:05.361 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.361 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.361 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:05.361 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.203520+0000", 2024-03-30T13:47:05.361 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.054498904000000001 2024-03-30T13:47:05.361 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.361 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.361 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:05.361 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.203629+0000", 2024-03-30T13:47:05.362 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00010940799999999999 2024-03-30T13:47:05.362 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.362 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.362 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,6", 2024-03-30T13:47:05.362 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.203741+0000", 2024-03-30T13:47:05.362 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011225100000000001 2024-03-30T13:47:05.362 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.362 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.362 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.363 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.211923+0000", 2024-03-30T13:47:05.363 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0081817850000000004 2024-03-30T13:47:05.363 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.363 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.363 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:05.363 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.211980+0000", 2024-03-30T13:47:05.363 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7259999999999997e-05 2024-03-30T13:47:05.363 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.363 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.363 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.363 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.266173+0000", 2024-03-30T13:47:05.364 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.054192136000000002 2024-03-30T13:47:05.364 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.364 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.364 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:05.364 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.266271+0000", 2024-03-30T13:47:05.364 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.8014e-05 2024-03-30T13:47:05.364 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.364 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.364 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:05.364 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.266321+0000", 2024-03-30T13:47:05.364 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0540000000000001e-05 2024-03-30T13:47:05.364 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.364 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.364 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.365 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.365 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.365 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:153509 7.1 7:880831da:::benchmark_data_smithi043_86989_object31171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-30T13:47:05.365 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:04.225047+0000", 2024-03-30T13:47:05.365 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.85284998499999998, 2024-03-30T13:47:05.365 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.14003331899999999, 2024-03-30T13:47:05.365 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:05.365 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:05.365 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:05.365 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:05.365 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:05.365 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:05.365 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 153509 2024-03-30T13:47:05.365 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.366 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:05.366 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.366 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:05.366 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.225047+0000", 2024-03-30T13:47:05.366 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.366 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.366 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.366 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:05.366 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.225047+0000", 2024-03-30T13:47:05.366 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:05.366 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.366 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.366 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:05.367 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.225048+0000", 2024-03-30T13:47:05.367 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5099999999999999e-07 2024-03-30T13:47:05.367 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.367 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.367 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:05.367 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.225051+0000", 2024-03-30T13:47:05.367 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.1690000000000001e-06 2024-03-30T13:47:05.367 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.367 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.367 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:05.367 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.262676+0000", 2024-03-30T13:47:05.367 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.037625604 2024-03-30T13:47:05.367 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.367 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.367 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:05.368 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.262697+0000", 2024-03-30T13:47:05.368 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.0665e-05 2024-03-30T13:47:05.368 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.368 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.368 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:05.368 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.267106+0000", 2024-03-30T13:47:05.368 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0044088210000000003 2024-03-30T13:47:05.368 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.368 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.368 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:05.368 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.267250+0000", 2024-03-30T13:47:05.368 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00014389200000000001 2024-03-30T13:47:05.368 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.368 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.368 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,6", 2024-03-30T13:47:05.368 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.267381+0000", 2024-03-30T13:47:05.369 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013087099999999999 2024-03-30T13:47:05.369 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.369 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.369 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:05.369 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.269923+0000", 2024-03-30T13:47:05.369 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0025421250000000001 2024-03-30T13:47:05.369 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.369 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.369 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.369 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.280556+0000", 2024-03-30T13:47:05.369 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.010633304 2024-03-30T13:47:05.369 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.369 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.369 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:05.369 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.364968+0000", 2024-03-30T13:47:05.370 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.084412155000000003 2024-03-30T13:47:05.370 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.370 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.370 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:05.370 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.365004+0000", 2024-03-30T13:47:05.370 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.5973000000000003e-05 2024-03-30T13:47:05.370 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:05.370 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:05.370 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:05.370 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:04.365080+0000", 2024-03-30T13:47:05.370 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.6088999999999996e-05 2024-03-30T13:47:05.370 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.370 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.370 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.371 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:05.371 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:05.371 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:47:05.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:05.372 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:05.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:05.397 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:47:05.474 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:05.525 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:05.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:05.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:05.628 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:05.658 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:05.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:05.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:05.760 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:05.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:05.800 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:05.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:05.901 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:05.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:05.973 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:05.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:06.078 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:06.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:06.111 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:06.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:06.212 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:06.255 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:06.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:06.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:06.265 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:47:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:47:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:47:06.358 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:06.399 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:06.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:06.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:06.501 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:06.530 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:06.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:06.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:06.579 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-30T13:47:06.588 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:47:06.588 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:47:06.588 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:47:06.588 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:47:06.588 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:47:06.589 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:47:06.589 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:47:06.592 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:47:06.608 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:47:06.608 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:47:06.608 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:47:06.608 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:47:06.608 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:47:06.608 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:47:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:47:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:47:06.632 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:06.668 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:06.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:06.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:06.769 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:06.807 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:06 smithi043 ceph-mon[29963]: pgmap v155: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 855 op/s 2024-03-30T13:47:06.807 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:06 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1753848414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:47:06.807 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:06 smithi043 ceph-mon[42099]: pgmap v155: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 855 op/s 2024-03-30T13:47:06.808 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:06 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1753848414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:47:06.808 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:06.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:06.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:06.910 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:06.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:06.963 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:06.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:06.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:06 smithi113 ceph-mon[34341]: pgmap v155: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 855 op/s 2024-03-30T13:47:06.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:06 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1753848414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:47:07.064 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:07.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:07.111 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:07.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:07.212 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:07.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:07.252 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:07.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:07.353 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:07.393 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:07.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:07.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:07.454 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:47:07.495 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:07.562 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:07.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:07.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:07.664 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:07.683 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-03-30T13:47:07.715 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:07.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:07.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:07.763 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:07 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4147721302' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:47:07.763 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:07 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:47:07.763 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:07 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:07.764 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:07 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:07.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:07 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/4147721302' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:47:07.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:07 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:47:07.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:07 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:07.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:07 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:07.818 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:07.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:07.856 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:07.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:07.949 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:07 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/4147721302' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:47:07.950 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:07 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:47:07.950 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:07 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:07.950 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:07 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:07.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:08.006 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:08.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:08.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:08.108 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:08.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:08.141 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:08.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:08.242 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:08.268 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:08.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:08.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:08.370 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:08.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:08.416 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:08.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:08.517 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:08.546 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:08.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:08.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:08.648 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:08.693 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:08.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:08.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:08.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:47:08 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:47:08.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:08 smithi113 ceph-mon[34341]: pgmap v156: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 880 B/s rd, 0 B/s wr, 736 op/s 2024-03-30T13:47:08.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:08 smithi113 ceph-mon[34341]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:47:08.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:08 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:47:08.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:08 smithi113 ceph-mon[34341]: osdmap e125: 8 total, 8 up, 8 in 2024-03-30T13:47:08.795 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:08.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:08.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:08.942 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:08.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:08.987 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:08.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:09.088 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:09.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:09.127 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:09.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:09.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:08 smithi043 ceph-mon[29963]: pgmap v156: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 880 B/s rd, 0 B/s wr, 736 op/s 2024-03-30T13:47:09.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:08 smithi043 ceph-mon[29963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:47:09.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:08 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:47:09.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:08 smithi043 ceph-mon[29963]: osdmap e125: 8 total, 8 up, 8 in 2024-03-30T13:47:09.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:08 smithi043 ceph-mon[42099]: pgmap v156: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 880 B/s rd, 0 B/s wr, 736 op/s 2024-03-30T13:47:09.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:08 smithi043 ceph-mon[42099]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:47:09.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:08 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:47:09.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:08 smithi043 ceph-mon[42099]: osdmap e125: 8 total, 8 up, 8 in 2024-03-30T13:47:09.229 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:09.260 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:09.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:09.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:09.362 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:09.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:09.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:09.512 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:09.532 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:47:09.543 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:09.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:09.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:09.554 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:47:09.554 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:47:09.645 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:09.672 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:09.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:09.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:09.774 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:09.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:09.804 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:09.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:09.858 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:09 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:47:09.859 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:09 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:47:09.905 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:09.949 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:09 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:47:09.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:09.967 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:09.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:10.068 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:10.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:10.104 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:10.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:10.206 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:10.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:10.267 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:10.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:10.312 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-03-30T13:47:10.368 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:10.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:10.407 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:10.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:10.508 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:no valid command found; 10 closest matches: 2024-03-30T13:47:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:0 2024-03-30T13:47:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:1 2024-03-30T13:47:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:2 2024-03-30T13:47:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:abort 2024-03-30T13:47:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:assert 2024-03-30T13:47:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:bench [] [] [] [] 2024-03-30T13:47:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs debug_inject_read_zeros 2024-03-30T13:47:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs files list 2024-03-30T13:47:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs stats 2024-03-30T13:47:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:bluestore allocator dump block 2024-03-30T13:47:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: invalid command 2024-03-30T13:47:10.565 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:47:10.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:10.584 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:10.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:10.617 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:47:10.685 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:10.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:10.718 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:10.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:10.735 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T13:47:10.819 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:10.847 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:10.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:10.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:10.868 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:10 smithi043 ceph-mon[29963]: pgmap v158: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 905 op/s 2024-03-30T13:47:10.868 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:10 smithi043 ceph-mon[42099]: pgmap v158: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 905 op/s 2024-03-30T13:47:10.949 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:10.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:10 smithi113 ceph-mon[34341]: pgmap v158: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 905 op/s 2024-03-30T13:47:10.975 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:10.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:11.076 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:11.130 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:11.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:11.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:11.231 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:11.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:11.280 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:11.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:11.381 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:11.422 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:11.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:11.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:11.524 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:11.576 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:11.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:11.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:11.678 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:11.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:11.731 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:11.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:11.832 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:11.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:11.865 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:11.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:11.967 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:12.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:12.015 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:12.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:12.119 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:12.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:12.174 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:12.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:12.204 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:47:12.205 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-30T13:47:12.205 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 'trim stale osdmaps' 2024-03-30T13:47:12.219 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:47:12.275 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:12.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:12.302 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:12.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:12.403 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:12.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:12.452 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:12.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:12.553 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:12.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:12.623 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:12.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:12.724 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:12.754 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:12.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:12.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:12.773 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:12 smithi043 ceph-mon[29963]: pgmap v159: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 905 op/s 2024-03-30T13:47:12.773 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:12 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:12.773 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:12 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:12.773 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:12 smithi043 ceph-mon[42099]: pgmap v159: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 905 op/s 2024-03-30T13:47:12.773 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:12 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:12.773 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:12 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:12.787 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:12 smithi113 ceph-mon[34341]: pgmap v159: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 905 op/s 2024-03-30T13:47:12.787 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:12 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:12.787 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:12 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:12.856 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:12.885 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:12.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:12.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:12.987 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:13.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:13.016 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:13.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:13.118 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:13.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:13.151 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:13.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:13.244 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:47:13.244 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:47:13.253 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:13.289 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:13.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:13.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:13.391 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:13.430 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:13.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:13.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:13.532 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:13.578 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:13.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:13.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:13.680 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:13.715 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:13.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:13.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:13.817 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:13.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:13.858 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:13.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:13.863 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:13 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:47:13.863 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:13 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:47:13.864 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:13 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:47:13.864 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:13 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:13.865 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:13 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:47:13.865 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:13 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:47:13.865 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:13 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:47:13.865 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:13 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:13.938 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:13 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:47:13.938 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:13 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:47:13.938 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:13 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:47:13.938 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:13 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:13.959 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:13.976 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-03-30T13:47:14.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:14.012 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:14.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:14.113 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:14.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:14.160 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:14.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:14.261 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:14.297 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:47:14.300 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:14.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:14.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:14.402 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:14.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:14.431 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:14.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:14.532 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:14.573 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:14.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:14.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:14.675 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:14.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:14.722 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:14.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:14.823 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:14.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:14.863 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:14.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:14.964 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:14.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:14.998 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:14.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:15.091 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:14 smithi043 ceph-mon[42099]: pgmap v160: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 915 op/s 2024-03-30T13:47:15.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:14 smithi043 ceph-mon[29963]: pgmap v160: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 915 op/s 2024-03-30T13:47:15.099 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:15.132 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:15.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:15.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:15.206 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:14 smithi113 ceph-mon[34341]: pgmap v160: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 915 op/s 2024-03-30T13:47:15.233 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:15.261 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:15.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:15.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:15.363 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:15.402 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:15.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:15.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:15.459 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:47:15.504 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:15.508 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:47:15.530 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:15.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:15.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:15.553 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:47:15.605 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:47:15.632 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:15.654 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:47:15.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:15.676 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:15.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:15.701 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:47:15.746 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:47:15.777 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:15.796 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:47:15.802 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:47:15.803 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:47:15.803 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:47:15.803 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:47:15.803 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:47:15.803 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:47:15.803 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:47:15.803 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:47:15.829 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:15.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:15.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:15.930 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:15.959 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:15.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:15.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:15.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:15 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:47:15.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:15 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:47:15.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:15 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3727430199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:47:16.061 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:16.082 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:47:16.086 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:16.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:16.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:16.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:15 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:47:16.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:15 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:47:16.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:15 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3727430199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:47:16.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:15 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:47:16.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:15 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:47:16.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:15 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3727430199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:47:16.188 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:16.237 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:16.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:16.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:16.290 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:47:16.291 INFO:teuthology.orchestra.run.smithi113.stdout: "size": 20, 2024-03-30T13:47:16.291 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 600, 2024-03-30T13:47:16.291 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [ 2024-03-30T13:47:16.291 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.291 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:18366 7.0 7:02272c26:::benchmark_data_smithi043_86989_object18365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-30T13:47:16.291 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:45:07.224995+0000", 2024-03-30T13:47:16.291 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 129.06184783800001, 2024-03-30T13:47:16.291 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.071909846999999999, 2024-03-30T13:47:16.291 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:16.291 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:16.291 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:16.291 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:16.292 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:16.292 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:16.292 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 18366 2024-03-30T13:47:16.292 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.292 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:16.292 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.292 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:16.292 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.224995+0000", 2024-03-30T13:47:16.292 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.292 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.292 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.292 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:16.292 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.224995+0000", 2024-03-30T13:47:16.292 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.293 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.293 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.293 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:16.293 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.224997+0000", 2024-03-30T13:47:16.293 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.68e-06 2024-03-30T13:47:16.293 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.293 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.293 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:16.293 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.225051+0000", 2024-03-30T13:47:16.293 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3777999999999999e-05 2024-03-30T13:47:16.293 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.293 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.293 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:16.293 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.225063+0000", 2024-03-30T13:47:16.294 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2027e-05 2024-03-30T13:47:16.294 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.294 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.294 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:16.294 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.225069+0000", 2024-03-30T13:47:16.294 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6899999999999997e-06 2024-03-30T13:47:16.294 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.294 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.294 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:16.294 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.225113+0000", 2024-03-30T13:47:16.294 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3933000000000001e-05 2024-03-30T13:47:16.294 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.294 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.295 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:16.295 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.225252+0000", 2024-03-30T13:47:16.295 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013896700000000001 2024-03-30T13:47:16.295 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.295 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.295 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:47:16.295 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.225360+0000", 2024-03-30T13:47:16.295 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000108393 2024-03-30T13:47:16.295 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.295 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.295 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:16.295 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.226132+0000", 2024-03-30T13:47:16.295 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00077158399999999996 2024-03-30T13:47:16.295 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.296 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.296 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.296 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.227157+0000", 2024-03-30T13:47:16.296 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0010249840000000001 2024-03-30T13:47:16.296 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.296 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.296 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.296 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.296790+0000", 2024-03-30T13:47:16.296 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.069633466000000005 2024-03-30T13:47:16.296 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.296 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.296 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:16.296 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.296856+0000", 2024-03-30T13:47:16.296 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5970999999999996e-05 2024-03-30T13:47:16.296 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.297 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.297 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:16.297 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.296905+0000", 2024-03-30T13:47:16.297 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9373999999999999e-05 2024-03-30T13:47:16.297 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.297 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.297 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.297 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.297 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.297 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:19902 7.0 7:01c3a0b0:::benchmark_data_smithi043_86989_object19901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-30T13:47:16.297 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:45:07.989962+0000", 2024-03-30T13:47:16.297 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 128.296881181, 2024-03-30T13:47:16.297 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.051843736000000001, 2024-03-30T13:47:16.297 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:16.298 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:16.298 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:16.298 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:16.298 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:16.298 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:16.298 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 19902 2024-03-30T13:47:16.298 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.298 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:16.298 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.298 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:16.298 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.989962+0000", 2024-03-30T13:47:16.298 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.298 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.299 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.299 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:16.299 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.989962+0000", 2024-03-30T13:47:16.299 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.299 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.299 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.299 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:16.299 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.989964+0000", 2024-03-30T13:47:16.299 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.9489999999999999e-06 2024-03-30T13:47:16.300 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.300 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.300 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:16.300 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.990029+0000", 2024-03-30T13:47:16.300 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.444e-05 2024-03-30T13:47:16.300 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.300 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.300 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:16.300 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.990042+0000", 2024-03-30T13:47:16.300 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3465999999999999e-05 2024-03-30T13:47:16.300 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.300 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.300 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:16.300 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.990050+0000", 2024-03-30T13:47:16.301 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.2139999999999996e-06 2024-03-30T13:47:16.301 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.301 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.301 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:16.301 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.990123+0000", 2024-03-30T13:47:16.301 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.3254e-05 2024-03-30T13:47:16.301 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.301 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.301 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:16.301 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.990289+0000", 2024-03-30T13:47:16.301 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00016593900000000001 2024-03-30T13:47:16.301 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.301 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.301 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:47:16.301 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.990435+0000", 2024-03-30T13:47:16.302 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00014511600000000001 2024-03-30T13:47:16.302 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.302 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.302 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:16.302 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.991664+0000", 2024-03-30T13:47:16.302 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0012299170000000001 2024-03-30T13:47:16.302 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.302 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.302 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.302 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:07.992193+0000", 2024-03-30T13:47:16.302 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00052825699999999997 2024-03-30T13:47:16.302 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.302 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.303 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.303 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:08.041730+0000", 2024-03-30T13:47:16.303 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.049537734999999999 2024-03-30T13:47:16.303 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.303 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.303 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:16.303 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:08.041766+0000", 2024-03-30T13:47:16.303 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.5404999999999997e-05 2024-03-30T13:47:16.303 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.303 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.303 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:16.303 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:08.041806+0000", 2024-03-30T13:47:16.303 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.0043999999999998e-05 2024-03-30T13:47:16.303 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.304 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.304 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.304 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.304 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.304 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:22329 7.0 7:05055661:::benchmark_data_smithi043_86989_object22328:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-30T13:47:16.304 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:45:09.136749+0000", 2024-03-30T13:47:16.304 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 127.15009455800001, 2024-03-30T13:47:16.304 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.079349797999999999, 2024-03-30T13:47:16.304 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:16.304 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:16.305 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:16.305 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:16.306 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:16.306 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:16.306 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 22329 2024-03-30T13:47:16.306 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.306 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:16.306 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.306 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:16.306 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.136749+0000", 2024-03-30T13:47:16.306 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.306 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.306 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.306 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:16.306 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.136749+0000", 2024-03-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.136750+0000", 2024-03-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0640000000000001e-06 2024-03-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.136810+0000", 2024-03-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9842000000000001e-05 2024-03-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:16.307 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.136820+0000", 2024-03-30T13:47:16.308 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0505e-05 2024-03-30T13:47:16.308 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.308 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.308 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:16.308 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.136826+0000", 2024-03-30T13:47:16.308 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0750000000000003e-06 2024-03-30T13:47:16.308 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.308 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.308 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:16.308 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.136864+0000", 2024-03-30T13:47:16.308 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.7468000000000001e-05 2024-03-30T13:47:16.308 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.308 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.309 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:16.309 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.136943+0000", 2024-03-30T13:47:16.309 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.9567999999999994e-05 2024-03-30T13:47:16.309 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.309 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.309 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:47:16.309 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.137061+0000", 2024-03-30T13:47:16.309 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000117378 2024-03-30T13:47:16.309 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.309 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.309 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:16.309 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.137761+0000", 2024-03-30T13:47:16.309 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00070024400000000004 2024-03-30T13:47:16.310 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.310 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.310 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.310 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.138365+0000", 2024-03-30T13:47:16.310 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00060360399999999999 2024-03-30T13:47:16.310 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.310 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.310 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.310 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.215982+0000", 2024-03-30T13:47:16.310 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.077617647999999997 2024-03-30T13:47:16.310 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.310 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.310 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:16.311 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.216053+0000", 2024-03-30T13:47:16.311 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.0603000000000001e-05 2024-03-30T13:47:16.311 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.311 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.311 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:16.311 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.216099+0000", 2024-03-30T13:47:16.311 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.5799e-05 2024-03-30T13:47:16.311 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.311 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.311 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.311 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.311 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.311 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:27097 7.0 7:09b1ad83:::benchmark_data_smithi043_86989_object27096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-30T13:47:16.311 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:45:11.444689+0000", 2024-03-30T13:47:16.312 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 124.842154767, 2024-03-30T13:47:16.312 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.048868647000000001, 2024-03-30T13:47:16.312 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:16.312 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:16.312 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:16.312 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:16.312 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:16.312 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:16.312 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 27097 2024-03-30T13:47:16.312 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.312 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:16.312 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.312 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:16.312 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:11.444689+0000", 2024-03-30T13:47:16.313 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.313 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.313 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.313 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:16.313 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:11.444689+0000", 2024-03-30T13:47:16.313 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.313 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.313 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.313 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:16.313 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:11.444690+0000", 2024-03-30T13:47:16.313 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.204e-06 2024-03-30T13:47:16.313 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.313 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.313 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:16.314 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:11.444734+0000", 2024-03-30T13:47:16.314 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.4413999999999999e-05 2024-03-30T13:47:16.314 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.314 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.314 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:16.314 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:11.444747+0000", 2024-03-30T13:47:16.314 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2875e-05 2024-03-30T13:47:16.314 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.314 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.314 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:16.314 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:11.444754+0000", 2024-03-30T13:47:16.314 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.4139999999999997e-06 2024-03-30T13:47:16.314 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.314 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.315 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:16.315 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:11.444798+0000", 2024-03-30T13:47:16.315 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3333e-05 2024-03-30T13:47:16.315 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.315 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.315 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:16.315 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:11.444902+0000", 2024-03-30T13:47:16.315 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000104345 2024-03-30T13:47:16.315 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.315 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.315 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:47:16.315 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:11.445050+0000", 2024-03-30T13:47:16.315 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00014810800000000001 2024-03-30T13:47:16.315 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.316 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.316 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:16.316 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:11.446367+0000", 2024-03-30T13:47:16.316 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001316865 2024-03-30T13:47:16.316 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.316 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.316 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.316 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:11.447174+0000", 2024-03-30T13:47:16.316 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00080717300000000001 2024-03-30T13:47:16.316 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.316 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.316 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.316 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:11.493452+0000", 2024-03-30T13:47:16.317 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.046277480000000003 2024-03-30T13:47:16.317 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.317 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.317 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:16.317 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:11.493511+0000", 2024-03-30T13:47:16.317 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9605999999999998e-05 2024-03-30T13:47:16.317 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.317 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.317 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:16.317 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:11.493557+0000", 2024-03-30T13:47:16.317 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.583e-05 2024-03-30T13:47:16.317 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.317 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.317 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.317 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.318 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.318 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:61995 7.0 7:0437627d:::benchmark_data_smithi043_86989_object61994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-30T13:47:16.318 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:45:33.954329+0000", 2024-03-30T13:47:16.318 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 102.332514729, 2024-03-30T13:47:16.318 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.090862954999999995, 2024-03-30T13:47:16.318 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:16.318 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:16.318 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:16.318 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:16.318 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:16.318 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:16.318 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 61995 2024-03-30T13:47:16.318 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.319 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:16.319 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.319 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:16.319 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.954329+0000", 2024-03-30T13:47:16.319 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.319 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.319 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.319 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:16.319 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.954329+0000", 2024-03-30T13:47:16.319 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.319 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.319 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.319 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:16.319 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.954330+0000", 2024-03-30T13:47:16.320 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.1200000000000001e-07 2024-03-30T13:47:16.320 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.320 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.320 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:16.320 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.954380+0000", 2024-03-30T13:47:16.320 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0637999999999999e-05 2024-03-30T13:47:16.320 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.320 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.320 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:16.320 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.954390+0000", 2024-03-30T13:47:16.320 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.645e-06 2024-03-30T13:47:16.320 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.320 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.320 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:16.320 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.954395+0000", 2024-03-30T13:47:16.321 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3419999999999998e-06 2024-03-30T13:47:16.321 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.321 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.321 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:16.321 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.954442+0000", 2024-03-30T13:47:16.321 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7296999999999998e-05 2024-03-30T13:47:16.321 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.321 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.321 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:16.321 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.954618+0000", 2024-03-30T13:47:16.321 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00017529199999999999 2024-03-30T13:47:16.321 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.321 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.321 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:47:16.322 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.954763+0000", 2024-03-30T13:47:16.322 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000145196 2024-03-30T13:47:16.322 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.322 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.322 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:16.322 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.957069+0000", 2024-03-30T13:47:16.322 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00230583 2024-03-30T13:47:16.322 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.322 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.322 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.322 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.959940+0000", 2024-03-30T13:47:16.322 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0028708969999999999 2024-03-30T13:47:16.322 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.323 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.323 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.323 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:34.045064+0000", 2024-03-30T13:47:16.323 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.085124209000000006 2024-03-30T13:47:16.323 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.323 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.323 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:16.323 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:34.045153+0000", 2024-03-30T13:47:16.323 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.8701000000000003e-05 2024-03-30T13:47:16.323 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.323 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.323 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:16.323 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:34.045192+0000", 2024-03-30T13:47:16.323 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.8995999999999997e-05 2024-03-30T13:47:16.324 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.324 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.324 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.324 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.324 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.324 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:62003 7.0 7:0229d695:::benchmark_data_smithi043_86989_object62002:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-30T13:47:16.324 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:45:33.965096+0000", 2024-03-30T13:47:16.324 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 102.32174747000001, 2024-03-30T13:47:16.324 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.086717765000000002, 2024-03-30T13:47:16.324 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:16.324 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:16.324 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:16.324 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:16.324 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:16.325 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:16.325 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 62003 2024-03-30T13:47:16.325 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.325 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:16.325 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.325 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:16.325 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.965096+0000", 2024-03-30T13:47:16.325 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.325 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.325 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.325 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:16.325 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.965096+0000", 2024-03-30T13:47:16.325 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.325 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.325 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.326 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:16.326 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.965097+0000", 2024-03-30T13:47:16.326 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1820000000000001e-06 2024-03-30T13:47:16.326 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.326 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.326 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:16.326 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.965152+0000", 2024-03-30T13:47:16.326 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4993e-05 2024-03-30T13:47:16.326 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.326 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.326 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:16.326 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.965164+0000", 2024-03-30T13:47:16.326 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1474999999999999e-05 2024-03-30T13:47:16.327 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.327 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.327 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:16.327 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.965169+0000", 2024-03-30T13:47:16.327 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8690000000000001e-06 2024-03-30T13:47:16.327 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.327 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.327 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:16.327 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.965211+0000", 2024-03-30T13:47:16.327 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1322000000000002e-05 2024-03-30T13:47:16.327 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.327 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.327 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:16.327 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.965305+0000", 2024-03-30T13:47:16.328 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4408999999999997e-05 2024-03-30T13:47:16.328 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.328 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.328 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:47:16.328 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.965394+0000", 2024-03-30T13:47:16.328 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.8945999999999998e-05 2024-03-30T13:47:16.328 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.328 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.328 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:16.328 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.967037+0000", 2024-03-30T13:47:16.328 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001642527 2024-03-30T13:47:16.328 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.328 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.328 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.329 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.967294+0000", 2024-03-30T13:47:16.329 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000257457 2024-03-30T13:47:16.329 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.329 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.329 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.329 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:34.051715+0000", 2024-03-30T13:47:16.329 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.084420766999999994 2024-03-30T13:47:16.329 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.329 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.329 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:16.329 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:34.051766+0000", 2024-03-30T13:47:16.329 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1371e-05 2024-03-30T13:47:16.329 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.330 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.330 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:16.330 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:34.051814+0000", 2024-03-30T13:47:16.330 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7447000000000002e-05 2024-03-30T13:47:16.330 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.330 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.330 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.330 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.330 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.330 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:63511 7.0 7:0c7c7d6b:::benchmark_data_smithi043_86989_object63510:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-30T13:47:16.330 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:45:35.200203+0000", 2024-03-30T13:47:16.330 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 101.086639945, 2024-03-30T13:47:16.330 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.056164680000000002, 2024-03-30T13:47:16.331 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:16.331 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:16.331 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:16.331 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:16.331 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:16.331 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:16.331 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 63511 2024-03-30T13:47:16.331 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.331 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:16.331 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.331 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:16.331 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:35.200203+0000", 2024-03-30T13:47:16.331 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.331 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.331 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.332 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:16.332 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:35.200203+0000", 2024-03-30T13:47:16.332 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.332 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.332 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.332 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:16.332 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:35.200205+0000", 2024-03-30T13:47:16.332 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.176e-06 2024-03-30T13:47:16.332 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.332 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.332 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:16.332 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:35.200252+0000", 2024-03-30T13:47:16.332 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7049999999999998e-05 2024-03-30T13:47:16.333 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.333 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.333 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:16.333 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:35.200263+0000", 2024-03-30T13:47:16.333 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1293e-05 2024-03-30T13:47:16.333 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.333 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.333 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:16.333 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:35.200271+0000", 2024-03-30T13:47:16.333 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.1999999999999994e-06 2024-03-30T13:47:16.333 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.333 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.333 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:16.333 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:35.200321+0000", 2024-03-30T13:47:16.334 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9645999999999999e-05 2024-03-30T13:47:16.334 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.334 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.334 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:16.334 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:35.200418+0000", 2024-03-30T13:47:16.334 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.7411000000000006e-05 2024-03-30T13:47:16.334 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.334 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.334 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:47:16.334 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:35.200536+0000", 2024-03-30T13:47:16.334 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000118214 2024-03-30T13:47:16.334 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.334 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.334 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:16.335 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:35.202645+0000", 2024-03-30T13:47:16.335 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0021086870000000001 2024-03-30T13:47:16.335 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.335 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.335 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.335 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:35.204125+0000", 2024-03-30T13:47:16.335 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001479667 2024-03-30T13:47:16.335 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.335 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.335 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.335 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:35.256278+0000", 2024-03-30T13:47:16.335 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.052153508000000001 2024-03-30T13:47:16.335 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.336 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.336 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:16.336 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:35.256330+0000", 2024-03-30T13:47:16.336 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1728999999999999e-05 2024-03-30T13:47:16.336 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.336 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.336 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:16.336 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:35.256368+0000", 2024-03-30T13:47:16.336 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.8099000000000002e-05 2024-03-30T13:47:16.336 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.336 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.336 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.336 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.336 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.337 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:97889 7.0 7:0f8024e2:::benchmark_data_smithi043_86989_object97888:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-03-30T13:47:16.337 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:05.973368+0000", 2024-03-30T13:47:16.337 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 70.313475546000006, 2024-03-30T13:47:16.337 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.072580213000000005, 2024-03-30T13:47:16.337 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:16.337 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:16.337 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:16.337 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:16.337 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:16.337 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:16.337 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 97889 2024-03-30T13:47:16.337 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.337 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:16.337 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.338 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:16.338 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:05.973368+0000", 2024-03-30T13:47:16.338 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.338 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.338 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.338 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:16.338 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:05.973368+0000", 2024-03-30T13:47:16.338 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.338 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.338 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.338 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:16.338 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:05.973369+0000", 2024-03-30T13:47:16.338 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.375e-06 2024-03-30T13:47:16.339 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.339 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.339 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:16.339 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:05.973399+0000", 2024-03-30T13:47:16.339 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.0175000000000002e-05 2024-03-30T13:47:16.339 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.339 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.339 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:16.339 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:05.973418+0000", 2024-03-30T13:47:16.339 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.8411e-05 2024-03-30T13:47:16.339 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.339 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.339 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:16.339 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:05.973427+0000", 2024-03-30T13:47:16.339 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.3789999999999995e-06 2024-03-30T13:47:16.340 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.340 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.340 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:16.340 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:05.973486+0000", 2024-03-30T13:47:16.340 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9035e-05 2024-03-30T13:47:16.340 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.340 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.340 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:16.340 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:05.973600+0000", 2024-03-30T13:47:16.340 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000114082 2024-03-30T13:47:16.340 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.340 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.340 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:47:16.341 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:05.973740+0000", 2024-03-30T13:47:16.341 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000139278 2024-03-30T13:47:16.341 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.341 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.341 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:16.341 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:05.976827+0000", 2024-03-30T13:47:16.341 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.003087598 2024-03-30T13:47:16.341 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.341 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.341 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.341 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:05.978165+0000", 2024-03-30T13:47:16.341 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001337544 2024-03-30T13:47:16.341 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.342 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.342 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.342 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:06.045858+0000", 2024-03-30T13:47:16.342 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.067693368000000004 2024-03-30T13:47:16.342 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.342 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.342 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:16.342 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:06.045896+0000", 2024-03-30T13:47:16.342 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.8333e-05 2024-03-30T13:47:16.342 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.342 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.342 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:16.342 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:06.045948+0000", 2024-03-30T13:47:16.343 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1635000000000003e-05 2024-03-30T13:47:16.343 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.343 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.343 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.343 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.343 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.343 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:98436 7.0 7:0ba9ebd1:::benchmark_data_smithi043_86989_object98435:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-03-30T13:47:16.343 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:06.574879+0000", 2024-03-30T13:47:16.343 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 69.711964054000006, 2024-03-30T13:47:16.343 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.063047743000000003, 2024-03-30T13:47:16.343 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:16.343 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:16.343 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:16.343 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:16.344 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:16.344 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:16.344 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 98436 2024-03-30T13:47:16.344 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.344 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:16.344 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.344 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:16.344 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:06.574879+0000", 2024-03-30T13:47:16.344 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.344 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.344 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.344 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:16.344 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:06.574879+0000", 2024-03-30T13:47:16.344 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.345 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.345 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.345 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:16.345 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:06.574881+0000", 2024-03-30T13:47:16.345 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.725e-06 2024-03-30T13:47:16.345 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.345 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.345 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:16.345 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:06.574911+0000", 2024-03-30T13:47:16.345 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.9889000000000001e-05 2024-03-30T13:47:16.345 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.345 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.345 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:16.345 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:06.574926+0000", 2024-03-30T13:47:16.346 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5073e-05 2024-03-30T13:47:16.346 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.346 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.346 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:16.346 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:06.574934+0000", 2024-03-30T13:47:16.346 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.4130000000000006e-06 2024-03-30T13:47:16.346 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.346 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.346 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:16.346 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:06.575085+0000", 2024-03-30T13:47:16.346 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00015023200000000001 2024-03-30T13:47:16.346 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.346 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.347 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:16.347 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:06.575236+0000", 2024-03-30T13:47:16.347 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000151528 2024-03-30T13:47:16.347 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.347 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.347 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:16.349 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:47:16.349 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:06.575513+0000", 2024-03-30T13:47:16.349 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000276827 2024-03-30T13:47:16.349 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.349 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.349 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:16.349 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:06.578484+0000", 2024-03-30T13:47:16.349 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.002971511 2024-03-30T13:47:16.349 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.349 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.349 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.349 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:06.580760+0000", 2024-03-30T13:47:16.350 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0022758689999999998 2024-03-30T13:47:16.350 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.350 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.350 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.350 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:06.637822+0000", 2024-03-30T13:47:16.350 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.057061662999999999 2024-03-30T13:47:16.350 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.350 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.350 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:16.350 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:06.637883+0000", 2024-03-30T13:47:16.350 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0782999999999997e-05 2024-03-30T13:47:16.350 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.350 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.351 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:16.351 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:06.637927+0000", 2024-03-30T13:47:16.351 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.4230000000000002e-05 2024-03-30T13:47:16.351 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.351 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.351 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.351 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.351 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.351 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:124615 7.0 7:0cf4596f:::benchmark_data_smithi043_86989_object2277:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T13:47:16.351 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:32.760937+0000", 2024-03-30T13:47:16.351 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 43.525906282000001, 2024-03-30T13:47:16.351 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.048785037000000003, 2024-03-30T13:47:16.351 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:16.352 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:16.352 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:16.352 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:16.352 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:16.352 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:16.352 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 124615 2024-03-30T13:47:16.352 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.352 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:16.352 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.352 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:16.352 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:32.760937+0000", 2024-03-30T13:47:16.352 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.352 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.353 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.353 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:16.353 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:32.760937+0000", 2024-03-30T13:47:16.353 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.353 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.353 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.353 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:16.353 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:32.760939+0000", 2024-03-30T13:47:16.353 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4750000000000001e-06 2024-03-30T13:47:16.353 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.353 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.353 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:16.353 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:32.760944+0000", 2024-03-30T13:47:16.353 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8239999999999996e-06 2024-03-30T13:47:16.354 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.354 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.354 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:16.354 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:32.760956+0000", 2024-03-30T13:47:16.354 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1975e-05 2024-03-30T13:47:16.354 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.354 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.354 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:16.354 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:32.760962+0000", 2024-03-30T13:47:16.354 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9290000000000003e-06 2024-03-30T13:47:16.354 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.354 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.354 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:16.355 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:32.762620+0000", 2024-03-30T13:47:16.355 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001657946 2024-03-30T13:47:16.355 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.355 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.355 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:16.355 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:32.762742+0000", 2024-03-30T13:47:16.355 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012171999999999999 2024-03-30T13:47:16.355 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.355 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.355 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:47:16.355 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:32.762863+0000", 2024-03-30T13:47:16.355 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000121568 2024-03-30T13:47:16.355 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.355 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.355 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:16.356 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:32.770128+0000", 2024-03-30T13:47:16.356 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0072640129999999997 2024-03-30T13:47:16.356 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.356 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.356 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.356 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:32.772390+0000", 2024-03-30T13:47:16.356 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0022624619999999998 2024-03-30T13:47:16.356 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.356 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.356 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.356 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:32.809595+0000", 2024-03-30T13:47:16.356 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.037205272999999997 2024-03-30T13:47:16.356 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.357 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.357 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:16.357 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:32.809649+0000", 2024-03-30T13:47:16.357 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4088000000000001e-05 2024-03-30T13:47:16.357 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.357 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.357 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:16.357 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:32.809722+0000", 2024-03-30T13:47:16.357 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.2763999999999997e-05 2024-03-30T13:47:16.357 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.357 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.357 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.357 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.357 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.358 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:125768 7.0 7:0e11e2db:::benchmark_data_smithi043_86989_object3430:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T13:47:16.358 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:33.940593+0000", 2024-03-30T13:47:16.358 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 42.346250390999998, 2024-03-30T13:47:16.358 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.063252480999999999, 2024-03-30T13:47:16.358 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:16.358 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:16.358 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:16.358 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:16.358 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:16.358 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:16.358 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 125768 2024-03-30T13:47:16.358 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.358 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:16.358 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.359 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:16.359 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.940593+0000", 2024-03-30T13:47:16.359 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.359 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.359 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.359 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:16.359 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.940593+0000", 2024-03-30T13:47:16.359 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.359 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.359 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.359 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:16.359 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.940595+0000", 2024-03-30T13:47:16.359 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.917e-06 2024-03-30T13:47:16.359 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.359 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.360 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:16.360 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.940601+0000", 2024-03-30T13:47:16.360 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9909999999999999e-06 2024-03-30T13:47:16.360 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.360 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.360 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:16.360 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.940612+0000", 2024-03-30T13:47:16.360 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1092e-05 2024-03-30T13:47:16.360 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.360 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.360 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:16.360 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.940620+0000", 2024-03-30T13:47:16.360 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8359999999999992e-06 2024-03-30T13:47:16.360 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.361 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.361 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:16.361 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.948413+0000", 2024-03-30T13:47:16.361 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0077932779999999998 2024-03-30T13:47:16.361 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.361 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.361 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:16.361 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.948512+0000", 2024-03-30T13:47:16.361 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.8939999999999998e-05 2024-03-30T13:47:16.361 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.361 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.361 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:47:16.361 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.948631+0000", 2024-03-30T13:47:16.361 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000118507 2024-03-30T13:47:16.362 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.362 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.362 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.362 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.955726+0000", 2024-03-30T13:47:16.362 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0070958499999999999 2024-03-30T13:47:16.362 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.362 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.362 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:16.362 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.957866+0000", 2024-03-30T13:47:16.362 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0021399610000000001 2024-03-30T13:47:16.362 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.362 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.362 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.362 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:34.003723+0000", 2024-03-30T13:47:16.363 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.045857032999999998 2024-03-30T13:47:16.363 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.363 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.363 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:16.363 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:34.003780+0000", 2024-03-30T13:47:16.363 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6508000000000002e-05 2024-03-30T13:47:16.363 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.363 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.363 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:16.363 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:34.003845+0000", 2024-03-30T13:47:16.363 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5568000000000006e-05 2024-03-30T13:47:16.363 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.363 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.363 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.364 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.364 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.364 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:125774 7.0 7:076c59d5:::benchmark_data_smithi043_86989_object3436:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T13:47:16.364 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:33.944180+0000", 2024-03-30T13:47:16.364 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 42.342663373999997, 2024-03-30T13:47:16.364 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.066432890999999994, 2024-03-30T13:47:16.364 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:16.364 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:16.364 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:16.364 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:16.364 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:16.364 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:16.364 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 125774 2024-03-30T13:47:16.364 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.364 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:16.365 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.365 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:16.365 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.944180+0000", 2024-03-30T13:47:16.365 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.365 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.365 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.365 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:16.365 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.944180+0000", 2024-03-30T13:47:16.365 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.365 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.365 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.365 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:16.365 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.944181+0000", 2024-03-30T13:47:16.365 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1909999999999999e-06 2024-03-30T13:47:16.366 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.366 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.366 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:16.366 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.944187+0000", 2024-03-30T13:47:16.366 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0610000000000001e-06 2024-03-30T13:47:16.366 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.366 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.366 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:16.366 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.944203+0000", 2024-03-30T13:47:16.366 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5611e-05 2024-03-30T13:47:16.366 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.366 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.366 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:16.366 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.944212+0000", 2024-03-30T13:47:16.367 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.0359999999999995e-06 2024-03-30T13:47:16.367 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.367 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.367 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:16.367 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.953240+0000", 2024-03-30T13:47:16.367 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0090283580000000002 2024-03-30T13:47:16.367 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.367 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.367 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:16.367 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.953369+0000", 2024-03-30T13:47:16.367 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000129028 2024-03-30T13:47:16.367 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.367 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.367 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:47:16.368 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.953511+0000", 2024-03-30T13:47:16.368 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00014195700000000001 2024-03-30T13:47:16.368 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.368 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.368 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:16.368 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.957874+0000", 2024-03-30T13:47:16.368 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0043627140000000002 2024-03-30T13:47:16.368 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.368 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.368 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.368 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.958264+0000", 2024-03-30T13:47:16.368 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00038991699999999999 2024-03-30T13:47:16.368 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.368 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.369 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.369 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:34.010471+0000", 2024-03-30T13:47:16.369 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.052207152999999999 2024-03-30T13:47:16.369 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.369 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.369 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:16.369 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:34.010550+0000", 2024-03-30T13:47:16.369 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8745000000000006e-05 2024-03-30T13:47:16.369 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.369 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.369 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:16.369 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:34.010613+0000", 2024-03-30T13:47:16.369 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3120000000000006e-05 2024-03-30T13:47:16.369 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:125788 7.0 7:06a743d3:::benchmark_data_smithi043_86989_object3450:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T13:47:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:33.953967+0000", 2024-03-30T13:47:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 42.332876558000002, 2024-03-30T13:47:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.056771308999999999, 2024-03-30T13:47:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:16.371 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 125788 2024-03-30T13:47:16.371 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.371 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:16.371 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.371 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:16.371 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.953967+0000", 2024-03-30T13:47:16.371 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.371 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.371 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.371 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:16.371 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.953967+0000", 2024-03-30T13:47:16.371 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.371 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.371 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.372 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:16.372 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.953968+0000", 2024-03-30T13:47:16.372 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.341e-06 2024-03-30T13:47:16.372 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.372 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.372 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:16.372 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.953974+0000", 2024-03-30T13:47:16.372 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.412e-06 2024-03-30T13:47:16.372 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.372 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.372 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:16.372 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.953989+0000", 2024-03-30T13:47:16.372 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5085999999999999e-05 2024-03-30T13:47:16.372 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.373 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.373 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:16.373 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.953997+0000", 2024-03-30T13:47:16.373 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.0790000000000001e-06 2024-03-30T13:47:16.373 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.373 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.373 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:16.373 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.955767+0000", 2024-03-30T13:47:16.373 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001770347 2024-03-30T13:47:16.373 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.373 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.373 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:16.373 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.955879+0000", 2024-03-30T13:47:16.373 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011242 2024-03-30T13:47:16.373 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.374 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.374 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:47:16.374 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.956000+0000", 2024-03-30T13:47:16.374 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000120895 2024-03-30T13:47:16.374 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.374 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.374 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.374 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.958298+0000", 2024-03-30T13:47:16.374 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.002297285 2024-03-30T13:47:16.374 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.374 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.374 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:16.374 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:33.958548+0000", 2024-03-30T13:47:16.374 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00024996000000000003 2024-03-30T13:47:16.375 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.375 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.375 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.375 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:34.010661+0000", 2024-03-30T13:47:16.375 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.052113598999999997 2024-03-30T13:47:16.375 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.375 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.375 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:16.375 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:34.010702+0000", 2024-03-30T13:47:16.375 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1162999999999999e-05 2024-03-30T13:47:16.375 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.375 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.375 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:16.375 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:34.010738+0000", 2024-03-30T13:47:16.376 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.5722000000000001e-05 2024-03-30T13:47:16.376 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.376 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.376 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.376 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.376 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.376 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:131453 7.0 7:04b35a9d:::benchmark_data_smithi043_86989_object9115:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T13:47:16.376 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:39.388543+0000", 2024-03-30T13:47:16.376 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 36.898300085000002, 2024-03-30T13:47:16.376 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.054741994000000002, 2024-03-30T13:47:16.376 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:16.376 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:16.376 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:16.376 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:16.377 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:16.377 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:16.377 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 131453 2024-03-30T13:47:16.377 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.377 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:16.377 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.377 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:16.377 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.388543+0000", 2024-03-30T13:47:16.377 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.377 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.377 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.377 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:16.377 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.388543+0000", 2024-03-30T13:47:16.377 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.378 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.378 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.378 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:16.378 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.388544+0000", 2024-03-30T13:47:16.378 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.085e-06 2024-03-30T13:47:16.378 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.378 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.378 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:16.378 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.388549+0000", 2024-03-30T13:47:16.378 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9629999999999997e-06 2024-03-30T13:47:16.378 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.378 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.378 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:16.378 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.388560+0000", 2024-03-30T13:47:16.379 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1161e-05 2024-03-30T13:47:16.379 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.379 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.379 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:16.379 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.388566+0000", 2024-03-30T13:47:16.379 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.801e-06 2024-03-30T13:47:16.379 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.379 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.379 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:16.379 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.388622+0000", 2024-03-30T13:47:16.379 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5290000000000001e-05 2024-03-30T13:47:16.379 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.379 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.380 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:16.380 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.388754+0000", 2024-03-30T13:47:16.380 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013217400000000001 2024-03-30T13:47:16.380 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.380 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.380 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:47:16.380 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.388867+0000", 2024-03-30T13:47:16.380 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011345699999999999 2024-03-30T13:47:16.380 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.380 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.380 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:16.380 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.391645+0000", 2024-03-30T13:47:16.380 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0027780840000000001 2024-03-30T13:47:16.380 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.381 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.381 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.381 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.391778+0000", 2024-03-30T13:47:16.381 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013242000000000001 2024-03-30T13:47:16.381 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.381 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.381 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.381 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.443120+0000", 2024-03-30T13:47:16.381 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.051342266999999997 2024-03-30T13:47:16.381 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.381 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.381 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:16.381 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.443220+0000", 2024-03-30T13:47:16.381 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.9804999999999997e-05 2024-03-30T13:47:16.382 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.382 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.382 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:16.382 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.443285+0000", 2024-03-30T13:47:16.382 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5487000000000005e-05 2024-03-30T13:47:16.382 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.382 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.382 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.382 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.382 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.382 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:131564 7.0 7:031570b0:::benchmark_data_smithi043_86989_object9226:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T13:47:16.382 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:39.532529+0000", 2024-03-30T13:47:16.382 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 36.754314237000003, 2024-03-30T13:47:16.383 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.075576455000000001, 2024-03-30T13:47:16.383 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:16.383 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:16.383 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:16.383 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:16.383 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:16.383 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:16.383 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 131564 2024-03-30T13:47:16.383 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.383 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:16.383 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.383 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:16.383 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.532529+0000", 2024-03-30T13:47:16.383 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.384 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.384 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.384 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:16.384 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.532529+0000", 2024-03-30T13:47:16.384 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.384 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.384 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.384 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:16.384 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.532530+0000", 2024-03-30T13:47:16.384 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.13e-06 2024-03-30T13:47:16.384 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.384 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.384 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:16.385 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.532535+0000", 2024-03-30T13:47:16.385 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.899e-06 2024-03-30T13:47:16.385 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.385 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.385 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:16.385 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.532545+0000", 2024-03-30T13:47:16.385 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0127e-05 2024-03-30T13:47:16.385 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.385 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.385 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:16.385 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.532553+0000", 2024-03-30T13:47:16.385 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.4730000000000003e-06 2024-03-30T13:47:16.385 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.385 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.386 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:16.386 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.532599+0000", 2024-03-30T13:47:16.386 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6424000000000001e-05 2024-03-30T13:47:16.386 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.386 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.386 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:16.386 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.532695+0000", 2024-03-30T13:47:16.386 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.5408999999999994e-05 2024-03-30T13:47:16.386 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.386 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.386 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:47:16.386 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.532775+0000", 2024-03-30T13:47:16.386 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.0661000000000006e-05 2024-03-30T13:47:16.387 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.387 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.387 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:16.387 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.538859+0000", 2024-03-30T13:47:16.387 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0060833459999999999 2024-03-30T13:47:16.387 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.387 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.387 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.387 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.541931+0000", 2024-03-30T13:47:16.387 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0030721709999999998 2024-03-30T13:47:16.387 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.387 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.387 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.387 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.608029+0000", 2024-03-30T13:47:16.388 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.066098373000000002 2024-03-30T13:47:16.388 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.388 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.388 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:16.388 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.608069+0000", 2024-03-30T13:47:16.388 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.9762999999999999e-05 2024-03-30T13:47:16.388 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.388 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.388 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:16.388 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.608106+0000", 2024-03-30T13:47:16.388 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.6678999999999999e-05 2024-03-30T13:47:16.388 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.388 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.388 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.389 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.389 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.389 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:131567 7.0 7:0171ba3d:::benchmark_data_smithi043_86989_object9229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T13:47:16.389 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:39.535444+0000", 2024-03-30T13:47:16.389 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 36.751399550000002, 2024-03-30T13:47:16.389 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.072807943999999999, 2024-03-30T13:47:16.389 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:16.389 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:16.389 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:16.389 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:16.389 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:16.389 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:16.389 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 131567 2024-03-30T13:47:16.389 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.389 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:16.390 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.390 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:16.390 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.535444+0000", 2024-03-30T13:47:16.390 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.390 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.390 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.390 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:16.390 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.535444+0000", 2024-03-30T13:47:16.390 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.390 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.390 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.390 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:16.390 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.535445+0000", 2024-03-30T13:47:16.391 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.8000000000000004e-07 2024-03-30T13:47:16.391 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.391 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.391 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:16.391 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.535450+0000", 2024-03-30T13:47:16.391 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4249999999999997e-06 2024-03-30T13:47:16.391 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.391 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.391 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:16.391 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.535465+0000", 2024-03-30T13:47:16.391 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.505e-05 2024-03-30T13:47:16.391 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.391 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.391 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:16.392 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.535474+0000", 2024-03-30T13:47:16.392 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.7169999999999997e-06 2024-03-30T13:47:16.392 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.392 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.392 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:16.392 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.535559+0000", 2024-03-30T13:47:16.392 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5337e-05 2024-03-30T13:47:16.392 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.392 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.392 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:16.392 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.535683+0000", 2024-03-30T13:47:16.392 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012394700000000001 2024-03-30T13:47:16.392 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.392 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.393 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:47:16.393 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.535796+0000", 2024-03-30T13:47:16.393 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011249800000000001 2024-03-30T13:47:16.393 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.393 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.393 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:16.393 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.539399+0000", 2024-03-30T13:47:16.393 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.003603635 2024-03-30T13:47:16.393 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.393 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.393 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.393 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.587167+0000", 2024-03-30T13:47:16.393 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.047768127000000001 2024-03-30T13:47:16.393 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.394 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.394 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.394 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.608149+0000", 2024-03-30T13:47:16.394 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.020981132 2024-03-30T13:47:16.394 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.394 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.394 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:16.394 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.608205+0000", 2024-03-30T13:47:16.394 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6798000000000001e-05 2024-03-30T13:47:16.394 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.394 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.394 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:16.394 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:39.608252+0000", 2024-03-30T13:47:16.394 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6397999999999999e-05 2024-03-30T13:47:16.395 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.395 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.395 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.395 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.395 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.395 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:146016 7.0 7:0523db38:::benchmark_data_smithi043_86989_object23678:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-30T13:47:16.395 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:55.848026+0000", 2024-03-30T13:47:16.395 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 20.438817471, 2024-03-30T13:47:16.395 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.075848738999999998, 2024-03-30T13:47:16.395 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:16.395 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:16.395 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:16.395 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:16.396 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:16.396 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:16.396 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 146016 2024-03-30T13:47:16.396 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.396 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:16.396 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.396 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:16.396 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.848026+0000", 2024-03-30T13:47:16.396 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.396 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.396 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.396 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:16.396 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.848026+0000", 2024-03-30T13:47:16.396 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.396 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.397 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.397 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:16.397 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.848028+0000", 2024-03-30T13:47:16.397 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.7430000000000001e-06 2024-03-30T13:47:16.397 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.397 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.397 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:16.397 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.848036+0000", 2024-03-30T13:47:16.397 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5399999999999996e-06 2024-03-30T13:47:16.397 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.397 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.397 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:16.397 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.848055+0000", 2024-03-30T13:47:16.397 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.9196000000000001e-05 2024-03-30T13:47:16.398 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.398 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.398 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:16.398 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.848066+0000", 2024-03-30T13:47:16.398 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0963e-05 2024-03-30T13:47:16.398 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.398 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.398 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:16.398 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.848139+0000", 2024-03-30T13:47:16.398 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.2745999999999998e-05 2024-03-30T13:47:16.398 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.398 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.398 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:16.399 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.848279+0000", 2024-03-30T13:47:16.399 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013969000000000001 2024-03-30T13:47:16.399 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.399 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.399 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:47:16.399 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.848422+0000", 2024-03-30T13:47:16.399 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000143005 2024-03-30T13:47:16.399 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.399 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.399 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:16.399 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.851302+0000", 2024-03-30T13:47:16.399 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0028803890000000001 2024-03-30T13:47:16.399 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.400 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.400 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.400 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.851764+0000", 2024-03-30T13:47:16.400 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000461955 2024-03-30T13:47:16.400 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.400 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.400 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.400 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.923771+0000", 2024-03-30T13:47:16.400 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.072006596000000006 2024-03-30T13:47:16.400 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.400 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.400 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:16.400 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.923824+0000", 2024-03-30T13:47:16.400 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2997000000000001e-05 2024-03-30T13:47:16.401 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.401 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.401 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:16.401 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.923875+0000", 2024-03-30T13:47:16.401 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0918999999999998e-05 2024-03-30T13:47:16.401 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.401 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.401 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.401 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.401 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.401 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:146027 7.0 7:0624f1a7:::benchmark_data_smithi043_86989_object23689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-30T13:47:16.401 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:55.859056+0000", 2024-03-30T13:47:16.401 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 20.427787224999999, 2024-03-30T13:47:16.401 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.079134809, 2024-03-30T13:47:16.402 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:16.402 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:16.402 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:16.402 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:16.402 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:16.402 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:16.402 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 146027 2024-03-30T13:47:16.402 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.402 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:16.402 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.402 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:16.402 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.859056+0000", 2024-03-30T13:47:16.402 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.402 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.403 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.403 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:16.403 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.859056+0000", 2024-03-30T13:47:16.403 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.403 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.403 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.403 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:16.403 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.859058+0000", 2024-03-30T13:47:16.403 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.57e-06 2024-03-30T13:47:16.403 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.403 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.403 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:16.403 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.859065+0000", 2024-03-30T13:47:16.403 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.6210000000000004e-06 2024-03-30T13:47:16.403 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.404 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.404 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:16.404 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.859079+0000", 2024-03-30T13:47:16.404 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3961000000000001e-05 2024-03-30T13:47:16.404 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.404 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.404 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:16.404 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.859088+0000", 2024-03-30T13:47:16.404 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.9709999999999993e-06 2024-03-30T13:47:16.404 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.404 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.404 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:16.404 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.859158+0000", 2024-03-30T13:47:16.405 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.0240999999999994e-05 2024-03-30T13:47:16.405 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.405 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.405 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:16.405 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.859302+0000", 2024-03-30T13:47:16.405 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00014327200000000001 2024-03-30T13:47:16.405 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.405 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.405 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:47:16.405 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.859432+0000", 2024-03-30T13:47:16.405 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000129932 2024-03-30T13:47:16.405 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.405 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.405 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.406 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.862282+0000", 2024-03-30T13:47:16.406 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0028500969999999998 2024-03-30T13:47:16.406 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.406 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.406 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:16.406 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.862313+0000", 2024-03-30T13:47:16.406 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.1152000000000003e-05 2024-03-30T13:47:16.406 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.406 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.406 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.406 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.938112+0000", 2024-03-30T13:47:16.406 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.075798692000000001 2024-03-30T13:47:16.406 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.406 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.407 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:16.407 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.938149+0000", 2024-03-30T13:47:16.407 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.7839000000000001e-05 2024-03-30T13:47:16.407 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.407 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.407 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:16.407 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.938191+0000", 2024-03-30T13:47:16.407 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1461000000000002e-05 2024-03-30T13:47:16.407 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.407 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.407 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.407 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.407 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.407 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:149183 7.0 7:0bb2867c:::benchmark_data_smithi043_86989_object26845:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-30T13:47:16.408 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:59.222265+0000", 2024-03-30T13:47:16.408 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 17.064577841999998, 2024-03-30T13:47:16.408 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.049204349000000001, 2024-03-30T13:47:16.408 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:16.408 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:16.408 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:16.408 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:16.408 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:16.408 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:16.408 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 149183 2024-03-30T13:47:16.408 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.408 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:16.408 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.408 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:16.409 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.222265+0000", 2024-03-30T13:47:16.409 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.409 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.409 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.409 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:16.409 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.222265+0000", 2024-03-30T13:47:16.409 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.409 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.409 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.409 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:16.409 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.222267+0000", 2024-03-30T13:47:16.409 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.389e-06 2024-03-30T13:47:16.409 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.409 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.410 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:16.410 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.222273+0000", 2024-03-30T13:47:16.410 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1059999999999996e-06 2024-03-30T13:47:16.410 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.410 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.410 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:16.410 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.222287+0000", 2024-03-30T13:47:16.410 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4080000000000001e-05 2024-03-30T13:47:16.410 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.410 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.410 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:16.410 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.222296+0000", 2024-03-30T13:47:16.410 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.8750000000000006e-06 2024-03-30T13:47:16.410 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.411 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.411 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:16.411 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.222629+0000", 2024-03-30T13:47:16.411 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00033306200000000002 2024-03-30T13:47:16.411 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.411 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.411 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:16.411 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.222754+0000", 2024-03-30T13:47:16.411 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012536900000000001 2024-03-30T13:47:16.411 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.411 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.411 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:47:16.411 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.222885+0000", 2024-03-30T13:47:16.411 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000130166 2024-03-30T13:47:16.411 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.412 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.412 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.412 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.226617+0000", 2024-03-30T13:47:16.412 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0037326299999999998 2024-03-30T13:47:16.412 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.412 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.412 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:16.412 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.228297+0000", 2024-03-30T13:47:16.412 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0016797279999999999 2024-03-30T13:47:16.412 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.412 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.412 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.412 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.271378+0000", 2024-03-30T13:47:16.412 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.043080809999999997 2024-03-30T13:47:16.413 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.413 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.413 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:16.413 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.271428+0000", 2024-03-30T13:47:16.413 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0658999999999997e-05 2024-03-30T13:47:16.413 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.413 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.413 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:16.413 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:59.271470+0000", 2024-03-30T13:47:16.413 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1474999999999998e-05 2024-03-30T13:47:16.413 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.413 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.413 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.413 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.414 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.414 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:151187 7.0 7:016989f8:::benchmark_data_smithi043_86989_object28849:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-30T13:47:16.414 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:01.678912+0000", 2024-03-30T13:47:16.414 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 14.607931338, 2024-03-30T13:47:16.414 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.045592858, 2024-03-30T13:47:16.414 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:16.414 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:16.414 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:16.414 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:16.414 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:16.414 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:16.414 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 151187 2024-03-30T13:47:16.414 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.414 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:16.415 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.415 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:16.415 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.678912+0000", 2024-03-30T13:47:16.415 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.415 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.415 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.415 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:16.415 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.678912+0000", 2024-03-30T13:47:16.415 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:16.415 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.415 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.415 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:16.415 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.678913+0000", 2024-03-30T13:47:16.415 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4899999999999999e-06 2024-03-30T13:47:16.416 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.416 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.416 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:16.416 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.678919+0000", 2024-03-30T13:47:16.416 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8819999999999998e-06 2024-03-30T13:47:16.416 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.416 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.416 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:16.416 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.678934+0000", 2024-03-30T13:47:16.416 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4508999999999999e-05 2024-03-30T13:47:16.416 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.416 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.416 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:16.416 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.678943+0000", 2024-03-30T13:47:16.417 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.7120000000000006e-06 2024-03-30T13:47:16.417 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.417 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.417 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:16.417 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.678999+0000", 2024-03-30T13:47:16.417 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6875e-05 2024-03-30T13:47:16.417 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.417 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.417 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:16.418 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:47:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:47:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:47:16.419 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.679140+0000", 2024-03-30T13:47:16.419 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00014065799999999999 2024-03-30T13:47:16.419 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.419 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.419 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:47:16.419 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.679291+0000", 2024-03-30T13:47:16.419 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00015055600000000001 2024-03-30T13:47:16.419 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.419 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.419 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:16.419 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.682211+0000", 2024-03-30T13:47:16.419 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0029199320000000001 2024-03-30T13:47:16.419 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.419 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.420 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.420 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.682672+0000", 2024-03-30T13:47:16.420 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00046096499999999998 2024-03-30T13:47:16.420 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.420 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.420 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:16.420 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.724402+0000", 2024-03-30T13:47:16.420 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.041730742000000001 2024-03-30T13:47:16.420 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.420 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.420 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:16.420 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.724469+0000", 2024-03-30T13:47:16.420 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.7070000000000007e-05 2024-03-30T13:47:16.420 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:16.421 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:16.421 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:16.421 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:01.724505+0000", 2024-03-30T13:47:16.421 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.5466999999999998e-05 2024-03-30T13:47:16.421 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.421 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.421 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.421 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:16.421 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:16.421 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:47:16.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:16.422 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:16.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:16.523 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:16.554 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:16.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:16.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:16.655 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:16.685 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:16.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:16.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:16.787 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:16.815 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:16 smithi043 ceph-mon[29963]: pgmap v161: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 979 op/s 2024-03-30T13:47:16.815 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:16 smithi043 ceph-mon[29963]: osdmap e126: 8 total, 8 up, 8 in 2024-03-30T13:47:16.815 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:16 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3165496448' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:47:16.817 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:16 smithi043 ceph-mon[42099]: pgmap v161: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 979 op/s 2024-03-30T13:47:16.817 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:16 smithi043 ceph-mon[42099]: osdmap e126: 8 total, 8 up, 8 in 2024-03-30T13:47:16.817 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:16 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3165496448' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:47:16.818 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T13:47:16.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:16.831 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:16.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:16.933 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:16.964 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:16.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:16.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:17.066 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:17.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:17.096 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:17.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:17.197 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:16 smithi113 ceph-mon[34341]: pgmap v161: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 979 op/s 2024-03-30T13:47:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:16 smithi113 ceph-mon[34341]: osdmap e126: 8 total, 8 up, 8 in 2024-03-30T13:47:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:16 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3165496448' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:47:17.238 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:17.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:17.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:17.340 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:17.368 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:17.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:17.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:17.410 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-30T13:47:17.471 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:17.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:17.519 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:17.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:17.620 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:17.656 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:17.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:17.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:17.758 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:17.781 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:47:17.801 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:17.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:17.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:17.903 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:17.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:17.928 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:17.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:18.029 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:18.060 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:17 smithi113 ceph-mon[34341]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:47:18.060 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:17 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3165496448' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:47:18.060 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:17 smithi113 ceph-mon[34341]: osdmap e127: 8 total, 8 up, 8 in 2024-03-30T13:47:18.067 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:18.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:18.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:18.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:17 smithi043 ceph-mon[29963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:47:18.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:17 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3165496448' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:47:18.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:17 smithi043 ceph-mon[29963]: osdmap e127: 8 total, 8 up, 8 in 2024-03-30T13:47:18.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:17 smithi043 ceph-mon[42099]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:47:18.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:17 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3165496448' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:47:18.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:17 smithi043 ceph-mon[42099]: osdmap e127: 8 total, 8 up, 8 in 2024-03-30T13:47:18.169 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:18.214 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:18.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:18.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:18.316 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:18.359 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:18.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:18.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:18.461 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:18.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:18.505 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:18.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:18.601 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:47:18.606 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:18.639 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:18.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:18.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:18.698 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:47:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:47:18.741 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:18.771 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:18.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:18.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:18.873 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:18.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:18.898 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:18.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:18.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:18 smithi113 ceph-mon[34341]: 4.2 scrub starts 2024-03-30T13:47:18.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:18 smithi113 ceph-mon[34341]: 4.2 scrub ok 2024-03-30T13:47:18.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:18 smithi113 ceph-mon[34341]: pgmap v164: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 627 op/s 2024-03-30T13:47:18.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:18 smithi113 ceph-mon[34341]: 5.1f scrub starts 2024-03-30T13:47:18.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:18 smithi113 ceph-mon[34341]: 5.1f scrub ok 2024-03-30T13:47:18.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:18 smithi113 ceph-mon[34341]: 7.6 scrub starts 2024-03-30T13:47:18.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:18 smithi113 ceph-mon[34341]: 7.1 scrub starts 2024-03-30T13:47:18.999 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:19.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:19.028 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:19.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:19.129 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:19.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:18 smithi043 ceph-mon[29963]: 4.2 scrub starts 2024-03-30T13:47:19.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:18 smithi043 ceph-mon[29963]: 4.2 scrub ok 2024-03-30T13:47:19.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:18 smithi043 ceph-mon[29963]: pgmap v164: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 627 op/s 2024-03-30T13:47:19.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:18 smithi043 ceph-mon[29963]: 5.1f scrub starts 2024-03-30T13:47:19.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:18 smithi043 ceph-mon[29963]: 5.1f scrub ok 2024-03-30T13:47:19.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:18 smithi043 ceph-mon[29963]: 7.6 scrub starts 2024-03-30T13:47:19.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:18 smithi043 ceph-mon[29963]: 7.1 scrub starts 2024-03-30T13:47:19.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:18 smithi043 ceph-mon[42099]: 4.2 scrub starts 2024-03-30T13:47:19.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:18 smithi043 ceph-mon[42099]: 4.2 scrub ok 2024-03-30T13:47:19.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:18 smithi043 ceph-mon[42099]: pgmap v164: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 627 op/s 2024-03-30T13:47:19.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:18 smithi043 ceph-mon[42099]: 5.1f scrub starts 2024-03-30T13:47:19.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:18 smithi043 ceph-mon[42099]: 5.1f scrub ok 2024-03-30T13:47:19.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:18 smithi043 ceph-mon[42099]: 7.6 scrub starts 2024-03-30T13:47:19.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:18 smithi043 ceph-mon[42099]: 7.1 scrub starts 2024-03-30T13:47:19.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:19.175 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:19.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:19.275 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:19.306 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:19.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:19.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:19.408 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:19.448 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:19.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:19.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:19.550 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:19.590 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:19.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:19.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:19.692 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:19.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:19.755 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:19.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:19.766 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:47:19.766 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [ 2024-03-30T13:47:19.766 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:19.766 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:168910 7.7 7:e3c82a6d:::benchmark_data_smithi043_86989_object46572:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-30T13:47:19.766 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:19.764296+0000", 2024-03-30T13:47:19.766 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.00055058899999999998, 2024-03-30T13:47:19.767 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00057348000000000002, 2024-03-30T13:47:19.767 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:19.767 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:19.767 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "waiting for sub ops", 2024-03-30T13:47:19.767 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:19.767 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:19.767 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:19.767 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 168910 2024-03-30T13:47:19.767 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:19.767 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:19.767 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:19.767 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:19.767 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:19.764296+0000", 2024-03-30T13:47:19.768 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:19.768 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:19.768 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:19.768 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:19.768 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:19.764296+0000", 2024-03-30T13:47:19.768 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:19.768 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:19.768 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:19.768 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:19.768 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:19.764297+0000", 2024-03-30T13:47:19.768 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0449999999999999e-06 2024-03-30T13:47:19.768 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:19.768 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:19.768 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:19.768 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:19.764302+0000", 2024-03-30T13:47:19.769 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0599999999999998e-06 2024-03-30T13:47:19.769 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:19.769 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:19.769 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:19.769 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:19.764316+0000", 2024-03-30T13:47:19.769 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3472000000000001e-05 2024-03-30T13:47:19.769 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:19.769 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:19.769 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:19.769 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:19.764322+0000", 2024-03-30T13:47:19.769 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.4439999999999996e-06 2024-03-30T13:47:19.769 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:19.769 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:19.769 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:19.770 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:19.764370+0000", 2024-03-30T13:47:19.770 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7613e-05 2024-03-30T13:47:19.770 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:19.770 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:19.770 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:19.770 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:19.764469+0000", 2024-03-30T13:47:19.770 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.8943000000000005e-05 2024-03-30T13:47:19.770 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:19.770 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:19.770 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,3", 2024-03-30T13:47:19.770 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:19.764549+0000", 2024-03-30T13:47:19.770 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.0611999999999993e-05 2024-03-30T13:47:19.770 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:19.770 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:19.770 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:19.771 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:19.771 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-03-30T13:47:19.771 INFO:teuthology.orchestra.run.smithi113.stdout: "num_ops": 1 2024-03-30T13:47:19.771 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:47:19.856 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:19.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:19.887 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:19.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:19.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:19 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:47:19.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:19 smithi043 ceph-mon[29963]: 4.e scrub starts 2024-03-30T13:47:19.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:19 smithi043 ceph-mon[29963]: 4.e scrub ok 2024-03-30T13:47:19.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:19 smithi043 ceph-mon[29963]: 7.1 scrub ok 2024-03-30T13:47:19.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:19 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:47:19.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:19 smithi043 ceph-mon[42099]: 4.e scrub starts 2024-03-30T13:47:19.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:19 smithi043 ceph-mon[42099]: 4.e scrub ok 2024-03-30T13:47:19.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:19 smithi043 ceph-mon[42099]: 7.1 scrub ok 2024-03-30T13:47:19.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:19 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:47:19.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:19 smithi113 ceph-mon[34341]: 4.e scrub starts 2024-03-30T13:47:19.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:19 smithi113 ceph-mon[34341]: 4.e scrub ok 2024-03-30T13:47:19.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:19 smithi113 ceph-mon[34341]: 7.1 scrub ok 2024-03-30T13:47:19.989 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:20.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:20.029 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:20.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:20.130 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:20.161 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:20.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:20.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:20.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:20.323 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:20.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:20.424 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:20.467 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:20.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:20.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:20.526 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-03-30T13:47:20.573 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:20.613 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:20.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:20.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:20.715 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:20.744 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:47:20.746 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:20.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:20.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:20.848 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:20.870 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:47:20.905 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:20.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:20.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:20.908 INFO:teuthology.orchestra.run.smithi043.stderr:no valid command found; 10 closest matches: 2024-03-30T13:47:20.908 INFO:teuthology.orchestra.run.smithi043.stderr:0 2024-03-30T13:47:20.908 INFO:teuthology.orchestra.run.smithi043.stderr:1 2024-03-30T13:47:20.908 INFO:teuthology.orchestra.run.smithi043.stderr:2 2024-03-30T13:47:20.908 INFO:teuthology.orchestra.run.smithi043.stderr:abort 2024-03-30T13:47:20.909 INFO:teuthology.orchestra.run.smithi043.stderr:assert 2024-03-30T13:47:20.909 INFO:teuthology.orchestra.run.smithi043.stderr:bench [] [] [] [] 2024-03-30T13:47:20.909 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs debug_inject_read_zeros 2024-03-30T13:47:20.909 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs files list 2024-03-30T13:47:20.909 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs stats 2024-03-30T13:47:20.909 INFO:teuthology.orchestra.run.smithi043.stderr:bluestore allocator dump block 2024-03-30T13:47:20.909 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: invalid command 2024-03-30T13:47:21.009 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:21.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:20 smithi043 ceph-mon[29963]: pgmap v165: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-30T13:47:21.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:20 smithi043 ceph-mon[29963]: 4.15 scrub starts 2024-03-30T13:47:21.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:20 smithi043 ceph-mon[29963]: 4.15 scrub ok 2024-03-30T13:47:21.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:20 smithi043 ceph-mon[29963]: 7.6 scrub ok 2024-03-30T13:47:21.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:20 smithi043 ceph-mon[29963]: 4.1c scrub starts 2024-03-30T13:47:21.044 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:20 smithi043 ceph-mon[29963]: 4.1c scrub ok 2024-03-30T13:47:21.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:20 smithi043 ceph-mon[42099]: pgmap v165: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-30T13:47:21.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:20 smithi043 ceph-mon[42099]: 4.15 scrub starts 2024-03-30T13:47:21.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:20 smithi043 ceph-mon[42099]: 4.15 scrub ok 2024-03-30T13:47:21.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:20 smithi043 ceph-mon[42099]: 7.6 scrub ok 2024-03-30T13:47:21.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:20 smithi043 ceph-mon[42099]: 4.1c scrub starts 2024-03-30T13:47:21.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:20 smithi043 ceph-mon[42099]: 4.1c scrub ok 2024-03-30T13:47:21.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:21.046 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:21.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:21.147 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:21.151 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:20 smithi113 ceph-mon[34341]: pgmap v165: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-30T13:47:21.151 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:20 smithi113 ceph-mon[34341]: 4.15 scrub starts 2024-03-30T13:47:21.151 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:20 smithi113 ceph-mon[34341]: 4.15 scrub ok 2024-03-30T13:47:21.152 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:20 smithi113 ceph-mon[34341]: 7.6 scrub ok 2024-03-30T13:47:21.152 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:20 smithi113 ceph-mon[34341]: 4.1c scrub starts 2024-03-30T13:47:21.152 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:20 smithi113 ceph-mon[34341]: 4.1c scrub ok 2024-03-30T13:47:21.188 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:21.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:21.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:21.289 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:21.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:21.313 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:21.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:21.415 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:21.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:21.450 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:21.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:21.551 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:21.589 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:21.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:21.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:21.691 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:21.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:21.740 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:21.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:21.841 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:21.869 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:21.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:21.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:21.972 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:22.003 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:22.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:22.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:22.105 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:22.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:22.150 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:22.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:22.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:21 smithi043 ceph-mon[29963]: 5.7 scrub starts 2024-03-30T13:47:22.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:21 smithi043 ceph-mon[29963]: 5.7 scrub ok 2024-03-30T13:47:22.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:21 smithi043 ceph-mon[29963]: 3.6 scrub starts 2024-03-30T13:47:22.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:21 smithi043 ceph-mon[29963]: 3.6 scrub ok 2024-03-30T13:47:22.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:21 smithi043 ceph-mon[29963]: 4.1f scrub starts 2024-03-30T13:47:22.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:21 smithi043 ceph-mon[29963]: 4.1f scrub ok 2024-03-30T13:47:22.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:21 smithi043 ceph-mon[29963]: 5.1 scrub starts 2024-03-30T13:47:22.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:21 smithi043 ceph-mon[29963]: 5.1 scrub ok 2024-03-30T13:47:22.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:21 smithi043 ceph-mon[42099]: 5.7 scrub starts 2024-03-30T13:47:22.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:21 smithi043 ceph-mon[42099]: 5.7 scrub ok 2024-03-30T13:47:22.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:21 smithi043 ceph-mon[42099]: 3.6 scrub starts 2024-03-30T13:47:22.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:21 smithi043 ceph-mon[42099]: 3.6 scrub ok 2024-03-30T13:47:22.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:21 smithi043 ceph-mon[42099]: 4.1f scrub starts 2024-03-30T13:47:22.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:21 smithi043 ceph-mon[42099]: 4.1f scrub ok 2024-03-30T13:47:22.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:21 smithi043 ceph-mon[42099]: 5.1 scrub starts 2024-03-30T13:47:22.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:21 smithi043 ceph-mon[42099]: 5.1 scrub ok 2024-03-30T13:47:22.211 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T13:47:22.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:21 smithi113 ceph-mon[34341]: 5.7 scrub starts 2024-03-30T13:47:22.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:21 smithi113 ceph-mon[34341]: 5.7 scrub ok 2024-03-30T13:47:22.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:21 smithi113 ceph-mon[34341]: 3.6 scrub starts 2024-03-30T13:47:22.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:21 smithi113 ceph-mon[34341]: 3.6 scrub ok 2024-03-30T13:47:22.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:21 smithi113 ceph-mon[34341]: 4.1f scrub starts 2024-03-30T13:47:22.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:21 smithi113 ceph-mon[34341]: 4.1f scrub ok 2024-03-30T13:47:22.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:21 smithi113 ceph-mon[34341]: 5.1 scrub starts 2024-03-30T13:47:22.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:21 smithi113 ceph-mon[34341]: 5.1 scrub ok 2024-03-30T13:47:22.251 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:22.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:22.309 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:22.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:22.410 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:22.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:22.448 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:22.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:22.550 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:22.636 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:22.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:22.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:22.738 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:22.789 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:22.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:22.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:22.853 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:22 smithi113 ceph-mon[34341]: pgmap v166: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-30T13:47:22.891 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:22.926 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:47:22.927 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [], 2024-03-30T13:47:22.927 INFO:teuthology.orchestra.run.smithi113.stdout: "complaint_time": 30, 2024-03-30T13:47:22.927 INFO:teuthology.orchestra.run.smithi113.stdout: "num_blocked_ops": 0 2024-03-30T13:47:22.927 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:47:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:22.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:22.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:23.036 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:23.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:23.066 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:23.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:23.069 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:47:23.069 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-30T13:47:23.069 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 'trim stale osdmaps' 2024-03-30T13:47:23.111 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:47:23.135 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:22 smithi043 ceph-mon[29963]: pgmap v166: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-30T13:47:23.136 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:22 smithi043 ceph-mon[42099]: pgmap v166: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-30T13:47:23.167 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:23.223 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:23.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:23.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:23.324 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:23.356 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:23.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:23.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:23.458 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:23.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:23.507 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:23.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:23.608 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:23.640 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:23.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:23.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:23.731 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-03-30T13:47:23.742 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:23.786 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:23.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:23.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:23.888 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:23.932 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:23.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:23.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:23.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:23 smithi113 ceph-mon[34341]: 3.c scrub starts 2024-03-30T13:47:23.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:23 smithi113 ceph-mon[34341]: 3.c scrub ok 2024-03-30T13:47:23.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:23 smithi113 ceph-mon[34341]: 7.7 scrub starts 2024-03-30T13:47:23.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:23 smithi113 ceph-mon[34341]: 3.0 scrub starts 2024-03-30T13:47:23.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:23 smithi113 ceph-mon[34341]: 3.0 scrub ok 2024-03-30T13:47:23.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:23 smithi113 ceph-mon[34341]: 5.0 scrub starts 2024-03-30T13:47:23.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:23 smithi113 ceph-mon[34341]: 5.0 scrub ok 2024-03-30T13:47:24.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:23 smithi043 ceph-mon[29963]: 3.c scrub starts 2024-03-30T13:47:24.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:23 smithi043 ceph-mon[29963]: 3.c scrub ok 2024-03-30T13:47:24.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:23 smithi043 ceph-mon[29963]: 7.7 scrub starts 2024-03-30T13:47:24.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:23 smithi043 ceph-mon[29963]: 3.0 scrub starts 2024-03-30T13:47:24.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:23 smithi043 ceph-mon[29963]: 3.0 scrub ok 2024-03-30T13:47:24.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:23 smithi043 ceph-mon[29963]: 5.0 scrub starts 2024-03-30T13:47:24.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:23 smithi043 ceph-mon[29963]: 5.0 scrub ok 2024-03-30T13:47:24.019 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:23 smithi043 ceph-mon[42099]: 3.c scrub starts 2024-03-30T13:47:24.020 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:23 smithi043 ceph-mon[42099]: 3.c scrub ok 2024-03-30T13:47:24.020 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:23 smithi043 ceph-mon[42099]: 7.7 scrub starts 2024-03-30T13:47:24.020 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:23 smithi043 ceph-mon[42099]: 3.0 scrub starts 2024-03-30T13:47:24.020 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:23 smithi043 ceph-mon[42099]: 3.0 scrub ok 2024-03-30T13:47:24.020 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:23 smithi043 ceph-mon[42099]: 5.0 scrub starts 2024-03-30T13:47:24.020 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:23 smithi043 ceph-mon[42099]: 5.0 scrub ok 2024-03-30T13:47:24.034 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:24.072 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:24.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:24.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:24.074 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:47:24.175 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:24.204 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:24.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:24.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:24.305 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:24.341 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:24.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:24.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:24.443 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:24.463 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:47:24.463 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:47:24.463 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:47:24.463 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:47:24.463 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:47:24.463 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:47:24.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:24.478 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:24.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:24.493 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:47:24.543 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:47:24.552 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:47:24.552 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:47:24.552 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:47:24.552 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:47:24.552 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:47:24.552 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:47:24.552 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:47:24.552 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:47:24.579 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:24.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:24.610 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:24.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:24.711 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:24.748 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:24.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:24.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:24.850 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:24.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:24.879 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:24.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:24.981 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:25.010 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:25.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:25.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:25.111 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:25.143 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:25.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:25.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:25.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:24 smithi043 ceph-mon[29963]: pgmap v167: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 732 op/s 2024-03-30T13:47:25.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:24 smithi043 ceph-mon[29963]: 4.a scrub starts 2024-03-30T13:47:25.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:24 smithi043 ceph-mon[29963]: 4.a scrub ok 2024-03-30T13:47:25.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:24 smithi043 ceph-mon[29963]: 4.b scrub starts 2024-03-30T13:47:25.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:24 smithi043 ceph-mon[29963]: 4.b scrub ok 2024-03-30T13:47:25.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:24 smithi043 ceph-mon[29963]: 6.1a scrub starts 2024-03-30T13:47:25.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:24 smithi043 ceph-mon[29963]: 6.1a scrub ok 2024-03-30T13:47:25.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:24 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/935079198' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:47:25.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:24 smithi043 ceph-mon[42099]: pgmap v167: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 732 op/s 2024-03-30T13:47:25.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:24 smithi043 ceph-mon[42099]: 4.a scrub starts 2024-03-30T13:47:25.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:24 smithi043 ceph-mon[42099]: 4.a scrub ok 2024-03-30T13:47:25.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:24 smithi043 ceph-mon[42099]: 4.b scrub starts 2024-03-30T13:47:25.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:24 smithi043 ceph-mon[42099]: 4.b scrub ok 2024-03-30T13:47:25.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:24 smithi043 ceph-mon[42099]: 6.1a scrub starts 2024-03-30T13:47:25.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:24 smithi043 ceph-mon[42099]: 6.1a scrub ok 2024-03-30T13:47:25.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:24 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/935079198' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:47:25.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:24 smithi113 ceph-mon[34341]: pgmap v167: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 732 op/s 2024-03-30T13:47:25.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:24 smithi113 ceph-mon[34341]: 4.a scrub starts 2024-03-30T13:47:25.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:24 smithi113 ceph-mon[34341]: 4.a scrub ok 2024-03-30T13:47:25.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:24 smithi113 ceph-mon[34341]: 4.b scrub starts 2024-03-30T13:47:25.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:24 smithi113 ceph-mon[34341]: 4.b scrub ok 2024-03-30T13:47:25.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:24 smithi113 ceph-mon[34341]: 6.1a scrub starts 2024-03-30T13:47:25.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:24 smithi113 ceph-mon[34341]: 6.1a scrub ok 2024-03-30T13:47:25.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:24 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/935079198' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:47:25.245 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:25.285 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:25.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:25.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:25.387 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:25.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:25.425 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:25.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:25.526 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:25.567 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:25.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:25.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:25.669 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:25.726 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:25.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:25.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:25.828 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:25.860 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:25.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:25.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:25.962 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:25.990 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:25.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:25.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:26.004 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:25 smithi043 ceph-mon[29963]: 2.0 scrub starts 2024-03-30T13:47:26.004 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:25 smithi043 ceph-mon[29963]: 2.0 scrub ok 2024-03-30T13:47:26.004 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:25 smithi043 ceph-mon[29963]: 7.7 scrub ok 2024-03-30T13:47:26.004 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:25 smithi043 ceph-mon[29963]: 4.1b scrub starts 2024-03-30T13:47:26.004 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:25 smithi043 ceph-mon[29963]: 4.1b scrub ok 2024-03-30T13:47:26.004 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:25 smithi043 ceph-mon[29963]: 5.13 scrub starts 2024-03-30T13:47:26.004 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:25 smithi043 ceph-mon[29963]: 5.13 scrub ok 2024-03-30T13:47:26.005 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:25 smithi043 ceph-mon[42099]: 2.0 scrub starts 2024-03-30T13:47:26.005 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:25 smithi043 ceph-mon[42099]: 2.0 scrub ok 2024-03-30T13:47:26.005 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:25 smithi043 ceph-mon[42099]: 7.7 scrub ok 2024-03-30T13:47:26.005 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:25 smithi043 ceph-mon[42099]: 4.1b scrub starts 2024-03-30T13:47:26.005 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:25 smithi043 ceph-mon[42099]: 4.1b scrub ok 2024-03-30T13:47:26.005 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:25 smithi043 ceph-mon[42099]: 5.13 scrub starts 2024-03-30T13:47:26.005 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:25 smithi043 ceph-mon[42099]: 5.13 scrub ok 2024-03-30T13:47:26.092 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:26.122 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:26.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:26.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:26.171 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:25 smithi113 ceph-mon[34341]: 2.0 scrub starts 2024-03-30T13:47:26.171 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:25 smithi113 ceph-mon[34341]: 2.0 scrub ok 2024-03-30T13:47:26.171 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:25 smithi113 ceph-mon[34341]: 7.7 scrub ok 2024-03-30T13:47:26.171 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:25 smithi113 ceph-mon[34341]: 4.1b scrub starts 2024-03-30T13:47:26.171 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:25 smithi113 ceph-mon[34341]: 4.1b scrub ok 2024-03-30T13:47:26.171 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:25 smithi113 ceph-mon[34341]: 5.13 scrub starts 2024-03-30T13:47:26.171 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:25 smithi113 ceph-mon[34341]: 5.13 scrub ok 2024-03-30T13:47:26.224 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:26.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:26.252 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:26.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:26.278 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:47:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:47:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:47:26.354 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:26.363 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:47:26.418 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:26.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:26.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:26.520 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:26.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:26.585 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:26.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:26.687 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:26.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:26.734 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:26.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:26.808 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:47:26.808 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:47:26.835 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:26.867 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:26.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:26.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:26.870 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-03-30T13:47:26.969 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:27.008 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:27.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:27.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:27.109 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:27.139 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:27.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:27.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:27.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:26 smithi043 ceph-mon[29963]: pgmap v168: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-30T13:47:27.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:26 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2631853793' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:47:27.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:26 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:47:27.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:26 smithi043 ceph-mon[42099]: pgmap v168: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-30T13:47:27.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:26 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2631853793' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:47:27.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:26 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:47:27.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:26 smithi113 ceph-mon[34341]: pgmap v168: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-30T13:47:27.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:26 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2631853793' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:47:27.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:26 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:47:27.241 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:27.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:27.279 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:27.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:27.380 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:27.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:27.425 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:27.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:27.526 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:27.566 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:27.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:27.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:27.669 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:27.697 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:27.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:27.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:27.715 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:47:27.798 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:27.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:27.844 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:27.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:27.926 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-30T13:47:27.946 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:27.991 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:27.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:27.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:28.093 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:28.133 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:28.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:28.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:28.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:27 smithi043 ceph-mon[29963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:47:28.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:27 smithi043 ceph-mon[29963]: Cluster is now healthy 2024-03-30T13:47:28.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:27 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:47:28.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:27 smithi043 ceph-mon[29963]: osdmap e128: 8 total, 8 up, 8 in 2024-03-30T13:47:28.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:28.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:47:28.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:27 smithi043 ceph-mon[42099]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:47:28.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:27 smithi043 ceph-mon[42099]: Cluster is now healthy 2024-03-30T13:47:28.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:27 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:47:28.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:27 smithi043 ceph-mon[42099]: osdmap e128: 8 total, 8 up, 8 in 2024-03-30T13:47:28.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:28.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:47:28.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:27 smithi113 ceph-mon[34341]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:47:28.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:27 smithi113 ceph-mon[34341]: Cluster is now healthy 2024-03-30T13:47:28.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:27 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:47:28.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:27 smithi113 ceph-mon[34341]: osdmap e128: 8 total, 8 up, 8 in 2024-03-30T13:47:28.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:28.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:47:28.235 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:28.244 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:47:28.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:28.263 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:28.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:28.364 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:28.400 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:28.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:28.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:28.502 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:28.544 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:28.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:28.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:28.646 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:28.674 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:28.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:28.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:28.693 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:47:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:47:28.776 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:28.818 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:28.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:28.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:28.920 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:28.959 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:28.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:28.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:28.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:28 smithi113 ceph-mon[34341]: pgmap v170: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:47:28.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:28 smithi113 ceph-mon[34341]: 3.17 scrub starts 2024-03-30T13:47:28.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:28 smithi113 ceph-mon[34341]: 3.17 scrub ok 2024-03-30T13:47:28.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:28 smithi113 ceph-mon[34341]: 5.a deep-scrub starts 2024-03-30T13:47:28.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:28 smithi113 ceph-mon[34341]: 5.a deep-scrub ok 2024-03-30T13:47:29.061 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:29.093 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:29.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:29.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:29.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:28 smithi043 ceph-mon[29963]: pgmap v170: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:47:29.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:28 smithi043 ceph-mon[29963]: 3.17 scrub starts 2024-03-30T13:47:29.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:28 smithi043 ceph-mon[29963]: 3.17 scrub ok 2024-03-30T13:47:29.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:28 smithi043 ceph-mon[29963]: 5.a deep-scrub starts 2024-03-30T13:47:29.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:28 smithi043 ceph-mon[29963]: 5.a deep-scrub ok 2024-03-30T13:47:29.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:28 smithi043 ceph-mon[42099]: pgmap v170: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:47:29.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:28 smithi043 ceph-mon[42099]: 3.17 scrub starts 2024-03-30T13:47:29.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:28 smithi043 ceph-mon[42099]: 3.17 scrub ok 2024-03-30T13:47:29.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:28 smithi043 ceph-mon[42099]: 5.a deep-scrub starts 2024-03-30T13:47:29.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:28 smithi043 ceph-mon[42099]: 5.a deep-scrub ok 2024-03-30T13:47:29.195 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:29.233 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:29.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:29.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:29.335 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:29.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:29.375 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:29.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:29.476 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:29.516 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:29.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:29.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:29.618 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:29.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:29.655 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:29.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:29.756 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:29.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:29.790 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:29.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:29.891 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:29.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:29.932 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:29.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:30.033 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:30.061 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:30.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:30.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:30.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:29 smithi043 ceph-mon[29963]: 5.19 scrub starts 2024-03-30T13:47:30.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:29 smithi043 ceph-mon[29963]: 5.19 scrub ok 2024-03-30T13:47:30.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:29 smithi043 ceph-mon[29963]: 3.1c scrub starts 2024-03-30T13:47:30.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:29 smithi043 ceph-mon[29963]: 3.1c scrub ok 2024-03-30T13:47:30.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:29 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:47:30.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:29 smithi043 ceph-mon[29963]: 4.6 deep-scrub starts 2024-03-30T13:47:30.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:29 smithi043 ceph-mon[29963]: 4.6 deep-scrub ok 2024-03-30T13:47:30.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:29 smithi043 ceph-mon[42099]: 5.19 scrub starts 2024-03-30T13:47:30.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:29 smithi043 ceph-mon[42099]: 5.19 scrub ok 2024-03-30T13:47:30.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:29 smithi043 ceph-mon[42099]: 3.1c scrub starts 2024-03-30T13:47:30.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:29 smithi043 ceph-mon[42099]: 3.1c scrub ok 2024-03-30T13:47:30.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:29 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:47:30.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:29 smithi043 ceph-mon[42099]: 4.6 deep-scrub starts 2024-03-30T13:47:30.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:29 smithi043 ceph-mon[42099]: 4.6 deep-scrub ok 2024-03-30T13:47:30.162 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:30.194 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:30.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:30.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:30.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:29 smithi113 ceph-mon[34341]: 5.19 scrub starts 2024-03-30T13:47:30.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:29 smithi113 ceph-mon[34341]: 5.19 scrub ok 2024-03-30T13:47:30.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:29 smithi113 ceph-mon[34341]: 3.1c scrub starts 2024-03-30T13:47:30.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:29 smithi113 ceph-mon[34341]: 3.1c scrub ok 2024-03-30T13:47:30.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:29 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:47:30.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:29 smithi113 ceph-mon[34341]: 4.6 deep-scrub starts 2024-03-30T13:47:30.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:29 smithi113 ceph-mon[34341]: 4.6 deep-scrub ok 2024-03-30T13:47:30.296 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:30.323 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:30.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:30.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:30.329 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:47:30.329 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:47:30.425 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:30.453 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:30.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:30.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:30.555 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:30.595 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:30.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:30.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:30.697 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:30.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:30.735 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:30.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:30.836 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:30.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:30.878 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:30.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:30.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:30 smithi113 ceph-mon[34341]: 3.19 scrub starts 2024-03-30T13:47:30.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:30 smithi113 ceph-mon[34341]: 3.19 scrub ok 2024-03-30T13:47:30.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:30 smithi113 ceph-mon[34341]: pgmap v171: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:47:30.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:30 smithi113 ceph-mon[34341]: 5.5 scrub starts 2024-03-30T13:47:30.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:30 smithi113 ceph-mon[34341]: 5.5 scrub ok 2024-03-30T13:47:30.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:30 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:47:30.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:30 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:47:30.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:30 smithi113 ceph-mon[34341]: 4.17 scrub starts 2024-03-30T13:47:30.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:30 smithi113 ceph-mon[34341]: 4.17 scrub ok 2024-03-30T13:47:30.979 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:31.017 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:31.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:31.051 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-03-30T13:47:31.118 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:31.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:30 smithi043 ceph-mon[42099]: 3.19 scrub starts 2024-03-30T13:47:31.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:30 smithi043 ceph-mon[42099]: 3.19 scrub ok 2024-03-30T13:47:31.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:30 smithi043 ceph-mon[42099]: pgmap v171: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:47:31.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:30 smithi043 ceph-mon[42099]: 5.5 scrub starts 2024-03-30T13:47:31.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:30 smithi043 ceph-mon[42099]: 5.5 scrub ok 2024-03-30T13:47:31.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:30 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:47:31.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:30 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:47:31.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:30 smithi043 ceph-mon[42099]: 4.17 scrub starts 2024-03-30T13:47:31.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:30 smithi043 ceph-mon[42099]: 4.17 scrub ok 2024-03-30T13:47:31.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:30 smithi043 ceph-mon[29963]: 3.19 scrub starts 2024-03-30T13:47:31.152 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:30 smithi043 ceph-mon[29963]: 3.19 scrub ok 2024-03-30T13:47:31.152 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:30 smithi043 ceph-mon[29963]: pgmap v171: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:47:31.152 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:30 smithi043 ceph-mon[29963]: 5.5 scrub starts 2024-03-30T13:47:31.152 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:30 smithi043 ceph-mon[29963]: 5.5 scrub ok 2024-03-30T13:47:31.152 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:30 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:47:31.152 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:30 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:47:31.152 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:30 smithi043 ceph-mon[29963]: 4.17 scrub starts 2024-03-30T13:47:31.152 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:30 smithi043 ceph-mon[29963]: 4.17 scrub ok 2024-03-30T13:47:31.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:31.153 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:31.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:31.255 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:31.303 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:31.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:31.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:31.372 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:47:31.405 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:31.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:31.447 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:31.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:31.548 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:31.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:31.580 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:31.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:31.681 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:31.716 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:31.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:31.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:31.818 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:31.848 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:31.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:31.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:31.950 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:31.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:31.978 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:31.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:32.081 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:32.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:32.111 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:32.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:32.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:31 smithi043 ceph-mon[29963]: 4.12 scrub starts 2024-03-30T13:47:32.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:31 smithi043 ceph-mon[29963]: 4.12 scrub ok 2024-03-30T13:47:32.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:31 smithi043 ceph-mon[29963]: osdmap e129: 8 total, 8 up, 8 in 2024-03-30T13:47:32.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:31 smithi043 ceph-mon[29963]: 5.b scrub starts 2024-03-30T13:47:32.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:31 smithi043 ceph-mon[29963]: 5.b scrub ok 2024-03-30T13:47:32.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:31 smithi043 ceph-mon[42099]: 4.12 scrub starts 2024-03-30T13:47:32.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:31 smithi043 ceph-mon[42099]: 4.12 scrub ok 2024-03-30T13:47:32.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:31 smithi043 ceph-mon[42099]: osdmap e129: 8 total, 8 up, 8 in 2024-03-30T13:47:32.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:31 smithi043 ceph-mon[42099]: 5.b scrub starts 2024-03-30T13:47:32.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:31 smithi043 ceph-mon[42099]: 5.b scrub ok 2024-03-30T13:47:32.212 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:32.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:31 smithi113 ceph-mon[34341]: 4.12 scrub starts 2024-03-30T13:47:32.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:31 smithi113 ceph-mon[34341]: 4.12 scrub ok 2024-03-30T13:47:32.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:31 smithi113 ceph-mon[34341]: osdmap e129: 8 total, 8 up, 8 in 2024-03-30T13:47:32.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:31 smithi113 ceph-mon[34341]: 5.b scrub starts 2024-03-30T13:47:32.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:31 smithi113 ceph-mon[34341]: 5.b scrub ok 2024-03-30T13:47:32.237 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:32.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:32.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:32.338 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:32.394 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:32.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:32.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:47:32.496 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:32.527 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:32.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:32.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:32.585 INFO:teuthology.orchestra.run.smithi043.stderr:no valid command found; 10 closest matches: 2024-03-30T13:47:32.585 INFO:teuthology.orchestra.run.smithi043.stderr:0 2024-03-30T13:47:32.585 INFO:teuthology.orchestra.run.smithi043.stderr:1 2024-03-30T13:47:32.585 INFO:teuthology.orchestra.run.smithi043.stderr:2 2024-03-30T13:47:32.585 INFO:teuthology.orchestra.run.smithi043.stderr:abort 2024-03-30T13:47:32.585 INFO:teuthology.orchestra.run.smithi043.stderr:assert 2024-03-30T13:47:32.585 INFO:teuthology.orchestra.run.smithi043.stderr:bench [] [] [] [] 2024-03-30T13:47:32.585 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs debug_inject_read_zeros 2024-03-30T13:47:32.585 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs files list 2024-03-30T13:47:32.585 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs stats 2024-03-30T13:47:32.585 INFO:teuthology.orchestra.run.smithi043.stderr:bluestore allocator dump block 2024-03-30T13:47:32.585 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: invalid command 2024-03-30T13:47:32.629 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:32.659 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:32.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:32.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:32.761 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:32.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:32.786 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:32.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:32.887 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:32.910 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:32.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:32.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:33.012 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:33.046 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:33.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:33.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:33.148 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:33.177 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:33.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:33.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:33.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:32 smithi043 ceph-mon[29963]: 7.5 deep-scrub starts 2024-03-30T13:47:33.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:32 smithi043 ceph-mon[29963]: pgmap v173: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-30T13:47:33.197 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:32 smithi043 ceph-mon[29963]: 2.2 scrub starts 2024-03-30T13:47:33.197 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:32 smithi043 ceph-mon[29963]: 2.2 scrub ok 2024-03-30T13:47:33.197 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:32 smithi043 ceph-mon[29963]: 6.3 deep-scrub starts 2024-03-30T13:47:33.197 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:32 smithi043 ceph-mon[29963]: 6.3 deep-scrub ok 2024-03-30T13:47:33.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:32 smithi043 ceph-mon[42099]: 7.5 deep-scrub starts 2024-03-30T13:47:33.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:32 smithi043 ceph-mon[42099]: pgmap v173: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-30T13:47:33.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:32 smithi043 ceph-mon[42099]: 2.2 scrub starts 2024-03-30T13:47:33.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:32 smithi043 ceph-mon[42099]: 2.2 scrub ok 2024-03-30T13:47:33.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:32 smithi043 ceph-mon[42099]: 6.3 deep-scrub starts 2024-03-30T13:47:33.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:32 smithi043 ceph-mon[42099]: 6.3 deep-scrub ok 2024-03-30T13:47:33.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:32 smithi113 ceph-mon[34341]: 7.5 deep-scrub starts 2024-03-30T13:47:33.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:32 smithi113 ceph-mon[34341]: pgmap v173: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-30T13:47:33.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:32 smithi113 ceph-mon[34341]: 2.2 scrub starts 2024-03-30T13:47:33.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:32 smithi113 ceph-mon[34341]: 2.2 scrub ok 2024-03-30T13:47:33.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:32 smithi113 ceph-mon[34341]: 6.3 deep-scrub starts 2024-03-30T13:47:33.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:32 smithi113 ceph-mon[34341]: 6.3 deep-scrub ok 2024-03-30T13:47:33.279 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:33.306 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:33.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:33.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:33.411 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:33.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:33.439 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:33.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:33.540 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:33.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:33.583 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:33.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:33.684 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:33.730 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:33.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:33.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:33.832 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:33.860 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:33.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:33.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:33.924 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T13:47:33.952 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:47:33.953 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-30T13:47:33.953 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 'trim stale osdmaps' 2024-03-30T13:47:33.962 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:33.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:33.987 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:33.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:34.087 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:34.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:34.115 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:34.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:34.216 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:34.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:33 smithi113 ceph-mon[34341]: 4.16 scrub starts 2024-03-30T13:47:34.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:34 smithi113 ceph-mon[34341]: 4.16 scrub ok 2024-03-30T13:47:34.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:34 smithi113 ceph-mon[34341]: 4.1 scrub starts 2024-03-30T13:47:34.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:34 smithi113 ceph-mon[34341]: 4.1 scrub ok 2024-03-30T13:47:34.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:34 smithi113 ceph-mon[34341]: 7.3 scrub starts 2024-03-30T13:47:34.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:34.280 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:34.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:34.382 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:34.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:33 smithi043 ceph-mon[29963]: 4.16 scrub starts 2024-03-30T13:47:34.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:33 smithi043 ceph-mon[29963]: 4.16 scrub ok 2024-03-30T13:47:34.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:33 smithi043 ceph-mon[29963]: 4.1 scrub starts 2024-03-30T13:47:34.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:33 smithi043 ceph-mon[29963]: 4.1 scrub ok 2024-03-30T13:47:34.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:33 smithi043 ceph-mon[29963]: 7.3 scrub starts 2024-03-30T13:47:34.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:33 smithi043 ceph-mon[42099]: 4.16 scrub starts 2024-03-30T13:47:34.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:33 smithi043 ceph-mon[42099]: 4.16 scrub ok 2024-03-30T13:47:34.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:33 smithi043 ceph-mon[42099]: 4.1 scrub starts 2024-03-30T13:47:34.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:33 smithi043 ceph-mon[42099]: 4.1 scrub ok 2024-03-30T13:47:34.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:33 smithi043 ceph-mon[42099]: 7.3 scrub starts 2024-03-30T13:47:34.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:34.425 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:34.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:34.525 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:34.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:34.568 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:34.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:34.669 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:34.737 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:47:34.739 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:34.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:34.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:34.841 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:34.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:34.888 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:34.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:34.989 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:35.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:35.029 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:35.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:35.130 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:35.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:34 smithi043 ceph-mon[29963]: 3.4 scrub starts 2024-03-30T13:47:35.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:34 smithi043 ceph-mon[29963]: 3.4 scrub ok 2024-03-30T13:47:35.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:34 smithi043 ceph-mon[29963]: pgmap v174: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 685 op/s 2024-03-30T13:47:35.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:34 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:47:35.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:34 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:47:35.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:34 smithi043 ceph-mon[29963]: 7.a scrub starts 2024-03-30T13:47:35.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:34 smithi043 ceph-mon[42099]: 3.4 scrub starts 2024-03-30T13:47:35.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:34 smithi043 ceph-mon[42099]: 3.4 scrub ok 2024-03-30T13:47:35.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:34 smithi043 ceph-mon[42099]: pgmap v174: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 685 op/s 2024-03-30T13:47:35.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:34 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:47:35.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:34 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:47:35.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:34 smithi043 ceph-mon[42099]: 7.a scrub starts 2024-03-30T13:47:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:35.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:35.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:35.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:34 smithi113 ceph-mon[34341]: 3.4 scrub starts 2024-03-30T13:47:35.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:34 smithi113 ceph-mon[34341]: 3.4 scrub ok 2024-03-30T13:47:35.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:34 smithi113 ceph-mon[34341]: pgmap v174: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 685 op/s 2024-03-30T13:47:35.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:34 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:47:35.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:34 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:47:35.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:34 smithi113 ceph-mon[34341]: 7.a scrub starts 2024-03-30T13:47:35.268 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:35.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:35.312 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:35.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:35.413 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:35.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:35.461 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:35.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:35.562 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:35.591 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:35.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:35.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:35.693 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:35.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:35.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:35.836 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:35.863 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:35.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:35.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:35.964 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:36.003 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:36.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:36.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:36.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:35 smithi043 ceph-mon[29963]: 7.3 scrub ok 2024-03-30T13:47:36.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:35 smithi043 ceph-mon[29963]: 5.3 scrub starts 2024-03-30T13:47:36.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:35 smithi043 ceph-mon[29963]: 5.3 scrub ok 2024-03-30T13:47:36.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:35 smithi043 ceph-mon[29963]: osdmap e130: 8 total, 8 up, 8 in 2024-03-30T13:47:36.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:35 smithi043 ceph-mon[29963]: pgmap v176: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-30T13:47:36.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:35 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3275719247' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:47:36.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:35 smithi043 ceph-mon[42099]: 7.3 scrub ok 2024-03-30T13:47:36.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:35 smithi043 ceph-mon[42099]: 5.3 scrub starts 2024-03-30T13:47:36.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:35 smithi043 ceph-mon[42099]: 5.3 scrub ok 2024-03-30T13:47:36.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:35 smithi043 ceph-mon[42099]: osdmap e130: 8 total, 8 up, 8 in 2024-03-30T13:47:36.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:35 smithi043 ceph-mon[42099]: pgmap v176: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-30T13:47:36.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:35 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3275719247' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:47:36.104 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:36.130 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:36.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:36.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:36.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:35 smithi113 ceph-mon[34341]: 7.3 scrub ok 2024-03-30T13:47:36.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:35 smithi113 ceph-mon[34341]: 5.3 scrub starts 2024-03-30T13:47:36.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:35 smithi113 ceph-mon[34341]: 5.3 scrub ok 2024-03-30T13:47:36.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:35 smithi113 ceph-mon[34341]: osdmap e130: 8 total, 8 up, 8 in 2024-03-30T13:47:36.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:35 smithi113 ceph-mon[34341]: pgmap v176: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-30T13:47:36.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:35 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3275719247' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:47:36.232 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:36.261 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:36.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:36.342 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:47:36.342 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:47:36.342 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:47:36.342 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:47:36.342 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:47:36.342 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:47:36.342 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:47:36.342 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:47:36.343 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:47:36.343 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:47:36.343 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:47:36.343 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:47:36.343 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:47:36.343 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:47:36.343 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:47:36.343 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:47:36.364 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:36.391 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:36.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:36.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:36.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:47:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:47:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:47:36.492 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:36.525 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:36.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:36.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:36.561 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:47:36.563 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:47:36.563 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [], 2024-03-30T13:47:36.563 INFO:teuthology.orchestra.run.smithi113.stdout: "complaint_time": 30, 2024-03-30T13:47:36.563 INFO:teuthology.orchestra.run.smithi113.stdout: "num_blocked_ops": 0 2024-03-30T13:47:36.563 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:47:36.628 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:36.683 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:36.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:36.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:36.785 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:36.813 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:36.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:36.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:36.915 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:36.951 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:36.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:36.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:37.052 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:37.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:37.098 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:37.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:37.199 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:37.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:36 smithi113 ceph-mon[34341]: 7.a scrub ok 2024-03-30T13:47:37.242 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:37.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:37.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:37.303 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:36 smithi043 ceph-mon[29963]: 7.a scrub ok 2024-03-30T13:47:37.304 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:36 smithi043 ceph-mon[42099]: 7.a scrub ok 2024-03-30T13:47:37.310 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-03-30T13:47:37.344 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:37.382 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:37.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:37.484 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:37.523 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:37.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:37.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:37.625 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:37.633 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:47:37.666 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:37.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:37.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:37.767 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:37.804 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:37.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:37.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:37.906 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:37.948 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:37.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:37.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:38.050 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:38.070 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:37 smithi113 ceph-mon[34341]: 1.0 scrub starts 2024-03-30T13:47:38.070 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:37 smithi113 ceph-mon[34341]: 1.0 scrub ok 2024-03-30T13:47:38.070 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:37 smithi113 ceph-mon[34341]: 3.1a scrub starts 2024-03-30T13:47:38.070 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:37 smithi113 ceph-mon[34341]: 3.1a scrub ok 2024-03-30T13:47:38.070 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:37 smithi113 ceph-mon[34341]: pgmap v177: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 533 op/s 2024-03-30T13:47:38.093 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:38.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:38.195 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:38.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:38.227 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:38.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:38.328 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:38.357 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:38.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:38.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:37 smithi043 ceph-mon[29963]: 1.0 scrub starts 2024-03-30T13:47:38.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:37 smithi043 ceph-mon[29963]: 1.0 scrub ok 2024-03-30T13:47:38.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:37 smithi043 ceph-mon[29963]: 3.1a scrub starts 2024-03-30T13:47:38.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:37 smithi043 ceph-mon[29963]: 3.1a scrub ok 2024-03-30T13:47:38.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:37 smithi043 ceph-mon[29963]: pgmap v177: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 533 op/s 2024-03-30T13:47:38.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:37 smithi043 ceph-mon[42099]: 1.0 scrub starts 2024-03-30T13:47:38.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:37 smithi043 ceph-mon[42099]: 1.0 scrub ok 2024-03-30T13:47:38.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:37 smithi043 ceph-mon[42099]: 3.1a scrub starts 2024-03-30T13:47:38.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:37 smithi043 ceph-mon[42099]: 3.1a scrub ok 2024-03-30T13:47:38.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:37 smithi043 ceph-mon[42099]: pgmap v177: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 533 op/s 2024-03-30T13:47:38.458 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:38.490 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:38.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:38.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:38.592 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:38.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:38.629 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:38.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:38.730 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:38.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:38.760 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:38.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:38.861 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:38.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:38.909 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:38.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:38.951 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:47:38 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:47:39.010 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:39.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:39.042 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:39.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:39.107 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:38 smithi043 ceph-mon[29963]: 6.1 scrub starts 2024-03-30T13:47:39.107 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:38 smithi043 ceph-mon[29963]: 6.1 scrub ok 2024-03-30T13:47:39.107 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:38 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:47:39.107 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:38 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:47:39.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:38 smithi043 ceph-mon[29963]: 3.1d deep-scrub starts 2024-03-30T13:47:39.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:38 smithi043 ceph-mon[29963]: 3.1d deep-scrub ok 2024-03-30T13:47:39.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:38 smithi043 ceph-mon[29963]: osdmap e131: 8 total, 8 up, 8 in 2024-03-30T13:47:39.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:38 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:47:39.108 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:38 smithi043 ceph-mon[42099]: 6.1 scrub starts 2024-03-30T13:47:39.108 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:38 smithi043 ceph-mon[42099]: 6.1 scrub ok 2024-03-30T13:47:39.108 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:38 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:47:39.108 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:38 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:47:39.108 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:38 smithi043 ceph-mon[42099]: 3.1d deep-scrub starts 2024-03-30T13:47:39.108 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:38 smithi043 ceph-mon[42099]: 3.1d deep-scrub ok 2024-03-30T13:47:39.108 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:38 smithi043 ceph-mon[42099]: osdmap e131: 8 total, 8 up, 8 in 2024-03-30T13:47:39.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:38 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:47:39.144 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:39.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:39.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:39.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:38 smithi113 ceph-mon[34341]: 6.1 scrub starts 2024-03-30T13:47:39.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:38 smithi113 ceph-mon[34341]: 6.1 scrub ok 2024-03-30T13:47:39.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:38 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:47:39.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:38 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:47:39.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:38 smithi113 ceph-mon[34341]: 3.1d deep-scrub starts 2024-03-30T13:47:39.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:38 smithi113 ceph-mon[34341]: 3.1d deep-scrub ok 2024-03-30T13:47:39.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:38 smithi113 ceph-mon[34341]: osdmap e131: 8 total, 8 up, 8 in 2024-03-30T13:47:39.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:38 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:47:39.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:39.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:39.332 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:39.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:39.433 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:39.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:39.469 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:39.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:39.570 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:39.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:39.598 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:39.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:39.699 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:39.725 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:39.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:39.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:39.827 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:39.835 INFO:teuthology.orchestra.run.smithi043.stderr:no valid command found; 10 closest matches: 2024-03-30T13:47:39.835 INFO:teuthology.orchestra.run.smithi043.stderr:0 2024-03-30T13:47:39.835 INFO:teuthology.orchestra.run.smithi043.stderr:1 2024-03-30T13:47:39.835 INFO:teuthology.orchestra.run.smithi043.stderr:2 2024-03-30T13:47:39.835 INFO:teuthology.orchestra.run.smithi043.stderr:abort 2024-03-30T13:47:39.835 INFO:teuthology.orchestra.run.smithi043.stderr:assert 2024-03-30T13:47:39.835 INFO:teuthology.orchestra.run.smithi043.stderr:bench [] [] [] [] 2024-03-30T13:47:39.836 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs debug_inject_read_zeros 2024-03-30T13:47:39.836 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs files list 2024-03-30T13:47:39.836 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs stats 2024-03-30T13:47:39.836 INFO:teuthology.orchestra.run.smithi043.stderr:bluestore allocator dump block 2024-03-30T13:47:39.836 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: invalid command 2024-03-30T13:47:39.878 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:39.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:39.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:39.980 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:39.984 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:47:40.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:40.022 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:40.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:40.123 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:40.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:40.166 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:40.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:40.248 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:39 smithi113 ceph-mon[34341]: 5.f scrub starts 2024-03-30T13:47:40.249 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:39 smithi113 ceph-mon[34341]: 5.f scrub ok 2024-03-30T13:47:40.249 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:39 smithi113 ceph-mon[34341]: pgmap v179: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-03-30T13:47:40.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:39 smithi043 ceph-mon[29963]: 5.f scrub starts 2024-03-30T13:47:40.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:39 smithi043 ceph-mon[29963]: 5.f scrub ok 2024-03-30T13:47:40.255 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:39 smithi043 ceph-mon[29963]: pgmap v179: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-03-30T13:47:40.255 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:39 smithi043 ceph-mon[42099]: 5.f scrub starts 2024-03-30T13:47:40.255 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:39 smithi043 ceph-mon[42099]: 5.f scrub ok 2024-03-30T13:47:40.255 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:39 smithi043 ceph-mon[42099]: pgmap v179: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-03-30T13:47:40.269 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:40.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:40.296 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:40.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:40.398 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:40.419 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:47:40.419 INFO:teuthology.orchestra.run.smithi113.stdout: "size": 20, 2024-03-30T13:47:40.420 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 600, 2024-03-30T13:47:40.420 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [ 2024-03-30T13:47:40.420 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.420 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:145825 7.5 7:a688f4b0:::benchmark_data_smithi043_86989_object23487:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-30T13:47:40.420 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:55.513638+0000", 2024-03-30T13:47:40.420 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 44.90134424, 2024-03-30T13:47:40.420 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.155179499, 2024-03-30T13:47:40.420 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:40.420 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:40.420 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:40.420 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:40.420 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:40.421 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:40.421 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 145825 2024-03-30T13:47:40.421 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.421 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:40.421 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.421 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:40.421 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.513638+0000", 2024-03-30T13:47:40.421 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.421 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.421 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.421 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:40.421 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.513638+0000", 2024-03-30T13:47:40.421 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.422 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.422 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.422 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:40.422 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.513639+0000", 2024-03-30T13:47:40.422 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1790000000000001e-06 2024-03-30T13:47:40.422 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.422 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.422 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:40.422 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.513644+0000", 2024-03-30T13:47:40.422 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2460000000000003e-06 2024-03-30T13:47:40.422 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.422 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.423 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:40.423 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.513656+0000", 2024-03-30T13:47:40.423 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1983e-05 2024-03-30T13:47:40.423 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.423 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.423 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:40.423 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.513663+0000", 2024-03-30T13:47:40.423 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8700000000000003e-06 2024-03-30T13:47:40.423 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.423 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.423 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:40.424 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.513718+0000", 2024-03-30T13:47:40.424 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4533000000000001e-05 2024-03-30T13:47:40.424 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.424 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.424 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:40.424 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.513852+0000", 2024-03-30T13:47:40.424 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013423100000000001 2024-03-30T13:47:40.424 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.424 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.424 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:47:40.424 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.513986+0000", 2024-03-30T13:47:40.424 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000133397 2024-03-30T13:47:40.425 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.425 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.425 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:40.425 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.516904+0000", 2024-03-30T13:47:40.425 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0029180730000000002 2024-03-30T13:47:40.425 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.425 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.425 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.425 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.517328+0000", 2024-03-30T13:47:40.426 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00042470099999999998 2024-03-30T13:47:40.426 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.426 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.426 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.426 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.668714+0000", 2024-03-30T13:47:40.426 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15138600599999999 2024-03-30T13:47:40.426 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.426 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.426 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:40.426 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.668767+0000", 2024-03-30T13:47:40.426 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2853999999999999e-05 2024-03-30T13:47:40.426 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.427 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.427 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:40.427 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.668818+0000", 2024-03-30T13:47:40.427 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0426000000000001e-05 2024-03-30T13:47:40.427 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.427 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.427 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.427 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.427 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.427 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:184866 7.5 7:a99b6bae:::benchmark_data_smithi043_86989_object62528:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-30T13:47:40.427 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:36.958864+0000", 2024-03-30T13:47:40.427 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 3.4561187169999998, 2024-03-30T13:47:40.427 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15833154399999999, 2024-03-30T13:47:40.428 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:40.428 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:40.428 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:40.428 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:40.428 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:40.428 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:40.428 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 184866 2024-03-30T13:47:40.428 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.428 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:40.428 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.428 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:40.428 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.958864+0000", 2024-03-30T13:47:40.428 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.429 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.429 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.429 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:40.429 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.958864+0000", 2024-03-30T13:47:40.429 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.429 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.429 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.429 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:40.429 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.958865+0000", 2024-03-30T13:47:40.429 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.345e-06 2024-03-30T13:47:40.430 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.430 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.430 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:40.430 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.958870+0000", 2024-03-30T13:47:40.430 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8049999999999997e-06 2024-03-30T13:47:40.430 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.430 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.430 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:40.430 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.958881+0000", 2024-03-30T13:47:40.430 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1724e-05 2024-03-30T13:47:40.430 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.430 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.430 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:40.431 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.958889+0000", 2024-03-30T13:47:40.431 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.5109999999999997e-06 2024-03-30T13:47:40.431 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.431 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.431 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:40.431 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.009980+0000", 2024-03-30T13:47:40.431 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.051090871000000003 2024-03-30T13:47:40.431 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.431 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.431 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:40.431 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.010096+0000", 2024-03-30T13:47:40.431 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011582300000000001 2024-03-30T13:47:40.431 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.432 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.432 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:47:40.432 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.010173+0000", 2024-03-30T13:47:40.432 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.7374999999999997e-05 2024-03-30T13:47:40.432 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.432 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.432 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:40.432 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.015369+0000", 2024-03-30T13:47:40.432 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0051959789999999999 2024-03-30T13:47:40.432 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.432 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.432 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.432 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.048919+0000", 2024-03-30T13:47:40.432 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.033550096000000001 2024-03-30T13:47:40.433 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.433 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.433 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.433 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.117068+0000", 2024-03-30T13:47:40.433 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.068148743999999997 2024-03-30T13:47:40.433 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.433 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.433 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:40.433 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.117131+0000", 2024-03-30T13:47:40.433 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3268000000000005e-05 2024-03-30T13:47:40.433 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.434 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.434 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:40.434 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.117195+0000", 2024-03-30T13:47:40.434 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.4003000000000004e-05 2024-03-30T13:47:40.434 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.434 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.434 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.434 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.435 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.435 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:184867 7.5 7:a079a26e:::benchmark_data_smithi043_86989_object62529:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-30T13:47:40.435 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:36.961802+0000", 2024-03-30T13:47:40.435 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 3.4531800320000001, 2024-03-30T13:47:40.435 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15550554899999999, 2024-03-30T13:47:40.435 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:40.436 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:40.436 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:40.436 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:40.436 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:40.436 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:40.436 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 184867 2024-03-30T13:47:40.436 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.436 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:40.436 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.436 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:40.436 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.961802+0000", 2024-03-30T13:47:40.436 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.437 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.437 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.437 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:40.437 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.961802+0000", 2024-03-30T13:47:40.437 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.437 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.437 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.437 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:40.437 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.961804+0000", 2024-03-30T13:47:40.437 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3170000000000001e-06 2024-03-30T13:47:40.437 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.437 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.437 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:40.438 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.961807+0000", 2024-03-30T13:47:40.438 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.6909999999999999e-06 2024-03-30T13:47:40.438 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.438 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.438 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:40.438 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.961817+0000", 2024-03-30T13:47:40.438 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.5349999999999993e-06 2024-03-30T13:47:40.438 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.438 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.438 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:40.438 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.961824+0000", 2024-03-30T13:47:40.438 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.7340000000000002e-06 2024-03-30T13:47:40.438 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.439 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.439 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:40.439 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.012549+0000", 2024-03-30T13:47:40.439 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.050725943000000003 2024-03-30T13:47:40.439 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.439 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.439 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:40.439 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.012671+0000", 2024-03-30T13:47:40.439 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000121777 2024-03-30T13:47:40.439 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.439 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.439 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:47:40.440 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.012803+0000", 2024-03-30T13:47:40.440 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013154 2024-03-30T13:47:40.440 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.440 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.440 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:40.440 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.015537+0000", 2024-03-30T13:47:40.440 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.002733709 2024-03-30T13:47:40.440 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.440 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.440 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.440 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.058704+0000", 2024-03-30T13:47:40.440 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.043167533000000001 2024-03-30T13:47:40.440 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.440 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.441 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.441 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.117242+0000", 2024-03-30T13:47:40.441 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.058537759000000002 2024-03-30T13:47:40.441 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.441 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.441 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:40.441 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.117276+0000", 2024-03-30T13:47:40.442 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.4131000000000002e-05 2024-03-30T13:47:40.442 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.442 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.442 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:40.442 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.117308+0000", 2024-03-30T13:47:40.442 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.188e-05 2024-03-30T13:47:40.442 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.443 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.443 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.443 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.443 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.443 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:186577 7.5 7:a290ff28:::benchmark_data_smithi043_86989_object64239:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-30T13:47:40.443 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:38.859881+0000", 2024-03-30T13:47:40.443 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.555101268, 2024-03-30T13:47:40.444 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.16084641199999999, 2024-03-30T13:47:40.444 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:40.444 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:40.444 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:40.444 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:40.444 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:40.445 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:40.445 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 186577 2024-03-30T13:47:40.445 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.445 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:40.445 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.445 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:40.446 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.859881+0000", 2024-03-30T13:47:40.446 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.446 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.446 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.446 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:40.446 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.859881+0000", 2024-03-30T13:47:40.447 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.447 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.447 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.447 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:40.447 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.859882+0000", 2024-03-30T13:47:40.447 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.119e-06 2024-03-30T13:47:40.447 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.447 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.448 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:40.448 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.859887+0000", 2024-03-30T13:47:40.448 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0499999999999999e-06 2024-03-30T13:47:40.448 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.448 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.448 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:40.448 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.859898+0000", 2024-03-30T13:47:40.448 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0665e-05 2024-03-30T13:47:40.449 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.449 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.449 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:40.449 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.859907+0000", 2024-03-30T13:47:40.449 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6799999999999999e-06 2024-03-30T13:47:40.449 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.450 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.450 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:40.450 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.900822+0000", 2024-03-30T13:47:40.450 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.040915002999999998 2024-03-30T13:47:40.450 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.451 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.451 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:40.451 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.900937+0000", 2024-03-30T13:47:40.451 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000115683 2024-03-30T13:47:40.451 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.451 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.451 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:47:40.451 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.901067+0000", 2024-03-30T13:47:40.451 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000130242 2024-03-30T13:47:40.452 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.452 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.452 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:40.452 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.905083+0000", 2024-03-30T13:47:40.452 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0040151739999999998 2024-03-30T13:47:40.452 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.452 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.452 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.452 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.943163+0000", 2024-03-30T13:47:40.452 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.038080073999999998 2024-03-30T13:47:40.452 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.452 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.452 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.452 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.020632+0000", 2024-03-30T13:47:40.453 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.077468843999999995 2024-03-30T13:47:40.453 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.453 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.453 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:40.453 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.020669+0000", 2024-03-30T13:47:40.453 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.7403000000000001e-05 2024-03-30T13:47:40.453 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.453 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.453 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:40.453 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.020727+0000", 2024-03-30T13:47:40.453 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8474999999999998e-05 2024-03-30T13:47:40.453 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.453 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.454 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.454 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.454 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.454 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:186884 7.5 7:a73a07c4:::benchmark_data_smithi043_86989_object64546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-30T13:47:40.454 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.217916+0000", 2024-03-30T13:47:40.454 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.1970662750000001, 2024-03-30T13:47:40.454 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.22441243499999999, 2024-03-30T13:47:40.454 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:40.454 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:40.454 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:40.454 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:40.455 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:40.455 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:40.455 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 186884 2024-03-30T13:47:40.455 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.455 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:40.455 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.455 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:40.455 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.217916+0000", 2024-03-30T13:47:40.455 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.455 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.455 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.455 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:40.455 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.217916+0000", 2024-03-30T13:47:40.455 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.455 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.456 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.456 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:40.456 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.217917+0000", 2024-03-30T13:47:40.456 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.0100000000000003e-07 2024-03-30T13:47:40.456 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.456 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.456 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:40.458 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.217921+0000", 2024-03-30T13:47:40.458 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.997e-06 2024-03-30T13:47:40.459 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.459 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.459 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:40.459 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.217930+0000", 2024-03-30T13:47:40.459 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.5289999999999997e-06 2024-03-30T13:47:40.459 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.459 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.459 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:40.459 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.217936+0000", 2024-03-30T13:47:40.459 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1420000000000001e-06 2024-03-30T13:47:40.460 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.460 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.460 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:40.460 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.217979+0000", 2024-03-30T13:47:40.460 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.2970999999999999e-05 2024-03-30T13:47:40.461 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.461 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.461 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:40.461 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.218077+0000", 2024-03-30T13:47:40.461 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.8084999999999993e-05 2024-03-30T13:47:40.461 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.461 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.461 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:47:40.461 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.218150+0000", 2024-03-30T13:47:40.461 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.3424999999999996e-05 2024-03-30T13:47:40.461 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.461 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.462 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:40.462 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.222912+0000", 2024-03-30T13:47:40.462 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.004761927 2024-03-30T13:47:40.462 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.462 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.462 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.462 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.241362+0000", 2024-03-30T13:47:40.462 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.018450139000000001 2024-03-30T13:47:40.462 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.462 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.462 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.463 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442214+0000", 2024-03-30T13:47:40.463 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.200851737 2024-03-30T13:47:40.463 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.463 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.463 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:40.463 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442271+0000", 2024-03-30T13:47:40.463 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6712999999999997e-05 2024-03-30T13:47:40.463 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.463 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.463 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:40.463 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442328+0000", 2024-03-30T13:47:40.463 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7868999999999998e-05 2024-03-30T13:47:40.463 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.463 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.464 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.464 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.464 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.464 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:186886 7.5 7:a2cfd65b:::benchmark_data_smithi043_86989_object64548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-30T13:47:40.464 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.219014+0000", 2024-03-30T13:47:40.464 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.1959681369999999, 2024-03-30T13:47:40.464 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.22343902400000001, 2024-03-30T13:47:40.464 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:40.464 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:40.464 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:40.464 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:40.464 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:40.464 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:40.464 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 186886 2024-03-30T13:47:40.465 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.465 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:40.465 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.465 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:40.465 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.219014+0000", 2024-03-30T13:47:40.465 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.465 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.465 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.465 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:40.465 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.219014+0000", 2024-03-30T13:47:40.465 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.465 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.465 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.465 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:40.466 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.219015+0000", 2024-03-30T13:47:40.466 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.06e-07 2024-03-30T13:47:40.466 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.466 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.466 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:40.466 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.219019+0000", 2024-03-30T13:47:40.466 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.464e-06 2024-03-30T13:47:40.466 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.466 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.466 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:40.466 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.219028+0000", 2024-03-30T13:47:40.466 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.9260000000000006e-06 2024-03-30T13:47:40.467 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.467 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.467 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:40.467 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.219035+0000", 2024-03-30T13:47:40.467 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6660000000000002e-06 2024-03-30T13:47:40.467 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.467 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.467 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:40.467 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.220296+0000", 2024-03-30T13:47:40.467 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0012610950000000001 2024-03-30T13:47:40.467 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.467 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.468 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:40.468 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.220412+0000", 2024-03-30T13:47:40.468 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000116413 2024-03-30T13:47:40.468 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.468 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.468 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:47:40.468 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.220542+0000", 2024-03-30T13:47:40.468 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012981700000000001 2024-03-30T13:47:40.468 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.468 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.468 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:40.468 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.223017+0000", 2024-03-30T13:47:40.469 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0024748909999999999 2024-03-30T13:47:40.469 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.469 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.469 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.469 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.241404+0000", 2024-03-30T13:47:40.469 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.018386848000000001 2024-03-30T13:47:40.469 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.469 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.469 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.469 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442373+0000", 2024-03-30T13:47:40.469 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.20096920300000001 2024-03-30T13:47:40.469 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.469 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.470 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:40.470 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442406+0000", 2024-03-30T13:47:40.470 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2446000000000001e-05 2024-03-30T13:47:40.470 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.470 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.470 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:40.470 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442453+0000", 2024-03-30T13:47:40.470 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7549000000000002e-05 2024-03-30T13:47:40.470 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.470 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.470 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.470 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.470 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.471 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:186896 7.5 7:a006897a:::benchmark_data_smithi043_86989_object64558:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-30T13:47:40.471 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.232075+0000", 2024-03-30T13:47:40.471 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.182906877, 2024-03-30T13:47:40.471 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.21065776999999999, 2024-03-30T13:47:40.471 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:40.471 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:40.471 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:40.471 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:40.471 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:40.471 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:40.471 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 186896 2024-03-30T13:47:40.471 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.472 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:40.472 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.472 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:40.472 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.232075+0000", 2024-03-30T13:47:40.472 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.472 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.472 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.472 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:40.472 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.232075+0000", 2024-03-30T13:47:40.472 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.472 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.472 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.472 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:40.473 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.232076+0000", 2024-03-30T13:47:40.473 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0410000000000001e-06 2024-03-30T13:47:40.473 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.473 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.473 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:40.473 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.232081+0000", 2024-03-30T13:47:40.473 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.2980000000000002e-06 2024-03-30T13:47:40.473 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.473 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.474 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:40.474 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.232090+0000", 2024-03-30T13:47:40.474 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.952e-06 2024-03-30T13:47:40.474 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.474 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.474 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:40.474 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.232095+0000", 2024-03-30T13:47:40.477 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2560000000000002e-06 2024-03-30T13:47:40.477 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.477 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.478 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:40.478 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.232144+0000", 2024-03-30T13:47:40.478 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9450999999999998e-05 2024-03-30T13:47:40.478 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.478 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.478 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:40.479 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.232248+0000", 2024-03-30T13:47:40.479 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000103526 2024-03-30T13:47:40.479 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.479 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.479 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:47:40.479 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.232333+0000", 2024-03-30T13:47:40.479 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5550000000000006e-05 2024-03-30T13:47:40.479 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.479 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.479 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:40.479 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.234450+0000", 2024-03-30T13:47:40.480 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0021169990000000001 2024-03-30T13:47:40.480 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.480 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.480 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.480 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.241435+0000", 2024-03-30T13:47:40.480 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0069840379999999997 2024-03-30T13:47:40.480 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.480 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.480 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.480 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442614+0000", 2024-03-30T13:47:40.480 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.20117961100000001 2024-03-30T13:47:40.480 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.481 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.481 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:40.481 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442675+0000", 2024-03-30T13:47:40.481 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0401e-05 2024-03-30T13:47:40.481 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.481 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.481 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:40.481 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442733+0000", 2024-03-30T13:47:40.481 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8647000000000003e-05 2024-03-30T13:47:40.481 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.481 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.481 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.482 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.482 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.482 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:186905 7.5 7:ab740115:::benchmark_data_smithi043_86989_object64567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-30T13:47:40.482 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.242589+0000", 2024-03-30T13:47:40.482 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.172393604, 2024-03-30T13:47:40.482 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.200289468, 2024-03-30T13:47:40.482 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:40.482 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:40.482 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:40.482 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:40.482 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:40.483 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:40.483 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 186905 2024-03-30T13:47:40.483 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.483 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:40.483 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.483 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:40.483 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.242589+0000", 2024-03-30T13:47:40.483 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.483 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.483 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.483 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:40.483 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.242589+0000", 2024-03-30T13:47:40.484 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.484 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.484 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.484 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:40.484 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.242590+0000", 2024-03-30T13:47:40.485 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.091e-06 2024-03-30T13:47:40.485 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.485 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.485 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:40.485 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.242595+0000", 2024-03-30T13:47:40.485 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8840000000000002e-06 2024-03-30T13:47:40.486 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.486 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.487 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:40.487 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.242606+0000", 2024-03-30T13:47:40.487 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1384e-05 2024-03-30T13:47:40.487 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.487 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.487 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:40.488 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.242612+0000", 2024-03-30T13:47:40.488 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7699999999999998e-06 2024-03-30T13:47:40.488 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.488 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.488 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:40.488 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.242660+0000", 2024-03-30T13:47:40.488 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8554999999999998e-05 2024-03-30T13:47:40.488 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.488 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.489 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:40.489 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.242789+0000", 2024-03-30T13:47:40.489 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000128918 2024-03-30T13:47:40.489 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.489 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.489 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:47:40.489 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.242917+0000", 2024-03-30T13:47:40.489 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012774100000000001 2024-03-30T13:47:40.489 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.489 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.489 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:40.489 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245127+0000", 2024-03-30T13:47:40.489 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0022098980000000001 2024-03-30T13:47:40.489 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.490 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.490 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.490 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.249020+0000", 2024-03-30T13:47:40.490 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0038933969999999998 2024-03-30T13:47:40.490 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.490 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.490 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.490 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442788+0000", 2024-03-30T13:47:40.490 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.19376775800000001 2024-03-30T13:47:40.490 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.490 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.490 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:40.491 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442835+0000", 2024-03-30T13:47:40.491 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6813000000000001e-05 2024-03-30T13:47:40.491 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.491 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.491 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:40.491 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442878+0000", 2024-03-30T13:47:40.491 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3259e-05 2024-03-30T13:47:40.491 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.491 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.491 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.491 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.491 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.492 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:186909 7.5 7:ac8b9ff7:::benchmark_data_smithi043_86989_object64571:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-30T13:47:40.492 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.245217+0000", 2024-03-30T13:47:40.492 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.1697650070000001, 2024-03-30T13:47:40.492 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.19782831200000001, 2024-03-30T13:47:40.492 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:40.492 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:40.492 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:40.492 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:40.492 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:40.492 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:40.492 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 186909 2024-03-30T13:47:40.492 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.492 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:40.493 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.493 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:40.493 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245217+0000", 2024-03-30T13:47:40.493 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.493 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.493 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.493 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:40.493 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245217+0000", 2024-03-30T13:47:40.493 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.493 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.493 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.493 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:40.493 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245228+0000", 2024-03-30T13:47:40.493 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.042e-05 2024-03-30T13:47:40.494 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.494 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.494 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:40.494 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245232+0000", 2024-03-30T13:47:40.494 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7550000000000002e-06 2024-03-30T13:47:40.494 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.494 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.494 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:40.494 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245243+0000", 2024-03-30T13:47:40.494 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0342e-05 2024-03-30T13:47:40.494 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.494 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.494 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:40.495 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245248+0000", 2024-03-30T13:47:40.495 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.592e-06 2024-03-30T13:47:40.495 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.495 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.495 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:40.495 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245284+0000", 2024-03-30T13:47:40.495 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.5901000000000001e-05 2024-03-30T13:47:40.495 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.495 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.495 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:40.495 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245362+0000", 2024-03-30T13:47:40.495 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.7622999999999999e-05 2024-03-30T13:47:40.495 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.495 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.496 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:47:40.496 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245437+0000", 2024-03-30T13:47:40.496 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.4944999999999995e-05 2024-03-30T13:47:40.496 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.496 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.496 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:40.496 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.248971+0000", 2024-03-30T13:47:40.496 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0035339450000000001 2024-03-30T13:47:40.496 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.496 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.496 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.496 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.251184+0000", 2024-03-30T13:47:40.496 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.002213095 2024-03-30T13:47:40.497 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.497 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.497 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.497 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442940+0000", 2024-03-30T13:47:40.497 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.19175640099999999 2024-03-30T13:47:40.497 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.497 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.497 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:40.497 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442991+0000", 2024-03-30T13:47:40.497 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0535999999999999e-05 2024-03-30T13:47:40.497 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.497 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.497 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:40.498 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443046+0000", 2024-03-30T13:47:40.498 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4756999999999998e-05 2024-03-30T13:47:40.498 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.498 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.498 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.498 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.498 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.498 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:186910 7.5 7:a7b0943c:::benchmark_data_smithi043_86989_object64572:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-30T13:47:40.498 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.245266+0000", 2024-03-30T13:47:40.498 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.1697166320000001, 2024-03-30T13:47:40.498 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.19788240700000001, 2024-03-30T13:47:40.498 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:40.499 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:40.499 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:40.499 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:40.499 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:40.499 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:40.499 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 186910 2024-03-30T13:47:40.499 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.499 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:40.499 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.499 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:40.499 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245266+0000", 2024-03-30T13:47:40.500 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.500 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.500 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.500 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:40.500 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245266+0000", 2024-03-30T13:47:40.500 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.500 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.500 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.500 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:40.500 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245266+0000", 2024-03-30T13:47:40.500 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.96e-07 2024-03-30T13:47:40.500 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.500 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.500 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:40.501 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245268+0000", 2024-03-30T13:47:40.501 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.9700000000000002e-06 2024-03-30T13:47:40.501 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.501 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.501 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:40.501 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245273+0000", 2024-03-30T13:47:40.501 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1279999999999999e-06 2024-03-30T13:47:40.501 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.501 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.501 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:40.501 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245277+0000", 2024-03-30T13:47:40.502 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.0559999999999998e-06 2024-03-30T13:47:40.502 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.502 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.502 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:40.502 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.247136+0000", 2024-03-30T13:47:40.503 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0018592789999999999 2024-03-30T13:47:40.503 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.503 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.503 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:40.503 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.247232+0000", 2024-03-30T13:47:40.503 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.5748000000000006e-05 2024-03-30T13:47:40.503 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.503 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.504 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:47:40.504 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.247318+0000", 2024-03-30T13:47:40.504 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5373999999999999e-05 2024-03-30T13:47:40.504 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.504 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.504 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:40.504 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.249088+0000", 2024-03-30T13:47:40.504 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001770766 2024-03-30T13:47:40.504 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.504 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.505 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.505 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.252085+0000", 2024-03-30T13:47:40.505 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0029968500000000001 2024-03-30T13:47:40.505 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.505 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.505 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.505 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443081+0000", 2024-03-30T13:47:40.505 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.19099596399999999 2024-03-30T13:47:40.506 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.506 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.506 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:40.506 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443110+0000", 2024-03-30T13:47:40.506 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.8690000000000001e-05 2024-03-30T13:47:40.506 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.506 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.506 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:40.506 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443148+0000", 2024-03-30T13:47:40.506 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.8186000000000003e-05 2024-03-30T13:47:40.506 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.506 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.507 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.507 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.507 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.507 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:186926 7.5 7:a7188cf2:::benchmark_data_smithi043_86989_object64588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-30T13:47:40.507 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.267148+0000", 2024-03-30T13:47:40.507 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.1478347330000001, 2024-03-30T13:47:40.507 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.176160026, 2024-03-30T13:47:40.507 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:40.507 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:40.507 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:40.507 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:40.508 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:40.508 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:40.508 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 186926 2024-03-30T13:47:40.508 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.508 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:40.508 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.508 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:40.508 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.267148+0000", 2024-03-30T13:47:40.508 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.508 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.508 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.508 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:40.508 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.267148+0000", 2024-03-30T13:47:40.509 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.509 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.509 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.509 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:40.509 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.267149+0000", 2024-03-30T13:47:40.509 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1990000000000001e-06 2024-03-30T13:47:40.509 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.509 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.509 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:40.509 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.267153+0000", 2024-03-30T13:47:40.510 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6619999999999996e-06 2024-03-30T13:47:40.510 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.510 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.510 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:40.510 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.267164+0000", 2024-03-30T13:47:40.510 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0678e-05 2024-03-30T13:47:40.510 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.510 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.510 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:40.510 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.267171+0000", 2024-03-30T13:47:40.510 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.9999999999999999e-06 2024-03-30T13:47:40.510 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.511 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.511 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:40.511 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.267208+0000", 2024-03-30T13:47:40.511 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.6990000000000003e-05 2024-03-30T13:47:40.511 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.511 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.511 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:40.511 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.267317+0000", 2024-03-30T13:47:40.511 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000109308 2024-03-30T13:47:40.511 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.511 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.511 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:47:40.511 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.267412+0000", 2024-03-30T13:47:40.512 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4116000000000005e-05 2024-03-30T13:47:40.512 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.512 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.512 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:40.512 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.269599+0000", 2024-03-30T13:47:40.512 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.002187128 2024-03-30T13:47:40.512 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.512 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.512 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.512 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.270659+0000", 2024-03-30T13:47:40.512 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0010600189999999999 2024-03-30T13:47:40.512 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.512 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.512 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.513 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443194+0000", 2024-03-30T13:47:40.513 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.172535569 2024-03-30T13:47:40.513 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.513 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.513 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:40.513 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443253+0000", 2024-03-30T13:47:40.513 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8888000000000003e-05 2024-03-30T13:47:40.513 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.513 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.513 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:40.513 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443308+0000", 2024-03-30T13:47:40.513 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4469000000000003e-05 2024-03-30T13:47:40.513 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.514 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.514 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.514 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.514 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.514 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop_reply(client.14817.0:186884 7.5 e131/98 ondisk, result = 0)", 2024-03-30T13:47:40.514 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.288030+0000", 2024-03-30T13:47:40.514 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.1269523349999999, 2024-03-30T13:47:40.514 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.154306057, 2024-03-30T13:47:40.514 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:40.514 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:40.514 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:47:40.514 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:40.514 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.515 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:40.515 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.288030+0000", 2024-03-30T13:47:40.515 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.515 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.515 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.515 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:40.515 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.288030+0000", 2024-03-30T13:47:40.515 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.515 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.515 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.515 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:40.515 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.288031+0000", 2024-03-30T13:47:40.515 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6400000000000002e-07 2024-03-30T13:47:40.516 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.516 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.516 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:40.516 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.288035+0000", 2024-03-30T13:47:40.516 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3930000000000001e-06 2024-03-30T13:47:40.516 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.516 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.516 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:40.516 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.288046+0000", 2024-03-30T13:47:40.516 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0811e-05 2024-03-30T13:47:40.516 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.516 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.516 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:40.516 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.288051+0000", 2024-03-30T13:47:40.517 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2270000000000001e-06 2024-03-30T13:47:40.517 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.517 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.517 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:40.517 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442192+0000", 2024-03-30T13:47:40.517 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.154141159 2024-03-30T13:47:40.517 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.517 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.517 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:40.517 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442201+0000", 2024-03-30T13:47:40.517 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.3249999999999997e-06 2024-03-30T13:47:40.517 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.517 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.518 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:40.518 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442336+0000", 2024-03-30T13:47:40.518 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000134578 2024-03-30T13:47:40.518 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.518 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.518 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.518 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.518 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.518 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop_reply(client.14817.0:186886 7.5 e131/98 ondisk, result = 0)", 2024-03-30T13:47:40.518 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.291698+0000", 2024-03-30T13:47:40.518 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.1232845, 2024-03-30T13:47:40.519 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.150873435, 2024-03-30T13:47:40.519 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:40.519 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:40.519 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:47:40.519 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:40.519 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.519 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:40.519 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291698+0000", 2024-03-30T13:47:40.519 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.519 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.519 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.519 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:40.520 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291698+0000", 2024-03-30T13:47:40.520 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.520 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.520 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.520 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:40.520 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291699+0000", 2024-03-30T13:47:40.520 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6400000000000001e-07 2024-03-30T13:47:40.520 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.520 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.520 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:40.520 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291705+0000", 2024-03-30T13:47:40.521 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5740000000000001e-06 2024-03-30T13:47:40.521 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.521 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.521 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:40.521 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291719+0000", 2024-03-30T13:47:40.521 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4139e-05 2024-03-30T13:47:40.521 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.521 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.521 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:40.521 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291727+0000", 2024-03-30T13:47:40.521 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8730000000000007e-06 2024-03-30T13:47:40.521 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.521 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.521 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:40.522 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442364+0000", 2024-03-30T13:47:40.522 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15063685199999999 2024-03-30T13:47:40.522 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.522 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.522 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:40.522 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442369+0000", 2024-03-30T13:47:40.522 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2120000000000002e-06 2024-03-30T13:47:40.522 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.522 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.522 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:40.522 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442571+0000", 2024-03-30T13:47:40.522 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00020192100000000001 2024-03-30T13:47:40.522 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.523 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.523 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.523 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.523 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.523 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop_reply(client.14817.0:186896 7.5 e131/98 ondisk, result = 0)", 2024-03-30T13:47:40.523 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.291749+0000", 2024-03-30T13:47:40.523 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.123233277, 2024-03-30T13:47:40.523 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15099047900000001, 2024-03-30T13:47:40.523 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:40.523 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:40.523 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:47:40.524 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:40.524 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.524 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:40.524 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291749+0000", 2024-03-30T13:47:40.524 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.524 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.524 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.524 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:40.524 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291749+0000", 2024-03-30T13:47:40.524 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.524 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.524 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.524 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:40.524 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291749+0000", 2024-03-30T13:47:40.525 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.8799999999999998e-07 2024-03-30T13:47:40.525 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.525 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.525 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:40.525 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291752+0000", 2024-03-30T13:47:40.525 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.9179999999999998e-06 2024-03-30T13:47:40.525 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.525 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.525 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:40.525 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291759+0000", 2024-03-30T13:47:40.525 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5690000000000001e-06 2024-03-30T13:47:40.526 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.526 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.526 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:40.526 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291764+0000", 2024-03-30T13:47:40.526 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9030000000000003e-06 2024-03-30T13:47:40.527 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.527 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.527 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:40.527 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442598+0000", 2024-03-30T13:47:40.527 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15083411999999999 2024-03-30T13:47:40.527 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.528 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.528 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:40.528 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442607+0000", 2024-03-30T13:47:40.528 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.3000000000000007e-06 2024-03-30T13:47:40.528 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.528 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.528 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:40.528 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442740+0000", 2024-03-30T13:47:40.528 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013228099999999999 2024-03-30T13:47:40.529 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.529 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.529 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.529 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.529 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.529 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop_reply(client.14817.0:186905 7.5 e131/98 ondisk, result = 0)", 2024-03-30T13:47:40.530 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.291773+0000", 2024-03-30T13:47:40.530 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.123209449, 2024-03-30T13:47:40.530 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15111850800000001, 2024-03-30T13:47:40.530 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:40.530 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:40.530 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:47:40.530 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:40.530 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.531 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:40.531 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291773+0000", 2024-03-30T13:47:40.531 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.531 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.531 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.531 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:40.531 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291773+0000", 2024-03-30T13:47:40.531 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.531 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.531 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.531 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:40.531 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291773+0000", 2024-03-30T13:47:40.531 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.28e-07 2024-03-30T13:47:40.532 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.532 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.532 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:40.532 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291775+0000", 2024-03-30T13:47:40.532 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.26e-06 2024-03-30T13:47:40.532 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.532 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.532 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:40.532 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291781+0000", 2024-03-30T13:47:40.532 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6330000000000001e-06 2024-03-30T13:47:40.532 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.532 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.532 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:40.532 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291788+0000", 2024-03-30T13:47:40.533 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1849999999999998e-06 2024-03-30T13:47:40.533 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.533 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.533 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:40.533 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442773+0000", 2024-03-30T13:47:40.533 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15098513 2024-03-30T13:47:40.533 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.533 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.533 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:40.533 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442782+0000", 2024-03-30T13:47:40.533 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5839999999999995e-06 2024-03-30T13:47:40.533 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.533 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.533 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:40.534 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442891+0000", 2024-03-30T13:47:40.534 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00010948800000000001 2024-03-30T13:47:40.534 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.534 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.534 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.534 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.534 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.534 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop_reply(client.14817.0:186909 7.5 e131/98 ondisk, result = 0)", 2024-03-30T13:47:40.534 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.291831+0000", 2024-03-30T13:47:40.534 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.1231517520000001, 2024-03-30T13:47:40.535 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15122132399999999, 2024-03-30T13:47:40.535 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:40.535 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:40.535 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:47:40.535 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:40.535 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.535 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:40.535 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291831+0000", 2024-03-30T13:47:40.535 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.535 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.535 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.535 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:40.535 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291831+0000", 2024-03-30T13:47:40.535 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.536 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.536 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.536 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:40.536 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291831+0000", 2024-03-30T13:47:40.536 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.41e-07 2024-03-30T13:47:40.536 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.536 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.536 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:40.536 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291834+0000", 2024-03-30T13:47:40.536 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.7099999999999999e-06 2024-03-30T13:47:40.536 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.536 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.536 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:40.536 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291841+0000", 2024-03-30T13:47:40.537 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.43e-06 2024-03-30T13:47:40.537 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.537 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.537 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:40.537 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291847+0000", 2024-03-30T13:47:40.537 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5280000000000003e-06 2024-03-30T13:47:40.537 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.537 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.537 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:40.537 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442927+0000", 2024-03-30T13:47:40.537 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15108020599999999 2024-03-30T13:47:40.537 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.537 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.537 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:40.538 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442934+0000", 2024-03-30T13:47:40.538 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.6869999999999994e-06 2024-03-30T13:47:40.538 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.538 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.538 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:40.538 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443052+0000", 2024-03-30T13:47:40.538 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000117422 2024-03-30T13:47:40.538 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.538 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.538 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.538 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.538 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.539 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop_reply(client.14817.0:186910 7.5 e131/98 ondisk, result = 0)", 2024-03-30T13:47:40.539 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.291937+0000", 2024-03-30T13:47:40.539 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.1230451560000001, 2024-03-30T13:47:40.539 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.151214919, 2024-03-30T13:47:40.539 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:40.539 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:40.539 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:47:40.539 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:40.539 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.539 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:40.539 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291937+0000", 2024-03-30T13:47:40.539 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.539 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.540 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.540 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:40.540 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291937+0000", 2024-03-30T13:47:40.540 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.540 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.540 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.540 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:40.540 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291937+0000", 2024-03-30T13:47:40.540 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.9400000000000001e-07 2024-03-30T13:47:40.540 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.540 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.540 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:40.540 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291940+0000", 2024-03-30T13:47:40.541 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.4269999999999998e-06 2024-03-30T13:47:40.541 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.541 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.541 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:40.541 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291947+0000", 2024-03-30T13:47:40.541 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1019999999999999e-06 2024-03-30T13:47:40.541 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.541 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.542 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:40.542 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291952+0000", 2024-03-30T13:47:40.542 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6750000000000001e-06 2024-03-30T13:47:40.542 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.542 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.543 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:40.543 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443074+0000", 2024-03-30T13:47:40.543 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15112245299999999 2024-03-30T13:47:40.543 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.543 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.543 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:40.543 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443078+0000", 2024-03-30T13:47:40.544 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.9840000000000003e-06 2024-03-30T13:47:40.544 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.544 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.544 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:40.544 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443152+0000", 2024-03-30T13:47:40.544 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.3984000000000002e-05 2024-03-30T13:47:40.544 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.544 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.544 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.544 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.544 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.544 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:186951 7.5 7:a7e7bfd7:::benchmark_data_smithi043_86989_object64613:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-30T13:47:40.544 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.292317+0000", 2024-03-30T13:47:40.545 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.1226657, 2024-03-30T13:47:40.545 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.181689726, 2024-03-30T13:47:40.545 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:40.545 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:40.545 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:40.545 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:40.545 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:40.545 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:40.545 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 186951 2024-03-30T13:47:40.545 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.545 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:40.545 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.545 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:40.545 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.292317+0000", 2024-03-30T13:47:40.546 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.546 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.546 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.546 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:40.546 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.292317+0000", 2024-03-30T13:47:40.546 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.546 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.546 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.546 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:40.546 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.292318+0000", 2024-03-30T13:47:40.546 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.624e-06 2024-03-30T13:47:40.546 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.547 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.547 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:40.547 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.292323+0000", 2024-03-30T13:47:40.547 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.4150000000000001e-06 2024-03-30T13:47:40.547 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.547 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.547 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:40.547 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.292334+0000", 2024-03-30T13:47:40.547 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1192000000000001e-05 2024-03-30T13:47:40.547 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.547 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.547 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:40.547 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.292342+0000", 2024-03-30T13:47:40.547 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.7540000000000006e-06 2024-03-30T13:47:40.548 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.548 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.548 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:40.548 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.446331+0000", 2024-03-30T13:47:40.548 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.153989391 2024-03-30T13:47:40.548 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.548 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.548 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:40.548 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.446450+0000", 2024-03-30T13:47:40.548 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000118868 2024-03-30T13:47:40.548 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.548 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:40.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:40.550 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:47:40.550 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.446530+0000", 2024-03-30T13:47:40.550 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.0203000000000004e-05 2024-03-30T13:47:40.550 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.550 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.550 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:40.550 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.450029+0000", 2024-03-30T13:47:40.550 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00349863 2024-03-30T13:47:40.550 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.550 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.551 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.551 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.464588+0000", 2024-03-30T13:47:40.551 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.014559631999999999 2024-03-30T13:47:40.551 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.551 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.551 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.551 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.473921+0000", 2024-03-30T13:47:40.551 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0093331649999999992 2024-03-30T13:47:40.551 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.552 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.552 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:40.552 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.473961+0000", 2024-03-30T13:47:40.552 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.9443000000000002e-05 2024-03-30T13:47:40.552 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.552 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.552 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:40.552 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.474006+0000", 2024-03-30T13:47:40.552 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.5408999999999998e-05 2024-03-30T13:47:40.552 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.552 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.552 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.552 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.553 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.553 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop_reply(client.14817.0:186926 7.5 e131/98 ondisk, result = 0)", 2024-03-30T13:47:40.553 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.296461+0000", 2024-03-30T13:47:40.553 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.1185215550000001, 2024-03-30T13:47:40.553 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.146852222, 2024-03-30T13:47:40.553 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:40.553 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:40.553 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:47:40.553 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:40.553 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.553 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:40.553 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.296461+0000", 2024-03-30T13:47:40.553 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.553 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.554 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.554 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:40.554 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.296461+0000", 2024-03-30T13:47:40.554 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.554 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.554 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.554 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:40.554 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.296461+0000", 2024-03-30T13:47:40.554 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3900000000000004e-07 2024-03-30T13:47:40.554 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.555 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.555 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:40.555 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.296465+0000", 2024-03-30T13:47:40.555 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.162e-06 2024-03-30T13:47:40.555 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.555 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.555 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:40.555 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.296473+0000", 2024-03-30T13:47:40.555 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.1960000000000008e-06 2024-03-30T13:47:40.555 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.555 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.555 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:40.555 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.296478+0000", 2024-03-30T13:47:40.555 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6300000000000003e-06 2024-03-30T13:47:40.556 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.556 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.556 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:40.556 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443182+0000", 2024-03-30T13:47:40.556 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.14670403200000001 2024-03-30T13:47:40.556 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.556 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.556 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:40.556 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443189+0000", 2024-03-30T13:47:40.556 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6699999999999997e-06 2024-03-30T13:47:40.556 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.556 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.556 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:40.557 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443313+0000", 2024-03-30T13:47:40.557 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000123893 2024-03-30T13:47:40.557 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.557 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.557 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.557 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.557 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.557 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:186962 7.5 7:a78141fa:::benchmark_data_smithi043_86989_object64624:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-30T13:47:40.557 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.303756+0000", 2024-03-30T13:47:40.557 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.111226048, 2024-03-30T13:47:40.557 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.17041271799999999, 2024-03-30T13:47:40.557 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:47:40.557 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:47:40.557 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:47:40.557 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:47:40.558 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:47:40.558 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:47:40.558 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 186962 2024-03-30T13:47:40.558 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.558 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:47:40.558 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.558 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:47:40.558 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.303756+0000", 2024-03-30T13:47:40.558 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.558 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.558 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.558 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:47:40.558 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.303756+0000", 2024-03-30T13:47:40.559 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:47:40.559 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.559 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.559 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:47:40.559 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.303758+0000", 2024-03-30T13:47:40.559 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3790000000000001e-06 2024-03-30T13:47:40.559 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.559 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.559 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:47:40.559 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.303762+0000", 2024-03-30T13:47:40.559 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7160000000000002e-06 2024-03-30T13:47:40.559 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.559 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.560 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:47:40.560 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.303774+0000", 2024-03-30T13:47:40.560 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1916000000000001e-05 2024-03-30T13:47:40.560 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.560 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.560 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:47:40.560 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.303783+0000", 2024-03-30T13:47:40.560 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.7760000000000003e-06 2024-03-30T13:47:40.560 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.561 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.561 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:47:40.561 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.448546+0000", 2024-03-30T13:47:40.561 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.14476262400000001 2024-03-30T13:47:40.561 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.561 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.561 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:47:40.561 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.448623+0000", 2024-03-30T13:47:40.561 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.7118000000000003e-05 2024-03-30T13:47:40.561 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.562 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.562 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:47:40.562 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.448692+0000", 2024-03-30T13:47:40.562 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.9321999999999998e-05 2024-03-30T13:47:40.562 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.562 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.562 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:47:40.562 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.454432+0000", 2024-03-30T13:47:40.562 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0057398090000000002 2024-03-30T13:47:40.562 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.563 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.563 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.563 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.464644+0000", 2024-03-30T13:47:40.563 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.010211981 2024-03-30T13:47:40.563 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.563 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.563 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:47:40.563 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.474058+0000", 2024-03-30T13:47:40.563 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0094136930000000008 2024-03-30T13:47:40.563 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.563 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.563 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:47:40.563 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.474113+0000", 2024-03-30T13:47:40.563 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5714999999999998e-05 2024-03-30T13:47:40.564 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:47:40.564 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:47:40.564 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:47:40.564 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.474169+0000", 2024-03-30T13:47:40.564 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5668999999999998e-05 2024-03-30T13:47:40.564 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.564 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.564 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.564 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:47:40.564 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:47:40.564 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:47:40.650 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:40.680 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:40.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:40.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:40.781 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:40.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:40.812 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:40.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:40.913 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:40.952 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:40.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:40.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:41.007 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:47:41.008 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-30T13:47:41.008 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 'trim stale osdmaps' 2024-03-30T13:47:41.017 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-03-30T13:47:41.055 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:41.083 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:41.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:41.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:41.185 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:41.213 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:41.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:41.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:41.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:41 smithi113 ceph-mon[34341]: 4.1e scrub starts 2024-03-30T13:47:41.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:41 smithi113 ceph-mon[34341]: 4.1e scrub ok 2024-03-30T13:47:41.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:41 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/216530438' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:47:41.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:41 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:47:41.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:41 smithi113 ceph-mon[34341]: 6.17 scrub starts 2024-03-30T13:47:41.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:41 smithi113 ceph-mon[34341]: 6.17 scrub ok 2024-03-30T13:47:41.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:41 smithi113 ceph-mon[34341]: 3.18 scrub starts 2024-03-30T13:47:41.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:41 smithi113 ceph-mon[34341]: 3.18 scrub ok 2024-03-30T13:47:41.316 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:41.369 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:41 smithi043 ceph-mon[29963]: 4.1e scrub starts 2024-03-30T13:47:41.369 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:41 smithi043 ceph-mon[29963]: 4.1e scrub ok 2024-03-30T13:47:41.369 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:41 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/216530438' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:47:41.369 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:41 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:47:41.370 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:41 smithi043 ceph-mon[29963]: 6.17 scrub starts 2024-03-30T13:47:41.370 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:41 smithi043 ceph-mon[29963]: 6.17 scrub ok 2024-03-30T13:47:41.370 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:41 smithi043 ceph-mon[29963]: 3.18 scrub starts 2024-03-30T13:47:41.370 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:41 smithi043 ceph-mon[29963]: 3.18 scrub ok 2024-03-30T13:47:41.373 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:41 smithi043 ceph-mon[42099]: 4.1e scrub starts 2024-03-30T13:47:41.373 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:41 smithi043 ceph-mon[42099]: 4.1e scrub ok 2024-03-30T13:47:41.373 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:41 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/216530438' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:47:41.373 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:41 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:47:41.373 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:41 smithi043 ceph-mon[42099]: 6.17 scrub starts 2024-03-30T13:47:41.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:41 smithi043 ceph-mon[42099]: 6.17 scrub ok 2024-03-30T13:47:41.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:41 smithi043 ceph-mon[42099]: 3.18 scrub starts 2024-03-30T13:47:41.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:41 smithi043 ceph-mon[42099]: 3.18 scrub ok 2024-03-30T13:47:41.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:41.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:41.476 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:41.505 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:41.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:41.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:41.543 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-30T13:47:41.608 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:41.635 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:41.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:41.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:41.737 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:41.811 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:41.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:41.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:41.852 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:47:41.913 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:41.971 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:41.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:41.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:42.074 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:42.122 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:42.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:42.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:42.224 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:42.251 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:42.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:42.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:42.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:42 smithi043 ceph-mon[42099]: 5.1d scrub starts 2024-03-30T13:47:42.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:42 smithi043 ceph-mon[42099]: 5.1d scrub ok 2024-03-30T13:47:42.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:42 smithi043 ceph-mon[42099]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:47:42.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:42 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:47:42.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:42 smithi043 ceph-mon[42099]: osdmap e132: 8 total, 8 up, 8 in 2024-03-30T13:47:42.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:42 smithi043 ceph-mon[42099]: pgmap v181: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 797 B/s rd, 0 B/s wr, 623 op/s 2024-03-30T13:47:42.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:42 smithi043 ceph-mon[29963]: 5.1d scrub starts 2024-03-30T13:47:42.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:42 smithi043 ceph-mon[29963]: 5.1d scrub ok 2024-03-30T13:47:42.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:42 smithi043 ceph-mon[29963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:47:42.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:42 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:47:42.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:42 smithi043 ceph-mon[29963]: osdmap e132: 8 total, 8 up, 8 in 2024-03-30T13:47:42.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:42 smithi043 ceph-mon[29963]: pgmap v181: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 797 B/s rd, 0 B/s wr, 623 op/s 2024-03-30T13:47:42.354 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:42.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:42.409 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:42.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:42.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:42 smithi113 ceph-mon[34341]: 5.1d scrub starts 2024-03-30T13:47:42.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:42 smithi113 ceph-mon[34341]: 5.1d scrub ok 2024-03-30T13:47:42.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:42 smithi113 ceph-mon[34341]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:47:42.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:42 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:47:42.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:42 smithi113 ceph-mon[34341]: osdmap e132: 8 total, 8 up, 8 in 2024-03-30T13:47:42.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:42 smithi113 ceph-mon[34341]: pgmap v181: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 797 B/s rd, 0 B/s wr, 623 op/s 2024-03-30T13:47:42.510 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:42.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:42.540 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:42.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:42.641 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:42.686 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:42.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:42.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:42.692 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:47:42.789 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:42.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:42.820 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:42.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:42.921 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:42.950 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:42.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:42.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:43.052 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:43.081 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:43.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:43.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:43.183 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:43.204 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:47:43.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:43.249 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:43.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:43.350 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:43.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:43.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:43.504 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:43.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:43.550 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:43.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:43.652 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:43.695 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:43.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:43.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:43.797 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:43.824 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T13:47:43.825 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:43.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:43.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:43.927 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:43.954 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:43.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:43.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:44.023 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:43 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:44.024 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:43 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:47:44.024 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:43 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:44.024 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:43 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:47:44.055 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:44.081 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:44.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:44.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:44.183 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:44.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:43 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:47:44.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:43 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:47:44.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:44.225 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:44.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:44.326 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:44.342 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:47:44.342 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:47:44.342 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:47:44.342 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:47:44.342 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:47:44.355 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:44.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:44.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:44.375 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:47:44.414 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:47:44.456 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:47:44.457 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:44.470 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:47:44.470 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:47:44.470 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:47:44.470 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:47:44.470 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:47:44.470 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:47:44.471 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:47:44.471 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:47:44.485 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:44.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:44.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:44.587 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:44.616 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:44.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:44.718 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:44.747 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:44.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:44.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:44.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:44.894 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:44.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:44.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:44.996 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:45.031 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:45.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:45.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:45.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:44 smithi043 ceph-mon[29963]: pgmap v182: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 534 op/s 2024-03-30T13:47:45.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:44 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2616195807' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:47:45.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:44 smithi043 ceph-mon[42099]: pgmap v182: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 534 op/s 2024-03-30T13:47:45.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:44 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2616195807' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:47:45.133 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:45.160 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:45.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:45.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:45.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:44 smithi113 ceph-mon[34341]: pgmap v182: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 534 op/s 2024-03-30T13:47:45.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:44 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2616195807' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:47:45.262 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:45.299 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:45.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:45.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:45.400 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:45.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:45.429 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:45.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:45.530 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:45.560 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:45.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:45.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:45.661 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:45.697 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:45.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:45.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:45.799 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:45.837 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:45.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:45.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:45.939 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:45.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:45.967 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:45.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:46.068 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:46.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:45 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:47:46.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:45 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:47:46.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:45 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:47:46.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:45 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:47:46.106 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:46.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:46.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:46.208 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:46.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:45 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:47:46.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:45 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:47:46.236 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:46.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:46.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:46.337 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:46.376 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:46.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:46.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:46.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:47:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:47:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:47:46.478 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:46.522 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:46.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:46.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:46.624 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:46.673 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:46.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:46.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:46.775 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:46.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:46.819 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:46.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:46.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:46 smithi043 ceph-mon[29963]: pgmap v183: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:47:46.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:46 smithi043 ceph-mon[29963]: osdmap e133: 8 total, 8 up, 8 in 2024-03-30T13:47:46.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:46 smithi043 ceph-mon[42099]: pgmap v183: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:47:46.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:46 smithi043 ceph-mon[42099]: osdmap e133: 8 total, 8 up, 8 in 2024-03-30T13:47:46.920 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:46.948 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:46.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:46.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:47.026 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:47:47.050 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:47.111 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:47.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:47.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:47.213 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:47.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:46 smithi113 ceph-mon[34341]: pgmap v183: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:47:47.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:46 smithi113 ceph-mon[34341]: osdmap e133: 8 total, 8 up, 8 in 2024-03-30T13:47:47.239 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:47.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:47.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:47.341 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:47.368 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:47.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:47.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:47.470 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:47.496 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:47.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:47.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:47.599 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:47.636 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:47.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:47.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:47.738 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:47.776 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:47.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:47.878 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:47.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:47.922 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:47.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:48.023 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:48.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:48.065 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:48.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:48.167 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:48.192 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:47:48.192 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:47:48.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:48.197 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:48.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:48.275 INFO:teuthology.orchestra.run.smithi043.stderr:no valid command found; 10 closest matches: 2024-03-30T13:47:48.276 INFO:teuthology.orchestra.run.smithi043.stderr:0 2024-03-30T13:47:48.276 INFO:teuthology.orchestra.run.smithi043.stderr:1 2024-03-30T13:47:48.276 INFO:teuthology.orchestra.run.smithi043.stderr:2 2024-03-30T13:47:48.276 INFO:teuthology.orchestra.run.smithi043.stderr:abort 2024-03-30T13:47:48.276 INFO:teuthology.orchestra.run.smithi043.stderr:assert 2024-03-30T13:47:48.276 INFO:teuthology.orchestra.run.smithi043.stderr:bench [] [] [] [] 2024-03-30T13:47:48.276 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs debug_inject_read_zeros 2024-03-30T13:47:48.276 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs files list 2024-03-30T13:47:48.276 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs stats 2024-03-30T13:47:48.276 INFO:teuthology.orchestra.run.smithi043.stderr:bluestore allocator dump block 2024-03-30T13:47:48.276 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: invalid command 2024-03-30T13:47:48.299 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:48.330 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:48.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:48.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:48.432 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:48.479 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:48.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:48.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:48.581 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:48.622 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:48.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:48.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:48.724 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:48.767 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:48.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:48.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:48.869 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:48.911 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:48.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:48.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:48.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:48 smithi113 ceph-mon[34341]: pgmap v185: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 520 op/s 2024-03-30T13:47:48.972 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:47:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:47:49.013 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:49.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:49.040 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:49.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:49.071 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:47:49.110 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:48 smithi043 ceph-mon[29963]: pgmap v185: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 520 op/s 2024-03-30T13:47:49.111 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:48 smithi043 ceph-mon[42099]: pgmap v185: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 520 op/s 2024-03-30T13:47:49.142 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:49.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:49.185 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:49.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:49.248 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:47:49.249 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-30T13:47:49.249 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 'trim stale osdmaps' 2024-03-30T13:47:49.286 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:49.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:49.335 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:49.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:49.436 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:49.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:49.487 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:49.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:49.588 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:49.615 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:49.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:49.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:49.717 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:49.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:49.746 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:49.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:49.847 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:49.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:49.880 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:49.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:49.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:49 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:47:49.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:49 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:47:49.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:49 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:47:49.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:49 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:47:49.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:49 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:47:49.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:49 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:47:49.980 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:49.986 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-03-30T13:47:50.023 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:50.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:50.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:50.125 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:50.159 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:50.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:50.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:50.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:49 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:47:50.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:49 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:47:50.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:49 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:47:50.261 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:50.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:50.292 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:50.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:50.393 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:50.437 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:50.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:50.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:50.539 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:50.576 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:50.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:50.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:50.678 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:50.727 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:50.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:50.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:50.828 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:50.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:50.868 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:50.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:50.969 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:51.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:51.009 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:51.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:51.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:50 smithi043 ceph-mon[29963]: pgmap v186: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:47:51.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:50 smithi043 ceph-mon[29963]: osdmap e134: 8 total, 8 up, 8 in 2024-03-30T13:47:51.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:50 smithi043 ceph-mon[42099]: pgmap v186: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:47:51.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:50 smithi043 ceph-mon[42099]: osdmap e134: 8 total, 8 up, 8 in 2024-03-30T13:47:51.110 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:51.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:51.139 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:51.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:51.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:50 smithi113 ceph-mon[34341]: pgmap v186: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:47:51.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:50 smithi113 ceph-mon[34341]: osdmap e134: 8 total, 8 up, 8 in 2024-03-30T13:47:51.223 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:47:51.227 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:47:51.239 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:51.267 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:51.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:51.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:51.369 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:51.399 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:51.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:51.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:51.502 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:51.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:51.564 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:51.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:51.665 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:51.695 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:51.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:51.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:51.797 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:51.837 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:51.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:51.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:51.939 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:51.968 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:51.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:51.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:52.070 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:52.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:52.111 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:52.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:52.212 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:52.236 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:52.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:52.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:52.338 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:52.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:52.375 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:52.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:52.476 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:52.513 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:52.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:52.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:52.615 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:52.644 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:52.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:52.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:52.745 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:52.785 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:52.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:52.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:52.886 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:52.928 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:52.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:52.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:53.030 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:53.066 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:53.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:53.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:53.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:52 smithi043 ceph-mon[29963]: 7.5 deep-scrub ok 2024-03-30T13:47:53.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:52 smithi043 ceph-mon[29963]: pgmap v188: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-03-30T13:47:53.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:52 smithi043 ceph-mon[42099]: 7.5 deep-scrub ok 2024-03-30T13:47:53.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:52 smithi043 ceph-mon[42099]: pgmap v188: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-03-30T13:47:53.168 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:53.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:53.212 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:53.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:53.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:52 smithi113 ceph-mon[34341]: 7.5 deep-scrub ok 2024-03-30T13:47:53.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:52 smithi113 ceph-mon[34341]: pgmap v188: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-03-30T13:47:53.280 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:47:53.313 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:53.345 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:53.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:53.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:53.447 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:53.474 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:53.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:53.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:53.575 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:53.605 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:53.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:53.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:53.707 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:53.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:53.749 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:53.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:53.771 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-03-30T13:47:53.850 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:53.892 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:53.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:53.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:53.993 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:54.024 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:54.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:54.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:54.125 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:54.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:53 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2343423796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:47:54.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:53 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2343423796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:47:54.161 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:54.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:54.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:54.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:53 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2343423796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:47:54.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:54.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:54.311 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:54.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:54.413 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:54.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:54.445 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:54.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:54.546 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:54.585 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:54.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:54.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:54.689 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:54.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:54.730 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:54.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:54.831 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:54.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:54.866 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:54.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:54.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:54 smithi043 ceph-mon[29963]: pgmap v189: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 574 op/s 2024-03-30T13:47:54.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:54 smithi043 ceph-mon[29963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:47:54.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:54 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2343423796' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:47:54.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:54 smithi043 ceph-mon[29963]: osdmap e135: 8 total, 8 up, 8 in 2024-03-30T13:47:54.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:54 smithi043 ceph-mon[42099]: pgmap v189: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 574 op/s 2024-03-30T13:47:54.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:54 smithi043 ceph-mon[42099]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:47:54.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:54 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2343423796' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:47:54.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:54 smithi043 ceph-mon[42099]: osdmap e135: 8 total, 8 up, 8 in 2024-03-30T13:47:54.967 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:54.995 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:54.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:54.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:55.097 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:55.137 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:55.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:55.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:55.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:54 smithi113 ceph-mon[34341]: pgmap v189: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 574 op/s 2024-03-30T13:47:55.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:54 smithi113 ceph-mon[34341]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:47:55.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:54 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2343423796' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:47:55.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:54 smithi113 ceph-mon[34341]: osdmap e135: 8 total, 8 up, 8 in 2024-03-30T13:47:55.238 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:55.266 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:55.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:55.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:55.368 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:55.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:55.392 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:55.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:55.493 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:55.522 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:55.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:55.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:55.624 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:55.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:55.661 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:55.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:55.762 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:55.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:55.799 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:55.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:55.900 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:55.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:55.935 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:55.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:56.036 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:56.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:56.065 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:56.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:56.166 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:56.195 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:56.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:56.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:56.297 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:56.326 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:56.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:56.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:56.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:47:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:47:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:47:56.428 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:56.452 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:56.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:56.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:56.554 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:56.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:56.582 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:56.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:56.683 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:56.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:56.718 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:56.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:56.819 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:56.844 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:56.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:56.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:56.945 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:56.986 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T13:47:56.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:56.988 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:56.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:57.089 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:57.120 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:57.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:57.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:57.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:56 smithi043 ceph-mon[29963]: pgmap v191: 148 pgs: 148 active+clean; 2.8 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-30T13:47:57.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:56 smithi043 ceph-mon[42099]: pgmap v191: 148 pgs: 148 active+clean; 2.8 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-30T13:47:57.222 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:47:57.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:56 smithi113 ceph-mon[34341]: pgmap v191: 148 pgs: 148 active+clean; 2.8 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-30T13:47:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:57.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:57.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:47:57.373 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:57.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:57.407 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:57.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:57.508 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:57.537 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:57.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:57.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:57.639 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:57.685 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:57.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:57.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:57.787 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:57.814 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:57.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:57.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:57.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:57 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:47:57.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:57 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:47:57.915 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:57.946 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:57.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:57.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:58.048 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:58.080 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:58.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:58.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:58.181 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:58.211 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:58.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:58.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:58.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:57 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:47:58.313 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:58.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:58.345 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:58.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:58.446 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:58.473 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:58.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:58.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:58.576 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:58.605 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:58.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:58.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:58.706 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:47:58.735 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:58.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:58.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:47:58.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:47:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:47:58.837 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:58.890 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:58.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:58.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:58.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:58 smithi043 ceph-mon[29963]: pgmap v192: 148 pgs: 148 active+clean; 2.8 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 583 op/s 2024-03-30T13:47:58.980 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:58 smithi043 ceph-mon[42099]: pgmap v192: 148 pgs: 148 active+clean; 2.8 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 583 op/s 2024-03-30T13:47:58.993 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:47:59.046 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:59.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:59.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:47:59.148 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:59.179 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:59.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:59.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:59.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:58 smithi113 ceph-mon[34341]: pgmap v192: 148 pgs: 148 active+clean; 2.8 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 583 op/s 2024-03-30T13:47:59.282 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:47:59.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:59.312 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:59.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:47:59.413 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:59.467 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:59.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:59.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:47:59.568 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:47:59.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:59.610 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:47:59.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:47:59.688 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:47:59.711 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:47:59.736 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:47:59.739 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:59.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:59.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:47:59.790 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:47:59.836 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:47:59.842 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:47:59.869 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:47:59.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:47:59.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:47:59.874 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:47:59.914 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:47:59.915 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:59 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:47:59.915 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:47:59 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4096097237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:47:59.915 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:59 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:47:59.915 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:47:59 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/4096097237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:47:59.952 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:47:59.970 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:47:59.995 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:48:00.003 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:48:00.003 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:48:00.004 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:48:00.004 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:48:00.004 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:48:00.004 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:48:00.004 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:48:00.004 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:48:00.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:00.018 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:00.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:00.119 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:00.163 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:00.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:00.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:00.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:59 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:48:00.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:47:59 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/4096097237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:48:00.265 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:00.295 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:00.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:00.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:00.396 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:00.427 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:00.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:00.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:00.529 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:00.559 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:00.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:00.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:00.661 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:00.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:00.699 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:00.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:00.800 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:00.801 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:48:00.833 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:00.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:00.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:00.935 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:00.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:00.981 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:00.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:01.082 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:01.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:00 smithi043 ceph-mon[29963]: pgmap v193: 148 pgs: 148 active+clean; 2.4 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:48:01.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:00 smithi043 ceph-mon[42099]: pgmap v193: 148 pgs: 148 active+clean; 2.4 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:48:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:01.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:01.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:01.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:00 smithi113 ceph-mon[34341]: pgmap v193: 148 pgs: 148 active+clean; 2.4 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:48:01.227 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:01.276 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:01.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:01.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:01.378 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:01.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:01.427 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:01.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:01.528 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:01.557 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:01.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:01.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:01.658 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:01.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:01.715 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:01.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:01.815 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:01.856 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:01.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:01.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:01.958 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:02.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:02.018 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:02.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:02.119 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:02.148 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:02.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:02.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:02.250 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:02.321 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:02.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:02.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:02.423 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:02.449 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:02.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:02.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:02.551 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:02.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:02.578 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:02.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:02.679 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:02.718 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:02.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:02.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:02.820 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:02.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:02.859 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:02.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:02.960 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:02.986 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:02.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:02.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:03.070 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:02 smithi043 ceph-mon[29963]: pgmap v194: 148 pgs: 148 active+clean; 2.4 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 980 op/s 2024-03-30T13:48:03.070 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:02 smithi043 ceph-mon[42099]: pgmap v194: 148 pgs: 148 active+clean; 2.4 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 980 op/s 2024-03-30T13:48:03.089 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:03.117 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:03.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:03.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:03.216 INFO:teuthology.orchestra.run.smithi043.stderr:no valid command found; 10 closest matches: 2024-03-30T13:48:03.216 INFO:teuthology.orchestra.run.smithi043.stderr:0 2024-03-30T13:48:03.216 INFO:teuthology.orchestra.run.smithi043.stderr:1 2024-03-30T13:48:03.216 INFO:teuthology.orchestra.run.smithi043.stderr:2 2024-03-30T13:48:03.216 INFO:teuthology.orchestra.run.smithi043.stderr:abort 2024-03-30T13:48:03.216 INFO:teuthology.orchestra.run.smithi043.stderr:assert 2024-03-30T13:48:03.216 INFO:teuthology.orchestra.run.smithi043.stderr:bench [] [] [] [] 2024-03-30T13:48:03.216 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs debug_inject_read_zeros 2024-03-30T13:48:03.216 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs files list 2024-03-30T13:48:03.216 INFO:teuthology.orchestra.run.smithi043.stderr:bluefs stats 2024-03-30T13:48:03.216 INFO:teuthology.orchestra.run.smithi043.stderr:bluestore allocator dump block 2024-03-30T13:48:03.216 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: invalid command 2024-03-30T13:48:03.218 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:03.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:02 smithi113 ceph-mon[34341]: pgmap v194: 148 pgs: 148 active+clean; 2.4 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 980 op/s 2024-03-30T13:48:03.247 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:03.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:03.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:03.349 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:03.378 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:03.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:03.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:03.480 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:03.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:03.509 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:03.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:03.610 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:03.653 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:03.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:03.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:03.747 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:48:03.755 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:03.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:03.842 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:03.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:03.944 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:03.969 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:03.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:03.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:04.070 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:04.099 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:04.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:04.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:04.201 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:04.230 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:04.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:04.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:04.332 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:04.358 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:04.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:04.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:04.460 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:04.488 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:04.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:04.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:04.590 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:04.617 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:04.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:04.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:04.719 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:04.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:04.760 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:04.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:04.861 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:04.882 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:48:04.882 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T13:48:04.883 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-03-30T13:48:04.883 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-03-30T13:48:04.883 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:48:04.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:04.900 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:04.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:05.001 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:05.043 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:05.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:05.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:05.145 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:05.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:04 smithi043 ceph-mon[29963]: pgmap v195: 148 pgs: 148 active+clean; 2.4 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 985 op/s 2024-03-30T13:48:05.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:04 smithi043 ceph-mon[42099]: pgmap v195: 148 pgs: 148 active+clean; 2.4 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 985 op/s 2024-03-30T13:48:05.173 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:05.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:05.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:05.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:04 smithi113 ceph-mon[34341]: pgmap v195: 148 pgs: 148 active+clean; 2.4 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 985 op/s 2024-03-30T13:48:05.276 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:05.303 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:05.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:05.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:05.405 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:05.441 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:05.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:05.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:05.545 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:05.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:05.588 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:05.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:05.596 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-03-30T13:48:05.689 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:05.717 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:05.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:05.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:05.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:05.883 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:05.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:05.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:05.985 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:06.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:06.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:05 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:48:06.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:05 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:48:06.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:05 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:48:06.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:05 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:48:06.133 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:06.162 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:06.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:06.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:06.208 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:48:06.209 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-30T13:48:06.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:05 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:48:06.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:05 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:48:06.264 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:06.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:06.319 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:06.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:06.394 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:48:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:48:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:48:06.421 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:06.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:06.467 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:06.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:06.568 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:06.598 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:06.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:06.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:06.700 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:06.729 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:06.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:06.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:06.818 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:48:06.822 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:48:06.831 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:06.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:06.904 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:06.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:06.972 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:06 smithi043 ceph-mon[29963]: pgmap v196: 148 pgs: 148 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 881 B/s rd, 0 B/s wr, 975 op/s 2024-03-30T13:48:06.972 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:06 smithi043 ceph-mon[29963]: osdmap e136: 8 total, 8 up, 8 in 2024-03-30T13:48:06.974 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:06 smithi043 ceph-mon[42099]: pgmap v196: 148 pgs: 148 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 881 B/s rd, 0 B/s wr, 975 op/s 2024-03-30T13:48:06.974 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:06 smithi043 ceph-mon[42099]: osdmap e136: 8 total, 8 up, 8 in 2024-03-30T13:48:07.004 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:07.055 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:07.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:07.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:07.156 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:07.206 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:07.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:07.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:07.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:06 smithi113 ceph-mon[34341]: pgmap v196: 148 pgs: 148 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 881 B/s rd, 0 B/s wr, 975 op/s 2024-03-30T13:48:07.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:06 smithi113 ceph-mon[34341]: osdmap e136: 8 total, 8 up, 8 in 2024-03-30T13:48:07.307 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:07.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:07.346 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:07.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:07.447 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:07.476 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:07.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:07.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:07.578 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:07.610 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:07.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:07.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:07.711 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:07.752 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:07.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:07.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:07.854 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:07.875 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T13:48:07.884 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:07.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:07.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:07.985 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:08.016 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:08.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:08.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:08.117 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:08.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:07 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3915814187' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:48:08.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:07 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:48:08.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:07 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3915814187' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:48:08.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:07 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:48:08.151 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:08.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:08.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:08.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:07 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3915814187' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:48:08.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:07 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:48:08.253 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:08.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:08.298 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:08.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:08.399 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:08.433 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:08.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:08.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:08.535 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:08.591 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:08.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:08.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:08.693 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:08.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:08.744 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:08.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:08.845 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:08.850 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:48:08 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:48:08.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:08.885 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:08.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:08.986 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:09.025 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:09.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:09.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:09.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:09.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:08 smithi043 ceph-mon[29963]: pgmap v198: 148 pgs: 148 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-30T13:48:09.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:08 smithi043 ceph-mon[29963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:48:09.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:08 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:48:09.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:08 smithi043 ceph-mon[29963]: osdmap e137: 8 total, 8 up, 8 in 2024-03-30T13:48:09.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:08 smithi043 ceph-mon[29963]: 7.d scrub starts 2024-03-30T13:48:09.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:08 smithi043 ceph-mon[42099]: pgmap v198: 148 pgs: 148 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-30T13:48:09.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:08 smithi043 ceph-mon[42099]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:48:09.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:08 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:48:09.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:08 smithi043 ceph-mon[42099]: osdmap e137: 8 total, 8 up, 8 in 2024-03-30T13:48:09.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:08 smithi043 ceph-mon[42099]: 7.d scrub starts 2024-03-30T13:48:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:09.171 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:09.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:09.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:08 smithi113 ceph-mon[34341]: pgmap v198: 148 pgs: 148 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-30T13:48:09.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:08 smithi113 ceph-mon[34341]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:48:09.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:08 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:48:09.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:08 smithi113 ceph-mon[34341]: osdmap e137: 8 total, 8 up, 8 in 2024-03-30T13:48:09.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:08 smithi113 ceph-mon[34341]: 7.d scrub starts 2024-03-30T13:48:09.272 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:09.303 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:09.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:09.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:09.405 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:09.436 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:09.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:09.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:09.538 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:09.567 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:09.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:09.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:09.669 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:09.696 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:09.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:09.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:09.798 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:09.828 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:09.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:09.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:09.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:09 smithi043 ceph-mon[29963]: 5.4 scrub starts 2024-03-30T13:48:09.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:09 smithi043 ceph-mon[29963]: 5.4 scrub ok 2024-03-30T13:48:09.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:09 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:48:09.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:09 smithi043 ceph-mon[29963]: 7.d scrub ok 2024-03-30T13:48:09.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:09 smithi043 ceph-mon[29963]: 7.c scrub starts 2024-03-30T13:48:09.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:09 smithi043 ceph-mon[29963]: 5.18 scrub starts 2024-03-30T13:48:09.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:09 smithi043 ceph-mon[29963]: 5.18 scrub ok 2024-03-30T13:48:09.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:09 smithi043 ceph-mon[29963]: 6.f scrub starts 2024-03-30T13:48:09.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:09 smithi043 ceph-mon[29963]: 6.f scrub ok 2024-03-30T13:48:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:09 smithi043 ceph-mon[42099]: 5.4 scrub starts 2024-03-30T13:48:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:09 smithi043 ceph-mon[42099]: 5.4 scrub ok 2024-03-30T13:48:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:09 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:48:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:09 smithi043 ceph-mon[42099]: 7.d scrub ok 2024-03-30T13:48:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:09 smithi043 ceph-mon[42099]: 7.c scrub starts 2024-03-30T13:48:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:09 smithi043 ceph-mon[42099]: 5.18 scrub starts 2024-03-30T13:48:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:09 smithi043 ceph-mon[42099]: 5.18 scrub ok 2024-03-30T13:48:09.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:09 smithi043 ceph-mon[42099]: 6.f scrub starts 2024-03-30T13:48:09.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:09 smithi043 ceph-mon[42099]: 6.f scrub ok 2024-03-30T13:48:09.929 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:09.969 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:09.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:09.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:10.071 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:10.115 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:10.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:10.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:10.216 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:10.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:09 smithi113 ceph-mon[34341]: 5.4 scrub starts 2024-03-30T13:48:10.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:09 smithi113 ceph-mon[34341]: 5.4 scrub ok 2024-03-30T13:48:10.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:09 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:48:10.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:09 smithi113 ceph-mon[34341]: 7.d scrub ok 2024-03-30T13:48:10.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:09 smithi113 ceph-mon[34341]: 7.c scrub starts 2024-03-30T13:48:10.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:09 smithi113 ceph-mon[34341]: 5.18 scrub starts 2024-03-30T13:48:10.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:09 smithi113 ceph-mon[34341]: 5.18 scrub ok 2024-03-30T13:48:10.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:09 smithi113 ceph-mon[34341]: 6.f scrub starts 2024-03-30T13:48:10.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:09 smithi113 ceph-mon[34341]: 6.f scrub ok 2024-03-30T13:48:10.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:10.269 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:10.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:10.305 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:48:10.355 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:48:10.370 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:10.397 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:48:10.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:10.398 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:10.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:10.447 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:48:10.492 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:48:10.499 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:10.532 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:48:10.543 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:10.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:10.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:10.570 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:48:10.609 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:48:10.617 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:48:10.617 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:48:10.617 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:48:10.617 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:48:10.617 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:48:10.617 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:48:10.618 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:48:10.618 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:48:10.645 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:10.688 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:10.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:10.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:10.790 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:10.834 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:10.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:10.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:10.936 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:10.975 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:10.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:10.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:11.077 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:11.116 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:11.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:11.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:11.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:10 smithi043 ceph-mon[29963]: pgmap v200: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-30T13:48:11.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:10 smithi043 ceph-mon[29963]: 7.c scrub ok 2024-03-30T13:48:11.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:10 smithi043 ceph-mon[29963]: 6.2 scrub starts 2024-03-30T13:48:11.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:10 smithi043 ceph-mon[29963]: 6.2 scrub ok 2024-03-30T13:48:11.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:10 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1050294366' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:48:11.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:10 smithi043 ceph-mon[42099]: pgmap v200: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-30T13:48:11.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:10 smithi043 ceph-mon[42099]: 7.c scrub ok 2024-03-30T13:48:11.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:10 smithi043 ceph-mon[42099]: 6.2 scrub starts 2024-03-30T13:48:11.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:10 smithi043 ceph-mon[42099]: 6.2 scrub ok 2024-03-30T13:48:11.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:10 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1050294366' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:48:11.210 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T13:48:11.210 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:48:11.210 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-03-30T13:48:11.210 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-30T13:48:11.210 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-30T13:48:11.210 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-03-30T13:48:11.217 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:11.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:10 smithi113 ceph-mon[34341]: pgmap v200: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-30T13:48:11.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:10 smithi113 ceph-mon[34341]: 7.c scrub ok 2024-03-30T13:48:11.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:10 smithi113 ceph-mon[34341]: 6.2 scrub starts 2024-03-30T13:48:11.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:10 smithi113 ceph-mon[34341]: 6.2 scrub ok 2024-03-30T13:48:11.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:10 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1050294366' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:48:11.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:11.256 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:11.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:11.357 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:11.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:11.404 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:11.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:11.505 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:11.539 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:11.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:11.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:11.551 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:48:11.640 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:11.676 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:11.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:11.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:11.778 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:11.819 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:11.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:11.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:11.921 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:11.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:11.968 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:11.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:11.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:11 smithi113 ceph-mon[34341]: 6.4 scrub starts 2024-03-30T13:48:11.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:11 smithi113 ceph-mon[34341]: 6.4 scrub ok 2024-03-30T13:48:11.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:11 smithi113 ceph-mon[34341]: 6.b scrub starts 2024-03-30T13:48:11.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:11 smithi113 ceph-mon[34341]: 6.b scrub ok 2024-03-30T13:48:11.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:11 smithi113 ceph-mon[34341]: 3.a scrub starts 2024-03-30T13:48:11.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:11 smithi113 ceph-mon[34341]: 3.a scrub ok 2024-03-30T13:48:12.069 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:12.112 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:12.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:12.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:11 smithi043 ceph-mon[29963]: 6.4 scrub starts 2024-03-30T13:48:12.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:11 smithi043 ceph-mon[29963]: 6.4 scrub ok 2024-03-30T13:48:12.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:11 smithi043 ceph-mon[29963]: 6.b scrub starts 2024-03-30T13:48:12.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:11 smithi043 ceph-mon[29963]: 6.b scrub ok 2024-03-30T13:48:12.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:11 smithi043 ceph-mon[29963]: 3.a scrub starts 2024-03-30T13:48:12.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:11 smithi043 ceph-mon[29963]: 3.a scrub ok 2024-03-30T13:48:12.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:11 smithi043 ceph-mon[42099]: 6.4 scrub starts 2024-03-30T13:48:12.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:11 smithi043 ceph-mon[42099]: 6.4 scrub ok 2024-03-30T13:48:12.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:11 smithi043 ceph-mon[42099]: 6.b scrub starts 2024-03-30T13:48:12.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:11 smithi043 ceph-mon[42099]: 6.b scrub ok 2024-03-30T13:48:12.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:11 smithi043 ceph-mon[42099]: 3.a scrub starts 2024-03-30T13:48:12.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:11 smithi043 ceph-mon[42099]: 3.a scrub ok 2024-03-30T13:48:12.213 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:12.254 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:12.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:12.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:12.356 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:12.393 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:12.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:12.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:12.495 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:12.530 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:12.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:12.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:12.632 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:12.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:12.663 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:12.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:12.764 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:12.808 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:12.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:12.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:12.910 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:12.938 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:12.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:12.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:13.039 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:13.066 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:13.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:13.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:13.068 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T13:48:13.121 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:48:13.121 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:48:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:12 smithi043 ceph-mon[29963]: 5.9 scrub starts 2024-03-30T13:48:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:12 smithi043 ceph-mon[29963]: 5.9 scrub ok 2024-03-30T13:48:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:12 smithi043 ceph-mon[29963]: pgmap v201: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-30T13:48:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:12 smithi043 ceph-mon[29963]: 7.2 scrub starts 2024-03-30T13:48:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:12 smithi043 ceph-mon[29963]: 6.0 scrub starts 2024-03-30T13:48:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:12 smithi043 ceph-mon[29963]: 6.0 scrub ok 2024-03-30T13:48:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:12 smithi043 ceph-mon[29963]: 4.9 scrub starts 2024-03-30T13:48:13.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:12 smithi043 ceph-mon[29963]: 4.9 scrub ok 2024-03-30T13:48:13.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:12 smithi043 ceph-mon[29963]: 7.2 scrub ok 2024-03-30T13:48:13.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:12 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:48:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:12 smithi043 ceph-mon[42099]: 5.9 scrub starts 2024-03-30T13:48:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:12 smithi043 ceph-mon[42099]: 5.9 scrub ok 2024-03-30T13:48:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:12 smithi043 ceph-mon[42099]: pgmap v201: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-30T13:48:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:12 smithi043 ceph-mon[42099]: 7.2 scrub starts 2024-03-30T13:48:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:12 smithi043 ceph-mon[42099]: 6.0 scrub starts 2024-03-30T13:48:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:12 smithi043 ceph-mon[42099]: 6.0 scrub ok 2024-03-30T13:48:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:12 smithi043 ceph-mon[42099]: 4.9 scrub starts 2024-03-30T13:48:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:12 smithi043 ceph-mon[42099]: 4.9 scrub ok 2024-03-30T13:48:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:12 smithi043 ceph-mon[42099]: 7.2 scrub ok 2024-03-30T13:48:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:12 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:48:13.168 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:13.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:12 smithi113 ceph-mon[34341]: 5.9 scrub starts 2024-03-30T13:48:13.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:12 smithi113 ceph-mon[34341]: 5.9 scrub ok 2024-03-30T13:48:13.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:12 smithi113 ceph-mon[34341]: pgmap v201: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-30T13:48:13.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:12 smithi113 ceph-mon[34341]: 7.2 scrub starts 2024-03-30T13:48:13.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:12 smithi113 ceph-mon[34341]: 6.0 scrub starts 2024-03-30T13:48:13.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:12 smithi113 ceph-mon[34341]: 6.0 scrub ok 2024-03-30T13:48:13.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:12 smithi113 ceph-mon[34341]: 4.9 scrub starts 2024-03-30T13:48:13.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:12 smithi113 ceph-mon[34341]: 4.9 scrub ok 2024-03-30T13:48:13.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:12 smithi113 ceph-mon[34341]: 7.2 scrub ok 2024-03-30T13:48:13.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:12 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:48:13.235 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:13.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:13.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:13.337 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:13.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:13.408 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:13.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:13.509 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:13.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:13.539 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:13.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:13.640 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:13.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:13.694 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:13.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:13.795 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:13.835 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:13.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:13.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:13.937 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:13.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:13 smithi113 ceph-mon[34341]: 3.11 scrub starts 2024-03-30T13:48:13.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:13 smithi113 ceph-mon[34341]: 3.11 scrub ok 2024-03-30T13:48:13.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:13 smithi113 ceph-mon[34341]: 3.1 scrub starts 2024-03-30T13:48:13.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:13 smithi113 ceph-mon[34341]: 3.1 scrub ok 2024-03-30T13:48:13.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:13 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:48:13.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:13 smithi113 ceph-mon[34341]: 5.2 scrub starts 2024-03-30T13:48:13.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:13 smithi113 ceph-mon[34341]: 5.2 scrub ok 2024-03-30T13:48:13.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:13 smithi113 ceph-mon[34341]: 5.6 scrub starts 2024-03-30T13:48:13.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:13 smithi113 ceph-mon[34341]: 5.6 scrub ok 2024-03-30T13:48:13.975 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:13.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:14.077 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:14.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:14.114 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:14.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:14.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:13 smithi043 ceph-mon[29963]: 3.11 scrub starts 2024-03-30T13:48:14.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:13 smithi043 ceph-mon[29963]: 3.11 scrub ok 2024-03-30T13:48:14.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:13 smithi043 ceph-mon[29963]: 3.1 scrub starts 2024-03-30T13:48:14.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:13 smithi043 ceph-mon[29963]: 3.1 scrub ok 2024-03-30T13:48:14.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:13 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:48:14.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:13 smithi043 ceph-mon[29963]: 5.2 scrub starts 2024-03-30T13:48:14.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:13 smithi043 ceph-mon[29963]: 5.2 scrub ok 2024-03-30T13:48:14.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:13 smithi043 ceph-mon[29963]: 5.6 scrub starts 2024-03-30T13:48:14.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:13 smithi043 ceph-mon[29963]: 5.6 scrub ok 2024-03-30T13:48:14.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:13 smithi043 ceph-mon[42099]: 3.11 scrub starts 2024-03-30T13:48:14.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:13 smithi043 ceph-mon[42099]: 3.11 scrub ok 2024-03-30T13:48:14.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:13 smithi043 ceph-mon[42099]: 3.1 scrub starts 2024-03-30T13:48:14.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:13 smithi043 ceph-mon[42099]: 3.1 scrub ok 2024-03-30T13:48:14.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:13 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:48:14.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:13 smithi043 ceph-mon[42099]: 5.2 scrub starts 2024-03-30T13:48:14.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:13 smithi043 ceph-mon[42099]: 5.2 scrub ok 2024-03-30T13:48:14.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:13 smithi043 ceph-mon[42099]: 5.6 scrub starts 2024-03-30T13:48:14.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:13 smithi043 ceph-mon[42099]: 5.6 scrub ok 2024-03-30T13:48:14.153 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-30T13:48:14.215 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:14.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:14.276 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:14.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:14.297 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:48:14.377 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:14.380 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:48:14.381 INFO:teuthology.orchestra.run.smithi113.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-30T13:48:14.381 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:48:14.408 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:14.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:14.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:14.426 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:48:14.510 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:14.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:14.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:14.702 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:14.782 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:14.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:14.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:14.884 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:14.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:14.932 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:14.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:15.033 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:15.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:15.080 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:15.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:15.179 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:14 smithi113 ceph-mon[34341]: pgmap v202: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.8 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 691 op/s 2024-03-30T13:48:15.179 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:14 smithi113 ceph-mon[34341]: 7.0 scrub starts 2024-03-30T13:48:15.179 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:14 smithi113 ceph-mon[34341]: 7.0 scrub ok 2024-03-30T13:48:15.180 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:14 smithi113 ceph-mon[34341]: 5.e scrub starts 2024-03-30T13:48:15.180 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:14 smithi113 ceph-mon[34341]: 5.e scrub ok 2024-03-30T13:48:15.180 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:14 smithi113 ceph-mon[34341]: 6.14 scrub starts 2024-03-30T13:48:15.180 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:14 smithi113 ceph-mon[34341]: 6.14 scrub ok 2024-03-30T13:48:15.180 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:14 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:48:15.180 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:14 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:48:15.181 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:15.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:15.217 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:15.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:15.318 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:15.351 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:15.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:15.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:15.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:14 smithi043 ceph-mon[29963]: pgmap v202: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.8 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 691 op/s 2024-03-30T13:48:15.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:14 smithi043 ceph-mon[29963]: 7.0 scrub starts 2024-03-30T13:48:15.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:14 smithi043 ceph-mon[29963]: 7.0 scrub ok 2024-03-30T13:48:15.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:14 smithi043 ceph-mon[29963]: 5.e scrub starts 2024-03-30T13:48:15.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:14 smithi043 ceph-mon[29963]: 5.e scrub ok 2024-03-30T13:48:15.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:14 smithi043 ceph-mon[29963]: 6.14 scrub starts 2024-03-30T13:48:15.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:14 smithi043 ceph-mon[29963]: 6.14 scrub ok 2024-03-30T13:48:15.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:14 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:48:15.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:14 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:48:15.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:14 smithi043 ceph-mon[42099]: pgmap v202: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.8 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 691 op/s 2024-03-30T13:48:15.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:14 smithi043 ceph-mon[42099]: 7.0 scrub starts 2024-03-30T13:48:15.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:14 smithi043 ceph-mon[42099]: 7.0 scrub ok 2024-03-30T13:48:15.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:14 smithi043 ceph-mon[42099]: 5.e scrub starts 2024-03-30T13:48:15.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:14 smithi043 ceph-mon[42099]: 5.e scrub ok 2024-03-30T13:48:15.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:14 smithi043 ceph-mon[42099]: 6.14 scrub starts 2024-03-30T13:48:15.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:14 smithi043 ceph-mon[42099]: 6.14 scrub ok 2024-03-30T13:48:15.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:14 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:48:15.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:14 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:48:15.453 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:15.491 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:15.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:15.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:15.593 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:15.636 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:15.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:15.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:15.738 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:15.783 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:15.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:15.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:15.885 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:15.942 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:15.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:15.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:16.044 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:16.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:16.093 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:16.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:16.098 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:15 smithi043 ceph-mon[29963]: 4.8 scrub starts 2024-03-30T13:48:16.098 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:15 smithi043 ceph-mon[29963]: 4.8 scrub ok 2024-03-30T13:48:16.098 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:15 smithi043 ceph-mon[29963]: 5.1e scrub starts 2024-03-30T13:48:16.099 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:15 smithi043 ceph-mon[29963]: 5.1e scrub ok 2024-03-30T13:48:16.099 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:15 smithi043 ceph-mon[29963]: 3.9 scrub starts 2024-03-30T13:48:16.099 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:15 smithi043 ceph-mon[29963]: 3.9 scrub ok 2024-03-30T13:48:16.099 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:15 smithi043 ceph-mon[29963]: 5.d scrub starts 2024-03-30T13:48:16.099 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:15 smithi043 ceph-mon[29963]: 5.d scrub ok 2024-03-30T13:48:16.099 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:15 smithi043 ceph-mon[42099]: 4.8 scrub starts 2024-03-30T13:48:16.099 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:15 smithi043 ceph-mon[42099]: 4.8 scrub ok 2024-03-30T13:48:16.099 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:15 smithi043 ceph-mon[42099]: 5.1e scrub starts 2024-03-30T13:48:16.099 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:15 smithi043 ceph-mon[42099]: 5.1e scrub ok 2024-03-30T13:48:16.100 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:15 smithi043 ceph-mon[42099]: 3.9 scrub starts 2024-03-30T13:48:16.100 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:15 smithi043 ceph-mon[42099]: 3.9 scrub ok 2024-03-30T13:48:16.100 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:15 smithi043 ceph-mon[42099]: 5.d scrub starts 2024-03-30T13:48:16.100 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:15 smithi043 ceph-mon[42099]: 5.d scrub ok 2024-03-30T13:48:16.194 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:16.222 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:16.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:15 smithi113 ceph-mon[34341]: 4.8 scrub starts 2024-03-30T13:48:16.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:15 smithi113 ceph-mon[34341]: 4.8 scrub ok 2024-03-30T13:48:16.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:15 smithi113 ceph-mon[34341]: 5.1e scrub starts 2024-03-30T13:48:16.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:15 smithi113 ceph-mon[34341]: 5.1e scrub ok 2024-03-30T13:48:16.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:15 smithi113 ceph-mon[34341]: 3.9 scrub starts 2024-03-30T13:48:16.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:15 smithi113 ceph-mon[34341]: 3.9 scrub ok 2024-03-30T13:48:16.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:15 smithi113 ceph-mon[34341]: 5.d scrub starts 2024-03-30T13:48:16.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:15 smithi113 ceph-mon[34341]: 5.d scrub ok 2024-03-30T13:48:16.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:16.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:16.325 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:16.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:16.371 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:16.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:16.401 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:48:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:48:16.472 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:16.523 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:16.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:16.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:16.624 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:16.661 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:16.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:16.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:16.763 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:16.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:16.807 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:16.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:16.908 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:16.936 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:16.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:16.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:16.961 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:48:16.976 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:48:17.038 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:17.068 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:17.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:17.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:17.170 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:17.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:17.215 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:17.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:16 smithi113 ceph-mon[34341]: 3.e scrub starts 2024-03-30T13:48:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:16 smithi113 ceph-mon[34341]: 3.e scrub ok 2024-03-30T13:48:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:16 smithi113 ceph-mon[34341]: pgmap v203: 148 pgs: 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-30T13:48:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:16 smithi113 ceph-mon[34341]: 5.16 scrub starts 2024-03-30T13:48:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:16 smithi113 ceph-mon[34341]: 5.16 scrub ok 2024-03-30T13:48:17.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:16 smithi113 ceph-mon[34341]: 4.3 scrub starts 2024-03-30T13:48:17.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:16 smithi113 ceph-mon[34341]: 4.3 scrub ok 2024-03-30T13:48:17.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:16 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:48:17.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:16 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:48:17.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:16 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:48:17.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:16 smithi113 ceph-mon[34341]: 5.8 scrub starts 2024-03-30T13:48:17.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:16 smithi113 ceph-mon[34341]: 5.8 scrub ok 2024-03-30T13:48:17.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[29963]: 3.e scrub starts 2024-03-30T13:48:17.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[29963]: 3.e scrub ok 2024-03-30T13:48:17.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[29963]: pgmap v203: 148 pgs: 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-30T13:48:17.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[29963]: 5.16 scrub starts 2024-03-30T13:48:17.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[29963]: 5.16 scrub ok 2024-03-30T13:48:17.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[29963]: 4.3 scrub starts 2024-03-30T13:48:17.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[29963]: 4.3 scrub ok 2024-03-30T13:48:17.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:48:17.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:48:17.255 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:48:17.255 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[29963]: 5.8 scrub starts 2024-03-30T13:48:17.255 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[29963]: 5.8 scrub ok 2024-03-30T13:48:17.255 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[42099]: 3.e scrub starts 2024-03-30T13:48:17.255 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[42099]: 3.e scrub ok 2024-03-30T13:48:17.255 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[42099]: pgmap v203: 148 pgs: 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-30T13:48:17.255 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[42099]: 5.16 scrub starts 2024-03-30T13:48:17.255 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[42099]: 5.16 scrub ok 2024-03-30T13:48:17.255 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[42099]: 4.3 scrub starts 2024-03-30T13:48:17.256 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[42099]: 4.3 scrub ok 2024-03-30T13:48:17.256 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:48:17.256 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:48:17.256 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:48:17.256 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[42099]: 5.8 scrub starts 2024-03-30T13:48:17.256 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:16 smithi043 ceph-mon[42099]: 5.8 scrub ok 2024-03-30T13:48:17.315 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:17.368 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:17.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:17.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:17.470 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:17.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:17.517 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:17.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:17.618 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:17.662 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:17.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:17.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:17.764 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:17.792 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:17.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:17.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:17.895 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:17.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:17.925 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:17.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:18.026 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:18.041 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:17 smithi043 ceph-mon[29963]: 6.16 scrub starts 2024-03-30T13:48:18.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:17 smithi043 ceph-mon[29963]: 6.16 scrub ok 2024-03-30T13:48:18.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:17 smithi043 ceph-mon[42099]: 6.16 scrub starts 2024-03-30T13:48:18.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:17 smithi043 ceph-mon[42099]: 6.16 scrub ok 2024-03-30T13:48:18.056 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:18.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:18.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:18.158 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:18.186 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:18.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:18.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:18.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:17 smithi113 ceph-mon[34341]: 6.16 scrub starts 2024-03-30T13:48:18.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:17 smithi113 ceph-mon[34341]: 6.16 scrub ok 2024-03-30T13:48:18.289 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:18.318 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:18.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:18.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:18.420 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:18.450 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:18.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:18.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:18.552 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:18.580 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:18.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:18.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:18.681 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:18.707 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:18.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:18.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:18.809 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:18.847 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:18.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:18.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:18.923 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:48:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:48:18.950 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:18.991 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:18.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:18.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:19.092 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:19.107 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:18 smithi043 ceph-mon[29963]: pgmap v204: 148 pgs: 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:48:19.107 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:18 smithi043 ceph-mon[29963]: 7.8 scrub starts 2024-03-30T13:48:19.107 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:18 smithi043 ceph-mon[42099]: pgmap v204: 148 pgs: 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:48:19.107 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:18 smithi043 ceph-mon[42099]: 7.8 scrub starts 2024-03-30T13:48:19.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:19.135 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:19.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:19.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:18 smithi113 ceph-mon[34341]: pgmap v204: 148 pgs: 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:48:19.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:18 smithi113 ceph-mon[34341]: 7.8 scrub starts 2024-03-30T13:48:19.236 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:19.266 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:19.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:19.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:19.367 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:19.399 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:19.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:19.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:19.501 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:19.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:19.538 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:19.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:19.639 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:19.673 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:19.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:19.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:19.774 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:19.813 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:19.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:19.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:19.915 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:19.954 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:19.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:19.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:20.056 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:20.086 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:20.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:20.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:20.188 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:20.221 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:20.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:20.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:20.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:19 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:48:20.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:19 smithi113 ceph-mon[34341]: 7.8 scrub ok 2024-03-30T13:48:20.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:19 smithi113 ceph-mon[34341]: 2.1 scrub starts 2024-03-30T13:48:20.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:19 smithi113 ceph-mon[34341]: 2.1 scrub ok 2024-03-30T13:48:20.222 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T13:48:20.223 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:48:20.223 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-03-30T13:48:20.223 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 reset_purged_snaps_last 2024-03-30T13:48:20.263 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:19 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:48:20.263 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:19 smithi043 ceph-mon[29963]: 7.8 scrub ok 2024-03-30T13:48:20.263 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:19 smithi043 ceph-mon[29963]: 2.1 scrub starts 2024-03-30T13:48:20.263 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:19 smithi043 ceph-mon[29963]: 2.1 scrub ok 2024-03-30T13:48:20.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:19 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:48:20.264 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:19 smithi043 ceph-mon[42099]: 7.8 scrub ok 2024-03-30T13:48:20.264 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:19 smithi043 ceph-mon[42099]: 2.1 scrub starts 2024-03-30T13:48:20.264 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:19 smithi043 ceph-mon[42099]: 2.1 scrub ok 2024-03-30T13:48:20.323 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:20.354 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:20.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:20.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:20.455 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:20.482 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:20.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:20.584 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:20.614 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:20.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:20.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:20.716 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:20.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:20.748 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:20.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:20.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:20.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:20.894 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:20.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:20.919 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:48:20.919 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:48:20.995 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:21.021 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:21.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:21.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:21.123 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:21.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:20 smithi043 ceph-mon[42099]: 4.7 scrub starts 2024-03-30T13:48:21.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:20 smithi043 ceph-mon[42099]: 4.7 scrub ok 2024-03-30T13:48:21.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:20 smithi043 ceph-mon[42099]: pgmap v205: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-30T13:48:21.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:20 smithi043 ceph-mon[42099]: 6.d scrub starts 2024-03-30T13:48:21.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:20 smithi043 ceph-mon[42099]: 6.d scrub ok 2024-03-30T13:48:21.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:20 smithi043 ceph-mon[42099]: 3.1f scrub starts 2024-03-30T13:48:21.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:20 smithi043 ceph-mon[42099]: 3.1f scrub ok 2024-03-30T13:48:21.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:20 smithi043 ceph-mon[42099]: 3.f scrub starts 2024-03-30T13:48:21.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:20 smithi043 ceph-mon[42099]: 3.f scrub ok 2024-03-30T13:48:21.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:20 smithi043 ceph-mon[29963]: 4.7 scrub starts 2024-03-30T13:48:21.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:20 smithi043 ceph-mon[29963]: 4.7 scrub ok 2024-03-30T13:48:21.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:20 smithi043 ceph-mon[29963]: pgmap v205: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-30T13:48:21.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:20 smithi043 ceph-mon[29963]: 6.d scrub starts 2024-03-30T13:48:21.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:20 smithi043 ceph-mon[29963]: 6.d scrub ok 2024-03-30T13:48:21.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:20 smithi043 ceph-mon[29963]: 3.1f scrub starts 2024-03-30T13:48:21.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:20 smithi043 ceph-mon[29963]: 3.1f scrub ok 2024-03-30T13:48:21.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:20 smithi043 ceph-mon[29963]: 3.f scrub starts 2024-03-30T13:48:21.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:20 smithi043 ceph-mon[29963]: 3.f scrub ok 2024-03-30T13:48:21.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:21.153 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:21.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:21.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:20 smithi113 ceph-mon[34341]: 4.7 scrub starts 2024-03-30T13:48:21.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:20 smithi113 ceph-mon[34341]: 4.7 scrub ok 2024-03-30T13:48:21.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:20 smithi113 ceph-mon[34341]: pgmap v205: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-30T13:48:21.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:20 smithi113 ceph-mon[34341]: 6.d scrub starts 2024-03-30T13:48:21.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:20 smithi113 ceph-mon[34341]: 6.d scrub ok 2024-03-30T13:48:21.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:20 smithi113 ceph-mon[34341]: 3.1f scrub starts 2024-03-30T13:48:21.224 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:20 smithi113 ceph-mon[34341]: 3.1f scrub ok 2024-03-30T13:48:21.224 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:20 smithi113 ceph-mon[34341]: 3.f scrub starts 2024-03-30T13:48:21.224 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:20 smithi113 ceph-mon[34341]: 3.f scrub ok 2024-03-30T13:48:21.255 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:21.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:21.295 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:21.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:21.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:21.444 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:21.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:21.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:21.546 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:21.574 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:21.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:21.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:21.676 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:21.728 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:21.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:21.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:21.829 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:21.865 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:21.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:21.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:21.963 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-03-30T13:48:21.967 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:21.996 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:21.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:21.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:22.098 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:22.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:22.139 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:22.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:22.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:21 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2511524231' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:48:22.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:21 smithi043 ceph-mon[29963]: pgmap v206: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 837 op/s 2024-03-30T13:48:22.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:21 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2511524231' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:48:22.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:21 smithi043 ceph-mon[42099]: pgmap v206: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 837 op/s 2024-03-30T13:48:22.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:21 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2511524231' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:48:22.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:21 smithi113 ceph-mon[34341]: pgmap v206: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 837 op/s 2024-03-30T13:48:22.241 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:22.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:22.283 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:22.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:22.384 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:22.413 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:22.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:22.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:22.424 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-03-30T13:48:22.515 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:22.543 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:22.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:22.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:22.645 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:22.671 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:22.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:22.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:22.773 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:22.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:22.804 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:22.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:22.905 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:22.946 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:22.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:22.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:23.048 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:23.076 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:23.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:23.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:23.178 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:23.205 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[42099]: 4.14 scrub starts 2024-03-30T13:48:23.205 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[42099]: 4.14 scrub ok 2024-03-30T13:48:23.205 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[42099]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:48:23.205 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[42099]: Cluster is now healthy 2024-03-30T13:48:23.206 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2511524231' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:48:23.206 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[42099]: osdmap e138: 8 total, 8 up, 8 in 2024-03-30T13:48:23.206 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[42099]: 4.13 scrub starts 2024-03-30T13:48:23.206 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[42099]: 4.13 scrub ok 2024-03-30T13:48:23.206 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:48:23.206 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:48:23.206 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[42099]: 6.15 scrub starts 2024-03-30T13:48:23.207 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[42099]: 6.15 scrub ok 2024-03-30T13:48:23.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[29963]: 4.14 scrub starts 2024-03-30T13:48:23.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[29963]: 4.14 scrub ok 2024-03-30T13:48:23.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[29963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:48:23.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[29963]: Cluster is now healthy 2024-03-30T13:48:23.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2511524231' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:48:23.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[29963]: osdmap e138: 8 total, 8 up, 8 in 2024-03-30T13:48:23.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[29963]: 4.13 scrub starts 2024-03-30T13:48:23.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[29963]: 4.13 scrub ok 2024-03-30T13:48:23.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:48:23.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:48:23.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[29963]: 6.15 scrub starts 2024-03-30T13:48:23.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:22 smithi043 ceph-mon[29963]: 6.15 scrub ok 2024-03-30T13:48:23.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:23.209 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:23.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:23.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:22 smithi113 ceph-mon[34341]: 4.14 scrub starts 2024-03-30T13:48:23.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:22 smithi113 ceph-mon[34341]: 4.14 scrub ok 2024-03-30T13:48:23.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:22 smithi113 ceph-mon[34341]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:48:23.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:22 smithi113 ceph-mon[34341]: Cluster is now healthy 2024-03-30T13:48:23.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:22 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2511524231' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:48:23.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:22 smithi113 ceph-mon[34341]: osdmap e138: 8 total, 8 up, 8 in 2024-03-30T13:48:23.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:22 smithi113 ceph-mon[34341]: 4.13 scrub starts 2024-03-30T13:48:23.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:22 smithi113 ceph-mon[34341]: 4.13 scrub ok 2024-03-30T13:48:23.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:22 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:48:23.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:22 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:48:23.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:22 smithi113 ceph-mon[34341]: 6.15 scrub starts 2024-03-30T13:48:23.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:22 smithi113 ceph-mon[34341]: 6.15 scrub ok 2024-03-30T13:48:23.311 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:23.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:23.343 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:23.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:23.444 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:23.470 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:23.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:23.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:23.535 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:48:23.572 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:23.591 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:48:23.592 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:48:23.597 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:23.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:23.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:23.636 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:48:23.680 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:48:23.699 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:23.727 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:48:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:23.751 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:23.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:23.764 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:48:23.808 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:48:23.848 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:48:23.851 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:23.879 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:23.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:23.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:23.892 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:48:23.919 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:48:23.920 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:48:23.920 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:48:23.920 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:48:23.920 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:48:23.920 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:48:23.920 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:48:23.920 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:48:23.969 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:23 smithi043 ceph-mon[29963]: 3.8 scrub starts 2024-03-30T13:48:23.969 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:23 smithi043 ceph-mon[29963]: 3.8 scrub ok 2024-03-30T13:48:23.970 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:23 smithi043 ceph-mon[29963]: osdmap e139: 8 total, 8 up, 8 in 2024-03-30T13:48:23.970 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:23 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:48:23.970 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:23 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:48:23.970 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:23 smithi043 ceph-mon[29963]: 6.12 scrub starts 2024-03-30T13:48:23.970 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:23 smithi043 ceph-mon[29963]: 6.12 scrub ok 2024-03-30T13:48:23.970 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:23 smithi043 ceph-mon[29963]: pgmap v209: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 667 op/s 2024-03-30T13:48:23.970 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:23 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1786841006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:48:23.970 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:23 smithi043 ceph-mon[42099]: 3.8 scrub starts 2024-03-30T13:48:23.970 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:23 smithi043 ceph-mon[42099]: 3.8 scrub ok 2024-03-30T13:48:23.970 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:23 smithi043 ceph-mon[42099]: osdmap e139: 8 total, 8 up, 8 in 2024-03-30T13:48:23.971 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:23 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:48:23.971 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:23 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:48:23.971 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:23 smithi043 ceph-mon[42099]: 6.12 scrub starts 2024-03-30T13:48:23.971 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:23 smithi043 ceph-mon[42099]: 6.12 scrub ok 2024-03-30T13:48:23.971 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:23 smithi043 ceph-mon[42099]: pgmap v209: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 667 op/s 2024-03-30T13:48:23.971 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:23 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1786841006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:48:23.980 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:24.025 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:24.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:24.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:24.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:24.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:24.168 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:24.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:24.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:23 smithi113 ceph-mon[34341]: 3.8 scrub starts 2024-03-30T13:48:24.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:23 smithi113 ceph-mon[34341]: 3.8 scrub ok 2024-03-30T13:48:24.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:23 smithi113 ceph-mon[34341]: osdmap e139: 8 total, 8 up, 8 in 2024-03-30T13:48:24.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:23 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:48:24.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:23 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:48:24.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:23 smithi113 ceph-mon[34341]: 6.12 scrub starts 2024-03-30T13:48:24.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:23 smithi113 ceph-mon[34341]: 6.12 scrub ok 2024-03-30T13:48:24.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:23 smithi113 ceph-mon[34341]: pgmap v209: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 667 op/s 2024-03-30T13:48:24.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:23 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1786841006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:48:24.270 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:24.293 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:24.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:24.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:24.394 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:24.432 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:24.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:24.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:24.534 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:24.576 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:24.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:24.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:24.678 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:24.707 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:24.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:24.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:24.809 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:24.839 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:24.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:24.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:24.942 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:24.988 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:24.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:24.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:25.090 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:25.117 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:25.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:25.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:25.219 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:25.253 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:25.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T13:48:25.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:25.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:25.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[29963]: 5.c scrub starts 2024-03-30T13:48:25.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[29963]: 5.c scrub ok 2024-03-30T13:48:25.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[29963]: 5.15 scrub starts 2024-03-30T13:48:25.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[29963]: 5.15 scrub ok 2024-03-30T13:48:25.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[29963]: 6.c scrub starts 2024-03-30T13:48:25.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[29963]: 6.c scrub ok 2024-03-30T13:48:25.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[29963]: 6.10 scrub starts 2024-03-30T13:48:25.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[29963]: 6.10 scrub ok 2024-03-30T13:48:25.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[29963]: osdmap e140: 8 total, 8 up, 8 in 2024-03-30T13:48:25.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[29963]: 6.8 scrub starts 2024-03-30T13:48:25.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[29963]: 6.8 scrub ok 2024-03-30T13:48:25.287 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[42099]: 5.c scrub starts 2024-03-30T13:48:25.287 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[42099]: 5.c scrub ok 2024-03-30T13:48:25.287 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[42099]: 5.15 scrub starts 2024-03-30T13:48:25.287 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[42099]: 5.15 scrub ok 2024-03-30T13:48:25.287 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[42099]: 6.c scrub starts 2024-03-30T13:48:25.287 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[42099]: 6.c scrub ok 2024-03-30T13:48:25.287 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[42099]: 6.10 scrub starts 2024-03-30T13:48:25.287 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[42099]: 6.10 scrub ok 2024-03-30T13:48:25.287 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[42099]: osdmap e140: 8 total, 8 up, 8 in 2024-03-30T13:48:25.287 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[42099]: 6.8 scrub starts 2024-03-30T13:48:25.287 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:24 smithi043 ceph-mon[42099]: 6.8 scrub ok 2024-03-30T13:48:25.356 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:25.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:25.381 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:25.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:25.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:24 smithi113 ceph-mon[34341]: 5.c scrub starts 2024-03-30T13:48:25.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:24 smithi113 ceph-mon[34341]: 5.c scrub ok 2024-03-30T13:48:25.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:24 smithi113 ceph-mon[34341]: 5.15 scrub starts 2024-03-30T13:48:25.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:24 smithi113 ceph-mon[34341]: 5.15 scrub ok 2024-03-30T13:48:25.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:24 smithi113 ceph-mon[34341]: 6.c scrub starts 2024-03-30T13:48:25.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:24 smithi113 ceph-mon[34341]: 6.c scrub ok 2024-03-30T13:48:25.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:24 smithi113 ceph-mon[34341]: 6.10 scrub starts 2024-03-30T13:48:25.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:24 smithi113 ceph-mon[34341]: 6.10 scrub ok 2024-03-30T13:48:25.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:24 smithi113 ceph-mon[34341]: osdmap e140: 8 total, 8 up, 8 in 2024-03-30T13:48:25.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:24 smithi113 ceph-mon[34341]: 6.8 scrub starts 2024-03-30T13:48:25.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:24 smithi113 ceph-mon[34341]: 6.8 scrub ok 2024-03-30T13:48:25.482 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:25.511 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:25.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:25.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:25.613 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:25.656 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:25.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:25.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:25.758 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:25.787 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:25.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:25.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:25.889 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:25.939 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:25.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:25.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:26.041 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:26.080 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:26.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:26.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:26.101 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[29963]: 6.1d scrub starts 2024-03-30T13:48:26.101 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[29963]: 6.1d scrub ok 2024-03-30T13:48:26.101 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[29963]: 7.9 scrub starts 2024-03-30T13:48:26.101 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[29963]: 4.5 scrub starts 2024-03-30T13:48:26.101 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[29963]: 4.5 scrub ok 2024-03-30T13:48:26.101 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[29963]: 5.1c scrub starts 2024-03-30T13:48:26.101 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[29963]: 5.1c scrub ok 2024-03-30T13:48:26.101 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[29963]: 7.9 scrub ok 2024-03-30T13:48:26.101 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[29963]: 6.1c scrub starts 2024-03-30T13:48:26.102 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[29963]: 6.1c scrub ok 2024-03-30T13:48:26.102 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[29963]: pgmap v211: 148 pgs: 148 active+clean; 918 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 863 op/s 2024-03-30T13:48:26.102 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[42099]: 6.1d scrub starts 2024-03-30T13:48:26.102 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[42099]: 6.1d scrub ok 2024-03-30T13:48:26.102 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[42099]: 7.9 scrub starts 2024-03-30T13:48:26.102 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[42099]: 4.5 scrub starts 2024-03-30T13:48:26.102 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[42099]: 4.5 scrub ok 2024-03-30T13:48:26.102 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[42099]: 5.1c scrub starts 2024-03-30T13:48:26.102 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[42099]: 5.1c scrub ok 2024-03-30T13:48:26.102 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[42099]: 7.9 scrub ok 2024-03-30T13:48:26.102 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[42099]: 6.1c scrub starts 2024-03-30T13:48:26.102 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[42099]: 6.1c scrub ok 2024-03-30T13:48:26.103 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:25 smithi043 ceph-mon[42099]: pgmap v211: 148 pgs: 148 active+clean; 918 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 863 op/s 2024-03-30T13:48:26.182 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:26.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:26.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:26.324 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:26.361 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:26.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:26.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:26.401 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:48:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:48:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:48:26.463 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:26.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:25 smithi113 ceph-mon[34341]: 6.1d scrub starts 2024-03-30T13:48:26.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:25 smithi113 ceph-mon[34341]: 6.1d scrub ok 2024-03-30T13:48:26.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:25 smithi113 ceph-mon[34341]: 7.9 scrub starts 2024-03-30T13:48:26.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:25 smithi113 ceph-mon[34341]: 4.5 scrub starts 2024-03-30T13:48:26.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:25 smithi113 ceph-mon[34341]: 4.5 scrub ok 2024-03-30T13:48:26.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:25 smithi113 ceph-mon[34341]: 5.1c scrub starts 2024-03-30T13:48:26.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:25 smithi113 ceph-mon[34341]: 5.1c scrub ok 2024-03-30T13:48:26.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:25 smithi113 ceph-mon[34341]: 7.9 scrub ok 2024-03-30T13:48:26.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:25 smithi113 ceph-mon[34341]: 6.1c scrub starts 2024-03-30T13:48:26.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:25 smithi113 ceph-mon[34341]: 6.1c scrub ok 2024-03-30T13:48:26.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:25 smithi113 ceph-mon[34341]: pgmap v211: 148 pgs: 148 active+clean; 918 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 863 op/s 2024-03-30T13:48:26.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:26.505 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:26.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:26.606 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:26.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:26.639 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:26.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:26.739 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:26.770 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:26.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:26.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:26.872 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:26.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:26.916 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:26.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:27.017 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:27.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:27.063 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:27.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:27.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:26 smithi043 ceph-mon[29963]: 3.16 deep-scrub starts 2024-03-30T13:48:27.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:26 smithi043 ceph-mon[29963]: 3.16 deep-scrub ok 2024-03-30T13:48:27.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:26 smithi043 ceph-mon[29963]: 6.1f scrub starts 2024-03-30T13:48:27.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:26 smithi043 ceph-mon[29963]: 6.1f scrub ok 2024-03-30T13:48:27.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:26 smithi043 ceph-mon[42099]: 3.16 deep-scrub starts 2024-03-30T13:48:27.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:26 smithi043 ceph-mon[42099]: 3.16 deep-scrub ok 2024-03-30T13:48:27.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:26 smithi043 ceph-mon[42099]: 6.1f scrub starts 2024-03-30T13:48:27.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:26 smithi043 ceph-mon[42099]: 6.1f scrub ok 2024-03-30T13:48:27.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:27.205 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:27.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:27.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:27.307 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:27.337 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:27.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:27.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:27.439 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:27.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:26 smithi113 ceph-mon[34341]: 3.16 deep-scrub starts 2024-03-30T13:48:27.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:26 smithi113 ceph-mon[34341]: 3.16 deep-scrub ok 2024-03-30T13:48:27.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:26 smithi113 ceph-mon[34341]: 6.1f scrub starts 2024-03-30T13:48:27.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:26 smithi113 ceph-mon[34341]: 6.1f scrub ok 2024-03-30T13:48:27.481 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:27.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:27.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:27.583 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:27.625 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:27.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:27.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:27.727 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:27.757 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:27.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:27.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:27.859 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:27.886 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:27.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:27.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:27.989 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:28.019 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:28.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:28.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:28.070 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:27 smithi043 ceph-mon[29963]: 6.1b scrub starts 2024-03-30T13:48:28.070 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:27 smithi043 ceph-mon[29963]: 6.1b scrub ok 2024-03-30T13:48:28.070 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:27 smithi043 ceph-mon[29963]: 4.d scrub starts 2024-03-30T13:48:28.070 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:27 smithi043 ceph-mon[29963]: 4.d scrub ok 2024-03-30T13:48:28.070 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:27 smithi043 ceph-mon[29963]: pgmap v212: 148 pgs: 148 active+clean; 918 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 863 op/s 2024-03-30T13:48:28.070 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:48:28.070 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:48:28.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:27 smithi043 ceph-mon[42099]: 6.1b scrub starts 2024-03-30T13:48:28.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:27 smithi043 ceph-mon[42099]: 6.1b scrub ok 2024-03-30T13:48:28.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:27 smithi043 ceph-mon[42099]: 4.d scrub starts 2024-03-30T13:48:28.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:27 smithi043 ceph-mon[42099]: 4.d scrub ok 2024-03-30T13:48:28.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:27 smithi043 ceph-mon[42099]: pgmap v212: 148 pgs: 148 active+clean; 918 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 863 op/s 2024-03-30T13:48:28.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:48:28.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:48:28.072 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:48:28.121 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:28.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:28.156 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:28.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:28.257 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:28.288 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:28.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:28.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:28.315 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:48:28.389 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:28.417 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:28.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:28.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:28.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:27 smithi113 ceph-mon[34341]: 6.1b scrub starts 2024-03-30T13:48:28.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:27 smithi113 ceph-mon[34341]: 6.1b scrub ok 2024-03-30T13:48:28.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:27 smithi113 ceph-mon[34341]: 4.d scrub starts 2024-03-30T13:48:28.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:27 smithi113 ceph-mon[34341]: 4.d scrub ok 2024-03-30T13:48:28.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:27 smithi113 ceph-mon[34341]: pgmap v212: 148 pgs: 148 active+clean; 918 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 863 op/s 2024-03-30T13:48:28.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:48:28.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:48:28.519 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:28.571 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:28.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:28.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:28.655 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:48:28.655 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T13:48:28.655 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-03-30T13:48:28.655 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-03-30T13:48:28.655 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:48:28.673 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:28.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:28.722 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:28.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:28.823 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:28.873 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:28.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:28.972 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:48:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:48:28.974 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:29.017 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:29.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:29.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:29.120 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:29.153 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:29.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:29.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:29.229 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:29 smithi043 ceph-mon[29963]: 4.4 scrub starts 2024-03-30T13:48:29.229 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:29 smithi043 ceph-mon[29963]: 4.4 scrub ok 2024-03-30T13:48:29.229 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:29 smithi043 ceph-mon[29963]: 6.7 scrub starts 2024-03-30T13:48:29.229 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:29 smithi043 ceph-mon[29963]: 6.7 scrub ok 2024-03-30T13:48:29.229 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:29 smithi043 ceph-mon[29963]: 4.c scrub starts 2024-03-30T13:48:29.229 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:29 smithi043 ceph-mon[29963]: 4.c scrub ok 2024-03-30T13:48:29.230 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:29 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:48:29.230 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:29 smithi043 ceph-mon[42099]: 4.4 scrub starts 2024-03-30T13:48:29.230 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:29 smithi043 ceph-mon[42099]: 4.4 scrub ok 2024-03-30T13:48:29.230 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:29 smithi043 ceph-mon[42099]: 6.7 scrub starts 2024-03-30T13:48:29.230 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:29 smithi043 ceph-mon[42099]: 6.7 scrub ok 2024-03-30T13:48:29.230 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:29 smithi043 ceph-mon[42099]: 4.c scrub starts 2024-03-30T13:48:29.230 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:29 smithi043 ceph-mon[42099]: 4.c scrub ok 2024-03-30T13:48:29.230 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:29 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:48:29.256 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:29.296 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:29.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:29.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:29.398 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:29.428 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:29.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:29.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:29.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:28 smithi113 ceph-mon[34341]: 4.4 scrub starts 2024-03-30T13:48:29.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:28 smithi113 ceph-mon[34341]: 4.4 scrub ok 2024-03-30T13:48:29.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:28 smithi113 ceph-mon[34341]: 6.7 scrub starts 2024-03-30T13:48:29.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:29 smithi113 ceph-mon[34341]: 6.7 scrub ok 2024-03-30T13:48:29.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:29 smithi113 ceph-mon[34341]: 4.c scrub starts 2024-03-30T13:48:29.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:29 smithi113 ceph-mon[34341]: 4.c scrub ok 2024-03-30T13:48:29.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:29 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:48:29.531 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:29.569 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:29.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:29.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:29.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:29.719 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 reset_purged_snaps_last 2024-03-30T13:48:29.731 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:29.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:29.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:29.833 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:29.863 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:29.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:29.965 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:30.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:30.033 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:30.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:30.134 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:30.194 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:30.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:30.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:30.244 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:30 smithi043 ceph-mon[29963]: 7.b deep-scrub starts 2024-03-30T13:48:30.244 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:30 smithi043 ceph-mon[29963]: 6.a scrub starts 2024-03-30T13:48:30.244 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:30 smithi043 ceph-mon[29963]: 6.a scrub ok 2024-03-30T13:48:30.244 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:30 smithi043 ceph-mon[29963]: 3.12 deep-scrub starts 2024-03-30T13:48:30.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:30 smithi043 ceph-mon[29963]: 3.12 deep-scrub ok 2024-03-30T13:48:30.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:30 smithi043 ceph-mon[29963]: pgmap v213: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 599 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-03-30T13:48:30.245 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:30 smithi043 ceph-mon[42099]: 7.b deep-scrub starts 2024-03-30T13:48:30.245 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:30 smithi043 ceph-mon[42099]: 6.a scrub starts 2024-03-30T13:48:30.245 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:30 smithi043 ceph-mon[42099]: 6.a scrub ok 2024-03-30T13:48:30.245 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:30 smithi043 ceph-mon[42099]: 3.12 deep-scrub starts 2024-03-30T13:48:30.245 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:30 smithi043 ceph-mon[42099]: 3.12 deep-scrub ok 2024-03-30T13:48:30.245 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:30 smithi043 ceph-mon[42099]: pgmap v213: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 599 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-03-30T13:48:30.297 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:30.330 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:30.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:30.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:30.432 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:30.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:30 smithi113 ceph-mon[34341]: 7.b deep-scrub starts 2024-03-30T13:48:30.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:30 smithi113 ceph-mon[34341]: 6.a scrub starts 2024-03-30T13:48:30.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:30 smithi113 ceph-mon[34341]: 6.a scrub ok 2024-03-30T13:48:30.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:30 smithi113 ceph-mon[34341]: 3.12 deep-scrub starts 2024-03-30T13:48:30.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:30 smithi113 ceph-mon[34341]: 3.12 deep-scrub ok 2024-03-30T13:48:30.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:30 smithi113 ceph-mon[34341]: pgmap v213: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 599 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-03-30T13:48:30.476 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:30.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:30.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:30.578 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:30.619 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:30.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:30.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:30.721 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:30.762 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:30.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:30.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:30.864 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:30.900 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:30.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:30.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:31.002 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:31.029 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:31.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:31.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:31.131 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:31.159 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:31.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:31.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:31.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:31 smithi113 ceph-mon[34341]: 3.15 deep-scrub starts 2024-03-30T13:48:31.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:31 smithi113 ceph-mon[34341]: 3.15 deep-scrub ok 2024-03-30T13:48:31.261 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:31.299 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:31.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:31.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:31.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:31 smithi043 ceph-mon[29963]: 3.15 deep-scrub starts 2024-03-30T13:48:31.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:31 smithi043 ceph-mon[29963]: 3.15 deep-scrub ok 2024-03-30T13:48:31.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:31 smithi043 ceph-mon[42099]: 3.15 deep-scrub starts 2024-03-30T13:48:31.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:31 smithi043 ceph-mon[42099]: 3.15 deep-scrub ok 2024-03-30T13:48:31.401 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:31.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:31.445 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:31.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:31.546 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:31.586 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:31.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:31.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:31.688 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:31.716 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:31.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:31.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:31.818 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:31.847 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:31.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:31.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:31.949 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:31.976 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:31.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:31.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:32.078 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:32.108 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:32.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:32.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:32.210 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:32.238 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:32.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:32.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:32.340 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:32.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:32.387 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:32.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:32.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[29963]: 4.f scrub starts 2024-03-30T13:48:32.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[29963]: 5.1b scrub starts 2024-03-30T13:48:32.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[29963]: 5.1b scrub ok 2024-03-30T13:48:32.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[29963]: 4.f scrub ok 2024-03-30T13:48:32.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[29963]: 5.17 scrub starts 2024-03-30T13:48:32.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[29963]: 5.17 scrub ok 2024-03-30T13:48:32.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[29963]: 6.18 deep-scrub starts 2024-03-30T13:48:32.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[29963]: 6.18 deep-scrub ok 2024-03-30T13:48:32.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[29963]: 3.10 scrub starts 2024-03-30T13:48:32.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[29963]: 3.10 scrub ok 2024-03-30T13:48:32.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[29963]: 7.e scrub starts 2024-03-30T13:48:32.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[29963]: pgmap v214: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 599 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-30T13:48:32.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[29963]: 7.e scrub ok 2024-03-30T13:48:32.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[42099]: 4.f scrub starts 2024-03-30T13:48:32.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[42099]: 5.1b scrub starts 2024-03-30T13:48:32.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[42099]: 5.1b scrub ok 2024-03-30T13:48:32.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[42099]: 4.f scrub ok 2024-03-30T13:48:32.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[42099]: 5.17 scrub starts 2024-03-30T13:48:32.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[42099]: 5.17 scrub ok 2024-03-30T13:48:32.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[42099]: 6.18 deep-scrub starts 2024-03-30T13:48:32.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[42099]: 6.18 deep-scrub ok 2024-03-30T13:48:32.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[42099]: 3.10 scrub starts 2024-03-30T13:48:32.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[42099]: 3.10 scrub ok 2024-03-30T13:48:32.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[42099]: 7.e scrub starts 2024-03-30T13:48:32.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[42099]: pgmap v214: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 599 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-30T13:48:32.402 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:32 smithi043 ceph-mon[42099]: 7.e scrub ok 2024-03-30T13:48:32.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:32 smithi113 ceph-mon[34341]: 4.f scrub starts 2024-03-30T13:48:32.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:32 smithi113 ceph-mon[34341]: 5.1b scrub starts 2024-03-30T13:48:32.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:32 smithi113 ceph-mon[34341]: 5.1b scrub ok 2024-03-30T13:48:32.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:32 smithi113 ceph-mon[34341]: 4.f scrub ok 2024-03-30T13:48:32.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:32 smithi113 ceph-mon[34341]: 5.17 scrub starts 2024-03-30T13:48:32.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:32 smithi113 ceph-mon[34341]: 5.17 scrub ok 2024-03-30T13:48:32.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:32 smithi113 ceph-mon[34341]: 6.18 deep-scrub starts 2024-03-30T13:48:32.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:32 smithi113 ceph-mon[34341]: 6.18 deep-scrub ok 2024-03-30T13:48:32.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:32 smithi113 ceph-mon[34341]: 3.10 scrub starts 2024-03-30T13:48:32.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:32 smithi113 ceph-mon[34341]: 3.10 scrub ok 2024-03-30T13:48:32.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:32 smithi113 ceph-mon[34341]: 7.e scrub starts 2024-03-30T13:48:32.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:32 smithi113 ceph-mon[34341]: pgmap v214: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 599 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-30T13:48:32.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:32 smithi113 ceph-mon[34341]: 7.e scrub ok 2024-03-30T13:48:32.488 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:32.517 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:32.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:32.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:32.619 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:32.657 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:32.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:32.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:32.748 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-03-30T13:48:32.759 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:32.789 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:32.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:32.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:32.792 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:48:32.891 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:32.961 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:32.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:32.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:33.067 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:33.093 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:33.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:33.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:33.195 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:33.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:33.240 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:33.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:33.342 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:33.370 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:33.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:33.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:33.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:33 smithi043 ceph-mon[29963]: 5.12 scrub starts 2024-03-30T13:48:33.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:33 smithi043 ceph-mon[29963]: 5.12 scrub ok 2024-03-30T13:48:33.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:33 smithi043 ceph-mon[29963]: 7.b deep-scrub ok 2024-03-30T13:48:33.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:33 smithi043 ceph-mon[29963]: 4.1a scrub starts 2024-03-30T13:48:33.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:33 smithi043 ceph-mon[29963]: 4.1a scrub ok 2024-03-30T13:48:33.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:33 smithi043 ceph-mon[42099]: 5.12 scrub starts 2024-03-30T13:48:33.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:33 smithi043 ceph-mon[42099]: 5.12 scrub ok 2024-03-30T13:48:33.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:33 smithi043 ceph-mon[42099]: 7.b deep-scrub ok 2024-03-30T13:48:33.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:33 smithi043 ceph-mon[42099]: 4.1a scrub starts 2024-03-30T13:48:33.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:33 smithi043 ceph-mon[42099]: 4.1a scrub ok 2024-03-30T13:48:33.470 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:48:33.472 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:33.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:33 smithi113 ceph-mon[34341]: 5.12 scrub starts 2024-03-30T13:48:33.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:33 smithi113 ceph-mon[34341]: 5.12 scrub ok 2024-03-30T13:48:33.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:33 smithi113 ceph-mon[34341]: 7.b deep-scrub ok 2024-03-30T13:48:33.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:33 smithi113 ceph-mon[34341]: 4.1a scrub starts 2024-03-30T13:48:33.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:33 smithi113 ceph-mon[34341]: 4.1a scrub ok 2024-03-30T13:48:33.498 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:33.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:33.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:33.600 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:33.626 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:33.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:33.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:33.727 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:33.754 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:33.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:33.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:33.856 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:33.904 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:33.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:33.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:34.006 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:34.033 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:34.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:34.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:34.135 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:34.191 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:34.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:34.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:34.257 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:34 smithi043 ceph-mon[29963]: 3.1b scrub starts 2024-03-30T13:48:34.257 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:34 smithi043 ceph-mon[29963]: 3.1b scrub ok 2024-03-30T13:48:34.257 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:34 smithi043 ceph-mon[29963]: pgmap v215: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 581 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:48:34.258 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:34 smithi043 ceph-mon[42099]: 3.1b scrub starts 2024-03-30T13:48:34.258 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:34 smithi043 ceph-mon[42099]: 3.1b scrub ok 2024-03-30T13:48:34.258 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:34 smithi043 ceph-mon[42099]: pgmap v215: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 581 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:48:34.295 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:34.341 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:34.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:34.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:34.443 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:34.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:34 smithi113 ceph-mon[34341]: 3.1b scrub starts 2024-03-30T13:48:34.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:34 smithi113 ceph-mon[34341]: 3.1b scrub ok 2024-03-30T13:48:34.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:34 smithi113 ceph-mon[34341]: pgmap v215: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 581 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-30T13:48:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:34.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:34.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:34.590 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:34.617 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:34.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:34.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:34.719 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:34.758 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:34.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:34.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:34.820 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:48:34.859 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:34.887 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:34.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:34.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:34.989 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:35.028 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:35.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:35.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:35.131 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:35.162 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:35.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:35.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:35.264 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:35.294 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:35.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:35.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:35.387 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:35 smithi043 ceph-mon[29963]: 6.13 scrub starts 2024-03-30T13:48:35.387 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:35 smithi043 ceph-mon[29963]: 6.13 scrub ok 2024-03-30T13:48:35.387 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:35 smithi043 ceph-mon[29963]: 6.1e scrub starts 2024-03-30T13:48:35.387 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:35 smithi043 ceph-mon[29963]: 6.1e scrub ok 2024-03-30T13:48:35.387 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:35 smithi043 ceph-mon[29963]: 5.10 scrub starts 2024-03-30T13:48:35.387 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:35 smithi043 ceph-mon[29963]: 5.10 scrub ok 2024-03-30T13:48:35.387 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:35 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/937416218' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:48:35.388 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:35 smithi043 ceph-mon[42099]: 6.13 scrub starts 2024-03-30T13:48:35.388 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:35 smithi043 ceph-mon[42099]: 6.13 scrub ok 2024-03-30T13:48:35.388 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:35 smithi043 ceph-mon[42099]: 6.1e scrub starts 2024-03-30T13:48:35.388 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:35 smithi043 ceph-mon[42099]: 6.1e scrub ok 2024-03-30T13:48:35.388 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:35 smithi043 ceph-mon[42099]: 5.10 scrub starts 2024-03-30T13:48:35.388 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:35 smithi043 ceph-mon[42099]: 5.10 scrub ok 2024-03-30T13:48:35.388 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:35 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/937416218' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:48:35.397 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:35.426 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:35.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:35.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:35.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:35 smithi113 ceph-mon[34341]: 6.13 scrub starts 2024-03-30T13:48:35.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:35 smithi113 ceph-mon[34341]: 6.13 scrub ok 2024-03-30T13:48:35.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:35 smithi113 ceph-mon[34341]: 6.1e scrub starts 2024-03-30T13:48:35.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:35 smithi113 ceph-mon[34341]: 6.1e scrub ok 2024-03-30T13:48:35.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:35 smithi113 ceph-mon[34341]: 5.10 scrub starts 2024-03-30T13:48:35.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:35 smithi113 ceph-mon[34341]: 5.10 scrub ok 2024-03-30T13:48:35.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:35 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/937416218' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:48:35.531 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:35.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:35.573 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:35.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:35.674 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:35.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:35.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:35.818 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:35.846 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:35.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:35.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:35.949 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:35.991 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:35.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:35.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:36.093 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:36.122 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:36.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:36.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:36.175 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-03-30T13:48:36.224 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:36.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:36.265 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:36.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:36.366 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:36.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:36 smithi043 ceph-mon[29963]: 7.4 scrub starts 2024-03-30T13:48:36.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:36 smithi043 ceph-mon[29963]: 5.11 scrub starts 2024-03-30T13:48:36.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:36 smithi043 ceph-mon[29963]: 5.11 scrub ok 2024-03-30T13:48:36.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:36 smithi043 ceph-mon[29963]: 7.4 scrub ok 2024-03-30T13:48:36.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:36 smithi043 ceph-mon[29963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:48:36.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:36 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/937416218' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:48:36.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:36 smithi043 ceph-mon[29963]: osdmap e141: 8 total, 8 up, 8 in 2024-03-30T13:48:36.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:36 smithi043 ceph-mon[29963]: pgmap v217: 148 pgs: 148 active+clean; 318 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 960 op/s 2024-03-30T13:48:36.401 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:48:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:48:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:48:36.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:36 smithi043 ceph-mon[42099]: 7.4 scrub starts 2024-03-30T13:48:36.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:36 smithi043 ceph-mon[42099]: 5.11 scrub starts 2024-03-30T13:48:36.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:36 smithi043 ceph-mon[42099]: 5.11 scrub ok 2024-03-30T13:48:36.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:36 smithi043 ceph-mon[42099]: 7.4 scrub ok 2024-03-30T13:48:36.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:36 smithi043 ceph-mon[42099]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:48:36.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:36 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/937416218' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:48:36.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:36 smithi043 ceph-mon[42099]: osdmap e141: 8 total, 8 up, 8 in 2024-03-30T13:48:36.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:36 smithi043 ceph-mon[42099]: pgmap v217: 148 pgs: 148 active+clean; 318 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 960 op/s 2024-03-30T13:48:36.411 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:36.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:36.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:36.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:36 smithi113 ceph-mon[34341]: 7.4 scrub starts 2024-03-30T13:48:36.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:36 smithi113 ceph-mon[34341]: 5.11 scrub starts 2024-03-30T13:48:36.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:36 smithi113 ceph-mon[34341]: 5.11 scrub ok 2024-03-30T13:48:36.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:36 smithi113 ceph-mon[34341]: 7.4 scrub ok 2024-03-30T13:48:36.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:36 smithi113 ceph-mon[34341]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:48:36.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:36 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/937416218' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:48:36.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:36 smithi113 ceph-mon[34341]: osdmap e141: 8 total, 8 up, 8 in 2024-03-30T13:48:36.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:36 smithi113 ceph-mon[34341]: pgmap v217: 148 pgs: 148 active+clean; 318 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 960 op/s 2024-03-30T13:48:36.512 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:36.555 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:36.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:36.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:36.657 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:36.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:36.686 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:36.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:36.787 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:36.827 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:36.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:36.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:36.929 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:36.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:36.959 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:36.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:37.060 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:37.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:37.107 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:37.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:37.209 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:37.241 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:37.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:37.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:37.343 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:37.375 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:37.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:37.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:37.477 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:37.506 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:37.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:37.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:37.608 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:37.638 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:37.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:37.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:37.740 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:37.768 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:37.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:37.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:37.870 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:37.901 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:37.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:37.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:38.003 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:38.043 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:38.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:38.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:38.145 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:38.183 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:38.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:38.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:38.285 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:38.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:38.316 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:38.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:38.418 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:38.458 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:38.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:38.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:38.559 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:38.576 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Removed 122335 objects 2024-03-30T13:48:38.576 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Clean up completed and total clean up time :128.157 2024-03-30T13:48:38.588 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:38.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:38.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:38.690 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:38.714 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:38.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:38.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:38.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:38 smithi113 ceph-mon[34341]: pgmap v218: 148 pgs: 148 active+clean; 318 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 960 op/s 2024-03-30T13:48:38.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:48:38 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:48:38.800 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T13:48:38.813 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:38 smithi043 ceph-mon[29963]: pgmap v218: 148 pgs: 148 active+clean; 318 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 960 op/s 2024-03-30T13:48:38.813 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:38 smithi043 ceph-mon[42099]: pgmap v218: 148 pgs: 148 active+clean; 318 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 960 op/s 2024-03-30T13:48:38.816 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:38.839 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:38.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:38.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:38.941 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:38.964 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:38.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:39.066 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:39.091 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:39.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:39.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:39.193 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:39.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:39.217 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:39.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:39.318 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:39.343 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:39.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:39.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:39.445 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:39.469 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:39.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:39.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:39.571 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:39.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:39.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:39.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:39 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:48:39.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:39 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:48:39.697 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:39.721 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:39.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:39.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:39.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:39 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:48:39.823 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:39.850 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:39.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:39.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:39.951 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:39.967 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-03-30T13:48:39.967 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-03-30T13:48:39.977 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:39.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:39.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:40.079 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:40.102 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:40.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:40.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:40.204 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:40.233 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:40.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:40.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:40.335 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:40.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:40.360 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:40.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:40.461 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:40.489 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:40.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:40.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:40.590 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:40.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:40 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:48:40.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:40 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:48:40.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:40 smithi043 ceph-mon[29963]: pgmap v219: 148 pgs: 148 active+clean; 5.3 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 950 op/s 2024-03-30T13:48:40.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:40 smithi043 ceph-mon[29963]: osdmap e142: 8 total, 8 up, 8 in 2024-03-30T13:48:40.610 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:40 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:48:40.610 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:40 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:48:40.610 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:40 smithi043 ceph-mon[42099]: pgmap v219: 148 pgs: 148 active+clean; 5.3 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 950 op/s 2024-03-30T13:48:40.610 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:40 smithi043 ceph-mon[42099]: osdmap e142: 8 total, 8 up, 8 in 2024-03-30T13:48:40.617 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:40.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:40.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:40.719 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:40.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:40 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:48:40.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:40 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:48:40.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:40 smithi113 ceph-mon[34341]: pgmap v219: 148 pgs: 148 active+clean; 5.3 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 950 op/s 2024-03-30T13:48:40.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:40 smithi113 ceph-mon[34341]: osdmap e142: 8 total, 8 up, 8 in 2024-03-30T13:48:40.742 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:40.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:40.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:40.824 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:48:40.843 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:48:40.844 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:40.863 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:48:40.872 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:40.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:40.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:40.883 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:48:40.903 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:48:40.925 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:48:40.946 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:48:40.966 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:48:40.968 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:48:40.968 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:48:40.968 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:48:40.968 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:48:40.968 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:48:40.968 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:48:40.968 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:48:40.968 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:48:40.974 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:41.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:41.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:41.034 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:48:41.034 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:48:41.104 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:41.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:41.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:41.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:41.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:41.256 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:41.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:41.357 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:41.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:41.483 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:41.506 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:41.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:41.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:41.607 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:41.635 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:41.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:41.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:41.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:41 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1539893083' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:48:41.737 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:41.761 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:41.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:41.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:41.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:41.887 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:41.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:41.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:41.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:41 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1539893083' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:48:41.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:41 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1539893083' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:48:41.988 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:42.013 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:42.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:42.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:42.114 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:42.138 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:42.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:42.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:42.240 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:42.263 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:42.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:42.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:42.365 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:42.393 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:42.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:42.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:42.495 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:42.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:42.524 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:42.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:42.625 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:42.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:42.654 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:42.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:42.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:42 smithi113 ceph-mon[34341]: pgmap v221: 148 pgs: 148 active+clean; 5.3 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-30T13:48:42.755 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:42.780 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:42.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:42.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:42.843 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:42 smithi043 ceph-mon[29963]: pgmap v221: 148 pgs: 148 active+clean; 5.3 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-30T13:48:42.844 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:42 smithi043 ceph-mon[42099]: pgmap v221: 148 pgs: 148 active+clean; 5.3 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-30T13:48:42.870 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 reset_purged_snaps_last 2024-03-30T13:48:42.882 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:42.896 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:48:42.925 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:42.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:42.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:43.027 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:43.052 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:43.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:43.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:43.154 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:43.177 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:43.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:43.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:43.279 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:43.306 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:43.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:43.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:43.408 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:43.432 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:43.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:43.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:43.534 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:43.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:43.559 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:43.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:43.660 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:43.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:43.689 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:43.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:43.729 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:48:43.790 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:43.818 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:43.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:43.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:43.919 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:43.943 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:43.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:43.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:44.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:43 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:48:44.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:43 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:48:44.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:43 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:48:44.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:43 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:48:44.045 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:44.067 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-30T13:48:44.073 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:44.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:44.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:44.175 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:44.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:44.203 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:44.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:44.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:43 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:48:44.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:43 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:48:44.304 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:44.330 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:44.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:44.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:44.432 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:44.458 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:44.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:44.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:44.523 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:48:44.560 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:44.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:44.586 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:44.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:44.687 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:44.712 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:44.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:44.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:44.814 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:44.840 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:44.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:44.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:44.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:44 smithi043 ceph-mon[29963]: pgmap v222: 148 pgs: 148 active+clean; 4.3 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 614 B/s rd, 0 B/s wr, 602 op/s 2024-03-30T13:48:44.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:44 smithi043 ceph-mon[42099]: pgmap v222: 148 pgs: 148 active+clean; 4.3 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 614 B/s rd, 0 B/s wr, 602 op/s 2024-03-30T13:48:44.942 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:44.968 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:44.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:44.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:45.070 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:45.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:45.095 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:45.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:45.196 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:45.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:44 smithi113 ceph-mon[34341]: pgmap v222: 148 pgs: 148 active+clean; 4.3 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 614 B/s rd, 0 B/s wr, 602 op/s 2024-03-30T13:48:45.223 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:45.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:45.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:45.325 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:45.348 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:45.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:45.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:45.448 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:48:45.450 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:45.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:45.478 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:45.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:45.579 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:45.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:45.604 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:45.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:45.705 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:45.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:45.729 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:45.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:45.830 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:45.854 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:45.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:45.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:45.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:45.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:45.980 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:45.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:46.082 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:46.105 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:46.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:46.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:46.207 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:46.237 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:46.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:46.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:46.339 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:46.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:46.363 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:46.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:46.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:48:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:48:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:48:46.464 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:46.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:46.489 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:46.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:46.590 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:46.618 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:46.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:46.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:46.720 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:46.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:46.747 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:46.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:46.848 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:46.873 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:46.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:46.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:46.974 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:46.998 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:46.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:46.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:47.100 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:47.113 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:46 smithi043 ceph-mon[29963]: pgmap v223: 148 pgs: 148 active+clean; 582 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 508 op/s 2024-03-30T13:48:47.113 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:46 smithi043 ceph-mon[42099]: pgmap v223: 148 pgs: 148 active+clean; 582 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 508 op/s 2024-03-30T13:48:47.128 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:47.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:47.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:47.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:46 smithi113 ceph-mon[34341]: pgmap v223: 148 pgs: 148 active+clean; 582 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 508 op/s 2024-03-30T13:48:47.230 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:47.253 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:47.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:47.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:47.355 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:47.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:47.385 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:47.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:47.487 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:47.514 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:47.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:47.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:47.616 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:47.641 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:47.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:47.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:47.743 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:47.767 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:47.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:47.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:47.842 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-03-30T13:48:47.868 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:47.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:47.893 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:47.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:47.994 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:48.017 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:48.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:48.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:48.119 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:48.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:48.149 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:48.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:48.250 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:48.277 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:48.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:48.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:48.379 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:48.407 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:48.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:48.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:48.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:48 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:48:48.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:48 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:48:48.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:48 smithi043 ceph-mon[29963]: pgmap v224: 148 pgs: 148 active+clean; 582 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 508 op/s 2024-03-30T13:48:48.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:48 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2684832825' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:48:48.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:48 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:48:48.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:48 smithi043 ceph-mon[29963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:48:48.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:48 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:48:48.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:48 smithi043 ceph-mon[29963]: osdmap e143: 8 total, 8 up, 8 in 2024-03-30T13:48:48.441 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:48 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:48:48.441 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:48 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:48:48.441 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:48 smithi043 ceph-mon[42099]: pgmap v224: 148 pgs: 148 active+clean; 582 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 508 op/s 2024-03-30T13:48:48.441 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:48 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2684832825' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:48:48.441 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:48 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:48:48.441 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:48 smithi043 ceph-mon[42099]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:48:48.441 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:48 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:48:48.441 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:48 smithi043 ceph-mon[42099]: osdmap e143: 8 total, 8 up, 8 in 2024-03-30T13:48:48.510 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:48.537 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:48.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:48.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:48.640 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:48.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:48.666 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:48.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:48.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:48:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:48:48.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:48 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:48:48.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:48 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:48:48.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:48 smithi113 ceph-mon[34341]: pgmap v224: 148 pgs: 148 active+clean; 582 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 508 op/s 2024-03-30T13:48:48.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:48 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2684832825' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:48:48.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:48 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:48:48.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:48 smithi113 ceph-mon[34341]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:48:48.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:48 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:48:48.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:48 smithi113 ceph-mon[34341]: osdmap e143: 8 total, 8 up, 8 in 2024-03-30T13:48:48.767 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:48.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:48.795 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:48.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:48.896 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:48.920 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:48.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:48.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:49.022 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:49.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:49.047 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:49.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:49.148 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:49.172 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:49.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:49.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:49.274 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:49.302 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:49.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:49.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:49.404 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:49.430 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:49.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:49.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:49.532 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:49.558 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:49.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:49.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:49.621 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:49 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:48:49.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:49 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:48:49.660 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:49.678 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:49 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:48:49.687 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:49.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:49.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:49.789 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:49.813 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:49.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:49.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:49.826 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:48:49.915 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:49.939 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:49.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:49.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:50.041 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:50.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:50.065 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:50.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:50.166 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:50.191 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:50.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:50.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:50.259 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 reset_purged_snaps_last 2024-03-30T13:48:50.292 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:50.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:50.317 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:50.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:50.419 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:50.443 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:50.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:50.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:50.472 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:48:50.544 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:50.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:50.570 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:50.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:50.576 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:50 smithi043 ceph-mon[29963]: pgmap v226: 148 pgs: 148 active+clean; 582 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-30T13:48:50.576 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:50 smithi043 ceph-mon[42099]: pgmap v226: 148 pgs: 148 active+clean; 582 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-30T13:48:50.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:50.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:50.699 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:50.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:50.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:50 smithi113 ceph-mon[34341]: pgmap v226: 148 pgs: 148 active+clean; 582 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-30T13:48:50.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:50.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:50.827 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:50.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:50.928 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:50.952 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:50.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:50.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:51.054 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:51.078 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:51.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:51.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:51.118 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T13:48:51.181 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:51.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:51.206 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:51.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:51.307 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:51.331 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:51.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:51.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:51.424 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'unique_pool_0' removed 2024-03-30T13:48:51.432 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:51.475 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:51.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:51.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:51.577 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:51.604 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:51.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:51.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:51.705 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:51.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:51 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:48:51.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:51 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:48:51.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:51 smithi113 ceph-mon[34341]: osdmap e144: 8 total, 8 up, 8 in 2024-03-30T13:48:51.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:51 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1887644412' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T13:48:51.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:51 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T13:48:51.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:51.730 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:51.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:51.831 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:51.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:51.856 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:51.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:51.862 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:51 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:48:51.863 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:51 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:48:51.863 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:51 smithi043 ceph-mon[29963]: osdmap e144: 8 total, 8 up, 8 in 2024-03-30T13:48:51.863 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:51 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1887644412' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T13:48:51.863 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:51 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T13:48:51.863 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:51 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:48:51.863 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:51 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:48:51.863 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:51 smithi043 ceph-mon[42099]: osdmap e144: 8 total, 8 up, 8 in 2024-03-30T13:48:51.863 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:51 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1887644412' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T13:48:51.863 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:51 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T13:48:51.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:51.980 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:51.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:51.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:52.083 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:52.106 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:52.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:52.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:52.208 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:52.234 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:52.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:52.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:52.307 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-30T13:48:52.308 INFO:tasks.radosbench:Beginning radosbench... 2024-03-30T13:48:52.308 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-03-30T13:48:52.308 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool create unique_pool_1 16 2024-03-30T13:48:52.338 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:52.362 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:52.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:52.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:52.464 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:52.489 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:52.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:52.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:52.591 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:52.608 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:52 smithi043 ceph-mon[42099]: pgmap v228: 148 pgs: 148 active+clean; 582 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-30T13:48:52.608 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:52 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T13:48:52.609 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:52 smithi043 ceph-mon[42099]: osdmap e145: 8 total, 8 up, 8 in 2024-03-30T13:48:52.609 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:52 smithi043 ceph-mon[29963]: pgmap v228: 148 pgs: 148 active+clean; 582 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-30T13:48:52.609 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:52 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T13:48:52.609 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:52 smithi043 ceph-mon[29963]: osdmap e145: 8 total, 8 up, 8 in 2024-03-30T13:48:52.615 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:52.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:52.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:52.717 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:52.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:52 smithi113 ceph-mon[34341]: pgmap v228: 148 pgs: 148 active+clean; 582 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-30T13:48:52.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:52 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T13:48:52.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:52 smithi113 ceph-mon[34341]: osdmap e145: 8 total, 8 up, 8 in 2024-03-30T13:48:52.741 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:52.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:52.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:52.845 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:52.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:52.871 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:52.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:52.972 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:52.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:52.996 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:52.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:53.050 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:48:53.097 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:53.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:53.121 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:53.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:53.222 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:53.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:53.248 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:53.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:53.349 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:53.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:53.374 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:53.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:53.475 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:53.498 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:53.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:53.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:53.600 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:53.614 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:48:53.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:53.631 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:53.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:53.732 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:53.756 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:53.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:53.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:53.858 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:53.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:53.884 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:53.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:53.985 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:54.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:54.011 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:54.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:54.112 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:54.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:54.137 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:54.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:54.238 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:54.262 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:54.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:54.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:54.364 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:54.389 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:54.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:54.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:54.410 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:48:54.434 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:48:54.449 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:48:54.469 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:48:54.488 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:48:54.491 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:54.508 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:48:54.515 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:54.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:54.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:54.528 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:48:54.548 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:48:54.549 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:48:54.550 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:48:54.550 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:48:54.550 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:48:54.550 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:48:54.550 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:48:54.550 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:48:54.550 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:48:54.617 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:54.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:54.642 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:54.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:54.661 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:48:54.663 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:54 smithi043 ceph-mon[29963]: pgmap v230: 132 pgs: 132 active+clean; 582 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-30T13:48:54.663 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:54 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2804146863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:48:54.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:54 smithi043 ceph-mon[42099]: pgmap v230: 132 pgs: 132 active+clean; 582 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-30T13:48:54.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:54 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2804146863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:48:54.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:54 smithi113 ceph-mon[34341]: pgmap v230: 132 pgs: 132 active+clean; 582 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-30T13:48:54.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:54 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2804146863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:48:54.743 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:54.767 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:54.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:54.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:54.869 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:54.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:54.894 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:54.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:54.995 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:55.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:55.020 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:55.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:55.122 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:55.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:55.146 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:55.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:55.247 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:55.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:55.272 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:55.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:55.373 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:55.400 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:55.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:55.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:55.502 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:55.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:55.529 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:55.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:55.630 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:55.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:55.660 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:55.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:55.761 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:48:55.789 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:55.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:55.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:48:55.891 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:55.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:55.916 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:55.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:56.017 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:56.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:56.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:56.143 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:56.149 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:48:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:48:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:48:56.167 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:56.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:56.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:56.269 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:56.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:56.293 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:56.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:56.394 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:56.423 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:56.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:56.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:56.525 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:56.552 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:56.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:56.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:56.654 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:56.678 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:56.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:56.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:56.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:56 smithi043 ceph-mon[29963]: pgmap v231: 132 pgs: 132 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:48:56.702 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:56 smithi043 ceph-mon[42099]: pgmap v231: 132 pgs: 132 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:48:56.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:56 smithi113 ceph-mon[34341]: pgmap v231: 132 pgs: 132 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:48:56.781 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:48:56.804 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:56.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:56.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:48:56.906 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:56.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:56.935 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:56.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:56.974 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:48:56.974 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T13:48:56.974 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 0 2024-03-30T13:48:56.974 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:48:57.037 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:57.062 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:57.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:57.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:57.164 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:57.191 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:57.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:57.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:57.293 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:57.321 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:57.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:57.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:57.378 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:48:57.423 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:57.449 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:57.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:57.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:57.551 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:57.581 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:57.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:57.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:57.683 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:57.711 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:57.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:57.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:57.812 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:57.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:57.842 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:57.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:57.943 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:57.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:57.968 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:57.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:58.069 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:48:58.097 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:58.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:58.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:48:58.199 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:58.223 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:58.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:58.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:58.325 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:48:58.350 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:58.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:58.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:48:58.452 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:58.475 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:58.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:58.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:58.571 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:58 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:48:58.571 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:58 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:48:58.571 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:58 smithi043 ceph-mon[29963]: pgmap v232: 132 pgs: 132 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-30T13:48:58.571 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:58 smithi043 ceph-mon[29963]: osdmap e146: 8 total, 8 up, 8 in 2024-03-30T13:48:58.571 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:58 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:48:58.572 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:58 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:48:58.572 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:58 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:48:58.572 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:58 smithi043 ceph-mon[42099]: pgmap v232: 132 pgs: 132 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-30T13:48:58.572 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:58 smithi043 ceph-mon[42099]: osdmap e146: 8 total, 8 up, 8 in 2024-03-30T13:48:58.572 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:58 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:48:58.578 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:58.608 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:58.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:58.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:58.710 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:48:58.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:58 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:48:58.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:58 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:48:58.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:58 smithi113 ceph-mon[34341]: pgmap v232: 132 pgs: 132 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-30T13:48:58.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:58 smithi113 ceph-mon[34341]: osdmap e146: 8 total, 8 up, 8 in 2024-03-30T13:48:58.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:58 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:48:58.723 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:48:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:48:58.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:58.735 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:58.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:48:58.836 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:58.864 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:58.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:58.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:58.966 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:58.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:58.992 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:58.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:59.093 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:59.121 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:59.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:59.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:59.221 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-03-30T13:48:59.222 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:59.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:59.251 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:59.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:59.352 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:59.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:59.379 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:59.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:59.480 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:48:59.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:59.506 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:59.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:48:59.611 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:59.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:59.636 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:59.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:59.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:48:59 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:48:59.738 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:48:59.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:59.762 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:48:59.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:48:59.863 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:48:59.889 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:48:59.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:48:59.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:48:59.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:48:59 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:48:59.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:48:59 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:48:59.991 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:00.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:00.019 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:00.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:00.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:00.145 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:00.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:00.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:49:00.247 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:00.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:00.272 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:00.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:00.373 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:00.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:00.399 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:00.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:00.443 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'unique_pool_1' created 2024-03-30T13:49:00.499 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:00.523 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:00.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:00.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:00.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:00 smithi043 ceph-mon[29963]: pgmap v234: 132 pgs: 132 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T13:49:00.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:00 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1108919840' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-30T13:49:00.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:00 smithi043 ceph-mon[42099]: pgmap v234: 132 pgs: 132 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T13:49:00.596 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:00 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1108919840' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-30T13:49:00.625 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:00.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:00.653 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:00.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:00.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:00 smithi113 ceph-mon[34341]: pgmap v234: 132 pgs: 132 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T13:49:00.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:00 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1108919840' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-30T13:49:00.755 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:00.778 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:00.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:00.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:00.880 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:00.909 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:00.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:00.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:01.011 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:01.034 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:01.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:01.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:01.136 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:01.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:01.162 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:01.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:01.259 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 reset_purged_snaps_last 2024-03-30T13:49:01.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:01.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:01.326 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:01.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:01.427 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:01.456 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:01.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:01.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:01.506 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:01 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1108919840' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-30T13:49:01.506 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:01 smithi043 ceph-mon[29963]: osdmap e147: 8 total, 8 up, 8 in 2024-03-30T13:49:01.507 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:01 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1108919840' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-30T13:49:01.507 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:01 smithi043 ceph-mon[42099]: osdmap e147: 8 total, 8 up, 8 in 2024-03-30T13:49:01.558 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:01.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:01.583 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:01.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:01.684 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:01.710 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:01.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:01.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:01.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:01 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1108919840' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-30T13:49:01.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:01 smithi113 ceph-mon[34341]: osdmap e147: 8 total, 8 up, 8 in 2024-03-30T13:49:01.812 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:01.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:01.838 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:01.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:01.939 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:01.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:01.965 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:01.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:02.066 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:02.092 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:02.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:02.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:02.185 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-03-30T13:49:02.193 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:02.219 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:02.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:02.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:02.321 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:02.346 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:02.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:02.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:02.349 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:49:02.448 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:02.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:02.473 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:02.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:02.574 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:02.599 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:02.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:02.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:02.701 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:02.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:02 smithi113 ceph-mon[34341]: pgmap v236: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:49:02.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:02 smithi113 ceph-mon[34341]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:49:02.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:02 smithi113 ceph-mon[34341]: osdmap e148: 8 total, 8 up, 8 in 2024-03-30T13:49:02.725 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:02.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:02.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:02.744 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:02 smithi043 ceph-mon[29963]: pgmap v236: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:49:02.744 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:02 smithi043 ceph-mon[29963]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:49:02.744 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:02 smithi043 ceph-mon[29963]: osdmap e148: 8 total, 8 up, 8 in 2024-03-30T13:49:02.744 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:02 smithi043 ceph-mon[42099]: pgmap v236: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:49:02.744 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:02 smithi043 ceph-mon[42099]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:49:02.745 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:02 smithi043 ceph-mon[42099]: osdmap e148: 8 total, 8 up, 8 in 2024-03-30T13:49:02.826 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:02.855 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:02.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:02.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:02.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:02.981 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:02.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:02.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:03.051 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:49:03.083 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:03.111 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:03.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:03.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:03.213 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:03.236 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:03.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:03.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:03.338 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:03.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:03.366 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:03.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:03.457 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T13:49:03.468 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:03.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:03.498 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:03.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:03.599 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:03.625 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:03.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:03.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:03 smithi113 ceph-mon[34341]: osdmap e149: 8 total, 8 up, 8 in 2024-03-30T13:49:03.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:03 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2499526495' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:49:03.728 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:03.753 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:03.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:03.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:03.797 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:03 smithi043 ceph-mon[29963]: osdmap e149: 8 total, 8 up, 8 in 2024-03-30T13:49:03.797 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:03 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2499526495' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:49:03.797 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:03 smithi043 ceph-mon[42099]: osdmap e149: 8 total, 8 up, 8 in 2024-03-30T13:49:03.797 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:03 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2499526495' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:49:03.855 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:03.880 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:03.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:03.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:03.981 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:04.009 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:04.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:04.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:04.111 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:04.137 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:04.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:04.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:04.240 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:04.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:04.266 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:04.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:04.367 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:04.394 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:04.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:04.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:04.441 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:49:04.495 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:04.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:04.522 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:04.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:04.623 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:04.647 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:04.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:04.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:04.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:04 smithi043 ceph-mon[29963]: pgmap v239: 148 pgs: 13 unknown, 135 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-03-30T13:49:04.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:04 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:49:04.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:04 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:49:04.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:04 smithi043 ceph-mon[29963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:49:04.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:04 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2499526495' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:49:04.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:04 smithi043 ceph-mon[29963]: osdmap e150: 8 total, 8 up, 8 in 2024-03-30T13:49:04.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:04 smithi043 ceph-mon[42099]: pgmap v239: 148 pgs: 13 unknown, 135 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-03-30T13:49:04.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:04 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:49:04.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:04 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:49:04.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:04 smithi043 ceph-mon[42099]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:49:04.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:04 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2499526495' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:49:04.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:04 smithi043 ceph-mon[42099]: osdmap e150: 8 total, 8 up, 8 in 2024-03-30T13:49:04.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:04 smithi113 ceph-mon[34341]: pgmap v239: 148 pgs: 13 unknown, 135 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-03-30T13:49:04.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:04 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:49:04.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:04 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:49:04.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:04 smithi113 ceph-mon[34341]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:49:04.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:04 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2499526495' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:49:04.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:04 smithi113 ceph-mon[34341]: osdmap e150: 8 total, 8 up, 8 in 2024-03-30T13:49:04.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:04.773 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:04.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:04.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:04.875 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:04.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:04.901 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:04.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:05.002 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:05.026 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:05.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:05.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:05.128 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:05.151 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:05.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:05.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:05.252 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:05.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:05.281 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:05.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:05.382 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:05.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:05.410 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:05.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:05.511 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:05.536 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:05.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:05.638 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:05.666 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:05.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:05.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:05.768 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:05.791 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:05.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:05.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:05.893 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:05.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:05.917 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:05.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:06.018 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:06.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:06.043 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:06.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:06.144 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:06.168 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:06.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:06.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:06.270 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:06.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:06.295 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:06.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:06.330 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:49:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:49:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:49:06.397 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:06.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:06.422 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:06.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:06.523 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:06.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:06.547 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:06.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:06.648 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:06.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:06 smithi043 ceph-mon[29963]: pgmap v241: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T13:49:06.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:06 smithi043 ceph-mon[42099]: pgmap v241: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T13:49:06.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:06.672 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:06.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:06.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:06 smithi113 ceph-mon[34341]: pgmap v241: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T13:49:06.773 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:06.801 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:06.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:06.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:06.888 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:49:06.888 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T13:49:06.888 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-03-30T13:49:06.889 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-03-30T13:49:06.889 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:49:06.903 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:06.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:06.932 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:06.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:06.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T13:49:07.033 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:07.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:07.058 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:07.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:07.159 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:07.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:07.190 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:07.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:07.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:07.316 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:07.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:07.417 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:07.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:07.441 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:07.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:07.542 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:07.552 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-03-30T13:49:07.571 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:07.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:07.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:07.673 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:07.700 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:07.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:07.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:07.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:07 smithi113 ceph-mon[34341]: 6.e scrub starts 2024-03-30T13:49:07.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:07 smithi113 ceph-mon[34341]: 6.e scrub ok 2024-03-30T13:49:07.801 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:07 smithi043 ceph-mon[29963]: 6.e scrub starts 2024-03-30T13:49:07.801 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:07 smithi043 ceph-mon[29963]: 6.e scrub ok 2024-03-30T13:49:07.802 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:07 smithi043 ceph-mon[42099]: 6.e scrub starts 2024-03-30T13:49:07.802 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:07 smithi043 ceph-mon[42099]: 6.e scrub ok 2024-03-30T13:49:07.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:07.827 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:07.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:07.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:07.929 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:07.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:07.958 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:07.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:08.059 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:08.082 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:08.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:08.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:08.184 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:08.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:08.210 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:08.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:08.311 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:08.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:08.340 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:08.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:08.441 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:08.465 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:08.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:08.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:08.567 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:08.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:08.596 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:08.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:08.697 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:08.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:49:08 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:49:08.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:08 smithi113 ceph-mon[34341]: pgmap v242: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:49:08.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:08 smithi113 ceph-mon[34341]: 3.13 scrub starts 2024-03-30T13:49:08.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:08 smithi113 ceph-mon[34341]: 3.13 scrub ok 2024-03-30T13:49:08.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:08 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1928385972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:49:08.724 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:08.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:08.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:08.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:08.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:08.852 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:08.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:08.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:08 smithi043 ceph-mon[29963]: pgmap v242: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:49:08.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:08 smithi043 ceph-mon[29963]: 3.13 scrub starts 2024-03-30T13:49:08.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:08 smithi043 ceph-mon[29963]: 3.13 scrub ok 2024-03-30T13:49:08.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:08 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1928385972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:49:08.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:08 smithi043 ceph-mon[42099]: pgmap v242: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:49:08.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:08 smithi043 ceph-mon[42099]: 3.13 scrub starts 2024-03-30T13:49:08.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:08 smithi043 ceph-mon[42099]: 3.13 scrub ok 2024-03-30T13:49:08.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:08 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1928385972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:49:08.954 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:08.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:08.983 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:08.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:09.084 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:09.108 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:09.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:09.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:09.210 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:09.234 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:49:09.234 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:49:09.234 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:49:09.234 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:49:09.234 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:49:09.234 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:49:09.234 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:49:09.235 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:49:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:49:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:49:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:49:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:49:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:49:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:49:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:49:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:49:09.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:09.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:09.337 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:09.360 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:09.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:09.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:09.462 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:09.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:09.492 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:09.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:09.593 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:09.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:09.622 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:09.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:09.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:09 smithi113 ceph-mon[34341]: 4.11 scrub starts 2024-03-30T13:49:09.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:09 smithi113 ceph-mon[34341]: 4.11 scrub ok 2024-03-30T13:49:09.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:09 smithi113 ceph-mon[34341]: 3.3 scrub starts 2024-03-30T13:49:09.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:09 smithi113 ceph-mon[34341]: 3.3 scrub ok 2024-03-30T13:49:09.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:09 smithi113 ceph-mon[34341]: 4.1c scrub starts 2024-03-30T13:49:09.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:09 smithi113 ceph-mon[34341]: 4.1c scrub ok 2024-03-30T13:49:09.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:09 smithi113 ceph-mon[34341]: 4.2 scrub starts 2024-03-30T13:49:09.724 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:09 smithi113 ceph-mon[34341]: 4.2 scrub ok 2024-03-30T13:49:09.724 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:09 smithi113 ceph-mon[34341]: 3.5 scrub starts 2024-03-30T13:49:09.724 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:09 smithi113 ceph-mon[34341]: 3.5 scrub ok 2024-03-30T13:49:09.724 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:09 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:49:09.724 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:09.749 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:09.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:09.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:09.851 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:09.879 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:09.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:09.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:09.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[42099]: 4.11 scrub starts 2024-03-30T13:49:09.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[42099]: 4.11 scrub ok 2024-03-30T13:49:09.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[42099]: 3.3 scrub starts 2024-03-30T13:49:09.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[42099]: 3.3 scrub ok 2024-03-30T13:49:09.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[42099]: 4.1c scrub starts 2024-03-30T13:49:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[42099]: 4.1c scrub ok 2024-03-30T13:49:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[42099]: 4.2 scrub starts 2024-03-30T13:49:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[42099]: 4.2 scrub ok 2024-03-30T13:49:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[42099]: 3.5 scrub starts 2024-03-30T13:49:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[42099]: 3.5 scrub ok 2024-03-30T13:49:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:49:09.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[29963]: 4.11 scrub starts 2024-03-30T13:49:09.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[29963]: 4.11 scrub ok 2024-03-30T13:49:09.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[29963]: 3.3 scrub starts 2024-03-30T13:49:09.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[29963]: 3.3 scrub ok 2024-03-30T13:49:09.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[29963]: 4.1c scrub starts 2024-03-30T13:49:09.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[29963]: 4.1c scrub ok 2024-03-30T13:49:09.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[29963]: 4.2 scrub starts 2024-03-30T13:49:09.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[29963]: 4.2 scrub ok 2024-03-30T13:49:09.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[29963]: 3.5 scrub starts 2024-03-30T13:49:09.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[29963]: 3.5 scrub ok 2024-03-30T13:49:09.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:09 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:49:09.981 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:10.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:10.006 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:10.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:10.107 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:10.130 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:10.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:10.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:10.232 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:10.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:10.259 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:10.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:10.360 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:10.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:10.385 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:10.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:10.486 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:10.509 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:10.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:10.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:10.611 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:10.634 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:10.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:10.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:10.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:10 smithi113 ceph-mon[34341]: 3.14 scrub starts 2024-03-30T13:49:10.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:10 smithi113 ceph-mon[34341]: 3.14 scrub ok 2024-03-30T13:49:10.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:10 smithi113 ceph-mon[34341]: pgmap v243: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T13:49:10.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:10 smithi113 ceph-mon[34341]: 5.19 scrub starts 2024-03-30T13:49:10.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:10 smithi113 ceph-mon[34341]: 5.19 scrub ok 2024-03-30T13:49:10.736 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:10.761 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:10.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:10.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:10.855 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:10 smithi043 ceph-mon[29963]: 3.14 scrub starts 2024-03-30T13:49:10.856 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:10 smithi043 ceph-mon[29963]: 3.14 scrub ok 2024-03-30T13:49:10.856 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:10 smithi043 ceph-mon[29963]: pgmap v243: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T13:49:10.856 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:10 smithi043 ceph-mon[29963]: 5.19 scrub starts 2024-03-30T13:49:10.856 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:10 smithi043 ceph-mon[29963]: 5.19 scrub ok 2024-03-30T13:49:10.856 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:10 smithi043 ceph-mon[42099]: 3.14 scrub starts 2024-03-30T13:49:10.856 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:10 smithi043 ceph-mon[42099]: 3.14 scrub ok 2024-03-30T13:49:10.856 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:10 smithi043 ceph-mon[42099]: pgmap v243: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T13:49:10.856 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:10 smithi043 ceph-mon[42099]: 5.19 scrub starts 2024-03-30T13:49:10.856 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:10 smithi043 ceph-mon[42099]: 5.19 scrub ok 2024-03-30T13:49:10.863 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:10.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:10.892 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:10.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:10.993 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:11.018 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:11.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:11.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:11.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:11.144 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:11.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:11.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:11.245 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:11.246 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:49:11.273 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:11.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:11.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:11.375 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:11.399 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:11.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:11.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:11.474 INFO:teuthology.orchestra.run.smithi043.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-03-30T13:49:11.501 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:11.526 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:11.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:11.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:11.628 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:11.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:11.653 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:11.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:11.754 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:11.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:11.778 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:11.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:11.866 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:49:11.879 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:11.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:11 smithi043 ceph-mon[29963]: 4.18 scrub starts 2024-03-30T13:49:11.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:11 smithi043 ceph-mon[29963]: 4.18 scrub ok 2024-03-30T13:49:11.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:11 smithi043 ceph-mon[29963]: 3.19 scrub starts 2024-03-30T13:49:11.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:11 smithi043 ceph-mon[29963]: 3.19 scrub ok 2024-03-30T13:49:11.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:11 smithi043 ceph-mon[29963]: 5.1a scrub starts 2024-03-30T13:49:11.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:11 smithi043 ceph-mon[29963]: 5.1a scrub ok 2024-03-30T13:49:11.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:11 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:49:11.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:11 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:49:11.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:11 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2985892035' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:49:11.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:11 smithi043 ceph-mon[42099]: 4.18 scrub starts 2024-03-30T13:49:11.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:11 smithi043 ceph-mon[42099]: 4.18 scrub ok 2024-03-30T13:49:11.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:11 smithi043 ceph-mon[42099]: 3.19 scrub starts 2024-03-30T13:49:11.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:11 smithi043 ceph-mon[42099]: 3.19 scrub ok 2024-03-30T13:49:11.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:11 smithi043 ceph-mon[42099]: 5.1a scrub starts 2024-03-30T13:49:11.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:11 smithi043 ceph-mon[42099]: 5.1a scrub ok 2024-03-30T13:49:11.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:11 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:49:11.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:11 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:49:11.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:11 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2985892035' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:49:11.907 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:11.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:11.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:11.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:11 smithi113 ceph-mon[34341]: 4.18 scrub starts 2024-03-30T13:49:11.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:11 smithi113 ceph-mon[34341]: 4.18 scrub ok 2024-03-30T13:49:11.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:11 smithi113 ceph-mon[34341]: 3.19 scrub starts 2024-03-30T13:49:11.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:11 smithi113 ceph-mon[34341]: 3.19 scrub ok 2024-03-30T13:49:11.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:11 smithi113 ceph-mon[34341]: 5.1a scrub starts 2024-03-30T13:49:11.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:11 smithi113 ceph-mon[34341]: 5.1a scrub ok 2024-03-30T13:49:11.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:11 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:49:11.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:11 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:49:11.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:11 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2985892035' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:49:11.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:49:12.009 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:12.040 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:12.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:12.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:12.142 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:12.170 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:12.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:12.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:12.272 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:12.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:12.296 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:12.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:12.397 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:12.412 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 reset_purged_snaps_last 2024-03-30T13:49:12.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:12.431 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:12.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:12.533 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:12.557 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:12.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:12.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:12.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:12 smithi043 ceph-mon[42099]: pgmap v244: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T13:49:12.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:12 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2985892035' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-30T13:49:12.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:12 smithi043 ceph-mon[42099]: osdmap e151: 8 total, 8 up, 8 in 2024-03-30T13:49:12.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:12 smithi043 ceph-mon[42099]: 3.d scrub starts 2024-03-30T13:49:12.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:12 smithi043 ceph-mon[42099]: 3.d scrub ok 2024-03-30T13:49:12.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:12 smithi043 ceph-mon[29963]: pgmap v244: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T13:49:12.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:12 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2985892035' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-30T13:49:12.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:12 smithi043 ceph-mon[29963]: osdmap e151: 8 total, 8 up, 8 in 2024-03-30T13:49:12.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:12 smithi043 ceph-mon[29963]: 3.d scrub starts 2024-03-30T13:49:12.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:12 smithi043 ceph-mon[29963]: 3.d scrub ok 2024-03-30T13:49:12.659 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:12.684 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:12.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:12.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:12.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:12 smithi113 ceph-mon[34341]: pgmap v244: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T13:49:12.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:12 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2985892035' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-30T13:49:12.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:12 smithi113 ceph-mon[34341]: osdmap e151: 8 total, 8 up, 8 in 2024-03-30T13:49:12.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:12 smithi113 ceph-mon[34341]: 3.d scrub starts 2024-03-30T13:49:12.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:12 smithi113 ceph-mon[34341]: 3.d scrub ok 2024-03-30T13:49:12.786 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:12.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:12.810 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:12.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:12.911 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:12.940 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:12.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:13.042 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:13.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:13.066 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:13.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:13.167 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:13.195 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:13.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:13.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:13.297 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:13.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:13.326 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:13.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:13.427 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:13.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:13.454 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:13.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:13.555 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:13.579 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:13.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:13.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:13.681 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:13.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:13.706 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:13.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:13.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:13 smithi113 ceph-mon[34341]: 4.0 scrub starts 2024-03-30T13:49:13.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:13 smithi113 ceph-mon[34341]: 4.0 scrub ok 2024-03-30T13:49:13.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:13 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:49:13.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:13 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:49:13.807 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:13.836 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:13.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:13.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:13.888 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:13 smithi043 ceph-mon[29963]: 4.0 scrub starts 2024-03-30T13:49:13.888 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:13 smithi043 ceph-mon[29963]: 4.0 scrub ok 2024-03-30T13:49:13.888 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:13 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:49:13.888 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:13 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:49:13.888 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:13 smithi043 ceph-mon[42099]: 4.0 scrub starts 2024-03-30T13:49:13.888 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:13 smithi043 ceph-mon[42099]: 4.0 scrub ok 2024-03-30T13:49:13.888 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:13 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:49:13.889 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:13 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:49:13.938 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:13.961 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:13.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:13.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:14.063 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:14.091 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:14.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:14.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:14.193 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:14.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:14.219 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:14.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:14.320 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:14.344 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:14.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:14.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:14.446 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:14.474 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:14.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:14.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:14.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:14.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:14.702 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:14.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:14 smithi113 ceph-mon[34341]: 6.9 scrub starts 2024-03-30T13:49:14.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:14 smithi113 ceph-mon[34341]: 6.9 scrub ok 2024-03-30T13:49:14.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:14 smithi113 ceph-mon[34341]: 5.a scrub starts 2024-03-30T13:49:14.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:14 smithi113 ceph-mon[34341]: 5.a scrub ok 2024-03-30T13:49:14.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:14 smithi113 ceph-mon[34341]: pgmap v246: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 514 B/s rd, 0 op/s 2024-03-30T13:49:14.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:14 smithi113 ceph-mon[34341]: 6.5 scrub starts 2024-03-30T13:49:14.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:14 smithi113 ceph-mon[34341]: 6.5 scrub ok 2024-03-30T13:49:14.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:14 smithi113 ceph-mon[34341]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T13:49:14.725 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:14.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:14.736 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:14 smithi043 ceph-mon[29963]: 6.9 scrub starts 2024-03-30T13:49:14.736 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:14 smithi043 ceph-mon[29963]: 6.9 scrub ok 2024-03-30T13:49:14.736 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:14 smithi043 ceph-mon[29963]: 5.a scrub starts 2024-03-30T13:49:14.736 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:14 smithi043 ceph-mon[29963]: 5.a scrub ok 2024-03-30T13:49:14.736 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:14 smithi043 ceph-mon[29963]: pgmap v246: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 514 B/s rd, 0 op/s 2024-03-30T13:49:14.736 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:14 smithi043 ceph-mon[29963]: 6.5 scrub starts 2024-03-30T13:49:14.736 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:14 smithi043 ceph-mon[29963]: 6.5 scrub ok 2024-03-30T13:49:14.736 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:14 smithi043 ceph-mon[29963]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T13:49:14.737 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:14 smithi043 ceph-mon[42099]: 6.9 scrub starts 2024-03-30T13:49:14.737 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:14 smithi043 ceph-mon[42099]: 6.9 scrub ok 2024-03-30T13:49:14.737 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:14 smithi043 ceph-mon[42099]: 5.a scrub starts 2024-03-30T13:49:14.737 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:14 smithi043 ceph-mon[42099]: 5.a scrub ok 2024-03-30T13:49:14.737 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:14 smithi043 ceph-mon[42099]: pgmap v246: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 514 B/s rd, 0 op/s 2024-03-30T13:49:14.737 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:14 smithi043 ceph-mon[42099]: 6.5 scrub starts 2024-03-30T13:49:14.737 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:14 smithi043 ceph-mon[42099]: 6.5 scrub ok 2024-03-30T13:49:14.737 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:14 smithi043 ceph-mon[42099]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T13:49:14.828 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:14.853 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:14.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:14.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:14.915 DEBUG:teuthology.orchestra.run.smithi043:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-03-30T13:49:14.918 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-30T13:49:14.918 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-30T13:49:14.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:14.968 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:hints = 1 2024-03-30T13:49:14.968 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-30T13:49:14.968 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Object prefix: benchmark_data_smithi043_119832 2024-03-30T13:49:14.969 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T13:49:14.969 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 0 0 0 0 0 0 - 0 2024-03-30T13:49:14.987 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:14.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:14.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:15.089 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:15.123 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:15.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:15.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:15.225 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:15.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:15.267 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:15.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:15.368 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:15.403 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:15.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:15.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:15.505 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:15.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:15.537 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:15.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:15.638 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:15.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:15.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:15.783 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:15.822 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:15.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:15.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:15.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:15 smithi043 ceph-mon[42099]: 4.6 scrub starts 2024-03-30T13:49:15.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:15 smithi043 ceph-mon[42099]: 4.6 scrub ok 2024-03-30T13:49:15.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:15 smithi043 ceph-mon[42099]: 4.1f scrub starts 2024-03-30T13:49:15.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:15 smithi043 ceph-mon[42099]: 4.1f scrub ok 2024-03-30T13:49:15.840 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:15 smithi043 ceph-mon[29963]: 4.6 scrub starts 2024-03-30T13:49:15.840 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:15 smithi043 ceph-mon[29963]: 4.6 scrub ok 2024-03-30T13:49:15.840 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:15 smithi043 ceph-mon[29963]: 4.1f scrub starts 2024-03-30T13:49:15.840 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:15 smithi043 ceph-mon[29963]: 4.1f scrub ok 2024-03-30T13:49:15.923 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:15.945 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:49:15.951 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:49:15.953 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:15.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:15.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:15.969 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 1 16 3280 3264 203.988 204 0.00610885 0.0048763 2024-03-30T13:49:15.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:15 smithi113 ceph-mon[34341]: 4.6 scrub starts 2024-03-30T13:49:15.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:15 smithi113 ceph-mon[34341]: 4.6 scrub ok 2024-03-30T13:49:15.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:15 smithi113 ceph-mon[34341]: 4.1f scrub starts 2024-03-30T13:49:15.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:15 smithi113 ceph-mon[34341]: 4.1f scrub ok 2024-03-30T13:49:16.055 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:16.103 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:16.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:16.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:16.205 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:16.240 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:16.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:16.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:16.341 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:16.375 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:16.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:16.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:16.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:49:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:49:16.476 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:16.499 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-03-30T13:49:16.531 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:16.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:16.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:16.633 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:16.666 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:16.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:16.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:16.707 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-mon[29963]: 3.6 scrub starts 2024-03-30T13:49:16.707 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-mon[29963]: 3.6 scrub ok 2024-03-30T13:49:16.707 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-mon[29963]: pgmap v247: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:49:16.707 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3188748161' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:49:16.707 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:49:16.707 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-mon[29963]: 2.0 scrub starts 2024-03-30T13:49:16.707 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-mon[29963]: 2.0 scrub ok 2024-03-30T13:49:16.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-mon[29963]: 5.1f scrub starts 2024-03-30T13:49:16.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-mon[29963]: 5.1f scrub ok 2024-03-30T13:49:16.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:49:16.708 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-mon[42099]: 3.6 scrub starts 2024-03-30T13:49:16.708 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-mon[42099]: 3.6 scrub ok 2024-03-30T13:49:16.708 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-mon[42099]: pgmap v247: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:49:16.708 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3188748161' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:49:16.708 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:49:16.708 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-mon[42099]: 2.0 scrub starts 2024-03-30T13:49:16.708 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-mon[42099]: 2.0 scrub ok 2024-03-30T13:49:16.708 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-mon[42099]: 5.1f scrub starts 2024-03-30T13:49:16.708 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-mon[42099]: 5.1f scrub ok 2024-03-30T13:49:16.709 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:16 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:49:16.768 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:16.822 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:16.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:16.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:16.831 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:49:16.831 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:49:16.924 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:16.961 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:16.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:16.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:16.969 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 2 16 6153 6137 191.767 179.562 0.00551066 0.00519887 2024-03-30T13:49:16.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:16 smithi113 ceph-mon[34341]: 3.6 scrub starts 2024-03-30T13:49:16.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:16 smithi113 ceph-mon[34341]: 3.6 scrub ok 2024-03-30T13:49:16.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:16 smithi113 ceph-mon[34341]: pgmap v247: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:49:16.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:16 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3188748161' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:49:16.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:16 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:49:16.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:16 smithi113 ceph-mon[34341]: 2.0 scrub starts 2024-03-30T13:49:16.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:16 smithi113 ceph-mon[34341]: 2.0 scrub ok 2024-03-30T13:49:16.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:16 smithi113 ceph-mon[34341]: 5.1f scrub starts 2024-03-30T13:49:16.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:16 smithi113 ceph-mon[34341]: 5.1f scrub ok 2024-03-30T13:49:16.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:16 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:49:17.063 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:17.100 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:17.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:17.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:17.202 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:17.245 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:17.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:17.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:17.347 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:17.372 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:17.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:17.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:17.474 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:17.512 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:17.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:17.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:17.614 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:17.653 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:17.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:17.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:17.755 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:17.795 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:17.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:17.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:17.897 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:17.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[29963]: 3.0 scrub starts 2024-03-30T13:49:17.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[29963]: 3.0 scrub ok 2024-03-30T13:49:17.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[29963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:49:17.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[29963]: Cluster is now healthy 2024-03-30T13:49:17.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:49:17.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[29963]: osdmap e152: 8 total, 8 up, 8 in 2024-03-30T13:49:17.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[29963]: 4.1d scrub starts 2024-03-30T13:49:17.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[29963]: 4.1d scrub ok 2024-03-30T13:49:17.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:49:17.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:49:17.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[29963]: osdmap e153: 8 total, 8 up, 8 in 2024-03-30T13:49:17.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[42099]: 3.0 scrub starts 2024-03-30T13:49:17.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[42099]: 3.0 scrub ok 2024-03-30T13:49:17.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[42099]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:49:17.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[42099]: Cluster is now healthy 2024-03-30T13:49:17.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:49:17.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[42099]: osdmap e152: 8 total, 8 up, 8 in 2024-03-30T13:49:17.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[42099]: 4.1d scrub starts 2024-03-30T13:49:17.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[42099]: 4.1d scrub ok 2024-03-30T13:49:17.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:49:17.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:49:17.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:17 smithi043 ceph-mon[42099]: osdmap e153: 8 total, 8 up, 8 in 2024-03-30T13:49:17.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:17.934 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:17.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:17.968 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 3 16 8909 8893 185.256 172.25 0.00467271 0.0053887 2024-03-30T13:49:17.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:17 smithi113 ceph-mon[34341]: 3.0 scrub starts 2024-03-30T13:49:17.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:17 smithi113 ceph-mon[34341]: 3.0 scrub ok 2024-03-30T13:49:17.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:17 smithi113 ceph-mon[34341]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:49:17.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:17 smithi113 ceph-mon[34341]: Cluster is now healthy 2024-03-30T13:49:17.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:17 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:49:17.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:17 smithi113 ceph-mon[34341]: osdmap e152: 8 total, 8 up, 8 in 2024-03-30T13:49:17.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:17 smithi113 ceph-mon[34341]: 4.1d scrub starts 2024-03-30T13:49:17.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:17 smithi113 ceph-mon[34341]: 4.1d scrub ok 2024-03-30T13:49:17.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:17 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:49:17.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:17 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:49:17.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:17 smithi113 ceph-mon[34341]: osdmap e153: 8 total, 8 up, 8 in 2024-03-30T13:49:18.035 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:18.071 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:18.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:18.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:18.173 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:18.208 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:18.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:18.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:18.310 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:18.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:18.359 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:18.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:18.460 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:18.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:18.496 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:18.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:18.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:18 smithi043 ceph-mon[29963]: 5.3 scrub starts 2024-03-30T13:49:18.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:18 smithi043 ceph-mon[29963]: 5.3 scrub ok 2024-03-30T13:49:18.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:18 smithi043 ceph-mon[29963]: pgmap v250: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T13:49:18.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:18 smithi043 ceph-mon[29963]: 4.a scrub starts 2024-03-30T13:49:18.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:18 smithi043 ceph-mon[29963]: 4.a scrub ok 2024-03-30T13:49:18.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:18 smithi043 ceph-mon[42099]: 5.3 scrub starts 2024-03-30T13:49:18.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:18 smithi043 ceph-mon[42099]: 5.3 scrub ok 2024-03-30T13:49:18.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:18 smithi043 ceph-mon[42099]: pgmap v250: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T13:49:18.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:18 smithi043 ceph-mon[42099]: 4.a scrub starts 2024-03-30T13:49:18.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:18 smithi043 ceph-mon[42099]: 4.a scrub ok 2024-03-30T13:49:18.597 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:18.637 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:18.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:18.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:18.740 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:18.785 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:18.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:18.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:18.887 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:18.913 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:18.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:18.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:18.969 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 4 15 11877 11862 185.329 185.562 0.00786351 0.00538856 2024-03-30T13:49:18.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:18 smithi113 ceph-mon[34341]: 5.3 scrub starts 2024-03-30T13:49:18.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:18 smithi113 ceph-mon[34341]: 5.3 scrub ok 2024-03-30T13:49:18.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:18 smithi113 ceph-mon[34341]: pgmap v250: 148 pgs: 148 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T13:49:18.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:18 smithi113 ceph-mon[34341]: 4.a scrub starts 2024-03-30T13:49:18.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:18 smithi113 ceph-mon[34341]: 4.a scrub ok 2024-03-30T13:49:18.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:49:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:49:19.015 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:19.057 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:19.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:19.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:19.158 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:19.198 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:19.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:19.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:19.300 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:19.334 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:19.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:19.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:19.436 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:19.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:19.462 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:19.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:19.563 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:19.597 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:19.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:19.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:19.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:19 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:49:19.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:19 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:49:19.698 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:19.739 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:19.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:19.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:19.841 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:19.873 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:19 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:49:19.876 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:19.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:19.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:19.969 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 5 16 14659 14643 183.022 173.812 0.00455897 0.00545725 2024-03-30T13:49:19.977 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:20.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:20.009 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:20.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:20.110 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:20.136 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:20.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:20.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:20.238 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:20.276 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:20.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:20.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:20.378 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:20.410 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:20.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:20.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:20.512 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:20.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:20.547 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:20.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:20.648 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:20.680 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:20.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:20.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:20.763 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:20 smithi043 ceph-mon[29963]: 5.5 deep-scrub starts 2024-03-30T13:49:20.764 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:20 smithi043 ceph-mon[29963]: 5.5 deep-scrub ok 2024-03-30T13:49:20.764 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:20 smithi043 ceph-mon[29963]: pgmap v251: 148 pgs: 148 active+clean; 680 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-30T13:49:20.764 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:20 smithi043 ceph-mon[29963]: 5.b scrub starts 2024-03-30T13:49:20.764 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:20 smithi043 ceph-mon[29963]: 5.b scrub ok 2024-03-30T13:49:20.764 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:20 smithi043 ceph-mon[29963]: 3.4 scrub starts 2024-03-30T13:49:20.764 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:20 smithi043 ceph-mon[29963]: 3.4 scrub ok 2024-03-30T13:49:20.764 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:20 smithi043 ceph-mon[29963]: 1.0 scrub starts 2024-03-30T13:49:20.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:20 smithi043 ceph-mon[29963]: 1.0 scrub ok 2024-03-30T13:49:20.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:20 smithi043 ceph-mon[42099]: 5.5 deep-scrub starts 2024-03-30T13:49:20.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:20 smithi043 ceph-mon[42099]: 5.5 deep-scrub ok 2024-03-30T13:49:20.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:20 smithi043 ceph-mon[42099]: pgmap v251: 148 pgs: 148 active+clean; 680 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-30T13:49:20.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:20 smithi043 ceph-mon[42099]: 5.b scrub starts 2024-03-30T13:49:20.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:20 smithi043 ceph-mon[42099]: 5.b scrub ok 2024-03-30T13:49:20.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:20 smithi043 ceph-mon[42099]: 3.4 scrub starts 2024-03-30T13:49:20.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:20 smithi043 ceph-mon[42099]: 3.4 scrub ok 2024-03-30T13:49:20.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:20 smithi043 ceph-mon[42099]: 1.0 scrub starts 2024-03-30T13:49:20.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:20 smithi043 ceph-mon[42099]: 1.0 scrub ok 2024-03-30T13:49:20.783 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:20.786 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:49:20.823 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:20.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:20.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:20.825 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:49:20.866 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:49:20.913 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:49:20.926 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:20.956 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:20.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:20.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:20.959 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:49:20.969 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 6 16 17239 17223 179.391 161.25 0.00720234 0.00556503 2024-03-30T13:49:20.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:20 smithi113 ceph-mon[34341]: 5.5 deep-scrub starts 2024-03-30T13:49:20.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:20 smithi113 ceph-mon[34341]: 5.5 deep-scrub ok 2024-03-30T13:49:20.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:20 smithi113 ceph-mon[34341]: pgmap v251: 148 pgs: 148 active+clean; 680 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-30T13:49:20.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:20 smithi113 ceph-mon[34341]: 5.b scrub starts 2024-03-30T13:49:20.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:20 smithi113 ceph-mon[34341]: 5.b scrub ok 2024-03-30T13:49:20.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:20 smithi113 ceph-mon[34341]: 3.4 scrub starts 2024-03-30T13:49:20.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:20 smithi113 ceph-mon[34341]: 3.4 scrub ok 2024-03-30T13:49:20.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:20 smithi113 ceph-mon[34341]: 1.0 scrub starts 2024-03-30T13:49:20.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:20 smithi113 ceph-mon[34341]: 1.0 scrub ok 2024-03-30T13:49:21.001 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:49:21.044 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:49:21.058 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:21.083 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:49:21.087 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:49:21.087 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:49:21.087 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:49:21.087 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:49:21.087 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:49:21.087 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:49:21.088 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:49:21.088 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:49:21.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:21.095 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:21.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:21.197 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:21.233 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:21.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:21.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:21.334 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:21.375 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:21.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:21.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:21.477 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:21.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:21.513 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:21.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:21.615 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:21.648 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:21.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:21.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:21.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:21 smithi043 ceph-mon[29963]: 3.1e deep-scrub starts 2024-03-30T13:49:21.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:21 smithi043 ceph-mon[29963]: 3.1e deep-scrub ok 2024-03-30T13:49:21.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:21 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2395713484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:49:21.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:21 smithi043 ceph-mon[42099]: 3.1e deep-scrub starts 2024-03-30T13:49:21.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:21 smithi043 ceph-mon[42099]: 3.1e deep-scrub ok 2024-03-30T13:49:21.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:21 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2395713484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:49:21.750 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:21.781 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:21.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:21.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:21.882 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:21.913 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:21.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:21.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:21.969 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 7 16 19731 19715 176.012 155.75 0.00456067 0.00567509 2024-03-30T13:49:21.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:21 smithi113 ceph-mon[34341]: 3.1e deep-scrub starts 2024-03-30T13:49:21.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:21 smithi113 ceph-mon[34341]: 3.1e deep-scrub ok 2024-03-30T13:49:21.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:21 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2395713484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:49:22.016 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:22.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:22.060 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:22.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:22.161 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:22.173 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-30T13:49:22.199 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:22.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:22.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:22.302 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:22.333 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:22.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:22.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:22.435 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:22.475 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T13:49:22.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:22.487 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:22.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:22.588 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:22.615 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:22.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:22.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:22.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:22 smithi043 ceph-mon[29963]: pgmap v252: 148 pgs: 148 active+clean; 680 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-30T13:49:22.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:22 smithi043 ceph-mon[29963]: 6.1 scrub starts 2024-03-30T13:49:22.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:22 smithi043 ceph-mon[29963]: 6.1 scrub ok 2024-03-30T13:49:22.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:22 smithi043 ceph-mon[29963]: 6.3 scrub starts 2024-03-30T13:49:22.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:22 smithi043 ceph-mon[29963]: 6.3 scrub ok 2024-03-30T13:49:22.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:22 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:49:22.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:22 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:49:22.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:22 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:49:22.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:22 smithi043 ceph-mon[29963]: osdmap e154: 8 total, 8 up, 8 in 2024-03-30T13:49:22.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:22 smithi043 ceph-mon[42099]: pgmap v252: 148 pgs: 148 active+clean; 680 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-30T13:49:22.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:22 smithi043 ceph-mon[42099]: 6.1 scrub starts 2024-03-30T13:49:22.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:22 smithi043 ceph-mon[42099]: 6.1 scrub ok 2024-03-30T13:49:22.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:22 smithi043 ceph-mon[42099]: 6.3 scrub starts 2024-03-30T13:49:22.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:22 smithi043 ceph-mon[42099]: 6.3 scrub ok 2024-03-30T13:49:22.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:22 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:49:22.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:22 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:49:22.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:22 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:49:22.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:22 smithi043 ceph-mon[42099]: osdmap e154: 8 total, 8 up, 8 in 2024-03-30T13:49:22.717 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:22.761 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:22.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:22.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:22.863 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:22.896 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:22.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:22.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:22.969 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 8 16 22038 22022 172.033 144.188 0.00640826 0.00580797 2024-03-30T13:49:22.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:22 smithi113 ceph-mon[34341]: pgmap v252: 148 pgs: 148 active+clean; 680 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-30T13:49:22.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:22 smithi113 ceph-mon[34341]: 6.1 scrub starts 2024-03-30T13:49:22.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:22 smithi113 ceph-mon[34341]: 6.1 scrub ok 2024-03-30T13:49:22.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:22 smithi113 ceph-mon[34341]: 6.3 scrub starts 2024-03-30T13:49:22.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:22 smithi113 ceph-mon[34341]: 6.3 scrub ok 2024-03-30T13:49:22.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:22 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:49:22.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:22 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:49:22.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:22 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:49:22.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:22 smithi113 ceph-mon[34341]: osdmap e154: 8 total, 8 up, 8 in 2024-03-30T13:49:22.998 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:23.035 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:23.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:23.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:23.137 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:23.171 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:23.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:23.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:23.273 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:23.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:23.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:23.398 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:23.433 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:23.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:23.536 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:23.574 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:23.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:23.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:23.676 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:23.716 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:23.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:23.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:23.818 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:23.859 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:23.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:23.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:23.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:23 smithi043 ceph-mon[42099]: 4.1e deep-scrub starts 2024-03-30T13:49:23.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:23 smithi043 ceph-mon[42099]: 4.1e deep-scrub ok 2024-03-30T13:49:23.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:23 smithi043 ceph-mon[42099]: 5.8 scrub starts 2024-03-30T13:49:23.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:23 smithi043 ceph-mon[42099]: 5.8 scrub ok 2024-03-30T13:49:23.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:23 smithi043 ceph-mon[29963]: 4.1e deep-scrub starts 2024-03-30T13:49:23.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:23 smithi043 ceph-mon[29963]: 4.1e deep-scrub ok 2024-03-30T13:49:23.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:23 smithi043 ceph-mon[29963]: 5.8 scrub starts 2024-03-30T13:49:23.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:23 smithi043 ceph-mon[29963]: 5.8 scrub ok 2024-03-30T13:49:23.961 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:23.969 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 9 15 24332 24317 168.854 143.438 0.0122815 0.00591493 2024-03-30T13:49:23.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:23 smithi113 ceph-mon[34341]: 4.1e deep-scrub starts 2024-03-30T13:49:23.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:23 smithi113 ceph-mon[34341]: 4.1e deep-scrub ok 2024-03-30T13:49:23.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:23 smithi113 ceph-mon[34341]: 5.8 scrub starts 2024-03-30T13:49:23.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:23 smithi113 ceph-mon[34341]: 5.8 scrub ok 2024-03-30T13:49:24.018 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:24.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:24.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:24.056 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:49:24.121 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:24.138 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:49:24.165 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:49:24.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:24.216 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:24.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:24.317 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:24.347 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:24.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:24.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:24.449 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:24.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:24.482 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:24.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:24.581 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:24 smithi043 ceph-mon[29963]: 4.b scrub starts 2024-03-30T13:49:24.581 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:24 smithi043 ceph-mon[29963]: 4.b scrub ok 2024-03-30T13:49:24.581 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:24 smithi043 ceph-mon[29963]: 5.1 scrub starts 2024-03-30T13:49:24.581 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:24 smithi043 ceph-mon[29963]: 5.1 scrub ok 2024-03-30T13:49:24.581 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:24 smithi043 ceph-mon[29963]: pgmap v254: 148 pgs: 148 active+clean; 828 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 740 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-30T13:49:24.581 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:24 smithi043 ceph-mon[29963]: 6.19 scrub starts 2024-03-30T13:49:24.581 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:24 smithi043 ceph-mon[29963]: 6.19 scrub ok 2024-03-30T13:49:24.582 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:24 smithi043 ceph-mon[29963]: 5.14 scrub starts 2024-03-30T13:49:24.582 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:24 smithi043 ceph-mon[29963]: 5.14 scrub ok 2024-03-30T13:49:24.582 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:24 smithi043 ceph-mon[42099]: 4.b scrub starts 2024-03-30T13:49:24.582 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:24 smithi043 ceph-mon[42099]: 4.b scrub ok 2024-03-30T13:49:24.582 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:24 smithi043 ceph-mon[42099]: 5.1 scrub starts 2024-03-30T13:49:24.582 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:24 smithi043 ceph-mon[42099]: 5.1 scrub ok 2024-03-30T13:49:24.583 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:24 smithi043 ceph-mon[42099]: pgmap v254: 148 pgs: 148 active+clean; 828 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 740 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-30T13:49:24.583 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:24 smithi043 ceph-mon[42099]: 6.19 scrub starts 2024-03-30T13:49:24.583 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:24 smithi043 ceph-mon[42099]: 6.19 scrub ok 2024-03-30T13:49:24.583 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:24 smithi043 ceph-mon[42099]: 5.14 scrub starts 2024-03-30T13:49:24.583 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:24 smithi043 ceph-mon[42099]: 5.14 scrub ok 2024-03-30T13:49:24.583 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:24.615 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:24.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:24.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:24.717 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:24.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:24.767 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:24.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:24.868 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:24.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:24.908 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:24.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:24.969 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 10 16 26380 26364 164.762 127.938 0.00198001 0.00605545 2024-03-30T13:49:24.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:24 smithi113 ceph-mon[34341]: 4.b scrub starts 2024-03-30T13:49:24.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:24 smithi113 ceph-mon[34341]: 4.b scrub ok 2024-03-30T13:49:24.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:24 smithi113 ceph-mon[34341]: 5.1 scrub starts 2024-03-30T13:49:24.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:24 smithi113 ceph-mon[34341]: 5.1 scrub ok 2024-03-30T13:49:24.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:24 smithi113 ceph-mon[34341]: pgmap v254: 148 pgs: 148 active+clean; 828 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 740 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-30T13:49:24.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:24 smithi113 ceph-mon[34341]: 6.19 scrub starts 2024-03-30T13:49:24.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:24 smithi113 ceph-mon[34341]: 6.19 scrub ok 2024-03-30T13:49:24.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:24 smithi113 ceph-mon[34341]: 5.14 scrub starts 2024-03-30T13:49:24.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:24 smithi113 ceph-mon[34341]: 5.14 scrub ok 2024-03-30T13:49:25.010 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:25.043 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:25.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:25.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:25.144 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:25.176 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:25.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:25.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:25.278 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:25.305 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:25.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:25.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:25.407 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:25.438 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:25.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:25.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:25.540 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:25.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:25.564 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:25.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:25.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:25 smithi043 ceph-mon[29963]: 4.e scrub starts 2024-03-30T13:49:25.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:25 smithi043 ceph-mon[29963]: 4.e scrub ok 2024-03-30T13:49:25.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:25 smithi043 ceph-mon[29963]: 6.11 deep-scrub starts 2024-03-30T13:49:25.604 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:25 smithi043 ceph-mon[42099]: 4.e scrub starts 2024-03-30T13:49:25.604 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:25 smithi043 ceph-mon[42099]: 4.e scrub ok 2024-03-30T13:49:25.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:25 smithi043 ceph-mon[42099]: 6.11 deep-scrub starts 2024-03-30T13:49:25.605 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:25 smithi043 ceph-mon[29963]: 6.11 deep-scrub ok 2024-03-30T13:49:25.665 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:25.704 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:25.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:25.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:25.806 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:25.838 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:25.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:25.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:25.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:25 smithi043 ceph-mon[29963]: 2.1 scrub starts 2024-03-30T13:49:25.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:25 smithi043 ceph-mon[29963]: 2.1 scrub ok 2024-03-30T13:49:25.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:25 smithi043 ceph-mon[42099]: 6.11 deep-scrub ok 2024-03-30T13:49:25.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:25 smithi043 ceph-mon[42099]: 2.1 scrub starts 2024-03-30T13:49:25.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:25 smithi043 ceph-mon[42099]: 2.1 scrub ok 2024-03-30T13:49:25.940 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:25.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:25.966 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:25.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:25.969 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 11 15 28102 28087 159.572 107.688 0.1039 0.00624746 2024-03-30T13:49:25.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:25 smithi113 ceph-mon[34341]: 4.e scrub starts 2024-03-30T13:49:25.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:25 smithi113 ceph-mon[34341]: 4.e scrub ok 2024-03-30T13:49:25.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:25 smithi113 ceph-mon[34341]: 6.11 deep-scrub starts 2024-03-30T13:49:25.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:25 smithi113 ceph-mon[34341]: 6.11 deep-scrub ok 2024-03-30T13:49:25.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:25 smithi113 ceph-mon[34341]: 2.1 scrub starts 2024-03-30T13:49:25.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:25 smithi113 ceph-mon[34341]: 2.1 scrub ok 2024-03-30T13:49:26.067 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:26.099 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:26.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:26.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:26.201 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:26.240 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:26.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:26.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:26.342 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:26.376 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:26.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:26.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:26.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:49:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:49:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:49:26.478 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:26.507 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:26.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:26.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:26.608 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:26.642 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:26.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:26.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:26.743 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:26.783 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:26.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:26.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:26.885 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:26 smithi043 ceph-mon[29963]: 4.16 scrub starts 2024-03-30T13:49:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:26 smithi043 ceph-mon[29963]: 4.16 scrub ok 2024-03-30T13:49:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:26 smithi043 ceph-mon[29963]: 6.1a scrub starts 2024-03-30T13:49:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:26 smithi043 ceph-mon[29963]: 6.1a scrub ok 2024-03-30T13:49:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:26 smithi043 ceph-mon[29963]: pgmap v255: 148 pgs: 148 active+clean; 1.4 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 183 MiB/s wr, 2.93k op/s 2024-03-30T13:49:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:26 smithi043 ceph-mon[29963]: 5.4 scrub starts 2024-03-30T13:49:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:26 smithi043 ceph-mon[29963]: 5.4 scrub ok 2024-03-30T13:49:26.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:26 smithi043 ceph-mon[42099]: 4.16 scrub starts 2024-03-30T13:49:26.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:26 smithi043 ceph-mon[42099]: 4.16 scrub ok 2024-03-30T13:49:26.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:26 smithi043 ceph-mon[42099]: 6.1a scrub starts 2024-03-30T13:49:26.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:26 smithi043 ceph-mon[42099]: 6.1a scrub ok 2024-03-30T13:49:26.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:26 smithi043 ceph-mon[42099]: pgmap v255: 148 pgs: 148 active+clean; 1.4 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 183 MiB/s wr, 2.93k op/s 2024-03-30T13:49:26.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:26 smithi043 ceph-mon[42099]: 5.4 scrub starts 2024-03-30T13:49:26.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:26 smithi043 ceph-mon[42099]: 5.4 scrub ok 2024-03-30T13:49:26.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:26.924 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:26.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:26.969 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 12 16 30053 30037 156.43 121.875 0.00838568 0.00638682 2024-03-30T13:49:26.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:26 smithi113 ceph-mon[34341]: 4.16 scrub starts 2024-03-30T13:49:26.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:26 smithi113 ceph-mon[34341]: 4.16 scrub ok 2024-03-30T13:49:26.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:26 smithi113 ceph-mon[34341]: 6.1a scrub starts 2024-03-30T13:49:26.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:26 smithi113 ceph-mon[34341]: 6.1a scrub ok 2024-03-30T13:49:26.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:26 smithi113 ceph-mon[34341]: pgmap v255: 148 pgs: 148 active+clean; 1.4 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 183 MiB/s wr, 2.93k op/s 2024-03-30T13:49:26.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:26 smithi113 ceph-mon[34341]: 5.4 scrub starts 2024-03-30T13:49:26.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:26 smithi113 ceph-mon[34341]: 5.4 scrub ok 2024-03-30T13:49:27.026 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:27.063 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:27.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:27.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:27.165 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:27.201 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:27.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:27.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:27.303 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:27.337 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:27.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:27.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:27.439 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:27.469 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:27.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:27.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:27.572 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:27.605 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:27.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:27.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:27.706 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:27.730 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:27.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:27.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:27.831 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:27.866 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:27.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:27.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:27.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:27 smithi043 ceph-mon[29963]: 4.10 scrub starts 2024-03-30T13:49:27.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:27 smithi043 ceph-mon[29963]: 4.10 scrub ok 2024-03-30T13:49:27.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:27 smithi043 ceph-mon[29963]: 3.17 scrub starts 2024-03-30T13:49:27.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:27 smithi043 ceph-mon[29963]: 3.17 scrub ok 2024-03-30T13:49:27.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:27 smithi043 ceph-mon[29963]: 4.1 scrub starts 2024-03-30T13:49:27.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:27 smithi043 ceph-mon[29963]: 4.1 scrub ok 2024-03-30T13:49:27.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:27 smithi043 ceph-mon[29963]: 4.12 scrub starts 2024-03-30T13:49:27.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:27 smithi043 ceph-mon[29963]: 4.12 scrub ok 2024-03-30T13:49:27.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:27 smithi043 ceph-mon[29963]: osdmap e155: 8 total, 8 up, 8 in 2024-03-30T13:49:27.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:27 smithi043 ceph-mon[42099]: 4.10 scrub starts 2024-03-30T13:49:27.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:27 smithi043 ceph-mon[42099]: 4.10 scrub ok 2024-03-30T13:49:27.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:27 smithi043 ceph-mon[42099]: 3.17 scrub starts 2024-03-30T13:49:27.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:27 smithi043 ceph-mon[42099]: 3.17 scrub ok 2024-03-30T13:49:27.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:27 smithi043 ceph-mon[42099]: 4.1 scrub starts 2024-03-30T13:49:27.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:27 smithi043 ceph-mon[42099]: 4.1 scrub ok 2024-03-30T13:49:27.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:27 smithi043 ceph-mon[42099]: 4.12 scrub starts 2024-03-30T13:49:27.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:27 smithi043 ceph-mon[42099]: 4.12 scrub ok 2024-03-30T13:49:27.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:27 smithi043 ceph-mon[42099]: osdmap e155: 8 total, 8 up, 8 in 2024-03-30T13:49:27.969 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:27.971 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 13 16 31742 31726 152.517 105.562 0.00528343 0.006551 2024-03-30T13:49:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:27 smithi113 ceph-mon[34341]: 4.10 scrub starts 2024-03-30T13:49:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:27 smithi113 ceph-mon[34341]: 4.10 scrub ok 2024-03-30T13:49:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:27 smithi113 ceph-mon[34341]: 3.17 scrub starts 2024-03-30T13:49:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:27 smithi113 ceph-mon[34341]: 3.17 scrub ok 2024-03-30T13:49:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:27 smithi113 ceph-mon[34341]: 4.1 scrub starts 2024-03-30T13:49:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:27 smithi113 ceph-mon[34341]: 4.1 scrub ok 2024-03-30T13:49:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:27 smithi113 ceph-mon[34341]: 4.12 scrub starts 2024-03-30T13:49:27.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:27 smithi113 ceph-mon[34341]: 4.12 scrub ok 2024-03-30T13:49:27.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:27 smithi113 ceph-mon[34341]: osdmap e155: 8 total, 8 up, 8 in 2024-03-30T13:49:28.001 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:28.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:28.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:28.103 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:28.143 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:28.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:28.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:28.245 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:28.293 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:28.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:28.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:28.395 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:28.430 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:28.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:28.531 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:28.575 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:28.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:28.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:28.677 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:28.715 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:28.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:28.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:28.817 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:28.856 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:28.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:28.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:28.870 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[29963]: pgmap v256: 148 pgs: 148 active+clean; 1.4 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 147 MiB/s wr, 2.35k op/s 2024-03-30T13:49:28.870 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[29963]: 3.1c scrub starts 2024-03-30T13:49:28.871 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[29963]: 3.1c scrub ok 2024-03-30T13:49:28.871 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[29963]: 5.9 scrub starts 2024-03-30T13:49:28.871 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[29963]: 5.9 scrub ok 2024-03-30T13:49:28.871 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:49:28.871 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:49:28.871 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[29963]: 5.1e scrub starts 2024-03-30T13:49:28.871 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[29963]: 5.1e scrub ok 2024-03-30T13:49:28.871 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[29963]: 5.6 deep-scrub starts 2024-03-30T13:49:28.871 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[29963]: 5.6 deep-scrub ok 2024-03-30T13:49:28.871 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:49:28.871 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:49:28.872 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[42099]: pgmap v256: 148 pgs: 148 active+clean; 1.4 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 147 MiB/s wr, 2.35k op/s 2024-03-30T13:49:28.872 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[42099]: 3.1c scrub starts 2024-03-30T13:49:28.872 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[42099]: 3.1c scrub ok 2024-03-30T13:49:28.872 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[42099]: 5.9 scrub starts 2024-03-30T13:49:28.872 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[42099]: 5.9 scrub ok 2024-03-30T13:49:28.872 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:49:28.872 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:49:28.872 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[42099]: 5.1e scrub starts 2024-03-30T13:49:28.872 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[42099]: 5.1e scrub ok 2024-03-30T13:49:28.872 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[42099]: 5.6 deep-scrub starts 2024-03-30T13:49:28.872 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[42099]: 5.6 deep-scrub ok 2024-03-30T13:49:28.872 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:49:28.872 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:28 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:49:28.958 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:28.970 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 14 15 33802 33787 150.823 128.812 0.00678835 0.00662437 2024-03-30T13:49:28.972 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:49:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:49:28.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:28 smithi113 ceph-mon[34341]: pgmap v256: 148 pgs: 148 active+clean; 1.4 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 147 MiB/s wr, 2.35k op/s 2024-03-30T13:49:28.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:28 smithi113 ceph-mon[34341]: 3.1c scrub starts 2024-03-30T13:49:28.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:28 smithi113 ceph-mon[34341]: 3.1c scrub ok 2024-03-30T13:49:28.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:28 smithi113 ceph-mon[34341]: 5.9 scrub starts 2024-03-30T13:49:28.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:28 smithi113 ceph-mon[34341]: 5.9 scrub ok 2024-03-30T13:49:28.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:49:28.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:49:28.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:28 smithi113 ceph-mon[34341]: 5.1e scrub starts 2024-03-30T13:49:28.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:28 smithi113 ceph-mon[34341]: 5.1e scrub ok 2024-03-30T13:49:28.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:28 smithi113 ceph-mon[34341]: 5.6 deep-scrub starts 2024-03-30T13:49:28.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:28 smithi113 ceph-mon[34341]: 5.6 deep-scrub ok 2024-03-30T13:49:28.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:28 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:49:28.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:28 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:49:28.990 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:28.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:28.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:29.092 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:29.135 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:29.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:29.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:29.206 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 reset_purged_snaps_last 2024-03-30T13:49:29.237 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:29.298 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:29.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:29.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:29.400 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:29.433 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:29.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:29.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:29.535 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:29.565 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:29.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:29.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:29.667 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:29.695 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:29.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:29.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:29.797 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:29.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:29.831 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:29.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:29.932 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:29.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:29.965 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:29.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:29.970 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 15 15 35721 35706 148.763 119.938 0.00833951 0.0067165 2024-03-30T13:49:30.066 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:30.098 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:30.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:30.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:30.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:29 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:49:30.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:29 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:49:30.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:29 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:49:30.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:29 smithi043 ceph-mon[29963]: 6.0 scrub starts 2024-03-30T13:49:30.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:29 smithi043 ceph-mon[29963]: 6.0 scrub ok 2024-03-30T13:49:30.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:29 smithi043 ceph-mon[29963]: osdmap e156: 8 total, 8 up, 8 in 2024-03-30T13:49:30.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:29 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:49:30.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:29 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:49:30.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:29 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:49:30.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:29 smithi043 ceph-mon[42099]: 6.0 scrub starts 2024-03-30T13:49:30.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:29 smithi043 ceph-mon[42099]: 6.0 scrub ok 2024-03-30T13:49:30.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:29 smithi043 ceph-mon[42099]: osdmap e156: 8 total, 8 up, 8 in 2024-03-30T13:49:30.200 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:30.213 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:49:30.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:29 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:49:30.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:29 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:49:30.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:29 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:49:30.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:29 smithi113 ceph-mon[34341]: 6.0 scrub starts 2024-03-30T13:49:30.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:29 smithi113 ceph-mon[34341]: 6.0 scrub ok 2024-03-30T13:49:30.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:29 smithi113 ceph-mon[34341]: osdmap e156: 8 total, 8 up, 8 in 2024-03-30T13:49:30.233 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:30.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:30.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:30.334 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:30.369 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:30.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:30.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:30.470 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:30.505 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:30.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:30.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:30.607 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:30.639 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:30.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:30.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:30.741 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:30.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:30.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:30.881 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:30.937 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:49:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:30.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:30.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:30.971 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 16 16 37598 37582 146.788 117.25 0.0133975 0.00680685 2024-03-30T13:49:31.047 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:30 smithi043 ceph-mon[29963]: pgmap v258: 148 pgs: 148 active+clean; 2.0 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 173 MiB/s wr, 2.77k op/s 2024-03-30T13:49:31.047 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:30 smithi043 ceph-mon[29963]: 6.17 scrub starts 2024-03-30T13:49:31.047 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:30 smithi043 ceph-mon[29963]: 6.17 scrub ok 2024-03-30T13:49:31.047 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:30 smithi043 ceph-mon[42099]: pgmap v258: 148 pgs: 148 active+clean; 2.0 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 173 MiB/s wr, 2.77k op/s 2024-03-30T13:49:31.047 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:30 smithi043 ceph-mon[42099]: 6.17 scrub starts 2024-03-30T13:49:31.048 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:30 smithi043 ceph-mon[42099]: 6.17 scrub ok 2024-03-30T13:49:31.066 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:31.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:31.113 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:31.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:31.214 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:31.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:30 smithi113 ceph-mon[34341]: pgmap v258: 148 pgs: 148 active+clean; 2.0 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 173 MiB/s wr, 2.77k op/s 2024-03-30T13:49:31.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:30 smithi113 ceph-mon[34341]: 6.17 scrub starts 2024-03-30T13:49:31.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:30 smithi113 ceph-mon[34341]: 6.17 scrub ok 2024-03-30T13:49:31.251 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:31.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:31.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:31.353 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:31.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:31.399 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:31.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:31.500 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:31.530 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:31.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:31.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:31.632 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:31.665 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:31.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:31.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:31.767 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:31.804 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:31.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:31.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:31.906 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:31.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:31.936 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:31.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:31.970 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 17 16 39247 39231 144.216 103.062 0.0043849 0.00692912 2024-03-30T13:49:32.037 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:32.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:32.085 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:32.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:32.186 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:32.216 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:32.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:32.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:31 smithi113 ceph-mon[34341]: 5.1d deep-scrub starts 2024-03-30T13:49:32.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:31 smithi113 ceph-mon[34341]: 5.1d deep-scrub ok 2024-03-30T13:49:32.318 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:32.357 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:31 smithi043 ceph-mon[29963]: 5.1d deep-scrub starts 2024-03-30T13:49:32.357 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:31 smithi043 ceph-mon[29963]: 5.1d deep-scrub ok 2024-03-30T13:49:32.358 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:31 smithi043 ceph-mon[42099]: 5.1d deep-scrub starts 2024-03-30T13:49:32.358 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:31 smithi043 ceph-mon[42099]: 5.1d deep-scrub ok 2024-03-30T13:49:32.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:32.359 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:32.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:32.460 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:32.497 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:32.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:32.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:32.599 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:32.639 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:32.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:32.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:32.742 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:32.781 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:32.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:32.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:32.883 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:32.926 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:32.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:32.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:32.972 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 18 16 40986 40970 142.232 108.688 0.00745883 0.00702515 2024-03-30T13:49:33.028 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:33.071 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:33.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:33.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:33.173 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:33.206 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:33.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:33.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:33.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:32 smithi113 ceph-mon[34341]: pgmap v260: 148 pgs: 148 active+clean; 2.0 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 155 MiB/s wr, 2.48k op/s 2024-03-30T13:49:33.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:32 smithi113 ceph-mon[34341]: 4.7 scrub starts 2024-03-30T13:49:33.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:32 smithi113 ceph-mon[34341]: 4.7 scrub ok 2024-03-30T13:49:33.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:32 smithi113 ceph-mon[34341]: 5.7 scrub starts 2024-03-30T13:49:33.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:32 smithi113 ceph-mon[34341]: 5.7 scrub ok 2024-03-30T13:49:33.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:32 smithi113 ceph-mon[34341]: 6.8 scrub starts 2024-03-30T13:49:33.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:32 smithi113 ceph-mon[34341]: 6.8 scrub ok 2024-03-30T13:49:33.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:32 smithi113 ceph-mon[34341]: osdmap e157: 8 total, 8 up, 8 in 2024-03-30T13:49:33.307 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:33.321 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:32 smithi043 ceph-mon[29963]: pgmap v260: 148 pgs: 148 active+clean; 2.0 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 155 MiB/s wr, 2.48k op/s 2024-03-30T13:49:33.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:32 smithi043 ceph-mon[29963]: 4.7 scrub starts 2024-03-30T13:49:33.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:32 smithi043 ceph-mon[29963]: 4.7 scrub ok 2024-03-30T13:49:33.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:32 smithi043 ceph-mon[29963]: 5.7 scrub starts 2024-03-30T13:49:33.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:32 smithi043 ceph-mon[29963]: 5.7 scrub ok 2024-03-30T13:49:33.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:32 smithi043 ceph-mon[29963]: 6.8 scrub starts 2024-03-30T13:49:33.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:32 smithi043 ceph-mon[29963]: 6.8 scrub ok 2024-03-30T13:49:33.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:32 smithi043 ceph-mon[29963]: osdmap e157: 8 total, 8 up, 8 in 2024-03-30T13:49:33.322 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:32 smithi043 ceph-mon[42099]: pgmap v260: 148 pgs: 148 active+clean; 2.0 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 155 MiB/s wr, 2.48k op/s 2024-03-30T13:49:33.322 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:32 smithi043 ceph-mon[42099]: 4.7 scrub starts 2024-03-30T13:49:33.322 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:32 smithi043 ceph-mon[42099]: 4.7 scrub ok 2024-03-30T13:49:33.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:32 smithi043 ceph-mon[42099]: 5.7 scrub starts 2024-03-30T13:49:33.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:32 smithi043 ceph-mon[42099]: 5.7 scrub ok 2024-03-30T13:49:33.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:32 smithi043 ceph-mon[42099]: 6.8 scrub starts 2024-03-30T13:49:33.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:32 smithi043 ceph-mon[42099]: 6.8 scrub ok 2024-03-30T13:49:33.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:32 smithi043 ceph-mon[42099]: osdmap e157: 8 total, 8 up, 8 in 2024-03-30T13:49:33.339 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:33.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:33.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:33.441 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:33.472 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:33.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:33.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:33.575 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:33.625 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:33.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:33.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:33.727 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:33.762 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:33.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:33.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:33.863 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:33.892 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:33.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:33.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:33.972 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 19 15 42816 42801 140.769 114.438 0.00518426 0.00709809 2024-03-30T13:49:33.994 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:34.031 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:49:34.033 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:34.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:34.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:34.073 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:49:34.115 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:49:34.135 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:34.162 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:49:34.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:34.181 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:34.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:34.204 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:49:34.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:33 smithi043 ceph-mon[29963]: 6.15 scrub starts 2024-03-30T13:49:34.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:33 smithi043 ceph-mon[29963]: 6.15 scrub ok 2024-03-30T13:49:34.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:33 smithi043 ceph-mon[29963]: 4.1b scrub starts 2024-03-30T13:49:34.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:33 smithi043 ceph-mon[29963]: 4.1b scrub ok 2024-03-30T13:49:34.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:33 smithi043 ceph-mon[29963]: 3.b scrub starts 2024-03-30T13:49:34.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:33 smithi043 ceph-mon[29963]: 3.b scrub ok 2024-03-30T13:49:34.205 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:33 smithi043 ceph-mon[42099]: 6.15 scrub starts 2024-03-30T13:49:34.205 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:33 smithi043 ceph-mon[42099]: 6.15 scrub ok 2024-03-30T13:49:34.205 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:33 smithi043 ceph-mon[42099]: 4.1b scrub starts 2024-03-30T13:49:34.205 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:33 smithi043 ceph-mon[42099]: 4.1b scrub ok 2024-03-30T13:49:34.205 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:33 smithi043 ceph-mon[42099]: 3.b scrub starts 2024-03-30T13:49:34.205 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:33 smithi043 ceph-mon[42099]: 3.b scrub ok 2024-03-30T13:49:34.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:33 smithi113 ceph-mon[34341]: 6.15 scrub starts 2024-03-30T13:49:34.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:33 smithi113 ceph-mon[34341]: 6.15 scrub ok 2024-03-30T13:49:34.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:33 smithi113 ceph-mon[34341]: 4.1b scrub starts 2024-03-30T13:49:34.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:33 smithi113 ceph-mon[34341]: 4.1b scrub ok 2024-03-30T13:49:34.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:33 smithi113 ceph-mon[34341]: 3.b scrub starts 2024-03-30T13:49:34.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:33 smithi113 ceph-mon[34341]: 3.b scrub ok 2024-03-30T13:49:34.247 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:49:34.282 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:34.291 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:49:34.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:34.327 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:34.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:34.331 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:49:34.334 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:49:34.335 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:49:34.335 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:49:34.335 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:49:34.335 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:49:34.335 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:49:34.335 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:49:34.335 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:49:34.428 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:34.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:34.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:34.574 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:34.609 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:34.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:34.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:34.710 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:34.744 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:34.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:34.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:34.846 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:34.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:34.877 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:34.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:34.972 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-03-30T13:49:34.971435+0000 min lat: 0.00129646 max lat: 0.178336 avg lat: 0.00717853 2024-03-30T13:49:34.972 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T13:49:34.972 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 20 16 44568 44552 139.202 109.438 0.00797316 0.00717853 2024-03-30T13:49:34.978 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:35.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:35.009 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:35.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:35.072 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:49:35.072 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:49:35.110 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:35.141 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:35.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:35.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:35.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:34 smithi113 ceph-mon[34341]: pgmap v262: 148 pgs: 148 active+clean; 2.1 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-30T13:49:35.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:34 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3920736511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:49:35.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:34 smithi113 ceph-mon[34341]: 6.6 scrub starts 2024-03-30T13:49:35.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:34 smithi113 ceph-mon[34341]: 6.6 scrub ok 2024-03-30T13:49:35.229 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:34 smithi043 ceph-mon[29963]: pgmap v262: 148 pgs: 148 active+clean; 2.1 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-30T13:49:35.229 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:34 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3920736511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:49:35.229 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:34 smithi043 ceph-mon[29963]: 6.6 scrub starts 2024-03-30T13:49:35.230 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:34 smithi043 ceph-mon[29963]: 6.6 scrub ok 2024-03-30T13:49:35.230 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:34 smithi043 ceph-mon[42099]: pgmap v262: 148 pgs: 148 active+clean; 2.1 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-30T13:49:35.230 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:34 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3920736511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:49:35.230 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:34 smithi043 ceph-mon[42099]: 6.6 scrub starts 2024-03-30T13:49:35.230 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:34 smithi043 ceph-mon[42099]: 6.6 scrub ok 2024-03-30T13:49:35.243 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:35.274 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:35.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:35.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:35.376 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:35.416 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:35.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:35.518 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:35.553 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:35.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:35.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:35.655 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:35.693 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:35.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:35.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:35.795 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:35.837 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:35.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:35.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:35.939 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:35.973 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 21 15 46249 46234 137.575 105.125 0.0045752 0.00726199 2024-03-30T13:49:35.977 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:35.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:35.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:36.079 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:36.108 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:36.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:36.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:36.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:35 smithi043 ceph-mon[29963]: 2.2 scrub starts 2024-03-30T13:49:36.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:35 smithi043 ceph-mon[29963]: 2.2 scrub ok 2024-03-30T13:49:36.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:35 smithi043 ceph-mon[29963]: 5.2 scrub starts 2024-03-30T13:49:36.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:35 smithi043 ceph-mon[29963]: 5.2 scrub ok 2024-03-30T13:49:36.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:35 smithi043 ceph-mon[42099]: 2.2 scrub starts 2024-03-30T13:49:36.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:35 smithi043 ceph-mon[42099]: 2.2 scrub ok 2024-03-30T13:49:36.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:35 smithi043 ceph-mon[42099]: 5.2 scrub starts 2024-03-30T13:49:36.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:35 smithi043 ceph-mon[42099]: 5.2 scrub ok 2024-03-30T13:49:36.150 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:49:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:49:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:49:36.210 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:36.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:35 smithi113 ceph-mon[34341]: 2.2 scrub starts 2024-03-30T13:49:36.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:35 smithi113 ceph-mon[34341]: 2.2 scrub ok 2024-03-30T13:49:36.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:35 smithi113 ceph-mon[34341]: 5.2 scrub starts 2024-03-30T13:49:36.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:35 smithi113 ceph-mon[34341]: 5.2 scrub ok 2024-03-30T13:49:36.252 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:36.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:36.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:36.354 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:36.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:36.396 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:36.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:36.497 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:36.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:36.533 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:36.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:36.633 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:36.669 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:36.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:36.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:36.771 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:36.809 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-03-30T13:49:36.821 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:36.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:36.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:36.923 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:36.954 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:36.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:36.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:36.973 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 22 15 48047 48032 136.427 112.375 0.00600289 0.00732446 2024-03-30T13:49:37.027 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:36 smithi043 ceph-mon[29963]: pgmap v263: 148 pgs: 148 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 146 MiB/s wr, 2.34k op/s 2024-03-30T13:49:37.027 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:36 smithi043 ceph-mon[29963]: 3.11 scrub starts 2024-03-30T13:49:37.027 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:36 smithi043 ceph-mon[29963]: 3.11 scrub ok 2024-03-30T13:49:37.027 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:36 smithi043 ceph-mon[29963]: 5.d scrub starts 2024-03-30T13:49:37.027 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:36 smithi043 ceph-mon[29963]: 5.d scrub ok 2024-03-30T13:49:37.027 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:36 smithi043 ceph-mon[29963]: 3.7 scrub starts 2024-03-30T13:49:37.027 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:36 smithi043 ceph-mon[29963]: 3.7 scrub ok 2024-03-30T13:49:37.028 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:36 smithi043 ceph-mon[29963]: 5.10 scrub starts 2024-03-30T13:49:37.028 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:36 smithi043 ceph-mon[29963]: 5.10 scrub ok 2024-03-30T13:49:37.032 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:36 smithi043 ceph-mon[42099]: pgmap v263: 148 pgs: 148 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 146 MiB/s wr, 2.34k op/s 2024-03-30T13:49:37.032 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:36 smithi043 ceph-mon[42099]: 3.11 scrub starts 2024-03-30T13:49:37.032 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:36 smithi043 ceph-mon[42099]: 3.11 scrub ok 2024-03-30T13:49:37.032 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:36 smithi043 ceph-mon[42099]: 5.d scrub starts 2024-03-30T13:49:37.032 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:36 smithi043 ceph-mon[42099]: 5.d scrub ok 2024-03-30T13:49:37.032 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:36 smithi043 ceph-mon[42099]: 3.7 scrub starts 2024-03-30T13:49:37.032 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:36 smithi043 ceph-mon[42099]: 3.7 scrub ok 2024-03-30T13:49:37.032 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:36 smithi043 ceph-mon[42099]: 5.10 scrub starts 2024-03-30T13:49:37.032 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:36 smithi043 ceph-mon[42099]: 5.10 scrub ok 2024-03-30T13:49:37.057 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:37.082 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 reset_purged_snaps_last 2024-03-30T13:49:37.105 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:37.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:37.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:37.207 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:37.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:36 smithi113 ceph-mon[34341]: pgmap v263: 148 pgs: 148 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 146 MiB/s wr, 2.34k op/s 2024-03-30T13:49:37.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:36 smithi113 ceph-mon[34341]: 3.11 scrub starts 2024-03-30T13:49:37.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:36 smithi113 ceph-mon[34341]: 3.11 scrub ok 2024-03-30T13:49:37.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:36 smithi113 ceph-mon[34341]: 5.d scrub starts 2024-03-30T13:49:37.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:36 smithi113 ceph-mon[34341]: 5.d scrub ok 2024-03-30T13:49:37.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:36 smithi113 ceph-mon[34341]: 3.7 scrub starts 2024-03-30T13:49:37.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:36 smithi113 ceph-mon[34341]: 3.7 scrub ok 2024-03-30T13:49:37.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:36 smithi113 ceph-mon[34341]: 5.10 scrub starts 2024-03-30T13:49:37.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:36 smithi113 ceph-mon[34341]: 5.10 scrub ok 2024-03-30T13:49:37.238 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:37.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:37.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:37.340 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:37.363 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:37.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:37.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:37.465 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:37.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:37.498 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:37.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:37.599 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:37.645 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:37.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:37.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:37.748 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:37.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:37.783 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:37.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:37.884 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:37.921 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:37.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:37.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:37.973 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 23 16 49423 49407 134.232 85.9375 0.00651358 0.00744433 2024-03-30T13:49:38.024 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:38.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:38.060 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:38.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:38.161 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:38.197 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:38.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:38.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:38.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:37 smithi113 ceph-mon[34341]: 5.11 scrub starts 2024-03-30T13:49:38.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:37 smithi113 ceph-mon[34341]: 5.11 scrub ok 2024-03-30T13:49:38.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:37 smithi113 ceph-mon[34341]: osdmap e158: 8 total, 8 up, 8 in 2024-03-30T13:49:38.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:37 smithi113 ceph-mon[34341]: 3.e scrub starts 2024-03-30T13:49:38.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:37 smithi113 ceph-mon[34341]: 3.e scrub ok 2024-03-30T13:49:38.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:37 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/739741608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:49:38.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:37 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:49:38.299 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:38.330 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:38.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:38.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:38.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:37 smithi043 ceph-mon[29963]: 5.11 scrub starts 2024-03-30T13:49:38.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:37 smithi043 ceph-mon[29963]: 5.11 scrub ok 2024-03-30T13:49:38.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:37 smithi043 ceph-mon[29963]: osdmap e158: 8 total, 8 up, 8 in 2024-03-30T13:49:38.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:37 smithi043 ceph-mon[29963]: 3.e scrub starts 2024-03-30T13:49:38.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:37 smithi043 ceph-mon[29963]: 3.e scrub ok 2024-03-30T13:49:38.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:37 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/739741608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:49:38.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:37 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:49:38.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:37 smithi043 ceph-mon[42099]: 5.11 scrub starts 2024-03-30T13:49:38.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:37 smithi043 ceph-mon[42099]: 5.11 scrub ok 2024-03-30T13:49:38.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:37 smithi043 ceph-mon[42099]: osdmap e158: 8 total, 8 up, 8 in 2024-03-30T13:49:38.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:37 smithi043 ceph-mon[42099]: 3.e scrub starts 2024-03-30T13:49:38.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:37 smithi043 ceph-mon[42099]: 3.e scrub ok 2024-03-30T13:49:38.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:37 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/739741608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:49:38.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:37 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:49:38.432 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:38.443 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-03-30T13:49:38.477 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:38.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:38.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:38.579 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:38.613 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:38.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:38.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:38.714 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:38.745 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:38.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:38.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:38.846 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:38.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:38.890 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:38.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:38.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:49:38 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:49:38.944 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:38 smithi043 ceph-mon[29963]: pgmap v265: 148 pgs: 148 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 640 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-30T13:49:38.944 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:38 smithi043 ceph-mon[29963]: 5.c scrub starts 2024-03-30T13:49:38.944 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:38 smithi043 ceph-mon[29963]: 5.c scrub ok 2024-03-30T13:49:38.944 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:38 smithi043 ceph-mon[29963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:49:38.944 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:38 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:49:38.944 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:38 smithi043 ceph-mon[29963]: osdmap e159: 8 total, 8 up, 8 in 2024-03-30T13:49:38.973 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 24 16 51158 51142 133.157 108.438 0.0100391 0.00750491 2024-03-30T13:49:38.991 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:39.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:39.030 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:39.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:39.132 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:39.141 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:49:39.147 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:49:39.185 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:39.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:39.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:39.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:38 smithi113 ceph-mon[34341]: pgmap v265: 148 pgs: 148 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 640 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-30T13:49:39.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:38 smithi113 ceph-mon[34341]: 5.c scrub starts 2024-03-30T13:49:39.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:38 smithi113 ceph-mon[34341]: 5.c scrub ok 2024-03-30T13:49:39.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:38 smithi113 ceph-mon[34341]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:49:39.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:38 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:49:39.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:38 smithi113 ceph-mon[34341]: osdmap e159: 8 total, 8 up, 8 in 2024-03-30T13:49:39.257 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:38 smithi043 ceph-mon[42099]: pgmap v265: 148 pgs: 148 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 640 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-30T13:49:39.258 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:38 smithi043 ceph-mon[42099]: 5.c scrub starts 2024-03-30T13:49:39.258 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:38 smithi043 ceph-mon[42099]: 5.c scrub ok 2024-03-30T13:49:39.258 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:38 smithi043 ceph-mon[42099]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:49:39.258 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:38 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:49:39.258 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:38 smithi043 ceph-mon[42099]: osdmap e159: 8 total, 8 up, 8 in 2024-03-30T13:49:39.286 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:39.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:39.328 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:39.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:39.429 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:39.452 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:39.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:39.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:39.554 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:39.594 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:39.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:39.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:39.696 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:39.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:39.733 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:39.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:39.834 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:39.874 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:39.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:39.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:39.973 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 25 16 52726 52710 131.75 98 0.00532004 0.00758489 2024-03-30T13:49:39.977 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:40.009 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:40.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:40.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:40.110 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:40.152 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:40.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:40.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:40.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:39 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:49:40.254 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:40.286 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:40.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:40.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:40.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:39 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:49:40.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:39 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:49:40.389 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:40.421 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:40.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:40.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:40.522 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:40.551 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:40.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:40.653 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:40.685 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:40.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:40.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:40.787 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:40.817 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:40.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:40.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:40.856 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:49:40.919 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:40.962 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:40.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:40.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:40.973 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 26 16 54321 54305 130.516 99.6875 0.00419398 0.00765468 2024-03-30T13:49:41.064 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:41.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:41.108 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:41.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:41.209 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:41.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:40 smithi113 ceph-mon[34341]: pgmap v267: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 156 MiB/s wr, 2.50k op/s 2024-03-30T13:49:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:41.263 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:41.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:41.364 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:41.379 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:40 smithi043 ceph-mon[29963]: pgmap v267: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 156 MiB/s wr, 2.50k op/s 2024-03-30T13:49:41.379 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:40 smithi043 ceph-mon[42099]: pgmap v267: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 156 MiB/s wr, 2.50k op/s 2024-03-30T13:49:41.410 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:41.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:41.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:41.512 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:41.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:41.551 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:41.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:41.652 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:41.690 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:41.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:41.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:41.792 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:41.828 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:41.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:41.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:41.929 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:41.963 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:41.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:41.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:41.974 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 27 16 55880 55864 129.291 97.4375 0.0107717 0.0077282 2024-03-30T13:49:42.036 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T13:49:42.065 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:42.076 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:41 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:49:42.077 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:41 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:49:42.077 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:41 smithi043 ceph-mon[29963]: osdmap e160: 8 total, 8 up, 8 in 2024-03-30T13:49:42.077 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:41 smithi043 ceph-mon[29963]: pgmap v269: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-03-30T13:49:42.077 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:41 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:49:42.077 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:41 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:49:42.077 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:41 smithi043 ceph-mon[42099]: osdmap e160: 8 total, 8 up, 8 in 2024-03-30T13:49:42.077 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:41 smithi043 ceph-mon[42099]: pgmap v269: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-03-30T13:49:42.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:42.119 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:42.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:42.220 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:42.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:41 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:49:42.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:41 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:49:42.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:41 smithi113 ceph-mon[34341]: osdmap e160: 8 total, 8 up, 8 in 2024-03-30T13:49:42.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:41 smithi113 ceph-mon[34341]: pgmap v269: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-03-30T13:49:42.255 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:42.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:42.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:42.357 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:42.393 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:42.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:42.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:42.494 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:42.536 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:42.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:42.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:42.637 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:42.670 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:42.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:42.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:42.772 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:42.804 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:42.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:42.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:42.905 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:42.956 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:42.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:42.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:42.973 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 28 16 57410 57394 128.089 95.625 0.00341798 0.00780113 2024-03-30T13:49:43.058 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:43.110 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:43.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:43.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:43.212 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:43.259 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:43.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:43.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:43.360 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:43.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:43.405 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:43.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:43.481 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:49:43.481 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:49:43.506 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:43.529 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:43.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:43.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:43.631 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:43.672 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:43.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:43.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:43.774 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:43.806 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:43.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:43.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:43.908 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:43.942 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:43.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:43.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:43.973 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 29 16 58818 58802 126.706 88 0.0062242 0.0078869 2024-03-30T13:49:44.044 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:44.079 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:44.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:44.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:44.181 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:44.218 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:44.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:44.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:44.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:43 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:49:44.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:43 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:49:44.320 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:44.360 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:44.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:44.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:44.381 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:43 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:49:44.381 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:43 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:49:44.381 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:43 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:49:44.381 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:43 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:49:44.462 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:44.494 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:44.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:44.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:44.595 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:44.633 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:44.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:44.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:44.735 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:44.776 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:44.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:44.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:44.877 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:44.910 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:44.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:44.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:44.974 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 30 15 60400 60385 125.78 98.9375 0.00977584 0.00794531 2024-03-30T13:49:45.012 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:45.049 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:45.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:45.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:45.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:44 smithi043 ceph-mon[29963]: pgmap v270: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-03-30T13:49:45.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:44 smithi043 ceph-mon[42099]: pgmap v270: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-03-30T13:49:45.151 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:45.187 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:45.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:45.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:45.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:44 smithi113 ceph-mon[34341]: pgmap v270: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-03-30T13:49:45.289 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:45.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:45.329 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:45.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:45.430 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:45.466 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:45.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:45.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:45.568 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:45.591 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:45.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:45.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:45.693 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:45.728 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:45.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:45.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:45.830 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:45.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:45.860 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:45.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:45.961 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:45.974 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 31 16 61810 61794 124.564 88.0625 0.00813002 0.00802263 2024-03-30T13:49:46.001 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:46.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:46.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:46.103 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:46.137 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:46.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:46.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:46.239 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:46.274 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:46.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:46.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:46.376 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:46.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:49:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:49:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:49:46.408 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:46.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:46.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:46.510 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:46.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:46.535 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:46.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:46.636 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:46.674 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:46.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:46.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:46.776 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:46.815 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:46.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:46.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:46.917 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:46.957 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:46.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:46.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:46.974 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 32 16 63178 63162 123.343 85.5 0.00918839 0.00810234 2024-03-30T13:49:47.059 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:47.097 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:47.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:47.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:47.199 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:47.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:46 smithi113 ceph-mon[34341]: pgmap v271: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-03-30T13:49:47.238 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:47.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:47.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:47.340 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:47.377 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:47.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:47.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:47.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:46 smithi043 ceph-mon[29963]: pgmap v271: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-03-30T13:49:47.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:46 smithi043 ceph-mon[42099]: pgmap v271: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-03-30T13:49:47.480 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:47.512 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:47.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:47.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:47.570 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-03-30T13:49:47.586 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:49:47.614 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:47.631 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:49:47.647 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:47.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:47.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:47.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:47.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:47.818 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:47.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:47.919 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:47.951 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:47.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:47.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:47.974 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 33 16 64577 64561 122.255 87.4375 0.0191724 0.00817387 2024-03-30T13:49:48.053 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:48.102 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:48.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:48.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:48.203 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:48.247 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:48.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:48.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:48.349 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:48.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:48.399 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:48.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:48.500 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:48.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:48.534 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:48.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:48.635 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:48.668 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:48.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:48.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:48.770 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:48.800 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:48.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:48.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:48.902 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:48.919 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:49:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:49:48.939 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:48.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:48.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:48.974 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 34 16 66036 66020 121.341 91.1875 0.00422221 0.00823607 2024-03-30T13:49:49.041 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:49.073 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:49.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:49.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:49.174 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:49.208 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:49.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:49.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:49.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:48 smithi113 ceph-mon[34341]: pgmap v272: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 569 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-30T13:49:49.310 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:49.349 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:49.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:49.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:49.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:48 smithi043 ceph-mon[29963]: pgmap v272: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 569 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-30T13:49:49.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:48 smithi043 ceph-mon[42099]: pgmap v272: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 569 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-30T13:49:49.451 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:49.489 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:49.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:49.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:49.591 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:49.629 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:49.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:49.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:49.731 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:49.810 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:49.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:49.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:49.862 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T13:49:49.862 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:49:49.862 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-30T13:49:49.863 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.000000 2024-03-30T13:49:49.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 2 0 2024-03-30T13:49:49.912 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:49.943 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:49.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:49.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:49.974 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 35 16 67406 67390 120.32 85.625 0.0214809 0.00830558 2024-03-30T13:49:50.044 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:50.068 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:50.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:50.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:50.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:49 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:49:50.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:49 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:49:50.170 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:50.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:50.203 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:50.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:50.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:49 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:49:50.305 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:50.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:50.347 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:50.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:50.448 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:50.486 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:50.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:50.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:50.588 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:50.612 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:50.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:50.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:50.714 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:50.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:50.749 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:50.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:50.850 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:50.890 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:50.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:50.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:50.974 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 36 16 68541 68525 118.948 70.9375 0.0141267 0.00840211 2024-03-30T13:49:50.992 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:51.025 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:51.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:51.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:51.126 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:51.178 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:51.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:51.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:51.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:50 smithi113 ceph-mon[34341]: pgmap v273: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-30T13:49:51.280 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:51.313 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:51.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:51.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:51.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:50 smithi043 ceph-mon[29963]: pgmap v273: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-30T13:49:51.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:50 smithi043 ceph-mon[42099]: pgmap v273: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-30T13:49:51.415 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:51.451 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:51.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:51.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:51.553 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:51.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:51.588 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:51.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:51.689 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:51.729 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:51.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:51.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:51.831 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:51.840 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:49:51.862 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:51.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:51.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:51.963 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:51.974 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 37 16 69890 69874 118.012 84.3125 0.00644197 0.00846836 2024-03-30T13:49:51.994 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:51.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:51.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:52.096 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:52.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:52.147 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:52.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:52.248 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:52.283 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:52.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:52.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:52.385 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:52.420 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:52.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:52.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:52.522 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:52.554 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:52.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:52.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:52.656 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:52.697 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:52.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:52.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:52.799 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:52.840 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:52.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:52.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:52.857 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:49:52.894 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:49:52.938 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:49:52.942 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:52.973 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:52.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:52.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:52.976 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 38 16 71204 71188 117.062 82.125 0.00578909 0.00853724 2024-03-30T13:49:52.997 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:49:53.037 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:49:53.075 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:53.077 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:49:53.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:53.108 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:53.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:53.119 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:49:53.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:52 smithi043 ceph-mon[29963]: pgmap v274: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 979 B/s rd, 90 MiB/s wr, 1.43k op/s 2024-03-30T13:49:53.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:52 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1847989458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:49:53.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:52 smithi043 ceph-mon[42099]: pgmap v274: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 979 B/s rd, 90 MiB/s wr, 1.43k op/s 2024-03-30T13:49:53.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:52 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1847989458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:49:53.159 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:49:53.199 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:49:53.207 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:49:53.207 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:49:53.207 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:49:53.207 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:49:53.207 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:49:53.207 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:49:53.207 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:49:53.207 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:49:53.210 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:53.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:52 smithi113 ceph-mon[34341]: pgmap v274: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 979 B/s rd, 90 MiB/s wr, 1.43k op/s 2024-03-30T13:49:53.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:52 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1847989458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:49:53.241 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:53.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:53.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:53.343 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:53.382 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:53.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:53.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:53.484 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:53.518 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:53.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:53.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:53.619 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:53.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:53.657 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:53.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:53.758 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:53.798 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:53.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:53.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:53.900 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:53.925 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:53.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:53.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:53.976 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 39 16 72405 72389 115.985 75.0625 0.00943418 0.00861637 2024-03-30T13:49:54.028 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:54.068 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:54.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:54.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:54.170 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:54.212 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:54.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:54.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:54.314 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:54.347 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:54.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:54.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:54.449 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:54.484 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:54.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:54.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:54.586 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:54.625 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:54.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:54.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:54.727 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:54.790 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:54.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:54.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:54.892 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:54.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:54.930 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:54.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:54.967 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-03-30T13:49:54.976 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-03-30T13:49:54.975836+0000 min lat: 0.00129646 max lat: 0.178336 avg lat: 0.008675 2024-03-30T13:49:54.976 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T13:49:54.976 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 40 16 73761 73745 115.204 84.75 0.0130068 0.008675 2024-03-30T13:49:55.032 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:55.062 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:55.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:55.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:55.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:54 smithi043 ceph-mon[29963]: pgmap v275: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-30T13:49:55.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:54 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3682473193' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:49:55.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:54 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:49:55.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:54 smithi043 ceph-mon[42099]: pgmap v275: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-30T13:49:55.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:54 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3682473193' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:49:55.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:54 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:49:55.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:55.200 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:55.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:55.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:55.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:54 smithi113 ceph-mon[34341]: pgmap v275: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-30T13:49:55.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:54 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3682473193' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:49:55.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:54 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:49:55.302 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:55.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:55.339 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:55.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:55.440 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:55.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:55.469 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:55.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:55.570 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:55.597 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:55.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:55.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:55.699 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:55.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:55.738 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:55.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:55.839 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:55.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:55.870 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:55.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:55.971 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:55.976 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 41 16 75014 74998 114.304 78.3125 0.0160202 0.00874382 2024-03-30T13:49:56.004 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:56.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:56.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:56.106 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:49:56.147 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:56.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:56.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:49:56.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:55 smithi113 ceph-mon[34341]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:49:56.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:55 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:49:56.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:55 smithi113 ceph-mon[34341]: osdmap e161: 8 total, 8 up, 8 in 2024-03-30T13:49:56.249 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:56.288 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:56.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:56.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:56.390 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:56.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:55 smithi043 ceph-mon[29963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:49:56.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:55 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:49:56.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:55 smithi043 ceph-mon[29963]: osdmap e161: 8 total, 8 up, 8 in 2024-03-30T13:49:56.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:55 smithi043 ceph-mon[42099]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:49:56.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:55 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:49:56.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:55 smithi043 ceph-mon[42099]: osdmap e161: 8 total, 8 up, 8 in 2024-03-30T13:49:56.400 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:49:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:49:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:49:56.424 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:56.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:56.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:56.526 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:56.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:56.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:56.666 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:56.696 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:56.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:56.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:56.798 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:56.838 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:56.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:56.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:56.940 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:56.963 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:56.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:56.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:56.976 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 42 16 76278 76262 113.464 79 0.00298163 0.00880172 2024-03-30T13:49:57.066 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:57.104 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:57.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:57.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:57.186 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:49:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-03-30T13:49:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-03-30T13:49:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-03-30T13:49:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:61996 7.a 7:523a71f3:::benchmark_data_smithi043_86989_object61995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-30T13:49:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:33.954381+0000", 2024-03-30T13:49:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 261.58175068899999, 2024-03-30T13:49:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.10224989299999999, 2024-03-30T13:49:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:49:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:49:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:49:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:49:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:49:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:49:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61996 2024-03-30T13:49:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:49:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:49:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:33.954381+0000", 2024-03-30T13:49:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:49:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:33.954381+0000", 2024-03-30T13:49:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:49:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:33.954382+0000", 2024-03-30T13:49:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9200000000000001e-07 2024-03-30T13:49:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:49:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:33.954400+0000", 2024-03-30T13:49:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7961e-05 2024-03-30T13:49:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:49:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:33.954407+0000", 2024-03-30T13:49:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4320000000000002e-06 2024-03-30T13:49:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:49:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:33.954414+0000", 2024-03-30T13:49:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2269999999999998e-06 2024-03-30T13:49:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:49:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:34.045724+0000", 2024-03-30T13:49:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.091310721999999997 2024-03-30T13:49:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:49:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:34.045885+0000", 2024-03-30T13:49:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016039900000000001 2024-03-30T13:49:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,5", 2024-03-30T13:49:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:34.046031+0000", 2024-03-30T13:49:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014583200000000001 2024-03-30T13:49:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:34.056419+0000", 2024-03-30T13:49:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010388312 2024-03-30T13:49:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:49:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:34.056433+0000", 2024-03-30T13:49:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.397e-05 2024-03-30T13:49:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:34.056513+0000", 2024-03-30T13:49:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0436e-05 2024-03-30T13:49:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:49:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:34.056589+0000", 2024-03-30T13:49:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5341999999999998e-05 2024-03-30T13:49:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:49:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:34.056631+0000", 2024-03-30T13:49:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2667999999999999e-05 2024-03-30T13:49:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:62000 7.a 7:50effac1:::benchmark_data_smithi043_86989_object61999:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-30T13:49:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:45:33.960032+0000", 2024-03-30T13:49:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 261.57609966899997, 2024-03-30T13:49:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.10204147399999999, 2024-03-30T13:49:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:49:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:49:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:49:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:49:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:49:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:49:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 62000 2024-03-30T13:49:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:49:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:49:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:33.960032+0000", 2024-03-30T13:49:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:49:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:33.960032+0000", 2024-03-30T13:49:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:49:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:33.960034+0000", 2024-03-30T13:49:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4670000000000001e-06 2024-03-30T13:49:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:49:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:33.960059+0000", 2024-03-30T13:49:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5299999999999998e-05 2024-03-30T13:49:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:49:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:34.044648+0000", 2024-03-30T13:49:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.084589261999999998 2024-03-30T13:49:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:49:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:34.044662+0000", 2024-03-30T13:49:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3088000000000001e-05 2024-03-30T13:49:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:49:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:34.051081+0000", 2024-03-30T13:49:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0064193439999999996 2024-03-30T13:49:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:49:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:34.051207+0000", 2024-03-30T13:49:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012598399999999999 2024-03-30T13:49:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,5", 2024-03-30T13:49:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:34.051344+0000", 2024-03-30T13:49:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013694 2024-03-30T13:49:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:49:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:34.056642+0000", 2024-03-30T13:49:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0052981310000000002 2024-03-30T13:49:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:34.056675+0000", 2024-03-30T13:49:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3458999999999999e-05 2024-03-30T13:49:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:34.059634+0000", 2024-03-30T13:49:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0029581070000000002 2024-03-30T13:49:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:49:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:34.059686+0000", 2024-03-30T13:49:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2002000000000001e-05 2024-03-30T13:49:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:49:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:45:34.062074+0000", 2024-03-30T13:49:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0023883899999999998 2024-03-30T13:49:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(osd.4.0:11271 7.1 e123/98 7:85602f8f:::benchmark_data_smithi043_86989_object67343:head v 123'11157, mlcod=123'11152)", 2024-03-30T13:49:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:46:55.515434+0000", 2024-03-30T13:49:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 180.020698544, 2024-03-30T13:49:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13478109999999999, 2024-03-30T13:49:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:49:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:49:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:49:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:49:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:49:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.515434+0000", 2024-03-30T13:49:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:49:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.515434+0000", 2024-03-30T13:49:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:49:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.515434+0000", 2024-03-30T13:49:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2600000000000002e-07 2024-03-30T13:49:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:49:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.515442+0000", 2024-03-30T13:49:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4189999999999996e-06 2024-03-30T13:49:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:49:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.515453+0000", 2024-03-30T13:49:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0892e-05 2024-03-30T13:49:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:49:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.515463+0000", 2024-03-30T13:49:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0876999999999999e-05 2024-03-30T13:49:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:49:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.515954+0000", 2024-03-30T13:49:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00049101600000000004 2024-03-30T13:49:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:49:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.515971+0000", 2024-03-30T13:49:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6481e-05 2024-03-30T13:49:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:49:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.650155+0000", 2024-03-30T13:49:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.134184213 2024-03-30T13:49:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:49:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.650215+0000", 2024-03-30T13:49:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9475999999999998e-05 2024-03-30T13:49:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(osd.4.0:11272 7.1 e123/98 7:856090b7:::benchmark_data_smithi043_86989_object96645:head v 123'11158, mlcod=123'11152)", 2024-03-30T13:49:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:46:55.516923+0000", 2024-03-30T13:49:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 180.01920873200001, 2024-03-30T13:49:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.136447866, 2024-03-30T13:49:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:49:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:49:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:49:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:49:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:49:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.516923+0000", 2024-03-30T13:49:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:49:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.516923+0000", 2024-03-30T13:49:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:49:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.516924+0000", 2024-03-30T13:49:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4000000000000002e-07 2024-03-30T13:49:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:49:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.516930+0000", 2024-03-30T13:49:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8749999999999997e-06 2024-03-30T13:49:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:49:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.516939+0000", 2024-03-30T13:49:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.9020000000000005e-06 2024-03-30T13:49:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:49:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.516945+0000", 2024-03-30T13:49:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9329999999999998e-06 2024-03-30T13:49:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:49:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.518082+0000", 2024-03-30T13:49:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0011373449999999999 2024-03-30T13:49:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:49:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.518094+0000", 2024-03-30T13:49:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1798e-05 2024-03-30T13:49:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:49:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.653334+0000", 2024-03-30T13:49:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13524011399999999 2024-03-30T13:49:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:49:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.653371+0000", 2024-03-30T13:49:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7358999999999999e-05 2024-03-30T13:49:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.14817.0:145828 7.7 e123/98 7:e63862af:::benchmark_data_smithi043_86989_object23490:head v 123'9067, mlcod=123'9065)", 2024-03-30T13:49:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:46:55.517569+0000", 2024-03-30T13:49:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 180.01856296099999, 2024-03-30T13:49:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.133007652, 2024-03-30T13:49:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:49:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:49:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:49:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:49:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:49:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.517569+0000", 2024-03-30T13:49:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:49:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.517569+0000", 2024-03-30T13:49:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:49:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.517570+0000", 2024-03-30T13:49:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8999999999999996e-07 2024-03-30T13:49:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:49:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.517576+0000", 2024-03-30T13:49:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7679999999999997e-06 2024-03-30T13:49:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:49:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.517585+0000", 2024-03-30T13:49:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7939999999999997e-06 2024-03-30T13:49:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:49:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.517592+0000", 2024-03-30T13:49:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3799999999999999e-06 2024-03-30T13:49:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:49:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.517641+0000", 2024-03-30T13:49:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8726000000000001e-05 2024-03-30T13:49:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:49:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.517660+0000", 2024-03-30T13:49:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9848999999999999e-05 2024-03-30T13:49:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:49:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.650537+0000", 2024-03-30T13:49:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13287669299999999 2024-03-30T13:49:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:49:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.650577+0000", 2024-03-30T13:49:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9752000000000002e-05 2024-03-30T13:49:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(osd.4.0:11273 7.1 e123/98 7:8560b898:::benchmark_data_smithi043_86989_object43301:head v 123'11159, mlcod=123'11152)", 2024-03-30T13:49:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:46:55.519084+0000", 2024-03-30T13:49:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 180.01704807600001, 2024-03-30T13:49:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.143160654, 2024-03-30T13:49:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:49:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:49:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:49:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:49:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:49:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.519084+0000", 2024-03-30T13:49:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:49:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.519084+0000", 2024-03-30T13:49:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:49:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.519085+0000", 2024-03-30T13:49:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4600000000000005e-07 2024-03-30T13:49:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:49:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.519090+0000", 2024-03-30T13:49:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3900000000000001e-06 2024-03-30T13:49:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:49:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.519098+0000", 2024-03-30T13:49:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4440000000000008e-06 2024-03-30T13:49:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:49:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.519105+0000", 2024-03-30T13:49:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1260000000000003e-06 2024-03-30T13:49:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:49:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.650245+0000", 2024-03-30T13:49:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.131140794 2024-03-30T13:49:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:49:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.650271+0000", 2024-03-30T13:49:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5627e-05 2024-03-30T13:49:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:49:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.662204+0000", 2024-03-30T13:49:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011933365 2024-03-30T13:49:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:49:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.662245+0000", 2024-03-30T13:49:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0361999999999997e-05 2024-03-30T13:49:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:149331 7.3 7:c3e78d49:::benchmark_data_smithi043_86989_object26993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-30T13:49:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:46:59.426934+0000", 2024-03-30T13:49:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 176.10919854100001, 2024-03-30T13:49:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.11952345, 2024-03-30T13:49:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:49:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:49:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:49:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:49:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:49:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:49:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 149331 2024-03-30T13:49:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:49:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:49:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.426934+0000", 2024-03-30T13:49:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:49:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.426934+0000", 2024-03-30T13:49:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:49:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.426935+0000", 2024-03-30T13:49:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2929999999999999e-06 2024-03-30T13:49:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:49:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.426941+0000", 2024-03-30T13:49:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3729999999999998e-06 2024-03-30T13:49:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:49:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.426952+0000", 2024-03-30T13:49:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0457e-05 2024-03-30T13:49:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:49:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.426959+0000", 2024-03-30T13:49:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.357e-06 2024-03-30T13:49:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:49:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.428227+0000", 2024-03-30T13:49:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00126778 2024-03-30T13:49:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:49:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.428351+0000", 2024-03-30T13:49:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012371699999999999 2024-03-30T13:49:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,4", 2024-03-30T13:49:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.428459+0000", 2024-03-30T13:49:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010809799999999999 2024-03-30T13:49:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:49:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.434941+0000", 2024-03-30T13:49:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0064828150000000003 2024-03-30T13:49:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.437526+0000", 2024-03-30T13:49:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0025843960000000001 2024-03-30T13:49:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.546291+0000", 2024-03-30T13:49:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.108764786 2024-03-30T13:49:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:49:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.546390+0000", 2024-03-30T13:49:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9483999999999998e-05 2024-03-30T13:49:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:49:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.546457+0000", 2024-03-30T13:49:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6894e-05 2024-03-30T13:49:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:149334 7.3 7:cd30d1fa:::benchmark_data_smithi043_86989_object26996:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-30T13:49:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:46:59.430880+0000", 2024-03-30T13:49:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 176.10525257399999, 2024-03-30T13:49:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.115787641, 2024-03-30T13:49:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:49:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:49:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:49:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:49:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:49:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:49:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 149334 2024-03-30T13:49:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:49:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:49:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.430880+0000", 2024-03-30T13:49:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:49:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.430880+0000", 2024-03-30T13:49:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:49:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.430880+0000", 2024-03-30T13:49:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8100000000000002e-07 2024-03-30T13:49:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:49:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.430885+0000", 2024-03-30T13:49:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.498e-06 2024-03-30T13:49:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:49:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.430895+0000", 2024-03-30T13:49:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0035e-05 2024-03-30T13:49:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:49:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.430902+0000", 2024-03-30T13:49:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2350000000000002e-06 2024-03-30T13:49:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:49:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.432101+0000", 2024-03-30T13:49:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001198541 2024-03-30T13:49:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:49:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.432213+0000", 2024-03-30T13:49:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000112828 2024-03-30T13:49:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,4", 2024-03-30T13:49:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.432321+0000", 2024-03-30T13:49:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010727500000000001 2024-03-30T13:49:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:49:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.434976+0000", 2024-03-30T13:49:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026550250000000001 2024-03-30T13:49:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.437691+0000", 2024-03-30T13:49:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0027153630000000002 2024-03-30T13:49:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.546577+0000", 2024-03-30T13:49:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.108885736 2024-03-30T13:49:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:49:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.546631+0000", 2024-03-30T13:49:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3875999999999997e-05 2024-03-30T13:49:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:49:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:59.546667+0000", 2024-03-30T13:49:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6548000000000003e-05 2024-03-30T13:49:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:27417 8.b 8:daff8372:::benchmark_data_smithi043_119832_object27416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-30T13:49:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:49:25.470238+0000", 2024-03-30T13:49:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 30.065894220000001, 2024-03-30T13:49:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.135372724, 2024-03-30T13:49:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:49:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:49:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:49:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:49:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:49:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:49:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 27417 2024-03-30T13:49:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:49:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:49:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.470238+0000", 2024-03-30T13:49:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:49:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.470238+0000", 2024-03-30T13:49:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:49:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.470239+0000", 2024-03-30T13:49:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5090000000000001e-06 2024-03-30T13:49:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:49:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.470261+0000", 2024-03-30T13:49:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.128e-05 2024-03-30T13:49:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:49:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.470373+0000", 2024-03-30T13:49:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011277000000000001 2024-03-30T13:49:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:49:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.470382+0000", 2024-03-30T13:49:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.9479999999999997e-06 2024-03-30T13:49:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:49:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.470444+0000", 2024-03-30T13:49:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1328999999999995e-05 2024-03-30T13:49:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:49:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.470577+0000", 2024-03-30T13:49:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000133096 2024-03-30T13:49:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:49:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.470707+0000", 2024-03-30T13:49:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013005199999999999 2024-03-30T13:49:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.474205+0000", 2024-03-30T13:49:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0034984679999999998 2024-03-30T13:49:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:49:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.477381+0000", 2024-03-30T13:49:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0031755189999999999 2024-03-30T13:49:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.605507+0000", 2024-03-30T13:49:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.12812605599999999 2024-03-30T13:49:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:49:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.605564+0000", 2024-03-30T13:49:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6776e-05 2024-03-30T13:49:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:49:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.605611+0000", 2024-03-30T13:49:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6921e-05 2024-03-30T13:49:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:27427 8.b 8:d199c970:::benchmark_data_smithi043_119832_object27426:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-30T13:49:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:49:25.473999+0000", 2024-03-30T13:49:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 30.062132622, 2024-03-30T13:49:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13176854399999999, 2024-03-30T13:49:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:49:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:49:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:49:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:49:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:49:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:49:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 27427 2024-03-30T13:49:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:49:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:49:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.473999+0000", 2024-03-30T13:49:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:49:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.473999+0000", 2024-03-30T13:49:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:49:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.474001+0000", 2024-03-30T13:49:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.265e-06 2024-03-30T13:49:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:49:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.474025+0000", 2024-03-30T13:49:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4073000000000001e-05 2024-03-30T13:49:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:49:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.474035+0000", 2024-03-30T13:49:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7969999999999995e-06 2024-03-30T13:49:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:49:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.474042+0000", 2024-03-30T13:49:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6350000000000006e-06 2024-03-30T13:49:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:49:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.474231+0000", 2024-03-30T13:49:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018868300000000001 2024-03-30T13:49:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:49:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.474366+0000", 2024-03-30T13:49:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013525700000000001 2024-03-30T13:49:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:49:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.474545+0000", 2024-03-30T13:49:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017867 2024-03-30T13:49:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:49:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.477395+0000", 2024-03-30T13:49:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0028505399999999999 2024-03-30T13:49:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.477479+0000", 2024-03-30T13:49:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3350000000000007e-05 2024-03-30T13:49:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.605660+0000", 2024-03-30T13:49:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.12818110699999999 2024-03-30T13:49:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:49:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.605729+0000", 2024-03-30T13:49:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8887999999999995e-05 2024-03-30T13:49:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:49:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.605768+0000", 2024-03-30T13:49:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9279000000000001e-05 2024-03-30T13:49:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.243 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.243 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:27430 8.b 8:dfc354e5:::benchmark_data_smithi043_119832_object27429:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-30T13:49:57.243 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:49:25.474854+0000", 2024-03-30T13:49:57.243 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 30.061278432000002, 2024-03-30T13:49:57.243 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13112748699999999, 2024-03-30T13:49:57.243 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:49:57.243 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:49:57.243 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:49:57.243 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:49:57.243 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:49:57.243 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:49:57.244 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 27430 2024-03-30T13:49:57.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.244 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:49:57.244 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.244 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:49:57.244 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.474854+0000", 2024-03-30T13:49:57.244 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.244 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.244 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:49:57.244 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.474854+0000", 2024-03-30T13:49:57.244 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.244 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.245 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:49:57.245 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.474855+0000", 2024-03-30T13:49:57.245 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1009999999999999e-06 2024-03-30T13:49:57.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.245 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:49:57.245 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.474882+0000", 2024-03-30T13:49:57.245 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7387999999999999e-05 2024-03-30T13:49:57.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.245 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:49:57.245 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.474893+0000", 2024-03-30T13:49:57.245 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1246999999999999e-05 2024-03-30T13:49:57.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:49:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.474902+0000", 2024-03-30T13:49:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2889999999999998e-06 2024-03-30T13:49:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:49:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.475592+0000", 2024-03-30T13:49:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00069049699999999999 2024-03-30T13:49:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:49:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.475718+0000", 2024-03-30T13:49:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012552599999999999 2024-03-30T13:49:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:49:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.475847+0000", 2024-03-30T13:49:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012957800000000001 2024-03-30T13:49:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:49:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.477401+0000", 2024-03-30T13:49:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001553764 2024-03-30T13:49:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.477655+0000", 2024-03-30T13:49:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00025373699999999997 2024-03-30T13:49:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.605820+0000", 2024-03-30T13:49:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.12816543899999999 2024-03-30T13:49:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:49:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.605899+0000", 2024-03-30T13:49:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8660999999999998e-05 2024-03-30T13:49:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:49:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.605981+0000", 2024-03-30T13:49:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2260000000000002e-05 2024-03-30T13:49:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:27451 8.b 8:de066841:::benchmark_data_smithi043_119832_object27450:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-30T13:49:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:49:25.484033+0000", 2024-03-30T13:49:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 30.052099067, 2024-03-30T13:49:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.122424249, 2024-03-30T13:49:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:49:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:49:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:49:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:49:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:49:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:49:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 27451 2024-03-30T13:49:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:49:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:49:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.484033+0000", 2024-03-30T13:49:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:49:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.484033+0000", 2024-03-30T13:49:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:49:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.484034+0000", 2024-03-30T13:49:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0529999999999999e-06 2024-03-30T13:49:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:49:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.484059+0000", 2024-03-30T13:49:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4573999999999999e-05 2024-03-30T13:49:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:49:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.484068+0000", 2024-03-30T13:49:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4299999999999995e-06 2024-03-30T13:49:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:49:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.484074+0000", 2024-03-30T13:49:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4360000000000001e-06 2024-03-30T13:49:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:49:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.484116+0000", 2024-03-30T13:49:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2184000000000001e-05 2024-03-30T13:49:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:49:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.484197+0000", 2024-03-30T13:49:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1184000000000001e-05 2024-03-30T13:49:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:49:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.484284+0000", 2024-03-30T13:49:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6780000000000006e-05 2024-03-30T13:49:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:49:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.485226+0000", 2024-03-30T13:49:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00094229800000000005 2024-03-30T13:49:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.603645+0000", 2024-03-30T13:49:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.11841882400000001 2024-03-30T13:49:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.606361+0000", 2024-03-30T13:49:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0027164490000000001 2024-03-30T13:49:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:49:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.606416+0000", 2024-03-30T13:49:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4333000000000003e-05 2024-03-30T13:49:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:49:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.606457+0000", 2024-03-30T13:49:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1703999999999999e-05 2024-03-30T13:49:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop_reply(client.24977.0:27451 8.b e154/147 ondisk, result = 0)", 2024-03-30T13:49:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:49:25.485416+0000", 2024-03-30T13:49:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 30.050715779000001, 2024-03-30T13:49:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.11823744899999999, 2024-03-30T13:49:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:49:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:49:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T13:49:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:49:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:49:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.485416+0000", 2024-03-30T13:49:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:49:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.485416+0000", 2024-03-30T13:49:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:49:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.485418+0000", 2024-03-30T13:49:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3939999999999999e-06 2024-03-30T13:49:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:49:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.485426+0000", 2024-03-30T13:49:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1030000000000002e-06 2024-03-30T13:49:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:49:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.485447+0000", 2024-03-30T13:49:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1305999999999999e-05 2024-03-30T13:49:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:49:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.485460+0000", 2024-03-30T13:49:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3196999999999999e-05 2024-03-30T13:49:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:49:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.603629+0000", 2024-03-30T13:49:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.118168255 2024-03-30T13:49:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:49:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.603638+0000", 2024-03-30T13:49:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0589999999999992e-06 2024-03-30T13:49:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:49:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.603654+0000", 2024-03-30T13:49:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6135000000000001e-05 2024-03-30T13:49:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.24977.0:28082 8.8 e154/147 8:1b8ed0ff:::benchmark_data_smithi043_119832_object28081:head v 154'1798, mlcod=154'1794)", 2024-03-30T13:49:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:49:25.865637+0000", 2024-03-30T13:49:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 29.670495175999999, 2024-03-30T13:49:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.10129766599999999, 2024-03-30T13:49:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:49:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:49:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:49:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:49:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:49:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.865637+0000", 2024-03-30T13:49:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:49:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.865637+0000", 2024-03-30T13:49:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:49:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.865638+0000", 2024-03-30T13:49:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7400000000000003e-07 2024-03-30T13:49:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:49:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.865753+0000", 2024-03-30T13:49:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000115232 2024-03-30T13:49:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:49:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.865762+0000", 2024-03-30T13:49:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.9730000000000003e-06 2024-03-30T13:49:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:49:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.865768+0000", 2024-03-30T13:49:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7549999999999997e-06 2024-03-30T13:49:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:49:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.867047+0000", 2024-03-30T13:49:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0012784070000000001 2024-03-30T13:49:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:49:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.867065+0000", 2024-03-30T13:49:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8573e-05 2024-03-30T13:49:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:49:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.966044+0000", 2024-03-30T13:49:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.098978269999999993 2024-03-30T13:49:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:49:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.966935+0000", 2024-03-30T13:49:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00089088199999999996 2024-03-30T13:49:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:28085 8.b 8:db2fbbe0:::benchmark_data_smithi043_119832_object28084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-30T13:49:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:49:25.865671+0000", 2024-03-30T13:49:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 29.670460996999999, 2024-03-30T13:49:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.10047708900000001, 2024-03-30T13:49:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:49:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:49:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:49:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:49:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:49:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:49:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 28085 2024-03-30T13:49:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:49:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:49:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.865671+0000", 2024-03-30T13:49:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:49:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.865671+0000", 2024-03-30T13:49:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:49:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.865672+0000", 2024-03-30T13:49:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0580000000000001e-06 2024-03-30T13:49:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:49:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.865694+0000", 2024-03-30T13:49:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2089000000000001e-05 2024-03-30T13:49:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:49:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.865705+0000", 2024-03-30T13:49:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1072e-05 2024-03-30T13:49:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:49:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.865713+0000", 2024-03-30T13:49:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9580000000000007e-06 2024-03-30T13:49:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:49:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.865792+0000", 2024-03-30T13:49:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8454000000000005e-05 2024-03-30T13:49:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:49:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.865915+0000", 2024-03-30T13:49:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000123738 2024-03-30T13:49:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:49:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.866246+0000", 2024-03-30T13:49:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00033010100000000002 2024-03-30T13:49:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.965850+0000", 2024-03-30T13:49:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.099604430999999993 2024-03-30T13:49:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.965972+0000", 2024-03-30T13:49:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000121741 2024-03-30T13:49:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:49:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.965988+0000", 2024-03-30T13:49:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6540000000000001e-05 2024-03-30T13:49:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:49:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.966101+0000", 2024-03-30T13:49:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011261500000000001 2024-03-30T13:49:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:49:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.966148+0000", 2024-03-30T13:49:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7292000000000001e-05 2024-03-30T13:49:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:28086 8.b 8:d18ebee2:::benchmark_data_smithi043_119832_object28085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-30T13:49:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:49:25.865797+0000", 2024-03-30T13:49:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 29.670335513000001, 2024-03-30T13:49:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.105596654, 2024-03-30T13:49:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:49:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:49:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:49:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:49:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:49:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:49:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 28086 2024-03-30T13:49:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:49:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:49:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.865797+0000", 2024-03-30T13:49:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:49:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.865797+0000", 2024-03-30T13:49:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:49:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.865797+0000", 2024-03-30T13:49:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9100000000000003e-07 2024-03-30T13:49:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:49:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.865817+0000", 2024-03-30T13:49:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9644e-05 2024-03-30T13:49:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:49:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.865826+0000", 2024-03-30T13:49:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1549999999999996e-06 2024-03-30T13:49:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:49:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.865833+0000", 2024-03-30T13:49:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8029999999999999e-06 2024-03-30T13:49:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:49:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.964171+0000", 2024-03-30T13:49:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.098338378000000004 2024-03-30T13:49:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:49:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.964302+0000", 2024-03-30T13:49:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000130493 2024-03-30T13:49:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:49:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.964440+0000", 2024-03-30T13:49:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000138646 2024-03-30T13:49:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:49:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.966156+0000", 2024-03-30T13:49:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001715821 2024-03-30T13:49:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.968013+0000", 2024-03-30T13:49:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001856529 2024-03-30T13:49:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.970378+0000", 2024-03-30T13:49:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002365607 2024-03-30T13:49:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:49:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.970435+0000", 2024-03-30T13:49:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6267000000000002e-05 2024-03-30T13:49:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:49:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:25.971393+0000", 2024-03-30T13:49:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00095861999999999998 2024-03-30T13:49:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:31276 8.b 8:d05106bb:::benchmark_data_smithi043_119832_object31275:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-30T13:49:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:49:27.602579+0000", 2024-03-30T13:49:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 27.933553030999999, 2024-03-30T13:49:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17785183600000001, 2024-03-30T13:49:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:49:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:49:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:49:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:49:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:49:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:49:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 31276 2024-03-30T13:49:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:49:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:49:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.602579+0000", 2024-03-30T13:49:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:49:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.602579+0000", 2024-03-30T13:49:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:49:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.602581+0000", 2024-03-30T13:49:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.79e-06 2024-03-30T13:49:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:49:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.602606+0000", 2024-03-30T13:49:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5021e-05 2024-03-30T13:49:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:49:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.602618+0000", 2024-03-30T13:49:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2568000000000001e-05 2024-03-30T13:49:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:49:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.602627+0000", 2024-03-30T13:49:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6859999999999995e-06 2024-03-30T13:49:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:49:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.602688+0000", 2024-03-30T13:49:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1011999999999998e-05 2024-03-30T13:49:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:49:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.603023+0000", 2024-03-30T13:49:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000334937 2024-03-30T13:49:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:49:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.603224+0000", 2024-03-30T13:49:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000201227 2024-03-30T13:49:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.605455+0000", 2024-03-30T13:49:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022307920000000001 2024-03-30T13:49:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:49:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.605467+0000", 2024-03-30T13:49:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2232000000000001e-05 2024-03-30T13:49:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780257+0000", 2024-03-30T13:49:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.174789747 2024-03-30T13:49:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:49:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780377+0000", 2024-03-30T13:49:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012008800000000001 2024-03-30T13:49:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:49:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780431+0000", 2024-03-30T13:49:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3736000000000002e-05 2024-03-30T13:49:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:31289 8.b 8:d1f0aa81:::benchmark_data_smithi043_119832_object31288:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-30T13:49:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:49:27.605999+0000", 2024-03-30T13:49:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 27.930132651000001, 2024-03-30T13:49:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17461035799999999, 2024-03-30T13:49:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:49:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:49:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:49:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:49:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:49:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:49:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 31289 2024-03-30T13:49:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:49:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:49:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.605999+0000", 2024-03-30T13:49:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:49:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.605999+0000", 2024-03-30T13:49:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:49:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.606001+0000", 2024-03-30T13:49:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.567e-06 2024-03-30T13:49:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:49:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.606020+0000", 2024-03-30T13:49:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8974000000000002e-05 2024-03-30T13:49:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:49:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.606034+0000", 2024-03-30T13:49:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.403e-05 2024-03-30T13:49:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:49:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.606042+0000", 2024-03-30T13:49:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8720000000000002e-06 2024-03-30T13:49:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:49:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.606114+0000", 2024-03-30T13:49:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2198000000000002e-05 2024-03-30T13:49:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:49:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.606316+0000", 2024-03-30T13:49:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00020206900000000001 2024-03-30T13:49:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:49:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.606465+0000", 2024-03-30T13:49:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014919400000000001 2024-03-30T13:49:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:49:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.608103+0000", 2024-03-30T13:49:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0016373049999999999 2024-03-30T13:49:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.610843+0000", 2024-03-30T13:49:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0027407769999999998 2024-03-30T13:49:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780498+0000", 2024-03-30T13:49:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.16965456300000001 2024-03-30T13:49:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:49:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780565+0000", 2024-03-30T13:49:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6587999999999993e-05 2024-03-30T13:49:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:49:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780610+0000", 2024-03-30T13:49:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5221e-05 2024-03-30T13:49:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:31302 8.b 8:db19a970:::benchmark_data_smithi043_119832_object31301:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-30T13:49:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:49:27.614761+0000", 2024-03-30T13:49:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 27.921371569000002, 2024-03-30T13:49:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.16602650699999999, 2024-03-30T13:49:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:49:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:49:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:49:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:49:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:49:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:49:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 31302 2024-03-30T13:49:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:49:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:49:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.614761+0000", 2024-03-30T13:49:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:49:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.614761+0000", 2024-03-30T13:49:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:49:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.614762+0000", 2024-03-30T13:49:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3999999999999999e-06 2024-03-30T13:49:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:49:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.614796+0000", 2024-03-30T13:49:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4400000000000003e-05 2024-03-30T13:49:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:49:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.614809+0000", 2024-03-30T13:49:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2699e-05 2024-03-30T13:49:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:49:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.614817+0000", 2024-03-30T13:49:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5299999999999999e-06 2024-03-30T13:49:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:49:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.615011+0000", 2024-03-30T13:49:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00019433699999999999 2024-03-30T13:49:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:49:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.615129+0000", 2024-03-30T13:49:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011833900000000001 2024-03-30T13:49:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:49:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.615547+0000", 2024-03-30T13:49:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00041808699999999998 2024-03-30T13:49:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:49:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.617305+0000", 2024-03-30T13:49:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0017581070000000001 2024-03-30T13:49:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.618095+0000", 2024-03-30T13:49:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00078949499999999998 2024-03-30T13:49:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780674+0000", 2024-03-30T13:49:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.16257946000000001 2024-03-30T13:49:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:49:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780743+0000", 2024-03-30T13:49:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8725000000000004e-05 2024-03-30T13:49:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:49:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780787+0000", 2024-03-30T13:49:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3927999999999997e-05 2024-03-30T13:49:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:31311 8.b 8:de8ee756:::benchmark_data_smithi043_119832_object31310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-30T13:49:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:49:27.618998+0000", 2024-03-30T13:49:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 27.917133843999999, 2024-03-30T13:49:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.16195649300000001, 2024-03-30T13:49:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:49:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:49:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:49:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:49:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:49:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:49:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 31311 2024-03-30T13:49:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:49:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:49:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.618998+0000", 2024-03-30T13:49:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:49:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.618998+0000", 2024-03-30T13:49:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:49:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:49:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.619000+0000", 2024-03-30T13:49:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5969999999999999e-06 2024-03-30T13:49:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:49:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.619022+0000", 2024-03-30T13:49:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2180000000000001e-05 2024-03-30T13:49:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:49:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.619036+0000", 2024-03-30T13:49:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3726999999999999e-05 2024-03-30T13:49:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:49:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.619044+0000", 2024-03-30T13:49:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2609999999999993e-06 2024-03-30T13:49:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:49:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.619105+0000", 2024-03-30T13:49:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0843e-05 2024-03-30T13:49:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:49:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.619236+0000", 2024-03-30T13:49:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013116500000000001 2024-03-30T13:49:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:49:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.619385+0000", 2024-03-30T13:49:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014934799999999999 2024-03-30T13:49:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:49:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.620675+0000", 2024-03-30T13:49:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0012892890000000001 2024-03-30T13:49:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.621289+0000", 2024-03-30T13:49:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00061401499999999998 2024-03-30T13:49:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:49:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780836+0000", 2024-03-30T13:49:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.15954684799999999 2024-03-30T13:49:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:49:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780917+0000", 2024-03-30T13:49:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1414000000000004e-05 2024-03-30T13:49:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:49:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:49:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:49:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780955+0000", 2024-03-30T13:49:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7805999999999997e-05 2024-03-30T13:49:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:49:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:49:57.302 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:49:57.302 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:49:57.303 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:56 smithi113 ceph-mon[34341]: pgmap v277: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-30T13:49:57.336 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:57.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:57.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:49:57.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:56 smithi043 ceph-mon[29963]: pgmap v277: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-30T13:49:57.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:56 smithi043 ceph-mon[42099]: pgmap v277: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-30T13:49:57.438 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:57.468 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:57.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:57.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:57.570 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:57.610 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:57.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:57.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:57.712 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:57.743 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:57.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:57.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:57.844 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:57.877 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:57.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:57.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:57.975 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-30T13:49:57.976 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 43 16 77606 77590 112.755 83 0.00576348 0.00886268 2024-03-30T13:49:57.979 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:58.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:58.068 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:58.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:58.169 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:49:58.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:58.204 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:58.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:49:58.305 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:58.343 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:49:58.351 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:58.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:58.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:58.453 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:58.496 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:58.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:58.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:58.598 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:58.644 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:58.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:58.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:58.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:58 smithi043 ceph-mon[29963]: pgmap v278: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-30T13:49:58.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:58 smithi043 ceph-mon[29963]: osdmap e162: 8 total, 8 up, 8 in 2024-03-30T13:49:58.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:58 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:49:58.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:58 smithi043 ceph-mon[42099]: pgmap v278: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-30T13:49:58.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:58 smithi043 ceph-mon[42099]: osdmap e162: 8 total, 8 up, 8 in 2024-03-30T13:49:58.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:58 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:49:58.670 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:58 smithi113 ceph-mon[34341]: pgmap v278: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-30T13:49:58.670 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:58 smithi113 ceph-mon[34341]: osdmap e162: 8 total, 8 up, 8 in 2024-03-30T13:49:58.670 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:58 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:49:58.670 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:49:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:49:58.746 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:58.799 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:58.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:58.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:58.900 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:49:58.942 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:58.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:58.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:49:58.976 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 44 16 78885 78869 112.009 79.9375 0.00886521 0.00892337 2024-03-30T13:49:59.045 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:59.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:59.083 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:59.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:59.184 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:59.227 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:59.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:59.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:59.329 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:49:59.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:59.362 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:59.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:49:59.463 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:49:59.488 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:49:59.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:59.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:49:59.590 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:59.626 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:59.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:59.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:49:59 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:49:59.728 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:49:59.770 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:59.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:59.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:49:59.872 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:49:59.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:49:59 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:49:59.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:49:59 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:49:59.914 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:49:59.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:49:59.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:49:59.977 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 45 16 79954 79938 111.005 66.8125 0.0121562 0.00900337 2024-03-30T13:50:00.015 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:00.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:00.053 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:00.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:00.154 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:00.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:00.194 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:00.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:00.295 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:00.323 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:00.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:00.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:00.425 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:00.450 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.2 primary-affinity to 0 (802) 2024-03-30T13:50:00.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:00.467 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:00.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:00.509 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:00 smithi113 ceph-mon[34341]: pgmap v280: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-03-30T13:50:00.509 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:00 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2497440956' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]: dispatch 2024-03-30T13:50:00.509 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:00 smithi113 ceph-mon[34341]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-30T13:50:00.509 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:00 smithi113 ceph-mon[34341]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-30T13:50:00.568 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:50:00.616 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:00.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:00.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:50:00.701 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:50:00.701 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [ 2024-03-30T13:50:00.701 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.701 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop(client.24977.0:80777 8.5 e163/147 8:a79387a0:::benchmark_data_smithi043_119832_object80776:head v 163'5150, mlcod=163'5148)", 2024-03-30T13:50:00.701 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:00.694887+0000", 2024-03-30T13:50:00.702 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.0026398250000000002, 2024-03-30T13:50:00.702 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00267395, 2024-03-30T13:50:00.702 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:50:00.702 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:50:00.702 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:50:00.702 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:50:00.702 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.702 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:50:00.702 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.694887+0000", 2024-03-30T13:50:00.702 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:50:00.702 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.703 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.703 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:50:00.703 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.694887+0000", 2024-03-30T13:50:00.703 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:50:00.703 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.703 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.703 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:50:00.703 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.694888+0000", 2024-03-30T13:50:00.703 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.4199999999999995e-07 2024-03-30T13:50:00.703 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.703 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.703 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:50:00.703 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.694913+0000", 2024-03-30T13:50:00.703 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.5562999999999999e-05 2024-03-30T13:50:00.704 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.704 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.704 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:50:00.704 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.694926+0000", 2024-03-30T13:50:00.704 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2593e-05 2024-03-30T13:50:00.704 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.704 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.704 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:50:00.704 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.694934+0000", 2024-03-30T13:50:00.704 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.4460000000000001e-06 2024-03-30T13:50:00.704 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.704 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.704 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:50:00.704 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.696280+0000", 2024-03-30T13:50:00.705 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001346058 2024-03-30T13:50:00.705 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.705 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.705 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:50:00.705 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.696299+0000", 2024-03-30T13:50:00.705 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.8845e-05 2024-03-30T13:50:00.705 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:50:00.705 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:50:00.705 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:50:00.705 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.705 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.705 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop(client.24977.0:80774 8.b e163/147 8:d69d014c:::benchmark_data_smithi043_119832_object80773:head v 163'4979, mlcod=163'4977)", 2024-03-30T13:50:00.705 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:00.696097+0000", 2024-03-30T13:50:00.705 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.001429498, 2024-03-30T13:50:00.706 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0015392800000000001, 2024-03-30T13:50:00.706 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:50:00.706 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:50:00.706 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:50:00.706 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:50:00.706 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.706 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:50:00.706 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.696097+0000", 2024-03-30T13:50:00.706 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:50:00.706 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.706 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.706 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:50:00.706 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.696097+0000", 2024-03-30T13:50:00.707 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:50:00.707 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.707 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.707 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:50:00.707 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.696098+0000", 2024-03-30T13:50:00.707 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8999999999999995e-07 2024-03-30T13:50:00.707 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.707 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.707 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:50:00.707 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.696127+0000", 2024-03-30T13:50:00.707 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.9422000000000001e-05 2024-03-30T13:50:00.707 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.707 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.708 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:50:00.708 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.696141+0000", 2024-03-30T13:50:00.708 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.358e-05 2024-03-30T13:50:00.708 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.708 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.708 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:50:00.708 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.696148+0000", 2024-03-30T13:50:00.708 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.7460000000000003e-06 2024-03-30T13:50:00.708 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.708 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.708 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:50:00.708 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.696213+0000", 2024-03-30T13:50:00.708 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5474000000000004e-05 2024-03-30T13:50:00.708 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.709 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.709 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:50:00.709 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.696243+0000", 2024-03-30T13:50:00.709 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.0090000000000002e-05 2024-03-30T13:50:00.709 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:50:00.709 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:50:00.709 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:50:00.709 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.709 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.709 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:80773 8.2 8:4691ee08:::benchmark_data_smithi043_119832_object80772:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-03-30T13:50:00.709 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:00.690331+0000", 2024-03-30T13:50:00.709 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.0071952580000000004, 2024-03-30T13:50:00.709 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0073803510000000003, 2024-03-30T13:50:00.709 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:50:00.710 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:50:00.710 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "waiting for sub ops", 2024-03-30T13:50:00.710 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:50:00.710 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:50:00.710 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:50:00.710 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 80773 2024-03-30T13:50:00.710 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.710 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:50:00.710 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.710 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:50:00.710 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.690331+0000", 2024-03-30T13:50:00.710 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:50:00.710 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.711 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.711 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:50:00.711 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.690331+0000", 2024-03-30T13:50:00.711 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:50:00.711 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.711 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.711 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:50:00.711 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.690333+0000", 2024-03-30T13:50:00.711 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.7710000000000001e-06 2024-03-30T13:50:00.711 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.711 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.712 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:50:00.712 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.690377+0000", 2024-03-30T13:50:00.712 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.367e-05 2024-03-30T13:50:00.712 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.712 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.712 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:50:00.712 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.690391+0000", 2024-03-30T13:50:00.712 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4078e-05 2024-03-30T13:50:00.712 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.712 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.712 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:50:00.713 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.690399+0000", 2024-03-30T13:50:00.713 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.2139999999999996e-06 2024-03-30T13:50:00.713 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.713 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.713 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:50:00.713 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.690467+0000", 2024-03-30T13:50:00.713 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.7618000000000002e-05 2024-03-30T13:50:00.714 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.714 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.714 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:50:00.714 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.690600+0000", 2024-03-30T13:50:00.714 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013304500000000001 2024-03-30T13:50:00.714 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.714 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.714 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,5", 2024-03-30T13:50:00.714 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.690730+0000", 2024-03-30T13:50:00.714 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012992599999999999 2024-03-30T13:50:00.714 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.714 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.715 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:50:00.715 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.693543+0000", 2024-03-30T13:50:00.715 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0028131459999999999 2024-03-30T13:50:00.715 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.715 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.715 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:50:00.715 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.695595+0000", 2024-03-30T13:50:00.715 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0020520909999999998 2024-03-30T13:50:00.715 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:50:00.715 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:50:00.715 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:50:00.716 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.716 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.716 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop(client.24977.0:80767 8.4 e163/147 8:218d37ea:::benchmark_data_smithi043_119832_object80766:head v 163'5121, mlcod=163'5117)", 2024-03-30T13:50:00.716 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:00.693752+0000", 2024-03-30T13:50:00.716 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.0037744089999999998, 2024-03-30T13:50:00.716 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0040553840000000004, 2024-03-30T13:50:00.716 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:50:00.716 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:50:00.716 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:50:00.716 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:50:00.716 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.716 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:50:00.717 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.693752+0000", 2024-03-30T13:50:00.717 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:50:00.717 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.717 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.717 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:50:00.717 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.693752+0000", 2024-03-30T13:50:00.717 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:50:00.717 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.717 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.717 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:50:00.717 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.693753+0000", 2024-03-30T13:50:00.717 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.018e-06 2024-03-30T13:50:00.717 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.718 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.718 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:50:00.718 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.693791+0000", 2024-03-30T13:50:00.718 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.7336999999999998e-05 2024-03-30T13:50:00.718 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.718 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.718 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:50:00.718 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.693804+0000", 2024-03-30T13:50:00.718 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3454e-05 2024-03-30T13:50:00.718 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.718 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.718 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:50:00.719 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.693813+0000", 2024-03-30T13:50:00.719 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5560000000000008e-06 2024-03-30T13:50:00.719 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.719 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.719 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:50:00.719 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.693874+0000", 2024-03-30T13:50:00.719 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1189e-05 2024-03-30T13:50:00.719 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:00.719 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:00.719 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:50:00.719 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.693899+0000", 2024-03-30T13:50:00.719 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.4868e-05 2024-03-30T13:50:00.720 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:50:00.720 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:50:00.720 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:50:00.720 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:50:00.720 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-03-30T13:50:00.720 INFO:teuthology.orchestra.run.smithi113.stdout: "num_ops": 4 2024-03-30T13:50:00.720 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:50:00.720 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:00.759 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:00.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:00.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:00.783 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T13:50:00.845 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:00 smithi043 ceph-mon[42099]: pgmap v280: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-03-30T13:50:00.845 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:00 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2497440956' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]: dispatch 2024-03-30T13:50:00.845 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:00 smithi043 ceph-mon[42099]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-30T13:50:00.845 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:00 smithi043 ceph-mon[42099]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-30T13:50:00.846 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:00 smithi043 ceph-mon[29963]: pgmap v280: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-03-30T13:50:00.846 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:00 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2497440956' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]: dispatch 2024-03-30T13:50:00.846 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:00 smithi043 ceph-mon[29963]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-30T13:50:00.846 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:00 smithi043 ceph-mon[29963]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-30T13:50:00.861 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:00.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:00.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:00.977 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 46 16 81061 81045 110.096 69.1875 0.0225167 0.00907761 2024-03-30T13:50:01.007 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:01.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:01.055 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:01.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:01.156 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:01.209 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:01.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:01.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:01.312 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:01.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:01.344 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:01.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:01.409 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:50:01.444 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:01.466 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:01 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2497440956' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]': finished 2024-03-30T13:50:01.466 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:01 smithi043 ceph-mon[42099]: osdmap e163: 8 total, 8 up, 8 in 2024-03-30T13:50:01.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:01.503 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:01.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:01.604 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:01.659 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:01.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:01.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:01.718 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:01 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2497440956' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]': finished 2024-03-30T13:50:01.718 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:01 smithi043 ceph-mon[29963]: osdmap e163: 8 total, 8 up, 8 in 2024-03-30T13:50:01.718 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:01 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:50:01.719 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:01 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:50:01.719 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:01 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:50:01.719 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:01 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:50:01.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:01 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2497440956' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]': finished 2024-03-30T13:50:01.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:01 smithi113 ceph-mon[34341]: osdmap e163: 8 total, 8 up, 8 in 2024-03-30T13:50:01.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:01 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:50:01.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:01 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:50:01.762 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:01.792 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:01.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:01.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:01.894 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:50:01.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:01.945 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:01.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:50:01.977 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 47 16 82084 82068 109.114 63.9375 0.0107964 0.00915934 2024-03-30T13:50:02.047 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:02.076 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:02.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:02.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:02.178 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:02.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:02.217 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:02.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:02.251 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-03-30T13:50:02.318 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:02.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:02.356 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:02.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:02.457 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:02.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:02.503 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:02.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:02.603 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:02.608 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:50:02.638 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:50:02.642 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:02.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:02.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:02.687 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:02 smithi043 ceph-mon[29963]: pgmap v282: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 792 B/s rd, 62 MiB/s wr, 995 op/s 2024-03-30T13:50:02.688 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:02 smithi043 ceph-mon[29963]: osdmap e164: 8 total, 8 up, 8 in 2024-03-30T13:50:02.688 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:02 smithi043 ceph-mon[42099]: pgmap v282: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 792 B/s rd, 62 MiB/s wr, 995 op/s 2024-03-30T13:50:02.688 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:02 smithi043 ceph-mon[42099]: osdmap e164: 8 total, 8 up, 8 in 2024-03-30T13:50:02.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:02 smithi113 ceph-mon[34341]: pgmap v282: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 792 B/s rd, 62 MiB/s wr, 995 op/s 2024-03-30T13:50:02.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:02 smithi113 ceph-mon[34341]: osdmap e164: 8 total, 8 up, 8 in 2024-03-30T13:50:02.744 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:02.779 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:02.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:02.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:02.881 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:02.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:02.916 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:02.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:02.977 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 48 16 83223 83207 108.323 71.1875 0.017945 0.00922633 2024-03-30T13:50:03.017 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:03.049 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:03.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:03.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:03.151 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:03.185 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:03.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:03.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:03.287 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:03.326 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:03.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:03.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:03.428 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:03.465 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:03.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:03.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:03.567 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:50:03.616 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:03.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:03.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:50:03.718 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:03.727 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:50:03.746 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:03.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:03.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:03.848 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:03.904 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:03.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:03.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:03.977 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 49 16 84406 84390 107.621 73.9375 0.0113597 0.00928729 2024-03-30T13:50:04.006 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:50:04.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:04.052 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:04.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:50:04.153 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:04.180 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:04.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:04.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:04.285 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:04.326 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:04.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:04.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:04.428 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:50:04.469 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:04.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:04.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:50:04.496 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T13:50:04.531 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:04 smithi113 ceph-mon[34341]: pgmap v284: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-30T13:50:04.531 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:04 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2581241442' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:50:04.571 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:50:04.613 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:04.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:04.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:50:04.624 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:50:04.624 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [], 2024-03-30T13:50:04.624 INFO:teuthology.orchestra.run.smithi113.stdout: "complaint_time": 30, 2024-03-30T13:50:04.625 INFO:teuthology.orchestra.run.smithi113.stdout: "num_blocked_ops": 0 2024-03-30T13:50:04.625 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:50:04.715 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:04.749 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:04.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:04.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:04.846 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:04 smithi043 ceph-mon[29963]: pgmap v284: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-30T13:50:04.846 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:04 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2581241442' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:50:04.846 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:04 smithi043 ceph-mon[42099]: pgmap v284: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-30T13:50:04.846 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:04 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2581241442' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:50:04.851 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:04.888 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:04.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:04.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:04.977 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 50 16 85588 85572 106.946 73.875 0.0113404 0.00934526 2024-03-30T13:50:04.994 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:05.035 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:05.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:05.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:05.137 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:05.177 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:05.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:05.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:05.278 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T13:50:05.320 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:05.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:05.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T13:50:05.422 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:05.463 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:05.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:05.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:05.558 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-03-30T13:50:05.564 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:05.634 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:05.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:05.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:05.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:05 smithi043 ceph-mon[29963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:50:05.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:05 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2581241442' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:50:05.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:05 smithi043 ceph-mon[29963]: osdmap e165: 8 total, 8 up, 8 in 2024-03-30T13:50:05.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:05 smithi043 ceph-mon[29963]: 4.15 scrub starts 2024-03-30T13:50:05.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:05 smithi043 ceph-mon[29963]: 4.15 scrub ok 2024-03-30T13:50:05.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:05 smithi043 ceph-mon[29963]: 4.5 scrub starts 2024-03-30T13:50:05.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:05 smithi043 ceph-mon[29963]: 4.5 scrub ok 2024-03-30T13:50:05.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:05 smithi043 ceph-mon[42099]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:50:05.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:05 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2581241442' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:50:05.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:05 smithi043 ceph-mon[42099]: osdmap e165: 8 total, 8 up, 8 in 2024-03-30T13:50:05.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:05 smithi043 ceph-mon[42099]: 4.15 scrub starts 2024-03-30T13:50:05.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:05 smithi043 ceph-mon[42099]: 4.15 scrub ok 2024-03-30T13:50:05.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:05 smithi043 ceph-mon[42099]: 4.5 scrub starts 2024-03-30T13:50:05.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:05 smithi043 ceph-mon[42099]: 4.5 scrub ok 2024-03-30T13:50:05.736 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:05.777 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:05.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:05.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:05.878 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:05.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:05.938 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:05.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:05.967 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:50:05.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:05 smithi113 ceph-mon[34341]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:50:05.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:05 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2581241442' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:50:05.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:05 smithi113 ceph-mon[34341]: osdmap e165: 8 total, 8 up, 8 in 2024-03-30T13:50:05.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:05 smithi113 ceph-mon[34341]: 4.15 scrub starts 2024-03-30T13:50:05.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:05 smithi113 ceph-mon[34341]: 4.15 scrub ok 2024-03-30T13:50:05.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:05 smithi113 ceph-mon[34341]: 4.5 scrub starts 2024-03-30T13:50:05.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:05 smithi113 ceph-mon[34341]: 4.5 scrub ok 2024-03-30T13:50:05.977 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 51 16 86796 86780 106.329 75.5 0.0102157 0.00939983 2024-03-30T13:50:06.040 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:06.078 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:06.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:06.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:06.180 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:06.218 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:06.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:06.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:06.320 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:06.360 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:06.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:06.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:06.375 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T13:50:06.375 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:50:06.375 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T13:50:06.375 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-30T13:50:06.376 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.2 2024-03-30T13:50:06.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:50:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:50:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:50:06.462 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:06.495 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:06.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:06.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:06.502 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:50:06.554 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:50:06.597 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:06.598 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T13:50:06.626 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:06.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:06.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:06.639 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:50:06.681 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:50:06.681 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:06 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:50:06.681 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:06 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:50:06.681 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:06 smithi043 ceph-mon[29963]: 3.2 scrub starts 2024-03-30T13:50:06.681 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:06 smithi043 ceph-mon[29963]: 3.2 scrub ok 2024-03-30T13:50:06.681 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:06 smithi043 ceph-mon[29963]: pgmap v286: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 682 B/s rd, 58 MiB/s wr, 929 op/s 2024-03-30T13:50:06.682 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:06 smithi043 ceph-mon[29963]: osdmap e166: 8 total, 8 up, 8 in 2024-03-30T13:50:06.682 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:06 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1792977374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:50:06.682 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:06 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:50:06.682 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:06 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:50:06.682 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:06 smithi043 ceph-mon[42099]: 3.2 scrub starts 2024-03-30T13:50:06.682 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:06 smithi043 ceph-mon[42099]: 3.2 scrub ok 2024-03-30T13:50:06.682 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:06 smithi043 ceph-mon[42099]: pgmap v286: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 682 B/s rd, 58 MiB/s wr, 929 op/s 2024-03-30T13:50:06.682 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:06 smithi043 ceph-mon[42099]: osdmap e166: 8 total, 8 up, 8 in 2024-03-30T13:50:06.682 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:06 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1792977374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:50:06.683 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:50:06 smithi043 systemd[1]: Stopping Ceph osd.2 for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T13:50:06.721 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:50:06.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:06 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:50:06.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:06 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:50:06.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:06 smithi113 ceph-mon[34341]: 3.2 scrub starts 2024-03-30T13:50:06.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:06 smithi113 ceph-mon[34341]: 3.2 scrub ok 2024-03-30T13:50:06.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:06 smithi113 ceph-mon[34341]: pgmap v286: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 682 B/s rd, 58 MiB/s wr, 929 op/s 2024-03-30T13:50:06.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:06 smithi113 ceph-mon[34341]: osdmap e166: 8 total, 8 up, 8 in 2024-03-30T13:50:06.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:06 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1792977374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:50:06.728 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:06.762 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:50:06.763 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:06.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:06.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:06.800 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:50:06.804 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:50:06.804 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:50:06.804 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T13:50:06.804 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:50:06.804 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:50:06.804 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:50:06.805 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:50:06.805 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:50:06.865 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:06.901 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:06.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:06.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:06.978 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 52 16 87915 87899 105.629 69.9375 0.014044 0.0094616 2024-03-30T13:50:07.003 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:07.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:07.044 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:07.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:07.145 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:07.182 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:07.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:07.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:07.284 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:07.310 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T13:50:07.330 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:07.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:07.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:07.399 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:50:07 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2[59302]: 2024-03-30T13:50:07.126+0000 7f59e935a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T13:50:07.399 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:50:07 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2[59302]: 2024-03-30T13:50:07.126+0000 7f59e935a700 -1 osd.2 166 *** Got signal Terminated *** 2024-03-30T13:50:07.400 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:50:07 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2[59302]: 2024-03-30T13:50:07.126+0000 7f59e935a700 -1 osd.2 166 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T13:50:07.432 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:07.479 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:07.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:07.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:07.580 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:07.604 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:07.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:07.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:07.706 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:07.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:07.731 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:07.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:07.785 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:07 smithi113 ceph-mon[34341]: osd.2 marked itself down and dead 2024-03-30T13:50:07.832 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:07.838 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:50:07.838 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:50:07.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:07.857 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:07.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:07.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:07 smithi043 ceph-mon[29963]: osd.2 marked itself down and dead 2024-03-30T13:50:07.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:07 smithi043 ceph-mon[42099]: osd.2 marked itself down and dead 2024-03-30T13:50:07.958 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:07.977 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 53 16 88563 88547 104.401 40.5 0.00380682 0.00949003 2024-03-30T13:50:07.982 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:07.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:07.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:08.007 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:50:08.084 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:08.107 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:08.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:08.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:08.209 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:08.232 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:08.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:08.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:08.334 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:08.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:08.358 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:08.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:08.459 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:08.483 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:08.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:08.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:08.512 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-30T13:50:08.585 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:08.625 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:08.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:08.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:08.649 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:50:08 smithi043 podman[125799]: 2024-03-30 13:50:08.267378115 +0000 UTC m=+1.470954340 container died 2ef34f1a5835fd1de68b93fb8eb941690cc6fb19ed4c156d2c67b7cdac24b047 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2, GIT_CLEAN=True, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-30T13:50:08.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:08 smithi043 ceph-mon[29963]: 3.1a scrub starts 2024-03-30T13:50:08.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:08 smithi043 ceph-mon[29963]: 3.1a scrub ok 2024-03-30T13:50:08.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:08 smithi043 ceph-mon[29963]: pgmap v288: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 682 B/s rd, 58 MiB/s wr, 929 op/s 2024-03-30T13:50:08.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:08 smithi043 ceph-mon[29963]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T13:50:08.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:08 smithi043 ceph-mon[29963]: osdmap e167: 8 total, 7 up, 8 in 2024-03-30T13:50:08.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:08 smithi043 ceph-mon[42099]: 3.1a scrub starts 2024-03-30T13:50:08.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:08 smithi043 ceph-mon[42099]: 3.1a scrub ok 2024-03-30T13:50:08.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:08 smithi043 ceph-mon[42099]: pgmap v288: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 682 B/s rd, 58 MiB/s wr, 929 op/s 2024-03-30T13:50:08.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:08 smithi043 ceph-mon[42099]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T13:50:08.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:08 smithi043 ceph-mon[42099]: osdmap e167: 8 total, 7 up, 8 in 2024-03-30T13:50:08.727 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:08.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:08.775 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:08.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:08.876 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:08.908 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:08.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:08.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:08.913 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:50:08.972 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:50:08 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:50:08.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:08 smithi113 ceph-mon[34341]: 3.1a scrub starts 2024-03-30T13:50:08.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:08 smithi113 ceph-mon[34341]: 3.1a scrub ok 2024-03-30T13:50:08.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:08 smithi113 ceph-mon[34341]: pgmap v288: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 682 B/s rd, 58 MiB/s wr, 929 op/s 2024-03-30T13:50:08.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:08 smithi113 ceph-mon[34341]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T13:50:08.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:08 smithi113 ceph-mon[34341]: osdmap e167: 8 total, 7 up, 8 in 2024-03-30T13:50:08.977 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 54 15 89085 89070 103.073 32.6875 0.00700175 0.00969693 2024-03-30T13:50:09.010 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:09.048 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:09.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:09.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:09.150 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:09.185 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:09.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:09.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:09.287 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:09.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:09.323 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:09.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:09.424 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:09.461 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:09.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:09.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:09.563 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:09.603 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:09.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:09.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:09.705 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:09.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:09 smithi113 ceph-mon[34341]: 6.16 scrub starts 2024-03-30T13:50:09.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:09 smithi113 ceph-mon[34341]: 6.16 scrub ok 2024-03-30T13:50:09.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:09 smithi113 ceph-mon[34341]: 5.e scrub starts 2024-03-30T13:50:09.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:09 smithi113 ceph-mon[34341]: 5.e scrub ok 2024-03-30T13:50:09.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:09 smithi113 ceph-mon[34341]: osdmap e168: 8 total, 7 up, 8 in 2024-03-30T13:50:09.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:09 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:50:09.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:09 smithi113 ceph-mon[34341]: 3.a scrub starts 2024-03-30T13:50:09.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:09 smithi113 ceph-mon[34341]: 3.a scrub ok 2024-03-30T13:50:09.740 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:09.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:09.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:09.842 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:09.880 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:09.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:09.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:09 smithi043 ceph-mon[29963]: 6.16 scrub starts 2024-03-30T13:50:09.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:09 smithi043 ceph-mon[29963]: 6.16 scrub ok 2024-03-30T13:50:09.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:09 smithi043 ceph-mon[29963]: 5.e scrub starts 2024-03-30T13:50:09.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:09 smithi043 ceph-mon[29963]: 5.e scrub ok 2024-03-30T13:50:09.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:09 smithi043 ceph-mon[29963]: osdmap e168: 8 total, 7 up, 8 in 2024-03-30T13:50:09.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:09 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:50:09.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:09 smithi043 ceph-mon[29963]: 3.a scrub starts 2024-03-30T13:50:09.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:09 smithi043 ceph-mon[29963]: 3.a scrub ok 2024-03-30T13:50:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:09 smithi043 ceph-mon[42099]: 6.16 scrub starts 2024-03-30T13:50:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:09 smithi043 ceph-mon[42099]: 6.16 scrub ok 2024-03-30T13:50:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:09 smithi043 ceph-mon[42099]: 5.e scrub starts 2024-03-30T13:50:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:09 smithi043 ceph-mon[42099]: 5.e scrub ok 2024-03-30T13:50:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:09 smithi043 ceph-mon[42099]: osdmap e168: 8 total, 7 up, 8 in 2024-03-30T13:50:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:09 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:50:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:09 smithi043 ceph-mon[42099]: 3.a scrub starts 2024-03-30T13:50:09.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:09 smithi043 ceph-mon[42099]: 3.a scrub ok 2024-03-30T13:50:09.960 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:50:09.977 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 55 16 90354 90338 102.639 79.25 0.0175768 0.0097379 2024-03-30T13:50:09.981 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:10.017 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:10.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:10.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:10.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:10.171 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:10.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:10.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:10.273 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:10.339 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:10.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:10.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:10.399 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:50:10 smithi043 podman[125799]: 2024-03-30 13:50:10.202933067 +0000 UTC m=+3.406509298 container cleanup 2ef34f1a5835fd1de68b93fb8eb941690cc6fb19ed4c156d2c67b7cdac24b047 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-30T13:50:10.401 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:50:10 smithi043 bash[125799]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2 2024-03-30T13:50:10.442 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:10.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:10.480 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:10.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:10.581 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:10.616 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:10.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:10.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:10.717 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:10.753 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:10.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:10.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:10.781 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:10 smithi113 ceph-mon[34341]: 5.18 scrub starts 2024-03-30T13:50:10.781 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:10 smithi113 ceph-mon[34341]: 5.18 scrub ok 2024-03-30T13:50:10.781 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:10 smithi113 ceph-mon[34341]: pgmap v291: 148 pgs: 12 active+undersized+degraded, 21 peering, 11 active+undersized, 104 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 60 MiB/s wr, 961 op/s; 22311/266904 objects degraded (8.359%) 2024-03-30T13:50:10.781 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:10 smithi113 ceph-mon[34341]: Health check failed: Degraded data redundancy: 22311/266904 objects degraded (8.359%), 12 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:10.781 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:10 smithi113 ceph-mon[34341]: 5.f scrub starts 2024-03-30T13:50:10.781 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:10 smithi113 ceph-mon[34341]: 5.f scrub ok 2024-03-30T13:50:10.781 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:10 smithi113 ceph-mon[34341]: 6.4 scrub starts 2024-03-30T13:50:10.781 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:10 smithi113 ceph-mon[34341]: 6.4 scrub ok 2024-03-30T13:50:10.855 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:10.871 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:50:10.871 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:50:10.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:10.899 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:10.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:10.900 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:50:10 smithi043 podman[126016]: 2024-03-30 13:50:10.404701302 +0000 UTC m=+2.314627037 container remove 2ef34f1a5835fd1de68b93fb8eb941690cc6fb19ed4c156d2c67b7cdac24b047 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-30T13:50:10.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:10 smithi043 ceph-mon[29963]: 5.18 scrub starts 2024-03-30T13:50:10.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:10 smithi043 ceph-mon[29963]: 5.18 scrub ok 2024-03-30T13:50:10.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:10 smithi043 ceph-mon[29963]: pgmap v291: 148 pgs: 12 active+undersized+degraded, 21 peering, 11 active+undersized, 104 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 60 MiB/s wr, 961 op/s; 22311/266904 objects degraded (8.359%) 2024-03-30T13:50:10.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:10 smithi043 ceph-mon[29963]: Health check failed: Degraded data redundancy: 22311/266904 objects degraded (8.359%), 12 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:10.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:10 smithi043 ceph-mon[29963]: 5.f scrub starts 2024-03-30T13:50:10.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:10 smithi043 ceph-mon[29963]: 5.f scrub ok 2024-03-30T13:50:10.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:10 smithi043 ceph-mon[29963]: 6.4 scrub starts 2024-03-30T13:50:10.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:10 smithi043 ceph-mon[29963]: 6.4 scrub ok 2024-03-30T13:50:10.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:10 smithi043 ceph-mon[42099]: 5.18 scrub starts 2024-03-30T13:50:10.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:10 smithi043 ceph-mon[42099]: 5.18 scrub ok 2024-03-30T13:50:10.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:10 smithi043 ceph-mon[42099]: pgmap v291: 148 pgs: 12 active+undersized+degraded, 21 peering, 11 active+undersized, 104 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 60 MiB/s wr, 961 op/s; 22311/266904 objects degraded (8.359%) 2024-03-30T13:50:10.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:10 smithi043 ceph-mon[42099]: Health check failed: Degraded data redundancy: 22311/266904 objects degraded (8.359%), 12 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:10.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:10 smithi043 ceph-mon[42099]: 5.f scrub starts 2024-03-30T13:50:10.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:10 smithi043 ceph-mon[42099]: 5.f scrub ok 2024-03-30T13:50:10.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:10 smithi043 ceph-mon[42099]: 6.4 scrub starts 2024-03-30T13:50:10.902 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:10 smithi043 ceph-mon[42099]: 6.4 scrub ok 2024-03-30T13:50:10.978 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 56 16 91661 91645 102.265 81.6875 0.0158786 0.00977343 2024-03-30T13:50:11.000 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:11.045 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:11.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:11.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:11.146 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:11.181 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:11.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:11.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:11.282 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:11.322 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:11.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:11.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:11.424 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:11.461 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:11.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:11.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:11.563 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:11.586 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-03-30T13:50:11.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:11.610 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:11.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:11.711 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:11.746 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:11.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:11.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:11.848 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:11.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:11.886 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:11.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:11.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:11 smithi043 ceph-mon[29963]: 4.9 scrub starts 2024-03-30T13:50:11.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:11 smithi043 ceph-mon[29963]: 4.9 scrub ok 2024-03-30T13:50:11.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:11 smithi043 ceph-mon[29963]: 3.c scrub starts 2024-03-30T13:50:11.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:11 smithi043 ceph-mon[29963]: 3.c scrub ok 2024-03-30T13:50:11.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:11 smithi043 ceph-mon[29963]: 4.f scrub starts 2024-03-30T13:50:11.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:11 smithi043 ceph-mon[29963]: 4.f scrub ok 2024-03-30T13:50:11.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:11 smithi043 ceph-mon[42099]: 4.9 scrub starts 2024-03-30T13:50:11.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:11 smithi043 ceph-mon[42099]: 4.9 scrub ok 2024-03-30T13:50:11.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:11 smithi043 ceph-mon[42099]: 3.c scrub starts 2024-03-30T13:50:11.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:11 smithi043 ceph-mon[42099]: 3.c scrub ok 2024-03-30T13:50:11.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:11 smithi043 ceph-mon[42099]: 4.f scrub starts 2024-03-30T13:50:11.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:11 smithi043 ceph-mon[42099]: 4.f scrub ok 2024-03-30T13:50:11.953 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:50:11.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:11 smithi113 ceph-mon[34341]: 4.9 scrub starts 2024-03-30T13:50:11.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:11 smithi113 ceph-mon[34341]: 4.9 scrub ok 2024-03-30T13:50:11.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:11 smithi113 ceph-mon[34341]: 3.c scrub starts 2024-03-30T13:50:11.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:11 smithi113 ceph-mon[34341]: 3.c scrub ok 2024-03-30T13:50:11.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:11 smithi113 ceph-mon[34341]: 4.f scrub starts 2024-03-30T13:50:11.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:11 smithi113 ceph-mon[34341]: 4.f scrub ok 2024-03-30T13:50:11.978 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 57 16 92845 92829 101.769 74 0.017924 0.00982081 2024-03-30T13:50:11.987 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:12.014 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:12.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:12.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:12.115 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:12.153 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:12.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:12.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:12.255 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:12.279 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:12.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:12.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:12.381 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:12.419 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:12.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:12.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:12.521 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:12.557 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:12.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:12.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:12.659 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:12.694 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:12.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:12.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:12.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:12 smithi113 ceph-mon[34341]: pgmap v292: 148 pgs: 12 active+undersized+degraded, 21 peering, 11 active+undersized, 104 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 49 MiB/s wr, 789 op/s; 22311/266904 objects degraded (8.359%) 2024-03-30T13:50:12.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:12 smithi113 ceph-mon[34341]: 6.1d scrub starts 2024-03-30T13:50:12.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:12 smithi113 ceph-mon[34341]: 6.1d scrub ok 2024-03-30T13:50:12.796 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:12.808 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:50:12.871 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:12.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:12.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:12.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:12 smithi043 ceph-mon[42099]: pgmap v292: 148 pgs: 12 active+undersized+degraded, 21 peering, 11 active+undersized, 104 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 49 MiB/s wr, 789 op/s; 22311/266904 objects degraded (8.359%) 2024-03-30T13:50:12.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:12 smithi043 ceph-mon[42099]: 6.1d scrub starts 2024-03-30T13:50:12.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:12 smithi043 ceph-mon[42099]: 6.1d scrub ok 2024-03-30T13:50:12.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:12 smithi043 ceph-mon[29963]: pgmap v292: 148 pgs: 12 active+undersized+degraded, 21 peering, 11 active+undersized, 104 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 49 MiB/s wr, 789 op/s; 22311/266904 objects degraded (8.359%) 2024-03-30T13:50:12.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:12 smithi043 ceph-mon[29963]: 6.1d scrub starts 2024-03-30T13:50:12.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:12 smithi043 ceph-mon[29963]: 6.1d scrub ok 2024-03-30T13:50:12.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:12.978 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 58 15 94034 94019 101.297 74.375 0.00740251 0.00986717 2024-03-30T13:50:13.017 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:13.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:13.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:13.119 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:13.152 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:13.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:13.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:13.296 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:13.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:13.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:13.398 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:13.436 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:13.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:13.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:13.538 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:13.575 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:13.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:13.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:13.677 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:13.709 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:13.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:13.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:13.810 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:13.845 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:13.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:13.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:13.947 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:13.978 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 59 16 95343 95327 100.965 81.75 0.00482005 0.00989951 2024-03-30T13:50:13.995 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:13.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:13.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:14.068 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:13 smithi113 ceph-mon[34341]: 3.1d scrub starts 2024-03-30T13:50:14.068 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:13 smithi113 ceph-mon[34341]: 3.1d scrub ok 2024-03-30T13:50:14.069 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:13 smithi113 ceph-mon[34341]: 3.10 scrub starts 2024-03-30T13:50:14.069 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:13 smithi113 ceph-mon[34341]: 3.10 scrub ok 2024-03-30T13:50:14.069 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:13 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:50:14.069 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:13 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:50:14.069 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:13 smithi113 ceph-mon[34341]: 4.c scrub starts 2024-03-30T13:50:14.069 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:13 smithi113 ceph-mon[34341]: 4.c scrub ok 2024-03-30T13:50:14.069 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:13 smithi113 ceph-mon[34341]: 5.13 scrub starts 2024-03-30T13:50:14.069 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:13 smithi113 ceph-mon[34341]: 5.13 scrub ok 2024-03-30T13:50:14.098 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:14.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:13 smithi043 ceph-mon[29963]: 3.1d scrub starts 2024-03-30T13:50:14.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:13 smithi043 ceph-mon[29963]: 3.1d scrub ok 2024-03-30T13:50:14.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:13 smithi043 ceph-mon[29963]: 3.10 scrub starts 2024-03-30T13:50:14.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:13 smithi043 ceph-mon[29963]: 3.10 scrub ok 2024-03-30T13:50:14.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:13 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:50:14.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:13 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:50:14.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:13 smithi043 ceph-mon[29963]: 4.c scrub starts 2024-03-30T13:50:14.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:13 smithi043 ceph-mon[29963]: 4.c scrub ok 2024-03-30T13:50:14.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:13 smithi043 ceph-mon[29963]: 5.13 scrub starts 2024-03-30T13:50:14.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:13 smithi043 ceph-mon[29963]: 5.13 scrub ok 2024-03-30T13:50:14.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:13 smithi043 ceph-mon[42099]: 3.1d scrub starts 2024-03-30T13:50:14.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:13 smithi043 ceph-mon[42099]: 3.1d scrub ok 2024-03-30T13:50:14.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:13 smithi043 ceph-mon[42099]: 3.10 scrub starts 2024-03-30T13:50:14.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:13 smithi043 ceph-mon[42099]: 3.10 scrub ok 2024-03-30T13:50:14.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:13 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:50:14.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:13 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:50:14.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:13 smithi043 ceph-mon[42099]: 4.c scrub starts 2024-03-30T13:50:14.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:13 smithi043 ceph-mon[42099]: 4.c scrub ok 2024-03-30T13:50:14.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:13 smithi043 ceph-mon[42099]: 5.13 scrub starts 2024-03-30T13:50:14.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:13 smithi043 ceph-mon[42099]: 5.13 scrub ok 2024-03-30T13:50:14.144 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:14.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:14.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:14.246 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:14.282 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:50:14.282 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:50:14.283 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:14.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:14.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:14.385 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:14.421 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:14.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:14.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:14.523 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:14.558 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:14.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:14.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:14.660 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:14.695 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:14.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:14.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:14.796 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:14.835 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:14.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:14.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:14.937 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:14.970 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:50:14 smithi043 podman[126554]: 2024-03-30 13:50:14.703493132 +0000 UTC m=+0.563533986 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T13:50:14.970 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:50:14 smithi043 podman[126554]: 2024-03-30 13:50:14.820839489 +0000 UTC m=+0.680880386 container create d592489e4f3dc28f38f2213f9a5d148b2cbe494ff0965fc6c535000f09082863 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-03-30T13:50:14.978 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-03-30T13:50:14.977946+0000 min lat: 0.00129646 max lat: 1.05462 avg lat: 0.00992685 2024-03-30T13:50:14.978 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T13:50:14.978 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 60 16 96693 96677 100.689 84.375 0.00628573 0.00992685 2024-03-30T13:50:14.985 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:14.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:14.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:14.994 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-03-30T13:50:15.042 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-03-30T13:50:15.087 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:15.127 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:15.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:15.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:15.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:14 smithi043 ceph-mon[29963]: pgmap v293: 148 pgs: 14 active+undersized+degraded, 17 peering, 13 active+undersized, 104 active+clean; 5.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 645 B/s rd, 46 MiB/s wr, 738 op/s; 34150/270267 objects degraded (12.636%); 8.1 KiB/s, 0 objects/s recovering 2024-03-30T13:50:15.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:14 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/747932846' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:50:15.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:14 smithi043 ceph-mon[29963]: 4.1a scrub starts 2024-03-30T13:50:15.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:14 smithi043 ceph-mon[29963]: 4.1a scrub ok 2024-03-30T13:50:15.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:14 smithi043 ceph-mon[29963]: 6.10 scrub starts 2024-03-30T13:50:15.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:14 smithi043 ceph-mon[29963]: 6.10 scrub ok 2024-03-30T13:50:15.223 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:14 smithi043 ceph-mon[42099]: pgmap v293: 148 pgs: 14 active+undersized+degraded, 17 peering, 13 active+undersized, 104 active+clean; 5.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 645 B/s rd, 46 MiB/s wr, 738 op/s; 34150/270267 objects degraded (12.636%); 8.1 KiB/s, 0 objects/s recovering 2024-03-30T13:50:15.223 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:14 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/747932846' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:50:15.223 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:14 smithi043 ceph-mon[42099]: 4.1a scrub starts 2024-03-30T13:50:15.223 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:14 smithi043 ceph-mon[42099]: 4.1a scrub ok 2024-03-30T13:50:15.223 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:14 smithi043 ceph-mon[42099]: 6.10 scrub starts 2024-03-30T13:50:15.223 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:14 smithi043 ceph-mon[42099]: 6.10 scrub ok 2024-03-30T13:50:15.229 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:15.269 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:15.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:15.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:15.371 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:15.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:15.417 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:15.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:15.435 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:50:15.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:14 smithi113 ceph-mon[34341]: pgmap v293: 148 pgs: 14 active+undersized+degraded, 17 peering, 13 active+undersized, 104 active+clean; 5.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 645 B/s rd, 46 MiB/s wr, 738 op/s; 34150/270267 objects degraded (12.636%); 8.1 KiB/s, 0 objects/s recovering 2024-03-30T13:50:15.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:14 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/747932846' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:50:15.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:14 smithi113 ceph-mon[34341]: 4.1a scrub starts 2024-03-30T13:50:15.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:14 smithi113 ceph-mon[34341]: 4.1a scrub ok 2024-03-30T13:50:15.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:14 smithi113 ceph-mon[34341]: 6.10 scrub starts 2024-03-30T13:50:15.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:14 smithi113 ceph-mon[34341]: 6.10 scrub ok 2024-03-30T13:50:15.518 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:15.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:15.556 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:15.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:15.651 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:50:15 smithi043 podman[126554]: 2024-03-30 13:50:15.327335026 +0000 UTC m=+1.187375880 container init d592489e4f3dc28f38f2213f9a5d148b2cbe494ff0965fc6c535000f09082863 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-4e50bbc, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-30T13:50:15.651 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:50:15 smithi043 podman[126554]: 2024-03-30 13:50:15.338293944 +0000 UTC m=+1.198334791 container start d592489e4f3dc28f38f2213f9a5d148b2cbe494ff0965fc6c535000f09082863 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2) 2024-03-30T13:50:15.651 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:50:15 smithi043 podman[126554]: 2024-03-30 13:50:15.435475032 +0000 UTC m=+1.295515884 container attach d592489e4f3dc28f38f2213f9a5d148b2cbe494ff0965fc6c535000f09082863 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-30T13:50:15.657 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:15.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:15.694 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:15.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:15.795 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:15.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:15.833 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:15.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:15.934 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:15.970 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:15.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:15.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:15.978 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 61 16 97978 97962 100.354 80.3125 0.0064185 0.0099599 2024-03-30T13:50:16.072 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:16.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:15 smithi043 ceph-mon[29963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:50:16.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:15 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/747932846' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:50:16.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:15 smithi043 ceph-mon[29963]: osdmap e169: 8 total, 7 up, 8 in 2024-03-30T13:50:16.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:15 smithi043 ceph-mon[29963]: 5.12 scrub starts 2024-03-30T13:50:16.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:15 smithi043 ceph-mon[29963]: 5.12 scrub ok 2024-03-30T13:50:16.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:15 smithi043 ceph-mon[42099]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:50:16.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:15 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/747932846' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:50:16.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:15 smithi043 ceph-mon[42099]: osdmap e169: 8 total, 7 up, 8 in 2024-03-30T13:50:16.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:15 smithi043 ceph-mon[42099]: 5.12 scrub starts 2024-03-30T13:50:16.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:15 smithi043 ceph-mon[42099]: 5.12 scrub ok 2024-03-30T13:50:16.113 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:16.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:16.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:16.215 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:16.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:15 smithi113 ceph-mon[34341]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:50:16.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:15 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/747932846' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:50:16.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:15 smithi113 ceph-mon[34341]: osdmap e169: 8 total, 7 up, 8 in 2024-03-30T13:50:16.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:15 smithi113 ceph-mon[34341]: 5.12 scrub starts 2024-03-30T13:50:16.223 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:15 smithi113 ceph-mon[34341]: 5.12 scrub ok 2024-03-30T13:50:16.256 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:16.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:16.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:16.358 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:16.392 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:16.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:16.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:16.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:50:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:50:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:50:16.399 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 13:50:16 smithi043 podman[126554]: 2024-03-30 13:50:16.258259329 +0000 UTC m=+2.118300174 container died d592489e4f3dc28f38f2213f9a5d148b2cbe494ff0965fc6c535000f09082863 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-30T13:50:16.494 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:16.533 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:16.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:16.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:16.635 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:16.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:16.675 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:16.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:16.776 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:16.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:16.818 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:16.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:16.919 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:16.953 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:16.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:16.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:16.978 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 62 16 99348 99332 100.117 85.625 0.00720913 0.00998339 2024-03-30T13:50:17.055 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.2.service' 2024-03-30T13:50:17.055 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:17.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:17.090 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:17.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:17.127 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T13:50:17.127 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-30T13:50:17.127 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-03-30T13:50:17.127 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi043.front.sepia.ceph.com 2024-03-30T13:50:17.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -n osd.2 --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-03-30T13:50:17.191 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:16 smithi113 ceph-mon[34341]: pgmap v295: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 35712/285498 objects degraded (12.509%); 8.1 KiB/s, 0 objects/s recovering 2024-03-30T13:50:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:16 smithi113 ceph-mon[34341]: 4.8 scrub starts 2024-03-30T13:50:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:16 smithi113 ceph-mon[34341]: 4.8 scrub ok 2024-03-30T13:50:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:16 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 35712/285498 objects degraded (12.509%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:16 smithi113 ceph-mon[34341]: 6.1e scrub starts 2024-03-30T13:50:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:16 smithi113 ceph-mon[34341]: 6.1e scrub ok 2024-03-30T13:50:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:16 smithi113 ceph-mon[34341]: 8.7 scrub starts 2024-03-30T13:50:17.226 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:17.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:17.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:17.327 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:17.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:17.372 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:17.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:17.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:16 smithi043 ceph-mon[29963]: pgmap v295: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 35712/285498 objects degraded (12.509%); 8.1 KiB/s, 0 objects/s recovering 2024-03-30T13:50:17.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:16 smithi043 ceph-mon[29963]: 4.8 scrub starts 2024-03-30T13:50:17.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:16 smithi043 ceph-mon[29963]: 4.8 scrub ok 2024-03-30T13:50:17.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:16 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 35712/285498 objects degraded (12.509%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:17.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:16 smithi043 ceph-mon[29963]: 6.1e scrub starts 2024-03-30T13:50:17.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:16 smithi043 ceph-mon[29963]: 6.1e scrub ok 2024-03-30T13:50:17.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:16 smithi043 ceph-mon[29963]: 8.7 scrub starts 2024-03-30T13:50:17.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:16 smithi043 ceph-mon[42099]: pgmap v295: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 35712/285498 objects degraded (12.509%); 8.1 KiB/s, 0 objects/s recovering 2024-03-30T13:50:17.399 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:16 smithi043 ceph-mon[42099]: 4.8 scrub starts 2024-03-30T13:50:17.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:16 smithi043 ceph-mon[42099]: 4.8 scrub ok 2024-03-30T13:50:17.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:16 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 35712/285498 objects degraded (12.509%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:17.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:16 smithi043 ceph-mon[42099]: 6.1e scrub starts 2024-03-30T13:50:17.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:16 smithi043 ceph-mon[42099]: 6.1e scrub ok 2024-03-30T13:50:17.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:16 smithi043 ceph-mon[42099]: 8.7 scrub starts 2024-03-30T13:50:17.473 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:17.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:17.515 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:17.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:17.592 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:50:17.592 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:50:17.616 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:17.656 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:17.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:17.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:17.758 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:17.790 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:17.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:17.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:17.892 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:17.923 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:17.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:17.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:17.978 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 63 16 100460 100444 99.6308 69.5 0.0307795 0.0100319 2024-03-30T13:50:18.025 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:18.060 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:18.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:18.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:18.162 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:18.194 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:18.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:18.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:18.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T13:50:18.296 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:18.338 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:18.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:18.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:18.428 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:50:18.428 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:50:18.428 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:50:18.428 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:50:18.428 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:50:18.429 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:50:18.429 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:50:18.429 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:50:18.429 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:50:18.429 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:50:18.429 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:50:18.429 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:50:18.429 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:50:18.429 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:50:18.429 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:50:18.440 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:18.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:18 smithi113 ceph-mon[34341]: pgmap v296: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 573 B/s rd, 64 MiB/s wr, 1.02k op/s; 35712/285498 objects degraded (12.509%); 7.2 KiB/s, 0 objects/s recovering 2024-03-30T13:50:18.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:18 smithi113 ceph-mon[34341]: osdmap e170: 8 total, 7 up, 8 in 2024-03-30T13:50:18.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:18 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3331922532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:50:18.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:18 smithi113 ceph-mon[34341]: 3.16 scrub starts 2024-03-30T13:50:18.472 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:18 smithi113 ceph-mon[34341]: 3.16 scrub ok 2024-03-30T13:50:18.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:18 smithi113 ceph-mon[34341]: 8.3 scrub starts 2024-03-30T13:50:18.474 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:18.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:18.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:18.537 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-30T13:50:18.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:18.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:18.615 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:18.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:18.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:18 smithi043 ceph-mon[29963]: pgmap v296: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 573 B/s rd, 64 MiB/s wr, 1.02k op/s; 35712/285498 objects degraded (12.509%); 7.2 KiB/s, 0 objects/s recovering 2024-03-30T13:50:18.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:18 smithi043 ceph-mon[29963]: osdmap e170: 8 total, 7 up, 8 in 2024-03-30T13:50:18.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:18 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3331922532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:50:18.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:18 smithi043 ceph-mon[29963]: 3.16 scrub starts 2024-03-30T13:50:18.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:18 smithi043 ceph-mon[29963]: 3.16 scrub ok 2024-03-30T13:50:18.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:18 smithi043 ceph-mon[29963]: 8.3 scrub starts 2024-03-30T13:50:18.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:18 smithi043 ceph-mon[42099]: pgmap v296: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 573 B/s rd, 64 MiB/s wr, 1.02k op/s; 35712/285498 objects degraded (12.509%); 7.2 KiB/s, 0 objects/s recovering 2024-03-30T13:50:18.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:18 smithi043 ceph-mon[42099]: osdmap e170: 8 total, 7 up, 8 in 2024-03-30T13:50:18.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:18 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3331922532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:50:18.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:18 smithi043 ceph-mon[42099]: 3.16 scrub starts 2024-03-30T13:50:18.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:18 smithi043 ceph-mon[42099]: 3.16 scrub ok 2024-03-30T13:50:18.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:18 smithi043 ceph-mon[42099]: 8.3 scrub starts 2024-03-30T13:50:18.716 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:18.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:18.756 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:18.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:18.857 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:18.894 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:18.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:18.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:18.921 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:50:18.971 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:50:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:50:18.978 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 64 16 101518 101502 99.1072 66.125 0.00431169 0.0100799 2024-03-30T13:50:18.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:19.022 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:19.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:19.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:19.123 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:19.158 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:19.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:19.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:19.260 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:19.300 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:19.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:19.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:19.402 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:19.435 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:19.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:19.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:19.537 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:19.569 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:19.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:19.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:19.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:19.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:19.701 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:19.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:19.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:19 smithi113 ceph-mon[34341]: 8.7 scrub ok 2024-03-30T13:50:19.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:19 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:50:19.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:19.816 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/osd.2/config 2024-03-30T13:50:19.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:19.838 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:19.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:19.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:19 smithi043 ceph-mon[29963]: 8.7 scrub ok 2024-03-30T13:50:19.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:19 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:50:19.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:19 smithi043 ceph-mon[42099]: 8.7 scrub ok 2024-03-30T13:50:19.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:19 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:50:19.939 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:19.978 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:19.979 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 65 16 102518 102502 98.544 62.5 0.00727978 0.0101431 2024-03-30T13:50:19.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:19.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:20.080 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:20.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:20.118 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:20.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:20.219 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:20.254 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:20.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:20.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:20.356 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:20.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:20.394 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:20.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:20.495 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:20.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:20.530 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:20.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:20.631 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:20.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:20.670 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:20.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:20.709 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:20 smithi043 ceph-mon[29963]: pgmap v298: 148 pgs: 1 active+clean+scrubbing, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s; 38077/304296 objects degraded (12.513%); 8.0 KiB/s, 0 objects/s recovering 2024-03-30T13:50:20.709 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:20 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:50:20.709 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:20 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:50:20.709 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:20 smithi043 ceph-mon[42099]: pgmap v298: 148 pgs: 1 active+clean+scrubbing, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s; 38077/304296 objects degraded (12.513%); 8.0 KiB/s, 0 objects/s recovering 2024-03-30T13:50:20.709 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:20 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:50:20.709 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:20 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:50:20.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:20 smithi113 ceph-mon[34341]: pgmap v298: 148 pgs: 1 active+clean+scrubbing, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s; 38077/304296 objects degraded (12.513%); 8.0 KiB/s, 0 objects/s recovering 2024-03-30T13:50:20.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:20 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:50:20.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:20 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:50:20.753 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:50:20.753 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:50:20.771 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:20.805 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:20.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:20.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:20.907 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:20.949 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:20.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:20.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:20.978 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 66 16 103561 103545 98.0385 65.1875 0.0121648 0.0101917 2024-03-30T13:50:21.051 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:21.089 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:21.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:21.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:21.181 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:50:21.182 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [ 2024-03-30T13:50:21.182 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.182 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop(client.24977.0:103687 8.5 e171/147 8:aed460f2:::benchmark_data_smithi043_119832_object103686:head v 171'6627, mlcod=171'6619)", 2024-03-30T13:50:21.182 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:21.162684+0000", 2024-03-30T13:50:21.182 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.016705833, 2024-03-30T13:50:21.182 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.016734440999999999, 2024-03-30T13:50:21.182 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:50:21.182 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:50:21.182 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:50:21.182 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:50:21.182 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.182 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:50:21.183 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.162684+0000", 2024-03-30T13:50:21.183 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:50:21.183 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.183 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.183 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:50:21.183 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.162684+0000", 2024-03-30T13:50:21.183 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:50:21.183 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.183 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.183 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:50:21.183 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.162685+0000", 2024-03-30T13:50:21.183 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8400000000000003e-07 2024-03-30T13:50:21.183 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.183 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.184 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:50:21.184 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.162711+0000", 2024-03-30T13:50:21.184 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.6364999999999999e-05 2024-03-30T13:50:21.184 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.184 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.184 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:50:21.184 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.162726+0000", 2024-03-30T13:50:21.184 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5235000000000001e-05 2024-03-30T13:50:21.184 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.184 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.184 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:50:21.184 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.162734+0000", 2024-03-30T13:50:21.184 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.5340000000000002e-06 2024-03-30T13:50:21.185 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.185 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.185 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:50:21.185 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.164963+0000", 2024-03-30T13:50:21.185 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0022287639999999998 2024-03-30T13:50:21.185 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.185 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.185 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:50:21.185 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.165006+0000", 2024-03-30T13:50:21.185 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3449999999999999e-05 2024-03-30T13:50:21.185 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:50:21.185 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:50:21.185 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:50:21.185 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.186 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.186 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop(client.24977.0:103698 8.d e171/167 8:b6dceae5:::benchmark_data_smithi043_119832_object103697:head v 171'6382, mlcod=171'6379)", 2024-03-30T13:50:21.186 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:21.167826+0000", 2024-03-30T13:50:21.186 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.011564196000000001, 2024-03-30T13:50:21.186 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.011670893999999999, 2024-03-30T13:50:21.186 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:50:21.186 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:50:21.186 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:50:21.186 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:50:21.186 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.186 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:50:21.186 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.167826+0000", 2024-03-30T13:50:21.186 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:50:21.187 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.187 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.187 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:50:21.187 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.167826+0000", 2024-03-30T13:50:21.187 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:50:21.187 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.187 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.187 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:50:21.187 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.167827+0000", 2024-03-30T13:50:21.187 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.026e-06 2024-03-30T13:50:21.187 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.187 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.187 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:50:21.187 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.167849+0000", 2024-03-30T13:50:21.188 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.1997e-05 2024-03-30T13:50:21.188 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.188 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.188 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:50:21.188 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.167863+0000", 2024-03-30T13:50:21.188 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4233000000000001e-05 2024-03-30T13:50:21.188 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.188 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.188 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:50:21.188 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.167871+0000", 2024-03-30T13:50:21.188 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.2970000000000004e-06 2024-03-30T13:50:21.188 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.188 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.188 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:50:21.189 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.168367+0000", 2024-03-30T13:50:21.189 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00049542800000000001 2024-03-30T13:50:21.189 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.189 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.189 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:50:21.189 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.168400+0000", 2024-03-30T13:50:21.189 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.3442000000000003e-05 2024-03-30T13:50:21.189 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:50:21.189 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:50:21.189 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:50:21.189 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.189 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.189 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop(client.24977.0:103704 8.6 e171/147 8:663083f7:::benchmark_data_smithi043_119832_object103703:head v 171'6424, mlcod=171'6422)", 2024-03-30T13:50:21.189 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:21.176256+0000", 2024-03-30T13:50:21.190 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.0031339940000000002, 2024-03-30T13:50:21.190 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0033010000000000001, 2024-03-30T13:50:21.190 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:50:21.190 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:50:21.190 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:50:21.190 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:50:21.190 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.190 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:50:21.190 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.176256+0000", 2024-03-30T13:50:21.190 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:50:21.190 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.190 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.190 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:50:21.190 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.176256+0000", 2024-03-30T13:50:21.191 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:50:21.191 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.191 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.191 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:50:21.191 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.176257+0000", 2024-03-30T13:50:21.191 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.085e-06 2024-03-30T13:50:21.191 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.191 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.191 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:50:21.191 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.176301+0000", 2024-03-30T13:50:21.191 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3962999999999999e-05 2024-03-30T13:50:21.191 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.191 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.191 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:50:21.191 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.176318+0000", 2024-03-30T13:50:21.192 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.7125e-05 2024-03-30T13:50:21.192 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.192 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.192 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:50:21.192 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.176327+0000", 2024-03-30T13:50:21.192 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.9670000000000007e-06 2024-03-30T13:50:21.192 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.192 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.192 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:50:21.192 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.176404+0000", 2024-03-30T13:50:21.192 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.6705999999999994e-05 2024-03-30T13:50:21.192 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:50:21.192 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:50:21.192 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:50:21.193 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.176425+0000", 2024-03-30T13:50:21.193 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.1393999999999999e-05 2024-03-30T13:50:21.193 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:50:21.193 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:50:21.193 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:50:21.193 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:50:21.193 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-03-30T13:50:21.193 INFO:teuthology.orchestra.run.smithi113.stdout: "num_ops": 3 2024-03-30T13:50:21.193 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:50:21.193 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:21.223 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:21.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:21.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:21.325 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:21.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:21.363 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:21.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:21.466 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:21.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:21.497 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:21.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:21.597 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:21.632 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:21.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:21.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:21.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:21 smithi043 ceph-mon[29963]: osdmap e171: 8 total, 7 up, 8 in 2024-03-30T13:50:21.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:21 smithi043 ceph-mon[29963]: 8.5 deep-scrub starts 2024-03-30T13:50:21.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:21 smithi043 ceph-mon[29963]: 8.3 scrub ok 2024-03-30T13:50:21.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:21 smithi043 ceph-mon[42099]: osdmap e171: 8 total, 7 up, 8 in 2024-03-30T13:50:21.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:21 smithi043 ceph-mon[42099]: 8.5 deep-scrub starts 2024-03-30T13:50:21.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:21 smithi043 ceph-mon[42099]: 8.3 scrub ok 2024-03-30T13:50:21.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:21 smithi113 ceph-mon[34341]: osdmap e171: 8 total, 7 up, 8 in 2024-03-30T13:50:21.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:21 smithi113 ceph-mon[34341]: 8.5 deep-scrub starts 2024-03-30T13:50:21.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:21 smithi113 ceph-mon[34341]: 8.3 scrub ok 2024-03-30T13:50:21.734 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:21.768 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:21.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:21.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:21.869 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:21.910 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:21.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:21.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:21.978 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 67 16 104600 104584 97.5445 64.9375 0.00971125 0.0102469 2024-03-30T13:50:22.011 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-30T13:50:22.041 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:50:22.052 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:22.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:22.052 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-03-30T13:50:22.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:22.154 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:22.190 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:22.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:22.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:22.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:22.324 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:22.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:22.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:22.426 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:22.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:22.486 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:22.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:22.524 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:22 smithi043 ceph-mon[29963]: 3.15 scrub starts 2024-03-30T13:50:22.524 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:22 smithi043 ceph-mon[29963]: 3.15 scrub ok 2024-03-30T13:50:22.524 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:22 smithi043 ceph-mon[29963]: 4.17 scrub starts 2024-03-30T13:50:22.524 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:22 smithi043 ceph-mon[29963]: 4.17 scrub ok 2024-03-30T13:50:22.524 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:22 smithi043 ceph-mon[29963]: pgmap v300: 148 pgs: 1 active+clean+scrubbing, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 795 B/s rd, 61 MiB/s wr, 974 op/s; 38077/304296 objects degraded (12.513%) 2024-03-30T13:50:22.524 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:22 smithi043 ceph-mon[29963]: 5.16 scrub starts 2024-03-30T13:50:22.524 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:22 smithi043 ceph-mon[29963]: 5.16 scrub ok 2024-03-30T13:50:22.524 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:22 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:50:22.524 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:22 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 38077/304296 objects degraded (12.513%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:22.524 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:22 smithi043 ceph-mon[29963]: osdmap e172: 8 total, 7 up, 8 in 2024-03-30T13:50:22.525 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:22 smithi043 ceph-mon[42099]: 3.15 scrub starts 2024-03-30T13:50:22.525 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:22 smithi043 ceph-mon[42099]: 3.15 scrub ok 2024-03-30T13:50:22.525 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:22 smithi043 ceph-mon[42099]: 4.17 scrub starts 2024-03-30T13:50:22.525 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:22 smithi043 ceph-mon[42099]: 4.17 scrub ok 2024-03-30T13:50:22.525 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:22 smithi043 ceph-mon[42099]: pgmap v300: 148 pgs: 1 active+clean+scrubbing, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 795 B/s rd, 61 MiB/s wr, 974 op/s; 38077/304296 objects degraded (12.513%) 2024-03-30T13:50:22.525 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:22 smithi043 ceph-mon[42099]: 5.16 scrub starts 2024-03-30T13:50:22.525 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:22 smithi043 ceph-mon[42099]: 5.16 scrub ok 2024-03-30T13:50:22.525 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:22 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:50:22.525 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:22 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 38077/304296 objects degraded (12.513%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:22.525 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:22 smithi043 ceph-mon[42099]: osdmap e172: 8 total, 7 up, 8 in 2024-03-30T13:50:22.547 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:50:22.587 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:22.621 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:22.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:22.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:22.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:22 smithi113 ceph-mon[34341]: 3.15 scrub starts 2024-03-30T13:50:22.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:22 smithi113 ceph-mon[34341]: 3.15 scrub ok 2024-03-30T13:50:22.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:22 smithi113 ceph-mon[34341]: 4.17 scrub starts 2024-03-30T13:50:22.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:22 smithi113 ceph-mon[34341]: 4.17 scrub ok 2024-03-30T13:50:22.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:22 smithi113 ceph-mon[34341]: pgmap v300: 148 pgs: 1 active+clean+scrubbing, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 795 B/s rd, 61 MiB/s wr, 974 op/s; 38077/304296 objects degraded (12.513%) 2024-03-30T13:50:22.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:22 smithi113 ceph-mon[34341]: 5.16 scrub starts 2024-03-30T13:50:22.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:22 smithi113 ceph-mon[34341]: 5.16 scrub ok 2024-03-30T13:50:22.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:22 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:50:22.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:22 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 38077/304296 objects degraded (12.513%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:22.723 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:22 smithi113 ceph-mon[34341]: osdmap e172: 8 total, 7 up, 8 in 2024-03-30T13:50:22.723 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:22.754 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:50:22.760 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:22.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:22.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:22.862 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:22.911 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:22.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:22.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:22.978 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 68 16 105505 105489 96.9418 56.5625 0.0182869 0.0103092 2024-03-30T13:50:23.014 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:23.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:23.072 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:23.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:23.173 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:23.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:23.209 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:23.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:23.310 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:23.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:23.351 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:23.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:23.452 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:23.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:23.496 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:23.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:23.597 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:23.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:23.635 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:23.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:23.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:23 smithi043 ceph-mon[29963]: 3.14 scrub starts 2024-03-30T13:50:23.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:23 smithi043 ceph-mon[29963]: 3.14 scrub ok 2024-03-30T13:50:23.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:23 smithi043 ceph-mon[29963]: 6.d scrub starts 2024-03-30T13:50:23.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:23 smithi043 ceph-mon[29963]: 6.d scrub ok 2024-03-30T13:50:23.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:23 smithi043 ceph-mon[42099]: 3.14 scrub starts 2024-03-30T13:50:23.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:23 smithi043 ceph-mon[42099]: 3.14 scrub ok 2024-03-30T13:50:23.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:23 smithi043 ceph-mon[42099]: 6.d scrub starts 2024-03-30T13:50:23.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:23 smithi043 ceph-mon[42099]: 6.d scrub ok 2024-03-30T13:50:23.736 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:23.761 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:23.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:23.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:23.861 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:50:23.862 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:23.893 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:23.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:23.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:23 smithi113 ceph-mon[34341]: 3.14 scrub starts 2024-03-30T13:50:23.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:23 smithi113 ceph-mon[34341]: 3.14 scrub ok 2024-03-30T13:50:23.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:23 smithi113 ceph-mon[34341]: 6.d scrub starts 2024-03-30T13:50:23.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:23 smithi113 ceph-mon[34341]: 6.d scrub ok 2024-03-30T13:50:23.979 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 69 16 106425 106409 96.3701 57.5 0.00303646 0.0103626 2024-03-30T13:50:23.994 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:24.028 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:24.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:24.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:24.129 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:24.180 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:24.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:24.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:24.282 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:24.319 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:24.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:24.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:24.423 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:24.450 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:24.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:24.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:24.552 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:24.593 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:24.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:24.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:24.695 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:24.730 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:24.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:24.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:24.778 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:24 smithi043 ceph-mon[29963]: 3.1 scrub starts 2024-03-30T13:50:24.778 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:24 smithi043 ceph-mon[29963]: 3.1 scrub ok 2024-03-30T13:50:24.778 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:24 smithi043 ceph-mon[29963]: pgmap v302: 148 pgs: 1 active+clean+scrubbing, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 75 MiB/s wr, 1.20k op/s; 38672/307041 objects degraded (12.595%) 2024-03-30T13:50:24.778 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:24 smithi043 ceph-mon[29963]: 3.18 scrub starts 2024-03-30T13:50:24.778 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:24 smithi043 ceph-mon[29963]: 3.18 scrub ok 2024-03-30T13:50:24.778 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:24 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:50:24.778 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:24 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:50:24.778 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:24 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:50:24.778 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:24 smithi043 ceph-mon[42099]: 3.1 scrub starts 2024-03-30T13:50:24.778 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:24 smithi043 ceph-mon[42099]: 3.1 scrub ok 2024-03-30T13:50:24.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:24 smithi043 ceph-mon[42099]: pgmap v302: 148 pgs: 1 active+clean+scrubbing, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 75 MiB/s wr, 1.20k op/s; 38672/307041 objects degraded (12.595%) 2024-03-30T13:50:24.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:24 smithi043 ceph-mon[42099]: 3.18 scrub starts 2024-03-30T13:50:24.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:24 smithi043 ceph-mon[42099]: 3.18 scrub ok 2024-03-30T13:50:24.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:24 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:50:24.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:24 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:50:24.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:24 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:50:24.831 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:24.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:24.873 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:24.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:24.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:24 smithi113 ceph-mon[34341]: 3.1 scrub starts 2024-03-30T13:50:24.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:24 smithi113 ceph-mon[34341]: 3.1 scrub ok 2024-03-30T13:50:24.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:24 smithi113 ceph-mon[34341]: pgmap v302: 148 pgs: 1 active+clean+scrubbing, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 75 MiB/s wr, 1.20k op/s; 38672/307041 objects degraded (12.595%) 2024-03-30T13:50:24.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:24 smithi113 ceph-mon[34341]: 3.18 scrub starts 2024-03-30T13:50:24.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:24 smithi113 ceph-mon[34341]: 3.18 scrub ok 2024-03-30T13:50:24.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:24 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:50:24.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:24 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:50:24.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:24 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:50:24.974 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:24.979 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 70 16 107438 107422 95.8978 63.3125 0.00899233 0.010423 2024-03-30T13:50:25.007 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:25.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:25.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:25.070 INFO:teuthology.orchestra.run.smithi043.stdout:8.f 2024-03-30T13:50:25.070 INFO:teuthology.orchestra.run.smithi043.stdout:8.d 2024-03-30T13:50:25.070 INFO:teuthology.orchestra.run.smithi043.stdout:8.a 2024-03-30T13:50:25.071 INFO:teuthology.orchestra.run.smithi043.stdout:8.4 2024-03-30T13:50:25.071 INFO:teuthology.orchestra.run.smithi043.stdout:8.1 2024-03-30T13:50:25.071 INFO:teuthology.orchestra.run.smithi043.stdout:6.e 2024-03-30T13:50:25.071 INFO:teuthology.orchestra.run.smithi043.stdout:4.6 2024-03-30T13:50:25.071 INFO:teuthology.orchestra.run.smithi043.stdout:6.17 2024-03-30T13:50:25.071 INFO:teuthology.orchestra.run.smithi043.stdout:4.1c 2024-03-30T13:50:25.071 INFO:teuthology.orchestra.run.smithi043.stdout:5.a 2024-03-30T13:50:25.071 INFO:teuthology.orchestra.run.smithi043.stdout:5.1e 2024-03-30T13:50:25.071 INFO:teuthology.orchestra.run.smithi043.stdout:3.9 2024-03-30T13:50:25.071 INFO:teuthology.orchestra.run.smithi043.stdout:3.5 2024-03-30T13:50:25.072 INFO:teuthology.orchestra.run.smithi043.stdout:3.4 2024-03-30T13:50:25.072 INFO:teuthology.orchestra.run.smithi043.stdout:3.1a 2024-03-30T13:50:25.072 INFO:teuthology.orchestra.run.smithi043.stdout:5.1c 2024-03-30T13:50:25.072 INFO:teuthology.orchestra.run.smithi043.stdout:3.0 2024-03-30T13:50:25.072 INFO:teuthology.orchestra.run.smithi043.stdout:4.13 2024-03-30T13:50:25.072 INFO:teuthology.orchestra.run.smithi043.stdout:3.1f 2024-03-30T13:50:25.072 INFO:teuthology.orchestra.run.smithi043.stdout:3.13 2024-03-30T13:50:25.072 INFO:teuthology.orchestra.run.smithi043.stdout:4.12 2024-03-30T13:50:25.072 INFO:teuthology.orchestra.run.smithi043.stdout:6.9 2024-03-30T13:50:25.072 INFO:teuthology.orchestra.run.smithi043.stdout:2.1 2024-03-30T13:50:25.072 INFO:teuthology.orchestra.run.smithi043.stdout:5.14 2024-03-30T13:50:25.072 INFO:teuthology.orchestra.run.smithi043.stdout:5.b 2024-03-30T13:50:25.073 INFO:teuthology.orchestra.run.smithi043.stdout:3.1e 2024-03-30T13:50:25.073 INFO:teuthology.orchestra.run.smithi043.stdout:4.14 2024-03-30T13:50:25.073 INFO:teuthology.orchestra.run.smithi043.stdout:5.d 2024-03-30T13:50:25.073 INFO:teuthology.orchestra.run.smithi043.stdout:6.f 2024-03-30T13:50:25.073 INFO:teuthology.orchestra.run.smithi043.stdout:6.2 2024-03-30T13:50:25.073 INFO:teuthology.orchestra.run.smithi043.stdout:4.d 2024-03-30T13:50:25.073 INFO:teuthology.orchestra.run.smithi043.stdout:5.6 2024-03-30T13:50:25.073 INFO:teuthology.orchestra.run.smithi043.stdout:6.12 2024-03-30T13:50:25.073 INFO:teuthology.orchestra.run.smithi043.stdout:4.19 2024-03-30T13:50:25.073 INFO:teuthology.orchestra.run.smithi043.stdout:5.8 2024-03-30T13:50:25.074 INFO:teuthology.orchestra.run.smithi043.stdout:6.0 2024-03-30T13:50:25.074 INFO:teuthology.orchestra.run.smithi043.stdout:5.4 2024-03-30T13:50:25.074 INFO:teuthology.orchestra.run.smithi043.stdout:6.14 2024-03-30T13:50:25.074 INFO:teuthology.orchestra.run.smithi043.stdout:8.e 2024-03-30T13:50:25.074 INFO:teuthology.orchestra.run.smithi043.stdout:6.1 2024-03-30T13:50:25.074 INFO:teuthology.orchestra.run.smithi043.stdout:6.3 2024-03-30T13:50:25.074 INFO:teuthology.orchestra.run.smithi043.stdout:4.4 2024-03-30T13:50:25.074 INFO:teuthology.orchestra.run.smithi043.stdout:6.1c 2024-03-30T13:50:25.074 INFO:teuthology.orchestra.run.smithi043.stdout:6.8 2024-03-30T13:50:25.110 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:25.147 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:25.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:25.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:25.249 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:25.285 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:25.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:25.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:25.387 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:25.426 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:25.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:25.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:25.504 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-03-30T13:50:25.527 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:25.566 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:25.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:25.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:25.668 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:25.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:25.710 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:25.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:25.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:25.817 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:25 smithi113 ceph-mon[34341]: 3.f scrub starts 2024-03-30T13:50:25.817 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:25 smithi113 ceph-mon[34341]: 3.f scrub ok 2024-03-30T13:50:25.817 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:25 smithi113 ceph-mon[34341]: 3.12 scrub starts 2024-03-30T13:50:25.817 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:25 smithi113 ceph-mon[34341]: 3.12 scrub ok 2024-03-30T13:50:25.817 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:25 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1995322060' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:50:25.817 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:25 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:50:25.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:25 smithi043 ceph-mon[29963]: 3.f scrub starts 2024-03-30T13:50:25.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:25 smithi043 ceph-mon[29963]: 3.f scrub ok 2024-03-30T13:50:25.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:25 smithi043 ceph-mon[29963]: 3.12 scrub starts 2024-03-30T13:50:25.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:25 smithi043 ceph-mon[29963]: 3.12 scrub ok 2024-03-30T13:50:25.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:25 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1995322060' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:50:25.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:25 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:50:25.831 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:25 smithi043 ceph-mon[42099]: 3.f scrub starts 2024-03-30T13:50:25.831 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:25 smithi043 ceph-mon[42099]: 3.f scrub ok 2024-03-30T13:50:25.832 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:25 smithi043 ceph-mon[42099]: 3.12 scrub starts 2024-03-30T13:50:25.832 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:25 smithi043 ceph-mon[42099]: 3.12 scrub ok 2024-03-30T13:50:25.832 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:25 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1995322060' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:50:25.832 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:25 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:50:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:25.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:25.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:25.946 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:25.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:25.979 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:25.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:25.979 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 71 16 108500 108484 95.4819 66.375 0.0164229 0.0104674 2024-03-30T13:50:26.080 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:26.112 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:26.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:26.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:26.214 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:26.242 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:26.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:26.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:26.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:26.372 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:26.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:26.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:26.398 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:50:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:50:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:50:26.474 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:26.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:26.508 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:26.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:26.519 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:50:26.520 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [], 2024-03-30T13:50:26.520 INFO:teuthology.orchestra.run.smithi113.stdout: "complaint_time": 30, 2024-03-30T13:50:26.520 INFO:teuthology.orchestra.run.smithi113.stdout: "num_blocked_ops": 0 2024-03-30T13:50:26.520 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:50:26.608 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:26.645 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:26.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:26.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:26.746 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:26.778 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:26.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:26.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:26.880 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:26.895 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:26 smithi113 ceph-mon[34341]: pgmap v303: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.41k op/s; 39913/319242 objects degraded (12.502%) 2024-03-30T13:50:26.895 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:26 smithi113 ceph-mon[34341]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:50:26.895 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:26 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:50:26.895 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:26 smithi113 ceph-mon[34341]: osdmap e173: 8 total, 7 up, 8 in 2024-03-30T13:50:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:26 smithi043 ceph-mon[29963]: pgmap v303: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.41k op/s; 39913/319242 objects degraded (12.502%) 2024-03-30T13:50:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:26 smithi043 ceph-mon[29963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:50:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:26 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:50:26.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:26 smithi043 ceph-mon[29963]: osdmap e173: 8 total, 7 up, 8 in 2024-03-30T13:50:26.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:26 smithi043 ceph-mon[42099]: pgmap v303: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.41k op/s; 39913/319242 objects degraded (12.502%) 2024-03-30T13:50:26.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:26 smithi043 ceph-mon[42099]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:50:26.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:26 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:50:26.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:26 smithi043 ceph-mon[42099]: osdmap e173: 8 total, 7 up, 8 in 2024-03-30T13:50:26.912 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:26.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:26.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:26.979 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 72 16 109524 109508 95.0447 64 0.00545349 0.0105139 2024-03-30T13:50:27.016 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:27.052 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:27.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:27.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:27.154 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:27.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:27.198 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:27.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:27.240 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-03-30T13:50:27.299 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:27.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:27.340 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:27.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:27.441 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:27.487 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:27.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:27.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:27.589 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:27.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:27.640 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:27.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:27.661 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:50:27.740 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:27.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:27.781 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:27.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:27.800 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:27 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:50:27.800 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:27 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:50:27.800 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:27 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 39913/319242 objects degraded (12.502%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:27.800 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:27 smithi113 ceph-mon[34341]: osdmap e174: 8 total, 7 up, 8 in 2024-03-30T13:50:27.882 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:27.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:27 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:50:27.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:27 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:50:27.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:27 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 39913/319242 objects degraded (12.502%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:27.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:27 smithi043 ceph-mon[29963]: osdmap e174: 8 total, 7 up, 8 in 2024-03-30T13:50:27.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:27 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:50:27.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:27 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:50:27.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:27 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 39913/319242 objects degraded (12.502%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:27.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:27 smithi043 ceph-mon[42099]: osdmap e174: 8 total, 7 up, 8 in 2024-03-30T13:50:27.913 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:27.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:27.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:27.979 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 73 16 110557 110541 94.627 64.5625 0.00703835 0.0105625 2024-03-30T13:50:28.015 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:28.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:28.051 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:28.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:28.152 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:28.188 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:28.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:28.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:28.213 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -n osd.2 --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 8.e --file /var/log/ceph/exp.8.e.2 2024-03-30T13:50:28.291 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:28.320 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:28.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:28.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:28.422 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:28.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:28.460 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:28.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:28.561 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:28.602 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:28.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:28.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:28.704 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:28.747 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:28.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:28.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:28.849 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:28.887 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:28.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:28.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:28.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:28 smithi043 ceph-mon[29963]: pgmap v306: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 830 op/s; 39913/319242 objects degraded (12.502%) 2024-03-30T13:50:28.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:50:28.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:28 smithi043 ceph-mon[42099]: pgmap v306: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 830 op/s; 39913/319242 objects degraded (12.502%) 2024-03-30T13:50:28.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:50:28.951 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/osd.2/config 2024-03-30T13:50:28.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:28 smithi113 ceph-mon[34341]: pgmap v306: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 830 op/s; 39913/319242 objects degraded (12.502%) 2024-03-30T13:50:28.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:50:28.972 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:50:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:50:28.979 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 74 16 111565 111549 94.1996 63 0.00467103 0.0106055 2024-03-30T13:50:28.989 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:29.018 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:29.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:29.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:29.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:29.153 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:29.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:29.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:29.255 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:29.287 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:29.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:29.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:29.389 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:29.422 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:29.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:29.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:29.524 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:29.556 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:50:29.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:29.557 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:29.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:29.589 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:50:29.629 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:50:29.658 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:29.666 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:50:29.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:29.696 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:29.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:29.700 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:50:29.738 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:50:29.738 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:29 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:50:29.738 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:29 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:50:29.783 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:50:29.783 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:50:29.783 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:50:29.783 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:50:29.783 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:50:29.783 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:50:29.783 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:50:29.783 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:50:29.784 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:50:29.798 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:29.835 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:29.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:29.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:29.937 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:29.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:29 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:50:29.978 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:29.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:29.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:29.979 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 75 16 112674 112658 93.8677 69.3125 0.0150026 0.0106475 2024-03-30T13:50:30.080 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:30.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:30.125 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:30.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:30.226 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:30.257 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:30.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:30.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:30.359 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:30.397 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:30.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:30.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:30.498 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:30.534 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:30.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:30.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:30.630 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:30 smithi113 ceph-mon[34341]: pgmap v307: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 41791/334356 objects degraded (12.499%) 2024-03-30T13:50:30.630 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:30 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1347110587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:50:30.636 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:30.674 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:30.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:30.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:30.776 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:30.781 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:50:30.781 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:50:30.811 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:30.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:30.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:30.879 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:30 smithi043 ceph-mon[29963]: pgmap v307: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 41791/334356 objects degraded (12.499%) 2024-03-30T13:50:30.879 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:30 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1347110587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:50:30.879 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:30 smithi043 ceph-mon[42099]: pgmap v307: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 41791/334356 objects degraded (12.499%) 2024-03-30T13:50:30.879 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:30 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1347110587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:50:30.912 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:30.945 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:30.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:30.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:30.977 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T13:50:30.979 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 76 16 113746 113730 93.5142 67 0.0095771 0.0106885 2024-03-30T13:50:31.048 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:31.085 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:31.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:31.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:31.152 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:50:31.152 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:50:31.187 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:31.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:31.227 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:31.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:31.312 INFO:teuthology.orchestra.run.smithi043.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-30T13:50:31.328 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:31.367 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:31.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:31.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:31.469 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:31.505 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:31.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:31.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:31.607 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:31.640 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:31.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:31.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:31.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:31 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:50:31.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:31 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:50:31.722 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:31 smithi113 ceph-mon[34341]: osdmap e175: 8 total, 7 up, 8 in 2024-03-30T13:50:31.742 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:31.780 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:31.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:31.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:31.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:31 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:50:31.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:31 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:50:31.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:31 smithi043 ceph-mon[29963]: osdmap e175: 8 total, 7 up, 8 in 2024-03-30T13:50:31.831 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:31 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:50:31.831 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:31 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:50:31.831 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:31 smithi043 ceph-mon[42099]: osdmap e175: 8 total, 7 up, 8 in 2024-03-30T13:50:31.840 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:50:31.882 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:31.920 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:31.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:31.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:31.956 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-30T13:50:31.979 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 77 16 114772 114756 93.1324 64.125 0.0347524 0.0107299 2024-03-30T13:50:32.021 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:32.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:32.056 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:32.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:32.077 INFO:teuthology.orchestra.run.smithi043.stderr:Exporting 8.e info 8.e( v 166'5643 (0'0,166'5643] local-lis/les=147/148 n=5643 ec=147/147 lis/c=147/147 les/c/f=148/148/0 sis=147) 2024-03-30T13:50:32.131 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70000158:::benchmark_data_smithi043_119832_object18973:head# 2024-03-30T13:50:32.131 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7000934d:::benchmark_data_smithi043_119832_object50764:head# 2024-03-30T13:50:32.131 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7000e5ef:::benchmark_data_smithi043_119832_object87691:head# 2024-03-30T13:50:32.132 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7002fd95:::benchmark_data_smithi043_119832_object267:head# 2024-03-30T13:50:32.132 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:700302d5:::benchmark_data_smithi043_119832_object31432:head# 2024-03-30T13:50:32.132 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70030c19:::benchmark_data_smithi043_119832_object40003:head# 2024-03-30T13:50:32.133 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7003dcb3:::benchmark_data_smithi043_119832_object28164:head# 2024-03-30T13:50:32.133 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:700524b2:::benchmark_data_smithi043_119832_object23052:head# 2024-03-30T13:50:32.133 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7005811f:::benchmark_data_smithi043_119832_object38752:head# 2024-03-30T13:50:32.133 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:700661f5:::benchmark_data_smithi043_119832_object20795:head# 2024-03-30T13:50:32.133 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7007a27b:::benchmark_data_smithi043_119832_object35591:head# 2024-03-30T13:50:32.134 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70082362:::benchmark_data_smithi043_119832_object13125:head# 2024-03-30T13:50:32.134 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7008ca39:::benchmark_data_smithi043_119832_object21970:head# 2024-03-30T13:50:32.135 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70099af1:::benchmark_data_smithi043_119832_object20472:head# 2024-03-30T13:50:32.135 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:700a1cbb:::benchmark_data_smithi043_119832_object17699:head# 2024-03-30T13:50:32.135 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:700b3b1c:::benchmark_data_smithi043_119832_object77210:head# 2024-03-30T13:50:32.135 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:700f3efe:::benchmark_data_smithi043_119832_object79719:head# 2024-03-30T13:50:32.135 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70105bf1:::benchmark_data_smithi043_119832_object3840:head# 2024-03-30T13:50:32.137 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7010fb6a:::benchmark_data_smithi043_119832_object8497:head# 2024-03-30T13:50:32.137 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70117fa6:::benchmark_data_smithi043_119832_object4021:head# 2024-03-30T13:50:32.140 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7011d777:::benchmark_data_smithi043_119832_object71475:head# 2024-03-30T13:50:32.141 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7012d397:::benchmark_data_smithi043_119832_object87870:head# 2024-03-30T13:50:32.141 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7012fe35:::benchmark_data_smithi043_119832_object38493:head# 2024-03-30T13:50:32.141 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7013a700:::benchmark_data_smithi043_119832_object58094:head# 2024-03-30T13:50:32.141 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7013b1f4:::benchmark_data_smithi043_119832_object22506:head# 2024-03-30T13:50:32.141 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7015d905:::benchmark_data_smithi043_119832_object67941:head# 2024-03-30T13:50:32.142 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7015df1c:::benchmark_data_smithi043_119832_object20267:head# 2024-03-30T13:50:32.144 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70162494:::benchmark_data_smithi043_119832_object25481:head# 2024-03-30T13:50:32.144 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70165efa:::benchmark_data_smithi043_119832_object30799:head# 2024-03-30T13:50:32.148 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7016f0fc:::benchmark_data_smithi043_119832_object17695:head# 2024-03-30T13:50:32.148 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70170670:::benchmark_data_smithi043_119832_object21158:head# 2024-03-30T13:50:32.152 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70171c83:::benchmark_data_smithi043_119832_object55411:head# 2024-03-30T13:50:32.152 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70186dc7:::benchmark_data_smithi043_119832_object64168:head# 2024-03-30T13:50:32.152 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7019f491:::benchmark_data_smithi043_119832_object53571:head# 2024-03-30T13:50:32.152 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:701a2b9e:::benchmark_data_smithi043_119832_object40090:head# 2024-03-30T13:50:32.153 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:701a8775:::benchmark_data_smithi043_119832_object66566:head# 2024-03-30T13:50:32.153 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:701a8e2e:::benchmark_data_smithi043_119832_object65994:head# 2024-03-30T13:50:32.154 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:701aa75a:::benchmark_data_smithi043_119832_object15185:head# 2024-03-30T13:50:32.155 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:701abb32:::benchmark_data_smithi043_119832_object53518:head# 2024-03-30T13:50:32.157 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:701b1149:::benchmark_data_smithi043_119832_object71946:head# 2024-03-30T13:50:32.157 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:701b1850:::benchmark_data_smithi043_119832_object68311:head# 2024-03-30T13:50:32.157 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:701bcdb4:::benchmark_data_smithi043_119832_object38562:head# 2024-03-30T13:50:32.157 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:701c2dc5:::benchmark_data_smithi043_119832_object55407:head# 2024-03-30T13:50:32.157 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:32.158 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:701c820c:::benchmark_data_smithi043_119832_object38358:head# 2024-03-30T13:50:32.159 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:701d4d59:::benchmark_data_smithi043_119832_object8526:head# 2024-03-30T13:50:32.160 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:701d4f27:::benchmark_data_smithi043_119832_object23902:head# 2024-03-30T13:50:32.160 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:701dbff6:::benchmark_data_smithi043_119832_object59109:head# 2024-03-30T13:50:32.162 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:701e0fbc:::benchmark_data_smithi043_119832_object81436:head# 2024-03-30T13:50:32.162 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:701e4a7f:::benchmark_data_smithi043_119832_object1088:head# 2024-03-30T13:50:32.162 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:701e52ae:::benchmark_data_smithi043_119832_object6158:head# 2024-03-30T13:50:32.162 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:701ece4c:::benchmark_data_smithi043_119832_object14107:head# 2024-03-30T13:50:32.164 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:701f683e:::benchmark_data_smithi043_119832_object53424:head# 2024-03-30T13:50:32.165 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:701ff537:::benchmark_data_smithi043_119832_object67966:head# 2024-03-30T13:50:32.167 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7020108f:::benchmark_data_smithi043_119832_object5976:head# 2024-03-30T13:50:32.168 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70201efc:::benchmark_data_smithi043_119832_object65255:head# 2024-03-30T13:50:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7021fa21:::benchmark_data_smithi043_119832_object67019:head# 2024-03-30T13:50:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70224df2:::benchmark_data_smithi043_119832_object4362:head# 2024-03-30T13:50:32.170 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7022669d:::benchmark_data_smithi043_119832_object81668:head# 2024-03-30T13:50:32.171 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7023d4b9:::benchmark_data_smithi043_119832_object80302:head# 2024-03-30T13:50:32.171 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70247df2:::benchmark_data_smithi043_119832_object68457:head# 2024-03-30T13:50:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70261550:::benchmark_data_smithi043_119832_object69358:head# 2024-03-30T13:50:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70266a5c:::benchmark_data_smithi043_119832_object6750:head# 2024-03-30T13:50:32.173 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70267f9b:::benchmark_data_smithi043_119832_object23830:head# 2024-03-30T13:50:32.173 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:702750eb:::benchmark_data_smithi043_119832_object29381:head# 2024-03-30T13:50:32.174 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70282d5b:::benchmark_data_smithi043_119832_object86953:head# 2024-03-30T13:50:32.175 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7029058f:::benchmark_data_smithi043_119832_object74484:head# 2024-03-30T13:50:32.175 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70292585:::benchmark_data_smithi043_119832_object71642:head# 2024-03-30T13:50:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7029ff00:::benchmark_data_smithi043_119832_object11792:head# 2024-03-30T13:50:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:702ab07a:::benchmark_data_smithi043_119832_object85843:head# 2024-03-30T13:50:32.177 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:702ac466:::benchmark_data_smithi043_119832_object63525:head# 2024-03-30T13:50:32.185 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:702cae14:::benchmark_data_smithi043_119832_object49346:head# 2024-03-30T13:50:32.185 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:702cbcd1:::benchmark_data_smithi043_119832_object74508:head# 2024-03-30T13:50:32.185 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:702cc0fc:::benchmark_data_smithi043_119832_object24364:head# 2024-03-30T13:50:32.186 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:702d5e6c:::benchmark_data_smithi043_119832_object41726:head# 2024-03-30T13:50:32.186 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:702dd9dd:::benchmark_data_smithi043_119832_object60814:head# 2024-03-30T13:50:32.186 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:702f716a:::benchmark_data_smithi043_119832_object17559:head# 2024-03-30T13:50:32.186 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70302403:::benchmark_data_smithi043_119832_object48850:head# 2024-03-30T13:50:32.186 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7030b5f3:::benchmark_data_smithi043_119832_object60422:head# 2024-03-30T13:50:32.186 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7032237a:::benchmark_data_smithi043_119832_object3778:head# 2024-03-30T13:50:32.186 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70325169:::benchmark_data_smithi043_119832_object48828:head# 2024-03-30T13:50:32.186 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7032c809:::benchmark_data_smithi043_119832_object86159:head# 2024-03-30T13:50:32.189 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70338a85:::benchmark_data_smithi043_119832_object72378:head# 2024-03-30T13:50:32.190 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7033c6c3:::benchmark_data_smithi043_119832_object33681:head# 2024-03-30T13:50:32.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:32.191 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:703516d7:::benchmark_data_smithi043_119832_object72166:head# 2024-03-30T13:50:32.191 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7035a7dd:::benchmark_data_smithi043_119832_object35756:head# 2024-03-30T13:50:32.191 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:32.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:32.196 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70370c4e:::benchmark_data_smithi043_119832_object10274:head# 2024-03-30T13:50:32.196 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70376e29:::benchmark_data_smithi043_119832_object40609:head# 2024-03-30T13:50:32.198 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7037ba92:::benchmark_data_smithi043_119832_object22655:head# 2024-03-30T13:50:32.199 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7038b2a3:::benchmark_data_smithi043_119832_object22922:head# 2024-03-30T13:50:32.199 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7038bd71:::benchmark_data_smithi043_119832_object75604:head# 2024-03-30T13:50:32.199 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:703907dd:::benchmark_data_smithi043_119832_object12505:head# 2024-03-30T13:50:32.199 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:703a931b:::benchmark_data_smithi043_119832_object56093:head# 2024-03-30T13:50:32.199 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:703b1c3b:::benchmark_data_smithi043_119832_object5019:head# 2024-03-30T13:50:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:703ba6ef:::benchmark_data_smithi043_119832_object70671:head# 2024-03-30T13:50:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:703bb785:::benchmark_data_smithi043_119832_object16830:head# 2024-03-30T13:50:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:703be694:::benchmark_data_smithi043_119832_object41863:head# 2024-03-30T13:50:32.202 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:703d3dad:::benchmark_data_smithi043_119832_object68090:head# 2024-03-30T13:50:32.202 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:703d404c:::benchmark_data_smithi043_119832_object38401:head# 2024-03-30T13:50:32.203 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:703e7bae:::benchmark_data_smithi043_119832_object70506:head# 2024-03-30T13:50:32.203 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:703eb67f:::benchmark_data_smithi043_119832_object66252:head# 2024-03-30T13:50:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:703f4e56:::benchmark_data_smithi043_119832_object39459:head# 2024-03-30T13:50:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:703fedad:::benchmark_data_smithi043_119832_object62353:head# 2024-03-30T13:50:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7040b70c:::benchmark_data_smithi043_119832_object82720:head# 2024-03-30T13:50:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:704102bd:::benchmark_data_smithi043_119832_object12240:head# 2024-03-30T13:50:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70415d6b:::benchmark_data_smithi043_119832_object19797:head# 2024-03-30T13:50:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70418f64:::benchmark_data_smithi043_119832_object32940:head# 2024-03-30T13:50:32.205 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70429bfd:::benchmark_data_smithi043_119832_object24366:head# 2024-03-30T13:50:32.205 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70429e0e:::benchmark_data_smithi043_119832_object36155:head# 2024-03-30T13:50:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7042ecfa:::benchmark_data_smithi043_119832_object51882:head# 2024-03-30T13:50:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:704304b5:::benchmark_data_smithi043_119832_object24332:head# 2024-03-30T13:50:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7043a762:::benchmark_data_smithi043_119832_object63381:head# 2024-03-30T13:50:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70444018:::benchmark_data_smithi043_119832_object67962:head# 2024-03-30T13:50:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7044a1ac:::benchmark_data_smithi043_119832_object68883:head# 2024-03-30T13:50:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7046f8db:::benchmark_data_smithi043_119832_object10074:head# 2024-03-30T13:50:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:704714b1:::benchmark_data_smithi043_119832_object24882:head# 2024-03-30T13:50:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7047f20b:::benchmark_data_smithi043_119832_object51806:head# 2024-03-30T13:50:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70483c3d:::benchmark_data_smithi043_119832_object66041:head# 2024-03-30T13:50:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70483ea8:::benchmark_data_smithi043_119832_object4470:head# 2024-03-30T13:50:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70485431:::benchmark_data_smithi043_119832_object55891:head# 2024-03-30T13:50:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70492822:::benchmark_data_smithi043_119832_object72084:head# 2024-03-30T13:50:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:704ac926:::benchmark_data_smithi043_119832_object59140:head# 2024-03-30T13:50:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:704b1226:::benchmark_data_smithi043_119832_object62507:head# 2024-03-30T13:50:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:704ba457:::benchmark_data_smithi043_119832_object32137:head# 2024-03-30T13:50:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:704e145a:::benchmark_data_smithi043_119832_object54945:head# 2024-03-30T13:50:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:704e4cbc:::benchmark_data_smithi043_119832_object48274:head# 2024-03-30T13:50:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:704e8bd2:::benchmark_data_smithi043_119832_object51696:head# 2024-03-30T13:50:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:704f462e:::benchmark_data_smithi043_119832_object41961:head# 2024-03-30T13:50:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7050dd7e:::benchmark_data_smithi043_119832_object66475:head# 2024-03-30T13:50:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70513bf9:::benchmark_data_smithi043_119832_object32250:head# 2024-03-30T13:50:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:705265a0:::benchmark_data_smithi043_119832_object37095:head# 2024-03-30T13:50:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70539985:::benchmark_data_smithi043_119832_object65407:head# 2024-03-30T13:50:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:705436f8:::benchmark_data_smithi043_119832_object51124:head# 2024-03-30T13:50:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70547be4:::benchmark_data_smithi043_119832_object15347:head# 2024-03-30T13:50:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70548204:::benchmark_data_smithi043_119832_object10563:head# 2024-03-30T13:50:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7054bbd4:::benchmark_data_smithi043_119832_object33888:head# 2024-03-30T13:50:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7054c55b:::benchmark_data_smithi043_119832_object30627:head# 2024-03-30T13:50:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7054e3c3:::benchmark_data_smithi043_119832_object48094:head# 2024-03-30T13:50:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7054fc08:::benchmark_data_smithi043_119832_object87693:head# 2024-03-30T13:50:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70552c1b:::benchmark_data_smithi043_119832_object15938:head# 2024-03-30T13:50:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7055c7d1:::benchmark_data_smithi043_119832_object27254:head# 2024-03-30T13:50:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7056f996:::benchmark_data_smithi043_119832_object46124:head# 2024-03-30T13:50:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7057b245:::benchmark_data_smithi043_119832_object70710:head# 2024-03-30T13:50:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7058d295:::benchmark_data_smithi043_119832_object17164:head# 2024-03-30T13:50:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:705a1c72:::benchmark_data_smithi043_119832_object45424:head# 2024-03-30T13:50:32.221 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:705a32c5:::benchmark_data_smithi043_119832_object55003:head# 2024-03-30T13:50:32.221 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:705a59f1:::benchmark_data_smithi043_119832_object85196:head# 2024-03-30T13:50:32.222 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:705b7ab0:::benchmark_data_smithi043_119832_object42903:head# 2024-03-30T13:50:32.222 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:705ca55f:::benchmark_data_smithi043_119832_object81810:head# 2024-03-30T13:50:32.223 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:705cc23a:::benchmark_data_smithi043_119832_object78555:head# 2024-03-30T13:50:32.224 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:705dcff6:::benchmark_data_smithi043_119832_object70008:head# 2024-03-30T13:50:32.224 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:705e3337:::benchmark_data_smithi043_119832_object58953:head# 2024-03-30T13:50:32.226 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:705e3500:::benchmark_data_smithi043_119832_object39816:head# 2024-03-30T13:50:32.226 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7060670c:::benchmark_data_smithi043_119832_object47767:head# 2024-03-30T13:50:32.226 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7060eae8:::benchmark_data_smithi043_119832_object52757:head# 2024-03-30T13:50:32.226 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70622d53:::benchmark_data_smithi043_119832_object61765:head# 2024-03-30T13:50:32.227 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7062d3b2:::benchmark_data_smithi043_119832_object2175:head# 2024-03-30T13:50:32.227 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7062de97:::benchmark_data_smithi043_119832_object11735:head# 2024-03-30T13:50:32.227 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70642dce:::benchmark_data_smithi043_119832_object84848:head# 2024-03-30T13:50:32.228 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70646b44:::benchmark_data_smithi043_119832_object12451:head# 2024-03-30T13:50:32.228 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70646c52:::benchmark_data_smithi043_119832_object21960:head# 2024-03-30T13:50:32.228 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70650196:::benchmark_data_smithi043_119832_object30814:head# 2024-03-30T13:50:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7065babf:::benchmark_data_smithi043_119832_object33364:head# 2024-03-30T13:50:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7066585c:::benchmark_data_smithi043_119832_object77356:head# 2024-03-30T13:50:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70676f11:::benchmark_data_smithi043_119832_object9682:head# 2024-03-30T13:50:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70688a1e:::benchmark_data_smithi043_119832_object727:head# 2024-03-30T13:50:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:706932f6:::benchmark_data_smithi043_119832_object24000:head# 2024-03-30T13:50:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70694f86:::benchmark_data_smithi043_119832_object64024:head# 2024-03-30T13:50:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7069b25d:::benchmark_data_smithi043_119832_object37646:head# 2024-03-30T13:50:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7069c825:::benchmark_data_smithi043_119832_object11748:head# 2024-03-30T13:50:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7069ef07:::benchmark_data_smithi043_119832_object29295:head# 2024-03-30T13:50:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:706b33e4:::benchmark_data_smithi043_119832_object59395:head# 2024-03-30T13:50:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:706cf053:::benchmark_data_smithi043_119832_object48848:head# 2024-03-30T13:50:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:706ee211:::benchmark_data_smithi043_119832_object47998:head# 2024-03-30T13:50:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:706ff8f4:::benchmark_data_smithi043_119832_object44698:head# 2024-03-30T13:50:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70703846:::benchmark_data_smithi043_119832_object56524:head# 2024-03-30T13:50:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:707117a7:::benchmark_data_smithi043_119832_object84158:head# 2024-03-30T13:50:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7071726b:::benchmark_data_smithi043_119832_object42655:head# 2024-03-30T13:50:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70718a69:::benchmark_data_smithi043_119832_object79242:head# 2024-03-30T13:50:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7072269f:::benchmark_data_smithi043_119832_object16099:head# 2024-03-30T13:50:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7073d8f8:::benchmark_data_smithi043_119832_object59024:head# 2024-03-30T13:50:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70746d21:::benchmark_data_smithi043_119832_object19859:head# 2024-03-30T13:50:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70746d6e:::benchmark_data_smithi043_119832_object63845:head# 2024-03-30T13:50:32.237 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70750a8d:::benchmark_data_smithi043_119832_object50798:head# 2024-03-30T13:50:32.237 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70751932:::benchmark_data_smithi043_119832_object39021:head# 2024-03-30T13:50:32.237 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70754ad7:::benchmark_data_smithi043_119832_object79760:head# 2024-03-30T13:50:32.242 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70754c5c:::benchmark_data_smithi043_119832_object14577:head# 2024-03-30T13:50:32.242 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7076339f:::benchmark_data_smithi043_119832_object81960:head# 2024-03-30T13:50:32.242 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7077a8da:::benchmark_data_smithi043_119832_object80898:head# 2024-03-30T13:50:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70783d8e:::benchmark_data_smithi043_119832_object8162:head# 2024-03-30T13:50:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70786f92:::benchmark_data_smithi043_119832_object62181:head# 2024-03-30T13:50:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:707bdc9a:::benchmark_data_smithi043_119832_object5120:head# 2024-03-30T13:50:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:707cdc0c:::benchmark_data_smithi043_119832_object53081:head# 2024-03-30T13:50:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:707cfbaa:::benchmark_data_smithi043_119832_object4083:head# 2024-03-30T13:50:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:707de8a5:::benchmark_data_smithi043_119832_object15869:head# 2024-03-30T13:50:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:707f961f:::benchmark_data_smithi043_119832_object57637:head# 2024-03-30T13:50:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:707fd2fd:::benchmark_data_smithi043_119832_object68309:head# 2024-03-30T13:50:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70805f0b:::benchmark_data_smithi043_119832_object77740:head# 2024-03-30T13:50:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7080a57a:::benchmark_data_smithi043_119832_object2761:head# 2024-03-30T13:50:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7084024a:::benchmark_data_smithi043_119832_object67844:head# 2024-03-30T13:50:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70845849:::benchmark_data_smithi043_119832_object38060:head# 2024-03-30T13:50:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70851472:::benchmark_data_smithi043_119832_object259:head# 2024-03-30T13:50:32.244 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7085d40b:::benchmark_data_smithi043_119832_object19132:head# 2024-03-30T13:50:32.244 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7086b525:::benchmark_data_smithi043_119832_object8398:head# 2024-03-30T13:50:32.245 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70877aaf:::benchmark_data_smithi043_119832_object72934:head# 2024-03-30T13:50:32.246 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7087bb97:::benchmark_data_smithi043_119832_object75452:head# 2024-03-30T13:50:32.247 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7087df40:::benchmark_data_smithi043_119832_object57784:head# 2024-03-30T13:50:32.247 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7087fd3a:::benchmark_data_smithi043_119832_object75481:head# 2024-03-30T13:50:32.247 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:708986e9:::benchmark_data_smithi043_119832_object79178:head# 2024-03-30T13:50:32.247 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:708aa26e:::benchmark_data_smithi043_119832_object76126:head# 2024-03-30T13:50:32.247 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:708aae08:::benchmark_data_smithi043_119832_object62427:head# 2024-03-30T13:50:32.247 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:708b5cfa:::benchmark_data_smithi043_119832_object59929:head# 2024-03-30T13:50:32.249 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:708c5c1d:::benchmark_data_smithi043_119832_object29676:head# 2024-03-30T13:50:32.249 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:708cb745:::benchmark_data_smithi043_119832_object26989:head# 2024-03-30T13:50:32.249 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:708d54b4:::benchmark_data_smithi043_119832_object10641:head# 2024-03-30T13:50:32.249 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:708ec144:::benchmark_data_smithi043_119832_object25517:head# 2024-03-30T13:50:32.249 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70917e62:::benchmark_data_smithi043_119832_object67993:head# 2024-03-30T13:50:32.250 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7092d789:::benchmark_data_smithi043_119832_object19021:head# 2024-03-30T13:50:32.250 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70938452:::benchmark_data_smithi043_119832_object69961:head# 2024-03-30T13:50:32.251 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7093dfb6:::benchmark_data_smithi043_119832_object27635:head# 2024-03-30T13:50:32.251 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7093ecd4:::benchmark_data_smithi043_119832_object88067:head# 2024-03-30T13:50:32.251 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70945205:::benchmark_data_smithi043_119832_object52379:head# 2024-03-30T13:50:32.252 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70948d97:::benchmark_data_smithi043_119832_object36752:head# 2024-03-30T13:50:32.252 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7094a7cf:::benchmark_data_smithi043_119832_object59364:head# 2024-03-30T13:50:32.252 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70950fd2:::benchmark_data_smithi043_119832_object73674:head# 2024-03-30T13:50:32.252 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:709561b0:::benchmark_data_smithi043_119832_object31682:head# 2024-03-30T13:50:32.252 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7095c2be:::benchmark_data_smithi043_119832_object64763:head# 2024-03-30T13:50:32.253 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7095ce32:::benchmark_data_smithi043_119832_object3364:head# 2024-03-30T13:50:32.254 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70968d5b:::benchmark_data_smithi043_119832_object35319:head# 2024-03-30T13:50:32.254 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7098b108:::benchmark_data_smithi043_119832_object80590:head# 2024-03-30T13:50:32.255 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7099ff83:::benchmark_data_smithi043_119832_object19991:head# 2024-03-30T13:50:32.255 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:709a0779:::benchmark_data_smithi043_119832_object67392:head# 2024-03-30T13:50:32.255 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:709a5293:::benchmark_data_smithi043_119832_object82167:head# 2024-03-30T13:50:32.259 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:709b6af8:::benchmark_data_smithi043_119832_object21404:head# 2024-03-30T13:50:32.259 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:709bda35:::benchmark_data_smithi043_119832_object3561:head# 2024-03-30T13:50:32.259 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:709c025e:::benchmark_data_smithi043_119832_object64115:head# 2024-03-30T13:50:32.259 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:709cf48a:::benchmark_data_smithi043_119832_object36693:head# 2024-03-30T13:50:32.259 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:709ddc74:::benchmark_data_smithi043_119832_object77748:head# 2024-03-30T13:50:32.259 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:709f1fa1:::benchmark_data_smithi043_119832_object10113:head# 2024-03-30T13:50:32.259 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:709f88bc:::benchmark_data_smithi043_119832_object16494:head# 2024-03-30T13:50:32.259 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70a034a4:::benchmark_data_smithi043_119832_object897:head# 2024-03-30T13:50:32.260 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70a1db6b:::benchmark_data_smithi043_119832_object48209:head# 2024-03-30T13:50:32.261 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70a38d81:::benchmark_data_smithi043_119832_object32327:head# 2024-03-30T13:50:32.261 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70a47492:::benchmark_data_smithi043_119832_object51529:head# 2024-03-30T13:50:32.261 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70a4bdcf:::benchmark_data_smithi043_119832_object38762:head# 2024-03-30T13:50:32.261 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70a50fec:::benchmark_data_smithi043_119832_object22742:head# 2024-03-30T13:50:32.261 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70a5129d:::benchmark_data_smithi043_119832_object80737:head# 2024-03-30T13:50:32.267 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70a5c00c:::benchmark_data_smithi043_119832_object34030:head# 2024-03-30T13:50:32.267 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70a6c002:::benchmark_data_smithi043_119832_object49758:head# 2024-03-30T13:50:32.267 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70a70f59:::benchmark_data_smithi043_119832_object72116:head# 2024-03-30T13:50:32.267 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70a769e9:::benchmark_data_smithi043_119832_object77992:head# 2024-03-30T13:50:32.267 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70a85cf7:::benchmark_data_smithi043_119832_object34679:head# 2024-03-30T13:50:32.267 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70a9f6a3:::benchmark_data_smithi043_119832_object14636:head# 2024-03-30T13:50:32.268 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70aa29fa:::benchmark_data_smithi043_119832_object4937:head# 2024-03-30T13:50:32.268 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70aa5d52:::benchmark_data_smithi043_119832_object48431:head# 2024-03-30T13:50:32.268 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70ab304c:::benchmark_data_smithi043_119832_object29482:head# 2024-03-30T13:50:32.268 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70ad1471:::benchmark_data_smithi043_119832_object25485:head# 2024-03-30T13:50:32.268 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70ad6cbf:::benchmark_data_smithi043_119832_object73639:head# 2024-03-30T13:50:32.268 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70add2bb:::benchmark_data_smithi043_119832_object61063:head# 2024-03-30T13:50:32.268 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70aef3d4:::benchmark_data_smithi043_119832_object5944:head# 2024-03-30T13:50:32.268 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70afc60b:::benchmark_data_smithi043_119832_object11682:head# 2024-03-30T13:50:32.268 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70afc9b6:::benchmark_data_smithi043_119832_object27669:head# 2024-03-30T13:50:32.268 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70b1bed3:::benchmark_data_smithi043_119832_object45763:head# 2024-03-30T13:50:32.268 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70b3814e:::benchmark_data_smithi043_119832_object50392:head# 2024-03-30T13:50:32.270 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70b42d59:::benchmark_data_smithi043_119832_object13368:head# 2024-03-30T13:50:32.270 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70b4f9a1:::benchmark_data_smithi043_119832_object58200:head# 2024-03-30T13:50:32.270 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70b62b19:::benchmark_data_smithi043_119832_object9968:head# 2024-03-30T13:50:32.270 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70b670fd:::benchmark_data_smithi043_119832_object79704:head# 2024-03-30T13:50:32.270 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70b94acb:::benchmark_data_smithi043_119832_object63717:head# 2024-03-30T13:50:32.270 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70baa44e:::benchmark_data_smithi043_119832_object45230:head# 2024-03-30T13:50:32.270 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70bb1389:::benchmark_data_smithi043_119832_object59065:head# 2024-03-30T13:50:32.271 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70bbfd3b:::benchmark_data_smithi043_119832_object7711:head# 2024-03-30T13:50:32.271 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70bc2998:::benchmark_data_smithi043_119832_object6554:head# 2024-03-30T13:50:32.271 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70bc75ae:::benchmark_data_smithi043_119832_object48290:head# 2024-03-30T13:50:32.271 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70bc9ebb:::benchmark_data_smithi043_119832_object22082:head# 2024-03-30T13:50:32.271 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70be30af:::benchmark_data_smithi043_119832_object78392:head# 2024-03-30T13:50:32.273 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70be7634:::benchmark_data_smithi043_119832_object10794:head# 2024-03-30T13:50:32.273 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70c087a4:::benchmark_data_smithi043_119832_object57937:head# 2024-03-30T13:50:32.273 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70c10e1c:::benchmark_data_smithi043_119832_object66540:head# 2024-03-30T13:50:32.273 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70c2f2c9:::benchmark_data_smithi043_119832_object86996:head# 2024-03-30T13:50:32.273 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70c31d98:::benchmark_data_smithi043_119832_object35299:head# 2024-03-30T13:50:32.274 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70c3f9ae:::benchmark_data_smithi043_119832_object82018:head# 2024-03-30T13:50:32.277 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70c55ecc:::benchmark_data_smithi043_119832_object44192:head# 2024-03-30T13:50:32.277 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70c56df9:::benchmark_data_smithi043_119832_object65020:head# 2024-03-30T13:50:32.277 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70c6de3f:::benchmark_data_smithi043_119832_object21705:head# 2024-03-30T13:50:32.277 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70c7d9c5:::benchmark_data_smithi043_119832_object2143:head# 2024-03-30T13:50:32.277 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70c86faf:::benchmark_data_smithi043_119832_object16047:head# 2024-03-30T13:50:32.277 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70c8c0c6:::benchmark_data_smithi043_119832_object912:head# 2024-03-30T13:50:32.278 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70c979a7:::benchmark_data_smithi043_119832_object88175:head# 2024-03-30T13:50:32.278 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70c99758:::benchmark_data_smithi043_119832_object43887:head# 2024-03-30T13:50:32.283 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70c9ca50:::benchmark_data_smithi043_119832_object58743:head# 2024-03-30T13:50:32.283 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70cafd2a:::benchmark_data_smithi043_119832_object76917:head# 2024-03-30T13:50:32.283 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70cb4c2f:::benchmark_data_smithi043_119832_object4345:head# 2024-03-30T13:50:32.283 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70cb91d5:::benchmark_data_smithi043_119832_object43790:head# 2024-03-30T13:50:32.283 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70cb9f74:::benchmark_data_smithi043_119832_object55391:head# 2024-03-30T13:50:32.283 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70cca98b:::benchmark_data_smithi043_119832_object69819:head# 2024-03-30T13:50:32.283 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70cce172:::benchmark_data_smithi043_119832_object13764:head# 2024-03-30T13:50:32.283 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70ce08ed:::benchmark_data_smithi043_119832_object49618:head# 2024-03-30T13:50:32.283 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70ce2460:::benchmark_data_smithi043_119832_object20776:head# 2024-03-30T13:50:32.283 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70ce2e1b:::benchmark_data_smithi043_119832_object72936:head# 2024-03-30T13:50:32.283 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70ce78ca:::benchmark_data_smithi043_119832_object38090:head# 2024-03-30T13:50:32.284 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70cebad9:::benchmark_data_smithi043_119832_object5897:head# 2024-03-30T13:50:32.284 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70d10792:::benchmark_data_smithi043_119832_object83269:head# 2024-03-30T13:50:32.285 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70d15097:::benchmark_data_smithi043_119832_object71872:head# 2024-03-30T13:50:32.285 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70d19ece:::benchmark_data_smithi043_119832_object34017:head# 2024-03-30T13:50:32.285 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70d3443f:::benchmark_data_smithi043_119832_object18548:head# 2024-03-30T13:50:32.285 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70d3cdc6:::benchmark_data_smithi043_119832_object55617:head# 2024-03-30T13:50:32.286 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70d47814:::benchmark_data_smithi043_119832_object26067:head# 2024-03-30T13:50:32.286 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70d5d79f:::benchmark_data_smithi043_119832_object87366:head# 2024-03-30T13:50:32.293 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70d6a086:::benchmark_data_smithi043_119832_object47062:head# 2024-03-30T13:50:32.293 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70d99e65:::benchmark_data_smithi043_119832_object20407:head# 2024-03-30T13:50:32.293 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70da1ae2:::benchmark_data_smithi043_119832_object75461:head# 2024-03-30T13:50:32.293 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70da627c:::benchmark_data_smithi043_119832_object54673:head# 2024-03-30T13:50:32.293 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70da7361:::benchmark_data_smithi043_119832_object37378:head# 2024-03-30T13:50:32.293 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70daf31c:::benchmark_data_smithi043_119832_object53232:head# 2024-03-30T13:50:32.293 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70dc94ee:::benchmark_data_smithi043_119832_object61401:head# 2024-03-30T13:50:32.293 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70dcb1c6:::benchmark_data_smithi043_119832_object33503:head# 2024-03-30T13:50:32.293 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70e042a5:::benchmark_data_smithi043_119832_object20107:head# 2024-03-30T13:50:32.293 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70e0e0d4:::benchmark_data_smithi043_119832_object80393:head# 2024-03-30T13:50:32.294 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70e1ba07:::benchmark_data_smithi043_119832_object19574:head# 2024-03-30T13:50:32.294 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70e1d288:::benchmark_data_smithi043_119832_object9599:head# 2024-03-30T13:50:32.294 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70e206ba:::benchmark_data_smithi043_119832_object7001:head# 2024-03-30T13:50:32.294 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70e219e1:::benchmark_data_smithi043_119832_object81061:head# 2024-03-30T13:50:32.294 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70e26308:::benchmark_data_smithi043_119832_object23998:head# 2024-03-30T13:50:32.294 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:32.299 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70e29d14:::benchmark_data_smithi043_119832_object32775:head# 2024-03-30T13:50:32.299 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70e38142:::benchmark_data_smithi043_119832_object24846:head# 2024-03-30T13:50:32.299 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70e4587a:::benchmark_data_smithi043_119832_object88453:head# 2024-03-30T13:50:32.299 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70e5e6ef:::benchmark_data_smithi043_119832_object82622:head# 2024-03-30T13:50:32.299 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70e62442:::benchmark_data_smithi043_119832_object21132:head# 2024-03-30T13:50:32.299 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70e73619:::benchmark_data_smithi043_119832_object55971:head# 2024-03-30T13:50:32.299 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70e882ce:::benchmark_data_smithi043_119832_object34507:head# 2024-03-30T13:50:32.299 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70e8e27c:::benchmark_data_smithi043_119832_object51036:head# 2024-03-30T13:50:32.299 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70ead256:::benchmark_data_smithi043_119832_object34228:head# 2024-03-30T13:50:32.299 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70eb5d03:::benchmark_data_smithi043_119832_object1108:head# 2024-03-30T13:50:32.300 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70ebf3ec:::benchmark_data_smithi043_119832_object47705:head# 2024-03-30T13:50:32.301 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70edfb2a:::benchmark_data_smithi043_119832_object34899:head# 2024-03-30T13:50:32.301 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70ee2108:::benchmark_data_smithi043_119832_object20191:head# 2024-03-30T13:50:32.301 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70ee7917:::benchmark_data_smithi043_119832_object5337:head# 2024-03-30T13:50:32.301 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70f02eca:::benchmark_data_smithi043_119832_object16048:head# 2024-03-30T13:50:32.301 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70f138f7:::benchmark_data_smithi043_119832_object26823:head# 2024-03-30T13:50:32.303 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70f29105:::benchmark_data_smithi043_119832_object38709:head# 2024-03-30T13:50:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70f31c31:::benchmark_data_smithi043_119832_object46513:head# 2024-03-30T13:50:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70f3f507:::benchmark_data_smithi043_119832_object41724:head# 2024-03-30T13:50:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70f5c6a8:::benchmark_data_smithi043_119832_object62709:head# 2024-03-30T13:50:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70f5ce29:::benchmark_data_smithi043_119832_object53902:head# 2024-03-30T13:50:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70f5ed6e:::benchmark_data_smithi043_119832_object54328:head# 2024-03-30T13:50:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70f6e0a9:::benchmark_data_smithi043_119832_object16385:head# 2024-03-30T13:50:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70f82676:::benchmark_data_smithi043_119832_object1196:head# 2024-03-30T13:50:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70f83397:::benchmark_data_smithi043_119832_object47870:head# 2024-03-30T13:50:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70f9942c:::benchmark_data_smithi043_119832_object1709:head# 2024-03-30T13:50:32.305 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70f9ddba:::benchmark_data_smithi043_119832_object62744:head# 2024-03-30T13:50:32.305 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70fb24e1:::benchmark_data_smithi043_119832_object78848:head# 2024-03-30T13:50:32.306 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70fbf716:::benchmark_data_smithi043_119832_object70397:head# 2024-03-30T13:50:32.306 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70fcdeac:::benchmark_data_smithi043_119832_object26237:head# 2024-03-30T13:50:32.306 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70ff8460:::benchmark_data_smithi043_119832_object76464:head# 2024-03-30T13:50:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:70ffef75:::benchmark_data_smithi043_119832_object85648:head# 2024-03-30T13:50:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:710016fe:::benchmark_data_smithi043_119832_object54701:head# 2024-03-30T13:50:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7101010a:::benchmark_data_smithi043_119832_object10297:head# 2024-03-30T13:50:32.309 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71022245:::benchmark_data_smithi043_119832_object76540:head# 2024-03-30T13:50:32.309 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7104bb68:::benchmark_data_smithi043_119832_object7549:head# 2024-03-30T13:50:32.309 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7107470e:::benchmark_data_smithi043_119832_object88377:head# 2024-03-30T13:50:32.309 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7107bf2b:::benchmark_data_smithi043_119832_object7467:head# 2024-03-30T13:50:32.309 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71094308:::benchmark_data_smithi043_119832_object52532:head# 2024-03-30T13:50:32.311 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:710956f8:::benchmark_data_smithi043_119832_object27462:head# 2024-03-30T13:50:32.311 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:710990e7:::benchmark_data_smithi043_119832_object45600:head# 2024-03-30T13:50:32.311 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71099a65:::benchmark_data_smithi043_119832_object64074:head# 2024-03-30T13:50:32.311 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:710a5ba4:::benchmark_data_smithi043_119832_object7533:head# 2024-03-30T13:50:32.312 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:710b6f72:::benchmark_data_smithi043_119832_object81996:head# 2024-03-30T13:50:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:710bc09f:::benchmark_data_smithi043_119832_object36925:head# 2024-03-30T13:50:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:710c014e:::benchmark_data_smithi043_119832_object49760:head# 2024-03-30T13:50:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:710c3a4f:::benchmark_data_smithi043_119832_object12763:head# 2024-03-30T13:50:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:710ca45b:::benchmark_data_smithi043_119832_object21226:head# 2024-03-30T13:50:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:710def2d:::benchmark_data_smithi043_119832_object11462:head# 2024-03-30T13:50:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:710df8b5:::benchmark_data_smithi043_119832_object72039:head# 2024-03-30T13:50:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:710ffa5c:::benchmark_data_smithi043_119832_object19437:head# 2024-03-30T13:50:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:711122cb:::benchmark_data_smithi043_119832_object11569:head# 2024-03-30T13:50:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7111bfde:::benchmark_data_smithi043_119832_object30126:head# 2024-03-30T13:50:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7111dc0d:::benchmark_data_smithi043_119832_object46779:head# 2024-03-30T13:50:32.315 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:711373a6:::benchmark_data_smithi043_119832_object87265:head# 2024-03-30T13:50:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7113f233:::benchmark_data_smithi043_119832_object17894:head# 2024-03-30T13:50:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:711490cf:::benchmark_data_smithi043_119832_object21673:head# 2024-03-30T13:50:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7115ef30:::benchmark_data_smithi043_119832_object14694:head# 2024-03-30T13:50:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7115fc4d:::benchmark_data_smithi043_119832_object28441:head# 2024-03-30T13:50:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7119a60d:::benchmark_data_smithi043_119832_object33770:head# 2024-03-30T13:50:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:711a3e7f:::benchmark_data_smithi043_119832_object10336:head# 2024-03-30T13:50:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:711b328c:::benchmark_data_smithi043_119832_object42876:head# 2024-03-30T13:50:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:711b43a7:::benchmark_data_smithi043_119832_object6477:head# 2024-03-30T13:50:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:711c37de:::benchmark_data_smithi043_119832_object83531:head# 2024-03-30T13:50:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:711c4fb3:::benchmark_data_smithi043_119832_object49139:head# 2024-03-30T13:50:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:711d0cd8:::benchmark_data_smithi043_119832_object4107:head# 2024-03-30T13:50:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:711d64f0:::benchmark_data_smithi043_119832_object36049:head# 2024-03-30T13:50:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:711eec02:::benchmark_data_smithi043_119832_object58892:head# 2024-03-30T13:50:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:711f4e2e:::benchmark_data_smithi043_119832_object4698:head# 2024-03-30T13:50:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:711fdb50:::benchmark_data_smithi043_119832_object11468:head# 2024-03-30T13:50:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7120421e:::benchmark_data_smithi043_119832_object51142:head# 2024-03-30T13:50:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71205f7a:::benchmark_data_smithi043_119832_object5065:head# 2024-03-30T13:50:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:712095e6:::benchmark_data_smithi043_119832_object65099:head# 2024-03-30T13:50:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71212c97:::benchmark_data_smithi043_119832_object59864:head# 2024-03-30T13:50:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71214d5e:::benchmark_data_smithi043_119832_object83756:head# 2024-03-30T13:50:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71217fdd:::benchmark_data_smithi043_119832_object11549:head# 2024-03-30T13:50:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7121b09e:::benchmark_data_smithi043_119832_object76525:head# 2024-03-30T13:50:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7121fa92:::benchmark_data_smithi043_119832_object67585:head# 2024-03-30T13:50:32.324 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7123ed1f:::benchmark_data_smithi043_119832_object34320:head# 2024-03-30T13:50:32.324 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:712444ff:::benchmark_data_smithi043_119832_object60733:head# 2024-03-30T13:50:32.324 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:712477ee:::benchmark_data_smithi043_119832_object24768:head# 2024-03-30T13:50:32.324 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:712493f3:::benchmark_data_smithi043_119832_object28693:head# 2024-03-30T13:50:32.324 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7125882a:::benchmark_data_smithi043_119832_object46992:head# 2024-03-30T13:50:32.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:32.333 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:32.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:32.364 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7125c0b8:::benchmark_data_smithi043_119832_object3089:head# 2024-03-30T13:50:32.364 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71272364:::benchmark_data_smithi043_119832_object22552:head# 2024-03-30T13:50:32.364 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71274fd7:::benchmark_data_smithi043_119832_object6305:head# 2024-03-30T13:50:32.364 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:712774b5:::benchmark_data_smithi043_119832_object37311:head# 2024-03-30T13:50:32.364 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:712828c8:::benchmark_data_smithi043_119832_object45857:head# 2024-03-30T13:50:32.364 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:712865c6:::benchmark_data_smithi043_119832_object8731:head# 2024-03-30T13:50:32.364 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7128ccf8:::benchmark_data_smithi043_119832_object51690:head# 2024-03-30T13:50:32.364 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:712aa8ab:::benchmark_data_smithi043_119832_object33800:head# 2024-03-30T13:50:32.364 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:712afd7c:::benchmark_data_smithi043_119832_object24238:head# 2024-03-30T13:50:32.364 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:712b7563:::benchmark_data_smithi043_119832_object80332:head# 2024-03-30T13:50:32.364 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:712c8ca0:::benchmark_data_smithi043_119832_object21965:head# 2024-03-30T13:50:32.364 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:712cc413:::benchmark_data_smithi043_119832_object15303:head# 2024-03-30T13:50:32.364 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:712cd225:::benchmark_data_smithi043_119832_object69661:head# 2024-03-30T13:50:32.365 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:712ebae4:::benchmark_data_smithi043_119832_object79379:head# 2024-03-30T13:50:32.365 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71300ffa:::benchmark_data_smithi043_119832_object15979:head# 2024-03-30T13:50:32.365 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71320613:::benchmark_data_smithi043_119832_object30974:head# 2024-03-30T13:50:32.365 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71320a28:::benchmark_data_smithi043_119832_object4059:head# 2024-03-30T13:50:32.365 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7133a44b:::benchmark_data_smithi043_119832_object59885:head# 2024-03-30T13:50:32.365 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7133cd58:::benchmark_data_smithi043_119832_object71162:head# 2024-03-30T13:50:32.365 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7134ff20:::benchmark_data_smithi043_119832_object56807:head# 2024-03-30T13:50:32.365 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7135db38:::benchmark_data_smithi043_119832_object58720:head# 2024-03-30T13:50:32.365 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71360cb2:::benchmark_data_smithi043_119832_object75923:head# 2024-03-30T13:50:32.365 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71362067:::benchmark_data_smithi043_119832_object34116:head# 2024-03-30T13:50:32.365 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71372b28:::benchmark_data_smithi043_119832_object31587:head# 2024-03-30T13:50:32.365 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71376e76:::benchmark_data_smithi043_119832_object15543:head# 2024-03-30T13:50:32.365 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7138cab5:::benchmark_data_smithi043_119832_object21335:head# 2024-03-30T13:50:32.366 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7139089e:::benchmark_data_smithi043_119832_object547:head# 2024-03-30T13:50:32.366 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:713a8b79:::benchmark_data_smithi043_119832_object8771:head# 2024-03-30T13:50:32.366 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:713a9566:::benchmark_data_smithi043_119832_object72205:head# 2024-03-30T13:50:32.366 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:713bad7d:::benchmark_data_smithi043_119832_object72540:head# 2024-03-30T13:50:32.366 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:713c6b4d:::benchmark_data_smithi043_119832_object15619:head# 2024-03-30T13:50:32.366 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:713d7d44:::benchmark_data_smithi043_119832_object52742:head# 2024-03-30T13:50:32.366 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:713da36d:::benchmark_data_smithi043_119832_object45157:head# 2024-03-30T13:50:32.366 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:713da72c:::benchmark_data_smithi043_119832_object4799:head# 2024-03-30T13:50:32.366 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:713da835:::benchmark_data_smithi043_119832_object20280:head# 2024-03-30T13:50:32.366 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:713db775:::benchmark_data_smithi043_119832_object57322:head# 2024-03-30T13:50:32.366 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:713e1b03:::benchmark_data_smithi043_119832_object30206:head# 2024-03-30T13:50:32.366 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:713e788a:::benchmark_data_smithi043_119832_object59252:head# 2024-03-30T13:50:32.366 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:713ec4e0:::benchmark_data_smithi043_119832_object39617:head# 2024-03-30T13:50:32.367 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:713ed01a:::benchmark_data_smithi043_119832_object5130:head# 2024-03-30T13:50:32.367 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:713f2f89:::benchmark_data_smithi043_119832_object7468:head# 2024-03-30T13:50:32.367 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71404ad2:::benchmark_data_smithi043_119832_object79162:head# 2024-03-30T13:50:32.367 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7140644d:::benchmark_data_smithi043_119832_object31137:head# 2024-03-30T13:50:32.367 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71406837:::benchmark_data_smithi043_119832_object64817:head# 2024-03-30T13:50:32.367 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71411166:::benchmark_data_smithi043_119832_object24962:head# 2024-03-30T13:50:32.367 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7141359a:::benchmark_data_smithi043_119832_object36428:head# 2024-03-30T13:50:32.367 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7142ae08:::benchmark_data_smithi043_119832_object15807:head# 2024-03-30T13:50:32.367 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:714310a2:::benchmark_data_smithi043_119832_object48425:head# 2024-03-30T13:50:32.367 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71432a5d:::benchmark_data_smithi043_119832_object67009:head# 2024-03-30T13:50:32.367 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71437ac5:::benchmark_data_smithi043_119832_object22057:head# 2024-03-30T13:50:32.367 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7143ded1:::benchmark_data_smithi043_119832_object15458:head# 2024-03-30T13:50:32.367 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71449bb7:::benchmark_data_smithi043_119832_object45867:head# 2024-03-30T13:50:32.367 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7144bc10:::benchmark_data_smithi043_119832_object58667:head# 2024-03-30T13:50:32.368 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7144da3c:::benchmark_data_smithi043_119832_object65589:head# 2024-03-30T13:50:32.368 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7144edad:::benchmark_data_smithi043_119832_object37121:head# 2024-03-30T13:50:32.368 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71452f52:::benchmark_data_smithi043_119832_object62860:head# 2024-03-30T13:50:32.368 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71457a4a:::benchmark_data_smithi043_119832_object84600:head# 2024-03-30T13:50:32.368 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7145a23c:::benchmark_data_smithi043_119832_object67517:head# 2024-03-30T13:50:32.368 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7145a9ae:::benchmark_data_smithi043_119832_object31876:head# 2024-03-30T13:50:32.368 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7145b26a:::benchmark_data_smithi043_119832_object57261:head# 2024-03-30T13:50:32.368 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7145c1e3:::benchmark_data_smithi043_119832_object85969:head# 2024-03-30T13:50:32.368 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71464e84:::benchmark_data_smithi043_119832_object72564:head# 2024-03-30T13:50:32.368 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71471de9:::benchmark_data_smithi043_119832_object45630:head# 2024-03-30T13:50:32.368 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7148e5d6:::benchmark_data_smithi043_119832_object8775:head# 2024-03-30T13:50:32.368 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7148eb83:::benchmark_data_smithi043_119832_object7511:head# 2024-03-30T13:50:32.368 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7148f9e2:::benchmark_data_smithi043_119832_object13893:head# 2024-03-30T13:50:32.369 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:714a96b4:::benchmark_data_smithi043_119832_object2644:head# 2024-03-30T13:50:32.369 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:714ab284:::benchmark_data_smithi043_119832_object26243:head# 2024-03-30T13:50:32.369 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:714c405c:::benchmark_data_smithi043_119832_object68341:head# 2024-03-30T13:50:32.369 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:714d241c:::benchmark_data_smithi043_119832_object69106:head# 2024-03-30T13:50:32.369 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:714db556:::benchmark_data_smithi043_119832_object46592:head# 2024-03-30T13:50:32.369 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:714eca97:::benchmark_data_smithi043_119832_object56452:head# 2024-03-30T13:50:32.369 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:714fef1b:::benchmark_data_smithi043_119832_object38516:head# 2024-03-30T13:50:32.369 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71504dee:::benchmark_data_smithi043_119832_object22041:head# 2024-03-30T13:50:32.369 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7151c032:::benchmark_data_smithi043_119832_object80040:head# 2024-03-30T13:50:32.369 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71534eed:::benchmark_data_smithi043_119832_object82424:head# 2024-03-30T13:50:32.369 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7153cc72:::benchmark_data_smithi043_119832_object68307:head# 2024-03-30T13:50:32.369 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:715478aa:::benchmark_data_smithi043_119832_object25084:head# 2024-03-30T13:50:32.369 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7155f84d:::benchmark_data_smithi043_119832_object16639:head# 2024-03-30T13:50:32.369 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71563390:::benchmark_data_smithi043_119832_object1237:head# 2024-03-30T13:50:32.370 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:715656c8:::benchmark_data_smithi043_119832_object30577:head# 2024-03-30T13:50:32.370 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7156a402:::benchmark_data_smithi043_119832_object56146:head# 2024-03-30T13:50:32.370 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71574e68:::benchmark_data_smithi043_119832_object61042:head# 2024-03-30T13:50:32.370 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71575bb2:::benchmark_data_smithi043_119832_object79695:head# 2024-03-30T13:50:32.370 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71576283:::benchmark_data_smithi043_119832_object21717:head# 2024-03-30T13:50:32.370 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7157ca54:::benchmark_data_smithi043_119832_object17332:head# 2024-03-30T13:50:32.370 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7157d061:::benchmark_data_smithi043_119832_object18703:head# 2024-03-30T13:50:32.370 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71594ee6:::benchmark_data_smithi043_119832_object10725:head# 2024-03-30T13:50:32.370 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7159cb0b:::benchmark_data_smithi043_119832_object86669:head# 2024-03-30T13:50:32.370 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:715ab001:::benchmark_data_smithi043_119832_object70849:head# 2024-03-30T13:50:32.370 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:715c3c6f:::benchmark_data_smithi043_119832_object21564:head# 2024-03-30T13:50:32.370 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:715d8b87:::benchmark_data_smithi043_119832_object16933:head# 2024-03-30T13:50:32.370 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:715dc948:::benchmark_data_smithi043_119832_object12669:head# 2024-03-30T13:50:32.370 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:715f9647:::benchmark_data_smithi043_119832_object4484:head# 2024-03-30T13:50:32.371 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:716073fb:::benchmark_data_smithi043_119832_object67906:head# 2024-03-30T13:50:32.371 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:716137bf:::benchmark_data_smithi043_119832_object17099:head# 2024-03-30T13:50:32.371 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7161bedd:::benchmark_data_smithi043_119832_object74331:head# 2024-03-30T13:50:32.371 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71628fdd:::benchmark_data_smithi043_119832_object17188:head# 2024-03-30T13:50:32.371 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7162b9d5:::benchmark_data_smithi043_119832_object86332:head# 2024-03-30T13:50:32.371 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7165d25f:::benchmark_data_smithi043_119832_object61692:head# 2024-03-30T13:50:32.371 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7166d261:::benchmark_data_smithi043_119832_object45925:head# 2024-03-30T13:50:32.371 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:716752f6:::benchmark_data_smithi043_119832_object88251:head# 2024-03-30T13:50:32.371 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71675d02:::benchmark_data_smithi043_119832_object25737:head# 2024-03-30T13:50:32.371 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71675fd5:::benchmark_data_smithi043_119832_object73009:head# 2024-03-30T13:50:32.371 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7169a490:::benchmark_data_smithi043_119832_object59279:head# 2024-03-30T13:50:32.371 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7169e685:::benchmark_data_smithi043_119832_object17952:head# 2024-03-30T13:50:32.371 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:716a3f24:::benchmark_data_smithi043_119832_object53361:head# 2024-03-30T13:50:32.371 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:716a793d:::benchmark_data_smithi043_119832_object1193:head# 2024-03-30T13:50:32.372 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:716ace89:::benchmark_data_smithi043_119832_object51542:head# 2024-03-30T13:50:32.372 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:716ca3ad:::benchmark_data_smithi043_119832_object65372:head# 2024-03-30T13:50:32.372 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:716d8bc9:::benchmark_data_smithi043_119832_object62653:head# 2024-03-30T13:50:32.372 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:716de828:::benchmark_data_smithi043_119832_object60030:head# 2024-03-30T13:50:32.372 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:716e4864:::benchmark_data_smithi043_119832_object56954:head# 2024-03-30T13:50:32.372 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:716ef3b3:::benchmark_data_smithi043_119832_object86156:head# 2024-03-30T13:50:32.372 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7170629b:::benchmark_data_smithi043_119832_object4990:head# 2024-03-30T13:50:32.372 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:717094ac:::benchmark_data_smithi043_119832_object27454:head# 2024-03-30T13:50:32.372 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7171b99e:::benchmark_data_smithi043_119832_object60432:head# 2024-03-30T13:50:32.372 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71724304:::benchmark_data_smithi043_119832_object84739:head# 2024-03-30T13:50:32.372 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71734e2c:::benchmark_data_smithi043_119832_object12924:head# 2024-03-30T13:50:32.379 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7173b2d7:::benchmark_data_smithi043_119832_object15848:head# 2024-03-30T13:50:32.379 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7174a0f2:::benchmark_data_smithi043_119832_object76932:head# 2024-03-30T13:50:32.379 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7174a3e2:::benchmark_data_smithi043_119832_object28306:head# 2024-03-30T13:50:32.380 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7175468d:::benchmark_data_smithi043_119832_object7827:head# 2024-03-30T13:50:32.380 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7175635f:::benchmark_data_smithi043_119832_object72170:head# 2024-03-30T13:50:32.380 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7176bf93:::benchmark_data_smithi043_119832_object76490:head# 2024-03-30T13:50:32.380 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:717707da:::benchmark_data_smithi043_119832_object72399:head# 2024-03-30T13:50:32.380 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7177b760:::benchmark_data_smithi043_119832_object415:head# 2024-03-30T13:50:32.380 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7177f0ad:::benchmark_data_smithi043_119832_object52042:head# 2024-03-30T13:50:32.380 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71782385:::benchmark_data_smithi043_119832_object60641:head# 2024-03-30T13:50:32.380 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:717916bd:::benchmark_data_smithi043_119832_object7714:head# 2024-03-30T13:50:32.380 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:717996ae:::benchmark_data_smithi043_119832_object22226:head# 2024-03-30T13:50:32.381 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7179c933:::benchmark_data_smithi043_119832_object42231:head# 2024-03-30T13:50:32.381 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7179d6b6:::benchmark_data_smithi043_119832_object85814:head# 2024-03-30T13:50:32.381 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:717b9fd2:::benchmark_data_smithi043_119832_object23815:head# 2024-03-30T13:50:32.381 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:717c53c2:::benchmark_data_smithi043_119832_object23186:head# 2024-03-30T13:50:32.381 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:717c5e1f:::benchmark_data_smithi043_119832_object69156:head# 2024-03-30T13:50:32.381 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:717cf5e3:::benchmark_data_smithi043_119832_object18851:head# 2024-03-30T13:50:32.381 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:717e6271:::benchmark_data_smithi043_119832_object83323:head# 2024-03-30T13:50:32.381 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:717ff251:::benchmark_data_smithi043_119832_object55367:head# 2024-03-30T13:50:32.381 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71802c64:::benchmark_data_smithi043_119832_object10978:head# 2024-03-30T13:50:32.381 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:718108d6:::benchmark_data_smithi043_119832_object66142:head# 2024-03-30T13:50:32.381 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7181787e:::benchmark_data_smithi043_119832_object63630:head# 2024-03-30T13:50:32.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7183150d:::benchmark_data_smithi043_119832_object56386:head# 2024-03-30T13:50:32.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71843250:::benchmark_data_smithi043_119832_object49605:head# 2024-03-30T13:50:32.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:718463e2:::benchmark_data_smithi043_119832_object29536:head# 2024-03-30T13:50:32.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71859183:::benchmark_data_smithi043_119832_object68907:head# 2024-03-30T13:50:32.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71865853:::benchmark_data_smithi043_119832_object10575:head# 2024-03-30T13:50:32.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7186ce50:::benchmark_data_smithi043_119832_object77351:head# 2024-03-30T13:50:32.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:718790f0:::benchmark_data_smithi043_119832_object5090:head# 2024-03-30T13:50:32.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:718876cd:::benchmark_data_smithi043_119832_object63638:head# 2024-03-30T13:50:32.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:718adc57:::benchmark_data_smithi043_119832_object67986:head# 2024-03-30T13:50:32.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:718d06dd:::benchmark_data_smithi043_119832_object79693:head# 2024-03-30T13:50:32.383 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:718d1750:::benchmark_data_smithi043_119832_object31131:head# 2024-03-30T13:50:32.383 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:718e81ac:::benchmark_data_smithi043_119832_object35318:head# 2024-03-30T13:50:32.383 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:718ec4a4:::benchmark_data_smithi043_119832_object40143:head# 2024-03-30T13:50:32.383 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:718ec887:::benchmark_data_smithi043_119832_object2097:head# 2024-03-30T13:50:32.383 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:718f27c6:::benchmark_data_smithi043_119832_object51234:head# 2024-03-30T13:50:32.383 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:718f5ea9:::benchmark_data_smithi043_119832_object39236:head# 2024-03-30T13:50:32.383 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71908699:::benchmark_data_smithi043_119832_object42161:head# 2024-03-30T13:50:32.383 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71931b77:::benchmark_data_smithi043_119832_object27294:head# 2024-03-30T13:50:32.383 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71937fb0:::benchmark_data_smithi043_119832_object51533:head# 2024-03-30T13:50:32.385 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:719384de:::benchmark_data_smithi043_119832_object22402:head# 2024-03-30T13:50:32.386 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71938604:::benchmark_data_smithi043_119832_object76667:head# 2024-03-30T13:50:32.386 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7193eeda:::benchmark_data_smithi043_119832_object64103:head# 2024-03-30T13:50:32.386 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71941de6:::benchmark_data_smithi043_119832_object83701:head# 2024-03-30T13:50:32.386 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71955e9b:::benchmark_data_smithi043_119832_object62670:head# 2024-03-30T13:50:32.386 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7195b5ff:::benchmark_data_smithi043_119832_object15340:head# 2024-03-30T13:50:32.386 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:719725bf:::benchmark_data_smithi043_119832_object20822:head# 2024-03-30T13:50:32.386 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:719794dc:::benchmark_data_smithi043_119832_object19047:head# 2024-03-30T13:50:32.386 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:719869dd:::benchmark_data_smithi043_119832_object9035:head# 2024-03-30T13:50:32.386 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:719893af:::benchmark_data_smithi043_119832_object28060:head# 2024-03-30T13:50:32.386 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7198e463:::benchmark_data_smithi043_119832_object73068:head# 2024-03-30T13:50:32.387 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71993fe8:::benchmark_data_smithi043_119832_object6802:head# 2024-03-30T13:50:32.387 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7199efed:::benchmark_data_smithi043_119832_object19931:head# 2024-03-30T13:50:32.388 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:719a266e:::benchmark_data_smithi043_119832_object40329:head# 2024-03-30T13:50:32.388 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:719b9508:::benchmark_data_smithi043_119832_object33743:head# 2024-03-30T13:50:32.388 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:719bafc3:::benchmark_data_smithi043_119832_object54690:head# 2024-03-30T13:50:32.388 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:719c513f:::benchmark_data_smithi043_119832_object27531:head# 2024-03-30T13:50:32.388 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:719cc789:::benchmark_data_smithi043_119832_object84563:head# 2024-03-30T13:50:32.388 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:719cf843:::benchmark_data_smithi043_119832_object54838:head# 2024-03-30T13:50:32.389 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a032d9:::benchmark_data_smithi043_119832_object15962:head# 2024-03-30T13:50:32.389 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a051bf:::benchmark_data_smithi043_119832_object81379:head# 2024-03-30T13:50:32.389 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a0f357:::benchmark_data_smithi043_119832_object12332:head# 2024-03-30T13:50:32.390 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a16b95:::benchmark_data_smithi043_119832_object69971:head# 2024-03-30T13:50:32.390 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a1f896:::benchmark_data_smithi043_119832_object18224:head# 2024-03-30T13:50:32.390 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a27cc1:::benchmark_data_smithi043_119832_object60053:head# 2024-03-30T13:50:32.391 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a2cb22:::benchmark_data_smithi043_119832_object67053:head# 2024-03-30T13:50:32.393 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a329dc:::benchmark_data_smithi043_119832_object12914:head# 2024-03-30T13:50:32.393 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a338cb:::benchmark_data_smithi043_119832_object40353:head# 2024-03-30T13:50:32.393 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a37b80:::benchmark_data_smithi043_119832_object31345:head# 2024-03-30T13:50:32.393 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a37dbb:::benchmark_data_smithi043_119832_object24080:head# 2024-03-30T13:50:32.393 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a55489:::benchmark_data_smithi043_119832_object66734:head# 2024-03-30T13:50:32.394 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a5d5df:::benchmark_data_smithi043_119832_object72528:head# 2024-03-30T13:50:32.396 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a779e5:::benchmark_data_smithi043_119832_object1029:head# 2024-03-30T13:50:32.397 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a79df7:::benchmark_data_smithi043_119832_object51393:head# 2024-03-30T13:50:32.397 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a7ac54:::benchmark_data_smithi043_119832_object36813:head# 2024-03-30T13:50:32.397 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a8a73c:::benchmark_data_smithi043_119832_object13566:head# 2024-03-30T13:50:32.398 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a8e173:::benchmark_data_smithi043_119832_object21542:head# 2024-03-30T13:50:32.399 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a9164e:::benchmark_data_smithi043_119832_object36692:head# 2024-03-30T13:50:32.399 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a93533:::benchmark_data_smithi043_119832_object56622:head# 2024-03-30T13:50:32.399 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a93754:::benchmark_data_smithi043_119832_object53923:head# 2024-03-30T13:50:32.399 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a959b0:::benchmark_data_smithi043_119832_object75664:head# 2024-03-30T13:50:32.400 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71a9d278:::benchmark_data_smithi043_119832_object13750:head# 2024-03-30T13:50:32.400 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71aab8d8:::benchmark_data_smithi043_119832_object83930:head# 2024-03-30T13:50:32.400 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71aaf963:::benchmark_data_smithi043_119832_object86367:head# 2024-03-30T13:50:32.401 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71ab38c6:::benchmark_data_smithi043_119832_object39410:head# 2024-03-30T13:50:32.401 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71ab672b:::benchmark_data_smithi043_119832_object86548:head# 2024-03-30T13:50:32.402 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71ac4947:::benchmark_data_smithi043_119832_object7237:head# 2024-03-30T13:50:32.402 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71adca02:::benchmark_data_smithi043_119832_object31439:head# 2024-03-30T13:50:32.403 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71ae7e4a:::benchmark_data_smithi043_119832_object1382:head# 2024-03-30T13:50:32.404 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71ae82cb:::benchmark_data_smithi043_119832_object57352:head# 2024-03-30T13:50:32.404 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71ae8a17:::benchmark_data_smithi043_119832_object52212:head# 2024-03-30T13:50:32.404 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71aea1c3:::benchmark_data_smithi043_119832_object18362:head# 2024-03-30T13:50:32.404 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71b161b3:::benchmark_data_smithi043_119832_object84006:head# 2024-03-30T13:50:32.405 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71b5d518:::benchmark_data_smithi043_119832_object30337:head# 2024-03-30T13:50:32.405 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71b5fed0:::benchmark_data_smithi043_119832_object35777:head# 2024-03-30T13:50:32.405 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71b7abb1:::benchmark_data_smithi043_119832_object82356:head# 2024-03-30T13:50:32.406 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71b8e8e4:::benchmark_data_smithi043_119832_object34596:head# 2024-03-30T13:50:32.406 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71b99b59:::benchmark_data_smithi043_119832_object85775:head# 2024-03-30T13:50:32.407 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71bab083:::benchmark_data_smithi043_119832_object83716:head# 2024-03-30T13:50:32.407 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71bc8784:::benchmark_data_smithi043_119832_object44910:head# 2024-03-30T13:50:32.408 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71bcf0ed:::benchmark_data_smithi043_119832_object24694:head# 2024-03-30T13:50:32.408 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71be6695:::benchmark_data_smithi043_119832_object43485:head# 2024-03-30T13:50:32.408 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71bf0fa4:::benchmark_data_smithi043_119832_object4988:head# 2024-03-30T13:50:32.410 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71c02f22:::benchmark_data_smithi043_119832_object57462:head# 2024-03-30T13:50:32.410 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71c0611d:::benchmark_data_smithi043_119832_object59044:head# 2024-03-30T13:50:32.410 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71c1c97a:::benchmark_data_smithi043_119832_object37983:head# 2024-03-30T13:50:32.410 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71c235cb:::benchmark_data_smithi043_119832_object45435:head# 2024-03-30T13:50:32.410 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71c32f55:::benchmark_data_smithi043_119832_object63065:head# 2024-03-30T13:50:32.411 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71c33dff:::benchmark_data_smithi043_119832_object41478:head# 2024-03-30T13:50:32.411 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71c3cdfc:::benchmark_data_smithi043_119832_object63302:head# 2024-03-30T13:50:32.411 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71c3df76:::benchmark_data_smithi043_119832_object74277:head# 2024-03-30T13:50:32.412 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71c4d0f0:::benchmark_data_smithi043_119832_object81541:head# 2024-03-30T13:50:32.412 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71c5d95e:::benchmark_data_smithi043_119832_object5152:head# 2024-03-30T13:50:32.412 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71c6a143:::benchmark_data_smithi043_119832_object77535:head# 2024-03-30T13:50:32.413 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71c6e14c:::benchmark_data_smithi043_119832_object83887:head# 2024-03-30T13:50:32.413 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71c7d89a:::benchmark_data_smithi043_119832_object5923:head# 2024-03-30T13:50:32.413 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71c822ed:::benchmark_data_smithi043_119832_object23718:head# 2024-03-30T13:50:32.414 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71c84a40:::benchmark_data_smithi043_119832_object27592:head# 2024-03-30T13:50:32.414 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71ca13ed:::benchmark_data_smithi043_119832_object58093:head# 2024-03-30T13:50:32.415 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71cadec2:::benchmark_data_smithi043_119832_object28165:head# 2024-03-30T13:50:32.415 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71cb4882:::benchmark_data_smithi043_119832_object23702:head# 2024-03-30T13:50:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71cd97f1:::benchmark_data_smithi043_119832_object32835:head# 2024-03-30T13:50:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71ce073c:::benchmark_data_smithi043_119832_object38759:head# 2024-03-30T13:50:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71ce6b30:::benchmark_data_smithi043_119832_object84451:head# 2024-03-30T13:50:32.420 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71cf1146:::benchmark_data_smithi043_119832_object70640:head# 2024-03-30T13:50:32.420 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71d0920d:::benchmark_data_smithi043_119832_object87510:head# 2024-03-30T13:50:32.420 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71d21565:::benchmark_data_smithi043_119832_object10632:head# 2024-03-30T13:50:32.421 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71d27b7f:::benchmark_data_smithi043_119832_object63118:head# 2024-03-30T13:50:32.421 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71d2d524:::benchmark_data_smithi043_119832_object865:head# 2024-03-30T13:50:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71d30a4f:::benchmark_data_smithi043_119832_object50932:head# 2024-03-30T13:50:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71d3e665:::benchmark_data_smithi043_119832_object753:head# 2024-03-30T13:50:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71d45f0b:::benchmark_data_smithi043_119832_object73851:head# 2024-03-30T13:50:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71d50bc7:::benchmark_data_smithi043_119832_object32118:head# 2024-03-30T13:50:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71d578b2:::benchmark_data_smithi043_119832_object21889:head# 2024-03-30T13:50:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71d59c5c:::benchmark_data_smithi043_119832_object78740:head# 2024-03-30T13:50:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71d5a9d4:::benchmark_data_smithi043_119832_object67465:head# 2024-03-30T13:50:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71d6fc8c:::benchmark_data_smithi043_119832_object29094:head# 2024-03-30T13:50:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71d70688:::benchmark_data_smithi043_119832_object23572:head# 2024-03-30T13:50:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71d72e2c:::benchmark_data_smithi043_119832_object20670:head# 2024-03-30T13:50:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71d7e527:::benchmark_data_smithi043_119832_object70711:head# 2024-03-30T13:50:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71d8566b:::benchmark_data_smithi043_119832_object5744:head# 2024-03-30T13:50:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71d9b77a:::benchmark_data_smithi043_119832_object3074:head# 2024-03-30T13:50:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71db2b84:::benchmark_data_smithi043_119832_object51989:head# 2024-03-30T13:50:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71dc4d52:::benchmark_data_smithi043_119832_object58422:head# 2024-03-30T13:50:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71dc966a:::benchmark_data_smithi043_119832_object18201:head# 2024-03-30T13:50:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71dc9f91:::benchmark_data_smithi043_119832_object2122:head# 2024-03-30T13:50:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71ddb5f7:::benchmark_data_smithi043_119832_object20309:head# 2024-03-30T13:50:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71de09c4:::benchmark_data_smithi043_119832_object42982:head# 2024-03-30T13:50:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71de312a:::benchmark_data_smithi043_119832_object80184:head# 2024-03-30T13:50:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71df6165:::benchmark_data_smithi043_119832_object15884:head# 2024-03-30T13:50:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71e041b6:::benchmark_data_smithi043_119832_object70593:head# 2024-03-30T13:50:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71e16739:::benchmark_data_smithi043_119832_object46627:head# 2024-03-30T13:50:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71e3bde5:::benchmark_data_smithi043_119832_object530:head# 2024-03-30T13:50:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71e444e4:::benchmark_data_smithi043_119832_object79150:head# 2024-03-30T13:50:32.434 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71e4e03e:::benchmark_data_smithi043_119832_object36077:head# 2024-03-30T13:50:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71e4e3b3:::benchmark_data_smithi043_119832_object21826:head# 2024-03-30T13:50:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71e5bb80:::benchmark_data_smithi043_119832_object21456:head# 2024-03-30T13:50:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71e64fa5:::benchmark_data_smithi043_119832_object85134:head# 2024-03-30T13:50:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71e993f5:::benchmark_data_smithi043_119832_object14386:head# 2024-03-30T13:50:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71e9e05d:::benchmark_data_smithi043_119832_object52587:head# 2024-03-30T13:50:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71ea3f9a:::benchmark_data_smithi043_119832_object9973:head# 2024-03-30T13:50:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71ea68cc:::benchmark_data_smithi043_119832_object886:head# 2024-03-30T13:50:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71eaaa0e:::benchmark_data_smithi043_119832_object78207:head# 2024-03-30T13:50:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71ecae5e:::benchmark_data_smithi043_119832_object86961:head# 2024-03-30T13:50:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71edf2ce:::benchmark_data_smithi043_119832_object49915:head# 2024-03-30T13:50:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71ee9c5f:::benchmark_data_smithi043_119832_object25743:head# 2024-03-30T13:50:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71efe9c6:::benchmark_data_smithi043_119832_object60793:head# 2024-03-30T13:50:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71f158c2:::benchmark_data_smithi043_119832_object32572:head# 2024-03-30T13:50:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71f211b2:::benchmark_data_smithi043_119832_object83645:head# 2024-03-30T13:50:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71f2b6a5:::benchmark_data_smithi043_119832_object52360:head# 2024-03-30T13:50:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71f35739:::benchmark_data_smithi043_119832_object6731:head# 2024-03-30T13:50:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71f3c87d:::benchmark_data_smithi043_119832_object30045:head# 2024-03-30T13:50:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71f45683:::benchmark_data_smithi043_119832_object34437:head# 2024-03-30T13:50:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71f6dbad:::benchmark_data_smithi043_119832_object53719:head# 2024-03-30T13:50:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71f77e0a:::benchmark_data_smithi043_119832_object35042:head# 2024-03-30T13:50:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71f88fc5:::benchmark_data_smithi043_119832_object52038:head# 2024-03-30T13:50:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71fa9c20:::benchmark_data_smithi043_119832_object47374:head# 2024-03-30T13:50:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71fb8a81:::benchmark_data_smithi043_119832_object70576:head# 2024-03-30T13:50:32.443 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71fd279b:::benchmark_data_smithi043_119832_object15629:head# 2024-03-30T13:50:32.443 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:71ff2e8d:::benchmark_data_smithi043_119832_object29451:head# 2024-03-30T13:50:32.443 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7200937a:::benchmark_data_smithi043_119832_object16115:head# 2024-03-30T13:50:32.443 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7200a944:::benchmark_data_smithi043_119832_object60893:head# 2024-03-30T13:50:32.444 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7200d813:::benchmark_data_smithi043_119832_object45356:head# 2024-03-30T13:50:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7200d822:::benchmark_data_smithi043_119832_object18720:head# 2024-03-30T13:50:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72017289:::benchmark_data_smithi043_119832_object12588:head# 2024-03-30T13:50:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7201cdd1:::benchmark_data_smithi043_119832_object80585:head# 2024-03-30T13:50:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72030284:::benchmark_data_smithi043_119832_object19844:head# 2024-03-30T13:50:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:720312fe:::benchmark_data_smithi043_119832_object11432:head# 2024-03-30T13:50:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:720400c6:::benchmark_data_smithi043_119832_object77672:head# 2024-03-30T13:50:32.447 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72044203:::benchmark_data_smithi043_119832_object5273:head# 2024-03-30T13:50:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7204671b:::benchmark_data_smithi043_119832_object43218:head# 2024-03-30T13:50:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7204be6e:::benchmark_data_smithi043_119832_object59469:head# 2024-03-30T13:50:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7204f9e1:::benchmark_data_smithi043_119832_object72043:head# 2024-03-30T13:50:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72060532:::benchmark_data_smithi043_119832_object41928:head# 2024-03-30T13:50:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72079ed3:::benchmark_data_smithi043_119832_object51386:head# 2024-03-30T13:50:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72082531:::benchmark_data_smithi043_119832_object68997:head# 2024-03-30T13:50:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72085f6e:::benchmark_data_smithi043_119832_object50947:head# 2024-03-30T13:50:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7208ef75:::benchmark_data_smithi043_119832_object13124:head# 2024-03-30T13:50:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:720a1308:::benchmark_data_smithi043_119832_object49851:head# 2024-03-30T13:50:32.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:32.477 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:32.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:32.479 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:720a6a46:::benchmark_data_smithi043_119832_object28074:head# 2024-03-30T13:50:32.479 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:720cc65d:::benchmark_data_smithi043_119832_object52024:head# 2024-03-30T13:50:32.479 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:720dd06b:::benchmark_data_smithi043_119832_object66597:head# 2024-03-30T13:50:32.479 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:720e4c24:::benchmark_data_smithi043_119832_object933:head# 2024-03-30T13:50:32.479 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:720e8c05:::benchmark_data_smithi043_119832_object20648:head# 2024-03-30T13:50:32.479 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:720eba39:::benchmark_data_smithi043_119832_object87502:head# 2024-03-30T13:50:32.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:720f470e:::benchmark_data_smithi043_119832_object68563:head# 2024-03-30T13:50:32.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:721170ff:::benchmark_data_smithi043_119832_object82701:head# 2024-03-30T13:50:32.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:721203c5:::benchmark_data_smithi043_119832_object3689:head# 2024-03-30T13:50:32.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72126193:::benchmark_data_smithi043_119832_object24245:head# 2024-03-30T13:50:32.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72127ac7:::benchmark_data_smithi043_119832_object6754:head# 2024-03-30T13:50:32.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72141339:::benchmark_data_smithi043_119832_object3679:head# 2024-03-30T13:50:32.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7214a646:::benchmark_data_smithi043_119832_object14488:head# 2024-03-30T13:50:32.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72151981:::benchmark_data_smithi043_119832_object53428:head# 2024-03-30T13:50:32.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7215c492:::benchmark_data_smithi043_119832_object48863:head# 2024-03-30T13:50:32.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72175647:::benchmark_data_smithi043_119832_object48461:head# 2024-03-30T13:50:32.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7217844e:::benchmark_data_smithi043_119832_object53688:head# 2024-03-30T13:50:32.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7217b34d:::benchmark_data_smithi043_119832_object73229:head# 2024-03-30T13:50:32.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7217c34b:::benchmark_data_smithi043_119832_object81875:head# 2024-03-30T13:50:32.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7217fcde:::benchmark_data_smithi043_119832_object1099:head# 2024-03-30T13:50:32.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7219468a:::benchmark_data_smithi043_119832_object59765:head# 2024-03-30T13:50:32.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:721962af:::benchmark_data_smithi043_119832_object2070:head# 2024-03-30T13:50:32.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7219ad9e:::benchmark_data_smithi043_119832_object70423:head# 2024-03-30T13:50:32.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:721b8c32:::benchmark_data_smithi043_119832_object23736:head# 2024-03-30T13:50:32.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:721d0162:::benchmark_data_smithi043_119832_object34752:head# 2024-03-30T13:50:32.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:721d76e6:::benchmark_data_smithi043_119832_object7541:head# 2024-03-30T13:50:32.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:721df83b:::benchmark_data_smithi043_119832_object42980:head# 2024-03-30T13:50:32.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:721e7cda:::benchmark_data_smithi043_119832_object68863:head# 2024-03-30T13:50:32.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72203385:::benchmark_data_smithi043_119832_object25474:head# 2024-03-30T13:50:32.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:722039e3:::benchmark_data_smithi043_119832_object24172:head# 2024-03-30T13:50:32.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72215472:::benchmark_data_smithi043_119832_object21631:head# 2024-03-30T13:50:32.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72218d54:::benchmark_data_smithi043_119832_object34174:head# 2024-03-30T13:50:32.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7222c4f2:::benchmark_data_smithi043_119832_object4707:head# 2024-03-30T13:50:32.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72231673:::benchmark_data_smithi043_119832_object1021:head# 2024-03-30T13:50:32.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72232628:::benchmark_data_smithi043_119832_object80731:head# 2024-03-30T13:50:32.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72232ff7:::benchmark_data_smithi043_119832_object3680:head# 2024-03-30T13:50:32.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7224e5a7:::benchmark_data_smithi043_119832_object7231:head# 2024-03-30T13:50:32.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7226c900:::benchmark_data_smithi043_119832_object19703:head# 2024-03-30T13:50:32.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7226e04e:::benchmark_data_smithi043_119832_object20013:head# 2024-03-30T13:50:32.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72272e92:::benchmark_data_smithi043_119832_object6651:head# 2024-03-30T13:50:32.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72293555:::benchmark_data_smithi043_119832_object79664:head# 2024-03-30T13:50:32.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72294c66:::benchmark_data_smithi043_119832_object30151:head# 2024-03-30T13:50:32.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72297bb6:::benchmark_data_smithi043_119832_object6254:head# 2024-03-30T13:50:32.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:722a9a82:::benchmark_data_smithi043_119832_object15584:head# 2024-03-30T13:50:32.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:722ab2df:::benchmark_data_smithi043_119832_object35310:head# 2024-03-30T13:50:32.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:722b2d2d:::benchmark_data_smithi043_119832_object21772:head# 2024-03-30T13:50:32.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:722b4737:::benchmark_data_smithi043_119832_object60377:head# 2024-03-30T13:50:32.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:722b74c5:::benchmark_data_smithi043_119832_object5764:head# 2024-03-30T13:50:32.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:722bda2c:::benchmark_data_smithi043_119832_object18737:head# 2024-03-30T13:50:32.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:722c94c3:::benchmark_data_smithi043_119832_object46007:head# 2024-03-30T13:50:32.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:722d4758:::benchmark_data_smithi043_119832_object52524:head# 2024-03-30T13:50:32.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:722d4995:::benchmark_data_smithi043_119832_object76856:head# 2024-03-30T13:50:32.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:722f3420:::benchmark_data_smithi043_119832_object28332:head# 2024-03-30T13:50:32.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:722fd433:::benchmark_data_smithi043_119832_object86078:head# 2024-03-30T13:50:32.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:722fe2c2:::benchmark_data_smithi043_119832_object34614:head# 2024-03-30T13:50:32.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:722fe81e:::benchmark_data_smithi043_119832_object67022:head# 2024-03-30T13:50:32.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72309fab:::benchmark_data_smithi043_119832_object37170:head# 2024-03-30T13:50:32.484 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7232ad28:::benchmark_data_smithi043_119832_object83604:head# 2024-03-30T13:50:32.484 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7234065a:::benchmark_data_smithi043_119832_object79988:head# 2024-03-30T13:50:32.484 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72356f1b:::benchmark_data_smithi043_119832_object60131:head# 2024-03-30T13:50:32.484 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72366587:::benchmark_data_smithi043_119832_object22716:head# 2024-03-30T13:50:32.484 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72367af7:::benchmark_data_smithi043_119832_object59289:head# 2024-03-30T13:50:32.484 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7237646e:::benchmark_data_smithi043_119832_object12980:head# 2024-03-30T13:50:32.484 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72381a32:::benchmark_data_smithi043_119832_object35709:head# 2024-03-30T13:50:32.484 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72385369:::benchmark_data_smithi043_119832_object9666:head# 2024-03-30T13:50:32.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72385a2a:::benchmark_data_smithi043_119832_object71361:head# 2024-03-30T13:50:32.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:723869f8:::benchmark_data_smithi043_119832_object57279:head# 2024-03-30T13:50:32.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7238900f:::benchmark_data_smithi043_119832_object47463:head# 2024-03-30T13:50:32.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:723cd8d8:::benchmark_data_smithi043_119832_object5419:head# 2024-03-30T13:50:32.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:723d3866:::benchmark_data_smithi043_119832_object80061:head# 2024-03-30T13:50:32.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:723db22d:::benchmark_data_smithi043_119832_object86253:head# 2024-03-30T13:50:32.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:723df91c:::benchmark_data_smithi043_119832_object67799:head# 2024-03-30T13:50:32.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:723e60ac:::benchmark_data_smithi043_119832_object37779:head# 2024-03-30T13:50:32.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:723ee32a:::benchmark_data_smithi043_119832_object24342:head# 2024-03-30T13:50:32.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:723f64c7:::benchmark_data_smithi043_119832_object7418:head# 2024-03-30T13:50:32.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:723fed10:::benchmark_data_smithi043_119832_object88139:head# 2024-03-30T13:50:32.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7240182e:::benchmark_data_smithi043_119832_object39273:head# 2024-03-30T13:50:32.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7241d437:::benchmark_data_smithi043_119832_object37406:head# 2024-03-30T13:50:32.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7242efa9:::benchmark_data_smithi043_119832_object62666:head# 2024-03-30T13:50:32.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72431466:::benchmark_data_smithi043_119832_object44975:head# 2024-03-30T13:50:32.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:724321b8:::benchmark_data_smithi043_119832_object20109:head# 2024-03-30T13:50:32.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7245247a:::benchmark_data_smithi043_119832_object79368:head# 2024-03-30T13:50:32.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72453a87:::benchmark_data_smithi043_119832_object57254:head# 2024-03-30T13:50:32.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7245bfdc:::benchmark_data_smithi043_119832_object43705:head# 2024-03-30T13:50:32.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7247dc1b:::benchmark_data_smithi043_119832_object63395:head# 2024-03-30T13:50:32.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72480c59:::benchmark_data_smithi043_119832_object61012:head# 2024-03-30T13:50:32.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:724a29c5:::benchmark_data_smithi043_119832_object22884:head# 2024-03-30T13:50:32.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:724ad046:::benchmark_data_smithi043_119832_object34071:head# 2024-03-30T13:50:32.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:724c7cee:::benchmark_data_smithi043_119832_object19959:head# 2024-03-30T13:50:32.495 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:724c9d63:::benchmark_data_smithi043_119832_object71675:head# 2024-03-30T13:50:32.495 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7250994e:::benchmark_data_smithi043_119832_object57193:head# 2024-03-30T13:50:32.495 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7250adbe:::benchmark_data_smithi043_119832_object18151:head# 2024-03-30T13:50:32.495 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:725168a1:::benchmark_data_smithi043_119832_object72455:head# 2024-03-30T13:50:32.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:725168e3:::benchmark_data_smithi043_119832_object9586:head# 2024-03-30T13:50:32.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72519f00:::benchmark_data_smithi043_119832_object74789:head# 2024-03-30T13:50:32.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72530fc8:::benchmark_data_smithi043_119832_object64545:head# 2024-03-30T13:50:32.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72537238:::benchmark_data_smithi043_119832_object16034:head# 2024-03-30T13:50:32.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72543727:::benchmark_data_smithi043_119832_object4625:head# 2024-03-30T13:50:32.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7254c880:::benchmark_data_smithi043_119832_object51744:head# 2024-03-30T13:50:32.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72552cd5:::benchmark_data_smithi043_119832_object49566:head# 2024-03-30T13:50:32.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72554bf6:::benchmark_data_smithi043_119832_object52100:head# 2024-03-30T13:50:32.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72559acf:::benchmark_data_smithi043_119832_object39027:head# 2024-03-30T13:50:32.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7255b73d:::benchmark_data_smithi043_119832_object55496:head# 2024-03-30T13:50:32.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72567902:::benchmark_data_smithi043_119832_object24032:head# 2024-03-30T13:50:32.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7256c12e:::benchmark_data_smithi043_119832_object14655:head# 2024-03-30T13:50:32.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72574052:::benchmark_data_smithi043_119832_object12000:head# 2024-03-30T13:50:32.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7257b59a:::benchmark_data_smithi043_119832_object15972:head# 2024-03-30T13:50:32.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7259390d:::benchmark_data_smithi043_119832_object44012:head# 2024-03-30T13:50:32.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:725b5c21:::benchmark_data_smithi043_119832_object76910:head# 2024-03-30T13:50:32.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:725ba407:::benchmark_data_smithi043_119832_object37050:head# 2024-03-30T13:50:32.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:725c33c8:::benchmark_data_smithi043_119832_object39061:head# 2024-03-30T13:50:32.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:725ded6c:::benchmark_data_smithi043_119832_object16105:head# 2024-03-30T13:50:32.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7260bd22:::benchmark_data_smithi043_119832_object29265:head# 2024-03-30T13:50:32.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7260f77f:::benchmark_data_smithi043_119832_object52514:head# 2024-03-30T13:50:32.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72618de4:::benchmark_data_smithi043_119832_object50607:head# 2024-03-30T13:50:32.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:726585c8:::benchmark_data_smithi043_119832_object88194:head# 2024-03-30T13:50:32.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7265eb20:::benchmark_data_smithi043_119832_object69751:head# 2024-03-30T13:50:32.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7266f19b:::benchmark_data_smithi043_119832_object87733:head# 2024-03-30T13:50:32.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7267860b:::benchmark_data_smithi043_119832_object450:head# 2024-03-30T13:50:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7267d4a5:::benchmark_data_smithi043_119832_object10176:head# 2024-03-30T13:50:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7267d9cb:::benchmark_data_smithi043_119832_object23316:head# 2024-03-30T13:50:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72693192:::benchmark_data_smithi043_119832_object88454:head# 2024-03-30T13:50:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72696e22:::benchmark_data_smithi043_119832_object84887:head# 2024-03-30T13:50:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72698415:::benchmark_data_smithi043_119832_object74843:head# 2024-03-30T13:50:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:726adbdd:::benchmark_data_smithi043_119832_object33990:head# 2024-03-30T13:50:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:726b8d40:::benchmark_data_smithi043_119832_object85147:head# 2024-03-30T13:50:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:726dd3eb:::benchmark_data_smithi043_119832_object8957:head# 2024-03-30T13:50:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:726e2663:::benchmark_data_smithi043_119832_object46554:head# 2024-03-30T13:50:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:726e2837:::benchmark_data_smithi043_119832_object20426:head# 2024-03-30T13:50:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:726fb125:::benchmark_data_smithi043_119832_object47098:head# 2024-03-30T13:50:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72701077:::benchmark_data_smithi043_119832_object33070:head# 2024-03-30T13:50:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72702fca:::benchmark_data_smithi043_119832_object77220:head# 2024-03-30T13:50:32.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72709a56:::benchmark_data_smithi043_119832_object87639:head# 2024-03-30T13:50:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7271171c:::benchmark_data_smithi043_119832_object60317:head# 2024-03-30T13:50:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7275231b:::benchmark_data_smithi043_119832_object75821:head# 2024-03-30T13:50:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72756421:::benchmark_data_smithi043_119832_object44834:head# 2024-03-30T13:50:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7275ed0e:::benchmark_data_smithi043_119832_object13337:head# 2024-03-30T13:50:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7275f3c3:::benchmark_data_smithi043_119832_object78009:head# 2024-03-30T13:50:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7276b1b2:::benchmark_data_smithi043_119832_object21730:head# 2024-03-30T13:50:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7276dde4:::benchmark_data_smithi043_119832_object33940:head# 2024-03-30T13:50:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:727718d4:::benchmark_data_smithi043_119832_object74972:head# 2024-03-30T13:50:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72772b26:::benchmark_data_smithi043_119832_object40577:head# 2024-03-30T13:50:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7278c2d2:::benchmark_data_smithi043_119832_object53071:head# 2024-03-30T13:50:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7278e248:::benchmark_data_smithi043_119832_object47885:head# 2024-03-30T13:50:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7279b028:::benchmark_data_smithi043_119832_object32335:head# 2024-03-30T13:50:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:727a1e63:::benchmark_data_smithi043_119832_object48699:head# 2024-03-30T13:50:32.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:727aa488:::benchmark_data_smithi043_119832_object85426:head# 2024-03-30T13:50:32.519 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:727ad602:::benchmark_data_smithi043_119832_object30373:head# 2024-03-30T13:50:32.519 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:727b73b5:::benchmark_data_smithi043_119832_object9750:head# 2024-03-30T13:50:32.519 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:727c40c6:::benchmark_data_smithi043_119832_object5532:head# 2024-03-30T13:50:32.531 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:727f2f8b:::benchmark_data_smithi043_119832_object15517:head# 2024-03-30T13:50:32.531 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:727fa240:::benchmark_data_smithi043_119832_object73913:head# 2024-03-30T13:50:32.531 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:728026e5:::benchmark_data_smithi043_119832_object23285:head# 2024-03-30T13:50:32.531 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7282c8a0:::benchmark_data_smithi043_119832_object28452:head# 2024-03-30T13:50:32.531 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72830666:::benchmark_data_smithi043_119832_object35674:head# 2024-03-30T13:50:32.531 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:728348aa:::benchmark_data_smithi043_119832_object64765:head# 2024-03-30T13:50:32.531 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7283bc61:::benchmark_data_smithi043_119832_object1145:head# 2024-03-30T13:50:32.532 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7284d6fb:::benchmark_data_smithi043_119832_object50253:head# 2024-03-30T13:50:32.532 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7284e1cb:::benchmark_data_smithi043_119832_object83014:head# 2024-03-30T13:50:32.532 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72851760:::benchmark_data_smithi043_119832_object51972:head# 2024-03-30T13:50:32.532 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:728678e7:::benchmark_data_smithi043_119832_object82995:head# 2024-03-30T13:50:32.532 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7287059f:::benchmark_data_smithi043_119832_object33978:head# 2024-03-30T13:50:32.532 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7287c19d:::benchmark_data_smithi043_119832_object10981:head# 2024-03-30T13:50:32.532 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7287fde6:::benchmark_data_smithi043_119832_object6893:head# 2024-03-30T13:50:32.532 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7289442f:::benchmark_data_smithi043_119832_object30829:head# 2024-03-30T13:50:32.532 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:728a11db:::benchmark_data_smithi043_119832_object78449:head# 2024-03-30T13:50:32.532 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:728a5977:::benchmark_data_smithi043_119832_object77998:head# 2024-03-30T13:50:32.532 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:728a7d1d:::benchmark_data_smithi043_119832_object17063:head# 2024-03-30T13:50:32.533 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:728bc4f0:::benchmark_data_smithi043_119832_object54543:head# 2024-03-30T13:50:32.533 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:728c0211:::benchmark_data_smithi043_119832_object19360:head# 2024-03-30T13:50:32.533 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:728c1843:::benchmark_data_smithi043_119832_object53458:head# 2024-03-30T13:50:32.533 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:728d3445:::benchmark_data_smithi043_119832_object19207:head# 2024-03-30T13:50:32.533 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:728d7fac:::benchmark_data_smithi043_119832_object59856:head# 2024-03-30T13:50:32.533 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:728e19b5:::benchmark_data_smithi043_119832_object83310:head# 2024-03-30T13:50:32.533 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:728e6625:::benchmark_data_smithi043_119832_object6658:head# 2024-03-30T13:50:32.533 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:728e90f9:::benchmark_data_smithi043_119832_object19417:head# 2024-03-30T13:50:32.533 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:728ea2c4:::benchmark_data_smithi043_119832_object48738:head# 2024-03-30T13:50:32.534 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:728f5f1b:::benchmark_data_smithi043_119832_object58025:head# 2024-03-30T13:50:32.534 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:729010db:::benchmark_data_smithi043_119832_object7788:head# 2024-03-30T13:50:32.534 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72912d25:::benchmark_data_smithi043_119832_object3997:head# 2024-03-30T13:50:32.534 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72938817:::benchmark_data_smithi043_119832_object87879:head# 2024-03-30T13:50:32.534 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72951016:::benchmark_data_smithi043_119832_object71259:head# 2024-03-30T13:50:32.534 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72957455:::benchmark_data_smithi043_119832_object40805:head# 2024-03-30T13:50:32.534 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7295e2d0:::benchmark_data_smithi043_119832_object50751:head# 2024-03-30T13:50:32.534 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72965cd7:::benchmark_data_smithi043_119832_object84603:head# 2024-03-30T13:50:32.541 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7296b66b:::benchmark_data_smithi043_119832_object57326:head# 2024-03-30T13:50:32.541 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72975e3c:::benchmark_data_smithi043_119832_object6298:head# 2024-03-30T13:50:32.541 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7297c121:::benchmark_data_smithi043_119832_object88150:head# 2024-03-30T13:50:32.541 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:729874d7:::benchmark_data_smithi043_119832_object80317:head# 2024-03-30T13:50:32.541 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7298b442:::benchmark_data_smithi043_119832_object45011:head# 2024-03-30T13:50:32.541 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7298b588:::benchmark_data_smithi043_119832_object56846:head# 2024-03-30T13:50:32.541 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:729a0068:::benchmark_data_smithi043_119832_object31798:head# 2024-03-30T13:50:32.541 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:729a1a79:::benchmark_data_smithi043_119832_object21973:head# 2024-03-30T13:50:32.541 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:729b3e46:::benchmark_data_smithi043_119832_object72978:head# 2024-03-30T13:50:32.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:729bcf2a:::benchmark_data_smithi043_119832_object87725:head# 2024-03-30T13:50:32.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:729ddca0:::benchmark_data_smithi043_119832_object41112:head# 2024-03-30T13:50:32.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:729ec102:::benchmark_data_smithi043_119832_object22645:head# 2024-03-30T13:50:32.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:729f3736:::benchmark_data_smithi043_119832_object6528:head# 2024-03-30T13:50:32.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:729f8b8f:::benchmark_data_smithi043_119832_object81249:head# 2024-03-30T13:50:32.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:729ff89d:::benchmark_data_smithi043_119832_object27351:head# 2024-03-30T13:50:32.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72a04d8e:::benchmark_data_smithi043_119832_object54247:head# 2024-03-30T13:50:32.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72a1a27d:::benchmark_data_smithi043_119832_object21087:head# 2024-03-30T13:50:32.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72a24430:::benchmark_data_smithi043_119832_object8471:head# 2024-03-30T13:50:32.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72a2b0d5:::benchmark_data_smithi043_119832_object71896:head# 2024-03-30T13:50:32.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72a363cc:::benchmark_data_smithi043_119832_object22178:head# 2024-03-30T13:50:32.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72a44876:::benchmark_data_smithi043_119832_object29038:head# 2024-03-30T13:50:32.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72a491e8:::benchmark_data_smithi043_119832_object65287:head# 2024-03-30T13:50:32.546 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72a4eb5e:::benchmark_data_smithi043_119832_object57259:head# 2024-03-30T13:50:32.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72a7535e:::benchmark_data_smithi043_119832_object70274:head# 2024-03-30T13:50:32.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72a7fe0f:::benchmark_data_smithi043_119832_object6988:head# 2024-03-30T13:50:32.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72a8d292:::benchmark_data_smithi043_119832_object65996:head# 2024-03-30T13:50:32.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72aa0768:::benchmark_data_smithi043_119832_object28466:head# 2024-03-30T13:50:32.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72aa762e:::benchmark_data_smithi043_119832_object42507:head# 2024-03-30T13:50:32.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72acdf35:::benchmark_data_smithi043_119832_object70856:head# 2024-03-30T13:50:32.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72ad8721:::benchmark_data_smithi043_119832_object8884:head# 2024-03-30T13:50:32.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72adfcfb:::benchmark_data_smithi043_119832_object9834:head# 2024-03-30T13:50:32.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72af6604:::benchmark_data_smithi043_119832_object37653:head# 2024-03-30T13:50:32.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72b243b2:::benchmark_data_smithi043_119832_object70575:head# 2024-03-30T13:50:32.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72b35de5:::benchmark_data_smithi043_119832_object55461:head# 2024-03-30T13:50:32.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72b412d7:::benchmark_data_smithi043_119832_object45954:head# 2024-03-30T13:50:32.550 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72b4a0a0:::benchmark_data_smithi043_119832_object42614:head# 2024-03-30T13:50:32.550 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72b4e71b:::benchmark_data_smithi043_119832_object45556:head# 2024-03-30T13:50:32.550 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72b5123c:::benchmark_data_smithi043_119832_object73144:head# 2024-03-30T13:50:32.550 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72b5d1f4:::benchmark_data_smithi043_119832_object37002:head# 2024-03-30T13:50:32.550 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72b61d2e:::benchmark_data_smithi043_119832_object57689:head# 2024-03-30T13:50:32.550 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72b6dc05:::benchmark_data_smithi043_119832_object42512:head# 2024-03-30T13:50:32.551 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72b724c0:::benchmark_data_smithi043_119832_object32669:head# 2024-03-30T13:50:32.551 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72b9e383:::benchmark_data_smithi043_119832_object74156:head# 2024-03-30T13:50:32.553 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72b9f283:::benchmark_data_smithi043_119832_object76165:head# 2024-03-30T13:50:32.553 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72ba2427:::benchmark_data_smithi043_119832_object81526:head# 2024-03-30T13:50:32.553 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72bc49ad:::benchmark_data_smithi043_119832_object55573:head# 2024-03-30T13:50:32.553 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72bccb1c:::benchmark_data_smithi043_119832_object32438:head# 2024-03-30T13:50:32.553 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72bce37b:::benchmark_data_smithi043_119832_object43680:head# 2024-03-30T13:50:32.553 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72bde24a:::benchmark_data_smithi043_119832_object36238:head# 2024-03-30T13:50:32.554 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72beec6d:::benchmark_data_smithi043_119832_object64961:head# 2024-03-30T13:50:32.559 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72bfef37:::benchmark_data_smithi043_119832_object36889:head# 2024-03-30T13:50:32.559 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72c0b4c1:::benchmark_data_smithi043_119832_object36404:head# 2024-03-30T13:50:32.559 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72c15537:::benchmark_data_smithi043_119832_object47774:head# 2024-03-30T13:50:32.559 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72c39afa:::benchmark_data_smithi043_119832_object16395:head# 2024-03-30T13:50:32.559 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72c3e4dc:::benchmark_data_smithi043_119832_object79822:head# 2024-03-30T13:50:32.559 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72c4b36a:::benchmark_data_smithi043_119832_object87268:head# 2024-03-30T13:50:32.560 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72c501a0:::benchmark_data_smithi043_119832_object41558:head# 2024-03-30T13:50:32.560 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72c54ad8:::benchmark_data_smithi043_119832_object43906:head# 2024-03-30T13:50:32.560 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72c6c8f5:::benchmark_data_smithi043_119832_object19855:head# 2024-03-30T13:50:32.560 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72c7c227:::benchmark_data_smithi043_119832_object2219:head# 2024-03-30T13:50:32.560 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72ca8bca:::benchmark_data_smithi043_119832_object41173:head# 2024-03-30T13:50:32.560 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72cab48a:::benchmark_data_smithi043_119832_object29511:head# 2024-03-30T13:50:32.560 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72cb538d:::benchmark_data_smithi043_119832_object37738:head# 2024-03-30T13:50:32.569 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72cb593f:::benchmark_data_smithi043_119832_object87860:head# 2024-03-30T13:50:32.569 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72cb8237:::benchmark_data_smithi043_119832_object53218:head# 2024-03-30T13:50:32.569 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72cbed10:::benchmark_data_smithi043_119832_object11336:head# 2024-03-30T13:50:32.569 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72cecee6:::benchmark_data_smithi043_119832_object17794:head# 2024-03-30T13:50:32.569 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72cf4473:::benchmark_data_smithi043_119832_object12447:head# 2024-03-30T13:50:32.570 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72d010c1:::benchmark_data_smithi043_119832_object84816:head# 2024-03-30T13:50:32.570 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72d05282:::benchmark_data_smithi043_119832_object7246:head# 2024-03-30T13:50:32.570 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72d21e13:::benchmark_data_smithi043_119832_object78905:head# 2024-03-30T13:50:32.570 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72d381a7:::benchmark_data_smithi043_119832_object52254:head# 2024-03-30T13:50:32.570 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72d4f1e6:::benchmark_data_smithi043_119832_object50638:head# 2024-03-30T13:50:32.570 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72d55263:::benchmark_data_smithi043_119832_object10801:head# 2024-03-30T13:50:32.570 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72d72374:::benchmark_data_smithi043_119832_object68494:head# 2024-03-30T13:50:32.570 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72d7c5a6:::benchmark_data_smithi043_119832_object39864:head# 2024-03-30T13:50:32.570 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72d95452:::benchmark_data_smithi043_119832_object2012:head# 2024-03-30T13:50:32.571 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72db9bb9:::benchmark_data_smithi043_119832_object34242:head# 2024-03-30T13:50:32.571 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72dbd13e:::benchmark_data_smithi043_119832_object60091:head# 2024-03-30T13:50:32.571 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72dbf928:::benchmark_data_smithi043_119832_object8163:head# 2024-03-30T13:50:32.571 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72deda5c:::benchmark_data_smithi043_119832_object83450:head# 2024-03-30T13:50:32.571 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72dee5cc:::benchmark_data_smithi043_119832_object2495:head# 2024-03-30T13:50:32.571 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72df1ed9:::benchmark_data_smithi043_119832_object64894:head# 2024-03-30T13:50:32.571 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72dfd4fd:::benchmark_data_smithi043_119832_object84964:head# 2024-03-30T13:50:32.571 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72e22954:::benchmark_data_smithi043_119832_object54683:head# 2024-03-30T13:50:32.571 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72e490ee:::benchmark_data_smithi043_119832_object27641:head# 2024-03-30T13:50:32.583 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:32.592 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72e6958b:::benchmark_data_smithi043_119832_object22067:head# 2024-03-30T13:50:32.592 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72e6b0d9:::benchmark_data_smithi043_119832_object71117:head# 2024-03-30T13:50:32.592 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72e70699:::benchmark_data_smithi043_119832_object15798:head# 2024-03-30T13:50:32.592 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72e71b6e:::benchmark_data_smithi043_119832_object24471:head# 2024-03-30T13:50:32.593 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72e7c062:::benchmark_data_smithi043_119832_object43823:head# 2024-03-30T13:50:32.593 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72e867e4:::benchmark_data_smithi043_119832_object45149:head# 2024-03-30T13:50:32.593 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72e891c9:::benchmark_data_smithi043_119832_object28278:head# 2024-03-30T13:50:32.593 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72ea23a6:::benchmark_data_smithi043_119832_object42948:head# 2024-03-30T13:50:32.593 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72eace4f:::benchmark_data_smithi043_119832_object64007:head# 2024-03-30T13:50:32.593 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72eb1443:::benchmark_data_smithi043_119832_object76172:head# 2024-03-30T13:50:32.593 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72ec8d5f:::benchmark_data_smithi043_119832_object65241:head# 2024-03-30T13:50:32.593 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72eca005:::benchmark_data_smithi043_119832_object26356:head# 2024-03-30T13:50:32.593 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72ed6129:::benchmark_data_smithi043_119832_object20627:head# 2024-03-30T13:50:32.593 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72ed671a:::benchmark_data_smithi043_119832_object17725:head# 2024-03-30T13:50:32.593 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72ed77d8:::benchmark_data_smithi043_119832_object26054:head# 2024-03-30T13:50:32.593 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72ef1ef7:::benchmark_data_smithi043_119832_object75792:head# 2024-03-30T13:50:32.593 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72f3cd4d:::benchmark_data_smithi043_119832_object74465:head# 2024-03-30T13:50:32.594 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72f4649d:::benchmark_data_smithi043_119832_object69791:head# 2024-03-30T13:50:32.594 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72f54125:::benchmark_data_smithi043_119832_object82631:head# 2024-03-30T13:50:32.594 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72f54dac:::benchmark_data_smithi043_119832_object40118:head# 2024-03-30T13:50:32.594 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72f5695d:::benchmark_data_smithi043_119832_object24127:head# 2024-03-30T13:50:32.594 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72f66ee2:::benchmark_data_smithi043_119832_object6518:head# 2024-03-30T13:50:32.594 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72f723f8:::benchmark_data_smithi043_119832_object25268:head# 2024-03-30T13:50:32.594 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72f747ef:::benchmark_data_smithi043_119832_object71385:head# 2024-03-30T13:50:32.594 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72f7bb54:::benchmark_data_smithi043_119832_object87086:head# 2024-03-30T13:50:32.594 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72f7c5a1:::benchmark_data_smithi043_119832_object78418:head# 2024-03-30T13:50:32.594 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72f8c310:::benchmark_data_smithi043_119832_object34051:head# 2024-03-30T13:50:32.594 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72f926bc:::benchmark_data_smithi043_119832_object26302:head# 2024-03-30T13:50:32.594 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72f9c618:::benchmark_data_smithi043_119832_object8976:head# 2024-03-30T13:50:32.594 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72fa1abc:::benchmark_data_smithi043_119832_object58616:head# 2024-03-30T13:50:32.595 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72fa3377:::benchmark_data_smithi043_119832_object36948:head# 2024-03-30T13:50:32.595 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72fa4966:::benchmark_data_smithi043_119832_object43326:head# 2024-03-30T13:50:32.595 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72fa7067:::benchmark_data_smithi043_119832_object76183:head# 2024-03-30T13:50:32.595 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72fad31f:::benchmark_data_smithi043_119832_object35464:head# 2024-03-30T13:50:32.595 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72fb3af0:::benchmark_data_smithi043_119832_object8269:head# 2024-03-30T13:50:32.595 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72fb7a19:::benchmark_data_smithi043_119832_object64856:head# 2024-03-30T13:50:32.596 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72fcd559:::benchmark_data_smithi043_119832_object42399:head# 2024-03-30T13:50:32.596 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72fcf02a:::benchmark_data_smithi043_119832_object49798:head# 2024-03-30T13:50:32.596 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72ff4329:::benchmark_data_smithi043_119832_object9161:head# 2024-03-30T13:50:32.596 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72ff57bc:::benchmark_data_smithi043_119832_object42387:head# 2024-03-30T13:50:32.596 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:72ffe97d:::benchmark_data_smithi043_119832_object3915:head# 2024-03-30T13:50:32.621 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7300535a:::benchmark_data_smithi043_119832_object7180:head# 2024-03-30T13:50:32.621 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73017667:::benchmark_data_smithi043_119832_object9391:head# 2024-03-30T13:50:32.621 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7301d8b7:::benchmark_data_smithi043_119832_object11028:head# 2024-03-30T13:50:32.621 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73022386:::benchmark_data_smithi043_119832_object6354:head# 2024-03-30T13:50:32.622 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73037998:::benchmark_data_smithi043_119832_object83758:head# 2024-03-30T13:50:32.622 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73038a14:::benchmark_data_smithi043_119832_object69514:head# 2024-03-30T13:50:32.622 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:730510e1:::benchmark_data_smithi043_119832_object19457:head# 2024-03-30T13:50:32.622 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73063627:::benchmark_data_smithi043_119832_object72316:head# 2024-03-30T13:50:32.622 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:730699aa:::benchmark_data_smithi043_119832_object69756:head# 2024-03-30T13:50:32.622 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73069a3f:::benchmark_data_smithi043_119832_object57224:head# 2024-03-30T13:50:32.622 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73074886:::benchmark_data_smithi043_119832_object81655:head# 2024-03-30T13:50:32.622 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7307bc44:::benchmark_data_smithi043_119832_object34602:head# 2024-03-30T13:50:32.622 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7307c0a7:::benchmark_data_smithi043_119832_object15744:head# 2024-03-30T13:50:32.622 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:730a4f9d:::benchmark_data_smithi043_119832_object1481:head# 2024-03-30T13:50:32.622 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:730a927d:::benchmark_data_smithi043_119832_object44218:head# 2024-03-30T13:50:32.622 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:730a96dc:::benchmark_data_smithi043_119832_object85144:head# 2024-03-30T13:50:32.622 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:730b8b42:::benchmark_data_smithi043_119832_object83247:head# 2024-03-30T13:50:32.622 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:730c4fe8:::benchmark_data_smithi043_119832_object25443:head# 2024-03-30T13:50:32.623 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:730d35d6:::benchmark_data_smithi043_119832_object15558:head# 2024-03-30T13:50:32.623 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:730d3b3b:::benchmark_data_smithi043_119832_object63482:head# 2024-03-30T13:50:32.623 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:730e22c2:::benchmark_data_smithi043_119832_object9441:head# 2024-03-30T13:50:32.623 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73107587:::benchmark_data_smithi043_119832_object80644:head# 2024-03-30T13:50:32.623 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73113b7d:::benchmark_data_smithi043_119832_object62927:head# 2024-03-30T13:50:32.623 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73114bd0:::benchmark_data_smithi043_119832_object82965:head# 2024-03-30T13:50:32.623 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:731158e5:::benchmark_data_smithi043_119832_object25216:head# 2024-03-30T13:50:32.623 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7312713b:::benchmark_data_smithi043_119832_object50090:head# 2024-03-30T13:50:32.623 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7312c3b2:::benchmark_data_smithi043_119832_object26660:head# 2024-03-30T13:50:32.623 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73140916:::benchmark_data_smithi043_119832_object47311:head# 2024-03-30T13:50:32.623 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73149983:::benchmark_data_smithi043_119832_object32296:head# 2024-03-30T13:50:32.623 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7314a2b6:::benchmark_data_smithi043_119832_object31309:head# 2024-03-30T13:50:32.623 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7314b576:::benchmark_data_smithi043_119832_object17833:head# 2024-03-30T13:50:32.623 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7314ff98:::benchmark_data_smithi043_119832_object26907:head# 2024-03-30T13:50:32.624 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73163be2:::benchmark_data_smithi043_119832_object44938:head# 2024-03-30T13:50:32.624 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73174dcd:::benchmark_data_smithi043_119832_object68699:head# 2024-03-30T13:50:32.624 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73175960:::benchmark_data_smithi043_119832_object25480:head# 2024-03-30T13:50:32.624 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73181118:::benchmark_data_smithi043_119832_object8669:head# 2024-03-30T13:50:32.624 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7318443a:::benchmark_data_smithi043_119832_object27332:head# 2024-03-30T13:50:32.624 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7319caf6:::benchmark_data_smithi043_119832_object56080:head# 2024-03-30T13:50:32.624 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7319d7e2:::benchmark_data_smithi043_119832_object79011:head# 2024-03-30T13:50:32.624 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7319f93a:::benchmark_data_smithi043_119832_object53810:head# 2024-03-30T13:50:32.624 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:731a74ae:::benchmark_data_smithi043_119832_object42478:head# 2024-03-30T13:50:32.624 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:731ab7e3:::benchmark_data_smithi043_119832_object71718:head# 2024-03-30T13:50:32.624 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:731b4e34:::benchmark_data_smithi043_119832_object12869:head# 2024-03-30T13:50:32.624 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:731c2c86:::benchmark_data_smithi043_119832_object79509:head# 2024-03-30T13:50:32.624 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:731cf4ca:::benchmark_data_smithi043_119832_object22626:head# 2024-03-30T13:50:32.624 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:731e1bc9:::benchmark_data_smithi043_119832_object11273:head# 2024-03-30T13:50:32.624 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:731fe26e:::benchmark_data_smithi043_119832_object74095:head# 2024-03-30T13:50:32.625 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7320bf4f:::benchmark_data_smithi043_119832_object61971:head# 2024-03-30T13:50:32.625 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73212542:::benchmark_data_smithi043_119832_object57852:head# 2024-03-30T13:50:32.625 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7321b613:::benchmark_data_smithi043_119832_object36239:head# 2024-03-30T13:50:32.625 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73227700:::benchmark_data_smithi043_119832_object1076:head# 2024-03-30T13:50:32.625 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7322e784:::benchmark_data_smithi043_119832_object67819:head# 2024-03-30T13:50:32.625 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7323026f:::benchmark_data_smithi043_119832_object53851:head# 2024-03-30T13:50:32.625 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73231467:::benchmark_data_smithi043_119832_object64675:head# 2024-03-30T13:50:32.625 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73240fd6:::benchmark_data_smithi043_119832_object2623:head# 2024-03-30T13:50:32.625 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73243aca:::benchmark_data_smithi043_119832_object51110:head# 2024-03-30T13:50:32.625 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73246a21:::benchmark_data_smithi043_119832_object41215:head# 2024-03-30T13:50:32.625 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7324db09:::benchmark_data_smithi043_119832_object49978:head# 2024-03-30T13:50:32.625 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7324fa62:::benchmark_data_smithi043_119832_object16452:head# 2024-03-30T13:50:32.625 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73254f5a:::benchmark_data_smithi043_119832_object58646:head# 2024-03-30T13:50:32.625 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73266ecc:::benchmark_data_smithi043_119832_object30849:head# 2024-03-30T13:50:32.626 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7327ca30:::benchmark_data_smithi043_119832_object44480:head# 2024-03-30T13:50:32.626 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:732808a7:::benchmark_data_smithi043_119832_object6652:head# 2024-03-30T13:50:32.626 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73287f10:::benchmark_data_smithi043_119832_object35788:head# 2024-03-30T13:50:32.626 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73294864:::benchmark_data_smithi043_119832_object38593:head# 2024-03-30T13:50:32.626 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7329f015:::benchmark_data_smithi043_119832_object28204:head# 2024-03-30T13:50:32.626 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:732a61fd:::benchmark_data_smithi043_119832_object83940:head# 2024-03-30T13:50:32.626 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:732b3e8c:::benchmark_data_smithi043_119832_object86327:head# 2024-03-30T13:50:32.626 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:732b59e6:::benchmark_data_smithi043_119832_object70621:head# 2024-03-30T13:50:32.626 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:732b80a9:::benchmark_data_smithi043_119832_object56180:head# 2024-03-30T13:50:32.626 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:732bcd9d:::benchmark_data_smithi043_119832_object36562:head# 2024-03-30T13:50:32.626 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:732c095f:::benchmark_data_smithi043_119832_object25772:head# 2024-03-30T13:50:32.626 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:732c381a:::benchmark_data_smithi043_119832_object57504:head# 2024-03-30T13:50:32.626 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:732c3ece:::benchmark_data_smithi043_119832_object41486:head# 2024-03-30T13:50:32.626 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:732c7c01:::benchmark_data_smithi043_119832_object82316:head# 2024-03-30T13:50:32.626 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:732f059e:::benchmark_data_smithi043_119832_object4381:head# 2024-03-30T13:50:32.627 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:732f63ff:::benchmark_data_smithi043_119832_object72463:head# 2024-03-30T13:50:32.627 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:732f85ae:::benchmark_data_smithi043_119832_object67389:head# 2024-03-30T13:50:32.627 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:732f97a9:::benchmark_data_smithi043_119832_object10500:head# 2024-03-30T13:50:32.627 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7330a849:::benchmark_data_smithi043_119832_object13621:head# 2024-03-30T13:50:32.627 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73311fb9:::benchmark_data_smithi043_119832_object83141:head# 2024-03-30T13:50:32.627 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7331d23f:::benchmark_data_smithi043_119832_object74772:head# 2024-03-30T13:50:32.627 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7332b116:::benchmark_data_smithi043_119832_object54282:head# 2024-03-30T13:50:32.627 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7332f55b:::benchmark_data_smithi043_119832_object73263:head# 2024-03-30T13:50:32.627 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:733357b3:::benchmark_data_smithi043_119832_object78950:head# 2024-03-30T13:50:32.627 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7333d8fb:::benchmark_data_smithi043_119832_object601:head# 2024-03-30T13:50:32.627 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7333ff34:::benchmark_data_smithi043_119832_object82420:head# 2024-03-30T13:50:32.629 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7336ae5b:::benchmark_data_smithi043_119832_object19398:head# 2024-03-30T13:50:32.629 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7336b956:::benchmark_data_smithi043_119832_object3962:head# 2024-03-30T13:50:32.630 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7336d5dd:::benchmark_data_smithi043_119832_object71141:head# 2024-03-30T13:50:32.630 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:733788de:::benchmark_data_smithi043_119832_object8760:head# 2024-03-30T13:50:32.630 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7337bceb:::benchmark_data_smithi043_119832_object51013:head# 2024-03-30T13:50:32.630 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:733a6408:::benchmark_data_smithi043_119832_object60913:head# 2024-03-30T13:50:32.630 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:733b19f6:::benchmark_data_smithi043_119832_object66614:head# 2024-03-30T13:50:32.630 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:733b8ea0:::benchmark_data_smithi043_119832_object70088:head# 2024-03-30T13:50:32.630 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:32.631 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:733bf09d:::benchmark_data_smithi043_119832_object72109:head# 2024-03-30T13:50:32.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:32.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:32.647 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:733c3342:::benchmark_data_smithi043_119832_object74992:head# 2024-03-30T13:50:32.647 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:733c3846:::benchmark_data_smithi043_119832_object18377:head# 2024-03-30T13:50:32.647 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:733c8623:::benchmark_data_smithi043_119832_object72704:head# 2024-03-30T13:50:32.647 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:733e5115:::benchmark_data_smithi043_119832_object83740:head# 2024-03-30T13:50:32.647 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:733e7b11:::benchmark_data_smithi043_119832_object11502:head# 2024-03-30T13:50:32.647 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:733ea3f6:::benchmark_data_smithi043_119832_object79912:head# 2024-03-30T13:50:32.648 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:733ef5f9:::benchmark_data_smithi043_119832_object51267:head# 2024-03-30T13:50:32.648 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:733f558f:::benchmark_data_smithi043_119832_object13023:head# 2024-03-30T13:50:32.648 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:733f8c5f:::benchmark_data_smithi043_119832_object2994:head# 2024-03-30T13:50:32.648 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:734099f6:::benchmark_data_smithi043_119832_object17246:head# 2024-03-30T13:50:32.648 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73418ce6:::benchmark_data_smithi043_119832_object22250:head# 2024-03-30T13:50:32.648 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:734614b7:::benchmark_data_smithi043_119832_object18992:head# 2024-03-30T13:50:32.648 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73461772:::benchmark_data_smithi043_119832_object71477:head# 2024-03-30T13:50:32.648 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:734646dd:::benchmark_data_smithi043_119832_object62985:head# 2024-03-30T13:50:32.648 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7346f719:::benchmark_data_smithi043_119832_object7799:head# 2024-03-30T13:50:32.648 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73474a25:::benchmark_data_smithi043_119832_object59690:head# 2024-03-30T13:50:32.648 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73475475:::benchmark_data_smithi043_119832_object81864:head# 2024-03-30T13:50:32.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:32 smithi043 ceph-mon[29963]: pgmap v309: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 840 op/s; 41791/334356 objects degraded (12.499%) 2024-03-30T13:50:32.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:32 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 41791/334356 objects degraded (12.499%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:32.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:32 smithi043 ceph-mon[29963]: osdmap e176: 8 total, 7 up, 8 in 2024-03-30T13:50:32.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:32 smithi043 ceph-mon[42099]: pgmap v309: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 840 op/s; 41791/334356 objects degraded (12.499%) 2024-03-30T13:50:32.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:32 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 41791/334356 objects degraded (12.499%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:32.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:32 smithi043 ceph-mon[42099]: osdmap e176: 8 total, 7 up, 8 in 2024-03-30T13:50:32.739 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:32.786 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:50:32.840 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7348fb77:::benchmark_data_smithi043_119832_object51249:head# 2024-03-30T13:50:32.840 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:734a541a:::benchmark_data_smithi043_119832_object37485:head# 2024-03-30T13:50:32.840 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:734a7513:::benchmark_data_smithi043_119832_object58372:head# 2024-03-30T13:50:32.841 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:734a853f:::benchmark_data_smithi043_119832_object60:head# 2024-03-30T13:50:32.841 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:734aa94a:::benchmark_data_smithi043_119832_object13322:head# 2024-03-30T13:50:32.841 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:734ad730:::benchmark_data_smithi043_119832_object23908:head# 2024-03-30T13:50:32.841 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:734cc906:::benchmark_data_smithi043_119832_object85022:head# 2024-03-30T13:50:32.841 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:734ee960:::benchmark_data_smithi043_119832_object14213:head# 2024-03-30T13:50:32.841 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73501579:::benchmark_data_smithi043_119832_object30007:head# 2024-03-30T13:50:32.841 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:735159e8:::benchmark_data_smithi043_119832_object72817:head# 2024-03-30T13:50:32.841 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73519575:::benchmark_data_smithi043_119832_object39263:head# 2024-03-30T13:50:32.841 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73558e5c:::benchmark_data_smithi043_119832_object65654:head# 2024-03-30T13:50:32.841 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73561ac1:::benchmark_data_smithi043_119832_object1640:head# 2024-03-30T13:50:32.841 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7356fd38:::benchmark_data_smithi043_119832_object14587:head# 2024-03-30T13:50:32.841 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7357d542:::benchmark_data_smithi043_119832_object59136:head# 2024-03-30T13:50:32.841 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:735922a4:::benchmark_data_smithi043_119832_object23372:head# 2024-03-30T13:50:32.842 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:735a81f2:::benchmark_data_smithi043_119832_object14737:head# 2024-03-30T13:50:32.842 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:735b5a7c:::benchmark_data_smithi043_119832_object9534:head# 2024-03-30T13:50:32.842 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:735b7fdb:::benchmark_data_smithi043_119832_object14988:head# 2024-03-30T13:50:32.842 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:735b9aa7:::benchmark_data_smithi043_119832_object71484:head# 2024-03-30T13:50:32.842 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:735c117e:::benchmark_data_smithi043_119832_object39789:head# 2024-03-30T13:50:32.842 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:735d19fd:::benchmark_data_smithi043_119832_object60226:head# 2024-03-30T13:50:32.842 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:735d92a6:::benchmark_data_smithi043_119832_object37216:head# 2024-03-30T13:50:32.842 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:735dd05f:::benchmark_data_smithi043_119832_object50128:head# 2024-03-30T13:50:32.842 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:735eda65:::benchmark_data_smithi043_119832_object19052:head# 2024-03-30T13:50:32.842 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:735f7450:::benchmark_data_smithi043_119832_object4479:head# 2024-03-30T13:50:32.842 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7360d6ea:::benchmark_data_smithi043_119832_object8139:head# 2024-03-30T13:50:32.842 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7360ff4f:::benchmark_data_smithi043_119832_object62899:head# 2024-03-30T13:50:32.843 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:736165a7:::benchmark_data_smithi043_119832_object85189:head# 2024-03-30T13:50:32.843 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7361735d:::benchmark_data_smithi043_119832_object35431:head# 2024-03-30T13:50:32.843 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:736194aa:::benchmark_data_smithi043_119832_object69904:head# 2024-03-30T13:50:32.843 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7362321d:::benchmark_data_smithi043_119832_object58574:head# 2024-03-30T13:50:32.843 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:736296ff:::benchmark_data_smithi043_119832_object52269:head# 2024-03-30T13:50:32.843 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:736297c9:::benchmark_data_smithi043_119832_object37449:head# 2024-03-30T13:50:32.843 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7362d1b8:::benchmark_data_smithi043_119832_object18784:head# 2024-03-30T13:50:32.843 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:736377c6:::benchmark_data_smithi043_119832_object81175:head# 2024-03-30T13:50:32.843 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7363ac1d:::benchmark_data_smithi043_119832_object18874:head# 2024-03-30T13:50:32.843 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:736457b6:::benchmark_data_smithi043_119832_object19007:head# 2024-03-30T13:50:32.843 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73647797:::benchmark_data_smithi043_119832_object60577:head# 2024-03-30T13:50:32.843 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7364a527:::benchmark_data_smithi043_119832_object20471:head# 2024-03-30T13:50:32.844 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7364b7c7:::benchmark_data_smithi043_119832_object43778:head# 2024-03-30T13:50:32.844 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7366fe68:::benchmark_data_smithi043_119832_object86711:head# 2024-03-30T13:50:32.844 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73672bba:::benchmark_data_smithi043_119832_object31053:head# 2024-03-30T13:50:32.844 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:736896b6:::benchmark_data_smithi043_119832_object66778:head# 2024-03-30T13:50:32.844 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:736a03b2:::benchmark_data_smithi043_119832_object22088:head# 2024-03-30T13:50:32.844 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:736a161c:::benchmark_data_smithi043_119832_object59502:head# 2024-03-30T13:50:32.844 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:736c3456:::benchmark_data_smithi043_119832_object6519:head# 2024-03-30T13:50:32.844 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:736e227d:::benchmark_data_smithi043_119832_object32011:head# 2024-03-30T13:50:32.844 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:736e2db2:::benchmark_data_smithi043_119832_object32045:head# 2024-03-30T13:50:32.844 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:736f1c99:::benchmark_data_smithi043_119832_object33028:head# 2024-03-30T13:50:32.844 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:736fdefe:::benchmark_data_smithi043_119832_object63743:head# 2024-03-30T13:50:32.844 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:736fe47c:::benchmark_data_smithi043_119832_object1347:head# 2024-03-30T13:50:32.844 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73707ee2:::benchmark_data_smithi043_119832_object78662:head# 2024-03-30T13:50:32.844 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73723e1b:::benchmark_data_smithi043_119832_object25500:head# 2024-03-30T13:50:32.845 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7372cd90:::benchmark_data_smithi043_119832_object47538:head# 2024-03-30T13:50:32.845 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:737308d0:::benchmark_data_smithi043_119832_object86835:head# 2024-03-30T13:50:32.845 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73740ee1:::benchmark_data_smithi043_119832_object45724:head# 2024-03-30T13:50:32.845 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73744e6c:::benchmark_data_smithi043_119832_object16764:head# 2024-03-30T13:50:32.845 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7374b460:::benchmark_data_smithi043_119832_object6312:head# 2024-03-30T13:50:32.845 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7374ef8b:::benchmark_data_smithi043_119832_object35139:head# 2024-03-30T13:50:32.845 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73751983:::benchmark_data_smithi043_119832_object1493:head# 2024-03-30T13:50:32.845 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7375ecf8:::benchmark_data_smithi043_119832_object68317:head# 2024-03-30T13:50:32.845 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7378a76e:::benchmark_data_smithi043_119832_object33188:head# 2024-03-30T13:50:32.845 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7379addf:::benchmark_data_smithi043_119832_object6939:head# 2024-03-30T13:50:32.845 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7379e150:::benchmark_data_smithi043_119832_object23988:head# 2024-03-30T13:50:32.845 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:737a2e2e:::benchmark_data_smithi043_119832_object16430:head# 2024-03-30T13:50:32.845 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:737a95b9:::benchmark_data_smithi043_119832_object20304:head# 2024-03-30T13:50:32.846 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:737aee55:::benchmark_data_smithi043_119832_object11945:head# 2024-03-30T13:50:32.846 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:737d5537:::benchmark_data_smithi043_119832_object8035:head# 2024-03-30T13:50:32.846 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:737f8fab:::benchmark_data_smithi043_119832_object73693:head# 2024-03-30T13:50:32.846 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73800740:::benchmark_data_smithi043_119832_object56133:head# 2024-03-30T13:50:32.846 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:738105e7:::benchmark_data_smithi043_119832_object39682:head# 2024-03-30T13:50:32.846 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7381a759:::benchmark_data_smithi043_119832_object10935:head# 2024-03-30T13:50:32.846 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7382b8bf:::benchmark_data_smithi043_119832_object66535:head# 2024-03-30T13:50:32.846 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:738387df:::benchmark_data_smithi043_119832_object78801:head# 2024-03-30T13:50:32.846 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7383af91:::benchmark_data_smithi043_119832_object52869:head# 2024-03-30T13:50:32.846 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73850537:::benchmark_data_smithi043_119832_object58206:head# 2024-03-30T13:50:32.846 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73853284:::benchmark_data_smithi043_119832_object62115:head# 2024-03-30T13:50:32.846 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7385329f:::benchmark_data_smithi043_119832_object44911:head# 2024-03-30T13:50:32.846 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7385371e:::benchmark_data_smithi043_119832_object50961:head# 2024-03-30T13:50:32.846 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7385795d:::benchmark_data_smithi043_119832_object14156:head# 2024-03-30T13:50:32.847 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7385b005:::benchmark_data_smithi043_119832_object43212:head# 2024-03-30T13:50:32.847 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7386a61e:::benchmark_data_smithi043_119832_object2395:head# 2024-03-30T13:50:32.847 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7387ea95:::benchmark_data_smithi043_119832_object41665:head# 2024-03-30T13:50:32.847 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7388c4b4:::benchmark_data_smithi043_119832_object27485:head# 2024-03-30T13:50:32.847 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7389d81f:::benchmark_data_smithi043_119832_object12765:head# 2024-03-30T13:50:32.847 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7389e5ec:::benchmark_data_smithi043_119832_object76660:head# 2024-03-30T13:50:32.847 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7389f278:::benchmark_data_smithi043_119832_object65418:head# 2024-03-30T13:50:32.847 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:738a20dd:::benchmark_data_smithi043_119832_object49732:head# 2024-03-30T13:50:32.847 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:738a972b:::benchmark_data_smithi043_119832_object62169:head# 2024-03-30T13:50:32.847 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:738a9a8d:::benchmark_data_smithi043_119832_object14704:head# 2024-03-30T13:50:32.847 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:738bdd7a:::benchmark_data_smithi043_119832_object14435:head# 2024-03-30T13:50:32.848 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:738be96e:::benchmark_data_smithi043_119832_object24568:head# 2024-03-30T13:50:32.848 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:738c2e69:::benchmark_data_smithi043_119832_object60573:head# 2024-03-30T13:50:32.848 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:738c77a4:::benchmark_data_smithi043_119832_object80634:head# 2024-03-30T13:50:32.848 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:738cc439:::benchmark_data_smithi043_119832_object54578:head# 2024-03-30T13:50:32.848 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:738d40d3:::benchmark_data_smithi043_119832_object87817:head# 2024-03-30T13:50:32.848 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:738e7b07:::benchmark_data_smithi043_119832_object27339:head# 2024-03-30T13:50:32.848 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:738e7c5b:::benchmark_data_smithi043_119832_object23020:head# 2024-03-30T13:50:32.848 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:738f19df:::benchmark_data_smithi043_119832_object74168:head# 2024-03-30T13:50:32.848 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:738f8b3d:::benchmark_data_smithi043_119832_object23965:head# 2024-03-30T13:50:32.848 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7390bf2f:::benchmark_data_smithi043_119832_object3176:head# 2024-03-30T13:50:32.848 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7390f0ad:::benchmark_data_smithi043_119832_object85820:head# 2024-03-30T13:50:32.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73915c61:::benchmark_data_smithi043_119832_object30093:head# 2024-03-30T13:50:32.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73916048:::benchmark_data_smithi043_119832_object11555:head# 2024-03-30T13:50:32.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7391e551:::benchmark_data_smithi043_119832_object26194:head# 2024-03-30T13:50:32.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73926f1f:::benchmark_data_smithi043_119832_object74280:head# 2024-03-30T13:50:32.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73938ed7:::benchmark_data_smithi043_119832_object73093:head# 2024-03-30T13:50:32.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7393b865:::benchmark_data_smithi043_119832_object33082:head# 2024-03-30T13:50:32.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:739406ea:::benchmark_data_smithi043_119832_object20320:head# 2024-03-30T13:50:32.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7394e117:::benchmark_data_smithi043_119832_object39866:head# 2024-03-30T13:50:32.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73952040:::benchmark_data_smithi043_119832_object65895:head# 2024-03-30T13:50:32.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7395ea15:::benchmark_data_smithi043_119832_object79028:head# 2024-03-30T13:50:32.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:739627e4:::benchmark_data_smithi043_119832_object65561:head# 2024-03-30T13:50:32.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73983e48:::benchmark_data_smithi043_119832_object47116:head# 2024-03-30T13:50:32.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:739865d4:::benchmark_data_smithi043_119832_object76317:head# 2024-03-30T13:50:32.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:739899f8:::benchmark_data_smithi043_119832_object39485:head# 2024-03-30T13:50:32.850 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:739948c2:::benchmark_data_smithi043_119832_object4190:head# 2024-03-30T13:50:32.850 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73998586:::benchmark_data_smithi043_119832_object37838:head# 2024-03-30T13:50:32.850 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7399dd46:::benchmark_data_smithi043_119832_object35278:head# 2024-03-30T13:50:32.850 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:739a6447:::benchmark_data_smithi043_119832_object31722:head# 2024-03-30T13:50:32.850 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:739b5351:::benchmark_data_smithi043_119832_object49007:head# 2024-03-30T13:50:32.850 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:739c837d:::benchmark_data_smithi043_119832_object74228:head# 2024-03-30T13:50:32.850 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:739cc8fc:::benchmark_data_smithi043_119832_object45390:head# 2024-03-30T13:50:32.850 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:739db323:::benchmark_data_smithi043_119832_object68551:head# 2024-03-30T13:50:32.850 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:739dd069:::benchmark_data_smithi043_119832_object46174:head# 2024-03-30T13:50:32.851 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:739dd2b4:::benchmark_data_smithi043_119832_object81428:head# 2024-03-30T13:50:32.851 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:739dd651:::benchmark_data_smithi043_119832_object53246:head# 2024-03-30T13:50:32.851 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:739eddcb:::benchmark_data_smithi043_119832_object51617:head# 2024-03-30T13:50:32.851 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:739f2df1:::benchmark_data_smithi043_119832_object68202:head# 2024-03-30T13:50:32.851 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:739ffa2a:::benchmark_data_smithi043_119832_object65602:head# 2024-03-30T13:50:32.851 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73a15dac:::benchmark_data_smithi043_119832_object87224:head# 2024-03-30T13:50:32.851 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73a1ed77:::benchmark_data_smithi043_119832_object82223:head# 2024-03-30T13:50:32.851 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73a4b94d:::benchmark_data_smithi043_119832_object25121:head# 2024-03-30T13:50:32.851 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73a509a9:::benchmark_data_smithi043_119832_object50781:head# 2024-03-30T13:50:32.851 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73a9b63b:::benchmark_data_smithi043_119832_object35645:head# 2024-03-30T13:50:32.851 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73aa0c5c:::benchmark_data_smithi043_119832_object67492:head# 2024-03-30T13:50:32.851 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73aa7eb4:::benchmark_data_smithi043_119832_object57354:head# 2024-03-30T13:50:32.851 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73ac4b46:::benchmark_data_smithi043_119832_object77523:head# 2024-03-30T13:50:32.851 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73aca0b9:::benchmark_data_smithi043_119832_object45603:head# 2024-03-30T13:50:32.852 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73acbc8c:::benchmark_data_smithi043_119832_object64249:head# 2024-03-30T13:50:32.852 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73ade79a:::benchmark_data_smithi043_119832_object48277:head# 2024-03-30T13:50:32.852 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73ae0e34:::benchmark_data_smithi043_119832_object50830:head# 2024-03-30T13:50:32.852 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73ae2867:::benchmark_data_smithi043_119832_object81219:head# 2024-03-30T13:50:32.852 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73aead8b:::benchmark_data_smithi043_119832_object3450:head# 2024-03-30T13:50:32.852 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73afd023:::benchmark_data_smithi043_119832_object2666:head# 2024-03-30T13:50:32.852 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73b0e0dd:::benchmark_data_smithi043_119832_object4889:head# 2024-03-30T13:50:32.852 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73b191aa:::benchmark_data_smithi043_119832_object67801:head# 2024-03-30T13:50:32.852 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73b1afe9:::benchmark_data_smithi043_119832_object5878:head# 2024-03-30T13:50:32.853 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73b200ff:::benchmark_data_smithi043_119832_object9397:head# 2024-03-30T13:50:32.853 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73b27049:::benchmark_data_smithi043_119832_object8789:head# 2024-03-30T13:50:32.853 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73b2e61a:::benchmark_data_smithi043_119832_object18708:head# 2024-03-30T13:50:32.853 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73b364fb:::benchmark_data_smithi043_119832_object4552:head# 2024-03-30T13:50:32.853 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73b52e2a:::benchmark_data_smithi043_119832_object24702:head# 2024-03-30T13:50:32.853 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73b6544f:::benchmark_data_smithi043_119832_object55364:head# 2024-03-30T13:50:32.853 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73b71fd4:::benchmark_data_smithi043_119832_object18557:head# 2024-03-30T13:50:32.853 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73b7c03d:::benchmark_data_smithi043_119832_object87450:head# 2024-03-30T13:50:32.853 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73b90f38:::benchmark_data_smithi043_119832_object48416:head# 2024-03-30T13:50:32.853 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73b9ba24:::benchmark_data_smithi043_119832_object57930:head# 2024-03-30T13:50:32.853 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73bacefc:::benchmark_data_smithi043_119832_object38996:head# 2024-03-30T13:50:32.853 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73bb2da3:::benchmark_data_smithi043_119832_object40960:head# 2024-03-30T13:50:32.853 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73bb9077:::benchmark_data_smithi043_119832_object63736:head# 2024-03-30T13:50:32.854 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73bc5176:::benchmark_data_smithi043_119832_object46822:head# 2024-03-30T13:50:32.854 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73be5620:::benchmark_data_smithi043_119832_object75120:head# 2024-03-30T13:50:32.854 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73c09c54:::benchmark_data_smithi043_119832_object41878:head# 2024-03-30T13:50:32.854 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73c153ac:::benchmark_data_smithi043_119832_object14976:head# 2024-03-30T13:50:32.854 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73c1737f:::benchmark_data_smithi043_119832_object80859:head# 2024-03-30T13:50:32.854 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73c17498:::benchmark_data_smithi043_119832_object6650:head# 2024-03-30T13:50:32.854 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73c1cbd2:::benchmark_data_smithi043_119832_object14991:head# 2024-03-30T13:50:32.854 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73c23558:::benchmark_data_smithi043_119832_object39254:head# 2024-03-30T13:50:32.854 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73c2b512:::benchmark_data_smithi043_119832_object19093:head# 2024-03-30T13:50:32.854 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73c35971:::benchmark_data_smithi043_119832_object22908:head# 2024-03-30T13:50:32.854 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73c3a923:::benchmark_data_smithi043_119832_object31850:head# 2024-03-30T13:50:32.855 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73c482c0:::benchmark_data_smithi043_119832_object60648:head# 2024-03-30T13:50:32.855 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73c56b50:::benchmark_data_smithi043_119832_object70239:head# 2024-03-30T13:50:32.855 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73c5cecc:::benchmark_data_smithi043_119832_object8941:head# 2024-03-30T13:50:32.855 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73c65fcf:::benchmark_data_smithi043_119832_object9146:head# 2024-03-30T13:50:32.855 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73c74184:::benchmark_data_smithi043_119832_object32133:head# 2024-03-30T13:50:32.855 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73c8d6e9:::benchmark_data_smithi043_119832_object74713:head# 2024-03-30T13:50:32.855 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73c93a96:::benchmark_data_smithi043_119832_object79671:head# 2024-03-30T13:50:32.855 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73c9b7f4:::benchmark_data_smithi043_119832_object8776:head# 2024-03-30T13:50:32.855 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73c9ce0e:::benchmark_data_smithi043_119832_object1271:head# 2024-03-30T13:50:32.855 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73ca0d1a:::benchmark_data_smithi043_119832_object34598:head# 2024-03-30T13:50:32.855 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73cb4513:::benchmark_data_smithi043_119832_object66744:head# 2024-03-30T13:50:32.856 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73cc8328:::benchmark_data_smithi043_119832_object25924:head# 2024-03-30T13:50:32.856 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73cd94a3:::benchmark_data_smithi043_119832_object23836:head# 2024-03-30T13:50:32.856 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73ce2bca:::benchmark_data_smithi043_119832_object69361:head# 2024-03-30T13:50:32.856 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73ce8bb5:::benchmark_data_smithi043_119832_object81255:head# 2024-03-30T13:50:32.856 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73cf5661:::benchmark_data_smithi043_119832_object71023:head# 2024-03-30T13:50:32.856 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73d04972:::benchmark_data_smithi043_119832_object63547:head# 2024-03-30T13:50:32.856 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73d04e02:::benchmark_data_smithi043_119832_object34544:head# 2024-03-30T13:50:32.856 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73d14499:::benchmark_data_smithi043_119832_object123:head# 2024-03-30T13:50:32.856 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73d33c63:::benchmark_data_smithi043_119832_object76507:head# 2024-03-30T13:50:32.856 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73d37b41:::benchmark_data_smithi043_119832_object80738:head# 2024-03-30T13:50:32.857 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73d3c143:::benchmark_data_smithi043_119832_object54676:head# 2024-03-30T13:50:32.857 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73d423b1:::benchmark_data_smithi043_119832_object81564:head# 2024-03-30T13:50:32.857 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73d4e762:::benchmark_data_smithi043_119832_object53016:head# 2024-03-30T13:50:32.857 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73d69fc9:::benchmark_data_smithi043_119832_object17467:head# 2024-03-30T13:50:32.857 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73d72b92:::benchmark_data_smithi043_119832_object72975:head# 2024-03-30T13:50:32.857 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73d76e94:::benchmark_data_smithi043_119832_object69452:head# 2024-03-30T13:50:32.857 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73d77b81:::benchmark_data_smithi043_119832_object43668:head# 2024-03-30T13:50:32.857 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73d85b18:::benchmark_data_smithi043_119832_object35454:head# 2024-03-30T13:50:32.857 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73d9b15b:::benchmark_data_smithi043_119832_object22684:head# 2024-03-30T13:50:32.857 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73dcca66:::benchmark_data_smithi043_119832_object10484:head# 2024-03-30T13:50:32.857 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73dccfa5:::benchmark_data_smithi043_119832_object77064:head# 2024-03-30T13:50:32.858 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73dd2a41:::benchmark_data_smithi043_119832_object33453:head# 2024-03-30T13:50:32.858 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73dd7696:::benchmark_data_smithi043_119832_object55834:head# 2024-03-30T13:50:32.858 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73de8dd5:::benchmark_data_smithi043_119832_object43535:head# 2024-03-30T13:50:32.858 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73dee8f0:::benchmark_data_smithi043_119832_object59939:head# 2024-03-30T13:50:32.858 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73defe7f:::benchmark_data_smithi043_119832_object32199:head# 2024-03-30T13:50:32.858 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73df1d7c:::benchmark_data_smithi043_119832_object56739:head# 2024-03-30T13:50:32.858 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73df6f64:::benchmark_data_smithi043_119832_object85180:head# 2024-03-30T13:50:32.858 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73dfd489:::benchmark_data_smithi043_119832_object71330:head# 2024-03-30T13:50:32.858 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73e1b1ca:::benchmark_data_smithi043_119832_object51100:head# 2024-03-30T13:50:32.859 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73e25baa:::benchmark_data_smithi043_119832_object61100:head# 2024-03-30T13:50:32.859 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73e2d277:::benchmark_data_smithi043_119832_object54996:head# 2024-03-30T13:50:32.859 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73e2ef9a:::benchmark_data_smithi043_119832_object30739:head# 2024-03-30T13:50:32.859 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73e2f95e:::benchmark_data_smithi043_119832_object53103:head# 2024-03-30T13:50:32.859 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73e6eaf3:::benchmark_data_smithi043_119832_object82623:head# 2024-03-30T13:50:32.859 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73e743ca:::benchmark_data_smithi043_119832_object65198:head# 2024-03-30T13:50:32.859 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73e91a9e:::benchmark_data_smithi043_119832_object49068:head# 2024-03-30T13:50:32.860 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73eadd36:::benchmark_data_smithi043_119832_object67516:head# 2024-03-30T13:50:32.860 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73eb892e:::benchmark_data_smithi043_119832_object88235:head# 2024-03-30T13:50:32.860 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73ec87b7:::benchmark_data_smithi043_119832_object81868:head# 2024-03-30T13:50:32.860 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73ec911f:::benchmark_data_smithi043_119832_object20975:head# 2024-03-30T13:50:32.860 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73ef4eb1:::benchmark_data_smithi043_119832_object37274:head# 2024-03-30T13:50:32.860 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73f1a80c:::benchmark_data_smithi043_119832_object74002:head# 2024-03-30T13:50:32.860 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73f1a8ba:::benchmark_data_smithi043_119832_object48759:head# 2024-03-30T13:50:32.860 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73f2c0f9:::benchmark_data_smithi043_119832_object64510:head# 2024-03-30T13:50:32.860 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73f49ebf:::benchmark_data_smithi043_119832_object15659:head# 2024-03-30T13:50:32.860 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73f4d967:::benchmark_data_smithi043_119832_object82543:head# 2024-03-30T13:50:32.860 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73f602be:::benchmark_data_smithi043_119832_object82548:head# 2024-03-30T13:50:32.860 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73f6706a:::benchmark_data_smithi043_119832_object60879:head# 2024-03-30T13:50:32.860 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73f70379:::benchmark_data_smithi043_119832_object418:head# 2024-03-30T13:50:32.861 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73f723ca:::benchmark_data_smithi043_119832_object70960:head# 2024-03-30T13:50:32.861 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73f9876a:::benchmark_data_smithi043_119832_object31243:head# 2024-03-30T13:50:32.861 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73f9dd5b:::benchmark_data_smithi043_119832_object29737:head# 2024-03-30T13:50:32.861 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73fadcec:::benchmark_data_smithi043_119832_object31489:head# 2024-03-30T13:50:32.861 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73fb3cd5:::benchmark_data_smithi043_119832_object74939:head# 2024-03-30T13:50:32.861 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73fb7c0d:::benchmark_data_smithi043_119832_object2763:head# 2024-03-30T13:50:32.861 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73fc024c:::benchmark_data_smithi043_119832_object60995:head# 2024-03-30T13:50:32.861 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73fc3968:::benchmark_data_smithi043_119832_object74951:head# 2024-03-30T13:50:32.861 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73fd7bed:::benchmark_data_smithi043_119832_object87458:head# 2024-03-30T13:50:32.861 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73fe1a58:::benchmark_data_smithi043_119832_object79155:head# 2024-03-30T13:50:32.861 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73feba33:::benchmark_data_smithi043_119832_object82472:head# 2024-03-30T13:50:32.861 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73feedd8:::benchmark_data_smithi043_119832_object83114:head# 2024-03-30T13:50:32.861 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:73ff697f:::benchmark_data_smithi043_119832_object56562:head# 2024-03-30T13:50:32.862 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7401bce5:::benchmark_data_smithi043_119832_object16706:head# 2024-03-30T13:50:32.862 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:740270c0:::benchmark_data_smithi043_119832_object602:head# 2024-03-30T13:50:32.862 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:740406ae:::benchmark_data_smithi043_119832_object32475:head# 2024-03-30T13:50:32.862 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:740594ea:::benchmark_data_smithi043_119832_object76886:head# 2024-03-30T13:50:32.862 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7405e4d0:::benchmark_data_smithi043_119832_object11870:head# 2024-03-30T13:50:32.862 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7405ec61:::benchmark_data_smithi043_119832_object20777:head# 2024-03-30T13:50:32.862 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7405fa1e:::benchmark_data_smithi043_119832_object45492:head# 2024-03-30T13:50:32.862 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:740705cd:::benchmark_data_smithi043_119832_object53322:head# 2024-03-30T13:50:32.862 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74079c3a:::benchmark_data_smithi043_119832_object2049:head# 2024-03-30T13:50:32.862 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7407d5be:::benchmark_data_smithi043_119832_object77453:head# 2024-03-30T13:50:32.862 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:740a805b:::benchmark_data_smithi043_119832_object10693:head# 2024-03-30T13:50:32.862 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:740ac55b:::benchmark_data_smithi043_119832_object52253:head# 2024-03-30T13:50:32.862 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:740b761b:::benchmark_data_smithi043_119832_object77692:head# 2024-03-30T13:50:32.863 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:740c639a:::benchmark_data_smithi043_119832_object63595:head# 2024-03-30T13:50:32.863 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:740c8eca:::benchmark_data_smithi043_119832_object49121:head# 2024-03-30T13:50:32.863 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:740d50f8:::benchmark_data_smithi043_119832_object13145:head# 2024-03-30T13:50:32.863 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:740da3d0:::benchmark_data_smithi043_119832_object85401:head# 2024-03-30T13:50:32.863 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:740f7a33:::benchmark_data_smithi043_119832_object7362:head# 2024-03-30T13:50:32.863 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:740fb036:::benchmark_data_smithi043_119832_object30033:head# 2024-03-30T13:50:32.863 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:740febf7:::benchmark_data_smithi043_119832_object76970:head# 2024-03-30T13:50:32.863 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7410b57d:::benchmark_data_smithi043_119832_object82605:head# 2024-03-30T13:50:32.863 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7412b460:::benchmark_data_smithi043_119832_object11290:head# 2024-03-30T13:50:32.863 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7416982f:::benchmark_data_smithi043_119832_object37816:head# 2024-03-30T13:50:32.863 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74184d08:::benchmark_data_smithi043_119832_object87664:head# 2024-03-30T13:50:32.863 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7418d0a4:::benchmark_data_smithi043_119832_object49056:head# 2024-03-30T13:50:32.863 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:741ab28d:::benchmark_data_smithi043_119832_object40822:head# 2024-03-30T13:50:32.863 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:741b1841:::benchmark_data_smithi043_119832_object85423:head# 2024-03-30T13:50:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:741b6e17:::benchmark_data_smithi043_119832_object38374:head# 2024-03-30T13:50:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:741ca002:::benchmark_data_smithi043_119832_object86955:head# 2024-03-30T13:50:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:741d17ba:::benchmark_data_smithi043_119832_object61831:head# 2024-03-30T13:50:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:741d77bc:::benchmark_data_smithi043_119832_object54117:head# 2024-03-30T13:50:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:741df49c:::benchmark_data_smithi043_119832_object64428:head# 2024-03-30T13:50:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:741df92a:::benchmark_data_smithi043_119832_object13148:head# 2024-03-30T13:50:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:741f6c1f:::benchmark_data_smithi043_119832_object27232:head# 2024-03-30T13:50:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:741f76c0:::benchmark_data_smithi043_119832_object76712:head# 2024-03-30T13:50:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:741f8726:::benchmark_data_smithi043_119832_object23336:head# 2024-03-30T13:50:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7421550a:::benchmark_data_smithi043_119832_object80130:head# 2024-03-30T13:50:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74217387:::benchmark_data_smithi043_119832_object4855:head# 2024-03-30T13:50:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7421819b:::benchmark_data_smithi043_119832_object86872:head# 2024-03-30T13:50:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7424ac0b:::benchmark_data_smithi043_119832_object23335:head# 2024-03-30T13:50:32.864 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7424bd1b:::benchmark_data_smithi043_119832_object77410:head# 2024-03-30T13:50:32.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74279516:::benchmark_data_smithi043_119832_object50788:head# 2024-03-30T13:50:32.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7427c966:::benchmark_data_smithi043_119832_object8434:head# 2024-03-30T13:50:32.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74283119:::benchmark_data_smithi043_119832_object37663:head# 2024-03-30T13:50:32.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7428a750:::benchmark_data_smithi043_119832_object23107:head# 2024-03-30T13:50:32.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7428cd28:::benchmark_data_smithi043_119832_object62135:head# 2024-03-30T13:50:32.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:742ac255:::benchmark_data_smithi043_119832_object56087:head# 2024-03-30T13:50:32.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:742bddb0:::benchmark_data_smithi043_119832_object36517:head# 2024-03-30T13:50:32.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:742c35b8:::benchmark_data_smithi043_119832_object5170:head# 2024-03-30T13:50:32.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:742c806c:::benchmark_data_smithi043_119832_object77804:head# 2024-03-30T13:50:32.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:742db022:::benchmark_data_smithi043_119832_object32392:head# 2024-03-30T13:50:32.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:742dc1ba:::benchmark_data_smithi043_119832_object72347:head# 2024-03-30T13:50:32.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:742e7d73:::benchmark_data_smithi043_119832_object86829:head# 2024-03-30T13:50:32.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:742e8c5b:::benchmark_data_smithi043_119832_object45429:head# 2024-03-30T13:50:32.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:742efb2b:::benchmark_data_smithi043_119832_object70098:head# 2024-03-30T13:50:32.866 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:742efd5a:::benchmark_data_smithi043_119832_object29297:head# 2024-03-30T13:50:32.866 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:742f15f2:::benchmark_data_smithi043_119832_object13039:head# 2024-03-30T13:50:32.866 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:742f50c9:::benchmark_data_smithi043_119832_object47194:head# 2024-03-30T13:50:32.866 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:742fd84b:::benchmark_data_smithi043_119832_object73282:head# 2024-03-30T13:50:32.866 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:742ff03c:::benchmark_data_smithi043_119832_object55117:head# 2024-03-30T13:50:32.866 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:743148ab:::benchmark_data_smithi043_119832_object51783:head# 2024-03-30T13:50:32.866 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:743202b8:::benchmark_data_smithi043_119832_object15205:head# 2024-03-30T13:50:32.866 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74324b7b:::benchmark_data_smithi043_119832_object41042:head# 2024-03-30T13:50:32.866 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7433309d:::benchmark_data_smithi043_119832_object17006:head# 2024-03-30T13:50:32.866 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74343668:::benchmark_data_smithi043_119832_object26419:head# 2024-03-30T13:50:32.866 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7434c2e3:::benchmark_data_smithi043_119832_object73634:head# 2024-03-30T13:50:32.866 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:743571a8:::benchmark_data_smithi043_119832_object38660:head# 2024-03-30T13:50:32.866 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7436d649:::benchmark_data_smithi043_119832_object21165:head# 2024-03-30T13:50:32.866 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:743793ca:::benchmark_data_smithi043_119832_object14946:head# 2024-03-30T13:50:32.867 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7437ea17:::benchmark_data_smithi043_119832_object18657:head# 2024-03-30T13:50:32.867 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:743a13ef:::benchmark_data_smithi043_119832_object13421:head# 2024-03-30T13:50:32.867 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:743ad00b:::benchmark_data_smithi043_119832_object36694:head# 2024-03-30T13:50:32.867 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:743aefe1:::benchmark_data_smithi043_119832_object16800:head# 2024-03-30T13:50:32.867 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:743c0e2c:::benchmark_data_smithi043_119832_object17715:head# 2024-03-30T13:50:32.867 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:743cba9f:::benchmark_data_smithi043_119832_object1974:head# 2024-03-30T13:50:32.867 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:743da4e4:::benchmark_data_smithi043_119832_object81703:head# 2024-03-30T13:50:32.867 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:743de688:::benchmark_data_smithi043_119832_object78584:head# 2024-03-30T13:50:32.867 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:743fbc4a:::benchmark_data_smithi043_119832_object61986:head# 2024-03-30T13:50:32.867 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7440e616:::benchmark_data_smithi043_119832_object58579:head# 2024-03-30T13:50:32.867 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74411a2f:::benchmark_data_smithi043_119832_object22985:head# 2024-03-30T13:50:32.867 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74414014:::benchmark_data_smithi043_119832_object28318:head# 2024-03-30T13:50:32.867 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7441b9fb:::benchmark_data_smithi043_119832_object18861:head# 2024-03-30T13:50:32.867 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74421bbb:::benchmark_data_smithi043_119832_object17226:head# 2024-03-30T13:50:32.868 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74428061:::benchmark_data_smithi043_119832_object21243:head# 2024-03-30T13:50:32.868 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74449de0:::benchmark_data_smithi043_119832_object13055:head# 2024-03-30T13:50:32.868 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74455052:::benchmark_data_smithi043_119832_object41796:head# 2024-03-30T13:50:32.868 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74455c98:::benchmark_data_smithi043_119832_object68312:head# 2024-03-30T13:50:32.868 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7445f733:::benchmark_data_smithi043_119832_object61111:head# 2024-03-30T13:50:32.868 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74462d08:::benchmark_data_smithi043_119832_object43936:head# 2024-03-30T13:50:32.868 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7446c7ca:::benchmark_data_smithi043_119832_object20915:head# 2024-03-30T13:50:32.868 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7446ca58:::benchmark_data_smithi043_119832_object71332:head# 2024-03-30T13:50:32.868 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7446cf8e:::benchmark_data_smithi043_119832_object30904:head# 2024-03-30T13:50:32.868 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74470cf5:::benchmark_data_smithi043_119832_object83225:head# 2024-03-30T13:50:32.868 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:744898d8:::benchmark_data_smithi043_119832_object40790:head# 2024-03-30T13:50:32.868 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74489a67:::benchmark_data_smithi043_119832_object8638:head# 2024-03-30T13:50:32.868 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74491d81:::benchmark_data_smithi043_119832_object8979:head# 2024-03-30T13:50:32.868 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:744932d1:::benchmark_data_smithi043_119832_object54286:head# 2024-03-30T13:50:32.869 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7449863a:::benchmark_data_smithi043_119832_object14069:head# 2024-03-30T13:50:32.869 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7449b3c0:::benchmark_data_smithi043_119832_object29419:head# 2024-03-30T13:50:32.869 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7449d547:::benchmark_data_smithi043_119832_object32234:head# 2024-03-30T13:50:32.869 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:744a665c:::benchmark_data_smithi043_119832_object62262:head# 2024-03-30T13:50:32.869 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:744d4bc5:::benchmark_data_smithi043_119832_object51420:head# 2024-03-30T13:50:32.869 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:744d8662:::benchmark_data_smithi043_119832_object13259:head# 2024-03-30T13:50:32.869 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:744fa93c:::benchmark_data_smithi043_119832_object84957:head# 2024-03-30T13:50:32.869 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74501c98:::benchmark_data_smithi043_119832_object79035:head# 2024-03-30T13:50:32.869 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74502db5:::benchmark_data_smithi043_119832_object31196:head# 2024-03-30T13:50:32.869 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74504f45:::benchmark_data_smithi043_119832_object84327:head# 2024-03-30T13:50:32.869 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7450bd0d:::benchmark_data_smithi043_119832_object68273:head# 2024-03-30T13:50:32.869 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7450f99c:::benchmark_data_smithi043_119832_object79502:head# 2024-03-30T13:50:32.869 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74517617:::benchmark_data_smithi043_119832_object55035:head# 2024-03-30T13:50:32.870 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74522237:::benchmark_data_smithi043_119832_object50448:head# 2024-03-30T13:50:32.870 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7452a293:::benchmark_data_smithi043_119832_object57226:head# 2024-03-30T13:50:32.870 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7454f802:::benchmark_data_smithi043_119832_object21171:head# 2024-03-30T13:50:32.870 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7457763c:::benchmark_data_smithi043_119832_object36895:head# 2024-03-30T13:50:32.870 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7458526a:::benchmark_data_smithi043_119832_object31069:head# 2024-03-30T13:50:32.870 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7458f315:::benchmark_data_smithi043_119832_object2562:head# 2024-03-30T13:50:32.870 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74593d00:::benchmark_data_smithi043_119832_object58196:head# 2024-03-30T13:50:32.870 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7459d759:::benchmark_data_smithi043_119832_object61096:head# 2024-03-30T13:50:32.870 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:745a9267:::benchmark_data_smithi043_119832_object47308:head# 2024-03-30T13:50:32.870 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:745b0bf0:::benchmark_data_smithi043_119832_object26213:head# 2024-03-30T13:50:32.870 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:745beff6:::benchmark_data_smithi043_119832_object31513:head# 2024-03-30T13:50:32.870 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:745dc8e7:::benchmark_data_smithi043_119832_object63253:head# 2024-03-30T13:50:32.870 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:745f0ba5:::benchmark_data_smithi043_119832_object72885:head# 2024-03-30T13:50:32.871 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7461526d:::benchmark_data_smithi043_119832_object39694:head# 2024-03-30T13:50:32.871 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7461b4e2:::benchmark_data_smithi043_119832_object16332:head# 2024-03-30T13:50:32.871 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74623004:::benchmark_data_smithi043_119832_object41654:head# 2024-03-30T13:50:32.871 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74631496:::benchmark_data_smithi043_119832_object53938:head# 2024-03-30T13:50:32.871 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:746418c2:::benchmark_data_smithi043_119832_object45709:head# 2024-03-30T13:50:32.871 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7464a66c:::benchmark_data_smithi043_119832_object13498:head# 2024-03-30T13:50:32.871 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7464a852:::benchmark_data_smithi043_119832_object8036:head# 2024-03-30T13:50:32.871 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7464f1ef:::benchmark_data_smithi043_119832_object42157:head# 2024-03-30T13:50:32.871 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74659957:::benchmark_data_smithi043_119832_object42671:head# 2024-03-30T13:50:32.871 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7465c81e:::benchmark_data_smithi043_119832_object75206:head# 2024-03-30T13:50:32.871 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74681802:::benchmark_data_smithi043_119832_object25804:head# 2024-03-30T13:50:32.871 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:746c0ca0:::benchmark_data_smithi043_119832_object20410:head# 2024-03-30T13:50:32.871 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:746c922b:::benchmark_data_smithi043_119832_object72228:head# 2024-03-30T13:50:32.871 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:746ca874:::benchmark_data_smithi043_119832_object56420:head# 2024-03-30T13:50:32.872 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:746d0b42:::benchmark_data_smithi043_119832_object61588:head# 2024-03-30T13:50:32.872 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:746d14b1:::benchmark_data_smithi043_119832_object22679:head# 2024-03-30T13:50:32.872 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:746d3ca3:::benchmark_data_smithi043_119832_object21064:head# 2024-03-30T13:50:32.872 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:746d6dd7:::benchmark_data_smithi043_119832_object6385:head# 2024-03-30T13:50:32.872 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:746d8fea:::benchmark_data_smithi043_119832_object67809:head# 2024-03-30T13:50:32.872 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:746dc4a6:::benchmark_data_smithi043_119832_object68078:head# 2024-03-30T13:50:32.872 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74701463:::benchmark_data_smithi043_119832_object44660:head# 2024-03-30T13:50:32.872 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7470c81e:::benchmark_data_smithi043_119832_object64956:head# 2024-03-30T13:50:32.872 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74710475:::benchmark_data_smithi043_119832_object11311:head# 2024-03-30T13:50:32.872 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74719c0e:::benchmark_data_smithi043_119832_object84835:head# 2024-03-30T13:50:32.872 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74720a2f:::benchmark_data_smithi043_119832_object7879:head# 2024-03-30T13:50:32.872 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74729397:::benchmark_data_smithi043_119832_object67772:head# 2024-03-30T13:50:32.872 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:747352e1:::benchmark_data_smithi043_119832_object58674:head# 2024-03-30T13:50:32.873 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7473d260:::benchmark_data_smithi043_119832_object1317:head# 2024-03-30T13:50:32.892 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74749ad6:::benchmark_data_smithi043_119832_object80643:head# 2024-03-30T13:50:32.893 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7474b842:::benchmark_data_smithi043_119832_object2585:head# 2024-03-30T13:50:32.893 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74752c96:::benchmark_data_smithi043_119832_object87769:head# 2024-03-30T13:50:32.893 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7475f79a:::benchmark_data_smithi043_119832_object31319:head# 2024-03-30T13:50:32.893 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74766ee3:::benchmark_data_smithi043_119832_object38434:head# 2024-03-30T13:50:32.893 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7476ed1d:::benchmark_data_smithi043_119832_object23735:head# 2024-03-30T13:50:32.893 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74773f5e:::benchmark_data_smithi043_119832_object12119:head# 2024-03-30T13:50:32.893 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7477889a:::benchmark_data_smithi043_119832_object13185:head# 2024-03-30T13:50:32.893 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7478ad32:::benchmark_data_smithi043_119832_object6879:head# 2024-03-30T13:50:32.894 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7479d42e:::benchmark_data_smithi043_119832_object30769:head# 2024-03-30T13:50:32.894 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:747bcedd:::benchmark_data_smithi043_119832_object47752:head# 2024-03-30T13:50:32.894 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:747c8c8d:::benchmark_data_smithi043_119832_object32897:head# 2024-03-30T13:50:32.894 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:747e4a78:::benchmark_data_smithi043_119832_object38329:head# 2024-03-30T13:50:32.894 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:747f08de:::benchmark_data_smithi043_119832_object29150:head# 2024-03-30T13:50:32.894 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:747f1a35:::benchmark_data_smithi043_119832_object17907:head# 2024-03-30T13:50:32.894 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:747f9129:::benchmark_data_smithi043_119832_object46363:head# 2024-03-30T13:50:32.894 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7480382e:::benchmark_data_smithi043_119832_object54895:head# 2024-03-30T13:50:32.894 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7485381f:::benchmark_data_smithi043_119832_object77034:head# 2024-03-30T13:50:32.894 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7485e486:::benchmark_data_smithi043_119832_object38066:head# 2024-03-30T13:50:32.894 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74862fdb:::benchmark_data_smithi043_119832_object32322:head# 2024-03-30T13:50:32.895 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7486399a:::benchmark_data_smithi043_119832_object11882:head# 2024-03-30T13:50:32.895 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:748742e7:::benchmark_data_smithi043_119832_object76627:head# 2024-03-30T13:50:32.895 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:748803e9:::benchmark_data_smithi043_119832_object75236:head# 2024-03-30T13:50:32.895 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7489593f:::benchmark_data_smithi043_119832_object2627:head# 2024-03-30T13:50:32.895 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74896771:::benchmark_data_smithi043_119832_object67597:head# 2024-03-30T13:50:32.895 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74896816:::benchmark_data_smithi043_119832_object7869:head# 2024-03-30T13:50:32.895 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7489bb48:::benchmark_data_smithi043_119832_object22846:head# 2024-03-30T13:50:32.895 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7489d593:::benchmark_data_smithi043_119832_object66880:head# 2024-03-30T13:50:32.895 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:748a0a3c:::benchmark_data_smithi043_119832_object41407:head# 2024-03-30T13:50:32.896 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:748aa2b6:::benchmark_data_smithi043_119832_object47290:head# 2024-03-30T13:50:32.896 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:748b3491:::benchmark_data_smithi043_119832_object30795:head# 2024-03-30T13:50:32.896 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:748c4a51:::benchmark_data_smithi043_119832_object3097:head# 2024-03-30T13:50:32.896 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:748c58d7:::benchmark_data_smithi043_119832_object79825:head# 2024-03-30T13:50:32.896 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:748e335e:::benchmark_data_smithi043_119832_object32476:head# 2024-03-30T13:50:32.896 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:749060e7:::benchmark_data_smithi043_119832_object29064:head# 2024-03-30T13:50:32.896 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74927cf5:::benchmark_data_smithi043_119832_object68432:head# 2024-03-30T13:50:32.896 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7492d3f3:::benchmark_data_smithi043_119832_object82828:head# 2024-03-30T13:50:32.896 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74944047:::benchmark_data_smithi043_119832_object47712:head# 2024-03-30T13:50:32.897 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7495fadf:::benchmark_data_smithi043_119832_object23727:head# 2024-03-30T13:50:32.897 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:749849e2:::benchmark_data_smithi043_119832_object70116:head# 2024-03-30T13:50:32.897 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74985b3a:::benchmark_data_smithi043_119832_object39506:head# 2024-03-30T13:50:32.897 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74991a7b:::benchmark_data_smithi043_119832_object79900:head# 2024-03-30T13:50:32.897 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7499cecb:::benchmark_data_smithi043_119832_object80822:head# 2024-03-30T13:50:32.897 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7499f712:::benchmark_data_smithi043_119832_object9650:head# 2024-03-30T13:50:32.897 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:749b3693:::benchmark_data_smithi043_119832_object20713:head# 2024-03-30T13:50:32.897 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:749ba314:::benchmark_data_smithi043_119832_object44560:head# 2024-03-30T13:50:32.897 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:749d7d82:::benchmark_data_smithi043_119832_object14038:head# 2024-03-30T13:50:32.898 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:749db2cf:::benchmark_data_smithi043_119832_object61427:head# 2024-03-30T13:50:32.898 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:749f1dd8:::benchmark_data_smithi043_119832_object85487:head# 2024-03-30T13:50:32.898 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:749fb908:::benchmark_data_smithi043_119832_object61685:head# 2024-03-30T13:50:32.898 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:749fe9d5:::benchmark_data_smithi043_119832_object25988:head# 2024-03-30T13:50:32.898 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74a05c44:::benchmark_data_smithi043_119832_object83948:head# 2024-03-30T13:50:32.898 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74a20d2a:::benchmark_data_smithi043_119832_object75486:head# 2024-03-30T13:50:32.898 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74a31bad:::benchmark_data_smithi043_119832_object58019:head# 2024-03-30T13:50:32.898 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74a32dfc:::benchmark_data_smithi043_119832_object50427:head# 2024-03-30T13:50:32.898 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74a57ca9:::benchmark_data_smithi043_119832_object60211:head# 2024-03-30T13:50:32.898 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74a5f050:::benchmark_data_smithi043_119832_object4704:head# 2024-03-30T13:50:32.898 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74a65eea:::benchmark_data_smithi043_119832_object70333:head# 2024-03-30T13:50:32.898 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74a67ca8:::benchmark_data_smithi043_119832_object38172:head# 2024-03-30T13:50:32.898 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74a6904b:::benchmark_data_smithi043_119832_object62906:head# 2024-03-30T13:50:32.899 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74a6b3dd:::benchmark_data_smithi043_119832_object54674:head# 2024-03-30T13:50:32.899 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74a6c7e2:::benchmark_data_smithi043_119832_object15325:head# 2024-03-30T13:50:32.899 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74a79405:::benchmark_data_smithi043_119832_object66754:head# 2024-03-30T13:50:32.899 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74a9025d:::benchmark_data_smithi043_119832_object88159:head# 2024-03-30T13:50:32.899 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74abe13f:::benchmark_data_smithi043_119832_object79320:head# 2024-03-30T13:50:32.899 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74ac5127:::benchmark_data_smithi043_119832_object34941:head# 2024-03-30T13:50:32.899 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74ace5f9:::benchmark_data_smithi043_119832_object32904:head# 2024-03-30T13:50:32.899 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74addf30:::benchmark_data_smithi043_119832_object29757:head# 2024-03-30T13:50:32.899 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74aee9e6:::benchmark_data_smithi043_119832_object6227:head# 2024-03-30T13:50:32.899 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74af1b19:::benchmark_data_smithi043_119832_object79082:head# 2024-03-30T13:50:32.899 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74af2d6c:::benchmark_data_smithi043_119832_object88395:head# 2024-03-30T13:50:32.901 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74af8755:::benchmark_data_smithi043_119832_object46390:head# 2024-03-30T13:50:32.901 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74b0328d:::benchmark_data_smithi043_119832_object62378:head# 2024-03-30T13:50:32.901 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74b0c081:::benchmark_data_smithi043_119832_object14883:head# 2024-03-30T13:50:32.901 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74b13fc7:::benchmark_data_smithi043_119832_object65252:head# 2024-03-30T13:50:32.901 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74b2aae7:::benchmark_data_smithi043_119832_object64506:head# 2024-03-30T13:50:32.901 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74b2b412:::benchmark_data_smithi043_119832_object51979:head# 2024-03-30T13:50:32.901 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74b32d11:::benchmark_data_smithi043_119832_object83588:head# 2024-03-30T13:50:32.901 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74b36e79:::benchmark_data_smithi043_119832_object54853:head# 2024-03-30T13:50:32.901 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74b47f10:::benchmark_data_smithi043_119832_object35893:head# 2024-03-30T13:50:32.902 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74b7196d:::benchmark_data_smithi043_119832_object53153:head# 2024-03-30T13:50:32.902 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74b74844:::benchmark_data_smithi043_119832_object22801:head# 2024-03-30T13:50:32.902 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74b7c2d7:::benchmark_data_smithi043_119832_object84174:head# 2024-03-30T13:50:32.902 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74b916c1:::benchmark_data_smithi043_119832_object66494:head# 2024-03-30T13:50:32.902 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74b9353e:::benchmark_data_smithi043_119832_object32526:head# 2024-03-30T13:50:32.902 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74ba566e:::benchmark_data_smithi043_119832_object58747:head# 2024-03-30T13:50:32.902 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74bb718b:::benchmark_data_smithi043_119832_object27492:head# 2024-03-30T13:50:32.902 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74bcee50:::benchmark_data_smithi043_119832_object31233:head# 2024-03-30T13:50:32.902 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74bd98da:::benchmark_data_smithi043_119832_object7977:head# 2024-03-30T13:50:32.902 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74be7f4d:::benchmark_data_smithi043_119832_object17180:head# 2024-03-30T13:50:32.902 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74bf3ef7:::benchmark_data_smithi043_119832_object70888:head# 2024-03-30T13:50:32.902 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74c0e5db:::benchmark_data_smithi043_119832_object14906:head# 2024-03-30T13:50:32.902 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74c19816:::benchmark_data_smithi043_119832_object16963:head# 2024-03-30T13:50:32.903 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74c1ee42:::benchmark_data_smithi043_119832_object3116:head# 2024-03-30T13:50:32.903 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74c1f3b8:::benchmark_data_smithi043_119832_object32528:head# 2024-03-30T13:50:32.903 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74c24d87:::benchmark_data_smithi043_119832_object14501:head# 2024-03-30T13:50:32.903 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:32.904 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74c2c873:::benchmark_data_smithi043_119832_object7944:head# 2024-03-30T13:50:32.904 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74c306b1:::benchmark_data_smithi043_119832_object27098:head# 2024-03-30T13:50:32.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:32.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:32.932 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74c3f5a7:::benchmark_data_smithi043_119832_object87147:head# 2024-03-30T13:50:32.932 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74c43398:::benchmark_data_smithi043_119832_object80207:head# 2024-03-30T13:50:32.932 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74c46838:::benchmark_data_smithi043_119832_object11601:head# 2024-03-30T13:50:32.932 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74c4ebfd:::benchmark_data_smithi043_119832_object1101:head# 2024-03-30T13:50:32.932 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74c4f651:::benchmark_data_smithi043_119832_object70257:head# 2024-03-30T13:50:32.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74c51ade:::benchmark_data_smithi043_119832_object34111:head# 2024-03-30T13:50:32.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74c54744:::benchmark_data_smithi043_119832_object85091:head# 2024-03-30T13:50:32.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74c5e10c:::benchmark_data_smithi043_119832_object78225:head# 2024-03-30T13:50:32.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74c64f71:::benchmark_data_smithi043_119832_object63893:head# 2024-03-30T13:50:32.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74c6a50e:::benchmark_data_smithi043_119832_object35444:head# 2024-03-30T13:50:32.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74c7a8a3:::benchmark_data_smithi043_119832_object27326:head# 2024-03-30T13:50:32.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74c7b8ac:::benchmark_data_smithi043_119832_object86909:head# 2024-03-30T13:50:32.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74c98dc3:::benchmark_data_smithi043_119832_object8324:head# 2024-03-30T13:50:32.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74ca4cf1:::benchmark_data_smithi043_119832_object23485:head# 2024-03-30T13:50:32.934 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74cb03df:::benchmark_data_smithi043_119832_object61455:head# 2024-03-30T13:50:32.934 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74cb15e2:::benchmark_data_smithi043_119832_object46964:head# 2024-03-30T13:50:32.934 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74cd4a6f:::benchmark_data_smithi043_119832_object41520:head# 2024-03-30T13:50:32.934 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74ce262a:::benchmark_data_smithi043_119832_object81962:head# 2024-03-30T13:50:32.934 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74ceb1e0:::benchmark_data_smithi043_119832_object41873:head# 2024-03-30T13:50:32.934 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74cec1dd:::benchmark_data_smithi043_119832_object45096:head# 2024-03-30T13:50:32.934 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74cee7d8:::benchmark_data_smithi043_119832_object13719:head# 2024-03-30T13:50:32.934 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74cefea9:::benchmark_data_smithi043_119832_object18980:head# 2024-03-30T13:50:32.934 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74d0017e:::benchmark_data_smithi043_119832_object22305:head# 2024-03-30T13:50:32.934 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74d0ab62:::benchmark_data_smithi043_119832_object20925:head# 2024-03-30T13:50:32.934 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74d0b6f7:::benchmark_data_smithi043_119832_object69139:head# 2024-03-30T13:50:32.935 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74d1bbd4:::benchmark_data_smithi043_119832_object85989:head# 2024-03-30T13:50:32.935 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74d2291f:::benchmark_data_smithi043_119832_object51940:head# 2024-03-30T13:50:32.935 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74d22ef5:::benchmark_data_smithi043_119832_object23408:head# 2024-03-30T13:50:32.935 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74d3d2ec:::benchmark_data_smithi043_119832_object85840:head# 2024-03-30T13:50:32.935 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74d3d3cd:::benchmark_data_smithi043_119832_object57104:head# 2024-03-30T13:50:32.935 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74d4437b:::benchmark_data_smithi043_119832_object79156:head# 2024-03-30T13:50:32.935 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74d49aee:::benchmark_data_smithi043_119832_object13328:head# 2024-03-30T13:50:32.935 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74d53688:::benchmark_data_smithi043_119832_object34821:head# 2024-03-30T13:50:32.935 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74d5d540:::benchmark_data_smithi043_119832_object51256:head# 2024-03-30T13:50:32.935 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74d61a20:::benchmark_data_smithi043_119832_object3824:head# 2024-03-30T13:50:32.935 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74d62faa:::benchmark_data_smithi043_119832_object74338:head# 2024-03-30T13:50:32.935 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74d6c175:::benchmark_data_smithi043_119832_object52453:head# 2024-03-30T13:50:32.935 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74d72172:::benchmark_data_smithi043_119832_object82577:head# 2024-03-30T13:50:32.936 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74d77f23:::benchmark_data_smithi043_119832_object52620:head# 2024-03-30T13:50:32.936 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74d7d1ba:::benchmark_data_smithi043_119832_object49602:head# 2024-03-30T13:50:32.936 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74db9a4d:::benchmark_data_smithi043_119832_object32326:head# 2024-03-30T13:50:32.936 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74db9bb8:::benchmark_data_smithi043_119832_object56051:head# 2024-03-30T13:50:32.936 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74dc926c:::benchmark_data_smithi043_119832_object9033:head# 2024-03-30T13:50:32.936 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74ddcb49:::benchmark_data_smithi043_119832_object82932:head# 2024-03-30T13:50:32.936 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74df0767:::benchmark_data_smithi043_119832_object63499:head# 2024-03-30T13:50:32.936 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74df13cf:::benchmark_data_smithi043_119832_object31101:head# 2024-03-30T13:50:32.936 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74df35bb:::benchmark_data_smithi043_119832_object58375:head# 2024-03-30T13:50:32.936 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74df7dd5:::benchmark_data_smithi043_119832_object84882:head# 2024-03-30T13:50:32.936 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74dff8f5:::benchmark_data_smithi043_119832_object67079:head# 2024-03-30T13:50:32.936 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74e03cd3:::benchmark_data_smithi043_119832_object71364:head# 2024-03-30T13:50:32.936 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74e0bac8:::benchmark_data_smithi043_119832_object44477:head# 2024-03-30T13:50:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74e137f9:::benchmark_data_smithi043_119832_object57992:head# 2024-03-30T13:50:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74e35f7c:::benchmark_data_smithi043_119832_object45719:head# 2024-03-30T13:50:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74e3ec1c:::benchmark_data_smithi043_119832_object19161:head# 2024-03-30T13:50:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74e51dc7:::benchmark_data_smithi043_119832_object10160:head# 2024-03-30T13:50:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74e53ef0:::benchmark_data_smithi043_119832_object39340:head# 2024-03-30T13:50:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74e5eda0:::benchmark_data_smithi043_119832_object2501:head# 2024-03-30T13:50:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74e65b6b:::benchmark_data_smithi043_119832_object24375:head# 2024-03-30T13:50:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74e690a4:::benchmark_data_smithi043_119832_object72844:head# 2024-03-30T13:50:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74e6a5ed:::benchmark_data_smithi043_119832_object5187:head# 2024-03-30T13:50:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74e731ce:::benchmark_data_smithi043_119832_object37866:head# 2024-03-30T13:50:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74e80d23:::benchmark_data_smithi043_119832_object28826:head# 2024-03-30T13:50:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74e814dc:::benchmark_data_smithi043_119832_object26808:head# 2024-03-30T13:50:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74e8bc30:::benchmark_data_smithi043_119832_object14161:head# 2024-03-30T13:50:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74ea1a8d:::benchmark_data_smithi043_119832_object16734:head# 2024-03-30T13:50:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74ea752a:::benchmark_data_smithi043_119832_object55689:head# 2024-03-30T13:50:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74ea757c:::benchmark_data_smithi043_119832_object72903:head# 2024-03-30T13:50:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74ead0c7:::benchmark_data_smithi043_119832_object12037:head# 2024-03-30T13:50:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74eb11ae:::benchmark_data_smithi043_119832_object75905:head# 2024-03-30T13:50:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74ebb5dd:::benchmark_data_smithi043_119832_object53427:head# 2024-03-30T13:50:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74ec0f2a:::benchmark_data_smithi043_119832_object8733:head# 2024-03-30T13:50:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74ece889:::benchmark_data_smithi043_119832_object18806:head# 2024-03-30T13:50:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74ed1dc6:::benchmark_data_smithi043_119832_object50658:head# 2024-03-30T13:50:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74edfea3:::benchmark_data_smithi043_119832_object85557:head# 2024-03-30T13:50:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74ef0964:::benchmark_data_smithi043_119832_object9130:head# 2024-03-30T13:50:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74ef9df0:::benchmark_data_smithi043_119832_object66774:head# 2024-03-30T13:50:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74efeec7:::benchmark_data_smithi043_119832_object7055:head# 2024-03-30T13:50:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74f08335:::benchmark_data_smithi043_119832_object16249:head# 2024-03-30T13:50:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74f08af4:::benchmark_data_smithi043_119832_object69566:head# 2024-03-30T13:50:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74f14aa4:::benchmark_data_smithi043_119832_object80020:head# 2024-03-30T13:50:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74f270c0:::benchmark_data_smithi043_119832_object61619:head# 2024-03-30T13:50:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74f317b5:::benchmark_data_smithi043_119832_object8619:head# 2024-03-30T13:50:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74f4c0a4:::benchmark_data_smithi043_119832_object56422:head# 2024-03-30T13:50:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74f58fd1:::benchmark_data_smithi043_119832_object29484:head# 2024-03-30T13:50:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74f5cd6d:::benchmark_data_smithi043_119832_object88225:head# 2024-03-30T13:50:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74f6e5c1:::benchmark_data_smithi043_119832_object8465:head# 2024-03-30T13:50:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74f77217:::benchmark_data_smithi043_119832_object44826:head# 2024-03-30T13:50:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74fac1c4:::benchmark_data_smithi043_119832_object38178:head# 2024-03-30T13:50:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74fac6f6:::benchmark_data_smithi043_119832_object25887:head# 2024-03-30T13:50:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74fdec1e:::benchmark_data_smithi043_119832_object49745:head# 2024-03-30T13:50:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74fe0023:::benchmark_data_smithi043_119832_object65750:head# 2024-03-30T13:50:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:74fe3c0f:::benchmark_data_smithi043_119832_object47085:head# 2024-03-30T13:50:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7500585c:::benchmark_data_smithi043_119832_object3698:head# 2024-03-30T13:50:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7500ee6e:::benchmark_data_smithi043_119832_object25309:head# 2024-03-30T13:50:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75015ed1:::benchmark_data_smithi043_119832_object68803:head# 2024-03-30T13:50:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:750161a6:::benchmark_data_smithi043_119832_object87447:head# 2024-03-30T13:50:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:750355b0:::benchmark_data_smithi043_119832_object34902:head# 2024-03-30T13:50:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7503844a:::benchmark_data_smithi043_119832_object67383:head# 2024-03-30T13:50:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75046f99:::benchmark_data_smithi043_119832_object17353:head# 2024-03-30T13:50:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7504d303:::benchmark_data_smithi043_119832_object46757:head# 2024-03-30T13:50:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7504dfdf:::benchmark_data_smithi043_119832_object58623:head# 2024-03-30T13:50:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7505ebd7:::benchmark_data_smithi043_119832_object33488:head# 2024-03-30T13:50:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7506b915:::benchmark_data_smithi043_119832_object25790:head# 2024-03-30T13:50:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7506be30:::benchmark_data_smithi043_119832_object32562:head# 2024-03-30T13:50:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7506ec29:::benchmark_data_smithi043_119832_object13190:head# 2024-03-30T13:50:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75075761:::benchmark_data_smithi043_119832_object80917:head# 2024-03-30T13:50:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75079fe1:::benchmark_data_smithi043_119832_object40266:head# 2024-03-30T13:50:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7507c34e:::benchmark_data_smithi043_119832_object21144:head# 2024-03-30T13:50:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75080175:::benchmark_data_smithi043_119832_object18788:head# 2024-03-30T13:50:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75080819:::benchmark_data_smithi043_119832_object72793:head# 2024-03-30T13:50:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7508f4f0:::benchmark_data_smithi043_119832_object74122:head# 2024-03-30T13:50:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:750997db:::benchmark_data_smithi043_119832_object32987:head# 2024-03-30T13:50:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7509e7b8:::benchmark_data_smithi043_119832_object60309:head# 2024-03-30T13:50:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:750b02e6:::benchmark_data_smithi043_119832_object85915:head# 2024-03-30T13:50:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:750b4043:::benchmark_data_smithi043_119832_object22415:head# 2024-03-30T13:50:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:750be059:::benchmark_data_smithi043_119832_object35528:head# 2024-03-30T13:50:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:750bf078:::benchmark_data_smithi043_119832_object13260:head# 2024-03-30T13:50:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:750d4278:::benchmark_data_smithi043_119832_object69724:head# 2024-03-30T13:50:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:750d8436:::benchmark_data_smithi043_119832_object4416:head# 2024-03-30T13:50:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:750ddd97:::benchmark_data_smithi043_119832_object52572:head# 2024-03-30T13:50:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:750df975:::benchmark_data_smithi043_119832_object49808:head# 2024-03-30T13:50:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:750e1696:::benchmark_data_smithi043_119832_object44439:head# 2024-03-30T13:50:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:750f18c4:::benchmark_data_smithi043_119832_object76763:head# 2024-03-30T13:50:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:750f86f9:::benchmark_data_smithi043_119832_object78355:head# 2024-03-30T13:50:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75128c20:::benchmark_data_smithi043_119832_object51378:head# 2024-03-30T13:50:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7514346b:::benchmark_data_smithi043_119832_object87595:head# 2024-03-30T13:50:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7514e32d:::benchmark_data_smithi043_119832_object74112:head# 2024-03-30T13:50:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75170f8b:::benchmark_data_smithi043_119832_object12102:head# 2024-03-30T13:50:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7518d212:::benchmark_data_smithi043_119832_object37597:head# 2024-03-30T13:50:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7518ddeb:::benchmark_data_smithi043_119832_object40193:head# 2024-03-30T13:50:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:751a2db0:::benchmark_data_smithi043_119832_object59111:head# 2024-03-30T13:50:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:751a5138:::benchmark_data_smithi043_119832_object76919:head# 2024-03-30T13:50:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:751c207b:::benchmark_data_smithi043_119832_object71370:head# 2024-03-30T13:50:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:751c575c:::benchmark_data_smithi043_119832_object27667:head# 2024-03-30T13:50:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:751d2c2c:::benchmark_data_smithi043_119832_object1511:head# 2024-03-30T13:50:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:751dba47:::benchmark_data_smithi043_119832_object26831:head# 2024-03-30T13:50:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:751fe2b0:::benchmark_data_smithi043_119832_object16065:head# 2024-03-30T13:50:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:751ff981:::benchmark_data_smithi043_119832_object53888:head# 2024-03-30T13:50:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:752019a1:::benchmark_data_smithi043_119832_object17067:head# 2024-03-30T13:50:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:752055bd:::benchmark_data_smithi043_119832_object77298:head# 2024-03-30T13:50:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7521d60f:::benchmark_data_smithi043_119832_object71428:head# 2024-03-30T13:50:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7522413c:::benchmark_data_smithi043_119832_object19472:head# 2024-03-30T13:50:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7524e4c2:::benchmark_data_smithi043_119832_object21782:head# 2024-03-30T13:50:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7525bdd5:::benchmark_data_smithi043_119832_object18080:head# 2024-03-30T13:50:32.976 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:32 smithi113 ceph-mon[34341]: pgmap v309: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 840 op/s; 41791/334356 objects degraded (12.499%) 2024-03-30T13:50:32.976 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:32 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 41791/334356 objects degraded (12.499%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:32.976 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:32 smithi113 ceph-mon[34341]: osdmap e176: 8 total, 7 up, 8 in 2024-03-30T13:50:32.988 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7526357c:::benchmark_data_smithi043_119832_object39325:head# 2024-03-30T13:50:32.988 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75263d32:::benchmark_data_smithi043_119832_object43590:head# 2024-03-30T13:50:32.988 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:752707d0:::benchmark_data_smithi043_119832_object64492:head# 2024-03-30T13:50:32.988 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7527e9e8:::benchmark_data_smithi043_119832_object78187:head# 2024-03-30T13:50:32.988 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7528f14c:::benchmark_data_smithi043_119832_object60827:head# 2024-03-30T13:50:32.988 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75292d82:::benchmark_data_smithi043_119832_object27582:head# 2024-03-30T13:50:32.988 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75296988:::benchmark_data_smithi043_119832_object82845:head# 2024-03-30T13:50:32.988 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:752a7007:::benchmark_data_smithi043_119832_object55772:head# 2024-03-30T13:50:32.988 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:752ba09a:::benchmark_data_smithi043_119832_object6471:head# 2024-03-30T13:50:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:752bd9f4:::benchmark_data_smithi043_119832_object39440:head# 2024-03-30T13:50:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:752be8e8:::benchmark_data_smithi043_119832_object84131:head# 2024-03-30T13:50:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:752dbd55:::benchmark_data_smithi043_119832_object29441:head# 2024-03-30T13:50:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:752f52af:::benchmark_data_smithi043_119832_object5340:head# 2024-03-30T13:50:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:752fcc60:::benchmark_data_smithi043_119832_object71494:head# 2024-03-30T13:50:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:752ff68b:::benchmark_data_smithi043_119832_object72794:head# 2024-03-30T13:50:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:753103a4:::benchmark_data_smithi043_119832_object76893:head# 2024-03-30T13:50:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75320db0:::benchmark_data_smithi043_119832_object78811:head# 2024-03-30T13:50:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:753222aa:::benchmark_data_smithi043_119832_object74809:head# 2024-03-30T13:50:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75329b72:::benchmark_data_smithi043_119832_object66915:head# 2024-03-30T13:50:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7532b309:::benchmark_data_smithi043_119832_object18268:head# 2024-03-30T13:50:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:753444bf:::benchmark_data_smithi043_119832_object12576:head# 2024-03-30T13:50:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:753519d5:::benchmark_data_smithi043_119832_object31118:head# 2024-03-30T13:50:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75354f64:::benchmark_data_smithi043_119832_object49789:head# 2024-03-30T13:50:32.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75357785:::benchmark_data_smithi043_119832_object8449:head# 2024-03-30T13:50:32.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7537a950:::benchmark_data_smithi043_119832_object87936:head# 2024-03-30T13:50:32.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75382af5:::benchmark_data_smithi043_119832_object26576:head# 2024-03-30T13:50:32.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7538cbdf:::benchmark_data_smithi043_119832_object37515:head# 2024-03-30T13:50:32.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7539bab0:::benchmark_data_smithi043_119832_object79400:head# 2024-03-30T13:50:32.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:753a0e99:::benchmark_data_smithi043_119832_object3694:head# 2024-03-30T13:50:32.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:753a19d3:::benchmark_data_smithi043_119832_object64898:head# 2024-03-30T13:50:32.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:753a4576:::benchmark_data_smithi043_119832_object39443:head# 2024-03-30T13:50:32.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:753a840b:::benchmark_data_smithi043_119832_object80101:head# 2024-03-30T13:50:32.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:753b4a73:::benchmark_data_smithi043_119832_object23396:head# 2024-03-30T13:50:32.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:753bedb7:::benchmark_data_smithi043_119832_object78294:head# 2024-03-30T13:50:32.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:753c002f:::benchmark_data_smithi043_119832_object15082:head# 2024-03-30T13:50:32.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:753c0b4e:::benchmark_data_smithi043_119832_object40336:head# 2024-03-30T13:50:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:753c198d:::benchmark_data_smithi043_119832_object72806:head# 2024-03-30T13:50:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:753d5a5b:::benchmark_data_smithi043_119832_object12629:head# 2024-03-30T13:50:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:753d889c:::benchmark_data_smithi043_119832_object64539:head# 2024-03-30T13:50:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:753e9ce2:::benchmark_data_smithi043_119832_object10828:head# 2024-03-30T13:50:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:753f3624:::benchmark_data_smithi043_119832_object77040:head# 2024-03-30T13:50:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:753f86f7:::benchmark_data_smithi043_119832_object17340:head# 2024-03-30T13:50:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75403c38:::benchmark_data_smithi043_119832_object82272:head# 2024-03-30T13:50:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7540456f:::benchmark_data_smithi043_119832_object8756:head# 2024-03-30T13:50:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75405e6d:::benchmark_data_smithi043_119832_object15487:head# 2024-03-30T13:50:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75406065:::benchmark_data_smithi043_119832_object33721:head# 2024-03-30T13:50:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7541f871:::benchmark_data_smithi043_119832_object16936:head# 2024-03-30T13:50:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:754279e8:::benchmark_data_smithi043_119832_object59216:head# 2024-03-30T13:50:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7542e1ec:::benchmark_data_smithi043_119832_object45541:head# 2024-03-30T13:50:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7542ec25:::benchmark_data_smithi043_119832_object46591:head# 2024-03-30T13:50:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75434f68:::benchmark_data_smithi043_119832_object48539:head# 2024-03-30T13:50:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7544a3f3:::benchmark_data_smithi043_119832_object81632:head# 2024-03-30T13:50:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7544ecb7:::benchmark_data_smithi043_119832_object86331:head# 2024-03-30T13:50:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:754591ef:::benchmark_data_smithi043_119832_object40037:head# 2024-03-30T13:50:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7545cfa6:::benchmark_data_smithi043_119832_object12971:head# 2024-03-30T13:50:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7546a1d2:::benchmark_data_smithi043_119832_object78452:head# 2024-03-30T13:50:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75479e18:::benchmark_data_smithi043_119832_object59300:head# 2024-03-30T13:50:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7547da53:::benchmark_data_smithi043_119832_object64337:head# 2024-03-30T13:50:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:754858e8:::benchmark_data_smithi043_119832_object84837:head# 2024-03-30T13:50:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75491cd2:::benchmark_data_smithi043_119832_object5407:head# 2024-03-30T13:50:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7549c8a9:::benchmark_data_smithi043_119832_object46653:head# 2024-03-30T13:50:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:754b15ec:::benchmark_data_smithi043_119832_object14880:head# 2024-03-30T13:50:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:754b3358:::benchmark_data_smithi043_119832_object31644:head# 2024-03-30T13:50:32.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:754b3919:::benchmark_data_smithi043_119832_object66482:head# 2024-03-30T13:50:32.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:754c29d0:::benchmark_data_smithi043_119832_object87886:head# 2024-03-30T13:50:32.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:754e84da:::benchmark_data_smithi043_119832_object56838:head# 2024-03-30T13:50:32.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75517ae8:::benchmark_data_smithi043_119832_object11263:head# 2024-03-30T13:50:32.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7551fcf7:::benchmark_data_smithi043_119832_object3284:head# 2024-03-30T13:50:32.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75538484:::benchmark_data_smithi043_119832_object85165:head# 2024-03-30T13:50:32.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7553cd8d:::benchmark_data_smithi043_119832_object61095:head# 2024-03-30T13:50:32.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7554131b:::benchmark_data_smithi043_119832_object12858:head# 2024-03-30T13:50:32.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75541bf0:::benchmark_data_smithi043_119832_object67168:head# 2024-03-30T13:50:32.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7554976c:::benchmark_data_smithi043_119832_object85415:head# 2024-03-30T13:50:32.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7554be85:::benchmark_data_smithi043_119832_object81374:head# 2024-03-30T13:50:32.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7556135c:::benchmark_data_smithi043_119832_object79107:head# 2024-03-30T13:50:32.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75565a2a:::benchmark_data_smithi043_119832_object53013:head# 2024-03-30T13:50:32.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:755864a2:::benchmark_data_smithi043_119832_object22379:head# 2024-03-30T13:50:32.994 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:755874c8:::benchmark_data_smithi043_119832_object72556:head# 2024-03-30T13:50:32.994 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75592d9d:::benchmark_data_smithi043_119832_object31848:head# 2024-03-30T13:50:32.994 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:755a2941:::benchmark_data_smithi043_119832_object49643:head# 2024-03-30T13:50:32.994 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:755a6c11:::benchmark_data_smithi043_119832_object35890:head# 2024-03-30T13:50:32.994 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:755a74a6:::benchmark_data_smithi043_119832_object85908:head# 2024-03-30T13:50:32.994 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:755ac514:::benchmark_data_smithi043_119832_object60196:head# 2024-03-30T13:50:32.994 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:755b62e7:::benchmark_data_smithi043_119832_object9328:head# 2024-03-30T13:50:32.994 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:755c92ce:::benchmark_data_smithi043_119832_object85606:head# 2024-03-30T13:50:32.994 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:755d209b:::benchmark_data_smithi043_119832_object65882:head# 2024-03-30T13:50:32.994 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:755d44b1:::benchmark_data_smithi043_119832_object45197:head# 2024-03-30T13:50:32.994 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:755db233:::benchmark_data_smithi043_119832_object54243:head# 2024-03-30T13:50:32.994 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:755f7db2:::benchmark_data_smithi043_119832_object70637:head# 2024-03-30T13:50:32.995 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:755fc5f2:::benchmark_data_smithi043_119832_object57713:head# 2024-03-30T13:50:32.995 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:756215ef:::benchmark_data_smithi043_119832_object57516:head# 2024-03-30T13:50:32.995 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75625266:::benchmark_data_smithi043_119832_object4905:head# 2024-03-30T13:50:32.995 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75643e33:::benchmark_data_smithi043_119832_object60555:head# 2024-03-30T13:50:32.995 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75648f5b:::benchmark_data_smithi043_119832_object6264:head# 2024-03-30T13:50:32.995 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:756581d0:::benchmark_data_smithi043_119832_object38870:head# 2024-03-30T13:50:32.995 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75663ad1:::benchmark_data_smithi043_119832_object51812:head# 2024-03-30T13:50:32.995 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:756a7a29:::benchmark_data_smithi043_119832_object35747:head# 2024-03-30T13:50:32.995 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:756af728:::benchmark_data_smithi043_119832_object80014:head# 2024-03-30T13:50:32.996 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:756c02f8:::benchmark_data_smithi043_119832_object9890:head# 2024-03-30T13:50:32.996 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:756c77a4:::benchmark_data_smithi043_119832_object12999:head# 2024-03-30T13:50:32.996 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:756c7eac:::benchmark_data_smithi043_119832_object65744:head# 2024-03-30T13:50:32.996 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:756c9fb2:::benchmark_data_smithi043_119832_object49228:head# 2024-03-30T13:50:32.996 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:756d5cc9:::benchmark_data_smithi043_119832_object8092:head# 2024-03-30T13:50:32.996 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:756db4a1:::benchmark_data_smithi043_119832_object35811:head# 2024-03-30T13:50:32.996 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:756f2bbb:::benchmark_data_smithi043_119832_object11770:head# 2024-03-30T13:50:32.996 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:756f393e:::benchmark_data_smithi043_119832_object925:head# 2024-03-30T13:50:32.996 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:756f7f09:::benchmark_data_smithi043_119832_object13066:head# 2024-03-30T13:50:32.996 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:756feb1f:::benchmark_data_smithi043_119832_object11254:head# 2024-03-30T13:50:32.996 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7571a3c6:::benchmark_data_smithi043_119832_object4431:head# 2024-03-30T13:50:32.996 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7571a4aa:::benchmark_data_smithi043_119832_object54049:head# 2024-03-30T13:50:32.996 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75722098:::benchmark_data_smithi043_119832_object8996:head# 2024-03-30T13:50:32.997 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75729b64:::benchmark_data_smithi043_119832_object37096:head# 2024-03-30T13:50:32.997 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:757323bb:::benchmark_data_smithi043_119832_object68680:head# 2024-03-30T13:50:32.997 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7574223e:::benchmark_data_smithi043_119832_object71603:head# 2024-03-30T13:50:32.997 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7574839e:::benchmark_data_smithi043_119832_object82856:head# 2024-03-30T13:50:32.997 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7574aeb1:::benchmark_data_smithi043_119832_object11108:head# 2024-03-30T13:50:32.997 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:757514e0:::benchmark_data_smithi043_119832_object28425:head# 2024-03-30T13:50:32.997 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7575d996:::benchmark_data_smithi043_119832_object10355:head# 2024-03-30T13:50:32.997 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:757613f7:::benchmark_data_smithi043_119832_object70583:head# 2024-03-30T13:50:32.997 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75765ea0:::benchmark_data_smithi043_119832_object73394:head# 2024-03-30T13:50:32.997 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:757799c9:::benchmark_data_smithi043_119832_object54820:head# 2024-03-30T13:50:32.997 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7577d9a1:::benchmark_data_smithi043_119832_object3552:head# 2024-03-30T13:50:32.997 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7577f4ee:::benchmark_data_smithi043_119832_object16523:head# 2024-03-30T13:50:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:757821b5:::benchmark_data_smithi043_119832_object14676:head# 2024-03-30T13:50:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:757842ab:::benchmark_data_smithi043_119832_object45988:head# 2024-03-30T13:50:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7578af54:::benchmark_data_smithi043_119832_object55736:head# 2024-03-30T13:50:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:757955a7:::benchmark_data_smithi043_119832_object54927:head# 2024-03-30T13:50:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75796105:::benchmark_data_smithi043_119832_object44110:head# 2024-03-30T13:50:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:757a5d03:::benchmark_data_smithi043_119832_object84377:head# 2024-03-30T13:50:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:757bd389:::benchmark_data_smithi043_119832_object158:head# 2024-03-30T13:50:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:757c4226:::benchmark_data_smithi043_119832_object73246:head# 2024-03-30T13:50:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:757c8f1e:::benchmark_data_smithi043_119832_object52760:head# 2024-03-30T13:50:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:757d51f6:::benchmark_data_smithi043_119832_object44319:head# 2024-03-30T13:50:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:757e0eae:::benchmark_data_smithi043_119832_object28073:head# 2024-03-30T13:50:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:757e48e0:::benchmark_data_smithi043_119832_object81186:head# 2024-03-30T13:50:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:757ea339:::benchmark_data_smithi043_119832_object13991:head# 2024-03-30T13:50:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75801faf:::benchmark_data_smithi043_119832_object6278:head# 2024-03-30T13:50:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7580de1f:::benchmark_data_smithi043_119832_object35858:head# 2024-03-30T13:50:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75820343:::benchmark_data_smithi043_119832_object12177:head# 2024-03-30T13:50:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75829d9e:::benchmark_data_smithi043_119832_object9077:head# 2024-03-30T13:50:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75834473:::benchmark_data_smithi043_119832_object35686:head# 2024-03-30T13:50:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:758485b9:::benchmark_data_smithi043_119832_object29182:head# 2024-03-30T13:50:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7585cde2:::benchmark_data_smithi043_119832_object9647:head# 2024-03-30T13:50:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7585e321:::benchmark_data_smithi043_119832_object86237:head# 2024-03-30T13:50:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7586688f:::benchmark_data_smithi043_119832_object35373:head# 2024-03-30T13:50:33.000 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75867244:::benchmark_data_smithi043_119832_object68168:head# 2024-03-30T13:50:33.000 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7586d909:::benchmark_data_smithi043_119832_object4854:head# 2024-03-30T13:50:33.000 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:758747ba:::benchmark_data_smithi043_119832_object58480:head# 2024-03-30T13:50:33.000 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7587da1e:::benchmark_data_smithi043_119832_object15790:head# 2024-03-30T13:50:33.000 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7589242f:::benchmark_data_smithi043_119832_object43459:head# 2024-03-30T13:50:33.000 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7589826e:::benchmark_data_smithi043_119832_object6615:head# 2024-03-30T13:50:33.000 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7589dcac:::benchmark_data_smithi043_119832_object85301:head# 2024-03-30T13:50:33.000 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:758a2b59:::benchmark_data_smithi043_119832_object42885:head# 2024-03-30T13:50:33.000 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:758a907a:::benchmark_data_smithi043_119832_object9730:head# 2024-03-30T13:50:33.000 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:758b39a3:::benchmark_data_smithi043_119832_object65224:head# 2024-03-30T13:50:33.000 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:758bc3dd:::benchmark_data_smithi043_119832_object42716:head# 2024-03-30T13:50:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:758d8d54:::benchmark_data_smithi043_119832_object45445:head# 2024-03-30T13:50:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:758f455b:::benchmark_data_smithi043_119832_object2549:head# 2024-03-30T13:50:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:758f7dde:::benchmark_data_smithi043_119832_object29738:head# 2024-03-30T13:50:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7590c8b6:::benchmark_data_smithi043_119832_object82074:head# 2024-03-30T13:50:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75927097:::benchmark_data_smithi043_119832_object57473:head# 2024-03-30T13:50:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75929476:::benchmark_data_smithi043_119832_object31881:head# 2024-03-30T13:50:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75932034:::benchmark_data_smithi043_119832_object71892:head# 2024-03-30T13:50:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75940b25:::benchmark_data_smithi043_119832_object1488:head# 2024-03-30T13:50:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:759428cc:::benchmark_data_smithi043_119832_object54963:head# 2024-03-30T13:50:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7594c04e:::benchmark_data_smithi043_119832_object61979:head# 2024-03-30T13:50:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:759556b6:::benchmark_data_smithi043_119832_object76341:head# 2024-03-30T13:50:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7595a000:::benchmark_data_smithi043_119832_object81372:head# 2024-03-30T13:50:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7595b6b4:::benchmark_data_smithi043_119832_object50192:head# 2024-03-30T13:50:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7595ea76:::benchmark_data_smithi043_119832_object20048:head# 2024-03-30T13:50:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75960f8d:::benchmark_data_smithi043_119832_object85278:head# 2024-03-30T13:50:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75961ccc:::benchmark_data_smithi043_119832_object208:head# 2024-03-30T13:50:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7597606a:::benchmark_data_smithi043_119832_object9587:head# 2024-03-30T13:50:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7597af4c:::benchmark_data_smithi043_119832_object10067:head# 2024-03-30T13:50:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75989358:::benchmark_data_smithi043_119832_object23824:head# 2024-03-30T13:50:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7598bb17:::benchmark_data_smithi043_119832_object28363:head# 2024-03-30T13:50:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7598c0c5:::benchmark_data_smithi043_119832_object40777:head# 2024-03-30T13:50:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7598d61a:::benchmark_data_smithi043_119832_object77076:head# 2024-03-30T13:50:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:759bddd6:::benchmark_data_smithi043_119832_object20655:head# 2024-03-30T13:50:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:759bdf68:::benchmark_data_smithi043_119832_object28528:head# 2024-03-30T13:50:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:759ca551:::benchmark_data_smithi043_119832_object33936:head# 2024-03-30T13:50:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:759ca9f8:::benchmark_data_smithi043_119832_object32213:head# 2024-03-30T13:50:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:759d23f5:::benchmark_data_smithi043_119832_object53479:head# 2024-03-30T13:50:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:759d3954:::benchmark_data_smithi043_119832_object3505:head# 2024-03-30T13:50:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:759f45db:::benchmark_data_smithi043_119832_object15658:head# 2024-03-30T13:50:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75a26e1e:::benchmark_data_smithi043_119832_object36283:head# 2024-03-30T13:50:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75a42721:::benchmark_data_smithi043_119832_object83033:head# 2024-03-30T13:50:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75a4e937:::benchmark_data_smithi043_119832_object10452:head# 2024-03-30T13:50:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75a5cb65:::benchmark_data_smithi043_119832_object66807:head# 2024-03-30T13:50:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75a5cbf9:::benchmark_data_smithi043_119832_object52361:head# 2024-03-30T13:50:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75a6a1e9:::benchmark_data_smithi043_119832_object9600:head# 2024-03-30T13:50:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75a769b2:::benchmark_data_smithi043_119832_object10648:head# 2024-03-30T13:50:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75a81d9d:::benchmark_data_smithi043_119832_object14119:head# 2024-03-30T13:50:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75a83aa6:::benchmark_data_smithi043_119832_object72588:head# 2024-03-30T13:50:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75a90cf2:::benchmark_data_smithi043_119832_object24305:head# 2024-03-30T13:50:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75a9adc1:::benchmark_data_smithi043_119832_object73750:head# 2024-03-30T13:50:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75ab7d7d:::benchmark_data_smithi043_119832_object45454:head# 2024-03-30T13:50:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75abc499:::benchmark_data_smithi043_119832_object66358:head# 2024-03-30T13:50:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75abd3c2:::benchmark_data_smithi043_119832_object57252:head# 2024-03-30T13:50:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75ac0f88:::benchmark_data_smithi043_119832_object38000:head# 2024-03-30T13:50:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75ad8d4b:::benchmark_data_smithi043_119832_object27530:head# 2024-03-30T13:50:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75b0021d:::benchmark_data_smithi043_119832_object31914:head# 2024-03-30T13:50:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75b03217:::benchmark_data_smithi043_119832_object27086:head# 2024-03-30T13:50:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75b03c78:::benchmark_data_smithi043_119832_object56602:head# 2024-03-30T13:50:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75b04446:::benchmark_data_smithi043_119832_object70006:head# 2024-03-30T13:50:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75b0cb01:::benchmark_data_smithi043_119832_object44538:head# 2024-03-30T13:50:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75b0dfea:::benchmark_data_smithi043_119832_object50577:head# 2024-03-30T13:50:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75b0f546:::benchmark_data_smithi043_119832_object41136:head# 2024-03-30T13:50:33.006 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75b1396e:::benchmark_data_smithi043_119832_object56202:head# 2024-03-30T13:50:33.006 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75b3f4d6:::benchmark_data_smithi043_119832_object31540:head# 2024-03-30T13:50:33.006 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75b3f934:::benchmark_data_smithi043_119832_object23700:head# 2024-03-30T13:50:33.006 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75b47c37:::benchmark_data_smithi043_119832_object52624:head# 2024-03-30T13:50:33.006 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75b76082:::benchmark_data_smithi043_119832_object38583:head# 2024-03-30T13:50:33.006 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75b84ad6:::benchmark_data_smithi043_119832_object35865:head# 2024-03-30T13:50:33.006 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75b94524:::benchmark_data_smithi043_119832_object58759:head# 2024-03-30T13:50:33.006 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75bd72f7:::benchmark_data_smithi043_119832_object57154:head# 2024-03-30T13:50:33.006 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75bdff34:::benchmark_data_smithi043_119832_object42527:head# 2024-03-30T13:50:33.007 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75be08dd:::benchmark_data_smithi043_119832_object52689:head# 2024-03-30T13:50:33.007 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75be2653:::benchmark_data_smithi043_119832_object66322:head# 2024-03-30T13:50:33.007 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75c02a63:::benchmark_data_smithi043_119832_object6052:head# 2024-03-30T13:50:33.007 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75c19149:::benchmark_data_smithi043_119832_object40439:head# 2024-03-30T13:50:33.007 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75c28d4d:::benchmark_data_smithi043_119832_object18943:head# 2024-03-30T13:50:33.007 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75c31f87:::benchmark_data_smithi043_119832_object10893:head# 2024-03-30T13:50:33.007 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75c33068:::benchmark_data_smithi043_119832_object46543:head# 2024-03-30T13:50:33.007 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75c50bb3:::benchmark_data_smithi043_119832_object86589:head# 2024-03-30T13:50:33.007 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75c50dec:::benchmark_data_smithi043_119832_object37379:head# 2024-03-30T13:50:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75c5666e:::benchmark_data_smithi043_119832_object68638:head# 2024-03-30T13:50:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75c5d8bf:::benchmark_data_smithi043_119832_object35474:head# 2024-03-30T13:50:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75c71503:::benchmark_data_smithi043_119832_object17241:head# 2024-03-30T13:50:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75c8024d:::benchmark_data_smithi043_119832_object76546:head# 2024-03-30T13:50:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75c868fb:::benchmark_data_smithi043_119832_object68411:head# 2024-03-30T13:50:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75c89c78:::benchmark_data_smithi043_119832_object49264:head# 2024-03-30T13:50:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75c8aa0e:::benchmark_data_smithi043_119832_object29877:head# 2024-03-30T13:50:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75c9fe47:::benchmark_data_smithi043_119832_object87608:head# 2024-03-30T13:50:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75cb340e:::benchmark_data_smithi043_119832_object17975:head# 2024-03-30T13:50:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75cbebdd:::benchmark_data_smithi043_119832_object3774:head# 2024-03-30T13:50:33.009 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 78 16 115748 115732 92.7182 61 0.00918385 0.010779 2024-03-30T13:50:33.011 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:33.018 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75cd9e91:::benchmark_data_smithi043_119832_object35407:head# 2024-03-30T13:50:33.018 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75d08b0e:::benchmark_data_smithi043_119832_object51081:head# 2024-03-30T13:50:33.018 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75d0aa30:::benchmark_data_smithi043_119832_object51148:head# 2024-03-30T13:50:33.018 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75d1a2cd:::benchmark_data_smithi043_119832_object6347:head# 2024-03-30T13:50:33.018 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75d21dc9:::benchmark_data_smithi043_119832_object62554:head# 2024-03-30T13:50:33.018 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75d2cff9:::benchmark_data_smithi043_119832_object68659:head# 2024-03-30T13:50:33.018 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75d31824:::benchmark_data_smithi043_119832_object39294:head# 2024-03-30T13:50:33.018 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75d410ad:::benchmark_data_smithi043_119832_object85665:head# 2024-03-30T13:50:33.018 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75d69a34:::benchmark_data_smithi043_119832_object15130:head# 2024-03-30T13:50:33.018 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75d76580:::benchmark_data_smithi043_119832_object51439:head# 2024-03-30T13:50:33.018 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75d7f0b3:::benchmark_data_smithi043_119832_object32388:head# 2024-03-30T13:50:33.018 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75d7f305:::benchmark_data_smithi043_119832_object17349:head# 2024-03-30T13:50:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75d95d72:::benchmark_data_smithi043_119832_object44545:head# 2024-03-30T13:50:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75d99e5c:::benchmark_data_smithi043_119832_object82841:head# 2024-03-30T13:50:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75db38e6:::benchmark_data_smithi043_119832_object18766:head# 2024-03-30T13:50:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75dcb176:::benchmark_data_smithi043_119832_object72050:head# 2024-03-30T13:50:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75dcc7d2:::benchmark_data_smithi043_119832_object22650:head# 2024-03-30T13:50:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75df1d11:::benchmark_data_smithi043_119832_object81432:head# 2024-03-30T13:50:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75dffbe7:::benchmark_data_smithi043_119832_object44513:head# 2024-03-30T13:50:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75e001c9:::benchmark_data_smithi043_119832_object31135:head# 2024-03-30T13:50:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75e1a293:::benchmark_data_smithi043_119832_object53491:head# 2024-03-30T13:50:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75e1b55a:::benchmark_data_smithi043_119832_object75313:head# 2024-03-30T13:50:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75e29355:::benchmark_data_smithi043_119832_object13101:head# 2024-03-30T13:50:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75e33a10:::benchmark_data_smithi043_119832_object85561:head# 2024-03-30T13:50:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75e43ae5:::benchmark_data_smithi043_119832_object88523:head# 2024-03-30T13:50:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75e5e4ce:::benchmark_data_smithi043_119832_object58679:head# 2024-03-30T13:50:33.020 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75e6944c:::benchmark_data_smithi043_119832_object66616:head# 2024-03-30T13:50:33.020 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75e9bb0e:::benchmark_data_smithi043_119832_object67061:head# 2024-03-30T13:50:33.020 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75ea360c:::benchmark_data_smithi043_119832_object82613:head# 2024-03-30T13:50:33.020 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75ea82a3:::benchmark_data_smithi043_119832_object69693:head# 2024-03-30T13:50:33.020 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75eb8654:::benchmark_data_smithi043_119832_object69247:head# 2024-03-30T13:50:33.020 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75ebe66e:::benchmark_data_smithi043_119832_object66780:head# 2024-03-30T13:50:33.020 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75ec390f:::benchmark_data_smithi043_119832_object66260:head# 2024-03-30T13:50:33.020 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75ec3fdd:::benchmark_data_smithi043_119832_object39573:head# 2024-03-30T13:50:33.020 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75ecb146:::benchmark_data_smithi043_119832_object84283:head# 2024-03-30T13:50:33.020 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75ecbbf6:::benchmark_data_smithi043_119832_object39167:head# 2024-03-30T13:50:33.020 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75eda286:::benchmark_data_smithi043_119832_object27000:head# 2024-03-30T13:50:33.020 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75edb8a2:::benchmark_data_smithi043_119832_object9419:head# 2024-03-30T13:50:33.020 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75ee94b0:::benchmark_data_smithi043_119832_object46552:head# 2024-03-30T13:50:33.021 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75eefbc5:::benchmark_data_smithi043_119832_object66674:head# 2024-03-30T13:50:33.021 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75efaf27:::benchmark_data_smithi043_119832_object57843:head# 2024-03-30T13:50:33.021 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75f0a837:::benchmark_data_smithi043_119832_object34188:head# 2024-03-30T13:50:33.021 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75f0c60f:::benchmark_data_smithi043_119832_object2158:head# 2024-03-30T13:50:33.021 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75f17bf6:::benchmark_data_smithi043_119832_object51838:head# 2024-03-30T13:50:33.021 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75f45946:::benchmark_data_smithi043_119832_object82962:head# 2024-03-30T13:50:33.021 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75f6ebda:::benchmark_data_smithi043_119832_object82198:head# 2024-03-30T13:50:33.021 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75f779e4:::benchmark_data_smithi043_119832_object65330:head# 2024-03-30T13:50:33.021 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75f7afee:::benchmark_data_smithi043_119832_object71414:head# 2024-03-30T13:50:33.021 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75fa3e18:::benchmark_data_smithi043_119832_object65488:head# 2024-03-30T13:50:33.021 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75fa9b35:::benchmark_data_smithi043_119832_object34832:head# 2024-03-30T13:50:33.021 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75fb708b:::benchmark_data_smithi043_119832_object32920:head# 2024-03-30T13:50:33.021 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75fbdaaa:::benchmark_data_smithi043_119832_object75624:head# 2024-03-30T13:50:33.021 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75fc0ca8:::benchmark_data_smithi043_119832_object79188:head# 2024-03-30T13:50:33.022 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75fc7291:::benchmark_data_smithi043_119832_object34801:head# 2024-03-30T13:50:33.022 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75fd2b79:::benchmark_data_smithi043_119832_object88409:head# 2024-03-30T13:50:33.022 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75fdb73a:::benchmark_data_smithi043_119832_object49425:head# 2024-03-30T13:50:33.022 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75fdcee2:::benchmark_data_smithi043_119832_object56039:head# 2024-03-30T13:50:33.022 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75fe7cff:::benchmark_data_smithi043_119832_object44314:head# 2024-03-30T13:50:33.022 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75fe970e:::benchmark_data_smithi043_119832_object40974:head# 2024-03-30T13:50:33.022 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75fefc1d:::benchmark_data_smithi043_119832_object34353:head# 2024-03-30T13:50:33.022 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75ff8147:::benchmark_data_smithi043_119832_object65381:head# 2024-03-30T13:50:33.022 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:75ffa5f6:::benchmark_data_smithi043_119832_object17685:head# 2024-03-30T13:50:33.022 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7600bded:::benchmark_data_smithi043_119832_object30051:head# 2024-03-30T13:50:33.022 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:760139b4:::benchmark_data_smithi043_119832_object48737:head# 2024-03-30T13:50:33.022 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76019836:::benchmark_data_smithi043_119832_object77030:head# 2024-03-30T13:50:33.023 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7602f3b0:::benchmark_data_smithi043_119832_object46794:head# 2024-03-30T13:50:33.023 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76047105:::benchmark_data_smithi043_119832_object54128:head# 2024-03-30T13:50:33.023 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76056729:::benchmark_data_smithi043_119832_object287:head# 2024-03-30T13:50:33.023 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:760582cb:::benchmark_data_smithi043_119832_object59468:head# 2024-03-30T13:50:33.023 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76062ae4:::benchmark_data_smithi043_119832_object9941:head# 2024-03-30T13:50:33.023 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76066544:::benchmark_data_smithi043_119832_object81539:head# 2024-03-30T13:50:33.023 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7606fd6e:::benchmark_data_smithi043_119832_object12079:head# 2024-03-30T13:50:33.023 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76072932:::benchmark_data_smithi043_119832_object80589:head# 2024-03-30T13:50:33.023 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76080247:::benchmark_data_smithi043_119832_object30058:head# 2024-03-30T13:50:33.023 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:760906ec:::benchmark_data_smithi043_119832_object28534:head# 2024-03-30T13:50:33.023 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:760c3911:::benchmark_data_smithi043_119832_object73959:head# 2024-03-30T13:50:33.023 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:760d093b:::benchmark_data_smithi043_119832_object33071:head# 2024-03-30T13:50:33.023 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:760d1a78:::benchmark_data_smithi043_119832_object5943:head# 2024-03-30T13:50:33.023 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76101f69:::benchmark_data_smithi043_119832_object539:head# 2024-03-30T13:50:33.023 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76146576:::benchmark_data_smithi043_119832_object59635:head# 2024-03-30T13:50:33.024 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76147d3a:::benchmark_data_smithi043_119832_object28581:head# 2024-03-30T13:50:33.024 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76153244:::benchmark_data_smithi043_119832_object62246:head# 2024-03-30T13:50:33.024 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7615e38f:::benchmark_data_smithi043_119832_object52868:head# 2024-03-30T13:50:33.024 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:761619d4:::benchmark_data_smithi043_119832_object12859:head# 2024-03-30T13:50:33.024 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7616e48c:::benchmark_data_smithi043_119832_object36557:head# 2024-03-30T13:50:33.027 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:761792cb:::benchmark_data_smithi043_119832_object6771:head# 2024-03-30T13:50:33.027 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7617bf6b:::benchmark_data_smithi043_119832_object17986:head# 2024-03-30T13:50:33.027 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7618b947:::benchmark_data_smithi043_119832_object47281:head# 2024-03-30T13:50:33.027 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76197d91:::benchmark_data_smithi043_119832_object78971:head# 2024-03-30T13:50:33.028 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76198398:::benchmark_data_smithi043_119832_object18771:head# 2024-03-30T13:50:33.028 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:761a4200:::benchmark_data_smithi043_119832_object25143:head# 2024-03-30T13:50:33.028 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:761a5f60:::benchmark_data_smithi043_119832_object31238:head# 2024-03-30T13:50:33.028 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:761a8f25:::benchmark_data_smithi043_119832_object75608:head# 2024-03-30T13:50:33.028 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:761afff3:::benchmark_data_smithi043_119832_object63750:head# 2024-03-30T13:50:33.028 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:761d3bd2:::benchmark_data_smithi043_119832_object87791:head# 2024-03-30T13:50:33.028 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:761d95e4:::benchmark_data_smithi043_119832_object5450:head# 2024-03-30T13:50:33.028 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:761dba87:::benchmark_data_smithi043_119832_object82264:head# 2024-03-30T13:50:33.028 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:761f90d6:::benchmark_data_smithi043_119832_object22587:head# 2024-03-30T13:50:33.028 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:761fb115:::benchmark_data_smithi043_119832_object24044:head# 2024-03-30T13:50:33.028 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:762082d3:::benchmark_data_smithi043_119832_object13179:head# 2024-03-30T13:50:33.028 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76216bd7:::benchmark_data_smithi043_119832_object12229:head# 2024-03-30T13:50:33.028 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7621a525:::benchmark_data_smithi043_119832_object36460:head# 2024-03-30T13:50:33.029 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7621d5cc:::benchmark_data_smithi043_119832_object16752:head# 2024-03-30T13:50:33.029 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:762204b9:::benchmark_data_smithi043_119832_object52633:head# 2024-03-30T13:50:33.029 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76229c89:::benchmark_data_smithi043_119832_object16097:head# 2024-03-30T13:50:33.029 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76232eb8:::benchmark_data_smithi043_119832_object82082:head# 2024-03-30T13:50:33.029 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7623ec71:::benchmark_data_smithi043_119832_object71057:head# 2024-03-30T13:50:33.030 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76245a57:::benchmark_data_smithi043_119832_object72781:head# 2024-03-30T13:50:33.030 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7624babb:::benchmark_data_smithi043_119832_object80763:head# 2024-03-30T13:50:33.030 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76254cc3:::benchmark_data_smithi043_119832_object39615:head# 2024-03-30T13:50:33.030 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7625efb0:::benchmark_data_smithi043_119832_object19856:head# 2024-03-30T13:50:33.030 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76268ae0:::benchmark_data_smithi043_119832_object56215:head# 2024-03-30T13:50:33.030 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:762699da:::benchmark_data_smithi043_119832_object53381:head# 2024-03-30T13:50:33.031 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:762abfc6:::benchmark_data_smithi043_119832_object4121:head# 2024-03-30T13:50:33.032 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:762d7d48:::benchmark_data_smithi043_119832_object43914:head# 2024-03-30T13:50:33.032 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:762d9893:::benchmark_data_smithi043_119832_object16794:head# 2024-03-30T13:50:33.033 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:762fea3d:::benchmark_data_smithi043_119832_object26978:head# 2024-03-30T13:50:33.033 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7630b7d5:::benchmark_data_smithi043_119832_object17803:head# 2024-03-30T13:50:33.033 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7630f6ce:::benchmark_data_smithi043_119832_object16765:head# 2024-03-30T13:50:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76322364:::benchmark_data_smithi043_119832_object40864:head# 2024-03-30T13:50:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76323f94:::benchmark_data_smithi043_119832_object15402:head# 2024-03-30T13:50:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7632776e:::benchmark_data_smithi043_119832_object41890:head# 2024-03-30T13:50:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76334aab:::benchmark_data_smithi043_119832_object56368:head# 2024-03-30T13:50:33.035 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76352e02:::benchmark_data_smithi043_119832_object86959:head# 2024-03-30T13:50:33.035 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76355681:::benchmark_data_smithi043_119832_object61812:head# 2024-03-30T13:50:33.035 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7636476a:::benchmark_data_smithi043_119832_object23803:head# 2024-03-30T13:50:33.036 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7636aef4:::benchmark_data_smithi043_119832_object40555:head# 2024-03-30T13:50:33.036 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7636fd2a:::benchmark_data_smithi043_119832_object55436:head# 2024-03-30T13:50:33.037 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7637e9e3:::benchmark_data_smithi043_119832_object2917:head# 2024-03-30T13:50:33.037 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7638868b:::benchmark_data_smithi043_119832_object68602:head# 2024-03-30T13:50:33.038 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76398ec7:::benchmark_data_smithi043_119832_object43503:head# 2024-03-30T13:50:33.038 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:763a7cd5:::benchmark_data_smithi043_119832_object78665:head# 2024-03-30T13:50:33.039 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:763b419b:::benchmark_data_smithi043_119832_object33934:head# 2024-03-30T13:50:33.039 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:763bc8a8:::benchmark_data_smithi043_119832_object31190:head# 2024-03-30T13:50:33.039 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:763d6eb4:::benchmark_data_smithi043_119832_object79053:head# 2024-03-30T13:50:33.040 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:763dd59a:::benchmark_data_smithi043_119832_object69489:head# 2024-03-30T13:50:33.044 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:763f14c1:::benchmark_data_smithi043_119832_object84910:head# 2024-03-30T13:50:33.044 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:763f79d0:::benchmark_data_smithi043_119832_object37624:head# 2024-03-30T13:50:33.044 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:763fc9c1:::benchmark_data_smithi043_119832_object40076:head# 2024-03-30T13:50:33.044 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:764014cc:::benchmark_data_smithi043_119832_object39115:head# 2024-03-30T13:50:33.044 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7640502d:::benchmark_data_smithi043_119832_object59413:head# 2024-03-30T13:50:33.045 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7640fea2:::benchmark_data_smithi043_119832_object59240:head# 2024-03-30T13:50:33.045 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:764127dd:::benchmark_data_smithi043_119832_object85310:head# 2024-03-30T13:50:33.045 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76420f15:::benchmark_data_smithi043_119832_object33097:head# 2024-03-30T13:50:33.045 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76422354:::benchmark_data_smithi043_119832_object30070:head# 2024-03-30T13:50:33.046 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76428ec6:::benchmark_data_smithi043_119832_object74356:head# 2024-03-30T13:50:33.047 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7644649d:::benchmark_data_smithi043_119832_object8793:head# 2024-03-30T13:50:33.047 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7645c5d6:::benchmark_data_smithi043_119832_object30646:head# 2024-03-30T13:50:33.047 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7647c245:::benchmark_data_smithi043_119832_object63576:head# 2024-03-30T13:50:33.047 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7649c8d6:::benchmark_data_smithi043_119832_object30344:head# 2024-03-30T13:50:33.047 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7649fc7f:::benchmark_data_smithi043_119832_object63199:head# 2024-03-30T13:50:33.048 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:764a1686:::benchmark_data_smithi043_119832_object87093:head# 2024-03-30T13:50:33.048 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:764a5fec:::benchmark_data_smithi043_119832_object33581:head# 2024-03-30T13:50:33.052 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:764a9c3a:::benchmark_data_smithi043_119832_object50036:head# 2024-03-30T13:50:33.052 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:764ba238:::benchmark_data_smithi043_119832_object4612:head# 2024-03-30T13:50:33.052 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:764bc414:::benchmark_data_smithi043_119832_object39055:head# 2024-03-30T13:50:33.052 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:764bca1b:::benchmark_data_smithi043_119832_object78622:head# 2024-03-30T13:50:33.052 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:764bdef3:::benchmark_data_smithi043_119832_object20317:head# 2024-03-30T13:50:33.052 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:764c3efd:::benchmark_data_smithi043_119832_object32197:head# 2024-03-30T13:50:33.052 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:764c60c9:::benchmark_data_smithi043_119832_object14195:head# 2024-03-30T13:50:33.052 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:764e5ab6:::benchmark_data_smithi043_119832_object20148:head# 2024-03-30T13:50:33.053 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:764f24bc:::benchmark_data_smithi043_119832_object79942:head# 2024-03-30T13:50:33.053 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:765025f2:::benchmark_data_smithi043_119832_object22989:head# 2024-03-30T13:50:33.053 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76513618:::benchmark_data_smithi043_119832_object81941:head# 2024-03-30T13:50:33.053 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:765159a9:::benchmark_data_smithi043_119832_object60431:head# 2024-03-30T13:50:33.054 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7651cf43:::benchmark_data_smithi043_119832_object23105:head# 2024-03-30T13:50:33.054 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7653e190:::benchmark_data_smithi043_119832_object6446:head# 2024-03-30T13:50:33.054 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7653e3a6:::benchmark_data_smithi043_119832_object76541:head# 2024-03-30T13:50:33.054 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7654d296:::benchmark_data_smithi043_119832_object12853:head# 2024-03-30T13:50:33.054 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76565583:::benchmark_data_smithi043_119832_object64604:head# 2024-03-30T13:50:33.055 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:765692a5:::benchmark_data_smithi043_119832_object39543:head# 2024-03-30T13:50:33.055 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76575590:::benchmark_data_smithi043_119832_object45939:head# 2024-03-30T13:50:33.055 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7657ad2e:::benchmark_data_smithi043_119832_object83943:head# 2024-03-30T13:50:33.056 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7658a5c9:::benchmark_data_smithi043_119832_object325:head# 2024-03-30T13:50:33.056 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76591b24:::benchmark_data_smithi043_119832_object60649:head# 2024-03-30T13:50:33.057 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7659ce77:::benchmark_data_smithi043_119832_object35688:head# 2024-03-30T13:50:33.057 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:765b1bed:::benchmark_data_smithi043_119832_object25844:head# 2024-03-30T13:50:33.058 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:765ba667:::benchmark_data_smithi043_119832_object30350:head# 2024-03-30T13:50:33.058 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:765c9a15:::benchmark_data_smithi043_119832_object79933:head# 2024-03-30T13:50:33.059 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:765cef09:::benchmark_data_smithi043_119832_object70812:head# 2024-03-30T13:50:33.059 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:765d4613:::benchmark_data_smithi043_119832_object59400:head# 2024-03-30T13:50:33.059 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:765e14c5:::benchmark_data_smithi043_119832_object60557:head# 2024-03-30T13:50:33.060 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:765e360e:::benchmark_data_smithi043_119832_object22585:head# 2024-03-30T13:50:33.060 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:765ee4f9:::benchmark_data_smithi043_119832_object3969:head# 2024-03-30T13:50:33.061 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7660224e:::benchmark_data_smithi043_119832_object53634:head# 2024-03-30T13:50:33.061 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:766110f8:::benchmark_data_smithi043_119832_object8852:head# 2024-03-30T13:50:33.062 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76615004:::benchmark_data_smithi043_119832_object65499:head# 2024-03-30T13:50:33.062 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:766245bb:::benchmark_data_smithi043_119832_object26280:head# 2024-03-30T13:50:33.062 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76631350:::benchmark_data_smithi043_119832_object76590:head# 2024-03-30T13:50:33.063 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:766327e2:::benchmark_data_smithi043_119832_object71128:head# 2024-03-30T13:50:33.063 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7663b6d1:::benchmark_data_smithi043_119832_object83846:head# 2024-03-30T13:50:33.064 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76641ff8:::benchmark_data_smithi043_119832_object30264:head# 2024-03-30T13:50:33.064 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76650744:::benchmark_data_smithi043_119832_object62488:head# 2024-03-30T13:50:33.065 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76657468:::benchmark_data_smithi043_119832_object310:head# 2024-03-30T13:50:33.065 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7665cd20:::benchmark_data_smithi043_119832_object74854:head# 2024-03-30T13:50:33.066 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7666139e:::benchmark_data_smithi043_119832_object73828:head# 2024-03-30T13:50:33.066 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76664592:::benchmark_data_smithi043_119832_object4265:head# 2024-03-30T13:50:33.067 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76664b6d:::benchmark_data_smithi043_119832_object86082:head# 2024-03-30T13:50:33.067 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7669e968:::benchmark_data_smithi043_119832_object75662:head# 2024-03-30T13:50:33.068 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:766ca19e:::benchmark_data_smithi043_119832_object15125:head# 2024-03-30T13:50:33.068 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:766d6aac:::benchmark_data_smithi043_119832_object3622:head# 2024-03-30T13:50:33.068 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:766ddeea:::benchmark_data_smithi043_119832_object53859:head# 2024-03-30T13:50:33.069 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:766f17a1:::benchmark_data_smithi043_119832_object79481:head# 2024-03-30T13:50:33.070 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:766f1887:::benchmark_data_smithi043_119832_object30792:head# 2024-03-30T13:50:33.070 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7671b5a8:::benchmark_data_smithi043_119832_object65638:head# 2024-03-30T13:50:33.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:33.071 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:33.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:33.076 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:767238de:::benchmark_data_smithi043_119832_object59081:head# 2024-03-30T13:50:33.076 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76724b75:::benchmark_data_smithi043_119832_object21007:head# 2024-03-30T13:50:33.076 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:767251d2:::benchmark_data_smithi043_119832_object63506:head# 2024-03-30T13:50:33.076 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7672eb93:::benchmark_data_smithi043_119832_object68670:head# 2024-03-30T13:50:33.076 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7672f382:::benchmark_data_smithi043_119832_object83982:head# 2024-03-30T13:50:33.077 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76733cf7:::benchmark_data_smithi043_119832_object25845:head# 2024-03-30T13:50:33.077 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7673df8d:::benchmark_data_smithi043_119832_object33422:head# 2024-03-30T13:50:33.077 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7673f468:::benchmark_data_smithi043_119832_object76964:head# 2024-03-30T13:50:33.077 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76753aeb:::benchmark_data_smithi043_119832_object65739:head# 2024-03-30T13:50:33.077 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7675723d:::benchmark_data_smithi043_119832_object22265:head# 2024-03-30T13:50:33.077 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7676163f:::benchmark_data_smithi043_119832_object46973:head# 2024-03-30T13:50:33.077 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:767738f6:::benchmark_data_smithi043_119832_object85608:head# 2024-03-30T13:50:33.077 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:767a5949:::benchmark_data_smithi043_119832_object42325:head# 2024-03-30T13:50:33.077 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:767af637:::benchmark_data_smithi043_119832_object47169:head# 2024-03-30T13:50:33.080 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:767b65df:::benchmark_data_smithi043_119832_object48513:head# 2024-03-30T13:50:33.080 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:767bccfa:::benchmark_data_smithi043_119832_object62660:head# 2024-03-30T13:50:33.080 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:767bf803:::benchmark_data_smithi043_119832_object80636:head# 2024-03-30T13:50:33.080 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:767c7da9:::benchmark_data_smithi043_119832_object46915:head# 2024-03-30T13:50:33.080 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:767cb73c:::benchmark_data_smithi043_119832_object40942:head# 2024-03-30T13:50:33.080 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:767e3c57:::benchmark_data_smithi043_119832_object9663:head# 2024-03-30T13:50:33.080 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:767e5003:::benchmark_data_smithi043_119832_object78968:head# 2024-03-30T13:50:33.080 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:767e8d5d:::benchmark_data_smithi043_119832_object16429:head# 2024-03-30T13:50:33.082 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:767ea3dd:::benchmark_data_smithi043_119832_object80477:head# 2024-03-30T13:50:33.082 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:767ee0dd:::benchmark_data_smithi043_119832_object83066:head# 2024-03-30T13:50:33.082 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:767f1514:::benchmark_data_smithi043_119832_object61043:head# 2024-03-30T13:50:33.083 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:767fd2f5:::benchmark_data_smithi043_119832_object47727:head# 2024-03-30T13:50:33.083 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:767ff754:::benchmark_data_smithi043_119832_object72157:head# 2024-03-30T13:50:33.083 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7680efb3:::benchmark_data_smithi043_119832_object54930:head# 2024-03-30T13:50:33.083 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7681158d:::benchmark_data_smithi043_119832_object12683:head# 2024-03-30T13:50:33.083 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7681febc:::benchmark_data_smithi043_119832_object10771:head# 2024-03-30T13:50:33.084 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76821f8f:::benchmark_data_smithi043_119832_object55734:head# 2024-03-30T13:50:33.084 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76822cbe:::benchmark_data_smithi043_119832_object59902:head# 2024-03-30T13:50:33.084 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7684dba6:::benchmark_data_smithi043_119832_object3519:head# 2024-03-30T13:50:33.084 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7685037e:::benchmark_data_smithi043_119832_object80815:head# 2024-03-30T13:50:33.086 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7686d120:::benchmark_data_smithi043_119832_object41722:head# 2024-03-30T13:50:33.086 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7686ea6b:::benchmark_data_smithi043_119832_object54729:head# 2024-03-30T13:50:33.086 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76874eec:::benchmark_data_smithi043_119832_object35290:head# 2024-03-30T13:50:33.086 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76879bd5:::benchmark_data_smithi043_119832_object40432:head# 2024-03-30T13:50:33.086 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76884694:::benchmark_data_smithi043_119832_object38286:head# 2024-03-30T13:50:33.087 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:768a4c04:::benchmark_data_smithi043_119832_object87375:head# 2024-03-30T13:50:33.087 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:768a7e55:::benchmark_data_smithi043_119832_object63574:head# 2024-03-30T13:50:33.087 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:768b78eb:::benchmark_data_smithi043_119832_object35540:head# 2024-03-30T13:50:33.088 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:768c48d7:::benchmark_data_smithi043_119832_object69834:head# 2024-03-30T13:50:33.088 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:768c5856:::benchmark_data_smithi043_119832_object130:head# 2024-03-30T13:50:33.088 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:768df144:::benchmark_data_smithi043_119832_object81676:head# 2024-03-30T13:50:33.088 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:768fcea4:::benchmark_data_smithi043_119832_object74634:head# 2024-03-30T13:50:33.089 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:768ffad4:::benchmark_data_smithi043_119832_object38889:head# 2024-03-30T13:50:33.089 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7690b413:::benchmark_data_smithi043_119832_object34260:head# 2024-03-30T13:50:33.089 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7690def8:::benchmark_data_smithi043_119832_object8842:head# 2024-03-30T13:50:33.090 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76910179:::benchmark_data_smithi043_119832_object17982:head# 2024-03-30T13:50:33.090 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7691975e:::benchmark_data_smithi043_119832_object73937:head# 2024-03-30T13:50:33.090 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7691e15a:::benchmark_data_smithi043_119832_object64211:head# 2024-03-30T13:50:33.090 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76931591:::benchmark_data_smithi043_119832_object9758:head# 2024-03-30T13:50:33.091 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7693406f:::benchmark_data_smithi043_119832_object3365:head# 2024-03-30T13:50:33.091 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76934514:::benchmark_data_smithi043_119832_object37892:head# 2024-03-30T13:50:33.091 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7693e6ab:::benchmark_data_smithi043_119832_object70043:head# 2024-03-30T13:50:33.091 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:769583ad:::benchmark_data_smithi043_119832_object25832:head# 2024-03-30T13:50:33.091 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76963bc0:::benchmark_data_smithi043_119832_object33864:head# 2024-03-30T13:50:33.092 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:769650a0:::benchmark_data_smithi043_119832_object9573:head# 2024-03-30T13:50:33.092 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7696abba:::benchmark_data_smithi043_119832_object7106:head# 2024-03-30T13:50:33.092 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76972901:::benchmark_data_smithi043_119832_object17693:head# 2024-03-30T13:50:33.092 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76985bf1:::benchmark_data_smithi043_119832_object15196:head# 2024-03-30T13:50:33.092 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76995378:::benchmark_data_smithi043_119832_object221:head# 2024-03-30T13:50:33.092 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:769a403d:::benchmark_data_smithi043_119832_object38285:head# 2024-03-30T13:50:33.093 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:769a421e:::benchmark_data_smithi043_119832_object20275:head# 2024-03-30T13:50:33.093 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:769ad85f:::benchmark_data_smithi043_119832_object65102:head# 2024-03-30T13:50:33.094 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:769e5450:::benchmark_data_smithi043_119832_object54862:head# 2024-03-30T13:50:33.094 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:769eb229:::benchmark_data_smithi043_119832_object60757:head# 2024-03-30T13:50:33.095 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:769fccf9:::benchmark_data_smithi043_119832_object57994:head# 2024-03-30T13:50:33.095 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76a07c30:::benchmark_data_smithi043_119832_object6339:head# 2024-03-30T13:50:33.095 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76a19318:::benchmark_data_smithi043_119832_object66408:head# 2024-03-30T13:50:33.095 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76a3d06a:::benchmark_data_smithi043_119832_object56670:head# 2024-03-30T13:50:33.095 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76a47a4c:::benchmark_data_smithi043_119832_object17527:head# 2024-03-30T13:50:33.096 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76a4c671:::benchmark_data_smithi043_119832_object37398:head# 2024-03-30T13:50:33.096 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76a5555d:::benchmark_data_smithi043_119832_object69479:head# 2024-03-30T13:50:33.096 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76a5be4d:::benchmark_data_smithi043_119832_object69851:head# 2024-03-30T13:50:33.097 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76a6a4dd:::benchmark_data_smithi043_119832_object63792:head# 2024-03-30T13:50:33.097 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76a7008e:::benchmark_data_smithi043_119832_object73382:head# 2024-03-30T13:50:33.098 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76a74271:::benchmark_data_smithi043_119832_object32014:head# 2024-03-30T13:50:33.099 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76a98abc:::benchmark_data_smithi043_119832_object4201:head# 2024-03-30T13:50:33.099 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76a9bf2c:::benchmark_data_smithi043_119832_object36716:head# 2024-03-30T13:50:33.099 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76aafb00:::benchmark_data_smithi043_119832_object4621:head# 2024-03-30T13:50:33.100 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76ab8b9c:::benchmark_data_smithi043_119832_object45040:head# 2024-03-30T13:50:33.100 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76ac9fc4:::benchmark_data_smithi043_119832_object654:head# 2024-03-30T13:50:33.100 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76acb9a3:::benchmark_data_smithi043_119832_object47322:head# 2024-03-30T13:50:33.101 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76ad27de:::benchmark_data_smithi043_119832_object24433:head# 2024-03-30T13:50:33.101 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76ad3c0e:::benchmark_data_smithi043_119832_object17977:head# 2024-03-30T13:50:33.101 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76ada1ad:::benchmark_data_smithi043_119832_object37686:head# 2024-03-30T13:50:33.102 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76adff47:::benchmark_data_smithi043_119832_object47589:head# 2024-03-30T13:50:33.102 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76ae45fd:::benchmark_data_smithi043_119832_object26210:head# 2024-03-30T13:50:33.102 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76af469f:::benchmark_data_smithi043_119832_object60642:head# 2024-03-30T13:50:33.103 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76b0a09c:::benchmark_data_smithi043_119832_object26256:head# 2024-03-30T13:50:33.103 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76b3ff4d:::benchmark_data_smithi043_119832_object9164:head# 2024-03-30T13:50:33.105 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76b44ae9:::benchmark_data_smithi043_119832_object56916:head# 2024-03-30T13:50:33.105 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76b6e26e:::benchmark_data_smithi043_119832_object39577:head# 2024-03-30T13:50:33.105 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76b738d0:::benchmark_data_smithi043_119832_object31256:head# 2024-03-30T13:50:33.105 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76b89397:::benchmark_data_smithi043_119832_object85712:head# 2024-03-30T13:50:33.105 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76b9451b:::benchmark_data_smithi043_119832_object44825:head# 2024-03-30T13:50:33.110 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76b97727:::benchmark_data_smithi043_119832_object48179:head# 2024-03-30T13:50:33.110 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76ba1621:::benchmark_data_smithi043_119832_object86487:head# 2024-03-30T13:50:33.110 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76bac342:::benchmark_data_smithi043_119832_object19334:head# 2024-03-30T13:50:33.110 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76bb71df:::benchmark_data_smithi043_119832_object14126:head# 2024-03-30T13:50:33.110 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76bb845e:::benchmark_data_smithi043_119832_object50341:head# 2024-03-30T13:50:33.110 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76bc54f9:::benchmark_data_smithi043_119832_object41652:head# 2024-03-30T13:50:33.110 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76bcd23a:::benchmark_data_smithi043_119832_object68544:head# 2024-03-30T13:50:33.110 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76bd131d:::benchmark_data_smithi043_119832_object41312:head# 2024-03-30T13:50:33.110 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76bda616:::benchmark_data_smithi043_119832_object34583:head# 2024-03-30T13:50:33.111 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76bfb7ad:::benchmark_data_smithi043_119832_object480:head# 2024-03-30T13:50:33.111 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76c00a1d:::benchmark_data_smithi043_119832_object8928:head# 2024-03-30T13:50:33.111 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76c07dbd:::benchmark_data_smithi043_119832_object4694:head# 2024-03-30T13:50:33.111 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76c0e758:::benchmark_data_smithi043_119832_object16268:head# 2024-03-30T13:50:33.111 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76c28885:::benchmark_data_smithi043_119832_object9893:head# 2024-03-30T13:50:33.111 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76c2d2fb:::benchmark_data_smithi043_119832_object60149:head# 2024-03-30T13:50:33.111 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76c400cd:::benchmark_data_smithi043_119832_object84478:head# 2024-03-30T13:50:33.111 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76c4bf6c:::benchmark_data_smithi043_119832_object56112:head# 2024-03-30T13:50:33.111 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76c5782c:::benchmark_data_smithi043_119832_object27710:head# 2024-03-30T13:50:33.111 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76c5c00c:::benchmark_data_smithi043_119832_object55686:head# 2024-03-30T13:50:33.111 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76c6236a:::benchmark_data_smithi043_119832_object7964:head# 2024-03-30T13:50:33.112 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76c6b4ed:::benchmark_data_smithi043_119832_object60375:head# 2024-03-30T13:50:33.112 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76c6c387:::benchmark_data_smithi043_119832_object29341:head# 2024-03-30T13:50:33.113 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76c71da1:::benchmark_data_smithi043_119832_object33159:head# 2024-03-30T13:50:33.113 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76c8111e:::benchmark_data_smithi043_119832_object14767:head# 2024-03-30T13:50:33.113 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76c8c6c5:::benchmark_data_smithi043_119832_object33580:head# 2024-03-30T13:50:33.113 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76ca4902:::benchmark_data_smithi043_119832_object46717:head# 2024-03-30T13:50:33.114 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76cb2928:::benchmark_data_smithi043_119832_object69222:head# 2024-03-30T13:50:33.114 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76cd3a34:::benchmark_data_smithi043_119832_object47867:head# 2024-03-30T13:50:33.114 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76cf0ae1:::benchmark_data_smithi043_119832_object78045:head# 2024-03-30T13:50:33.114 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76d0eec4:::benchmark_data_smithi043_119832_object31566:head# 2024-03-30T13:50:33.114 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76d15f1b:::benchmark_data_smithi043_119832_object58344:head# 2024-03-30T13:50:33.115 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76d1c4a2:::benchmark_data_smithi043_119832_object88191:head# 2024-03-30T13:50:33.115 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76d2cc43:::benchmark_data_smithi043_119832_object11853:head# 2024-03-30T13:50:33.115 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76d38395:::benchmark_data_smithi043_119832_object52951:head# 2024-03-30T13:50:33.115 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76d4c7b1:::benchmark_data_smithi043_119832_object28537:head# 2024-03-30T13:50:33.115 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76d5e1bc:::benchmark_data_smithi043_119832_object86640:head# 2024-03-30T13:50:33.116 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76d61e75:::benchmark_data_smithi043_119832_object31392:head# 2024-03-30T13:50:33.116 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76d69b2f:::benchmark_data_smithi043_119832_object69550:head# 2024-03-30T13:50:33.116 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76d76458:::benchmark_data_smithi043_119832_object50349:head# 2024-03-30T13:50:33.116 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76d823dd:::benchmark_data_smithi043_119832_object39053:head# 2024-03-30T13:50:33.117 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76d82b95:::benchmark_data_smithi043_119832_object71263:head# 2024-03-30T13:50:33.118 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76d8a73e:::benchmark_data_smithi043_119832_object31004:head# 2024-03-30T13:50:33.118 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76d907fe:::benchmark_data_smithi043_119832_object80341:head# 2024-03-30T13:50:33.119 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76d91e8c:::benchmark_data_smithi043_119832_object84160:head# 2024-03-30T13:50:33.119 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76d98a88:::benchmark_data_smithi043_119832_object20014:head# 2024-03-30T13:50:33.121 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76d9c560:::benchmark_data_smithi043_119832_object25530:head# 2024-03-30T13:50:33.121 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76d9e6ef:::benchmark_data_smithi043_119832_object79849:head# 2024-03-30T13:50:33.121 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76da5c09:::benchmark_data_smithi043_119832_object85762:head# 2024-03-30T13:50:33.121 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76da8827:::benchmark_data_smithi043_119832_object15214:head# 2024-03-30T13:50:33.121 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76da9899:::benchmark_data_smithi043_119832_object73279:head# 2024-03-30T13:50:33.121 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76dd2461:::benchmark_data_smithi043_119832_object68326:head# 2024-03-30T13:50:33.121 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76dd7d4b:::benchmark_data_smithi043_119832_object25694:head# 2024-03-30T13:50:33.121 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76de6b69:::benchmark_data_smithi043_119832_object23039:head# 2024-03-30T13:50:33.122 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76df5472:::benchmark_data_smithi043_119832_object75271:head# 2024-03-30T13:50:33.122 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76df82e9:::benchmark_data_smithi043_119832_object37084:head# 2024-03-30T13:50:33.122 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76e05e9c:::benchmark_data_smithi043_119832_object57063:head# 2024-03-30T13:50:33.123 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76e08791:::benchmark_data_smithi043_119832_object86401:head# 2024-03-30T13:50:33.123 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76e0c8a4:::benchmark_data_smithi043_119832_object33297:head# 2024-03-30T13:50:33.123 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76e2fcab:::benchmark_data_smithi043_119832_object75617:head# 2024-03-30T13:50:33.124 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76e34371:::benchmark_data_smithi043_119832_object49141:head# 2024-03-30T13:50:33.124 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76e4551d:::benchmark_data_smithi043_119832_object18083:head# 2024-03-30T13:50:33.125 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76e4ada5:::benchmark_data_smithi043_119832_object85844:head# 2024-03-30T13:50:33.125 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76e5cd2c:::benchmark_data_smithi043_119832_object77785:head# 2024-03-30T13:50:33.127 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76e723a3:::benchmark_data_smithi043_119832_object35271:head# 2024-03-30T13:50:33.127 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76e841ce:::benchmark_data_smithi043_119832_object590:head# 2024-03-30T13:50:33.127 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76e84ab8:::benchmark_data_smithi043_119832_object24634:head# 2024-03-30T13:50:33.127 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76e89b81:::benchmark_data_smithi043_119832_object33822:head# 2024-03-30T13:50:33.128 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76e8a688:::benchmark_data_smithi043_119832_object79785:head# 2024-03-30T13:50:33.128 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76e944b6:::benchmark_data_smithi043_119832_object36640:head# 2024-03-30T13:50:33.131 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76e9c877:::benchmark_data_smithi043_119832_object82580:head# 2024-03-30T13:50:33.131 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76ea0454:::benchmark_data_smithi043_119832_object76109:head# 2024-03-30T13:50:33.131 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76ea95d1:::benchmark_data_smithi043_119832_object71068:head# 2024-03-30T13:50:33.131 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76eb268f:::benchmark_data_smithi043_119832_object81852:head# 2024-03-30T13:50:33.131 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76eb8424:::benchmark_data_smithi043_119832_object20882:head# 2024-03-30T13:50:33.132 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76eba114:::benchmark_data_smithi043_119832_object63435:head# 2024-03-30T13:50:33.132 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76ec3284:::benchmark_data_smithi043_119832_object41113:head# 2024-03-30T13:50:33.132 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76ecb0eb:::benchmark_data_smithi043_119832_object20274:head# 2024-03-30T13:50:33.134 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76ecc339:::benchmark_data_smithi043_119832_object72311:head# 2024-03-30T13:50:33.134 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76ed4101:::benchmark_data_smithi043_119832_object3191:head# 2024-03-30T13:50:33.134 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f002b0:::benchmark_data_smithi043_119832_object54677:head# 2024-03-30T13:50:33.134 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f02114:::benchmark_data_smithi043_119832_object34563:head# 2024-03-30T13:50:33.134 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f02b75:::benchmark_data_smithi043_119832_object41627:head# 2024-03-30T13:50:33.141 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f07f2f:::benchmark_data_smithi043_119832_object1618:head# 2024-03-30T13:50:33.141 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f0f74a:::benchmark_data_smithi043_119832_object80937:head# 2024-03-30T13:50:33.141 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f1aeb8:::benchmark_data_smithi043_119832_object68987:head# 2024-03-30T13:50:33.142 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f1d12f:::benchmark_data_smithi043_119832_object5786:head# 2024-03-30T13:50:33.142 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f27e73:::benchmark_data_smithi043_119832_object13944:head# 2024-03-30T13:50:33.142 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f4f329:::benchmark_data_smithi043_119832_object46576:head# 2024-03-30T13:50:33.142 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f54b6b:::benchmark_data_smithi043_119832_object11376:head# 2024-03-30T13:50:33.142 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f55e37:::benchmark_data_smithi043_119832_object32742:head# 2024-03-30T13:50:33.142 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f5803a:::benchmark_data_smithi043_119832_object22935:head# 2024-03-30T13:50:33.142 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f5b2f7:::benchmark_data_smithi043_119832_object31797:head# 2024-03-30T13:50:33.142 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f5fc18:::benchmark_data_smithi043_119832_object66246:head# 2024-03-30T13:50:33.142 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f630f5:::benchmark_data_smithi043_119832_object62881:head# 2024-03-30T13:50:33.143 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f64720:::benchmark_data_smithi043_119832_object46650:head# 2024-03-30T13:50:33.143 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f6cf66:::benchmark_data_smithi043_119832_object20978:head# 2024-03-30T13:50:33.143 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f6d614:::benchmark_data_smithi043_119832_object38898:head# 2024-03-30T13:50:33.143 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f8347b:::benchmark_data_smithi043_119832_object63811:head# 2024-03-30T13:50:33.144 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f9665e:::benchmark_data_smithi043_119832_object7983:head# 2024-03-30T13:50:33.145 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76f97e5c:::benchmark_data_smithi043_119832_object87744:head# 2024-03-30T13:50:33.145 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76fac369:::benchmark_data_smithi043_119832_object58601:head# 2024-03-30T13:50:33.145 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76fc6e60:::benchmark_data_smithi043_119832_object52328:head# 2024-03-30T13:50:33.145 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76fcb48f:::benchmark_data_smithi043_119832_object39033:head# 2024-03-30T13:50:33.145 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76fe2fa6:::benchmark_data_smithi043_119832_object12162:head# 2024-03-30T13:50:33.145 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76fe4c4b:::benchmark_data_smithi043_119832_object83524:head# 2024-03-30T13:50:33.145 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76fed906:::benchmark_data_smithi043_119832_object39813:head# 2024-03-30T13:50:33.145 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76ffac7d:::benchmark_data_smithi043_119832_object50057:head# 2024-03-30T13:50:33.147 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76ffacc4:::benchmark_data_smithi043_119832_object15504:head# 2024-03-30T13:50:33.147 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76ffb1f1:::benchmark_data_smithi043_119832_object33725:head# 2024-03-30T13:50:33.147 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76ffd41a:::benchmark_data_smithi043_119832_object75430:head# 2024-03-30T13:50:33.147 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:76fff381:::benchmark_data_smithi043_119832_object43415:head# 2024-03-30T13:50:33.147 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77006830:::benchmark_data_smithi043_119832_object10930:head# 2024-03-30T13:50:33.147 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:770152a7:::benchmark_data_smithi043_119832_object14287:head# 2024-03-30T13:50:33.147 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77016b82:::benchmark_data_smithi043_119832_object3368:head# 2024-03-30T13:50:33.150 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7702ac8b:::benchmark_data_smithi043_119832_object7483:head# 2024-03-30T13:50:33.150 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77042aaa:::benchmark_data_smithi043_119832_object53132:head# 2024-03-30T13:50:33.150 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77049168:::benchmark_data_smithi043_119832_object65336:head# 2024-03-30T13:50:33.150 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77062880:::benchmark_data_smithi043_119832_object6747:head# 2024-03-30T13:50:33.150 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77070a54:::benchmark_data_smithi043_119832_object5356:head# 2024-03-30T13:50:33.151 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7707ecbe:::benchmark_data_smithi043_119832_object2950:head# 2024-03-30T13:50:33.151 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7707efe4:::benchmark_data_smithi043_119832_object39096:head# 2024-03-30T13:50:33.155 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:770883c6:::benchmark_data_smithi043_119832_object517:head# 2024-03-30T13:50:33.155 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7708d62a:::benchmark_data_smithi043_119832_object30691:head# 2024-03-30T13:50:33.155 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77097002:::benchmark_data_smithi043_119832_object81773:head# 2024-03-30T13:50:33.155 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:770b1eb2:::benchmark_data_smithi043_119832_object83109:head# 2024-03-30T13:50:33.155 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:770b979c:::benchmark_data_smithi043_119832_object25006:head# 2024-03-30T13:50:33.155 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:770c4fb1:::benchmark_data_smithi043_119832_object20483:head# 2024-03-30T13:50:33.155 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:770c6cd9:::benchmark_data_smithi043_119832_object56518:head# 2024-03-30T13:50:33.155 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:770c8544:::benchmark_data_smithi043_119832_object74323:head# 2024-03-30T13:50:33.155 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:770ce4b5:::benchmark_data_smithi043_119832_object43321:head# 2024-03-30T13:50:33.155 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77116d18:::benchmark_data_smithi043_119832_object64529:head# 2024-03-30T13:50:33.156 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77121d23:::benchmark_data_smithi043_119832_object29754:head# 2024-03-30T13:50:33.157 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7712b2d5:::benchmark_data_smithi043_119832_object45613:head# 2024-03-30T13:50:33.157 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77134d5c:::benchmark_data_smithi043_119832_object39194:head# 2024-03-30T13:50:33.157 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7713f5ee:::benchmark_data_smithi043_119832_object79323:head# 2024-03-30T13:50:33.157 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77141761:::benchmark_data_smithi043_119832_object950:head# 2024-03-30T13:50:33.157 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:771523e0:::benchmark_data_smithi043_119832_object44821:head# 2024-03-30T13:50:33.158 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77152455:::benchmark_data_smithi043_119832_object83219:head# 2024-03-30T13:50:33.158 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77157337:::benchmark_data_smithi043_119832_object82817:head# 2024-03-30T13:50:33.158 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:771819b8:::benchmark_data_smithi043_119832_object25236:head# 2024-03-30T13:50:33.159 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77188be9:::benchmark_data_smithi043_119832_object61988:head# 2024-03-30T13:50:33.159 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7718925f:::benchmark_data_smithi043_119832_object3929:head# 2024-03-30T13:50:33.160 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:771983ef:::benchmark_data_smithi043_119832_object22014:head# 2024-03-30T13:50:33.160 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:771a95b2:::benchmark_data_smithi043_119832_object55149:head# 2024-03-30T13:50:33.161 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:771b4f2e:::benchmark_data_smithi043_119832_object34925:head# 2024-03-30T13:50:33.161 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:771da743:::benchmark_data_smithi043_119832_object22093:head# 2024-03-30T13:50:33.161 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:771dccac:::benchmark_data_smithi043_119832_object7409:head# 2024-03-30T13:50:33.163 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:771e3640:::benchmark_data_smithi043_119832_object17668:head# 2024-03-30T13:50:33.163 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:771ed650:::benchmark_data_smithi043_119832_object49130:head# 2024-03-30T13:50:33.163 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:771f8760:::benchmark_data_smithi043_119832_object85917:head# 2024-03-30T13:50:33.171 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77200cfc:::benchmark_data_smithi043_119832_object19385:head# 2024-03-30T13:50:33.171 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7720917f:::benchmark_data_smithi043_119832_object81895:head# 2024-03-30T13:50:33.171 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:772151c3:::benchmark_data_smithi043_119832_object14463:head# 2024-03-30T13:50:33.171 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7721be89:::benchmark_data_smithi043_119832_object84455:head# 2024-03-30T13:50:33.171 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77237883:::benchmark_data_smithi043_119832_object74960:head# 2024-03-30T13:50:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77239b59:::benchmark_data_smithi043_119832_object65753:head# 2024-03-30T13:50:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77261a2c:::benchmark_data_smithi043_119832_object33891:head# 2024-03-30T13:50:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77263b0c:::benchmark_data_smithi043_119832_object56907:head# 2024-03-30T13:50:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7726f43f:::benchmark_data_smithi043_119832_object83532:head# 2024-03-30T13:50:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:772743ed:::benchmark_data_smithi043_119832_object66717:head# 2024-03-30T13:50:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77275cf0:::benchmark_data_smithi043_119832_object36972:head# 2024-03-30T13:50:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7728fcdc:::benchmark_data_smithi043_119832_object25364:head# 2024-03-30T13:50:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:772a5bc8:::benchmark_data_smithi043_119832_object12651:head# 2024-03-30T13:50:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:772acb51:::benchmark_data_smithi043_119832_object16460:head# 2024-03-30T13:50:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:772d4ba6:::benchmark_data_smithi043_119832_object28448:head# 2024-03-30T13:50:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:772e4aa8:::benchmark_data_smithi043_119832_object6812:head# 2024-03-30T13:50:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:772f0577:::benchmark_data_smithi043_119832_object411:head# 2024-03-30T13:50:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:772ff33e:::benchmark_data_smithi043_119832_object53078:head# 2024-03-30T13:50:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77301c7f:::benchmark_data_smithi043_119832_object46930:head# 2024-03-30T13:50:33.173 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7730b1ff:::benchmark_data_smithi043_119832_object44193:head# 2024-03-30T13:50:33.173 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77310717:::benchmark_data_smithi043_119832_object77541:head# 2024-03-30T13:50:33.173 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7731d238:::benchmark_data_smithi043_119832_object39541:head# 2024-03-30T13:50:33.173 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:33.175 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7731f221:::benchmark_data_smithi043_119832_object81776:head# 2024-03-30T13:50:33.175 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:773293e6:::benchmark_data_smithi043_119832_object86028:head# 2024-03-30T13:50:33.175 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7732bef2:::benchmark_data_smithi043_119832_object82733:head# 2024-03-30T13:50:33.176 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7733356e:::benchmark_data_smithi043_119832_object67654:head# 2024-03-30T13:50:33.176 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77335485:::benchmark_data_smithi043_119832_object37186:head# 2024-03-30T13:50:33.176 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77336c96:::benchmark_data_smithi043_119832_object54334:head# 2024-03-30T13:50:33.176 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7733c366:::benchmark_data_smithi043_119832_object77406:head# 2024-03-30T13:50:33.176 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77345c50:::benchmark_data_smithi043_119832_object77454:head# 2024-03-30T13:50:33.176 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77358e9f:::benchmark_data_smithi043_119832_object2902:head# 2024-03-30T13:50:33.176 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77360432:::benchmark_data_smithi043_119832_object82523:head# 2024-03-30T13:50:33.176 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77374298:::benchmark_data_smithi043_119832_object21135:head# 2024-03-30T13:50:33.177 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77382ef3:::benchmark_data_smithi043_119832_object45206:head# 2024-03-30T13:50:33.180 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:773934fa:::benchmark_data_smithi043_119832_object71022:head# 2024-03-30T13:50:33.180 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77398191:::benchmark_data_smithi043_119832_object39837:head# 2024-03-30T13:50:33.180 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:773b14e8:::benchmark_data_smithi043_119832_object80115:head# 2024-03-30T13:50:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:773ba2bc:::benchmark_data_smithi043_119832_object68529:head# 2024-03-30T13:50:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:773c9a07:::benchmark_data_smithi043_119832_object74880:head# 2024-03-30T13:50:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:773db4f5:::benchmark_data_smithi043_119832_object39671:head# 2024-03-30T13:50:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:773dda23:::benchmark_data_smithi043_119832_object44794:head# 2024-03-30T13:50:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:773ddaf3:::benchmark_data_smithi043_119832_object78248:head# 2024-03-30T13:50:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:773f6089:::benchmark_data_smithi043_119832_object20221:head# 2024-03-30T13:50:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:773f6d37:::benchmark_data_smithi043_119832_object10212:head# 2024-03-30T13:50:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7740a9cb:::benchmark_data_smithi043_119832_object63670:head# 2024-03-30T13:50:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7740eb85:::benchmark_data_smithi043_119832_object34061:head# 2024-03-30T13:50:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77413b5a:::benchmark_data_smithi043_119832_object3374:head# 2024-03-30T13:50:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77419f44:::benchmark_data_smithi043_119832_object35441:head# 2024-03-30T13:50:33.182 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:774249c7:::benchmark_data_smithi043_119832_object74823:head# 2024-03-30T13:50:33.191 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7742af75:::benchmark_data_smithi043_119832_object70991:head# 2024-03-30T13:50:33.191 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77432d55:::benchmark_data_smithi043_119832_object70168:head# 2024-03-30T13:50:33.191 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77436f55:::benchmark_data_smithi043_119832_object37632:head# 2024-03-30T13:50:33.191 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77441236:::benchmark_data_smithi043_119832_object37394:head# 2024-03-30T13:50:33.192 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77447675:::benchmark_data_smithi043_119832_object7124:head# 2024-03-30T13:50:33.192 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7746c3d4:::benchmark_data_smithi043_119832_object24936:head# 2024-03-30T13:50:33.192 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77472ea9:::benchmark_data_smithi043_119832_object2900:head# 2024-03-30T13:50:33.192 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77484394:::benchmark_data_smithi043_119832_object10971:head# 2024-03-30T13:50:33.192 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77494a4f:::benchmark_data_smithi043_119832_object57920:head# 2024-03-30T13:50:33.192 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:774a02d7:::benchmark_data_smithi043_119832_object23695:head# 2024-03-30T13:50:33.192 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:774a6645:::benchmark_data_smithi043_119832_object5226:head# 2024-03-30T13:50:33.192 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:774a864d:::benchmark_data_smithi043_119832_object77050:head# 2024-03-30T13:50:33.192 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:774bbbe2:::benchmark_data_smithi043_119832_object48474:head# 2024-03-30T13:50:33.192 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:774c9e75:::benchmark_data_smithi043_119832_object6876:head# 2024-03-30T13:50:33.192 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:774e443c:::benchmark_data_smithi043_119832_object37200:head# 2024-03-30T13:50:33.193 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:774fcfa5:::benchmark_data_smithi043_119832_object79636:head# 2024-03-30T13:50:33.193 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:774fe019:::benchmark_data_smithi043_119832_object12031:head# 2024-03-30T13:50:33.193 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77501267:::benchmark_data_smithi043_119832_object10263:head# 2024-03-30T13:50:33.193 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7750a4d3:::benchmark_data_smithi043_119832_object72223:head# 2024-03-30T13:50:33.195 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7750c401:::benchmark_data_smithi043_119832_object4750:head# 2024-03-30T13:50:33.195 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:775141c6:::benchmark_data_smithi043_119832_object25391:head# 2024-03-30T13:50:33.195 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77524707:::benchmark_data_smithi043_119832_object65715:head# 2024-03-30T13:50:33.195 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7752df78:::benchmark_data_smithi043_119832_object57951:head# 2024-03-30T13:50:33.195 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7753740e:::benchmark_data_smithi043_119832_object20311:head# 2024-03-30T13:50:33.202 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7753fcaa:::benchmark_data_smithi043_119832_object17733:head# 2024-03-30T13:50:33.202 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7754b3f9:::benchmark_data_smithi043_119832_object21331:head# 2024-03-30T13:50:33.202 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:775507be:::benchmark_data_smithi043_119832_object39522:head# 2024-03-30T13:50:33.202 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77551bc1:::benchmark_data_smithi043_119832_object15695:head# 2024-03-30T13:50:33.202 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77559dbd:::benchmark_data_smithi043_119832_object28242:head# 2024-03-30T13:50:33.202 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:775609d5:::benchmark_data_smithi043_119832_object85001:head# 2024-03-30T13:50:33.202 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7758d742:::benchmark_data_smithi043_119832_object79331:head# 2024-03-30T13:50:33.202 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7759623c:::benchmark_data_smithi043_119832_object44534:head# 2024-03-30T13:50:33.202 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:775a4efa:::benchmark_data_smithi043_119832_object58217:head# 2024-03-30T13:50:33.203 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:775a929b:::benchmark_data_smithi043_119832_object57219:head# 2024-03-30T13:50:33.203 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:775d0171:::benchmark_data_smithi043_119832_object28590:head# 2024-03-30T13:50:33.203 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:775d7062:::benchmark_data_smithi043_119832_object27711:head# 2024-03-30T13:50:33.203 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:775e292e:::benchmark_data_smithi043_119832_object42030:head# 2024-03-30T13:50:33.203 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:775e69a5:::benchmark_data_smithi043_119832_object30193:head# 2024-03-30T13:50:33.203 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:775ee9bb:::benchmark_data_smithi043_119832_object41602:head# 2024-03-30T13:50:33.203 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77607de8:::benchmark_data_smithi043_119832_object638:head# 2024-03-30T13:50:33.203 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7760b7fc:::benchmark_data_smithi043_119832_object37769:head# 2024-03-30T13:50:33.203 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7761b85e:::benchmark_data_smithi043_119832_object17189:head# 2024-03-30T13:50:33.203 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:776227fd:::benchmark_data_smithi043_119832_object82719:head# 2024-03-30T13:50:33.203 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:776269b3:::benchmark_data_smithi043_119832_object11948:head# 2024-03-30T13:50:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77626e69:::benchmark_data_smithi043_119832_object25421:head# 2024-03-30T13:50:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77631c0f:::benchmark_data_smithi043_119832_object86004:head# 2024-03-30T13:50:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77639347:::benchmark_data_smithi043_119832_object43465:head# 2024-03-30T13:50:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77639db3:::benchmark_data_smithi043_119832_object45063:head# 2024-03-30T13:50:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:776570ac:::benchmark_data_smithi043_119832_object69455:head# 2024-03-30T13:50:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77659985:::benchmark_data_smithi043_119832_object55861:head# 2024-03-30T13:50:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77672cb7:::benchmark_data_smithi043_119832_object22663:head# 2024-03-30T13:50:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77681d51:::benchmark_data_smithi043_119832_object6840:head# 2024-03-30T13:50:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77683fbc:::benchmark_data_smithi043_119832_object84473:head# 2024-03-30T13:50:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7768b7d0:::benchmark_data_smithi043_119832_object69107:head# 2024-03-30T13:50:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77690ffc:::benchmark_data_smithi043_119832_object76619:head# 2024-03-30T13:50:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77691632:::benchmark_data_smithi043_119832_object1870:head# 2024-03-30T13:50:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:776955f3:::benchmark_data_smithi043_119832_object16705:head# 2024-03-30T13:50:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7769eb2a:::benchmark_data_smithi043_119832_object33342:head# 2024-03-30T13:50:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:776a714d:::benchmark_data_smithi043_119832_object68451:head# 2024-03-30T13:50:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:776bf424:::benchmark_data_smithi043_119832_object43498:head# 2024-03-30T13:50:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:776c7669:::benchmark_data_smithi043_119832_object47001:head# 2024-03-30T13:50:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:776d8d81:::benchmark_data_smithi043_119832_object74897:head# 2024-03-30T13:50:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:776ef386:::benchmark_data_smithi043_119832_object54346:head# 2024-03-30T13:50:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7771b326:::benchmark_data_smithi043_119832_object76391:head# 2024-03-30T13:50:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:777226d1:::benchmark_data_smithi043_119832_object31641:head# 2024-03-30T13:50:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7773b15d:::benchmark_data_smithi043_119832_object6685:head# 2024-03-30T13:50:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:777442dc:::benchmark_data_smithi043_119832_object85008:head# 2024-03-30T13:50:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:777469e7:::benchmark_data_smithi043_119832_object64891:head# 2024-03-30T13:50:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:777510b6:::benchmark_data_smithi043_119832_object47017:head# 2024-03-30T13:50:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77758f8c:::benchmark_data_smithi043_119832_object906:head# 2024-03-30T13:50:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77762d46:::benchmark_data_smithi043_119832_object84314:head# 2024-03-30T13:50:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:777648c0:::benchmark_data_smithi043_119832_object119:head# 2024-03-30T13:50:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:777710a6:::benchmark_data_smithi043_119832_object54048:head# 2024-03-30T13:50:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:777800e0:::benchmark_data_smithi043_119832_object36314:head# 2024-03-30T13:50:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7778fc23:::benchmark_data_smithi043_119832_object25561:head# 2024-03-30T13:50:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:777a2575:::benchmark_data_smithi043_119832_object73320:head# 2024-03-30T13:50:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:777aa0fb:::benchmark_data_smithi043_119832_object6649:head# 2024-03-30T13:50:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:777b9d91:::benchmark_data_smithi043_119832_object22818:head# 2024-03-30T13:50:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:777ba0bf:::benchmark_data_smithi043_119832_object19298:head# 2024-03-30T13:50:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:777db56d:::benchmark_data_smithi043_119832_object41635:head# 2024-03-30T13:50:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:777e067e:::benchmark_data_smithi043_119832_object21597:head# 2024-03-30T13:50:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:777e9e72:::benchmark_data_smithi043_119832_object58148:head# 2024-03-30T13:50:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:777ffc71:::benchmark_data_smithi043_119832_object21002:head# 2024-03-30T13:50:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7781020c:::benchmark_data_smithi043_119832_object39647:head# 2024-03-30T13:50:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7781a09f:::benchmark_data_smithi043_119832_object34795:head# 2024-03-30T13:50:33.221 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7782adb0:::benchmark_data_smithi043_119832_object45977:head# 2024-03-30T13:50:33.221 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:778428b7:::benchmark_data_smithi043_119832_object72258:head# 2024-03-30T13:50:33.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:33.222 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:33.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:33.225 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7784779b:::benchmark_data_smithi043_119832_object53229:head# 2024-03-30T13:50:33.226 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77855a18:::benchmark_data_smithi043_119832_object80384:head# 2024-03-30T13:50:33.226 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7787241d:::benchmark_data_smithi043_119832_object36746:head# 2024-03-30T13:50:33.226 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7787d7be:::benchmark_data_smithi043_119832_object14565:head# 2024-03-30T13:50:33.226 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77889a38:::benchmark_data_smithi043_119832_object64654:head# 2024-03-30T13:50:33.226 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7789582c:::benchmark_data_smithi043_119832_object67085:head# 2024-03-30T13:50:33.226 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77898d72:::benchmark_data_smithi043_119832_object58597:head# 2024-03-30T13:50:33.226 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:778a14b8:::benchmark_data_smithi043_119832_object34971:head# 2024-03-30T13:50:33.226 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:778a2c82:::benchmark_data_smithi043_119832_object11492:head# 2024-03-30T13:50:33.226 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:778bbe92:::benchmark_data_smithi043_119832_object41770:head# 2024-03-30T13:50:33.226 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:778c3118:::benchmark_data_smithi043_119832_object9236:head# 2024-03-30T13:50:33.227 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:778d58eb:::benchmark_data_smithi043_119832_object48735:head# 2024-03-30T13:50:33.227 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:778d841b:::benchmark_data_smithi043_119832_object8917:head# 2024-03-30T13:50:33.230 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:778e0ac4:::benchmark_data_smithi043_119832_object59056:head# 2024-03-30T13:50:33.230 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:778e9700:::benchmark_data_smithi043_119832_object60388:head# 2024-03-30T13:50:33.230 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:778ec452:::benchmark_data_smithi043_119832_object38011:head# 2024-03-30T13:50:33.230 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:778f1b1a:::benchmark_data_smithi043_119832_object12189:head# 2024-03-30T13:50:33.230 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:778f4d7a:::benchmark_data_smithi043_119832_object11754:head# 2024-03-30T13:50:33.230 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77907deb:::benchmark_data_smithi043_119832_object71172:head# 2024-03-30T13:50:33.230 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7792338f:::benchmark_data_smithi043_119832_object57683:head# 2024-03-30T13:50:33.231 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77925468:::benchmark_data_smithi043_119832_object44440:head# 2024-03-30T13:50:33.231 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7792f64e:::benchmark_data_smithi043_119832_object5044:head# 2024-03-30T13:50:33.231 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77935a71:::benchmark_data_smithi043_119832_object67360:head# 2024-03-30T13:50:33.231 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77938002:::benchmark_data_smithi043_119832_object77257:head# 2024-03-30T13:50:33.232 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77955263:::benchmark_data_smithi043_119832_object50231:head# 2024-03-30T13:50:33.232 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77956286:::benchmark_data_smithi043_119832_object7671:head# 2024-03-30T13:50:33.232 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77957c0c:::benchmark_data_smithi043_119832_object33791:head# 2024-03-30T13:50:33.232 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7796622e:::benchmark_data_smithi043_119832_object32187:head# 2024-03-30T13:50:33.233 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7796fbd9:::benchmark_data_smithi043_119832_object71461:head# 2024-03-30T13:50:33.233 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:779702c0:::benchmark_data_smithi043_119832_object33009:head# 2024-03-30T13:50:33.233 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77977ab5:::benchmark_data_smithi043_119832_object36885:head# 2024-03-30T13:50:33.233 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:779783ca:::benchmark_data_smithi043_119832_object44919:head# 2024-03-30T13:50:33.234 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7798687d:::benchmark_data_smithi043_119832_object35847:head# 2024-03-30T13:50:33.234 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7799d27b:::benchmark_data_smithi043_119832_object2479:head# 2024-03-30T13:50:33.235 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:779a82dc:::benchmark_data_smithi043_119832_object22866:head# 2024-03-30T13:50:33.235 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:779a8a1d:::benchmark_data_smithi043_119832_object80534:head# 2024-03-30T13:50:33.237 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:779acdde:::benchmark_data_smithi043_119832_object39548:head# 2024-03-30T13:50:33.237 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:779af97f:::benchmark_data_smithi043_119832_object17814:head# 2024-03-30T13:50:33.237 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:779b1d90:::benchmark_data_smithi043_119832_object33796:head# 2024-03-30T13:50:33.237 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:779b87ee:::benchmark_data_smithi043_119832_object36739:head# 2024-03-30T13:50:33.238 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:779cd5c0:::benchmark_data_smithi043_119832_object61629:head# 2024-03-30T13:50:33.238 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:779cda23:::benchmark_data_smithi043_119832_object49190:head# 2024-03-30T13:50:33.238 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:779ce7cd:::benchmark_data_smithi043_119832_object35725:head# 2024-03-30T13:50:33.239 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:779cfd70:::benchmark_data_smithi043_119832_object87499:head# 2024-03-30T13:50:33.239 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:779e2008:::benchmark_data_smithi043_119832_object61711:head# 2024-03-30T13:50:33.240 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:779e45cd:::benchmark_data_smithi043_119832_object73904:head# 2024-03-30T13:50:33.240 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:779e5020:::benchmark_data_smithi043_119832_object7591:head# 2024-03-30T13:50:33.241 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:779ed28c:::benchmark_data_smithi043_119832_object24692:head# 2024-03-30T13:50:33.241 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:779f3cea:::benchmark_data_smithi043_119832_object62566:head# 2024-03-30T13:50:33.242 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77a00abd:::benchmark_data_smithi043_119832_object71547:head# 2024-03-30T13:50:33.242 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77a07221:::benchmark_data_smithi043_119832_object9456:head# 2024-03-30T13:50:33.242 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77a0f8da:::benchmark_data_smithi043_119832_object88302:head# 2024-03-30T13:50:33.243 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77a1faa8:::benchmark_data_smithi043_119832_object60928:head# 2024-03-30T13:50:33.243 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77a211f1:::benchmark_data_smithi043_119832_object30602:head# 2024-03-30T13:50:33.244 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77a30230:::benchmark_data_smithi043_119832_object29597:head# 2024-03-30T13:50:33.244 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77a3dbdd:::benchmark_data_smithi043_119832_object22446:head# 2024-03-30T13:50:33.244 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77a430b4:::benchmark_data_smithi043_119832_object31567:head# 2024-03-30T13:50:33.245 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77a4bbc3:::benchmark_data_smithi043_119832_object43021:head# 2024-03-30T13:50:33.246 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77a4ee4d:::benchmark_data_smithi043_119832_object24241:head# 2024-03-30T13:50:33.246 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77a5766c:::benchmark_data_smithi043_119832_object23035:head# 2024-03-30T13:50:33.247 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77a588a4:::benchmark_data_smithi043_119832_object60256:head# 2024-03-30T13:50:33.247 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77a60718:::benchmark_data_smithi043_119832_object15361:head# 2024-03-30T13:50:33.248 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77a62333:::benchmark_data_smithi043_119832_object10377:head# 2024-03-30T13:50:33.248 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77a6dea9:::benchmark_data_smithi043_119832_object30195:head# 2024-03-30T13:50:33.249 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77a73e71:::benchmark_data_smithi043_119832_object4282:head# 2024-03-30T13:50:33.249 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77a8bc99:::benchmark_data_smithi043_119832_object11573:head# 2024-03-30T13:50:33.250 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77a8f2b4:::benchmark_data_smithi043_119832_object88228:head# 2024-03-30T13:50:33.250 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77a94698:::benchmark_data_smithi043_119832_object15637:head# 2024-03-30T13:50:33.250 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77a98913:::benchmark_data_smithi043_119832_object39521:head# 2024-03-30T13:50:33.253 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77aa03ea:::benchmark_data_smithi043_119832_object65664:head# 2024-03-30T13:50:33.253 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77aa2390:::benchmark_data_smithi043_119832_object72783:head# 2024-03-30T13:50:33.253 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77aa7ff5:::benchmark_data_smithi043_119832_object66076:head# 2024-03-30T13:50:33.253 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77ab0772:::benchmark_data_smithi043_119832_object23514:head# 2024-03-30T13:50:33.253 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77ac24d9:::benchmark_data_smithi043_119832_object48076:head# 2024-03-30T13:50:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77ac7994:::benchmark_data_smithi043_119832_object47056:head# 2024-03-30T13:50:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77add3c8:::benchmark_data_smithi043_119832_object46065:head# 2024-03-30T13:50:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77ae7c6d:::benchmark_data_smithi043_119832_object24205:head# 2024-03-30T13:50:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77afddbf:::benchmark_data_smithi043_119832_object64595:head# 2024-03-30T13:50:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77b02d5d:::benchmark_data_smithi043_119832_object64922:head# 2024-03-30T13:50:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77b1105c:::benchmark_data_smithi043_119832_object1071:head# 2024-03-30T13:50:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77b1e1ba:::benchmark_data_smithi043_119832_object34996:head# 2024-03-30T13:50:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77b2d237:::benchmark_data_smithi043_119832_object31978:head# 2024-03-30T13:50:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77b454a9:::benchmark_data_smithi043_119832_object3515:head# 2024-03-30T13:50:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77b47316:::benchmark_data_smithi043_119832_object26801:head# 2024-03-30T13:50:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77b49e78:::benchmark_data_smithi043_119832_object25272:head# 2024-03-30T13:50:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77b60545:::benchmark_data_smithi043_119832_object54443:head# 2024-03-30T13:50:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77b7d2b8:::benchmark_data_smithi043_119832_object88201:head# 2024-03-30T13:50:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77b8279e:::benchmark_data_smithi043_119832_object67925:head# 2024-03-30T13:50:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77b86451:::benchmark_data_smithi043_119832_object62688:head# 2024-03-30T13:50:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77b9e078:::benchmark_data_smithi043_119832_object79059:head# 2024-03-30T13:50:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77ba3d43:::benchmark_data_smithi043_119832_object64001:head# 2024-03-30T13:50:33.262 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77ba6811:::benchmark_data_smithi043_119832_object30388:head# 2024-03-30T13:50:33.262 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77baa23a:::benchmark_data_smithi043_119832_object88030:head# 2024-03-30T13:50:33.262 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77bab356:::benchmark_data_smithi043_119832_object58575:head# 2024-03-30T13:50:33.262 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77bb8786:::benchmark_data_smithi043_119832_object32210:head# 2024-03-30T13:50:33.264 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77bbb3dd:::benchmark_data_smithi043_119832_object29042:head# 2024-03-30T13:50:33.265 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77bd4b49:::benchmark_data_smithi043_119832_object24316:head# 2024-03-30T13:50:33.265 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77bdc2d2:::benchmark_data_smithi043_119832_object28761:head# 2024-03-30T13:50:33.265 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77bee5d9:::benchmark_data_smithi043_119832_object52083:head# 2024-03-30T13:50:33.265 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77bf475c:::benchmark_data_smithi043_119832_object77199:head# 2024-03-30T13:50:33.265 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77bf6e13:::benchmark_data_smithi043_119832_object22388:head# 2024-03-30T13:50:33.265 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77bf70dd:::benchmark_data_smithi043_119832_object44460:head# 2024-03-30T13:50:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77bfa088:::benchmark_data_smithi043_119832_object83385:head# 2024-03-30T13:50:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77c00e46:::benchmark_data_smithi043_119832_object79186:head# 2024-03-30T13:50:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77c01f77:::benchmark_data_smithi043_119832_object53408:head# 2024-03-30T13:50:33.267 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77c238ce:::benchmark_data_smithi043_119832_object32:head# 2024-03-30T13:50:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77c271e5:::benchmark_data_smithi043_119832_object67496:head# 2024-03-30T13:50:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77c31908:::benchmark_data_smithi043_119832_object4274:head# 2024-03-30T13:50:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77c394a2:::benchmark_data_smithi043_119832_object81382:head# 2024-03-30T13:50:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77c3aff8:::benchmark_data_smithi043_119832_object47242:head# 2024-03-30T13:50:33.269 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77c3e712:::benchmark_data_smithi043_119832_object53108:head# 2024-03-30T13:50:33.269 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77c48b76:::benchmark_data_smithi043_119832_object10795:head# 2024-03-30T13:50:33.269 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77c564c3:::benchmark_data_smithi043_119832_object81582:head# 2024-03-30T13:50:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77c62720:::benchmark_data_smithi043_119832_object65892:head# 2024-03-30T13:50:33.272 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77c66b69:::benchmark_data_smithi043_119832_object60244:head# 2024-03-30T13:50:33.272 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77c7aa85:::benchmark_data_smithi043_119832_object47368:head# 2024-03-30T13:50:33.272 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77c91853:::benchmark_data_smithi043_119832_object64585:head# 2024-03-30T13:50:33.273 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77ca9eb1:::benchmark_data_smithi043_119832_object36999:head# 2024-03-30T13:50:33.273 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77cafff7:::benchmark_data_smithi043_119832_object5110:head# 2024-03-30T13:50:33.273 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77cc8067:::benchmark_data_smithi043_119832_object43332:head# 2024-03-30T13:50:33.273 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77cca033:::benchmark_data_smithi043_119832_object44059:head# 2024-03-30T13:50:33.274 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77ccb086:::benchmark_data_smithi043_119832_object74241:head# 2024-03-30T13:50:33.274 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77cdab46:::benchmark_data_smithi043_119832_object76753:head# 2024-03-30T13:50:33.274 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77ce974f:::benchmark_data_smithi043_119832_object17249:head# 2024-03-30T13:50:33.276 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77ce9f74:::benchmark_data_smithi043_119832_object2901:head# 2024-03-30T13:50:33.276 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77cfbf65:::benchmark_data_smithi043_119832_object62757:head# 2024-03-30T13:50:33.276 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77cfdee8:::benchmark_data_smithi043_119832_object43775:head# 2024-03-30T13:50:33.276 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77d21faf:::benchmark_data_smithi043_119832_object23543:head# 2024-03-30T13:50:33.276 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77d2d96e:::benchmark_data_smithi043_119832_object34526:head# 2024-03-30T13:50:33.277 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77d2edee:::benchmark_data_smithi043_119832_object65337:head# 2024-03-30T13:50:33.278 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77d4296e:::benchmark_data_smithi043_119832_object85480:head# 2024-03-30T13:50:33.278 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77d4a633:::benchmark_data_smithi043_119832_object39252:head# 2024-03-30T13:50:33.278 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77d4d1f6:::benchmark_data_smithi043_119832_object2194:head# 2024-03-30T13:50:33.278 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77d53232:::benchmark_data_smithi043_119832_object67928:head# 2024-03-30T13:50:33.278 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77d70529:::benchmark_data_smithi043_119832_object22523:head# 2024-03-30T13:50:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77d70773:::benchmark_data_smithi043_119832_object442:head# 2024-03-30T13:50:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77d7da19:::benchmark_data_smithi043_119832_object41381:head# 2024-03-30T13:50:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77d827bc:::benchmark_data_smithi043_119832_object67189:head# 2024-03-30T13:50:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77d8c65d:::benchmark_data_smithi043_119832_object31287:head# 2024-03-30T13:50:33.281 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77dc19ae:::benchmark_data_smithi043_119832_object1381:head# 2024-03-30T13:50:33.281 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77dc6748:::benchmark_data_smithi043_119832_object42901:head# 2024-03-30T13:50:33.281 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77dcf96b:::benchmark_data_smithi043_119832_object81188:head# 2024-03-30T13:50:33.282 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77de7cd7:::benchmark_data_smithi043_119832_object65952:head# 2024-03-30T13:50:33.282 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77e17c05:::benchmark_data_smithi043_119832_object11455:head# 2024-03-30T13:50:33.282 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77e21bc9:::benchmark_data_smithi043_119832_object5175:head# 2024-03-30T13:50:33.283 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77e3f62b:::benchmark_data_smithi043_119832_object12187:head# 2024-03-30T13:50:33.283 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77e41e78:::benchmark_data_smithi043_119832_object8415:head# 2024-03-30T13:50:33.283 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77e47e20:::benchmark_data_smithi043_119832_object8730:head# 2024-03-30T13:50:33.283 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77e497ef:::benchmark_data_smithi043_119832_object37453:head# 2024-03-30T13:50:33.283 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77e4ba3b:::benchmark_data_smithi043_119832_object27930:head# 2024-03-30T13:50:33.284 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77e510e9:::benchmark_data_smithi043_119832_object3056:head# 2024-03-30T13:50:33.284 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77e53d21:::benchmark_data_smithi043_119832_object36938:head# 2024-03-30T13:50:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77e54635:::benchmark_data_smithi043_119832_object85123:head# 2024-03-30T13:50:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77e5bcc3:::benchmark_data_smithi043_119832_object15252:head# 2024-03-30T13:50:33.287 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77e5ef2e:::benchmark_data_smithi043_119832_object67940:head# 2024-03-30T13:50:33.287 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77e655b3:::benchmark_data_smithi043_119832_object12881:head# 2024-03-30T13:50:33.287 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77e94f50:::benchmark_data_smithi043_119832_object36076:head# 2024-03-30T13:50:33.287 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77e97bff:::benchmark_data_smithi043_119832_object62472:head# 2024-03-30T13:50:33.287 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77ea291a:::benchmark_data_smithi043_119832_object82003:head# 2024-03-30T13:50:33.287 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77eab349:::benchmark_data_smithi043_119832_object50379:head# 2024-03-30T13:50:33.287 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77ee0e7b:::benchmark_data_smithi043_119832_object55135:head# 2024-03-30T13:50:33.287 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77eecec9:::benchmark_data_smithi043_119832_object54271:head# 2024-03-30T13:50:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77ef12fe:::benchmark_data_smithi043_119832_object70370:head# 2024-03-30T13:50:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77ef35f8:::benchmark_data_smithi043_119832_object77684:head# 2024-03-30T13:50:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77f10188:::benchmark_data_smithi043_119832_object33694:head# 2024-03-30T13:50:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77f15ed4:::benchmark_data_smithi043_119832_object61964:head# 2024-03-30T13:50:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77f252ba:::benchmark_data_smithi043_119832_object87153:head# 2024-03-30T13:50:33.290 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77f3416e:::benchmark_data_smithi043_119832_object51512:head# 2024-03-30T13:50:33.290 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77f34bc5:::benchmark_data_smithi043_119832_object40208:head# 2024-03-30T13:50:33.290 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77f40b0a:::benchmark_data_smithi043_119832_object24308:head# 2024-03-30T13:50:33.291 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77f473ab:::benchmark_data_smithi043_119832_object42605:head# 2024-03-30T13:50:33.291 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77f47d4a:::benchmark_data_smithi043_119832_object17610:head# 2024-03-30T13:50:33.291 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77f4a1e7:::benchmark_data_smithi043_119832_object61654:head# 2024-03-30T13:50:33.291 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77f4cd3e:::benchmark_data_smithi043_119832_object59847:head# 2024-03-30T13:50:33.291 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77f512bd:::benchmark_data_smithi043_119832_object46639:head# 2024-03-30T13:50:33.292 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77f5414c:::benchmark_data_smithi043_119832_object37329:head# 2024-03-30T13:50:33.292 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77f57e8d:::benchmark_data_smithi043_119832_object15357:head# 2024-03-30T13:50:33.292 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77f69a13:::benchmark_data_smithi043_119832_object45265:head# 2024-03-30T13:50:33.293 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77f87896:::benchmark_data_smithi043_119832_object16114:head# 2024-03-30T13:50:33.293 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77f878ae:::benchmark_data_smithi043_119832_object64748:head# 2024-03-30T13:50:33.293 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77f9016a:::benchmark_data_smithi043_119832_object53400:head# 2024-03-30T13:50:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77f91570:::benchmark_data_smithi043_119832_object9313:head# 2024-03-30T13:50:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77f9182d:::benchmark_data_smithi043_119832_object67682:head# 2024-03-30T13:50:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77f99cd8:::benchmark_data_smithi043_119832_object76434:head# 2024-03-30T13:50:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77fa338e:::benchmark_data_smithi043_119832_object50316:head# 2024-03-30T13:50:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77fa8519:::benchmark_data_smithi043_119832_object25105:head# 2024-03-30T13:50:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77faad45:::benchmark_data_smithi043_119832_object61604:head# 2024-03-30T13:50:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77faca7b:::benchmark_data_smithi043_119832_object22153:head# 2024-03-30T13:50:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77fb009b:::benchmark_data_smithi043_119832_object63786:head# 2024-03-30T13:50:33.298 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77fbef9f:::benchmark_data_smithi043_119832_object51753:head# 2024-03-30T13:50:33.298 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77fc6800:::benchmark_data_smithi043_119832_object3508:head# 2024-03-30T13:50:33.298 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77fcd1c0:::benchmark_data_smithi043_119832_object25418:head# 2024-03-30T13:50:33.298 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77fcd6c7:::benchmark_data_smithi043_119832_object25493:head# 2024-03-30T13:50:33.298 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77fd6e9b:::benchmark_data_smithi043_119832_object4336:head# 2024-03-30T13:50:33.299 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77fda5f7:::benchmark_data_smithi043_119832_object69944:head# 2024-03-30T13:50:33.299 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77fe3ef5:::benchmark_data_smithi043_119832_object17670:head# 2024-03-30T13:50:33.299 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77fece0f:::benchmark_data_smithi043_119832_object11615:head# 2024-03-30T13:50:33.299 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:77fef65d:::benchmark_data_smithi043_119832_object74794:head# 2024-03-30T13:50:33.300 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:780198f2:::benchmark_data_smithi043_119832_object48870:head# 2024-03-30T13:50:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78021205:::benchmark_data_smithi043_119832_object13293:head# 2024-03-30T13:50:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7802ed90:::benchmark_data_smithi043_119832_object30225:head# 2024-03-30T13:50:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78046b20:::benchmark_data_smithi043_119832_object32886:head# 2024-03-30T13:50:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78047a9b:::benchmark_data_smithi043_119832_object58977:head# 2024-03-30T13:50:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7804d608:::benchmark_data_smithi043_119832_object18323:head# 2024-03-30T13:50:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:780779a7:::benchmark_data_smithi043_119832_object87286:head# 2024-03-30T13:50:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:780791a0:::benchmark_data_smithi043_119832_object59042:head# 2024-03-30T13:50:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7807e7dc:::benchmark_data_smithi043_119832_object61004:head# 2024-03-30T13:50:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7808e1a3:::benchmark_data_smithi043_119832_object46510:head# 2024-03-30T13:50:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:780993d9:::benchmark_data_smithi043_119832_object85395:head# 2024-03-30T13:50:33.304 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:780a41b2:::benchmark_data_smithi043_119832_object20004:head# 2024-03-30T13:50:33.304 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:780ad6cb:::benchmark_data_smithi043_119832_object2340:head# 2024-03-30T13:50:33.304 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:780c787d:::benchmark_data_smithi043_119832_object15865:head# 2024-03-30T13:50:33.310 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:780c9a21:::benchmark_data_smithi043_119832_object72483:head# 2024-03-30T13:50:33.310 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:780cdac2:::benchmark_data_smithi043_119832_object18114:head# 2024-03-30T13:50:33.310 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:780e73ff:::benchmark_data_smithi043_119832_object86373:head# 2024-03-30T13:50:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:780f1950:::benchmark_data_smithi043_119832_object61266:head# 2024-03-30T13:50:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:780f3e71:::benchmark_data_smithi043_119832_object33569:head# 2024-03-30T13:50:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78101913:::benchmark_data_smithi043_119832_object46144:head# 2024-03-30T13:50:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:781064b0:::benchmark_data_smithi043_119832_object52703:head# 2024-03-30T13:50:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7810f910:::benchmark_data_smithi043_119832_object45281:head# 2024-03-30T13:50:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78110e58:::benchmark_data_smithi043_119832_object49116:head# 2024-03-30T13:50:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:781295fb:::benchmark_data_smithi043_119832_object76451:head# 2024-03-30T13:50:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7812f811:::benchmark_data_smithi043_119832_object46171:head# 2024-03-30T13:50:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:781310eb:::benchmark_data_smithi043_119832_object34613:head# 2024-03-30T13:50:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7813660d:::benchmark_data_smithi043_119832_object41553:head# 2024-03-30T13:50:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78146b1e:::benchmark_data_smithi043_119832_object34317:head# 2024-03-30T13:50:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78165b3e:::benchmark_data_smithi043_119832_object39955:head# 2024-03-30T13:50:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:781665eb:::benchmark_data_smithi043_119832_object69023:head# 2024-03-30T13:50:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78166601:::benchmark_data_smithi043_119832_object59689:head# 2024-03-30T13:50:33.312 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:781676d4:::benchmark_data_smithi043_119832_object7604:head# 2024-03-30T13:50:33.312 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78178136:::benchmark_data_smithi043_119832_object48061:head# 2024-03-30T13:50:33.312 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78178b18:::benchmark_data_smithi043_119832_object32046:head# 2024-03-30T13:50:33.314 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:781816b0:::benchmark_data_smithi043_119832_object35446:head# 2024-03-30T13:50:33.314 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78196561:::benchmark_data_smithi043_119832_object28607:head# 2024-03-30T13:50:33.314 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:781a5a31:::benchmark_data_smithi043_119832_object83035:head# 2024-03-30T13:50:33.315 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:781b746e:::benchmark_data_smithi043_119832_object75291:head# 2024-03-30T13:50:33.315 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:781d4540:::benchmark_data_smithi043_119832_object21506:head# 2024-03-30T13:50:33.315 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:781e4614:::benchmark_data_smithi043_119832_object12773:head# 2024-03-30T13:50:33.315 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:781fdcc9:::benchmark_data_smithi043_119832_object68848:head# 2024-03-30T13:50:33.315 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:782019fe:::benchmark_data_smithi043_119832_object27789:head# 2024-03-30T13:50:33.315 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78240d6e:::benchmark_data_smithi043_119832_object85033:head# 2024-03-30T13:50:33.315 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78263daa:::benchmark_data_smithi043_119832_object21258:head# 2024-03-30T13:50:33.315 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7826d968:::benchmark_data_smithi043_119832_object85749:head# 2024-03-30T13:50:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:782759cc:::benchmark_data_smithi043_119832_object15726:head# 2024-03-30T13:50:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78278f64:::benchmark_data_smithi043_119832_object51904:head# 2024-03-30T13:50:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78288581:::benchmark_data_smithi043_119832_object63264:head# 2024-03-30T13:50:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78294f2b:::benchmark_data_smithi043_119832_object41792:head# 2024-03-30T13:50:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7829dfd0:::benchmark_data_smithi043_119832_object25933:head# 2024-03-30T13:50:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:782af2f7:::benchmark_data_smithi043_119832_object68773:head# 2024-03-30T13:50:33.319 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:782b131b:::benchmark_data_smithi043_119832_object72539:head# 2024-03-30T13:50:33.319 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:782bc400:::benchmark_data_smithi043_119832_object60411:head# 2024-03-30T13:50:33.319 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:782c002b:::benchmark_data_smithi043_119832_object23447:head# 2024-03-30T13:50:33.319 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:782c6542:::benchmark_data_smithi043_119832_object83649:head# 2024-03-30T13:50:33.319 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:782c6f0d:::benchmark_data_smithi043_119832_object41820:head# 2024-03-30T13:50:33.321 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:782d4f16:::benchmark_data_smithi043_119832_object37654:head# 2024-03-30T13:50:33.321 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:782e19c3:::benchmark_data_smithi043_119832_object80926:head# 2024-03-30T13:50:33.321 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:782e3275:::benchmark_data_smithi043_119832_object44365:head# 2024-03-30T13:50:33.321 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:782e395b:::benchmark_data_smithi043_119832_object35031:head# 2024-03-30T13:50:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:782e45ea:::benchmark_data_smithi043_119832_object69051:head# 2024-03-30T13:50:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:782eb018:::benchmark_data_smithi043_119832_object86444:head# 2024-03-30T13:50:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:782eb520:::benchmark_data_smithi043_119832_object45532:head# 2024-03-30T13:50:33.323 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:782efe7d:::benchmark_data_smithi043_119832_object2448:head# 2024-03-30T13:50:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:782f6faa:::benchmark_data_smithi043_119832_object11542:head# 2024-03-30T13:50:33.325 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:782f98d5:::benchmark_data_smithi043_119832_object72559:head# 2024-03-30T13:50:33.325 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:782ff87c:::benchmark_data_smithi043_119832_object37087:head# 2024-03-30T13:50:33.325 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:783021fc:::benchmark_data_smithi043_119832_object21100:head# 2024-03-30T13:50:33.328 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78302216:::benchmark_data_smithi043_119832_object11728:head# 2024-03-30T13:50:33.328 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7830c442:::benchmark_data_smithi043_119832_object71782:head# 2024-03-30T13:50:33.328 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78315939:::benchmark_data_smithi043_119832_object8536:head# 2024-03-30T13:50:33.328 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78324358:::benchmark_data_smithi043_119832_object19805:head# 2024-03-30T13:50:33.328 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7832f4c0:::benchmark_data_smithi043_119832_object73071:head# 2024-03-30T13:50:33.328 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78338601:::benchmark_data_smithi043_119832_object76450:head# 2024-03-30T13:50:33.329 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7833b572:::benchmark_data_smithi043_119832_object77357:head# 2024-03-30T13:50:33.329 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7835ac7e:::benchmark_data_smithi043_119832_object25179:head# 2024-03-30T13:50:33.335 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7835f64d:::benchmark_data_smithi043_119832_object34389:head# 2024-03-30T13:50:33.335 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7835fb00:::benchmark_data_smithi043_119832_object56482:head# 2024-03-30T13:50:33.335 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7837349e:::benchmark_data_smithi043_119832_object57884:head# 2024-03-30T13:50:33.336 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78373848:::benchmark_data_smithi043_119832_object40788:head# 2024-03-30T13:50:33.336 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78380e4a:::benchmark_data_smithi043_119832_object57873:head# 2024-03-30T13:50:33.336 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78382298:::benchmark_data_smithi043_119832_object41152:head# 2024-03-30T13:50:33.336 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:783869f4:::benchmark_data_smithi043_119832_object23206:head# 2024-03-30T13:50:33.336 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7838adf4:::benchmark_data_smithi043_119832_object19426:head# 2024-03-30T13:50:33.336 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7839e7a0:::benchmark_data_smithi043_119832_object48043:head# 2024-03-30T13:50:33.336 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:783abc77:::benchmark_data_smithi043_119832_object14457:head# 2024-03-30T13:50:33.336 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:783acf3d:::benchmark_data_smithi043_119832_object87080:head# 2024-03-30T13:50:33.336 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:783b4237:::benchmark_data_smithi043_119832_object51644:head# 2024-03-30T13:50:33.337 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:783cde34:::benchmark_data_smithi043_119832_object76459:head# 2024-03-30T13:50:33.337 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:783d7370:::benchmark_data_smithi043_119832_object86075:head# 2024-03-30T13:50:33.337 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:783f93a7:::benchmark_data_smithi043_119832_object66675:head# 2024-03-30T13:50:33.337 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:784054aa:::benchmark_data_smithi043_119832_object75993:head# 2024-03-30T13:50:33.337 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78414016:::benchmark_data_smithi043_119832_object7159:head# 2024-03-30T13:50:33.337 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7841681b:::benchmark_data_smithi043_119832_object4673:head# 2024-03-30T13:50:33.337 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7841e92a:::benchmark_data_smithi043_119832_object56867:head# 2024-03-30T13:50:33.337 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7841ec5f:::benchmark_data_smithi043_119832_object25826:head# 2024-03-30T13:50:33.337 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78432fe8:::benchmark_data_smithi043_119832_object25849:head# 2024-03-30T13:50:33.343 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7843b31f:::benchmark_data_smithi043_119832_object29605:head# 2024-03-30T13:50:33.343 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78442698:::benchmark_data_smithi043_119832_object48767:head# 2024-03-30T13:50:33.343 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78444c2d:::benchmark_data_smithi043_119832_object14853:head# 2024-03-30T13:50:33.343 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78452858:::benchmark_data_smithi043_119832_object71959:head# 2024-03-30T13:50:33.343 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:784547a7:::benchmark_data_smithi043_119832_object61328:head# 2024-03-30T13:50:33.343 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78461aba:::benchmark_data_smithi043_119832_object13251:head# 2024-03-30T13:50:33.343 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:784640b1:::benchmark_data_smithi043_119832_object82723:head# 2024-03-30T13:50:33.343 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78473007:::benchmark_data_smithi043_119832_object34766:head# 2024-03-30T13:50:33.343 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78490d64:::benchmark_data_smithi043_119832_object79787:head# 2024-03-30T13:50:33.343 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:784aa92e:::benchmark_data_smithi043_119832_object45827:head# 2024-03-30T13:50:33.343 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:784b5d03:::benchmark_data_smithi043_119832_object12982:head# 2024-03-30T13:50:33.343 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:784b8770:::benchmark_data_smithi043_119832_object61304:head# 2024-03-30T13:50:33.343 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:784bb559:::benchmark_data_smithi043_119832_object12786:head# 2024-03-30T13:50:33.344 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:784bec48:::benchmark_data_smithi043_119832_object72424:head# 2024-03-30T13:50:33.344 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:784dc48a:::benchmark_data_smithi043_119832_object80367:head# 2024-03-30T13:50:33.344 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:784ef07a:::benchmark_data_smithi043_119832_object85738:head# 2024-03-30T13:50:33.344 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:784f3671:::benchmark_data_smithi043_119832_object61747:head# 2024-03-30T13:50:33.344 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7850406c:::benchmark_data_smithi043_119832_object78224:head# 2024-03-30T13:50:33.344 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7851817b:::benchmark_data_smithi043_119832_object31478:head# 2024-03-30T13:50:33.344 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78525a98:::benchmark_data_smithi043_119832_object81360:head# 2024-03-30T13:50:33.344 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78525a9d:::benchmark_data_smithi043_119832_object51069:head# 2024-03-30T13:50:33.344 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7853be32:::benchmark_data_smithi043_119832_object55055:head# 2024-03-30T13:50:33.344 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7853bf98:::benchmark_data_smithi043_119832_object78360:head# 2024-03-30T13:50:33.349 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7855b2e5:::benchmark_data_smithi043_119832_object77544:head# 2024-03-30T13:50:33.349 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7855c429:::benchmark_data_smithi043_119832_object755:head# 2024-03-30T13:50:33.350 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78568fdf:::benchmark_data_smithi043_119832_object76788:head# 2024-03-30T13:50:33.350 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7856b7f3:::benchmark_data_smithi043_119832_object51691:head# 2024-03-30T13:50:33.350 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:785787e0:::benchmark_data_smithi043_119832_object63510:head# 2024-03-30T13:50:33.350 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7858f23d:::benchmark_data_smithi043_119832_object30170:head# 2024-03-30T13:50:33.350 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7859a09e:::benchmark_data_smithi043_119832_object80886:head# 2024-03-30T13:50:33.350 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7859e6d5:::benchmark_data_smithi043_119832_object27378:head# 2024-03-30T13:50:33.350 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:785c515b:::benchmark_data_smithi043_119832_object8605:head# 2024-03-30T13:50:33.350 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:785ca3d9:::benchmark_data_smithi043_119832_object74636:head# 2024-03-30T13:50:33.350 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:785d29e1:::benchmark_data_smithi043_119832_object73533:head# 2024-03-30T13:50:33.350 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:785d7ee3:::benchmark_data_smithi043_119832_object74973:head# 2024-03-30T13:50:33.350 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:785e0638:::benchmark_data_smithi043_119832_object43872:head# 2024-03-30T13:50:33.350 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:785e1802:::benchmark_data_smithi043_119832_object77647:head# 2024-03-30T13:50:33.350 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:785e4c12:::benchmark_data_smithi043_119832_object2524:head# 2024-03-30T13:50:33.351 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:785f419d:::benchmark_data_smithi043_119832_object86511:head# 2024-03-30T13:50:33.352 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78608bab:::benchmark_data_smithi043_119832_object27490:head# 2024-03-30T13:50:33.352 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7861916d:::benchmark_data_smithi043_119832_object64436:head# 2024-03-30T13:50:33.352 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7861c57b:::benchmark_data_smithi043_119832_object17232:head# 2024-03-30T13:50:33.352 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78622844:::benchmark_data_smithi043_119832_object66263:head# 2024-03-30T13:50:33.352 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7862783e:::benchmark_data_smithi043_119832_object25037:head# 2024-03-30T13:50:33.353 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7862ea1f:::benchmark_data_smithi043_119832_object21225:head# 2024-03-30T13:50:33.353 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:786492b5:::benchmark_data_smithi043_119832_object58156:head# 2024-03-30T13:50:33.353 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7864fa22:::benchmark_data_smithi043_119832_object3715:head# 2024-03-30T13:50:33.354 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78654d82:::benchmark_data_smithi043_119832_object27700:head# 2024-03-30T13:50:33.354 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:786659f7:::benchmark_data_smithi043_119832_object35138:head# 2024-03-30T13:50:33.355 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7866a65d:::benchmark_data_smithi043_119832_object82388:head# 2024-03-30T13:50:33.355 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78670cfa:::benchmark_data_smithi043_119832_object63226:head# 2024-03-30T13:50:33.356 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78671501:::benchmark_data_smithi043_119832_object18519:head# 2024-03-30T13:50:33.356 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7868f52f:::benchmark_data_smithi043_119832_object63098:head# 2024-03-30T13:50:33.356 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78692e8c:::benchmark_data_smithi043_119832_object38844:head# 2024-03-30T13:50:33.356 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:786cf821:::benchmark_data_smithi043_119832_object16663:head# 2024-03-30T13:50:33.357 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:786dd718:::benchmark_data_smithi043_119832_object10246:head# 2024-03-30T13:50:33.358 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:786e58eb:::benchmark_data_smithi043_119832_object5930:head# 2024-03-30T13:50:33.358 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:786ec6ee:::benchmark_data_smithi043_119832_object41687:head# 2024-03-30T13:50:33.358 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:786f3efc:::benchmark_data_smithi043_119832_object1851:head# 2024-03-30T13:50:33.359 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7870b0d5:::benchmark_data_smithi043_119832_object51558:head# 2024-03-30T13:50:33.359 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7871663d:::benchmark_data_smithi043_119832_object73881:head# 2024-03-30T13:50:33.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:33.361 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:33.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7871e341:::benchmark_data_smithi043_119832_object55403:head# 2024-03-30T13:50:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7871e4ad:::benchmark_data_smithi043_119832_object85488:head# 2024-03-30T13:50:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78726a56:::benchmark_data_smithi043_119832_object23081:head# 2024-03-30T13:50:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7876cd06:::benchmark_data_smithi043_119832_object7064:head# 2024-03-30T13:50:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7876f049:::benchmark_data_smithi043_119832_object36418:head# 2024-03-30T13:50:33.363 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78777f23:::benchmark_data_smithi043_119832_object55167:head# 2024-03-30T13:50:33.363 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7878066d:::benchmark_data_smithi043_119832_object57446:head# 2024-03-30T13:50:33.363 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7878a54c:::benchmark_data_smithi043_119832_object14452:head# 2024-03-30T13:50:33.363 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7878c890:::benchmark_data_smithi043_119832_object6963:head# 2024-03-30T13:50:33.363 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:787957fb:::benchmark_data_smithi043_119832_object23643:head# 2024-03-30T13:50:33.363 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7879943f:::benchmark_data_smithi043_119832_object7389:head# 2024-03-30T13:50:33.366 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:787a2203:::benchmark_data_smithi043_119832_object58189:head# 2024-03-30T13:50:33.366 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:787a37e6:::benchmark_data_smithi043_119832_object610:head# 2024-03-30T13:50:33.366 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:787ba0ac:::benchmark_data_smithi043_119832_object50072:head# 2024-03-30T13:50:33.366 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:787c008a:::benchmark_data_smithi043_119832_object45142:head# 2024-03-30T13:50:33.366 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:787c8caa:::benchmark_data_smithi043_119832_object49910:head# 2024-03-30T13:50:33.366 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:787ca7bc:::benchmark_data_smithi043_119832_object66233:head# 2024-03-30T13:50:33.367 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:787cf92e:::benchmark_data_smithi043_119832_object12885:head# 2024-03-30T13:50:33.367 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:787e8d06:::benchmark_data_smithi043_119832_object13171:head# 2024-03-30T13:50:33.367 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:787f7480:::benchmark_data_smithi043_119832_object42188:head# 2024-03-30T13:50:33.367 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78805f31:::benchmark_data_smithi043_119832_object76718:head# 2024-03-30T13:50:33.367 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78807660:::benchmark_data_smithi043_119832_object8971:head# 2024-03-30T13:50:33.367 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78814453:::benchmark_data_smithi043_119832_object42335:head# 2024-03-30T13:50:33.369 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7881b70e:::benchmark_data_smithi043_119832_object15667:head# 2024-03-30T13:50:33.369 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7881ed9d:::benchmark_data_smithi043_119832_object63031:head# 2024-03-30T13:50:33.369 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7883a34b:::benchmark_data_smithi043_119832_object15352:head# 2024-03-30T13:50:33.369 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7883c20a:::benchmark_data_smithi043_119832_object1422:head# 2024-03-30T13:50:33.369 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7883d982:::benchmark_data_smithi043_119832_object33911:head# 2024-03-30T13:50:33.371 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:788633a0:::benchmark_data_smithi043_119832_object42506:head# 2024-03-30T13:50:33.371 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:788696fb:::benchmark_data_smithi043_119832_object70427:head# 2024-03-30T13:50:33.372 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7886ce66:::benchmark_data_smithi043_119832_object78470:head# 2024-03-30T13:50:33.372 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7887c005:::benchmark_data_smithi043_119832_object46334:head# 2024-03-30T13:50:33.372 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7888709d:::benchmark_data_smithi043_119832_object46960:head# 2024-03-30T13:50:33.375 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78890a16:::benchmark_data_smithi043_119832_object72271:head# 2024-03-30T13:50:33.375 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:788a0871:::benchmark_data_smithi043_119832_object65036:head# 2024-03-30T13:50:33.375 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:788a1f04:::benchmark_data_smithi043_119832_object779:head# 2024-03-30T13:50:33.375 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:788ac7fb:::benchmark_data_smithi043_119832_object41220:head# 2024-03-30T13:50:33.375 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:788afa30:::benchmark_data_smithi043_119832_object14718:head# 2024-03-30T13:50:33.375 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:788b1904:::benchmark_data_smithi043_119832_object65004:head# 2024-03-30T13:50:33.375 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:788b3c1c:::benchmark_data_smithi043_119832_object6073:head# 2024-03-30T13:50:33.381 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:788b4c27:::benchmark_data_smithi043_119832_object46986:head# 2024-03-30T13:50:33.381 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:788c77f9:::benchmark_data_smithi043_119832_object16144:head# 2024-03-30T13:50:33.381 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:788c9d69:::benchmark_data_smithi043_119832_object83983:head# 2024-03-30T13:50:33.381 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:788d1056:::benchmark_data_smithi043_119832_object21116:head# 2024-03-30T13:50:33.381 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:789088fc:::benchmark_data_smithi043_119832_object6928:head# 2024-03-30T13:50:33.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78929c87:::benchmark_data_smithi043_119832_object13031:head# 2024-03-30T13:50:33.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7894f32d:::benchmark_data_smithi043_119832_object38134:head# 2024-03-30T13:50:33.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7895f734:::benchmark_data_smithi043_119832_object70536:head# 2024-03-30T13:50:33.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7897711b:::benchmark_data_smithi043_119832_object71383:head# 2024-03-30T13:50:33.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78990106:::benchmark_data_smithi043_119832_object41318:head# 2024-03-30T13:50:33.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78999514:::benchmark_data_smithi043_119832_object81244:head# 2024-03-30T13:50:33.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:789acae4:::benchmark_data_smithi043_119832_object55498:head# 2024-03-30T13:50:33.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:789bc2fb:::benchmark_data_smithi043_119832_object76852:head# 2024-03-30T13:50:33.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:789cadc7:::benchmark_data_smithi043_119832_object60526:head# 2024-03-30T13:50:33.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:789cc65d:::benchmark_data_smithi043_119832_object23402:head# 2024-03-30T13:50:33.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:789ccacf:::benchmark_data_smithi043_119832_object25985:head# 2024-03-30T13:50:33.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:789d657a:::benchmark_data_smithi043_119832_object58528:head# 2024-03-30T13:50:33.382 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:789de0f5:::benchmark_data_smithi043_119832_object8540:head# 2024-03-30T13:50:33.385 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:789df816:::benchmark_data_smithi043_119832_object56981:head# 2024-03-30T13:50:33.385 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:789e17f6:::benchmark_data_smithi043_119832_object8246:head# 2024-03-30T13:50:33.385 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:789e6b99:::benchmark_data_smithi043_119832_object28768:head# 2024-03-30T13:50:33.385 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:789e87aa:::benchmark_data_smithi043_119832_object78422:head# 2024-03-30T13:50:33.385 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:789f858b:::benchmark_data_smithi043_119832_object4494:head# 2024-03-30T13:50:33.385 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:789ffd24:::benchmark_data_smithi043_119832_object62004:head# 2024-03-30T13:50:33.388 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78a14621:::benchmark_data_smithi043_119832_object69357:head# 2024-03-30T13:50:33.388 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78a15f67:::benchmark_data_smithi043_119832_object39113:head# 2024-03-30T13:50:33.388 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78a16460:::benchmark_data_smithi043_119832_object52510:head# 2024-03-30T13:50:33.389 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78a19334:::benchmark_data_smithi043_119832_object77505:head# 2024-03-30T13:50:33.389 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78a1a434:::benchmark_data_smithi043_119832_object21942:head# 2024-03-30T13:50:33.389 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78a1d31d:::benchmark_data_smithi043_119832_object20061:head# 2024-03-30T13:50:33.389 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78a33cc2:::benchmark_data_smithi043_119832_object51580:head# 2024-03-30T13:50:33.389 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78a4cc56:::benchmark_data_smithi043_119832_object5104:head# 2024-03-30T13:50:33.389 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78a56b0a:::benchmark_data_smithi043_119832_object78911:head# 2024-03-30T13:50:33.392 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78a63408:::benchmark_data_smithi043_119832_object46088:head# 2024-03-30T13:50:33.392 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78a66c8b:::benchmark_data_smithi043_119832_object66726:head# 2024-03-30T13:50:33.393 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78a708c7:::benchmark_data_smithi043_119832_object23051:head# 2024-03-30T13:50:33.393 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78a7e561:::benchmark_data_smithi043_119832_object30451:head# 2024-03-30T13:50:33.393 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78a84ca6:::benchmark_data_smithi043_119832_object31959:head# 2024-03-30T13:50:33.393 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78a89824:::benchmark_data_smithi043_119832_object19677:head# 2024-03-30T13:50:33.393 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78a95b68:::benchmark_data_smithi043_119832_object79587:head# 2024-03-30T13:50:33.393 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78aa49fb:::benchmark_data_smithi043_119832_object67898:head# 2024-03-30T13:50:33.393 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78aaa3df:::benchmark_data_smithi043_119832_object85549:head# 2024-03-30T13:50:33.393 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78ac8950:::benchmark_data_smithi043_119832_object17596:head# 2024-03-30T13:50:33.395 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78ad9bd4:::benchmark_data_smithi043_119832_object9662:head# 2024-03-30T13:50:33.395 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78adec18:::benchmark_data_smithi043_119832_object5817:head# 2024-03-30T13:50:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78ae745a:::benchmark_data_smithi043_119832_object25567:head# 2024-03-30T13:50:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78af2b44:::benchmark_data_smithi043_119832_object37229:head# 2024-03-30T13:50:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78afa2e1:::benchmark_data_smithi043_119832_object76206:head# 2024-03-30T13:50:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78afc2ba:::benchmark_data_smithi043_119832_object61630:head# 2024-03-30T13:50:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78b005bf:::benchmark_data_smithi043_119832_object82911:head# 2024-03-30T13:50:33.397 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78b1cb4b:::benchmark_data_smithi043_119832_object51769:head# 2024-03-30T13:50:33.397 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78b39ed6:::benchmark_data_smithi043_119832_object31648:head# 2024-03-30T13:50:33.397 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78b3ed9e:::benchmark_data_smithi043_119832_object75614:head# 2024-03-30T13:50:33.397 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78b46e23:::benchmark_data_smithi043_119832_object34763:head# 2024-03-30T13:50:33.397 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78b48685:::benchmark_data_smithi043_119832_object23939:head# 2024-03-30T13:50:33.397 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78b48fce:::benchmark_data_smithi043_119832_object42953:head# 2024-03-30T13:50:33.398 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78b4fa46:::benchmark_data_smithi043_119832_object5057:head# 2024-03-30T13:50:33.398 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78b50df1:::benchmark_data_smithi043_119832_object61623:head# 2024-03-30T13:50:33.398 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78b522a6:::benchmark_data_smithi043_119832_object47881:head# 2024-03-30T13:50:33.399 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78b579d7:::benchmark_data_smithi043_119832_object86357:head# 2024-03-30T13:50:33.399 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78b601d5:::benchmark_data_smithi043_119832_object67938:head# 2024-03-30T13:50:33.399 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78b7d91d:::benchmark_data_smithi043_119832_object27097:head# 2024-03-30T13:50:33.399 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78b7ef7c:::benchmark_data_smithi043_119832_object50085:head# 2024-03-30T13:50:33.400 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78b96502:::benchmark_data_smithi043_119832_object43114:head# 2024-03-30T13:50:33.400 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78baa9ad:::benchmark_data_smithi043_119832_object15018:head# 2024-03-30T13:50:33.400 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78bb296e:::benchmark_data_smithi043_119832_object86452:head# 2024-03-30T13:50:33.400 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78bbb036:::benchmark_data_smithi043_119832_object80088:head# 2024-03-30T13:50:33.400 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78bc00c1:::benchmark_data_smithi043_119832_object21557:head# 2024-03-30T13:50:33.400 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78bc5560:::benchmark_data_smithi043_119832_object17184:head# 2024-03-30T13:50:33.401 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78bcc878:::benchmark_data_smithi043_119832_object9179:head# 2024-03-30T13:50:33.401 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78bd10df:::benchmark_data_smithi043_119832_object40334:head# 2024-03-30T13:50:33.401 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78bd1ef2:::benchmark_data_smithi043_119832_object3657:head# 2024-03-30T13:50:33.401 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78bdd661:::benchmark_data_smithi043_119832_object76334:head# 2024-03-30T13:50:33.401 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78bde2a5:::benchmark_data_smithi043_119832_object58742:head# 2024-03-30T13:50:33.401 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78bde864:::benchmark_data_smithi043_119832_object68696:head# 2024-03-30T13:50:33.403 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78beafe1:::benchmark_data_smithi043_119832_object29620:head# 2024-03-30T13:50:33.403 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78bed774:::benchmark_data_smithi043_119832_object71800:head# 2024-03-30T13:50:33.403 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78c0abf9:::benchmark_data_smithi043_119832_object77445:head# 2024-03-30T13:50:33.403 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78c172d6:::benchmark_data_smithi043_119832_object16691:head# 2024-03-30T13:50:33.403 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78c1ca3d:::benchmark_data_smithi043_119832_object83833:head# 2024-03-30T13:50:33.404 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78c2364f:::benchmark_data_smithi043_119832_object85251:head# 2024-03-30T13:50:33.404 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78c2df48:::benchmark_data_smithi043_119832_object11939:head# 2024-03-30T13:50:33.404 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78c3997c:::benchmark_data_smithi043_119832_object48825:head# 2024-03-30T13:50:33.404 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78c3d76f:::benchmark_data_smithi043_119832_object44929:head# 2024-03-30T13:50:33.405 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78c568d1:::benchmark_data_smithi043_119832_object38549:head# 2024-03-30T13:50:33.405 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78c5fe71:::benchmark_data_smithi043_119832_object594:head# 2024-03-30T13:50:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78c61b76:::benchmark_data_smithi043_119832_object573:head# 2024-03-30T13:50:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78c64ec3:::benchmark_data_smithi043_119832_object31192:head# 2024-03-30T13:50:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78c702aa:::benchmark_data_smithi043_119832_object13620:head# 2024-03-30T13:50:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78c710d7:::benchmark_data_smithi043_119832_object14431:head# 2024-03-30T13:50:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78c78f2e:::benchmark_data_smithi043_119832_object8028:head# 2024-03-30T13:50:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78c989a6:::benchmark_data_smithi043_119832_object52630:head# 2024-03-30T13:50:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78c9b2b1:::benchmark_data_smithi043_119832_object26204:head# 2024-03-30T13:50:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78ca421f:::benchmark_data_smithi043_119832_object44206:head# 2024-03-30T13:50:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78cbddab:::benchmark_data_smithi043_119832_object42159:head# 2024-03-30T13:50:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78cc83d8:::benchmark_data_smithi043_119832_object85595:head# 2024-03-30T13:50:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78ce5c4d:::benchmark_data_smithi043_119832_object17691:head# 2024-03-30T13:50:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78cfc2d3:::benchmark_data_smithi043_119832_object61022:head# 2024-03-30T13:50:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78d0b525:::benchmark_data_smithi043_119832_object82742:head# 2024-03-30T13:50:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78d14659:::benchmark_data_smithi043_119832_object12641:head# 2024-03-30T13:50:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78d1ecb5:::benchmark_data_smithi043_119832_object239:head# 2024-03-30T13:50:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78d30ced:::benchmark_data_smithi043_119832_object60227:head# 2024-03-30T13:50:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78d33fac:::benchmark_data_smithi043_119832_object71313:head# 2024-03-30T13:50:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78d477ab:::benchmark_data_smithi043_119832_object32634:head# 2024-03-30T13:50:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78d54623:::benchmark_data_smithi043_119832_object77167:head# 2024-03-30T13:50:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78d57e18:::benchmark_data_smithi043_119832_object5959:head# 2024-03-30T13:50:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78d5ba25:::benchmark_data_smithi043_119832_object34958:head# 2024-03-30T13:50:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78d5ce8d:::benchmark_data_smithi043_119832_object14703:head# 2024-03-30T13:50:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78d64647:::benchmark_data_smithi043_119832_object59323:head# 2024-03-30T13:50:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78d74099:::benchmark_data_smithi043_119832_object658:head# 2024-03-30T13:50:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78d851b7:::benchmark_data_smithi043_119832_object70808:head# 2024-03-30T13:50:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78d8c31d:::benchmark_data_smithi043_119832_object36750:head# 2024-03-30T13:50:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78d97fec:::benchmark_data_smithi043_119832_object26871:head# 2024-03-30T13:50:33.412 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78da8577:::benchmark_data_smithi043_119832_object80967:head# 2024-03-30T13:50:33.412 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78dc02c1:::benchmark_data_smithi043_119832_object59966:head# 2024-03-30T13:50:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78dc7296:::benchmark_data_smithi043_119832_object58685:head# 2024-03-30T13:50:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78dc9631:::benchmark_data_smithi043_119832_object17401:head# 2024-03-30T13:50:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78dd479b:::benchmark_data_smithi043_119832_object28012:head# 2024-03-30T13:50:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78dd9f07:::benchmark_data_smithi043_119832_object70208:head# 2024-03-30T13:50:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78ddd439:::benchmark_data_smithi043_119832_object81650:head# 2024-03-30T13:50:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78de4670:::benchmark_data_smithi043_119832_object13446:head# 2024-03-30T13:50:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78de9a65:::benchmark_data_smithi043_119832_object74065:head# 2024-03-30T13:50:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78deada7:::benchmark_data_smithi043_119832_object65271:head# 2024-03-30T13:50:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78e229d6:::benchmark_data_smithi043_119832_object27954:head# 2024-03-30T13:50:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78e279da:::benchmark_data_smithi043_119832_object13052:head# 2024-03-30T13:50:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78e33aaf:::benchmark_data_smithi043_119832_object32071:head# 2024-03-30T13:50:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78e37fd4:::benchmark_data_smithi043_119832_object62963:head# 2024-03-30T13:50:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78e48fb2:::benchmark_data_smithi043_119832_object29349:head# 2024-03-30T13:50:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78e746a7:::benchmark_data_smithi043_119832_object61371:head# 2024-03-30T13:50:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78e81442:::benchmark_data_smithi043_119832_object3157:head# 2024-03-30T13:50:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78e9595b:::benchmark_data_smithi043_119832_object36975:head# 2024-03-30T13:50:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78ea1183:::benchmark_data_smithi043_119832_object31129:head# 2024-03-30T13:50:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78ea4238:::benchmark_data_smithi043_119832_object70463:head# 2024-03-30T13:50:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78eb1e59:::benchmark_data_smithi043_119832_object82450:head# 2024-03-30T13:50:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78eb2624:::benchmark_data_smithi043_119832_object62162:head# 2024-03-30T13:50:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78ec043e:::benchmark_data_smithi043_119832_object28811:head# 2024-03-30T13:50:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78ec8ffe:::benchmark_data_smithi043_119832_object31280:head# 2024-03-30T13:50:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78edab5b:::benchmark_data_smithi043_119832_object4220:head# 2024-03-30T13:50:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78edb02b:::benchmark_data_smithi043_119832_object11851:head# 2024-03-30T13:50:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78ee4c66:::benchmark_data_smithi043_119832_object22128:head# 2024-03-30T13:50:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78f13142:::benchmark_data_smithi043_119832_object39313:head# 2024-03-30T13:50:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78f14ca4:::benchmark_data_smithi043_119832_object10739:head# 2024-03-30T13:50:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78f17ad1:::benchmark_data_smithi043_119832_object69990:head# 2024-03-30T13:50:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78f1d378:::benchmark_data_smithi043_119832_object13660:head# 2024-03-30T13:50:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78f3f95b:::benchmark_data_smithi043_119832_object59947:head# 2024-03-30T13:50:33.421 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78f4612d:::benchmark_data_smithi043_119832_object34351:head# 2024-03-30T13:50:33.421 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78f58c6d:::benchmark_data_smithi043_119832_object18186:head# 2024-03-30T13:50:33.421 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78f5c5f6:::benchmark_data_smithi043_119832_object43489:head# 2024-03-30T13:50:33.422 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78f787ca:::benchmark_data_smithi043_119832_object3683:head# 2024-03-30T13:50:33.422 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78f7e3b3:::benchmark_data_smithi043_119832_object604:head# 2024-03-30T13:50:33.422 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78f86a66:::benchmark_data_smithi043_119832_object51130:head# 2024-03-30T13:50:33.422 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78f8883a:::benchmark_data_smithi043_119832_object66396:head# 2024-03-30T13:50:33.422 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78f9a75e:::benchmark_data_smithi043_119832_object39353:head# 2024-03-30T13:50:33.422 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78f9f279:::benchmark_data_smithi043_119832_object54042:head# 2024-03-30T13:50:33.422 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78f9fbaa:::benchmark_data_smithi043_119832_object49235:head# 2024-03-30T13:50:33.423 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78fa54cb:::benchmark_data_smithi043_119832_object7273:head# 2024-03-30T13:50:33.423 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78fac415:::benchmark_data_smithi043_119832_object31759:head# 2024-03-30T13:50:33.424 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78fb7354:::benchmark_data_smithi043_119832_object8169:head# 2024-03-30T13:50:33.424 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78fc8084:::benchmark_data_smithi043_119832_object25441:head# 2024-03-30T13:50:33.424 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78fd64f3:::benchmark_data_smithi043_119832_object21058:head# 2024-03-30T13:50:33.424 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78fdf767:::benchmark_data_smithi043_119832_object25902:head# 2024-03-30T13:50:33.424 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78fe4a80:::benchmark_data_smithi043_119832_object80149:head# 2024-03-30T13:50:33.425 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78fee2a6:::benchmark_data_smithi043_119832_object26197:head# 2024-03-30T13:50:33.425 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78fefdd3:::benchmark_data_smithi043_119832_object16284:head# 2024-03-30T13:50:33.425 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:78ff14b5:::benchmark_data_smithi043_119832_object29524:head# 2024-03-30T13:50:33.425 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:790152a5:::benchmark_data_smithi043_119832_object40719:head# 2024-03-30T13:50:33.450 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:790297f7:::benchmark_data_smithi043_119832_object50502:head# 2024-03-30T13:50:33.450 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7902b3be:::benchmark_data_smithi043_119832_object78137:head# 2024-03-30T13:50:33.450 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7902b56b:::benchmark_data_smithi043_119832_object5551:head# 2024-03-30T13:50:33.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79030e6c:::benchmark_data_smithi043_119832_object78569:head# 2024-03-30T13:50:33.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7903541a:::benchmark_data_smithi043_119832_object75768:head# 2024-03-30T13:50:33.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7903e2ad:::benchmark_data_smithi043_119832_object67831:head# 2024-03-30T13:50:33.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7904552c:::benchmark_data_smithi043_119832_object13642:head# 2024-03-30T13:50:33.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79054555:::benchmark_data_smithi043_119832_object40645:head# 2024-03-30T13:50:33.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79064181:::benchmark_data_smithi043_119832_object36266:head# 2024-03-30T13:50:33.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7907807c:::benchmark_data_smithi043_119832_object85877:head# 2024-03-30T13:50:33.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7908024e:::benchmark_data_smithi043_119832_object45267:head# 2024-03-30T13:50:33.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7909ad17:::benchmark_data_smithi043_119832_object82355:head# 2024-03-30T13:50:33.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:790a55aa:::benchmark_data_smithi043_119832_object85691:head# 2024-03-30T13:50:33.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:790b17de:::benchmark_data_smithi043_119832_object20981:head# 2024-03-30T13:50:33.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:790b6128:::benchmark_data_smithi043_119832_object52898:head# 2024-03-30T13:50:33.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:790bc3ee:::benchmark_data_smithi043_119832_object46940:head# 2024-03-30T13:50:33.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:790dc2b0:::benchmark_data_smithi043_119832_object82310:head# 2024-03-30T13:50:33.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:790e38e9:::benchmark_data_smithi043_119832_object68607:head# 2024-03-30T13:50:33.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:790e7301:::benchmark_data_smithi043_119832_object63163:head# 2024-03-30T13:50:33.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:790f6f4d:::benchmark_data_smithi043_119832_object69181:head# 2024-03-30T13:50:33.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:790fcbfe:::benchmark_data_smithi043_119832_object24745:head# 2024-03-30T13:50:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79107704:::benchmark_data_smithi043_119832_object39969:head# 2024-03-30T13:50:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79111c32:::benchmark_data_smithi043_119832_object25684:head# 2024-03-30T13:50:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7911a941:::benchmark_data_smithi043_119832_object88060:head# 2024-03-30T13:50:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7912a844:::benchmark_data_smithi043_119832_object18644:head# 2024-03-30T13:50:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7912c68e:::benchmark_data_smithi043_119832_object47953:head# 2024-03-30T13:50:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79149bd6:::benchmark_data_smithi043_119832_object60396:head# 2024-03-30T13:50:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7914e728:::benchmark_data_smithi043_119832_object87508:head# 2024-03-30T13:50:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7916a141:::benchmark_data_smithi043_119832_object70993:head# 2024-03-30T13:50:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79176bc6:::benchmark_data_smithi043_119832_object69345:head# 2024-03-30T13:50:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:791784c4:::benchmark_data_smithi043_119832_object15522:head# 2024-03-30T13:50:33.454 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79189cc3:::benchmark_data_smithi043_119832_object59231:head# 2024-03-30T13:50:33.454 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:791a4bfb:::benchmark_data_smithi043_119832_object55691:head# 2024-03-30T13:50:33.454 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:791b4736:::benchmark_data_smithi043_119832_object49826:head# 2024-03-30T13:50:33.454 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:791c04d4:::benchmark_data_smithi043_119832_object37785:head# 2024-03-30T13:50:33.454 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:791c18eb:::benchmark_data_smithi043_119832_object73950:head# 2024-03-30T13:50:33.454 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:791c6013:::benchmark_data_smithi043_119832_object41323:head# 2024-03-30T13:50:33.454 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:791cd740:::benchmark_data_smithi043_119832_object80297:head# 2024-03-30T13:50:33.454 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:791d6b49:::benchmark_data_smithi043_119832_object81387:head# 2024-03-30T13:50:33.454 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:791d7c82:::benchmark_data_smithi043_119832_object15189:head# 2024-03-30T13:50:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:791de215:::benchmark_data_smithi043_119832_object78431:head# 2024-03-30T13:50:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:791e9812:::benchmark_data_smithi043_119832_object53877:head# 2024-03-30T13:50:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:791f0314:::benchmark_data_smithi043_119832_object17253:head# 2024-03-30T13:50:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:791f8da0:::benchmark_data_smithi043_119832_object33921:head# 2024-03-30T13:50:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:791fc27e:::benchmark_data_smithi043_119832_object58736:head# 2024-03-30T13:50:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:791fc2d6:::benchmark_data_smithi043_119832_object48857:head# 2024-03-30T13:50:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79203572:::benchmark_data_smithi043_119832_object28604:head# 2024-03-30T13:50:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79207b6d:::benchmark_data_smithi043_119832_object5748:head# 2024-03-30T13:50:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7921ad29:::benchmark_data_smithi043_119832_object76359:head# 2024-03-30T13:50:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7923420b:::benchmark_data_smithi043_119832_object27441:head# 2024-03-30T13:50:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:792535f5:::benchmark_data_smithi043_119832_object71523:head# 2024-03-30T13:50:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7925af7d:::benchmark_data_smithi043_119832_object64468:head# 2024-03-30T13:50:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7928af38:::benchmark_data_smithi043_119832_object45824:head# 2024-03-30T13:50:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7928e87e:::benchmark_data_smithi043_119832_object19341:head# 2024-03-30T13:50:33.456 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7928f4db:::benchmark_data_smithi043_119832_object40027:head# 2024-03-30T13:50:33.456 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:792b08f9:::benchmark_data_smithi043_119832_object47415:head# 2024-03-30T13:50:33.456 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:792b1bee:::benchmark_data_smithi043_119832_object19538:head# 2024-03-30T13:50:33.456 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:792bf7c5:::benchmark_data_smithi043_119832_object76665:head# 2024-03-30T13:50:33.456 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:792ca53e:::benchmark_data_smithi043_119832_object49369:head# 2024-03-30T13:50:33.456 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:792cd1f0:::benchmark_data_smithi043_119832_object31834:head# 2024-03-30T13:50:33.456 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:792f4147:::benchmark_data_smithi043_119832_object67843:head# 2024-03-30T13:50:33.456 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:792f91f8:::benchmark_data_smithi043_119832_object69709:head# 2024-03-30T13:50:33.456 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7930850a:::benchmark_data_smithi043_119832_object65519:head# 2024-03-30T13:50:33.456 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79310a5b:::benchmark_data_smithi043_119832_object14079:head# 2024-03-30T13:50:33.456 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79315653:::benchmark_data_smithi043_119832_object50723:head# 2024-03-30T13:50:33.461 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79333820:::benchmark_data_smithi043_119832_object38323:head# 2024-03-30T13:50:33.461 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7936763f:::benchmark_data_smithi043_119832_object63310:head# 2024-03-30T13:50:33.461 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:793690b2:::benchmark_data_smithi043_119832_object10142:head# 2024-03-30T13:50:33.461 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7936e207:::benchmark_data_smithi043_119832_object14839:head# 2024-03-30T13:50:33.461 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79375f1a:::benchmark_data_smithi043_119832_object61899:head# 2024-03-30T13:50:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79392662:::benchmark_data_smithi043_119832_object3786:head# 2024-03-30T13:50:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7939ac2a:::benchmark_data_smithi043_119832_object78002:head# 2024-03-30T13:50:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:793a65e0:::benchmark_data_smithi043_119832_object22728:head# 2024-03-30T13:50:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:793ecde0:::benchmark_data_smithi043_119832_object26702:head# 2024-03-30T13:50:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:793ef0bf:::benchmark_data_smithi043_119832_object79568:head# 2024-03-30T13:50:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:793f1736:::benchmark_data_smithi043_119832_object51725:head# 2024-03-30T13:50:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:793fe3c9:::benchmark_data_smithi043_119832_object12276:head# 2024-03-30T13:50:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79402344:::benchmark_data_smithi043_119832_object44590:head# 2024-03-30T13:50:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79406c87:::benchmark_data_smithi043_119832_object22637:head# 2024-03-30T13:50:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7940e1f9:::benchmark_data_smithi043_119832_object41588:head# 2024-03-30T13:50:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79410546:::benchmark_data_smithi043_119832_object49538:head# 2024-03-30T13:50:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79418161:::benchmark_data_smithi043_119832_object87325:head# 2024-03-30T13:50:33.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79427fba:::benchmark_data_smithi043_119832_object64636:head# 2024-03-30T13:50:33.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7943cb3b:::benchmark_data_smithi043_119832_object45542:head# 2024-03-30T13:50:33.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7944124e:::benchmark_data_smithi043_119832_object63621:head# 2024-03-30T13:50:33.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79461fdd:::benchmark_data_smithi043_119832_object31818:head# 2024-03-30T13:50:33.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79474507:::benchmark_data_smithi043_119832_object53676:head# 2024-03-30T13:50:33.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7949628b:::benchmark_data_smithi043_119832_object33109:head# 2024-03-30T13:50:33.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:794afc53:::benchmark_data_smithi043_119832_object51824:head# 2024-03-30T13:50:33.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:794d39fd:::benchmark_data_smithi043_119832_object51074:head# 2024-03-30T13:50:33.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:794d5590:::benchmark_data_smithi043_119832_object61551:head# 2024-03-30T13:50:33.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:794e4424:::benchmark_data_smithi043_119832_object10617:head# 2024-03-30T13:50:33.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:794e6f9c:::benchmark_data_smithi043_119832_object55079:head# 2024-03-30T13:50:33.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:794fe4fd:::benchmark_data_smithi043_119832_object69995:head# 2024-03-30T13:50:33.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:795119d2:::benchmark_data_smithi043_119832_object42005:head# 2024-03-30T13:50:33.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7951e446:::benchmark_data_smithi043_119832_object50331:head# 2024-03-30T13:50:33.464 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79520719:::benchmark_data_smithi043_119832_object72721:head# 2024-03-30T13:50:33.464 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7952c37f:::benchmark_data_smithi043_119832_object21732:head# 2024-03-30T13:50:33.464 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7953ff0b:::benchmark_data_smithi043_119832_object52154:head# 2024-03-30T13:50:33.464 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:795502f9:::benchmark_data_smithi043_119832_object3281:head# 2024-03-30T13:50:33.464 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:33.467 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7955be91:::benchmark_data_smithi043_119832_object17173:head# 2024-03-30T13:50:33.467 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79579bb1:::benchmark_data_smithi043_119832_object21125:head# 2024-03-30T13:50:33.467 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79595383:::benchmark_data_smithi043_119832_object1057:head# 2024-03-30T13:50:33.467 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:795ad6f0:::benchmark_data_smithi043_119832_object44191:head# 2024-03-30T13:50:33.467 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:795b1914:::benchmark_data_smithi043_119832_object14272:head# 2024-03-30T13:50:33.467 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:795b26ca:::benchmark_data_smithi043_119832_object54660:head# 2024-03-30T13:50:33.468 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:795b6dd1:::benchmark_data_smithi043_119832_object48947:head# 2024-03-30T13:50:33.468 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:795bcace:::benchmark_data_smithi043_119832_object58565:head# 2024-03-30T13:50:33.468 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:795c4f18:::benchmark_data_smithi043_119832_object83204:head# 2024-03-30T13:50:33.468 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:795d04d5:::benchmark_data_smithi043_119832_object39267:head# 2024-03-30T13:50:33.468 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:795d9917:::benchmark_data_smithi043_119832_object53210:head# 2024-03-30T13:50:33.468 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:795fd6a5:::benchmark_data_smithi043_119832_object47278:head# 2024-03-30T13:50:33.468 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:796110ac:::benchmark_data_smithi043_119832_object42819:head# 2024-03-30T13:50:33.468 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:796251c6:::benchmark_data_smithi043_119832_object59228:head# 2024-03-30T13:50:33.468 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7963e69a:::benchmark_data_smithi043_119832_object31094:head# 2024-03-30T13:50:33.470 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79647947:::benchmark_data_smithi043_119832_object37470:head# 2024-03-30T13:50:33.470 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7965706f:::benchmark_data_smithi043_119832_object52676:head# 2024-03-30T13:50:33.470 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7965f15c:::benchmark_data_smithi043_119832_object29408:head# 2024-03-30T13:50:33.471 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79665fb4:::benchmark_data_smithi043_119832_object56199:head# 2024-03-30T13:50:33.471 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7967dc54:::benchmark_data_smithi043_119832_object80178:head# 2024-03-30T13:50:33.471 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7967e5a3:::benchmark_data_smithi043_119832_object57153:head# 2024-03-30T13:50:33.472 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:796aa4b8:::benchmark_data_smithi043_119832_object6595:head# 2024-03-30T13:50:33.472 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:796b1df4:::benchmark_data_smithi043_119832_object20247:head# 2024-03-30T13:50:33.472 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:796b2853:::benchmark_data_smithi043_119832_object34869:head# 2024-03-30T13:50:33.474 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:796b6f89:::benchmark_data_smithi043_119832_object1498:head# 2024-03-30T13:50:33.474 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:796cf235:::benchmark_data_smithi043_119832_object74654:head# 2024-03-30T13:50:33.474 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:796d1113:::benchmark_data_smithi043_119832_object35663:head# 2024-03-30T13:50:33.474 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:796eb1eb:::benchmark_data_smithi043_119832_object56831:head# 2024-03-30T13:50:33.474 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:796fb189:::benchmark_data_smithi043_119832_object47341:head# 2024-03-30T13:50:33.474 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:797091b7:::benchmark_data_smithi043_119832_object68552:head# 2024-03-30T13:50:33.475 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7971f192:::benchmark_data_smithi043_119832_object21163:head# 2024-03-30T13:50:33.476 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7972507d:::benchmark_data_smithi043_119832_object3222:head# 2024-03-30T13:50:33.476 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7972756e:::benchmark_data_smithi043_119832_object64100:head# 2024-03-30T13:50:33.478 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7972a8ce:::benchmark_data_smithi043_119832_object69918:head# 2024-03-30T13:50:33.478 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7973bde0:::benchmark_data_smithi043_119832_object65478:head# 2024-03-30T13:50:33.478 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7974d3ee:::benchmark_data_smithi043_119832_object68025:head# 2024-03-30T13:50:33.478 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7975b2d4:::benchmark_data_smithi043_119832_object67811:head# 2024-03-30T13:50:33.478 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79767781:::benchmark_data_smithi043_119832_object64926:head# 2024-03-30T13:50:33.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7977b6c9:::benchmark_data_smithi043_119832_object6727:head# 2024-03-30T13:50:33.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79783b32:::benchmark_data_smithi043_119832_object12412:head# 2024-03-30T13:50:33.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79794de0:::benchmark_data_smithi043_119832_object32033:head# 2024-03-30T13:50:33.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:797b0533:::benchmark_data_smithi043_119832_object61997:head# 2024-03-30T13:50:33.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:797c340e:::benchmark_data_smithi043_119832_object32611:head# 2024-03-30T13:50:33.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:797d70b8:::benchmark_data_smithi043_119832_object63799:head# 2024-03-30T13:50:33.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:797e26c7:::benchmark_data_smithi043_119832_object50389:head# 2024-03-30T13:50:33.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:797e4dfe:::benchmark_data_smithi043_119832_object67715:head# 2024-03-30T13:50:33.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:797e5cf8:::benchmark_data_smithi043_119832_object86399:head# 2024-03-30T13:50:33.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:797f3514:::benchmark_data_smithi043_119832_object29373:head# 2024-03-30T13:50:33.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:797f7654:::benchmark_data_smithi043_119832_object87531:head# 2024-03-30T13:50:33.485 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:797fabce:::benchmark_data_smithi043_119832_object66753:head# 2024-03-30T13:50:33.485 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:797ff94f:::benchmark_data_smithi043_119832_object25626:head# 2024-03-30T13:50:33.485 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7980e0d1:::benchmark_data_smithi043_119832_object21490:head# 2024-03-30T13:50:33.485 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79812263:::benchmark_data_smithi043_119832_object27874:head# 2024-03-30T13:50:33.485 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:798177fc:::benchmark_data_smithi043_119832_object73816:head# 2024-03-30T13:50:33.486 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79827505:::benchmark_data_smithi043_119832_object41408:head# 2024-03-30T13:50:33.488 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79831053:::benchmark_data_smithi043_119832_object6813:head# 2024-03-30T13:50:33.488 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7983c652:::benchmark_data_smithi043_119832_object15830:head# 2024-03-30T13:50:33.488 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7984a59b:::benchmark_data_smithi043_119832_object72216:head# 2024-03-30T13:50:33.488 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7984a92e:::benchmark_data_smithi043_119832_object25349:head# 2024-03-30T13:50:33.488 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79853eb4:::benchmark_data_smithi043_119832_object35172:head# 2024-03-30T13:50:33.488 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7986796a:::benchmark_data_smithi043_119832_object66016:head# 2024-03-30T13:50:33.488 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79878980:::benchmark_data_smithi043_119832_object26072:head# 2024-03-30T13:50:33.490 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7987ddc8:::benchmark_data_smithi043_119832_object23330:head# 2024-03-30T13:50:33.490 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79884ca7:::benchmark_data_smithi043_119832_object81405:head# 2024-03-30T13:50:33.490 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7989a75e:::benchmark_data_smithi043_119832_object65711:head# 2024-03-30T13:50:33.490 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:798a513e:::benchmark_data_smithi043_119832_object37040:head# 2024-03-30T13:50:33.490 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:798a9804:::benchmark_data_smithi043_119832_object3347:head# 2024-03-30T13:50:33.491 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:798ae180:::benchmark_data_smithi043_119832_object61308:head# 2024-03-30T13:50:33.491 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:798afdf1:::benchmark_data_smithi043_119832_object51447:head# 2024-03-30T13:50:33.491 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:798b3fbf:::benchmark_data_smithi043_119832_object86904:head# 2024-03-30T13:50:33.491 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:798b7eed:::benchmark_data_smithi043_119832_object48921:head# 2024-03-30T13:50:33.492 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:798bacc9:::benchmark_data_smithi043_119832_object20259:head# 2024-03-30T13:50:33.492 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:798d8f18:::benchmark_data_smithi043_119832_object88288:head# 2024-03-30T13:50:33.492 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:798e57e0:::benchmark_data_smithi043_119832_object14359:head# 2024-03-30T13:50:33.498 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:798e93f5:::benchmark_data_smithi043_119832_object69064:head# 2024-03-30T13:50:33.498 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:798e9b50:::benchmark_data_smithi043_119832_object17048:head# 2024-03-30T13:50:33.498 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:798f525d:::benchmark_data_smithi043_119832_object18429:head# 2024-03-30T13:50:33.498 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:798fd92b:::benchmark_data_smithi043_119832_object79503:head# 2024-03-30T13:50:33.498 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:798fdef2:::benchmark_data_smithi043_119832_object32861:head# 2024-03-30T13:50:33.498 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7990b1fc:::benchmark_data_smithi043_119832_object17042:head# 2024-03-30T13:50:33.498 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7991ec4e:::benchmark_data_smithi043_119832_object84941:head# 2024-03-30T13:50:33.498 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7991f661:::benchmark_data_smithi043_119832_object73876:head# 2024-03-30T13:50:33.498 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:799322d5:::benchmark_data_smithi043_119832_object6263:head# 2024-03-30T13:50:33.499 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7993a8bb:::benchmark_data_smithi043_119832_object28678:head# 2024-03-30T13:50:33.499 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79959058:::benchmark_data_smithi043_119832_object78746:head# 2024-03-30T13:50:33.499 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:799632e9:::benchmark_data_smithi043_119832_object16559:head# 2024-03-30T13:50:33.499 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79976655:::benchmark_data_smithi043_119832_object41583:head# 2024-03-30T13:50:33.501 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7998a2cc:::benchmark_data_smithi043_119832_object53460:head# 2024-03-30T13:50:33.501 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7998b50b:::benchmark_data_smithi043_119832_object38437:head# 2024-03-30T13:50:33.501 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:799922b5:::benchmark_data_smithi043_119832_object42781:head# 2024-03-30T13:50:33.501 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79994a46:::benchmark_data_smithi043_119832_object73105:head# 2024-03-30T13:50:33.501 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:799ad006:::benchmark_data_smithi043_119832_object1512:head# 2024-03-30T13:50:33.501 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:799bce63:::benchmark_data_smithi043_119832_object9228:head# 2024-03-30T13:50:33.501 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:799d4069:::benchmark_data_smithi043_119832_object22595:head# 2024-03-30T13:50:33.501 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:799e2255:::benchmark_data_smithi043_119832_object45234:head# 2024-03-30T13:50:33.504 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:799eb555:::benchmark_data_smithi043_119832_object66749:head# 2024-03-30T13:50:33.504 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:799eda35:::benchmark_data_smithi043_119832_object79866:head# 2024-03-30T13:50:33.504 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:799f4faa:::benchmark_data_smithi043_119832_object27435:head# 2024-03-30T13:50:33.504 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:799f7f1c:::benchmark_data_smithi043_119832_object11814:head# 2024-03-30T13:50:33.504 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:799f8717:::benchmark_data_smithi043_119832_object5711:head# 2024-03-30T13:50:33.504 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79a1c509:::benchmark_data_smithi043_119832_object5552:head# 2024-03-30T13:50:33.504 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79a2448d:::benchmark_data_smithi043_119832_object71286:head# 2024-03-30T13:50:33.505 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79a41be0:::benchmark_data_smithi043_119832_object13657:head# 2024-03-30T13:50:33.505 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79a463e6:::benchmark_data_smithi043_119832_object28988:head# 2024-03-30T13:50:33.505 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79a6df30:::benchmark_data_smithi043_119832_object39757:head# 2024-03-30T13:50:33.506 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79a8690c:::benchmark_data_smithi043_119832_object11758:head# 2024-03-30T13:50:33.506 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79a87058:::benchmark_data_smithi043_119832_object58682:head# 2024-03-30T13:50:33.506 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79a9a1f4:::benchmark_data_smithi043_119832_object69645:head# 2024-03-30T13:50:33.507 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79aab952:::benchmark_data_smithi043_119832_object41995:head# 2024-03-30T13:50:33.507 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79ab1892:::benchmark_data_smithi043_119832_object84449:head# 2024-03-30T13:50:33.507 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79abb8d1:::benchmark_data_smithi043_119832_object7739:head# 2024-03-30T13:50:33.507 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79ac1de2:::benchmark_data_smithi043_119832_object71137:head# 2024-03-30T13:50:33.508 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79ace356:::benchmark_data_smithi043_119832_object28105:head# 2024-03-30T13:50:33.508 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79adec33:::benchmark_data_smithi043_119832_object39046:head# 2024-03-30T13:50:33.509 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79af4c69:::benchmark_data_smithi043_119832_object21153:head# 2024-03-30T13:50:33.510 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79aff49a:::benchmark_data_smithi043_119832_object24307:head# 2024-03-30T13:50:33.510 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b07377:::benchmark_data_smithi043_119832_object28288:head# 2024-03-30T13:50:33.510 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b07902:::benchmark_data_smithi043_119832_object81611:head# 2024-03-30T13:50:33.511 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b0e3e3:::benchmark_data_smithi043_119832_object83468:head# 2024-03-30T13:50:33.511 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b154d3:::benchmark_data_smithi043_119832_object75961:head# 2024-03-30T13:50:33.511 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b176db:::benchmark_data_smithi043_119832_object48575:head# 2024-03-30T13:50:33.512 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b28b50:::benchmark_data_smithi043_119832_object48835:head# 2024-03-30T13:50:33.512 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b317df:::benchmark_data_smithi043_119832_object49085:head# 2024-03-30T13:50:33.512 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b31dd5:::benchmark_data_smithi043_119832_object27252:head# 2024-03-30T13:50:33.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:33.513 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:33.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:33.541 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b38fb7:::benchmark_data_smithi043_119832_object10324:head# 2024-03-30T13:50:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b39743:::benchmark_data_smithi043_119832_object20428:head# 2024-03-30T13:50:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b3a1ab:::benchmark_data_smithi043_119832_object4891:head# 2024-03-30T13:50:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b4141b:::benchmark_data_smithi043_119832_object54503:head# 2024-03-30T13:50:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b46c6d:::benchmark_data_smithi043_119832_object73979:head# 2024-03-30T13:50:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b47b2f:::benchmark_data_smithi043_119832_object86197:head# 2024-03-30T13:50:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b693df:::benchmark_data_smithi043_119832_object23697:head# 2024-03-30T13:50:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b6c102:::benchmark_data_smithi043_119832_object46615:head# 2024-03-30T13:50:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b7d353:::benchmark_data_smithi043_119832_object85520:head# 2024-03-30T13:50:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b82ad1:::benchmark_data_smithi043_119832_object51821:head# 2024-03-30T13:50:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b8b770:::benchmark_data_smithi043_119832_object35065:head# 2024-03-30T13:50:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b90e2c:::benchmark_data_smithi043_119832_object75394:head# 2024-03-30T13:50:33.542 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79b9190f:::benchmark_data_smithi043_119832_object16713:head# 2024-03-30T13:50:33.543 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79ba2487:::benchmark_data_smithi043_119832_object11496:head# 2024-03-30T13:50:33.543 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79ba3cc4:::benchmark_data_smithi043_119832_object48341:head# 2024-03-30T13:50:33.543 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79ba8cb1:::benchmark_data_smithi043_119832_object41205:head# 2024-03-30T13:50:33.543 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79bd0268:::benchmark_data_smithi043_119832_object11854:head# 2024-03-30T13:50:33.543 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79bfb46d:::benchmark_data_smithi043_119832_object61491:head# 2024-03-30T13:50:33.543 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79c01479:::benchmark_data_smithi043_119832_object37424:head# 2024-03-30T13:50:33.543 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79c0c925:::benchmark_data_smithi043_119832_object74181:head# 2024-03-30T13:50:33.543 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79c2cb90:::benchmark_data_smithi043_119832_object28956:head# 2024-03-30T13:50:33.543 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79c3a5df:::benchmark_data_smithi043_119832_object75266:head# 2024-03-30T13:50:33.543 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79c4055c:::benchmark_data_smithi043_119832_object59366:head# 2024-03-30T13:50:33.543 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79c57982:::benchmark_data_smithi043_119832_object85863:head# 2024-03-30T13:50:33.543 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79c81244:::benchmark_data_smithi043_119832_object13774:head# 2024-03-30T13:50:33.543 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79c86f8e:::benchmark_data_smithi043_119832_object22917:head# 2024-03-30T13:50:33.544 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79c98f5e:::benchmark_data_smithi043_119832_object22347:head# 2024-03-30T13:50:33.544 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79cafb93:::benchmark_data_smithi043_119832_object82323:head# 2024-03-30T13:50:33.544 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79cb5484:::benchmark_data_smithi043_119832_object83999:head# 2024-03-30T13:50:33.544 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79cb6219:::benchmark_data_smithi043_119832_object53478:head# 2024-03-30T13:50:33.544 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79cd4995:::benchmark_data_smithi043_119832_object76770:head# 2024-03-30T13:50:33.544 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79cd8346:::benchmark_data_smithi043_119832_object19549:head# 2024-03-30T13:50:33.544 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79ceb312:::benchmark_data_smithi043_119832_object58324:head# 2024-03-30T13:50:33.544 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79ceeb05:::benchmark_data_smithi043_119832_object80979:head# 2024-03-30T13:50:33.544 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79cfa701:::benchmark_data_smithi043_119832_object76270:head# 2024-03-30T13:50:33.544 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79d08755:::benchmark_data_smithi043_119832_object4260:head# 2024-03-30T13:50:33.544 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79d1145e:::benchmark_data_smithi043_119832_object57603:head# 2024-03-30T13:50:33.544 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79d1f7e9:::benchmark_data_smithi043_119832_object37879:head# 2024-03-30T13:50:33.544 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79d242a6:::benchmark_data_smithi043_119832_object56629:head# 2024-03-30T13:50:33.544 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79d2a217:::benchmark_data_smithi043_119832_object33962:head# 2024-03-30T13:50:33.545 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79d2af99:::benchmark_data_smithi043_119832_object72991:head# 2024-03-30T13:50:33.545 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79d407ba:::benchmark_data_smithi043_119832_object22568:head# 2024-03-30T13:50:33.545 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79d5598f:::benchmark_data_smithi043_119832_object43076:head# 2024-03-30T13:50:33.545 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79d5d485:::benchmark_data_smithi043_119832_object71647:head# 2024-03-30T13:50:33.545 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79d5d86a:::benchmark_data_smithi043_119832_object13221:head# 2024-03-30T13:50:33.545 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79d5ee81:::benchmark_data_smithi043_119832_object2136:head# 2024-03-30T13:50:33.545 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79d63443:::benchmark_data_smithi043_119832_object17862:head# 2024-03-30T13:50:33.545 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79d69e42:::benchmark_data_smithi043_119832_object83841:head# 2024-03-30T13:50:33.545 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79d6a844:::benchmark_data_smithi043_119832_object16292:head# 2024-03-30T13:50:33.545 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79d6ad96:::benchmark_data_smithi043_119832_object88214:head# 2024-03-30T13:50:33.545 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79d7f9a0:::benchmark_data_smithi043_119832_object53872:head# 2024-03-30T13:50:33.545 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79d80d8f:::benchmark_data_smithi043_119832_object12973:head# 2024-03-30T13:50:33.545 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79da5f02:::benchmark_data_smithi043_119832_object41269:head# 2024-03-30T13:50:33.546 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79db358c:::benchmark_data_smithi043_119832_object72849:head# 2024-03-30T13:50:33.546 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79db3688:::benchmark_data_smithi043_119832_object15923:head# 2024-03-30T13:50:33.546 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79db9fb1:::benchmark_data_smithi043_119832_object10972:head# 2024-03-30T13:50:33.546 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79dc2950:::benchmark_data_smithi043_119832_object10471:head# 2024-03-30T13:50:33.546 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79dce77d:::benchmark_data_smithi043_119832_object55523:head# 2024-03-30T13:50:33.546 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79dd3cd7:::benchmark_data_smithi043_119832_object78317:head# 2024-03-30T13:50:33.546 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79dd55b7:::benchmark_data_smithi043_119832_object35612:head# 2024-03-30T13:50:33.546 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79ddefa0:::benchmark_data_smithi043_119832_object48350:head# 2024-03-30T13:50:33.546 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79de15c9:::benchmark_data_smithi043_119832_object41226:head# 2024-03-30T13:50:33.546 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79e17ec0:::benchmark_data_smithi043_119832_object5395:head# 2024-03-30T13:50:33.546 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79e33749:::benchmark_data_smithi043_119832_object78612:head# 2024-03-30T13:50:33.546 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79e45be2:::benchmark_data_smithi043_119832_object71067:head# 2024-03-30T13:50:33.546 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79e536ed:::benchmark_data_smithi043_119832_object78092:head# 2024-03-30T13:50:33.546 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79e7466a:::benchmark_data_smithi043_119832_object59473:head# 2024-03-30T13:50:33.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79e774c2:::benchmark_data_smithi043_119832_object57223:head# 2024-03-30T13:50:33.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79e87f83:::benchmark_data_smithi043_119832_object80709:head# 2024-03-30T13:50:33.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79eb1ec3:::benchmark_data_smithi043_119832_object43635:head# 2024-03-30T13:50:33.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79eb9995:::benchmark_data_smithi043_119832_object72526:head# 2024-03-30T13:50:33.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79ebc027:::benchmark_data_smithi043_119832_object58181:head# 2024-03-30T13:50:33.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79ebf917:::benchmark_data_smithi043_119832_object41311:head# 2024-03-30T13:50:33.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79ecc080:::benchmark_data_smithi043_119832_object84056:head# 2024-03-30T13:50:33.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79ece052:::benchmark_data_smithi043_119832_object33112:head# 2024-03-30T13:50:33.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79ed6765:::benchmark_data_smithi043_119832_object25402:head# 2024-03-30T13:50:33.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79ed70c2:::benchmark_data_smithi043_119832_object6947:head# 2024-03-30T13:50:33.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79ed852d:::benchmark_data_smithi043_119832_object12648:head# 2024-03-30T13:50:33.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79f06c4a:::benchmark_data_smithi043_119832_object23253:head# 2024-03-30T13:50:33.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79f08fdc:::benchmark_data_smithi043_119832_object11454:head# 2024-03-30T13:50:33.547 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79f0ce0a:::benchmark_data_smithi043_119832_object8061:head# 2024-03-30T13:50:33.548 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79f29841:::benchmark_data_smithi043_119832_object12317:head# 2024-03-30T13:50:33.548 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79f2a79b:::benchmark_data_smithi043_119832_object76425:head# 2024-03-30T13:50:33.548 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79f55294:::benchmark_data_smithi043_119832_object3894:head# 2024-03-30T13:50:33.548 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79f65405:::benchmark_data_smithi043_119832_object22593:head# 2024-03-30T13:50:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79f75ebc:::benchmark_data_smithi043_119832_object19502:head# 2024-03-30T13:50:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79f7f7a8:::benchmark_data_smithi043_119832_object64619:head# 2024-03-30T13:50:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79f86fcb:::benchmark_data_smithi043_119832_object86376:head# 2024-03-30T13:50:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79f8f62b:::benchmark_data_smithi043_119832_object29136:head# 2024-03-30T13:50:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79f91eb5:::benchmark_data_smithi043_119832_object64283:head# 2024-03-30T13:50:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79f980e5:::benchmark_data_smithi043_119832_object5902:head# 2024-03-30T13:50:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79f9b64a:::benchmark_data_smithi043_119832_object28670:head# 2024-03-30T13:50:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79f9f170:::benchmark_data_smithi043_119832_object18773:head# 2024-03-30T13:50:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79fe18ec:::benchmark_data_smithi043_119832_object36576:head# 2024-03-30T13:50:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79ff25fd:::benchmark_data_smithi043_119832_object36113:head# 2024-03-30T13:50:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:79ffec33:::benchmark_data_smithi043_119832_object70610:head# 2024-03-30T13:50:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a005fca:::benchmark_data_smithi043_119832_object43002:head# 2024-03-30T13:50:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a0246da:::benchmark_data_smithi043_119832_object79920:head# 2024-03-30T13:50:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a03c68a:::benchmark_data_smithi043_119832_object14672:head# 2024-03-30T13:50:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a041218:::benchmark_data_smithi043_119832_object21880:head# 2024-03-30T13:50:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a045804:::benchmark_data_smithi043_119832_object80691:head# 2024-03-30T13:50:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a04d17f:::benchmark_data_smithi043_119832_object31723:head# 2024-03-30T13:50:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a05986d:::benchmark_data_smithi043_119832_object45904:head# 2024-03-30T13:50:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a064428:::benchmark_data_smithi043_119832_object39733:head# 2024-03-30T13:50:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a07325f:::benchmark_data_smithi043_119832_object54753:head# 2024-03-30T13:50:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a0772a4:::benchmark_data_smithi043_119832_object57481:head# 2024-03-30T13:50:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a07755a:::benchmark_data_smithi043_119832_object52480:head# 2024-03-30T13:50:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a080357:::benchmark_data_smithi043_119832_object71254:head# 2024-03-30T13:50:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a082800:::benchmark_data_smithi043_119832_object57825:head# 2024-03-30T13:50:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a08bf7e:::benchmark_data_smithi043_119832_object1642:head# 2024-03-30T13:50:33.574 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a0904cf:::benchmark_data_smithi043_119832_object15993:head# 2024-03-30T13:50:33.574 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a09547a:::benchmark_data_smithi043_119832_object2616:head# 2024-03-30T13:50:33.574 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a09e954:::benchmark_data_smithi043_119832_object24246:head# 2024-03-30T13:50:33.574 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a0bc4ac:::benchmark_data_smithi043_119832_object64706:head# 2024-03-30T13:50:33.574 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a0dcac1:::benchmark_data_smithi043_119832_object40815:head# 2024-03-30T13:50:33.574 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a0ded86:::benchmark_data_smithi043_119832_object10060:head# 2024-03-30T13:50:33.574 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a0e9a27:::benchmark_data_smithi043_119832_object84861:head# 2024-03-30T13:50:33.574 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a0edf52:::benchmark_data_smithi043_119832_object69667:head# 2024-03-30T13:50:33.574 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a0fdead:::benchmark_data_smithi043_119832_object77944:head# 2024-03-30T13:50:33.574 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a1081c2:::benchmark_data_smithi043_119832_object32988:head# 2024-03-30T13:50:33.574 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a108eec:::benchmark_data_smithi043_119832_object28943:head# 2024-03-30T13:50:33.574 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a11e170:::benchmark_data_smithi043_119832_object10920:head# 2024-03-30T13:50:33.574 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a121598:::benchmark_data_smithi043_119832_object43873:head# 2024-03-30T13:50:33.574 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a12bcf6:::benchmark_data_smithi043_119832_object57090:head# 2024-03-30T13:50:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a15464b:::benchmark_data_smithi043_119832_object36390:head# 2024-03-30T13:50:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a162abe:::benchmark_data_smithi043_119832_object13376:head# 2024-03-30T13:50:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a169c1a:::benchmark_data_smithi043_119832_object58499:head# 2024-03-30T13:50:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a17ec06:::benchmark_data_smithi043_119832_object29926:head# 2024-03-30T13:50:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a17f582:::benchmark_data_smithi043_119832_object32776:head# 2024-03-30T13:50:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a181156:::benchmark_data_smithi043_119832_object78793:head# 2024-03-30T13:50:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a18af0e:::benchmark_data_smithi043_119832_object25453:head# 2024-03-30T13:50:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a1975aa:::benchmark_data_smithi043_119832_object76092:head# 2024-03-30T13:50:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a1b705d:::benchmark_data_smithi043_119832_object59975:head# 2024-03-30T13:50:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a1c258f:::benchmark_data_smithi043_119832_object21944:head# 2024-03-30T13:50:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a1c5803:::benchmark_data_smithi043_119832_object66109:head# 2024-03-30T13:50:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a1df009:::benchmark_data_smithi043_119832_object28477:head# 2024-03-30T13:50:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a1e5de1:::benchmark_data_smithi043_119832_object46801:head# 2024-03-30T13:50:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a1ed7a7:::benchmark_data_smithi043_119832_object45843:head# 2024-03-30T13:50:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a1edb9b:::benchmark_data_smithi043_119832_object47178:head# 2024-03-30T13:50:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a1f16e2:::benchmark_data_smithi043_119832_object59505:head# 2024-03-30T13:50:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a219b67:::benchmark_data_smithi043_119832_object52449:head# 2024-03-30T13:50:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a21c92a:::benchmark_data_smithi043_119832_object69964:head# 2024-03-30T13:50:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a228aa3:::benchmark_data_smithi043_119832_object81723:head# 2024-03-30T13:50:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a23e41e:::benchmark_data_smithi043_119832_object77033:head# 2024-03-30T13:50:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a243857:::benchmark_data_smithi043_119832_object53978:head# 2024-03-30T13:50:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a25bd1b:::benchmark_data_smithi043_119832_object72994:head# 2024-03-30T13:50:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a2730ad:::benchmark_data_smithi043_119832_object18245:head# 2024-03-30T13:50:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a28cd62:::benchmark_data_smithi043_119832_object2575:head# 2024-03-30T13:50:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a2bdb70:::benchmark_data_smithi043_119832_object9614:head# 2024-03-30T13:50:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a2d14ea:::benchmark_data_smithi043_119832_object29228:head# 2024-03-30T13:50:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a2dc1c7:::benchmark_data_smithi043_119832_object6716:head# 2024-03-30T13:50:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a2e0188:::benchmark_data_smithi043_119832_object87554:head# 2024-03-30T13:50:33.577 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a2e1362:::benchmark_data_smithi043_119832_object37902:head# 2024-03-30T13:50:33.577 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a2f8255:::benchmark_data_smithi043_119832_object50874:head# 2024-03-30T13:50:33.577 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a30428b:::benchmark_data_smithi043_119832_object43806:head# 2024-03-30T13:50:33.577 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a304fec:::benchmark_data_smithi043_119832_object19040:head# 2024-03-30T13:50:33.605 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a309050:::benchmark_data_smithi043_119832_object77287:head# 2024-03-30T13:50:33.605 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a30ecd0:::benchmark_data_smithi043_119832_object78288:head# 2024-03-30T13:50:33.605 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a30f453:::benchmark_data_smithi043_119832_object37110:head# 2024-03-30T13:50:33.605 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a315984:::benchmark_data_smithi043_119832_object54087:head# 2024-03-30T13:50:33.605 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a32b2d7:::benchmark_data_smithi043_119832_object28767:head# 2024-03-30T13:50:33.605 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a330de5:::benchmark_data_smithi043_119832_object23152:head# 2024-03-30T13:50:33.605 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a341283:::benchmark_data_smithi043_119832_object79442:head# 2024-03-30T13:50:33.605 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a3481e4:::benchmark_data_smithi043_119832_object69828:head# 2024-03-30T13:50:33.605 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a34a4ad:::benchmark_data_smithi043_119832_object73332:head# 2024-03-30T13:50:33.605 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a35943d:::benchmark_data_smithi043_119832_object31671:head# 2024-03-30T13:50:33.606 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a3667ce:::benchmark_data_smithi043_119832_object67614:head# 2024-03-30T13:50:33.606 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a368271:::benchmark_data_smithi043_119832_object2088:head# 2024-03-30T13:50:33.606 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a37502c:::benchmark_data_smithi043_119832_object88459:head# 2024-03-30T13:50:33.606 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a378335:::benchmark_data_smithi043_119832_object12008:head# 2024-03-30T13:50:33.606 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a37bae1:::benchmark_data_smithi043_119832_object79595:head# 2024-03-30T13:50:33.606 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a3a1feb:::benchmark_data_smithi043_119832_object49054:head# 2024-03-30T13:50:33.606 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a3b51b0:::benchmark_data_smithi043_119832_object10420:head# 2024-03-30T13:50:33.606 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a3d6cba:::benchmark_data_smithi043_119832_object76512:head# 2024-03-30T13:50:33.606 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a3da9ea:::benchmark_data_smithi043_119832_object50694:head# 2024-03-30T13:50:33.606 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a3de33b:::benchmark_data_smithi043_119832_object43106:head# 2024-03-30T13:50:33.606 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a3e36f5:::benchmark_data_smithi043_119832_object78048:head# 2024-03-30T13:50:33.606 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a4129af:::benchmark_data_smithi043_119832_object62024:head# 2024-03-30T13:50:33.606 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a432965:::benchmark_data_smithi043_119832_object60051:head# 2024-03-30T13:50:33.607 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a437b7a:::benchmark_data_smithi043_119832_object40423:head# 2024-03-30T13:50:33.607 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a43d132:::benchmark_data_smithi043_119832_object11070:head# 2024-03-30T13:50:33.607 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a43d3a8:::benchmark_data_smithi043_119832_object71223:head# 2024-03-30T13:50:33.607 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a441b3a:::benchmark_data_smithi043_119832_object54462:head# 2024-03-30T13:50:33.607 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a44efbc:::benchmark_data_smithi043_119832_object54367:head# 2024-03-30T13:50:33.607 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a46ab46:::benchmark_data_smithi043_119832_object66742:head# 2024-03-30T13:50:33.607 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a471621:::benchmark_data_smithi043_119832_object1730:head# 2024-03-30T13:50:33.607 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a47546d:::benchmark_data_smithi043_119832_object42288:head# 2024-03-30T13:50:33.607 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a477aee:::benchmark_data_smithi043_119832_object64698:head# 2024-03-30T13:50:33.607 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a4998aa:::benchmark_data_smithi043_119832_object77613:head# 2024-03-30T13:50:33.607 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a49fe85:::benchmark_data_smithi043_119832_object36550:head# 2024-03-30T13:50:33.607 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a4a05e1:::benchmark_data_smithi043_119832_object50791:head# 2024-03-30T13:50:33.607 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a4a48c6:::benchmark_data_smithi043_119832_object4762:head# 2024-03-30T13:50:33.607 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a4b26d3:::benchmark_data_smithi043_119832_object40679:head# 2024-03-30T13:50:33.608 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a4c1162:::benchmark_data_smithi043_119832_object13726:head# 2024-03-30T13:50:33.608 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a4ccfd6:::benchmark_data_smithi043_119832_object49434:head# 2024-03-30T13:50:33.608 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a4d91f9:::benchmark_data_smithi043_119832_object44262:head# 2024-03-30T13:50:33.608 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a4e4258:::benchmark_data_smithi043_119832_object10347:head# 2024-03-30T13:50:33.608 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a4f5d2b:::benchmark_data_smithi043_119832_object43160:head# 2024-03-30T13:50:33.608 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a4fd0c2:::benchmark_data_smithi043_119832_object55103:head# 2024-03-30T13:50:33.608 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a515ac9:::benchmark_data_smithi043_119832_object38815:head# 2024-03-30T13:50:33.608 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a53106f:::benchmark_data_smithi043_119832_object17019:head# 2024-03-30T13:50:33.608 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a537cc4:::benchmark_data_smithi043_119832_object57625:head# 2024-03-30T13:50:33.608 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a545992:::benchmark_data_smithi043_119832_object3617:head# 2024-03-30T13:50:33.608 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a545a4e:::benchmark_data_smithi043_119832_object45881:head# 2024-03-30T13:50:33.608 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a552677:::benchmark_data_smithi043_119832_object52495:head# 2024-03-30T13:50:33.608 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a555c0c:::benchmark_data_smithi043_119832_object26339:head# 2024-03-30T13:50:33.608 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a56a135:::benchmark_data_smithi043_119832_object30224:head# 2024-03-30T13:50:33.608 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a572b14:::benchmark_data_smithi043_119832_object81719:head# 2024-03-30T13:50:33.609 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a5792f4:::benchmark_data_smithi043_119832_object52087:head# 2024-03-30T13:50:33.609 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a57fbb7:::benchmark_data_smithi043_119832_object77230:head# 2024-03-30T13:50:33.609 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a583160:::benchmark_data_smithi043_119832_object74649:head# 2024-03-30T13:50:33.609 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a586d87:::benchmark_data_smithi043_119832_object52987:head# 2024-03-30T13:50:33.609 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a59c63a:::benchmark_data_smithi043_119832_object38145:head# 2024-03-30T13:50:33.609 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a59e5d6:::benchmark_data_smithi043_119832_object64184:head# 2024-03-30T13:50:33.609 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a5b4164:::benchmark_data_smithi043_119832_object59169:head# 2024-03-30T13:50:33.609 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a5d7be0:::benchmark_data_smithi043_119832_object1788:head# 2024-03-30T13:50:33.609 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a5d823e:::benchmark_data_smithi043_119832_object41546:head# 2024-03-30T13:50:33.609 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a5d85de:::benchmark_data_smithi043_119832_object65261:head# 2024-03-30T13:50:33.609 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a5d9223:::benchmark_data_smithi043_119832_object36024:head# 2024-03-30T13:50:33.609 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a5f6be4:::benchmark_data_smithi043_119832_object12300:head# 2024-03-30T13:50:33.609 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a602179:::benchmark_data_smithi043_119832_object37349:head# 2024-03-30T13:50:33.609 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a617982:::benchmark_data_smithi043_119832_object72180:head# 2024-03-30T13:50:33.609 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a61c241:::benchmark_data_smithi043_119832_object46465:head# 2024-03-30T13:50:33.610 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a6265c3:::benchmark_data_smithi043_119832_object64307:head# 2024-03-30T13:50:33.610 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a646406:::benchmark_data_smithi043_119832_object14641:head# 2024-03-30T13:50:33.610 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a6471dd:::benchmark_data_smithi043_119832_object65792:head# 2024-03-30T13:50:33.610 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a650153:::benchmark_data_smithi043_119832_object11312:head# 2024-03-30T13:50:33.610 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a65147b:::benchmark_data_smithi043_119832_object86864:head# 2024-03-30T13:50:33.610 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a654218:::benchmark_data_smithi043_119832_object81247:head# 2024-03-30T13:50:33.610 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a654b64:::benchmark_data_smithi043_119832_object59115:head# 2024-03-30T13:50:33.610 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a658ad3:::benchmark_data_smithi043_119832_object71368:head# 2024-03-30T13:50:33.610 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a65c8a7:::benchmark_data_smithi043_119832_object6692:head# 2024-03-30T13:50:33.610 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a65e00d:::benchmark_data_smithi043_119832_object51080:head# 2024-03-30T13:50:33.610 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a65f77d:::benchmark_data_smithi043_119832_object88340:head# 2024-03-30T13:50:33.610 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a6620e9:::benchmark_data_smithi043_119832_object42906:head# 2024-03-30T13:50:33.610 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a67963a:::benchmark_data_smithi043_119832_object11937:head# 2024-03-30T13:50:33.610 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a67fb1f:::benchmark_data_smithi043_119832_object80633:head# 2024-03-30T13:50:33.611 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a68fe32:::benchmark_data_smithi043_119832_object41608:head# 2024-03-30T13:50:33.611 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a69260e:::benchmark_data_smithi043_119832_object42119:head# 2024-03-30T13:50:33.611 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a69d21e:::benchmark_data_smithi043_119832_object42897:head# 2024-03-30T13:50:33.611 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a6a17ee:::benchmark_data_smithi043_119832_object38585:head# 2024-03-30T13:50:33.611 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a6a6114:::benchmark_data_smithi043_119832_object5994:head# 2024-03-30T13:50:33.611 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a6bca4d:::benchmark_data_smithi043_119832_object70526:head# 2024-03-30T13:50:33.611 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a6bff74:::benchmark_data_smithi043_119832_object65423:head# 2024-03-30T13:50:33.611 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a6c3660:::benchmark_data_smithi043_119832_object20467:head# 2024-03-30T13:50:33.611 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a6c39c3:::benchmark_data_smithi043_119832_object19919:head# 2024-03-30T13:50:33.611 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a6c73cd:::benchmark_data_smithi043_119832_object52252:head# 2024-03-30T13:50:33.618 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:33.627 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a6d64b1:::benchmark_data_smithi043_119832_object1255:head# 2024-03-30T13:50:33.627 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a6d82df:::benchmark_data_smithi043_119832_object84293:head# 2024-03-30T13:50:33.627 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a6dfd0a:::benchmark_data_smithi043_119832_object29487:head# 2024-03-30T13:50:33.627 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a6eeec2:::benchmark_data_smithi043_119832_object64823:head# 2024-03-30T13:50:33.627 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a71b8aa:::benchmark_data_smithi043_119832_object41525:head# 2024-03-30T13:50:33.627 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a725e44:::benchmark_data_smithi043_119832_object38412:head# 2024-03-30T13:50:33.627 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a72bacc:::benchmark_data_smithi043_119832_object50813:head# 2024-03-30T13:50:33.627 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a74f065:::benchmark_data_smithi043_119832_object60172:head# 2024-03-30T13:50:33.628 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a7553b2:::benchmark_data_smithi043_119832_object50705:head# 2024-03-30T13:50:33.628 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a75ba95:::benchmark_data_smithi043_119832_object28077:head# 2024-03-30T13:50:33.628 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a7685fe:::benchmark_data_smithi043_119832_object44563:head# 2024-03-30T13:50:33.628 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a76e687:::benchmark_data_smithi043_119832_object51413:head# 2024-03-30T13:50:33.628 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a7792a2:::benchmark_data_smithi043_119832_object66467:head# 2024-03-30T13:50:33.628 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a793190:::benchmark_data_smithi043_119832_object38277:head# 2024-03-30T13:50:33.628 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a793b0e:::benchmark_data_smithi043_119832_object8435:head# 2024-03-30T13:50:33.628 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a795fd5:::benchmark_data_smithi043_119832_object50582:head# 2024-03-30T13:50:33.628 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a798a76:::benchmark_data_smithi043_119832_object81903:head# 2024-03-30T13:50:33.628 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a7ac307:::benchmark_data_smithi043_119832_object88070:head# 2024-03-30T13:50:33.628 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a7b921a:::benchmark_data_smithi043_119832_object2725:head# 2024-03-30T13:50:33.628 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a7ba168:::benchmark_data_smithi043_119832_object554:head# 2024-03-30T13:50:33.629 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a7c1b99:::benchmark_data_smithi043_119832_object16648:head# 2024-03-30T13:50:33.629 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a7c451b:::benchmark_data_smithi043_119832_object41217:head# 2024-03-30T13:50:33.629 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a7c8d13:::benchmark_data_smithi043_119832_object80392:head# 2024-03-30T13:50:33.629 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a7c94de:::benchmark_data_smithi043_119832_object23962:head# 2024-03-30T13:50:33.629 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a7d74c0:::benchmark_data_smithi043_119832_object20758:head# 2024-03-30T13:50:33.629 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a7d8295:::benchmark_data_smithi043_119832_object52497:head# 2024-03-30T13:50:33.629 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a814e43:::benchmark_data_smithi043_119832_object68562:head# 2024-03-30T13:50:33.629 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a837838:::benchmark_data_smithi043_119832_object40784:head# 2024-03-30T13:50:33.629 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a845524:::benchmark_data_smithi043_119832_object37730:head# 2024-03-30T13:50:33.629 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a85236f:::benchmark_data_smithi043_119832_object15183:head# 2024-03-30T13:50:33.629 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a858678:::benchmark_data_smithi043_119832_object28438:head# 2024-03-30T13:50:33.629 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a85b40b:::benchmark_data_smithi043_119832_object69576:head# 2024-03-30T13:50:33.629 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a862ad2:::benchmark_data_smithi043_119832_object85209:head# 2024-03-30T13:50:33.630 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a86835b:::benchmark_data_smithi043_119832_object2641:head# 2024-03-30T13:50:33.630 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a86dc52:::benchmark_data_smithi043_119832_object57949:head# 2024-03-30T13:50:33.630 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a874238:::benchmark_data_smithi043_119832_object87488:head# 2024-03-30T13:50:33.630 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a8954b0:::benchmark_data_smithi043_119832_object79871:head# 2024-03-30T13:50:33.630 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a89d8bd:::benchmark_data_smithi043_119832_object70303:head# 2024-03-30T13:50:33.630 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a89ee37:::benchmark_data_smithi043_119832_object21324:head# 2024-03-30T13:50:33.630 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a8a1a97:::benchmark_data_smithi043_119832_object63427:head# 2024-03-30T13:50:33.630 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a8a413c:::benchmark_data_smithi043_119832_object17388:head# 2024-03-30T13:50:33.637 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a8abce4:::benchmark_data_smithi043_119832_object4335:head# 2024-03-30T13:50:33.637 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a8b611e:::benchmark_data_smithi043_119832_object42571:head# 2024-03-30T13:50:33.637 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a8bca32:::benchmark_data_smithi043_119832_object32819:head# 2024-03-30T13:50:33.637 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a8c1737:::benchmark_data_smithi043_119832_object73932:head# 2024-03-30T13:50:33.637 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a8c5754:::benchmark_data_smithi043_119832_object14025:head# 2024-03-30T13:50:33.638 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a8c6941:::benchmark_data_smithi043_119832_object85821:head# 2024-03-30T13:50:33.638 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a8d3114:::benchmark_data_smithi043_119832_object26727:head# 2024-03-30T13:50:33.638 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a8d3347:::benchmark_data_smithi043_119832_object43315:head# 2024-03-30T13:50:33.638 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a8d42f6:::benchmark_data_smithi043_119832_object4581:head# 2024-03-30T13:50:33.638 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a8ea2fe:::benchmark_data_smithi043_119832_object9759:head# 2024-03-30T13:50:33.638 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a8f15fb:::benchmark_data_smithi043_119832_object85981:head# 2024-03-30T13:50:33.638 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a8f1f83:::benchmark_data_smithi043_119832_object26836:head# 2024-03-30T13:50:33.638 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a8f7407:::benchmark_data_smithi043_119832_object71759:head# 2024-03-30T13:50:33.638 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a90b3a9:::benchmark_data_smithi043_119832_object24410:head# 2024-03-30T13:50:33.639 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a914d16:::benchmark_data_smithi043_119832_object85527:head# 2024-03-30T13:50:33.639 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a921891:::benchmark_data_smithi043_119832_object77681:head# 2024-03-30T13:50:33.639 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a9223d5:::benchmark_data_smithi043_119832_object38680:head# 2024-03-30T13:50:33.639 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a926399:::benchmark_data_smithi043_119832_object71200:head# 2024-03-30T13:50:33.639 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a928d9a:::benchmark_data_smithi043_119832_object56960:head# 2024-03-30T13:50:33.639 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a92a519:::benchmark_data_smithi043_119832_object10414:head# 2024-03-30T13:50:33.639 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a9301b8:::benchmark_data_smithi043_119832_object7358:head# 2024-03-30T13:50:33.639 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a932e37:::benchmark_data_smithi043_119832_object71398:head# 2024-03-30T13:50:33.639 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a93973c:::benchmark_data_smithi043_119832_object21458:head# 2024-03-30T13:50:33.639 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a95b276:::benchmark_data_smithi043_119832_object85534:head# 2024-03-30T13:50:33.639 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a961628:::benchmark_data_smithi043_119832_object64703:head# 2024-03-30T13:50:33.639 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a963ccc:::benchmark_data_smithi043_119832_object62902:head# 2024-03-30T13:50:33.639 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a96900d:::benchmark_data_smithi043_119832_object74675:head# 2024-03-30T13:50:33.639 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a98f5ae:::benchmark_data_smithi043_119832_object37074:head# 2024-03-30T13:50:33.640 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a999f29:::benchmark_data_smithi043_119832_object81920:head# 2024-03-30T13:50:33.640 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a9a94fa:::benchmark_data_smithi043_119832_object68318:head# 2024-03-30T13:50:33.640 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a9ac920:::benchmark_data_smithi043_119832_object79966:head# 2024-03-30T13:50:33.640 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a9bf5ff:::benchmark_data_smithi043_119832_object4095:head# 2024-03-30T13:50:33.640 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a9ce3de:::benchmark_data_smithi043_119832_object15345:head# 2024-03-30T13:50:33.640 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a9d25b7:::benchmark_data_smithi043_119832_object2423:head# 2024-03-30T13:50:33.640 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a9d8a04:::benchmark_data_smithi043_119832_object22632:head# 2024-03-30T13:50:33.640 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a9e364d:::benchmark_data_smithi043_119832_object70468:head# 2024-03-30T13:50:33.640 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a9f4457:::benchmark_data_smithi043_119832_object5179:head# 2024-03-30T13:50:33.640 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7a9f7776:::benchmark_data_smithi043_119832_object66086:head# 2024-03-30T13:50:33.640 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aa0ca6d:::benchmark_data_smithi043_119832_object86567:head# 2024-03-30T13:50:33.640 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aa14fcf:::benchmark_data_smithi043_119832_object32261:head# 2024-03-30T13:50:33.640 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aa17976:::benchmark_data_smithi043_119832_object83519:head# 2024-03-30T13:50:33.643 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aa26ff3:::benchmark_data_smithi043_119832_object67683:head# 2024-03-30T13:50:33.643 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aa35262:::benchmark_data_smithi043_119832_object7750:head# 2024-03-30T13:50:33.643 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aa380cf:::benchmark_data_smithi043_119832_object75591:head# 2024-03-30T13:50:33.643 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aa4576d:::benchmark_data_smithi043_119832_object42084:head# 2024-03-30T13:50:33.644 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aa4cbbb:::benchmark_data_smithi043_119832_object31534:head# 2024-03-30T13:50:33.644 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aa589a8:::benchmark_data_smithi043_119832_object48153:head# 2024-03-30T13:50:33.644 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aa5996b:::benchmark_data_smithi043_119832_object48685:head# 2024-03-30T13:50:33.644 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aa6f9de:::benchmark_data_smithi043_119832_object72324:head# 2024-03-30T13:50:33.644 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aa7bbbf:::benchmark_data_smithi043_119832_object6897:head# 2024-03-30T13:50:33.644 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aa8bba1:::benchmark_data_smithi043_119832_object7132:head# 2024-03-30T13:50:33.644 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aa8d466:::benchmark_data_smithi043_119832_object35718:head# 2024-03-30T13:50:33.644 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aa93f48:::benchmark_data_smithi043_119832_object53909:head# 2024-03-30T13:50:33.644 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aaa120a:::benchmark_data_smithi043_119832_object44656:head# 2024-03-30T13:50:33.644 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aab9a1b:::benchmark_data_smithi043_119832_object80355:head# 2024-03-30T13:50:33.644 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aabfa75:::benchmark_data_smithi043_119832_object53162:head# 2024-03-30T13:50:33.645 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aad6409:::benchmark_data_smithi043_119832_object48296:head# 2024-03-30T13:50:33.645 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aae582d:::benchmark_data_smithi043_119832_object42750:head# 2024-03-30T13:50:33.645 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aaecb19:::benchmark_data_smithi043_119832_object63738:head# 2024-03-30T13:50:33.646 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aaf0436:::benchmark_data_smithi043_119832_object71091:head# 2024-03-30T13:50:33.646 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aaf129a:::benchmark_data_smithi043_119832_object54529:head# 2024-03-30T13:50:33.651 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aafcf4d:::benchmark_data_smithi043_119832_object629:head# 2024-03-30T13:50:33.651 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ab240e6:::benchmark_data_smithi043_119832_object33710:head# 2024-03-30T13:50:33.651 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ab3f7be:::benchmark_data_smithi043_119832_object44838:head# 2024-03-30T13:50:33.651 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ab4a447:::benchmark_data_smithi043_119832_object24785:head# 2024-03-30T13:50:33.654 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ab4c6cd:::benchmark_data_smithi043_119832_object26827:head# 2024-03-30T13:50:33.654 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ab53a25:::benchmark_data_smithi043_119832_object22812:head# 2024-03-30T13:50:33.654 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ab63756:::benchmark_data_smithi043_119832_object26480:head# 2024-03-30T13:50:33.654 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ab65e35:::benchmark_data_smithi043_119832_object34035:head# 2024-03-30T13:50:33.654 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ab6f44f:::benchmark_data_smithi043_119832_object78143:head# 2024-03-30T13:50:33.654 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ab74166:::benchmark_data_smithi043_119832_object50429:head# 2024-03-30T13:50:33.657 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ab85a19:::benchmark_data_smithi043_119832_object51315:head# 2024-03-30T13:50:33.657 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ab90f97:::benchmark_data_smithi043_119832_object68071:head# 2024-03-30T13:50:33.657 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7abb0006:::benchmark_data_smithi043_119832_object57629:head# 2024-03-30T13:50:33.657 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7abb287b:::benchmark_data_smithi043_119832_object79771:head# 2024-03-30T13:50:33.657 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7abb474a:::benchmark_data_smithi043_119832_object11850:head# 2024-03-30T13:50:33.657 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7abecd41:::benchmark_data_smithi043_119832_object22524:head# 2024-03-30T13:50:33.657 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7abeda0c:::benchmark_data_smithi043_119832_object79941:head# 2024-03-30T13:50:33.657 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ac02eb4:::benchmark_data_smithi043_119832_object78724:head# 2024-03-30T13:50:33.663 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ac30d47:::benchmark_data_smithi043_119832_object86757:head# 2024-03-30T13:50:33.663 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ac327ef:::benchmark_data_smithi043_119832_object82872:head# 2024-03-30T13:50:33.663 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ac42674:::benchmark_data_smithi043_119832_object54261:head# 2024-03-30T13:50:33.663 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ac43b1c:::benchmark_data_smithi043_119832_object86219:head# 2024-03-30T13:50:33.663 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ac5641d:::benchmark_data_smithi043_119832_object33805:head# 2024-03-30T13:50:33.663 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ac5783d:::benchmark_data_smithi043_119832_object60783:head# 2024-03-30T13:50:33.663 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ac5b62c:::benchmark_data_smithi043_119832_object18380:head# 2024-03-30T13:50:33.664 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ac70084:::benchmark_data_smithi043_119832_object75015:head# 2024-03-30T13:50:33.664 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ac8437c:::benchmark_data_smithi043_119832_object40572:head# 2024-03-30T13:50:33.664 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ac88cd3:::benchmark_data_smithi043_119832_object73618:head# 2024-03-30T13:50:33.664 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ac8f204:::benchmark_data_smithi043_119832_object9117:head# 2024-03-30T13:50:33.664 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ac91db4:::benchmark_data_smithi043_119832_object76078:head# 2024-03-30T13:50:33.664 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ac94ca7:::benchmark_data_smithi043_119832_object28275:head# 2024-03-30T13:50:33.664 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ac9621c:::benchmark_data_smithi043_119832_object54506:head# 2024-03-30T13:50:33.664 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ac9adab:::benchmark_data_smithi043_119832_object19762:head# 2024-03-30T13:50:33.664 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ac9c1c0:::benchmark_data_smithi043_119832_object67410:head# 2024-03-30T13:50:33.664 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aca8540:::benchmark_data_smithi043_119832_object45786:head# 2024-03-30T13:50:33.664 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7acbdce1:::benchmark_data_smithi043_119832_object78529:head# 2024-03-30T13:50:33.666 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:50:33.675 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7acbfb77:::benchmark_data_smithi043_119832_object31704:head# 2024-03-30T13:50:33.675 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7acc2215:::benchmark_data_smithi043_119832_object18349:head# 2024-03-30T13:50:33.675 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7acc94b9:::benchmark_data_smithi043_119832_object16796:head# 2024-03-30T13:50:33.675 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7acfaae5:::benchmark_data_smithi043_119832_object87575:head# 2024-03-30T13:50:33.675 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7acff29e:::benchmark_data_smithi043_119832_object77269:head# 2024-03-30T13:50:33.675 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ad04bad:::benchmark_data_smithi043_119832_object33739:head# 2024-03-30T13:50:33.675 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ad1701c:::benchmark_data_smithi043_119832_object53623:head# 2024-03-30T13:50:33.675 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ad341e4:::benchmark_data_smithi043_119832_object305:head# 2024-03-30T13:50:33.675 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ad4ca31:::benchmark_data_smithi043_119832_object67010:head# 2024-03-30T13:50:33.675 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ad4d592:::benchmark_data_smithi043_119832_object37113:head# 2024-03-30T13:50:33.675 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ad54caf:::benchmark_data_smithi043_119832_object29819:head# 2024-03-30T13:50:33.676 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ad571d9:::benchmark_data_smithi043_119832_object42709:head# 2024-03-30T13:50:33.676 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ad5f2f5:::benchmark_data_smithi043_119832_object13084:head# 2024-03-30T13:50:33.676 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ad69e96:::benchmark_data_smithi043_119832_object9282:head# 2024-03-30T13:50:33.676 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ad753fc:::benchmark_data_smithi043_119832_object35607:head# 2024-03-30T13:50:33.676 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ad84fd5:::benchmark_data_smithi043_119832_object57562:head# 2024-03-30T13:50:33.676 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ad8e6c6:::benchmark_data_smithi043_119832_object61118:head# 2024-03-30T13:50:33.676 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ad9fb33:::benchmark_data_smithi043_119832_object541:head# 2024-03-30T13:50:33.676 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ada4ebf:::benchmark_data_smithi043_119832_object79236:head# 2024-03-30T13:50:33.676 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7adc4667:::benchmark_data_smithi043_119832_object29525:head# 2024-03-30T13:50:33.676 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7adece91:::benchmark_data_smithi043_119832_object22414:head# 2024-03-30T13:50:33.676 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ae10229:::benchmark_data_smithi043_119832_object64815:head# 2024-03-30T13:50:33.676 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ae1e6a7:::benchmark_data_smithi043_119832_object63315:head# 2024-03-30T13:50:33.676 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ae28c3a:::benchmark_data_smithi043_119832_object21497:head# 2024-03-30T13:50:33.676 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ae2bc01:::benchmark_data_smithi043_119832_object69726:head# 2024-03-30T13:50:33.677 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ae49e6b:::benchmark_data_smithi043_119832_object62771:head# 2024-03-30T13:50:33.677 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ae5213e:::benchmark_data_smithi043_119832_object74732:head# 2024-03-30T13:50:33.677 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ae5d014:::benchmark_data_smithi043_119832_object59037:head# 2024-03-30T13:50:33.677 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ae60d80:::benchmark_data_smithi043_119832_object75176:head# 2024-03-30T13:50:33.680 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ae6dff3:::benchmark_data_smithi043_119832_object72199:head# 2024-03-30T13:50:33.680 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ae70593:::benchmark_data_smithi043_119832_object40073:head# 2024-03-30T13:50:33.680 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ae71659:::benchmark_data_smithi043_119832_object44571:head# 2024-03-30T13:50:33.680 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ae77314:::benchmark_data_smithi043_119832_object88505:head# 2024-03-30T13:50:33.680 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ae777bb:::benchmark_data_smithi043_119832_object45401:head# 2024-03-30T13:50:33.680 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ae85b36:::benchmark_data_smithi043_119832_object39855:head# 2024-03-30T13:50:33.680 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ae8e64c:::benchmark_data_smithi043_119832_object27979:head# 2024-03-30T13:50:33.681 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ae907a7:::benchmark_data_smithi043_119832_object51750:head# 2024-03-30T13:50:33.681 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ae97a6a:::benchmark_data_smithi043_119832_object43011:head# 2024-03-30T13:50:33.681 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ae996c5:::benchmark_data_smithi043_119832_object71236:head# 2024-03-30T13:50:33.681 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aea0652:::benchmark_data_smithi043_119832_object17995:head# 2024-03-30T13:50:33.681 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aeaa027:::benchmark_data_smithi043_119832_object78391:head# 2024-03-30T13:50:33.681 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aecd6a7:::benchmark_data_smithi043_119832_object19770:head# 2024-03-30T13:50:33.681 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aee418a:::benchmark_data_smithi043_119832_object29665:head# 2024-03-30T13:50:33.681 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aeec871:::benchmark_data_smithi043_119832_object47284:head# 2024-03-30T13:50:33.681 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aef4ef9:::benchmark_data_smithi043_119832_object13423:head# 2024-03-30T13:50:33.684 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7aefe1ac:::benchmark_data_smithi043_119832_object73755:head# 2024-03-30T13:50:33.684 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7af18aa6:::benchmark_data_smithi043_119832_object71747:head# 2024-03-30T13:50:33.684 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7af227c9:::benchmark_data_smithi043_119832_object31867:head# 2024-03-30T13:50:33.684 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7af2b07e:::benchmark_data_smithi043_119832_object88324:head# 2024-03-30T13:50:33.684 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7af2b9f3:::benchmark_data_smithi043_119832_object32573:head# 2024-03-30T13:50:33.684 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7af40a9f:::benchmark_data_smithi043_119832_object47110:head# 2024-03-30T13:50:33.684 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7af49a58:::benchmark_data_smithi043_119832_object49292:head# 2024-03-30T13:50:33.685 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7af4c1ec:::benchmark_data_smithi043_119832_object25526:head# 2024-03-30T13:50:33.685 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7af59b9b:::benchmark_data_smithi043_119832_object60507:head# 2024-03-30T13:50:33.685 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7af78d62:::benchmark_data_smithi043_119832_object36881:head# 2024-03-30T13:50:33.685 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7af82d0e:::benchmark_data_smithi043_119832_object38110:head# 2024-03-30T13:50:33.685 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7af82edd:::benchmark_data_smithi043_119832_object37457:head# 2024-03-30T13:50:33.686 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7af9e5fb:::benchmark_data_smithi043_119832_object24223:head# 2024-03-30T13:50:33.686 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7afae7c9:::benchmark_data_smithi043_119832_object82659:head# 2024-03-30T13:50:33.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:33.687 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:33.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:33.688 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7afb0b2c:::benchmark_data_smithi043_119832_object27734:head# 2024-03-30T13:50:33.688 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7afba001:::benchmark_data_smithi043_119832_object78038:head# 2024-03-30T13:50:33.688 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7afc206f:::benchmark_data_smithi043_119832_object64605:head# 2024-03-30T13:50:33.688 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7afc3df4:::benchmark_data_smithi043_119832_object41436:head# 2024-03-30T13:50:33.688 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7afe269e:::benchmark_data_smithi043_119832_object19409:head# 2024-03-30T13:50:33.689 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7affd612:::benchmark_data_smithi043_119832_object81380:head# 2024-03-30T13:50:33.690 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b00c90e:::benchmark_data_smithi043_119832_object71119:head# 2024-03-30T13:50:33.690 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b0561a6:::benchmark_data_smithi043_119832_object18119:head# 2024-03-30T13:50:33.690 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b05c80f:::benchmark_data_smithi043_119832_object34093:head# 2024-03-30T13:50:33.691 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b062123:::benchmark_data_smithi043_119832_object6033:head# 2024-03-30T13:50:33.692 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b0675e2:::benchmark_data_smithi043_119832_object32646:head# 2024-03-30T13:50:33.692 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b085642:::benchmark_data_smithi043_119832_object6967:head# 2024-03-30T13:50:33.692 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b09329a:::benchmark_data_smithi043_119832_object61837:head# 2024-03-30T13:50:33.692 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b0a162a:::benchmark_data_smithi043_119832_object40848:head# 2024-03-30T13:50:33.693 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b0b94fa:::benchmark_data_smithi043_119832_object74350:head# 2024-03-30T13:50:33.693 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b0bb62a:::benchmark_data_smithi043_119832_object61920:head# 2024-03-30T13:50:33.693 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b0be08e:::benchmark_data_smithi043_119832_object83154:head# 2024-03-30T13:50:33.693 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b0c392c:::benchmark_data_smithi043_119832_object16980:head# 2024-03-30T13:50:33.694 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b0c55bd:::benchmark_data_smithi043_119832_object44657:head# 2024-03-30T13:50:33.694 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b0d595b:::benchmark_data_smithi043_119832_object70102:head# 2024-03-30T13:50:33.695 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b0eab4f:::benchmark_data_smithi043_119832_object26736:head# 2024-03-30T13:50:33.696 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b0f8e78:::benchmark_data_smithi043_119832_object57750:head# 2024-03-30T13:50:33.696 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b11b60e:::benchmark_data_smithi043_119832_object27101:head# 2024-03-30T13:50:33.697 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b12290f:::benchmark_data_smithi043_119832_object38217:head# 2024-03-30T13:50:33.697 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b12f3c9:::benchmark_data_smithi043_119832_object4568:head# 2024-03-30T13:50:33.697 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b130cc9:::benchmark_data_smithi043_119832_object48104:head# 2024-03-30T13:50:33.697 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b14599d:::benchmark_data_smithi043_119832_object11025:head# 2024-03-30T13:50:33.697 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b14a523:::benchmark_data_smithi043_119832_object66425:head# 2024-03-30T13:50:33.698 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b15bbf2:::benchmark_data_smithi043_119832_object20442:head# 2024-03-30T13:50:33.698 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b15e1a0:::benchmark_data_smithi043_119832_object74109:head# 2024-03-30T13:50:33.702 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b15ebcf:::benchmark_data_smithi043_119832_object28129:head# 2024-03-30T13:50:33.702 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b169edc:::benchmark_data_smithi043_119832_object81933:head# 2024-03-30T13:50:33.702 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b173832:::benchmark_data_smithi043_119832_object59379:head# 2024-03-30T13:50:33.702 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b17ffbc:::benchmark_data_smithi043_119832_object78752:head# 2024-03-30T13:50:33.702 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b19c868:::benchmark_data_smithi043_119832_object8417:head# 2024-03-30T13:50:33.702 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b1a0525:::benchmark_data_smithi043_119832_object60887:head# 2024-03-30T13:50:33.702 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b1afe9c:::benchmark_data_smithi043_119832_object86958:head# 2024-03-30T13:50:33.702 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b1b0532:::benchmark_data_smithi043_119832_object79065:head# 2024-03-30T13:50:33.703 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b1c87fe:::benchmark_data_smithi043_119832_object8331:head# 2024-03-30T13:50:33.703 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b1d8f7e:::benchmark_data_smithi043_119832_object42536:head# 2024-03-30T13:50:33.703 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b1df204:::benchmark_data_smithi043_119832_object69291:head# 2024-03-30T13:50:33.703 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b1e397c:::benchmark_data_smithi043_119832_object69062:head# 2024-03-30T13:50:33.704 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b1e41d5:::benchmark_data_smithi043_119832_object65219:head# 2024-03-30T13:50:33.704 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b1f6854:::benchmark_data_smithi043_119832_object59797:head# 2024-03-30T13:50:33.704 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b201d84:::benchmark_data_smithi043_119832_object46181:head# 2024-03-30T13:50:33.704 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b20232a:::benchmark_data_smithi043_119832_object5372:head# 2024-03-30T13:50:33.708 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b226e89:::benchmark_data_smithi043_119832_object36156:head# 2024-03-30T13:50:33.708 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b22da26:::benchmark_data_smithi043_119832_object24655:head# 2024-03-30T13:50:33.708 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b232d3d:::benchmark_data_smithi043_119832_object57545:head# 2024-03-30T13:50:33.708 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b24d1aa:::benchmark_data_smithi043_119832_object29528:head# 2024-03-30T13:50:33.708 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b258cf3:::benchmark_data_smithi043_119832_object73065:head# 2024-03-30T13:50:33.708 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b25d908:::benchmark_data_smithi043_119832_object80840:head# 2024-03-30T13:50:33.708 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b25dae4:::benchmark_data_smithi043_119832_object8451:head# 2024-03-30T13:50:33.709 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b269002:::benchmark_data_smithi043_119832_object76147:head# 2024-03-30T13:50:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b276b96:::benchmark_data_smithi043_119832_object5985:head# 2024-03-30T13:50:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b284bd4:::benchmark_data_smithi043_119832_object48692:head# 2024-03-30T13:50:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b289b6d:::benchmark_data_smithi043_119832_object36568:head# 2024-03-30T13:50:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b28b6c7:::benchmark_data_smithi043_119832_object7666:head# 2024-03-30T13:50:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b29a8af:::benchmark_data_smithi043_119832_object28038:head# 2024-03-30T13:50:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b2a2a60:::benchmark_data_smithi043_119832_object11076:head# 2024-03-30T13:50:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b2c6422:::benchmark_data_smithi043_119832_object75353:head# 2024-03-30T13:50:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b2c9c21:::benchmark_data_smithi043_119832_object60000:head# 2024-03-30T13:50:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b2d2aaa:::benchmark_data_smithi043_119832_object65872:head# 2024-03-30T13:50:33.721 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b2d3586:::benchmark_data_smithi043_119832_object12121:head# 2024-03-30T13:50:33.721 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b2d4a4e:::benchmark_data_smithi043_119832_object36717:head# 2024-03-30T13:50:33.721 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b2de454:::benchmark_data_smithi043_119832_object38673:head# 2024-03-30T13:50:33.721 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b2e0d92:::benchmark_data_smithi043_119832_object86533:head# 2024-03-30T13:50:33.721 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b2f39d4:::benchmark_data_smithi043_119832_object73159:head# 2024-03-30T13:50:33.721 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b2f96f6:::benchmark_data_smithi043_119832_object47616:head# 2024-03-30T13:50:33.721 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b2ff51b:::benchmark_data_smithi043_119832_object65207:head# 2024-03-30T13:50:33.721 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b2ff99e:::benchmark_data_smithi043_119832_object12211:head# 2024-03-30T13:50:33.721 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b300861:::benchmark_data_smithi043_119832_object30697:head# 2024-03-30T13:50:33.721 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b301027:::benchmark_data_smithi043_119832_object13830:head# 2024-03-30T13:50:33.721 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b301982:::benchmark_data_smithi043_119832_object48064:head# 2024-03-30T13:50:33.721 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b303b22:::benchmark_data_smithi043_119832_object23776:head# 2024-03-30T13:50:33.721 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b308946:::benchmark_data_smithi043_119832_object49455:head# 2024-03-30T13:50:33.722 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b30edcb:::benchmark_data_smithi043_119832_object2101:head# 2024-03-30T13:50:33.722 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b31a7d2:::benchmark_data_smithi043_119832_object66411:head# 2024-03-30T13:50:33.722 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b31e586:::benchmark_data_smithi043_119832_object30002:head# 2024-03-30T13:50:33.722 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b31fe3b:::benchmark_data_smithi043_119832_object54515:head# 2024-03-30T13:50:33.722 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b3326b0:::benchmark_data_smithi043_119832_object55320:head# 2024-03-30T13:50:33.722 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b33bea4:::benchmark_data_smithi043_119832_object48326:head# 2024-03-30T13:50:33.723 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b34fec9:::benchmark_data_smithi043_119832_object8640:head# 2024-03-30T13:50:33.723 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b353a41:::benchmark_data_smithi043_119832_object53435:head# 2024-03-30T13:50:33.723 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b358b60:::benchmark_data_smithi043_119832_object79399:head# 2024-03-30T13:50:33.723 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b363cf8:::benchmark_data_smithi043_119832_object11356:head# 2024-03-30T13:50:33.723 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b3707fd:::benchmark_data_smithi043_119832_object81607:head# 2024-03-30T13:50:33.723 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b372238:::benchmark_data_smithi043_119832_object5336:head# 2024-03-30T13:50:33.725 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b387e08:::benchmark_data_smithi043_119832_object35639:head# 2024-03-30T13:50:33.726 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b39a30c:::benchmark_data_smithi043_119832_object27121:head# 2024-03-30T13:50:33.726 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b3ccd79:::benchmark_data_smithi043_119832_object65528:head# 2024-03-30T13:50:33.726 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b3cd976:::benchmark_data_smithi043_119832_object27188:head# 2024-03-30T13:50:33.726 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b3db904:::benchmark_data_smithi043_119832_object39747:head# 2024-03-30T13:50:33.726 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b3dd379:::benchmark_data_smithi043_119832_object71262:head# 2024-03-30T13:50:33.726 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b3e052f:::benchmark_data_smithi043_119832_object25541:head# 2024-03-30T13:50:33.726 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b3ec82f:::benchmark_data_smithi043_119832_object79511:head# 2024-03-30T13:50:33.726 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b3ede05:::benchmark_data_smithi043_119832_object45902:head# 2024-03-30T13:50:33.728 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b3efad9:::benchmark_data_smithi043_119832_object24239:head# 2024-03-30T13:50:33.729 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b3f99e7:::benchmark_data_smithi043_119832_object32230:head# 2024-03-30T13:50:33.729 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b402929:::benchmark_data_smithi043_119832_object30229:head# 2024-03-30T13:50:33.729 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b40455b:::benchmark_data_smithi043_119832_object51491:head# 2024-03-30T13:50:33.729 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b40fa27:::benchmark_data_smithi043_119832_object19615:head# 2024-03-30T13:50:33.729 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b410579:::benchmark_data_smithi043_119832_object38551:head# 2024-03-30T13:50:33.729 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b414f2c:::benchmark_data_smithi043_119832_object5169:head# 2024-03-30T13:50:33.729 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b425006:::benchmark_data_smithi043_119832_object74450:head# 2024-03-30T13:50:33.730 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b435402:::benchmark_data_smithi043_119832_object68868:head# 2024-03-30T13:50:33.730 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b43b1ff:::benchmark_data_smithi043_119832_object31244:head# 2024-03-30T13:50:33.730 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b441039:::benchmark_data_smithi043_119832_object15837:head# 2024-03-30T13:50:33.730 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b453c0e:::benchmark_data_smithi043_119832_object58225:head# 2024-03-30T13:50:33.733 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b454b80:::benchmark_data_smithi043_119832_object44831:head# 2024-03-30T13:50:33.733 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b45cc07:::benchmark_data_smithi043_119832_object4498:head# 2024-03-30T13:50:33.733 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b46712a:::benchmark_data_smithi043_119832_object57600:head# 2024-03-30T13:50:33.733 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b46c517:::benchmark_data_smithi043_119832_object50560:head# 2024-03-30T13:50:33.733 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b47a31f:::benchmark_data_smithi043_119832_object19020:head# 2024-03-30T13:50:33.733 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b4841d4:::benchmark_data_smithi043_119832_object9304:head# 2024-03-30T13:50:33.734 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b487e7c:::benchmark_data_smithi043_119832_object51403:head# 2024-03-30T13:50:33.734 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b49c6b5:::benchmark_data_smithi043_119832_object32111:head# 2024-03-30T13:50:33.734 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b4b2074:::benchmark_data_smithi043_119832_object87597:head# 2024-03-30T13:50:33.736 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b4d0d94:::benchmark_data_smithi043_119832_object23663:head# 2024-03-30T13:50:33.736 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b4d2b92:::benchmark_data_smithi043_119832_object72582:head# 2024-03-30T13:50:33.737 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b4d425b:::benchmark_data_smithi043_119832_object12636:head# 2024-03-30T13:50:33.737 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b4e28aa:::benchmark_data_smithi043_119832_object74500:head# 2024-03-30T13:50:33.737 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b4eb7a8:::benchmark_data_smithi043_119832_object69317:head# 2024-03-30T13:50:33.737 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b4f0bb8:::benchmark_data_smithi043_119832_object9365:head# 2024-03-30T13:50:33.737 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b4f2214:::benchmark_data_smithi043_119832_object56671:head# 2024-03-30T13:50:33.737 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b4f2f7b:::benchmark_data_smithi043_119832_object1724:head# 2024-03-30T13:50:33.737 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b4fae64:::benchmark_data_smithi043_119832_object84333:head# 2024-03-30T13:50:33.737 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b509868:::benchmark_data_smithi043_119832_object1906:head# 2024-03-30T13:50:33.737 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b50d7a5:::benchmark_data_smithi043_119832_object7709:head# 2024-03-30T13:50:33.737 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b51b653:::benchmark_data_smithi043_119832_object46300:head# 2024-03-30T13:50:33.737 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b54c74b:::benchmark_data_smithi043_119832_object23343:head# 2024-03-30T13:50:33.737 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b559a96:::benchmark_data_smithi043_119832_object88385:head# 2024-03-30T13:50:33.737 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b55f0f4:::benchmark_data_smithi043_119832_object79135:head# 2024-03-30T13:50:33.738 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b564ccc:::benchmark_data_smithi043_119832_object5257:head# 2024-03-30T13:50:33.738 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b57c27a:::benchmark_data_smithi043_119832_object28264:head# 2024-03-30T13:50:33.738 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b599fe2:::benchmark_data_smithi043_119832_object125:head# 2024-03-30T13:50:33.738 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b59a03e:::benchmark_data_smithi043_119832_object59875:head# 2024-03-30T13:50:33.738 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b5aaf5b:::benchmark_data_smithi043_119832_object77100:head# 2024-03-30T13:50:33.739 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b5d660c:::benchmark_data_smithi043_119832_object86528:head# 2024-03-30T13:50:33.739 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b5d9d08:::benchmark_data_smithi043_119832_object32737:head# 2024-03-30T13:50:33.739 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b5dd2d9:::benchmark_data_smithi043_119832_object65109:head# 2024-03-30T13:50:33.739 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b5e55d4:::benchmark_data_smithi043_119832_object71387:head# 2024-03-30T13:50:33.739 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b5f0377:::benchmark_data_smithi043_119832_object25798:head# 2024-03-30T13:50:33.740 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b5f0907:::benchmark_data_smithi043_119832_object26089:head# 2024-03-30T13:50:33.740 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b5f5d40:::benchmark_data_smithi043_119832_object30878:head# 2024-03-30T13:50:33.741 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b602ea6:::benchmark_data_smithi043_119832_object20525:head# 2024-03-30T13:50:33.741 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b607c44:::benchmark_data_smithi043_119832_object52613:head# 2024-03-30T13:50:33.742 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b616b3e:::benchmark_data_smithi043_119832_object40394:head# 2024-03-30T13:50:33.742 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b6251e9:::benchmark_data_smithi043_119832_object59513:head# 2024-03-30T13:50:33.743 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b644184:::benchmark_data_smithi043_119832_object87209:head# 2024-03-30T13:50:33.744 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b647bd1:::benchmark_data_smithi043_119832_object83741:head# 2024-03-30T13:50:33.744 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b648ef5:::benchmark_data_smithi043_119832_object542:head# 2024-03-30T13:50:33.744 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b654726:::benchmark_data_smithi043_119832_object64824:head# 2024-03-30T13:50:33.745 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b6557ff:::benchmark_data_smithi043_119832_object80293:head# 2024-03-30T13:50:33.745 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b665ce1:::benchmark_data_smithi043_119832_object48455:head# 2024-03-30T13:50:33.746 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b67b75b:::benchmark_data_smithi043_119832_object59263:head# 2024-03-30T13:50:33.747 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b6ac353:::benchmark_data_smithi043_119832_object79167:head# 2024-03-30T13:50:33.747 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b6b0d90:::benchmark_data_smithi043_119832_object88333:head# 2024-03-30T13:50:33.747 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b6bd0c9:::benchmark_data_smithi043_119832_object65557:head# 2024-03-30T13:50:33.747 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b6c26eb:::benchmark_data_smithi043_119832_object87974:head# 2024-03-30T13:50:33.748 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b6c5c9b:::benchmark_data_smithi043_119832_object18375:head# 2024-03-30T13:50:33.748 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b6f7dad:::benchmark_data_smithi043_119832_object56343:head# 2024-03-30T13:50:33.749 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b6fea02:::benchmark_data_smithi043_119832_object69292:head# 2024-03-30T13:50:33.753 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b70bed8:::benchmark_data_smithi043_119832_object20052:head# 2024-03-30T13:50:33.753 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b7391b6:::benchmark_data_smithi043_119832_object15045:head# 2024-03-30T13:50:33.753 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b73b109:::benchmark_data_smithi043_119832_object34898:head# 2024-03-30T13:50:33.753 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b754458:::benchmark_data_smithi043_119832_object80123:head# 2024-03-30T13:50:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b758633:::benchmark_data_smithi043_119832_object58993:head# 2024-03-30T13:50:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b77d3a1:::benchmark_data_smithi043_119832_object14257:head# 2024-03-30T13:50:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b78eb79:::benchmark_data_smithi043_119832_object81574:head# 2024-03-30T13:50:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b78f698:::benchmark_data_smithi043_119832_object33069:head# 2024-03-30T13:50:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b796df3:::benchmark_data_smithi043_119832_object16687:head# 2024-03-30T13:50:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b7b8495:::benchmark_data_smithi043_119832_object33559:head# 2024-03-30T13:50:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b7c41ec:::benchmark_data_smithi043_119832_object39022:head# 2024-03-30T13:50:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b7d12b5:::benchmark_data_smithi043_119832_object3488:head# 2024-03-30T13:50:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b7da18e:::benchmark_data_smithi043_119832_object76137:head# 2024-03-30T13:50:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b7e9ca8:::benchmark_data_smithi043_119832_object56344:head# 2024-03-30T13:50:33.755 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b7ec3d6:::benchmark_data_smithi043_119832_object63729:head# 2024-03-30T13:50:33.755 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b7f31ab:::benchmark_data_smithi043_119832_object87804:head# 2024-03-30T13:50:33.755 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b809d71:::benchmark_data_smithi043_119832_object56839:head# 2024-03-30T13:50:33.755 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b80a220:::benchmark_data_smithi043_119832_object68487:head# 2024-03-30T13:50:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b80faf8:::benchmark_data_smithi043_119832_object43233:head# 2024-03-30T13:50:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b8133c1:::benchmark_data_smithi043_119832_object67611:head# 2024-03-30T13:50:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b81fa45:::benchmark_data_smithi043_119832_object23483:head# 2024-03-30T13:50:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b83e302:::benchmark_data_smithi043_119832_object21493:head# 2024-03-30T13:50:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b84b61f:::benchmark_data_smithi043_119832_object52276:head# 2024-03-30T13:50:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b84e78d:::benchmark_data_smithi043_119832_object32906:head# 2024-03-30T13:50:33.757 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b857dfd:::benchmark_data_smithi043_119832_object25229:head# 2024-03-30T13:50:33.757 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b860ffe:::benchmark_data_smithi043_119832_object3024:head# 2024-03-30T13:50:33.757 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b86153e:::benchmark_data_smithi043_119832_object17947:head# 2024-03-30T13:50:33.759 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b8674a5:::benchmark_data_smithi043_119832_object3785:head# 2024-03-30T13:50:33.759 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b87249f:::benchmark_data_smithi043_119832_object83366:head# 2024-03-30T13:50:33.759 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b878cfe:::benchmark_data_smithi043_119832_object27769:head# 2024-03-30T13:50:33.761 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b89d9d6:::benchmark_data_smithi043_119832_object50862:head# 2024-03-30T13:50:33.763 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b8a2295:::benchmark_data_smithi043_119832_object10922:head# 2024-03-30T13:50:33.763 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b8c14f0:::benchmark_data_smithi043_119832_object44066:head# 2024-03-30T13:50:33.763 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b8cd3a5:::benchmark_data_smithi043_119832_object71601:head# 2024-03-30T13:50:33.767 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b8ed36b:::benchmark_data_smithi043_119832_object45768:head# 2024-03-30T13:50:33.767 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b8ede72:::benchmark_data_smithi043_119832_object40211:head# 2024-03-30T13:50:33.767 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b8fa433:::benchmark_data_smithi043_119832_object84321:head# 2024-03-30T13:50:33.767 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b90e0c7:::benchmark_data_smithi043_119832_object14020:head# 2024-03-30T13:50:33.767 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b911995:::benchmark_data_smithi043_119832_object81321:head# 2024-03-30T13:50:33.770 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b931100:::benchmark_data_smithi043_119832_object39923:head# 2024-03-30T13:50:33.770 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b9368e8:::benchmark_data_smithi043_119832_object62967:head# 2024-03-30T13:50:33.770 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b956e9e:::benchmark_data_smithi043_119832_object14881:head# 2024-03-30T13:50:33.770 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b97c1e4:::benchmark_data_smithi043_119832_object73780:head# 2024-03-30T13:50:33.770 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b986b06:::benchmark_data_smithi043_119832_object75145:head# 2024-03-30T13:50:33.770 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b9a9cee:::benchmark_data_smithi043_119832_object64964:head# 2024-03-30T13:50:33.770 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b9b74fa:::benchmark_data_smithi043_119832_object43316:head# 2024-03-30T13:50:33.770 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7b9c7305:::benchmark_data_smithi043_119832_object31183:head# 2024-03-30T13:50:33.770 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ba074cd:::benchmark_data_smithi043_119832_object6999:head# 2024-03-30T13:50:33.770 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ba157dc:::benchmark_data_smithi043_119832_object48211:head# 2024-03-30T13:50:33.771 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ba1aaeb:::benchmark_data_smithi043_119832_object65211:head# 2024-03-30T13:50:33.772 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ba3f5be:::benchmark_data_smithi043_119832_object43339:head# 2024-03-30T13:50:33.772 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ba3f947:::benchmark_data_smithi043_119832_object81009:head# 2024-03-30T13:50:33.772 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ba441d3:::benchmark_data_smithi043_119832_object57448:head# 2024-03-30T13:50:33.772 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ba55715:::benchmark_data_smithi043_119832_object44159:head# 2024-03-30T13:50:33.772 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ba5c113:::benchmark_data_smithi043_119832_object76974:head# 2024-03-30T13:50:33.773 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ba5cc1d:::benchmark_data_smithi043_119832_object32726:head# 2024-03-30T13:50:33.774 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ba6590c:::benchmark_data_smithi043_119832_object77795:head# 2024-03-30T13:50:33.774 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ba6f844:::benchmark_data_smithi043_119832_object21146:head# 2024-03-30T13:50:33.775 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ba7ea01:::benchmark_data_smithi043_119832_object61591:head# 2024-03-30T13:50:33.776 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ba84fdd:::benchmark_data_smithi043_119832_object82491:head# 2024-03-30T13:50:33.776 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bab3595:::benchmark_data_smithi043_119832_object33543:head# 2024-03-30T13:50:33.776 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bad6e3d:::benchmark_data_smithi043_119832_object1899:head# 2024-03-30T13:50:33.777 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bae644b:::benchmark_data_smithi043_119832_object25061:head# 2024-03-30T13:50:33.778 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7baeae25:::benchmark_data_smithi043_119832_object75273:head# 2024-03-30T13:50:33.778 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7baef375:::benchmark_data_smithi043_119832_object53126:head# 2024-03-30T13:50:33.779 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7baf3e19:::benchmark_data_smithi043_119832_object57405:head# 2024-03-30T13:50:33.779 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7baf43cc:::benchmark_data_smithi043_119832_object35402:head# 2024-03-30T13:50:33.780 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7baff155:::benchmark_data_smithi043_119832_object52799:head# 2024-03-30T13:50:33.781 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb09186:::benchmark_data_smithi043_119832_object8834:head# 2024-03-30T13:50:33.781 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb0bcae:::benchmark_data_smithi043_119832_object7077:head# 2024-03-30T13:50:33.784 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb11967:::benchmark_data_smithi043_119832_object16622:head# 2024-03-30T13:50:33.784 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb1384b:::benchmark_data_smithi043_119832_object63977:head# 2024-03-30T13:50:33.784 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb15880:::benchmark_data_smithi043_119832_object26084:head# 2024-03-30T13:50:33.784 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb2b789:::benchmark_data_smithi043_119832_object26712:head# 2024-03-30T13:50:33.784 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb3744f:::benchmark_data_smithi043_119832_object18890:head# 2024-03-30T13:50:33.784 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb44525:::benchmark_data_smithi043_119832_object70760:head# 2024-03-30T13:50:33.785 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb453b5:::benchmark_data_smithi043_119832_object32173:head# 2024-03-30T13:50:33.785 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb4f916:::benchmark_data_smithi043_119832_object43014:head# 2024-03-30T13:50:33.785 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb53c90:::benchmark_data_smithi043_119832_object77976:head# 2024-03-30T13:50:33.786 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb57035:::benchmark_data_smithi043_119832_object74185:head# 2024-03-30T13:50:33.788 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:33.789 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb5b81b:::benchmark_data_smithi043_119832_object32653:head# 2024-03-30T13:50:33.789 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb679cf:::benchmark_data_smithi043_119832_object64776:head# 2024-03-30T13:50:33.789 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb6bec8:::benchmark_data_smithi043_119832_object60347:head# 2024-03-30T13:50:33.790 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb71756:::benchmark_data_smithi043_119832_object13889:head# 2024-03-30T13:50:33.790 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb74519:::benchmark_data_smithi043_119832_object3026:head# 2024-03-30T13:50:33.790 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb82340:::benchmark_data_smithi043_119832_object15491:head# 2024-03-30T13:50:33.790 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb85b9d:::benchmark_data_smithi043_119832_object65426:head# 2024-03-30T13:50:33.790 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb9b786:::benchmark_data_smithi043_119832_object47008:head# 2024-03-30T13:50:33.790 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb9c65b:::benchmark_data_smithi043_119832_object21287:head# 2024-03-30T13:50:33.790 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bb9f410:::benchmark_data_smithi043_119832_object44550:head# 2024-03-30T13:50:33.791 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bbac90b:::benchmark_data_smithi043_119832_object46216:head# 2024-03-30T13:50:33.791 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bbc6b91:::benchmark_data_smithi043_119832_object86838:head# 2024-03-30T13:50:33.791 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bbca2e6:::benchmark_data_smithi043_119832_object58999:head# 2024-03-30T13:50:33.791 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bbd8c43:::benchmark_data_smithi043_119832_object67312:head# 2024-03-30T13:50:33.791 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bbdec37:::benchmark_data_smithi043_119832_object36029:head# 2024-03-30T13:50:33.792 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bbecdd3:::benchmark_data_smithi043_119832_object29253:head# 2024-03-30T13:50:33.792 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bbeeece:::benchmark_data_smithi043_119832_object81801:head# 2024-03-30T13:50:33.792 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bbfc25e:::benchmark_data_smithi043_119832_object6378:head# 2024-03-30T13:50:33.795 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bc0a777:::benchmark_data_smithi043_119832_object29343:head# 2024-03-30T13:50:33.795 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bc0fc86:::benchmark_data_smithi043_119832_object7340:head# 2024-03-30T13:50:33.795 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bc1bf48:::benchmark_data_smithi043_119832_object78501:head# 2024-03-30T13:50:33.795 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bc1c96a:::benchmark_data_smithi043_119832_object55917:head# 2024-03-30T13:50:33.795 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bc40772:::benchmark_data_smithi043_119832_object78843:head# 2024-03-30T13:50:33.795 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bc4632b:::benchmark_data_smithi043_119832_object49530:head# 2024-03-30T13:50:33.795 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bc4b6bc:::benchmark_data_smithi043_119832_object46221:head# 2024-03-30T13:50:33.795 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bc6aa23:::benchmark_data_smithi043_119832_object4001:head# 2024-03-30T13:50:33.796 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bc7906e:::benchmark_data_smithi043_119832_object88298:head# 2024-03-30T13:50:33.796 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bc7bef6:::benchmark_data_smithi043_119832_object85581:head# 2024-03-30T13:50:33.796 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bc7d033:::benchmark_data_smithi043_119832_object73871:head# 2024-03-30T13:50:33.796 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bc7e058:::benchmark_data_smithi043_119832_object34109:head# 2024-03-30T13:50:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bc931e0:::benchmark_data_smithi043_119832_object72063:head# 2024-03-30T13:50:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bcb3cfa:::benchmark_data_smithi043_119832_object67179:head# 2024-03-30T13:50:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bcc4681:::benchmark_data_smithi043_119832_object6818:head# 2024-03-30T13:50:33.798 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bce4742:::benchmark_data_smithi043_119832_object73276:head# 2024-03-30T13:50:33.798 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bce647b:::benchmark_data_smithi043_119832_object74260:head# 2024-03-30T13:50:33.798 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bcf347d:::benchmark_data_smithi043_119832_object65964:head# 2024-03-30T13:50:33.800 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bd079fb:::benchmark_data_smithi043_119832_object82937:head# 2024-03-30T13:50:33.800 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bd09031:::benchmark_data_smithi043_119832_object15540:head# 2024-03-30T13:50:33.800 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bd26537:::benchmark_data_smithi043_119832_object83944:head# 2024-03-30T13:50:33.800 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bd2aea9:::benchmark_data_smithi043_119832_object7091:head# 2024-03-30T13:50:33.800 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bd31bc7:::benchmark_data_smithi043_119832_object40133:head# 2024-03-30T13:50:33.800 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bd43b75:::benchmark_data_smithi043_119832_object79803:head# 2024-03-30T13:50:33.800 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bd52ef0:::benchmark_data_smithi043_119832_object7201:head# 2024-03-30T13:50:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bd65644:::benchmark_data_smithi043_119832_object62138:head# 2024-03-30T13:50:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bd7c149:::benchmark_data_smithi043_119832_object35133:head# 2024-03-30T13:50:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bd88bf2:::benchmark_data_smithi043_119832_object33527:head# 2024-03-30T13:50:33.803 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bd8c318:::benchmark_data_smithi043_119832_object11503:head# 2024-03-30T13:50:33.803 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bd9034a:::benchmark_data_smithi043_119832_object41570:head# 2024-03-30T13:50:33.803 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bd97a10:::benchmark_data_smithi043_119832_object3518:head# 2024-03-30T13:50:33.803 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bdce855:::benchmark_data_smithi043_119832_object50138:head# 2024-03-30T13:50:33.803 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bdd6400:::benchmark_data_smithi043_119832_object66813:head# 2024-03-30T13:50:33.804 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bde2fcf:::benchmark_data_smithi043_119832_object66930:head# 2024-03-30T13:50:33.804 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7be046c5:::benchmark_data_smithi043_119832_object10811:head# 2024-03-30T13:50:33.804 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7be0a620:::benchmark_data_smithi043_119832_object8537:head# 2024-03-30T13:50:33.805 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7be1886e:::benchmark_data_smithi043_119832_object3075:head# 2024-03-30T13:50:33.805 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7be25009:::benchmark_data_smithi043_119832_object87851:head# 2024-03-30T13:50:33.805 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7be450d9:::benchmark_data_smithi043_119832_object73011:head# 2024-03-30T13:50:33.806 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7be65a22:::benchmark_data_smithi043_119832_object61151:head# 2024-03-30T13:50:33.806 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7be68b4c:::benchmark_data_smithi043_119832_object68266:head# 2024-03-30T13:50:33.806 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7be73f05:::benchmark_data_smithi043_119832_object81679:head# 2024-03-30T13:50:33.806 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7be76383:::benchmark_data_smithi043_119832_object82795:head# 2024-03-30T13:50:33.807 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7be77acc:::benchmark_data_smithi043_119832_object58167:head# 2024-03-30T13:50:33.808 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7be835df:::benchmark_data_smithi043_119832_object67293:head# 2024-03-30T13:50:33.808 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7be9c983:::benchmark_data_smithi043_119832_object18619:head# 2024-03-30T13:50:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7be9f921:::benchmark_data_smithi043_119832_object60021:head# 2024-03-30T13:50:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7beb5a05:::benchmark_data_smithi043_119832_object37899:head# 2024-03-30T13:50:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bebaaa3:::benchmark_data_smithi043_119832_object9381:head# 2024-03-30T13:50:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bebf676:::benchmark_data_smithi043_119832_object24252:head# 2024-03-30T13:50:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bed1ef1:::benchmark_data_smithi043_119832_object35946:head# 2024-03-30T13:50:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7beec492:::benchmark_data_smithi043_119832_object86976:head# 2024-03-30T13:50:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7beee53a:::benchmark_data_smithi043_119832_object76151:head# 2024-03-30T13:50:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7beff993:::benchmark_data_smithi043_119832_object78390:head# 2024-03-30T13:50:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bf0021c:::benchmark_data_smithi043_119832_object47797:head# 2024-03-30T13:50:33.813 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bf0b1eb:::benchmark_data_smithi043_119832_object54743:head# 2024-03-30T13:50:33.813 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bf0f6d0:::benchmark_data_smithi043_119832_object8017:head# 2024-03-30T13:50:33.814 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bf3624c:::benchmark_data_smithi043_119832_object37910:head# 2024-03-30T13:50:33.814 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bf40d53:::benchmark_data_smithi043_119832_object21618:head# 2024-03-30T13:50:33.815 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bf4423a:::benchmark_data_smithi043_119832_object17301:head# 2024-03-30T13:50:33.815 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bf45bb1:::benchmark_data_smithi043_119832_object32366:head# 2024-03-30T13:50:33.815 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bf4c051:::benchmark_data_smithi043_119832_object66741:head# 2024-03-30T13:50:33.815 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bf65eff:::benchmark_data_smithi043_119832_object13172:head# 2024-03-30T13:50:33.816 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bf69350:::benchmark_data_smithi043_119832_object56754:head# 2024-03-30T13:50:33.817 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bf803a6:::benchmark_data_smithi043_119832_object52589:head# 2024-03-30T13:50:33.817 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bf98ec0:::benchmark_data_smithi043_119832_object47162:head# 2024-03-30T13:50:33.817 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bf9aba8:::benchmark_data_smithi043_119832_object82:head# 2024-03-30T13:50:33.818 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bf9b82c:::benchmark_data_smithi043_119832_object34712:head# 2024-03-30T13:50:33.818 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bfa1177:::benchmark_data_smithi043_119832_object36784:head# 2024-03-30T13:50:33.818 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bfa5856:::benchmark_data_smithi043_119832_object47033:head# 2024-03-30T13:50:33.818 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bfbd943:::benchmark_data_smithi043_119832_object30695:head# 2024-03-30T13:50:33.818 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bfbdf39:::benchmark_data_smithi043_119832_object52366:head# 2024-03-30T13:50:33.818 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bfc88dc:::benchmark_data_smithi043_119832_object12446:head# 2024-03-30T13:50:33.819 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bfe46fd:::benchmark_data_smithi043_119832_object60243:head# 2024-03-30T13:50:33.819 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bfed1e2:::benchmark_data_smithi043_119832_object32352:head# 2024-03-30T13:50:33.819 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7bff15dc:::benchmark_data_smithi043_119832_object18217:head# 2024-03-30T13:50:33.819 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c000bbf:::benchmark_data_smithi043_119832_object85178:head# 2024-03-30T13:50:33.819 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c00751d:::benchmark_data_smithi043_119832_object4084:head# 2024-03-30T13:50:33.820 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c018c71:::benchmark_data_smithi043_119832_object994:head# 2024-03-30T13:50:33.820 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c019eb1:::benchmark_data_smithi043_119832_object22579:head# 2024-03-30T13:50:33.820 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c028cb0:::benchmark_data_smithi043_119832_object60713:head# 2024-03-30T13:50:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c047a5c:::benchmark_data_smithi043_119832_object24145:head# 2024-03-30T13:50:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c04aa8f:::benchmark_data_smithi043_119832_object36255:head# 2024-03-30T13:50:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c051250:::benchmark_data_smithi043_119832_object78302:head# 2024-03-30T13:50:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c059a6c:::benchmark_data_smithi043_119832_object15456:head# 2024-03-30T13:50:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c05ac25:::benchmark_data_smithi043_119832_object60745:head# 2024-03-30T13:50:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c05ff15:::benchmark_data_smithi043_119832_object33670:head# 2024-03-30T13:50:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c078015:::benchmark_data_smithi043_119832_object84425:head# 2024-03-30T13:50:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c07a7aa:::benchmark_data_smithi043_119832_object72024:head# 2024-03-30T13:50:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c07d7a9:::benchmark_data_smithi043_119832_object78607:head# 2024-03-30T13:50:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c08484d:::benchmark_data_smithi043_119832_object64719:head# 2024-03-30T13:50:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c0857ba:::benchmark_data_smithi043_119832_object64072:head# 2024-03-30T13:50:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c099413:::benchmark_data_smithi043_119832_object15232:head# 2024-03-30T13:50:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c0a97e4:::benchmark_data_smithi043_119832_object19283:head# 2024-03-30T13:50:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c0a98b5:::benchmark_data_smithi043_119832_object60681:head# 2024-03-30T13:50:33.828 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c0ab6e9:::benchmark_data_smithi043_119832_object87530:head# 2024-03-30T13:50:33.828 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c0b2149:::benchmark_data_smithi043_119832_object42454:head# 2024-03-30T13:50:33.828 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c0c896c:::benchmark_data_smithi043_119832_object16193:head# 2024-03-30T13:50:33.829 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c0d70d4:::benchmark_data_smithi043_119832_object12994:head# 2024-03-30T13:50:33.830 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c0e5e96:::benchmark_data_smithi043_119832_object70979:head# 2024-03-30T13:50:33.830 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c0e766c:::benchmark_data_smithi043_119832_object3010:head# 2024-03-30T13:50:33.830 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c0ebfb6:::benchmark_data_smithi043_119832_object59825:head# 2024-03-30T13:50:33.830 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c0ef27b:::benchmark_data_smithi043_119832_object29643:head# 2024-03-30T13:50:33.830 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c0f5b6c:::benchmark_data_smithi043_119832_object76939:head# 2024-03-30T13:50:33.831 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c120874:::benchmark_data_smithi043_119832_object26768:head# 2024-03-30T13:50:33.831 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c12b96d:::benchmark_data_smithi043_119832_object77124:head# 2024-03-30T13:50:33.832 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c12cb8e:::benchmark_data_smithi043_119832_object53644:head# 2024-03-30T13:50:33.833 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c13cff1:::benchmark_data_smithi043_119832_object52918:head# 2024-03-30T13:50:33.833 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c15416a:::benchmark_data_smithi043_119832_object73337:head# 2024-03-30T13:50:33.833 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c15afc7:::benchmark_data_smithi043_119832_object32672:head# 2024-03-30T13:50:33.836 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c162c2e:::benchmark_data_smithi043_119832_object59874:head# 2024-03-30T13:50:33.837 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c169ac1:::benchmark_data_smithi043_119832_object23046:head# 2024-03-30T13:50:33.837 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c1756f3:::benchmark_data_smithi043_119832_object37341:head# 2024-03-30T13:50:33.837 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c183beb:::benchmark_data_smithi043_119832_object82134:head# 2024-03-30T13:50:33.837 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c18bc2e:::benchmark_data_smithi043_119832_object33338:head# 2024-03-30T13:50:33.837 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c19193b:::benchmark_data_smithi043_119832_object60523:head# 2024-03-30T13:50:33.837 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c1ac6e9:::benchmark_data_smithi043_119832_object67679:head# 2024-03-30T13:50:33.837 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c1b1863:::benchmark_data_smithi043_119832_object73480:head# 2024-03-30T13:50:33.837 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c1c0d41:::benchmark_data_smithi043_119832_object19273:head# 2024-03-30T13:50:33.837 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c1c4595:::benchmark_data_smithi043_119832_object14068:head# 2024-03-30T13:50:33.838 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c1c94ff:::benchmark_data_smithi043_119832_object79922:head# 2024-03-30T13:50:33.838 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c1e0f45:::benchmark_data_smithi043_119832_object79459:head# 2024-03-30T13:50:33.838 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:33.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:33.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:33.840 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c1e29e7:::benchmark_data_smithi043_119832_object3725:head# 2024-03-30T13:50:33.840 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c1e9775:::benchmark_data_smithi043_119832_object1536:head# 2024-03-30T13:50:33.840 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c1ee621:::benchmark_data_smithi043_119832_object69316:head# 2024-03-30T13:50:33.840 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c1f4e4c:::benchmark_data_smithi043_119832_object48845:head# 2024-03-30T13:50:33.841 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c1fc7e3:::benchmark_data_smithi043_119832_object26717:head# 2024-03-30T13:50:33.841 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c1ffb6f:::benchmark_data_smithi043_119832_object48799:head# 2024-03-30T13:50:33.842 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c20cef5:::benchmark_data_smithi043_119832_object4570:head# 2024-03-30T13:50:33.842 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c211af3:::benchmark_data_smithi043_119832_object40109:head# 2024-03-30T13:50:33.842 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c232cb4:::benchmark_data_smithi043_119832_object45065:head# 2024-03-30T13:50:33.842 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c23617b:::benchmark_data_smithi043_119832_object45136:head# 2024-03-30T13:50:33.842 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c238134:::benchmark_data_smithi043_119832_object39044:head# 2024-03-30T13:50:33.842 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c23b389:::benchmark_data_smithi043_119832_object56686:head# 2024-03-30T13:50:33.842 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c24ea3c:::benchmark_data_smithi043_119832_object35280:head# 2024-03-30T13:50:33.843 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c2507c4:::benchmark_data_smithi043_119832_object75111:head# 2024-03-30T13:50:33.844 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c259355:::benchmark_data_smithi043_119832_object63176:head# 2024-03-30T13:50:33.844 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c25aecb:::benchmark_data_smithi043_119832_object69458:head# 2024-03-30T13:50:33.844 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c25d44e:::benchmark_data_smithi043_119832_object26057:head# 2024-03-30T13:50:33.846 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c26dc28:::benchmark_data_smithi043_119832_object73066:head# 2024-03-30T13:50:33.846 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c274346:::benchmark_data_smithi043_119832_object71454:head# 2024-03-30T13:50:33.846 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c2772fd:::benchmark_data_smithi043_119832_object20356:head# 2024-03-30T13:50:33.846 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c27ee08:::benchmark_data_smithi043_119832_object39215:head# 2024-03-30T13:50:33.846 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c28beb5:::benchmark_data_smithi043_119832_object62489:head# 2024-03-30T13:50:33.846 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c29b143:::benchmark_data_smithi043_119832_object33088:head# 2024-03-30T13:50:33.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c2a59e6:::benchmark_data_smithi043_119832_object3361:head# 2024-03-30T13:50:33.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c2a96e6:::benchmark_data_smithi043_119832_object7265:head# 2024-03-30T13:50:33.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c2b3437:::benchmark_data_smithi043_119832_object23931:head# 2024-03-30T13:50:33.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c2c1dc4:::benchmark_data_smithi043_119832_object67250:head# 2024-03-30T13:50:33.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c2c8538:::benchmark_data_smithi043_119832_object15796:head# 2024-03-30T13:50:33.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c2caee8:::benchmark_data_smithi043_119832_object52203:head# 2024-03-30T13:50:33.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c2cc39e:::benchmark_data_smithi043_119832_object51282:head# 2024-03-30T13:50:33.849 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c2cd96a:::benchmark_data_smithi043_119832_object68971:head# 2024-03-30T13:50:33.851 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c3163fc:::benchmark_data_smithi043_119832_object2371:head# 2024-03-30T13:50:33.851 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c31fa7b:::benchmark_data_smithi043_119832_object61338:head# 2024-03-30T13:50:33.851 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c335cf8:::benchmark_data_smithi043_119832_object47935:head# 2024-03-30T13:50:33.851 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c338cdf:::benchmark_data_smithi043_119832_object40548:head# 2024-03-30T13:50:33.851 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c3462ee:::benchmark_data_smithi043_119832_object88087:head# 2024-03-30T13:50:33.852 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c34e18f:::benchmark_data_smithi043_119832_object51362:head# 2024-03-30T13:50:33.852 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c3510b3:::benchmark_data_smithi043_119832_object42394:head# 2024-03-30T13:50:33.852 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c35a663:::benchmark_data_smithi043_119832_object12097:head# 2024-03-30T13:50:33.852 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c35dc53:::benchmark_data_smithi043_119832_object56977:head# 2024-03-30T13:50:33.854 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c36608d:::benchmark_data_smithi043_119832_object4721:head# 2024-03-30T13:50:33.854 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c36eeaa:::benchmark_data_smithi043_119832_object51575:head# 2024-03-30T13:50:33.854 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c371d2b:::benchmark_data_smithi043_119832_object72500:head# 2024-03-30T13:50:33.854 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c38810c:::benchmark_data_smithi043_119832_object48710:head# 2024-03-30T13:50:33.855 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c3cbbcf:::benchmark_data_smithi043_119832_object27965:head# 2024-03-30T13:50:33.856 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c3d9d62:::benchmark_data_smithi043_119832_object60121:head# 2024-03-30T13:50:33.856 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c3eac8b:::benchmark_data_smithi043_119832_object72952:head# 2024-03-30T13:50:33.856 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c407612:::benchmark_data_smithi043_119832_object68377:head# 2024-03-30T13:50:33.856 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c4125c8:::benchmark_data_smithi043_119832_object65766:head# 2024-03-30T13:50:33.857 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c41fd2c:::benchmark_data_smithi043_119832_object54698:head# 2024-03-30T13:50:33.857 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c427d1e:::benchmark_data_smithi043_119832_object2991:head# 2024-03-30T13:50:33.857 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c42d2b6:::benchmark_data_smithi043_119832_object84592:head# 2024-03-30T13:50:33.859 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c43607a:::benchmark_data_smithi043_119832_object11921:head# 2024-03-30T13:50:33.859 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c43a5d0:::benchmark_data_smithi043_119832_object39983:head# 2024-03-30T13:50:33.859 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c4486eb:::benchmark_data_smithi043_119832_object5750:head# 2024-03-30T13:50:33.859 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c452f24:::benchmark_data_smithi043_119832_object16445:head# 2024-03-30T13:50:33.859 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c45ae3c:::benchmark_data_smithi043_119832_object40008:head# 2024-03-30T13:50:33.864 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c478c2c:::benchmark_data_smithi043_119832_object81080:head# 2024-03-30T13:50:33.864 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c47999f:::benchmark_data_smithi043_119832_object52205:head# 2024-03-30T13:50:33.864 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c4934c9:::benchmark_data_smithi043_119832_object49896:head# 2024-03-30T13:50:33.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c4bf6d1:::benchmark_data_smithi043_119832_object78507:head# 2024-03-30T13:50:33.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c4c1f40:::benchmark_data_smithi043_119832_object5690:head# 2024-03-30T13:50:33.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c4d30b2:::benchmark_data_smithi043_119832_object87426:head# 2024-03-30T13:50:33.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c4fb7d6:::benchmark_data_smithi043_119832_object50914:head# 2024-03-30T13:50:33.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c507166:::benchmark_data_smithi043_119832_object37817:head# 2024-03-30T13:50:33.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c50af26:::benchmark_data_smithi043_119832_object13445:head# 2024-03-30T13:50:33.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c5404a2:::benchmark_data_smithi043_119832_object120:head# 2024-03-30T13:50:33.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c54065b:::benchmark_data_smithi043_119832_object54830:head# 2024-03-30T13:50:33.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c541bcd:::benchmark_data_smithi043_119832_object26184:head# 2024-03-30T13:50:33.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c544d36:::benchmark_data_smithi043_119832_object79632:head# 2024-03-30T13:50:33.865 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c55805a:::benchmark_data_smithi043_119832_object55836:head# 2024-03-30T13:50:33.866 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c563cd6:::benchmark_data_smithi043_119832_object77909:head# 2024-03-30T13:50:33.867 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c574232:::benchmark_data_smithi043_119832_object50288:head# 2024-03-30T13:50:33.867 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c579784:::benchmark_data_smithi043_119832_object25408:head# 2024-03-30T13:50:33.867 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c59d399:::benchmark_data_smithi043_119832_object60253:head# 2024-03-30T13:50:33.867 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c5a88a3:::benchmark_data_smithi043_119832_object9014:head# 2024-03-30T13:50:33.867 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c5b5d04:::benchmark_data_smithi043_119832_object58791:head# 2024-03-30T13:50:33.867 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c5bfd7f:::benchmark_data_smithi043_119832_object36360:head# 2024-03-30T13:50:33.868 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c5c04f8:::benchmark_data_smithi043_119832_object63773:head# 2024-03-30T13:50:33.868 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c5c89ab:::benchmark_data_smithi043_119832_object15200:head# 2024-03-30T13:50:33.868 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c5d5b63:::benchmark_data_smithi043_119832_object13698:head# 2024-03-30T13:50:33.869 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c5e394c:::benchmark_data_smithi043_119832_object481:head# 2024-03-30T13:50:33.869 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c5e73a3:::benchmark_data_smithi043_119832_object3154:head# 2024-03-30T13:50:33.869 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c60a9e5:::benchmark_data_smithi043_119832_object49443:head# 2024-03-30T13:50:33.872 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c611e09:::benchmark_data_smithi043_119832_object58996:head# 2024-03-30T13:50:33.872 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c61992e:::benchmark_data_smithi043_119832_object69520:head# 2024-03-30T13:50:33.872 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c6437b6:::benchmark_data_smithi043_119832_object3732:head# 2024-03-30T13:50:33.872 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c64ef11:::benchmark_data_smithi043_119832_object61180:head# 2024-03-30T13:50:33.872 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c6802bc:::benchmark_data_smithi043_119832_object56427:head# 2024-03-30T13:50:33.872 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c688dc7:::benchmark_data_smithi043_119832_object1559:head# 2024-03-30T13:50:33.872 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c699de7:::benchmark_data_smithi043_119832_object64264:head# 2024-03-30T13:50:33.873 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c69b350:::benchmark_data_smithi043_119832_object39186:head# 2024-03-30T13:50:33.873 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c6b62a9:::benchmark_data_smithi043_119832_object13434:head# 2024-03-30T13:50:33.874 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c6c74c3:::benchmark_data_smithi043_119832_object29687:head# 2024-03-30T13:50:33.874 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c6cd02e:::benchmark_data_smithi043_119832_object18129:head# 2024-03-30T13:50:33.874 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c6e85a0:::benchmark_data_smithi043_119832_object17536:head# 2024-03-30T13:50:33.875 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c6ed269:::benchmark_data_smithi043_119832_object12289:head# 2024-03-30T13:50:33.875 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c6f0233:::benchmark_data_smithi043_119832_object59422:head# 2024-03-30T13:50:33.875 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c6f7455:::benchmark_data_smithi043_119832_object86663:head# 2024-03-30T13:50:33.877 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c7045fa:::benchmark_data_smithi043_119832_object39793:head# 2024-03-30T13:50:33.877 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c707583:::benchmark_data_smithi043_119832_object9269:head# 2024-03-30T13:50:33.877 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c712d7c:::benchmark_data_smithi043_119832_object48933:head# 2024-03-30T13:50:33.877 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c715f80:::benchmark_data_smithi043_119832_object40465:head# 2024-03-30T13:50:33.877 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c719ce1:::benchmark_data_smithi043_119832_object77228:head# 2024-03-30T13:50:33.879 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c71eb03:::benchmark_data_smithi043_119832_object21648:head# 2024-03-30T13:50:33.879 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c725b0f:::benchmark_data_smithi043_119832_object38586:head# 2024-03-30T13:50:33.879 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c726c52:::benchmark_data_smithi043_119832_object54485:head# 2024-03-30T13:50:33.880 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c727ce5:::benchmark_data_smithi043_119832_object53575:head# 2024-03-30T13:50:33.880 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c728584:::benchmark_data_smithi043_119832_object76158:head# 2024-03-30T13:50:33.880 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c7327d5:::benchmark_data_smithi043_119832_object77981:head# 2024-03-30T13:50:33.880 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c742c68:::benchmark_data_smithi043_119832_object62632:head# 2024-03-30T13:50:33.881 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c75d788:::benchmark_data_smithi043_119832_object30633:head# 2024-03-30T13:50:33.881 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c7601fa:::benchmark_data_smithi043_119832_object31038:head# 2024-03-30T13:50:33.881 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c764fae:::benchmark_data_smithi043_119832_object54413:head# 2024-03-30T13:50:33.882 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c773007:::benchmark_data_smithi043_119832_object23386:head# 2024-03-30T13:50:33.882 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c77a417:::benchmark_data_smithi043_119832_object53641:head# 2024-03-30T13:50:33.882 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c79a5fd:::benchmark_data_smithi043_119832_object14776:head# 2024-03-30T13:50:33.882 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c7a6646:::benchmark_data_smithi043_119832_object50481:head# 2024-03-30T13:50:33.882 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c7aec2e:::benchmark_data_smithi043_119832_object71056:head# 2024-03-30T13:50:33.882 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c7b6f76:::benchmark_data_smithi043_119832_object64285:head# 2024-03-30T13:50:33.882 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c7d20ad:::benchmark_data_smithi043_119832_object29309:head# 2024-03-30T13:50:33.883 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c7d4e17:::benchmark_data_smithi043_119832_object68559:head# 2024-03-30T13:50:33.883 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c7d5037:::benchmark_data_smithi043_119832_object46573:head# 2024-03-30T13:50:33.883 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c809d5a:::benchmark_data_smithi043_119832_object40953:head# 2024-03-30T13:50:33.883 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c813321:::benchmark_data_smithi043_119832_object24534:head# 2024-03-30T13:50:33.884 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c827e2b:::benchmark_data_smithi043_119832_object46505:head# 2024-03-30T13:50:33.884 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c830a28:::benchmark_data_smithi043_119832_object17577:head# 2024-03-30T13:50:33.884 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c8350b0:::benchmark_data_smithi043_119832_object62212:head# 2024-03-30T13:50:33.884 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c8378f1:::benchmark_data_smithi043_119832_object80902:head# 2024-03-30T13:50:33.889 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c83d78f:::benchmark_data_smithi043_119832_object52751:head# 2024-03-30T13:50:33.889 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c84c4f9:::benchmark_data_smithi043_119832_object44999:head# 2024-03-30T13:50:33.889 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c852903:::benchmark_data_smithi043_119832_object3714:head# 2024-03-30T13:50:33.889 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c867ae4:::benchmark_data_smithi043_119832_object51306:head# 2024-03-30T13:50:33.889 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c8680e7:::benchmark_data_smithi043_119832_object61303:head# 2024-03-30T13:50:33.889 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c87293c:::benchmark_data_smithi043_119832_object58024:head# 2024-03-30T13:50:33.889 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c89a441:::benchmark_data_smithi043_119832_object62877:head# 2024-03-30T13:50:33.890 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c89ae7b:::benchmark_data_smithi043_119832_object49619:head# 2024-03-30T13:50:33.890 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c8bb90d:::benchmark_data_smithi043_119832_object82350:head# 2024-03-30T13:50:33.890 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c8d4d65:::benchmark_data_smithi043_119832_object63957:head# 2024-03-30T13:50:33.890 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c8dedd5:::benchmark_data_smithi043_119832_object9468:head# 2024-03-30T13:50:33.890 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c906b7a:::benchmark_data_smithi043_119832_object34709:head# 2024-03-30T13:50:33.890 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c9168f6:::benchmark_data_smithi043_119832_object41099:head# 2024-03-30T13:50:33.890 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c922c0e:::benchmark_data_smithi043_119832_object63971:head# 2024-03-30T13:50:33.891 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c93096b:::benchmark_data_smithi043_119832_object39706:head# 2024-03-30T13:50:33.892 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c9583e2:::benchmark_data_smithi043_119832_object2215:head# 2024-03-30T13:50:33.892 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c95c636:::benchmark_data_smithi043_119832_object15087:head# 2024-03-30T13:50:33.892 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c95f85b:::benchmark_data_smithi043_119832_object38806:head# 2024-03-30T13:50:33.892 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c96bedd:::benchmark_data_smithi043_119832_object709:head# 2024-03-30T13:50:33.892 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c97c573:::benchmark_data_smithi043_119832_object51726:head# 2024-03-30T13:50:33.892 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c98e87e:::benchmark_data_smithi043_119832_object54029:head# 2024-03-30T13:50:33.893 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c98f68a:::benchmark_data_smithi043_119832_object26682:head# 2024-03-30T13:50:33.893 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c99d5ed:::benchmark_data_smithi043_119832_object68080:head# 2024-03-30T13:50:33.893 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c9a5840:::benchmark_data_smithi043_119832_object78861:head# 2024-03-30T13:50:33.893 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c9a9bdf:::benchmark_data_smithi043_119832_object66797:head# 2024-03-30T13:50:33.894 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c9b65a4:::benchmark_data_smithi043_119832_object43256:head# 2024-03-30T13:50:33.896 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c9d72b8:::benchmark_data_smithi043_119832_object80729:head# 2024-03-30T13:50:33.896 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c9ead6d:::benchmark_data_smithi043_119832_object18735:head# 2024-03-30T13:50:33.896 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c9edd70:::benchmark_data_smithi043_119832_object59080:head# 2024-03-30T13:50:33.896 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7c9fb839:::benchmark_data_smithi043_119832_object82515:head# 2024-03-30T13:50:33.896 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ca00092:::benchmark_data_smithi043_119832_object46207:head# 2024-03-30T13:50:33.896 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ca23d89:::benchmark_data_smithi043_119832_object75543:head# 2024-03-30T13:50:33.896 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ca32c3d:::benchmark_data_smithi043_119832_object86063:head# 2024-03-30T13:50:33.898 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ca3ec24:::benchmark_data_smithi043_119832_object15760:head# 2024-03-30T13:50:33.898 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ca3fa8c:::benchmark_data_smithi043_119832_object79945:head# 2024-03-30T13:50:33.898 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ca4afd9:::benchmark_data_smithi043_119832_object5167:head# 2024-03-30T13:50:33.898 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ca62115:::benchmark_data_smithi043_119832_object53267:head# 2024-03-30T13:50:33.898 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ca670c9:::benchmark_data_smithi043_119832_object72810:head# 2024-03-30T13:50:33.898 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ca69898:::benchmark_data_smithi043_119832_object44339:head# 2024-03-30T13:50:33.899 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ca751dc:::benchmark_data_smithi043_119832_object87876:head# 2024-03-30T13:50:33.900 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ca7dd7b:::benchmark_data_smithi043_119832_object47265:head# 2024-03-30T13:50:33.900 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ca8f68b:::benchmark_data_smithi043_119832_object84633:head# 2024-03-30T13:50:33.900 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ca98245:::benchmark_data_smithi043_119832_object46197:head# 2024-03-30T13:50:33.900 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ca9a3a7:::benchmark_data_smithi043_119832_object61391:head# 2024-03-30T13:50:33.900 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ca9abba:::benchmark_data_smithi043_119832_object43709:head# 2024-03-30T13:50:33.901 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ca9ae36:::benchmark_data_smithi043_119832_object44861:head# 2024-03-30T13:50:33.901 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ca9edc8:::benchmark_data_smithi043_119832_object13016:head# 2024-03-30T13:50:33.902 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cab3a17:::benchmark_data_smithi043_119832_object76560:head# 2024-03-30T13:50:33.904 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cab7cc2:::benchmark_data_smithi043_119832_object12942:head# 2024-03-30T13:50:33.904 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cabb3ca:::benchmark_data_smithi043_119832_object19359:head# 2024-03-30T13:50:33.904 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cabe792:::benchmark_data_smithi043_119832_object43604:head# 2024-03-30T13:50:33.905 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cad117c:::benchmark_data_smithi043_119832_object64644:head# 2024-03-30T13:50:33.905 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cadf752:::benchmark_data_smithi043_119832_object50504:head# 2024-03-30T13:50:33.906 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cb059e0:::benchmark_data_smithi043_119832_object55766:head# 2024-03-30T13:50:33.906 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cb0fbef:::benchmark_data_smithi043_119832_object12104:head# 2024-03-30T13:50:33.906 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cb15d97:::benchmark_data_smithi043_119832_object30973:head# 2024-03-30T13:50:33.906 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cb1f235:::benchmark_data_smithi043_119832_object86754:head# 2024-03-30T13:50:33.907 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cb2efda:::benchmark_data_smithi043_119832_object84313:head# 2024-03-30T13:50:33.907 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cb380a5:::benchmark_data_smithi043_119832_object21212:head# 2024-03-30T13:50:33.907 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cb3cf37:::benchmark_data_smithi043_119832_object44458:head# 2024-03-30T13:50:33.907 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cb459eb:::benchmark_data_smithi043_119832_object81051:head# 2024-03-30T13:50:33.908 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cb4ab99:::benchmark_data_smithi043_119832_object28774:head# 2024-03-30T13:50:33.908 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cb4d99d:::benchmark_data_smithi043_119832_object13431:head# 2024-03-30T13:50:33.908 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cb5654e:::benchmark_data_smithi043_119832_object36256:head# 2024-03-30T13:50:33.908 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cb71ada:::benchmark_data_smithi043_119832_object16306:head# 2024-03-30T13:50:33.908 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cb75d5b:::benchmark_data_smithi043_119832_object84027:head# 2024-03-30T13:50:33.909 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cb886d8:::benchmark_data_smithi043_119832_object31307:head# 2024-03-30T13:50:33.909 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cb88828:::benchmark_data_smithi043_119832_object10881:head# 2024-03-30T13:50:33.909 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cb8a83a:::benchmark_data_smithi043_119832_object46205:head# 2024-03-30T13:50:33.909 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cb935fb:::benchmark_data_smithi043_119832_object33779:head# 2024-03-30T13:50:33.910 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cba35b6:::benchmark_data_smithi043_119832_object4181:head# 2024-03-30T13:50:33.914 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cba855e:::benchmark_data_smithi043_119832_object12431:head# 2024-03-30T13:50:33.914 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cba9073:::benchmark_data_smithi043_119832_object31890:head# 2024-03-30T13:50:33.914 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cbacee0:::benchmark_data_smithi043_119832_object82340:head# 2024-03-30T13:50:33.914 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cbcbf4c:::benchmark_data_smithi043_119832_object12943:head# 2024-03-30T13:50:33.914 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cbd80a6:::benchmark_data_smithi043_119832_object63884:head# 2024-03-30T13:50:33.914 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cbe383e:::benchmark_data_smithi043_119832_object69846:head# 2024-03-30T13:50:33.915 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cbe46c7:::benchmark_data_smithi043_119832_object1228:head# 2024-03-30T13:50:33.915 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cbe7335:::benchmark_data_smithi043_119832_object9151:head# 2024-03-30T13:50:33.915 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cbef52b:::benchmark_data_smithi043_119832_object60087:head# 2024-03-30T13:50:33.915 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cbfd431:::benchmark_data_smithi043_119832_object48861:head# 2024-03-30T13:50:33.915 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cc195cc:::benchmark_data_smithi043_119832_object15400:head# 2024-03-30T13:50:33.918 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cc1c2df:::benchmark_data_smithi043_119832_object68476:head# 2024-03-30T13:50:33.918 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cc24130:::benchmark_data_smithi043_119832_object25093:head# 2024-03-30T13:50:33.919 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cc300ff:::benchmark_data_smithi043_119832_object18068:head# 2024-03-30T13:50:33.919 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cc3c3d0:::benchmark_data_smithi043_119832_object77827:head# 2024-03-30T13:50:33.919 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cc3fe7e:::benchmark_data_smithi043_119832_object29372:head# 2024-03-30T13:50:33.919 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cc49b66:::benchmark_data_smithi043_119832_object29681:head# 2024-03-30T13:50:33.919 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cc571ad:::benchmark_data_smithi043_119832_object20966:head# 2024-03-30T13:50:33.919 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cc5e623:::benchmark_data_smithi043_119832_object63196:head# 2024-03-30T13:50:33.919 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cc60f02:::benchmark_data_smithi043_119832_object7871:head# 2024-03-30T13:50:33.919 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cc64a2a:::benchmark_data_smithi043_119832_object248:head# 2024-03-30T13:50:33.919 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cc78cd2:::benchmark_data_smithi043_119832_object76150:head# 2024-03-30T13:50:33.919 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cc84426:::benchmark_data_smithi043_119832_object17314:head# 2024-03-30T13:50:33.919 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cc90965:::benchmark_data_smithi043_119832_object81788:head# 2024-03-30T13:50:33.920 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cc94a57:::benchmark_data_smithi043_119832_object23993:head# 2024-03-30T13:50:33.921 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cc9acd3:::benchmark_data_smithi043_119832_object41426:head# 2024-03-30T13:50:33.921 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cc9bcc2:::benchmark_data_smithi043_119832_object7286:head# 2024-03-30T13:50:33.921 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ccaadda:::benchmark_data_smithi043_119832_object70314:head# 2024-03-30T13:50:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ccc2e1a:::benchmark_data_smithi043_119832_object83412:head# 2024-03-30T13:50:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cccf63b:::benchmark_data_smithi043_119832_object1656:head# 2024-03-30T13:50:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cce2dfb:::benchmark_data_smithi043_119832_object2028:head# 2024-03-30T13:50:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cd05f37:::benchmark_data_smithi043_119832_object67116:head# 2024-03-30T13:50:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cd0ca71:::benchmark_data_smithi043_119832_object35682:head# 2024-03-30T13:50:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cd12e33:::benchmark_data_smithi043_119832_object8467:head# 2024-03-30T13:50:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cd1487a:::benchmark_data_smithi043_119832_object41327:head# 2024-03-30T13:50:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cd20b42:::benchmark_data_smithi043_119832_object2220:head# 2024-03-30T13:50:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cd264de:::benchmark_data_smithi043_119832_object71210:head# 2024-03-30T13:50:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cd26db8:::benchmark_data_smithi043_119832_object56866:head# 2024-03-30T13:50:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cd2f0c1:::benchmark_data_smithi043_119832_object36026:head# 2024-03-30T13:50:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cd301f0:::benchmark_data_smithi043_119832_object1878:head# 2024-03-30T13:50:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cd3cde5:::benchmark_data_smithi043_119832_object37278:head# 2024-03-30T13:50:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cd4fb83:::benchmark_data_smithi043_119832_object64275:head# 2024-03-30T13:50:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cd51735:::benchmark_data_smithi043_119832_object78990:head# 2024-03-30T13:50:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cd53322:::benchmark_data_smithi043_119832_object79127:head# 2024-03-30T13:50:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cd591fb:::benchmark_data_smithi043_119832_object66134:head# 2024-03-30T13:50:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cd6f3fc:::benchmark_data_smithi043_119832_object24823:head# 2024-03-30T13:50:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cd89d5e:::benchmark_data_smithi043_119832_object43357:head# 2024-03-30T13:50:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cd9364f:::benchmark_data_smithi043_119832_object39611:head# 2024-03-30T13:50:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cd974c6:::benchmark_data_smithi043_119832_object5871:head# 2024-03-30T13:50:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cd9d80f:::benchmark_data_smithi043_119832_object77233:head# 2024-03-30T13:50:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cda98a4:::benchmark_data_smithi043_119832_object82875:head# 2024-03-30T13:50:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cdaf3cd:::benchmark_data_smithi043_119832_object21240:head# 2024-03-30T13:50:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cdb9462:::benchmark_data_smithi043_119832_object75155:head# 2024-03-30T13:50:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cdbcbc2:::benchmark_data_smithi043_119832_object86336:head# 2024-03-30T13:50:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cdc4651:::benchmark_data_smithi043_119832_object10846:head# 2024-03-30T13:50:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cdc54a5:::benchmark_data_smithi043_119832_object60716:head# 2024-03-30T13:50:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cdc8bba:::benchmark_data_smithi043_119832_object17012:head# 2024-03-30T13:50:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cdcb445:::benchmark_data_smithi043_119832_object26188:head# 2024-03-30T13:50:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cdcd45c:::benchmark_data_smithi043_119832_object26544:head# 2024-03-30T13:50:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cddf80f:::benchmark_data_smithi043_119832_object27355:head# 2024-03-30T13:50:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ce01f1e:::benchmark_data_smithi043_119832_object66091:head# 2024-03-30T13:50:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ce02f3d:::benchmark_data_smithi043_119832_object20146:head# 2024-03-30T13:50:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ce05ba3:::benchmark_data_smithi043_119832_object18147:head# 2024-03-30T13:50:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ce12aaa:::benchmark_data_smithi043_119832_object21873:head# 2024-03-30T13:50:33.941 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:33.970 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ce19b69:::benchmark_data_smithi043_119832_object30481:head# 2024-03-30T13:50:33.970 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ce1e935:::benchmark_data_smithi043_119832_object79066:head# 2024-03-30T13:50:33.970 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ce21444:::benchmark_data_smithi043_119832_object80639:head# 2024-03-30T13:50:33.970 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ce28483:::benchmark_data_smithi043_119832_object10417:head# 2024-03-30T13:50:33.971 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ce3fcc2:::benchmark_data_smithi043_119832_object26782:head# 2024-03-30T13:50:33.971 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ce444f0:::benchmark_data_smithi043_119832_object77973:head# 2024-03-30T13:50:33.971 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ce589b1:::benchmark_data_smithi043_119832_object55393:head# 2024-03-30T13:50:33.971 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ce62561:::benchmark_data_smithi043_119832_object86276:head# 2024-03-30T13:50:33.971 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ce75942:::benchmark_data_smithi043_119832_object9925:head# 2024-03-30T13:50:33.971 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ce8b449:::benchmark_data_smithi043_119832_object66026:head# 2024-03-30T13:50:33.971 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ce95289:::benchmark_data_smithi043_119832_object44436:head# 2024-03-30T13:50:33.971 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ce96354:::benchmark_data_smithi043_119832_object51841:head# 2024-03-30T13:50:33.971 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cea19af:::benchmark_data_smithi043_119832_object69876:head# 2024-03-30T13:50:33.971 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ceb0bf2:::benchmark_data_smithi043_119832_object19875:head# 2024-03-30T13:50:33.971 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ceba4a6:::benchmark_data_smithi043_119832_object82297:head# 2024-03-30T13:50:33.971 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cec30ed:::benchmark_data_smithi043_119832_object54735:head# 2024-03-30T13:50:33.971 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ced3986:::benchmark_data_smithi043_119832_object86963:head# 2024-03-30T13:50:33.972 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ced4fb4:::benchmark_data_smithi043_119832_object43522:head# 2024-03-30T13:50:33.972 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cedc2c7:::benchmark_data_smithi043_119832_object56533:head# 2024-03-30T13:50:33.972 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cede28c:::benchmark_data_smithi043_119832_object81940:head# 2024-03-30T13:50:33.972 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cee3ed6:::benchmark_data_smithi043_119832_object35241:head# 2024-03-30T13:50:33.972 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cee53df:::benchmark_data_smithi043_119832_object68110:head# 2024-03-30T13:50:33.972 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cee6188:::benchmark_data_smithi043_119832_object65471:head# 2024-03-30T13:50:33.972 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ceea12c:::benchmark_data_smithi043_119832_object71838:head# 2024-03-30T13:50:33.972 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cef1b45:::benchmark_data_smithi043_119832_object6607:head# 2024-03-30T13:50:33.972 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cefcd19:::benchmark_data_smithi043_119832_object41167:head# 2024-03-30T13:50:33.972 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ceff10c:::benchmark_data_smithi043_119832_object13690:head# 2024-03-30T13:50:33.972 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cf0b053:::benchmark_data_smithi043_119832_object73707:head# 2024-03-30T13:50:33.972 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cf1043a:::benchmark_data_smithi043_119832_object76545:head# 2024-03-30T13:50:33.972 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cf24f7e:::benchmark_data_smithi043_119832_object59921:head# 2024-03-30T13:50:33.973 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cf2fee1:::benchmark_data_smithi043_119832_object42270:head# 2024-03-30T13:50:33.973 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cf326b8:::benchmark_data_smithi043_119832_object27288:head# 2024-03-30T13:50:33.973 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cf753cd:::benchmark_data_smithi043_119832_object73303:head# 2024-03-30T13:50:33.973 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cf76242:::benchmark_data_smithi043_119832_object6035:head# 2024-03-30T13:50:33.973 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cf7d338:::benchmark_data_smithi043_119832_object39460:head# 2024-03-30T13:50:33.973 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cfa6e1e:::benchmark_data_smithi043_119832_object51286:head# 2024-03-30T13:50:33.973 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cfb0aa5:::benchmark_data_smithi043_119832_object5139:head# 2024-03-30T13:50:33.973 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cfb995b:::benchmark_data_smithi043_119832_object58840:head# 2024-03-30T13:50:33.973 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cfcb234:::benchmark_data_smithi043_119832_object61236:head# 2024-03-30T13:50:33.973 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cfcbfb2:::benchmark_data_smithi043_119832_object19880:head# 2024-03-30T13:50:33.973 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cfef25a:::benchmark_data_smithi043_119832_object50483:head# 2024-03-30T13:50:33.973 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cff0411:::benchmark_data_smithi043_119832_object11922:head# 2024-03-30T13:50:33.973 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cff8c50:::benchmark_data_smithi043_119832_object44465:head# 2024-03-30T13:50:33.974 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7cffe7a5:::benchmark_data_smithi043_119832_object44744:head# 2024-03-30T13:50:33.974 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d00e610:::benchmark_data_smithi043_119832_object20987:head# 2024-03-30T13:50:33.974 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d022bdd:::benchmark_data_smithi043_119832_object60113:head# 2024-03-30T13:50:33.974 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d02d692:::benchmark_data_smithi043_119832_object75564:head# 2024-03-30T13:50:33.974 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d04f2fb:::benchmark_data_smithi043_119832_object24353:head# 2024-03-30T13:50:33.974 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d04fbae:::benchmark_data_smithi043_119832_object8016:head# 2024-03-30T13:50:33.974 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d053f1e:::benchmark_data_smithi043_119832_object76537:head# 2024-03-30T13:50:33.974 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d072d7e:::benchmark_data_smithi043_119832_object4309:head# 2024-03-30T13:50:33.974 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d07ea77:::benchmark_data_smithi043_119832_object60090:head# 2024-03-30T13:50:33.974 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d082da0:::benchmark_data_smithi043_119832_object31830:head# 2024-03-30T13:50:33.974 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d096f54:::benchmark_data_smithi043_119832_object22652:head# 2024-03-30T13:50:33.974 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d0a4824:::benchmark_data_smithi043_119832_object48986:head# 2024-03-30T13:50:33.974 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d0ab0ca:::benchmark_data_smithi043_119832_object4295:head# 2024-03-30T13:50:33.975 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d0b3b9d:::benchmark_data_smithi043_119832_object85193:head# 2024-03-30T13:50:33.975 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d0b85dd:::benchmark_data_smithi043_119832_object20644:head# 2024-03-30T13:50:33.975 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d0c6d1c:::benchmark_data_smithi043_119832_object57555:head# 2024-03-30T13:50:33.975 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d0c87a7:::benchmark_data_smithi043_119832_object2775:head# 2024-03-30T13:50:33.975 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d0c998d:::benchmark_data_smithi043_119832_object46651:head# 2024-03-30T13:50:33.975 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d11203c:::benchmark_data_smithi043_119832_object12208:head# 2024-03-30T13:50:33.975 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d1153d3:::benchmark_data_smithi043_119832_object79543:head# 2024-03-30T13:50:33.975 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d11bba1:::benchmark_data_smithi043_119832_object25637:head# 2024-03-30T13:50:33.975 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d11e406:::benchmark_data_smithi043_119832_object43030:head# 2024-03-30T13:50:33.975 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d12459e:::benchmark_data_smithi043_119832_object39449:head# 2024-03-30T13:50:33.975 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d135223:::benchmark_data_smithi043_119832_object87198:head# 2024-03-30T13:50:33.976 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d14898b:::benchmark_data_smithi043_119832_object15442:head# 2024-03-30T13:50:33.976 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d157fc9:::benchmark_data_smithi043_119832_object84017:head# 2024-03-30T13:50:33.976 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d16e521:::benchmark_data_smithi043_119832_object73709:head# 2024-03-30T13:50:33.976 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d171b32:::benchmark_data_smithi043_119832_object70965:head# 2024-03-30T13:50:33.976 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d17a5bd:::benchmark_data_smithi043_119832_object7713:head# 2024-03-30T13:50:33.976 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d17d547:::benchmark_data_smithi043_119832_object81913:head# 2024-03-30T13:50:33.976 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d1961f7:::benchmark_data_smithi043_119832_object54546:head# 2024-03-30T13:50:33.976 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d19a1d6:::benchmark_data_smithi043_119832_object65320:head# 2024-03-30T13:50:33.976 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d19c5d7:::benchmark_data_smithi043_119832_object44486:head# 2024-03-30T13:50:33.976 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d1afe48:::benchmark_data_smithi043_119832_object84323:head# 2024-03-30T13:50:33.976 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d1b0558:::benchmark_data_smithi043_119832_object77147:head# 2024-03-30T13:50:33.976 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d1b9f92:::benchmark_data_smithi043_119832_object41084:head# 2024-03-30T13:50:33.976 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d1c5232:::benchmark_data_smithi043_119832_object3530:head# 2024-03-30T13:50:33.976 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d1ca4b0:::benchmark_data_smithi043_119832_object23461:head# 2024-03-30T13:50:33.976 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d1d0f72:::benchmark_data_smithi043_119832_object24504:head# 2024-03-30T13:50:33.977 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d1d841d:::benchmark_data_smithi043_119832_object55497:head# 2024-03-30T13:50:33.977 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d1e2d08:::benchmark_data_smithi043_119832_object13963:head# 2024-03-30T13:50:33.977 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d1e62f4:::benchmark_data_smithi043_119832_object85021:head# 2024-03-30T13:50:33.977 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d21427a:::benchmark_data_smithi043_119832_object68719:head# 2024-03-30T13:50:33.977 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d21ff58:::benchmark_data_smithi043_119832_object66975:head# 2024-03-30T13:50:33.977 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d2253ab:::benchmark_data_smithi043_119832_object76782:head# 2024-03-30T13:50:33.977 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d23cedb:::benchmark_data_smithi043_119832_object33927:head# 2024-03-30T13:50:33.977 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d25aca1:::benchmark_data_smithi043_119832_object41348:head# 2024-03-30T13:50:33.977 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d2656db:::benchmark_data_smithi043_119832_object59098:head# 2024-03-30T13:50:33.977 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d278101:::benchmark_data_smithi043_119832_object86047:head# 2024-03-30T13:50:33.977 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d278650:::benchmark_data_smithi043_119832_object81334:head# 2024-03-30T13:50:33.977 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d282409:::benchmark_data_smithi043_119832_object3404:head# 2024-03-30T13:50:33.977 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d28f271:::benchmark_data_smithi043_119832_object32610:head# 2024-03-30T13:50:33.978 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d293b5a:::benchmark_data_smithi043_119832_object27546:head# 2024-03-30T13:50:33.978 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d2a23dc:::benchmark_data_smithi043_119832_object79181:head# 2024-03-30T13:50:33.978 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d2a79b2:::benchmark_data_smithi043_119832_object38126:head# 2024-03-30T13:50:33.978 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d2aacaf:::benchmark_data_smithi043_119832_object32709:head# 2024-03-30T13:50:33.978 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d2bb581:::benchmark_data_smithi043_119832_object29433:head# 2024-03-30T13:50:33.978 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d2bbe58:::benchmark_data_smithi043_119832_object64908:head# 2024-03-30T13:50:33.978 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d2bdf03:::benchmark_data_smithi043_119832_object83806:head# 2024-03-30T13:50:33.978 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d2beba0:::benchmark_data_smithi043_119832_object28591:head# 2024-03-30T13:50:33.978 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d2c32a0:::benchmark_data_smithi043_119832_object78498:head# 2024-03-30T13:50:33.978 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d2e364b:::benchmark_data_smithi043_119832_object43889:head# 2024-03-30T13:50:33.978 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d2e4fc6:::benchmark_data_smithi043_119832_object17519:head# 2024-03-30T13:50:33.978 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d2f1887:::benchmark_data_smithi043_119832_object20976:head# 2024-03-30T13:50:33.978 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d327795:::benchmark_data_smithi043_119832_object78362:head# 2024-03-30T13:50:33.978 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d32c076:::benchmark_data_smithi043_119832_object1989:head# 2024-03-30T13:50:33.979 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d334f86:::benchmark_data_smithi043_119832_object83277:head# 2024-03-30T13:50:33.979 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d34fea7:::benchmark_data_smithi043_119832_object46732:head# 2024-03-30T13:50:33.979 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d353eb8:::benchmark_data_smithi043_119832_object68831:head# 2024-03-30T13:50:33.979 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d378da0:::benchmark_data_smithi043_119832_object30945:head# 2024-03-30T13:50:33.979 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d388ed2:::benchmark_data_smithi043_119832_object88491:head# 2024-03-30T13:50:33.979 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d3acbba:::benchmark_data_smithi043_119832_object87479:head# 2024-03-30T13:50:33.979 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d3b0102:::benchmark_data_smithi043_119832_object64180:head# 2024-03-30T13:50:33.979 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d3b2df6:::benchmark_data_smithi043_119832_object69659:head# 2024-03-30T13:50:33.979 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d3c11a6:::benchmark_data_smithi043_119832_object4010:head# 2024-03-30T13:50:33.979 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d3c5806:::benchmark_data_smithi043_119832_object21406:head# 2024-03-30T13:50:33.979 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d3cf71c:::benchmark_data_smithi043_119832_object76575:head# 2024-03-30T13:50:33.979 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d3f7bf5:::benchmark_data_smithi043_119832_object13701:head# 2024-03-30T13:50:33.979 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d40e02b:::benchmark_data_smithi043_119832_object3847:head# 2024-03-30T13:50:33.987 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:33.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d410d54:::benchmark_data_smithi043_119832_object44923:head# 2024-03-30T13:50:33.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d42aa2a:::benchmark_data_smithi043_119832_object45482:head# 2024-03-30T13:50:33.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d43ce4e:::benchmark_data_smithi043_119832_object62170:head# 2024-03-30T13:50:33.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d453356:::benchmark_data_smithi043_119832_object68546:head# 2024-03-30T13:50:33.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d473a16:::benchmark_data_smithi043_119832_object15682:head# 2024-03-30T13:50:33.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d47815a:::benchmark_data_smithi043_119832_object66326:head# 2024-03-30T13:50:33.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d4886df:::benchmark_data_smithi043_119832_object18277:head# 2024-03-30T13:50:33.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d499364:::benchmark_data_smithi043_119832_object47166:head# 2024-03-30T13:50:33.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d4aeb70:::benchmark_data_smithi043_119832_object60390:head# 2024-03-30T13:50:33.990 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d4b1bee:::benchmark_data_smithi043_119832_object6566:head# 2024-03-30T13:50:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d4b2c5c:::benchmark_data_smithi043_119832_object21808:head# 2024-03-30T13:50:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d4b69b6:::benchmark_data_smithi043_119832_object69739:head# 2024-03-30T13:50:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d4bd048:::benchmark_data_smithi043_119832_object68651:head# 2024-03-30T13:50:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d4caf8b:::benchmark_data_smithi043_119832_object29307:head# 2024-03-30T13:50:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d4ce3c8:::benchmark_data_smithi043_119832_object76645:head# 2024-03-30T13:50:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d4d4636:::benchmark_data_smithi043_119832_object36605:head# 2024-03-30T13:50:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d4d8d6e:::benchmark_data_smithi043_119832_object74288:head# 2024-03-30T13:50:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d4e855e:::benchmark_data_smithi043_119832_object3626:head# 2024-03-30T13:50:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d4f3bcf:::benchmark_data_smithi043_119832_object63859:head# 2024-03-30T13:50:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d4fffd8:::benchmark_data_smithi043_119832_object75862:head# 2024-03-30T13:50:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d50dbf5:::benchmark_data_smithi043_119832_object73722:head# 2024-03-30T13:50:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d50f01c:::benchmark_data_smithi043_119832_object43788:head# 2024-03-30T13:50:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d511434:::benchmark_data_smithi043_119832_object16675:head# 2024-03-30T13:50:33.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d51425e:::benchmark_data_smithi043_119832_object70219:head# 2024-03-30T13:50:33.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d550096:::benchmark_data_smithi043_119832_object55318:head# 2024-03-30T13:50:33.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d55dfc6:::benchmark_data_smithi043_119832_object80838:head# 2024-03-30T13:50:33.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d562f61:::benchmark_data_smithi043_119832_object56843:head# 2024-03-30T13:50:33.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d5634e8:::benchmark_data_smithi043_119832_object88303:head# 2024-03-30T13:50:33.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d57e8e7:::benchmark_data_smithi043_119832_object73882:head# 2024-03-30T13:50:33.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d57fe5c:::benchmark_data_smithi043_119832_object27037:head# 2024-03-30T13:50:33.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d587c96:::benchmark_data_smithi043_119832_object46147:head# 2024-03-30T13:50:33.992 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d5945d8:::benchmark_data_smithi043_119832_object74695:head# 2024-03-30T13:50:33.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d59658b:::benchmark_data_smithi043_119832_object33396:head# 2024-03-30T13:50:33.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d59afac:::benchmark_data_smithi043_119832_object2299:head# 2024-03-30T13:50:33.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d59fb59:::benchmark_data_smithi043_119832_object49620:head# 2024-03-30T13:50:33.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d5af1b2:::benchmark_data_smithi043_119832_object4072:head# 2024-03-30T13:50:33.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d5c57df:::benchmark_data_smithi043_119832_object74446:head# 2024-03-30T13:50:33.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d5ce1f6:::benchmark_data_smithi043_119832_object72330:head# 2024-03-30T13:50:33.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d5f527d:::benchmark_data_smithi043_119832_object45966:head# 2024-03-30T13:50:33.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d617dfe:::benchmark_data_smithi043_119832_object47289:head# 2024-03-30T13:50:33.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d618b64:::benchmark_data_smithi043_119832_object40406:head# 2024-03-30T13:50:33.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d61a68c:::benchmark_data_smithi043_119832_object56350:head# 2024-03-30T13:50:33.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d625cf7:::benchmark_data_smithi043_119832_object44274:head# 2024-03-30T13:50:33.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d640b70:::benchmark_data_smithi043_119832_object51363:head# 2024-03-30T13:50:33.993 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d6436f3:::benchmark_data_smithi043_119832_object50579:head# 2024-03-30T13:50:33.994 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 79 16 116725 116709 92.3175 61.0625 0.0121386 0.0108246 2024-03-30T13:50:33.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:33.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:34.099 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:34.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:34.145 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:34.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:34.250 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:34.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:34.296 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:34.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:34.403 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:34.443 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d64d67f:::benchmark_data_smithi043_119832_object27074:head# 2024-03-30T13:50:34.443 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d65add2:::benchmark_data_smithi043_119832_object5312:head# 2024-03-30T13:50:34.444 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d65ef66:::benchmark_data_smithi043_119832_object78966:head# 2024-03-30T13:50:34.444 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d667b19:::benchmark_data_smithi043_119832_object78354:head# 2024-03-30T13:50:34.444 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d667d79:::benchmark_data_smithi043_119832_object32401:head# 2024-03-30T13:50:34.444 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d668fff:::benchmark_data_smithi043_119832_object39796:head# 2024-03-30T13:50:34.444 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d669305:::benchmark_data_smithi043_119832_object60241:head# 2024-03-30T13:50:34.444 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d66d7e9:::benchmark_data_smithi043_119832_object51392:head# 2024-03-30T13:50:34.444 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d676411:::benchmark_data_smithi043_119832_object73134:head# 2024-03-30T13:50:34.444 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d680f39:::benchmark_data_smithi043_119832_object11672:head# 2024-03-30T13:50:34.444 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d69f76c:::benchmark_data_smithi043_119832_object83179:head# 2024-03-30T13:50:34.444 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d6aa67d:::benchmark_data_smithi043_119832_object19193:head# 2024-03-30T13:50:34.444 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d6aee25:::benchmark_data_smithi043_119832_object65010:head# 2024-03-30T13:50:34.445 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d6c2708:::benchmark_data_smithi043_119832_object84945:head# 2024-03-30T13:50:34.445 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d6c3119:::benchmark_data_smithi043_119832_object57592:head# 2024-03-30T13:50:34.445 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d6c5662:::benchmark_data_smithi043_119832_object17951:head# 2024-03-30T13:50:34.445 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d6c9ea1:::benchmark_data_smithi043_119832_object69564:head# 2024-03-30T13:50:34.445 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d6cf41f:::benchmark_data_smithi043_119832_object54302:head# 2024-03-30T13:50:34.445 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d6d5cbd:::benchmark_data_smithi043_119832_object28813:head# 2024-03-30T13:50:34.445 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d6da292:::benchmark_data_smithi043_119832_object1291:head# 2024-03-30T13:50:34.445 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d6ef02c:::benchmark_data_smithi043_119832_object57763:head# 2024-03-30T13:50:34.445 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d6f283f:::benchmark_data_smithi043_119832_object57570:head# 2024-03-30T13:50:34.445 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d6f9d44:::benchmark_data_smithi043_119832_object74816:head# 2024-03-30T13:50:34.445 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d6ff4ce:::benchmark_data_smithi043_119832_object42855:head# 2024-03-30T13:50:34.445 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d708bd7:::benchmark_data_smithi043_119832_object61807:head# 2024-03-30T13:50:34.446 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d70a859:::benchmark_data_smithi043_119832_object8629:head# 2024-03-30T13:50:34.446 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d70ace4:::benchmark_data_smithi043_119832_object53194:head# 2024-03-30T13:50:34.446 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d712312:::benchmark_data_smithi043_119832_object35265:head# 2024-03-30T13:50:34.446 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d71cc78:::benchmark_data_smithi043_119832_object57837:head# 2024-03-30T13:50:34.446 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d7238db:::benchmark_data_smithi043_119832_object29735:head# 2024-03-30T13:50:34.446 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d735c4b:::benchmark_data_smithi043_119832_object9359:head# 2024-03-30T13:50:34.446 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d75f162:::benchmark_data_smithi043_119832_object83282:head# 2024-03-30T13:50:34.446 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d760d52:::benchmark_data_smithi043_119832_object6601:head# 2024-03-30T13:50:34.446 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d7667ef:::benchmark_data_smithi043_119832_object88104:head# 2024-03-30T13:50:34.446 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d77475e:::benchmark_data_smithi043_119832_object12526:head# 2024-03-30T13:50:34.446 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d7911c3:::benchmark_data_smithi043_119832_object61777:head# 2024-03-30T13:50:34.447 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d79777e:::benchmark_data_smithi043_119832_object33120:head# 2024-03-30T13:50:34.447 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d79b328:::benchmark_data_smithi043_119832_object18354:head# 2024-03-30T13:50:34.447 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d7b964f:::benchmark_data_smithi043_119832_object21278:head# 2024-03-30T13:50:34.447 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d7d1a72:::benchmark_data_smithi043_119832_object44897:head# 2024-03-30T13:50:34.447 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d7d7178:::benchmark_data_smithi043_119832_object3534:head# 2024-03-30T13:50:34.447 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d7db848:::benchmark_data_smithi043_119832_object25584:head# 2024-03-30T13:50:34.447 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d7e1995:::benchmark_data_smithi043_119832_object3406:head# 2024-03-30T13:50:34.447 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d7fcd4a:::benchmark_data_smithi043_119832_object13911:head# 2024-03-30T13:50:34.447 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d808035:::benchmark_data_smithi043_119832_object53451:head# 2024-03-30T13:50:34.447 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d8081b7:::benchmark_data_smithi043_119832_object78543:head# 2024-03-30T13:50:34.447 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d80fb37:::benchmark_data_smithi043_119832_object80941:head# 2024-03-30T13:50:34.448 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d80fe64:::benchmark_data_smithi043_119832_object72877:head# 2024-03-30T13:50:34.448 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d814a40:::benchmark_data_smithi043_119832_object1364:head# 2024-03-30T13:50:34.448 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d8173f9:::benchmark_data_smithi043_119832_object41930:head# 2024-03-30T13:50:34.448 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d81741f:::benchmark_data_smithi043_119832_object68613:head# 2024-03-30T13:50:34.448 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d828dfb:::benchmark_data_smithi043_119832_object29232:head# 2024-03-30T13:50:34.448 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d8300b5:::benchmark_data_smithi043_119832_object66320:head# 2024-03-30T13:50:34.448 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d83a12b:::benchmark_data_smithi043_119832_object61824:head# 2024-03-30T13:50:34.448 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d842775:::benchmark_data_smithi043_119832_object16545:head# 2024-03-30T13:50:34.448 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d8678e9:::benchmark_data_smithi043_119832_object75627:head# 2024-03-30T13:50:34.448 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d88cff9:::benchmark_data_smithi043_119832_object26590:head# 2024-03-30T13:50:34.448 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d8a8288:::benchmark_data_smithi043_119832_object49568:head# 2024-03-30T13:50:34.449 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d8a9b97:::benchmark_data_smithi043_119832_object77763:head# 2024-03-30T13:50:34.449 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d8b79f1:::benchmark_data_smithi043_119832_object15866:head# 2024-03-30T13:50:34.449 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d8c4488:::benchmark_data_smithi043_119832_object76672:head# 2024-03-30T13:50:34.449 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d8e0928:::benchmark_data_smithi043_119832_object82298:head# 2024-03-30T13:50:34.449 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d8e8342:::benchmark_data_smithi043_119832_object61196:head# 2024-03-30T13:50:34.449 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d8e8528:::benchmark_data_smithi043_119832_object34335:head# 2024-03-30T13:50:34.449 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d906b05:::benchmark_data_smithi043_119832_object42003:head# 2024-03-30T13:50:34.449 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d910ed3:::benchmark_data_smithi043_119832_object26650:head# 2024-03-30T13:50:34.449 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d9221b6:::benchmark_data_smithi043_119832_object88391:head# 2024-03-30T13:50:34.449 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d93cc8b:::benchmark_data_smithi043_119832_object80282:head# 2024-03-30T13:50:34.449 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d948974:::benchmark_data_smithi043_119832_object74568:head# 2024-03-30T13:50:34.449 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d96627b:::benchmark_data_smithi043_119832_object38197:head# 2024-03-30T13:50:34.450 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d96cfbb:::benchmark_data_smithi043_119832_object12090:head# 2024-03-30T13:50:34.450 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d976876:::benchmark_data_smithi043_119832_object88423:head# 2024-03-30T13:50:34.450 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d97bcd0:::benchmark_data_smithi043_119832_object13542:head# 2024-03-30T13:50:34.450 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d99cbf2:::benchmark_data_smithi043_119832_object44610:head# 2024-03-30T13:50:34.450 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d9a333d:::benchmark_data_smithi043_119832_object33918:head# 2024-03-30T13:50:34.450 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d9a7b8e:::benchmark_data_smithi043_119832_object77792:head# 2024-03-30T13:50:34.450 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d9abf81:::benchmark_data_smithi043_119832_object37726:head# 2024-03-30T13:50:34.450 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d9b4437:::benchmark_data_smithi043_119832_object51043:head# 2024-03-30T13:50:34.450 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d9b7c53:::benchmark_data_smithi043_119832_object60920:head# 2024-03-30T13:50:34.450 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d9c3ff6:::benchmark_data_smithi043_119832_object18758:head# 2024-03-30T13:50:34.450 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d9cd66d:::benchmark_data_smithi043_119832_object79266:head# 2024-03-30T13:50:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d9d412f:::benchmark_data_smithi043_119832_object7708:head# 2024-03-30T13:50:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d9db571:::benchmark_data_smithi043_119832_object81068:head# 2024-03-30T13:50:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d9e2842:::benchmark_data_smithi043_119832_object27914:head# 2024-03-30T13:50:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d9fb062:::benchmark_data_smithi043_119832_object20548:head# 2024-03-30T13:50:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d9fba38:::benchmark_data_smithi043_119832_object33475:head# 2024-03-30T13:50:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d9fd889:::benchmark_data_smithi043_119832_object81518:head# 2024-03-30T13:50:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7d9fe21e:::benchmark_data_smithi043_119832_object80298:head# 2024-03-30T13:50:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7da0d397:::benchmark_data_smithi043_119832_object27173:head# 2024-03-30T13:50:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7da2c8e9:::benchmark_data_smithi043_119832_object60749:head# 2024-03-30T13:50:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7da2e5c1:::benchmark_data_smithi043_119832_object20735:head# 2024-03-30T13:50:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7da3231b:::benchmark_data_smithi043_119832_object48421:head# 2024-03-30T13:50:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7da38602:::benchmark_data_smithi043_119832_object52919:head# 2024-03-30T13:50:34.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7da46a7b:::benchmark_data_smithi043_119832_object3113:head# 2024-03-30T13:50:34.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7da5b86c:::benchmark_data_smithi043_119832_object63474:head# 2024-03-30T13:50:34.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7da69114:::benchmark_data_smithi043_119832_object19847:head# 2024-03-30T13:50:34.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7da724a6:::benchmark_data_smithi043_119832_object75855:head# 2024-03-30T13:50:34.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7da7c618:::benchmark_data_smithi043_119832_object19127:head# 2024-03-30T13:50:34.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7da7d371:::benchmark_data_smithi043_119832_object44727:head# 2024-03-30T13:50:34.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7da8c7c5:::benchmark_data_smithi043_119832_object79893:head# 2024-03-30T13:50:34.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7da9faed:::benchmark_data_smithi043_119832_object84943:head# 2024-03-30T13:50:34.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7daaac3e:::benchmark_data_smithi043_119832_object3472:head# 2024-03-30T13:50:34.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dab3e9f:::benchmark_data_smithi043_119832_object5645:head# 2024-03-30T13:50:34.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dadeed2:::benchmark_data_smithi043_119832_object82584:head# 2024-03-30T13:50:34.452 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7daf8857:::benchmark_data_smithi043_119832_object77236:head# 2024-03-30T13:50:34.453 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dafea86:::benchmark_data_smithi043_119832_object69826:head# 2024-03-30T13:50:34.453 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7db05f06:::benchmark_data_smithi043_119832_object71328:head# 2024-03-30T13:50:34.453 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7db0eaaa:::benchmark_data_smithi043_119832_object71739:head# 2024-03-30T13:50:34.453 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7db12ed8:::benchmark_data_smithi043_119832_object34659:head# 2024-03-30T13:50:34.453 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7db3f317:::benchmark_data_smithi043_119832_object68677:head# 2024-03-30T13:50:34.453 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7db4c447:::benchmark_data_smithi043_119832_object85867:head# 2024-03-30T13:50:34.453 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7db4f453:::benchmark_data_smithi043_119832_object881:head# 2024-03-30T13:50:34.454 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7db4fd88:::benchmark_data_smithi043_119832_object49657:head# 2024-03-30T13:50:34.454 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7db58523:::benchmark_data_smithi043_119832_object35130:head# 2024-03-30T13:50:34.454 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7db5cc95:::benchmark_data_smithi043_119832_object15124:head# 2024-03-30T13:50:34.454 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7db64696:::benchmark_data_smithi043_119832_object68832:head# 2024-03-30T13:50:34.454 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7db69982:::benchmark_data_smithi043_119832_object79843:head# 2024-03-30T13:50:34.454 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7db796fe:::benchmark_data_smithi043_119832_object70107:head# 2024-03-30T13:50:34.454 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7db7c6c9:::benchmark_data_smithi043_119832_object39806:head# 2024-03-30T13:50:34.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7db7d878:::benchmark_data_smithi043_119832_object37339:head# 2024-03-30T13:50:34.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7db92a18:::benchmark_data_smithi043_119832_object33167:head# 2024-03-30T13:50:34.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7db96452:::benchmark_data_smithi043_119832_object85663:head# 2024-03-30T13:50:34.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dba483f:::benchmark_data_smithi043_119832_object84477:head# 2024-03-30T13:50:34.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dbaa0dc:::benchmark_data_smithi043_119832_object31879:head# 2024-03-30T13:50:34.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dbb224e:::benchmark_data_smithi043_119832_object2590:head# 2024-03-30T13:50:34.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dbb67fe:::benchmark_data_smithi043_119832_object59487:head# 2024-03-30T13:50:34.455 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dbbbfc3:::benchmark_data_smithi043_119832_object61112:head# 2024-03-30T13:50:34.456 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dbbe545:::benchmark_data_smithi043_119832_object33473:head# 2024-03-30T13:50:34.456 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dbc74f8:::benchmark_data_smithi043_119832_object54354:head# 2024-03-30T13:50:34.456 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dbcf116:::benchmark_data_smithi043_119832_object55000:head# 2024-03-30T13:50:34.456 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dbd9ac4:::benchmark_data_smithi043_119832_object46824:head# 2024-03-30T13:50:34.456 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dbda748:::benchmark_data_smithi043_119832_object63783:head# 2024-03-30T13:50:34.456 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dbe87c9:::benchmark_data_smithi043_119832_object48626:head# 2024-03-30T13:50:34.456 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dbe8bbd:::benchmark_data_smithi043_119832_object41776:head# 2024-03-30T13:50:34.456 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dbe98cc:::benchmark_data_smithi043_119832_object40149:head# 2024-03-30T13:50:34.457 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dbec3b9:::benchmark_data_smithi043_119832_object60936:head# 2024-03-30T13:50:34.457 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dbeeb57:::benchmark_data_smithi043_119832_object36212:head# 2024-03-30T13:50:34.457 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dbf9fb6:::benchmark_data_smithi043_119832_object71932:head# 2024-03-30T13:50:34.457 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dc00703:::benchmark_data_smithi043_119832_object83495:head# 2024-03-30T13:50:34.457 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dc1f24a:::benchmark_data_smithi043_119832_object25222:head# 2024-03-30T13:50:34.457 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dc2bd83:::benchmark_data_smithi043_119832_object33572:head# 2024-03-30T13:50:34.457 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dc2d009:::benchmark_data_smithi043_119832_object55850:head# 2024-03-30T13:50:34.457 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dc45673:::benchmark_data_smithi043_119832_object40726:head# 2024-03-30T13:50:34.457 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dc5cc1e:::benchmark_data_smithi043_119832_object55742:head# 2024-03-30T13:50:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dc66d4c:::benchmark_data_smithi043_119832_object6058:head# 2024-03-30T13:50:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dc8312d:::benchmark_data_smithi043_119832_object47849:head# 2024-03-30T13:50:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dc849d8:::benchmark_data_smithi043_119832_object40314:head# 2024-03-30T13:50:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dc945a7:::benchmark_data_smithi043_119832_object41306:head# 2024-03-30T13:50:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dcb9a32:::benchmark_data_smithi043_119832_object52173:head# 2024-03-30T13:50:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dcc6160:::benchmark_data_smithi043_119832_object79918:head# 2024-03-30T13:50:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dcd4d15:::benchmark_data_smithi043_119832_object64052:head# 2024-03-30T13:50:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dce93ff:::benchmark_data_smithi043_119832_object57228:head# 2024-03-30T13:50:34.459 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dcf5ac7:::benchmark_data_smithi043_119832_object48946:head# 2024-03-30T13:50:34.459 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dcfd991:::benchmark_data_smithi043_119832_object25860:head# 2024-03-30T13:50:34.459 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd00e34:::benchmark_data_smithi043_119832_object18450:head# 2024-03-30T13:50:34.459 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd1f3ef:::benchmark_data_smithi043_119832_object36843:head# 2024-03-30T13:50:34.459 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd25ecf:::benchmark_data_smithi043_119832_object62891:head# 2024-03-30T13:50:34.459 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd2e710:::benchmark_data_smithi043_119832_object36776:head# 2024-03-30T13:50:34.459 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd31772:::benchmark_data_smithi043_119832_object56265:head# 2024-03-30T13:50:34.459 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd428fe:::benchmark_data_smithi043_119832_object13478:head# 2024-03-30T13:50:34.459 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd4b319:::benchmark_data_smithi043_119832_object64361:head# 2024-03-30T13:50:34.459 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd4c915:::benchmark_data_smithi043_119832_object71983:head# 2024-03-30T13:50:34.460 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd5182e:::benchmark_data_smithi043_119832_object13068:head# 2024-03-30T13:50:34.460 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd5fa66:::benchmark_data_smithi043_119832_object39041:head# 2024-03-30T13:50:34.460 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd72982:::benchmark_data_smithi043_119832_object52264:head# 2024-03-30T13:50:34.460 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd73ac0:::benchmark_data_smithi043_119832_object30853:head# 2024-03-30T13:50:34.460 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd73dae:::benchmark_data_smithi043_119832_object25311:head# 2024-03-30T13:50:34.460 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd758be:::benchmark_data_smithi043_119832_object71943:head# 2024-03-30T13:50:34.460 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd762c6:::benchmark_data_smithi043_119832_object67324:head# 2024-03-30T13:50:34.460 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd7c4df:::benchmark_data_smithi043_119832_object58408:head# 2024-03-30T13:50:34.461 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd7e086:::benchmark_data_smithi043_119832_object88085:head# 2024-03-30T13:50:34.461 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd7f0a0:::benchmark_data_smithi043_119832_object34069:head# 2024-03-30T13:50:34.461 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd83108:::benchmark_data_smithi043_119832_object6024:head# 2024-03-30T13:50:34.461 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd868d7:::benchmark_data_smithi043_119832_object81363:head# 2024-03-30T13:50:34.461 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd8aa84:::benchmark_data_smithi043_119832_object51692:head# 2024-03-30T13:50:34.461 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dd9eb91:::benchmark_data_smithi043_119832_object14391:head# 2024-03-30T13:50:34.461 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ddcc70b:::benchmark_data_smithi043_119832_object36775:head# 2024-03-30T13:50:34.461 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dde2507:::benchmark_data_smithi043_119832_object5862:head# 2024-03-30T13:50:34.461 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dde6f4e:::benchmark_data_smithi043_119832_object55526:head# 2024-03-30T13:50:34.461 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ddfed85:::benchmark_data_smithi043_119832_object85745:head# 2024-03-30T13:50:34.461 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7de110ff:::benchmark_data_smithi043_119832_object26708:head# 2024-03-30T13:50:34.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7de13348:::benchmark_data_smithi043_119832_object4929:head# 2024-03-30T13:50:34.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7de14cbf:::benchmark_data_smithi043_119832_object69173:head# 2024-03-30T13:50:34.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7de298ad:::benchmark_data_smithi043_119832_object85670:head# 2024-03-30T13:50:34.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7de2bc60:::benchmark_data_smithi043_119832_object87994:head# 2024-03-30T13:50:34.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7de3d1a8:::benchmark_data_smithi043_119832_object24567:head# 2024-03-30T13:50:34.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7de3ef8f:::benchmark_data_smithi043_119832_object13266:head# 2024-03-30T13:50:34.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7de52e66:::benchmark_data_smithi043_119832_object31705:head# 2024-03-30T13:50:34.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7de574be:::benchmark_data_smithi043_119832_object71288:head# 2024-03-30T13:50:34.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7de5ac2f:::benchmark_data_smithi043_119832_object61735:head# 2024-03-30T13:50:34.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7de5e97a:::benchmark_data_smithi043_119832_object57310:head# 2024-03-30T13:50:34.462 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7de753f9:::benchmark_data_smithi043_119832_object43391:head# 2024-03-30T13:50:34.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7de7845c:::benchmark_data_smithi043_119832_object35059:head# 2024-03-30T13:50:34.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7de7d175:::benchmark_data_smithi043_119832_object4644:head# 2024-03-30T13:50:34.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7de7ee93:::benchmark_data_smithi043_119832_object45626:head# 2024-03-30T13:50:34.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7de86c87:::benchmark_data_smithi043_119832_object42619:head# 2024-03-30T13:50:34.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7de88b73:::benchmark_data_smithi043_119832_object65554:head# 2024-03-30T13:50:34.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7de8c4b3:::benchmark_data_smithi043_119832_object73434:head# 2024-03-30T13:50:34.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7de8f2a6:::benchmark_data_smithi043_119832_object47971:head# 2024-03-30T13:50:34.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7de9012f:::benchmark_data_smithi043_119832_object66255:head# 2024-03-30T13:50:34.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7deaf0f9:::benchmark_data_smithi043_119832_object80593:head# 2024-03-30T13:50:34.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7deb0de7:::benchmark_data_smithi043_119832_object643:head# 2024-03-30T13:50:34.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dec04b9:::benchmark_data_smithi043_119832_object62604:head# 2024-03-30T13:50:34.463 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dec83ab:::benchmark_data_smithi043_119832_object49719:head# 2024-03-30T13:50:34.464 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7deca62a:::benchmark_data_smithi043_119832_object83360:head# 2024-03-30T13:50:34.464 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7deccbba:::benchmark_data_smithi043_119832_object85219:head# 2024-03-30T13:50:34.464 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ded9c49:::benchmark_data_smithi043_119832_object46764:head# 2024-03-30T13:50:34.464 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7deec61d:::benchmark_data_smithi043_119832_object12548:head# 2024-03-30T13:50:34.464 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7deeec61:::benchmark_data_smithi043_119832_object28803:head# 2024-03-30T13:50:34.464 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7def15f2:::benchmark_data_smithi043_119832_object44327:head# 2024-03-30T13:50:34.464 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7def8c6b:::benchmark_data_smithi043_119832_object14793:head# 2024-03-30T13:50:34.464 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7df00095:::benchmark_data_smithi043_119832_object48052:head# 2024-03-30T13:50:34.464 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7df0df5b:::benchmark_data_smithi043_119832_object66733:head# 2024-03-30T13:50:34.464 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7df1006e:::benchmark_data_smithi043_119832_object31900:head# 2024-03-30T13:50:34.465 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7df13cd7:::benchmark_data_smithi043_119832_object3328:head# 2024-03-30T13:50:34.465 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7df16327:::benchmark_data_smithi043_119832_object81041:head# 2024-03-30T13:50:34.465 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7df16578:::benchmark_data_smithi043_119832_object62647:head# 2024-03-30T13:50:34.465 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7df2c9fe:::benchmark_data_smithi043_119832_object29156:head# 2024-03-30T13:50:34.465 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7df2fe36:::benchmark_data_smithi043_119832_object6027:head# 2024-03-30T13:50:34.465 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7df31c9f:::benchmark_data_smithi043_119832_object86163:head# 2024-03-30T13:50:34.465 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7df3a787:::benchmark_data_smithi043_119832_object62946:head# 2024-03-30T13:50:34.465 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7df48d1a:::benchmark_data_smithi043_119832_object12876:head# 2024-03-30T13:50:34.466 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7df68789:::benchmark_data_smithi043_119832_object87816:head# 2024-03-30T13:50:34.466 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7df7b521:::benchmark_data_smithi043_119832_object9371:head# 2024-03-30T13:50:34.466 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7df7e0d1:::benchmark_data_smithi043_119832_object4536:head# 2024-03-30T13:50:34.466 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7df8ada7:::benchmark_data_smithi043_119832_object53306:head# 2024-03-30T13:50:34.466 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7df8ee4d:::benchmark_data_smithi043_119832_object85435:head# 2024-03-30T13:50:34.466 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7df901e3:::benchmark_data_smithi043_119832_object41500:head# 2024-03-30T13:50:34.466 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dfa3417:::benchmark_data_smithi043_119832_object38472:head# 2024-03-30T13:50:34.466 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dfa9e92:::benchmark_data_smithi043_119832_object56451:head# 2024-03-30T13:50:34.467 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dfc7089:::benchmark_data_smithi043_119832_object7880:head# 2024-03-30T13:50:34.467 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dfc7a95:::benchmark_data_smithi043_119832_object48879:head# 2024-03-30T13:50:34.467 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dfca142:::benchmark_data_smithi043_119832_object65680:head# 2024-03-30T13:50:34.467 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dfd13d6:::benchmark_data_smithi043_119832_object46700:head# 2024-03-30T13:50:34.467 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dfd39db:::benchmark_data_smithi043_119832_object18590:head# 2024-03-30T13:50:34.467 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dfddd99:::benchmark_data_smithi043_119832_object41165:head# 2024-03-30T13:50:34.467 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7dff9a43:::benchmark_data_smithi043_119832_object60685:head# 2024-03-30T13:50:34.468 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e006412:::benchmark_data_smithi043_119832_object50096:head# 2024-03-30T13:50:34.468 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e006ac8:::benchmark_data_smithi043_119832_object72589:head# 2024-03-30T13:50:34.468 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e00a23b:::benchmark_data_smithi043_119832_object70683:head# 2024-03-30T13:50:34.468 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e00d335:::benchmark_data_smithi043_119832_object67666:head# 2024-03-30T13:50:34.468 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e0130a9:::benchmark_data_smithi043_119832_object37357:head# 2024-03-30T13:50:34.468 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e01c0a5:::benchmark_data_smithi043_119832_object24976:head# 2024-03-30T13:50:34.469 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e01d508:::benchmark_data_smithi043_119832_object86196:head# 2024-03-30T13:50:34.469 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e0402a2:::benchmark_data_smithi043_119832_object12279:head# 2024-03-30T13:50:34.469 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e05a3d4:::benchmark_data_smithi043_119832_object39178:head# 2024-03-30T13:50:34.469 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e0766dc:::benchmark_data_smithi043_119832_object78358:head# 2024-03-30T13:50:34.469 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e07f41d:::benchmark_data_smithi043_119832_object11511:head# 2024-03-30T13:50:34.469 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e086090:::benchmark_data_smithi043_119832_object88003:head# 2024-03-30T13:50:34.469 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e08ce93:::benchmark_data_smithi043_119832_object83671:head# 2024-03-30T13:50:34.469 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e08e20a:::benchmark_data_smithi043_119832_object16665:head# 2024-03-30T13:50:34.470 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e0b924e:::benchmark_data_smithi043_119832_object34609:head# 2024-03-30T13:50:34.470 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e0d8542:::benchmark_data_smithi043_119832_object75831:head# 2024-03-30T13:50:34.470 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e0e616a:::benchmark_data_smithi043_119832_object57127:head# 2024-03-30T13:50:34.470 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e0f0329:::benchmark_data_smithi043_119832_object9602:head# 2024-03-30T13:50:34.470 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e115996:::benchmark_data_smithi043_119832_object70241:head# 2024-03-30T13:50:34.470 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e13b47d:::benchmark_data_smithi043_119832_object17394:head# 2024-03-30T13:50:34.470 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e13df1b:::benchmark_data_smithi043_119832_object39102:head# 2024-03-30T13:50:34.470 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e144382:::benchmark_data_smithi043_119832_object35463:head# 2024-03-30T13:50:34.470 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e145579:::benchmark_data_smithi043_119832_object79504:head# 2024-03-30T13:50:34.470 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e15e106:::benchmark_data_smithi043_119832_object9191:head# 2024-03-30T13:50:34.470 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e165238:::benchmark_data_smithi043_119832_object60472:head# 2024-03-30T13:50:34.470 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e168b92:::benchmark_data_smithi043_119832_object5579:head# 2024-03-30T13:50:34.471 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e16ad7c:::benchmark_data_smithi043_119832_object74413:head# 2024-03-30T13:50:34.471 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e16c966:::benchmark_data_smithi043_119832_object82172:head# 2024-03-30T13:50:34.471 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e173046:::benchmark_data_smithi043_119832_object14699:head# 2024-03-30T13:50:34.471 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e185945:::benchmark_data_smithi043_119832_object28833:head# 2024-03-30T13:50:34.471 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e19bda5:::benchmark_data_smithi043_119832_object43666:head# 2024-03-30T13:50:34.471 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e1ab598:::benchmark_data_smithi043_119832_object36903:head# 2024-03-30T13:50:34.471 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e1ae7a8:::benchmark_data_smithi043_119832_object945:head# 2024-03-30T13:50:34.471 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e1b216a:::benchmark_data_smithi043_119832_object5572:head# 2024-03-30T13:50:34.471 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e1bd9dc:::benchmark_data_smithi043_119832_object48241:head# 2024-03-30T13:50:34.471 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e1cca44:::benchmark_data_smithi043_119832_object48184:head# 2024-03-30T13:50:34.471 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e1d50ce:::benchmark_data_smithi043_119832_object42691:head# 2024-03-30T13:50:34.471 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e1dfb11:::benchmark_data_smithi043_119832_object73379:head# 2024-03-30T13:50:34.472 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e1e5d29:::benchmark_data_smithi043_119832_object15564:head# 2024-03-30T13:50:34.472 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e1f9aee:::benchmark_data_smithi043_119832_object26909:head# 2024-03-30T13:50:34.472 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e20b5de:::benchmark_data_smithi043_119832_object62372:head# 2024-03-30T13:50:34.472 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e20efae:::benchmark_data_smithi043_119832_object742:head# 2024-03-30T13:50:34.472 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e20f1ad:::benchmark_data_smithi043_119832_object26328:head# 2024-03-30T13:50:34.472 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e213bf3:::benchmark_data_smithi043_119832_object11250:head# 2024-03-30T13:50:34.472 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e215dbf:::benchmark_data_smithi043_119832_object57432:head# 2024-03-30T13:50:34.472 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e233bee:::benchmark_data_smithi043_119832_object37882:head# 2024-03-30T13:50:34.472 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e241828:::benchmark_data_smithi043_119832_object68367:head# 2024-03-30T13:50:34.472 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e24d74f:::benchmark_data_smithi043_119832_object75479:head# 2024-03-30T13:50:34.472 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e251e83:::benchmark_data_smithi043_119832_object79124:head# 2024-03-30T13:50:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e25570c:::benchmark_data_smithi043_119832_object39534:head# 2024-03-30T13:50:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e259382:::benchmark_data_smithi043_119832_object53637:head# 2024-03-30T13:50:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e25a238:::benchmark_data_smithi043_119832_object63428:head# 2024-03-30T13:50:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e25aa32:::benchmark_data_smithi043_119832_object35933:head# 2024-03-30T13:50:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e25deae:::benchmark_data_smithi043_119832_object69226:head# 2024-03-30T13:50:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e263521:::benchmark_data_smithi043_119832_object58973:head# 2024-03-30T13:50:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e273684:::benchmark_data_smithi043_119832_object51703:head# 2024-03-30T13:50:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e279bd8:::benchmark_data_smithi043_119832_object24092:head# 2024-03-30T13:50:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e28d467:::benchmark_data_smithi043_119832_object58205:head# 2024-03-30T13:50:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e2a5f08:::benchmark_data_smithi043_119832_object59294:head# 2024-03-30T13:50:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e2c6044:::benchmark_data_smithi043_119832_object20621:head# 2024-03-30T13:50:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e2db6db:::benchmark_data_smithi043_119832_object49122:head# 2024-03-30T13:50:34.474 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e2e578f:::benchmark_data_smithi043_119832_object56169:head# 2024-03-30T13:50:34.474 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e30bcd0:::benchmark_data_smithi043_119832_object6410:head# 2024-03-30T13:50:34.474 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e30caad:::benchmark_data_smithi043_119832_object44986:head# 2024-03-30T13:50:34.474 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e312672:::benchmark_data_smithi043_119832_object73193:head# 2024-03-30T13:50:34.474 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e324c31:::benchmark_data_smithi043_119832_object23486:head# 2024-03-30T13:50:34.474 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e330b80:::benchmark_data_smithi043_119832_object64245:head# 2024-03-30T13:50:34.474 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e330d36:::benchmark_data_smithi043_119832_object47854:head# 2024-03-30T13:50:34.474 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e33734c:::benchmark_data_smithi043_119832_object17495:head# 2024-03-30T13:50:34.474 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e34467a:::benchmark_data_smithi043_119832_object12086:head# 2024-03-30T13:50:34.474 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e345fc7:::benchmark_data_smithi043_119832_object13360:head# 2024-03-30T13:50:34.474 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e3550cc:::benchmark_data_smithi043_119832_object3563:head# 2024-03-30T13:50:34.475 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e3561f7:::benchmark_data_smithi043_119832_object54401:head# 2024-03-30T13:50:34.475 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e35a624:::benchmark_data_smithi043_119832_object55144:head# 2024-03-30T13:50:34.475 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e368dc3:::benchmark_data_smithi043_119832_object37495:head# 2024-03-30T13:50:34.475 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e384a20:::benchmark_data_smithi043_119832_object6548:head# 2024-03-30T13:50:34.475 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e3ab692:::benchmark_data_smithi043_119832_object10069:head# 2024-03-30T13:50:34.475 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e3afef8:::benchmark_data_smithi043_119832_object47643:head# 2024-03-30T13:50:34.475 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e3b5504:::benchmark_data_smithi043_119832_object48915:head# 2024-03-30T13:50:34.475 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e3d40e3:::benchmark_data_smithi043_119832_object41472:head# 2024-03-30T13:50:34.475 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e3df940:::benchmark_data_smithi043_119832_object36472:head# 2024-03-30T13:50:34.475 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e3e59a9:::benchmark_data_smithi043_119832_object35596:head# 2024-03-30T13:50:34.475 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e3f4726:::benchmark_data_smithi043_119832_object34646:head# 2024-03-30T13:50:34.475 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e3fdf14:::benchmark_data_smithi043_119832_object88071:head# 2024-03-30T13:50:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e42b110:::benchmark_data_smithi043_119832_object48581:head# 2024-03-30T13:50:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e42ef3a:::benchmark_data_smithi043_119832_object33130:head# 2024-03-30T13:50:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e43018f:::benchmark_data_smithi043_119832_object44090:head# 2024-03-30T13:50:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e44bf7f:::benchmark_data_smithi043_119832_object2905:head# 2024-03-30T13:50:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e452bba:::benchmark_data_smithi043_119832_object61224:head# 2024-03-30T13:50:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e4562f0:::benchmark_data_smithi043_119832_object44391:head# 2024-03-30T13:50:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e465512:::benchmark_data_smithi043_119832_object65840:head# 2024-03-30T13:50:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e46c397:::benchmark_data_smithi043_119832_object3265:head# 2024-03-30T13:50:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e475978:::benchmark_data_smithi043_119832_object64226:head# 2024-03-30T13:50:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e4942d7:::benchmark_data_smithi043_119832_object20870:head# 2024-03-30T13:50:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e4ac814:::benchmark_data_smithi043_119832_object49112:head# 2024-03-30T13:50:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e4bafe6:::benchmark_data_smithi043_119832_object21397:head# 2024-03-30T13:50:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e4be72e:::benchmark_data_smithi043_119832_object61184:head# 2024-03-30T13:50:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e4cfeb3:::benchmark_data_smithi043_119832_object36830:head# 2024-03-30T13:50:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e4d2980:::benchmark_data_smithi043_119832_object17106:head# 2024-03-30T13:50:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e4d62da:::benchmark_data_smithi043_119832_object2199:head# 2024-03-30T13:50:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e4d6c38:::benchmark_data_smithi043_119832_object48852:head# 2024-03-30T13:50:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e4dcd89:::benchmark_data_smithi043_119832_object71113:head# 2024-03-30T13:50:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e4fac57:::benchmark_data_smithi043_119832_object42952:head# 2024-03-30T13:50:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e504cfb:::benchmark_data_smithi043_119832_object51519:head# 2024-03-30T13:50:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e51e051:::benchmark_data_smithi043_119832_object33635:head# 2024-03-30T13:50:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e522467:::benchmark_data_smithi043_119832_object42110:head# 2024-03-30T13:50:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e52fa37:::benchmark_data_smithi043_119832_object20418:head# 2024-03-30T13:50:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e5425aa:::benchmark_data_smithi043_119832_object8994:head# 2024-03-30T13:50:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e54558b:::benchmark_data_smithi043_119832_object68735:head# 2024-03-30T13:50:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e554287:::benchmark_data_smithi043_119832_object42881:head# 2024-03-30T13:50:34.478 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e59b970:::benchmark_data_smithi043_119832_object1822:head# 2024-03-30T13:50:34.478 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e59e13c:::benchmark_data_smithi043_119832_object44243:head# 2024-03-30T13:50:34.478 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e59f6ec:::benchmark_data_smithi043_119832_object72898:head# 2024-03-30T13:50:34.478 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e5a68c1:::benchmark_data_smithi043_119832_object62515:head# 2024-03-30T13:50:34.478 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e5aa046:::benchmark_data_smithi043_119832_object59977:head# 2024-03-30T13:50:34.478 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e5ac30e:::benchmark_data_smithi043_119832_object83125:head# 2024-03-30T13:50:34.478 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e5aebf2:::benchmark_data_smithi043_119832_object699:head# 2024-03-30T13:50:34.478 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e5b1103:::benchmark_data_smithi043_119832_object32021:head# 2024-03-30T13:50:34.478 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e5b45ed:::benchmark_data_smithi043_119832_object54912:head# 2024-03-30T13:50:34.478 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e5d8b75:::benchmark_data_smithi043_119832_object8194:head# 2024-03-30T13:50:34.478 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e5decc0:::benchmark_data_smithi043_119832_object9318:head# 2024-03-30T13:50:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e5f9b76:::benchmark_data_smithi043_119832_object74941:head# 2024-03-30T13:50:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e5fb5f4:::benchmark_data_smithi043_119832_object79304:head# 2024-03-30T13:50:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e5fd69c:::benchmark_data_smithi043_119832_object84354:head# 2024-03-30T13:50:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e6043d9:::benchmark_data_smithi043_119832_object8273:head# 2024-03-30T13:50:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e61b5e4:::benchmark_data_smithi043_119832_object85763:head# 2024-03-30T13:50:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e620987:::benchmark_data_smithi043_119832_object31727:head# 2024-03-30T13:50:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e625534:::benchmark_data_smithi043_119832_object70068:head# 2024-03-30T13:50:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e625e32:::benchmark_data_smithi043_119832_object30443:head# 2024-03-30T13:50:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e63e7d1:::benchmark_data_smithi043_119832_object80116:head# 2024-03-30T13:50:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e66f301:::benchmark_data_smithi043_119832_object63341:head# 2024-03-30T13:50:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e6755d8:::benchmark_data_smithi043_119832_object13534:head# 2024-03-30T13:50:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e6a0e91:::benchmark_data_smithi043_119832_object34219:head# 2024-03-30T13:50:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e6ae25f:::benchmark_data_smithi043_119832_object9337:head# 2024-03-30T13:50:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e6b098b:::benchmark_data_smithi043_119832_object85308:head# 2024-03-30T13:50:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e6b4e85:::benchmark_data_smithi043_119832_object74582:head# 2024-03-30T13:50:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e6d2594:::benchmark_data_smithi043_119832_object69472:head# 2024-03-30T13:50:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e6d910e:::benchmark_data_smithi043_119832_object72691:head# 2024-03-30T13:50:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e6f306c:::benchmark_data_smithi043_119832_object79101:head# 2024-03-30T13:50:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e702471:::benchmark_data_smithi043_119832_object52793:head# 2024-03-30T13:50:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e70f6d2:::benchmark_data_smithi043_119832_object52617:head# 2024-03-30T13:50:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e711cf7:::benchmark_data_smithi043_119832_object76860:head# 2024-03-30T13:50:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e72e9bd:::benchmark_data_smithi043_119832_object62371:head# 2024-03-30T13:50:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e73551a:::benchmark_data_smithi043_119832_object43066:head# 2024-03-30T13:50:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e736822:::benchmark_data_smithi043_119832_object8916:head# 2024-03-30T13:50:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e73fb84:::benchmark_data_smithi043_119832_object36496:head# 2024-03-30T13:50:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e74d776:::benchmark_data_smithi043_119832_object83876:head# 2024-03-30T13:50:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e764b64:::benchmark_data_smithi043_119832_object44880:head# 2024-03-30T13:50:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e7701e0:::benchmark_data_smithi043_119832_object48188:head# 2024-03-30T13:50:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e78ce51:::benchmark_data_smithi043_119832_object31046:head# 2024-03-30T13:50:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e7a0bc0:::benchmark_data_smithi043_119832_object56436:head# 2024-03-30T13:50:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e7a488b:::benchmark_data_smithi043_119832_object12071:head# 2024-03-30T13:50:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e7ab437:::benchmark_data_smithi043_119832_object64393:head# 2024-03-30T13:50:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e7b1d5f:::benchmark_data_smithi043_119832_object29962:head# 2024-03-30T13:50:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e7b6269:::benchmark_data_smithi043_119832_object86342:head# 2024-03-30T13:50:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e7c5ba2:::benchmark_data_smithi043_119832_object40054:head# 2024-03-30T13:50:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e7f5e76:::benchmark_data_smithi043_119832_object54714:head# 2024-03-30T13:50:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e80158d:::benchmark_data_smithi043_119832_object4495:head# 2024-03-30T13:50:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e80add2:::benchmark_data_smithi043_119832_object73745:head# 2024-03-30T13:50:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e80e1d6:::benchmark_data_smithi043_119832_object87186:head# 2024-03-30T13:50:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e8129a8:::benchmark_data_smithi043_119832_object26995:head# 2024-03-30T13:50:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e815182:::benchmark_data_smithi043_119832_object78118:head# 2024-03-30T13:50:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e84733c:::benchmark_data_smithi043_119832_object88430:head# 2024-03-30T13:50:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e849338:::benchmark_data_smithi043_119832_object74188:head# 2024-03-30T13:50:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e84abed:::benchmark_data_smithi043_119832_object32146:head# 2024-03-30T13:50:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e855489:::benchmark_data_smithi043_119832_object36761:head# 2024-03-30T13:50:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e862b73:::benchmark_data_smithi043_119832_object36521:head# 2024-03-30T13:50:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e864e26:::benchmark_data_smithi043_119832_object67415:head# 2024-03-30T13:50:34.482 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e86ccc5:::benchmark_data_smithi043_119832_object30746:head# 2024-03-30T13:50:34.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e8704a3:::benchmark_data_smithi043_119832_object14180:head# 2024-03-30T13:50:34.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e88152f:::benchmark_data_smithi043_119832_object35552:head# 2024-03-30T13:50:34.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e882512:::benchmark_data_smithi043_119832_object52274:head# 2024-03-30T13:50:34.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e8837ea:::benchmark_data_smithi043_119832_object3793:head# 2024-03-30T13:50:34.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e89301c:::benchmark_data_smithi043_119832_object61643:head# 2024-03-30T13:50:34.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e89b271:::benchmark_data_smithi043_119832_object18481:head# 2024-03-30T13:50:34.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e89df90:::benchmark_data_smithi043_119832_object27556:head# 2024-03-30T13:50:34.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e8a1b6f:::benchmark_data_smithi043_119832_object27440:head# 2024-03-30T13:50:34.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e8a2064:::benchmark_data_smithi043_119832_object9590:head# 2024-03-30T13:50:34.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e8cd4b1:::benchmark_data_smithi043_119832_object58317:head# 2024-03-30T13:50:34.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e8d8b09:::benchmark_data_smithi043_119832_object26464:head# 2024-03-30T13:50:34.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e8efb42:::benchmark_data_smithi043_119832_object32135:head# 2024-03-30T13:50:34.483 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e8fa29f:::benchmark_data_smithi043_119832_object43530:head# 2024-03-30T13:50:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e8fb446:::benchmark_data_smithi043_119832_object19612:head# 2024-03-30T13:50:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e919669:::benchmark_data_smithi043_119832_object9287:head# 2024-03-30T13:50:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e92f307:::benchmark_data_smithi043_119832_object57150:head# 2024-03-30T13:50:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e93765b:::benchmark_data_smithi043_119832_object41706:head# 2024-03-30T13:50:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e93bda1:::benchmark_data_smithi043_119832_object70677:head# 2024-03-30T13:50:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e94bdb0:::benchmark_data_smithi043_119832_object82251:head# 2024-03-30T13:50:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e955002:::benchmark_data_smithi043_119832_object61770:head# 2024-03-30T13:50:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e9598de:::benchmark_data_smithi043_119832_object78947:head# 2024-03-30T13:50:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e9669d2:::benchmark_data_smithi043_119832_object39959:head# 2024-03-30T13:50:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e96cadd:::benchmark_data_smithi043_119832_object66811:head# 2024-03-30T13:50:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e97bbec:::benchmark_data_smithi043_119832_object30965:head# 2024-03-30T13:50:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e985363:::benchmark_data_smithi043_119832_object22995:head# 2024-03-30T13:50:34.485 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e986747:::benchmark_data_smithi043_119832_object26731:head# 2024-03-30T13:50:34.485 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e99ebe9:::benchmark_data_smithi043_119832_object31552:head# 2024-03-30T13:50:34.485 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e9c7de2:::benchmark_data_smithi043_119832_object38428:head# 2024-03-30T13:50:34.485 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e9cc098:::benchmark_data_smithi043_119832_object47135:head# 2024-03-30T13:50:34.485 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e9d103d:::benchmark_data_smithi043_119832_object3601:head# 2024-03-30T13:50:34.485 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e9d613a:::benchmark_data_smithi043_119832_object35398:head# 2024-03-30T13:50:34.485 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e9defb5:::benchmark_data_smithi043_119832_object15804:head# 2024-03-30T13:50:34.485 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e9ed10e:::benchmark_data_smithi043_119832_object52208:head# 2024-03-30T13:50:34.485 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7e9f19fc:::benchmark_data_smithi043_119832_object15660:head# 2024-03-30T13:50:34.485 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ea2bc5d:::benchmark_data_smithi043_119832_object9551:head# 2024-03-30T13:50:34.485 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ea40070:::benchmark_data_smithi043_119832_object10238:head# 2024-03-30T13:50:34.486 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ea47313:::benchmark_data_smithi043_119832_object63159:head# 2024-03-30T13:50:34.486 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ea488ed:::benchmark_data_smithi043_119832_object84669:head# 2024-03-30T13:50:34.486 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ea4d62c:::benchmark_data_smithi043_119832_object83977:head# 2024-03-30T13:50:34.486 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ea5b573:::benchmark_data_smithi043_119832_object48435:head# 2024-03-30T13:50:34.486 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ea6289b:::benchmark_data_smithi043_119832_object32229:head# 2024-03-30T13:50:34.486 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ea6408f:::benchmark_data_smithi043_119832_object46536:head# 2024-03-30T13:50:34.486 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ea68c09:::benchmark_data_smithi043_119832_object26381:head# 2024-03-30T13:50:34.486 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ea6c26f:::benchmark_data_smithi043_119832_object28870:head# 2024-03-30T13:50:34.486 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ea7027c:::benchmark_data_smithi043_119832_object62001:head# 2024-03-30T13:50:34.486 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ea74d88:::benchmark_data_smithi043_119832_object87046:head# 2024-03-30T13:50:34.486 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ea7e0a2:::benchmark_data_smithi043_119832_object85590:head# 2024-03-30T13:50:34.486 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ea7ea13:::benchmark_data_smithi043_119832_object15277:head# 2024-03-30T13:50:34.487 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ea80cd6:::benchmark_data_smithi043_119832_object9856:head# 2024-03-30T13:50:34.487 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ea9665c:::benchmark_data_smithi043_119832_object10047:head# 2024-03-30T13:50:34.487 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eaaa109:::benchmark_data_smithi043_119832_object31342:head# 2024-03-30T13:50:34.487 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eabe2d0:::benchmark_data_smithi043_119832_object72176:head# 2024-03-30T13:50:34.487 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eac6f2d:::benchmark_data_smithi043_119832_object15503:head# 2024-03-30T13:50:34.487 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ead9602:::benchmark_data_smithi043_119832_object83151:head# 2024-03-30T13:50:34.487 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eadc5e8:::benchmark_data_smithi043_119832_object81546:head# 2024-03-30T13:50:34.487 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eaee4cc:::benchmark_data_smithi043_119832_object55151:head# 2024-03-30T13:50:34.487 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eafbc03:::benchmark_data_smithi043_119832_object33895:head# 2024-03-30T13:50:34.487 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eb00248:::benchmark_data_smithi043_119832_object26047:head# 2024-03-30T13:50:34.487 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eb0679c:::benchmark_data_smithi043_119832_object15035:head# 2024-03-30T13:50:34.487 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eb0bc3e:::benchmark_data_smithi043_119832_object85731:head# 2024-03-30T13:50:34.487 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eb105a2:::benchmark_data_smithi043_119832_object78836:head# 2024-03-30T13:50:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eb20201:::benchmark_data_smithi043_119832_object16795:head# 2024-03-30T13:50:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eb2ee2f:::benchmark_data_smithi043_119832_object7408:head# 2024-03-30T13:50:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eb34524:::benchmark_data_smithi043_119832_object22889:head# 2024-03-30T13:50:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eb40550:::benchmark_data_smithi043_119832_object74339:head# 2024-03-30T13:50:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eb4adc8:::benchmark_data_smithi043_119832_object69594:head# 2024-03-30T13:50:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eb4bcfe:::benchmark_data_smithi043_119832_object4957:head# 2024-03-30T13:50:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eb52d62:::benchmark_data_smithi043_119832_object70051:head# 2024-03-30T13:50:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eb5b19f:::benchmark_data_smithi043_119832_object79982:head# 2024-03-30T13:50:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eb659b8:::benchmark_data_smithi043_119832_object66788:head# 2024-03-30T13:50:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eb68e5b:::benchmark_data_smithi043_119832_object48432:head# 2024-03-30T13:50:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eb85bc2:::benchmark_data_smithi043_119832_object8121:head# 2024-03-30T13:50:34.488 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eb89104:::benchmark_data_smithi043_119832_object46190:head# 2024-03-30T13:50:34.489 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eb93da8:::benchmark_data_smithi043_119832_object84528:head# 2024-03-30T13:50:34.489 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eb97976:::benchmark_data_smithi043_119832_object46391:head# 2024-03-30T13:50:34.489 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eb9beb5:::benchmark_data_smithi043_119832_object21499:head# 2024-03-30T13:50:34.489 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eba5b3a:::benchmark_data_smithi043_119832_object33699:head# 2024-03-30T13:50:34.489 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eba688f:::benchmark_data_smithi043_119832_object61247:head# 2024-03-30T13:50:34.489 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ebb6b2b:::benchmark_data_smithi043_119832_object24851:head# 2024-03-30T13:50:34.489 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ebc2bec:::benchmark_data_smithi043_119832_object33577:head# 2024-03-30T13:50:34.489 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ebc5af9:::benchmark_data_smithi043_119832_object32465:head# 2024-03-30T13:50:34.489 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ebcabab:::benchmark_data_smithi043_119832_object78969:head# 2024-03-30T13:50:34.489 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ebdc825:::benchmark_data_smithi043_119832_object61910:head# 2024-03-30T13:50:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ebdd97a:::benchmark_data_smithi043_119832_object83618:head# 2024-03-30T13:50:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ebe14e0:::benchmark_data_smithi043_119832_object12165:head# 2024-03-30T13:50:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ebe3f37:::benchmark_data_smithi043_119832_object38294:head# 2024-03-30T13:50:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ebe706c:::benchmark_data_smithi043_119832_object13017:head# 2024-03-30T13:50:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ebec910:::benchmark_data_smithi043_119832_object17935:head# 2024-03-30T13:50:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ebf1795:::benchmark_data_smithi043_119832_object18160:head# 2024-03-30T13:50:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ebf18c3:::benchmark_data_smithi043_119832_object22861:head# 2024-03-30T13:50:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ebf6ee8:::benchmark_data_smithi043_119832_object70331:head# 2024-03-30T13:50:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ebfc569:::benchmark_data_smithi043_119832_object48634:head# 2024-03-30T13:50:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ebfcc5f:::benchmark_data_smithi043_119832_object22282:head# 2024-03-30T13:50:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ec0d73f:::benchmark_data_smithi043_119832_object88149:head# 2024-03-30T13:50:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ec1f1d5:::benchmark_data_smithi043_119832_object49336:head# 2024-03-30T13:50:34.491 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ec293ff:::benchmark_data_smithi043_119832_object22216:head# 2024-03-30T13:50:34.491 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ec3149a:::benchmark_data_smithi043_119832_object56233:head# 2024-03-30T13:50:34.491 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ec397ec:::benchmark_data_smithi043_119832_object7202:head# 2024-03-30T13:50:34.491 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ec3fa31:::benchmark_data_smithi043_119832_object84954:head# 2024-03-30T13:50:34.491 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ec4b526:::benchmark_data_smithi043_119832_object86051:head# 2024-03-30T13:50:34.491 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ec5f8b0:::benchmark_data_smithi043_119832_object65685:head# 2024-03-30T13:50:34.491 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ec5ff88:::benchmark_data_smithi043_119832_object19798:head# 2024-03-30T13:50:34.491 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ec72c27:::benchmark_data_smithi043_119832_object59732:head# 2024-03-30T13:50:34.491 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ec7357e:::benchmark_data_smithi043_119832_object73794:head# 2024-03-30T13:50:34.491 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ec751fa:::benchmark_data_smithi043_119832_object63323:head# 2024-03-30T13:50:34.492 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ec77ce5:::benchmark_data_smithi043_119832_object15496:head# 2024-03-30T13:50:34.492 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ec77dcc:::benchmark_data_smithi043_119832_object51018:head# 2024-03-30T13:50:34.492 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ec9617f:::benchmark_data_smithi043_119832_object56515:head# 2024-03-30T13:50:34.492 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eca6de4:::benchmark_data_smithi043_119832_object32633:head# 2024-03-30T13:50:34.492 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ecafda7:::benchmark_data_smithi043_119832_object49152:head# 2024-03-30T13:50:34.492 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ecbb3d0:::benchmark_data_smithi043_119832_object21843:head# 2024-03-30T13:50:34.492 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ecbff9b:::benchmark_data_smithi043_119832_object25194:head# 2024-03-30T13:50:34.492 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ecca1d6:::benchmark_data_smithi043_119832_object67363:head# 2024-03-30T13:50:34.492 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ecd77c1:::benchmark_data_smithi043_119832_object73607:head# 2024-03-30T13:50:34.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ecde1f9:::benchmark_data_smithi043_119832_object54065:head# 2024-03-30T13:50:34.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ece2ec0:::benchmark_data_smithi043_119832_object6822:head# 2024-03-30T13:50:34.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ece5d4f:::benchmark_data_smithi043_119832_object73819:head# 2024-03-30T13:50:34.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ece65d4:::benchmark_data_smithi043_119832_object88287:head# 2024-03-30T13:50:34.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ecf9ed0:::benchmark_data_smithi043_119832_object32248:head# 2024-03-30T13:50:34.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ed09b0a:::benchmark_data_smithi043_119832_object69363:head# 2024-03-30T13:50:34.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ed0e19c:::benchmark_data_smithi043_119832_object50780:head# 2024-03-30T13:50:34.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ed1109a:::benchmark_data_smithi043_119832_object8137:head# 2024-03-30T13:50:34.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ed1dfb2:::benchmark_data_smithi043_119832_object6282:head# 2024-03-30T13:50:34.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ed1e1e4:::benchmark_data_smithi043_119832_object5471:head# 2024-03-30T13:50:34.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ed1f626:::benchmark_data_smithi043_119832_object4339:head# 2024-03-30T13:50:34.493 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ed41569:::benchmark_data_smithi043_119832_object39212:head# 2024-03-30T13:50:34.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ed42bed:::benchmark_data_smithi043_119832_object4734:head# 2024-03-30T13:50:34.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ed4b3c5:::benchmark_data_smithi043_119832_object22672:head# 2024-03-30T13:50:34.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ed4bd23:::benchmark_data_smithi043_119832_object88320:head# 2024-03-30T13:50:34.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ed66b19:::benchmark_data_smithi043_119832_object12957:head# 2024-03-30T13:50:34.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ed716d3:::benchmark_data_smithi043_119832_object61830:head# 2024-03-30T13:50:34.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eda912a:::benchmark_data_smithi043_119832_object5875:head# 2024-03-30T13:50:34.494 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7edb95de:::benchmark_data_smithi043_119832_object19215:head# 2024-03-30T13:50:34.495 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eddd353:::benchmark_data_smithi043_119832_object78022:head# 2024-03-30T13:50:34.495 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ede0d96:::benchmark_data_smithi043_119832_object30377:head# 2024-03-30T13:50:34.495 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ede122d:::benchmark_data_smithi043_119832_object18811:head# 2024-03-30T13:50:34.495 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ee06f7e:::benchmark_data_smithi043_119832_object31353:head# 2024-03-30T13:50:34.495 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ee19b4f:::benchmark_data_smithi043_119832_object337:head# 2024-03-30T13:50:34.495 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ee1dbe9:::benchmark_data_smithi043_119832_object66435:head# 2024-03-30T13:50:34.495 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ee57d9d:::benchmark_data_smithi043_119832_object66965:head# 2024-03-30T13:50:34.495 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ee637eb:::benchmark_data_smithi043_119832_object83592:head# 2024-03-30T13:50:34.495 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ee724cf:::benchmark_data_smithi043_119832_object67361:head# 2024-03-30T13:50:34.495 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ee83475:::benchmark_data_smithi043_119832_object18095:head# 2024-03-30T13:50:34.495 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ee8bdb3:::benchmark_data_smithi043_119832_object45060:head# 2024-03-30T13:50:34.495 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ee8cbe1:::benchmark_data_smithi043_119832_object68979:head# 2024-03-30T13:50:34.496 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eea1a79:::benchmark_data_smithi043_119832_object58589:head# 2024-03-30T13:50:34.496 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eea1c52:::benchmark_data_smithi043_119832_object45210:head# 2024-03-30T13:50:34.496 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eeabed5:::benchmark_data_smithi043_119832_object24999:head# 2024-03-30T13:50:34.496 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eeb8261:::benchmark_data_smithi043_119832_object34704:head# 2024-03-30T13:50:34.496 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eed281b:::benchmark_data_smithi043_119832_object2825:head# 2024-03-30T13:50:34.496 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eed2a0a:::benchmark_data_smithi043_119832_object80380:head# 2024-03-30T13:50:34.496 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eedf82c:::benchmark_data_smithi043_119832_object63982:head# 2024-03-30T13:50:34.496 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eee8338:::benchmark_data_smithi043_119832_object26908:head# 2024-03-30T13:50:34.497 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eefef26:::benchmark_data_smithi043_119832_object70250:head# 2024-03-30T13:50:34.497 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ef1c22d:::benchmark_data_smithi043_119832_object6914:head# 2024-03-30T13:50:34.497 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ef27761:::benchmark_data_smithi043_119832_object8557:head# 2024-03-30T13:50:34.497 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ef2d9f7:::benchmark_data_smithi043_119832_object66707:head# 2024-03-30T13:50:34.497 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ef45a81:::benchmark_data_smithi043_119832_object35376:head# 2024-03-30T13:50:34.497 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ef4b832:::benchmark_data_smithi043_119832_object1279:head# 2024-03-30T13:50:34.497 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ef54ee8:::benchmark_data_smithi043_119832_object47240:head# 2024-03-30T13:50:34.497 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ef581e1:::benchmark_data_smithi043_119832_object22681:head# 2024-03-30T13:50:34.497 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ef5dceb:::benchmark_data_smithi043_119832_object87992:head# 2024-03-30T13:50:34.497 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ef636c8:::benchmark_data_smithi043_119832_object40939:head# 2024-03-30T13:50:34.498 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ef67cd6:::benchmark_data_smithi043_119832_object88188:head# 2024-03-30T13:50:34.498 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ef6b3da:::benchmark_data_smithi043_119832_object25822:head# 2024-03-30T13:50:34.498 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ef74732:::benchmark_data_smithi043_119832_object74262:head# 2024-03-30T13:50:34.498 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ef83d87:::benchmark_data_smithi043_119832_object27071:head# 2024-03-30T13:50:34.498 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ef9961c:::benchmark_data_smithi043_119832_object76202:head# 2024-03-30T13:50:34.498 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7efa2645:::benchmark_data_smithi043_119832_object46228:head# 2024-03-30T13:50:34.498 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7efa3533:::benchmark_data_smithi043_119832_object40752:head# 2024-03-30T13:50:34.498 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7efa994e:::benchmark_data_smithi043_119832_object31574:head# 2024-03-30T13:50:34.498 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7efb8c12:::benchmark_data_smithi043_119832_object16773:head# 2024-03-30T13:50:34.499 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7efbb9bd:::benchmark_data_smithi043_119832_object4586:head# 2024-03-30T13:50:34.499 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7efd9c3f:::benchmark_data_smithi043_119832_object71963:head# 2024-03-30T13:50:34.499 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7efe62e6:::benchmark_data_smithi043_119832_object11982:head# 2024-03-30T13:50:34.499 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7eff1d63:::benchmark_data_smithi043_119832_object78837:head# 2024-03-30T13:50:34.499 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7efffc37:::benchmark_data_smithi043_119832_object83000:head# 2024-03-30T13:50:34.499 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f0239c7:::benchmark_data_smithi043_119832_object79526:head# 2024-03-30T13:50:34.499 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f028188:::benchmark_data_smithi043_119832_object52231:head# 2024-03-30T13:50:34.499 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f02b80d:::benchmark_data_smithi043_119832_object77144:head# 2024-03-30T13:50:34.499 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f043b09:::benchmark_data_smithi043_119832_object12755:head# 2024-03-30T13:50:34.499 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f046d65:::benchmark_data_smithi043_119832_object1061:head# 2024-03-30T13:50:34.500 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f048bf6:::benchmark_data_smithi043_119832_object48739:head# 2024-03-30T13:50:34.500 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f056f71:::benchmark_data_smithi043_119832_object46185:head# 2024-03-30T13:50:34.500 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f0597ab:::benchmark_data_smithi043_119832_object65615:head# 2024-03-30T13:50:34.500 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f05993e:::benchmark_data_smithi043_119832_object72067:head# 2024-03-30T13:50:34.500 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f05e151:::benchmark_data_smithi043_119832_object43444:head# 2024-03-30T13:50:34.500 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f068b74:::benchmark_data_smithi043_119832_object20110:head# 2024-03-30T13:50:34.500 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f0856d5:::benchmark_data_smithi043_119832_object63481:head# 2024-03-30T13:50:34.500 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f092b49:::benchmark_data_smithi043_119832_object79189:head# 2024-03-30T13:50:34.501 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f0a17d1:::benchmark_data_smithi043_119832_object70743:head# 2024-03-30T13:50:34.501 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f0abad2:::benchmark_data_smithi043_119832_object39271:head# 2024-03-30T13:50:34.501 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f0bd711:::benchmark_data_smithi043_119832_object37505:head# 2024-03-30T13:50:34.501 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f0bee31:::benchmark_data_smithi043_119832_object43037:head# 2024-03-30T13:50:34.501 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f0cb477:::benchmark_data_smithi043_119832_object39072:head# 2024-03-30T13:50:34.501 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f0cb7a9:::benchmark_data_smithi043_119832_object48024:head# 2024-03-30T13:50:34.501 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f0ce97d:::benchmark_data_smithi043_119832_object65523:head# 2024-03-30T13:50:34.501 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f0dc52a:::benchmark_data_smithi043_119832_object81157:head# 2024-03-30T13:50:34.502 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f0f19cc:::benchmark_data_smithi043_119832_object48392:head# 2024-03-30T13:50:34.502 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f11bdc9:::benchmark_data_smithi043_119832_object80405:head# 2024-03-30T13:50:34.502 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f11ce3a:::benchmark_data_smithi043_119832_object36979:head# 2024-03-30T13:50:34.502 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f12209d:::benchmark_data_smithi043_119832_object50524:head# 2024-03-30T13:50:34.502 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f14c170:::benchmark_data_smithi043_119832_object1539:head# 2024-03-30T13:50:34.502 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f16aba8:::benchmark_data_smithi043_119832_object63722:head# 2024-03-30T13:50:34.502 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f18bfb6:::benchmark_data_smithi043_119832_object55018:head# 2024-03-30T13:50:34.502 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f196a38:::benchmark_data_smithi043_119832_object52850:head# 2024-03-30T13:50:34.503 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f19acac:::benchmark_data_smithi043_119832_object9934:head# 2024-03-30T13:50:34.503 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f19cf5d:::benchmark_data_smithi043_119832_object70411:head# 2024-03-30T13:50:34.503 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f19f23b:::benchmark_data_smithi043_119832_object73139:head# 2024-03-30T13:50:34.503 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f1a92da:::benchmark_data_smithi043_119832_object47710:head# 2024-03-30T13:50:34.503 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f1abc01:::benchmark_data_smithi043_119832_object40876:head# 2024-03-30T13:50:34.503 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f1b3588:::benchmark_data_smithi043_119832_object35905:head# 2024-03-30T13:50:34.503 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f1c6b92:::benchmark_data_smithi043_119832_object63626:head# 2024-03-30T13:50:34.504 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f1c83b8:::benchmark_data_smithi043_119832_object5981:head# 2024-03-30T13:50:34.504 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f1cc229:::benchmark_data_smithi043_119832_object39404:head# 2024-03-30T13:50:34.504 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f1d1c9e:::benchmark_data_smithi043_119832_object44572:head# 2024-03-30T13:50:34.504 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f1d5672:::benchmark_data_smithi043_119832_object42518:head# 2024-03-30T13:50:34.504 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f1e2bc2:::benchmark_data_smithi043_119832_object7518:head# 2024-03-30T13:50:34.504 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f1eea10:::benchmark_data_smithi043_119832_object23997:head# 2024-03-30T13:50:34.504 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f1f0eb0:::benchmark_data_smithi043_119832_object71708:head# 2024-03-30T13:50:34.505 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f1f3808:::benchmark_data_smithi043_119832_object31216:head# 2024-03-30T13:50:34.505 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f1fa8d2:::benchmark_data_smithi043_119832_object56514:head# 2024-03-30T13:50:34.505 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f1fed64:::benchmark_data_smithi043_119832_object18987:head# 2024-03-30T13:50:34.505 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f213f01:::benchmark_data_smithi043_119832_object24195:head# 2024-03-30T13:50:34.505 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f21fe28:::benchmark_data_smithi043_119832_object86679:head# 2024-03-30T13:50:34.505 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f2400e3:::benchmark_data_smithi043_119832_object73963:head# 2024-03-30T13:50:34.505 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f26828c:::benchmark_data_smithi043_119832_object38748:head# 2024-03-30T13:50:34.505 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f274acd:::benchmark_data_smithi043_119832_object24605:head# 2024-03-30T13:50:34.505 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f275e2f:::benchmark_data_smithi043_119832_object8953:head# 2024-03-30T13:50:34.505 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f279b6f:::benchmark_data_smithi043_119832_object68402:head# 2024-03-30T13:50:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f27bdd7:::benchmark_data_smithi043_119832_object49499:head# 2024-03-30T13:50:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f281ed7:::benchmark_data_smithi043_119832_object53426:head# 2024-03-30T13:50:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f2843dc:::benchmark_data_smithi043_119832_object60625:head# 2024-03-30T13:50:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f28bf0e:::benchmark_data_smithi043_119832_object4892:head# 2024-03-30T13:50:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f28d3a6:::benchmark_data_smithi043_119832_object26297:head# 2024-03-30T13:50:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f28e369:::benchmark_data_smithi043_119832_object2612:head# 2024-03-30T13:50:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f29b501:::benchmark_data_smithi043_119832_object59657:head# 2024-03-30T13:50:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f29cf9e:::benchmark_data_smithi043_119832_object19467:head# 2024-03-30T13:50:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f2a5351:::benchmark_data_smithi043_119832_object28478:head# 2024-03-30T13:50:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f2bce33:::benchmark_data_smithi043_119832_object51301:head# 2024-03-30T13:50:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f2c94a9:::benchmark_data_smithi043_119832_object37936:head# 2024-03-30T13:50:34.507 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f2cbe63:::benchmark_data_smithi043_119832_object38153:head# 2024-03-30T13:50:34.507 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f2d02e8:::benchmark_data_smithi043_119832_object55115:head# 2024-03-30T13:50:34.507 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f2e2738:::benchmark_data_smithi043_119832_object52065:head# 2024-03-30T13:50:34.507 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f2ec152:::benchmark_data_smithi043_119832_object54296:head# 2024-03-30T13:50:34.507 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f2ee9f6:::benchmark_data_smithi043_119832_object69686:head# 2024-03-30T13:50:34.507 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f2f6fa1:::benchmark_data_smithi043_119832_object23713:head# 2024-03-30T13:50:34.507 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f303f25:::benchmark_data_smithi043_119832_object34245:head# 2024-03-30T13:50:34.507 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f3051b1:::benchmark_data_smithi043_119832_object12772:head# 2024-03-30T13:50:34.507 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f30680b:::benchmark_data_smithi043_119832_object24259:head# 2024-03-30T13:50:34.507 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f316b9f:::benchmark_data_smithi043_119832_object50455:head# 2024-03-30T13:50:34.507 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f31e9fa:::benchmark_data_smithi043_119832_object10262:head# 2024-03-30T13:50:34.507 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f335c87:::benchmark_data_smithi043_119832_object39701:head# 2024-03-30T13:50:34.508 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f336143:::benchmark_data_smithi043_119832_object26709:head# 2024-03-30T13:50:34.508 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f33692f:::benchmark_data_smithi043_119832_object65856:head# 2024-03-30T13:50:34.508 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f338c58:::benchmark_data_smithi043_119832_object106:head# 2024-03-30T13:50:34.508 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f33c7ee:::benchmark_data_smithi043_119832_object79190:head# 2024-03-30T13:50:34.508 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f34ec56:::benchmark_data_smithi043_119832_object85500:head# 2024-03-30T13:50:34.508 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f371887:::benchmark_data_smithi043_119832_object85185:head# 2024-03-30T13:50:34.508 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f372900:::benchmark_data_smithi043_119832_object3294:head# 2024-03-30T13:50:34.508 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f37482a:::benchmark_data_smithi043_119832_object52390:head# 2024-03-30T13:50:34.508 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f37851d:::benchmark_data_smithi043_119832_object57066:head# 2024-03-30T13:50:34.508 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f37fa85:::benchmark_data_smithi043_119832_object37185:head# 2024-03-30T13:50:34.508 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f387980:::benchmark_data_smithi043_119832_object64280:head# 2024-03-30T13:50:34.508 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f3981b9:::benchmark_data_smithi043_119832_object28504:head# 2024-03-30T13:50:34.508 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f3a5044:::benchmark_data_smithi043_119832_object52576:head# 2024-03-30T13:50:34.509 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f3ab1cf:::benchmark_data_smithi043_119832_object41066:head# 2024-03-30T13:50:34.509 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f3b6963:::benchmark_data_smithi043_119832_object73270:head# 2024-03-30T13:50:34.509 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f3c4998:::benchmark_data_smithi043_119832_object68083:head# 2024-03-30T13:50:34.509 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f3d67d7:::benchmark_data_smithi043_119832_object82509:head# 2024-03-30T13:50:34.509 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f40492c:::benchmark_data_smithi043_119832_object48244:head# 2024-03-30T13:50:34.509 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f405fd3:::benchmark_data_smithi043_119832_object35543:head# 2024-03-30T13:50:34.509 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f4060ca:::benchmark_data_smithi043_119832_object249:head# 2024-03-30T13:50:34.509 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f408acb:::benchmark_data_smithi043_119832_object57065:head# 2024-03-30T13:50:34.509 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f40bc4f:::benchmark_data_smithi043_119832_object27908:head# 2024-03-30T13:50:34.509 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f40f2d9:::benchmark_data_smithi043_119832_object35643:head# 2024-03-30T13:50:34.509 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f414536:::benchmark_data_smithi043_119832_object20943:head# 2024-03-30T13:50:34.509 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f42bc18:::benchmark_data_smithi043_119832_object32020:head# 2024-03-30T13:50:34.509 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f430a76:::benchmark_data_smithi043_119832_object83162:head# 2024-03-30T13:50:34.510 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f43829d:::benchmark_data_smithi043_119832_object86170:head# 2024-03-30T13:50:34.510 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f43d41e:::benchmark_data_smithi043_119832_object15349:head# 2024-03-30T13:50:34.510 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f4424f9:::benchmark_data_smithi043_119832_object84175:head# 2024-03-30T13:50:34.510 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f447587:::benchmark_data_smithi043_119832_object81156:head# 2024-03-30T13:50:34.510 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f44fc5c:::benchmark_data_smithi043_119832_object33150:head# 2024-03-30T13:50:34.510 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f4595fe:::benchmark_data_smithi043_119832_object77328:head# 2024-03-30T13:50:34.510 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f45a65a:::benchmark_data_smithi043_119832_object53390:head# 2024-03-30T13:50:34.510 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f46184e:::benchmark_data_smithi043_119832_object75366:head# 2024-03-30T13:50:34.510 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f46c0a0:::benchmark_data_smithi043_119832_object53363:head# 2024-03-30T13:50:34.510 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f474a59:::benchmark_data_smithi043_119832_object18178:head# 2024-03-30T13:50:34.510 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f4845af:::benchmark_data_smithi043_119832_object81696:head# 2024-03-30T13:50:34.511 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f48b4b9:::benchmark_data_smithi043_119832_object7320:head# 2024-03-30T13:50:34.511 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f48c5f7:::benchmark_data_smithi043_119832_object16793:head# 2024-03-30T13:50:34.511 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f49937e:::benchmark_data_smithi043_119832_object40839:head# 2024-03-30T13:50:34.511 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f49c277:::benchmark_data_smithi043_119832_object85241:head# 2024-03-30T13:50:34.511 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f49c522:::benchmark_data_smithi043_119832_object79086:head# 2024-03-30T13:50:34.511 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f4a2e65:::benchmark_data_smithi043_119832_object75226:head# 2024-03-30T13:50:34.511 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f4a900b:::benchmark_data_smithi043_119832_object11395:head# 2024-03-30T13:50:34.511 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f4b556f:::benchmark_data_smithi043_119832_object82130:head# 2024-03-30T13:50:34.511 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f4b70ea:::benchmark_data_smithi043_119832_object80714:head# 2024-03-30T13:50:34.511 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f4b728b:::benchmark_data_smithi043_119832_object65116:head# 2024-03-30T13:50:34.511 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f4bb8e3:::benchmark_data_smithi043_119832_object71081:head# 2024-03-30T13:50:34.511 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f4bf923:::benchmark_data_smithi043_119832_object45812:head# 2024-03-30T13:50:34.511 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f4c46a9:::benchmark_data_smithi043_119832_object61589:head# 2024-03-30T13:50:34.512 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f4c87ff:::benchmark_data_smithi043_119832_object42888:head# 2024-03-30T13:50:34.512 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f4caf50:::benchmark_data_smithi043_119832_object8978:head# 2024-03-30T13:50:34.512 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f4d0fae:::benchmark_data_smithi043_119832_object49906:head# 2024-03-30T13:50:34.512 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f4d33e3:::benchmark_data_smithi043_119832_object100:head# 2024-03-30T13:50:34.512 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f4de78d:::benchmark_data_smithi043_119832_object24815:head# 2024-03-30T13:50:34.512 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f4fdd51:::benchmark_data_smithi043_119832_object11593:head# 2024-03-30T13:50:34.512 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f4ffe66:::benchmark_data_smithi043_119832_object80368:head# 2024-03-30T13:50:34.512 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f50fb0e:::benchmark_data_smithi043_119832_object85690:head# 2024-03-30T13:50:34.512 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f51c4f6:::benchmark_data_smithi043_119832_object80545:head# 2024-03-30T13:50:34.512 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f52ed37:::benchmark_data_smithi043_119832_object77090:head# 2024-03-30T13:50:34.512 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f5324ad:::benchmark_data_smithi043_119832_object2764:head# 2024-03-30T13:50:34.513 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f5417b6:::benchmark_data_smithi043_119832_object24651:head# 2024-03-30T13:50:34.513 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f566775:::benchmark_data_smithi043_119832_object64857:head# 2024-03-30T13:50:34.513 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f56b124:::benchmark_data_smithi043_119832_object21400:head# 2024-03-30T13:50:34.513 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f574290:::benchmark_data_smithi043_119832_object13550:head# 2024-03-30T13:50:34.513 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f57df27:::benchmark_data_smithi043_119832_object5730:head# 2024-03-30T13:50:34.513 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f58c764:::benchmark_data_smithi043_119832_object74993:head# 2024-03-30T13:50:34.513 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f58d976:::benchmark_data_smithi043_119832_object21929:head# 2024-03-30T13:50:34.513 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f59f588:::benchmark_data_smithi043_119832_object58193:head# 2024-03-30T13:50:34.513 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f5a0652:::benchmark_data_smithi043_119832_object56354:head# 2024-03-30T13:50:34.513 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f5a39e6:::benchmark_data_smithi043_119832_object54160:head# 2024-03-30T13:50:34.513 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f5ad552:::benchmark_data_smithi043_119832_object57672:head# 2024-03-30T13:50:34.513 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f5b8b5e:::benchmark_data_smithi043_119832_object25123:head# 2024-03-30T13:50:34.514 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f5ba9c3:::benchmark_data_smithi043_119832_object26099:head# 2024-03-30T13:50:34.514 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f5befc8:::benchmark_data_smithi043_119832_object13154:head# 2024-03-30T13:50:34.514 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f5c010d:::benchmark_data_smithi043_119832_object76368:head# 2024-03-30T13:50:34.514 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f5db587:::benchmark_data_smithi043_119832_object32913:head# 2024-03-30T13:50:34.514 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f5e3511:::benchmark_data_smithi043_119832_object68295:head# 2024-03-30T13:50:34.514 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f5e75e3:::benchmark_data_smithi043_119832_object71867:head# 2024-03-30T13:50:34.514 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f5e7da9:::benchmark_data_smithi043_119832_object56122:head# 2024-03-30T13:50:34.514 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f5e8833:::benchmark_data_smithi043_119832_object24359:head# 2024-03-30T13:50:34.514 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f5ee467:::benchmark_data_smithi043_119832_object31689:head# 2024-03-30T13:50:34.514 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f5f70ef:::benchmark_data_smithi043_119832_object44429:head# 2024-03-30T13:50:34.514 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f60d3f5:::benchmark_data_smithi043_119832_object23919:head# 2024-03-30T13:50:34.514 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f61b40e:::benchmark_data_smithi043_119832_object46723:head# 2024-03-30T13:50:34.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f625325:::benchmark_data_smithi043_119832_object56203:head# 2024-03-30T13:50:34.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f6376b4:::benchmark_data_smithi043_119832_object65348:head# 2024-03-30T13:50:34.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f641230:::benchmark_data_smithi043_119832_object41578:head# 2024-03-30T13:50:34.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f6476dc:::benchmark_data_smithi043_119832_object785:head# 2024-03-30T13:50:34.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f6482eb:::benchmark_data_smithi043_119832_object21820:head# 2024-03-30T13:50:34.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f6574ce:::benchmark_data_smithi043_119832_object59497:head# 2024-03-30T13:50:34.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f6645ab:::benchmark_data_smithi043_119832_object66791:head# 2024-03-30T13:50:34.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f682197:::benchmark_data_smithi043_119832_object43994:head# 2024-03-30T13:50:34.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f68c434:::benchmark_data_smithi043_119832_object76872:head# 2024-03-30T13:50:34.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f691222:::benchmark_data_smithi043_119832_object2421:head# 2024-03-30T13:50:34.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f6a1b71:::benchmark_data_smithi043_119832_object63763:head# 2024-03-30T13:50:34.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f6aad42:::benchmark_data_smithi043_119832_object67511:head# 2024-03-30T13:50:34.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f6bfb2a:::benchmark_data_smithi043_119832_object61488:head# 2024-03-30T13:50:34.515 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f6c4632:::benchmark_data_smithi043_119832_object27813:head# 2024-03-30T13:50:34.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f6ce309:::benchmark_data_smithi043_119832_object24778:head# 2024-03-30T13:50:34.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f6da891:::benchmark_data_smithi043_119832_object80093:head# 2024-03-30T13:50:34.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f6dd18b:::benchmark_data_smithi043_119832_object80254:head# 2024-03-30T13:50:34.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f6ea7c0:::benchmark_data_smithi043_119832_object48401:head# 2024-03-30T13:50:34.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f6ebeef:::benchmark_data_smithi043_119832_object50082:head# 2024-03-30T13:50:34.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f6f0e2e:::benchmark_data_smithi043_119832_object26966:head# 2024-03-30T13:50:34.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f6f6999:::benchmark_data_smithi043_119832_object38712:head# 2024-03-30T13:50:34.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f6fe401:::benchmark_data_smithi043_119832_object36347:head# 2024-03-30T13:50:34.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f72a822:::benchmark_data_smithi043_119832_object31494:head# 2024-03-30T13:50:34.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f72efc9:::benchmark_data_smithi043_119832_object68935:head# 2024-03-30T13:50:34.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f72fbda:::benchmark_data_smithi043_119832_object70127:head# 2024-03-30T13:50:34.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f73322d:::benchmark_data_smithi043_119832_object33619:head# 2024-03-30T13:50:34.516 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f741329:::benchmark_data_smithi043_119832_object84551:head# 2024-03-30T13:50:34.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f758ea8:::benchmark_data_smithi043_119832_object63498:head# 2024-03-30T13:50:34.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f78569d:::benchmark_data_smithi043_119832_object55907:head# 2024-03-30T13:50:34.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f788ba1:::benchmark_data_smithi043_119832_object25182:head# 2024-03-30T13:50:34.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f792afd:::benchmark_data_smithi043_119832_object50190:head# 2024-03-30T13:50:34.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f7c03fc:::benchmark_data_smithi043_119832_object25052:head# 2024-03-30T13:50:34.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f7c9a09:::benchmark_data_smithi043_119832_object22667:head# 2024-03-30T13:50:34.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f7f06c1:::benchmark_data_smithi043_119832_object57129:head# 2024-03-30T13:50:34.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f802b0f:::benchmark_data_smithi043_119832_object5835:head# 2024-03-30T13:50:34.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f804daf:::benchmark_data_smithi043_119832_object17052:head# 2024-03-30T13:50:34.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f80e7dc:::benchmark_data_smithi043_119832_object54634:head# 2024-03-30T13:50:34.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f80ff64:::benchmark_data_smithi043_119832_object79024:head# 2024-03-30T13:50:34.517 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f818660:::benchmark_data_smithi043_119832_object26262:head# 2024-03-30T13:50:34.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f83feae:::benchmark_data_smithi043_119832_object44570:head# 2024-03-30T13:50:34.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f845bb5:::benchmark_data_smithi043_119832_object11819:head# 2024-03-30T13:50:34.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f848cf3:::benchmark_data_smithi043_119832_object81495:head# 2024-03-30T13:50:34.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f84df8d:::benchmark_data_smithi043_119832_object8010:head# 2024-03-30T13:50:34.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f8698d1:::benchmark_data_smithi043_119832_object42712:head# 2024-03-30T13:50:34.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f86ee0c:::benchmark_data_smithi043_119832_object28270:head# 2024-03-30T13:50:34.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f885536:::benchmark_data_smithi043_119832_object39606:head# 2024-03-30T13:50:34.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f885b25:::benchmark_data_smithi043_119832_object15274:head# 2024-03-30T13:50:34.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f895ab8:::benchmark_data_smithi043_119832_object6514:head# 2024-03-30T13:50:34.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f89aa75:::benchmark_data_smithi043_119832_object82126:head# 2024-03-30T13:50:34.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f8a47c8:::benchmark_data_smithi043_119832_object60885:head# 2024-03-30T13:50:34.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f8bfce9:::benchmark_data_smithi043_119832_object54620:head# 2024-03-30T13:50:34.518 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f8c5571:::benchmark_data_smithi043_119832_object16205:head# 2024-03-30T13:50:34.519 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f8c89a3:::benchmark_data_smithi043_119832_object73273:head# 2024-03-30T13:50:34.519 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f8d8a58:::benchmark_data_smithi043_119832_object48810:head# 2024-03-30T13:50:34.519 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f8e5f6d:::benchmark_data_smithi043_119832_object58119:head# 2024-03-30T13:50:34.519 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f8ea9c1:::benchmark_data_smithi043_119832_object49811:head# 2024-03-30T13:50:34.519 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f8ec9e6:::benchmark_data_smithi043_119832_object81635:head# 2024-03-30T13:50:34.519 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f8fa481:::benchmark_data_smithi043_119832_object4671:head# 2024-03-30T13:50:34.519 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f8fabe4:::benchmark_data_smithi043_119832_object58257:head# 2024-03-30T13:50:34.519 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f912092:::benchmark_data_smithi043_119832_object61327:head# 2024-03-30T13:50:34.519 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f9242f8:::benchmark_data_smithi043_119832_object6109:head# 2024-03-30T13:50:34.519 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f936238:::benchmark_data_smithi043_119832_object54004:head# 2024-03-30T13:50:34.519 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f93d410:::benchmark_data_smithi043_119832_object59057:head# 2024-03-30T13:50:34.519 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f9439ad:::benchmark_data_smithi043_119832_object58211:head# 2024-03-30T13:50:34.519 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f9446b9:::benchmark_data_smithi043_119832_object10429:head# 2024-03-30T13:50:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f94827a:::benchmark_data_smithi043_119832_object87655:head# 2024-03-30T13:50:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f950595:::benchmark_data_smithi043_119832_object18374:head# 2024-03-30T13:50:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f954685:::benchmark_data_smithi043_119832_object32425:head# 2024-03-30T13:50:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f957ab9:::benchmark_data_smithi043_119832_object17713:head# 2024-03-30T13:50:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f9584ff:::benchmark_data_smithi043_119832_object68969:head# 2024-03-30T13:50:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f95b118:::benchmark_data_smithi043_119832_object60232:head# 2024-03-30T13:50:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f966d22:::benchmark_data_smithi043_119832_object36930:head# 2024-03-30T13:50:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f96a716:::benchmark_data_smithi043_119832_object30175:head# 2024-03-30T13:50:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f979bc1:::benchmark_data_smithi043_119832_object71756:head# 2024-03-30T13:50:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f97a620:::benchmark_data_smithi043_119832_object8326:head# 2024-03-30T13:50:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f982d2b:::benchmark_data_smithi043_119832_object32657:head# 2024-03-30T13:50:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f9845ea:::benchmark_data_smithi043_119832_object81732:head# 2024-03-30T13:50:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f98b2bb:::benchmark_data_smithi043_119832_object23620:head# 2024-03-30T13:50:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f98c06b:::benchmark_data_smithi043_119832_object12106:head# 2024-03-30T13:50:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f995a21:::benchmark_data_smithi043_119832_object27932:head# 2024-03-30T13:50:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f997bdf:::benchmark_data_smithi043_119832_object20590:head# 2024-03-30T13:50:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f99aaaf:::benchmark_data_smithi043_119832_object71956:head# 2024-03-30T13:50:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f9a07e6:::benchmark_data_smithi043_119832_object31117:head# 2024-03-30T13:50:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f9a57f3:::benchmark_data_smithi043_119832_object4387:head# 2024-03-30T13:50:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f9bfdda:::benchmark_data_smithi043_119832_object32205:head# 2024-03-30T13:50:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f9c0aca:::benchmark_data_smithi043_119832_object60461:head# 2024-03-30T13:50:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f9d5bf7:::benchmark_data_smithi043_119832_object13495:head# 2024-03-30T13:50:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f9d5e89:::benchmark_data_smithi043_119832_object80087:head# 2024-03-30T13:50:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f9f8480:::benchmark_data_smithi043_119832_object15315:head# 2024-03-30T13:50:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7f9fd370:::benchmark_data_smithi043_119832_object23721:head# 2024-03-30T13:50:34.521 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fa09e59:::benchmark_data_smithi043_119832_object70632:head# 2024-03-30T13:50:34.522 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fa3a2bf:::benchmark_data_smithi043_119832_object51516:head# 2024-03-30T13:50:34.522 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fa60d26:::benchmark_data_smithi043_119832_object59373:head# 2024-03-30T13:50:34.522 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fa61604:::benchmark_data_smithi043_119832_object68901:head# 2024-03-30T13:50:34.522 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fa6ee67:::benchmark_data_smithi043_119832_object25848:head# 2024-03-30T13:50:34.522 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fa702d1:::benchmark_data_smithi043_119832_object57620:head# 2024-03-30T13:50:34.522 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fa742bc:::benchmark_data_smithi043_119832_object56656:head# 2024-03-30T13:50:34.522 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fa7c948:::benchmark_data_smithi043_119832_object81837:head# 2024-03-30T13:50:34.522 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fa9be6d:::benchmark_data_smithi043_119832_object74771:head# 2024-03-30T13:50:34.522 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fac7234:::benchmark_data_smithi043_119832_object87243:head# 2024-03-30T13:50:34.522 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7facf107:::benchmark_data_smithi043_119832_object21415:head# 2024-03-30T13:50:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fad2b4b:::benchmark_data_smithi043_119832_object53986:head# 2024-03-30T13:50:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fae13c9:::benchmark_data_smithi043_119832_object11410:head# 2024-03-30T13:50:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fae6204:::benchmark_data_smithi043_119832_object33195:head# 2024-03-30T13:50:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fae8215:::benchmark_data_smithi043_119832_object40127:head# 2024-03-30T13:50:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7faf7302:::benchmark_data_smithi043_119832_object78779:head# 2024-03-30T13:50:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fb02f60:::benchmark_data_smithi043_119832_object38961:head# 2024-03-30T13:50:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fb034a1:::benchmark_data_smithi043_119832_object80473:head# 2024-03-30T13:50:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fb0bf12:::benchmark_data_smithi043_119832_object10517:head# 2024-03-30T13:50:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fb0e64d:::benchmark_data_smithi043_119832_object24697:head# 2024-03-30T13:50:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fb34ef9:::benchmark_data_smithi043_119832_object37606:head# 2024-03-30T13:50:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fb37f10:::benchmark_data_smithi043_119832_object74704:head# 2024-03-30T13:50:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fb385ac:::benchmark_data_smithi043_119832_object13844:head# 2024-03-30T13:50:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fb3f820:::benchmark_data_smithi043_119832_object67777:head# 2024-03-30T13:50:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fb42e7d:::benchmark_data_smithi043_119832_object67621:head# 2024-03-30T13:50:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fb4bd49:::benchmark_data_smithi043_119832_object34237:head# 2024-03-30T13:50:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fb51911:::benchmark_data_smithi043_119832_object13048:head# 2024-03-30T13:50:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fb55044:::benchmark_data_smithi043_119832_object1805:head# 2024-03-30T13:50:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fb75641:::benchmark_data_smithi043_119832_object4784:head# 2024-03-30T13:50:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fb7f4c5:::benchmark_data_smithi043_119832_object23229:head# 2024-03-30T13:50:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fb8801c:::benchmark_data_smithi043_119832_object260:head# 2024-03-30T13:50:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fb934de:::benchmark_data_smithi043_119832_object55021:head# 2024-03-30T13:50:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fb9f599:::benchmark_data_smithi043_119832_object21314:head# 2024-03-30T13:50:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fb9fb3d:::benchmark_data_smithi043_119832_object56784:head# 2024-03-30T13:50:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fba0026:::benchmark_data_smithi043_119832_object30690:head# 2024-03-30T13:50:34.525 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fba6a7f:::benchmark_data_smithi043_119832_object43136:head# 2024-03-30T13:50:34.525 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fbaf62c:::benchmark_data_smithi043_119832_object41695:head# 2024-03-30T13:50:34.525 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fbbbe1b:::benchmark_data_smithi043_119832_object41933:head# 2024-03-30T13:50:34.525 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fbc4cc2:::benchmark_data_smithi043_119832_object59855:head# 2024-03-30T13:50:34.525 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fbc4d02:::benchmark_data_smithi043_119832_object75801:head# 2024-03-30T13:50:34.525 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fbc5230:::benchmark_data_smithi043_119832_object9601:head# 2024-03-30T13:50:34.525 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fbde701:::benchmark_data_smithi043_119832_object27265:head# 2024-03-30T13:50:34.525 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fbe2667:::benchmark_data_smithi043_119832_object49063:head# 2024-03-30T13:50:34.525 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fbe3c00:::benchmark_data_smithi043_119832_object68353:head# 2024-03-30T13:50:34.525 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fbe67b3:::benchmark_data_smithi043_119832_object41587:head# 2024-03-30T13:50:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fbf0eac:::benchmark_data_smithi043_119832_object76322:head# 2024-03-30T13:50:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fbf75b0:::benchmark_data_smithi043_119832_object15767:head# 2024-03-30T13:50:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fc14c20:::benchmark_data_smithi043_119832_object62497:head# 2024-03-30T13:50:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fc15035:::benchmark_data_smithi043_119832_object23054:head# 2024-03-30T13:50:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fc24dc6:::benchmark_data_smithi043_119832_object70019:head# 2024-03-30T13:50:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fc3e611:::benchmark_data_smithi043_119832_object50537:head# 2024-03-30T13:50:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fc44bf7:::benchmark_data_smithi043_119832_object85682:head# 2024-03-30T13:50:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fc4907d:::benchmark_data_smithi043_119832_object35668:head# 2024-03-30T13:50:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fc49dd1:::benchmark_data_smithi043_119832_object31979:head# 2024-03-30T13:50:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fc5ce96:::benchmark_data_smithi043_119832_object45676:head# 2024-03-30T13:50:34.527 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fc674b8:::benchmark_data_smithi043_119832_object4415:head# 2024-03-30T13:50:34.527 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fc70e34:::benchmark_data_smithi043_119832_object54596:head# 2024-03-30T13:50:34.527 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fc725ab:::benchmark_data_smithi043_119832_object74179:head# 2024-03-30T13:50:34.527 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fc77e0e:::benchmark_data_smithi043_119832_object63261:head# 2024-03-30T13:50:34.527 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fc85329:::benchmark_data_smithi043_119832_object74343:head# 2024-03-30T13:50:34.527 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fc93f08:::benchmark_data_smithi043_119832_object24606:head# 2024-03-30T13:50:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fc9ec67:::benchmark_data_smithi043_119832_object24233:head# 2024-03-30T13:50:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fca10c2:::benchmark_data_smithi043_119832_object74469:head# 2024-03-30T13:50:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fccaf26:::benchmark_data_smithi043_119832_object34388:head# 2024-03-30T13:50:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fcd12eb:::benchmark_data_smithi043_119832_object3829:head# 2024-03-30T13:50:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fcd9f52:::benchmark_data_smithi043_119832_object2534:head# 2024-03-30T13:50:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fcdbd49:::benchmark_data_smithi043_119832_object2967:head# 2024-03-30T13:50:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fcf1288:::benchmark_data_smithi043_119832_object65285:head# 2024-03-30T13:50:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fcf4bf7:::benchmark_data_smithi043_119832_object72628:head# 2024-03-30T13:50:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fd0a83e:::benchmark_data_smithi043_119832_object7452:head# 2024-03-30T13:50:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fd1026b:::benchmark_data_smithi043_119832_object74621:head# 2024-03-30T13:50:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fd1903b:::benchmark_data_smithi043_119832_object56987:head# 2024-03-30T13:50:34.528 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fd1d366:::benchmark_data_smithi043_119832_object42607:head# 2024-03-30T13:50:34.529 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fd26dd2:::benchmark_data_smithi043_119832_object52523:head# 2024-03-30T13:50:34.529 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fd31c9f:::benchmark_data_smithi043_119832_object70342:head# 2024-03-30T13:50:34.529 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fd353f5:::benchmark_data_smithi043_119832_object81097:head# 2024-03-30T13:50:34.529 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fd3ec15:::benchmark_data_smithi043_119832_object17089:head# 2024-03-30T13:50:34.529 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fd41518:::benchmark_data_smithi043_119832_object29239:head# 2024-03-30T13:50:34.529 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fd464e0:::benchmark_data_smithi043_119832_object24898:head# 2024-03-30T13:50:34.529 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fd4dbf3:::benchmark_data_smithi043_119832_object70521:head# 2024-03-30T13:50:34.529 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fd60723:::benchmark_data_smithi043_119832_object47874:head# 2024-03-30T13:50:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fd6b929:::benchmark_data_smithi043_119832_object3011:head# 2024-03-30T13:50:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fd70125:::benchmark_data_smithi043_119832_object18684:head# 2024-03-30T13:50:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fd70966:::benchmark_data_smithi043_119832_object43265:head# 2024-03-30T13:50:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fd70a8f:::benchmark_data_smithi043_119832_object55016:head# 2024-03-30T13:50:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fd71399:::benchmark_data_smithi043_119832_object26612:head# 2024-03-30T13:50:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fd82dfc:::benchmark_data_smithi043_119832_object65693:head# 2024-03-30T13:50:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fd847eb:::benchmark_data_smithi043_119832_object21611:head# 2024-03-30T13:50:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fda45a6:::benchmark_data_smithi043_119832_object57244:head# 2024-03-30T13:50:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fda83f7:::benchmark_data_smithi043_119832_object32865:head# 2024-03-30T13:50:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fda8671:::benchmark_data_smithi043_119832_object31590:head# 2024-03-30T13:50:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fdaf329:::benchmark_data_smithi043_119832_object22628:head# 2024-03-30T13:50:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fdd24e1:::benchmark_data_smithi043_119832_object8517:head# 2024-03-30T13:50:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fdd5397:::benchmark_data_smithi043_119832_object29943:head# 2024-03-30T13:50:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fddf818:::benchmark_data_smithi043_119832_object73307:head# 2024-03-30T13:50:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fde0a57:::benchmark_data_smithi043_119832_object42141:head# 2024-03-30T13:50:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fde44bf:::benchmark_data_smithi043_119832_object54044:head# 2024-03-30T13:50:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fde6978:::benchmark_data_smithi043_119832_object85112:head# 2024-03-30T13:50:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fdec3ba:::benchmark_data_smithi043_119832_object8236:head# 2024-03-30T13:50:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fdf014c:::benchmark_data_smithi043_119832_object31951:head# 2024-03-30T13:50:34.532 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fdf0514:::benchmark_data_smithi043_119832_object29878:head# 2024-03-30T13:50:34.532 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fdf497a:::benchmark_data_smithi043_119832_object45741:head# 2024-03-30T13:50:34.532 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fe009aa:::benchmark_data_smithi043_119832_object10627:head# 2024-03-30T13:50:34.532 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fe04cf6:::benchmark_data_smithi043_119832_object74735:head# 2024-03-30T13:50:34.532 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fe06c85:::benchmark_data_smithi043_119832_object2827:head# 2024-03-30T13:50:34.532 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fe10208:::benchmark_data_smithi043_119832_object57640:head# 2024-03-30T13:50:34.532 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fe284c5:::benchmark_data_smithi043_119832_object3250:head# 2024-03-30T13:50:34.532 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fe3689b:::benchmark_data_smithi043_119832_object17053:head# 2024-03-30T13:50:34.532 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fe3e8ee:::benchmark_data_smithi043_119832_object86784:head# 2024-03-30T13:50:34.533 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fe3ef99:::benchmark_data_smithi043_119832_object38104:head# 2024-03-30T13:50:34.533 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fe50325:::benchmark_data_smithi043_119832_object70766:head# 2024-03-30T13:50:34.533 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fe57b7e:::benchmark_data_smithi043_119832_object63879:head# 2024-03-30T13:50:34.533 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fe6d227:::benchmark_data_smithi043_119832_object34792:head# 2024-03-30T13:50:34.533 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fe71531:::benchmark_data_smithi043_119832_object40319:head# 2024-03-30T13:50:34.533 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fe7b112:::benchmark_data_smithi043_119832_object20083:head# 2024-03-30T13:50:34.533 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fe94e7b:::benchmark_data_smithi043_119832_object27740:head# 2024-03-30T13:50:34.533 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fe97aab:::benchmark_data_smithi043_119832_object24762:head# 2024-03-30T13:50:34.533 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fea1094:::benchmark_data_smithi043_119832_object4136:head# 2024-03-30T13:50:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7feaca24:::benchmark_data_smithi043_119832_object72982:head# 2024-03-30T13:50:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7feb3e28:::benchmark_data_smithi043_119832_object62850:head# 2024-03-30T13:50:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7feb7424:::benchmark_data_smithi043_119832_object77603:head# 2024-03-30T13:50:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fed17bd:::benchmark_data_smithi043_119832_object19587:head# 2024-03-30T13:50:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fedacbd:::benchmark_data_smithi043_119832_object64790:head# 2024-03-30T13:50:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ff0380d:::benchmark_data_smithi043_119832_object48251:head# 2024-03-30T13:50:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ff09a48:::benchmark_data_smithi043_119832_object62202:head# 2024-03-30T13:50:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ff09bd4:::benchmark_data_smithi043_119832_object75183:head# 2024-03-30T13:50:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ff0f2f2:::benchmark_data_smithi043_119832_object63851:head# 2024-03-30T13:50:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ff25ec9:::benchmark_data_smithi043_119832_object61193:head# 2024-03-30T13:50:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ff2835b:::benchmark_data_smithi043_119832_object51674:head# 2024-03-30T13:50:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ff5b365:::benchmark_data_smithi043_119832_object60971:head# 2024-03-30T13:50:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ff5cdef:::benchmark_data_smithi043_119832_object43195:head# 2024-03-30T13:50:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ff5f5a7:::benchmark_data_smithi043_119832_object12665:head# 2024-03-30T13:50:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ff65e19:::benchmark_data_smithi043_119832_object78720:head# 2024-03-30T13:50:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ff67e55:::benchmark_data_smithi043_119832_object69590:head# 2024-03-30T13:50:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ff7e9d9:::benchmark_data_smithi043_119832_object67507:head# 2024-03-30T13:50:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ff7ec5c:::benchmark_data_smithi043_119832_object23365:head# 2024-03-30T13:50:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ff82af1:::benchmark_data_smithi043_119832_object59820:head# 2024-03-30T13:50:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ff8a942:::benchmark_data_smithi043_119832_object5662:head# 2024-03-30T13:50:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ff9f6a0:::benchmark_data_smithi043_119832_object20126:head# 2024-03-30T13:50:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ffa3e16:::benchmark_data_smithi043_119832_object54434:head# 2024-03-30T13:50:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ffa446e:::benchmark_data_smithi043_119832_object26860:head# 2024-03-30T13:50:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ffa78be:::benchmark_data_smithi043_119832_object3462:head# 2024-03-30T13:50:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ffab638:::benchmark_data_smithi043_119832_object59912:head# 2024-03-30T13:50:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ffb743e:::benchmark_data_smithi043_119832_object15221:head# 2024-03-30T13:50:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ffc5789:::benchmark_data_smithi043_119832_object11843:head# 2024-03-30T13:50:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ffcfedd:::benchmark_data_smithi043_119832_object993:head# 2024-03-30T13:50:34.537 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7ffdd7d8:::benchmark_data_smithi043_119832_object83187:head# 2024-03-30T13:50:34.537 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fff5512:::benchmark_data_smithi043_119832_object23472:head# 2024-03-30T13:50:34.537 INFO:teuthology.orchestra.run.smithi043.stderr:Read #8:7fff689d:::benchmark_data_smithi043_119832_object6608:head# 2024-03-30T13:50:34.537 INFO:teuthology.orchestra.run.smithi043.stderr:Export successful 2024-03-30T13:50:34.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:34.538 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:34.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:34.639 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:34.674 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:34.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:34.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:34.776 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:34.800 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:34.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:34.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:34.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:34 smithi043 ceph-mon[29963]: pgmap v311: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 1k op/s; 42459/337251 objects degraded (12.590%) 2024-03-30T13:50:34.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:34 smithi043 ceph-mon[42099]: pgmap v311: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 1k op/s; 42459/337251 objects degraded (12.590%) 2024-03-30T13:50:34.902 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:34.944 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:34.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:34.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:34.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:34 smithi113 ceph-mon[34341]: pgmap v311: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 1k op/s; 42459/337251 objects degraded (12.590%) 2024-03-30T13:50:34.981 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-03-30T13:50:34.981507+0000 min lat: 0.00129646 max lat: 1.05462 avg lat: 0.0108661 2024-03-30T13:50:34.981 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T13:50:34.981 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 80 16 117755 117739 91.9681 64.375 0.00643676 0.0108661 2024-03-30T13:50:35.047 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:35.087 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:35.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:35.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:35.189 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:35.223 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:35.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:35.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:35.325 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:35.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:35.361 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:35.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:35.462 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:35.502 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:35.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:35.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:35.603 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:35.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:35.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:35.738 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:35.776 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:35.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:35.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:35.878 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:35.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:35.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:35.981 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 81 16 118873 118857 91.6953 69.875 0.00839113 0.010901 2024-03-30T13:50:36.012 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:36.048 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:36.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:36.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:36.150 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:36.184 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:36.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:36.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:36.286 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:36.287 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:50:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:50:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:50:36.319 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:36.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:36.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:36.421 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:36.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:36.454 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:36.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:36.554 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-03-30T13:50:36.556 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:36.590 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:36.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:36.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:36.692 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:36.746 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:36.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:36.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:36.848 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:36.891 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:36.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:36.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:36 smithi043 ceph-mon[29963]: pgmap v312: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 43747/350016 objects degraded (12.499%) 2024-03-30T13:50:36.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:36 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/311563980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:50:36.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:36 smithi043 ceph-mon[42099]: pgmap v312: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 43747/350016 objects degraded (12.499%) 2024-03-30T13:50:36.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:36 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/311563980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:50:36.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:36 smithi113 ceph-mon[34341]: pgmap v312: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 43747/350016 objects degraded (12.499%) 2024-03-30T13:50:36.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:36 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/311563980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:50:36.981 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 82 16 120126 120110 91.532 78.3125 0.00511427 0.0109205 2024-03-30T13:50:36.993 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:37.028 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:37.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:37.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:37.129 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:37.165 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:37.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:37.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:37.267 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:37.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:37.305 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:37.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:37.406 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:37.442 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:37.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:37.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:37.545 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:37.579 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:37.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:37.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:37.681 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:37.715 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:37.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:37.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:37.817 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:37.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:37.853 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:37.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:37.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:37 smithi043 ceph-mon[42099]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:50:37.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:37 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/311563980' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:50:37.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:37 smithi043 ceph-mon[42099]: osdmap e177: 8 total, 7 up, 8 in 2024-03-30T13:50:37.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:37 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 43747/350016 objects degraded (12.499%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:37.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:37 smithi043 ceph-mon[42099]: osdmap e178: 8 total, 7 up, 8 in 2024-03-30T13:50:37.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:37 smithi043 ceph-mon[29963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:50:37.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:37 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/311563980' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:50:37.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:37 smithi043 ceph-mon[29963]: osdmap e177: 8 total, 7 up, 8 in 2024-03-30T13:50:37.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:37 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 43747/350016 objects degraded (12.499%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:37.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:37 smithi043 ceph-mon[29963]: osdmap e178: 8 total, 7 up, 8 in 2024-03-30T13:50:37.954 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:37.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:37 smithi113 ceph-mon[34341]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:50:37.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:37 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/311563980' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:50:37.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:37 smithi113 ceph-mon[34341]: osdmap e177: 8 total, 7 up, 8 in 2024-03-30T13:50:37.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:37 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 43747/350016 objects degraded (12.499%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:37.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:37 smithi113 ceph-mon[34341]: osdmap e178: 8 total, 7 up, 8 in 2024-03-30T13:50:37.981 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 83 16 121437 121421 91.4164 81.9375 0.00869737 0.0109345 2024-03-30T13:50:37.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:37.989 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:37.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:38.090 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:38.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:38.129 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:38.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:38.230 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:38.266 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:38.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:38.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:38.367 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:38.387 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -n osd.2 --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 8.e 2024-03-30T13:50:38.399 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:38.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:38.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:38.501 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:38.531 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:38.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:38.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:38.609 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:38 smithi043 ceph-mon[29963]: pgmap v314: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 740 B/s rd, 47 MiB/s wr, 755 op/s; 43747/350016 objects degraded (12.499%) 2024-03-30T13:50:38.609 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:38 smithi043 ceph-mon[42099]: pgmap v314: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 740 B/s rd, 47 MiB/s wr, 755 op/s; 43747/350016 objects degraded (12.499%) 2024-03-30T13:50:38.617 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:50:38.632 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:38.663 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:38.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:38.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:38.730 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:50:38.730 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:50:38.764 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:38.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:38.795 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:38.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:38.897 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:38.933 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/osd.2/config 2024-03-30T13:50:38.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:38.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:38.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:38 smithi113 ceph-mon[34341]: pgmap v314: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 740 B/s rd, 47 MiB/s wr, 755 op/s; 43747/350016 objects degraded (12.499%) 2024-03-30T13:50:38.972 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:50:38 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:50:38.981 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 84 16 122761 122745 91.3131 82.75 0.00572343 0.0109463 2024-03-30T13:50:39.047 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:39.086 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:39.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:39.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:39.188 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:39.223 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:39.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:39.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:39.324 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:39.365 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:39.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:39.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:39.467 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:39.504 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:39.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:39.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:39.605 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:39.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:39.648 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:39.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:39.749 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:39.786 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:39.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:39.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:39.814 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T13:50:39.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-03-30T13:50:39.852 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:39 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:50:39.853 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:39 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:50:39.888 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:39.923 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:39.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:39.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:39.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:39 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:50:39.982 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 85 16 124158 124142 91.266 87.3125 0.0186553 0.0109523 2024-03-30T13:50:40.025 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:40.102 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:40.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:40.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:40.204 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:40.237 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:40.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:40.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:40.339 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:40.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:40.376 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:40.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:40.477 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:40.515 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:40.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:40.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:40.616 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:40.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:40.654 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:40.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:40.755 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:40.794 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:40.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:40.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:40.896 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:40.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:40 smithi043 ceph-mon[29963]: pgmap v316: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 100 MiB/s wr, 1.60k op/s; 45945/367932 objects degraded (12.487%) 2024-03-30T13:50:40.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:40 smithi043 ceph-mon[42099]: pgmap v316: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 100 MiB/s wr, 1.60k op/s; 45945/367932 objects degraded (12.487%) 2024-03-30T13:50:40.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:41.017 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:41.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:41.017 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:40 smithi113 ceph-mon[34341]: pgmap v316: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 100 MiB/s wr, 1.60k op/s; 45945/367932 objects degraded (12.487%) 2024-03-30T13:50:41.018 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 86 16 125344 125328 91.0666 74.125 0.007664 0.0109765 2024-03-30T13:50:41.118 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:41.153 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:41.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:41.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:41.255 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:41.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:41.290 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:41.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:41.391 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:41.433 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:41.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:41.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:41.535 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:41.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:41.559 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:41.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:41.660 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:41.696 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:41.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:41.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:41.797 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:41.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:41.834 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:41.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:41.935 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:41.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:41.964 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:41.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:41.982 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 87 16 126509 126493 90.8567 72.8125 0.00676294 0.0110023 2024-03-30T13:50:42.065 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:42.103 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:42.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:42.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:42.205 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:42.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:42.243 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:42.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:42.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:42.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:42.382 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:42.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:42.483 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:42.519 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:42.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:42.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:42.621 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:42.647 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:42.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:42.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:42.749 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:42.782 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:42.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:42.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:42.884 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:42.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:42 smithi043 ceph-mon[29963]: pgmap v317: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 45945/367932 objects degraded (12.487%) 2024-03-30T13:50:42.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:42 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 45945/367932 objects degraded (12.487%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:42.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:42 smithi043 ceph-mon[42099]: pgmap v317: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 45945/367932 objects degraded (12.487%) 2024-03-30T13:50:42.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:42 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 45945/367932 objects degraded (12.487%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:42.918 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:42.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:42.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:42.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:42 smithi113 ceph-mon[34341]: pgmap v317: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 45945/367932 objects degraded (12.487%) 2024-03-30T13:50:42.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:42 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 45945/367932 objects degraded (12.487%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:42.982 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 88 16 127839 127823 90.7688 83.125 0.00640201 0.0110122 2024-03-30T13:50:43.020 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:43.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:43.055 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:43.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:43.156 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:43.187 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:43.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:43.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:43.289 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:43.329 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:43.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:43.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:43.430 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:43.467 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:43.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:43.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:43.569 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:43.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:43.606 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:43.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:43.707 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:43.748 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:43.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:43.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:43.850 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:43.887 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:43.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:43.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:43.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:43 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:50:43.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:43 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:50:43.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:43 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:50:43.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:43 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:50:43.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:43 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:50:43.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:43 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:50:43.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:43 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:50:43.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:43 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:50:43.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:43 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:50:43.982 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 89 16 129094 129078 90.6302 78.4375 0.00295189 0.0110262 2024-03-30T13:50:43.989 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:44.024 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:44.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:44.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:44.126 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:44.162 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:44.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:44.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:44.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:44.297 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:44.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:44.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:44.398 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:44.439 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:44.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:44.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:44.541 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:44.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:44.565 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:44.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:44.666 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:44.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:44.704 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:44.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:44.805 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:44.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:44.844 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:44.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:44.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:44 smithi043 ceph-mon[29963]: pgmap v318: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 56 MiB/s wr, 892 op/s; 46729/371430 objects degraded (12.581%) 2024-03-30T13:50:44.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:44 smithi043 ceph-mon[29963]: osdmap e179: 8 total, 7 up, 8 in 2024-03-30T13:50:44.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:44 smithi043 ceph-mon[42099]: pgmap v318: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 56 MiB/s wr, 892 op/s; 46729/371430 objects degraded (12.581%) 2024-03-30T13:50:44.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:44 smithi043 ceph-mon[42099]: osdmap e179: 8 total, 7 up, 8 in 2024-03-30T13:50:44.945 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:44.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:44 smithi113 ceph-mon[34341]: pgmap v318: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 56 MiB/s wr, 892 op/s; 46729/371430 objects degraded (12.581%) 2024-03-30T13:50:44.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:44 smithi113 ceph-mon[34341]: osdmap e179: 8 total, 7 up, 8 in 2024-03-30T13:50:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:44.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:44.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:44.982 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Total time run: 90.0089 2024-03-30T13:50:44.982 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Total writes made: 130351 2024-03-30T13:50:44.982 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Write size: 65536 2024-03-30T13:50:44.982 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Object size: 65536 2024-03-30T13:50:44.982 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Bandwidth (MB/sec): 90.5126 2024-03-30T13:50:44.983 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev Bandwidth: 32.4977 2024-03-30T13:50:44.983 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max bandwidth (MB/sec): 204 2024-03-30T13:50:44.983 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min bandwidth (MB/sec): 32.6875 2024-03-30T13:50:44.983 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Average IOPS: 1448 2024-03-30T13:50:44.983 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev IOPS: 519.963 2024-03-30T13:50:44.983 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max IOPS: 3264 2024-03-30T13:50:44.983 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min IOPS: 523 2024-03-30T13:50:44.983 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Average Latency(s): 0.0110441 2024-03-30T13:50:44.983 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev Latency(s): 0.0151409 2024-03-30T13:50:44.983 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max latency(s): 1.05462 2024-03-30T13:50:44.983 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min latency(s): 0.00129646 2024-03-30T13:50:44.988 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Cleaning up (deleting benchmark objects) 2024-03-30T13:50:45.080 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:45.112 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:45.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:45.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:45.214 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:45.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:45.247 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:45.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:45.348 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:45.385 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:45.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:45.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:45.487 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:45.524 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:45.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:45.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:45.625 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:45.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:45.660 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:45.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:45.761 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:45.793 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:45.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:45.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:45.895 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:45.926 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:45.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:45.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:46.028 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:46.061 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:46.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:46.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:46.163 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:46.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:46.209 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:46.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:46.310 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:46.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:46.351 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:46.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:46.399 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:50:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:50:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:50:46.452 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:46.486 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:46.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:46.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:46.588 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:46.623 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:46.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:46.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:46.656 INFO:teuthology.orchestra.run.smithi043.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-30T13:50:46.725 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:46.762 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:46.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:46.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:46.864 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:46.888 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:46 smithi043 ceph-mon[29963]: pgmap v320: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s; 48320/387039 objects degraded (12.485%) 2024-03-30T13:50:46.889 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:46 smithi043 ceph-mon[42099]: pgmap v320: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s; 48320/387039 objects degraded (12.485%) 2024-03-30T13:50:46.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:46.901 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:46.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:46.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:46 smithi113 ceph-mon[34341]: pgmap v320: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s; 48320/387039 objects degraded (12.485%) 2024-03-30T13:50:47.002 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:47.041 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:47.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:47.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:47.143 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:47.175 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:47.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:47.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:47.277 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:47.309 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:47.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:47.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:47.373 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:50:47.411 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:47.452 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:47.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:47.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:47.497 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:50:47.547 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:50:47.555 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:47.585 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:47.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:47.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:47.587 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:50:47.637 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:50:47.684 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:50:47.687 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:47.722 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:50:47.729 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:47.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:47.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:47.762 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:47 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 48320/387039 objects degraded (12.485%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:47.762 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:47 smithi043 ceph-mon[29963]: osdmap e180: 8 total, 7 up, 8 in 2024-03-30T13:50:47.762 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:47 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1329021294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:50:47.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:47 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 48320/387039 objects degraded (12.485%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:47.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:47 smithi043 ceph-mon[42099]: osdmap e180: 8 total, 7 up, 8 in 2024-03-30T13:50:47.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:47 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1329021294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:50:47.769 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:50:47.769 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:50:47.769 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:50:47.769 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:50:47.770 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:50:47.770 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:50:47.770 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:50:47.770 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:50:47.770 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:50:47.825 INFO:teuthology.orchestra.run.smithi043.stdout: marking collection for removal 2024-03-30T13:50:47.826 INFO:teuthology.orchestra.run.smithi043.stdout:setting '_remove' omap key 2024-03-30T13:50:47.826 INFO:teuthology.orchestra.run.smithi043.stdout:finish_remove_pgs 8.e_head removing 8.e 2024-03-30T13:50:47.830 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:47.878 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:47.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:47.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:47.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:47 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 48320/387039 objects degraded (12.485%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:47.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:47 smithi113 ceph-mon[34341]: osdmap e180: 8 total, 7 up, 8 in 2024-03-30T13:50:47.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:47 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1329021294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:50:47.980 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:48.002 INFO:teuthology.orchestra.run.smithi043.stdout:Remove successful 2024-03-30T13:50:48.014 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:48.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:48.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:48.116 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:48.152 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:48.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:48.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:48.254 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:48.285 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:48.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:48.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:48.387 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:48.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:48.424 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:48.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:48.525 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:48.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:48.564 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:48.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:48.665 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:48.700 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:48.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:48.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:48.720 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:50:48.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:48.836 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:48 smithi043 ceph-mon[29963]: pgmap v322: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 796 op/s; 48320/387039 objects degraded (12.485%) 2024-03-30T13:50:48.836 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:48 smithi043 ceph-mon[42099]: pgmap v322: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 796 op/s; 48320/387039 objects degraded (12.485%) 2024-03-30T13:50:48.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:48.838 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:48.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:48.939 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:48.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:48 smithi113 ceph-mon[34341]: pgmap v322: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 796 op/s; 48320/387039 objects degraded (12.485%) 2024-03-30T13:50:48.972 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:50:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:50:48.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:48.977 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:48.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:49.078 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:49.110 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:49.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:49.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:49.212 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:49.243 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:49.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:49.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:49.345 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:49.378 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:49.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:49.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:49.480 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:49.517 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:49.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:49.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:49.619 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:49.652 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:49.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:49.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:49.754 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:49.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:49.791 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:49.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:49.872 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:49 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:50:49.893 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:49.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:49 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:50:49.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:49 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:50:49.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:49.942 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:49.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:50.043 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:50.077 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:50.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:50.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:50.178 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:50.206 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:50.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:50.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:50.308 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:50.346 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:50.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:50.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:50.448 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:50.479 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:50.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:50.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:50.583 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:50.616 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:50.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:50.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:50.718 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:50.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:50.757 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:50.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:50.857 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:50.890 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:50.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:50.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:50.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:50 smithi043 ceph-mon[29963]: pgmap v323: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 62 MiB/s wr, 1.68k op/s; 46837/375144 objects degraded (12.485%) 2024-03-30T13:50:50.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:50 smithi043 ceph-mon[42099]: pgmap v323: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 62 MiB/s wr, 1.68k op/s; 46837/375144 objects degraded (12.485%) 2024-03-30T13:50:50.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:50 smithi113 ceph-mon[34341]: pgmap v323: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 62 MiB/s wr, 1.68k op/s; 46837/375144 objects degraded (12.485%) 2024-03-30T13:50:50.992 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:51.031 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:51.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:51.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:51.133 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:51.166 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:51.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:51.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:51.267 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:51.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:51.305 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:51.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:51.406 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:51.444 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:51.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:51.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:51.546 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:51.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:51.581 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:51.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:51.682 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:51.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:51.721 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:51.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:51.771 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:50:51.771 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:50:51.822 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:51.861 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:51.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:51.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:51.897 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -n osd.2 --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.8.e.2 2024-03-30T13:50:51.963 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:51.996 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:51.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:51.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:52.098 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:52.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:52.145 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:52.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:52.245 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:52.286 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:52.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:52.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:52.355 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/osd.2/config 2024-03-30T13:50:52.388 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:52.425 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:52.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:52.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:52.527 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:52.566 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:52.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:52.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:52.668 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:52.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:52.700 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:52.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:52.801 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:52.833 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:52.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:52.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:52.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:52 smithi043 ceph-mon[29963]: pgmap v324: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 53 MiB/s wr, 1.53k op/s; 46837/375144 objects degraded (12.485%) 2024-03-30T13:50:52.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:52 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 46837/375144 objects degraded (12.485%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:52.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:52 smithi043 ceph-mon[42099]: pgmap v324: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 53 MiB/s wr, 1.53k op/s; 46837/375144 objects degraded (12.485%) 2024-03-30T13:50:52.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:52 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 46837/375144 objects degraded (12.485%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:52.935 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:52.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:52 smithi113 ceph-mon[34341]: pgmap v324: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 53 MiB/s wr, 1.53k op/s; 46837/375144 objects degraded (12.485%) 2024-03-30T13:50:52.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:52 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 46837/375144 objects degraded (12.485%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:52.972 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:52.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:52.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:53.074 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:53.112 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:53.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:53.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:53.214 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:53.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:53.250 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:53.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:53.351 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:53.386 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:53.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:53.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:53.489 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:53.522 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:53.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:53.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:53.624 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:53.659 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:53.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:53.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:53.761 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:53.772 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:50:53.795 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:53.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:53.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:53.897 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:53.930 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:53.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:53.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:54.032 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:54.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:54.064 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:54.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:54.165 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:54.200 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:54.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:54.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:54.302 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:54.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:54.345 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:54.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:54.371 INFO:teuthology.orchestra.run.smithi043.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-30T13:50:54.446 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:54.479 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:54.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:54.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:54.581 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:54.615 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:54.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:54.636 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T13:50:54.716 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:54.762 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:54.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:54.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:54.864 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:54.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:54 smithi043 ceph-mon[29963]: pgmap v325: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 624 B/s rd, 35 MiB/s wr, 1.12k op/s; 46837/375144 objects degraded (12.485%) 2024-03-30T13:50:54.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:54 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/690235960' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:50:54.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:54 smithi043 ceph-mon[42099]: pgmap v325: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 624 B/s rd, 35 MiB/s wr, 1.12k op/s; 46837/375144 objects degraded (12.485%) 2024-03-30T13:50:54.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:54 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/690235960' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:50:54.901 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:54.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:54.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:54.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:54 smithi113 ceph-mon[34341]: pgmap v325: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 624 B/s rd, 35 MiB/s wr, 1.12k op/s; 46837/375144 objects degraded (12.485%) 2024-03-30T13:50:54.974 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:54 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/690235960' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:50:55.003 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:55.040 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:55.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:55.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:55.142 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:55.172 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:55.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:55.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:55.275 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:55.309 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:55.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:55.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:55.411 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:55.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:55.448 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:55.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:55.549 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:55.586 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:55.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:55.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:55.688 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:55.719 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:55.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:55.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:55.821 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:55.854 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:55.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:55.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:55.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:55 smithi043 ceph-mon[29963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:50:55.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:55 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/690235960' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:50:55.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:55 smithi043 ceph-mon[29963]: osdmap e181: 8 total, 7 up, 8 in 2024-03-30T13:50:55.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:55 smithi043 ceph-mon[29963]: 8.5 scrub starts 2024-03-30T13:50:55.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:55 smithi043 ceph-mon[29963]: 4.7 scrub starts 2024-03-30T13:50:55.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:55 smithi043 ceph-mon[29963]: 4.7 scrub ok 2024-03-30T13:50:55.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:55 smithi043 ceph-mon[42099]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:50:55.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:55 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/690235960' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:50:55.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:55 smithi043 ceph-mon[42099]: osdmap e181: 8 total, 7 up, 8 in 2024-03-30T13:50:55.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:55 smithi043 ceph-mon[42099]: 8.5 scrub starts 2024-03-30T13:50:55.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:55 smithi043 ceph-mon[42099]: 4.7 scrub starts 2024-03-30T13:50:55.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:55 smithi043 ceph-mon[42099]: 4.7 scrub ok 2024-03-30T13:50:55.956 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:55.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:55 smithi113 ceph-mon[34341]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:50:55.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:55 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/690235960' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:50:55.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:55 smithi113 ceph-mon[34341]: osdmap e181: 8 total, 7 up, 8 in 2024-03-30T13:50:55.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:55 smithi113 ceph-mon[34341]: 8.5 scrub starts 2024-03-30T13:50:55.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:55 smithi113 ceph-mon[34341]: 4.7 scrub starts 2024-03-30T13:50:55.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:55 smithi113 ceph-mon[34341]: 4.7 scrub ok 2024-03-30T13:50:56.000 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:56.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:56.102 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:56.148 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:56.148 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:50:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:50:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:50:56.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:56.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:56.249 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:50:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T13:50:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-03-30T13:50:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-03-30T13:50:56.249 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:50:56.249 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:56.286 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:56.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:56.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:56.388 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:56.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:56.426 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:56.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:56.528 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:56.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:56.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:56.663 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:56.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:56.705 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:56.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:56.806 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:56.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:56.848 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:56.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:56.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:56 smithi043 ceph-mon[29963]: pgmap v327: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 12 MiB/s wr, 1.74k op/s; 44241/354549 objects degraded (12.478%) 2024-03-30T13:50:56.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:56 smithi043 ceph-mon[29963]: 5.19 scrub starts 2024-03-30T13:50:56.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:56 smithi043 ceph-mon[29963]: 5.19 scrub ok 2024-03-30T13:50:56.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:56 smithi043 ceph-mon[42099]: pgmap v327: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 12 MiB/s wr, 1.74k op/s; 44241/354549 objects degraded (12.478%) 2024-03-30T13:50:56.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:56 smithi043 ceph-mon[42099]: 5.19 scrub starts 2024-03-30T13:50:56.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:56 smithi043 ceph-mon[42099]: 5.19 scrub ok 2024-03-30T13:50:56.949 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:56.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:56 smithi113 ceph-mon[34341]: pgmap v327: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 12 MiB/s wr, 1.74k op/s; 44241/354549 objects degraded (12.478%) 2024-03-30T13:50:56.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:56 smithi113 ceph-mon[34341]: 5.19 scrub starts 2024-03-30T13:50:56.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:56 smithi113 ceph-mon[34341]: 5.19 scrub ok 2024-03-30T13:50:56.991 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:56.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:56.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:57.045 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-03-30T13:50:57.093 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:57.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:57.138 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:57.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:57.239 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:57.257 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:50:57.279 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:57.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:57.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:57.381 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:57.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:57.421 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:57.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:57.495 INFO:teuthology.orchestra.run.smithi043.stdout:get_pg_num_history pg_num_history pg_num_history(e166 pg_nums {1={19=1},2={63=3},3={64=32},4={66=32},5={68=32},6={70=32},7={98=16},8={147=16}} deleted_pools 145,7) 2024-03-30T13:50:57.495 INFO:teuthology.orchestra.run.smithi043.stdout:Importing pgid 8.e 2024-03-30T13:50:57.503 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70000158:::benchmark_data_smithi043_119832_object18973:head# 2024-03-30T13:50:57.504 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.504 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7000934d:::benchmark_data_smithi043_119832_object50764:head# 2024-03-30T13:50:57.504 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.504 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7000e5ef:::benchmark_data_smithi043_119832_object87691:head# 2024-03-30T13:50:57.504 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.504 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7002fd95:::benchmark_data_smithi043_119832_object267:head# 2024-03-30T13:50:57.504 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.505 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:700302d5:::benchmark_data_smithi043_119832_object31432:head# 2024-03-30T13:50:57.505 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.505 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70030c19:::benchmark_data_smithi043_119832_object40003:head# 2024-03-30T13:50:57.505 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.506 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7003dcb3:::benchmark_data_smithi043_119832_object28164:head# 2024-03-30T13:50:57.506 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.507 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:700524b2:::benchmark_data_smithi043_119832_object23052:head# 2024-03-30T13:50:57.507 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.507 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7005811f:::benchmark_data_smithi043_119832_object38752:head# 2024-03-30T13:50:57.507 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.507 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:700661f5:::benchmark_data_smithi043_119832_object20795:head# 2024-03-30T13:50:57.507 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.508 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7007a27b:::benchmark_data_smithi043_119832_object35591:head# 2024-03-30T13:50:57.508 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.508 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70082362:::benchmark_data_smithi043_119832_object13125:head# 2024-03-30T13:50:57.508 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.508 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7008ca39:::benchmark_data_smithi043_119832_object21970:head# 2024-03-30T13:50:57.509 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.509 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70099af1:::benchmark_data_smithi043_119832_object20472:head# 2024-03-30T13:50:57.509 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.509 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:700a1cbb:::benchmark_data_smithi043_119832_object17699:head# 2024-03-30T13:50:57.509 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.509 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:700b3b1c:::benchmark_data_smithi043_119832_object77210:head# 2024-03-30T13:50:57.509 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.510 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:700f3efe:::benchmark_data_smithi043_119832_object79719:head# 2024-03-30T13:50:57.510 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.510 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70105bf1:::benchmark_data_smithi043_119832_object3840:head# 2024-03-30T13:50:57.510 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.511 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7010fb6a:::benchmark_data_smithi043_119832_object8497:head# 2024-03-30T13:50:57.511 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.511 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70117fa6:::benchmark_data_smithi043_119832_object4021:head# 2024-03-30T13:50:57.511 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.511 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7011d777:::benchmark_data_smithi043_119832_object71475:head# 2024-03-30T13:50:57.511 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.511 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7012d397:::benchmark_data_smithi043_119832_object87870:head# 2024-03-30T13:50:57.511 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.511 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7012fe35:::benchmark_data_smithi043_119832_object38493:head# 2024-03-30T13:50:57.511 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.512 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7013a700:::benchmark_data_smithi043_119832_object58094:head# 2024-03-30T13:50:57.513 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.513 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7013b1f4:::benchmark_data_smithi043_119832_object22506:head# 2024-03-30T13:50:57.513 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.513 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7015d905:::benchmark_data_smithi043_119832_object67941:head# 2024-03-30T13:50:57.513 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.513 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7015df1c:::benchmark_data_smithi043_119832_object20267:head# 2024-03-30T13:50:57.513 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.513 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70162494:::benchmark_data_smithi043_119832_object25481:head# 2024-03-30T13:50:57.513 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.513 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70165efa:::benchmark_data_smithi043_119832_object30799:head# 2024-03-30T13:50:57.513 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.513 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7016f0fc:::benchmark_data_smithi043_119832_object17695:head# 2024-03-30T13:50:57.513 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.515 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70170670:::benchmark_data_smithi043_119832_object21158:head# 2024-03-30T13:50:57.515 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.515 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70171c83:::benchmark_data_smithi043_119832_object55411:head# 2024-03-30T13:50:57.516 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.516 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70186dc7:::benchmark_data_smithi043_119832_object64168:head# 2024-03-30T13:50:57.516 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.516 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7019f491:::benchmark_data_smithi043_119832_object53571:head# 2024-03-30T13:50:57.516 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.516 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:701a2b9e:::benchmark_data_smithi043_119832_object40090:head# 2024-03-30T13:50:57.516 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.516 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:701a8775:::benchmark_data_smithi043_119832_object66566:head# 2024-03-30T13:50:57.516 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.516 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:701a8e2e:::benchmark_data_smithi043_119832_object65994:head# 2024-03-30T13:50:57.516 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.516 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:701aa75a:::benchmark_data_smithi043_119832_object15185:head# 2024-03-30T13:50:57.516 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.518 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:701abb32:::benchmark_data_smithi043_119832_object53518:head# 2024-03-30T13:50:57.518 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.518 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:701b1149:::benchmark_data_smithi043_119832_object71946:head# 2024-03-30T13:50:57.518 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.518 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:701b1850:::benchmark_data_smithi043_119832_object68311:head# 2024-03-30T13:50:57.518 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.518 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:701bcdb4:::benchmark_data_smithi043_119832_object38562:head# 2024-03-30T13:50:57.519 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.519 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:701c2dc5:::benchmark_data_smithi043_119832_object55407:head# 2024-03-30T13:50:57.519 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.519 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:701c820c:::benchmark_data_smithi043_119832_object38358:head# 2024-03-30T13:50:57.519 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.519 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:701d4d59:::benchmark_data_smithi043_119832_object8526:head# 2024-03-30T13:50:57.519 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.519 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:701d4f27:::benchmark_data_smithi043_119832_object23902:head# 2024-03-30T13:50:57.519 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.519 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:701dbff6:::benchmark_data_smithi043_119832_object59109:head# 2024-03-30T13:50:57.519 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.520 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:701e0fbc:::benchmark_data_smithi043_119832_object81436:head# 2024-03-30T13:50:57.520 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.520 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:701e4a7f:::benchmark_data_smithi043_119832_object1088:head# 2024-03-30T13:50:57.521 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.521 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:701e52ae:::benchmark_data_smithi043_119832_object6158:head# 2024-03-30T13:50:57.521 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.521 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:701ece4c:::benchmark_data_smithi043_119832_object14107:head# 2024-03-30T13:50:57.521 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.521 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:701f683e:::benchmark_data_smithi043_119832_object53424:head# 2024-03-30T13:50:57.521 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.521 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:701ff537:::benchmark_data_smithi043_119832_object67966:head# 2024-03-30T13:50:57.521 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.521 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:57.522 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7020108f:::benchmark_data_smithi043_119832_object5976:head# 2024-03-30T13:50:57.522 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.522 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70201efc:::benchmark_data_smithi043_119832_object65255:head# 2024-03-30T13:50:57.522 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.523 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7021fa21:::benchmark_data_smithi043_119832_object67019:head# 2024-03-30T13:50:57.523 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.523 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70224df2:::benchmark_data_smithi043_119832_object4362:head# 2024-03-30T13:50:57.523 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.523 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7022669d:::benchmark_data_smithi043_119832_object81668:head# 2024-03-30T13:50:57.523 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.524 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7023d4b9:::benchmark_data_smithi043_119832_object80302:head# 2024-03-30T13:50:57.524 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.524 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70247df2:::benchmark_data_smithi043_119832_object68457:head# 2024-03-30T13:50:57.524 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.524 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70261550:::benchmark_data_smithi043_119832_object69358:head# 2024-03-30T13:50:57.524 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.524 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70266a5c:::benchmark_data_smithi043_119832_object6750:head# 2024-03-30T13:50:57.525 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.525 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70267f9b:::benchmark_data_smithi043_119832_object23830:head# 2024-03-30T13:50:57.525 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.526 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:702750eb:::benchmark_data_smithi043_119832_object29381:head# 2024-03-30T13:50:57.526 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.526 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70282d5b:::benchmark_data_smithi043_119832_object86953:head# 2024-03-30T13:50:57.526 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.526 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7029058f:::benchmark_data_smithi043_119832_object74484:head# 2024-03-30T13:50:57.526 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.527 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70292585:::benchmark_data_smithi043_119832_object71642:head# 2024-03-30T13:50:57.527 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.527 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7029ff00:::benchmark_data_smithi043_119832_object11792:head# 2024-03-30T13:50:57.528 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.528 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:702ab07a:::benchmark_data_smithi043_119832_object85843:head# 2024-03-30T13:50:57.528 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.529 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:702ac466:::benchmark_data_smithi043_119832_object63525:head# 2024-03-30T13:50:57.529 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.529 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:702cae14:::benchmark_data_smithi043_119832_object49346:head# 2024-03-30T13:50:57.529 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.529 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:702cbcd1:::benchmark_data_smithi043_119832_object74508:head# 2024-03-30T13:50:57.529 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.529 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:702cc0fc:::benchmark_data_smithi043_119832_object24364:head# 2024-03-30T13:50:57.530 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.530 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:702d5e6c:::benchmark_data_smithi043_119832_object41726:head# 2024-03-30T13:50:57.530 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.530 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:702dd9dd:::benchmark_data_smithi043_119832_object60814:head# 2024-03-30T13:50:57.530 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.531 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:702f716a:::benchmark_data_smithi043_119832_object17559:head# 2024-03-30T13:50:57.531 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.531 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70302403:::benchmark_data_smithi043_119832_object48850:head# 2024-03-30T13:50:57.531 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.531 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7030b5f3:::benchmark_data_smithi043_119832_object60422:head# 2024-03-30T13:50:57.531 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.532 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7032237a:::benchmark_data_smithi043_119832_object3778:head# 2024-03-30T13:50:57.532 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.532 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70325169:::benchmark_data_smithi043_119832_object48828:head# 2024-03-30T13:50:57.533 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.533 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7032c809:::benchmark_data_smithi043_119832_object86159:head# 2024-03-30T13:50:57.533 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.533 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70338a85:::benchmark_data_smithi043_119832_object72378:head# 2024-03-30T13:50:57.533 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.533 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7033c6c3:::benchmark_data_smithi043_119832_object33681:head# 2024-03-30T13:50:57.533 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.534 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:703516d7:::benchmark_data_smithi043_119832_object72166:head# 2024-03-30T13:50:57.534 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.534 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7035a7dd:::benchmark_data_smithi043_119832_object35756:head# 2024-03-30T13:50:57.534 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.534 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70370c4e:::benchmark_data_smithi043_119832_object10274:head# 2024-03-30T13:50:57.534 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.535 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70376e29:::benchmark_data_smithi043_119832_object40609:head# 2024-03-30T13:50:57.535 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.535 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7037ba92:::benchmark_data_smithi043_119832_object22655:head# 2024-03-30T13:50:57.535 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.535 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7038b2a3:::benchmark_data_smithi043_119832_object22922:head# 2024-03-30T13:50:57.535 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.536 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7038bd71:::benchmark_data_smithi043_119832_object75604:head# 2024-03-30T13:50:57.536 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.536 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:703907dd:::benchmark_data_smithi043_119832_object12505:head# 2024-03-30T13:50:57.536 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.536 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:703a931b:::benchmark_data_smithi043_119832_object56093:head# 2024-03-30T13:50:57.536 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.537 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:703b1c3b:::benchmark_data_smithi043_119832_object5019:head# 2024-03-30T13:50:57.537 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.537 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:703ba6ef:::benchmark_data_smithi043_119832_object70671:head# 2024-03-30T13:50:57.537 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.537 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:703bb785:::benchmark_data_smithi043_119832_object16830:head# 2024-03-30T13:50:57.538 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.538 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:703be694:::benchmark_data_smithi043_119832_object41863:head# 2024-03-30T13:50:57.538 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.538 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:703d3dad:::benchmark_data_smithi043_119832_object68090:head# 2024-03-30T13:50:57.538 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.539 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:703d404c:::benchmark_data_smithi043_119832_object38401:head# 2024-03-30T13:50:57.539 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.539 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:703e7bae:::benchmark_data_smithi043_119832_object70506:head# 2024-03-30T13:50:57.539 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.539 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:703eb67f:::benchmark_data_smithi043_119832_object66252:head# 2024-03-30T13:50:57.540 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.542 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:703f4e56:::benchmark_data_smithi043_119832_object39459:head# 2024-03-30T13:50:57.542 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.543 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:703fedad:::benchmark_data_smithi043_119832_object62353:head# 2024-03-30T13:50:57.543 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.543 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7040b70c:::benchmark_data_smithi043_119832_object82720:head# 2024-03-30T13:50:57.543 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.543 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:704102bd:::benchmark_data_smithi043_119832_object12240:head# 2024-03-30T13:50:57.543 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.543 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70415d6b:::benchmark_data_smithi043_119832_object19797:head# 2024-03-30T13:50:57.543 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.543 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70418f64:::benchmark_data_smithi043_119832_object32940:head# 2024-03-30T13:50:57.543 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.543 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70429bfd:::benchmark_data_smithi043_119832_object24366:head# 2024-03-30T13:50:57.544 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.544 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70429e0e:::benchmark_data_smithi043_119832_object36155:head# 2024-03-30T13:50:57.544 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.546 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7042ecfa:::benchmark_data_smithi043_119832_object51882:head# 2024-03-30T13:50:57.547 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.547 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:704304b5:::benchmark_data_smithi043_119832_object24332:head# 2024-03-30T13:50:57.547 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.547 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7043a762:::benchmark_data_smithi043_119832_object63381:head# 2024-03-30T13:50:57.547 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.547 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70444018:::benchmark_data_smithi043_119832_object67962:head# 2024-03-30T13:50:57.547 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.547 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7044a1ac:::benchmark_data_smithi043_119832_object68883:head# 2024-03-30T13:50:57.547 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.547 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7046f8db:::benchmark_data_smithi043_119832_object10074:head# 2024-03-30T13:50:57.547 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.548 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:704714b1:::benchmark_data_smithi043_119832_object24882:head# 2024-03-30T13:50:57.548 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.548 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7047f20b:::benchmark_data_smithi043_119832_object51806:head# 2024-03-30T13:50:57.548 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.548 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70483c3d:::benchmark_data_smithi043_119832_object66041:head# 2024-03-30T13:50:57.548 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.548 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70483ea8:::benchmark_data_smithi043_119832_object4470:head# 2024-03-30T13:50:57.548 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.548 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70485431:::benchmark_data_smithi043_119832_object55891:head# 2024-03-30T13:50:57.548 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.549 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70492822:::benchmark_data_smithi043_119832_object72084:head# 2024-03-30T13:50:57.549 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.549 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:704ac926:::benchmark_data_smithi043_119832_object59140:head# 2024-03-30T13:50:57.549 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.549 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:704b1226:::benchmark_data_smithi043_119832_object62507:head# 2024-03-30T13:50:57.549 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.550 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:704ba457:::benchmark_data_smithi043_119832_object32137:head# 2024-03-30T13:50:57.550 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.550 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:704e145a:::benchmark_data_smithi043_119832_object54945:head# 2024-03-30T13:50:57.550 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.550 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:704e4cbc:::benchmark_data_smithi043_119832_object48274:head# 2024-03-30T13:50:57.551 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.551 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:704e8bd2:::benchmark_data_smithi043_119832_object51696:head# 2024-03-30T13:50:57.551 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.551 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:704f462e:::benchmark_data_smithi043_119832_object41961:head# 2024-03-30T13:50:57.551 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.551 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7050dd7e:::benchmark_data_smithi043_119832_object66475:head# 2024-03-30T13:50:57.551 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.551 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70513bf9:::benchmark_data_smithi043_119832_object32250:head# 2024-03-30T13:50:57.551 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.552 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:705265a0:::benchmark_data_smithi043_119832_object37095:head# 2024-03-30T13:50:57.552 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.552 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70539985:::benchmark_data_smithi043_119832_object65407:head# 2024-03-30T13:50:57.552 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.552 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:705436f8:::benchmark_data_smithi043_119832_object51124:head# 2024-03-30T13:50:57.552 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.552 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70547be4:::benchmark_data_smithi043_119832_object15347:head# 2024-03-30T13:50:57.552 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.552 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70548204:::benchmark_data_smithi043_119832_object10563:head# 2024-03-30T13:50:57.552 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.552 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7054bbd4:::benchmark_data_smithi043_119832_object33888:head# 2024-03-30T13:50:57.552 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.554 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7054c55b:::benchmark_data_smithi043_119832_object30627:head# 2024-03-30T13:50:57.554 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.554 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7054e3c3:::benchmark_data_smithi043_119832_object48094:head# 2024-03-30T13:50:57.554 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.554 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7054fc08:::benchmark_data_smithi043_119832_object87693:head# 2024-03-30T13:50:57.554 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.554 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70552c1b:::benchmark_data_smithi043_119832_object15938:head# 2024-03-30T13:50:57.554 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.554 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7055c7d1:::benchmark_data_smithi043_119832_object27254:head# 2024-03-30T13:50:57.554 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.556 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7056f996:::benchmark_data_smithi043_119832_object46124:head# 2024-03-30T13:50:57.556 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.556 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7057b245:::benchmark_data_smithi043_119832_object70710:head# 2024-03-30T13:50:57.556 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.556 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7058d295:::benchmark_data_smithi043_119832_object17164:head# 2024-03-30T13:50:57.556 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.556 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:705a1c72:::benchmark_data_smithi043_119832_object45424:head# 2024-03-30T13:50:57.557 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.557 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:705a32c5:::benchmark_data_smithi043_119832_object55003:head# 2024-03-30T13:50:57.557 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.557 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:705a59f1:::benchmark_data_smithi043_119832_object85196:head# 2024-03-30T13:50:57.557 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.557 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:705b7ab0:::benchmark_data_smithi043_119832_object42903:head# 2024-03-30T13:50:57.557 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.557 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:705ca55f:::benchmark_data_smithi043_119832_object81810:head# 2024-03-30T13:50:57.557 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.557 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:705cc23a:::benchmark_data_smithi043_119832_object78555:head# 2024-03-30T13:50:57.557 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.557 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:705dcff6:::benchmark_data_smithi043_119832_object70008:head# 2024-03-30T13:50:57.558 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:57.558 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.558 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:705e3337:::benchmark_data_smithi043_119832_object58953:head# 2024-03-30T13:50:57.559 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.559 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:705e3500:::benchmark_data_smithi043_119832_object39816:head# 2024-03-30T13:50:57.559 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:57.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:57.562 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7060670c:::benchmark_data_smithi043_119832_object47767:head# 2024-03-30T13:50:57.563 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.563 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7060eae8:::benchmark_data_smithi043_119832_object52757:head# 2024-03-30T13:50:57.563 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.563 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70622d53:::benchmark_data_smithi043_119832_object61765:head# 2024-03-30T13:50:57.563 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.563 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7062d3b2:::benchmark_data_smithi043_119832_object2175:head# 2024-03-30T13:50:57.563 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.563 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7062de97:::benchmark_data_smithi043_119832_object11735:head# 2024-03-30T13:50:57.563 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.563 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70642dce:::benchmark_data_smithi043_119832_object84848:head# 2024-03-30T13:50:57.563 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.563 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70646b44:::benchmark_data_smithi043_119832_object12451:head# 2024-03-30T13:50:57.563 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.563 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70646c52:::benchmark_data_smithi043_119832_object21960:head# 2024-03-30T13:50:57.564 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.564 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70650196:::benchmark_data_smithi043_119832_object30814:head# 2024-03-30T13:50:57.564 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.564 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7065babf:::benchmark_data_smithi043_119832_object33364:head# 2024-03-30T13:50:57.564 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.564 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7066585c:::benchmark_data_smithi043_119832_object77356:head# 2024-03-30T13:50:57.564 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.564 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70676f11:::benchmark_data_smithi043_119832_object9682:head# 2024-03-30T13:50:57.564 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.564 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70688a1e:::benchmark_data_smithi043_119832_object727:head# 2024-03-30T13:50:57.565 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.565 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:706932f6:::benchmark_data_smithi043_119832_object24000:head# 2024-03-30T13:50:57.565 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.565 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70694f86:::benchmark_data_smithi043_119832_object64024:head# 2024-03-30T13:50:57.565 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.565 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7069b25d:::benchmark_data_smithi043_119832_object37646:head# 2024-03-30T13:50:57.565 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.567 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7069c825:::benchmark_data_smithi043_119832_object11748:head# 2024-03-30T13:50:57.567 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.567 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7069ef07:::benchmark_data_smithi043_119832_object29295:head# 2024-03-30T13:50:57.567 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.567 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:706b33e4:::benchmark_data_smithi043_119832_object59395:head# 2024-03-30T13:50:57.567 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.567 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:706cf053:::benchmark_data_smithi043_119832_object48848:head# 2024-03-30T13:50:57.567 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.567 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:706ee211:::benchmark_data_smithi043_119832_object47998:head# 2024-03-30T13:50:57.567 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.567 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:706ff8f4:::benchmark_data_smithi043_119832_object44698:head# 2024-03-30T13:50:57.567 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.568 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70703846:::benchmark_data_smithi043_119832_object56524:head# 2024-03-30T13:50:57.568 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.568 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:707117a7:::benchmark_data_smithi043_119832_object84158:head# 2024-03-30T13:50:57.568 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.568 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7071726b:::benchmark_data_smithi043_119832_object42655:head# 2024-03-30T13:50:57.568 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.568 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70718a69:::benchmark_data_smithi043_119832_object79242:head# 2024-03-30T13:50:57.568 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.568 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7072269f:::benchmark_data_smithi043_119832_object16099:head# 2024-03-30T13:50:57.568 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.568 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7073d8f8:::benchmark_data_smithi043_119832_object59024:head# 2024-03-30T13:50:57.568 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.568 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70746d21:::benchmark_data_smithi043_119832_object19859:head# 2024-03-30T13:50:57.568 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.570 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70746d6e:::benchmark_data_smithi043_119832_object63845:head# 2024-03-30T13:50:57.570 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.570 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70750a8d:::benchmark_data_smithi043_119832_object50798:head# 2024-03-30T13:50:57.570 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.570 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70751932:::benchmark_data_smithi043_119832_object39021:head# 2024-03-30T13:50:57.570 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.570 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70754ad7:::benchmark_data_smithi043_119832_object79760:head# 2024-03-30T13:50:57.570 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.570 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70754c5c:::benchmark_data_smithi043_119832_object14577:head# 2024-03-30T13:50:57.570 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.570 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7076339f:::benchmark_data_smithi043_119832_object81960:head# 2024-03-30T13:50:57.571 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.571 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7077a8da:::benchmark_data_smithi043_119832_object80898:head# 2024-03-30T13:50:57.571 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.571 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70783d8e:::benchmark_data_smithi043_119832_object8162:head# 2024-03-30T13:50:57.571 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.571 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70786f92:::benchmark_data_smithi043_119832_object62181:head# 2024-03-30T13:50:57.571 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.571 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:707bdc9a:::benchmark_data_smithi043_119832_object5120:head# 2024-03-30T13:50:57.571 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.571 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:707cdc0c:::benchmark_data_smithi043_119832_object53081:head# 2024-03-30T13:50:57.571 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.572 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:707cfbaa:::benchmark_data_smithi043_119832_object4083:head# 2024-03-30T13:50:57.572 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.573 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:707de8a5:::benchmark_data_smithi043_119832_object15869:head# 2024-03-30T13:50:57.573 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.573 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:707f961f:::benchmark_data_smithi043_119832_object57637:head# 2024-03-30T13:50:57.573 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.573 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:707fd2fd:::benchmark_data_smithi043_119832_object68309:head# 2024-03-30T13:50:57.573 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.573 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70805f0b:::benchmark_data_smithi043_119832_object77740:head# 2024-03-30T13:50:57.573 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.573 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7080a57a:::benchmark_data_smithi043_119832_object2761:head# 2024-03-30T13:50:57.573 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.573 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7084024a:::benchmark_data_smithi043_119832_object67844:head# 2024-03-30T13:50:57.574 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.574 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70845849:::benchmark_data_smithi043_119832_object38060:head# 2024-03-30T13:50:57.574 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.574 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70851472:::benchmark_data_smithi043_119832_object259:head# 2024-03-30T13:50:57.574 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.574 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7085d40b:::benchmark_data_smithi043_119832_object19132:head# 2024-03-30T13:50:57.578 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.578 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7086b525:::benchmark_data_smithi043_119832_object8398:head# 2024-03-30T13:50:57.578 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.579 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70877aaf:::benchmark_data_smithi043_119832_object72934:head# 2024-03-30T13:50:57.579 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.579 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7087bb97:::benchmark_data_smithi043_119832_object75452:head# 2024-03-30T13:50:57.579 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.579 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7087df40:::benchmark_data_smithi043_119832_object57784:head# 2024-03-30T13:50:57.579 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.580 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7087fd3a:::benchmark_data_smithi043_119832_object75481:head# 2024-03-30T13:50:57.580 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.580 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:708986e9:::benchmark_data_smithi043_119832_object79178:head# 2024-03-30T13:50:57.580 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.580 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:708aa26e:::benchmark_data_smithi043_119832_object76126:head# 2024-03-30T13:50:57.580 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.580 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:708aae08:::benchmark_data_smithi043_119832_object62427:head# 2024-03-30T13:50:57.581 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.581 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:708b5cfa:::benchmark_data_smithi043_119832_object59929:head# 2024-03-30T13:50:57.581 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.581 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:708c5c1d:::benchmark_data_smithi043_119832_object29676:head# 2024-03-30T13:50:57.581 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.581 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:708cb745:::benchmark_data_smithi043_119832_object26989:head# 2024-03-30T13:50:57.582 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.582 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:708d54b4:::benchmark_data_smithi043_119832_object10641:head# 2024-03-30T13:50:57.582 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.582 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:708ec144:::benchmark_data_smithi043_119832_object25517:head# 2024-03-30T13:50:57.582 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.582 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70917e62:::benchmark_data_smithi043_119832_object67993:head# 2024-03-30T13:50:57.583 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.587 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7092d789:::benchmark_data_smithi043_119832_object19021:head# 2024-03-30T13:50:57.587 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.588 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70938452:::benchmark_data_smithi043_119832_object69961:head# 2024-03-30T13:50:57.588 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.588 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7093dfb6:::benchmark_data_smithi043_119832_object27635:head# 2024-03-30T13:50:57.588 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.588 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7093ecd4:::benchmark_data_smithi043_119832_object88067:head# 2024-03-30T13:50:57.588 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.589 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70945205:::benchmark_data_smithi043_119832_object52379:head# 2024-03-30T13:50:57.589 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.589 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70948d97:::benchmark_data_smithi043_119832_object36752:head# 2024-03-30T13:50:57.589 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.589 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7094a7cf:::benchmark_data_smithi043_119832_object59364:head# 2024-03-30T13:50:57.589 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.590 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70950fd2:::benchmark_data_smithi043_119832_object73674:head# 2024-03-30T13:50:57.590 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.590 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:709561b0:::benchmark_data_smithi043_119832_object31682:head# 2024-03-30T13:50:57.590 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.590 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7095c2be:::benchmark_data_smithi043_119832_object64763:head# 2024-03-30T13:50:57.590 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.591 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7095ce32:::benchmark_data_smithi043_119832_object3364:head# 2024-03-30T13:50:57.591 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.591 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70968d5b:::benchmark_data_smithi043_119832_object35319:head# 2024-03-30T13:50:57.591 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.591 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7098b108:::benchmark_data_smithi043_119832_object80590:head# 2024-03-30T13:50:57.591 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.592 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7099ff83:::benchmark_data_smithi043_119832_object19991:head# 2024-03-30T13:50:57.592 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.592 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:709a0779:::benchmark_data_smithi043_119832_object67392:head# 2024-03-30T13:50:57.592 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.592 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:709a5293:::benchmark_data_smithi043_119832_object82167:head# 2024-03-30T13:50:57.592 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.595 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:709b6af8:::benchmark_data_smithi043_119832_object21404:head# 2024-03-30T13:50:57.596 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.596 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:709bda35:::benchmark_data_smithi043_119832_object3561:head# 2024-03-30T13:50:57.596 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.596 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:709c025e:::benchmark_data_smithi043_119832_object64115:head# 2024-03-30T13:50:57.596 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.596 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:709cf48a:::benchmark_data_smithi043_119832_object36693:head# 2024-03-30T13:50:57.597 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.597 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:709ddc74:::benchmark_data_smithi043_119832_object77748:head# 2024-03-30T13:50:57.597 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.597 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:709f1fa1:::benchmark_data_smithi043_119832_object10113:head# 2024-03-30T13:50:57.597 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.597 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:709f88bc:::benchmark_data_smithi043_119832_object16494:head# 2024-03-30T13:50:57.597 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.598 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70a034a4:::benchmark_data_smithi043_119832_object897:head# 2024-03-30T13:50:57.598 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.598 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70a1db6b:::benchmark_data_smithi043_119832_object48209:head# 2024-03-30T13:50:57.598 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.598 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70a38d81:::benchmark_data_smithi043_119832_object32327:head# 2024-03-30T13:50:57.599 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.601 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70a47492:::benchmark_data_smithi043_119832_object51529:head# 2024-03-30T13:50:57.601 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.601 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70a4bdcf:::benchmark_data_smithi043_119832_object38762:head# 2024-03-30T13:50:57.601 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.601 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70a50fec:::benchmark_data_smithi043_119832_object22742:head# 2024-03-30T13:50:57.602 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.602 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70a5129d:::benchmark_data_smithi043_119832_object80737:head# 2024-03-30T13:50:57.602 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.602 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70a5c00c:::benchmark_data_smithi043_119832_object34030:head# 2024-03-30T13:50:57.602 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.603 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70a6c002:::benchmark_data_smithi043_119832_object49758:head# 2024-03-30T13:50:57.603 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.603 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70a70f59:::benchmark_data_smithi043_119832_object72116:head# 2024-03-30T13:50:57.603 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.603 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70a769e9:::benchmark_data_smithi043_119832_object77992:head# 2024-03-30T13:50:57.603 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.603 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70a85cf7:::benchmark_data_smithi043_119832_object34679:head# 2024-03-30T13:50:57.604 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.606 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70a9f6a3:::benchmark_data_smithi043_119832_object14636:head# 2024-03-30T13:50:57.606 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.606 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70aa29fa:::benchmark_data_smithi043_119832_object4937:head# 2024-03-30T13:50:57.606 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.607 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70aa5d52:::benchmark_data_smithi043_119832_object48431:head# 2024-03-30T13:50:57.607 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.607 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70ab304c:::benchmark_data_smithi043_119832_object29482:head# 2024-03-30T13:50:57.607 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.607 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70ad1471:::benchmark_data_smithi043_119832_object25485:head# 2024-03-30T13:50:57.607 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.608 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70ad6cbf:::benchmark_data_smithi043_119832_object73639:head# 2024-03-30T13:50:57.608 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.608 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70add2bb:::benchmark_data_smithi043_119832_object61063:head# 2024-03-30T13:50:57.608 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.608 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70aef3d4:::benchmark_data_smithi043_119832_object5944:head# 2024-03-30T13:50:57.608 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.609 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70afc60b:::benchmark_data_smithi043_119832_object11682:head# 2024-03-30T13:50:57.609 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.610 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70afc9b6:::benchmark_data_smithi043_119832_object27669:head# 2024-03-30T13:50:57.610 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.611 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70b1bed3:::benchmark_data_smithi043_119832_object45763:head# 2024-03-30T13:50:57.611 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.613 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70b3814e:::benchmark_data_smithi043_119832_object50392:head# 2024-03-30T13:50:57.613 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.614 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70b42d59:::benchmark_data_smithi043_119832_object13368:head# 2024-03-30T13:50:57.614 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.615 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70b4f9a1:::benchmark_data_smithi043_119832_object58200:head# 2024-03-30T13:50:57.615 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.615 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70b62b19:::benchmark_data_smithi043_119832_object9968:head# 2024-03-30T13:50:57.615 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.616 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70b670fd:::benchmark_data_smithi043_119832_object79704:head# 2024-03-30T13:50:57.616 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.617 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70b94acb:::benchmark_data_smithi043_119832_object63717:head# 2024-03-30T13:50:57.617 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.617 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70baa44e:::benchmark_data_smithi043_119832_object45230:head# 2024-03-30T13:50:57.618 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.618 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70bb1389:::benchmark_data_smithi043_119832_object59065:head# 2024-03-30T13:50:57.618 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.619 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70bbfd3b:::benchmark_data_smithi043_119832_object7711:head# 2024-03-30T13:50:57.619 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.620 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70bc2998:::benchmark_data_smithi043_119832_object6554:head# 2024-03-30T13:50:57.620 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.620 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70bc75ae:::benchmark_data_smithi043_119832_object48290:head# 2024-03-30T13:50:57.620 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.620 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70bc9ebb:::benchmark_data_smithi043_119832_object22082:head# 2024-03-30T13:50:57.621 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.623 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70be30af:::benchmark_data_smithi043_119832_object78392:head# 2024-03-30T13:50:57.623 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.623 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70be7634:::benchmark_data_smithi043_119832_object10794:head# 2024-03-30T13:50:57.623 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.623 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70c087a4:::benchmark_data_smithi043_119832_object57937:head# 2024-03-30T13:50:57.623 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.623 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70c10e1c:::benchmark_data_smithi043_119832_object66540:head# 2024-03-30T13:50:57.623 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.623 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70c2f2c9:::benchmark_data_smithi043_119832_object86996:head# 2024-03-30T13:50:57.624 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.624 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70c31d98:::benchmark_data_smithi043_119832_object35299:head# 2024-03-30T13:50:57.624 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.625 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70c3f9ae:::benchmark_data_smithi043_119832_object82018:head# 2024-03-30T13:50:57.626 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.626 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70c55ecc:::benchmark_data_smithi043_119832_object44192:head# 2024-03-30T13:50:57.626 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.628 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70c56df9:::benchmark_data_smithi043_119832_object65020:head# 2024-03-30T13:50:57.628 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.628 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70c6de3f:::benchmark_data_smithi043_119832_object21705:head# 2024-03-30T13:50:57.628 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.629 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70c7d9c5:::benchmark_data_smithi043_119832_object2143:head# 2024-03-30T13:50:57.629 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.629 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70c86faf:::benchmark_data_smithi043_119832_object16047:head# 2024-03-30T13:50:57.629 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.629 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70c8c0c6:::benchmark_data_smithi043_119832_object912:head# 2024-03-30T13:50:57.630 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.630 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70c979a7:::benchmark_data_smithi043_119832_object88175:head# 2024-03-30T13:50:57.630 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.630 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70c99758:::benchmark_data_smithi043_119832_object43887:head# 2024-03-30T13:50:57.630 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.632 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70c9ca50:::benchmark_data_smithi043_119832_object58743:head# 2024-03-30T13:50:57.633 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.633 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70cafd2a:::benchmark_data_smithi043_119832_object76917:head# 2024-03-30T13:50:57.633 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.633 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70cb4c2f:::benchmark_data_smithi043_119832_object4345:head# 2024-03-30T13:50:57.633 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.633 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70cb91d5:::benchmark_data_smithi043_119832_object43790:head# 2024-03-30T13:50:57.634 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.634 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70cb9f74:::benchmark_data_smithi043_119832_object55391:head# 2024-03-30T13:50:57.634 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.634 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70cca98b:::benchmark_data_smithi043_119832_object69819:head# 2024-03-30T13:50:57.634 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.635 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70cce172:::benchmark_data_smithi043_119832_object13764:head# 2024-03-30T13:50:57.635 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.635 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70ce08ed:::benchmark_data_smithi043_119832_object49618:head# 2024-03-30T13:50:57.635 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.635 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70ce2460:::benchmark_data_smithi043_119832_object20776:head# 2024-03-30T13:50:57.635 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.635 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70ce2e1b:::benchmark_data_smithi043_119832_object72936:head# 2024-03-30T13:50:57.635 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.636 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70ce78ca:::benchmark_data_smithi043_119832_object38090:head# 2024-03-30T13:50:57.636 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.637 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70cebad9:::benchmark_data_smithi043_119832_object5897:head# 2024-03-30T13:50:57.637 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.638 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70d10792:::benchmark_data_smithi043_119832_object83269:head# 2024-03-30T13:50:57.638 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.638 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70d15097:::benchmark_data_smithi043_119832_object71872:head# 2024-03-30T13:50:57.638 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.638 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70d19ece:::benchmark_data_smithi043_119832_object34017:head# 2024-03-30T13:50:57.638 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.638 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70d3443f:::benchmark_data_smithi043_119832_object18548:head# 2024-03-30T13:50:57.638 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.638 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70d3cdc6:::benchmark_data_smithi043_119832_object55617:head# 2024-03-30T13:50:57.638 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.638 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70d47814:::benchmark_data_smithi043_119832_object26067:head# 2024-03-30T13:50:57.638 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.639 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70d5d79f:::benchmark_data_smithi043_119832_object87366:head# 2024-03-30T13:50:57.639 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.639 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70d6a086:::benchmark_data_smithi043_119832_object47062:head# 2024-03-30T13:50:57.639 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.639 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70d99e65:::benchmark_data_smithi043_119832_object20407:head# 2024-03-30T13:50:57.639 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.640 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70da1ae2:::benchmark_data_smithi043_119832_object75461:head# 2024-03-30T13:50:57.640 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.640 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70da627c:::benchmark_data_smithi043_119832_object54673:head# 2024-03-30T13:50:57.640 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.640 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70da7361:::benchmark_data_smithi043_119832_object37378:head# 2024-03-30T13:50:57.640 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.640 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70daf31c:::benchmark_data_smithi043_119832_object53232:head# 2024-03-30T13:50:57.640 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.642 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70dc94ee:::benchmark_data_smithi043_119832_object61401:head# 2024-03-30T13:50:57.642 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.642 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70dcb1c6:::benchmark_data_smithi043_119832_object33503:head# 2024-03-30T13:50:57.642 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.643 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70e042a5:::benchmark_data_smithi043_119832_object20107:head# 2024-03-30T13:50:57.643 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.643 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70e0e0d4:::benchmark_data_smithi043_119832_object80393:head# 2024-03-30T13:50:57.643 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.644 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70e1ba07:::benchmark_data_smithi043_119832_object19574:head# 2024-03-30T13:50:57.644 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.644 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70e1d288:::benchmark_data_smithi043_119832_object9599:head# 2024-03-30T13:50:57.644 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.644 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70e206ba:::benchmark_data_smithi043_119832_object7001:head# 2024-03-30T13:50:57.644 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.645 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70e219e1:::benchmark_data_smithi043_119832_object81061:head# 2024-03-30T13:50:57.645 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.645 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70e26308:::benchmark_data_smithi043_119832_object23998:head# 2024-03-30T13:50:57.646 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.646 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70e29d14:::benchmark_data_smithi043_119832_object32775:head# 2024-03-30T13:50:57.646 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.647 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70e38142:::benchmark_data_smithi043_119832_object24846:head# 2024-03-30T13:50:57.647 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.647 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70e4587a:::benchmark_data_smithi043_119832_object88453:head# 2024-03-30T13:50:57.647 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.647 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70e5e6ef:::benchmark_data_smithi043_119832_object82622:head# 2024-03-30T13:50:57.647 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.647 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70e62442:::benchmark_data_smithi043_119832_object21132:head# 2024-03-30T13:50:57.648 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.648 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70e73619:::benchmark_data_smithi043_119832_object55971:head# 2024-03-30T13:50:57.648 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.648 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70e882ce:::benchmark_data_smithi043_119832_object34507:head# 2024-03-30T13:50:57.648 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.648 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70e8e27c:::benchmark_data_smithi043_119832_object51036:head# 2024-03-30T13:50:57.648 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.649 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70ead256:::benchmark_data_smithi043_119832_object34228:head# 2024-03-30T13:50:57.650 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.650 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70eb5d03:::benchmark_data_smithi043_119832_object1108:head# 2024-03-30T13:50:57.650 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.650 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70ebf3ec:::benchmark_data_smithi043_119832_object47705:head# 2024-03-30T13:50:57.650 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.650 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70edfb2a:::benchmark_data_smithi043_119832_object34899:head# 2024-03-30T13:50:57.650 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.651 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70ee2108:::benchmark_data_smithi043_119832_object20191:head# 2024-03-30T13:50:57.651 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.651 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70ee7917:::benchmark_data_smithi043_119832_object5337:head# 2024-03-30T13:50:57.651 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.651 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70f02eca:::benchmark_data_smithi043_119832_object16048:head# 2024-03-30T13:50:57.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:57 smithi043 ceph-mon[29963]: 8.2 scrub starts 2024-03-30T13:50:57.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:57 smithi043 ceph-mon[29963]: 3.1b scrub starts 2024-03-30T13:50:57.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:57 smithi043 ceph-mon[29963]: 3.1b scrub ok 2024-03-30T13:50:57.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:57 smithi043 ceph-mon[29963]: 5.c scrub starts 2024-03-30T13:50:57.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:57 smithi043 ceph-mon[29963]: 5.c scrub ok 2024-03-30T13:50:57.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:57 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 44241/354549 objects degraded (12.478%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:57.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:57 smithi043 ceph-mon[29963]: osdmap e182: 8 total, 7 up, 8 in 2024-03-30T13:50:57.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:57 smithi043 ceph-mon[42099]: 8.2 scrub starts 2024-03-30T13:50:57.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:57 smithi043 ceph-mon[42099]: 3.1b scrub starts 2024-03-30T13:50:57.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:57 smithi043 ceph-mon[42099]: 3.1b scrub ok 2024-03-30T13:50:57.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:57 smithi043 ceph-mon[42099]: 5.c scrub starts 2024-03-30T13:50:57.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:57 smithi043 ceph-mon[42099]: 5.c scrub ok 2024-03-30T13:50:57.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:57 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 44241/354549 objects degraded (12.478%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:57.653 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:57 smithi043 ceph-mon[42099]: osdmap e182: 8 total, 7 up, 8 in 2024-03-30T13:50:57.655 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.655 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70f138f7:::benchmark_data_smithi043_119832_object26823:head# 2024-03-30T13:50:57.655 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.655 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70f29105:::benchmark_data_smithi043_119832_object38709:head# 2024-03-30T13:50:57.655 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.655 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70f31c31:::benchmark_data_smithi043_119832_object46513:head# 2024-03-30T13:50:57.655 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.655 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70f3f507:::benchmark_data_smithi043_119832_object41724:head# 2024-03-30T13:50:57.655 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.656 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70f5c6a8:::benchmark_data_smithi043_119832_object62709:head# 2024-03-30T13:50:57.656 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.656 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70f5ce29:::benchmark_data_smithi043_119832_object53902:head# 2024-03-30T13:50:57.656 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.656 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70f5ed6e:::benchmark_data_smithi043_119832_object54328:head# 2024-03-30T13:50:57.656 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.656 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70f6e0a9:::benchmark_data_smithi043_119832_object16385:head# 2024-03-30T13:50:57.656 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.656 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70f82676:::benchmark_data_smithi043_119832_object1196:head# 2024-03-30T13:50:57.656 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.656 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70f83397:::benchmark_data_smithi043_119832_object47870:head# 2024-03-30T13:50:57.656 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.656 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70f9942c:::benchmark_data_smithi043_119832_object1709:head# 2024-03-30T13:50:57.656 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.658 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70f9ddba:::benchmark_data_smithi043_119832_object62744:head# 2024-03-30T13:50:57.658 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.658 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70fb24e1:::benchmark_data_smithi043_119832_object78848:head# 2024-03-30T13:50:57.658 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.658 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70fbf716:::benchmark_data_smithi043_119832_object70397:head# 2024-03-30T13:50:57.658 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.658 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70fcdeac:::benchmark_data_smithi043_119832_object26237:head# 2024-03-30T13:50:57.658 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.658 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70ff8460:::benchmark_data_smithi043_119832_object76464:head# 2024-03-30T13:50:57.658 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.659 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:70ffef75:::benchmark_data_smithi043_119832_object85648:head# 2024-03-30T13:50:57.659 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.659 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:710016fe:::benchmark_data_smithi043_119832_object54701:head# 2024-03-30T13:50:57.659 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.659 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7101010a:::benchmark_data_smithi043_119832_object10297:head# 2024-03-30T13:50:57.659 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.660 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71022245:::benchmark_data_smithi043_119832_object76540:head# 2024-03-30T13:50:57.660 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.660 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7104bb68:::benchmark_data_smithi043_119832_object7549:head# 2024-03-30T13:50:57.660 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.660 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:57.662 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7107470e:::benchmark_data_smithi043_119832_object88377:head# 2024-03-30T13:50:57.662 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.662 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7107bf2b:::benchmark_data_smithi043_119832_object7467:head# 2024-03-30T13:50:57.662 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.662 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71094308:::benchmark_data_smithi043_119832_object52532:head# 2024-03-30T13:50:57.662 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.666 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:710956f8:::benchmark_data_smithi043_119832_object27462:head# 2024-03-30T13:50:57.666 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.666 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:710990e7:::benchmark_data_smithi043_119832_object45600:head# 2024-03-30T13:50:57.666 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.666 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71099a65:::benchmark_data_smithi043_119832_object64074:head# 2024-03-30T13:50:57.666 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.666 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:710a5ba4:::benchmark_data_smithi043_119832_object7533:head# 2024-03-30T13:50:57.666 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.667 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:710b6f72:::benchmark_data_smithi043_119832_object81996:head# 2024-03-30T13:50:57.667 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.667 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:710bc09f:::benchmark_data_smithi043_119832_object36925:head# 2024-03-30T13:50:57.667 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.667 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:710c014e:::benchmark_data_smithi043_119832_object49760:head# 2024-03-30T13:50:57.667 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.667 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:710c3a4f:::benchmark_data_smithi043_119832_object12763:head# 2024-03-30T13:50:57.667 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.667 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:710ca45b:::benchmark_data_smithi043_119832_object21226:head# 2024-03-30T13:50:57.667 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.667 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:710def2d:::benchmark_data_smithi043_119832_object11462:head# 2024-03-30T13:50:57.667 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.668 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:710df8b5:::benchmark_data_smithi043_119832_object72039:head# 2024-03-30T13:50:57.668 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.668 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:710ffa5c:::benchmark_data_smithi043_119832_object19437:head# 2024-03-30T13:50:57.668 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.668 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:711122cb:::benchmark_data_smithi043_119832_object11569:head# 2024-03-30T13:50:57.669 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.669 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7111bfde:::benchmark_data_smithi043_119832_object30126:head# 2024-03-30T13:50:57.669 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.669 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7111dc0d:::benchmark_data_smithi043_119832_object46779:head# 2024-03-30T13:50:57.669 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.669 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:711373a6:::benchmark_data_smithi043_119832_object87265:head# 2024-03-30T13:50:57.669 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.669 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7113f233:::benchmark_data_smithi043_119832_object17894:head# 2024-03-30T13:50:57.670 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.670 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:711490cf:::benchmark_data_smithi043_119832_object21673:head# 2024-03-30T13:50:57.670 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.670 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7115ef30:::benchmark_data_smithi043_119832_object14694:head# 2024-03-30T13:50:57.670 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.670 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7115fc4d:::benchmark_data_smithi043_119832_object28441:head# 2024-03-30T13:50:57.671 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.671 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7119a60d:::benchmark_data_smithi043_119832_object33770:head# 2024-03-30T13:50:57.671 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.671 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:711a3e7f:::benchmark_data_smithi043_119832_object10336:head# 2024-03-30T13:50:57.671 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.672 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:711b328c:::benchmark_data_smithi043_119832_object42876:head# 2024-03-30T13:50:57.672 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.672 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:711b43a7:::benchmark_data_smithi043_119832_object6477:head# 2024-03-30T13:50:57.672 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.672 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:711c37de:::benchmark_data_smithi043_119832_object83531:head# 2024-03-30T13:50:57.672 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.673 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:711c4fb3:::benchmark_data_smithi043_119832_object49139:head# 2024-03-30T13:50:57.673 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.673 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:711d0cd8:::benchmark_data_smithi043_119832_object4107:head# 2024-03-30T13:50:57.673 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.673 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:711d64f0:::benchmark_data_smithi043_119832_object36049:head# 2024-03-30T13:50:57.674 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.674 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:711eec02:::benchmark_data_smithi043_119832_object58892:head# 2024-03-30T13:50:57.674 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.675 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:711f4e2e:::benchmark_data_smithi043_119832_object4698:head# 2024-03-30T13:50:57.675 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.676 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:711fdb50:::benchmark_data_smithi043_119832_object11468:head# 2024-03-30T13:50:57.676 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.677 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7120421e:::benchmark_data_smithi043_119832_object51142:head# 2024-03-30T13:50:57.677 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.678 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71205f7a:::benchmark_data_smithi043_119832_object5065:head# 2024-03-30T13:50:57.678 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.679 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:712095e6:::benchmark_data_smithi043_119832_object65099:head# 2024-03-30T13:50:57.679 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.680 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71212c97:::benchmark_data_smithi043_119832_object59864:head# 2024-03-30T13:50:57.680 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.680 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71214d5e:::benchmark_data_smithi043_119832_object83756:head# 2024-03-30T13:50:57.680 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.680 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71217fdd:::benchmark_data_smithi043_119832_object11549:head# 2024-03-30T13:50:57.680 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.680 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7121b09e:::benchmark_data_smithi043_119832_object76525:head# 2024-03-30T13:50:57.681 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.681 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T13:50:57.682 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7121fa92:::benchmark_data_smithi043_119832_object67585:head# 2024-03-30T13:50:57.682 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.682 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7123ed1f:::benchmark_data_smithi043_119832_object34320:head# 2024-03-30T13:50:57.682 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.682 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:712444ff:::benchmark_data_smithi043_119832_object60733:head# 2024-03-30T13:50:57.682 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.682 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:712477ee:::benchmark_data_smithi043_119832_object24768:head# 2024-03-30T13:50:57.683 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.683 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:712493f3:::benchmark_data_smithi043_119832_object28693:head# 2024-03-30T13:50:57.683 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.683 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7125882a:::benchmark_data_smithi043_119832_object46992:head# 2024-03-30T13:50:57.683 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.683 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7125c0b8:::benchmark_data_smithi043_119832_object3089:head# 2024-03-30T13:50:57.683 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.684 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71272364:::benchmark_data_smithi043_119832_object22552:head# 2024-03-30T13:50:57.684 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.684 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71274fd7:::benchmark_data_smithi043_119832_object6305:head# 2024-03-30T13:50:57.685 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.685 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:712774b5:::benchmark_data_smithi043_119832_object37311:head# 2024-03-30T13:50:57.685 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.685 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:712828c8:::benchmark_data_smithi043_119832_object45857:head# 2024-03-30T13:50:57.685 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.685 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:712865c6:::benchmark_data_smithi043_119832_object8731:head# 2024-03-30T13:50:57.685 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.685 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7128ccf8:::benchmark_data_smithi043_119832_object51690:head# 2024-03-30T13:50:57.686 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.686 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:712aa8ab:::benchmark_data_smithi043_119832_object33800:head# 2024-03-30T13:50:57.686 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.686 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:712afd7c:::benchmark_data_smithi043_119832_object24238:head# 2024-03-30T13:50:57.686 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.688 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:712b7563:::benchmark_data_smithi043_119832_object80332:head# 2024-03-30T13:50:57.688 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.688 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:712c8ca0:::benchmark_data_smithi043_119832_object21965:head# 2024-03-30T13:50:57.688 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.688 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:712cc413:::benchmark_data_smithi043_119832_object15303:head# 2024-03-30T13:50:57.688 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.688 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:712cd225:::benchmark_data_smithi043_119832_object69661:head# 2024-03-30T13:50:57.688 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.688 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:712ebae4:::benchmark_data_smithi043_119832_object79379:head# 2024-03-30T13:50:57.688 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.688 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71300ffa:::benchmark_data_smithi043_119832_object15979:head# 2024-03-30T13:50:57.689 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.689 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71320613:::benchmark_data_smithi043_119832_object30974:head# 2024-03-30T13:50:57.689 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.690 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71320a28:::benchmark_data_smithi043_119832_object4059:head# 2024-03-30T13:50:57.690 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.690 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7133a44b:::benchmark_data_smithi043_119832_object59885:head# 2024-03-30T13:50:57.690 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.690 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7133cd58:::benchmark_data_smithi043_119832_object71162:head# 2024-03-30T13:50:57.690 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.690 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7134ff20:::benchmark_data_smithi043_119832_object56807:head# 2024-03-30T13:50:57.690 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.691 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7135db38:::benchmark_data_smithi043_119832_object58720:head# 2024-03-30T13:50:57.691 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.691 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71360cb2:::benchmark_data_smithi043_119832_object75923:head# 2024-03-30T13:50:57.691 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.691 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71362067:::benchmark_data_smithi043_119832_object34116:head# 2024-03-30T13:50:57.691 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.691 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71372b28:::benchmark_data_smithi043_119832_object31587:head# 2024-03-30T13:50:57.691 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.692 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71376e76:::benchmark_data_smithi043_119832_object15543:head# 2024-03-30T13:50:57.692 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.692 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7138cab5:::benchmark_data_smithi043_119832_object21335:head# 2024-03-30T13:50:57.692 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.692 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7139089e:::benchmark_data_smithi043_119832_object547:head# 2024-03-30T13:50:57.692 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.692 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:713a8b79:::benchmark_data_smithi043_119832_object8771:head# 2024-03-30T13:50:57.692 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.693 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:713a9566:::benchmark_data_smithi043_119832_object72205:head# 2024-03-30T13:50:57.693 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.693 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:713bad7d:::benchmark_data_smithi043_119832_object72540:head# 2024-03-30T13:50:57.694 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.694 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:713c6b4d:::benchmark_data_smithi043_119832_object15619:head# 2024-03-30T13:50:57.694 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.694 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:713d7d44:::benchmark_data_smithi043_119832_object52742:head# 2024-03-30T13:50:57.694 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.695 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:713da36d:::benchmark_data_smithi043_119832_object45157:head# 2024-03-30T13:50:57.695 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.695 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:713da72c:::benchmark_data_smithi043_119832_object4799:head# 2024-03-30T13:50:57.695 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.695 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:713da835:::benchmark_data_smithi043_119832_object20280:head# 2024-03-30T13:50:57.695 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.695 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:713db775:::benchmark_data_smithi043_119832_object57322:head# 2024-03-30T13:50:57.695 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.696 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:713e1b03:::benchmark_data_smithi043_119832_object30206:head# 2024-03-30T13:50:57.696 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.696 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:713e788a:::benchmark_data_smithi043_119832_object59252:head# 2024-03-30T13:50:57.696 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.696 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:713ec4e0:::benchmark_data_smithi043_119832_object39617:head# 2024-03-30T13:50:57.696 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.697 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:713ed01a:::benchmark_data_smithi043_119832_object5130:head# 2024-03-30T13:50:57.697 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.697 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:713f2f89:::benchmark_data_smithi043_119832_object7468:head# 2024-03-30T13:50:57.697 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.697 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71404ad2:::benchmark_data_smithi043_119832_object79162:head# 2024-03-30T13:50:57.698 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.699 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7140644d:::benchmark_data_smithi043_119832_object31137:head# 2024-03-30T13:50:57.699 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.700 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71406837:::benchmark_data_smithi043_119832_object64817:head# 2024-03-30T13:50:57.700 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.700 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71411166:::benchmark_data_smithi043_119832_object24962:head# 2024-03-30T13:50:57.700 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.700 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7141359a:::benchmark_data_smithi043_119832_object36428:head# 2024-03-30T13:50:57.700 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.700 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7142ae08:::benchmark_data_smithi043_119832_object15807:head# 2024-03-30T13:50:57.700 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.700 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:714310a2:::benchmark_data_smithi043_119832_object48425:head# 2024-03-30T13:50:57.700 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.700 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71432a5d:::benchmark_data_smithi043_119832_object67009:head# 2024-03-30T13:50:57.700 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.701 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71437ac5:::benchmark_data_smithi043_119832_object22057:head# 2024-03-30T13:50:57.701 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.701 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7143ded1:::benchmark_data_smithi043_119832_object15458:head# 2024-03-30T13:50:57.701 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.701 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71449bb7:::benchmark_data_smithi043_119832_object45867:head# 2024-03-30T13:50:57.701 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.701 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7144bc10:::benchmark_data_smithi043_119832_object58667:head# 2024-03-30T13:50:57.702 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.702 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7144da3c:::benchmark_data_smithi043_119832_object65589:head# 2024-03-30T13:50:57.702 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.702 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7144edad:::benchmark_data_smithi043_119832_object37121:head# 2024-03-30T13:50:57.702 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.702 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71452f52:::benchmark_data_smithi043_119832_object62860:head# 2024-03-30T13:50:57.703 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.703 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71457a4a:::benchmark_data_smithi043_119832_object84600:head# 2024-03-30T13:50:57.703 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.703 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7145a23c:::benchmark_data_smithi043_119832_object67517:head# 2024-03-30T13:50:57.703 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.703 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7145a9ae:::benchmark_data_smithi043_119832_object31876:head# 2024-03-30T13:50:57.703 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.703 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7145b26a:::benchmark_data_smithi043_119832_object57261:head# 2024-03-30T13:50:57.704 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.704 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7145c1e3:::benchmark_data_smithi043_119832_object85969:head# 2024-03-30T13:50:57.704 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.704 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:57.705 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71464e84:::benchmark_data_smithi043_119832_object72564:head# 2024-03-30T13:50:57.705 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:57.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:57.706 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71471de9:::benchmark_data_smithi043_119832_object45630:head# 2024-03-30T13:50:57.706 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.706 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7148e5d6:::benchmark_data_smithi043_119832_object8775:head# 2024-03-30T13:50:57.707 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.707 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7148eb83:::benchmark_data_smithi043_119832_object7511:head# 2024-03-30T13:50:57.707 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.707 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7148f9e2:::benchmark_data_smithi043_119832_object13893:head# 2024-03-30T13:50:57.707 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.707 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:714a96b4:::benchmark_data_smithi043_119832_object2644:head# 2024-03-30T13:50:57.707 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.707 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:714ab284:::benchmark_data_smithi043_119832_object26243:head# 2024-03-30T13:50:57.707 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.708 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:714c405c:::benchmark_data_smithi043_119832_object68341:head# 2024-03-30T13:50:57.708 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.708 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:714d241c:::benchmark_data_smithi043_119832_object69106:head# 2024-03-30T13:50:57.708 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.708 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:714db556:::benchmark_data_smithi043_119832_object46592:head# 2024-03-30T13:50:57.708 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.708 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:714eca97:::benchmark_data_smithi043_119832_object56452:head# 2024-03-30T13:50:57.708 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.708 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:714fef1b:::benchmark_data_smithi043_119832_object38516:head# 2024-03-30T13:50:57.708 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.709 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71504dee:::benchmark_data_smithi043_119832_object22041:head# 2024-03-30T13:50:57.709 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.709 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7151c032:::benchmark_data_smithi043_119832_object80040:head# 2024-03-30T13:50:57.710 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.710 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71534eed:::benchmark_data_smithi043_119832_object82424:head# 2024-03-30T13:50:57.710 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.710 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7153cc72:::benchmark_data_smithi043_119832_object68307:head# 2024-03-30T13:50:57.710 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.710 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:715478aa:::benchmark_data_smithi043_119832_object25084:head# 2024-03-30T13:50:57.710 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.711 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7155f84d:::benchmark_data_smithi043_119832_object16639:head# 2024-03-30T13:50:57.711 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.711 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71563390:::benchmark_data_smithi043_119832_object1237:head# 2024-03-30T13:50:57.711 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.714 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:715656c8:::benchmark_data_smithi043_119832_object30577:head# 2024-03-30T13:50:57.714 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.714 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7156a402:::benchmark_data_smithi043_119832_object56146:head# 2024-03-30T13:50:57.714 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.714 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71574e68:::benchmark_data_smithi043_119832_object61042:head# 2024-03-30T13:50:57.714 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.715 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71575bb2:::benchmark_data_smithi043_119832_object79695:head# 2024-03-30T13:50:57.715 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.715 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71576283:::benchmark_data_smithi043_119832_object21717:head# 2024-03-30T13:50:57.715 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.715 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7157ca54:::benchmark_data_smithi043_119832_object17332:head# 2024-03-30T13:50:57.715 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.715 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7157d061:::benchmark_data_smithi043_119832_object18703:head# 2024-03-30T13:50:57.715 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.715 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71594ee6:::benchmark_data_smithi043_119832_object10725:head# 2024-03-30T13:50:57.715 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.715 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7159cb0b:::benchmark_data_smithi043_119832_object86669:head# 2024-03-30T13:50:57.715 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.715 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:715ab001:::benchmark_data_smithi043_119832_object70849:head# 2024-03-30T13:50:57.715 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.716 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:715c3c6f:::benchmark_data_smithi043_119832_object21564:head# 2024-03-30T13:50:57.716 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.716 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:715d8b87:::benchmark_data_smithi043_119832_object16933:head# 2024-03-30T13:50:57.716 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.716 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:715dc948:::benchmark_data_smithi043_119832_object12669:head# 2024-03-30T13:50:57.716 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.716 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:715f9647:::benchmark_data_smithi043_119832_object4484:head# 2024-03-30T13:50:57.716 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.716 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:716073fb:::benchmark_data_smithi043_119832_object67906:head# 2024-03-30T13:50:57.717 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.717 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:716137bf:::benchmark_data_smithi043_119832_object17099:head# 2024-03-30T13:50:57.717 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.717 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7161bedd:::benchmark_data_smithi043_119832_object74331:head# 2024-03-30T13:50:57.717 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.717 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71628fdd:::benchmark_data_smithi043_119832_object17188:head# 2024-03-30T13:50:57.717 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.718 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7162b9d5:::benchmark_data_smithi043_119832_object86332:head# 2024-03-30T13:50:57.718 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.718 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7165d25f:::benchmark_data_smithi043_119832_object61692:head# 2024-03-30T13:50:57.718 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.719 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7166d261:::benchmark_data_smithi043_119832_object45925:head# 2024-03-30T13:50:57.719 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.719 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:716752f6:::benchmark_data_smithi043_119832_object88251:head# 2024-03-30T13:50:57.719 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.719 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71675d02:::benchmark_data_smithi043_119832_object25737:head# 2024-03-30T13:50:57.719 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.720 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71675fd5:::benchmark_data_smithi043_119832_object73009:head# 2024-03-30T13:50:57.720 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.720 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7169a490:::benchmark_data_smithi043_119832_object59279:head# 2024-03-30T13:50:57.720 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.721 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7169e685:::benchmark_data_smithi043_119832_object17952:head# 2024-03-30T13:50:57.721 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.721 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:716a3f24:::benchmark_data_smithi043_119832_object53361:head# 2024-03-30T13:50:57.721 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.721 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:716a793d:::benchmark_data_smithi043_119832_object1193:head# 2024-03-30T13:50:57.721 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.722 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:716ace89:::benchmark_data_smithi043_119832_object51542:head# 2024-03-30T13:50:57.722 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.722 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:716ca3ad:::benchmark_data_smithi043_119832_object65372:head# 2024-03-30T13:50:57.722 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.722 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:716d8bc9:::benchmark_data_smithi043_119832_object62653:head# 2024-03-30T13:50:57.723 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.723 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:716de828:::benchmark_data_smithi043_119832_object60030:head# 2024-03-30T13:50:57.723 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.723 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:716e4864:::benchmark_data_smithi043_119832_object56954:head# 2024-03-30T13:50:57.723 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.724 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:716ef3b3:::benchmark_data_smithi043_119832_object86156:head# 2024-03-30T13:50:57.724 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.724 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7170629b:::benchmark_data_smithi043_119832_object4990:head# 2024-03-30T13:50:57.724 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.724 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:717094ac:::benchmark_data_smithi043_119832_object27454:head# 2024-03-30T13:50:57.724 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.725 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7171b99e:::benchmark_data_smithi043_119832_object60432:head# 2024-03-30T13:50:57.725 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.725 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71724304:::benchmark_data_smithi043_119832_object84739:head# 2024-03-30T13:50:57.725 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.725 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71734e2c:::benchmark_data_smithi043_119832_object12924:head# 2024-03-30T13:50:57.725 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.726 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7173b2d7:::benchmark_data_smithi043_119832_object15848:head# 2024-03-30T13:50:57.726 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.726 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7174a0f2:::benchmark_data_smithi043_119832_object76932:head# 2024-03-30T13:50:57.726 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.727 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7174a3e2:::benchmark_data_smithi043_119832_object28306:head# 2024-03-30T13:50:57.727 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.727 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7175468d:::benchmark_data_smithi043_119832_object7827:head# 2024-03-30T13:50:57.727 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.727 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7175635f:::benchmark_data_smithi043_119832_object72170:head# 2024-03-30T13:50:57.727 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.728 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7176bf93:::benchmark_data_smithi043_119832_object76490:head# 2024-03-30T13:50:57.728 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.728 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:717707da:::benchmark_data_smithi043_119832_object72399:head# 2024-03-30T13:50:57.728 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.729 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7177b760:::benchmark_data_smithi043_119832_object415:head# 2024-03-30T13:50:57.729 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.729 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7177f0ad:::benchmark_data_smithi043_119832_object52042:head# 2024-03-30T13:50:57.729 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.729 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71782385:::benchmark_data_smithi043_119832_object60641:head# 2024-03-30T13:50:57.729 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.730 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:717916bd:::benchmark_data_smithi043_119832_object7714:head# 2024-03-30T13:50:57.730 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.730 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:717996ae:::benchmark_data_smithi043_119832_object22226:head# 2024-03-30T13:50:57.730 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.730 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7179c933:::benchmark_data_smithi043_119832_object42231:head# 2024-03-30T13:50:57.730 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.730 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7179d6b6:::benchmark_data_smithi043_119832_object85814:head# 2024-03-30T13:50:57.731 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.731 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:717b9fd2:::benchmark_data_smithi043_119832_object23815:head# 2024-03-30T13:50:57.731 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.731 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:717c53c2:::benchmark_data_smithi043_119832_object23186:head# 2024-03-30T13:50:57.731 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.732 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:717c5e1f:::benchmark_data_smithi043_119832_object69156:head# 2024-03-30T13:50:57.732 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.733 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:717cf5e3:::benchmark_data_smithi043_119832_object18851:head# 2024-03-30T13:50:57.733 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.733 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:717e6271:::benchmark_data_smithi043_119832_object83323:head# 2024-03-30T13:50:57.733 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.733 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:717ff251:::benchmark_data_smithi043_119832_object55367:head# 2024-03-30T13:50:57.733 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.733 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71802c64:::benchmark_data_smithi043_119832_object10978:head# 2024-03-30T13:50:57.733 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.734 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:718108d6:::benchmark_data_smithi043_119832_object66142:head# 2024-03-30T13:50:57.734 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.734 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7181787e:::benchmark_data_smithi043_119832_object63630:head# 2024-03-30T13:50:57.734 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.734 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7183150d:::benchmark_data_smithi043_119832_object56386:head# 2024-03-30T13:50:57.734 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.735 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71843250:::benchmark_data_smithi043_119832_object49605:head# 2024-03-30T13:50:57.735 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.735 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:718463e2:::benchmark_data_smithi043_119832_object29536:head# 2024-03-30T13:50:57.735 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.735 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71859183:::benchmark_data_smithi043_119832_object68907:head# 2024-03-30T13:50:57.735 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.736 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71865853:::benchmark_data_smithi043_119832_object10575:head# 2024-03-30T13:50:57.736 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.736 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7186ce50:::benchmark_data_smithi043_119832_object77351:head# 2024-03-30T13:50:57.736 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.736 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:718790f0:::benchmark_data_smithi043_119832_object5090:head# 2024-03-30T13:50:57.737 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.737 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:718876cd:::benchmark_data_smithi043_119832_object63638:head# 2024-03-30T13:50:57.737 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.737 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:718adc57:::benchmark_data_smithi043_119832_object67986:head# 2024-03-30T13:50:57.737 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.737 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:718d06dd:::benchmark_data_smithi043_119832_object79693:head# 2024-03-30T13:50:57.737 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.738 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:718d1750:::benchmark_data_smithi043_119832_object31131:head# 2024-03-30T13:50:57.738 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.738 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:718e81ac:::benchmark_data_smithi043_119832_object35318:head# 2024-03-30T13:50:57.738 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.739 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:718ec4a4:::benchmark_data_smithi043_119832_object40143:head# 2024-03-30T13:50:57.739 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.739 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:718ec887:::benchmark_data_smithi043_119832_object2097:head# 2024-03-30T13:50:57.739 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.739 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:718f27c6:::benchmark_data_smithi043_119832_object51234:head# 2024-03-30T13:50:57.739 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.739 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:718f5ea9:::benchmark_data_smithi043_119832_object39236:head# 2024-03-30T13:50:57.740 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.740 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71908699:::benchmark_data_smithi043_119832_object42161:head# 2024-03-30T13:50:57.740 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.740 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71931b77:::benchmark_data_smithi043_119832_object27294:head# 2024-03-30T13:50:57.740 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.741 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71937fb0:::benchmark_data_smithi043_119832_object51533:head# 2024-03-30T13:50:57.741 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.741 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:719384de:::benchmark_data_smithi043_119832_object22402:head# 2024-03-30T13:50:57.741 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.741 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71938604:::benchmark_data_smithi043_119832_object76667:head# 2024-03-30T13:50:57.742 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.742 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7193eeda:::benchmark_data_smithi043_119832_object64103:head# 2024-03-30T13:50:57.742 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.742 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71941de6:::benchmark_data_smithi043_119832_object83701:head# 2024-03-30T13:50:57.743 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.743 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71955e9b:::benchmark_data_smithi043_119832_object62670:head# 2024-03-30T13:50:57.743 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.743 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7195b5ff:::benchmark_data_smithi043_119832_object15340:head# 2024-03-30T13:50:57.743 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.743 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:719725bf:::benchmark_data_smithi043_119832_object20822:head# 2024-03-30T13:50:57.744 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.744 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:719794dc:::benchmark_data_smithi043_119832_object19047:head# 2024-03-30T13:50:57.744 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.744 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:719869dd:::benchmark_data_smithi043_119832_object9035:head# 2024-03-30T13:50:57.744 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.744 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:719893af:::benchmark_data_smithi043_119832_object28060:head# 2024-03-30T13:50:57.744 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.745 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7198e463:::benchmark_data_smithi043_119832_object73068:head# 2024-03-30T13:50:57.745 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.745 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71993fe8:::benchmark_data_smithi043_119832_object6802:head# 2024-03-30T13:50:57.745 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.745 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7199efed:::benchmark_data_smithi043_119832_object19931:head# 2024-03-30T13:50:57.745 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.745 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:719a266e:::benchmark_data_smithi043_119832_object40329:head# 2024-03-30T13:50:57.746 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.746 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:719b9508:::benchmark_data_smithi043_119832_object33743:head# 2024-03-30T13:50:57.746 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.746 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:719bafc3:::benchmark_data_smithi043_119832_object54690:head# 2024-03-30T13:50:57.746 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.746 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:719c513f:::benchmark_data_smithi043_119832_object27531:head# 2024-03-30T13:50:57.746 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.747 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:719cc789:::benchmark_data_smithi043_119832_object84563:head# 2024-03-30T13:50:57.747 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.747 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:719cf843:::benchmark_data_smithi043_119832_object54838:head# 2024-03-30T13:50:57.747 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.747 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a032d9:::benchmark_data_smithi043_119832_object15962:head# 2024-03-30T13:50:57.748 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.748 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a051bf:::benchmark_data_smithi043_119832_object81379:head# 2024-03-30T13:50:57.748 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.748 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a0f357:::benchmark_data_smithi043_119832_object12332:head# 2024-03-30T13:50:57.748 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.748 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a16b95:::benchmark_data_smithi043_119832_object69971:head# 2024-03-30T13:50:57.749 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.749 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a1f896:::benchmark_data_smithi043_119832_object18224:head# 2024-03-30T13:50:57.749 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.749 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a27cc1:::benchmark_data_smithi043_119832_object60053:head# 2024-03-30T13:50:57.749 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.749 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a2cb22:::benchmark_data_smithi043_119832_object67053:head# 2024-03-30T13:50:57.749 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.750 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a329dc:::benchmark_data_smithi043_119832_object12914:head# 2024-03-30T13:50:57.750 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.750 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a338cb:::benchmark_data_smithi043_119832_object40353:head# 2024-03-30T13:50:57.750 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.750 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a37b80:::benchmark_data_smithi043_119832_object31345:head# 2024-03-30T13:50:57.750 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.751 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a37dbb:::benchmark_data_smithi043_119832_object24080:head# 2024-03-30T13:50:57.751 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.751 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a55489:::benchmark_data_smithi043_119832_object66734:head# 2024-03-30T13:50:57.751 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.751 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a5d5df:::benchmark_data_smithi043_119832_object72528:head# 2024-03-30T13:50:57.751 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.752 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a779e5:::benchmark_data_smithi043_119832_object1029:head# 2024-03-30T13:50:57.752 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.752 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a79df7:::benchmark_data_smithi043_119832_object51393:head# 2024-03-30T13:50:57.752 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.752 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a7ac54:::benchmark_data_smithi043_119832_object36813:head# 2024-03-30T13:50:57.753 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.753 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a8a73c:::benchmark_data_smithi043_119832_object13566:head# 2024-03-30T13:50:57.753 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.753 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a8e173:::benchmark_data_smithi043_119832_object21542:head# 2024-03-30T13:50:57.753 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.753 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a9164e:::benchmark_data_smithi043_119832_object36692:head# 2024-03-30T13:50:57.754 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.754 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a93533:::benchmark_data_smithi043_119832_object56622:head# 2024-03-30T13:50:57.754 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.754 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a93754:::benchmark_data_smithi043_119832_object53923:head# 2024-03-30T13:50:57.754 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.754 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a959b0:::benchmark_data_smithi043_119832_object75664:head# 2024-03-30T13:50:57.754 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.755 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71a9d278:::benchmark_data_smithi043_119832_object13750:head# 2024-03-30T13:50:57.755 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.755 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71aab8d8:::benchmark_data_smithi043_119832_object83930:head# 2024-03-30T13:50:57.755 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.755 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71aaf963:::benchmark_data_smithi043_119832_object86367:head# 2024-03-30T13:50:57.756 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.756 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71ab38c6:::benchmark_data_smithi043_119832_object39410:head# 2024-03-30T13:50:57.756 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.756 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71ab672b:::benchmark_data_smithi043_119832_object86548:head# 2024-03-30T13:50:57.756 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.756 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71ac4947:::benchmark_data_smithi043_119832_object7237:head# 2024-03-30T13:50:57.756 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.757 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71adca02:::benchmark_data_smithi043_119832_object31439:head# 2024-03-30T13:50:57.757 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.757 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71ae7e4a:::benchmark_data_smithi043_119832_object1382:head# 2024-03-30T13:50:57.757 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.757 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71ae82cb:::benchmark_data_smithi043_119832_object57352:head# 2024-03-30T13:50:57.758 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.758 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71ae8a17:::benchmark_data_smithi043_119832_object52212:head# 2024-03-30T13:50:57.758 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.758 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71aea1c3:::benchmark_data_smithi043_119832_object18362:head# 2024-03-30T13:50:57.758 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.758 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71b161b3:::benchmark_data_smithi043_119832_object84006:head# 2024-03-30T13:50:57.758 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.759 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71b5d518:::benchmark_data_smithi043_119832_object30337:head# 2024-03-30T13:50:57.759 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.759 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71b5fed0:::benchmark_data_smithi043_119832_object35777:head# 2024-03-30T13:50:57.759 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.759 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71b7abb1:::benchmark_data_smithi043_119832_object82356:head# 2024-03-30T13:50:57.760 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.760 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71b8e8e4:::benchmark_data_smithi043_119832_object34596:head# 2024-03-30T13:50:57.760 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.760 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71b99b59:::benchmark_data_smithi043_119832_object85775:head# 2024-03-30T13:50:57.760 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.761 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71bab083:::benchmark_data_smithi043_119832_object83716:head# 2024-03-30T13:50:57.761 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.761 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71bc8784:::benchmark_data_smithi043_119832_object44910:head# 2024-03-30T13:50:57.761 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.761 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71bcf0ed:::benchmark_data_smithi043_119832_object24694:head# 2024-03-30T13:50:57.762 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.762 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71be6695:::benchmark_data_smithi043_119832_object43485:head# 2024-03-30T13:50:57.762 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.762 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71bf0fa4:::benchmark_data_smithi043_119832_object4988:head# 2024-03-30T13:50:57.762 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.763 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71c02f22:::benchmark_data_smithi043_119832_object57462:head# 2024-03-30T13:50:57.763 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.763 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71c0611d:::benchmark_data_smithi043_119832_object59044:head# 2024-03-30T13:50:57.763 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.763 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71c1c97a:::benchmark_data_smithi043_119832_object37983:head# 2024-03-30T13:50:57.763 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.763 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71c235cb:::benchmark_data_smithi043_119832_object45435:head# 2024-03-30T13:50:57.764 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.764 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71c32f55:::benchmark_data_smithi043_119832_object63065:head# 2024-03-30T13:50:57.764 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.764 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71c33dff:::benchmark_data_smithi043_119832_object41478:head# 2024-03-30T13:50:57.764 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.764 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71c3cdfc:::benchmark_data_smithi043_119832_object63302:head# 2024-03-30T13:50:57.764 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.765 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71c3df76:::benchmark_data_smithi043_119832_object74277:head# 2024-03-30T13:50:57.765 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.765 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71c4d0f0:::benchmark_data_smithi043_119832_object81541:head# 2024-03-30T13:50:57.765 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.765 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71c5d95e:::benchmark_data_smithi043_119832_object5152:head# 2024-03-30T13:50:57.765 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.765 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71c6a143:::benchmark_data_smithi043_119832_object77535:head# 2024-03-30T13:50:57.766 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.766 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71c6e14c:::benchmark_data_smithi043_119832_object83887:head# 2024-03-30T13:50:57.766 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.766 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71c7d89a:::benchmark_data_smithi043_119832_object5923:head# 2024-03-30T13:50:57.766 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.766 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71c822ed:::benchmark_data_smithi043_119832_object23718:head# 2024-03-30T13:50:57.767 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.767 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71c84a40:::benchmark_data_smithi043_119832_object27592:head# 2024-03-30T13:50:57.767 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.767 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71ca13ed:::benchmark_data_smithi043_119832_object58093:head# 2024-03-30T13:50:57.767 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.767 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71cadec2:::benchmark_data_smithi043_119832_object28165:head# 2024-03-30T13:50:57.767 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.768 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71cb4882:::benchmark_data_smithi043_119832_object23702:head# 2024-03-30T13:50:57.768 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.768 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71cd97f1:::benchmark_data_smithi043_119832_object32835:head# 2024-03-30T13:50:57.768 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.769 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71ce073c:::benchmark_data_smithi043_119832_object38759:head# 2024-03-30T13:50:57.769 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.769 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71ce6b30:::benchmark_data_smithi043_119832_object84451:head# 2024-03-30T13:50:57.769 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.769 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71cf1146:::benchmark_data_smithi043_119832_object70640:head# 2024-03-30T13:50:57.769 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.769 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71d0920d:::benchmark_data_smithi043_119832_object87510:head# 2024-03-30T13:50:57.770 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.770 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71d21565:::benchmark_data_smithi043_119832_object10632:head# 2024-03-30T13:50:57.770 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.770 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71d27b7f:::benchmark_data_smithi043_119832_object63118:head# 2024-03-30T13:50:57.770 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.770 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71d2d524:::benchmark_data_smithi043_119832_object865:head# 2024-03-30T13:50:57.771 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.771 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71d30a4f:::benchmark_data_smithi043_119832_object50932:head# 2024-03-30T13:50:57.771 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.771 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71d3e665:::benchmark_data_smithi043_119832_object753:head# 2024-03-30T13:50:57.771 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.772 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71d45f0b:::benchmark_data_smithi043_119832_object73851:head# 2024-03-30T13:50:57.772 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.772 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71d50bc7:::benchmark_data_smithi043_119832_object32118:head# 2024-03-30T13:50:57.772 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.772 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71d578b2:::benchmark_data_smithi043_119832_object21889:head# 2024-03-30T13:50:57.772 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.773 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71d59c5c:::benchmark_data_smithi043_119832_object78740:head# 2024-03-30T13:50:57.773 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.773 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71d5a9d4:::benchmark_data_smithi043_119832_object67465:head# 2024-03-30T13:50:57.773 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.773 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71d6fc8c:::benchmark_data_smithi043_119832_object29094:head# 2024-03-30T13:50:57.773 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.774 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71d70688:::benchmark_data_smithi043_119832_object23572:head# 2024-03-30T13:50:57.774 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.774 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71d72e2c:::benchmark_data_smithi043_119832_object20670:head# 2024-03-30T13:50:57.774 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.775 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71d7e527:::benchmark_data_smithi043_119832_object70711:head# 2024-03-30T13:50:57.775 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.775 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71d8566b:::benchmark_data_smithi043_119832_object5744:head# 2024-03-30T13:50:57.775 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.776 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71d9b77a:::benchmark_data_smithi043_119832_object3074:head# 2024-03-30T13:50:57.776 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.776 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71db2b84:::benchmark_data_smithi043_119832_object51989:head# 2024-03-30T13:50:57.776 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.776 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71dc4d52:::benchmark_data_smithi043_119832_object58422:head# 2024-03-30T13:50:57.776 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.776 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71dc966a:::benchmark_data_smithi043_119832_object18201:head# 2024-03-30T13:50:57.776 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.777 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71dc9f91:::benchmark_data_smithi043_119832_object2122:head# 2024-03-30T13:50:57.777 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.777 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71ddb5f7:::benchmark_data_smithi043_119832_object20309:head# 2024-03-30T13:50:57.777 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.778 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71de09c4:::benchmark_data_smithi043_119832_object42982:head# 2024-03-30T13:50:57.778 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.778 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71de312a:::benchmark_data_smithi043_119832_object80184:head# 2024-03-30T13:50:57.778 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.778 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71df6165:::benchmark_data_smithi043_119832_object15884:head# 2024-03-30T13:50:57.778 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.779 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71e041b6:::benchmark_data_smithi043_119832_object70593:head# 2024-03-30T13:50:57.779 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.779 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71e16739:::benchmark_data_smithi043_119832_object46627:head# 2024-03-30T13:50:57.779 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.779 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71e3bde5:::benchmark_data_smithi043_119832_object530:head# 2024-03-30T13:50:57.779 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.780 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71e444e4:::benchmark_data_smithi043_119832_object79150:head# 2024-03-30T13:50:57.780 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.780 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71e4e03e:::benchmark_data_smithi043_119832_object36077:head# 2024-03-30T13:50:57.780 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.781 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71e4e3b3:::benchmark_data_smithi043_119832_object21826:head# 2024-03-30T13:50:57.781 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.781 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71e5bb80:::benchmark_data_smithi043_119832_object21456:head# 2024-03-30T13:50:57.781 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.781 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71e64fa5:::benchmark_data_smithi043_119832_object85134:head# 2024-03-30T13:50:57.781 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.781 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71e993f5:::benchmark_data_smithi043_119832_object14386:head# 2024-03-30T13:50:57.782 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.782 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71e9e05d:::benchmark_data_smithi043_119832_object52587:head# 2024-03-30T13:50:57.782 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.782 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71ea3f9a:::benchmark_data_smithi043_119832_object9973:head# 2024-03-30T13:50:57.782 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.783 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71ea68cc:::benchmark_data_smithi043_119832_object886:head# 2024-03-30T13:50:57.783 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.783 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71eaaa0e:::benchmark_data_smithi043_119832_object78207:head# 2024-03-30T13:50:57.783 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.783 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71ecae5e:::benchmark_data_smithi043_119832_object86961:head# 2024-03-30T13:50:57.783 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.784 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71edf2ce:::benchmark_data_smithi043_119832_object49915:head# 2024-03-30T13:50:57.784 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.784 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71ee9c5f:::benchmark_data_smithi043_119832_object25743:head# 2024-03-30T13:50:57.785 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.785 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71efe9c6:::benchmark_data_smithi043_119832_object60793:head# 2024-03-30T13:50:57.785 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.785 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71f158c2:::benchmark_data_smithi043_119832_object32572:head# 2024-03-30T13:50:57.785 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.785 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71f211b2:::benchmark_data_smithi043_119832_object83645:head# 2024-03-30T13:50:57.785 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.785 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71f2b6a5:::benchmark_data_smithi043_119832_object52360:head# 2024-03-30T13:50:57.786 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.786 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71f35739:::benchmark_data_smithi043_119832_object6731:head# 2024-03-30T13:50:57.786 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.787 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71f3c87d:::benchmark_data_smithi043_119832_object30045:head# 2024-03-30T13:50:57.787 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.787 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71f45683:::benchmark_data_smithi043_119832_object34437:head# 2024-03-30T13:50:57.787 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.787 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71f6dbad:::benchmark_data_smithi043_119832_object53719:head# 2024-03-30T13:50:57.787 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.787 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71f77e0a:::benchmark_data_smithi043_119832_object35042:head# 2024-03-30T13:50:57.787 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.788 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71f88fc5:::benchmark_data_smithi043_119832_object52038:head# 2024-03-30T13:50:57.788 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.788 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71fa9c20:::benchmark_data_smithi043_119832_object47374:head# 2024-03-30T13:50:57.788 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.789 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71fb8a81:::benchmark_data_smithi043_119832_object70576:head# 2024-03-30T13:50:57.789 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.789 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71fd279b:::benchmark_data_smithi043_119832_object15629:head# 2024-03-30T13:50:57.789 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.789 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:71ff2e8d:::benchmark_data_smithi043_119832_object29451:head# 2024-03-30T13:50:57.789 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.790 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7200937a:::benchmark_data_smithi043_119832_object16115:head# 2024-03-30T13:50:57.790 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.790 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7200a944:::benchmark_data_smithi043_119832_object60893:head# 2024-03-30T13:50:57.790 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.791 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7200d813:::benchmark_data_smithi043_119832_object45356:head# 2024-03-30T13:50:57.791 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.791 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7200d822:::benchmark_data_smithi043_119832_object18720:head# 2024-03-30T13:50:57.791 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.791 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72017289:::benchmark_data_smithi043_119832_object12588:head# 2024-03-30T13:50:57.791 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.791 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7201cdd1:::benchmark_data_smithi043_119832_object80585:head# 2024-03-30T13:50:57.791 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.791 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72030284:::benchmark_data_smithi043_119832_object19844:head# 2024-03-30T13:50:57.791 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.792 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:720312fe:::benchmark_data_smithi043_119832_object11432:head# 2024-03-30T13:50:57.792 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.792 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:720400c6:::benchmark_data_smithi043_119832_object77672:head# 2024-03-30T13:50:57.792 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.792 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72044203:::benchmark_data_smithi043_119832_object5273:head# 2024-03-30T13:50:57.792 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.793 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7204671b:::benchmark_data_smithi043_119832_object43218:head# 2024-03-30T13:50:57.793 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.793 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7204be6e:::benchmark_data_smithi043_119832_object59469:head# 2024-03-30T13:50:57.793 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.794 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7204f9e1:::benchmark_data_smithi043_119832_object72043:head# 2024-03-30T13:50:57.794 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.794 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72060532:::benchmark_data_smithi043_119832_object41928:head# 2024-03-30T13:50:57.794 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.794 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72079ed3:::benchmark_data_smithi043_119832_object51386:head# 2024-03-30T13:50:57.794 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.794 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72082531:::benchmark_data_smithi043_119832_object68997:head# 2024-03-30T13:50:57.794 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.794 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72085f6e:::benchmark_data_smithi043_119832_object50947:head# 2024-03-30T13:50:57.794 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.796 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7208ef75:::benchmark_data_smithi043_119832_object13124:head# 2024-03-30T13:50:57.796 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.796 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:720a1308:::benchmark_data_smithi043_119832_object49851:head# 2024-03-30T13:50:57.796 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.796 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:720a6a46:::benchmark_data_smithi043_119832_object28074:head# 2024-03-30T13:50:57.796 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.796 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:720cc65d:::benchmark_data_smithi043_119832_object52024:head# 2024-03-30T13:50:57.796 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.796 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:720dd06b:::benchmark_data_smithi043_119832_object66597:head# 2024-03-30T13:50:57.796 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.797 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:720e4c24:::benchmark_data_smithi043_119832_object933:head# 2024-03-30T13:50:57.797 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.797 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:720e8c05:::benchmark_data_smithi043_119832_object20648:head# 2024-03-30T13:50:57.797 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.798 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:720eba39:::benchmark_data_smithi043_119832_object87502:head# 2024-03-30T13:50:57.798 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.798 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:720f470e:::benchmark_data_smithi043_119832_object68563:head# 2024-03-30T13:50:57.798 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.798 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:721170ff:::benchmark_data_smithi043_119832_object82701:head# 2024-03-30T13:50:57.798 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.798 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:721203c5:::benchmark_data_smithi043_119832_object3689:head# 2024-03-30T13:50:57.799 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.799 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72126193:::benchmark_data_smithi043_119832_object24245:head# 2024-03-30T13:50:57.799 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.799 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72127ac7:::benchmark_data_smithi043_119832_object6754:head# 2024-03-30T13:50:57.799 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.800 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72141339:::benchmark_data_smithi043_119832_object3679:head# 2024-03-30T13:50:57.800 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.800 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7214a646:::benchmark_data_smithi043_119832_object14488:head# 2024-03-30T13:50:57.800 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.800 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72151981:::benchmark_data_smithi043_119832_object53428:head# 2024-03-30T13:50:57.800 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.800 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7215c492:::benchmark_data_smithi043_119832_object48863:head# 2024-03-30T13:50:57.800 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.800 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72175647:::benchmark_data_smithi043_119832_object48461:head# 2024-03-30T13:50:57.800 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.800 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7217844e:::benchmark_data_smithi043_119832_object53688:head# 2024-03-30T13:50:57.800 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.802 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7217b34d:::benchmark_data_smithi043_119832_object73229:head# 2024-03-30T13:50:57.802 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.802 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7217c34b:::benchmark_data_smithi043_119832_object81875:head# 2024-03-30T13:50:57.802 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.802 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7217fcde:::benchmark_data_smithi043_119832_object1099:head# 2024-03-30T13:50:57.802 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.802 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7219468a:::benchmark_data_smithi043_119832_object59765:head# 2024-03-30T13:50:57.802 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.802 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:721962af:::benchmark_data_smithi043_119832_object2070:head# 2024-03-30T13:50:57.802 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.802 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7219ad9e:::benchmark_data_smithi043_119832_object70423:head# 2024-03-30T13:50:57.802 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.802 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:721b8c32:::benchmark_data_smithi043_119832_object23736:head# 2024-03-30T13:50:57.803 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.803 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:721d0162:::benchmark_data_smithi043_119832_object34752:head# 2024-03-30T13:50:57.803 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.803 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:721d76e6:::benchmark_data_smithi043_119832_object7541:head# 2024-03-30T13:50:57.804 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.804 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:721df83b:::benchmark_data_smithi043_119832_object42980:head# 2024-03-30T13:50:57.804 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.804 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:721e7cda:::benchmark_data_smithi043_119832_object68863:head# 2024-03-30T13:50:57.804 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.804 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72203385:::benchmark_data_smithi043_119832_object25474:head# 2024-03-30T13:50:57.804 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.804 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:722039e3:::benchmark_data_smithi043_119832_object24172:head# 2024-03-30T13:50:57.804 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.805 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72215472:::benchmark_data_smithi043_119832_object21631:head# 2024-03-30T13:50:57.805 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.805 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72218d54:::benchmark_data_smithi043_119832_object34174:head# 2024-03-30T13:50:57.805 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.805 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7222c4f2:::benchmark_data_smithi043_119832_object4707:head# 2024-03-30T13:50:57.805 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.805 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72231673:::benchmark_data_smithi043_119832_object1021:head# 2024-03-30T13:50:57.806 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.806 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72232628:::benchmark_data_smithi043_119832_object80731:head# 2024-03-30T13:50:57.806 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.806 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72232ff7:::benchmark_data_smithi043_119832_object3680:head# 2024-03-30T13:50:57.806 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.806 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:50:57.807 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7224e5a7:::benchmark_data_smithi043_119832_object7231:head# 2024-03-30T13:50:57.807 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.807 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7226c900:::benchmark_data_smithi043_119832_object19703:head# 2024-03-30T13:50:57.807 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.807 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7226e04e:::benchmark_data_smithi043_119832_object20013:head# 2024-03-30T13:50:57.807 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.807 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72272e92:::benchmark_data_smithi043_119832_object6651:head# 2024-03-30T13:50:57.807 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.808 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72293555:::benchmark_data_smithi043_119832_object79664:head# 2024-03-30T13:50:57.808 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.808 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72294c66:::benchmark_data_smithi043_119832_object30151:head# 2024-03-30T13:50:57.808 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.809 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72297bb6:::benchmark_data_smithi043_119832_object6254:head# 2024-03-30T13:50:57.809 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.809 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:722a9a82:::benchmark_data_smithi043_119832_object15584:head# 2024-03-30T13:50:57.809 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.809 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:722ab2df:::benchmark_data_smithi043_119832_object35310:head# 2024-03-30T13:50:57.809 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.809 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:722b2d2d:::benchmark_data_smithi043_119832_object21772:head# 2024-03-30T13:50:57.809 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.812 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:722b4737:::benchmark_data_smithi043_119832_object60377:head# 2024-03-30T13:50:57.812 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.812 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:722b74c5:::benchmark_data_smithi043_119832_object5764:head# 2024-03-30T13:50:57.812 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.812 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:722bda2c:::benchmark_data_smithi043_119832_object18737:head# 2024-03-30T13:50:57.812 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.812 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:722c94c3:::benchmark_data_smithi043_119832_object46007:head# 2024-03-30T13:50:57.812 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.812 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:722d4758:::benchmark_data_smithi043_119832_object52524:head# 2024-03-30T13:50:57.812 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.813 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:722d4995:::benchmark_data_smithi043_119832_object76856:head# 2024-03-30T13:50:57.813 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.813 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:722f3420:::benchmark_data_smithi043_119832_object28332:head# 2024-03-30T13:50:57.813 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.813 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:722fd433:::benchmark_data_smithi043_119832_object86078:head# 2024-03-30T13:50:57.813 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.814 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:722fe2c2:::benchmark_data_smithi043_119832_object34614:head# 2024-03-30T13:50:57.814 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.814 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:722fe81e:::benchmark_data_smithi043_119832_object67022:head# 2024-03-30T13:50:57.814 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.814 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72309fab:::benchmark_data_smithi043_119832_object37170:head# 2024-03-30T13:50:57.814 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.814 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7232ad28:::benchmark_data_smithi043_119832_object83604:head# 2024-03-30T13:50:57.814 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.815 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7234065a:::benchmark_data_smithi043_119832_object79988:head# 2024-03-30T13:50:57.815 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.815 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72356f1b:::benchmark_data_smithi043_119832_object60131:head# 2024-03-30T13:50:57.815 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.815 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72366587:::benchmark_data_smithi043_119832_object22716:head# 2024-03-30T13:50:57.815 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.816 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72367af7:::benchmark_data_smithi043_119832_object59289:head# 2024-03-30T13:50:57.816 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.816 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7237646e:::benchmark_data_smithi043_119832_object12980:head# 2024-03-30T13:50:57.816 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.816 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72381a32:::benchmark_data_smithi043_119832_object35709:head# 2024-03-30T13:50:57.816 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.816 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72385369:::benchmark_data_smithi043_119832_object9666:head# 2024-03-30T13:50:57.816 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.816 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72385a2a:::benchmark_data_smithi043_119832_object71361:head# 2024-03-30T13:50:57.817 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.817 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:723869f8:::benchmark_data_smithi043_119832_object57279:head# 2024-03-30T13:50:57.817 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.817 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7238900f:::benchmark_data_smithi043_119832_object47463:head# 2024-03-30T13:50:57.817 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.818 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:723cd8d8:::benchmark_data_smithi043_119832_object5419:head# 2024-03-30T13:50:57.818 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.818 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:723d3866:::benchmark_data_smithi043_119832_object80061:head# 2024-03-30T13:50:57.818 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.818 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:723db22d:::benchmark_data_smithi043_119832_object86253:head# 2024-03-30T13:50:57.818 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.819 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:723df91c:::benchmark_data_smithi043_119832_object67799:head# 2024-03-30T13:50:57.819 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.819 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:723e60ac:::benchmark_data_smithi043_119832_object37779:head# 2024-03-30T13:50:57.819 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.819 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:723ee32a:::benchmark_data_smithi043_119832_object24342:head# 2024-03-30T13:50:57.819 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.819 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:723f64c7:::benchmark_data_smithi043_119832_object7418:head# 2024-03-30T13:50:57.819 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.819 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:723fed10:::benchmark_data_smithi043_119832_object88139:head# 2024-03-30T13:50:57.819 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.820 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7240182e:::benchmark_data_smithi043_119832_object39273:head# 2024-03-30T13:50:57.820 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.820 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7241d437:::benchmark_data_smithi043_119832_object37406:head# 2024-03-30T13:50:57.820 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.821 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7242efa9:::benchmark_data_smithi043_119832_object62666:head# 2024-03-30T13:50:57.821 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.821 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72431466:::benchmark_data_smithi043_119832_object44975:head# 2024-03-30T13:50:57.821 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.821 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:724321b8:::benchmark_data_smithi043_119832_object20109:head# 2024-03-30T13:50:57.821 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.821 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7245247a:::benchmark_data_smithi043_119832_object79368:head# 2024-03-30T13:50:57.821 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.822 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72453a87:::benchmark_data_smithi043_119832_object57254:head# 2024-03-30T13:50:57.822 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.822 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7245bfdc:::benchmark_data_smithi043_119832_object43705:head# 2024-03-30T13:50:57.822 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.822 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7247dc1b:::benchmark_data_smithi043_119832_object63395:head# 2024-03-30T13:50:57.822 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.822 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72480c59:::benchmark_data_smithi043_119832_object61012:head# 2024-03-30T13:50:57.822 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.823 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:724a29c5:::benchmark_data_smithi043_119832_object22884:head# 2024-03-30T13:50:57.823 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.823 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:724ad046:::benchmark_data_smithi043_119832_object34071:head# 2024-03-30T13:50:57.823 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.823 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:724c7cee:::benchmark_data_smithi043_119832_object19959:head# 2024-03-30T13:50:57.823 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.824 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:724c9d63:::benchmark_data_smithi043_119832_object71675:head# 2024-03-30T13:50:57.824 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.824 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7250994e:::benchmark_data_smithi043_119832_object57193:head# 2024-03-30T13:50:57.824 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.824 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7250adbe:::benchmark_data_smithi043_119832_object18151:head# 2024-03-30T13:50:57.824 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.824 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:725168a1:::benchmark_data_smithi043_119832_object72455:head# 2024-03-30T13:50:57.825 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.825 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:725168e3:::benchmark_data_smithi043_119832_object9586:head# 2024-03-30T13:50:57.825 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.825 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72519f00:::benchmark_data_smithi043_119832_object74789:head# 2024-03-30T13:50:57.826 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.826 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72530fc8:::benchmark_data_smithi043_119832_object64545:head# 2024-03-30T13:50:57.826 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.826 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72537238:::benchmark_data_smithi043_119832_object16034:head# 2024-03-30T13:50:57.826 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.826 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72543727:::benchmark_data_smithi043_119832_object4625:head# 2024-03-30T13:50:57.826 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.826 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7254c880:::benchmark_data_smithi043_119832_object51744:head# 2024-03-30T13:50:57.826 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.826 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72552cd5:::benchmark_data_smithi043_119832_object49566:head# 2024-03-30T13:50:57.826 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.827 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72554bf6:::benchmark_data_smithi043_119832_object52100:head# 2024-03-30T13:50:57.827 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.827 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72559acf:::benchmark_data_smithi043_119832_object39027:head# 2024-03-30T13:50:57.827 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.827 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7255b73d:::benchmark_data_smithi043_119832_object55496:head# 2024-03-30T13:50:57.827 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.828 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72567902:::benchmark_data_smithi043_119832_object24032:head# 2024-03-30T13:50:57.828 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.828 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7256c12e:::benchmark_data_smithi043_119832_object14655:head# 2024-03-30T13:50:57.828 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.829 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72574052:::benchmark_data_smithi043_119832_object12000:head# 2024-03-30T13:50:57.829 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.829 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7257b59a:::benchmark_data_smithi043_119832_object15972:head# 2024-03-30T13:50:57.829 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.829 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7259390d:::benchmark_data_smithi043_119832_object44012:head# 2024-03-30T13:50:57.829 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.829 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:725b5c21:::benchmark_data_smithi043_119832_object76910:head# 2024-03-30T13:50:57.829 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.829 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:725ba407:::benchmark_data_smithi043_119832_object37050:head# 2024-03-30T13:50:57.829 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.829 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:725c33c8:::benchmark_data_smithi043_119832_object39061:head# 2024-03-30T13:50:57.829 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.830 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:725ded6c:::benchmark_data_smithi043_119832_object16105:head# 2024-03-30T13:50:57.830 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.830 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7260bd22:::benchmark_data_smithi043_119832_object29265:head# 2024-03-30T13:50:57.830 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.831 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7260f77f:::benchmark_data_smithi043_119832_object52514:head# 2024-03-30T13:50:57.831 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.831 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72618de4:::benchmark_data_smithi043_119832_object50607:head# 2024-03-30T13:50:57.831 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.831 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:726585c8:::benchmark_data_smithi043_119832_object88194:head# 2024-03-30T13:50:57.831 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.832 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7265eb20:::benchmark_data_smithi043_119832_object69751:head# 2024-03-30T13:50:57.832 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.832 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7266f19b:::benchmark_data_smithi043_119832_object87733:head# 2024-03-30T13:50:57.832 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.832 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7267860b:::benchmark_data_smithi043_119832_object450:head# 2024-03-30T13:50:57.832 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.832 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7267d4a5:::benchmark_data_smithi043_119832_object10176:head# 2024-03-30T13:50:57.832 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.832 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7267d9cb:::benchmark_data_smithi043_119832_object23316:head# 2024-03-30T13:50:57.832 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.832 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72693192:::benchmark_data_smithi043_119832_object88454:head# 2024-03-30T13:50:57.833 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.833 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72696e22:::benchmark_data_smithi043_119832_object84887:head# 2024-03-30T13:50:57.833 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.833 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72698415:::benchmark_data_smithi043_119832_object74843:head# 2024-03-30T13:50:57.833 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.834 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:726adbdd:::benchmark_data_smithi043_119832_object33990:head# 2024-03-30T13:50:57.834 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.834 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:726b8d40:::benchmark_data_smithi043_119832_object85147:head# 2024-03-30T13:50:57.834 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.834 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:726dd3eb:::benchmark_data_smithi043_119832_object8957:head# 2024-03-30T13:50:57.834 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.835 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:726e2663:::benchmark_data_smithi043_119832_object46554:head# 2024-03-30T13:50:57.835 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.835 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:726e2837:::benchmark_data_smithi043_119832_object20426:head# 2024-03-30T13:50:57.835 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.835 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:726fb125:::benchmark_data_smithi043_119832_object47098:head# 2024-03-30T13:50:57.835 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.835 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72701077:::benchmark_data_smithi043_119832_object33070:head# 2024-03-30T13:50:57.835 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.835 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72702fca:::benchmark_data_smithi043_119832_object77220:head# 2024-03-30T13:50:57.835 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.835 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72709a56:::benchmark_data_smithi043_119832_object87639:head# 2024-03-30T13:50:57.836 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.836 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7271171c:::benchmark_data_smithi043_119832_object60317:head# 2024-03-30T13:50:57.837 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.837 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7275231b:::benchmark_data_smithi043_119832_object75821:head# 2024-03-30T13:50:57.837 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.837 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72756421:::benchmark_data_smithi043_119832_object44834:head# 2024-03-30T13:50:57.837 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.837 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7275ed0e:::benchmark_data_smithi043_119832_object13337:head# 2024-03-30T13:50:57.837 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.837 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7275f3c3:::benchmark_data_smithi043_119832_object78009:head# 2024-03-30T13:50:57.837 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.837 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7276b1b2:::benchmark_data_smithi043_119832_object21730:head# 2024-03-30T13:50:57.837 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.838 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7276dde4:::benchmark_data_smithi043_119832_object33940:head# 2024-03-30T13:50:57.838 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.838 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:727718d4:::benchmark_data_smithi043_119832_object74972:head# 2024-03-30T13:50:57.838 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.838 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72772b26:::benchmark_data_smithi043_119832_object40577:head# 2024-03-30T13:50:57.838 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.839 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7278c2d2:::benchmark_data_smithi043_119832_object53071:head# 2024-03-30T13:50:57.839 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.839 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7278e248:::benchmark_data_smithi043_119832_object47885:head# 2024-03-30T13:50:57.839 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.839 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7279b028:::benchmark_data_smithi043_119832_object32335:head# 2024-03-30T13:50:57.839 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.840 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:727a1e63:::benchmark_data_smithi043_119832_object48699:head# 2024-03-30T13:50:57.840 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.840 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:727aa488:::benchmark_data_smithi043_119832_object85426:head# 2024-03-30T13:50:57.840 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.840 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:727ad602:::benchmark_data_smithi043_119832_object30373:head# 2024-03-30T13:50:57.840 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.840 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:727b73b5:::benchmark_data_smithi043_119832_object9750:head# 2024-03-30T13:50:57.840 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.841 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:727c40c6:::benchmark_data_smithi043_119832_object5532:head# 2024-03-30T13:50:57.841 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.841 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:727f2f8b:::benchmark_data_smithi043_119832_object15517:head# 2024-03-30T13:50:57.841 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.841 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:727fa240:::benchmark_data_smithi043_119832_object73913:head# 2024-03-30T13:50:57.841 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.842 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:728026e5:::benchmark_data_smithi043_119832_object23285:head# 2024-03-30T13:50:57.842 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.842 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7282c8a0:::benchmark_data_smithi043_119832_object28452:head# 2024-03-30T13:50:57.842 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.842 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72830666:::benchmark_data_smithi043_119832_object35674:head# 2024-03-30T13:50:57.842 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.842 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:728348aa:::benchmark_data_smithi043_119832_object64765:head# 2024-03-30T13:50:57.842 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.843 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7283bc61:::benchmark_data_smithi043_119832_object1145:head# 2024-03-30T13:50:57.843 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.843 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7284d6fb:::benchmark_data_smithi043_119832_object50253:head# 2024-03-30T13:50:57.843 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.843 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7284e1cb:::benchmark_data_smithi043_119832_object83014:head# 2024-03-30T13:50:57.843 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.843 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72851760:::benchmark_data_smithi043_119832_object51972:head# 2024-03-30T13:50:57.844 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.844 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:728678e7:::benchmark_data_smithi043_119832_object82995:head# 2024-03-30T13:50:57.844 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.844 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7287059f:::benchmark_data_smithi043_119832_object33978:head# 2024-03-30T13:50:57.844 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.844 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7287c19d:::benchmark_data_smithi043_119832_object10981:head# 2024-03-30T13:50:57.844 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.844 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7287fde6:::benchmark_data_smithi043_119832_object6893:head# 2024-03-30T13:50:57.845 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.845 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7289442f:::benchmark_data_smithi043_119832_object30829:head# 2024-03-30T13:50:57.845 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.845 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:728a11db:::benchmark_data_smithi043_119832_object78449:head# 2024-03-30T13:50:57.845 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.847 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:728a5977:::benchmark_data_smithi043_119832_object77998:head# 2024-03-30T13:50:57.847 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.847 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:728a7d1d:::benchmark_data_smithi043_119832_object17063:head# 2024-03-30T13:50:57.847 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.847 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:728bc4f0:::benchmark_data_smithi043_119832_object54543:head# 2024-03-30T13:50:57.847 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.847 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:728c0211:::benchmark_data_smithi043_119832_object19360:head# 2024-03-30T13:50:57.847 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.847 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:728c1843:::benchmark_data_smithi043_119832_object53458:head# 2024-03-30T13:50:57.847 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.847 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:728d3445:::benchmark_data_smithi043_119832_object19207:head# 2024-03-30T13:50:57.847 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.848 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:728d7fac:::benchmark_data_smithi043_119832_object59856:head# 2024-03-30T13:50:57.848 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.848 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:728e19b5:::benchmark_data_smithi043_119832_object83310:head# 2024-03-30T13:50:57.848 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.848 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:728e6625:::benchmark_data_smithi043_119832_object6658:head# 2024-03-30T13:50:57.848 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.848 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:728e90f9:::benchmark_data_smithi043_119832_object19417:head# 2024-03-30T13:50:57.848 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.849 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:728ea2c4:::benchmark_data_smithi043_119832_object48738:head# 2024-03-30T13:50:57.849 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.849 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:728f5f1b:::benchmark_data_smithi043_119832_object58025:head# 2024-03-30T13:50:57.849 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.849 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:729010db:::benchmark_data_smithi043_119832_object7788:head# 2024-03-30T13:50:57.849 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.849 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72912d25:::benchmark_data_smithi043_119832_object3997:head# 2024-03-30T13:50:57.849 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.849 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72938817:::benchmark_data_smithi043_119832_object87879:head# 2024-03-30T13:50:57.849 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.850 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72951016:::benchmark_data_smithi043_119832_object71259:head# 2024-03-30T13:50:57.850 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.850 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72957455:::benchmark_data_smithi043_119832_object40805:head# 2024-03-30T13:50:57.850 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.850 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7295e2d0:::benchmark_data_smithi043_119832_object50751:head# 2024-03-30T13:50:57.851 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.851 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72965cd7:::benchmark_data_smithi043_119832_object84603:head# 2024-03-30T13:50:57.851 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.853 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7296b66b:::benchmark_data_smithi043_119832_object57326:head# 2024-03-30T13:50:57.853 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.854 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72975e3c:::benchmark_data_smithi043_119832_object6298:head# 2024-03-30T13:50:57.854 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.854 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7297c121:::benchmark_data_smithi043_119832_object88150:head# 2024-03-30T13:50:57.854 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.854 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:729874d7:::benchmark_data_smithi043_119832_object80317:head# 2024-03-30T13:50:57.854 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.854 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7298b442:::benchmark_data_smithi043_119832_object45011:head# 2024-03-30T13:50:57.854 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.854 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7298b588:::benchmark_data_smithi043_119832_object56846:head# 2024-03-30T13:50:57.854 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.854 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:729a0068:::benchmark_data_smithi043_119832_object31798:head# 2024-03-30T13:50:57.854 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.854 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:729a1a79:::benchmark_data_smithi043_119832_object21973:head# 2024-03-30T13:50:57.855 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.855 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:729b3e46:::benchmark_data_smithi043_119832_object72978:head# 2024-03-30T13:50:57.855 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.859 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:729bcf2a:::benchmark_data_smithi043_119832_object87725:head# 2024-03-30T13:50:57.859 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.859 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:729ddca0:::benchmark_data_smithi043_119832_object41112:head# 2024-03-30T13:50:57.859 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.859 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:729ec102:::benchmark_data_smithi043_119832_object22645:head# 2024-03-30T13:50:57.860 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.860 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:729f3736:::benchmark_data_smithi043_119832_object6528:head# 2024-03-30T13:50:57.860 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.860 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:729f8b8f:::benchmark_data_smithi043_119832_object81249:head# 2024-03-30T13:50:57.860 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.860 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:729ff89d:::benchmark_data_smithi043_119832_object27351:head# 2024-03-30T13:50:57.860 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.860 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72a04d8e:::benchmark_data_smithi043_119832_object54247:head# 2024-03-30T13:50:57.860 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.860 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72a1a27d:::benchmark_data_smithi043_119832_object21087:head# 2024-03-30T13:50:57.860 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.860 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72a24430:::benchmark_data_smithi043_119832_object8471:head# 2024-03-30T13:50:57.860 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.860 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72a2b0d5:::benchmark_data_smithi043_119832_object71896:head# 2024-03-30T13:50:57.861 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.861 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72a363cc:::benchmark_data_smithi043_119832_object22178:head# 2024-03-30T13:50:57.861 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.861 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72a44876:::benchmark_data_smithi043_119832_object29038:head# 2024-03-30T13:50:57.861 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.861 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72a491e8:::benchmark_data_smithi043_119832_object65287:head# 2024-03-30T13:50:57.861 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.861 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72a4eb5e:::benchmark_data_smithi043_119832_object57259:head# 2024-03-30T13:50:57.861 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.862 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72a7535e:::benchmark_data_smithi043_119832_object70274:head# 2024-03-30T13:50:57.862 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.862 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72a7fe0f:::benchmark_data_smithi043_119832_object6988:head# 2024-03-30T13:50:57.862 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.862 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72a8d292:::benchmark_data_smithi043_119832_object65996:head# 2024-03-30T13:50:57.862 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.862 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72aa0768:::benchmark_data_smithi043_119832_object28466:head# 2024-03-30T13:50:57.862 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.862 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72aa762e:::benchmark_data_smithi043_119832_object42507:head# 2024-03-30T13:50:57.863 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.863 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72acdf35:::benchmark_data_smithi043_119832_object70856:head# 2024-03-30T13:50:57.863 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.863 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72ad8721:::benchmark_data_smithi043_119832_object8884:head# 2024-03-30T13:50:57.863 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.863 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72adfcfb:::benchmark_data_smithi043_119832_object9834:head# 2024-03-30T13:50:57.863 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.863 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72af6604:::benchmark_data_smithi043_119832_object37653:head# 2024-03-30T13:50:57.863 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.863 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72b243b2:::benchmark_data_smithi043_119832_object70575:head# 2024-03-30T13:50:57.863 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.864 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72b35de5:::benchmark_data_smithi043_119832_object55461:head# 2024-03-30T13:50:57.864 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.864 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72b412d7:::benchmark_data_smithi043_119832_object45954:head# 2024-03-30T13:50:57.864 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.864 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72b4a0a0:::benchmark_data_smithi043_119832_object42614:head# 2024-03-30T13:50:57.864 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.864 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72b4e71b:::benchmark_data_smithi043_119832_object45556:head# 2024-03-30T13:50:57.865 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.865 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72b5123c:::benchmark_data_smithi043_119832_object73144:head# 2024-03-30T13:50:57.865 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.865 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72b5d1f4:::benchmark_data_smithi043_119832_object37002:head# 2024-03-30T13:50:57.865 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:57.866 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:57.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:50:57.867 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72b61d2e:::benchmark_data_smithi043_119832_object57689:head# 2024-03-30T13:50:57.867 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.867 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72b6dc05:::benchmark_data_smithi043_119832_object42512:head# 2024-03-30T13:50:57.867 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.867 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72b724c0:::benchmark_data_smithi043_119832_object32669:head# 2024-03-30T13:50:57.867 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.867 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72b9e383:::benchmark_data_smithi043_119832_object74156:head# 2024-03-30T13:50:57.867 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.868 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72b9f283:::benchmark_data_smithi043_119832_object76165:head# 2024-03-30T13:50:57.868 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.868 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72ba2427:::benchmark_data_smithi043_119832_object81526:head# 2024-03-30T13:50:57.868 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.868 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72bc49ad:::benchmark_data_smithi043_119832_object55573:head# 2024-03-30T13:50:57.868 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.868 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72bccb1c:::benchmark_data_smithi043_119832_object32438:head# 2024-03-30T13:50:57.868 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.868 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72bce37b:::benchmark_data_smithi043_119832_object43680:head# 2024-03-30T13:50:57.868 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.868 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72bde24a:::benchmark_data_smithi043_119832_object36238:head# 2024-03-30T13:50:57.869 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.869 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72beec6d:::benchmark_data_smithi043_119832_object64961:head# 2024-03-30T13:50:57.869 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.870 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72bfef37:::benchmark_data_smithi043_119832_object36889:head# 2024-03-30T13:50:57.870 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.870 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72c0b4c1:::benchmark_data_smithi043_119832_object36404:head# 2024-03-30T13:50:57.870 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.870 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72c15537:::benchmark_data_smithi043_119832_object47774:head# 2024-03-30T13:50:57.870 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.870 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72c39afa:::benchmark_data_smithi043_119832_object16395:head# 2024-03-30T13:50:57.870 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.870 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72c3e4dc:::benchmark_data_smithi043_119832_object79822:head# 2024-03-30T13:50:57.870 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.871 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72c4b36a:::benchmark_data_smithi043_119832_object87268:head# 2024-03-30T13:50:57.871 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.871 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72c501a0:::benchmark_data_smithi043_119832_object41558:head# 2024-03-30T13:50:57.871 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.871 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72c54ad8:::benchmark_data_smithi043_119832_object43906:head# 2024-03-30T13:50:57.871 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.871 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72c6c8f5:::benchmark_data_smithi043_119832_object19855:head# 2024-03-30T13:50:57.871 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.872 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72c7c227:::benchmark_data_smithi043_119832_object2219:head# 2024-03-30T13:50:57.872 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.873 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72ca8bca:::benchmark_data_smithi043_119832_object41173:head# 2024-03-30T13:50:57.873 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.873 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72cab48a:::benchmark_data_smithi043_119832_object29511:head# 2024-03-30T13:50:57.873 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.873 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72cb538d:::benchmark_data_smithi043_119832_object37738:head# 2024-03-30T13:50:57.873 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.873 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72cb593f:::benchmark_data_smithi043_119832_object87860:head# 2024-03-30T13:50:57.873 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.874 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72cb8237:::benchmark_data_smithi043_119832_object53218:head# 2024-03-30T13:50:57.874 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.875 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72cbed10:::benchmark_data_smithi043_119832_object11336:head# 2024-03-30T13:50:57.875 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.875 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72cecee6:::benchmark_data_smithi043_119832_object17794:head# 2024-03-30T13:50:57.875 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.875 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72cf4473:::benchmark_data_smithi043_119832_object12447:head# 2024-03-30T13:50:57.875 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.875 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72d010c1:::benchmark_data_smithi043_119832_object84816:head# 2024-03-30T13:50:57.875 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.875 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72d05282:::benchmark_data_smithi043_119832_object7246:head# 2024-03-30T13:50:57.876 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.877 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72d21e13:::benchmark_data_smithi043_119832_object78905:head# 2024-03-30T13:50:57.877 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.877 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72d381a7:::benchmark_data_smithi043_119832_object52254:head# 2024-03-30T13:50:57.877 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.877 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72d4f1e6:::benchmark_data_smithi043_119832_object50638:head# 2024-03-30T13:50:57.877 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.877 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72d55263:::benchmark_data_smithi043_119832_object10801:head# 2024-03-30T13:50:57.877 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.877 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72d72374:::benchmark_data_smithi043_119832_object68494:head# 2024-03-30T13:50:57.877 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.878 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72d7c5a6:::benchmark_data_smithi043_119832_object39864:head# 2024-03-30T13:50:57.879 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.879 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72d95452:::benchmark_data_smithi043_119832_object2012:head# 2024-03-30T13:50:57.879 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.879 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72db9bb9:::benchmark_data_smithi043_119832_object34242:head# 2024-03-30T13:50:57.879 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.879 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72dbd13e:::benchmark_data_smithi043_119832_object60091:head# 2024-03-30T13:50:57.879 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.879 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72dbf928:::benchmark_data_smithi043_119832_object8163:head# 2024-03-30T13:50:57.879 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.880 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72deda5c:::benchmark_data_smithi043_119832_object83450:head# 2024-03-30T13:50:57.880 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.880 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72dee5cc:::benchmark_data_smithi043_119832_object2495:head# 2024-03-30T13:50:57.880 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.880 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72df1ed9:::benchmark_data_smithi043_119832_object64894:head# 2024-03-30T13:50:57.880 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.880 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72dfd4fd:::benchmark_data_smithi043_119832_object84964:head# 2024-03-30T13:50:57.880 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.881 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72e22954:::benchmark_data_smithi043_119832_object54683:head# 2024-03-30T13:50:57.881 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.881 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72e490ee:::benchmark_data_smithi043_119832_object27641:head# 2024-03-30T13:50:57.881 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.881 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72e6958b:::benchmark_data_smithi043_119832_object22067:head# 2024-03-30T13:50:57.881 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.882 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72e6b0d9:::benchmark_data_smithi043_119832_object71117:head# 2024-03-30T13:50:57.882 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.882 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72e70699:::benchmark_data_smithi043_119832_object15798:head# 2024-03-30T13:50:57.882 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.883 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72e71b6e:::benchmark_data_smithi043_119832_object24471:head# 2024-03-30T13:50:57.883 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.883 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72e7c062:::benchmark_data_smithi043_119832_object43823:head# 2024-03-30T13:50:57.883 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.883 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72e867e4:::benchmark_data_smithi043_119832_object45149:head# 2024-03-30T13:50:57.883 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.884 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72e891c9:::benchmark_data_smithi043_119832_object28278:head# 2024-03-30T13:50:57.884 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.886 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72ea23a6:::benchmark_data_smithi043_119832_object42948:head# 2024-03-30T13:50:57.886 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.886 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72eace4f:::benchmark_data_smithi043_119832_object64007:head# 2024-03-30T13:50:57.886 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.886 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72eb1443:::benchmark_data_smithi043_119832_object76172:head# 2024-03-30T13:50:57.886 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.886 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72ec8d5f:::benchmark_data_smithi043_119832_object65241:head# 2024-03-30T13:50:57.886 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.887 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72eca005:::benchmark_data_smithi043_119832_object26356:head# 2024-03-30T13:50:57.887 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.887 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72ed6129:::benchmark_data_smithi043_119832_object20627:head# 2024-03-30T13:50:57.887 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.887 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72ed671a:::benchmark_data_smithi043_119832_object17725:head# 2024-03-30T13:50:57.887 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.888 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72ed77d8:::benchmark_data_smithi043_119832_object26054:head# 2024-03-30T13:50:57.888 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.888 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72ef1ef7:::benchmark_data_smithi043_119832_object75792:head# 2024-03-30T13:50:57.888 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.888 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72f3cd4d:::benchmark_data_smithi043_119832_object74465:head# 2024-03-30T13:50:57.888 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.888 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72f4649d:::benchmark_data_smithi043_119832_object69791:head# 2024-03-30T13:50:57.888 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.888 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72f54125:::benchmark_data_smithi043_119832_object82631:head# 2024-03-30T13:50:57.888 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.889 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72f54dac:::benchmark_data_smithi043_119832_object40118:head# 2024-03-30T13:50:57.889 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.889 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72f5695d:::benchmark_data_smithi043_119832_object24127:head# 2024-03-30T13:50:57.889 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.889 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72f66ee2:::benchmark_data_smithi043_119832_object6518:head# 2024-03-30T13:50:57.889 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.890 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72f723f8:::benchmark_data_smithi043_119832_object25268:head# 2024-03-30T13:50:57.890 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.890 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72f747ef:::benchmark_data_smithi043_119832_object71385:head# 2024-03-30T13:50:57.890 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.890 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72f7bb54:::benchmark_data_smithi043_119832_object87086:head# 2024-03-30T13:50:57.890 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.890 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72f7c5a1:::benchmark_data_smithi043_119832_object78418:head# 2024-03-30T13:50:57.890 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.891 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72f8c310:::benchmark_data_smithi043_119832_object34051:head# 2024-03-30T13:50:57.891 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.891 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72f926bc:::benchmark_data_smithi043_119832_object26302:head# 2024-03-30T13:50:57.891 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.891 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72f9c618:::benchmark_data_smithi043_119832_object8976:head# 2024-03-30T13:50:57.891 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.893 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72fa1abc:::benchmark_data_smithi043_119832_object58616:head# 2024-03-30T13:50:57.893 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.893 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72fa3377:::benchmark_data_smithi043_119832_object36948:head# 2024-03-30T13:50:57.893 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.893 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72fa4966:::benchmark_data_smithi043_119832_object43326:head# 2024-03-30T13:50:57.894 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.894 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72fa7067:::benchmark_data_smithi043_119832_object76183:head# 2024-03-30T13:50:57.894 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.894 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72fad31f:::benchmark_data_smithi043_119832_object35464:head# 2024-03-30T13:50:57.894 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.894 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72fb3af0:::benchmark_data_smithi043_119832_object8269:head# 2024-03-30T13:50:57.894 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.895 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72fb7a19:::benchmark_data_smithi043_119832_object64856:head# 2024-03-30T13:50:57.895 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.895 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72fcd559:::benchmark_data_smithi043_119832_object42399:head# 2024-03-30T13:50:57.895 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.895 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72fcf02a:::benchmark_data_smithi043_119832_object49798:head# 2024-03-30T13:50:57.895 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.895 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72ff4329:::benchmark_data_smithi043_119832_object9161:head# 2024-03-30T13:50:57.895 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.896 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72ff57bc:::benchmark_data_smithi043_119832_object42387:head# 2024-03-30T13:50:57.896 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.896 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:72ffe97d:::benchmark_data_smithi043_119832_object3915:head# 2024-03-30T13:50:57.896 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.896 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7300535a:::benchmark_data_smithi043_119832_object7180:head# 2024-03-30T13:50:57.896 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.897 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73017667:::benchmark_data_smithi043_119832_object9391:head# 2024-03-30T13:50:57.897 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.897 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7301d8b7:::benchmark_data_smithi043_119832_object11028:head# 2024-03-30T13:50:57.897 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.897 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73022386:::benchmark_data_smithi043_119832_object6354:head# 2024-03-30T13:50:57.897 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.898 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73037998:::benchmark_data_smithi043_119832_object83758:head# 2024-03-30T13:50:57.898 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.898 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73038a14:::benchmark_data_smithi043_119832_object69514:head# 2024-03-30T13:50:57.898 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.898 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:730510e1:::benchmark_data_smithi043_119832_object19457:head# 2024-03-30T13:50:57.898 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.898 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73063627:::benchmark_data_smithi043_119832_object72316:head# 2024-03-30T13:50:57.899 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.899 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:730699aa:::benchmark_data_smithi043_119832_object69756:head# 2024-03-30T13:50:57.899 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.899 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73069a3f:::benchmark_data_smithi043_119832_object57224:head# 2024-03-30T13:50:57.900 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.900 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73074886:::benchmark_data_smithi043_119832_object81655:head# 2024-03-30T13:50:57.900 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.900 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7307bc44:::benchmark_data_smithi043_119832_object34602:head# 2024-03-30T13:50:57.901 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.901 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7307c0a7:::benchmark_data_smithi043_119832_object15744:head# 2024-03-30T13:50:57.901 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.901 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:730a4f9d:::benchmark_data_smithi043_119832_object1481:head# 2024-03-30T13:50:57.901 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.902 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:730a927d:::benchmark_data_smithi043_119832_object44218:head# 2024-03-30T13:50:57.902 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.902 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:730a96dc:::benchmark_data_smithi043_119832_object85144:head# 2024-03-30T13:50:57.903 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.903 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:730b8b42:::benchmark_data_smithi043_119832_object83247:head# 2024-03-30T13:50:57.903 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.903 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:730c4fe8:::benchmark_data_smithi043_119832_object25443:head# 2024-03-30T13:50:57.903 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.903 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:730d35d6:::benchmark_data_smithi043_119832_object15558:head# 2024-03-30T13:50:57.903 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.904 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:730d3b3b:::benchmark_data_smithi043_119832_object63482:head# 2024-03-30T13:50:57.904 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.904 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:730e22c2:::benchmark_data_smithi043_119832_object9441:head# 2024-03-30T13:50:57.904 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.904 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73107587:::benchmark_data_smithi043_119832_object80644:head# 2024-03-30T13:50:57.904 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.904 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73113b7d:::benchmark_data_smithi043_119832_object62927:head# 2024-03-30T13:50:57.904 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.905 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73114bd0:::benchmark_data_smithi043_119832_object82965:head# 2024-03-30T13:50:57.906 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.906 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:731158e5:::benchmark_data_smithi043_119832_object25216:head# 2024-03-30T13:50:57.906 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.906 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7312713b:::benchmark_data_smithi043_119832_object50090:head# 2024-03-30T13:50:57.906 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.906 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7312c3b2:::benchmark_data_smithi043_119832_object26660:head# 2024-03-30T13:50:57.906 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.906 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73140916:::benchmark_data_smithi043_119832_object47311:head# 2024-03-30T13:50:57.906 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.908 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73149983:::benchmark_data_smithi043_119832_object32296:head# 2024-03-30T13:50:57.908 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.908 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7314a2b6:::benchmark_data_smithi043_119832_object31309:head# 2024-03-30T13:50:57.908 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.908 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7314b576:::benchmark_data_smithi043_119832_object17833:head# 2024-03-30T13:50:57.908 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.908 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7314ff98:::benchmark_data_smithi043_119832_object26907:head# 2024-03-30T13:50:57.908 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.908 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73163be2:::benchmark_data_smithi043_119832_object44938:head# 2024-03-30T13:50:57.908 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.908 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73174dcd:::benchmark_data_smithi043_119832_object68699:head# 2024-03-30T13:50:57.908 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.909 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73175960:::benchmark_data_smithi043_119832_object25480:head# 2024-03-30T13:50:57.909 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.910 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73181118:::benchmark_data_smithi043_119832_object8669:head# 2024-03-30T13:50:57.911 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.911 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7318443a:::benchmark_data_smithi043_119832_object27332:head# 2024-03-30T13:50:57.911 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.911 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7319caf6:::benchmark_data_smithi043_119832_object56080:head# 2024-03-30T13:50:57.911 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.911 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7319d7e2:::benchmark_data_smithi043_119832_object79011:head# 2024-03-30T13:50:57.911 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.911 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7319f93a:::benchmark_data_smithi043_119832_object53810:head# 2024-03-30T13:50:57.911 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.911 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:731a74ae:::benchmark_data_smithi043_119832_object42478:head# 2024-03-30T13:50:57.911 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.911 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:731ab7e3:::benchmark_data_smithi043_119832_object71718:head# 2024-03-30T13:50:57.911 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.912 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:731b4e34:::benchmark_data_smithi043_119832_object12869:head# 2024-03-30T13:50:57.912 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.914 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:731c2c86:::benchmark_data_smithi043_119832_object79509:head# 2024-03-30T13:50:57.914 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.914 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:731cf4ca:::benchmark_data_smithi043_119832_object22626:head# 2024-03-30T13:50:57.914 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.914 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:731e1bc9:::benchmark_data_smithi043_119832_object11273:head# 2024-03-30T13:50:57.914 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.914 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:731fe26e:::benchmark_data_smithi043_119832_object74095:head# 2024-03-30T13:50:57.914 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.914 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7320bf4f:::benchmark_data_smithi043_119832_object61971:head# 2024-03-30T13:50:57.914 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.914 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73212542:::benchmark_data_smithi043_119832_object57852:head# 2024-03-30T13:50:57.914 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.915 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7321b613:::benchmark_data_smithi043_119832_object36239:head# 2024-03-30T13:50:57.915 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.915 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73227700:::benchmark_data_smithi043_119832_object1076:head# 2024-03-30T13:50:57.915 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.915 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7322e784:::benchmark_data_smithi043_119832_object67819:head# 2024-03-30T13:50:57.915 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.915 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7323026f:::benchmark_data_smithi043_119832_object53851:head# 2024-03-30T13:50:57.915 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.915 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73231467:::benchmark_data_smithi043_119832_object64675:head# 2024-03-30T13:50:57.915 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.919 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73240fd6:::benchmark_data_smithi043_119832_object2623:head# 2024-03-30T13:50:57.919 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.919 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73243aca:::benchmark_data_smithi043_119832_object51110:head# 2024-03-30T13:50:57.919 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.919 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73246a21:::benchmark_data_smithi043_119832_object41215:head# 2024-03-30T13:50:57.920 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.920 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7324db09:::benchmark_data_smithi043_119832_object49978:head# 2024-03-30T13:50:57.920 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.920 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7324fa62:::benchmark_data_smithi043_119832_object16452:head# 2024-03-30T13:50:57.920 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.920 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73254f5a:::benchmark_data_smithi043_119832_object58646:head# 2024-03-30T13:50:57.920 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.920 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73266ecc:::benchmark_data_smithi043_119832_object30849:head# 2024-03-30T13:50:57.920 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.920 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7327ca30:::benchmark_data_smithi043_119832_object44480:head# 2024-03-30T13:50:57.920 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.920 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:732808a7:::benchmark_data_smithi043_119832_object6652:head# 2024-03-30T13:50:57.920 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.920 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73287f10:::benchmark_data_smithi043_119832_object35788:head# 2024-03-30T13:50:57.920 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.921 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73294864:::benchmark_data_smithi043_119832_object38593:head# 2024-03-30T13:50:57.921 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.921 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7329f015:::benchmark_data_smithi043_119832_object28204:head# 2024-03-30T13:50:57.921 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.921 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:732a61fd:::benchmark_data_smithi043_119832_object83940:head# 2024-03-30T13:50:57.921 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.921 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:732b3e8c:::benchmark_data_smithi043_119832_object86327:head# 2024-03-30T13:50:57.921 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.921 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:732b59e6:::benchmark_data_smithi043_119832_object70621:head# 2024-03-30T13:50:57.921 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.921 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:732b80a9:::benchmark_data_smithi043_119832_object56180:head# 2024-03-30T13:50:57.921 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.921 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:732bcd9d:::benchmark_data_smithi043_119832_object36562:head# 2024-03-30T13:50:57.921 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.921 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:732c095f:::benchmark_data_smithi043_119832_object25772:head# 2024-03-30T13:50:57.922 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.922 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:732c381a:::benchmark_data_smithi043_119832_object57504:head# 2024-03-30T13:50:57.922 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.922 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:732c3ece:::benchmark_data_smithi043_119832_object41486:head# 2024-03-30T13:50:57.922 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.922 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:732c7c01:::benchmark_data_smithi043_119832_object82316:head# 2024-03-30T13:50:57.922 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.922 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:732f059e:::benchmark_data_smithi043_119832_object4381:head# 2024-03-30T13:50:57.922 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.922 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:732f63ff:::benchmark_data_smithi043_119832_object72463:head# 2024-03-30T13:50:57.922 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.927 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:732f85ae:::benchmark_data_smithi043_119832_object67389:head# 2024-03-30T13:50:57.927 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.927 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:732f97a9:::benchmark_data_smithi043_119832_object10500:head# 2024-03-30T13:50:57.927 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.927 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7330a849:::benchmark_data_smithi043_119832_object13621:head# 2024-03-30T13:50:57.927 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.928 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73311fb9:::benchmark_data_smithi043_119832_object83141:head# 2024-03-30T13:50:57.928 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.928 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7331d23f:::benchmark_data_smithi043_119832_object74772:head# 2024-03-30T13:50:57.928 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.928 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7332b116:::benchmark_data_smithi043_119832_object54282:head# 2024-03-30T13:50:57.928 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.928 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7332f55b:::benchmark_data_smithi043_119832_object73263:head# 2024-03-30T13:50:57.928 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.928 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:733357b3:::benchmark_data_smithi043_119832_object78950:head# 2024-03-30T13:50:57.928 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.928 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7333d8fb:::benchmark_data_smithi043_119832_object601:head# 2024-03-30T13:50:57.928 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.928 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7333ff34:::benchmark_data_smithi043_119832_object82420:head# 2024-03-30T13:50:57.929 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.929 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7336ae5b:::benchmark_data_smithi043_119832_object19398:head# 2024-03-30T13:50:57.929 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.929 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7336b956:::benchmark_data_smithi043_119832_object3962:head# 2024-03-30T13:50:57.929 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.929 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7336d5dd:::benchmark_data_smithi043_119832_object71141:head# 2024-03-30T13:50:57.929 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.929 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:733788de:::benchmark_data_smithi043_119832_object8760:head# 2024-03-30T13:50:57.929 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.929 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7337bceb:::benchmark_data_smithi043_119832_object51013:head# 2024-03-30T13:50:57.929 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.929 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:733a6408:::benchmark_data_smithi043_119832_object60913:head# 2024-03-30T13:50:57.929 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.929 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:733b19f6:::benchmark_data_smithi043_119832_object66614:head# 2024-03-30T13:50:57.930 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.930 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:733b8ea0:::benchmark_data_smithi043_119832_object70088:head# 2024-03-30T13:50:57.930 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.930 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:733bf09d:::benchmark_data_smithi043_119832_object72109:head# 2024-03-30T13:50:57.930 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.930 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:733c3342:::benchmark_data_smithi043_119832_object74992:head# 2024-03-30T13:50:57.930 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.930 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:733c3846:::benchmark_data_smithi043_119832_object18377:head# 2024-03-30T13:50:57.930 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.930 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:733c8623:::benchmark_data_smithi043_119832_object72704:head# 2024-03-30T13:50:57.930 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.930 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:733e5115:::benchmark_data_smithi043_119832_object83740:head# 2024-03-30T13:50:57.930 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.930 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:733e7b11:::benchmark_data_smithi043_119832_object11502:head# 2024-03-30T13:50:57.931 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.931 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:733ea3f6:::benchmark_data_smithi043_119832_object79912:head# 2024-03-30T13:50:57.931 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.931 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:733ef5f9:::benchmark_data_smithi043_119832_object51267:head# 2024-03-30T13:50:57.931 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.931 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:733f558f:::benchmark_data_smithi043_119832_object13023:head# 2024-03-30T13:50:57.931 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.931 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:733f8c5f:::benchmark_data_smithi043_119832_object2994:head# 2024-03-30T13:50:57.931 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.931 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:734099f6:::benchmark_data_smithi043_119832_object17246:head# 2024-03-30T13:50:57.931 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.931 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73418ce6:::benchmark_data_smithi043_119832_object22250:head# 2024-03-30T13:50:57.931 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.931 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:734614b7:::benchmark_data_smithi043_119832_object18992:head# 2024-03-30T13:50:57.932 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.932 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73461772:::benchmark_data_smithi043_119832_object71477:head# 2024-03-30T13:50:57.932 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.932 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:734646dd:::benchmark_data_smithi043_119832_object62985:head# 2024-03-30T13:50:57.932 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.938 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7346f719:::benchmark_data_smithi043_119832_object7799:head# 2024-03-30T13:50:57.938 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.938 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73474a25:::benchmark_data_smithi043_119832_object59690:head# 2024-03-30T13:50:57.938 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.938 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73475475:::benchmark_data_smithi043_119832_object81864:head# 2024-03-30T13:50:57.938 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.938 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7348fb77:::benchmark_data_smithi043_119832_object51249:head# 2024-03-30T13:50:57.939 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.939 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:734a541a:::benchmark_data_smithi043_119832_object37485:head# 2024-03-30T13:50:57.939 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.939 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:734a7513:::benchmark_data_smithi043_119832_object58372:head# 2024-03-30T13:50:57.939 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.939 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:734a853f:::benchmark_data_smithi043_119832_object60:head# 2024-03-30T13:50:57.939 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.939 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:734aa94a:::benchmark_data_smithi043_119832_object13322:head# 2024-03-30T13:50:57.939 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.939 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:734ad730:::benchmark_data_smithi043_119832_object23908:head# 2024-03-30T13:50:57.939 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.939 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:734cc906:::benchmark_data_smithi043_119832_object85022:head# 2024-03-30T13:50:57.939 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.940 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:734ee960:::benchmark_data_smithi043_119832_object14213:head# 2024-03-30T13:50:57.940 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.940 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73501579:::benchmark_data_smithi043_119832_object30007:head# 2024-03-30T13:50:57.940 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.940 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:735159e8:::benchmark_data_smithi043_119832_object72817:head# 2024-03-30T13:50:57.940 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.940 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73519575:::benchmark_data_smithi043_119832_object39263:head# 2024-03-30T13:50:57.940 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.940 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73558e5c:::benchmark_data_smithi043_119832_object65654:head# 2024-03-30T13:50:57.940 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.940 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73561ac1:::benchmark_data_smithi043_119832_object1640:head# 2024-03-30T13:50:57.940 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.940 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7356fd38:::benchmark_data_smithi043_119832_object14587:head# 2024-03-30T13:50:57.940 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.940 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7357d542:::benchmark_data_smithi043_119832_object59136:head# 2024-03-30T13:50:57.941 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.941 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:735922a4:::benchmark_data_smithi043_119832_object23372:head# 2024-03-30T13:50:57.941 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.941 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:735a81f2:::benchmark_data_smithi043_119832_object14737:head# 2024-03-30T13:50:57.941 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.941 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:735b5a7c:::benchmark_data_smithi043_119832_object9534:head# 2024-03-30T13:50:57.941 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.941 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:735b7fdb:::benchmark_data_smithi043_119832_object14988:head# 2024-03-30T13:50:57.941 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.941 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:735b9aa7:::benchmark_data_smithi043_119832_object71484:head# 2024-03-30T13:50:57.941 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.941 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:735c117e:::benchmark_data_smithi043_119832_object39789:head# 2024-03-30T13:50:57.941 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.941 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:735d19fd:::benchmark_data_smithi043_119832_object60226:head# 2024-03-30T13:50:57.942 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.942 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:735d92a6:::benchmark_data_smithi043_119832_object37216:head# 2024-03-30T13:50:57.942 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.942 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:735dd05f:::benchmark_data_smithi043_119832_object50128:head# 2024-03-30T13:50:57.942 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.942 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:735eda65:::benchmark_data_smithi043_119832_object19052:head# 2024-03-30T13:50:57.942 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.942 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:735f7450:::benchmark_data_smithi043_119832_object4479:head# 2024-03-30T13:50:57.942 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.942 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7360d6ea:::benchmark_data_smithi043_119832_object8139:head# 2024-03-30T13:50:57.942 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.942 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7360ff4f:::benchmark_data_smithi043_119832_object62899:head# 2024-03-30T13:50:57.942 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.943 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:736165a7:::benchmark_data_smithi043_119832_object85189:head# 2024-03-30T13:50:57.943 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.943 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7361735d:::benchmark_data_smithi043_119832_object35431:head# 2024-03-30T13:50:57.943 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.943 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:736194aa:::benchmark_data_smithi043_119832_object69904:head# 2024-03-30T13:50:57.943 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.943 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7362321d:::benchmark_data_smithi043_119832_object58574:head# 2024-03-30T13:50:57.943 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.943 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:736296ff:::benchmark_data_smithi043_119832_object52269:head# 2024-03-30T13:50:57.943 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.943 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:736297c9:::benchmark_data_smithi043_119832_object37449:head# 2024-03-30T13:50:57.943 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.943 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7362d1b8:::benchmark_data_smithi043_119832_object18784:head# 2024-03-30T13:50:57.943 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.943 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:736377c6:::benchmark_data_smithi043_119832_object81175:head# 2024-03-30T13:50:57.944 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.944 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7363ac1d:::benchmark_data_smithi043_119832_object18874:head# 2024-03-30T13:50:57.944 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.944 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:736457b6:::benchmark_data_smithi043_119832_object19007:head# 2024-03-30T13:50:57.944 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.944 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73647797:::benchmark_data_smithi043_119832_object60577:head# 2024-03-30T13:50:57.944 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.944 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7364a527:::benchmark_data_smithi043_119832_object20471:head# 2024-03-30T13:50:57.944 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.945 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7364b7c7:::benchmark_data_smithi043_119832_object43778:head# 2024-03-30T13:50:57.945 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.945 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7366fe68:::benchmark_data_smithi043_119832_object86711:head# 2024-03-30T13:50:57.945 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.945 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73672bba:::benchmark_data_smithi043_119832_object31053:head# 2024-03-30T13:50:57.945 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.946 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:736896b6:::benchmark_data_smithi043_119832_object66778:head# 2024-03-30T13:50:57.946 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.946 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:736a03b2:::benchmark_data_smithi043_119832_object22088:head# 2024-03-30T13:50:57.946 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.946 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:736a161c:::benchmark_data_smithi043_119832_object59502:head# 2024-03-30T13:50:57.946 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.946 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:736c3456:::benchmark_data_smithi043_119832_object6519:head# 2024-03-30T13:50:57.946 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.946 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:736e227d:::benchmark_data_smithi043_119832_object32011:head# 2024-03-30T13:50:57.946 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.946 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:736e2db2:::benchmark_data_smithi043_119832_object32045:head# 2024-03-30T13:50:57.946 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.946 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:736f1c99:::benchmark_data_smithi043_119832_object33028:head# 2024-03-30T13:50:57.947 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.947 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:736fdefe:::benchmark_data_smithi043_119832_object63743:head# 2024-03-30T13:50:57.947 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.947 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:736fe47c:::benchmark_data_smithi043_119832_object1347:head# 2024-03-30T13:50:57.947 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.947 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73707ee2:::benchmark_data_smithi043_119832_object78662:head# 2024-03-30T13:50:57.947 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.947 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73723e1b:::benchmark_data_smithi043_119832_object25500:head# 2024-03-30T13:50:57.947 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.947 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7372cd90:::benchmark_data_smithi043_119832_object47538:head# 2024-03-30T13:50:57.947 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.947 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:737308d0:::benchmark_data_smithi043_119832_object86835:head# 2024-03-30T13:50:57.947 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.947 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73740ee1:::benchmark_data_smithi043_119832_object45724:head# 2024-03-30T13:50:57.947 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.948 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73744e6c:::benchmark_data_smithi043_119832_object16764:head# 2024-03-30T13:50:57.948 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.948 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7374b460:::benchmark_data_smithi043_119832_object6312:head# 2024-03-30T13:50:57.948 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.948 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7374ef8b:::benchmark_data_smithi043_119832_object35139:head# 2024-03-30T13:50:57.948 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.949 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73751983:::benchmark_data_smithi043_119832_object1493:head# 2024-03-30T13:50:57.949 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.949 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7375ecf8:::benchmark_data_smithi043_119832_object68317:head# 2024-03-30T13:50:57.949 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.949 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7378a76e:::benchmark_data_smithi043_119832_object33188:head# 2024-03-30T13:50:57.949 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.949 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7379addf:::benchmark_data_smithi043_119832_object6939:head# 2024-03-30T13:50:57.949 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.949 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7379e150:::benchmark_data_smithi043_119832_object23988:head# 2024-03-30T13:50:57.950 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.950 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:737a2e2e:::benchmark_data_smithi043_119832_object16430:head# 2024-03-30T13:50:57.950 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.950 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:737a95b9:::benchmark_data_smithi043_119832_object20304:head# 2024-03-30T13:50:57.950 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.951 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:737aee55:::benchmark_data_smithi043_119832_object11945:head# 2024-03-30T13:50:57.951 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.951 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:737d5537:::benchmark_data_smithi043_119832_object8035:head# 2024-03-30T13:50:57.951 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.951 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:737f8fab:::benchmark_data_smithi043_119832_object73693:head# 2024-03-30T13:50:57.951 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.951 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73800740:::benchmark_data_smithi043_119832_object56133:head# 2024-03-30T13:50:57.951 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.951 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:738105e7:::benchmark_data_smithi043_119832_object39682:head# 2024-03-30T13:50:57.951 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.952 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7381a759:::benchmark_data_smithi043_119832_object10935:head# 2024-03-30T13:50:57.952 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.952 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7382b8bf:::benchmark_data_smithi043_119832_object66535:head# 2024-03-30T13:50:57.952 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.952 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:738387df:::benchmark_data_smithi043_119832_object78801:head# 2024-03-30T13:50:57.952 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.952 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7383af91:::benchmark_data_smithi043_119832_object52869:head# 2024-03-30T13:50:57.953 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.953 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73850537:::benchmark_data_smithi043_119832_object58206:head# 2024-03-30T13:50:57.953 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.954 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73853284:::benchmark_data_smithi043_119832_object62115:head# 2024-03-30T13:50:57.954 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.954 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7385329f:::benchmark_data_smithi043_119832_object44911:head# 2024-03-30T13:50:57.954 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.954 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7385371e:::benchmark_data_smithi043_119832_object50961:head# 2024-03-30T13:50:57.954 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.954 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7385795d:::benchmark_data_smithi043_119832_object14156:head# 2024-03-30T13:50:57.954 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.955 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7385b005:::benchmark_data_smithi043_119832_object43212:head# 2024-03-30T13:50:57.956 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.956 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7386a61e:::benchmark_data_smithi043_119832_object2395:head# 2024-03-30T13:50:57.956 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.956 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7387ea95:::benchmark_data_smithi043_119832_object41665:head# 2024-03-30T13:50:57.956 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.956 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7388c4b4:::benchmark_data_smithi043_119832_object27485:head# 2024-03-30T13:50:57.956 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.957 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7389d81f:::benchmark_data_smithi043_119832_object12765:head# 2024-03-30T13:50:57.957 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.957 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7389e5ec:::benchmark_data_smithi043_119832_object76660:head# 2024-03-30T13:50:57.957 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.957 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7389f278:::benchmark_data_smithi043_119832_object65418:head# 2024-03-30T13:50:57.957 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.957 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:738a20dd:::benchmark_data_smithi043_119832_object49732:head# 2024-03-30T13:50:57.957 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.958 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:738a972b:::benchmark_data_smithi043_119832_object62169:head# 2024-03-30T13:50:57.958 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.958 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:738a9a8d:::benchmark_data_smithi043_119832_object14704:head# 2024-03-30T13:50:57.958 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.958 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:738bdd7a:::benchmark_data_smithi043_119832_object14435:head# 2024-03-30T13:50:57.958 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.959 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:738be96e:::benchmark_data_smithi043_119832_object24568:head# 2024-03-30T13:50:57.959 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.959 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:738c2e69:::benchmark_data_smithi043_119832_object60573:head# 2024-03-30T13:50:57.959 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.960 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:738c77a4:::benchmark_data_smithi043_119832_object80634:head# 2024-03-30T13:50:57.960 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.960 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:738cc439:::benchmark_data_smithi043_119832_object54578:head# 2024-03-30T13:50:57.960 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.960 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:738d40d3:::benchmark_data_smithi043_119832_object87817:head# 2024-03-30T13:50:57.961 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.961 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:738e7b07:::benchmark_data_smithi043_119832_object27339:head# 2024-03-30T13:50:57.961 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.961 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:738e7c5b:::benchmark_data_smithi043_119832_object23020:head# 2024-03-30T13:50:57.961 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.962 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:738f19df:::benchmark_data_smithi043_119832_object74168:head# 2024-03-30T13:50:57.962 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.962 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:738f8b3d:::benchmark_data_smithi043_119832_object23965:head# 2024-03-30T13:50:57.962 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.962 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7390bf2f:::benchmark_data_smithi043_119832_object3176:head# 2024-03-30T13:50:57.962 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.963 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7390f0ad:::benchmark_data_smithi043_119832_object85820:head# 2024-03-30T13:50:57.963 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.963 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73915c61:::benchmark_data_smithi043_119832_object30093:head# 2024-03-30T13:50:57.963 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.964 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73916048:::benchmark_data_smithi043_119832_object11555:head# 2024-03-30T13:50:57.964 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.964 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7391e551:::benchmark_data_smithi043_119832_object26194:head# 2024-03-30T13:50:57.964 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.964 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73926f1f:::benchmark_data_smithi043_119832_object74280:head# 2024-03-30T13:50:57.964 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.965 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73938ed7:::benchmark_data_smithi043_119832_object73093:head# 2024-03-30T13:50:57.965 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.965 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7393b865:::benchmark_data_smithi043_119832_object33082:head# 2024-03-30T13:50:57.965 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.965 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:739406ea:::benchmark_data_smithi043_119832_object20320:head# 2024-03-30T13:50:57.966 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.966 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7394e117:::benchmark_data_smithi043_119832_object39866:head# 2024-03-30T13:50:57.966 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.966 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:57.967 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73952040:::benchmark_data_smithi043_119832_object65895:head# 2024-03-30T13:50:57.967 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.967 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7395ea15:::benchmark_data_smithi043_119832_object79028:head# 2024-03-30T13:50:57.967 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.967 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:739627e4:::benchmark_data_smithi043_119832_object65561:head# 2024-03-30T13:50:57.967 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.968 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73983e48:::benchmark_data_smithi043_119832_object47116:head# 2024-03-30T13:50:57.968 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.968 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:739865d4:::benchmark_data_smithi043_119832_object76317:head# 2024-03-30T13:50:57.968 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.968 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:739899f8:::benchmark_data_smithi043_119832_object39485:head# 2024-03-30T13:50:57.969 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.969 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:739948c2:::benchmark_data_smithi043_119832_object4190:head# 2024-03-30T13:50:57.970 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.970 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73998586:::benchmark_data_smithi043_119832_object37838:head# 2024-03-30T13:50:57.970 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.970 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7399dd46:::benchmark_data_smithi043_119832_object35278:head# 2024-03-30T13:50:57.970 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.970 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:739a6447:::benchmark_data_smithi043_119832_object31722:head# 2024-03-30T13:50:57.970 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.971 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:739b5351:::benchmark_data_smithi043_119832_object49007:head# 2024-03-30T13:50:57.971 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:57 smithi113 ceph-mon[34341]: 8.2 scrub starts 2024-03-30T13:50:57.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:57 smithi113 ceph-mon[34341]: 3.1b scrub starts 2024-03-30T13:50:57.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:57 smithi113 ceph-mon[34341]: 3.1b scrub ok 2024-03-30T13:50:57.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:57 smithi113 ceph-mon[34341]: 5.c scrub starts 2024-03-30T13:50:57.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:57 smithi113 ceph-mon[34341]: 5.c scrub ok 2024-03-30T13:50:57.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:57 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 44241/354549 objects degraded (12.478%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:50:57.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:57 smithi113 ceph-mon[34341]: osdmap e182: 8 total, 7 up, 8 in 2024-03-30T13:50:57.973 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:739c837d:::benchmark_data_smithi043_119832_object74228:head# 2024-03-30T13:50:57.973 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.973 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:739cc8fc:::benchmark_data_smithi043_119832_object45390:head# 2024-03-30T13:50:57.973 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.973 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:739db323:::benchmark_data_smithi043_119832_object68551:head# 2024-03-30T13:50:57.973 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.973 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:739dd069:::benchmark_data_smithi043_119832_object46174:head# 2024-03-30T13:50:57.973 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.974 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:739dd2b4:::benchmark_data_smithi043_119832_object81428:head# 2024-03-30T13:50:57.974 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.974 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:739dd651:::benchmark_data_smithi043_119832_object53246:head# 2024-03-30T13:50:57.974 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.974 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:739eddcb:::benchmark_data_smithi043_119832_object51617:head# 2024-03-30T13:50:57.974 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.974 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:739f2df1:::benchmark_data_smithi043_119832_object68202:head# 2024-03-30T13:50:57.974 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.975 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:739ffa2a:::benchmark_data_smithi043_119832_object65602:head# 2024-03-30T13:50:57.975 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.975 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73a15dac:::benchmark_data_smithi043_119832_object87224:head# 2024-03-30T13:50:57.975 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.975 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73a1ed77:::benchmark_data_smithi043_119832_object82223:head# 2024-03-30T13:50:57.975 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.975 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73a4b94d:::benchmark_data_smithi043_119832_object25121:head# 2024-03-30T13:50:57.976 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.976 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73a509a9:::benchmark_data_smithi043_119832_object50781:head# 2024-03-30T13:50:57.976 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.976 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73a9b63b:::benchmark_data_smithi043_119832_object35645:head# 2024-03-30T13:50:57.976 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.976 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73aa0c5c:::benchmark_data_smithi043_119832_object67492:head# 2024-03-30T13:50:57.977 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.977 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73aa7eb4:::benchmark_data_smithi043_119832_object57354:head# 2024-03-30T13:50:57.977 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.977 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73ac4b46:::benchmark_data_smithi043_119832_object77523:head# 2024-03-30T13:50:57.977 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.977 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73aca0b9:::benchmark_data_smithi043_119832_object45603:head# 2024-03-30T13:50:57.977 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.977 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73acbc8c:::benchmark_data_smithi043_119832_object64249:head# 2024-03-30T13:50:57.978 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.980 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73ade79a:::benchmark_data_smithi043_119832_object48277:head# 2024-03-30T13:50:57.980 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.980 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73ae0e34:::benchmark_data_smithi043_119832_object50830:head# 2024-03-30T13:50:57.980 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.980 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73ae2867:::benchmark_data_smithi043_119832_object81219:head# 2024-03-30T13:50:57.980 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.980 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73aead8b:::benchmark_data_smithi043_119832_object3450:head# 2024-03-30T13:50:57.980 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.980 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73afd023:::benchmark_data_smithi043_119832_object2666:head# 2024-03-30T13:50:57.980 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.980 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73b0e0dd:::benchmark_data_smithi043_119832_object4889:head# 2024-03-30T13:50:57.981 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.981 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73b191aa:::benchmark_data_smithi043_119832_object67801:head# 2024-03-30T13:50:57.981 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.981 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73b1afe9:::benchmark_data_smithi043_119832_object5878:head# 2024-03-30T13:50:57.981 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.982 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73b200ff:::benchmark_data_smithi043_119832_object9397:head# 2024-03-30T13:50:57.982 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.982 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73b27049:::benchmark_data_smithi043_119832_object8789:head# 2024-03-30T13:50:57.982 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.982 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73b2e61a:::benchmark_data_smithi043_119832_object18708:head# 2024-03-30T13:50:57.982 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.982 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73b364fb:::benchmark_data_smithi043_119832_object4552:head# 2024-03-30T13:50:57.982 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.982 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73b52e2a:::benchmark_data_smithi043_119832_object24702:head# 2024-03-30T13:50:57.982 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.983 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73b6544f:::benchmark_data_smithi043_119832_object55364:head# 2024-03-30T13:50:57.983 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.983 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73b71fd4:::benchmark_data_smithi043_119832_object18557:head# 2024-03-30T13:50:57.983 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.983 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73b7c03d:::benchmark_data_smithi043_119832_object87450:head# 2024-03-30T13:50:57.983 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.983 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73b90f38:::benchmark_data_smithi043_119832_object48416:head# 2024-03-30T13:50:57.983 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.984 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73b9ba24:::benchmark_data_smithi043_119832_object57930:head# 2024-03-30T13:50:57.984 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.984 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73bacefc:::benchmark_data_smithi043_119832_object38996:head# 2024-03-30T13:50:57.984 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.984 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73bb2da3:::benchmark_data_smithi043_119832_object40960:head# 2024-03-30T13:50:57.984 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.985 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73bb9077:::benchmark_data_smithi043_119832_object63736:head# 2024-03-30T13:50:57.985 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.985 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73bc5176:::benchmark_data_smithi043_119832_object46822:head# 2024-03-30T13:50:57.985 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.985 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73be5620:::benchmark_data_smithi043_119832_object75120:head# 2024-03-30T13:50:57.986 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.986 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73c09c54:::benchmark_data_smithi043_119832_object41878:head# 2024-03-30T13:50:57.986 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.986 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73c153ac:::benchmark_data_smithi043_119832_object14976:head# 2024-03-30T13:50:57.986 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.986 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73c1737f:::benchmark_data_smithi043_119832_object80859:head# 2024-03-30T13:50:57.986 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.987 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73c17498:::benchmark_data_smithi043_119832_object6650:head# 2024-03-30T13:50:57.987 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.988 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73c1cbd2:::benchmark_data_smithi043_119832_object14991:head# 2024-03-30T13:50:57.988 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.988 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73c23558:::benchmark_data_smithi043_119832_object39254:head# 2024-03-30T13:50:57.988 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.988 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73c2b512:::benchmark_data_smithi043_119832_object19093:head# 2024-03-30T13:50:57.988 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.989 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73c35971:::benchmark_data_smithi043_119832_object22908:head# 2024-03-30T13:50:57.989 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.989 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73c3a923:::benchmark_data_smithi043_119832_object31850:head# 2024-03-30T13:50:57.989 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.990 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73c482c0:::benchmark_data_smithi043_119832_object60648:head# 2024-03-30T13:50:57.990 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.990 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73c56b50:::benchmark_data_smithi043_119832_object70239:head# 2024-03-30T13:50:57.990 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.990 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73c5cecc:::benchmark_data_smithi043_119832_object8941:head# 2024-03-30T13:50:57.991 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.991 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73c65fcf:::benchmark_data_smithi043_119832_object9146:head# 2024-03-30T13:50:57.991 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.992 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73c74184:::benchmark_data_smithi043_119832_object32133:head# 2024-03-30T13:50:57.992 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.992 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73c8d6e9:::benchmark_data_smithi043_119832_object74713:head# 2024-03-30T13:50:57.992 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.992 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73c93a96:::benchmark_data_smithi043_119832_object79671:head# 2024-03-30T13:50:57.992 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.992 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73c9b7f4:::benchmark_data_smithi043_119832_object8776:head# 2024-03-30T13:50:57.992 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.992 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73c9ce0e:::benchmark_data_smithi043_119832_object1271:head# 2024-03-30T13:50:57.993 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.993 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73ca0d1a:::benchmark_data_smithi043_119832_object34598:head# 2024-03-30T13:50:57.993 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.993 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73cb4513:::benchmark_data_smithi043_119832_object66744:head# 2024-03-30T13:50:57.993 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.994 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73cc8328:::benchmark_data_smithi043_119832_object25924:head# 2024-03-30T13:50:57.994 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.995 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73cd94a3:::benchmark_data_smithi043_119832_object23836:head# 2024-03-30T13:50:57.995 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.995 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73ce2bca:::benchmark_data_smithi043_119832_object69361:head# 2024-03-30T13:50:57.995 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.995 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73ce8bb5:::benchmark_data_smithi043_119832_object81255:head# 2024-03-30T13:50:57.995 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.996 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73cf5661:::benchmark_data_smithi043_119832_object71023:head# 2024-03-30T13:50:57.996 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.996 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73d04972:::benchmark_data_smithi043_119832_object63547:head# 2024-03-30T13:50:57.996 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.996 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73d04e02:::benchmark_data_smithi043_119832_object34544:head# 2024-03-30T13:50:57.997 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.997 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73d14499:::benchmark_data_smithi043_119832_object123:head# 2024-03-30T13:50:57.997 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.997 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73d33c63:::benchmark_data_smithi043_119832_object76507:head# 2024-03-30T13:50:57.997 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.997 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73d37b41:::benchmark_data_smithi043_119832_object80738:head# 2024-03-30T13:50:57.997 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.998 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73d3c143:::benchmark_data_smithi043_119832_object54676:head# 2024-03-30T13:50:57.998 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.998 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73d423b1:::benchmark_data_smithi043_119832_object81564:head# 2024-03-30T13:50:57.998 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.998 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73d4e762:::benchmark_data_smithi043_119832_object53016:head# 2024-03-30T13:50:57.998 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.998 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73d69fc9:::benchmark_data_smithi043_119832_object17467:head# 2024-03-30T13:50:57.998 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.999 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73d72b92:::benchmark_data_smithi043_119832_object72975:head# 2024-03-30T13:50:57.999 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.999 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73d76e94:::benchmark_data_smithi043_119832_object69452:head# 2024-03-30T13:50:57.999 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.999 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73d77b81:::benchmark_data_smithi043_119832_object43668:head# 2024-03-30T13:50:57.999 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:57.999 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73d85b18:::benchmark_data_smithi043_119832_object35454:head# 2024-03-30T13:50:57.999 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.000 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73d9b15b:::benchmark_data_smithi043_119832_object22684:head# 2024-03-30T13:50:58.000 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.000 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73dcca66:::benchmark_data_smithi043_119832_object10484:head# 2024-03-30T13:50:58.000 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.001 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73dccfa5:::benchmark_data_smithi043_119832_object77064:head# 2024-03-30T13:50:58.001 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.001 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73dd2a41:::benchmark_data_smithi043_119832_object33453:head# 2024-03-30T13:50:58.001 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.001 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73dd7696:::benchmark_data_smithi043_119832_object55834:head# 2024-03-30T13:50:58.001 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.003 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73de8dd5:::benchmark_data_smithi043_119832_object43535:head# 2024-03-30T13:50:58.003 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.003 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73dee8f0:::benchmark_data_smithi043_119832_object59939:head# 2024-03-30T13:50:58.003 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.003 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73defe7f:::benchmark_data_smithi043_119832_object32199:head# 2024-03-30T13:50:58.003 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.003 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73df1d7c:::benchmark_data_smithi043_119832_object56739:head# 2024-03-30T13:50:58.003 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.004 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73df6f64:::benchmark_data_smithi043_119832_object85180:head# 2024-03-30T13:50:58.004 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.004 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73dfd489:::benchmark_data_smithi043_119832_object71330:head# 2024-03-30T13:50:58.004 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.004 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73e1b1ca:::benchmark_data_smithi043_119832_object51100:head# 2024-03-30T13:50:58.004 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.004 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73e25baa:::benchmark_data_smithi043_119832_object61100:head# 2024-03-30T13:50:58.004 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.006 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73e2d277:::benchmark_data_smithi043_119832_object54996:head# 2024-03-30T13:50:58.006 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.006 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73e2ef9a:::benchmark_data_smithi043_119832_object30739:head# 2024-03-30T13:50:58.006 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.006 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73e2f95e:::benchmark_data_smithi043_119832_object53103:head# 2024-03-30T13:50:58.006 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.006 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73e6eaf3:::benchmark_data_smithi043_119832_object82623:head# 2024-03-30T13:50:58.006 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.006 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73e743ca:::benchmark_data_smithi043_119832_object65198:head# 2024-03-30T13:50:58.006 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.010 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73e91a9e:::benchmark_data_smithi043_119832_object49068:head# 2024-03-30T13:50:58.010 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.010 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73eadd36:::benchmark_data_smithi043_119832_object67516:head# 2024-03-30T13:50:58.010 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.010 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73eb892e:::benchmark_data_smithi043_119832_object88235:head# 2024-03-30T13:50:58.010 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.011 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73ec87b7:::benchmark_data_smithi043_119832_object81868:head# 2024-03-30T13:50:58.011 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.011 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73ec911f:::benchmark_data_smithi043_119832_object20975:head# 2024-03-30T13:50:58.011 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.011 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73ef4eb1:::benchmark_data_smithi043_119832_object37274:head# 2024-03-30T13:50:58.011 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.011 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73f1a80c:::benchmark_data_smithi043_119832_object74002:head# 2024-03-30T13:50:58.011 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.011 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73f1a8ba:::benchmark_data_smithi043_119832_object48759:head# 2024-03-30T13:50:58.011 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.011 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73f2c0f9:::benchmark_data_smithi043_119832_object64510:head# 2024-03-30T13:50:58.011 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.011 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73f49ebf:::benchmark_data_smithi043_119832_object15659:head# 2024-03-30T13:50:58.011 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.011 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73f4d967:::benchmark_data_smithi043_119832_object82543:head# 2024-03-30T13:50:58.012 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.012 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73f602be:::benchmark_data_smithi043_119832_object82548:head# 2024-03-30T13:50:58.012 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.012 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73f6706a:::benchmark_data_smithi043_119832_object60879:head# 2024-03-30T13:50:58.012 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.013 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73f70379:::benchmark_data_smithi043_119832_object418:head# 2024-03-30T13:50:58.013 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.013 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73f723ca:::benchmark_data_smithi043_119832_object70960:head# 2024-03-30T13:50:58.013 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.013 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73f9876a:::benchmark_data_smithi043_119832_object31243:head# 2024-03-30T13:50:58.013 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.013 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73f9dd5b:::benchmark_data_smithi043_119832_object29737:head# 2024-03-30T13:50:58.013 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.013 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73fadcec:::benchmark_data_smithi043_119832_object31489:head# 2024-03-30T13:50:58.013 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.013 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73fb3cd5:::benchmark_data_smithi043_119832_object74939:head# 2024-03-30T13:50:58.014 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.014 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73fb7c0d:::benchmark_data_smithi043_119832_object2763:head# 2024-03-30T13:50:58.014 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.014 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:58.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:58.015 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73fc024c:::benchmark_data_smithi043_119832_object60995:head# 2024-03-30T13:50:58.015 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:58.016 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73fc3968:::benchmark_data_smithi043_119832_object74951:head# 2024-03-30T13:50:58.016 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.016 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73fd7bed:::benchmark_data_smithi043_119832_object87458:head# 2024-03-30T13:50:58.016 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.016 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73fe1a58:::benchmark_data_smithi043_119832_object79155:head# 2024-03-30T13:50:58.016 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.016 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73feba33:::benchmark_data_smithi043_119832_object82472:head# 2024-03-30T13:50:58.016 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.016 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73feedd8:::benchmark_data_smithi043_119832_object83114:head# 2024-03-30T13:50:58.016 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.016 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:73ff697f:::benchmark_data_smithi043_119832_object56562:head# 2024-03-30T13:50:58.016 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.017 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7401bce5:::benchmark_data_smithi043_119832_object16706:head# 2024-03-30T13:50:58.017 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.017 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:740270c0:::benchmark_data_smithi043_119832_object602:head# 2024-03-30T13:50:58.017 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.018 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:740406ae:::benchmark_data_smithi043_119832_object32475:head# 2024-03-30T13:50:58.018 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.018 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:740594ea:::benchmark_data_smithi043_119832_object76886:head# 2024-03-30T13:50:58.018 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.018 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7405e4d0:::benchmark_data_smithi043_119832_object11870:head# 2024-03-30T13:50:58.018 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.018 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7405ec61:::benchmark_data_smithi043_119832_object20777:head# 2024-03-30T13:50:58.018 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.018 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7405fa1e:::benchmark_data_smithi043_119832_object45492:head# 2024-03-30T13:50:58.019 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.019 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:740705cd:::benchmark_data_smithi043_119832_object53322:head# 2024-03-30T13:50:58.019 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.021 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74079c3a:::benchmark_data_smithi043_119832_object2049:head# 2024-03-30T13:50:58.021 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.021 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7407d5be:::benchmark_data_smithi043_119832_object77453:head# 2024-03-30T13:50:58.021 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.021 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:740a805b:::benchmark_data_smithi043_119832_object10693:head# 2024-03-30T13:50:58.022 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.022 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:740ac55b:::benchmark_data_smithi043_119832_object52253:head# 2024-03-30T13:50:58.022 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.022 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:740b761b:::benchmark_data_smithi043_119832_object77692:head# 2024-03-30T13:50:58.022 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.022 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:740c639a:::benchmark_data_smithi043_119832_object63595:head# 2024-03-30T13:50:58.022 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.022 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:740c8eca:::benchmark_data_smithi043_119832_object49121:head# 2024-03-30T13:50:58.022 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.022 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:740d50f8:::benchmark_data_smithi043_119832_object13145:head# 2024-03-30T13:50:58.022 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.022 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:740da3d0:::benchmark_data_smithi043_119832_object85401:head# 2024-03-30T13:50:58.022 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.023 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:740f7a33:::benchmark_data_smithi043_119832_object7362:head# 2024-03-30T13:50:58.023 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.023 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:740fb036:::benchmark_data_smithi043_119832_object30033:head# 2024-03-30T13:50:58.023 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.024 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:740febf7:::benchmark_data_smithi043_119832_object76970:head# 2024-03-30T13:50:58.024 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.024 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7410b57d:::benchmark_data_smithi043_119832_object82605:head# 2024-03-30T13:50:58.024 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.024 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7412b460:::benchmark_data_smithi043_119832_object11290:head# 2024-03-30T13:50:58.024 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.024 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7416982f:::benchmark_data_smithi043_119832_object37816:head# 2024-03-30T13:50:58.025 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.025 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74184d08:::benchmark_data_smithi043_119832_object87664:head# 2024-03-30T13:50:58.025 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.025 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7418d0a4:::benchmark_data_smithi043_119832_object49056:head# 2024-03-30T13:50:58.025 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.025 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:741ab28d:::benchmark_data_smithi043_119832_object40822:head# 2024-03-30T13:50:58.025 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.026 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:741b1841:::benchmark_data_smithi043_119832_object85423:head# 2024-03-30T13:50:58.026 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.026 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:741b6e17:::benchmark_data_smithi043_119832_object38374:head# 2024-03-30T13:50:58.026 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.026 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:741ca002:::benchmark_data_smithi043_119832_object86955:head# 2024-03-30T13:50:58.026 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.027 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:741d17ba:::benchmark_data_smithi043_119832_object61831:head# 2024-03-30T13:50:58.027 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.027 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:741d77bc:::benchmark_data_smithi043_119832_object54117:head# 2024-03-30T13:50:58.027 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.027 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:741df49c:::benchmark_data_smithi043_119832_object64428:head# 2024-03-30T13:50:58.027 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.027 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:741df92a:::benchmark_data_smithi043_119832_object13148:head# 2024-03-30T13:50:58.028 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.028 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:741f6c1f:::benchmark_data_smithi043_119832_object27232:head# 2024-03-30T13:50:58.028 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.029 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:741f76c0:::benchmark_data_smithi043_119832_object76712:head# 2024-03-30T13:50:58.029 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.029 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:741f8726:::benchmark_data_smithi043_119832_object23336:head# 2024-03-30T13:50:58.029 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.029 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7421550a:::benchmark_data_smithi043_119832_object80130:head# 2024-03-30T13:50:58.029 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.029 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74217387:::benchmark_data_smithi043_119832_object4855:head# 2024-03-30T13:50:58.029 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.030 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7421819b:::benchmark_data_smithi043_119832_object86872:head# 2024-03-30T13:50:58.030 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.030 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7424ac0b:::benchmark_data_smithi043_119832_object23335:head# 2024-03-30T13:50:58.030 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.030 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7424bd1b:::benchmark_data_smithi043_119832_object77410:head# 2024-03-30T13:50:58.030 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.031 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74279516:::benchmark_data_smithi043_119832_object50788:head# 2024-03-30T13:50:58.031 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.031 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7427c966:::benchmark_data_smithi043_119832_object8434:head# 2024-03-30T13:50:58.031 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.031 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74283119:::benchmark_data_smithi043_119832_object37663:head# 2024-03-30T13:50:58.032 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.032 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7428a750:::benchmark_data_smithi043_119832_object23107:head# 2024-03-30T13:50:58.032 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.032 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7428cd28:::benchmark_data_smithi043_119832_object62135:head# 2024-03-30T13:50:58.032 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.032 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:742ac255:::benchmark_data_smithi043_119832_object56087:head# 2024-03-30T13:50:58.032 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.034 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:742bddb0:::benchmark_data_smithi043_119832_object36517:head# 2024-03-30T13:50:58.034 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.034 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:742c35b8:::benchmark_data_smithi043_119832_object5170:head# 2024-03-30T13:50:58.034 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.035 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:742c806c:::benchmark_data_smithi043_119832_object77804:head# 2024-03-30T13:50:58.035 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.040 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:742db022:::benchmark_data_smithi043_119832_object32392:head# 2024-03-30T13:50:58.040 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.040 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:742dc1ba:::benchmark_data_smithi043_119832_object72347:head# 2024-03-30T13:50:58.040 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.040 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:742e7d73:::benchmark_data_smithi043_119832_object86829:head# 2024-03-30T13:50:58.040 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.040 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:742e8c5b:::benchmark_data_smithi043_119832_object45429:head# 2024-03-30T13:50:58.040 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.040 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:742efb2b:::benchmark_data_smithi043_119832_object70098:head# 2024-03-30T13:50:58.040 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.040 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:742efd5a:::benchmark_data_smithi043_119832_object29297:head# 2024-03-30T13:50:58.041 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.041 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:742f15f2:::benchmark_data_smithi043_119832_object13039:head# 2024-03-30T13:50:58.041 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.041 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:742f50c9:::benchmark_data_smithi043_119832_object47194:head# 2024-03-30T13:50:58.041 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.041 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:742fd84b:::benchmark_data_smithi043_119832_object73282:head# 2024-03-30T13:50:58.041 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.041 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:742ff03c:::benchmark_data_smithi043_119832_object55117:head# 2024-03-30T13:50:58.041 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.041 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:743148ab:::benchmark_data_smithi043_119832_object51783:head# 2024-03-30T13:50:58.041 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.041 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:743202b8:::benchmark_data_smithi043_119832_object15205:head# 2024-03-30T13:50:58.041 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.041 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74324b7b:::benchmark_data_smithi043_119832_object41042:head# 2024-03-30T13:50:58.042 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.042 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7433309d:::benchmark_data_smithi043_119832_object17006:head# 2024-03-30T13:50:58.042 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.042 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74343668:::benchmark_data_smithi043_119832_object26419:head# 2024-03-30T13:50:58.042 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.044 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7434c2e3:::benchmark_data_smithi043_119832_object73634:head# 2024-03-30T13:50:58.044 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.045 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:743571a8:::benchmark_data_smithi043_119832_object38660:head# 2024-03-30T13:50:58.045 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.045 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7436d649:::benchmark_data_smithi043_119832_object21165:head# 2024-03-30T13:50:58.045 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.045 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:743793ca:::benchmark_data_smithi043_119832_object14946:head# 2024-03-30T13:50:58.045 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.045 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7437ea17:::benchmark_data_smithi043_119832_object18657:head# 2024-03-30T13:50:58.045 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.045 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:743a13ef:::benchmark_data_smithi043_119832_object13421:head# 2024-03-30T13:50:58.045 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.045 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:743ad00b:::benchmark_data_smithi043_119832_object36694:head# 2024-03-30T13:50:58.045 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.045 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:743aefe1:::benchmark_data_smithi043_119832_object16800:head# 2024-03-30T13:50:58.045 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.046 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:743c0e2c:::benchmark_data_smithi043_119832_object17715:head# 2024-03-30T13:50:58.046 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.046 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:743cba9f:::benchmark_data_smithi043_119832_object1974:head# 2024-03-30T13:50:58.046 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.046 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:743da4e4:::benchmark_data_smithi043_119832_object81703:head# 2024-03-30T13:50:58.046 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.046 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:743de688:::benchmark_data_smithi043_119832_object78584:head# 2024-03-30T13:50:58.046 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.046 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:743fbc4a:::benchmark_data_smithi043_119832_object61986:head# 2024-03-30T13:50:58.046 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.048 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7440e616:::benchmark_data_smithi043_119832_object58579:head# 2024-03-30T13:50:58.048 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.048 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74411a2f:::benchmark_data_smithi043_119832_object22985:head# 2024-03-30T13:50:58.048 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.048 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74414014:::benchmark_data_smithi043_119832_object28318:head# 2024-03-30T13:50:58.049 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.049 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7441b9fb:::benchmark_data_smithi043_119832_object18861:head# 2024-03-30T13:50:58.049 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.049 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74421bbb:::benchmark_data_smithi043_119832_object17226:head# 2024-03-30T13:50:58.049 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.049 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74428061:::benchmark_data_smithi043_119832_object21243:head# 2024-03-30T13:50:58.049 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.049 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74449de0:::benchmark_data_smithi043_119832_object13055:head# 2024-03-30T13:50:58.049 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.049 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74455052:::benchmark_data_smithi043_119832_object41796:head# 2024-03-30T13:50:58.049 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.049 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74455c98:::benchmark_data_smithi043_119832_object68312:head# 2024-03-30T13:50:58.049 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.049 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7445f733:::benchmark_data_smithi043_119832_object61111:head# 2024-03-30T13:50:58.050 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.051 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74462d08:::benchmark_data_smithi043_119832_object43936:head# 2024-03-30T13:50:58.051 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.051 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7446c7ca:::benchmark_data_smithi043_119832_object20915:head# 2024-03-30T13:50:58.051 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.051 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7446ca58:::benchmark_data_smithi043_119832_object71332:head# 2024-03-30T13:50:58.051 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.051 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7446cf8e:::benchmark_data_smithi043_119832_object30904:head# 2024-03-30T13:50:58.051 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.051 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74470cf5:::benchmark_data_smithi043_119832_object83225:head# 2024-03-30T13:50:58.051 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.051 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:744898d8:::benchmark_data_smithi043_119832_object40790:head# 2024-03-30T13:50:58.052 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.052 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74489a67:::benchmark_data_smithi043_119832_object8638:head# 2024-03-30T13:50:58.052 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.052 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74491d81:::benchmark_data_smithi043_119832_object8979:head# 2024-03-30T13:50:58.052 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.052 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:744932d1:::benchmark_data_smithi043_119832_object54286:head# 2024-03-30T13:50:58.053 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.053 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7449863a:::benchmark_data_smithi043_119832_object14069:head# 2024-03-30T13:50:58.053 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.053 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7449b3c0:::benchmark_data_smithi043_119832_object29419:head# 2024-03-30T13:50:58.053 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.054 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7449d547:::benchmark_data_smithi043_119832_object32234:head# 2024-03-30T13:50:58.054 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.054 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:744a665c:::benchmark_data_smithi043_119832_object62262:head# 2024-03-30T13:50:58.054 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.054 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:744d4bc5:::benchmark_data_smithi043_119832_object51420:head# 2024-03-30T13:50:58.054 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.055 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:744d8662:::benchmark_data_smithi043_119832_object13259:head# 2024-03-30T13:50:58.055 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.055 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:744fa93c:::benchmark_data_smithi043_119832_object84957:head# 2024-03-30T13:50:58.055 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.055 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74501c98:::benchmark_data_smithi043_119832_object79035:head# 2024-03-30T13:50:58.055 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.055 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74502db5:::benchmark_data_smithi043_119832_object31196:head# 2024-03-30T13:50:58.055 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.056 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74504f45:::benchmark_data_smithi043_119832_object84327:head# 2024-03-30T13:50:58.056 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.056 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7450bd0d:::benchmark_data_smithi043_119832_object68273:head# 2024-03-30T13:50:58.056 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.056 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7450f99c:::benchmark_data_smithi043_119832_object79502:head# 2024-03-30T13:50:58.056 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.057 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74517617:::benchmark_data_smithi043_119832_object55035:head# 2024-03-30T13:50:58.057 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.057 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74522237:::benchmark_data_smithi043_119832_object50448:head# 2024-03-30T13:50:58.057 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.057 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7452a293:::benchmark_data_smithi043_119832_object57226:head# 2024-03-30T13:50:58.057 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.058 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7454f802:::benchmark_data_smithi043_119832_object21171:head# 2024-03-30T13:50:58.058 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.058 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7457763c:::benchmark_data_smithi043_119832_object36895:head# 2024-03-30T13:50:58.058 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.058 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7458526a:::benchmark_data_smithi043_119832_object31069:head# 2024-03-30T13:50:58.058 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.059 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7458f315:::benchmark_data_smithi043_119832_object2562:head# 2024-03-30T13:50:58.059 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.059 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74593d00:::benchmark_data_smithi043_119832_object58196:head# 2024-03-30T13:50:58.059 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.060 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7459d759:::benchmark_data_smithi043_119832_object61096:head# 2024-03-30T13:50:58.060 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.060 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:745a9267:::benchmark_data_smithi043_119832_object47308:head# 2024-03-30T13:50:58.060 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.060 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:745b0bf0:::benchmark_data_smithi043_119832_object26213:head# 2024-03-30T13:50:58.060 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.060 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:745beff6:::benchmark_data_smithi043_119832_object31513:head# 2024-03-30T13:50:58.060 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.061 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:745dc8e7:::benchmark_data_smithi043_119832_object63253:head# 2024-03-30T13:50:58.061 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.061 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:745f0ba5:::benchmark_data_smithi043_119832_object72885:head# 2024-03-30T13:50:58.061 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.062 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7461526d:::benchmark_data_smithi043_119832_object39694:head# 2024-03-30T13:50:58.062 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.062 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7461b4e2:::benchmark_data_smithi043_119832_object16332:head# 2024-03-30T13:50:58.062 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.062 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74623004:::benchmark_data_smithi043_119832_object41654:head# 2024-03-30T13:50:58.063 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.063 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74631496:::benchmark_data_smithi043_119832_object53938:head# 2024-03-30T13:50:58.063 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.063 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:746418c2:::benchmark_data_smithi043_119832_object45709:head# 2024-03-30T13:50:58.063 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.063 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7464a66c:::benchmark_data_smithi043_119832_object13498:head# 2024-03-30T13:50:58.063 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.064 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7464a852:::benchmark_data_smithi043_119832_object8036:head# 2024-03-30T13:50:58.064 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.064 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7464f1ef:::benchmark_data_smithi043_119832_object42157:head# 2024-03-30T13:50:58.064 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.065 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74659957:::benchmark_data_smithi043_119832_object42671:head# 2024-03-30T13:50:58.065 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.065 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7465c81e:::benchmark_data_smithi043_119832_object75206:head# 2024-03-30T13:50:58.065 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.065 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74681802:::benchmark_data_smithi043_119832_object25804:head# 2024-03-30T13:50:58.066 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.066 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:746c0ca0:::benchmark_data_smithi043_119832_object20410:head# 2024-03-30T13:50:58.066 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.066 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:746c922b:::benchmark_data_smithi043_119832_object72228:head# 2024-03-30T13:50:58.066 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.066 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:746ca874:::benchmark_data_smithi043_119832_object56420:head# 2024-03-30T13:50:58.066 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.067 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:746d0b42:::benchmark_data_smithi043_119832_object61588:head# 2024-03-30T13:50:58.067 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.067 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:746d14b1:::benchmark_data_smithi043_119832_object22679:head# 2024-03-30T13:50:58.068 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.068 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:746d3ca3:::benchmark_data_smithi043_119832_object21064:head# 2024-03-30T13:50:58.068 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.068 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:746d6dd7:::benchmark_data_smithi043_119832_object6385:head# 2024-03-30T13:50:58.068 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.068 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:746d8fea:::benchmark_data_smithi043_119832_object67809:head# 2024-03-30T13:50:58.068 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.070 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:746dc4a6:::benchmark_data_smithi043_119832_object68078:head# 2024-03-30T13:50:58.070 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.070 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74701463:::benchmark_data_smithi043_119832_object44660:head# 2024-03-30T13:50:58.070 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.070 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7470c81e:::benchmark_data_smithi043_119832_object64956:head# 2024-03-30T13:50:58.070 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.070 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74710475:::benchmark_data_smithi043_119832_object11311:head# 2024-03-30T13:50:58.070 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.071 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74719c0e:::benchmark_data_smithi043_119832_object84835:head# 2024-03-30T13:50:58.071 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.071 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74720a2f:::benchmark_data_smithi043_119832_object7879:head# 2024-03-30T13:50:58.071 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.071 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74729397:::benchmark_data_smithi043_119832_object67772:head# 2024-03-30T13:50:58.071 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.072 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:747352e1:::benchmark_data_smithi043_119832_object58674:head# 2024-03-30T13:50:58.072 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.072 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7473d260:::benchmark_data_smithi043_119832_object1317:head# 2024-03-30T13:50:58.072 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.073 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74749ad6:::benchmark_data_smithi043_119832_object80643:head# 2024-03-30T13:50:58.073 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.073 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7474b842:::benchmark_data_smithi043_119832_object2585:head# 2024-03-30T13:50:58.073 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.074 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74752c96:::benchmark_data_smithi043_119832_object87769:head# 2024-03-30T13:50:58.074 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.074 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7475f79a:::benchmark_data_smithi043_119832_object31319:head# 2024-03-30T13:50:58.075 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.075 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74766ee3:::benchmark_data_smithi043_119832_object38434:head# 2024-03-30T13:50:58.075 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.075 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7476ed1d:::benchmark_data_smithi043_119832_object23735:head# 2024-03-30T13:50:58.075 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.075 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74773f5e:::benchmark_data_smithi043_119832_object12119:head# 2024-03-30T13:50:58.075 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.075 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7477889a:::benchmark_data_smithi043_119832_object13185:head# 2024-03-30T13:50:58.075 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.076 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7478ad32:::benchmark_data_smithi043_119832_object6879:head# 2024-03-30T13:50:58.076 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.076 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7479d42e:::benchmark_data_smithi043_119832_object30769:head# 2024-03-30T13:50:58.076 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.076 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:747bcedd:::benchmark_data_smithi043_119832_object47752:head# 2024-03-30T13:50:58.076 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.076 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:747c8c8d:::benchmark_data_smithi043_119832_object32897:head# 2024-03-30T13:50:58.076 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.076 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:747e4a78:::benchmark_data_smithi043_119832_object38329:head# 2024-03-30T13:50:58.076 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.077 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:747f08de:::benchmark_data_smithi043_119832_object29150:head# 2024-03-30T13:50:58.077 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.077 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:747f1a35:::benchmark_data_smithi043_119832_object17907:head# 2024-03-30T13:50:58.077 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.077 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:747f9129:::benchmark_data_smithi043_119832_object46363:head# 2024-03-30T13:50:58.077 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.078 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7480382e:::benchmark_data_smithi043_119832_object54895:head# 2024-03-30T13:50:58.078 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.078 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7485381f:::benchmark_data_smithi043_119832_object77034:head# 2024-03-30T13:50:58.078 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.078 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7485e486:::benchmark_data_smithi043_119832_object38066:head# 2024-03-30T13:50:58.079 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.079 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74862fdb:::benchmark_data_smithi043_119832_object32322:head# 2024-03-30T13:50:58.079 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.079 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7486399a:::benchmark_data_smithi043_119832_object11882:head# 2024-03-30T13:50:58.079 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.080 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:748742e7:::benchmark_data_smithi043_119832_object76627:head# 2024-03-30T13:50:58.080 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.080 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:748803e9:::benchmark_data_smithi043_119832_object75236:head# 2024-03-30T13:50:58.080 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.080 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7489593f:::benchmark_data_smithi043_119832_object2627:head# 2024-03-30T13:50:58.080 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.081 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74896771:::benchmark_data_smithi043_119832_object67597:head# 2024-03-30T13:50:58.081 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.081 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74896816:::benchmark_data_smithi043_119832_object7869:head# 2024-03-30T13:50:58.081 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.081 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7489bb48:::benchmark_data_smithi043_119832_object22846:head# 2024-03-30T13:50:58.081 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.082 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7489d593:::benchmark_data_smithi043_119832_object66880:head# 2024-03-30T13:50:58.082 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.082 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:748a0a3c:::benchmark_data_smithi043_119832_object41407:head# 2024-03-30T13:50:58.082 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.083 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:748aa2b6:::benchmark_data_smithi043_119832_object47290:head# 2024-03-30T13:50:58.083 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.083 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:748b3491:::benchmark_data_smithi043_119832_object30795:head# 2024-03-30T13:50:58.083 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.083 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:748c4a51:::benchmark_data_smithi043_119832_object3097:head# 2024-03-30T13:50:58.084 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.084 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:748c58d7:::benchmark_data_smithi043_119832_object79825:head# 2024-03-30T13:50:58.084 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.084 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:748e335e:::benchmark_data_smithi043_119832_object32476:head# 2024-03-30T13:50:58.084 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.085 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:749060e7:::benchmark_data_smithi043_119832_object29064:head# 2024-03-30T13:50:58.085 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.085 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74927cf5:::benchmark_data_smithi043_119832_object68432:head# 2024-03-30T13:50:58.085 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.086 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7492d3f3:::benchmark_data_smithi043_119832_object82828:head# 2024-03-30T13:50:58.086 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.086 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74944047:::benchmark_data_smithi043_119832_object47712:head# 2024-03-30T13:50:58.086 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.086 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7495fadf:::benchmark_data_smithi043_119832_object23727:head# 2024-03-30T13:50:58.087 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.087 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:749849e2:::benchmark_data_smithi043_119832_object70116:head# 2024-03-30T13:50:58.087 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.087 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74985b3a:::benchmark_data_smithi043_119832_object39506:head# 2024-03-30T13:50:58.087 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.088 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74991a7b:::benchmark_data_smithi043_119832_object79900:head# 2024-03-30T13:50:58.088 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.088 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7499cecb:::benchmark_data_smithi043_119832_object80822:head# 2024-03-30T13:50:58.088 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.089 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7499f712:::benchmark_data_smithi043_119832_object9650:head# 2024-03-30T13:50:58.089 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.089 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:749b3693:::benchmark_data_smithi043_119832_object20713:head# 2024-03-30T13:50:58.089 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.089 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:749ba314:::benchmark_data_smithi043_119832_object44560:head# 2024-03-30T13:50:58.089 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.089 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:749d7d82:::benchmark_data_smithi043_119832_object14038:head# 2024-03-30T13:50:58.089 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.090 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:749db2cf:::benchmark_data_smithi043_119832_object61427:head# 2024-03-30T13:50:58.090 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.090 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:749f1dd8:::benchmark_data_smithi043_119832_object85487:head# 2024-03-30T13:50:58.090 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.091 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:749fb908:::benchmark_data_smithi043_119832_object61685:head# 2024-03-30T13:50:58.091 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.091 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:749fe9d5:::benchmark_data_smithi043_119832_object25988:head# 2024-03-30T13:50:58.091 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.092 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74a05c44:::benchmark_data_smithi043_119832_object83948:head# 2024-03-30T13:50:58.092 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.092 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74a20d2a:::benchmark_data_smithi043_119832_object75486:head# 2024-03-30T13:50:58.093 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.093 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74a31bad:::benchmark_data_smithi043_119832_object58019:head# 2024-03-30T13:50:58.093 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.093 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74a32dfc:::benchmark_data_smithi043_119832_object50427:head# 2024-03-30T13:50:58.093 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.094 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74a57ca9:::benchmark_data_smithi043_119832_object60211:head# 2024-03-30T13:50:58.094 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.094 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74a5f050:::benchmark_data_smithi043_119832_object4704:head# 2024-03-30T13:50:58.094 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.094 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74a65eea:::benchmark_data_smithi043_119832_object70333:head# 2024-03-30T13:50:58.095 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.095 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74a67ca8:::benchmark_data_smithi043_119832_object38172:head# 2024-03-30T13:50:58.095 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.095 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74a6904b:::benchmark_data_smithi043_119832_object62906:head# 2024-03-30T13:50:58.095 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.096 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74a6b3dd:::benchmark_data_smithi043_119832_object54674:head# 2024-03-30T13:50:58.096 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.096 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74a6c7e2:::benchmark_data_smithi043_119832_object15325:head# 2024-03-30T13:50:58.096 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.096 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74a79405:::benchmark_data_smithi043_119832_object66754:head# 2024-03-30T13:50:58.096 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.097 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74a9025d:::benchmark_data_smithi043_119832_object88159:head# 2024-03-30T13:50:58.097 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.098 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74abe13f:::benchmark_data_smithi043_119832_object79320:head# 2024-03-30T13:50:58.098 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.098 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74ac5127:::benchmark_data_smithi043_119832_object34941:head# 2024-03-30T13:50:58.098 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.098 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74ace5f9:::benchmark_data_smithi043_119832_object32904:head# 2024-03-30T13:50:58.099 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.099 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74addf30:::benchmark_data_smithi043_119832_object29757:head# 2024-03-30T13:50:58.099 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.101 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74aee9e6:::benchmark_data_smithi043_119832_object6227:head# 2024-03-30T13:50:58.101 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.101 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74af1b19:::benchmark_data_smithi043_119832_object79082:head# 2024-03-30T13:50:58.102 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.102 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74af2d6c:::benchmark_data_smithi043_119832_object88395:head# 2024-03-30T13:50:58.102 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.102 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74af8755:::benchmark_data_smithi043_119832_object46390:head# 2024-03-30T13:50:58.102 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.102 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74b0328d:::benchmark_data_smithi043_119832_object62378:head# 2024-03-30T13:50:58.102 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.102 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74b0c081:::benchmark_data_smithi043_119832_object14883:head# 2024-03-30T13:50:58.102 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.102 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74b13fc7:::benchmark_data_smithi043_119832_object65252:head# 2024-03-30T13:50:58.102 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.102 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74b2aae7:::benchmark_data_smithi043_119832_object64506:head# 2024-03-30T13:50:58.102 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.105 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74b2b412:::benchmark_data_smithi043_119832_object51979:head# 2024-03-30T13:50:58.105 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.105 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74b32d11:::benchmark_data_smithi043_119832_object83588:head# 2024-03-30T13:50:58.105 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.105 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74b36e79:::benchmark_data_smithi043_119832_object54853:head# 2024-03-30T13:50:58.105 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.105 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74b47f10:::benchmark_data_smithi043_119832_object35893:head# 2024-03-30T13:50:58.105 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.105 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74b7196d:::benchmark_data_smithi043_119832_object53153:head# 2024-03-30T13:50:58.105 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.106 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74b74844:::benchmark_data_smithi043_119832_object22801:head# 2024-03-30T13:50:58.106 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.106 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74b7c2d7:::benchmark_data_smithi043_119832_object84174:head# 2024-03-30T13:50:58.106 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.106 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74b916c1:::benchmark_data_smithi043_119832_object66494:head# 2024-03-30T13:50:58.106 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.106 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74b9353e:::benchmark_data_smithi043_119832_object32526:head# 2024-03-30T13:50:58.106 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.106 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74ba566e:::benchmark_data_smithi043_119832_object58747:head# 2024-03-30T13:50:58.106 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.107 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74bb718b:::benchmark_data_smithi043_119832_object27492:head# 2024-03-30T13:50:58.107 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.107 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74bcee50:::benchmark_data_smithi043_119832_object31233:head# 2024-03-30T13:50:58.107 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.107 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74bd98da:::benchmark_data_smithi043_119832_object7977:head# 2024-03-30T13:50:58.107 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.107 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74be7f4d:::benchmark_data_smithi043_119832_object17180:head# 2024-03-30T13:50:58.108 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.108 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74bf3ef7:::benchmark_data_smithi043_119832_object70888:head# 2024-03-30T13:50:58.108 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.108 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74c0e5db:::benchmark_data_smithi043_119832_object14906:head# 2024-03-30T13:50:58.108 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.110 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74c19816:::benchmark_data_smithi043_119832_object16963:head# 2024-03-30T13:50:58.110 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.110 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74c1ee42:::benchmark_data_smithi043_119832_object3116:head# 2024-03-30T13:50:58.110 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.110 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74c1f3b8:::benchmark_data_smithi043_119832_object32528:head# 2024-03-30T13:50:58.110 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.110 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74c24d87:::benchmark_data_smithi043_119832_object14501:head# 2024-03-30T13:50:58.110 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.110 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74c2c873:::benchmark_data_smithi043_119832_object7944:head# 2024-03-30T13:50:58.111 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.111 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74c306b1:::benchmark_data_smithi043_119832_object27098:head# 2024-03-30T13:50:58.111 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.111 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74c3f5a7:::benchmark_data_smithi043_119832_object87147:head# 2024-03-30T13:50:58.111 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.112 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74c43398:::benchmark_data_smithi043_119832_object80207:head# 2024-03-30T13:50:58.112 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.112 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74c46838:::benchmark_data_smithi043_119832_object11601:head# 2024-03-30T13:50:58.112 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.112 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74c4ebfd:::benchmark_data_smithi043_119832_object1101:head# 2024-03-30T13:50:58.112 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.112 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74c4f651:::benchmark_data_smithi043_119832_object70257:head# 2024-03-30T13:50:58.112 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.112 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74c51ade:::benchmark_data_smithi043_119832_object34111:head# 2024-03-30T13:50:58.112 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.113 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74c54744:::benchmark_data_smithi043_119832_object85091:head# 2024-03-30T13:50:58.113 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.113 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74c5e10c:::benchmark_data_smithi043_119832_object78225:head# 2024-03-30T13:50:58.113 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.113 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74c64f71:::benchmark_data_smithi043_119832_object63893:head# 2024-03-30T13:50:58.113 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.113 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74c6a50e:::benchmark_data_smithi043_119832_object35444:head# 2024-03-30T13:50:58.113 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.114 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74c7a8a3:::benchmark_data_smithi043_119832_object27326:head# 2024-03-30T13:50:58.114 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.114 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74c7b8ac:::benchmark_data_smithi043_119832_object86909:head# 2024-03-30T13:50:58.114 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.116 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:58.116 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74c98dc3:::benchmark_data_smithi043_119832_object8324:head# 2024-03-30T13:50:58.116 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.116 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74ca4cf1:::benchmark_data_smithi043_119832_object23485:head# 2024-03-30T13:50:58.117 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.117 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74cb03df:::benchmark_data_smithi043_119832_object61455:head# 2024-03-30T13:50:58.117 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.117 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74cb15e2:::benchmark_data_smithi043_119832_object46964:head# 2024-03-30T13:50:58.117 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.117 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74cd4a6f:::benchmark_data_smithi043_119832_object41520:head# 2024-03-30T13:50:58.117 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.117 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74ce262a:::benchmark_data_smithi043_119832_object81962:head# 2024-03-30T13:50:58.117 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.119 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74ceb1e0:::benchmark_data_smithi043_119832_object41873:head# 2024-03-30T13:50:58.119 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.119 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74cec1dd:::benchmark_data_smithi043_119832_object45096:head# 2024-03-30T13:50:58.119 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.119 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74cee7d8:::benchmark_data_smithi043_119832_object13719:head# 2024-03-30T13:50:58.120 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.120 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74cefea9:::benchmark_data_smithi043_119832_object18980:head# 2024-03-30T13:50:58.120 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.120 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74d0017e:::benchmark_data_smithi043_119832_object22305:head# 2024-03-30T13:50:58.120 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.120 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74d0ab62:::benchmark_data_smithi043_119832_object20925:head# 2024-03-30T13:50:58.120 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.120 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74d0b6f7:::benchmark_data_smithi043_119832_object69139:head# 2024-03-30T13:50:58.120 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.120 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74d1bbd4:::benchmark_data_smithi043_119832_object85989:head# 2024-03-30T13:50:58.120 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.120 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74d2291f:::benchmark_data_smithi043_119832_object51940:head# 2024-03-30T13:50:58.120 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.122 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74d22ef5:::benchmark_data_smithi043_119832_object23408:head# 2024-03-30T13:50:58.122 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.122 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74d3d2ec:::benchmark_data_smithi043_119832_object85840:head# 2024-03-30T13:50:58.122 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.122 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74d3d3cd:::benchmark_data_smithi043_119832_object57104:head# 2024-03-30T13:50:58.122 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.122 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74d4437b:::benchmark_data_smithi043_119832_object79156:head# 2024-03-30T13:50:58.122 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.122 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74d49aee:::benchmark_data_smithi043_119832_object13328:head# 2024-03-30T13:50:58.122 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.122 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74d53688:::benchmark_data_smithi043_119832_object34821:head# 2024-03-30T13:50:58.122 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.124 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74d5d540:::benchmark_data_smithi043_119832_object51256:head# 2024-03-30T13:50:58.124 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.124 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74d61a20:::benchmark_data_smithi043_119832_object3824:head# 2024-03-30T13:50:58.124 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.124 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74d62faa:::benchmark_data_smithi043_119832_object74338:head# 2024-03-30T13:50:58.125 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.125 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74d6c175:::benchmark_data_smithi043_119832_object52453:head# 2024-03-30T13:50:58.125 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.125 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74d72172:::benchmark_data_smithi043_119832_object82577:head# 2024-03-30T13:50:58.126 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.126 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74d77f23:::benchmark_data_smithi043_119832_object52620:head# 2024-03-30T13:50:58.126 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.126 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74d7d1ba:::benchmark_data_smithi043_119832_object49602:head# 2024-03-30T13:50:58.126 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.126 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74db9a4d:::benchmark_data_smithi043_119832_object32326:head# 2024-03-30T13:50:58.126 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.126 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74db9bb8:::benchmark_data_smithi043_119832_object56051:head# 2024-03-30T13:50:58.126 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.126 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74dc926c:::benchmark_data_smithi043_119832_object9033:head# 2024-03-30T13:50:58.126 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.126 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74ddcb49:::benchmark_data_smithi043_119832_object82932:head# 2024-03-30T13:50:58.126 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.127 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74df0767:::benchmark_data_smithi043_119832_object63499:head# 2024-03-30T13:50:58.127 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.127 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74df13cf:::benchmark_data_smithi043_119832_object31101:head# 2024-03-30T13:50:58.127 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.127 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74df35bb:::benchmark_data_smithi043_119832_object58375:head# 2024-03-30T13:50:58.127 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.127 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74df7dd5:::benchmark_data_smithi043_119832_object84882:head# 2024-03-30T13:50:58.127 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.128 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74dff8f5:::benchmark_data_smithi043_119832_object67079:head# 2024-03-30T13:50:58.128 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.128 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74e03cd3:::benchmark_data_smithi043_119832_object71364:head# 2024-03-30T13:50:58.128 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.128 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74e0bac8:::benchmark_data_smithi043_119832_object44477:head# 2024-03-30T13:50:58.128 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.128 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74e137f9:::benchmark_data_smithi043_119832_object57992:head# 2024-03-30T13:50:58.129 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.129 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74e35f7c:::benchmark_data_smithi043_119832_object45719:head# 2024-03-30T13:50:58.129 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.129 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74e3ec1c:::benchmark_data_smithi043_119832_object19161:head# 2024-03-30T13:50:58.129 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.129 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74e51dc7:::benchmark_data_smithi043_119832_object10160:head# 2024-03-30T13:50:58.129 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.132 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74e53ef0:::benchmark_data_smithi043_119832_object39340:head# 2024-03-30T13:50:58.132 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.132 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74e5eda0:::benchmark_data_smithi043_119832_object2501:head# 2024-03-30T13:50:58.132 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.132 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74e65b6b:::benchmark_data_smithi043_119832_object24375:head# 2024-03-30T13:50:58.132 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.132 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74e690a4:::benchmark_data_smithi043_119832_object72844:head# 2024-03-30T13:50:58.132 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.133 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74e6a5ed:::benchmark_data_smithi043_119832_object5187:head# 2024-03-30T13:50:58.133 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.133 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74e731ce:::benchmark_data_smithi043_119832_object37866:head# 2024-03-30T13:50:58.133 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.133 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74e80d23:::benchmark_data_smithi043_119832_object28826:head# 2024-03-30T13:50:58.133 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.133 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74e814dc:::benchmark_data_smithi043_119832_object26808:head# 2024-03-30T13:50:58.133 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.133 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74e8bc30:::benchmark_data_smithi043_119832_object14161:head# 2024-03-30T13:50:58.133 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.134 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74ea1a8d:::benchmark_data_smithi043_119832_object16734:head# 2024-03-30T13:50:58.134 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.135 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74ea752a:::benchmark_data_smithi043_119832_object55689:head# 2024-03-30T13:50:58.135 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.135 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74ea757c:::benchmark_data_smithi043_119832_object72903:head# 2024-03-30T13:50:58.135 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.135 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74ead0c7:::benchmark_data_smithi043_119832_object12037:head# 2024-03-30T13:50:58.135 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.135 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74eb11ae:::benchmark_data_smithi043_119832_object75905:head# 2024-03-30T13:50:58.135 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.135 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74ebb5dd:::benchmark_data_smithi043_119832_object53427:head# 2024-03-30T13:50:58.135 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.135 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74ec0f2a:::benchmark_data_smithi043_119832_object8733:head# 2024-03-30T13:50:58.135 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.135 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74ece889:::benchmark_data_smithi043_119832_object18806:head# 2024-03-30T13:50:58.135 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.136 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74ed1dc6:::benchmark_data_smithi043_119832_object50658:head# 2024-03-30T13:50:58.136 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.136 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74edfea3:::benchmark_data_smithi043_119832_object85557:head# 2024-03-30T13:50:58.136 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.136 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74ef0964:::benchmark_data_smithi043_119832_object9130:head# 2024-03-30T13:50:58.136 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.137 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74ef9df0:::benchmark_data_smithi043_119832_object66774:head# 2024-03-30T13:50:58.137 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.137 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74efeec7:::benchmark_data_smithi043_119832_object7055:head# 2024-03-30T13:50:58.137 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.137 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74f08335:::benchmark_data_smithi043_119832_object16249:head# 2024-03-30T13:50:58.139 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.140 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74f08af4:::benchmark_data_smithi043_119832_object69566:head# 2024-03-30T13:50:58.140 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.140 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74f14aa4:::benchmark_data_smithi043_119832_object80020:head# 2024-03-30T13:50:58.140 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.140 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74f270c0:::benchmark_data_smithi043_119832_object61619:head# 2024-03-30T13:50:58.140 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.140 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74f317b5:::benchmark_data_smithi043_119832_object8619:head# 2024-03-30T13:50:58.140 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.140 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74f4c0a4:::benchmark_data_smithi043_119832_object56422:head# 2024-03-30T13:50:58.140 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.140 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74f58fd1:::benchmark_data_smithi043_119832_object29484:head# 2024-03-30T13:50:58.140 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.140 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74f5cd6d:::benchmark_data_smithi043_119832_object88225:head# 2024-03-30T13:50:58.141 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.141 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74f6e5c1:::benchmark_data_smithi043_119832_object8465:head# 2024-03-30T13:50:58.141 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.141 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74f77217:::benchmark_data_smithi043_119832_object44826:head# 2024-03-30T13:50:58.141 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.143 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74fac1c4:::benchmark_data_smithi043_119832_object38178:head# 2024-03-30T13:50:58.143 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.143 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74fac6f6:::benchmark_data_smithi043_119832_object25887:head# 2024-03-30T13:50:58.143 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.143 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74fdec1e:::benchmark_data_smithi043_119832_object49745:head# 2024-03-30T13:50:58.143 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.143 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74fe0023:::benchmark_data_smithi043_119832_object65750:head# 2024-03-30T13:50:58.143 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.143 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:74fe3c0f:::benchmark_data_smithi043_119832_object47085:head# 2024-03-30T13:50:58.143 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.143 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7500585c:::benchmark_data_smithi043_119832_object3698:head# 2024-03-30T13:50:58.143 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.144 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7500ee6e:::benchmark_data_smithi043_119832_object25309:head# 2024-03-30T13:50:58.144 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.144 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75015ed1:::benchmark_data_smithi043_119832_object68803:head# 2024-03-30T13:50:58.144 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.144 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:750161a6:::benchmark_data_smithi043_119832_object87447:head# 2024-03-30T13:50:58.144 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.144 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:750355b0:::benchmark_data_smithi043_119832_object34902:head# 2024-03-30T13:50:58.144 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.145 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7503844a:::benchmark_data_smithi043_119832_object67383:head# 2024-03-30T13:50:58.145 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.145 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75046f99:::benchmark_data_smithi043_119832_object17353:head# 2024-03-30T13:50:58.145 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.145 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7504d303:::benchmark_data_smithi043_119832_object46757:head# 2024-03-30T13:50:58.145 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.145 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7504dfdf:::benchmark_data_smithi043_119832_object58623:head# 2024-03-30T13:50:58.146 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.146 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7505ebd7:::benchmark_data_smithi043_119832_object33488:head# 2024-03-30T13:50:58.146 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.146 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7506b915:::benchmark_data_smithi043_119832_object25790:head# 2024-03-30T13:50:58.146 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.146 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7506be30:::benchmark_data_smithi043_119832_object32562:head# 2024-03-30T13:50:58.146 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.147 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7506ec29:::benchmark_data_smithi043_119832_object13190:head# 2024-03-30T13:50:58.147 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.147 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75075761:::benchmark_data_smithi043_119832_object80917:head# 2024-03-30T13:50:58.147 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.147 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75079fe1:::benchmark_data_smithi043_119832_object40266:head# 2024-03-30T13:50:58.147 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.147 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7507c34e:::benchmark_data_smithi043_119832_object21144:head# 2024-03-30T13:50:58.147 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.148 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75080175:::benchmark_data_smithi043_119832_object18788:head# 2024-03-30T13:50:58.148 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.148 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75080819:::benchmark_data_smithi043_119832_object72793:head# 2024-03-30T13:50:58.148 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.148 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7508f4f0:::benchmark_data_smithi043_119832_object74122:head# 2024-03-30T13:50:58.148 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.148 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:750997db:::benchmark_data_smithi043_119832_object32987:head# 2024-03-30T13:50:58.148 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.149 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7509e7b8:::benchmark_data_smithi043_119832_object60309:head# 2024-03-30T13:50:58.149 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.149 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:750b02e6:::benchmark_data_smithi043_119832_object85915:head# 2024-03-30T13:50:58.149 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.149 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:750b4043:::benchmark_data_smithi043_119832_object22415:head# 2024-03-30T13:50:58.149 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.150 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:750be059:::benchmark_data_smithi043_119832_object35528:head# 2024-03-30T13:50:58.150 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.150 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:750bf078:::benchmark_data_smithi043_119832_object13260:head# 2024-03-30T13:50:58.150 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.150 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:750d4278:::benchmark_data_smithi043_119832_object69724:head# 2024-03-30T13:50:58.150 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.151 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:750d8436:::benchmark_data_smithi043_119832_object4416:head# 2024-03-30T13:50:58.151 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.151 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:750ddd97:::benchmark_data_smithi043_119832_object52572:head# 2024-03-30T13:50:58.151 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.151 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:750df975:::benchmark_data_smithi043_119832_object49808:head# 2024-03-30T13:50:58.152 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.152 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:750e1696:::benchmark_data_smithi043_119832_object44439:head# 2024-03-30T13:50:58.152 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.152 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:750f18c4:::benchmark_data_smithi043_119832_object76763:head# 2024-03-30T13:50:58.152 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.152 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:750f86f9:::benchmark_data_smithi043_119832_object78355:head# 2024-03-30T13:50:58.152 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.153 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75128c20:::benchmark_data_smithi043_119832_object51378:head# 2024-03-30T13:50:58.153 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.153 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7514346b:::benchmark_data_smithi043_119832_object87595:head# 2024-03-30T13:50:58.153 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.154 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7514e32d:::benchmark_data_smithi043_119832_object74112:head# 2024-03-30T13:50:58.154 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.154 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75170f8b:::benchmark_data_smithi043_119832_object12102:head# 2024-03-30T13:50:58.154 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.154 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7518d212:::benchmark_data_smithi043_119832_object37597:head# 2024-03-30T13:50:58.154 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.155 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7518ddeb:::benchmark_data_smithi043_119832_object40193:head# 2024-03-30T13:50:58.155 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.155 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:751a2db0:::benchmark_data_smithi043_119832_object59111:head# 2024-03-30T13:50:58.155 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.156 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:751a5138:::benchmark_data_smithi043_119832_object76919:head# 2024-03-30T13:50:58.156 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.156 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:751c207b:::benchmark_data_smithi043_119832_object71370:head# 2024-03-30T13:50:58.156 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.156 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:751c575c:::benchmark_data_smithi043_119832_object27667:head# 2024-03-30T13:50:58.157 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.157 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:751d2c2c:::benchmark_data_smithi043_119832_object1511:head# 2024-03-30T13:50:58.157 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.158 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:751dba47:::benchmark_data_smithi043_119832_object26831:head# 2024-03-30T13:50:58.158 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.158 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:751fe2b0:::benchmark_data_smithi043_119832_object16065:head# 2024-03-30T13:50:58.158 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.158 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:751ff981:::benchmark_data_smithi043_119832_object53888:head# 2024-03-30T13:50:58.159 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.159 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:752019a1:::benchmark_data_smithi043_119832_object17067:head# 2024-03-30T13:50:58.159 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.159 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:752055bd:::benchmark_data_smithi043_119832_object77298:head# 2024-03-30T13:50:58.159 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.160 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7521d60f:::benchmark_data_smithi043_119832_object71428:head# 2024-03-30T13:50:58.160 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.160 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7522413c:::benchmark_data_smithi043_119832_object19472:head# 2024-03-30T13:50:58.160 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.160 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7524e4c2:::benchmark_data_smithi043_119832_object21782:head# 2024-03-30T13:50:58.160 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.161 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7525bdd5:::benchmark_data_smithi043_119832_object18080:head# 2024-03-30T13:50:58.161 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.161 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7526357c:::benchmark_data_smithi043_119832_object39325:head# 2024-03-30T13:50:58.161 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.161 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75263d32:::benchmark_data_smithi043_119832_object43590:head# 2024-03-30T13:50:58.161 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.162 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:58.162 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:752707d0:::benchmark_data_smithi043_119832_object64492:head# 2024-03-30T13:50:58.162 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.162 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7527e9e8:::benchmark_data_smithi043_119832_object78187:head# 2024-03-30T13:50:58.162 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:58.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:58.164 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7528f14c:::benchmark_data_smithi043_119832_object60827:head# 2024-03-30T13:50:58.164 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.164 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75292d82:::benchmark_data_smithi043_119832_object27582:head# 2024-03-30T13:50:58.164 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.164 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75296988:::benchmark_data_smithi043_119832_object82845:head# 2024-03-30T13:50:58.164 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.164 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:752a7007:::benchmark_data_smithi043_119832_object55772:head# 2024-03-30T13:50:58.165 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.165 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:752ba09a:::benchmark_data_smithi043_119832_object6471:head# 2024-03-30T13:50:58.165 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.165 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:752bd9f4:::benchmark_data_smithi043_119832_object39440:head# 2024-03-30T13:50:58.165 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.165 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:752be8e8:::benchmark_data_smithi043_119832_object84131:head# 2024-03-30T13:50:58.166 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.166 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:752dbd55:::benchmark_data_smithi043_119832_object29441:head# 2024-03-30T13:50:58.166 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.166 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:752f52af:::benchmark_data_smithi043_119832_object5340:head# 2024-03-30T13:50:58.166 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.166 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:752fcc60:::benchmark_data_smithi043_119832_object71494:head# 2024-03-30T13:50:58.166 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.167 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:752ff68b:::benchmark_data_smithi043_119832_object72794:head# 2024-03-30T13:50:58.167 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.167 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:753103a4:::benchmark_data_smithi043_119832_object76893:head# 2024-03-30T13:50:58.167 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.167 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75320db0:::benchmark_data_smithi043_119832_object78811:head# 2024-03-30T13:50:58.167 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.167 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:753222aa:::benchmark_data_smithi043_119832_object74809:head# 2024-03-30T13:50:58.168 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.168 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75329b72:::benchmark_data_smithi043_119832_object66915:head# 2024-03-30T13:50:58.168 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.168 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7532b309:::benchmark_data_smithi043_119832_object18268:head# 2024-03-30T13:50:58.168 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.168 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:753444bf:::benchmark_data_smithi043_119832_object12576:head# 2024-03-30T13:50:58.168 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.169 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:753519d5:::benchmark_data_smithi043_119832_object31118:head# 2024-03-30T13:50:58.169 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.169 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75354f64:::benchmark_data_smithi043_119832_object49789:head# 2024-03-30T13:50:58.169 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.169 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75357785:::benchmark_data_smithi043_119832_object8449:head# 2024-03-30T13:50:58.170 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.170 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7537a950:::benchmark_data_smithi043_119832_object87936:head# 2024-03-30T13:50:58.170 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.171 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75382af5:::benchmark_data_smithi043_119832_object26576:head# 2024-03-30T13:50:58.171 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.171 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7538cbdf:::benchmark_data_smithi043_119832_object37515:head# 2024-03-30T13:50:58.171 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.172 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7539bab0:::benchmark_data_smithi043_119832_object79400:head# 2024-03-30T13:50:58.172 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.172 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:753a0e99:::benchmark_data_smithi043_119832_object3694:head# 2024-03-30T13:50:58.172 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.172 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:753a19d3:::benchmark_data_smithi043_119832_object64898:head# 2024-03-30T13:50:58.172 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.172 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:753a4576:::benchmark_data_smithi043_119832_object39443:head# 2024-03-30T13:50:58.172 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.174 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:753a840b:::benchmark_data_smithi043_119832_object80101:head# 2024-03-30T13:50:58.174 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.174 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:753b4a73:::benchmark_data_smithi043_119832_object23396:head# 2024-03-30T13:50:58.174 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.174 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:753bedb7:::benchmark_data_smithi043_119832_object78294:head# 2024-03-30T13:50:58.175 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.175 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:753c002f:::benchmark_data_smithi043_119832_object15082:head# 2024-03-30T13:50:58.175 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.175 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:753c0b4e:::benchmark_data_smithi043_119832_object40336:head# 2024-03-30T13:50:58.175 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.175 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:753c198d:::benchmark_data_smithi043_119832_object72806:head# 2024-03-30T13:50:58.175 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.175 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:753d5a5b:::benchmark_data_smithi043_119832_object12629:head# 2024-03-30T13:50:58.175 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.177 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:753d889c:::benchmark_data_smithi043_119832_object64539:head# 2024-03-30T13:50:58.177 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.177 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:753e9ce2:::benchmark_data_smithi043_119832_object10828:head# 2024-03-30T13:50:58.177 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.177 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:753f3624:::benchmark_data_smithi043_119832_object77040:head# 2024-03-30T13:50:58.177 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.177 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:753f86f7:::benchmark_data_smithi043_119832_object17340:head# 2024-03-30T13:50:58.177 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.177 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75403c38:::benchmark_data_smithi043_119832_object82272:head# 2024-03-30T13:50:58.178 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.178 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7540456f:::benchmark_data_smithi043_119832_object8756:head# 2024-03-30T13:50:58.178 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.178 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75405e6d:::benchmark_data_smithi043_119832_object15487:head# 2024-03-30T13:50:58.178 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.178 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75406065:::benchmark_data_smithi043_119832_object33721:head# 2024-03-30T13:50:58.178 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.180 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7541f871:::benchmark_data_smithi043_119832_object16936:head# 2024-03-30T13:50:58.180 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.180 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:754279e8:::benchmark_data_smithi043_119832_object59216:head# 2024-03-30T13:50:58.180 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.180 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7542e1ec:::benchmark_data_smithi043_119832_object45541:head# 2024-03-30T13:50:58.180 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.180 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7542ec25:::benchmark_data_smithi043_119832_object46591:head# 2024-03-30T13:50:58.180 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.181 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75434f68:::benchmark_data_smithi043_119832_object48539:head# 2024-03-30T13:50:58.181 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.181 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7544a3f3:::benchmark_data_smithi043_119832_object81632:head# 2024-03-30T13:50:58.181 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.181 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7544ecb7:::benchmark_data_smithi043_119832_object86331:head# 2024-03-30T13:50:58.181 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.181 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:754591ef:::benchmark_data_smithi043_119832_object40037:head# 2024-03-30T13:50:58.181 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.181 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7545cfa6:::benchmark_data_smithi043_119832_object12971:head# 2024-03-30T13:50:58.181 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.182 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7546a1d2:::benchmark_data_smithi043_119832_object78452:head# 2024-03-30T13:50:58.182 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.182 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75479e18:::benchmark_data_smithi043_119832_object59300:head# 2024-03-30T13:50:58.182 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.182 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7547da53:::benchmark_data_smithi043_119832_object64337:head# 2024-03-30T13:50:58.182 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.183 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:754858e8:::benchmark_data_smithi043_119832_object84837:head# 2024-03-30T13:50:58.183 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.183 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75491cd2:::benchmark_data_smithi043_119832_object5407:head# 2024-03-30T13:50:58.183 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.183 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7549c8a9:::benchmark_data_smithi043_119832_object46653:head# 2024-03-30T13:50:58.183 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.184 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:754b15ec:::benchmark_data_smithi043_119832_object14880:head# 2024-03-30T13:50:58.184 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.184 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:754b3358:::benchmark_data_smithi043_119832_object31644:head# 2024-03-30T13:50:58.184 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.184 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:754b3919:::benchmark_data_smithi043_119832_object66482:head# 2024-03-30T13:50:58.184 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.184 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:754c29d0:::benchmark_data_smithi043_119832_object87886:head# 2024-03-30T13:50:58.184 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.185 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:754e84da:::benchmark_data_smithi043_119832_object56838:head# 2024-03-30T13:50:58.185 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.185 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75517ae8:::benchmark_data_smithi043_119832_object11263:head# 2024-03-30T13:50:58.185 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.185 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7551fcf7:::benchmark_data_smithi043_119832_object3284:head# 2024-03-30T13:50:58.185 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.185 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75538484:::benchmark_data_smithi043_119832_object85165:head# 2024-03-30T13:50:58.186 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.186 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7553cd8d:::benchmark_data_smithi043_119832_object61095:head# 2024-03-30T13:50:58.186 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.186 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7554131b:::benchmark_data_smithi043_119832_object12858:head# 2024-03-30T13:50:58.186 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.187 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75541bf0:::benchmark_data_smithi043_119832_object67168:head# 2024-03-30T13:50:58.187 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.187 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7554976c:::benchmark_data_smithi043_119832_object85415:head# 2024-03-30T13:50:58.187 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.188 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7554be85:::benchmark_data_smithi043_119832_object81374:head# 2024-03-30T13:50:58.188 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.188 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7556135c:::benchmark_data_smithi043_119832_object79107:head# 2024-03-30T13:50:58.188 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.188 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75565a2a:::benchmark_data_smithi043_119832_object53013:head# 2024-03-30T13:50:58.188 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.189 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:755864a2:::benchmark_data_smithi043_119832_object22379:head# 2024-03-30T13:50:58.189 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.189 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:755874c8:::benchmark_data_smithi043_119832_object72556:head# 2024-03-30T13:50:58.189 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.190 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75592d9d:::benchmark_data_smithi043_119832_object31848:head# 2024-03-30T13:50:58.190 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.191 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:755a2941:::benchmark_data_smithi043_119832_object49643:head# 2024-03-30T13:50:58.191 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.192 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:755a6c11:::benchmark_data_smithi043_119832_object35890:head# 2024-03-30T13:50:58.192 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.192 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:755a74a6:::benchmark_data_smithi043_119832_object85908:head# 2024-03-30T13:50:58.192 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.192 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:755ac514:::benchmark_data_smithi043_119832_object60196:head# 2024-03-30T13:50:58.192 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.193 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:755b62e7:::benchmark_data_smithi043_119832_object9328:head# 2024-03-30T13:50:58.193 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.193 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:755c92ce:::benchmark_data_smithi043_119832_object85606:head# 2024-03-30T13:50:58.193 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.193 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:755d209b:::benchmark_data_smithi043_119832_object65882:head# 2024-03-30T13:50:58.193 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.193 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:755d44b1:::benchmark_data_smithi043_119832_object45197:head# 2024-03-30T13:50:58.194 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.194 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:755db233:::benchmark_data_smithi043_119832_object54243:head# 2024-03-30T13:50:58.194 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.194 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:755f7db2:::benchmark_data_smithi043_119832_object70637:head# 2024-03-30T13:50:58.194 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.195 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:755fc5f2:::benchmark_data_smithi043_119832_object57713:head# 2024-03-30T13:50:58.195 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.195 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:756215ef:::benchmark_data_smithi043_119832_object57516:head# 2024-03-30T13:50:58.195 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.196 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75625266:::benchmark_data_smithi043_119832_object4905:head# 2024-03-30T13:50:58.196 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.196 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75643e33:::benchmark_data_smithi043_119832_object60555:head# 2024-03-30T13:50:58.196 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.197 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75648f5b:::benchmark_data_smithi043_119832_object6264:head# 2024-03-30T13:50:58.197 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.197 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:756581d0:::benchmark_data_smithi043_119832_object38870:head# 2024-03-30T13:50:58.197 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.197 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75663ad1:::benchmark_data_smithi043_119832_object51812:head# 2024-03-30T13:50:58.197 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.198 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:756a7a29:::benchmark_data_smithi043_119832_object35747:head# 2024-03-30T13:50:58.198 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.198 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:756af728:::benchmark_data_smithi043_119832_object80014:head# 2024-03-30T13:50:58.198 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.199 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:756c02f8:::benchmark_data_smithi043_119832_object9890:head# 2024-03-30T13:50:58.199 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.199 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:756c77a4:::benchmark_data_smithi043_119832_object12999:head# 2024-03-30T13:50:58.199 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.200 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:756c7eac:::benchmark_data_smithi043_119832_object65744:head# 2024-03-30T13:50:58.200 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.200 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:756c9fb2:::benchmark_data_smithi043_119832_object49228:head# 2024-03-30T13:50:58.200 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.200 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:756d5cc9:::benchmark_data_smithi043_119832_object8092:head# 2024-03-30T13:50:58.201 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.201 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:756db4a1:::benchmark_data_smithi043_119832_object35811:head# 2024-03-30T13:50:58.201 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.201 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:756f2bbb:::benchmark_data_smithi043_119832_object11770:head# 2024-03-30T13:50:58.201 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.201 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:756f393e:::benchmark_data_smithi043_119832_object925:head# 2024-03-30T13:50:58.202 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.202 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:756f7f09:::benchmark_data_smithi043_119832_object13066:head# 2024-03-30T13:50:58.202 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.202 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:756feb1f:::benchmark_data_smithi043_119832_object11254:head# 2024-03-30T13:50:58.202 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.203 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7571a3c6:::benchmark_data_smithi043_119832_object4431:head# 2024-03-30T13:50:58.203 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.203 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7571a4aa:::benchmark_data_smithi043_119832_object54049:head# 2024-03-30T13:50:58.203 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.203 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75722098:::benchmark_data_smithi043_119832_object8996:head# 2024-03-30T13:50:58.203 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.204 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75729b64:::benchmark_data_smithi043_119832_object37096:head# 2024-03-30T13:50:58.204 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.204 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:757323bb:::benchmark_data_smithi043_119832_object68680:head# 2024-03-30T13:50:58.204 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.204 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7574223e:::benchmark_data_smithi043_119832_object71603:head# 2024-03-30T13:50:58.204 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.204 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7574839e:::benchmark_data_smithi043_119832_object82856:head# 2024-03-30T13:50:58.204 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.205 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7574aeb1:::benchmark_data_smithi043_119832_object11108:head# 2024-03-30T13:50:58.205 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.205 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:757514e0:::benchmark_data_smithi043_119832_object28425:head# 2024-03-30T13:50:58.205 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.205 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7575d996:::benchmark_data_smithi043_119832_object10355:head# 2024-03-30T13:50:58.205 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.205 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:757613f7:::benchmark_data_smithi043_119832_object70583:head# 2024-03-30T13:50:58.205 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.206 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75765ea0:::benchmark_data_smithi043_119832_object73394:head# 2024-03-30T13:50:58.206 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.206 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:757799c9:::benchmark_data_smithi043_119832_object54820:head# 2024-03-30T13:50:58.206 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.207 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7577d9a1:::benchmark_data_smithi043_119832_object3552:head# 2024-03-30T13:50:58.207 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.207 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7577f4ee:::benchmark_data_smithi043_119832_object16523:head# 2024-03-30T13:50:58.207 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.207 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:757821b5:::benchmark_data_smithi043_119832_object14676:head# 2024-03-30T13:50:58.207 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.208 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:757842ab:::benchmark_data_smithi043_119832_object45988:head# 2024-03-30T13:50:58.208 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.208 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7578af54:::benchmark_data_smithi043_119832_object55736:head# 2024-03-30T13:50:58.208 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.208 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:757955a7:::benchmark_data_smithi043_119832_object54927:head# 2024-03-30T13:50:58.208 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.209 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75796105:::benchmark_data_smithi043_119832_object44110:head# 2024-03-30T13:50:58.209 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.209 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:757a5d03:::benchmark_data_smithi043_119832_object84377:head# 2024-03-30T13:50:58.209 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.209 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:757bd389:::benchmark_data_smithi043_119832_object158:head# 2024-03-30T13:50:58.209 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.210 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:757c4226:::benchmark_data_smithi043_119832_object73246:head# 2024-03-30T13:50:58.210 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.210 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:757c8f1e:::benchmark_data_smithi043_119832_object52760:head# 2024-03-30T13:50:58.210 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.211 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:757d51f6:::benchmark_data_smithi043_119832_object44319:head# 2024-03-30T13:50:58.211 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.211 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:757e0eae:::benchmark_data_smithi043_119832_object28073:head# 2024-03-30T13:50:58.211 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.212 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:757e48e0:::benchmark_data_smithi043_119832_object81186:head# 2024-03-30T13:50:58.212 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.212 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:757ea339:::benchmark_data_smithi043_119832_object13991:head# 2024-03-30T13:50:58.212 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.212 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75801faf:::benchmark_data_smithi043_119832_object6278:head# 2024-03-30T13:50:58.212 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.213 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7580de1f:::benchmark_data_smithi043_119832_object35858:head# 2024-03-30T13:50:58.213 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.213 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75820343:::benchmark_data_smithi043_119832_object12177:head# 2024-03-30T13:50:58.213 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.213 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75829d9e:::benchmark_data_smithi043_119832_object9077:head# 2024-03-30T13:50:58.213 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.213 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75834473:::benchmark_data_smithi043_119832_object35686:head# 2024-03-30T13:50:58.214 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.214 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:758485b9:::benchmark_data_smithi043_119832_object29182:head# 2024-03-30T13:50:58.214 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.214 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7585cde2:::benchmark_data_smithi043_119832_object9647:head# 2024-03-30T13:50:58.215 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.215 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7585e321:::benchmark_data_smithi043_119832_object86237:head# 2024-03-30T13:50:58.215 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.215 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7586688f:::benchmark_data_smithi043_119832_object35373:head# 2024-03-30T13:50:58.215 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.216 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75867244:::benchmark_data_smithi043_119832_object68168:head# 2024-03-30T13:50:58.216 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.216 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7586d909:::benchmark_data_smithi043_119832_object4854:head# 2024-03-30T13:50:58.216 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.216 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:758747ba:::benchmark_data_smithi043_119832_object58480:head# 2024-03-30T13:50:58.216 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.217 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7587da1e:::benchmark_data_smithi043_119832_object15790:head# 2024-03-30T13:50:58.217 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.217 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7589242f:::benchmark_data_smithi043_119832_object43459:head# 2024-03-30T13:50:58.217 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.217 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7589826e:::benchmark_data_smithi043_119832_object6615:head# 2024-03-30T13:50:58.217 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.218 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7589dcac:::benchmark_data_smithi043_119832_object85301:head# 2024-03-30T13:50:58.218 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.218 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:758a2b59:::benchmark_data_smithi043_119832_object42885:head# 2024-03-30T13:50:58.218 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.219 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:758a907a:::benchmark_data_smithi043_119832_object9730:head# 2024-03-30T13:50:58.219 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.219 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:758b39a3:::benchmark_data_smithi043_119832_object65224:head# 2024-03-30T13:50:58.219 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.219 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:758bc3dd:::benchmark_data_smithi043_119832_object42716:head# 2024-03-30T13:50:58.219 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.220 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:758d8d54:::benchmark_data_smithi043_119832_object45445:head# 2024-03-30T13:50:58.220 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.220 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:758f455b:::benchmark_data_smithi043_119832_object2549:head# 2024-03-30T13:50:58.220 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.220 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:758f7dde:::benchmark_data_smithi043_119832_object29738:head# 2024-03-30T13:50:58.220 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.221 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7590c8b6:::benchmark_data_smithi043_119832_object82074:head# 2024-03-30T13:50:58.221 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.221 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75927097:::benchmark_data_smithi043_119832_object57473:head# 2024-03-30T13:50:58.221 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.221 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75929476:::benchmark_data_smithi043_119832_object31881:head# 2024-03-30T13:50:58.221 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.222 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75932034:::benchmark_data_smithi043_119832_object71892:head# 2024-03-30T13:50:58.222 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.222 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75940b25:::benchmark_data_smithi043_119832_object1488:head# 2024-03-30T13:50:58.223 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.223 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:759428cc:::benchmark_data_smithi043_119832_object54963:head# 2024-03-30T13:50:58.223 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.223 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7594c04e:::benchmark_data_smithi043_119832_object61979:head# 2024-03-30T13:50:58.224 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.224 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:759556b6:::benchmark_data_smithi043_119832_object76341:head# 2024-03-30T13:50:58.224 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.224 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7595a000:::benchmark_data_smithi043_119832_object81372:head# 2024-03-30T13:50:58.224 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.225 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7595b6b4:::benchmark_data_smithi043_119832_object50192:head# 2024-03-30T13:50:58.225 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.225 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7595ea76:::benchmark_data_smithi043_119832_object20048:head# 2024-03-30T13:50:58.225 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.225 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75960f8d:::benchmark_data_smithi043_119832_object85278:head# 2024-03-30T13:50:58.225 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.225 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75961ccc:::benchmark_data_smithi043_119832_object208:head# 2024-03-30T13:50:58.225 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.227 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7597606a:::benchmark_data_smithi043_119832_object9587:head# 2024-03-30T13:50:58.227 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.227 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7597af4c:::benchmark_data_smithi043_119832_object10067:head# 2024-03-30T13:50:58.227 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.227 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75989358:::benchmark_data_smithi043_119832_object23824:head# 2024-03-30T13:50:58.227 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.227 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7598bb17:::benchmark_data_smithi043_119832_object28363:head# 2024-03-30T13:50:58.227 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.227 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7598c0c5:::benchmark_data_smithi043_119832_object40777:head# 2024-03-30T13:50:58.227 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.228 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7598d61a:::benchmark_data_smithi043_119832_object77076:head# 2024-03-30T13:50:58.228 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.229 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:759bddd6:::benchmark_data_smithi043_119832_object20655:head# 2024-03-30T13:50:58.229 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.229 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:759bdf68:::benchmark_data_smithi043_119832_object28528:head# 2024-03-30T13:50:58.229 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.229 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:759ca551:::benchmark_data_smithi043_119832_object33936:head# 2024-03-30T13:50:58.229 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.229 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:759ca9f8:::benchmark_data_smithi043_119832_object32213:head# 2024-03-30T13:50:58.229 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.230 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:759d23f5:::benchmark_data_smithi043_119832_object53479:head# 2024-03-30T13:50:58.230 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.230 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:759d3954:::benchmark_data_smithi043_119832_object3505:head# 2024-03-30T13:50:58.231 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.231 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:759f45db:::benchmark_data_smithi043_119832_object15658:head# 2024-03-30T13:50:58.231 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.231 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75a26e1e:::benchmark_data_smithi043_119832_object36283:head# 2024-03-30T13:50:58.231 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.231 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75a42721:::benchmark_data_smithi043_119832_object83033:head# 2024-03-30T13:50:58.231 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.232 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75a4e937:::benchmark_data_smithi043_119832_object10452:head# 2024-03-30T13:50:58.232 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.233 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75a5cb65:::benchmark_data_smithi043_119832_object66807:head# 2024-03-30T13:50:58.233 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.233 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75a5cbf9:::benchmark_data_smithi043_119832_object52361:head# 2024-03-30T13:50:58.233 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.233 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75a6a1e9:::benchmark_data_smithi043_119832_object9600:head# 2024-03-30T13:50:58.233 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.233 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75a769b2:::benchmark_data_smithi043_119832_object10648:head# 2024-03-30T13:50:58.233 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.233 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75a81d9d:::benchmark_data_smithi043_119832_object14119:head# 2024-03-30T13:50:58.233 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.234 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75a83aa6:::benchmark_data_smithi043_119832_object72588:head# 2024-03-30T13:50:58.235 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.235 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75a90cf2:::benchmark_data_smithi043_119832_object24305:head# 2024-03-30T13:50:58.235 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.235 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75a9adc1:::benchmark_data_smithi043_119832_object73750:head# 2024-03-30T13:50:58.235 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.235 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75ab7d7d:::benchmark_data_smithi043_119832_object45454:head# 2024-03-30T13:50:58.235 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.235 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75abc499:::benchmark_data_smithi043_119832_object66358:head# 2024-03-30T13:50:58.235 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.235 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75abd3c2:::benchmark_data_smithi043_119832_object57252:head# 2024-03-30T13:50:58.235 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.237 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75ac0f88:::benchmark_data_smithi043_119832_object38000:head# 2024-03-30T13:50:58.237 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.237 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75ad8d4b:::benchmark_data_smithi043_119832_object27530:head# 2024-03-30T13:50:58.237 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.237 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75b0021d:::benchmark_data_smithi043_119832_object31914:head# 2024-03-30T13:50:58.237 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.237 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75b03217:::benchmark_data_smithi043_119832_object27086:head# 2024-03-30T13:50:58.237 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.237 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75b03c78:::benchmark_data_smithi043_119832_object56602:head# 2024-03-30T13:50:58.237 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.237 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75b04446:::benchmark_data_smithi043_119832_object70006:head# 2024-03-30T13:50:58.237 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.239 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75b0cb01:::benchmark_data_smithi043_119832_object44538:head# 2024-03-30T13:50:58.239 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.239 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75b0dfea:::benchmark_data_smithi043_119832_object50577:head# 2024-03-30T13:50:58.239 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.239 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75b0f546:::benchmark_data_smithi043_119832_object41136:head# 2024-03-30T13:50:58.239 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.239 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75b1396e:::benchmark_data_smithi043_119832_object56202:head# 2024-03-30T13:50:58.239 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.239 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75b3f4d6:::benchmark_data_smithi043_119832_object31540:head# 2024-03-30T13:50:58.239 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.239 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75b3f934:::benchmark_data_smithi043_119832_object23700:head# 2024-03-30T13:50:58.240 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.240 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75b47c37:::benchmark_data_smithi043_119832_object52624:head# 2024-03-30T13:50:58.240 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.240 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75b76082:::benchmark_data_smithi043_119832_object38583:head# 2024-03-30T13:50:58.240 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.241 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75b84ad6:::benchmark_data_smithi043_119832_object35865:head# 2024-03-30T13:50:58.241 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.241 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75b94524:::benchmark_data_smithi043_119832_object58759:head# 2024-03-30T13:50:58.241 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.241 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75bd72f7:::benchmark_data_smithi043_119832_object57154:head# 2024-03-30T13:50:58.241 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.242 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75bdff34:::benchmark_data_smithi043_119832_object42527:head# 2024-03-30T13:50:58.242 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.242 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75be08dd:::benchmark_data_smithi043_119832_object52689:head# 2024-03-30T13:50:58.242 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.242 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75be2653:::benchmark_data_smithi043_119832_object66322:head# 2024-03-30T13:50:58.243 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.243 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75c02a63:::benchmark_data_smithi043_119832_object6052:head# 2024-03-30T13:50:58.243 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.247 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75c19149:::benchmark_data_smithi043_119832_object40439:head# 2024-03-30T13:50:58.248 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.248 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75c28d4d:::benchmark_data_smithi043_119832_object18943:head# 2024-03-30T13:50:58.248 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.248 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75c31f87:::benchmark_data_smithi043_119832_object10893:head# 2024-03-30T13:50:58.248 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.248 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75c33068:::benchmark_data_smithi043_119832_object46543:head# 2024-03-30T13:50:58.248 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.248 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75c50bb3:::benchmark_data_smithi043_119832_object86589:head# 2024-03-30T13:50:58.248 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.248 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75c50dec:::benchmark_data_smithi043_119832_object37379:head# 2024-03-30T13:50:58.248 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.248 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75c5666e:::benchmark_data_smithi043_119832_object68638:head# 2024-03-30T13:50:58.248 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.248 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75c5d8bf:::benchmark_data_smithi043_119832_object35474:head# 2024-03-30T13:50:58.249 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.249 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75c71503:::benchmark_data_smithi043_119832_object17241:head# 2024-03-30T13:50:58.249 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.249 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75c8024d:::benchmark_data_smithi043_119832_object76546:head# 2024-03-30T13:50:58.249 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.249 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75c868fb:::benchmark_data_smithi043_119832_object68411:head# 2024-03-30T13:50:58.249 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.249 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75c89c78:::benchmark_data_smithi043_119832_object49264:head# 2024-03-30T13:50:58.249 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.249 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75c8aa0e:::benchmark_data_smithi043_119832_object29877:head# 2024-03-30T13:50:58.249 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.249 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75c9fe47:::benchmark_data_smithi043_119832_object87608:head# 2024-03-30T13:50:58.249 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.250 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75cb340e:::benchmark_data_smithi043_119832_object17975:head# 2024-03-30T13:50:58.250 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.250 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75cbebdd:::benchmark_data_smithi043_119832_object3774:head# 2024-03-30T13:50:58.250 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.250 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75cd9e91:::benchmark_data_smithi043_119832_object35407:head# 2024-03-30T13:50:58.250 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.252 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75d08b0e:::benchmark_data_smithi043_119832_object51081:head# 2024-03-30T13:50:58.252 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.252 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75d0aa30:::benchmark_data_smithi043_119832_object51148:head# 2024-03-30T13:50:58.252 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.252 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75d1a2cd:::benchmark_data_smithi043_119832_object6347:head# 2024-03-30T13:50:58.252 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.252 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75d21dc9:::benchmark_data_smithi043_119832_object62554:head# 2024-03-30T13:50:58.252 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.252 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75d2cff9:::benchmark_data_smithi043_119832_object68659:head# 2024-03-30T13:50:58.252 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.252 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75d31824:::benchmark_data_smithi043_119832_object39294:head# 2024-03-30T13:50:58.252 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.253 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75d410ad:::benchmark_data_smithi043_119832_object85665:head# 2024-03-30T13:50:58.253 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.253 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75d69a34:::benchmark_data_smithi043_119832_object15130:head# 2024-03-30T13:50:58.253 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.253 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75d76580:::benchmark_data_smithi043_119832_object51439:head# 2024-03-30T13:50:58.253 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.253 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75d7f0b3:::benchmark_data_smithi043_119832_object32388:head# 2024-03-30T13:50:58.253 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.253 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75d7f305:::benchmark_data_smithi043_119832_object17349:head# 2024-03-30T13:50:58.253 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.253 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75d95d72:::benchmark_data_smithi043_119832_object44545:head# 2024-03-30T13:50:58.253 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.253 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75d99e5c:::benchmark_data_smithi043_119832_object82841:head# 2024-03-30T13:50:58.253 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.254 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75db38e6:::benchmark_data_smithi043_119832_object18766:head# 2024-03-30T13:50:58.254 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.254 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75dcb176:::benchmark_data_smithi043_119832_object72050:head# 2024-03-30T13:50:58.254 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.256 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75dcc7d2:::benchmark_data_smithi043_119832_object22650:head# 2024-03-30T13:50:58.256 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.256 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75df1d11:::benchmark_data_smithi043_119832_object81432:head# 2024-03-30T13:50:58.256 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.256 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75dffbe7:::benchmark_data_smithi043_119832_object44513:head# 2024-03-30T13:50:58.256 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.256 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75e001c9:::benchmark_data_smithi043_119832_object31135:head# 2024-03-30T13:50:58.256 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.256 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75e1a293:::benchmark_data_smithi043_119832_object53491:head# 2024-03-30T13:50:58.256 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.256 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75e1b55a:::benchmark_data_smithi043_119832_object75313:head# 2024-03-30T13:50:58.257 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.257 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75e29355:::benchmark_data_smithi043_119832_object13101:head# 2024-03-30T13:50:58.257 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.257 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75e33a10:::benchmark_data_smithi043_119832_object85561:head# 2024-03-30T13:50:58.257 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.257 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75e43ae5:::benchmark_data_smithi043_119832_object88523:head# 2024-03-30T13:50:58.257 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.257 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75e5e4ce:::benchmark_data_smithi043_119832_object58679:head# 2024-03-30T13:50:58.257 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.257 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75e6944c:::benchmark_data_smithi043_119832_object66616:head# 2024-03-30T13:50:58.257 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.257 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75e9bb0e:::benchmark_data_smithi043_119832_object67061:head# 2024-03-30T13:50:58.257 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.257 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75ea360c:::benchmark_data_smithi043_119832_object82613:head# 2024-03-30T13:50:58.258 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.258 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75ea82a3:::benchmark_data_smithi043_119832_object69693:head# 2024-03-30T13:50:58.258 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.261 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75eb8654:::benchmark_data_smithi043_119832_object69247:head# 2024-03-30T13:50:58.261 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.261 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75ebe66e:::benchmark_data_smithi043_119832_object66780:head# 2024-03-30T13:50:58.261 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.261 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75ec390f:::benchmark_data_smithi043_119832_object66260:head# 2024-03-30T13:50:58.261 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.261 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75ec3fdd:::benchmark_data_smithi043_119832_object39573:head# 2024-03-30T13:50:58.262 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.262 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75ecb146:::benchmark_data_smithi043_119832_object84283:head# 2024-03-30T13:50:58.262 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.262 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75ecbbf6:::benchmark_data_smithi043_119832_object39167:head# 2024-03-30T13:50:58.262 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.262 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75eda286:::benchmark_data_smithi043_119832_object27000:head# 2024-03-30T13:50:58.262 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.262 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75edb8a2:::benchmark_data_smithi043_119832_object9419:head# 2024-03-30T13:50:58.262 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.262 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75ee94b0:::benchmark_data_smithi043_119832_object46552:head# 2024-03-30T13:50:58.262 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.262 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75eefbc5:::benchmark_data_smithi043_119832_object66674:head# 2024-03-30T13:50:58.262 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.262 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75efaf27:::benchmark_data_smithi043_119832_object57843:head# 2024-03-30T13:50:58.263 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.263 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75f0a837:::benchmark_data_smithi043_119832_object34188:head# 2024-03-30T13:50:58.263 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.263 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75f0c60f:::benchmark_data_smithi043_119832_object2158:head# 2024-03-30T13:50:58.263 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.263 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75f17bf6:::benchmark_data_smithi043_119832_object51838:head# 2024-03-30T13:50:58.263 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.263 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75f45946:::benchmark_data_smithi043_119832_object82962:head# 2024-03-30T13:50:58.263 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.263 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75f6ebda:::benchmark_data_smithi043_119832_object82198:head# 2024-03-30T13:50:58.263 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.263 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75f779e4:::benchmark_data_smithi043_119832_object65330:head# 2024-03-30T13:50:58.263 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.264 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:50:58.269 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75f7afee:::benchmark_data_smithi043_119832_object71414:head# 2024-03-30T13:50:58.269 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.269 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75fa3e18:::benchmark_data_smithi043_119832_object65488:head# 2024-03-30T13:50:58.269 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.269 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75fa9b35:::benchmark_data_smithi043_119832_object34832:head# 2024-03-30T13:50:58.269 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.269 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75fb708b:::benchmark_data_smithi043_119832_object32920:head# 2024-03-30T13:50:58.269 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.269 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75fbdaaa:::benchmark_data_smithi043_119832_object75624:head# 2024-03-30T13:50:58.269 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.269 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75fc0ca8:::benchmark_data_smithi043_119832_object79188:head# 2024-03-30T13:50:58.269 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.269 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75fc7291:::benchmark_data_smithi043_119832_object34801:head# 2024-03-30T13:50:58.270 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.270 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75fd2b79:::benchmark_data_smithi043_119832_object88409:head# 2024-03-30T13:50:58.270 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.270 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75fdb73a:::benchmark_data_smithi043_119832_object49425:head# 2024-03-30T13:50:58.270 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.270 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75fdcee2:::benchmark_data_smithi043_119832_object56039:head# 2024-03-30T13:50:58.270 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.270 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75fe7cff:::benchmark_data_smithi043_119832_object44314:head# 2024-03-30T13:50:58.270 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.270 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75fe970e:::benchmark_data_smithi043_119832_object40974:head# 2024-03-30T13:50:58.270 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.270 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75fefc1d:::benchmark_data_smithi043_119832_object34353:head# 2024-03-30T13:50:58.270 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.271 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75ff8147:::benchmark_data_smithi043_119832_object65381:head# 2024-03-30T13:50:58.271 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.271 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:75ffa5f6:::benchmark_data_smithi043_119832_object17685:head# 2024-03-30T13:50:58.271 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.271 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7600bded:::benchmark_data_smithi043_119832_object30051:head# 2024-03-30T13:50:58.271 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.271 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:760139b4:::benchmark_data_smithi043_119832_object48737:head# 2024-03-30T13:50:58.271 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.271 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76019836:::benchmark_data_smithi043_119832_object77030:head# 2024-03-30T13:50:58.271 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.271 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7602f3b0:::benchmark_data_smithi043_119832_object46794:head# 2024-03-30T13:50:58.271 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.271 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76047105:::benchmark_data_smithi043_119832_object54128:head# 2024-03-30T13:50:58.271 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.272 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76056729:::benchmark_data_smithi043_119832_object287:head# 2024-03-30T13:50:58.272 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.272 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:760582cb:::benchmark_data_smithi043_119832_object59468:head# 2024-03-30T13:50:58.272 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.272 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76062ae4:::benchmark_data_smithi043_119832_object9941:head# 2024-03-30T13:50:58.272 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.272 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76066544:::benchmark_data_smithi043_119832_object81539:head# 2024-03-30T13:50:58.272 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.276 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7606fd6e:::benchmark_data_smithi043_119832_object12079:head# 2024-03-30T13:50:58.276 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.276 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76072932:::benchmark_data_smithi043_119832_object80589:head# 2024-03-30T13:50:58.276 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.276 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76080247:::benchmark_data_smithi043_119832_object30058:head# 2024-03-30T13:50:58.277 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.277 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:760906ec:::benchmark_data_smithi043_119832_object28534:head# 2024-03-30T13:50:58.277 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.277 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:760c3911:::benchmark_data_smithi043_119832_object73959:head# 2024-03-30T13:50:58.277 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.277 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:760d093b:::benchmark_data_smithi043_119832_object33071:head# 2024-03-30T13:50:58.277 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.277 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:760d1a78:::benchmark_data_smithi043_119832_object5943:head# 2024-03-30T13:50:58.277 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.277 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76101f69:::benchmark_data_smithi043_119832_object539:head# 2024-03-30T13:50:58.277 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.277 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76146576:::benchmark_data_smithi043_119832_object59635:head# 2024-03-30T13:50:58.277 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.278 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76147d3a:::benchmark_data_smithi043_119832_object28581:head# 2024-03-30T13:50:58.278 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.278 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76153244:::benchmark_data_smithi043_119832_object62246:head# 2024-03-30T13:50:58.278 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.278 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7615e38f:::benchmark_data_smithi043_119832_object52868:head# 2024-03-30T13:50:58.278 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.278 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:761619d4:::benchmark_data_smithi043_119832_object12859:head# 2024-03-30T13:50:58.278 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.278 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7616e48c:::benchmark_data_smithi043_119832_object36557:head# 2024-03-30T13:50:58.278 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.278 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:761792cb:::benchmark_data_smithi043_119832_object6771:head# 2024-03-30T13:50:58.278 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.278 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7617bf6b:::benchmark_data_smithi043_119832_object17986:head# 2024-03-30T13:50:58.278 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.279 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7618b947:::benchmark_data_smithi043_119832_object47281:head# 2024-03-30T13:50:58.279 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.279 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76197d91:::benchmark_data_smithi043_119832_object78971:head# 2024-03-30T13:50:58.279 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.279 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76198398:::benchmark_data_smithi043_119832_object18771:head# 2024-03-30T13:50:58.279 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.279 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:761a4200:::benchmark_data_smithi043_119832_object25143:head# 2024-03-30T13:50:58.279 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.282 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:761a5f60:::benchmark_data_smithi043_119832_object31238:head# 2024-03-30T13:50:58.282 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.282 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:761a8f25:::benchmark_data_smithi043_119832_object75608:head# 2024-03-30T13:50:58.283 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.283 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:761afff3:::benchmark_data_smithi043_119832_object63750:head# 2024-03-30T13:50:58.283 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.283 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:761d3bd2:::benchmark_data_smithi043_119832_object87791:head# 2024-03-30T13:50:58.283 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.283 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:761d95e4:::benchmark_data_smithi043_119832_object5450:head# 2024-03-30T13:50:58.283 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.283 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:761dba87:::benchmark_data_smithi043_119832_object82264:head# 2024-03-30T13:50:58.283 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.283 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:761f90d6:::benchmark_data_smithi043_119832_object22587:head# 2024-03-30T13:50:58.283 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.283 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:761fb115:::benchmark_data_smithi043_119832_object24044:head# 2024-03-30T13:50:58.283 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.284 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:762082d3:::benchmark_data_smithi043_119832_object13179:head# 2024-03-30T13:50:58.284 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.284 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76216bd7:::benchmark_data_smithi043_119832_object12229:head# 2024-03-30T13:50:58.284 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.284 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7621a525:::benchmark_data_smithi043_119832_object36460:head# 2024-03-30T13:50:58.284 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.284 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7621d5cc:::benchmark_data_smithi043_119832_object16752:head# 2024-03-30T13:50:58.284 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.284 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:762204b9:::benchmark_data_smithi043_119832_object52633:head# 2024-03-30T13:50:58.285 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.285 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76229c89:::benchmark_data_smithi043_119832_object16097:head# 2024-03-30T13:50:58.285 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.285 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76232eb8:::benchmark_data_smithi043_119832_object82082:head# 2024-03-30T13:50:58.285 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.286 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7623ec71:::benchmark_data_smithi043_119832_object71057:head# 2024-03-30T13:50:58.286 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.286 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76245a57:::benchmark_data_smithi043_119832_object72781:head# 2024-03-30T13:50:58.286 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.287 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7624babb:::benchmark_data_smithi043_119832_object80763:head# 2024-03-30T13:50:58.287 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.287 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76254cc3:::benchmark_data_smithi043_119832_object39615:head# 2024-03-30T13:50:58.287 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.287 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7625efb0:::benchmark_data_smithi043_119832_object19856:head# 2024-03-30T13:50:58.287 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.288 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76268ae0:::benchmark_data_smithi043_119832_object56215:head# 2024-03-30T13:50:58.288 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.288 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:762699da:::benchmark_data_smithi043_119832_object53381:head# 2024-03-30T13:50:58.288 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.288 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:762abfc6:::benchmark_data_smithi043_119832_object4121:head# 2024-03-30T13:50:58.288 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.289 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:762d7d48:::benchmark_data_smithi043_119832_object43914:head# 2024-03-30T13:50:58.289 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.289 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:762d9893:::benchmark_data_smithi043_119832_object16794:head# 2024-03-30T13:50:58.289 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.289 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:762fea3d:::benchmark_data_smithi043_119832_object26978:head# 2024-03-30T13:50:58.289 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.289 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7630b7d5:::benchmark_data_smithi043_119832_object17803:head# 2024-03-30T13:50:58.290 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.290 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7630f6ce:::benchmark_data_smithi043_119832_object16765:head# 2024-03-30T13:50:58.290 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.290 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76322364:::benchmark_data_smithi043_119832_object40864:head# 2024-03-30T13:50:58.290 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.291 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76323f94:::benchmark_data_smithi043_119832_object15402:head# 2024-03-30T13:50:58.291 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.291 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7632776e:::benchmark_data_smithi043_119832_object41890:head# 2024-03-30T13:50:58.291 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.292 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76334aab:::benchmark_data_smithi043_119832_object56368:head# 2024-03-30T13:50:58.292 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.292 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76352e02:::benchmark_data_smithi043_119832_object86959:head# 2024-03-30T13:50:58.292 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.292 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76355681:::benchmark_data_smithi043_119832_object61812:head# 2024-03-30T13:50:58.292 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.292 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7636476a:::benchmark_data_smithi043_119832_object23803:head# 2024-03-30T13:50:58.292 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.292 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7636aef4:::benchmark_data_smithi043_119832_object40555:head# 2024-03-30T13:50:58.292 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.292 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7636fd2a:::benchmark_data_smithi043_119832_object55436:head# 2024-03-30T13:50:58.293 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.293 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7637e9e3:::benchmark_data_smithi043_119832_object2917:head# 2024-03-30T13:50:58.293 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.293 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7638868b:::benchmark_data_smithi043_119832_object68602:head# 2024-03-30T13:50:58.293 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.293 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76398ec7:::benchmark_data_smithi043_119832_object43503:head# 2024-03-30T13:50:58.294 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.294 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:763a7cd5:::benchmark_data_smithi043_119832_object78665:head# 2024-03-30T13:50:58.294 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.294 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:763b419b:::benchmark_data_smithi043_119832_object33934:head# 2024-03-30T13:50:58.294 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.295 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:763bc8a8:::benchmark_data_smithi043_119832_object31190:head# 2024-03-30T13:50:58.295 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.295 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:763d6eb4:::benchmark_data_smithi043_119832_object79053:head# 2024-03-30T13:50:58.295 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.296 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:763dd59a:::benchmark_data_smithi043_119832_object69489:head# 2024-03-30T13:50:58.296 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.296 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:763f14c1:::benchmark_data_smithi043_119832_object84910:head# 2024-03-30T13:50:58.296 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.296 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:763f79d0:::benchmark_data_smithi043_119832_object37624:head# 2024-03-30T13:50:58.296 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.297 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:763fc9c1:::benchmark_data_smithi043_119832_object40076:head# 2024-03-30T13:50:58.297 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.297 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:764014cc:::benchmark_data_smithi043_119832_object39115:head# 2024-03-30T13:50:58.297 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.297 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7640502d:::benchmark_data_smithi043_119832_object59413:head# 2024-03-30T13:50:58.297 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.298 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7640fea2:::benchmark_data_smithi043_119832_object59240:head# 2024-03-30T13:50:58.298 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.298 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:764127dd:::benchmark_data_smithi043_119832_object85310:head# 2024-03-30T13:50:58.298 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.299 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76420f15:::benchmark_data_smithi043_119832_object33097:head# 2024-03-30T13:50:58.299 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.299 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76422354:::benchmark_data_smithi043_119832_object30070:head# 2024-03-30T13:50:58.299 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.300 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76428ec6:::benchmark_data_smithi043_119832_object74356:head# 2024-03-30T13:50:58.300 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.300 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7644649d:::benchmark_data_smithi043_119832_object8793:head# 2024-03-30T13:50:58.300 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.300 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7645c5d6:::benchmark_data_smithi043_119832_object30646:head# 2024-03-30T13:50:58.300 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.300 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7647c245:::benchmark_data_smithi043_119832_object63576:head# 2024-03-30T13:50:58.300 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.300 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7649c8d6:::benchmark_data_smithi043_119832_object30344:head# 2024-03-30T13:50:58.301 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.302 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7649fc7f:::benchmark_data_smithi043_119832_object63199:head# 2024-03-30T13:50:58.302 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.302 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:764a1686:::benchmark_data_smithi043_119832_object87093:head# 2024-03-30T13:50:58.302 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.302 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:764a5fec:::benchmark_data_smithi043_119832_object33581:head# 2024-03-30T13:50:58.302 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.302 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:764a9c3a:::benchmark_data_smithi043_119832_object50036:head# 2024-03-30T13:50:58.302 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.302 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:764ba238:::benchmark_data_smithi043_119832_object4612:head# 2024-03-30T13:50:58.302 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.302 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:764bc414:::benchmark_data_smithi043_119832_object39055:head# 2024-03-30T13:50:58.303 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.303 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:764bca1b:::benchmark_data_smithi043_119832_object78622:head# 2024-03-30T13:50:58.303 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.304 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:764bdef3:::benchmark_data_smithi043_119832_object20317:head# 2024-03-30T13:50:58.304 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.304 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:764c3efd:::benchmark_data_smithi043_119832_object32197:head# 2024-03-30T13:50:58.304 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.304 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:764c60c9:::benchmark_data_smithi043_119832_object14195:head# 2024-03-30T13:50:58.304 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:58.306 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:58.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:50:58.309 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:764e5ab6:::benchmark_data_smithi043_119832_object20148:head# 2024-03-30T13:50:58.309 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.309 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:764f24bc:::benchmark_data_smithi043_119832_object79942:head# 2024-03-30T13:50:58.309 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.309 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:765025f2:::benchmark_data_smithi043_119832_object22989:head# 2024-03-30T13:50:58.309 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.309 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76513618:::benchmark_data_smithi043_119832_object81941:head# 2024-03-30T13:50:58.309 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.309 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:765159a9:::benchmark_data_smithi043_119832_object60431:head# 2024-03-30T13:50:58.309 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.309 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7651cf43:::benchmark_data_smithi043_119832_object23105:head# 2024-03-30T13:50:58.309 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.309 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7653e190:::benchmark_data_smithi043_119832_object6446:head# 2024-03-30T13:50:58.310 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.310 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7653e3a6:::benchmark_data_smithi043_119832_object76541:head# 2024-03-30T13:50:58.310 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.310 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7654d296:::benchmark_data_smithi043_119832_object12853:head# 2024-03-30T13:50:58.310 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.310 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76565583:::benchmark_data_smithi043_119832_object64604:head# 2024-03-30T13:50:58.310 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.310 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:765692a5:::benchmark_data_smithi043_119832_object39543:head# 2024-03-30T13:50:58.310 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.310 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76575590:::benchmark_data_smithi043_119832_object45939:head# 2024-03-30T13:50:58.310 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.310 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7657ad2e:::benchmark_data_smithi043_119832_object83943:head# 2024-03-30T13:50:58.310 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.311 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7658a5c9:::benchmark_data_smithi043_119832_object325:head# 2024-03-30T13:50:58.311 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.311 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76591b24:::benchmark_data_smithi043_119832_object60649:head# 2024-03-30T13:50:58.311 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.312 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7659ce77:::benchmark_data_smithi043_119832_object35688:head# 2024-03-30T13:50:58.312 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.312 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:765b1bed:::benchmark_data_smithi043_119832_object25844:head# 2024-03-30T13:50:58.313 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.313 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:765ba667:::benchmark_data_smithi043_119832_object30350:head# 2024-03-30T13:50:58.313 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.313 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:765c9a15:::benchmark_data_smithi043_119832_object79933:head# 2024-03-30T13:50:58.313 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.313 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:765cef09:::benchmark_data_smithi043_119832_object70812:head# 2024-03-30T13:50:58.313 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.313 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:765d4613:::benchmark_data_smithi043_119832_object59400:head# 2024-03-30T13:50:58.313 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.313 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:765e14c5:::benchmark_data_smithi043_119832_object60557:head# 2024-03-30T13:50:58.313 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.313 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:765e360e:::benchmark_data_smithi043_119832_object22585:head# 2024-03-30T13:50:58.313 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.313 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:765ee4f9:::benchmark_data_smithi043_119832_object3969:head# 2024-03-30T13:50:58.314 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.314 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7660224e:::benchmark_data_smithi043_119832_object53634:head# 2024-03-30T13:50:58.314 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.314 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:766110f8:::benchmark_data_smithi043_119832_object8852:head# 2024-03-30T13:50:58.314 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.315 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76615004:::benchmark_data_smithi043_119832_object65499:head# 2024-03-30T13:50:58.315 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.315 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:766245bb:::benchmark_data_smithi043_119832_object26280:head# 2024-03-30T13:50:58.315 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.315 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76631350:::benchmark_data_smithi043_119832_object76590:head# 2024-03-30T13:50:58.316 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.316 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:766327e2:::benchmark_data_smithi043_119832_object71128:head# 2024-03-30T13:50:58.316 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.316 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7663b6d1:::benchmark_data_smithi043_119832_object83846:head# 2024-03-30T13:50:58.316 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.316 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76641ff8:::benchmark_data_smithi043_119832_object30264:head# 2024-03-30T13:50:58.316 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.316 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76650744:::benchmark_data_smithi043_119832_object62488:head# 2024-03-30T13:50:58.316 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.316 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76657468:::benchmark_data_smithi043_119832_object310:head# 2024-03-30T13:50:58.316 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.316 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7665cd20:::benchmark_data_smithi043_119832_object74854:head# 2024-03-30T13:50:58.316 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.317 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7666139e:::benchmark_data_smithi043_119832_object73828:head# 2024-03-30T13:50:58.317 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.317 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76664592:::benchmark_data_smithi043_119832_object4265:head# 2024-03-30T13:50:58.317 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.318 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76664b6d:::benchmark_data_smithi043_119832_object86082:head# 2024-03-30T13:50:58.318 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.318 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7669e968:::benchmark_data_smithi043_119832_object75662:head# 2024-03-30T13:50:58.318 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.318 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:766ca19e:::benchmark_data_smithi043_119832_object15125:head# 2024-03-30T13:50:58.318 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.319 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:766d6aac:::benchmark_data_smithi043_119832_object3622:head# 2024-03-30T13:50:58.319 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.320 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:766ddeea:::benchmark_data_smithi043_119832_object53859:head# 2024-03-30T13:50:58.320 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.320 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:766f17a1:::benchmark_data_smithi043_119832_object79481:head# 2024-03-30T13:50:58.320 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.320 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:766f1887:::benchmark_data_smithi043_119832_object30792:head# 2024-03-30T13:50:58.320 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.321 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7671b5a8:::benchmark_data_smithi043_119832_object65638:head# 2024-03-30T13:50:58.321 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.321 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:767238de:::benchmark_data_smithi043_119832_object59081:head# 2024-03-30T13:50:58.321 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.321 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76724b75:::benchmark_data_smithi043_119832_object21007:head# 2024-03-30T13:50:58.322 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.322 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:767251d2:::benchmark_data_smithi043_119832_object63506:head# 2024-03-30T13:50:58.322 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.322 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7672eb93:::benchmark_data_smithi043_119832_object68670:head# 2024-03-30T13:50:58.322 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.322 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7672f382:::benchmark_data_smithi043_119832_object83982:head# 2024-03-30T13:50:58.322 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.323 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76733cf7:::benchmark_data_smithi043_119832_object25845:head# 2024-03-30T13:50:58.324 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.324 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7673df8d:::benchmark_data_smithi043_119832_object33422:head# 2024-03-30T13:50:58.324 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.324 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7673f468:::benchmark_data_smithi043_119832_object76964:head# 2024-03-30T13:50:58.324 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.324 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76753aeb:::benchmark_data_smithi043_119832_object65739:head# 2024-03-30T13:50:58.324 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.324 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7675723d:::benchmark_data_smithi043_119832_object22265:head# 2024-03-30T13:50:58.324 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.324 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7676163f:::benchmark_data_smithi043_119832_object46973:head# 2024-03-30T13:50:58.324 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.326 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:767738f6:::benchmark_data_smithi043_119832_object85608:head# 2024-03-30T13:50:58.326 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.326 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:767a5949:::benchmark_data_smithi043_119832_object42325:head# 2024-03-30T13:50:58.326 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.326 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:767af637:::benchmark_data_smithi043_119832_object47169:head# 2024-03-30T13:50:58.326 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.326 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:767b65df:::benchmark_data_smithi043_119832_object48513:head# 2024-03-30T13:50:58.326 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.326 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:767bccfa:::benchmark_data_smithi043_119832_object62660:head# 2024-03-30T13:50:58.326 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.326 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:767bf803:::benchmark_data_smithi043_119832_object80636:head# 2024-03-30T13:50:58.326 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.328 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:767c7da9:::benchmark_data_smithi043_119832_object46915:head# 2024-03-30T13:50:58.328 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.328 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:767cb73c:::benchmark_data_smithi043_119832_object40942:head# 2024-03-30T13:50:58.328 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.328 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:767e3c57:::benchmark_data_smithi043_119832_object9663:head# 2024-03-30T13:50:58.328 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.328 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:767e5003:::benchmark_data_smithi043_119832_object78968:head# 2024-03-30T13:50:58.328 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.328 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:767e8d5d:::benchmark_data_smithi043_119832_object16429:head# 2024-03-30T13:50:58.328 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.328 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:767ea3dd:::benchmark_data_smithi043_119832_object80477:head# 2024-03-30T13:50:58.328 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.329 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:767ee0dd:::benchmark_data_smithi043_119832_object83066:head# 2024-03-30T13:50:58.329 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.329 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:767f1514:::benchmark_data_smithi043_119832_object61043:head# 2024-03-30T13:50:58.329 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.329 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:767fd2f5:::benchmark_data_smithi043_119832_object47727:head# 2024-03-30T13:50:58.330 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.330 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:767ff754:::benchmark_data_smithi043_119832_object72157:head# 2024-03-30T13:50:58.330 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.330 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7680efb3:::benchmark_data_smithi043_119832_object54930:head# 2024-03-30T13:50:58.330 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.331 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7681158d:::benchmark_data_smithi043_119832_object12683:head# 2024-03-30T13:50:58.331 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.331 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7681febc:::benchmark_data_smithi043_119832_object10771:head# 2024-03-30T13:50:58.331 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.331 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76821f8f:::benchmark_data_smithi043_119832_object55734:head# 2024-03-30T13:50:58.331 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.331 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76822cbe:::benchmark_data_smithi043_119832_object59902:head# 2024-03-30T13:50:58.331 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.331 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7684dba6:::benchmark_data_smithi043_119832_object3519:head# 2024-03-30T13:50:58.331 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.332 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7685037e:::benchmark_data_smithi043_119832_object80815:head# 2024-03-30T13:50:58.332 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.332 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7686d120:::benchmark_data_smithi043_119832_object41722:head# 2024-03-30T13:50:58.332 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.332 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7686ea6b:::benchmark_data_smithi043_119832_object54729:head# 2024-03-30T13:50:58.332 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.332 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76874eec:::benchmark_data_smithi043_119832_object35290:head# 2024-03-30T13:50:58.333 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.333 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76879bd5:::benchmark_data_smithi043_119832_object40432:head# 2024-03-30T13:50:58.333 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.333 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76884694:::benchmark_data_smithi043_119832_object38286:head# 2024-03-30T13:50:58.334 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.334 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:768a4c04:::benchmark_data_smithi043_119832_object87375:head# 2024-03-30T13:50:58.334 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.334 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:768a7e55:::benchmark_data_smithi043_119832_object63574:head# 2024-03-30T13:50:58.334 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.334 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:768b78eb:::benchmark_data_smithi043_119832_object35540:head# 2024-03-30T13:50:58.334 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.334 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:768c48d7:::benchmark_data_smithi043_119832_object69834:head# 2024-03-30T13:50:58.334 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.335 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:768c5856:::benchmark_data_smithi043_119832_object130:head# 2024-03-30T13:50:58.335 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.335 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:768df144:::benchmark_data_smithi043_119832_object81676:head# 2024-03-30T13:50:58.335 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.335 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:768fcea4:::benchmark_data_smithi043_119832_object74634:head# 2024-03-30T13:50:58.335 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.335 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:768ffad4:::benchmark_data_smithi043_119832_object38889:head# 2024-03-30T13:50:58.336 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.336 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7690b413:::benchmark_data_smithi043_119832_object34260:head# 2024-03-30T13:50:58.336 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.336 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7690def8:::benchmark_data_smithi043_119832_object8842:head# 2024-03-30T13:50:58.336 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.337 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76910179:::benchmark_data_smithi043_119832_object17982:head# 2024-03-30T13:50:58.337 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.337 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7691975e:::benchmark_data_smithi043_119832_object73937:head# 2024-03-30T13:50:58.337 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.337 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7691e15a:::benchmark_data_smithi043_119832_object64211:head# 2024-03-30T13:50:58.337 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.337 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76931591:::benchmark_data_smithi043_119832_object9758:head# 2024-03-30T13:50:58.337 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.337 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7693406f:::benchmark_data_smithi043_119832_object3365:head# 2024-03-30T13:50:58.337 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.337 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76934514:::benchmark_data_smithi043_119832_object37892:head# 2024-03-30T13:50:58.338 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.338 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7693e6ab:::benchmark_data_smithi043_119832_object70043:head# 2024-03-30T13:50:58.339 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.339 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:769583ad:::benchmark_data_smithi043_119832_object25832:head# 2024-03-30T13:50:58.339 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.339 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76963bc0:::benchmark_data_smithi043_119832_object33864:head# 2024-03-30T13:50:58.339 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.339 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:769650a0:::benchmark_data_smithi043_119832_object9573:head# 2024-03-30T13:50:58.339 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.339 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7696abba:::benchmark_data_smithi043_119832_object7106:head# 2024-03-30T13:50:58.339 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.339 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76972901:::benchmark_data_smithi043_119832_object17693:head# 2024-03-30T13:50:58.339 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.340 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76985bf1:::benchmark_data_smithi043_119832_object15196:head# 2024-03-30T13:50:58.340 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.340 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76995378:::benchmark_data_smithi043_119832_object221:head# 2024-03-30T13:50:58.340 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.341 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:769a403d:::benchmark_data_smithi043_119832_object38285:head# 2024-03-30T13:50:58.341 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.341 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:769a421e:::benchmark_data_smithi043_119832_object20275:head# 2024-03-30T13:50:58.341 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.341 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:769ad85f:::benchmark_data_smithi043_119832_object65102:head# 2024-03-30T13:50:58.341 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.341 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:769e5450:::benchmark_data_smithi043_119832_object54862:head# 2024-03-30T13:50:58.341 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.342 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:769eb229:::benchmark_data_smithi043_119832_object60757:head# 2024-03-30T13:50:58.342 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.342 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:769fccf9:::benchmark_data_smithi043_119832_object57994:head# 2024-03-30T13:50:58.342 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.342 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76a07c30:::benchmark_data_smithi043_119832_object6339:head# 2024-03-30T13:50:58.343 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.343 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76a19318:::benchmark_data_smithi043_119832_object66408:head# 2024-03-30T13:50:58.343 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.343 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76a3d06a:::benchmark_data_smithi043_119832_object56670:head# 2024-03-30T13:50:58.343 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.343 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76a47a4c:::benchmark_data_smithi043_119832_object17527:head# 2024-03-30T13:50:58.343 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.343 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76a4c671:::benchmark_data_smithi043_119832_object37398:head# 2024-03-30T13:50:58.343 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.344 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76a5555d:::benchmark_data_smithi043_119832_object69479:head# 2024-03-30T13:50:58.344 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.344 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76a5be4d:::benchmark_data_smithi043_119832_object69851:head# 2024-03-30T13:50:58.344 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.344 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76a6a4dd:::benchmark_data_smithi043_119832_object63792:head# 2024-03-30T13:50:58.344 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.345 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76a7008e:::benchmark_data_smithi043_119832_object73382:head# 2024-03-30T13:50:58.345 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.345 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76a74271:::benchmark_data_smithi043_119832_object32014:head# 2024-03-30T13:50:58.345 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.345 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76a98abc:::benchmark_data_smithi043_119832_object4201:head# 2024-03-30T13:50:58.345 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.345 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76a9bf2c:::benchmark_data_smithi043_119832_object36716:head# 2024-03-30T13:50:58.345 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.345 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76aafb00:::benchmark_data_smithi043_119832_object4621:head# 2024-03-30T13:50:58.345 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.346 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76ab8b9c:::benchmark_data_smithi043_119832_object45040:head# 2024-03-30T13:50:58.346 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.346 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76ac9fc4:::benchmark_data_smithi043_119832_object654:head# 2024-03-30T13:50:58.346 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.347 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76acb9a3:::benchmark_data_smithi043_119832_object47322:head# 2024-03-30T13:50:58.347 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.347 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76ad27de:::benchmark_data_smithi043_119832_object24433:head# 2024-03-30T13:50:58.347 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.347 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76ad3c0e:::benchmark_data_smithi043_119832_object17977:head# 2024-03-30T13:50:58.347 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.348 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76ada1ad:::benchmark_data_smithi043_119832_object37686:head# 2024-03-30T13:50:58.348 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.348 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76adff47:::benchmark_data_smithi043_119832_object47589:head# 2024-03-30T13:50:58.348 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.349 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76ae45fd:::benchmark_data_smithi043_119832_object26210:head# 2024-03-30T13:50:58.349 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.349 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76af469f:::benchmark_data_smithi043_119832_object60642:head# 2024-03-30T13:50:58.349 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.349 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76b0a09c:::benchmark_data_smithi043_119832_object26256:head# 2024-03-30T13:50:58.349 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.350 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76b3ff4d:::benchmark_data_smithi043_119832_object9164:head# 2024-03-30T13:50:58.350 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.350 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76b44ae9:::benchmark_data_smithi043_119832_object56916:head# 2024-03-30T13:50:58.350 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.350 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76b6e26e:::benchmark_data_smithi043_119832_object39577:head# 2024-03-30T13:50:58.350 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.350 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76b738d0:::benchmark_data_smithi043_119832_object31256:head# 2024-03-30T13:50:58.350 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.350 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76b89397:::benchmark_data_smithi043_119832_object85712:head# 2024-03-30T13:50:58.350 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.351 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76b9451b:::benchmark_data_smithi043_119832_object44825:head# 2024-03-30T13:50:58.352 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.352 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76b97727:::benchmark_data_smithi043_119832_object48179:head# 2024-03-30T13:50:58.352 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.352 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76ba1621:::benchmark_data_smithi043_119832_object86487:head# 2024-03-30T13:50:58.352 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.352 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76bac342:::benchmark_data_smithi043_119832_object19334:head# 2024-03-30T13:50:58.352 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.352 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76bb71df:::benchmark_data_smithi043_119832_object14126:head# 2024-03-30T13:50:58.352 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.352 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76bb845e:::benchmark_data_smithi043_119832_object50341:head# 2024-03-30T13:50:58.352 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.354 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76bc54f9:::benchmark_data_smithi043_119832_object41652:head# 2024-03-30T13:50:58.354 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.354 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76bcd23a:::benchmark_data_smithi043_119832_object68544:head# 2024-03-30T13:50:58.354 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.354 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76bd131d:::benchmark_data_smithi043_119832_object41312:head# 2024-03-30T13:50:58.355 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.355 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76bda616:::benchmark_data_smithi043_119832_object34583:head# 2024-03-30T13:50:58.355 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.355 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76bfb7ad:::benchmark_data_smithi043_119832_object480:head# 2024-03-30T13:50:58.355 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.355 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76c00a1d:::benchmark_data_smithi043_119832_object8928:head# 2024-03-30T13:50:58.355 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.355 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76c07dbd:::benchmark_data_smithi043_119832_object4694:head# 2024-03-30T13:50:58.355 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.355 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76c0e758:::benchmark_data_smithi043_119832_object16268:head# 2024-03-30T13:50:58.355 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.357 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76c28885:::benchmark_data_smithi043_119832_object9893:head# 2024-03-30T13:50:58.357 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.358 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76c2d2fb:::benchmark_data_smithi043_119832_object60149:head# 2024-03-30T13:50:58.358 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.358 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76c400cd:::benchmark_data_smithi043_119832_object84478:head# 2024-03-30T13:50:58.358 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.358 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76c4bf6c:::benchmark_data_smithi043_119832_object56112:head# 2024-03-30T13:50:58.358 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.358 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76c5782c:::benchmark_data_smithi043_119832_object27710:head# 2024-03-30T13:50:58.358 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.358 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76c5c00c:::benchmark_data_smithi043_119832_object55686:head# 2024-03-30T13:50:58.358 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.358 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76c6236a:::benchmark_data_smithi043_119832_object7964:head# 2024-03-30T13:50:58.358 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.359 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76c6b4ed:::benchmark_data_smithi043_119832_object60375:head# 2024-03-30T13:50:58.359 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.359 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76c6c387:::benchmark_data_smithi043_119832_object29341:head# 2024-03-30T13:50:58.359 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.361 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76c71da1:::benchmark_data_smithi043_119832_object33159:head# 2024-03-30T13:50:58.361 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.361 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76c8111e:::benchmark_data_smithi043_119832_object14767:head# 2024-03-30T13:50:58.361 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.361 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76c8c6c5:::benchmark_data_smithi043_119832_object33580:head# 2024-03-30T13:50:58.361 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.361 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76ca4902:::benchmark_data_smithi043_119832_object46717:head# 2024-03-30T13:50:58.361 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.362 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76cb2928:::benchmark_data_smithi043_119832_object69222:head# 2024-03-30T13:50:58.362 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.362 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76cd3a34:::benchmark_data_smithi043_119832_object47867:head# 2024-03-30T13:50:58.362 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.362 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76cf0ae1:::benchmark_data_smithi043_119832_object78045:head# 2024-03-30T13:50:58.362 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.362 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76d0eec4:::benchmark_data_smithi043_119832_object31566:head# 2024-03-30T13:50:58.362 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.362 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76d15f1b:::benchmark_data_smithi043_119832_object58344:head# 2024-03-30T13:50:58.362 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.362 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76d1c4a2:::benchmark_data_smithi043_119832_object88191:head# 2024-03-30T13:50:58.362 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.364 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76d2cc43:::benchmark_data_smithi043_119832_object11853:head# 2024-03-30T13:50:58.365 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.365 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76d38395:::benchmark_data_smithi043_119832_object52951:head# 2024-03-30T13:50:58.365 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.365 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76d4c7b1:::benchmark_data_smithi043_119832_object28537:head# 2024-03-30T13:50:58.365 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.365 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76d5e1bc:::benchmark_data_smithi043_119832_object86640:head# 2024-03-30T13:50:58.365 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.365 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76d61e75:::benchmark_data_smithi043_119832_object31392:head# 2024-03-30T13:50:58.365 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.365 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76d69b2f:::benchmark_data_smithi043_119832_object69550:head# 2024-03-30T13:50:58.365 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.365 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76d76458:::benchmark_data_smithi043_119832_object50349:head# 2024-03-30T13:50:58.366 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.366 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76d823dd:::benchmark_data_smithi043_119832_object39053:head# 2024-03-30T13:50:58.366 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.366 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76d82b95:::benchmark_data_smithi043_119832_object71263:head# 2024-03-30T13:50:58.366 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.366 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76d8a73e:::benchmark_data_smithi043_119832_object31004:head# 2024-03-30T13:50:58.366 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.366 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76d907fe:::benchmark_data_smithi043_119832_object80341:head# 2024-03-30T13:50:58.366 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.368 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76d91e8c:::benchmark_data_smithi043_119832_object84160:head# 2024-03-30T13:50:58.368 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.368 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76d98a88:::benchmark_data_smithi043_119832_object20014:head# 2024-03-30T13:50:58.368 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.368 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76d9c560:::benchmark_data_smithi043_119832_object25530:head# 2024-03-30T13:50:58.368 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.368 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76d9e6ef:::benchmark_data_smithi043_119832_object79849:head# 2024-03-30T13:50:58.368 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.369 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76da5c09:::benchmark_data_smithi043_119832_object85762:head# 2024-03-30T13:50:58.369 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.369 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76da8827:::benchmark_data_smithi043_119832_object15214:head# 2024-03-30T13:50:58.369 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.369 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76da9899:::benchmark_data_smithi043_119832_object73279:head# 2024-03-30T13:50:58.369 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.369 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76dd2461:::benchmark_data_smithi043_119832_object68326:head# 2024-03-30T13:50:58.369 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.369 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76dd7d4b:::benchmark_data_smithi043_119832_object25694:head# 2024-03-30T13:50:58.369 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.369 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76de6b69:::benchmark_data_smithi043_119832_object23039:head# 2024-03-30T13:50:58.369 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.370 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76df5472:::benchmark_data_smithi043_119832_object75271:head# 2024-03-30T13:50:58.370 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.370 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76df82e9:::benchmark_data_smithi043_119832_object37084:head# 2024-03-30T13:50:58.370 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.371 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76e05e9c:::benchmark_data_smithi043_119832_object57063:head# 2024-03-30T13:50:58.371 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.371 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76e08791:::benchmark_data_smithi043_119832_object86401:head# 2024-03-30T13:50:58.371 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.372 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76e0c8a4:::benchmark_data_smithi043_119832_object33297:head# 2024-03-30T13:50:58.372 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.372 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76e2fcab:::benchmark_data_smithi043_119832_object75617:head# 2024-03-30T13:50:58.372 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.373 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76e34371:::benchmark_data_smithi043_119832_object49141:head# 2024-03-30T13:50:58.373 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.373 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76e4551d:::benchmark_data_smithi043_119832_object18083:head# 2024-03-30T13:50:58.373 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.373 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76e4ada5:::benchmark_data_smithi043_119832_object85844:head# 2024-03-30T13:50:58.373 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.374 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76e5cd2c:::benchmark_data_smithi043_119832_object77785:head# 2024-03-30T13:50:58.374 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.374 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76e723a3:::benchmark_data_smithi043_119832_object35271:head# 2024-03-30T13:50:58.374 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.375 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76e841ce:::benchmark_data_smithi043_119832_object590:head# 2024-03-30T13:50:58.375 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.375 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76e84ab8:::benchmark_data_smithi043_119832_object24634:head# 2024-03-30T13:50:58.375 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.376 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76e89b81:::benchmark_data_smithi043_119832_object33822:head# 2024-03-30T13:50:58.376 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.376 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76e8a688:::benchmark_data_smithi043_119832_object79785:head# 2024-03-30T13:50:58.376 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.378 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76e944b6:::benchmark_data_smithi043_119832_object36640:head# 2024-03-30T13:50:58.378 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.379 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76e9c877:::benchmark_data_smithi043_119832_object82580:head# 2024-03-30T13:50:58.379 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.379 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76ea0454:::benchmark_data_smithi043_119832_object76109:head# 2024-03-30T13:50:58.379 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.379 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76ea95d1:::benchmark_data_smithi043_119832_object71068:head# 2024-03-30T13:50:58.379 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.381 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76eb268f:::benchmark_data_smithi043_119832_object81852:head# 2024-03-30T13:50:58.381 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.381 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76eb8424:::benchmark_data_smithi043_119832_object20882:head# 2024-03-30T13:50:58.381 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.381 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76eba114:::benchmark_data_smithi043_119832_object63435:head# 2024-03-30T13:50:58.381 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.381 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76ec3284:::benchmark_data_smithi043_119832_object41113:head# 2024-03-30T13:50:58.381 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.381 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76ecb0eb:::benchmark_data_smithi043_119832_object20274:head# 2024-03-30T13:50:58.381 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.381 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76ecc339:::benchmark_data_smithi043_119832_object72311:head# 2024-03-30T13:50:58.382 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.383 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76ed4101:::benchmark_data_smithi043_119832_object3191:head# 2024-03-30T13:50:58.383 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.383 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f002b0:::benchmark_data_smithi043_119832_object54677:head# 2024-03-30T13:50:58.384 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.384 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f02114:::benchmark_data_smithi043_119832_object34563:head# 2024-03-30T13:50:58.384 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.384 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f02b75:::benchmark_data_smithi043_119832_object41627:head# 2024-03-30T13:50:58.384 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.384 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f07f2f:::benchmark_data_smithi043_119832_object1618:head# 2024-03-30T13:50:58.384 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.384 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f0f74a:::benchmark_data_smithi043_119832_object80937:head# 2024-03-30T13:50:58.384 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.384 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f1aeb8:::benchmark_data_smithi043_119832_object68987:head# 2024-03-30T13:50:58.384 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.387 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f1d12f:::benchmark_data_smithi043_119832_object5786:head# 2024-03-30T13:50:58.387 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.387 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f27e73:::benchmark_data_smithi043_119832_object13944:head# 2024-03-30T13:50:58.387 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.387 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f4f329:::benchmark_data_smithi043_119832_object46576:head# 2024-03-30T13:50:58.387 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.387 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f54b6b:::benchmark_data_smithi043_119832_object11376:head# 2024-03-30T13:50:58.387 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.387 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f55e37:::benchmark_data_smithi043_119832_object32742:head# 2024-03-30T13:50:58.387 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.387 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f5803a:::benchmark_data_smithi043_119832_object22935:head# 2024-03-30T13:50:58.387 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.388 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f5b2f7:::benchmark_data_smithi043_119832_object31797:head# 2024-03-30T13:50:58.388 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.388 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f5fc18:::benchmark_data_smithi043_119832_object66246:head# 2024-03-30T13:50:58.388 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.388 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f630f5:::benchmark_data_smithi043_119832_object62881:head# 2024-03-30T13:50:58.388 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.388 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f64720:::benchmark_data_smithi043_119832_object46650:head# 2024-03-30T13:50:58.388 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.391 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f6cf66:::benchmark_data_smithi043_119832_object20978:head# 2024-03-30T13:50:58.391 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.391 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f6d614:::benchmark_data_smithi043_119832_object38898:head# 2024-03-30T13:50:58.391 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.392 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f8347b:::benchmark_data_smithi043_119832_object63811:head# 2024-03-30T13:50:58.392 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.392 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f9665e:::benchmark_data_smithi043_119832_object7983:head# 2024-03-30T13:50:58.392 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.392 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76f97e5c:::benchmark_data_smithi043_119832_object87744:head# 2024-03-30T13:50:58.392 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.392 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76fac369:::benchmark_data_smithi043_119832_object58601:head# 2024-03-30T13:50:58.392 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.392 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76fc6e60:::benchmark_data_smithi043_119832_object52328:head# 2024-03-30T13:50:58.392 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.392 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76fcb48f:::benchmark_data_smithi043_119832_object39033:head# 2024-03-30T13:50:58.392 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.392 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76fe2fa6:::benchmark_data_smithi043_119832_object12162:head# 2024-03-30T13:50:58.393 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.393 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76fe4c4b:::benchmark_data_smithi043_119832_object83524:head# 2024-03-30T13:50:58.393 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.393 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76fed906:::benchmark_data_smithi043_119832_object39813:head# 2024-03-30T13:50:58.393 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.393 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76ffac7d:::benchmark_data_smithi043_119832_object50057:head# 2024-03-30T13:50:58.393 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.393 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76ffacc4:::benchmark_data_smithi043_119832_object15504:head# 2024-03-30T13:50:58.393 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.397 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76ffb1f1:::benchmark_data_smithi043_119832_object33725:head# 2024-03-30T13:50:58.397 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.397 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76ffd41a:::benchmark_data_smithi043_119832_object75430:head# 2024-03-30T13:50:58.397 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.397 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:76fff381:::benchmark_data_smithi043_119832_object43415:head# 2024-03-30T13:50:58.397 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.397 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77006830:::benchmark_data_smithi043_119832_object10930:head# 2024-03-30T13:50:58.397 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.397 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:770152a7:::benchmark_data_smithi043_119832_object14287:head# 2024-03-30T13:50:58.397 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.397 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77016b82:::benchmark_data_smithi043_119832_object3368:head# 2024-03-30T13:50:58.398 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.398 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7702ac8b:::benchmark_data_smithi043_119832_object7483:head# 2024-03-30T13:50:58.398 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.398 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77042aaa:::benchmark_data_smithi043_119832_object53132:head# 2024-03-30T13:50:58.398 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.398 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77049168:::benchmark_data_smithi043_119832_object65336:head# 2024-03-30T13:50:58.398 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.398 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77062880:::benchmark_data_smithi043_119832_object6747:head# 2024-03-30T13:50:58.398 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.398 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77070a54:::benchmark_data_smithi043_119832_object5356:head# 2024-03-30T13:50:58.398 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.398 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7707ecbe:::benchmark_data_smithi043_119832_object2950:head# 2024-03-30T13:50:58.398 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.399 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7707efe4:::benchmark_data_smithi043_119832_object39096:head# 2024-03-30T13:50:58.399 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.399 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:770883c6:::benchmark_data_smithi043_119832_object517:head# 2024-03-30T13:50:58.399 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.399 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7708d62a:::benchmark_data_smithi043_119832_object30691:head# 2024-03-30T13:50:58.399 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.399 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77097002:::benchmark_data_smithi043_119832_object81773:head# 2024-03-30T13:50:58.399 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.403 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:770b1eb2:::benchmark_data_smithi043_119832_object83109:head# 2024-03-30T13:50:58.403 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.403 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:770b979c:::benchmark_data_smithi043_119832_object25006:head# 2024-03-30T13:50:58.403 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.403 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:770c4fb1:::benchmark_data_smithi043_119832_object20483:head# 2024-03-30T13:50:58.403 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.403 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:770c6cd9:::benchmark_data_smithi043_119832_object56518:head# 2024-03-30T13:50:58.403 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.403 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:770c8544:::benchmark_data_smithi043_119832_object74323:head# 2024-03-30T13:50:58.403 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.404 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:770ce4b5:::benchmark_data_smithi043_119832_object43321:head# 2024-03-30T13:50:58.404 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.404 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77116d18:::benchmark_data_smithi043_119832_object64529:head# 2024-03-30T13:50:58.404 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.404 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77121d23:::benchmark_data_smithi043_119832_object29754:head# 2024-03-30T13:50:58.404 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.404 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7712b2d5:::benchmark_data_smithi043_119832_object45613:head# 2024-03-30T13:50:58.404 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.404 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77134d5c:::benchmark_data_smithi043_119832_object39194:head# 2024-03-30T13:50:58.404 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.404 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7713f5ee:::benchmark_data_smithi043_119832_object79323:head# 2024-03-30T13:50:58.404 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.404 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77141761:::benchmark_data_smithi043_119832_object950:head# 2024-03-30T13:50:58.404 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.405 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:771523e0:::benchmark_data_smithi043_119832_object44821:head# 2024-03-30T13:50:58.405 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.405 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77152455:::benchmark_data_smithi043_119832_object83219:head# 2024-03-30T13:50:58.405 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.405 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77157337:::benchmark_data_smithi043_119832_object82817:head# 2024-03-30T13:50:58.405 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.405 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:771819b8:::benchmark_data_smithi043_119832_object25236:head# 2024-03-30T13:50:58.405 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.405 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77188be9:::benchmark_data_smithi043_119832_object61988:head# 2024-03-30T13:50:58.405 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.405 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7718925f:::benchmark_data_smithi043_119832_object3929:head# 2024-03-30T13:50:58.405 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.409 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:771983ef:::benchmark_data_smithi043_119832_object22014:head# 2024-03-30T13:50:58.409 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.409 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:771a95b2:::benchmark_data_smithi043_119832_object55149:head# 2024-03-30T13:50:58.409 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.409 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:771b4f2e:::benchmark_data_smithi043_119832_object34925:head# 2024-03-30T13:50:58.409 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.409 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:771da743:::benchmark_data_smithi043_119832_object22093:head# 2024-03-30T13:50:58.409 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.409 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:771dccac:::benchmark_data_smithi043_119832_object7409:head# 2024-03-30T13:50:58.409 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.409 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:771e3640:::benchmark_data_smithi043_119832_object17668:head# 2024-03-30T13:50:58.409 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.410 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:771ed650:::benchmark_data_smithi043_119832_object49130:head# 2024-03-30T13:50:58.410 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.410 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:771f8760:::benchmark_data_smithi043_119832_object85917:head# 2024-03-30T13:50:58.410 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.410 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77200cfc:::benchmark_data_smithi043_119832_object19385:head# 2024-03-30T13:50:58.410 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.410 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7720917f:::benchmark_data_smithi043_119832_object81895:head# 2024-03-30T13:50:58.410 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.410 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:772151c3:::benchmark_data_smithi043_119832_object14463:head# 2024-03-30T13:50:58.410 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.410 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7721be89:::benchmark_data_smithi043_119832_object84455:head# 2024-03-30T13:50:58.410 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.410 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77237883:::benchmark_data_smithi043_119832_object74960:head# 2024-03-30T13:50:58.410 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.411 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77239b59:::benchmark_data_smithi043_119832_object65753:head# 2024-03-30T13:50:58.411 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.411 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77261a2c:::benchmark_data_smithi043_119832_object33891:head# 2024-03-30T13:50:58.411 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.411 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77263b0c:::benchmark_data_smithi043_119832_object56907:head# 2024-03-30T13:50:58.411 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.411 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7726f43f:::benchmark_data_smithi043_119832_object83532:head# 2024-03-30T13:50:58.411 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.411 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:58.414 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:772743ed:::benchmark_data_smithi043_119832_object66717:head# 2024-03-30T13:50:58.414 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.414 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77275cf0:::benchmark_data_smithi043_119832_object36972:head# 2024-03-30T13:50:58.414 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.414 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7728fcdc:::benchmark_data_smithi043_119832_object25364:head# 2024-03-30T13:50:58.414 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.414 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:772a5bc8:::benchmark_data_smithi043_119832_object12651:head# 2024-03-30T13:50:58.414 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.414 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:772acb51:::benchmark_data_smithi043_119832_object16460:head# 2024-03-30T13:50:58.414 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.414 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:772d4ba6:::benchmark_data_smithi043_119832_object28448:head# 2024-03-30T13:50:58.415 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.415 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:772e4aa8:::benchmark_data_smithi043_119832_object6812:head# 2024-03-30T13:50:58.415 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.415 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:772f0577:::benchmark_data_smithi043_119832_object411:head# 2024-03-30T13:50:58.415 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.415 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:772ff33e:::benchmark_data_smithi043_119832_object53078:head# 2024-03-30T13:50:58.415 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.415 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77301c7f:::benchmark_data_smithi043_119832_object46930:head# 2024-03-30T13:50:58.415 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.415 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7730b1ff:::benchmark_data_smithi043_119832_object44193:head# 2024-03-30T13:50:58.415 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.415 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77310717:::benchmark_data_smithi043_119832_object77541:head# 2024-03-30T13:50:58.415 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.415 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7731d238:::benchmark_data_smithi043_119832_object39541:head# 2024-03-30T13:50:58.416 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.416 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7731f221:::benchmark_data_smithi043_119832_object81776:head# 2024-03-30T13:50:58.416 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.416 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:773293e6:::benchmark_data_smithi043_119832_object86028:head# 2024-03-30T13:50:58.416 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.416 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7732bef2:::benchmark_data_smithi043_119832_object82733:head# 2024-03-30T13:50:58.416 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.417 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7733356e:::benchmark_data_smithi043_119832_object67654:head# 2024-03-30T13:50:58.417 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.417 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77335485:::benchmark_data_smithi043_119832_object37186:head# 2024-03-30T13:50:58.417 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.417 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77336c96:::benchmark_data_smithi043_119832_object54334:head# 2024-03-30T13:50:58.417 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.417 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7733c366:::benchmark_data_smithi043_119832_object77406:head# 2024-03-30T13:50:58.417 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.418 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77345c50:::benchmark_data_smithi043_119832_object77454:head# 2024-03-30T13:50:58.418 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.418 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77358e9f:::benchmark_data_smithi043_119832_object2902:head# 2024-03-30T13:50:58.418 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.418 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77360432:::benchmark_data_smithi043_119832_object82523:head# 2024-03-30T13:50:58.418 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.419 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77374298:::benchmark_data_smithi043_119832_object21135:head# 2024-03-30T13:50:58.419 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.419 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77382ef3:::benchmark_data_smithi043_119832_object45206:head# 2024-03-30T13:50:58.419 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.419 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:773934fa:::benchmark_data_smithi043_119832_object71022:head# 2024-03-30T13:50:58.419 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.419 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77398191:::benchmark_data_smithi043_119832_object39837:head# 2024-03-30T13:50:58.419 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.422 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:773b14e8:::benchmark_data_smithi043_119832_object80115:head# 2024-03-30T13:50:58.422 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.422 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:773ba2bc:::benchmark_data_smithi043_119832_object68529:head# 2024-03-30T13:50:58.422 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.422 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:773c9a07:::benchmark_data_smithi043_119832_object74880:head# 2024-03-30T13:50:58.422 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.422 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:773db4f5:::benchmark_data_smithi043_119832_object39671:head# 2024-03-30T13:50:58.422 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.423 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:773dda23:::benchmark_data_smithi043_119832_object44794:head# 2024-03-30T13:50:58.423 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.423 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:773ddaf3:::benchmark_data_smithi043_119832_object78248:head# 2024-03-30T13:50:58.423 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.423 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:773f6089:::benchmark_data_smithi043_119832_object20221:head# 2024-03-30T13:50:58.423 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.423 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:773f6d37:::benchmark_data_smithi043_119832_object10212:head# 2024-03-30T13:50:58.423 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.424 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7740a9cb:::benchmark_data_smithi043_119832_object63670:head# 2024-03-30T13:50:58.424 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.424 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7740eb85:::benchmark_data_smithi043_119832_object34061:head# 2024-03-30T13:50:58.424 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.424 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77413b5a:::benchmark_data_smithi043_119832_object3374:head# 2024-03-30T13:50:58.424 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.424 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77419f44:::benchmark_data_smithi043_119832_object35441:head# 2024-03-30T13:50:58.424 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.424 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:774249c7:::benchmark_data_smithi043_119832_object74823:head# 2024-03-30T13:50:58.425 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.425 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7742af75:::benchmark_data_smithi043_119832_object70991:head# 2024-03-30T13:50:58.425 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.425 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77432d55:::benchmark_data_smithi043_119832_object70168:head# 2024-03-30T13:50:58.426 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.426 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77436f55:::benchmark_data_smithi043_119832_object37632:head# 2024-03-30T13:50:58.426 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.426 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77441236:::benchmark_data_smithi043_119832_object37394:head# 2024-03-30T13:50:58.426 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.426 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77447675:::benchmark_data_smithi043_119832_object7124:head# 2024-03-30T13:50:58.426 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.427 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7746c3d4:::benchmark_data_smithi043_119832_object24936:head# 2024-03-30T13:50:58.427 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.428 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77472ea9:::benchmark_data_smithi043_119832_object2900:head# 2024-03-30T13:50:58.428 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.428 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77484394:::benchmark_data_smithi043_119832_object10971:head# 2024-03-30T13:50:58.428 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.428 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77494a4f:::benchmark_data_smithi043_119832_object57920:head# 2024-03-30T13:50:58.428 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.428 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:774a02d7:::benchmark_data_smithi043_119832_object23695:head# 2024-03-30T13:50:58.428 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.428 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:774a6645:::benchmark_data_smithi043_119832_object5226:head# 2024-03-30T13:50:58.428 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.429 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:774a864d:::benchmark_data_smithi043_119832_object77050:head# 2024-03-30T13:50:58.429 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.429 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:774bbbe2:::benchmark_data_smithi043_119832_object48474:head# 2024-03-30T13:50:58.429 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.429 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:774c9e75:::benchmark_data_smithi043_119832_object6876:head# 2024-03-30T13:50:58.429 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.429 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:774e443c:::benchmark_data_smithi043_119832_object37200:head# 2024-03-30T13:50:58.429 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.430 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:774fcfa5:::benchmark_data_smithi043_119832_object79636:head# 2024-03-30T13:50:58.430 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.430 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:774fe019:::benchmark_data_smithi043_119832_object12031:head# 2024-03-30T13:50:58.430 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.430 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77501267:::benchmark_data_smithi043_119832_object10263:head# 2024-03-30T13:50:58.430 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.431 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7750a4d3:::benchmark_data_smithi043_119832_object72223:head# 2024-03-30T13:50:58.431 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.431 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7750c401:::benchmark_data_smithi043_119832_object4750:head# 2024-03-30T13:50:58.431 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.431 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:775141c6:::benchmark_data_smithi043_119832_object25391:head# 2024-03-30T13:50:58.431 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.432 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77524707:::benchmark_data_smithi043_119832_object65715:head# 2024-03-30T13:50:58.432 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.432 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7752df78:::benchmark_data_smithi043_119832_object57951:head# 2024-03-30T13:50:58.432 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.433 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7753740e:::benchmark_data_smithi043_119832_object20311:head# 2024-03-30T13:50:58.433 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.433 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7753fcaa:::benchmark_data_smithi043_119832_object17733:head# 2024-03-30T13:50:58.433 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.433 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7754b3f9:::benchmark_data_smithi043_119832_object21331:head# 2024-03-30T13:50:58.433 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.434 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:775507be:::benchmark_data_smithi043_119832_object39522:head# 2024-03-30T13:50:58.434 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.434 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77551bc1:::benchmark_data_smithi043_119832_object15695:head# 2024-03-30T13:50:58.434 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.434 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77559dbd:::benchmark_data_smithi043_119832_object28242:head# 2024-03-30T13:50:58.434 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.437 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:775609d5:::benchmark_data_smithi043_119832_object85001:head# 2024-03-30T13:50:58.437 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.437 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7758d742:::benchmark_data_smithi043_119832_object79331:head# 2024-03-30T13:50:58.437 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.437 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7759623c:::benchmark_data_smithi043_119832_object44534:head# 2024-03-30T13:50:58.437 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.438 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:775a4efa:::benchmark_data_smithi043_119832_object58217:head# 2024-03-30T13:50:58.438 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.438 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:775a929b:::benchmark_data_smithi043_119832_object57219:head# 2024-03-30T13:50:58.438 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.438 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:775d0171:::benchmark_data_smithi043_119832_object28590:head# 2024-03-30T13:50:58.438 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.438 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:775d7062:::benchmark_data_smithi043_119832_object27711:head# 2024-03-30T13:50:58.438 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.438 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:775e292e:::benchmark_data_smithi043_119832_object42030:head# 2024-03-30T13:50:58.438 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.438 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:775e69a5:::benchmark_data_smithi043_119832_object30193:head# 2024-03-30T13:50:58.438 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.438 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:775ee9bb:::benchmark_data_smithi043_119832_object41602:head# 2024-03-30T13:50:58.439 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.440 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77607de8:::benchmark_data_smithi043_119832_object638:head# 2024-03-30T13:50:58.440 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.441 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7760b7fc:::benchmark_data_smithi043_119832_object37769:head# 2024-03-30T13:50:58.441 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.441 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7761b85e:::benchmark_data_smithi043_119832_object17189:head# 2024-03-30T13:50:58.441 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.441 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:776227fd:::benchmark_data_smithi043_119832_object82719:head# 2024-03-30T13:50:58.441 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.441 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:776269b3:::benchmark_data_smithi043_119832_object11948:head# 2024-03-30T13:50:58.441 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.441 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77626e69:::benchmark_data_smithi043_119832_object25421:head# 2024-03-30T13:50:58.441 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.441 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77631c0f:::benchmark_data_smithi043_119832_object86004:head# 2024-03-30T13:50:58.441 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.441 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77639347:::benchmark_data_smithi043_119832_object43465:head# 2024-03-30T13:50:58.441 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.442 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77639db3:::benchmark_data_smithi043_119832_object45063:head# 2024-03-30T13:50:58.442 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.443 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:776570ac:::benchmark_data_smithi043_119832_object69455:head# 2024-03-30T13:50:58.443 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.443 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77659985:::benchmark_data_smithi043_119832_object55861:head# 2024-03-30T13:50:58.443 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.443 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77672cb7:::benchmark_data_smithi043_119832_object22663:head# 2024-03-30T13:50:58.444 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.444 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77681d51:::benchmark_data_smithi043_119832_object6840:head# 2024-03-30T13:50:58.444 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.444 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77683fbc:::benchmark_data_smithi043_119832_object84473:head# 2024-03-30T13:50:58.444 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.444 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7768b7d0:::benchmark_data_smithi043_119832_object69107:head# 2024-03-30T13:50:58.444 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.444 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77690ffc:::benchmark_data_smithi043_119832_object76619:head# 2024-03-30T13:50:58.444 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.444 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77691632:::benchmark_data_smithi043_119832_object1870:head# 2024-03-30T13:50:58.445 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.445 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:776955f3:::benchmark_data_smithi043_119832_object16705:head# 2024-03-30T13:50:58.445 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.445 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7769eb2a:::benchmark_data_smithi043_119832_object33342:head# 2024-03-30T13:50:58.445 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.445 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:776a714d:::benchmark_data_smithi043_119832_object68451:head# 2024-03-30T13:50:58.445 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.446 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:776bf424:::benchmark_data_smithi043_119832_object43498:head# 2024-03-30T13:50:58.446 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.446 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:776c7669:::benchmark_data_smithi043_119832_object47001:head# 2024-03-30T13:50:58.446 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.446 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:776d8d81:::benchmark_data_smithi043_119832_object74897:head# 2024-03-30T13:50:58.446 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.449 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:776ef386:::benchmark_data_smithi043_119832_object54346:head# 2024-03-30T13:50:58.449 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.449 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7771b326:::benchmark_data_smithi043_119832_object76391:head# 2024-03-30T13:50:58.450 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.450 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:777226d1:::benchmark_data_smithi043_119832_object31641:head# 2024-03-30T13:50:58.450 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.451 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7773b15d:::benchmark_data_smithi043_119832_object6685:head# 2024-03-30T13:50:58.451 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.451 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:777442dc:::benchmark_data_smithi043_119832_object85008:head# 2024-03-30T13:50:58.451 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.451 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:777469e7:::benchmark_data_smithi043_119832_object64891:head# 2024-03-30T13:50:58.451 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.451 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:777510b6:::benchmark_data_smithi043_119832_object47017:head# 2024-03-30T13:50:58.451 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.452 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77758f8c:::benchmark_data_smithi043_119832_object906:head# 2024-03-30T13:50:58.452 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.452 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77762d46:::benchmark_data_smithi043_119832_object84314:head# 2024-03-30T13:50:58.452 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.452 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:777648c0:::benchmark_data_smithi043_119832_object119:head# 2024-03-30T13:50:58.452 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:58.452 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.453 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:777710a6:::benchmark_data_smithi043_119832_object54048:head# 2024-03-30T13:50:58.453 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:58.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:58.457 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:777800e0:::benchmark_data_smithi043_119832_object36314:head# 2024-03-30T13:50:58.457 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.457 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7778fc23:::benchmark_data_smithi043_119832_object25561:head# 2024-03-30T13:50:58.457 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.457 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:777a2575:::benchmark_data_smithi043_119832_object73320:head# 2024-03-30T13:50:58.457 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.458 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:777aa0fb:::benchmark_data_smithi043_119832_object6649:head# 2024-03-30T13:50:58.458 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.458 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:777b9d91:::benchmark_data_smithi043_119832_object22818:head# 2024-03-30T13:50:58.458 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.458 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:777ba0bf:::benchmark_data_smithi043_119832_object19298:head# 2024-03-30T13:50:58.458 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.458 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:777db56d:::benchmark_data_smithi043_119832_object41635:head# 2024-03-30T13:50:58.458 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.458 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:777e067e:::benchmark_data_smithi043_119832_object21597:head# 2024-03-30T13:50:58.458 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.458 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:777e9e72:::benchmark_data_smithi043_119832_object58148:head# 2024-03-30T13:50:58.459 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.459 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:777ffc71:::benchmark_data_smithi043_119832_object21002:head# 2024-03-30T13:50:58.459 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.459 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7781020c:::benchmark_data_smithi043_119832_object39647:head# 2024-03-30T13:50:58.459 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.459 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7781a09f:::benchmark_data_smithi043_119832_object34795:head# 2024-03-30T13:50:58.459 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.459 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7782adb0:::benchmark_data_smithi043_119832_object45977:head# 2024-03-30T13:50:58.459 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.459 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:778428b7:::benchmark_data_smithi043_119832_object72258:head# 2024-03-30T13:50:58.459 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.459 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7784779b:::benchmark_data_smithi043_119832_object53229:head# 2024-03-30T13:50:58.460 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.460 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77855a18:::benchmark_data_smithi043_119832_object80384:head# 2024-03-30T13:50:58.460 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.460 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7787241d:::benchmark_data_smithi043_119832_object36746:head# 2024-03-30T13:50:58.460 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.460 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7787d7be:::benchmark_data_smithi043_119832_object14565:head# 2024-03-30T13:50:58.460 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.464 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77889a38:::benchmark_data_smithi043_119832_object64654:head# 2024-03-30T13:50:58.464 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.465 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7789582c:::benchmark_data_smithi043_119832_object67085:head# 2024-03-30T13:50:58.465 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.465 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77898d72:::benchmark_data_smithi043_119832_object58597:head# 2024-03-30T13:50:58.465 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.465 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:778a14b8:::benchmark_data_smithi043_119832_object34971:head# 2024-03-30T13:50:58.465 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.465 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:778a2c82:::benchmark_data_smithi043_119832_object11492:head# 2024-03-30T13:50:58.465 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.465 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:778bbe92:::benchmark_data_smithi043_119832_object41770:head# 2024-03-30T13:50:58.465 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.465 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:778c3118:::benchmark_data_smithi043_119832_object9236:head# 2024-03-30T13:50:58.465 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.466 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:778d58eb:::benchmark_data_smithi043_119832_object48735:head# 2024-03-30T13:50:58.466 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.466 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:778d841b:::benchmark_data_smithi043_119832_object8917:head# 2024-03-30T13:50:58.466 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.466 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:778e0ac4:::benchmark_data_smithi043_119832_object59056:head# 2024-03-30T13:50:58.466 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.466 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:778e9700:::benchmark_data_smithi043_119832_object60388:head# 2024-03-30T13:50:58.466 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.466 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:778ec452:::benchmark_data_smithi043_119832_object38011:head# 2024-03-30T13:50:58.466 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.466 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:778f1b1a:::benchmark_data_smithi043_119832_object12189:head# 2024-03-30T13:50:58.467 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.467 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:778f4d7a:::benchmark_data_smithi043_119832_object11754:head# 2024-03-30T13:50:58.467 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.467 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77907deb:::benchmark_data_smithi043_119832_object71172:head# 2024-03-30T13:50:58.467 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.467 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7792338f:::benchmark_data_smithi043_119832_object57683:head# 2024-03-30T13:50:58.467 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.467 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77925468:::benchmark_data_smithi043_119832_object44440:head# 2024-03-30T13:50:58.467 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.467 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7792f64e:::benchmark_data_smithi043_119832_object5044:head# 2024-03-30T13:50:58.467 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.467 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77935a71:::benchmark_data_smithi043_119832_object67360:head# 2024-03-30T13:50:58.467 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.468 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77938002:::benchmark_data_smithi043_119832_object77257:head# 2024-03-30T13:50:58.468 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.471 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77955263:::benchmark_data_smithi043_119832_object50231:head# 2024-03-30T13:50:58.471 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.471 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77956286:::benchmark_data_smithi043_119832_object7671:head# 2024-03-30T13:50:58.471 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.471 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77957c0c:::benchmark_data_smithi043_119832_object33791:head# 2024-03-30T13:50:58.472 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.472 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7796622e:::benchmark_data_smithi043_119832_object32187:head# 2024-03-30T13:50:58.472 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.472 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7796fbd9:::benchmark_data_smithi043_119832_object71461:head# 2024-03-30T13:50:58.472 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.472 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:779702c0:::benchmark_data_smithi043_119832_object33009:head# 2024-03-30T13:50:58.472 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.472 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77977ab5:::benchmark_data_smithi043_119832_object36885:head# 2024-03-30T13:50:58.472 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.472 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:779783ca:::benchmark_data_smithi043_119832_object44919:head# 2024-03-30T13:50:58.472 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.472 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7798687d:::benchmark_data_smithi043_119832_object35847:head# 2024-03-30T13:50:58.472 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.473 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7799d27b:::benchmark_data_smithi043_119832_object2479:head# 2024-03-30T13:50:58.473 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.473 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:779a82dc:::benchmark_data_smithi043_119832_object22866:head# 2024-03-30T13:50:58.473 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.473 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:779a8a1d:::benchmark_data_smithi043_119832_object80534:head# 2024-03-30T13:50:58.473 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.473 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:779acdde:::benchmark_data_smithi043_119832_object39548:head# 2024-03-30T13:50:58.473 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.473 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:779af97f:::benchmark_data_smithi043_119832_object17814:head# 2024-03-30T13:50:58.473 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.473 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:779b1d90:::benchmark_data_smithi043_119832_object33796:head# 2024-03-30T13:50:58.473 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.473 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:779b87ee:::benchmark_data_smithi043_119832_object36739:head# 2024-03-30T13:50:58.473 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.474 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:779cd5c0:::benchmark_data_smithi043_119832_object61629:head# 2024-03-30T13:50:58.474 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.474 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:779cda23:::benchmark_data_smithi043_119832_object49190:head# 2024-03-30T13:50:58.474 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.477 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:779ce7cd:::benchmark_data_smithi043_119832_object35725:head# 2024-03-30T13:50:58.477 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.477 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:779cfd70:::benchmark_data_smithi043_119832_object87499:head# 2024-03-30T13:50:58.477 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.477 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:779e2008:::benchmark_data_smithi043_119832_object61711:head# 2024-03-30T13:50:58.477 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.478 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:779e45cd:::benchmark_data_smithi043_119832_object73904:head# 2024-03-30T13:50:58.478 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.478 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:779e5020:::benchmark_data_smithi043_119832_object7591:head# 2024-03-30T13:50:58.478 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.478 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:779ed28c:::benchmark_data_smithi043_119832_object24692:head# 2024-03-30T13:50:58.478 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.478 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:779f3cea:::benchmark_data_smithi043_119832_object62566:head# 2024-03-30T13:50:58.478 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.478 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77a00abd:::benchmark_data_smithi043_119832_object71547:head# 2024-03-30T13:50:58.478 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.478 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77a07221:::benchmark_data_smithi043_119832_object9456:head# 2024-03-30T13:50:58.478 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.478 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77a0f8da:::benchmark_data_smithi043_119832_object88302:head# 2024-03-30T13:50:58.478 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.479 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77a1faa8:::benchmark_data_smithi043_119832_object60928:head# 2024-03-30T13:50:58.479 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.479 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77a211f1:::benchmark_data_smithi043_119832_object30602:head# 2024-03-30T13:50:58.479 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.479 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77a30230:::benchmark_data_smithi043_119832_object29597:head# 2024-03-30T13:50:58.479 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.479 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77a3dbdd:::benchmark_data_smithi043_119832_object22446:head# 2024-03-30T13:50:58.479 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.479 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77a430b4:::benchmark_data_smithi043_119832_object31567:head# 2024-03-30T13:50:58.479 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.479 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77a4bbc3:::benchmark_data_smithi043_119832_object43021:head# 2024-03-30T13:50:58.479 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.479 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77a4ee4d:::benchmark_data_smithi043_119832_object24241:head# 2024-03-30T13:50:58.479 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.482 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77a5766c:::benchmark_data_smithi043_119832_object23035:head# 2024-03-30T13:50:58.482 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.483 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77a588a4:::benchmark_data_smithi043_119832_object60256:head# 2024-03-30T13:50:58.483 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.483 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77a60718:::benchmark_data_smithi043_119832_object15361:head# 2024-03-30T13:50:58.483 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.483 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77a62333:::benchmark_data_smithi043_119832_object10377:head# 2024-03-30T13:50:58.483 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.483 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77a6dea9:::benchmark_data_smithi043_119832_object30195:head# 2024-03-30T13:50:58.483 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.483 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77a73e71:::benchmark_data_smithi043_119832_object4282:head# 2024-03-30T13:50:58.483 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.483 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77a8bc99:::benchmark_data_smithi043_119832_object11573:head# 2024-03-30T13:50:58.483 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.483 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77a8f2b4:::benchmark_data_smithi043_119832_object88228:head# 2024-03-30T13:50:58.483 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.483 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77a94698:::benchmark_data_smithi043_119832_object15637:head# 2024-03-30T13:50:58.484 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.484 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77a98913:::benchmark_data_smithi043_119832_object39521:head# 2024-03-30T13:50:58.484 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.484 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77aa03ea:::benchmark_data_smithi043_119832_object65664:head# 2024-03-30T13:50:58.484 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.484 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77aa2390:::benchmark_data_smithi043_119832_object72783:head# 2024-03-30T13:50:58.484 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.484 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77aa7ff5:::benchmark_data_smithi043_119832_object66076:head# 2024-03-30T13:50:58.484 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.484 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77ab0772:::benchmark_data_smithi043_119832_object23514:head# 2024-03-30T13:50:58.484 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.484 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77ac24d9:::benchmark_data_smithi043_119832_object48076:head# 2024-03-30T13:50:58.484 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.487 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77ac7994:::benchmark_data_smithi043_119832_object47056:head# 2024-03-30T13:50:58.487 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.487 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77add3c8:::benchmark_data_smithi043_119832_object46065:head# 2024-03-30T13:50:58.487 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.487 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77ae7c6d:::benchmark_data_smithi043_119832_object24205:head# 2024-03-30T13:50:58.487 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.487 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77afddbf:::benchmark_data_smithi043_119832_object64595:head# 2024-03-30T13:50:58.487 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.488 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77b02d5d:::benchmark_data_smithi043_119832_object64922:head# 2024-03-30T13:50:58.488 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.488 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77b1105c:::benchmark_data_smithi043_119832_object1071:head# 2024-03-30T13:50:58.488 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.488 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77b1e1ba:::benchmark_data_smithi043_119832_object34996:head# 2024-03-30T13:50:58.488 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.488 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77b2d237:::benchmark_data_smithi043_119832_object31978:head# 2024-03-30T13:50:58.488 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.488 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77b454a9:::benchmark_data_smithi043_119832_object3515:head# 2024-03-30T13:50:58.488 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.488 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77b47316:::benchmark_data_smithi043_119832_object26801:head# 2024-03-30T13:50:58.488 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.488 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77b49e78:::benchmark_data_smithi043_119832_object25272:head# 2024-03-30T13:50:58.488 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.489 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77b60545:::benchmark_data_smithi043_119832_object54443:head# 2024-03-30T13:50:58.489 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.489 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77b7d2b8:::benchmark_data_smithi043_119832_object88201:head# 2024-03-30T13:50:58.489 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.490 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77b8279e:::benchmark_data_smithi043_119832_object67925:head# 2024-03-30T13:50:58.490 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.490 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77b86451:::benchmark_data_smithi043_119832_object62688:head# 2024-03-30T13:50:58.490 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.491 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77b9e078:::benchmark_data_smithi043_119832_object79059:head# 2024-03-30T13:50:58.491 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.491 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77ba3d43:::benchmark_data_smithi043_119832_object64001:head# 2024-03-30T13:50:58.491 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.491 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77ba6811:::benchmark_data_smithi043_119832_object30388:head# 2024-03-30T13:50:58.491 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.491 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77baa23a:::benchmark_data_smithi043_119832_object88030:head# 2024-03-30T13:50:58.491 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.491 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77bab356:::benchmark_data_smithi043_119832_object58575:head# 2024-03-30T13:50:58.491 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.491 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77bb8786:::benchmark_data_smithi043_119832_object32210:head# 2024-03-30T13:50:58.491 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.491 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77bbb3dd:::benchmark_data_smithi043_119832_object29042:head# 2024-03-30T13:50:58.491 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.492 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77bd4b49:::benchmark_data_smithi043_119832_object24316:head# 2024-03-30T13:50:58.492 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.492 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77bdc2d2:::benchmark_data_smithi043_119832_object28761:head# 2024-03-30T13:50:58.492 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.494 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77bee5d9:::benchmark_data_smithi043_119832_object52083:head# 2024-03-30T13:50:58.494 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.494 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77bf475c:::benchmark_data_smithi043_119832_object77199:head# 2024-03-30T13:50:58.494 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.494 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77bf6e13:::benchmark_data_smithi043_119832_object22388:head# 2024-03-30T13:50:58.494 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.494 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77bf70dd:::benchmark_data_smithi043_119832_object44460:head# 2024-03-30T13:50:58.494 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.494 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77bfa088:::benchmark_data_smithi043_119832_object83385:head# 2024-03-30T13:50:58.494 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.495 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77c00e46:::benchmark_data_smithi043_119832_object79186:head# 2024-03-30T13:50:58.495 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.495 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77c01f77:::benchmark_data_smithi043_119832_object53408:head# 2024-03-30T13:50:58.495 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.495 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77c238ce:::benchmark_data_smithi043_119832_object32:head# 2024-03-30T13:50:58.495 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.495 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77c271e5:::benchmark_data_smithi043_119832_object67496:head# 2024-03-30T13:50:58.495 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.495 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77c31908:::benchmark_data_smithi043_119832_object4274:head# 2024-03-30T13:50:58.495 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.495 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77c394a2:::benchmark_data_smithi043_119832_object81382:head# 2024-03-30T13:50:58.495 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.498 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77c3aff8:::benchmark_data_smithi043_119832_object47242:head# 2024-03-30T13:50:58.498 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.498 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77c3e712:::benchmark_data_smithi043_119832_object53108:head# 2024-03-30T13:50:58.498 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.498 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77c48b76:::benchmark_data_smithi043_119832_object10795:head# 2024-03-30T13:50:58.498 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.498 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77c564c3:::benchmark_data_smithi043_119832_object81582:head# 2024-03-30T13:50:58.498 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.498 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77c62720:::benchmark_data_smithi043_119832_object65892:head# 2024-03-30T13:50:58.498 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.498 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77c66b69:::benchmark_data_smithi043_119832_object60244:head# 2024-03-30T13:50:58.498 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.499 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77c7aa85:::benchmark_data_smithi043_119832_object47368:head# 2024-03-30T13:50:58.499 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.499 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77c91853:::benchmark_data_smithi043_119832_object64585:head# 2024-03-30T13:50:58.499 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.499 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77ca9eb1:::benchmark_data_smithi043_119832_object36999:head# 2024-03-30T13:50:58.499 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.499 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77cafff7:::benchmark_data_smithi043_119832_object5110:head# 2024-03-30T13:50:58.499 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.499 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77cc8067:::benchmark_data_smithi043_119832_object43332:head# 2024-03-30T13:50:58.499 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.501 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77cca033:::benchmark_data_smithi043_119832_object44059:head# 2024-03-30T13:50:58.501 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.502 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77ccb086:::benchmark_data_smithi043_119832_object74241:head# 2024-03-30T13:50:58.502 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.502 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77cdab46:::benchmark_data_smithi043_119832_object76753:head# 2024-03-30T13:50:58.502 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.502 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77ce974f:::benchmark_data_smithi043_119832_object17249:head# 2024-03-30T13:50:58.502 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.502 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77ce9f74:::benchmark_data_smithi043_119832_object2901:head# 2024-03-30T13:50:58.502 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.502 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77cfbf65:::benchmark_data_smithi043_119832_object62757:head# 2024-03-30T13:50:58.502 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.502 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77cfdee8:::benchmark_data_smithi043_119832_object43775:head# 2024-03-30T13:50:58.502 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.502 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77d21faf:::benchmark_data_smithi043_119832_object23543:head# 2024-03-30T13:50:58.503 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.503 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77d2d96e:::benchmark_data_smithi043_119832_object34526:head# 2024-03-30T13:50:58.503 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.503 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77d2edee:::benchmark_data_smithi043_119832_object65337:head# 2024-03-30T13:50:58.503 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.503 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77d4296e:::benchmark_data_smithi043_119832_object85480:head# 2024-03-30T13:50:58.503 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.505 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77d4a633:::benchmark_data_smithi043_119832_object39252:head# 2024-03-30T13:50:58.505 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.505 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77d4d1f6:::benchmark_data_smithi043_119832_object2194:head# 2024-03-30T13:50:58.505 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.505 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77d53232:::benchmark_data_smithi043_119832_object67928:head# 2024-03-30T13:50:58.506 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.506 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77d70529:::benchmark_data_smithi043_119832_object22523:head# 2024-03-30T13:50:58.506 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.506 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77d70773:::benchmark_data_smithi043_119832_object442:head# 2024-03-30T13:50:58.506 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.506 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77d7da19:::benchmark_data_smithi043_119832_object41381:head# 2024-03-30T13:50:58.506 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.506 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77d827bc:::benchmark_data_smithi043_119832_object67189:head# 2024-03-30T13:50:58.506 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.506 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77d8c65d:::benchmark_data_smithi043_119832_object31287:head# 2024-03-30T13:50:58.506 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.506 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77dc19ae:::benchmark_data_smithi043_119832_object1381:head# 2024-03-30T13:50:58.506 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.506 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77dc6748:::benchmark_data_smithi043_119832_object42901:head# 2024-03-30T13:50:58.507 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.507 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77dcf96b:::benchmark_data_smithi043_119832_object81188:head# 2024-03-30T13:50:58.507 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.509 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77de7cd7:::benchmark_data_smithi043_119832_object65952:head# 2024-03-30T13:50:58.509 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.509 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77e17c05:::benchmark_data_smithi043_119832_object11455:head# 2024-03-30T13:50:58.509 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.509 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77e21bc9:::benchmark_data_smithi043_119832_object5175:head# 2024-03-30T13:50:58.509 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.509 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77e3f62b:::benchmark_data_smithi043_119832_object12187:head# 2024-03-30T13:50:58.509 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.510 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77e41e78:::benchmark_data_smithi043_119832_object8415:head# 2024-03-30T13:50:58.510 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.510 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77e47e20:::benchmark_data_smithi043_119832_object8730:head# 2024-03-30T13:50:58.510 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.510 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77e497ef:::benchmark_data_smithi043_119832_object37453:head# 2024-03-30T13:50:58.510 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.510 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77e4ba3b:::benchmark_data_smithi043_119832_object27930:head# 2024-03-30T13:50:58.510 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.510 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77e510e9:::benchmark_data_smithi043_119832_object3056:head# 2024-03-30T13:50:58.510 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.510 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77e53d21:::benchmark_data_smithi043_119832_object36938:head# 2024-03-30T13:50:58.510 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.510 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77e54635:::benchmark_data_smithi043_119832_object85123:head# 2024-03-30T13:50:58.511 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.513 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77e5bcc3:::benchmark_data_smithi043_119832_object15252:head# 2024-03-30T13:50:58.513 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.513 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77e5ef2e:::benchmark_data_smithi043_119832_object67940:head# 2024-03-30T13:50:58.513 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.513 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77e655b3:::benchmark_data_smithi043_119832_object12881:head# 2024-03-30T13:50:58.513 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.513 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77e94f50:::benchmark_data_smithi043_119832_object36076:head# 2024-03-30T13:50:58.513 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.513 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77e97bff:::benchmark_data_smithi043_119832_object62472:head# 2024-03-30T13:50:58.513 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.514 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77ea291a:::benchmark_data_smithi043_119832_object82003:head# 2024-03-30T13:50:58.514 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.514 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77eab349:::benchmark_data_smithi043_119832_object50379:head# 2024-03-30T13:50:58.514 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.514 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77ee0e7b:::benchmark_data_smithi043_119832_object55135:head# 2024-03-30T13:50:58.514 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.514 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77eecec9:::benchmark_data_smithi043_119832_object54271:head# 2024-03-30T13:50:58.514 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.514 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77ef12fe:::benchmark_data_smithi043_119832_object70370:head# 2024-03-30T13:50:58.514 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.514 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77ef35f8:::benchmark_data_smithi043_119832_object77684:head# 2024-03-30T13:50:58.514 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.517 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77f10188:::benchmark_data_smithi043_119832_object33694:head# 2024-03-30T13:50:58.517 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.517 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77f15ed4:::benchmark_data_smithi043_119832_object61964:head# 2024-03-30T13:50:58.517 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.517 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77f252ba:::benchmark_data_smithi043_119832_object87153:head# 2024-03-30T13:50:58.518 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.518 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77f3416e:::benchmark_data_smithi043_119832_object51512:head# 2024-03-30T13:50:58.518 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.518 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77f34bc5:::benchmark_data_smithi043_119832_object40208:head# 2024-03-30T13:50:58.518 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.518 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77f40b0a:::benchmark_data_smithi043_119832_object24308:head# 2024-03-30T13:50:58.518 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.518 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77f473ab:::benchmark_data_smithi043_119832_object42605:head# 2024-03-30T13:50:58.518 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.518 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77f47d4a:::benchmark_data_smithi043_119832_object17610:head# 2024-03-30T13:50:58.518 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.518 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77f4a1e7:::benchmark_data_smithi043_119832_object61654:head# 2024-03-30T13:50:58.518 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.519 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77f4cd3e:::benchmark_data_smithi043_119832_object59847:head# 2024-03-30T13:50:58.519 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.519 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77f512bd:::benchmark_data_smithi043_119832_object46639:head# 2024-03-30T13:50:58.519 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.519 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77f5414c:::benchmark_data_smithi043_119832_object37329:head# 2024-03-30T13:50:58.519 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.519 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77f57e8d:::benchmark_data_smithi043_119832_object15357:head# 2024-03-30T13:50:58.519 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.522 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77f69a13:::benchmark_data_smithi043_119832_object45265:head# 2024-03-30T13:50:58.522 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.522 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77f87896:::benchmark_data_smithi043_119832_object16114:head# 2024-03-30T13:50:58.522 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.522 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77f878ae:::benchmark_data_smithi043_119832_object64748:head# 2024-03-30T13:50:58.522 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.522 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77f9016a:::benchmark_data_smithi043_119832_object53400:head# 2024-03-30T13:50:58.522 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.522 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77f91570:::benchmark_data_smithi043_119832_object9313:head# 2024-03-30T13:50:58.522 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.523 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77f9182d:::benchmark_data_smithi043_119832_object67682:head# 2024-03-30T13:50:58.523 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.523 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77f99cd8:::benchmark_data_smithi043_119832_object76434:head# 2024-03-30T13:50:58.523 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.523 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77fa338e:::benchmark_data_smithi043_119832_object50316:head# 2024-03-30T13:50:58.523 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.523 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77fa8519:::benchmark_data_smithi043_119832_object25105:head# 2024-03-30T13:50:58.523 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.523 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77faad45:::benchmark_data_smithi043_119832_object61604:head# 2024-03-30T13:50:58.523 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.523 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77faca7b:::benchmark_data_smithi043_119832_object22153:head# 2024-03-30T13:50:58.523 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.523 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77fb009b:::benchmark_data_smithi043_119832_object63786:head# 2024-03-30T13:50:58.523 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.524 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77fbef9f:::benchmark_data_smithi043_119832_object51753:head# 2024-03-30T13:50:58.524 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.524 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77fc6800:::benchmark_data_smithi043_119832_object3508:head# 2024-03-30T13:50:58.524 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.525 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77fcd1c0:::benchmark_data_smithi043_119832_object25418:head# 2024-03-30T13:50:58.526 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.526 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77fcd6c7:::benchmark_data_smithi043_119832_object25493:head# 2024-03-30T13:50:58.526 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.526 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77fd6e9b:::benchmark_data_smithi043_119832_object4336:head# 2024-03-30T13:50:58.526 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.526 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77fda5f7:::benchmark_data_smithi043_119832_object69944:head# 2024-03-30T13:50:58.526 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.526 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77fe3ef5:::benchmark_data_smithi043_119832_object17670:head# 2024-03-30T13:50:58.526 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.526 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77fece0f:::benchmark_data_smithi043_119832_object11615:head# 2024-03-30T13:50:58.526 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.526 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:77fef65d:::benchmark_data_smithi043_119832_object74794:head# 2024-03-30T13:50:58.526 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.526 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:780198f2:::benchmark_data_smithi043_119832_object48870:head# 2024-03-30T13:50:58.527 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.527 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78021205:::benchmark_data_smithi043_119832_object13293:head# 2024-03-30T13:50:58.527 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.527 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7802ed90:::benchmark_data_smithi043_119832_object30225:head# 2024-03-30T13:50:58.527 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.528 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78046b20:::benchmark_data_smithi043_119832_object32886:head# 2024-03-30T13:50:58.528 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.528 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78047a9b:::benchmark_data_smithi043_119832_object58977:head# 2024-03-30T13:50:58.528 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.528 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7804d608:::benchmark_data_smithi043_119832_object18323:head# 2024-03-30T13:50:58.528 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.528 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:780779a7:::benchmark_data_smithi043_119832_object87286:head# 2024-03-30T13:50:58.528 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.528 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:780791a0:::benchmark_data_smithi043_119832_object59042:head# 2024-03-30T13:50:58.528 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.528 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7807e7dc:::benchmark_data_smithi043_119832_object61004:head# 2024-03-30T13:50:58.529 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.529 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7808e1a3:::benchmark_data_smithi043_119832_object46510:head# 2024-03-30T13:50:58.529 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.529 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:780993d9:::benchmark_data_smithi043_119832_object85395:head# 2024-03-30T13:50:58.530 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.530 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:780a41b2:::benchmark_data_smithi043_119832_object20004:head# 2024-03-30T13:50:58.530 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.530 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:780ad6cb:::benchmark_data_smithi043_119832_object2340:head# 2024-03-30T13:50:58.530 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.530 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:780c787d:::benchmark_data_smithi043_119832_object15865:head# 2024-03-30T13:50:58.530 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.530 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:780c9a21:::benchmark_data_smithi043_119832_object72483:head# 2024-03-30T13:50:58.531 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.531 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:780cdac2:::benchmark_data_smithi043_119832_object18114:head# 2024-03-30T13:50:58.531 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.531 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:780e73ff:::benchmark_data_smithi043_119832_object86373:head# 2024-03-30T13:50:58.531 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.531 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:780f1950:::benchmark_data_smithi043_119832_object61266:head# 2024-03-30T13:50:58.532 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.532 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:780f3e71:::benchmark_data_smithi043_119832_object33569:head# 2024-03-30T13:50:58.532 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.532 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78101913:::benchmark_data_smithi043_119832_object46144:head# 2024-03-30T13:50:58.532 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.533 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:781064b0:::benchmark_data_smithi043_119832_object52703:head# 2024-03-30T13:50:58.533 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.533 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7810f910:::benchmark_data_smithi043_119832_object45281:head# 2024-03-30T13:50:58.533 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.533 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78110e58:::benchmark_data_smithi043_119832_object49116:head# 2024-03-30T13:50:58.533 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.534 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:781295fb:::benchmark_data_smithi043_119832_object76451:head# 2024-03-30T13:50:58.534 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.534 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7812f811:::benchmark_data_smithi043_119832_object46171:head# 2024-03-30T13:50:58.534 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.535 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:781310eb:::benchmark_data_smithi043_119832_object34613:head# 2024-03-30T13:50:58.535 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.535 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7813660d:::benchmark_data_smithi043_119832_object41553:head# 2024-03-30T13:50:58.535 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.536 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78146b1e:::benchmark_data_smithi043_119832_object34317:head# 2024-03-30T13:50:58.536 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.536 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78165b3e:::benchmark_data_smithi043_119832_object39955:head# 2024-03-30T13:50:58.536 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.536 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:781665eb:::benchmark_data_smithi043_119832_object69023:head# 2024-03-30T13:50:58.536 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.537 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78166601:::benchmark_data_smithi043_119832_object59689:head# 2024-03-30T13:50:58.537 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.537 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:781676d4:::benchmark_data_smithi043_119832_object7604:head# 2024-03-30T13:50:58.537 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.537 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78178136:::benchmark_data_smithi043_119832_object48061:head# 2024-03-30T13:50:58.538 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.538 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78178b18:::benchmark_data_smithi043_119832_object32046:head# 2024-03-30T13:50:58.538 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.538 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:781816b0:::benchmark_data_smithi043_119832_object35446:head# 2024-03-30T13:50:58.538 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.539 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78196561:::benchmark_data_smithi043_119832_object28607:head# 2024-03-30T13:50:58.539 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.539 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:781a5a31:::benchmark_data_smithi043_119832_object83035:head# 2024-03-30T13:50:58.539 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.540 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:781b746e:::benchmark_data_smithi043_119832_object75291:head# 2024-03-30T13:50:58.540 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.540 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:781d4540:::benchmark_data_smithi043_119832_object21506:head# 2024-03-30T13:50:58.540 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.540 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:781e4614:::benchmark_data_smithi043_119832_object12773:head# 2024-03-30T13:50:58.540 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.541 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:781fdcc9:::benchmark_data_smithi043_119832_object68848:head# 2024-03-30T13:50:58.541 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.541 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:782019fe:::benchmark_data_smithi043_119832_object27789:head# 2024-03-30T13:50:58.541 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.541 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78240d6e:::benchmark_data_smithi043_119832_object85033:head# 2024-03-30T13:50:58.542 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.542 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78263daa:::benchmark_data_smithi043_119832_object21258:head# 2024-03-30T13:50:58.542 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.542 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7826d968:::benchmark_data_smithi043_119832_object85749:head# 2024-03-30T13:50:58.542 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.543 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:782759cc:::benchmark_data_smithi043_119832_object15726:head# 2024-03-30T13:50:58.543 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.543 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78278f64:::benchmark_data_smithi043_119832_object51904:head# 2024-03-30T13:50:58.543 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.544 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78288581:::benchmark_data_smithi043_119832_object63264:head# 2024-03-30T13:50:58.544 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.544 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78294f2b:::benchmark_data_smithi043_119832_object41792:head# 2024-03-30T13:50:58.544 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.544 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7829dfd0:::benchmark_data_smithi043_119832_object25933:head# 2024-03-30T13:50:58.544 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.545 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:782af2f7:::benchmark_data_smithi043_119832_object68773:head# 2024-03-30T13:50:58.545 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.545 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:782b131b:::benchmark_data_smithi043_119832_object72539:head# 2024-03-30T13:50:58.545 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.546 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:782bc400:::benchmark_data_smithi043_119832_object60411:head# 2024-03-30T13:50:58.546 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.546 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:782c002b:::benchmark_data_smithi043_119832_object23447:head# 2024-03-30T13:50:58.546 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.546 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:782c6542:::benchmark_data_smithi043_119832_object83649:head# 2024-03-30T13:50:58.547 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.547 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:782c6f0d:::benchmark_data_smithi043_119832_object41820:head# 2024-03-30T13:50:58.547 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.547 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:782d4f16:::benchmark_data_smithi043_119832_object37654:head# 2024-03-30T13:50:58.547 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.548 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:782e19c3:::benchmark_data_smithi043_119832_object80926:head# 2024-03-30T13:50:58.548 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.548 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:782e3275:::benchmark_data_smithi043_119832_object44365:head# 2024-03-30T13:50:58.549 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.549 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:782e395b:::benchmark_data_smithi043_119832_object35031:head# 2024-03-30T13:50:58.549 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.549 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:782e45ea:::benchmark_data_smithi043_119832_object69051:head# 2024-03-30T13:50:58.549 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.549 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:782eb018:::benchmark_data_smithi043_119832_object86444:head# 2024-03-30T13:50:58.549 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.550 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:782eb520:::benchmark_data_smithi043_119832_object45532:head# 2024-03-30T13:50:58.550 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.550 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:782efe7d:::benchmark_data_smithi043_119832_object2448:head# 2024-03-30T13:50:58.551 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.551 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:782f6faa:::benchmark_data_smithi043_119832_object11542:head# 2024-03-30T13:50:58.551 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.551 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:782f98d5:::benchmark_data_smithi043_119832_object72559:head# 2024-03-30T13:50:58.551 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.551 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:782ff87c:::benchmark_data_smithi043_119832_object37087:head# 2024-03-30T13:50:58.552 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.552 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:783021fc:::benchmark_data_smithi043_119832_object21100:head# 2024-03-30T13:50:58.552 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.552 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78302216:::benchmark_data_smithi043_119832_object11728:head# 2024-03-30T13:50:58.552 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.553 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7830c442:::benchmark_data_smithi043_119832_object71782:head# 2024-03-30T13:50:58.553 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.553 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78315939:::benchmark_data_smithi043_119832_object8536:head# 2024-03-30T13:50:58.553 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.554 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78324358:::benchmark_data_smithi043_119832_object19805:head# 2024-03-30T13:50:58.554 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.554 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7832f4c0:::benchmark_data_smithi043_119832_object73071:head# 2024-03-30T13:50:58.554 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.554 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:58.555 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78338601:::benchmark_data_smithi043_119832_object76450:head# 2024-03-30T13:50:58.555 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.555 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7833b572:::benchmark_data_smithi043_119832_object77357:head# 2024-03-30T13:50:58.555 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.556 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7835ac7e:::benchmark_data_smithi043_119832_object25179:head# 2024-03-30T13:50:58.556 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.556 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7835f64d:::benchmark_data_smithi043_119832_object34389:head# 2024-03-30T13:50:58.556 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.556 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7835fb00:::benchmark_data_smithi043_119832_object56482:head# 2024-03-30T13:50:58.556 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.557 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7837349e:::benchmark_data_smithi043_119832_object57884:head# 2024-03-30T13:50:58.557 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.557 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78373848:::benchmark_data_smithi043_119832_object40788:head# 2024-03-30T13:50:58.558 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.558 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78380e4a:::benchmark_data_smithi043_119832_object57873:head# 2024-03-30T13:50:58.558 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.558 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78382298:::benchmark_data_smithi043_119832_object41152:head# 2024-03-30T13:50:58.558 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.558 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:783869f4:::benchmark_data_smithi043_119832_object23206:head# 2024-03-30T13:50:58.558 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.558 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7838adf4:::benchmark_data_smithi043_119832_object19426:head# 2024-03-30T13:50:58.558 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.560 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7839e7a0:::benchmark_data_smithi043_119832_object48043:head# 2024-03-30T13:50:58.561 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.561 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:783abc77:::benchmark_data_smithi043_119832_object14457:head# 2024-03-30T13:50:58.561 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.561 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:783acf3d:::benchmark_data_smithi043_119832_object87080:head# 2024-03-30T13:50:58.561 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.561 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:783b4237:::benchmark_data_smithi043_119832_object51644:head# 2024-03-30T13:50:58.561 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.561 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:783cde34:::benchmark_data_smithi043_119832_object76459:head# 2024-03-30T13:50:58.561 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.561 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:783d7370:::benchmark_data_smithi043_119832_object86075:head# 2024-03-30T13:50:58.561 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.561 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:783f93a7:::benchmark_data_smithi043_119832_object66675:head# 2024-03-30T13:50:58.561 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.561 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:784054aa:::benchmark_data_smithi043_119832_object75993:head# 2024-03-30T13:50:58.561 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.562 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78414016:::benchmark_data_smithi043_119832_object7159:head# 2024-03-30T13:50:58.562 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.562 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7841681b:::benchmark_data_smithi043_119832_object4673:head# 2024-03-30T13:50:58.562 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.563 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7841e92a:::benchmark_data_smithi043_119832_object56867:head# 2024-03-30T13:50:58.563 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.564 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7841ec5f:::benchmark_data_smithi043_119832_object25826:head# 2024-03-30T13:50:58.564 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.564 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78432fe8:::benchmark_data_smithi043_119832_object25849:head# 2024-03-30T13:50:58.564 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.564 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7843b31f:::benchmark_data_smithi043_119832_object29605:head# 2024-03-30T13:50:58.564 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.565 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78442698:::benchmark_data_smithi043_119832_object48767:head# 2024-03-30T13:50:58.565 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.565 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78444c2d:::benchmark_data_smithi043_119832_object14853:head# 2024-03-30T13:50:58.565 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.566 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78452858:::benchmark_data_smithi043_119832_object71959:head# 2024-03-30T13:50:58.566 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.566 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:784547a7:::benchmark_data_smithi043_119832_object61328:head# 2024-03-30T13:50:58.566 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.566 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78461aba:::benchmark_data_smithi043_119832_object13251:head# 2024-03-30T13:50:58.567 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.568 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:784640b1:::benchmark_data_smithi043_119832_object82723:head# 2024-03-30T13:50:58.568 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.568 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78473007:::benchmark_data_smithi043_119832_object34766:head# 2024-03-30T13:50:58.568 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.568 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78490d64:::benchmark_data_smithi043_119832_object79787:head# 2024-03-30T13:50:58.568 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.568 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:784aa92e:::benchmark_data_smithi043_119832_object45827:head# 2024-03-30T13:50:58.568 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.569 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:784b5d03:::benchmark_data_smithi043_119832_object12982:head# 2024-03-30T13:50:58.569 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.569 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:784b8770:::benchmark_data_smithi043_119832_object61304:head# 2024-03-30T13:50:58.569 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.569 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:784bb559:::benchmark_data_smithi043_119832_object12786:head# 2024-03-30T13:50:58.569 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.570 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:784bec48:::benchmark_data_smithi043_119832_object72424:head# 2024-03-30T13:50:58.571 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.571 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:784dc48a:::benchmark_data_smithi043_119832_object80367:head# 2024-03-30T13:50:58.571 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.571 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:784ef07a:::benchmark_data_smithi043_119832_object85738:head# 2024-03-30T13:50:58.571 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.571 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:784f3671:::benchmark_data_smithi043_119832_object61747:head# 2024-03-30T13:50:58.571 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.572 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7850406c:::benchmark_data_smithi043_119832_object78224:head# 2024-03-30T13:50:58.572 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.572 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7851817b:::benchmark_data_smithi043_119832_object31478:head# 2024-03-30T13:50:58.572 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.572 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78525a98:::benchmark_data_smithi043_119832_object81360:head# 2024-03-30T13:50:58.572 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.572 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78525a9d:::benchmark_data_smithi043_119832_object51069:head# 2024-03-30T13:50:58.573 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.574 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7853be32:::benchmark_data_smithi043_119832_object55055:head# 2024-03-30T13:50:58.574 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.574 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7853bf98:::benchmark_data_smithi043_119832_object78360:head# 2024-03-30T13:50:58.574 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.574 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7855b2e5:::benchmark_data_smithi043_119832_object77544:head# 2024-03-30T13:50:58.574 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.574 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7855c429:::benchmark_data_smithi043_119832_object755:head# 2024-03-30T13:50:58.574 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.575 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78568fdf:::benchmark_data_smithi043_119832_object76788:head# 2024-03-30T13:50:58.575 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.575 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7856b7f3:::benchmark_data_smithi043_119832_object51691:head# 2024-03-30T13:50:58.576 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.576 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:785787e0:::benchmark_data_smithi043_119832_object63510:head# 2024-03-30T13:50:58.576 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.576 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7858f23d:::benchmark_data_smithi043_119832_object30170:head# 2024-03-30T13:50:58.576 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.576 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7859a09e:::benchmark_data_smithi043_119832_object80886:head# 2024-03-30T13:50:58.576 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.577 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7859e6d5:::benchmark_data_smithi043_119832_object27378:head# 2024-03-30T13:50:58.577 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.577 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:785c515b:::benchmark_data_smithi043_119832_object8605:head# 2024-03-30T13:50:58.577 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.577 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:785ca3d9:::benchmark_data_smithi043_119832_object74636:head# 2024-03-30T13:50:58.577 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.577 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:785d29e1:::benchmark_data_smithi043_119832_object73533:head# 2024-03-30T13:50:58.577 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.578 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:785d7ee3:::benchmark_data_smithi043_119832_object74973:head# 2024-03-30T13:50:58.578 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.578 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:785e0638:::benchmark_data_smithi043_119832_object43872:head# 2024-03-30T13:50:58.578 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.578 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:785e1802:::benchmark_data_smithi043_119832_object77647:head# 2024-03-30T13:50:58.579 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.579 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:785e4c12:::benchmark_data_smithi043_119832_object2524:head# 2024-03-30T13:50:58.579 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.579 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:785f419d:::benchmark_data_smithi043_119832_object86511:head# 2024-03-30T13:50:58.579 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.579 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78608bab:::benchmark_data_smithi043_119832_object27490:head# 2024-03-30T13:50:58.580 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.580 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7861916d:::benchmark_data_smithi043_119832_object64436:head# 2024-03-30T13:50:58.580 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.580 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7861c57b:::benchmark_data_smithi043_119832_object17232:head# 2024-03-30T13:50:58.580 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.580 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78622844:::benchmark_data_smithi043_119832_object66263:head# 2024-03-30T13:50:58.580 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.581 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7862783e:::benchmark_data_smithi043_119832_object25037:head# 2024-03-30T13:50:58.581 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.581 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7862ea1f:::benchmark_data_smithi043_119832_object21225:head# 2024-03-30T13:50:58.581 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.582 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:786492b5:::benchmark_data_smithi043_119832_object58156:head# 2024-03-30T13:50:58.582 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.582 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7864fa22:::benchmark_data_smithi043_119832_object3715:head# 2024-03-30T13:50:58.582 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.582 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78654d82:::benchmark_data_smithi043_119832_object27700:head# 2024-03-30T13:50:58.582 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.582 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:786659f7:::benchmark_data_smithi043_119832_object35138:head# 2024-03-30T13:50:58.582 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.583 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7866a65d:::benchmark_data_smithi043_119832_object82388:head# 2024-03-30T13:50:58.583 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.583 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78670cfa:::benchmark_data_smithi043_119832_object63226:head# 2024-03-30T13:50:58.583 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.583 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78671501:::benchmark_data_smithi043_119832_object18519:head# 2024-03-30T13:50:58.584 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.584 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7868f52f:::benchmark_data_smithi043_119832_object63098:head# 2024-03-30T13:50:58.584 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.584 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78692e8c:::benchmark_data_smithi043_119832_object38844:head# 2024-03-30T13:50:58.584 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.584 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:786cf821:::benchmark_data_smithi043_119832_object16663:head# 2024-03-30T13:50:58.584 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.585 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:786dd718:::benchmark_data_smithi043_119832_object10246:head# 2024-03-30T13:50:58.585 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.585 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:786e58eb:::benchmark_data_smithi043_119832_object5930:head# 2024-03-30T13:50:58.585 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.585 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:786ec6ee:::benchmark_data_smithi043_119832_object41687:head# 2024-03-30T13:50:58.585 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.585 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:786f3efc:::benchmark_data_smithi043_119832_object1851:head# 2024-03-30T13:50:58.585 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.587 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7870b0d5:::benchmark_data_smithi043_119832_object51558:head# 2024-03-30T13:50:58.587 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.587 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7871663d:::benchmark_data_smithi043_119832_object73881:head# 2024-03-30T13:50:58.587 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.587 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7871e341:::benchmark_data_smithi043_119832_object55403:head# 2024-03-30T13:50:58.587 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.587 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7871e4ad:::benchmark_data_smithi043_119832_object85488:head# 2024-03-30T13:50:58.587 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.587 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78726a56:::benchmark_data_smithi043_119832_object23081:head# 2024-03-30T13:50:58.588 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.588 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7876cd06:::benchmark_data_smithi043_119832_object7064:head# 2024-03-30T13:50:58.588 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.588 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7876f049:::benchmark_data_smithi043_119832_object36418:head# 2024-03-30T13:50:58.588 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.588 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78777f23:::benchmark_data_smithi043_119832_object55167:head# 2024-03-30T13:50:58.588 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.589 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7878066d:::benchmark_data_smithi043_119832_object57446:head# 2024-03-30T13:50:58.589 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.589 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7878a54c:::benchmark_data_smithi043_119832_object14452:head# 2024-03-30T13:50:58.589 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.589 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7878c890:::benchmark_data_smithi043_119832_object6963:head# 2024-03-30T13:50:58.590 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.590 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:787957fb:::benchmark_data_smithi043_119832_object23643:head# 2024-03-30T13:50:58.590 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.590 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7879943f:::benchmark_data_smithi043_119832_object7389:head# 2024-03-30T13:50:58.590 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.590 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:787a2203:::benchmark_data_smithi043_119832_object58189:head# 2024-03-30T13:50:58.590 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.590 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:787a37e6:::benchmark_data_smithi043_119832_object610:head# 2024-03-30T13:50:58.590 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.592 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:787ba0ac:::benchmark_data_smithi043_119832_object50072:head# 2024-03-30T13:50:58.592 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.592 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:787c008a:::benchmark_data_smithi043_119832_object45142:head# 2024-03-30T13:50:58.592 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.592 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:787c8caa:::benchmark_data_smithi043_119832_object49910:head# 2024-03-30T13:50:58.592 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.592 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:787ca7bc:::benchmark_data_smithi043_119832_object66233:head# 2024-03-30T13:50:58.592 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.592 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:787cf92e:::benchmark_data_smithi043_119832_object12885:head# 2024-03-30T13:50:58.592 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.592 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:787e8d06:::benchmark_data_smithi043_119832_object13171:head# 2024-03-30T13:50:58.592 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.593 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:787f7480:::benchmark_data_smithi043_119832_object42188:head# 2024-03-30T13:50:58.593 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:58.594 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:58.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:58.596 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78805f31:::benchmark_data_smithi043_119832_object76718:head# 2024-03-30T13:50:58.597 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.597 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78807660:::benchmark_data_smithi043_119832_object8971:head# 2024-03-30T13:50:58.597 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.597 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78814453:::benchmark_data_smithi043_119832_object42335:head# 2024-03-30T13:50:58.597 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.597 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7881b70e:::benchmark_data_smithi043_119832_object15667:head# 2024-03-30T13:50:58.597 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.597 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7881ed9d:::benchmark_data_smithi043_119832_object63031:head# 2024-03-30T13:50:58.597 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.597 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7883a34b:::benchmark_data_smithi043_119832_object15352:head# 2024-03-30T13:50:58.597 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.597 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7883c20a:::benchmark_data_smithi043_119832_object1422:head# 2024-03-30T13:50:58.597 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.598 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7883d982:::benchmark_data_smithi043_119832_object33911:head# 2024-03-30T13:50:58.598 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.598 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:788633a0:::benchmark_data_smithi043_119832_object42506:head# 2024-03-30T13:50:58.598 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.598 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:788696fb:::benchmark_data_smithi043_119832_object70427:head# 2024-03-30T13:50:58.598 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.598 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7886ce66:::benchmark_data_smithi043_119832_object78470:head# 2024-03-30T13:50:58.598 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.598 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7887c005:::benchmark_data_smithi043_119832_object46334:head# 2024-03-30T13:50:58.598 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.598 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7888709d:::benchmark_data_smithi043_119832_object46960:head# 2024-03-30T13:50:58.598 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.598 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78890a16:::benchmark_data_smithi043_119832_object72271:head# 2024-03-30T13:50:58.599 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.601 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:788a0871:::benchmark_data_smithi043_119832_object65036:head# 2024-03-30T13:50:58.601 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.601 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:788a1f04:::benchmark_data_smithi043_119832_object779:head# 2024-03-30T13:50:58.602 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.602 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:788ac7fb:::benchmark_data_smithi043_119832_object41220:head# 2024-03-30T13:50:58.602 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.602 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:788afa30:::benchmark_data_smithi043_119832_object14718:head# 2024-03-30T13:50:58.602 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.602 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:788b1904:::benchmark_data_smithi043_119832_object65004:head# 2024-03-30T13:50:58.602 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.602 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:788b3c1c:::benchmark_data_smithi043_119832_object6073:head# 2024-03-30T13:50:58.602 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.602 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:788b4c27:::benchmark_data_smithi043_119832_object46986:head# 2024-03-30T13:50:58.602 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.602 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:788c77f9:::benchmark_data_smithi043_119832_object16144:head# 2024-03-30T13:50:58.603 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.603 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:788c9d69:::benchmark_data_smithi043_119832_object83983:head# 2024-03-30T13:50:58.603 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.603 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:788d1056:::benchmark_data_smithi043_119832_object21116:head# 2024-03-30T13:50:58.603 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.603 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:789088fc:::benchmark_data_smithi043_119832_object6928:head# 2024-03-30T13:50:58.603 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.603 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78929c87:::benchmark_data_smithi043_119832_object13031:head# 2024-03-30T13:50:58.603 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.603 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7894f32d:::benchmark_data_smithi043_119832_object38134:head# 2024-03-30T13:50:58.603 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.603 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7895f734:::benchmark_data_smithi043_119832_object70536:head# 2024-03-30T13:50:58.603 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.606 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7897711b:::benchmark_data_smithi043_119832_object71383:head# 2024-03-30T13:50:58.606 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.606 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78990106:::benchmark_data_smithi043_119832_object41318:head# 2024-03-30T13:50:58.606 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.607 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78999514:::benchmark_data_smithi043_119832_object81244:head# 2024-03-30T13:50:58.607 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.607 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:789acae4:::benchmark_data_smithi043_119832_object55498:head# 2024-03-30T13:50:58.607 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.607 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:789bc2fb:::benchmark_data_smithi043_119832_object76852:head# 2024-03-30T13:50:58.607 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.607 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:789cadc7:::benchmark_data_smithi043_119832_object60526:head# 2024-03-30T13:50:58.607 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.607 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:789cc65d:::benchmark_data_smithi043_119832_object23402:head# 2024-03-30T13:50:58.607 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.607 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:789ccacf:::benchmark_data_smithi043_119832_object25985:head# 2024-03-30T13:50:58.607 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.607 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:789d657a:::benchmark_data_smithi043_119832_object58528:head# 2024-03-30T13:50:58.608 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.608 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:789de0f5:::benchmark_data_smithi043_119832_object8540:head# 2024-03-30T13:50:58.608 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.608 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:789df816:::benchmark_data_smithi043_119832_object56981:head# 2024-03-30T13:50:58.608 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.608 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:789e17f6:::benchmark_data_smithi043_119832_object8246:head# 2024-03-30T13:50:58.608 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.608 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:789e6b99:::benchmark_data_smithi043_119832_object28768:head# 2024-03-30T13:50:58.608 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.611 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:789e87aa:::benchmark_data_smithi043_119832_object78422:head# 2024-03-30T13:50:58.611 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.611 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:789f858b:::benchmark_data_smithi043_119832_object4494:head# 2024-03-30T13:50:58.611 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.611 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:789ffd24:::benchmark_data_smithi043_119832_object62004:head# 2024-03-30T13:50:58.611 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.611 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78a14621:::benchmark_data_smithi043_119832_object69357:head# 2024-03-30T13:50:58.611 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.611 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78a15f67:::benchmark_data_smithi043_119832_object39113:head# 2024-03-30T13:50:58.612 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.612 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78a16460:::benchmark_data_smithi043_119832_object52510:head# 2024-03-30T13:50:58.612 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.612 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78a19334:::benchmark_data_smithi043_119832_object77505:head# 2024-03-30T13:50:58.612 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.612 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78a1a434:::benchmark_data_smithi043_119832_object21942:head# 2024-03-30T13:50:58.612 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.612 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78a1d31d:::benchmark_data_smithi043_119832_object20061:head# 2024-03-30T13:50:58.612 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.612 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78a33cc2:::benchmark_data_smithi043_119832_object51580:head# 2024-03-30T13:50:58.612 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.612 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78a4cc56:::benchmark_data_smithi043_119832_object5104:head# 2024-03-30T13:50:58.612 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.612 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78a56b0a:::benchmark_data_smithi043_119832_object78911:head# 2024-03-30T13:50:58.613 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.613 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78a63408:::benchmark_data_smithi043_119832_object46088:head# 2024-03-30T13:50:58.613 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.615 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78a66c8b:::benchmark_data_smithi043_119832_object66726:head# 2024-03-30T13:50:58.616 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.616 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78a708c7:::benchmark_data_smithi043_119832_object23051:head# 2024-03-30T13:50:58.616 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.616 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78a7e561:::benchmark_data_smithi043_119832_object30451:head# 2024-03-30T13:50:58.616 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.616 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78a84ca6:::benchmark_data_smithi043_119832_object31959:head# 2024-03-30T13:50:58.616 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.616 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78a89824:::benchmark_data_smithi043_119832_object19677:head# 2024-03-30T13:50:58.616 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.616 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78a95b68:::benchmark_data_smithi043_119832_object79587:head# 2024-03-30T13:50:58.616 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.616 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78aa49fb:::benchmark_data_smithi043_119832_object67898:head# 2024-03-30T13:50:58.616 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.617 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78aaa3df:::benchmark_data_smithi043_119832_object85549:head# 2024-03-30T13:50:58.617 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.617 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78ac8950:::benchmark_data_smithi043_119832_object17596:head# 2024-03-30T13:50:58.617 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.617 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78ad9bd4:::benchmark_data_smithi043_119832_object9662:head# 2024-03-30T13:50:58.617 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.617 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78adec18:::benchmark_data_smithi043_119832_object5817:head# 2024-03-30T13:50:58.617 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.617 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78ae745a:::benchmark_data_smithi043_119832_object25567:head# 2024-03-30T13:50:58.617 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.617 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78af2b44:::benchmark_data_smithi043_119832_object37229:head# 2024-03-30T13:50:58.617 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.620 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78afa2e1:::benchmark_data_smithi043_119832_object76206:head# 2024-03-30T13:50:58.620 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.620 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78afc2ba:::benchmark_data_smithi043_119832_object61630:head# 2024-03-30T13:50:58.620 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.620 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78b005bf:::benchmark_data_smithi043_119832_object82911:head# 2024-03-30T13:50:58.620 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.620 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78b1cb4b:::benchmark_data_smithi043_119832_object51769:head# 2024-03-30T13:50:58.620 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.620 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78b39ed6:::benchmark_data_smithi043_119832_object31648:head# 2024-03-30T13:50:58.621 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.621 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78b3ed9e:::benchmark_data_smithi043_119832_object75614:head# 2024-03-30T13:50:58.621 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.621 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78b46e23:::benchmark_data_smithi043_119832_object34763:head# 2024-03-30T13:50:58.621 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.621 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78b48685:::benchmark_data_smithi043_119832_object23939:head# 2024-03-30T13:50:58.621 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.621 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78b48fce:::benchmark_data_smithi043_119832_object42953:head# 2024-03-30T13:50:58.621 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.621 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78b4fa46:::benchmark_data_smithi043_119832_object5057:head# 2024-03-30T13:50:58.621 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.621 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78b50df1:::benchmark_data_smithi043_119832_object61623:head# 2024-03-30T13:50:58.621 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.622 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78b522a6:::benchmark_data_smithi043_119832_object47881:head# 2024-03-30T13:50:58.622 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.624 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78b579d7:::benchmark_data_smithi043_119832_object86357:head# 2024-03-30T13:50:58.624 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.624 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78b601d5:::benchmark_data_smithi043_119832_object67938:head# 2024-03-30T13:50:58.624 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.624 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78b7d91d:::benchmark_data_smithi043_119832_object27097:head# 2024-03-30T13:50:58.624 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.624 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78b7ef7c:::benchmark_data_smithi043_119832_object50085:head# 2024-03-30T13:50:58.624 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.624 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78b96502:::benchmark_data_smithi043_119832_object43114:head# 2024-03-30T13:50:58.625 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.625 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78baa9ad:::benchmark_data_smithi043_119832_object15018:head# 2024-03-30T13:50:58.625 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.625 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78bb296e:::benchmark_data_smithi043_119832_object86452:head# 2024-03-30T13:50:58.625 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.625 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78bbb036:::benchmark_data_smithi043_119832_object80088:head# 2024-03-30T13:50:58.625 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.625 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78bc00c1:::benchmark_data_smithi043_119832_object21557:head# 2024-03-30T13:50:58.625 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.625 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78bc5560:::benchmark_data_smithi043_119832_object17184:head# 2024-03-30T13:50:58.625 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.625 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78bcc878:::benchmark_data_smithi043_119832_object9179:head# 2024-03-30T13:50:58.625 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.628 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78bd10df:::benchmark_data_smithi043_119832_object40334:head# 2024-03-30T13:50:58.628 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.628 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78bd1ef2:::benchmark_data_smithi043_119832_object3657:head# 2024-03-30T13:50:58.628 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.628 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78bdd661:::benchmark_data_smithi043_119832_object76334:head# 2024-03-30T13:50:58.628 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.628 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78bde2a5:::benchmark_data_smithi043_119832_object58742:head# 2024-03-30T13:50:58.628 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.628 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78bde864:::benchmark_data_smithi043_119832_object68696:head# 2024-03-30T13:50:58.628 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.628 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78beafe1:::benchmark_data_smithi043_119832_object29620:head# 2024-03-30T13:50:58.628 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.629 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78bed774:::benchmark_data_smithi043_119832_object71800:head# 2024-03-30T13:50:58.629 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.629 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78c0abf9:::benchmark_data_smithi043_119832_object77445:head# 2024-03-30T13:50:58.629 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.629 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78c172d6:::benchmark_data_smithi043_119832_object16691:head# 2024-03-30T13:50:58.629 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.629 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78c1ca3d:::benchmark_data_smithi043_119832_object83833:head# 2024-03-30T13:50:58.629 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.629 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78c2364f:::benchmark_data_smithi043_119832_object85251:head# 2024-03-30T13:50:58.629 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.631 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78c2df48:::benchmark_data_smithi043_119832_object11939:head# 2024-03-30T13:50:58.631 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.631 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78c3997c:::benchmark_data_smithi043_119832_object48825:head# 2024-03-30T13:50:58.631 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.631 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78c3d76f:::benchmark_data_smithi043_119832_object44929:head# 2024-03-30T13:50:58.631 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.631 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78c568d1:::benchmark_data_smithi043_119832_object38549:head# 2024-03-30T13:50:58.632 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.632 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78c5fe71:::benchmark_data_smithi043_119832_object594:head# 2024-03-30T13:50:58.632 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.632 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78c61b76:::benchmark_data_smithi043_119832_object573:head# 2024-03-30T13:50:58.632 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.632 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78c64ec3:::benchmark_data_smithi043_119832_object31192:head# 2024-03-30T13:50:58.632 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.632 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78c702aa:::benchmark_data_smithi043_119832_object13620:head# 2024-03-30T13:50:58.632 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.632 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78c710d7:::benchmark_data_smithi043_119832_object14431:head# 2024-03-30T13:50:58.632 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.634 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78c78f2e:::benchmark_data_smithi043_119832_object8028:head# 2024-03-30T13:50:58.634 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.634 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78c989a6:::benchmark_data_smithi043_119832_object52630:head# 2024-03-30T13:50:58.634 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.635 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78c9b2b1:::benchmark_data_smithi043_119832_object26204:head# 2024-03-30T13:50:58.635 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.635 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78ca421f:::benchmark_data_smithi043_119832_object44206:head# 2024-03-30T13:50:58.635 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.635 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78cbddab:::benchmark_data_smithi043_119832_object42159:head# 2024-03-30T13:50:58.635 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.635 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78cc83d8:::benchmark_data_smithi043_119832_object85595:head# 2024-03-30T13:50:58.635 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.635 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78ce5c4d:::benchmark_data_smithi043_119832_object17691:head# 2024-03-30T13:50:58.635 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.635 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78cfc2d3:::benchmark_data_smithi043_119832_object61022:head# 2024-03-30T13:50:58.635 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.635 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78d0b525:::benchmark_data_smithi043_119832_object82742:head# 2024-03-30T13:50:58.635 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.637 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78d14659:::benchmark_data_smithi043_119832_object12641:head# 2024-03-30T13:50:58.637 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.637 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78d1ecb5:::benchmark_data_smithi043_119832_object239:head# 2024-03-30T13:50:58.637 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.637 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78d30ced:::benchmark_data_smithi043_119832_object60227:head# 2024-03-30T13:50:58.638 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.638 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78d33fac:::benchmark_data_smithi043_119832_object71313:head# 2024-03-30T13:50:58.638 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.638 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78d477ab:::benchmark_data_smithi043_119832_object32634:head# 2024-03-30T13:50:58.638 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.638 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78d54623:::benchmark_data_smithi043_119832_object77167:head# 2024-03-30T13:50:58.638 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.638 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78d57e18:::benchmark_data_smithi043_119832_object5959:head# 2024-03-30T13:50:58.638 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.638 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78d5ba25:::benchmark_data_smithi043_119832_object34958:head# 2024-03-30T13:50:58.638 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.640 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78d5ce8d:::benchmark_data_smithi043_119832_object14703:head# 2024-03-30T13:50:58.640 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.640 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78d64647:::benchmark_data_smithi043_119832_object59323:head# 2024-03-30T13:50:58.640 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.640 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78d74099:::benchmark_data_smithi043_119832_object658:head# 2024-03-30T13:50:58.640 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.640 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78d851b7:::benchmark_data_smithi043_119832_object70808:head# 2024-03-30T13:50:58.640 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.640 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78d8c31d:::benchmark_data_smithi043_119832_object36750:head# 2024-03-30T13:50:58.641 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.641 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78d97fec:::benchmark_data_smithi043_119832_object26871:head# 2024-03-30T13:50:58.641 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.641 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78da8577:::benchmark_data_smithi043_119832_object80967:head# 2024-03-30T13:50:58.641 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.642 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78dc02c1:::benchmark_data_smithi043_119832_object59966:head# 2024-03-30T13:50:58.643 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.643 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78dc7296:::benchmark_data_smithi043_119832_object58685:head# 2024-03-30T13:50:58.643 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.643 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78dc9631:::benchmark_data_smithi043_119832_object17401:head# 2024-03-30T13:50:58.643 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.643 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78dd479b:::benchmark_data_smithi043_119832_object28012:head# 2024-03-30T13:50:58.643 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.643 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78dd9f07:::benchmark_data_smithi043_119832_object70208:head# 2024-03-30T13:50:58.643 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.643 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78ddd439:::benchmark_data_smithi043_119832_object81650:head# 2024-03-30T13:50:58.643 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.643 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78de4670:::benchmark_data_smithi043_119832_object13446:head# 2024-03-30T13:50:58.643 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.644 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78de9a65:::benchmark_data_smithi043_119832_object74065:head# 2024-03-30T13:50:58.644 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.645 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78deada7:::benchmark_data_smithi043_119832_object65271:head# 2024-03-30T13:50:58.645 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.645 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78e229d6:::benchmark_data_smithi043_119832_object27954:head# 2024-03-30T13:50:58.645 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.645 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78e279da:::benchmark_data_smithi043_119832_object13052:head# 2024-03-30T13:50:58.646 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.646 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78e33aaf:::benchmark_data_smithi043_119832_object32071:head# 2024-03-30T13:50:58.646 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.646 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78e37fd4:::benchmark_data_smithi043_119832_object62963:head# 2024-03-30T13:50:58.646 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.646 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78e48fb2:::benchmark_data_smithi043_119832_object29349:head# 2024-03-30T13:50:58.646 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.646 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78e746a7:::benchmark_data_smithi043_119832_object61371:head# 2024-03-30T13:50:58.646 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.648 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78e81442:::benchmark_data_smithi043_119832_object3157:head# 2024-03-30T13:50:58.648 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.648 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78e9595b:::benchmark_data_smithi043_119832_object36975:head# 2024-03-30T13:50:58.648 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.648 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78ea1183:::benchmark_data_smithi043_119832_object31129:head# 2024-03-30T13:50:58.648 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.648 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78ea4238:::benchmark_data_smithi043_119832_object70463:head# 2024-03-30T13:50:58.649 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.649 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78eb1e59:::benchmark_data_smithi043_119832_object82450:head# 2024-03-30T13:50:58.649 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.649 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78eb2624:::benchmark_data_smithi043_119832_object62162:head# 2024-03-30T13:50:58.649 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.649 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78ec043e:::benchmark_data_smithi043_119832_object28811:head# 2024-03-30T13:50:58.649 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.649 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78ec8ffe:::benchmark_data_smithi043_119832_object31280:head# 2024-03-30T13:50:58.649 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.651 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78edab5b:::benchmark_data_smithi043_119832_object4220:head# 2024-03-30T13:50:58.651 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.651 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78edb02b:::benchmark_data_smithi043_119832_object11851:head# 2024-03-30T13:50:58.651 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.651 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78ee4c66:::benchmark_data_smithi043_119832_object22128:head# 2024-03-30T13:50:58.651 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.651 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78f13142:::benchmark_data_smithi043_119832_object39313:head# 2024-03-30T13:50:58.651 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.651 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78f14ca4:::benchmark_data_smithi043_119832_object10739:head# 2024-03-30T13:50:58.652 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.652 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78f17ad1:::benchmark_data_smithi043_119832_object69990:head# 2024-03-30T13:50:58.652 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.652 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78f1d378:::benchmark_data_smithi043_119832_object13660:head# 2024-03-30T13:50:58.652 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.652 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78f3f95b:::benchmark_data_smithi043_119832_object59947:head# 2024-03-30T13:50:58.652 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.653 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78f4612d:::benchmark_data_smithi043_119832_object34351:head# 2024-03-30T13:50:58.653 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.653 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78f58c6d:::benchmark_data_smithi043_119832_object18186:head# 2024-03-30T13:50:58.653 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.653 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78f5c5f6:::benchmark_data_smithi043_119832_object43489:head# 2024-03-30T13:50:58.653 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.654 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78f787ca:::benchmark_data_smithi043_119832_object3683:head# 2024-03-30T13:50:58.654 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:58 smithi043 ceph-mon[42099]: 6.18 scrub starts 2024-03-30T13:50:58.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:58 smithi043 ceph-mon[42099]: 6.18 scrub ok 2024-03-30T13:50:58.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:58 smithi043 ceph-mon[42099]: pgmap v329: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 858 op/s; 44241/354549 objects degraded (12.478%) 2024-03-30T13:50:58.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:58 smithi043 ceph-mon[42099]: 8.5 scrub ok 2024-03-30T13:50:58.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:58 smithi043 ceph-mon[42099]: 5.0 scrub starts 2024-03-30T13:50:58.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:58 smithi043 ceph-mon[42099]: 5.0 scrub ok 2024-03-30T13:50:58.654 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:58 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:50:58.655 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78f7e3b3:::benchmark_data_smithi043_119832_object604:head# 2024-03-30T13:50:58.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:58 smithi043 ceph-mon[29963]: 6.18 scrub starts 2024-03-30T13:50:58.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:58 smithi043 ceph-mon[29963]: 6.18 scrub ok 2024-03-30T13:50:58.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:58 smithi043 ceph-mon[29963]: pgmap v329: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 858 op/s; 44241/354549 objects degraded (12.478%) 2024-03-30T13:50:58.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:58 smithi043 ceph-mon[29963]: 8.5 scrub ok 2024-03-30T13:50:58.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:58 smithi043 ceph-mon[29963]: 5.0 scrub starts 2024-03-30T13:50:58.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:58 smithi043 ceph-mon[29963]: 5.0 scrub ok 2024-03-30T13:50:58.656 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:58 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:50:58.659 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.659 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78f86a66:::benchmark_data_smithi043_119832_object51130:head# 2024-03-30T13:50:58.659 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.659 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78f8883a:::benchmark_data_smithi043_119832_object66396:head# 2024-03-30T13:50:58.659 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.659 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78f9a75e:::benchmark_data_smithi043_119832_object39353:head# 2024-03-30T13:50:58.660 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.660 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78f9f279:::benchmark_data_smithi043_119832_object54042:head# 2024-03-30T13:50:58.660 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.660 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78f9fbaa:::benchmark_data_smithi043_119832_object49235:head# 2024-03-30T13:50:58.660 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.660 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78fa54cb:::benchmark_data_smithi043_119832_object7273:head# 2024-03-30T13:50:58.660 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.660 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78fac415:::benchmark_data_smithi043_119832_object31759:head# 2024-03-30T13:50:58.660 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.660 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78fb7354:::benchmark_data_smithi043_119832_object8169:head# 2024-03-30T13:50:58.660 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.660 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78fc8084:::benchmark_data_smithi043_119832_object25441:head# 2024-03-30T13:50:58.660 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.660 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78fd64f3:::benchmark_data_smithi043_119832_object21058:head# 2024-03-30T13:50:58.661 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.661 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78fdf767:::benchmark_data_smithi043_119832_object25902:head# 2024-03-30T13:50:58.661 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.661 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78fe4a80:::benchmark_data_smithi043_119832_object80149:head# 2024-03-30T13:50:58.661 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.661 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78fee2a6:::benchmark_data_smithi043_119832_object26197:head# 2024-03-30T13:50:58.661 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.661 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78fefdd3:::benchmark_data_smithi043_119832_object16284:head# 2024-03-30T13:50:58.661 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.661 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:78ff14b5:::benchmark_data_smithi043_119832_object29524:head# 2024-03-30T13:50:58.661 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.661 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:790152a5:::benchmark_data_smithi043_119832_object40719:head# 2024-03-30T13:50:58.661 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.662 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:790297f7:::benchmark_data_smithi043_119832_object50502:head# 2024-03-30T13:50:58.662 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.662 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7902b3be:::benchmark_data_smithi043_119832_object78137:head# 2024-03-30T13:50:58.662 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.665 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7902b56b:::benchmark_data_smithi043_119832_object5551:head# 2024-03-30T13:50:58.665 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.665 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79030e6c:::benchmark_data_smithi043_119832_object78569:head# 2024-03-30T13:50:58.665 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.665 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7903541a:::benchmark_data_smithi043_119832_object75768:head# 2024-03-30T13:50:58.665 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.665 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7903e2ad:::benchmark_data_smithi043_119832_object67831:head# 2024-03-30T13:50:58.665 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.665 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7904552c:::benchmark_data_smithi043_119832_object13642:head# 2024-03-30T13:50:58.666 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.666 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79054555:::benchmark_data_smithi043_119832_object40645:head# 2024-03-30T13:50:58.666 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.666 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79064181:::benchmark_data_smithi043_119832_object36266:head# 2024-03-30T13:50:58.666 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.666 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7907807c:::benchmark_data_smithi043_119832_object85877:head# 2024-03-30T13:50:58.666 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.666 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7908024e:::benchmark_data_smithi043_119832_object45267:head# 2024-03-30T13:50:58.666 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.666 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7909ad17:::benchmark_data_smithi043_119832_object82355:head# 2024-03-30T13:50:58.666 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.666 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:790a55aa:::benchmark_data_smithi043_119832_object85691:head# 2024-03-30T13:50:58.666 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.667 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:790b17de:::benchmark_data_smithi043_119832_object20981:head# 2024-03-30T13:50:58.667 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.667 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:790b6128:::benchmark_data_smithi043_119832_object52898:head# 2024-03-30T13:50:58.667 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.667 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:790bc3ee:::benchmark_data_smithi043_119832_object46940:head# 2024-03-30T13:50:58.667 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.667 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:790dc2b0:::benchmark_data_smithi043_119832_object82310:head# 2024-03-30T13:50:58.667 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.667 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:790e38e9:::benchmark_data_smithi043_119832_object68607:head# 2024-03-30T13:50:58.667 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.667 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:790e7301:::benchmark_data_smithi043_119832_object63163:head# 2024-03-30T13:50:58.667 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.670 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:790f6f4d:::benchmark_data_smithi043_119832_object69181:head# 2024-03-30T13:50:58.670 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.670 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:790fcbfe:::benchmark_data_smithi043_119832_object24745:head# 2024-03-30T13:50:58.670 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.670 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79107704:::benchmark_data_smithi043_119832_object39969:head# 2024-03-30T13:50:58.670 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.671 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79111c32:::benchmark_data_smithi043_119832_object25684:head# 2024-03-30T13:50:58.671 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.671 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7911a941:::benchmark_data_smithi043_119832_object88060:head# 2024-03-30T13:50:58.671 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.671 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7912a844:::benchmark_data_smithi043_119832_object18644:head# 2024-03-30T13:50:58.671 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.671 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7912c68e:::benchmark_data_smithi043_119832_object47953:head# 2024-03-30T13:50:58.671 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.671 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79149bd6:::benchmark_data_smithi043_119832_object60396:head# 2024-03-30T13:50:58.671 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.671 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7914e728:::benchmark_data_smithi043_119832_object87508:head# 2024-03-30T13:50:58.671 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.671 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7916a141:::benchmark_data_smithi043_119832_object70993:head# 2024-03-30T13:50:58.671 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.672 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79176bc6:::benchmark_data_smithi043_119832_object69345:head# 2024-03-30T13:50:58.672 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.672 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:791784c4:::benchmark_data_smithi043_119832_object15522:head# 2024-03-30T13:50:58.672 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.672 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79189cc3:::benchmark_data_smithi043_119832_object59231:head# 2024-03-30T13:50:58.672 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.672 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:791a4bfb:::benchmark_data_smithi043_119832_object55691:head# 2024-03-30T13:50:58.672 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.675 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:791b4736:::benchmark_data_smithi043_119832_object49826:head# 2024-03-30T13:50:58.675 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.675 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:791c04d4:::benchmark_data_smithi043_119832_object37785:head# 2024-03-30T13:50:58.675 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.675 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:791c18eb:::benchmark_data_smithi043_119832_object73950:head# 2024-03-30T13:50:58.675 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.675 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:791c6013:::benchmark_data_smithi043_119832_object41323:head# 2024-03-30T13:50:58.675 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.675 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:791cd740:::benchmark_data_smithi043_119832_object80297:head# 2024-03-30T13:50:58.675 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.675 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:791d6b49:::benchmark_data_smithi043_119832_object81387:head# 2024-03-30T13:50:58.675 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.676 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:791d7c82:::benchmark_data_smithi043_119832_object15189:head# 2024-03-30T13:50:58.676 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.676 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:791de215:::benchmark_data_smithi043_119832_object78431:head# 2024-03-30T13:50:58.676 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.676 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:791e9812:::benchmark_data_smithi043_119832_object53877:head# 2024-03-30T13:50:58.676 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.676 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:791f0314:::benchmark_data_smithi043_119832_object17253:head# 2024-03-30T13:50:58.676 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.676 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:791f8da0:::benchmark_data_smithi043_119832_object33921:head# 2024-03-30T13:50:58.676 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.676 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:791fc27e:::benchmark_data_smithi043_119832_object58736:head# 2024-03-30T13:50:58.676 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.676 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:791fc2d6:::benchmark_data_smithi043_119832_object48857:head# 2024-03-30T13:50:58.677 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.679 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79203572:::benchmark_data_smithi043_119832_object28604:head# 2024-03-30T13:50:58.679 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.679 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79207b6d:::benchmark_data_smithi043_119832_object5748:head# 2024-03-30T13:50:58.679 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.680 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7921ad29:::benchmark_data_smithi043_119832_object76359:head# 2024-03-30T13:50:58.680 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.680 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7923420b:::benchmark_data_smithi043_119832_object27441:head# 2024-03-30T13:50:58.680 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.680 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:792535f5:::benchmark_data_smithi043_119832_object71523:head# 2024-03-30T13:50:58.680 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.680 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7925af7d:::benchmark_data_smithi043_119832_object64468:head# 2024-03-30T13:50:58.680 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.680 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7928af38:::benchmark_data_smithi043_119832_object45824:head# 2024-03-30T13:50:58.680 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.680 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7928e87e:::benchmark_data_smithi043_119832_object19341:head# 2024-03-30T13:50:58.680 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.680 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7928f4db:::benchmark_data_smithi043_119832_object40027:head# 2024-03-30T13:50:58.680 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.681 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:792b08f9:::benchmark_data_smithi043_119832_object47415:head# 2024-03-30T13:50:58.681 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.681 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:792b1bee:::benchmark_data_smithi043_119832_object19538:head# 2024-03-30T13:50:58.681 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.681 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:792bf7c5:::benchmark_data_smithi043_119832_object76665:head# 2024-03-30T13:50:58.681 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.681 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:792ca53e:::benchmark_data_smithi043_119832_object49369:head# 2024-03-30T13:50:58.681 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.683 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:792cd1f0:::benchmark_data_smithi043_119832_object31834:head# 2024-03-30T13:50:58.683 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.684 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:792f4147:::benchmark_data_smithi043_119832_object67843:head# 2024-03-30T13:50:58.684 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.684 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:792f91f8:::benchmark_data_smithi043_119832_object69709:head# 2024-03-30T13:50:58.684 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.684 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7930850a:::benchmark_data_smithi043_119832_object65519:head# 2024-03-30T13:50:58.684 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.684 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79310a5b:::benchmark_data_smithi043_119832_object14079:head# 2024-03-30T13:50:58.684 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.684 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79315653:::benchmark_data_smithi043_119832_object50723:head# 2024-03-30T13:50:58.684 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.684 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79333820:::benchmark_data_smithi043_119832_object38323:head# 2024-03-30T13:50:58.684 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.684 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7936763f:::benchmark_data_smithi043_119832_object63310:head# 2024-03-30T13:50:58.684 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.685 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:793690b2:::benchmark_data_smithi043_119832_object10142:head# 2024-03-30T13:50:58.685 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.685 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7936e207:::benchmark_data_smithi043_119832_object14839:head# 2024-03-30T13:50:58.685 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.685 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79375f1a:::benchmark_data_smithi043_119832_object61899:head# 2024-03-30T13:50:58.685 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.687 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79392662:::benchmark_data_smithi043_119832_object3786:head# 2024-03-30T13:50:58.687 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.687 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7939ac2a:::benchmark_data_smithi043_119832_object78002:head# 2024-03-30T13:50:58.687 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.688 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:793a65e0:::benchmark_data_smithi043_119832_object22728:head# 2024-03-30T13:50:58.688 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.688 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:793ecde0:::benchmark_data_smithi043_119832_object26702:head# 2024-03-30T13:50:58.688 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.688 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:793ef0bf:::benchmark_data_smithi043_119832_object79568:head# 2024-03-30T13:50:58.688 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.688 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:793f1736:::benchmark_data_smithi043_119832_object51725:head# 2024-03-30T13:50:58.688 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.688 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:793fe3c9:::benchmark_data_smithi043_119832_object12276:head# 2024-03-30T13:50:58.688 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.688 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79402344:::benchmark_data_smithi043_119832_object44590:head# 2024-03-30T13:50:58.688 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.688 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79406c87:::benchmark_data_smithi043_119832_object22637:head# 2024-03-30T13:50:58.688 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.689 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7940e1f9:::benchmark_data_smithi043_119832_object41588:head# 2024-03-30T13:50:58.689 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.689 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79410546:::benchmark_data_smithi043_119832_object49538:head# 2024-03-30T13:50:58.689 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.691 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79418161:::benchmark_data_smithi043_119832_object87325:head# 2024-03-30T13:50:58.691 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.691 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79427fba:::benchmark_data_smithi043_119832_object64636:head# 2024-03-30T13:50:58.691 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.691 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7943cb3b:::benchmark_data_smithi043_119832_object45542:head# 2024-03-30T13:50:58.691 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.691 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7944124e:::benchmark_data_smithi043_119832_object63621:head# 2024-03-30T13:50:58.691 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.691 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79461fdd:::benchmark_data_smithi043_119832_object31818:head# 2024-03-30T13:50:58.691 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.691 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79474507:::benchmark_data_smithi043_119832_object53676:head# 2024-03-30T13:50:58.691 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.691 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7949628b:::benchmark_data_smithi043_119832_object33109:head# 2024-03-30T13:50:58.692 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.692 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:794afc53:::benchmark_data_smithi043_119832_object51824:head# 2024-03-30T13:50:58.692 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.692 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:794d39fd:::benchmark_data_smithi043_119832_object51074:head# 2024-03-30T13:50:58.692 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.693 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:794d5590:::benchmark_data_smithi043_119832_object61551:head# 2024-03-30T13:50:58.694 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.694 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:794e4424:::benchmark_data_smithi043_119832_object10617:head# 2024-03-30T13:50:58.694 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.694 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:794e6f9c:::benchmark_data_smithi043_119832_object55079:head# 2024-03-30T13:50:58.694 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.694 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:794fe4fd:::benchmark_data_smithi043_119832_object69995:head# 2024-03-30T13:50:58.694 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.694 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:795119d2:::benchmark_data_smithi043_119832_object42005:head# 2024-03-30T13:50:58.694 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.694 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7951e446:::benchmark_data_smithi043_119832_object50331:head# 2024-03-30T13:50:58.694 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.694 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79520719:::benchmark_data_smithi043_119832_object72721:head# 2024-03-30T13:50:58.694 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.695 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7952c37f:::benchmark_data_smithi043_119832_object21732:head# 2024-03-30T13:50:58.695 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.696 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7953ff0b:::benchmark_data_smithi043_119832_object52154:head# 2024-03-30T13:50:58.697 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.697 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:795502f9:::benchmark_data_smithi043_119832_object3281:head# 2024-03-30T13:50:58.697 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.697 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7955be91:::benchmark_data_smithi043_119832_object17173:head# 2024-03-30T13:50:58.697 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.697 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79579bb1:::benchmark_data_smithi043_119832_object21125:head# 2024-03-30T13:50:58.697 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.697 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79595383:::benchmark_data_smithi043_119832_object1057:head# 2024-03-30T13:50:58.697 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.697 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:795ad6f0:::benchmark_data_smithi043_119832_object44191:head# 2024-03-30T13:50:58.697 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.697 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:795b1914:::benchmark_data_smithi043_119832_object14272:head# 2024-03-30T13:50:58.697 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.697 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:795b26ca:::benchmark_data_smithi043_119832_object54660:head# 2024-03-30T13:50:58.697 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.698 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:50:58.700 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:795b6dd1:::benchmark_data_smithi043_119832_object48947:head# 2024-03-30T13:50:58.700 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.700 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:795bcace:::benchmark_data_smithi043_119832_object58565:head# 2024-03-30T13:50:58.700 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.700 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:795c4f18:::benchmark_data_smithi043_119832_object83204:head# 2024-03-30T13:50:58.701 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.701 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:795d04d5:::benchmark_data_smithi043_119832_object39267:head# 2024-03-30T13:50:58.701 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.701 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:795d9917:::benchmark_data_smithi043_119832_object53210:head# 2024-03-30T13:50:58.701 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.701 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:795fd6a5:::benchmark_data_smithi043_119832_object47278:head# 2024-03-30T13:50:58.701 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.701 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:796110ac:::benchmark_data_smithi043_119832_object42819:head# 2024-03-30T13:50:58.701 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.701 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:796251c6:::benchmark_data_smithi043_119832_object59228:head# 2024-03-30T13:50:58.701 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.701 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7963e69a:::benchmark_data_smithi043_119832_object31094:head# 2024-03-30T13:50:58.701 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.702 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79647947:::benchmark_data_smithi043_119832_object37470:head# 2024-03-30T13:50:58.702 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.702 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7965706f:::benchmark_data_smithi043_119832_object52676:head# 2024-03-30T13:50:58.702 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.705 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7965f15c:::benchmark_data_smithi043_119832_object29408:head# 2024-03-30T13:50:58.705 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.705 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79665fb4:::benchmark_data_smithi043_119832_object56199:head# 2024-03-30T13:50:58.705 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.705 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7967dc54:::benchmark_data_smithi043_119832_object80178:head# 2024-03-30T13:50:58.705 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.705 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7967e5a3:::benchmark_data_smithi043_119832_object57153:head# 2024-03-30T13:50:58.705 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.705 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:796aa4b8:::benchmark_data_smithi043_119832_object6595:head# 2024-03-30T13:50:58.705 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.705 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:796b1df4:::benchmark_data_smithi043_119832_object20247:head# 2024-03-30T13:50:58.706 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.706 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:796b2853:::benchmark_data_smithi043_119832_object34869:head# 2024-03-30T13:50:58.706 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.706 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:796b6f89:::benchmark_data_smithi043_119832_object1498:head# 2024-03-30T13:50:58.706 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.706 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:796cf235:::benchmark_data_smithi043_119832_object74654:head# 2024-03-30T13:50:58.706 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.706 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:796d1113:::benchmark_data_smithi043_119832_object35663:head# 2024-03-30T13:50:58.706 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.706 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:796eb1eb:::benchmark_data_smithi043_119832_object56831:head# 2024-03-30T13:50:58.706 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.706 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:796fb189:::benchmark_data_smithi043_119832_object47341:head# 2024-03-30T13:50:58.706 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.706 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:797091b7:::benchmark_data_smithi043_119832_object68552:head# 2024-03-30T13:50:58.707 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.710 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7971f192:::benchmark_data_smithi043_119832_object21163:head# 2024-03-30T13:50:58.710 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.710 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7972507d:::benchmark_data_smithi043_119832_object3222:head# 2024-03-30T13:50:58.710 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.710 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7972756e:::benchmark_data_smithi043_119832_object64100:head# 2024-03-30T13:50:58.711 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.711 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7972a8ce:::benchmark_data_smithi043_119832_object69918:head# 2024-03-30T13:50:58.711 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.711 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7973bde0:::benchmark_data_smithi043_119832_object65478:head# 2024-03-30T13:50:58.711 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.711 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7974d3ee:::benchmark_data_smithi043_119832_object68025:head# 2024-03-30T13:50:58.711 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.711 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7975b2d4:::benchmark_data_smithi043_119832_object67811:head# 2024-03-30T13:50:58.711 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.711 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79767781:::benchmark_data_smithi043_119832_object64926:head# 2024-03-30T13:50:58.711 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.711 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7977b6c9:::benchmark_data_smithi043_119832_object6727:head# 2024-03-30T13:50:58.711 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.712 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79783b32:::benchmark_data_smithi043_119832_object12412:head# 2024-03-30T13:50:58.712 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.712 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79794de0:::benchmark_data_smithi043_119832_object32033:head# 2024-03-30T13:50:58.712 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.712 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:797b0533:::benchmark_data_smithi043_119832_object61997:head# 2024-03-30T13:50:58.712 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.712 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:797c340e:::benchmark_data_smithi043_119832_object32611:head# 2024-03-30T13:50:58.712 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.712 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:797d70b8:::benchmark_data_smithi043_119832_object63799:head# 2024-03-30T13:50:58.712 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.712 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:797e26c7:::benchmark_data_smithi043_119832_object50389:head# 2024-03-30T13:50:58.712 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.715 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:797e4dfe:::benchmark_data_smithi043_119832_object67715:head# 2024-03-30T13:50:58.715 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.715 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:797e5cf8:::benchmark_data_smithi043_119832_object86399:head# 2024-03-30T13:50:58.715 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.715 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:797f3514:::benchmark_data_smithi043_119832_object29373:head# 2024-03-30T13:50:58.715 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.715 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:797f7654:::benchmark_data_smithi043_119832_object87531:head# 2024-03-30T13:50:58.715 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.715 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:797fabce:::benchmark_data_smithi043_119832_object66753:head# 2024-03-30T13:50:58.716 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.716 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:797ff94f:::benchmark_data_smithi043_119832_object25626:head# 2024-03-30T13:50:58.716 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.716 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7980e0d1:::benchmark_data_smithi043_119832_object21490:head# 2024-03-30T13:50:58.716 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.716 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79812263:::benchmark_data_smithi043_119832_object27874:head# 2024-03-30T13:50:58.716 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.716 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:798177fc:::benchmark_data_smithi043_119832_object73816:head# 2024-03-30T13:50:58.716 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.716 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79827505:::benchmark_data_smithi043_119832_object41408:head# 2024-03-30T13:50:58.716 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.716 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79831053:::benchmark_data_smithi043_119832_object6813:head# 2024-03-30T13:50:58.716 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.716 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7983c652:::benchmark_data_smithi043_119832_object15830:head# 2024-03-30T13:50:58.717 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.717 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7984a59b:::benchmark_data_smithi043_119832_object72216:head# 2024-03-30T13:50:58.717 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7984a92e:::benchmark_data_smithi043_119832_object25349:head# 2024-03-30T13:50:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79853eb4:::benchmark_data_smithi043_119832_object35172:head# 2024-03-30T13:50:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7986796a:::benchmark_data_smithi043_119832_object66016:head# 2024-03-30T13:50:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79878980:::benchmark_data_smithi043_119832_object26072:head# 2024-03-30T13:50:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7987ddc8:::benchmark_data_smithi043_119832_object23330:head# 2024-03-30T13:50:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79884ca7:::benchmark_data_smithi043_119832_object81405:head# 2024-03-30T13:50:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.720 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7989a75e:::benchmark_data_smithi043_119832_object65711:head# 2024-03-30T13:50:58.720 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.720 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:798a513e:::benchmark_data_smithi043_119832_object37040:head# 2024-03-30T13:50:58.720 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.720 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:798a9804:::benchmark_data_smithi043_119832_object3347:head# 2024-03-30T13:50:58.720 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.720 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:798ae180:::benchmark_data_smithi043_119832_object61308:head# 2024-03-30T13:50:58.720 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.722 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:798afdf1:::benchmark_data_smithi043_119832_object51447:head# 2024-03-30T13:50:58.722 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.722 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:798b3fbf:::benchmark_data_smithi043_119832_object86904:head# 2024-03-30T13:50:58.722 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.722 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:798b7eed:::benchmark_data_smithi043_119832_object48921:head# 2024-03-30T13:50:58.722 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.722 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:798bacc9:::benchmark_data_smithi043_119832_object20259:head# 2024-03-30T13:50:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:798d8f18:::benchmark_data_smithi043_119832_object88288:head# 2024-03-30T13:50:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:798e57e0:::benchmark_data_smithi043_119832_object14359:head# 2024-03-30T13:50:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:798e93f5:::benchmark_data_smithi043_119832_object69064:head# 2024-03-30T13:50:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:798e9b50:::benchmark_data_smithi043_119832_object17048:head# 2024-03-30T13:50:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:798f525d:::benchmark_data_smithi043_119832_object18429:head# 2024-03-30T13:50:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:798fd92b:::benchmark_data_smithi043_119832_object79503:head# 2024-03-30T13:50:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:798fdef2:::benchmark_data_smithi043_119832_object32861:head# 2024-03-30T13:50:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7990b1fc:::benchmark_data_smithi043_119832_object17042:head# 2024-03-30T13:50:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7991ec4e:::benchmark_data_smithi043_119832_object84941:head# 2024-03-30T13:50:58.726 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.726 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7991f661:::benchmark_data_smithi043_119832_object73876:head# 2024-03-30T13:50:58.726 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.726 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:799322d5:::benchmark_data_smithi043_119832_object6263:head# 2024-03-30T13:50:58.726 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.726 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7993a8bb:::benchmark_data_smithi043_119832_object28678:head# 2024-03-30T13:50:58.726 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.726 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79959058:::benchmark_data_smithi043_119832_object78746:head# 2024-03-30T13:50:58.726 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.726 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:799632e9:::benchmark_data_smithi043_119832_object16559:head# 2024-03-30T13:50:58.726 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.728 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79976655:::benchmark_data_smithi043_119832_object41583:head# 2024-03-30T13:50:58.728 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.728 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7998a2cc:::benchmark_data_smithi043_119832_object53460:head# 2024-03-30T13:50:58.728 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.728 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7998b50b:::benchmark_data_smithi043_119832_object38437:head# 2024-03-30T13:50:58.728 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.728 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:799922b5:::benchmark_data_smithi043_119832_object42781:head# 2024-03-30T13:50:58.728 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.728 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79994a46:::benchmark_data_smithi043_119832_object73105:head# 2024-03-30T13:50:58.728 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.728 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:799ad006:::benchmark_data_smithi043_119832_object1512:head# 2024-03-30T13:50:58.729 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.729 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:799bce63:::benchmark_data_smithi043_119832_object9228:head# 2024-03-30T13:50:58.729 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.730 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:799d4069:::benchmark_data_smithi043_119832_object22595:head# 2024-03-30T13:50:58.730 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.730 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:799e2255:::benchmark_data_smithi043_119832_object45234:head# 2024-03-30T13:50:58.730 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.730 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:799eb555:::benchmark_data_smithi043_119832_object66749:head# 2024-03-30T13:50:58.730 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.730 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:799eda35:::benchmark_data_smithi043_119832_object79866:head# 2024-03-30T13:50:58.730 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.730 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:799f4faa:::benchmark_data_smithi043_119832_object27435:head# 2024-03-30T13:50:58.730 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.730 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:799f7f1c:::benchmark_data_smithi043_119832_object11814:head# 2024-03-30T13:50:58.731 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.732 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:799f8717:::benchmark_data_smithi043_119832_object5711:head# 2024-03-30T13:50:58.732 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.733 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79a1c509:::benchmark_data_smithi043_119832_object5552:head# 2024-03-30T13:50:58.733 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.733 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79a2448d:::benchmark_data_smithi043_119832_object71286:head# 2024-03-30T13:50:58.733 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.733 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79a41be0:::benchmark_data_smithi043_119832_object13657:head# 2024-03-30T13:50:58.733 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.733 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79a463e6:::benchmark_data_smithi043_119832_object28988:head# 2024-03-30T13:50:58.733 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.733 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79a6df30:::benchmark_data_smithi043_119832_object39757:head# 2024-03-30T13:50:58.733 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.733 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79a8690c:::benchmark_data_smithi043_119832_object11758:head# 2024-03-30T13:50:58.733 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.733 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79a87058:::benchmark_data_smithi043_119832_object58682:head# 2024-03-30T13:50:58.733 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:58.735 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:58.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:50:58.737 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79a9a1f4:::benchmark_data_smithi043_119832_object69645:head# 2024-03-30T13:50:58.737 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.737 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79aab952:::benchmark_data_smithi043_119832_object41995:head# 2024-03-30T13:50:58.737 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.737 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79ab1892:::benchmark_data_smithi043_119832_object84449:head# 2024-03-30T13:50:58.737 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.737 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79abb8d1:::benchmark_data_smithi043_119832_object7739:head# 2024-03-30T13:50:58.738 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.738 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79ac1de2:::benchmark_data_smithi043_119832_object71137:head# 2024-03-30T13:50:58.738 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.738 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79ace356:::benchmark_data_smithi043_119832_object28105:head# 2024-03-30T13:50:58.738 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.738 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79adec33:::benchmark_data_smithi043_119832_object39046:head# 2024-03-30T13:50:58.738 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.738 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79af4c69:::benchmark_data_smithi043_119832_object21153:head# 2024-03-30T13:50:58.738 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.738 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79aff49a:::benchmark_data_smithi043_119832_object24307:head# 2024-03-30T13:50:58.738 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.738 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b07377:::benchmark_data_smithi043_119832_object28288:head# 2024-03-30T13:50:58.738 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.738 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b07902:::benchmark_data_smithi043_119832_object81611:head# 2024-03-30T13:50:58.739 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.739 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b0e3e3:::benchmark_data_smithi043_119832_object83468:head# 2024-03-30T13:50:58.739 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.739 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b154d3:::benchmark_data_smithi043_119832_object75961:head# 2024-03-30T13:50:58.739 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.741 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b176db:::benchmark_data_smithi043_119832_object48575:head# 2024-03-30T13:50:58.741 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.741 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b28b50:::benchmark_data_smithi043_119832_object48835:head# 2024-03-30T13:50:58.741 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.741 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b317df:::benchmark_data_smithi043_119832_object49085:head# 2024-03-30T13:50:58.741 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.741 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b31dd5:::benchmark_data_smithi043_119832_object27252:head# 2024-03-30T13:50:58.741 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.741 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b38fb7:::benchmark_data_smithi043_119832_object10324:head# 2024-03-30T13:50:58.741 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.741 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b39743:::benchmark_data_smithi043_119832_object20428:head# 2024-03-30T13:50:58.742 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.742 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b3a1ab:::benchmark_data_smithi043_119832_object4891:head# 2024-03-30T13:50:58.742 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.742 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b4141b:::benchmark_data_smithi043_119832_object54503:head# 2024-03-30T13:50:58.742 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.742 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b46c6d:::benchmark_data_smithi043_119832_object73979:head# 2024-03-30T13:50:58.742 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.742 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b47b2f:::benchmark_data_smithi043_119832_object86197:head# 2024-03-30T13:50:58.742 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.744 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b693df:::benchmark_data_smithi043_119832_object23697:head# 2024-03-30T13:50:58.744 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.744 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b6c102:::benchmark_data_smithi043_119832_object46615:head# 2024-03-30T13:50:58.744 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.744 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b7d353:::benchmark_data_smithi043_119832_object85520:head# 2024-03-30T13:50:58.744 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.744 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b82ad1:::benchmark_data_smithi043_119832_object51821:head# 2024-03-30T13:50:58.744 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.744 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b8b770:::benchmark_data_smithi043_119832_object35065:head# 2024-03-30T13:50:58.744 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.744 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b90e2c:::benchmark_data_smithi043_119832_object75394:head# 2024-03-30T13:50:58.744 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.745 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79b9190f:::benchmark_data_smithi043_119832_object16713:head# 2024-03-30T13:50:58.745 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.746 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79ba2487:::benchmark_data_smithi043_119832_object11496:head# 2024-03-30T13:50:58.746 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.746 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79ba3cc4:::benchmark_data_smithi043_119832_object48341:head# 2024-03-30T13:50:58.746 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.746 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79ba8cb1:::benchmark_data_smithi043_119832_object41205:head# 2024-03-30T13:50:58.746 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.746 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79bd0268:::benchmark_data_smithi043_119832_object11854:head# 2024-03-30T13:50:58.746 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.747 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79bfb46d:::benchmark_data_smithi043_119832_object61491:head# 2024-03-30T13:50:58.747 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.747 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79c01479:::benchmark_data_smithi043_119832_object37424:head# 2024-03-30T13:50:58.747 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.748 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79c0c925:::benchmark_data_smithi043_119832_object74181:head# 2024-03-30T13:50:58.748 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.748 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79c2cb90:::benchmark_data_smithi043_119832_object28956:head# 2024-03-30T13:50:58.748 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.748 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79c3a5df:::benchmark_data_smithi043_119832_object75266:head# 2024-03-30T13:50:58.748 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.749 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79c4055c:::benchmark_data_smithi043_119832_object59366:head# 2024-03-30T13:50:58.749 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.749 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79c57982:::benchmark_data_smithi043_119832_object85863:head# 2024-03-30T13:50:58.749 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.749 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79c81244:::benchmark_data_smithi043_119832_object13774:head# 2024-03-30T13:50:58.749 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.750 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79c86f8e:::benchmark_data_smithi043_119832_object22917:head# 2024-03-30T13:50:58.751 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.751 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79c98f5e:::benchmark_data_smithi043_119832_object22347:head# 2024-03-30T13:50:58.751 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.751 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79cafb93:::benchmark_data_smithi043_119832_object82323:head# 2024-03-30T13:50:58.751 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.751 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79cb5484:::benchmark_data_smithi043_119832_object83999:head# 2024-03-30T13:50:58.751 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.751 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79cb6219:::benchmark_data_smithi043_119832_object53478:head# 2024-03-30T13:50:58.751 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.751 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79cd4995:::benchmark_data_smithi043_119832_object76770:head# 2024-03-30T13:50:58.751 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.751 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79cd8346:::benchmark_data_smithi043_119832_object19549:head# 2024-03-30T13:50:58.752 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.753 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79ceb312:::benchmark_data_smithi043_119832_object58324:head# 2024-03-30T13:50:58.753 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.753 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79ceeb05:::benchmark_data_smithi043_119832_object80979:head# 2024-03-30T13:50:58.753 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.753 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79cfa701:::benchmark_data_smithi043_119832_object76270:head# 2024-03-30T13:50:58.753 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.753 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79d08755:::benchmark_data_smithi043_119832_object4260:head# 2024-03-30T13:50:58.753 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.753 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79d1145e:::benchmark_data_smithi043_119832_object57603:head# 2024-03-30T13:50:58.753 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.754 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79d1f7e9:::benchmark_data_smithi043_119832_object37879:head# 2024-03-30T13:50:58.754 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.755 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79d242a6:::benchmark_data_smithi043_119832_object56629:head# 2024-03-30T13:50:58.755 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.755 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79d2a217:::benchmark_data_smithi043_119832_object33962:head# 2024-03-30T13:50:58.755 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.755 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79d2af99:::benchmark_data_smithi043_119832_object72991:head# 2024-03-30T13:50:58.755 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.756 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79d407ba:::benchmark_data_smithi043_119832_object22568:head# 2024-03-30T13:50:58.756 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.756 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79d5598f:::benchmark_data_smithi043_119832_object43076:head# 2024-03-30T13:50:58.756 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.756 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79d5d485:::benchmark_data_smithi043_119832_object71647:head# 2024-03-30T13:50:58.756 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.757 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79d5d86a:::benchmark_data_smithi043_119832_object13221:head# 2024-03-30T13:50:58.757 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.757 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79d5ee81:::benchmark_data_smithi043_119832_object2136:head# 2024-03-30T13:50:58.758 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.758 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79d63443:::benchmark_data_smithi043_119832_object17862:head# 2024-03-30T13:50:58.758 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.758 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79d69e42:::benchmark_data_smithi043_119832_object83841:head# 2024-03-30T13:50:58.758 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.758 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79d6a844:::benchmark_data_smithi043_119832_object16292:head# 2024-03-30T13:50:58.758 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.758 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79d6ad96:::benchmark_data_smithi043_119832_object88214:head# 2024-03-30T13:50:58.758 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.760 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79d7f9a0:::benchmark_data_smithi043_119832_object53872:head# 2024-03-30T13:50:58.760 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.760 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79d80d8f:::benchmark_data_smithi043_119832_object12973:head# 2024-03-30T13:50:58.760 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.760 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79da5f02:::benchmark_data_smithi043_119832_object41269:head# 2024-03-30T13:50:58.760 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.760 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79db358c:::benchmark_data_smithi043_119832_object72849:head# 2024-03-30T13:50:58.760 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.760 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79db3688:::benchmark_data_smithi043_119832_object15923:head# 2024-03-30T13:50:58.760 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.760 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79db9fb1:::benchmark_data_smithi043_119832_object10972:head# 2024-03-30T13:50:58.760 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.761 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79dc2950:::benchmark_data_smithi043_119832_object10471:head# 2024-03-30T13:50:58.761 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.762 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79dce77d:::benchmark_data_smithi043_119832_object55523:head# 2024-03-30T13:50:58.762 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.762 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79dd3cd7:::benchmark_data_smithi043_119832_object78317:head# 2024-03-30T13:50:58.762 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.762 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79dd55b7:::benchmark_data_smithi043_119832_object35612:head# 2024-03-30T13:50:58.763 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.763 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79ddefa0:::benchmark_data_smithi043_119832_object48350:head# 2024-03-30T13:50:58.763 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.763 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79de15c9:::benchmark_data_smithi043_119832_object41226:head# 2024-03-30T13:50:58.763 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.763 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79e17ec0:::benchmark_data_smithi043_119832_object5395:head# 2024-03-30T13:50:58.763 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.763 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79e33749:::benchmark_data_smithi043_119832_object78612:head# 2024-03-30T13:50:58.763 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.764 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79e45be2:::benchmark_data_smithi043_119832_object71067:head# 2024-03-30T13:50:58.765 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.765 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79e536ed:::benchmark_data_smithi043_119832_object78092:head# 2024-03-30T13:50:58.765 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.765 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79e7466a:::benchmark_data_smithi043_119832_object59473:head# 2024-03-30T13:50:58.765 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.765 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79e774c2:::benchmark_data_smithi043_119832_object57223:head# 2024-03-30T13:50:58.765 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.765 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79e87f83:::benchmark_data_smithi043_119832_object80709:head# 2024-03-30T13:50:58.765 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.765 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79eb1ec3:::benchmark_data_smithi043_119832_object43635:head# 2024-03-30T13:50:58.765 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.765 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79eb9995:::benchmark_data_smithi043_119832_object72526:head# 2024-03-30T13:50:58.765 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.767 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79ebc027:::benchmark_data_smithi043_119832_object58181:head# 2024-03-30T13:50:58.767 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.767 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79ebf917:::benchmark_data_smithi043_119832_object41311:head# 2024-03-30T13:50:58.767 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.767 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79ecc080:::benchmark_data_smithi043_119832_object84056:head# 2024-03-30T13:50:58.767 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.767 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79ece052:::benchmark_data_smithi043_119832_object33112:head# 2024-03-30T13:50:58.767 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.767 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79ed6765:::benchmark_data_smithi043_119832_object25402:head# 2024-03-30T13:50:58.767 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.767 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79ed70c2:::benchmark_data_smithi043_119832_object6947:head# 2024-03-30T13:50:58.768 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.769 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79ed852d:::benchmark_data_smithi043_119832_object12648:head# 2024-03-30T13:50:58.769 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.769 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79f06c4a:::benchmark_data_smithi043_119832_object23253:head# 2024-03-30T13:50:58.769 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.769 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79f08fdc:::benchmark_data_smithi043_119832_object11454:head# 2024-03-30T13:50:58.769 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.769 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79f0ce0a:::benchmark_data_smithi043_119832_object8061:head# 2024-03-30T13:50:58.769 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.769 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79f29841:::benchmark_data_smithi043_119832_object12317:head# 2024-03-30T13:50:58.769 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.770 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79f2a79b:::benchmark_data_smithi043_119832_object76425:head# 2024-03-30T13:50:58.770 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.771 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79f55294:::benchmark_data_smithi043_119832_object3894:head# 2024-03-30T13:50:58.771 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.771 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79f65405:::benchmark_data_smithi043_119832_object22593:head# 2024-03-30T13:50:58.771 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.771 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79f75ebc:::benchmark_data_smithi043_119832_object19502:head# 2024-03-30T13:50:58.771 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.771 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79f7f7a8:::benchmark_data_smithi043_119832_object64619:head# 2024-03-30T13:50:58.771 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.771 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79f86fcb:::benchmark_data_smithi043_119832_object86376:head# 2024-03-30T13:50:58.771 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.773 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79f8f62b:::benchmark_data_smithi043_119832_object29136:head# 2024-03-30T13:50:58.773 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.773 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79f91eb5:::benchmark_data_smithi043_119832_object64283:head# 2024-03-30T13:50:58.773 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.773 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79f980e5:::benchmark_data_smithi043_119832_object5902:head# 2024-03-30T13:50:58.773 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.773 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79f9b64a:::benchmark_data_smithi043_119832_object28670:head# 2024-03-30T13:50:58.773 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.773 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79f9f170:::benchmark_data_smithi043_119832_object18773:head# 2024-03-30T13:50:58.773 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.773 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79fe18ec:::benchmark_data_smithi043_119832_object36576:head# 2024-03-30T13:50:58.773 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.775 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79ff25fd:::benchmark_data_smithi043_119832_object36113:head# 2024-03-30T13:50:58.775 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.775 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:79ffec33:::benchmark_data_smithi043_119832_object70610:head# 2024-03-30T13:50:58.775 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.775 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a005fca:::benchmark_data_smithi043_119832_object43002:head# 2024-03-30T13:50:58.775 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.775 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a0246da:::benchmark_data_smithi043_119832_object79920:head# 2024-03-30T13:50:58.775 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.775 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a03c68a:::benchmark_data_smithi043_119832_object14672:head# 2024-03-30T13:50:58.775 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.776 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a041218:::benchmark_data_smithi043_119832_object21880:head# 2024-03-30T13:50:58.777 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.777 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a045804:::benchmark_data_smithi043_119832_object80691:head# 2024-03-30T13:50:58.777 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.777 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a04d17f:::benchmark_data_smithi043_119832_object31723:head# 2024-03-30T13:50:58.777 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.777 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a05986d:::benchmark_data_smithi043_119832_object45904:head# 2024-03-30T13:50:58.777 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.777 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a064428:::benchmark_data_smithi043_119832_object39733:head# 2024-03-30T13:50:58.777 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.777 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a07325f:::benchmark_data_smithi043_119832_object54753:head# 2024-03-30T13:50:58.777 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.779 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a0772a4:::benchmark_data_smithi043_119832_object57481:head# 2024-03-30T13:50:58.779 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.779 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a07755a:::benchmark_data_smithi043_119832_object52480:head# 2024-03-30T13:50:58.779 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.779 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a080357:::benchmark_data_smithi043_119832_object71254:head# 2024-03-30T13:50:58.779 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.779 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a082800:::benchmark_data_smithi043_119832_object57825:head# 2024-03-30T13:50:58.779 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.779 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a08bf7e:::benchmark_data_smithi043_119832_object1642:head# 2024-03-30T13:50:58.780 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.780 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a0904cf:::benchmark_data_smithi043_119832_object15993:head# 2024-03-30T13:50:58.780 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.780 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a09547a:::benchmark_data_smithi043_119832_object2616:head# 2024-03-30T13:50:58.780 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.781 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a09e954:::benchmark_data_smithi043_119832_object24246:head# 2024-03-30T13:50:58.781 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.781 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a0bc4ac:::benchmark_data_smithi043_119832_object64706:head# 2024-03-30T13:50:58.782 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.782 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a0dcac1:::benchmark_data_smithi043_119832_object40815:head# 2024-03-30T13:50:58.782 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.782 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a0ded86:::benchmark_data_smithi043_119832_object10060:head# 2024-03-30T13:50:58.782 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.782 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a0e9a27:::benchmark_data_smithi043_119832_object84861:head# 2024-03-30T13:50:58.782 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.782 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a0edf52:::benchmark_data_smithi043_119832_object69667:head# 2024-03-30T13:50:58.782 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.782 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a0fdead:::benchmark_data_smithi043_119832_object77944:head# 2024-03-30T13:50:58.782 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.783 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a1081c2:::benchmark_data_smithi043_119832_object32988:head# 2024-03-30T13:50:58.783 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.783 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a108eec:::benchmark_data_smithi043_119832_object28943:head# 2024-03-30T13:50:58.784 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.784 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a11e170:::benchmark_data_smithi043_119832_object10920:head# 2024-03-30T13:50:58.784 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.784 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a121598:::benchmark_data_smithi043_119832_object43873:head# 2024-03-30T13:50:58.784 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.784 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a12bcf6:::benchmark_data_smithi043_119832_object57090:head# 2024-03-30T13:50:58.784 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.785 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a15464b:::benchmark_data_smithi043_119832_object36390:head# 2024-03-30T13:50:58.785 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.785 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a162abe:::benchmark_data_smithi043_119832_object13376:head# 2024-03-30T13:50:58.785 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.785 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a169c1a:::benchmark_data_smithi043_119832_object58499:head# 2024-03-30T13:50:58.785 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.785 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a17ec06:::benchmark_data_smithi043_119832_object29926:head# 2024-03-30T13:50:58.786 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.786 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a17f582:::benchmark_data_smithi043_119832_object32776:head# 2024-03-30T13:50:58.786 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.787 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a181156:::benchmark_data_smithi043_119832_object78793:head# 2024-03-30T13:50:58.787 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.787 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a18af0e:::benchmark_data_smithi043_119832_object25453:head# 2024-03-30T13:50:58.787 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.787 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a1975aa:::benchmark_data_smithi043_119832_object76092:head# 2024-03-30T13:50:58.787 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.788 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a1b705d:::benchmark_data_smithi043_119832_object59975:head# 2024-03-30T13:50:58.788 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.788 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a1c258f:::benchmark_data_smithi043_119832_object21944:head# 2024-03-30T13:50:58.788 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.788 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a1c5803:::benchmark_data_smithi043_119832_object66109:head# 2024-03-30T13:50:58.788 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.789 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a1df009:::benchmark_data_smithi043_119832_object28477:head# 2024-03-30T13:50:58.789 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.789 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a1e5de1:::benchmark_data_smithi043_119832_object46801:head# 2024-03-30T13:50:58.789 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.789 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a1ed7a7:::benchmark_data_smithi043_119832_object45843:head# 2024-03-30T13:50:58.790 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.790 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a1edb9b:::benchmark_data_smithi043_119832_object47178:head# 2024-03-30T13:50:58.790 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.790 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a1f16e2:::benchmark_data_smithi043_119832_object59505:head# 2024-03-30T13:50:58.790 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.790 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a219b67:::benchmark_data_smithi043_119832_object52449:head# 2024-03-30T13:50:58.790 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.791 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a21c92a:::benchmark_data_smithi043_119832_object69964:head# 2024-03-30T13:50:58.791 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.791 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a228aa3:::benchmark_data_smithi043_119832_object81723:head# 2024-03-30T13:50:58.791 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.792 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a23e41e:::benchmark_data_smithi043_119832_object77033:head# 2024-03-30T13:50:58.792 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.792 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a243857:::benchmark_data_smithi043_119832_object53978:head# 2024-03-30T13:50:58.792 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.792 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a25bd1b:::benchmark_data_smithi043_119832_object72994:head# 2024-03-30T13:50:58.792 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.792 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a2730ad:::benchmark_data_smithi043_119832_object18245:head# 2024-03-30T13:50:58.792 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.793 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a28cd62:::benchmark_data_smithi043_119832_object2575:head# 2024-03-30T13:50:58.793 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.794 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a2bdb70:::benchmark_data_smithi043_119832_object9614:head# 2024-03-30T13:50:58.794 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.794 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a2d14ea:::benchmark_data_smithi043_119832_object29228:head# 2024-03-30T13:50:58.794 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.794 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a2dc1c7:::benchmark_data_smithi043_119832_object6716:head# 2024-03-30T13:50:58.794 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.794 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a2e0188:::benchmark_data_smithi043_119832_object87554:head# 2024-03-30T13:50:58.794 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.794 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a2e1362:::benchmark_data_smithi043_119832_object37902:head# 2024-03-30T13:50:58.794 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.795 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a2f8255:::benchmark_data_smithi043_119832_object50874:head# 2024-03-30T13:50:58.796 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.796 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a30428b:::benchmark_data_smithi043_119832_object43806:head# 2024-03-30T13:50:58.796 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.796 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a304fec:::benchmark_data_smithi043_119832_object19040:head# 2024-03-30T13:50:58.796 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.796 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a309050:::benchmark_data_smithi043_119832_object77287:head# 2024-03-30T13:50:58.796 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.796 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a30ecd0:::benchmark_data_smithi043_119832_object78288:head# 2024-03-30T13:50:58.796 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.796 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a30f453:::benchmark_data_smithi043_119832_object37110:head# 2024-03-30T13:50:58.796 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.797 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a315984:::benchmark_data_smithi043_119832_object54087:head# 2024-03-30T13:50:58.798 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.798 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a32b2d7:::benchmark_data_smithi043_119832_object28767:head# 2024-03-30T13:50:58.798 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.798 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a330de5:::benchmark_data_smithi043_119832_object23152:head# 2024-03-30T13:50:58.798 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.798 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a341283:::benchmark_data_smithi043_119832_object79442:head# 2024-03-30T13:50:58.798 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.798 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a3481e4:::benchmark_data_smithi043_119832_object69828:head# 2024-03-30T13:50:58.798 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.799 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a34a4ad:::benchmark_data_smithi043_119832_object73332:head# 2024-03-30T13:50:58.799 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.799 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a35943d:::benchmark_data_smithi043_119832_object31671:head# 2024-03-30T13:50:58.799 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.799 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a3667ce:::benchmark_data_smithi043_119832_object67614:head# 2024-03-30T13:50:58.800 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.800 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a368271:::benchmark_data_smithi043_119832_object2088:head# 2024-03-30T13:50:58.800 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.800 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a37502c:::benchmark_data_smithi043_119832_object88459:head# 2024-03-30T13:50:58.800 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.801 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a378335:::benchmark_data_smithi043_119832_object12008:head# 2024-03-30T13:50:58.801 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.801 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a37bae1:::benchmark_data_smithi043_119832_object79595:head# 2024-03-30T13:50:58.801 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.801 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a3a1feb:::benchmark_data_smithi043_119832_object49054:head# 2024-03-30T13:50:58.801 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.801 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a3b51b0:::benchmark_data_smithi043_119832_object10420:head# 2024-03-30T13:50:58.801 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.802 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a3d6cba:::benchmark_data_smithi043_119832_object76512:head# 2024-03-30T13:50:58.803 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.803 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a3da9ea:::benchmark_data_smithi043_119832_object50694:head# 2024-03-30T13:50:58.803 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.803 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a3de33b:::benchmark_data_smithi043_119832_object43106:head# 2024-03-30T13:50:58.803 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.803 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a3e36f5:::benchmark_data_smithi043_119832_object78048:head# 2024-03-30T13:50:58.803 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.803 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a4129af:::benchmark_data_smithi043_119832_object62024:head# 2024-03-30T13:50:58.803 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.804 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a432965:::benchmark_data_smithi043_119832_object60051:head# 2024-03-30T13:50:58.804 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.804 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a437b7a:::benchmark_data_smithi043_119832_object40423:head# 2024-03-30T13:50:58.804 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.804 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a43d132:::benchmark_data_smithi043_119832_object11070:head# 2024-03-30T13:50:58.804 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.805 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a43d3a8:::benchmark_data_smithi043_119832_object71223:head# 2024-03-30T13:50:58.805 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.806 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a441b3a:::benchmark_data_smithi043_119832_object54462:head# 2024-03-30T13:50:58.806 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.806 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a44efbc:::benchmark_data_smithi043_119832_object54367:head# 2024-03-30T13:50:58.806 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.806 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a46ab46:::benchmark_data_smithi043_119832_object66742:head# 2024-03-30T13:50:58.806 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.806 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a471621:::benchmark_data_smithi043_119832_object1730:head# 2024-03-30T13:50:58.806 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.807 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a47546d:::benchmark_data_smithi043_119832_object42288:head# 2024-03-30T13:50:58.807 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.807 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a477aee:::benchmark_data_smithi043_119832_object64698:head# 2024-03-30T13:50:58.807 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.807 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a4998aa:::benchmark_data_smithi043_119832_object77613:head# 2024-03-30T13:50:58.807 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.807 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a49fe85:::benchmark_data_smithi043_119832_object36550:head# 2024-03-30T13:50:58.808 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.808 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a4a05e1:::benchmark_data_smithi043_119832_object50791:head# 2024-03-30T13:50:58.808 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.808 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a4a48c6:::benchmark_data_smithi043_119832_object4762:head# 2024-03-30T13:50:58.808 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.809 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a4b26d3:::benchmark_data_smithi043_119832_object40679:head# 2024-03-30T13:50:58.809 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.809 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a4c1162:::benchmark_data_smithi043_119832_object13726:head# 2024-03-30T13:50:58.809 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.809 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a4ccfd6:::benchmark_data_smithi043_119832_object49434:head# 2024-03-30T13:50:58.809 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.810 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a4d91f9:::benchmark_data_smithi043_119832_object44262:head# 2024-03-30T13:50:58.810 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.810 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a4e4258:::benchmark_data_smithi043_119832_object10347:head# 2024-03-30T13:50:58.810 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.810 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a4f5d2b:::benchmark_data_smithi043_119832_object43160:head# 2024-03-30T13:50:58.810 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.811 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a4fd0c2:::benchmark_data_smithi043_119832_object55103:head# 2024-03-30T13:50:58.811 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.811 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a515ac9:::benchmark_data_smithi043_119832_object38815:head# 2024-03-30T13:50:58.811 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.811 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a53106f:::benchmark_data_smithi043_119832_object17019:head# 2024-03-30T13:50:58.811 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.812 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a537cc4:::benchmark_data_smithi043_119832_object57625:head# 2024-03-30T13:50:58.812 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.812 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a545992:::benchmark_data_smithi043_119832_object3617:head# 2024-03-30T13:50:58.812 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.812 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a545a4e:::benchmark_data_smithi043_119832_object45881:head# 2024-03-30T13:50:58.812 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.813 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a552677:::benchmark_data_smithi043_119832_object52495:head# 2024-03-30T13:50:58.813 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.813 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a555c0c:::benchmark_data_smithi043_119832_object26339:head# 2024-03-30T13:50:58.813 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.814 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a56a135:::benchmark_data_smithi043_119832_object30224:head# 2024-03-30T13:50:58.814 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.815 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a572b14:::benchmark_data_smithi043_119832_object81719:head# 2024-03-30T13:50:58.815 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.815 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a5792f4:::benchmark_data_smithi043_119832_object52087:head# 2024-03-30T13:50:58.815 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.815 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a57fbb7:::benchmark_data_smithi043_119832_object77230:head# 2024-03-30T13:50:58.815 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.815 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a583160:::benchmark_data_smithi043_119832_object74649:head# 2024-03-30T13:50:58.815 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.816 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a586d87:::benchmark_data_smithi043_119832_object52987:head# 2024-03-30T13:50:58.816 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.816 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a59c63a:::benchmark_data_smithi043_119832_object38145:head# 2024-03-30T13:50:58.816 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.816 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a59e5d6:::benchmark_data_smithi043_119832_object64184:head# 2024-03-30T13:50:58.816 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.816 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a5b4164:::benchmark_data_smithi043_119832_object59169:head# 2024-03-30T13:50:58.816 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.817 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a5d7be0:::benchmark_data_smithi043_119832_object1788:head# 2024-03-30T13:50:58.817 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.817 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a5d823e:::benchmark_data_smithi043_119832_object41546:head# 2024-03-30T13:50:58.817 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.817 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a5d85de:::benchmark_data_smithi043_119832_object65261:head# 2024-03-30T13:50:58.818 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.818 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a5d9223:::benchmark_data_smithi043_119832_object36024:head# 2024-03-30T13:50:58.818 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.818 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a5f6be4:::benchmark_data_smithi043_119832_object12300:head# 2024-03-30T13:50:58.819 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.819 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a602179:::benchmark_data_smithi043_119832_object37349:head# 2024-03-30T13:50:58.819 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.819 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a617982:::benchmark_data_smithi043_119832_object72180:head# 2024-03-30T13:50:58.819 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.819 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a61c241:::benchmark_data_smithi043_119832_object46465:head# 2024-03-30T13:50:58.819 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.820 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a6265c3:::benchmark_data_smithi043_119832_object64307:head# 2024-03-30T13:50:58.820 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.820 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a646406:::benchmark_data_smithi043_119832_object14641:head# 2024-03-30T13:50:58.820 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.820 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a6471dd:::benchmark_data_smithi043_119832_object65792:head# 2024-03-30T13:50:58.821 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.821 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a650153:::benchmark_data_smithi043_119832_object11312:head# 2024-03-30T13:50:58.821 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.821 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a65147b:::benchmark_data_smithi043_119832_object86864:head# 2024-03-30T13:50:58.821 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.821 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a654218:::benchmark_data_smithi043_119832_object81247:head# 2024-03-30T13:50:58.821 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.822 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a654b64:::benchmark_data_smithi043_119832_object59115:head# 2024-03-30T13:50:58.822 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.822 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a658ad3:::benchmark_data_smithi043_119832_object71368:head# 2024-03-30T13:50:58.822 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.823 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a65c8a7:::benchmark_data_smithi043_119832_object6692:head# 2024-03-30T13:50:58.823 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.823 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a65e00d:::benchmark_data_smithi043_119832_object51080:head# 2024-03-30T13:50:58.823 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.823 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a65f77d:::benchmark_data_smithi043_119832_object88340:head# 2024-03-30T13:50:58.824 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.824 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a6620e9:::benchmark_data_smithi043_119832_object42906:head# 2024-03-30T13:50:58.824 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.824 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a67963a:::benchmark_data_smithi043_119832_object11937:head# 2024-03-30T13:50:58.824 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.824 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a67fb1f:::benchmark_data_smithi043_119832_object80633:head# 2024-03-30T13:50:58.824 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.824 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a68fe32:::benchmark_data_smithi043_119832_object41608:head# 2024-03-30T13:50:58.824 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.825 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a69260e:::benchmark_data_smithi043_119832_object42119:head# 2024-03-30T13:50:58.825 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.825 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a69d21e:::benchmark_data_smithi043_119832_object42897:head# 2024-03-30T13:50:58.825 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.825 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a6a17ee:::benchmark_data_smithi043_119832_object38585:head# 2024-03-30T13:50:58.826 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.826 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a6a6114:::benchmark_data_smithi043_119832_object5994:head# 2024-03-30T13:50:58.826 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.826 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a6bca4d:::benchmark_data_smithi043_119832_object70526:head# 2024-03-30T13:50:58.826 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.827 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a6bff74:::benchmark_data_smithi043_119832_object65423:head# 2024-03-30T13:50:58.827 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.827 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a6c3660:::benchmark_data_smithi043_119832_object20467:head# 2024-03-30T13:50:58.827 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.828 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a6c39c3:::benchmark_data_smithi043_119832_object19919:head# 2024-03-30T13:50:58.828 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.828 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a6c73cd:::benchmark_data_smithi043_119832_object52252:head# 2024-03-30T13:50:58.828 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.828 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a6d64b1:::benchmark_data_smithi043_119832_object1255:head# 2024-03-30T13:50:58.828 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.829 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a6d82df:::benchmark_data_smithi043_119832_object84293:head# 2024-03-30T13:50:58.829 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.829 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a6dfd0a:::benchmark_data_smithi043_119832_object29487:head# 2024-03-30T13:50:58.829 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.829 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a6eeec2:::benchmark_data_smithi043_119832_object64823:head# 2024-03-30T13:50:58.829 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.830 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a71b8aa:::benchmark_data_smithi043_119832_object41525:head# 2024-03-30T13:50:58.831 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.831 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a725e44:::benchmark_data_smithi043_119832_object38412:head# 2024-03-30T13:50:58.831 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.831 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a72bacc:::benchmark_data_smithi043_119832_object50813:head# 2024-03-30T13:50:58.831 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.831 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a74f065:::benchmark_data_smithi043_119832_object60172:head# 2024-03-30T13:50:58.831 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.832 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a7553b2:::benchmark_data_smithi043_119832_object50705:head# 2024-03-30T13:50:58.832 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.832 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a75ba95:::benchmark_data_smithi043_119832_object28077:head# 2024-03-30T13:50:58.832 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.832 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a7685fe:::benchmark_data_smithi043_119832_object44563:head# 2024-03-30T13:50:58.832 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.832 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a76e687:::benchmark_data_smithi043_119832_object51413:head# 2024-03-30T13:50:58.833 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.833 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a7792a2:::benchmark_data_smithi043_119832_object66467:head# 2024-03-30T13:50:58.834 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.834 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a793190:::benchmark_data_smithi043_119832_object38277:head# 2024-03-30T13:50:58.834 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.834 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a793b0e:::benchmark_data_smithi043_119832_object8435:head# 2024-03-30T13:50:58.834 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.834 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a795fd5:::benchmark_data_smithi043_119832_object50582:head# 2024-03-30T13:50:58.834 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.834 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a798a76:::benchmark_data_smithi043_119832_object81903:head# 2024-03-30T13:50:58.834 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.835 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a7ac307:::benchmark_data_smithi043_119832_object88070:head# 2024-03-30T13:50:58.835 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.835 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a7b921a:::benchmark_data_smithi043_119832_object2725:head# 2024-03-30T13:50:58.836 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.836 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a7ba168:::benchmark_data_smithi043_119832_object554:head# 2024-03-30T13:50:58.836 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.836 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a7c1b99:::benchmark_data_smithi043_119832_object16648:head# 2024-03-30T13:50:58.836 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.836 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a7c451b:::benchmark_data_smithi043_119832_object41217:head# 2024-03-30T13:50:58.836 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.836 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:58.838 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a7c8d13:::benchmark_data_smithi043_119832_object80392:head# 2024-03-30T13:50:58.838 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.838 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a7c94de:::benchmark_data_smithi043_119832_object23962:head# 2024-03-30T13:50:58.838 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.838 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a7d74c0:::benchmark_data_smithi043_119832_object20758:head# 2024-03-30T13:50:58.839 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.839 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a7d8295:::benchmark_data_smithi043_119832_object52497:head# 2024-03-30T13:50:58.839 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.839 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a814e43:::benchmark_data_smithi043_119832_object68562:head# 2024-03-30T13:50:58.839 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.839 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a837838:::benchmark_data_smithi043_119832_object40784:head# 2024-03-30T13:50:58.839 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.839 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a845524:::benchmark_data_smithi043_119832_object37730:head# 2024-03-30T13:50:58.839 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.841 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a85236f:::benchmark_data_smithi043_119832_object15183:head# 2024-03-30T13:50:58.841 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.841 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a858678:::benchmark_data_smithi043_119832_object28438:head# 2024-03-30T13:50:58.841 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.841 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a85b40b:::benchmark_data_smithi043_119832_object69576:head# 2024-03-30T13:50:58.841 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.841 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a862ad2:::benchmark_data_smithi043_119832_object85209:head# 2024-03-30T13:50:58.841 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.841 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a86835b:::benchmark_data_smithi043_119832_object2641:head# 2024-03-30T13:50:58.841 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.841 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a86dc52:::benchmark_data_smithi043_119832_object57949:head# 2024-03-30T13:50:58.841 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.842 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a874238:::benchmark_data_smithi043_119832_object87488:head# 2024-03-30T13:50:58.842 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.842 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a8954b0:::benchmark_data_smithi043_119832_object79871:head# 2024-03-30T13:50:58.842 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.842 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a89d8bd:::benchmark_data_smithi043_119832_object70303:head# 2024-03-30T13:50:58.843 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.843 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a89ee37:::benchmark_data_smithi043_119832_object21324:head# 2024-03-30T13:50:58.843 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.843 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a8a1a97:::benchmark_data_smithi043_119832_object63427:head# 2024-03-30T13:50:58.843 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.843 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a8a413c:::benchmark_data_smithi043_119832_object17388:head# 2024-03-30T13:50:58.843 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.843 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a8abce4:::benchmark_data_smithi043_119832_object4335:head# 2024-03-30T13:50:58.844 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.844 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a8b611e:::benchmark_data_smithi043_119832_object42571:head# 2024-03-30T13:50:58.844 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.844 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a8bca32:::benchmark_data_smithi043_119832_object32819:head# 2024-03-30T13:50:58.844 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.844 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a8c1737:::benchmark_data_smithi043_119832_object73932:head# 2024-03-30T13:50:58.845 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.845 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a8c5754:::benchmark_data_smithi043_119832_object14025:head# 2024-03-30T13:50:58.846 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.846 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a8c6941:::benchmark_data_smithi043_119832_object85821:head# 2024-03-30T13:50:58.846 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.846 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a8d3114:::benchmark_data_smithi043_119832_object26727:head# 2024-03-30T13:50:58.846 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.847 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a8d3347:::benchmark_data_smithi043_119832_object43315:head# 2024-03-30T13:50:58.847 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.847 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a8d42f6:::benchmark_data_smithi043_119832_object4581:head# 2024-03-30T13:50:58.847 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.847 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a8ea2fe:::benchmark_data_smithi043_119832_object9759:head# 2024-03-30T13:50:58.847 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.847 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a8f15fb:::benchmark_data_smithi043_119832_object85981:head# 2024-03-30T13:50:58.847 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.849 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a8f1f83:::benchmark_data_smithi043_119832_object26836:head# 2024-03-30T13:50:58.849 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.849 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a8f7407:::benchmark_data_smithi043_119832_object71759:head# 2024-03-30T13:50:58.849 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.849 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a90b3a9:::benchmark_data_smithi043_119832_object24410:head# 2024-03-30T13:50:58.849 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.849 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a914d16:::benchmark_data_smithi043_119832_object85527:head# 2024-03-30T13:50:58.849 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.849 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a921891:::benchmark_data_smithi043_119832_object77681:head# 2024-03-30T13:50:58.849 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.850 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a9223d5:::benchmark_data_smithi043_119832_object38680:head# 2024-03-30T13:50:58.850 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.850 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a926399:::benchmark_data_smithi043_119832_object71200:head# 2024-03-30T13:50:58.850 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.850 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a928d9a:::benchmark_data_smithi043_119832_object56960:head# 2024-03-30T13:50:58.850 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.851 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a92a519:::benchmark_data_smithi043_119832_object10414:head# 2024-03-30T13:50:58.851 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.851 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a9301b8:::benchmark_data_smithi043_119832_object7358:head# 2024-03-30T13:50:58.851 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.851 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a932e37:::benchmark_data_smithi043_119832_object71398:head# 2024-03-30T13:50:58.851 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.851 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a93973c:::benchmark_data_smithi043_119832_object21458:head# 2024-03-30T13:50:58.851 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.852 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a95b276:::benchmark_data_smithi043_119832_object85534:head# 2024-03-30T13:50:58.852 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.852 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a961628:::benchmark_data_smithi043_119832_object64703:head# 2024-03-30T13:50:58.852 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.852 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a963ccc:::benchmark_data_smithi043_119832_object62902:head# 2024-03-30T13:50:58.853 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.853 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a96900d:::benchmark_data_smithi043_119832_object74675:head# 2024-03-30T13:50:58.853 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.853 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a98f5ae:::benchmark_data_smithi043_119832_object37074:head# 2024-03-30T13:50:58.853 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.854 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a999f29:::benchmark_data_smithi043_119832_object81920:head# 2024-03-30T13:50:58.854 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.854 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a9a94fa:::benchmark_data_smithi043_119832_object68318:head# 2024-03-30T13:50:58.854 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.855 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a9ac920:::benchmark_data_smithi043_119832_object79966:head# 2024-03-30T13:50:58.855 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.855 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a9bf5ff:::benchmark_data_smithi043_119832_object4095:head# 2024-03-30T13:50:58.855 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.855 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a9ce3de:::benchmark_data_smithi043_119832_object15345:head# 2024-03-30T13:50:58.855 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.856 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a9d25b7:::benchmark_data_smithi043_119832_object2423:head# 2024-03-30T13:50:58.856 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.856 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a9d8a04:::benchmark_data_smithi043_119832_object22632:head# 2024-03-30T13:50:58.856 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.857 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a9e364d:::benchmark_data_smithi043_119832_object70468:head# 2024-03-30T13:50:58.857 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.857 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a9f4457:::benchmark_data_smithi043_119832_object5179:head# 2024-03-30T13:50:58.857 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.857 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7a9f7776:::benchmark_data_smithi043_119832_object66086:head# 2024-03-30T13:50:58.858 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.858 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aa0ca6d:::benchmark_data_smithi043_119832_object86567:head# 2024-03-30T13:50:58.858 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.858 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aa14fcf:::benchmark_data_smithi043_119832_object32261:head# 2024-03-30T13:50:58.858 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.858 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aa17976:::benchmark_data_smithi043_119832_object83519:head# 2024-03-30T13:50:58.859 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.859 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aa26ff3:::benchmark_data_smithi043_119832_object67683:head# 2024-03-30T13:50:58.859 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.859 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aa35262:::benchmark_data_smithi043_119832_object7750:head# 2024-03-30T13:50:58.859 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.860 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aa380cf:::benchmark_data_smithi043_119832_object75591:head# 2024-03-30T13:50:58.860 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.860 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aa4576d:::benchmark_data_smithi043_119832_object42084:head# 2024-03-30T13:50:58.860 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.860 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aa4cbbb:::benchmark_data_smithi043_119832_object31534:head# 2024-03-30T13:50:58.861 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.861 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aa589a8:::benchmark_data_smithi043_119832_object48153:head# 2024-03-30T13:50:58.861 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.861 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aa5996b:::benchmark_data_smithi043_119832_object48685:head# 2024-03-30T13:50:58.861 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.861 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aa6f9de:::benchmark_data_smithi043_119832_object72324:head# 2024-03-30T13:50:58.861 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.862 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aa7bbbf:::benchmark_data_smithi043_119832_object6897:head# 2024-03-30T13:50:58.862 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.862 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aa8bba1:::benchmark_data_smithi043_119832_object7132:head# 2024-03-30T13:50:58.862 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.866 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aa8d466:::benchmark_data_smithi043_119832_object35718:head# 2024-03-30T13:50:58.866 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.866 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aa93f48:::benchmark_data_smithi043_119832_object53909:head# 2024-03-30T13:50:58.866 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.866 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aaa120a:::benchmark_data_smithi043_119832_object44656:head# 2024-03-30T13:50:58.866 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.866 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aab9a1b:::benchmark_data_smithi043_119832_object80355:head# 2024-03-30T13:50:58.866 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.866 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aabfa75:::benchmark_data_smithi043_119832_object53162:head# 2024-03-30T13:50:58.866 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.866 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aad6409:::benchmark_data_smithi043_119832_object48296:head# 2024-03-30T13:50:58.866 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.867 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aae582d:::benchmark_data_smithi043_119832_object42750:head# 2024-03-30T13:50:58.867 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.867 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aaecb19:::benchmark_data_smithi043_119832_object63738:head# 2024-03-30T13:50:58.867 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.869 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aaf0436:::benchmark_data_smithi043_119832_object71091:head# 2024-03-30T13:50:58.869 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.869 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aaf129a:::benchmark_data_smithi043_119832_object54529:head# 2024-03-30T13:50:58.869 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.869 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aafcf4d:::benchmark_data_smithi043_119832_object629:head# 2024-03-30T13:50:58.869 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.869 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ab240e6:::benchmark_data_smithi043_119832_object33710:head# 2024-03-30T13:50:58.870 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.870 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ab3f7be:::benchmark_data_smithi043_119832_object44838:head# 2024-03-30T13:50:58.870 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.870 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ab4a447:::benchmark_data_smithi043_119832_object24785:head# 2024-03-30T13:50:58.870 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.871 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ab4c6cd:::benchmark_data_smithi043_119832_object26827:head# 2024-03-30T13:50:58.871 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.871 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ab53a25:::benchmark_data_smithi043_119832_object22812:head# 2024-03-30T13:50:58.871 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.872 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ab63756:::benchmark_data_smithi043_119832_object26480:head# 2024-03-30T13:50:58.872 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.872 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ab65e35:::benchmark_data_smithi043_119832_object34035:head# 2024-03-30T13:50:58.872 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.873 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ab6f44f:::benchmark_data_smithi043_119832_object78143:head# 2024-03-30T13:50:58.873 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.873 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ab74166:::benchmark_data_smithi043_119832_object50429:head# 2024-03-30T13:50:58.873 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.873 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ab85a19:::benchmark_data_smithi043_119832_object51315:head# 2024-03-30T13:50:58.873 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.873 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ab90f97:::benchmark_data_smithi043_119832_object68071:head# 2024-03-30T13:50:58.874 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.876 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7abb0006:::benchmark_data_smithi043_119832_object57629:head# 2024-03-30T13:50:58.876 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.876 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7abb287b:::benchmark_data_smithi043_119832_object79771:head# 2024-03-30T13:50:58.876 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.876 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7abb474a:::benchmark_data_smithi043_119832_object11850:head# 2024-03-30T13:50:58.876 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.876 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7abecd41:::benchmark_data_smithi043_119832_object22524:head# 2024-03-30T13:50:58.877 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.877 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7abeda0c:::benchmark_data_smithi043_119832_object79941:head# 2024-03-30T13:50:58.877 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.877 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ac02eb4:::benchmark_data_smithi043_119832_object78724:head# 2024-03-30T13:50:58.877 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.877 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ac30d47:::benchmark_data_smithi043_119832_object86757:head# 2024-03-30T13:50:58.877 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.877 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ac327ef:::benchmark_data_smithi043_119832_object82872:head# 2024-03-30T13:50:58.877 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.878 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ac42674:::benchmark_data_smithi043_119832_object54261:head# 2024-03-30T13:50:58.878 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.878 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:58.878 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ac43b1c:::benchmark_data_smithi043_119832_object86219:head# 2024-03-30T13:50:58.878 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.879 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ac5641d:::benchmark_data_smithi043_119832_object33805:head# 2024-03-30T13:50:58.879 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:58.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:58.880 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ac5783d:::benchmark_data_smithi043_119832_object60783:head# 2024-03-30T13:50:58.880 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.880 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ac5b62c:::benchmark_data_smithi043_119832_object18380:head# 2024-03-30T13:50:58.880 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.887 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ac70084:::benchmark_data_smithi043_119832_object75015:head# 2024-03-30T13:50:58.887 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.887 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ac8437c:::benchmark_data_smithi043_119832_object40572:head# 2024-03-30T13:50:58.887 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.887 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ac88cd3:::benchmark_data_smithi043_119832_object73618:head# 2024-03-30T13:50:58.887 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.888 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ac8f204:::benchmark_data_smithi043_119832_object9117:head# 2024-03-30T13:50:58.888 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.888 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ac91db4:::benchmark_data_smithi043_119832_object76078:head# 2024-03-30T13:50:58.888 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.889 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ac94ca7:::benchmark_data_smithi043_119832_object28275:head# 2024-03-30T13:50:58.889 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.889 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ac9621c:::benchmark_data_smithi043_119832_object54506:head# 2024-03-30T13:50:58.889 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.890 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ac9adab:::benchmark_data_smithi043_119832_object19762:head# 2024-03-30T13:50:58.890 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.890 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ac9c1c0:::benchmark_data_smithi043_119832_object67410:head# 2024-03-30T13:50:58.890 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.890 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aca8540:::benchmark_data_smithi043_119832_object45786:head# 2024-03-30T13:50:58.891 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.891 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7acbdce1:::benchmark_data_smithi043_119832_object78529:head# 2024-03-30T13:50:58.891 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.891 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7acbfb77:::benchmark_data_smithi043_119832_object31704:head# 2024-03-30T13:50:58.891 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.891 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7acc2215:::benchmark_data_smithi043_119832_object18349:head# 2024-03-30T13:50:58.892 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.892 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7acc94b9:::benchmark_data_smithi043_119832_object16796:head# 2024-03-30T13:50:58.892 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.892 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7acfaae5:::benchmark_data_smithi043_119832_object87575:head# 2024-03-30T13:50:58.892 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.893 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7acff29e:::benchmark_data_smithi043_119832_object77269:head# 2024-03-30T13:50:58.893 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.893 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ad04bad:::benchmark_data_smithi043_119832_object33739:head# 2024-03-30T13:50:58.893 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.893 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ad1701c:::benchmark_data_smithi043_119832_object53623:head# 2024-03-30T13:50:58.894 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.894 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ad341e4:::benchmark_data_smithi043_119832_object305:head# 2024-03-30T13:50:58.894 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.894 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ad4ca31:::benchmark_data_smithi043_119832_object67010:head# 2024-03-30T13:50:58.894 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.895 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ad4d592:::benchmark_data_smithi043_119832_object37113:head# 2024-03-30T13:50:58.895 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.896 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ad54caf:::benchmark_data_smithi043_119832_object29819:head# 2024-03-30T13:50:58.896 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.897 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ad571d9:::benchmark_data_smithi043_119832_object42709:head# 2024-03-30T13:50:58.897 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.898 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ad5f2f5:::benchmark_data_smithi043_119832_object13084:head# 2024-03-30T13:50:58.898 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.898 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ad69e96:::benchmark_data_smithi043_119832_object9282:head# 2024-03-30T13:50:58.898 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.899 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ad753fc:::benchmark_data_smithi043_119832_object35607:head# 2024-03-30T13:50:58.899 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.899 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ad84fd5:::benchmark_data_smithi043_119832_object57562:head# 2024-03-30T13:50:58.899 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.900 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ad8e6c6:::benchmark_data_smithi043_119832_object61118:head# 2024-03-30T13:50:58.900 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.900 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ad9fb33:::benchmark_data_smithi043_119832_object541:head# 2024-03-30T13:50:58.900 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.901 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ada4ebf:::benchmark_data_smithi043_119832_object79236:head# 2024-03-30T13:50:58.901 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.901 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7adc4667:::benchmark_data_smithi043_119832_object29525:head# 2024-03-30T13:50:58.901 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.901 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7adece91:::benchmark_data_smithi043_119832_object22414:head# 2024-03-30T13:50:58.902 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.902 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ae10229:::benchmark_data_smithi043_119832_object64815:head# 2024-03-30T13:50:58.902 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.902 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ae1e6a7:::benchmark_data_smithi043_119832_object63315:head# 2024-03-30T13:50:58.902 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.902 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ae28c3a:::benchmark_data_smithi043_119832_object21497:head# 2024-03-30T13:50:58.903 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.903 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ae2bc01:::benchmark_data_smithi043_119832_object69726:head# 2024-03-30T13:50:58.903 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.903 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ae49e6b:::benchmark_data_smithi043_119832_object62771:head# 2024-03-30T13:50:58.903 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.903 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ae5213e:::benchmark_data_smithi043_119832_object74732:head# 2024-03-30T13:50:58.903 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.904 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ae5d014:::benchmark_data_smithi043_119832_object59037:head# 2024-03-30T13:50:58.904 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.904 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ae60d80:::benchmark_data_smithi043_119832_object75176:head# 2024-03-30T13:50:58.904 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.905 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ae6dff3:::benchmark_data_smithi043_119832_object72199:head# 2024-03-30T13:50:58.905 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.905 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ae70593:::benchmark_data_smithi043_119832_object40073:head# 2024-03-30T13:50:58.905 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.905 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ae71659:::benchmark_data_smithi043_119832_object44571:head# 2024-03-30T13:50:58.905 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.906 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ae77314:::benchmark_data_smithi043_119832_object88505:head# 2024-03-30T13:50:58.906 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.906 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ae777bb:::benchmark_data_smithi043_119832_object45401:head# 2024-03-30T13:50:58.906 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.907 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ae85b36:::benchmark_data_smithi043_119832_object39855:head# 2024-03-30T13:50:58.907 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.907 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ae8e64c:::benchmark_data_smithi043_119832_object27979:head# 2024-03-30T13:50:58.907 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.907 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ae907a7:::benchmark_data_smithi043_119832_object51750:head# 2024-03-30T13:50:58.907 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.908 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ae97a6a:::benchmark_data_smithi043_119832_object43011:head# 2024-03-30T13:50:58.908 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.908 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ae996c5:::benchmark_data_smithi043_119832_object71236:head# 2024-03-30T13:50:58.908 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.909 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aea0652:::benchmark_data_smithi043_119832_object17995:head# 2024-03-30T13:50:58.909 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.909 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aeaa027:::benchmark_data_smithi043_119832_object78391:head# 2024-03-30T13:50:58.909 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.909 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aecd6a7:::benchmark_data_smithi043_119832_object19770:head# 2024-03-30T13:50:58.909 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.909 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aee418a:::benchmark_data_smithi043_119832_object29665:head# 2024-03-30T13:50:58.909 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.910 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aeec871:::benchmark_data_smithi043_119832_object47284:head# 2024-03-30T13:50:58.910 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.910 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aef4ef9:::benchmark_data_smithi043_119832_object13423:head# 2024-03-30T13:50:58.910 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.911 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7aefe1ac:::benchmark_data_smithi043_119832_object73755:head# 2024-03-30T13:50:58.911 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.911 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7af18aa6:::benchmark_data_smithi043_119832_object71747:head# 2024-03-30T13:50:58.911 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.911 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7af227c9:::benchmark_data_smithi043_119832_object31867:head# 2024-03-30T13:50:58.911 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.912 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7af2b07e:::benchmark_data_smithi043_119832_object88324:head# 2024-03-30T13:50:58.912 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.912 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7af2b9f3:::benchmark_data_smithi043_119832_object32573:head# 2024-03-30T13:50:58.912 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.912 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7af40a9f:::benchmark_data_smithi043_119832_object47110:head# 2024-03-30T13:50:58.913 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.913 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7af49a58:::benchmark_data_smithi043_119832_object49292:head# 2024-03-30T13:50:58.913 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.913 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7af4c1ec:::benchmark_data_smithi043_119832_object25526:head# 2024-03-30T13:50:58.913 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.914 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7af59b9b:::benchmark_data_smithi043_119832_object60507:head# 2024-03-30T13:50:58.914 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.914 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7af78d62:::benchmark_data_smithi043_119832_object36881:head# 2024-03-30T13:50:58.914 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.914 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7af82d0e:::benchmark_data_smithi043_119832_object38110:head# 2024-03-30T13:50:58.914 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.915 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7af82edd:::benchmark_data_smithi043_119832_object37457:head# 2024-03-30T13:50:58.915 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.915 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7af9e5fb:::benchmark_data_smithi043_119832_object24223:head# 2024-03-30T13:50:58.915 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.916 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7afae7c9:::benchmark_data_smithi043_119832_object82659:head# 2024-03-30T13:50:58.916 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.916 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7afb0b2c:::benchmark_data_smithi043_119832_object27734:head# 2024-03-30T13:50:58.916 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.917 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7afba001:::benchmark_data_smithi043_119832_object78038:head# 2024-03-30T13:50:58.917 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.917 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7afc206f:::benchmark_data_smithi043_119832_object64605:head# 2024-03-30T13:50:58.917 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.917 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7afc3df4:::benchmark_data_smithi043_119832_object41436:head# 2024-03-30T13:50:58.917 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.918 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7afe269e:::benchmark_data_smithi043_119832_object19409:head# 2024-03-30T13:50:58.918 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.918 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7affd612:::benchmark_data_smithi043_119832_object81380:head# 2024-03-30T13:50:58.918 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.918 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b00c90e:::benchmark_data_smithi043_119832_object71119:head# 2024-03-30T13:50:58.918 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.919 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b0561a6:::benchmark_data_smithi043_119832_object18119:head# 2024-03-30T13:50:58.919 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.919 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b05c80f:::benchmark_data_smithi043_119832_object34093:head# 2024-03-30T13:50:58.919 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.919 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b062123:::benchmark_data_smithi043_119832_object6033:head# 2024-03-30T13:50:58.919 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.919 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b0675e2:::benchmark_data_smithi043_119832_object32646:head# 2024-03-30T13:50:58.920 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.920 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b085642:::benchmark_data_smithi043_119832_object6967:head# 2024-03-30T13:50:58.920 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.920 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b09329a:::benchmark_data_smithi043_119832_object61837:head# 2024-03-30T13:50:58.920 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.921 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b0a162a:::benchmark_data_smithi043_119832_object40848:head# 2024-03-30T13:50:58.921 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.921 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b0b94fa:::benchmark_data_smithi043_119832_object74350:head# 2024-03-30T13:50:58.921 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.921 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b0bb62a:::benchmark_data_smithi043_119832_object61920:head# 2024-03-30T13:50:58.922 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.922 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b0be08e:::benchmark_data_smithi043_119832_object83154:head# 2024-03-30T13:50:58.922 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.922 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b0c392c:::benchmark_data_smithi043_119832_object16980:head# 2024-03-30T13:50:58.922 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.922 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b0c55bd:::benchmark_data_smithi043_119832_object44657:head# 2024-03-30T13:50:58.923 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.923 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b0d595b:::benchmark_data_smithi043_119832_object70102:head# 2024-03-30T13:50:58.924 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.924 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b0eab4f:::benchmark_data_smithi043_119832_object26736:head# 2024-03-30T13:50:58.924 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.924 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b0f8e78:::benchmark_data_smithi043_119832_object57750:head# 2024-03-30T13:50:58.924 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.924 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b11b60e:::benchmark_data_smithi043_119832_object27101:head# 2024-03-30T13:50:58.924 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.925 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b12290f:::benchmark_data_smithi043_119832_object38217:head# 2024-03-30T13:50:58.925 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.925 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b12f3c9:::benchmark_data_smithi043_119832_object4568:head# 2024-03-30T13:50:58.925 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.925 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b130cc9:::benchmark_data_smithi043_119832_object48104:head# 2024-03-30T13:50:58.925 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.925 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b14599d:::benchmark_data_smithi043_119832_object11025:head# 2024-03-30T13:50:58.925 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.926 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b14a523:::benchmark_data_smithi043_119832_object66425:head# 2024-03-30T13:50:58.926 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.926 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b15bbf2:::benchmark_data_smithi043_119832_object20442:head# 2024-03-30T13:50:58.926 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.926 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b15e1a0:::benchmark_data_smithi043_119832_object74109:head# 2024-03-30T13:50:58.926 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.927 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b15ebcf:::benchmark_data_smithi043_119832_object28129:head# 2024-03-30T13:50:58.927 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.927 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b169edc:::benchmark_data_smithi043_119832_object81933:head# 2024-03-30T13:50:58.927 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.928 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b173832:::benchmark_data_smithi043_119832_object59379:head# 2024-03-30T13:50:58.928 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.928 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b17ffbc:::benchmark_data_smithi043_119832_object78752:head# 2024-03-30T13:50:58.928 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.928 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b19c868:::benchmark_data_smithi043_119832_object8417:head# 2024-03-30T13:50:58.928 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.928 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b1a0525:::benchmark_data_smithi043_119832_object60887:head# 2024-03-30T13:50:58.929 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.929 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b1afe9c:::benchmark_data_smithi043_119832_object86958:head# 2024-03-30T13:50:58.929 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.929 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b1b0532:::benchmark_data_smithi043_119832_object79065:head# 2024-03-30T13:50:58.929 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.930 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b1c87fe:::benchmark_data_smithi043_119832_object8331:head# 2024-03-30T13:50:58.930 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.930 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b1d8f7e:::benchmark_data_smithi043_119832_object42536:head# 2024-03-30T13:50:58.930 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.930 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b1df204:::benchmark_data_smithi043_119832_object69291:head# 2024-03-30T13:50:58.931 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.931 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b1e397c:::benchmark_data_smithi043_119832_object69062:head# 2024-03-30T13:50:58.931 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.931 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b1e41d5:::benchmark_data_smithi043_119832_object65219:head# 2024-03-30T13:50:58.931 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.931 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b1f6854:::benchmark_data_smithi043_119832_object59797:head# 2024-03-30T13:50:58.931 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.932 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b201d84:::benchmark_data_smithi043_119832_object46181:head# 2024-03-30T13:50:58.932 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.932 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b20232a:::benchmark_data_smithi043_119832_object5372:head# 2024-03-30T13:50:58.932 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.933 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b226e89:::benchmark_data_smithi043_119832_object36156:head# 2024-03-30T13:50:58.933 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.933 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b22da26:::benchmark_data_smithi043_119832_object24655:head# 2024-03-30T13:50:58.933 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.933 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b232d3d:::benchmark_data_smithi043_119832_object57545:head# 2024-03-30T13:50:58.933 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.934 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b24d1aa:::benchmark_data_smithi043_119832_object29528:head# 2024-03-30T13:50:58.934 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.934 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b258cf3:::benchmark_data_smithi043_119832_object73065:head# 2024-03-30T13:50:58.934 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.934 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b25d908:::benchmark_data_smithi043_119832_object80840:head# 2024-03-30T13:50:58.934 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.935 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b25dae4:::benchmark_data_smithi043_119832_object8451:head# 2024-03-30T13:50:58.935 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.935 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b269002:::benchmark_data_smithi043_119832_object76147:head# 2024-03-30T13:50:58.935 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.935 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b276b96:::benchmark_data_smithi043_119832_object5985:head# 2024-03-30T13:50:58.935 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.935 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b284bd4:::benchmark_data_smithi043_119832_object48692:head# 2024-03-30T13:50:58.935 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.936 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b289b6d:::benchmark_data_smithi043_119832_object36568:head# 2024-03-30T13:50:58.936 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.936 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b28b6c7:::benchmark_data_smithi043_119832_object7666:head# 2024-03-30T13:50:58.936 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.937 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b29a8af:::benchmark_data_smithi043_119832_object28038:head# 2024-03-30T13:50:58.937 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.937 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b2a2a60:::benchmark_data_smithi043_119832_object11076:head# 2024-03-30T13:50:58.937 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.937 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b2c6422:::benchmark_data_smithi043_119832_object75353:head# 2024-03-30T13:50:58.938 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.938 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b2c9c21:::benchmark_data_smithi043_119832_object60000:head# 2024-03-30T13:50:58.938 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.938 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b2d2aaa:::benchmark_data_smithi043_119832_object65872:head# 2024-03-30T13:50:58.938 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.939 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b2d3586:::benchmark_data_smithi043_119832_object12121:head# 2024-03-30T13:50:58.939 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.939 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b2d4a4e:::benchmark_data_smithi043_119832_object36717:head# 2024-03-30T13:50:58.939 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.939 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b2de454:::benchmark_data_smithi043_119832_object38673:head# 2024-03-30T13:50:58.939 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.940 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b2e0d92:::benchmark_data_smithi043_119832_object86533:head# 2024-03-30T13:50:58.940 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.940 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b2f39d4:::benchmark_data_smithi043_119832_object73159:head# 2024-03-30T13:50:58.940 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.940 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b2f96f6:::benchmark_data_smithi043_119832_object47616:head# 2024-03-30T13:50:58.940 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.941 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b2ff51b:::benchmark_data_smithi043_119832_object65207:head# 2024-03-30T13:50:58.941 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.941 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b2ff99e:::benchmark_data_smithi043_119832_object12211:head# 2024-03-30T13:50:58.941 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.942 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b300861:::benchmark_data_smithi043_119832_object30697:head# 2024-03-30T13:50:58.942 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.942 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b301027:::benchmark_data_smithi043_119832_object13830:head# 2024-03-30T13:50:58.942 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.943 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b301982:::benchmark_data_smithi043_119832_object48064:head# 2024-03-30T13:50:58.943 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.943 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b303b22:::benchmark_data_smithi043_119832_object23776:head# 2024-03-30T13:50:58.943 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.943 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b308946:::benchmark_data_smithi043_119832_object49455:head# 2024-03-30T13:50:58.943 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.944 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b30edcb:::benchmark_data_smithi043_119832_object2101:head# 2024-03-30T13:50:58.944 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.944 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b31a7d2:::benchmark_data_smithi043_119832_object66411:head# 2024-03-30T13:50:58.944 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.944 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b31e586:::benchmark_data_smithi043_119832_object30002:head# 2024-03-30T13:50:58.945 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.945 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b31fe3b:::benchmark_data_smithi043_119832_object54515:head# 2024-03-30T13:50:58.945 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.945 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b3326b0:::benchmark_data_smithi043_119832_object55320:head# 2024-03-30T13:50:58.945 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.945 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b33bea4:::benchmark_data_smithi043_119832_object48326:head# 2024-03-30T13:50:58.946 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.946 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b34fec9:::benchmark_data_smithi043_119832_object8640:head# 2024-03-30T13:50:58.946 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.946 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b353a41:::benchmark_data_smithi043_119832_object53435:head# 2024-03-30T13:50:58.946 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.947 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b358b60:::benchmark_data_smithi043_119832_object79399:head# 2024-03-30T13:50:58.947 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.947 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b363cf8:::benchmark_data_smithi043_119832_object11356:head# 2024-03-30T13:50:58.947 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.947 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b3707fd:::benchmark_data_smithi043_119832_object81607:head# 2024-03-30T13:50:58.947 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.947 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b372238:::benchmark_data_smithi043_119832_object5336:head# 2024-03-30T13:50:58.948 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.948 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b387e08:::benchmark_data_smithi043_119832_object35639:head# 2024-03-30T13:50:58.948 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.948 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b39a30c:::benchmark_data_smithi043_119832_object27121:head# 2024-03-30T13:50:58.948 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.948 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b3ccd79:::benchmark_data_smithi043_119832_object65528:head# 2024-03-30T13:50:58.948 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.948 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b3cd976:::benchmark_data_smithi043_119832_object27188:head# 2024-03-30T13:50:58.948 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.949 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b3db904:::benchmark_data_smithi043_119832_object39747:head# 2024-03-30T13:50:58.949 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.949 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b3dd379:::benchmark_data_smithi043_119832_object71262:head# 2024-03-30T13:50:58.949 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.950 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b3e052f:::benchmark_data_smithi043_119832_object25541:head# 2024-03-30T13:50:58.950 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.950 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b3ec82f:::benchmark_data_smithi043_119832_object79511:head# 2024-03-30T13:50:58.950 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.950 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b3ede05:::benchmark_data_smithi043_119832_object45902:head# 2024-03-30T13:50:58.950 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.951 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b3efad9:::benchmark_data_smithi043_119832_object24239:head# 2024-03-30T13:50:58.951 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.951 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b3f99e7:::benchmark_data_smithi043_119832_object32230:head# 2024-03-30T13:50:58.951 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.951 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b402929:::benchmark_data_smithi043_119832_object30229:head# 2024-03-30T13:50:58.951 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.952 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b40455b:::benchmark_data_smithi043_119832_object51491:head# 2024-03-30T13:50:58.952 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.952 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b40fa27:::benchmark_data_smithi043_119832_object19615:head# 2024-03-30T13:50:58.952 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.952 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b410579:::benchmark_data_smithi043_119832_object38551:head# 2024-03-30T13:50:58.952 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.953 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b414f2c:::benchmark_data_smithi043_119832_object5169:head# 2024-03-30T13:50:58.953 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.953 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b425006:::benchmark_data_smithi043_119832_object74450:head# 2024-03-30T13:50:58.953 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.954 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b435402:::benchmark_data_smithi043_119832_object68868:head# 2024-03-30T13:50:58.954 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.954 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b43b1ff:::benchmark_data_smithi043_119832_object31244:head# 2024-03-30T13:50:58.954 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.954 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b441039:::benchmark_data_smithi043_119832_object15837:head# 2024-03-30T13:50:58.954 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.955 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b453c0e:::benchmark_data_smithi043_119832_object58225:head# 2024-03-30T13:50:58.955 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.955 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b454b80:::benchmark_data_smithi043_119832_object44831:head# 2024-03-30T13:50:58.955 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.955 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b45cc07:::benchmark_data_smithi043_119832_object4498:head# 2024-03-30T13:50:58.955 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.956 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b46712a:::benchmark_data_smithi043_119832_object57600:head# 2024-03-30T13:50:58.956 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.956 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b46c517:::benchmark_data_smithi043_119832_object50560:head# 2024-03-30T13:50:58.956 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.956 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b47a31f:::benchmark_data_smithi043_119832_object19020:head# 2024-03-30T13:50:58.956 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.957 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b4841d4:::benchmark_data_smithi043_119832_object9304:head# 2024-03-30T13:50:58.957 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.957 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b487e7c:::benchmark_data_smithi043_119832_object51403:head# 2024-03-30T13:50:58.957 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.957 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b49c6b5:::benchmark_data_smithi043_119832_object32111:head# 2024-03-30T13:50:58.958 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.958 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b4b2074:::benchmark_data_smithi043_119832_object87597:head# 2024-03-30T13:50:58.958 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.958 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b4d0d94:::benchmark_data_smithi043_119832_object23663:head# 2024-03-30T13:50:58.958 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.958 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b4d2b92:::benchmark_data_smithi043_119832_object72582:head# 2024-03-30T13:50:58.958 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.959 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b4d425b:::benchmark_data_smithi043_119832_object12636:head# 2024-03-30T13:50:58.959 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.959 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b4e28aa:::benchmark_data_smithi043_119832_object74500:head# 2024-03-30T13:50:58.959 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.960 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b4eb7a8:::benchmark_data_smithi043_119832_object69317:head# 2024-03-30T13:50:58.960 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.960 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b4f0bb8:::benchmark_data_smithi043_119832_object9365:head# 2024-03-30T13:50:58.960 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.960 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b4f2214:::benchmark_data_smithi043_119832_object56671:head# 2024-03-30T13:50:58.960 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.960 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b4f2f7b:::benchmark_data_smithi043_119832_object1724:head# 2024-03-30T13:50:58.960 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.961 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b4fae64:::benchmark_data_smithi043_119832_object84333:head# 2024-03-30T13:50:58.961 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.961 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b509868:::benchmark_data_smithi043_119832_object1906:head# 2024-03-30T13:50:58.961 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.961 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b50d7a5:::benchmark_data_smithi043_119832_object7709:head# 2024-03-30T13:50:58.962 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.962 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b51b653:::benchmark_data_smithi043_119832_object46300:head# 2024-03-30T13:50:58.962 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.962 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b54c74b:::benchmark_data_smithi043_119832_object23343:head# 2024-03-30T13:50:58.962 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.963 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b559a96:::benchmark_data_smithi043_119832_object88385:head# 2024-03-30T13:50:58.963 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.963 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b55f0f4:::benchmark_data_smithi043_119832_object79135:head# 2024-03-30T13:50:58.963 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.964 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b564ccc:::benchmark_data_smithi043_119832_object5257:head# 2024-03-30T13:50:58.964 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.964 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b57c27a:::benchmark_data_smithi043_119832_object28264:head# 2024-03-30T13:50:58.964 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.964 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b599fe2:::benchmark_data_smithi043_119832_object125:head# 2024-03-30T13:50:58.964 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.964 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b59a03e:::benchmark_data_smithi043_119832_object59875:head# 2024-03-30T13:50:58.965 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.965 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b5aaf5b:::benchmark_data_smithi043_119832_object77100:head# 2024-03-30T13:50:58.965 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.965 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b5d660c:::benchmark_data_smithi043_119832_object86528:head# 2024-03-30T13:50:58.965 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.965 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b5d9d08:::benchmark_data_smithi043_119832_object32737:head# 2024-03-30T13:50:58.966 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.966 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b5dd2d9:::benchmark_data_smithi043_119832_object65109:head# 2024-03-30T13:50:58.966 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.966 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b5e55d4:::benchmark_data_smithi043_119832_object71387:head# 2024-03-30T13:50:58.966 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.967 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b5f0377:::benchmark_data_smithi043_119832_object25798:head# 2024-03-30T13:50:58.967 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.967 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b5f0907:::benchmark_data_smithi043_119832_object26089:head# 2024-03-30T13:50:58.967 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.968 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b5f5d40:::benchmark_data_smithi043_119832_object30878:head# 2024-03-30T13:50:58.968 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.968 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b602ea6:::benchmark_data_smithi043_119832_object20525:head# 2024-03-30T13:50:58.968 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.968 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b607c44:::benchmark_data_smithi043_119832_object52613:head# 2024-03-30T13:50:58.968 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.969 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b616b3e:::benchmark_data_smithi043_119832_object40394:head# 2024-03-30T13:50:58.969 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.969 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b6251e9:::benchmark_data_smithi043_119832_object59513:head# 2024-03-30T13:50:58.969 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.969 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b644184:::benchmark_data_smithi043_119832_object87209:head# 2024-03-30T13:50:58.969 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.969 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b647bd1:::benchmark_data_smithi043_119832_object83741:head# 2024-03-30T13:50:58.970 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.970 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b648ef5:::benchmark_data_smithi043_119832_object542:head# 2024-03-30T13:50:58.970 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.970 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b654726:::benchmark_data_smithi043_119832_object64824:head# 2024-03-30T13:50:58.971 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.971 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b6557ff:::benchmark_data_smithi043_119832_object80293:head# 2024-03-30T13:50:58.971 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.971 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:50:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:50:58.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:58 smithi113 ceph-mon[34341]: 6.18 scrub starts 2024-03-30T13:50:58.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:58 smithi113 ceph-mon[34341]: 6.18 scrub ok 2024-03-30T13:50:58.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:58 smithi113 ceph-mon[34341]: pgmap v329: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 858 op/s; 44241/354549 objects degraded (12.478%) 2024-03-30T13:50:58.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:58 smithi113 ceph-mon[34341]: 8.5 scrub ok 2024-03-30T13:50:58.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:58 smithi113 ceph-mon[34341]: 5.0 scrub starts 2024-03-30T13:50:58.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:58 smithi113 ceph-mon[34341]: 5.0 scrub ok 2024-03-30T13:50:58.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:58 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:50:58.972 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b665ce1:::benchmark_data_smithi043_119832_object48455:head# 2024-03-30T13:50:58.973 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.973 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b67b75b:::benchmark_data_smithi043_119832_object59263:head# 2024-03-30T13:50:58.973 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.973 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b6ac353:::benchmark_data_smithi043_119832_object79167:head# 2024-03-30T13:50:58.973 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.973 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b6b0d90:::benchmark_data_smithi043_119832_object88333:head# 2024-03-30T13:50:58.973 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.973 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b6bd0c9:::benchmark_data_smithi043_119832_object65557:head# 2024-03-30T13:50:58.973 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.973 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b6c26eb:::benchmark_data_smithi043_119832_object87974:head# 2024-03-30T13:50:58.973 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.974 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b6c5c9b:::benchmark_data_smithi043_119832_object18375:head# 2024-03-30T13:50:58.974 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.974 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b6f7dad:::benchmark_data_smithi043_119832_object56343:head# 2024-03-30T13:50:58.974 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.974 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b6fea02:::benchmark_data_smithi043_119832_object69292:head# 2024-03-30T13:50:58.974 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.975 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b70bed8:::benchmark_data_smithi043_119832_object20052:head# 2024-03-30T13:50:58.975 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.975 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b7391b6:::benchmark_data_smithi043_119832_object15045:head# 2024-03-30T13:50:58.975 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.975 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b73b109:::benchmark_data_smithi043_119832_object34898:head# 2024-03-30T13:50:58.975 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.976 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b754458:::benchmark_data_smithi043_119832_object80123:head# 2024-03-30T13:50:58.976 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.976 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b758633:::benchmark_data_smithi043_119832_object58993:head# 2024-03-30T13:50:58.976 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.976 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b77d3a1:::benchmark_data_smithi043_119832_object14257:head# 2024-03-30T13:50:58.976 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.976 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b78eb79:::benchmark_data_smithi043_119832_object81574:head# 2024-03-30T13:50:58.977 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.977 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b78f698:::benchmark_data_smithi043_119832_object33069:head# 2024-03-30T13:50:58.977 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.977 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b796df3:::benchmark_data_smithi043_119832_object16687:head# 2024-03-30T13:50:58.977 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.978 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b7b8495:::benchmark_data_smithi043_119832_object33559:head# 2024-03-30T13:50:58.978 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.978 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b7c41ec:::benchmark_data_smithi043_119832_object39022:head# 2024-03-30T13:50:58.978 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.978 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b7d12b5:::benchmark_data_smithi043_119832_object3488:head# 2024-03-30T13:50:58.978 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.979 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b7da18e:::benchmark_data_smithi043_119832_object76137:head# 2024-03-30T13:50:58.979 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.979 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b7e9ca8:::benchmark_data_smithi043_119832_object56344:head# 2024-03-30T13:50:58.979 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.979 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b7ec3d6:::benchmark_data_smithi043_119832_object63729:head# 2024-03-30T13:50:58.979 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.980 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b7f31ab:::benchmark_data_smithi043_119832_object87804:head# 2024-03-30T13:50:58.980 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.980 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b809d71:::benchmark_data_smithi043_119832_object56839:head# 2024-03-30T13:50:58.980 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.980 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:58.981 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b80a220:::benchmark_data_smithi043_119832_object68487:head# 2024-03-30T13:50:58.981 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.981 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b80faf8:::benchmark_data_smithi043_119832_object43233:head# 2024-03-30T13:50:58.981 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.981 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b8133c1:::benchmark_data_smithi043_119832_object67611:head# 2024-03-30T13:50:58.981 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.982 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b81fa45:::benchmark_data_smithi043_119832_object23483:head# 2024-03-30T13:50:58.982 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.982 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b83e302:::benchmark_data_smithi043_119832_object21493:head# 2024-03-30T13:50:58.982 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.982 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b84b61f:::benchmark_data_smithi043_119832_object52276:head# 2024-03-30T13:50:58.982 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.984 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b84e78d:::benchmark_data_smithi043_119832_object32906:head# 2024-03-30T13:50:58.984 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.984 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b857dfd:::benchmark_data_smithi043_119832_object25229:head# 2024-03-30T13:50:58.984 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.984 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b860ffe:::benchmark_data_smithi043_119832_object3024:head# 2024-03-30T13:50:58.984 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.984 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b86153e:::benchmark_data_smithi043_119832_object17947:head# 2024-03-30T13:50:58.984 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.984 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b8674a5:::benchmark_data_smithi043_119832_object3785:head# 2024-03-30T13:50:58.984 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.985 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b87249f:::benchmark_data_smithi043_119832_object83366:head# 2024-03-30T13:50:58.985 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.985 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b878cfe:::benchmark_data_smithi043_119832_object27769:head# 2024-03-30T13:50:58.985 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.985 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b89d9d6:::benchmark_data_smithi043_119832_object50862:head# 2024-03-30T13:50:58.985 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.986 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b8a2295:::benchmark_data_smithi043_119832_object10922:head# 2024-03-30T13:50:58.986 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.986 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b8c14f0:::benchmark_data_smithi043_119832_object44066:head# 2024-03-30T13:50:58.986 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.986 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b8cd3a5:::benchmark_data_smithi043_119832_object71601:head# 2024-03-30T13:50:58.986 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.987 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b8ed36b:::benchmark_data_smithi043_119832_object45768:head# 2024-03-30T13:50:58.987 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.987 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b8ede72:::benchmark_data_smithi043_119832_object40211:head# 2024-03-30T13:50:58.987 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.987 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b8fa433:::benchmark_data_smithi043_119832_object84321:head# 2024-03-30T13:50:58.987 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.988 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b90e0c7:::benchmark_data_smithi043_119832_object14020:head# 2024-03-30T13:50:58.988 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.988 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b911995:::benchmark_data_smithi043_119832_object81321:head# 2024-03-30T13:50:58.988 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.988 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b931100:::benchmark_data_smithi043_119832_object39923:head# 2024-03-30T13:50:58.988 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.989 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b9368e8:::benchmark_data_smithi043_119832_object62967:head# 2024-03-30T13:50:58.989 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.989 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b956e9e:::benchmark_data_smithi043_119832_object14881:head# 2024-03-30T13:50:58.989 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.989 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b97c1e4:::benchmark_data_smithi043_119832_object73780:head# 2024-03-30T13:50:58.989 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.990 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b986b06:::benchmark_data_smithi043_119832_object75145:head# 2024-03-30T13:50:58.990 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.990 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b9a9cee:::benchmark_data_smithi043_119832_object64964:head# 2024-03-30T13:50:58.990 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.991 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b9b74fa:::benchmark_data_smithi043_119832_object43316:head# 2024-03-30T13:50:58.991 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.991 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7b9c7305:::benchmark_data_smithi043_119832_object31183:head# 2024-03-30T13:50:58.991 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.991 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ba074cd:::benchmark_data_smithi043_119832_object6999:head# 2024-03-30T13:50:58.991 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.992 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ba157dc:::benchmark_data_smithi043_119832_object48211:head# 2024-03-30T13:50:58.992 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.992 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ba1aaeb:::benchmark_data_smithi043_119832_object65211:head# 2024-03-30T13:50:58.992 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.992 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ba3f5be:::benchmark_data_smithi043_119832_object43339:head# 2024-03-30T13:50:58.992 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.993 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ba3f947:::benchmark_data_smithi043_119832_object81009:head# 2024-03-30T13:50:58.993 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.993 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ba441d3:::benchmark_data_smithi043_119832_object57448:head# 2024-03-30T13:50:58.993 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.994 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ba55715:::benchmark_data_smithi043_119832_object44159:head# 2024-03-30T13:50:58.994 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.994 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ba5c113:::benchmark_data_smithi043_119832_object76974:head# 2024-03-30T13:50:58.994 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.994 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ba5cc1d:::benchmark_data_smithi043_119832_object32726:head# 2024-03-30T13:50:58.995 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.995 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ba6590c:::benchmark_data_smithi043_119832_object77795:head# 2024-03-30T13:50:58.995 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.995 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ba6f844:::benchmark_data_smithi043_119832_object21146:head# 2024-03-30T13:50:58.995 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.995 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ba7ea01:::benchmark_data_smithi043_119832_object61591:head# 2024-03-30T13:50:58.995 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.996 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ba84fdd:::benchmark_data_smithi043_119832_object82491:head# 2024-03-30T13:50:58.996 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.996 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bab3595:::benchmark_data_smithi043_119832_object33543:head# 2024-03-30T13:50:58.996 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.997 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bad6e3d:::benchmark_data_smithi043_119832_object1899:head# 2024-03-30T13:50:58.997 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.997 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bae644b:::benchmark_data_smithi043_119832_object25061:head# 2024-03-30T13:50:58.997 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.997 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7baeae25:::benchmark_data_smithi043_119832_object75273:head# 2024-03-30T13:50:58.997 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.997 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7baef375:::benchmark_data_smithi043_119832_object53126:head# 2024-03-30T13:50:58.998 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.998 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7baf3e19:::benchmark_data_smithi043_119832_object57405:head# 2024-03-30T13:50:58.998 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.998 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7baf43cc:::benchmark_data_smithi043_119832_object35402:head# 2024-03-30T13:50:58.998 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.999 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7baff155:::benchmark_data_smithi043_119832_object52799:head# 2024-03-30T13:50:58.999 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.999 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb09186:::benchmark_data_smithi043_119832_object8834:head# 2024-03-30T13:50:58.999 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:58.999 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb0bcae:::benchmark_data_smithi043_119832_object7077:head# 2024-03-30T13:50:59.000 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.000 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb11967:::benchmark_data_smithi043_119832_object16622:head# 2024-03-30T13:50:59.000 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.000 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb1384b:::benchmark_data_smithi043_119832_object63977:head# 2024-03-30T13:50:59.000 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.000 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb15880:::benchmark_data_smithi043_119832_object26084:head# 2024-03-30T13:50:59.000 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.001 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb2b789:::benchmark_data_smithi043_119832_object26712:head# 2024-03-30T13:50:59.001 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.001 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb3744f:::benchmark_data_smithi043_119832_object18890:head# 2024-03-30T13:50:59.001 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.001 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb44525:::benchmark_data_smithi043_119832_object70760:head# 2024-03-30T13:50:59.001 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.002 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb453b5:::benchmark_data_smithi043_119832_object32173:head# 2024-03-30T13:50:59.002 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.002 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb4f916:::benchmark_data_smithi043_119832_object43014:head# 2024-03-30T13:50:59.002 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.002 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb53c90:::benchmark_data_smithi043_119832_object77976:head# 2024-03-30T13:50:59.003 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.003 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb57035:::benchmark_data_smithi043_119832_object74185:head# 2024-03-30T13:50:59.003 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.003 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb5b81b:::benchmark_data_smithi043_119832_object32653:head# 2024-03-30T13:50:59.003 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.004 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb679cf:::benchmark_data_smithi043_119832_object64776:head# 2024-03-30T13:50:59.004 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.004 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb6bec8:::benchmark_data_smithi043_119832_object60347:head# 2024-03-30T13:50:59.004 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.005 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb71756:::benchmark_data_smithi043_119832_object13889:head# 2024-03-30T13:50:59.005 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.005 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb74519:::benchmark_data_smithi043_119832_object3026:head# 2024-03-30T13:50:59.005 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.005 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb82340:::benchmark_data_smithi043_119832_object15491:head# 2024-03-30T13:50:59.005 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.006 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb85b9d:::benchmark_data_smithi043_119832_object65426:head# 2024-03-30T13:50:59.007 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.007 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb9b786:::benchmark_data_smithi043_119832_object47008:head# 2024-03-30T13:50:59.007 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.007 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb9c65b:::benchmark_data_smithi043_119832_object21287:head# 2024-03-30T13:50:59.007 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.007 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bb9f410:::benchmark_data_smithi043_119832_object44550:head# 2024-03-30T13:50:59.007 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.007 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bbac90b:::benchmark_data_smithi043_119832_object46216:head# 2024-03-30T13:50:59.007 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.008 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bbc6b91:::benchmark_data_smithi043_119832_object86838:head# 2024-03-30T13:50:59.008 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.008 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bbca2e6:::benchmark_data_smithi043_119832_object58999:head# 2024-03-30T13:50:59.008 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.008 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bbd8c43:::benchmark_data_smithi043_119832_object67312:head# 2024-03-30T13:50:59.008 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.008 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bbdec37:::benchmark_data_smithi043_119832_object36029:head# 2024-03-30T13:50:59.008 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.009 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bbecdd3:::benchmark_data_smithi043_119832_object29253:head# 2024-03-30T13:50:59.009 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.009 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bbeeece:::benchmark_data_smithi043_119832_object81801:head# 2024-03-30T13:50:59.009 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.009 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bbfc25e:::benchmark_data_smithi043_119832_object6378:head# 2024-03-30T13:50:59.009 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.010 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bc0a777:::benchmark_data_smithi043_119832_object29343:head# 2024-03-30T13:50:59.011 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.011 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bc0fc86:::benchmark_data_smithi043_119832_object7340:head# 2024-03-30T13:50:59.011 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.011 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bc1bf48:::benchmark_data_smithi043_119832_object78501:head# 2024-03-30T13:50:59.011 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.011 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bc1c96a:::benchmark_data_smithi043_119832_object55917:head# 2024-03-30T13:50:59.011 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.011 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bc40772:::benchmark_data_smithi043_119832_object78843:head# 2024-03-30T13:50:59.011 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.012 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bc4632b:::benchmark_data_smithi043_119832_object49530:head# 2024-03-30T13:50:59.012 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:59.013 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bc4b6bc:::benchmark_data_smithi043_119832_object46221:head# 2024-03-30T13:50:59.013 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.013 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bc6aa23:::benchmark_data_smithi043_119832_object4001:head# 2024-03-30T13:50:59.013 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.013 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bc7906e:::benchmark_data_smithi043_119832_object88298:head# 2024-03-30T13:50:59.013 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.013 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bc7bef6:::benchmark_data_smithi043_119832_object85581:head# 2024-03-30T13:50:59.014 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.014 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bc7d033:::benchmark_data_smithi043_119832_object73871:head# 2024-03-30T13:50:59.014 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.014 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:59.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:59.015 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bc7e058:::benchmark_data_smithi043_119832_object34109:head# 2024-03-30T13:50:59.015 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.015 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bc931e0:::benchmark_data_smithi043_119832_object72063:head# 2024-03-30T13:50:59.015 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.015 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bcb3cfa:::benchmark_data_smithi043_119832_object67179:head# 2024-03-30T13:50:59.015 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.015 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bcc4681:::benchmark_data_smithi043_119832_object6818:head# 2024-03-30T13:50:59.015 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.015 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bce4742:::benchmark_data_smithi043_119832_object73276:head# 2024-03-30T13:50:59.016 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.016 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bce647b:::benchmark_data_smithi043_119832_object74260:head# 2024-03-30T13:50:59.016 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.017 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bcf347d:::benchmark_data_smithi043_119832_object65964:head# 2024-03-30T13:50:59.017 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.017 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bd079fb:::benchmark_data_smithi043_119832_object82937:head# 2024-03-30T13:50:59.017 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.017 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bd09031:::benchmark_data_smithi043_119832_object15540:head# 2024-03-30T13:50:59.017 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.017 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bd26537:::benchmark_data_smithi043_119832_object83944:head# 2024-03-30T13:50:59.017 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.017 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bd2aea9:::benchmark_data_smithi043_119832_object7091:head# 2024-03-30T13:50:59.017 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.017 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bd31bc7:::benchmark_data_smithi043_119832_object40133:head# 2024-03-30T13:50:59.018 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.019 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bd43b75:::benchmark_data_smithi043_119832_object79803:head# 2024-03-30T13:50:59.019 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.019 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bd52ef0:::benchmark_data_smithi043_119832_object7201:head# 2024-03-30T13:50:59.019 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.019 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bd65644:::benchmark_data_smithi043_119832_object62138:head# 2024-03-30T13:50:59.019 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.019 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bd7c149:::benchmark_data_smithi043_119832_object35133:head# 2024-03-30T13:50:59.019 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.019 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bd88bf2:::benchmark_data_smithi043_119832_object33527:head# 2024-03-30T13:50:59.019 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.020 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bd8c318:::benchmark_data_smithi043_119832_object11503:head# 2024-03-30T13:50:59.020 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.020 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bd9034a:::benchmark_data_smithi043_119832_object41570:head# 2024-03-30T13:50:59.020 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.020 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bd97a10:::benchmark_data_smithi043_119832_object3518:head# 2024-03-30T13:50:59.020 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.021 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bdce855:::benchmark_data_smithi043_119832_object50138:head# 2024-03-30T13:50:59.021 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.021 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bdd6400:::benchmark_data_smithi043_119832_object66813:head# 2024-03-30T13:50:59.021 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.021 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bde2fcf:::benchmark_data_smithi043_119832_object66930:head# 2024-03-30T13:50:59.021 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.021 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7be046c5:::benchmark_data_smithi043_119832_object10811:head# 2024-03-30T13:50:59.021 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.022 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7be0a620:::benchmark_data_smithi043_119832_object8537:head# 2024-03-30T13:50:59.022 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.022 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7be1886e:::benchmark_data_smithi043_119832_object3075:head# 2024-03-30T13:50:59.022 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.023 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7be25009:::benchmark_data_smithi043_119832_object87851:head# 2024-03-30T13:50:59.023 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.023 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7be450d9:::benchmark_data_smithi043_119832_object73011:head# 2024-03-30T13:50:59.023 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.023 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7be65a22:::benchmark_data_smithi043_119832_object61151:head# 2024-03-30T13:50:59.023 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.024 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7be68b4c:::benchmark_data_smithi043_119832_object68266:head# 2024-03-30T13:50:59.024 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.024 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7be73f05:::benchmark_data_smithi043_119832_object81679:head# 2024-03-30T13:50:59.024 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.024 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7be76383:::benchmark_data_smithi043_119832_object82795:head# 2024-03-30T13:50:59.024 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.024 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7be77acc:::benchmark_data_smithi043_119832_object58167:head# 2024-03-30T13:50:59.025 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.025 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7be835df:::benchmark_data_smithi043_119832_object67293:head# 2024-03-30T13:50:59.025 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.025 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7be9c983:::benchmark_data_smithi043_119832_object18619:head# 2024-03-30T13:50:59.025 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.026 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7be9f921:::benchmark_data_smithi043_119832_object60021:head# 2024-03-30T13:50:59.026 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.026 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7beb5a05:::benchmark_data_smithi043_119832_object37899:head# 2024-03-30T13:50:59.026 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.027 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bebaaa3:::benchmark_data_smithi043_119832_object9381:head# 2024-03-30T13:50:59.027 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.028 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bebf676:::benchmark_data_smithi043_119832_object24252:head# 2024-03-30T13:50:59.028 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.028 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bed1ef1:::benchmark_data_smithi043_119832_object35946:head# 2024-03-30T13:50:59.028 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.028 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7beec492:::benchmark_data_smithi043_119832_object86976:head# 2024-03-30T13:50:59.028 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.029 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7beee53a:::benchmark_data_smithi043_119832_object76151:head# 2024-03-30T13:50:59.029 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.029 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7beff993:::benchmark_data_smithi043_119832_object78390:head# 2024-03-30T13:50:59.029 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.029 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bf0021c:::benchmark_data_smithi043_119832_object47797:head# 2024-03-30T13:50:59.029 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.030 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bf0b1eb:::benchmark_data_smithi043_119832_object54743:head# 2024-03-30T13:50:59.030 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.030 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bf0f6d0:::benchmark_data_smithi043_119832_object8017:head# 2024-03-30T13:50:59.030 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.030 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bf3624c:::benchmark_data_smithi043_119832_object37910:head# 2024-03-30T13:50:59.030 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.031 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bf40d53:::benchmark_data_smithi043_119832_object21618:head# 2024-03-30T13:50:59.031 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.031 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bf4423a:::benchmark_data_smithi043_119832_object17301:head# 2024-03-30T13:50:59.031 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.031 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bf45bb1:::benchmark_data_smithi043_119832_object32366:head# 2024-03-30T13:50:59.031 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.031 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bf4c051:::benchmark_data_smithi043_119832_object66741:head# 2024-03-30T13:50:59.031 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.032 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bf65eff:::benchmark_data_smithi043_119832_object13172:head# 2024-03-30T13:50:59.032 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.032 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bf69350:::benchmark_data_smithi043_119832_object56754:head# 2024-03-30T13:50:59.032 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.032 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bf803a6:::benchmark_data_smithi043_119832_object52589:head# 2024-03-30T13:50:59.033 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.033 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bf98ec0:::benchmark_data_smithi043_119832_object47162:head# 2024-03-30T13:50:59.033 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.033 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bf9aba8:::benchmark_data_smithi043_119832_object82:head# 2024-03-30T13:50:59.033 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.033 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bf9b82c:::benchmark_data_smithi043_119832_object34712:head# 2024-03-30T13:50:59.033 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.033 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bfa1177:::benchmark_data_smithi043_119832_object36784:head# 2024-03-30T13:50:59.033 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.034 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bfa5856:::benchmark_data_smithi043_119832_object47033:head# 2024-03-30T13:50:59.034 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.034 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bfbd943:::benchmark_data_smithi043_119832_object30695:head# 2024-03-30T13:50:59.034 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.034 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bfbdf39:::benchmark_data_smithi043_119832_object52366:head# 2024-03-30T13:50:59.035 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.035 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bfc88dc:::benchmark_data_smithi043_119832_object12446:head# 2024-03-30T13:50:59.035 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.035 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bfe46fd:::benchmark_data_smithi043_119832_object60243:head# 2024-03-30T13:50:59.035 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.035 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bfed1e2:::benchmark_data_smithi043_119832_object32352:head# 2024-03-30T13:50:59.035 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.036 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7bff15dc:::benchmark_data_smithi043_119832_object18217:head# 2024-03-30T13:50:59.036 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.036 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c000bbf:::benchmark_data_smithi043_119832_object85178:head# 2024-03-30T13:50:59.036 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.036 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c00751d:::benchmark_data_smithi043_119832_object4084:head# 2024-03-30T13:50:59.036 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.037 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c018c71:::benchmark_data_smithi043_119832_object994:head# 2024-03-30T13:50:59.037 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.037 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c019eb1:::benchmark_data_smithi043_119832_object22579:head# 2024-03-30T13:50:59.037 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.037 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c028cb0:::benchmark_data_smithi043_119832_object60713:head# 2024-03-30T13:50:59.037 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.037 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c047a5c:::benchmark_data_smithi043_119832_object24145:head# 2024-03-30T13:50:59.037 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.038 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c04aa8f:::benchmark_data_smithi043_119832_object36255:head# 2024-03-30T13:50:59.038 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.038 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c051250:::benchmark_data_smithi043_119832_object78302:head# 2024-03-30T13:50:59.038 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.039 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c059a6c:::benchmark_data_smithi043_119832_object15456:head# 2024-03-30T13:50:59.039 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.039 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c05ac25:::benchmark_data_smithi043_119832_object60745:head# 2024-03-30T13:50:59.039 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.040 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c05ff15:::benchmark_data_smithi043_119832_object33670:head# 2024-03-30T13:50:59.040 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.040 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c078015:::benchmark_data_smithi043_119832_object84425:head# 2024-03-30T13:50:59.040 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.040 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c07a7aa:::benchmark_data_smithi043_119832_object72024:head# 2024-03-30T13:50:59.040 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.040 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c07d7a9:::benchmark_data_smithi043_119832_object78607:head# 2024-03-30T13:50:59.040 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.041 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c08484d:::benchmark_data_smithi043_119832_object64719:head# 2024-03-30T13:50:59.041 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.041 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c0857ba:::benchmark_data_smithi043_119832_object64072:head# 2024-03-30T13:50:59.041 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.041 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c099413:::benchmark_data_smithi043_119832_object15232:head# 2024-03-30T13:50:59.041 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.042 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c0a97e4:::benchmark_data_smithi043_119832_object19283:head# 2024-03-30T13:50:59.042 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.042 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c0a98b5:::benchmark_data_smithi043_119832_object60681:head# 2024-03-30T13:50:59.042 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.042 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c0ab6e9:::benchmark_data_smithi043_119832_object87530:head# 2024-03-30T13:50:59.042 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.042 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c0b2149:::benchmark_data_smithi043_119832_object42454:head# 2024-03-30T13:50:59.042 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.043 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c0c896c:::benchmark_data_smithi043_119832_object16193:head# 2024-03-30T13:50:59.043 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.044 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c0d70d4:::benchmark_data_smithi043_119832_object12994:head# 2024-03-30T13:50:59.044 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.044 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c0e5e96:::benchmark_data_smithi043_119832_object70979:head# 2024-03-30T13:50:59.044 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.044 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c0e766c:::benchmark_data_smithi043_119832_object3010:head# 2024-03-30T13:50:59.044 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.045 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c0ebfb6:::benchmark_data_smithi043_119832_object59825:head# 2024-03-30T13:50:59.045 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.045 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c0ef27b:::benchmark_data_smithi043_119832_object29643:head# 2024-03-30T13:50:59.045 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.045 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c0f5b6c:::benchmark_data_smithi043_119832_object76939:head# 2024-03-30T13:50:59.045 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.045 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c120874:::benchmark_data_smithi043_119832_object26768:head# 2024-03-30T13:50:59.045 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.046 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c12b96d:::benchmark_data_smithi043_119832_object77124:head# 2024-03-30T13:50:59.046 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.046 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c12cb8e:::benchmark_data_smithi043_119832_object53644:head# 2024-03-30T13:50:59.046 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.046 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c13cff1:::benchmark_data_smithi043_119832_object52918:head# 2024-03-30T13:50:59.046 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.047 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c15416a:::benchmark_data_smithi043_119832_object73337:head# 2024-03-30T13:50:59.047 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.047 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c15afc7:::benchmark_data_smithi043_119832_object32672:head# 2024-03-30T13:50:59.047 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.047 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c162c2e:::benchmark_data_smithi043_119832_object59874:head# 2024-03-30T13:50:59.047 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.048 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c169ac1:::benchmark_data_smithi043_119832_object23046:head# 2024-03-30T13:50:59.048 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.048 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c1756f3:::benchmark_data_smithi043_119832_object37341:head# 2024-03-30T13:50:59.048 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.048 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c183beb:::benchmark_data_smithi043_119832_object82134:head# 2024-03-30T13:50:59.048 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.049 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c18bc2e:::benchmark_data_smithi043_119832_object33338:head# 2024-03-30T13:50:59.049 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.049 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c19193b:::benchmark_data_smithi043_119832_object60523:head# 2024-03-30T13:50:59.049 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.049 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c1ac6e9:::benchmark_data_smithi043_119832_object67679:head# 2024-03-30T13:50:59.049 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.049 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c1b1863:::benchmark_data_smithi043_119832_object73480:head# 2024-03-30T13:50:59.049 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.050 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c1c0d41:::benchmark_data_smithi043_119832_object19273:head# 2024-03-30T13:50:59.050 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.050 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c1c4595:::benchmark_data_smithi043_119832_object14068:head# 2024-03-30T13:50:59.050 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.050 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c1c94ff:::benchmark_data_smithi043_119832_object79922:head# 2024-03-30T13:50:59.051 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.051 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c1e0f45:::benchmark_data_smithi043_119832_object79459:head# 2024-03-30T13:50:59.051 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.051 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c1e29e7:::benchmark_data_smithi043_119832_object3725:head# 2024-03-30T13:50:59.051 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.051 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c1e9775:::benchmark_data_smithi043_119832_object1536:head# 2024-03-30T13:50:59.051 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.052 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c1ee621:::benchmark_data_smithi043_119832_object69316:head# 2024-03-30T13:50:59.052 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.052 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c1f4e4c:::benchmark_data_smithi043_119832_object48845:head# 2024-03-30T13:50:59.052 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.052 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c1fc7e3:::benchmark_data_smithi043_119832_object26717:head# 2024-03-30T13:50:59.053 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.053 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c1ffb6f:::benchmark_data_smithi043_119832_object48799:head# 2024-03-30T13:50:59.053 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.054 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c20cef5:::benchmark_data_smithi043_119832_object4570:head# 2024-03-30T13:50:59.054 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.054 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c211af3:::benchmark_data_smithi043_119832_object40109:head# 2024-03-30T13:50:59.054 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.054 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c232cb4:::benchmark_data_smithi043_119832_object45065:head# 2024-03-30T13:50:59.054 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.054 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c23617b:::benchmark_data_smithi043_119832_object45136:head# 2024-03-30T13:50:59.054 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.055 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c238134:::benchmark_data_smithi043_119832_object39044:head# 2024-03-30T13:50:59.055 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.055 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c23b389:::benchmark_data_smithi043_119832_object56686:head# 2024-03-30T13:50:59.055 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.055 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c24ea3c:::benchmark_data_smithi043_119832_object35280:head# 2024-03-30T13:50:59.055 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.055 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c2507c4:::benchmark_data_smithi043_119832_object75111:head# 2024-03-30T13:50:59.056 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.056 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c259355:::benchmark_data_smithi043_119832_object63176:head# 2024-03-30T13:50:59.056 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.056 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c25aecb:::benchmark_data_smithi043_119832_object69458:head# 2024-03-30T13:50:59.057 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.057 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c25d44e:::benchmark_data_smithi043_119832_object26057:head# 2024-03-30T13:50:59.057 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.057 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c26dc28:::benchmark_data_smithi043_119832_object73066:head# 2024-03-30T13:50:59.057 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.057 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c274346:::benchmark_data_smithi043_119832_object71454:head# 2024-03-30T13:50:59.057 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.058 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c2772fd:::benchmark_data_smithi043_119832_object20356:head# 2024-03-30T13:50:59.058 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.058 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c27ee08:::benchmark_data_smithi043_119832_object39215:head# 2024-03-30T13:50:59.058 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.058 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c28beb5:::benchmark_data_smithi043_119832_object62489:head# 2024-03-30T13:50:59.058 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.059 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c29b143:::benchmark_data_smithi043_119832_object33088:head# 2024-03-30T13:50:59.059 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.059 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c2a59e6:::benchmark_data_smithi043_119832_object3361:head# 2024-03-30T13:50:59.059 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.062 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c2a96e6:::benchmark_data_smithi043_119832_object7265:head# 2024-03-30T13:50:59.062 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.062 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c2b3437:::benchmark_data_smithi043_119832_object23931:head# 2024-03-30T13:50:59.063 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.063 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c2c1dc4:::benchmark_data_smithi043_119832_object67250:head# 2024-03-30T13:50:59.063 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.063 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c2c8538:::benchmark_data_smithi043_119832_object15796:head# 2024-03-30T13:50:59.063 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.063 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c2caee8:::benchmark_data_smithi043_119832_object52203:head# 2024-03-30T13:50:59.063 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.063 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c2cc39e:::benchmark_data_smithi043_119832_object51282:head# 2024-03-30T13:50:59.063 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.063 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c2cd96a:::benchmark_data_smithi043_119832_object68971:head# 2024-03-30T13:50:59.063 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.063 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c3163fc:::benchmark_data_smithi043_119832_object2371:head# 2024-03-30T13:50:59.063 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.063 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c31fa7b:::benchmark_data_smithi043_119832_object61338:head# 2024-03-30T13:50:59.063 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.065 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c335cf8:::benchmark_data_smithi043_119832_object47935:head# 2024-03-30T13:50:59.065 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.065 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c338cdf:::benchmark_data_smithi043_119832_object40548:head# 2024-03-30T13:50:59.065 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.065 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c3462ee:::benchmark_data_smithi043_119832_object88087:head# 2024-03-30T13:50:59.065 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.065 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c34e18f:::benchmark_data_smithi043_119832_object51362:head# 2024-03-30T13:50:59.065 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.065 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c3510b3:::benchmark_data_smithi043_119832_object42394:head# 2024-03-30T13:50:59.065 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.065 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c35a663:::benchmark_data_smithi043_119832_object12097:head# 2024-03-30T13:50:59.065 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.065 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c35dc53:::benchmark_data_smithi043_119832_object56977:head# 2024-03-30T13:50:59.065 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.067 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c36608d:::benchmark_data_smithi043_119832_object4721:head# 2024-03-30T13:50:59.067 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.067 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c36eeaa:::benchmark_data_smithi043_119832_object51575:head# 2024-03-30T13:50:59.067 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.067 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c371d2b:::benchmark_data_smithi043_119832_object72500:head# 2024-03-30T13:50:59.067 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.067 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c38810c:::benchmark_data_smithi043_119832_object48710:head# 2024-03-30T13:50:59.067 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.067 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c3cbbcf:::benchmark_data_smithi043_119832_object27965:head# 2024-03-30T13:50:59.067 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.067 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c3d9d62:::benchmark_data_smithi043_119832_object60121:head# 2024-03-30T13:50:59.067 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.068 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c3eac8b:::benchmark_data_smithi043_119832_object72952:head# 2024-03-30T13:50:59.068 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.069 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c407612:::benchmark_data_smithi043_119832_object68377:head# 2024-03-30T13:50:59.069 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.069 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c4125c8:::benchmark_data_smithi043_119832_object65766:head# 2024-03-30T13:50:59.069 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.069 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c41fd2c:::benchmark_data_smithi043_119832_object54698:head# 2024-03-30T13:50:59.069 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.069 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c427d1e:::benchmark_data_smithi043_119832_object2991:head# 2024-03-30T13:50:59.069 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.070 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c42d2b6:::benchmark_data_smithi043_119832_object84592:head# 2024-03-30T13:50:59.070 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.070 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c43607a:::benchmark_data_smithi043_119832_object11921:head# 2024-03-30T13:50:59.070 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.070 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c43a5d0:::benchmark_data_smithi043_119832_object39983:head# 2024-03-30T13:50:59.070 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.071 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c4486eb:::benchmark_data_smithi043_119832_object5750:head# 2024-03-30T13:50:59.071 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.071 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c452f24:::benchmark_data_smithi043_119832_object16445:head# 2024-03-30T13:50:59.071 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.072 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c45ae3c:::benchmark_data_smithi043_119832_object40008:head# 2024-03-30T13:50:59.072 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.072 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c478c2c:::benchmark_data_smithi043_119832_object81080:head# 2024-03-30T13:50:59.072 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.072 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c47999f:::benchmark_data_smithi043_119832_object52205:head# 2024-03-30T13:50:59.072 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.072 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c4934c9:::benchmark_data_smithi043_119832_object49896:head# 2024-03-30T13:50:59.072 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.073 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c4bf6d1:::benchmark_data_smithi043_119832_object78507:head# 2024-03-30T13:50:59.073 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.073 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c4c1f40:::benchmark_data_smithi043_119832_object5690:head# 2024-03-30T13:50:59.073 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.073 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c4d30b2:::benchmark_data_smithi043_119832_object87426:head# 2024-03-30T13:50:59.073 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.073 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c4fb7d6:::benchmark_data_smithi043_119832_object50914:head# 2024-03-30T13:50:59.073 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.075 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c507166:::benchmark_data_smithi043_119832_object37817:head# 2024-03-30T13:50:59.075 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.075 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c50af26:::benchmark_data_smithi043_119832_object13445:head# 2024-03-30T13:50:59.075 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.075 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c5404a2:::benchmark_data_smithi043_119832_object120:head# 2024-03-30T13:50:59.075 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.075 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c54065b:::benchmark_data_smithi043_119832_object54830:head# 2024-03-30T13:50:59.075 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.075 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c541bcd:::benchmark_data_smithi043_119832_object26184:head# 2024-03-30T13:50:59.075 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.076 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c544d36:::benchmark_data_smithi043_119832_object79632:head# 2024-03-30T13:50:59.076 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.076 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c55805a:::benchmark_data_smithi043_119832_object55836:head# 2024-03-30T13:50:59.076 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.076 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c563cd6:::benchmark_data_smithi043_119832_object77909:head# 2024-03-30T13:50:59.077 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.077 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c574232:::benchmark_data_smithi043_119832_object50288:head# 2024-03-30T13:50:59.077 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.077 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c579784:::benchmark_data_smithi043_119832_object25408:head# 2024-03-30T13:50:59.078 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.078 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c59d399:::benchmark_data_smithi043_119832_object60253:head# 2024-03-30T13:50:59.078 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.078 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c5a88a3:::benchmark_data_smithi043_119832_object9014:head# 2024-03-30T13:50:59.078 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.078 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c5b5d04:::benchmark_data_smithi043_119832_object58791:head# 2024-03-30T13:50:59.078 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.079 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c5bfd7f:::benchmark_data_smithi043_119832_object36360:head# 2024-03-30T13:50:59.079 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.079 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c5c04f8:::benchmark_data_smithi043_119832_object63773:head# 2024-03-30T13:50:59.079 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.079 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c5c89ab:::benchmark_data_smithi043_119832_object15200:head# 2024-03-30T13:50:59.079 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.080 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c5d5b63:::benchmark_data_smithi043_119832_object13698:head# 2024-03-30T13:50:59.080 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.080 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c5e394c:::benchmark_data_smithi043_119832_object481:head# 2024-03-30T13:50:59.080 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.080 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c5e73a3:::benchmark_data_smithi043_119832_object3154:head# 2024-03-30T13:50:59.080 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.080 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c60a9e5:::benchmark_data_smithi043_119832_object49443:head# 2024-03-30T13:50:59.080 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.082 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c611e09:::benchmark_data_smithi043_119832_object58996:head# 2024-03-30T13:50:59.082 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.083 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c61992e:::benchmark_data_smithi043_119832_object69520:head# 2024-03-30T13:50:59.083 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.083 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c6437b6:::benchmark_data_smithi043_119832_object3732:head# 2024-03-30T13:50:59.083 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.084 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c64ef11:::benchmark_data_smithi043_119832_object61180:head# 2024-03-30T13:50:59.084 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.084 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c6802bc:::benchmark_data_smithi043_119832_object56427:head# 2024-03-30T13:50:59.084 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.084 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c688dc7:::benchmark_data_smithi043_119832_object1559:head# 2024-03-30T13:50:59.084 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.085 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c699de7:::benchmark_data_smithi043_119832_object64264:head# 2024-03-30T13:50:59.086 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.086 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c69b350:::benchmark_data_smithi043_119832_object39186:head# 2024-03-30T13:50:59.086 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.086 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c6b62a9:::benchmark_data_smithi043_119832_object13434:head# 2024-03-30T13:50:59.086 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.086 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c6c74c3:::benchmark_data_smithi043_119832_object29687:head# 2024-03-30T13:50:59.086 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.087 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c6cd02e:::benchmark_data_smithi043_119832_object18129:head# 2024-03-30T13:50:59.087 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.087 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c6e85a0:::benchmark_data_smithi043_119832_object17536:head# 2024-03-30T13:50:59.087 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.087 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c6ed269:::benchmark_data_smithi043_119832_object12289:head# 2024-03-30T13:50:59.087 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.087 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c6f0233:::benchmark_data_smithi043_119832_object59422:head# 2024-03-30T13:50:59.087 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.088 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c6f7455:::benchmark_data_smithi043_119832_object86663:head# 2024-03-30T13:50:59.088 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.088 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c7045fa:::benchmark_data_smithi043_119832_object39793:head# 2024-03-30T13:50:59.088 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.089 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c707583:::benchmark_data_smithi043_119832_object9269:head# 2024-03-30T13:50:59.089 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.089 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c712d7c:::benchmark_data_smithi043_119832_object48933:head# 2024-03-30T13:50:59.089 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.089 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c715f80:::benchmark_data_smithi043_119832_object40465:head# 2024-03-30T13:50:59.089 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.089 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c719ce1:::benchmark_data_smithi043_119832_object77228:head# 2024-03-30T13:50:59.090 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.090 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c71eb03:::benchmark_data_smithi043_119832_object21648:head# 2024-03-30T13:50:59.090 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.090 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c725b0f:::benchmark_data_smithi043_119832_object38586:head# 2024-03-30T13:50:59.090 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.090 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c726c52:::benchmark_data_smithi043_119832_object54485:head# 2024-03-30T13:50:59.091 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.091 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c727ce5:::benchmark_data_smithi043_119832_object53575:head# 2024-03-30T13:50:59.091 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.091 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c728584:::benchmark_data_smithi043_119832_object76158:head# 2024-03-30T13:50:59.091 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.091 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c7327d5:::benchmark_data_smithi043_119832_object77981:head# 2024-03-30T13:50:59.091 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.092 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c742c68:::benchmark_data_smithi043_119832_object62632:head# 2024-03-30T13:50:59.092 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.092 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c75d788:::benchmark_data_smithi043_119832_object30633:head# 2024-03-30T13:50:59.092 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.092 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c7601fa:::benchmark_data_smithi043_119832_object31038:head# 2024-03-30T13:50:59.092 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.092 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c764fae:::benchmark_data_smithi043_119832_object54413:head# 2024-03-30T13:50:59.092 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.093 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c773007:::benchmark_data_smithi043_119832_object23386:head# 2024-03-30T13:50:59.093 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.093 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c77a417:::benchmark_data_smithi043_119832_object53641:head# 2024-03-30T13:50:59.093 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.093 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c79a5fd:::benchmark_data_smithi043_119832_object14776:head# 2024-03-30T13:50:59.093 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.093 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c7a6646:::benchmark_data_smithi043_119832_object50481:head# 2024-03-30T13:50:59.093 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.094 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c7aec2e:::benchmark_data_smithi043_119832_object71056:head# 2024-03-30T13:50:59.094 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.094 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c7b6f76:::benchmark_data_smithi043_119832_object64285:head# 2024-03-30T13:50:59.094 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.094 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c7d20ad:::benchmark_data_smithi043_119832_object29309:head# 2024-03-30T13:50:59.094 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.094 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c7d4e17:::benchmark_data_smithi043_119832_object68559:head# 2024-03-30T13:50:59.094 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.095 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c7d5037:::benchmark_data_smithi043_119832_object46573:head# 2024-03-30T13:50:59.095 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.095 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c809d5a:::benchmark_data_smithi043_119832_object40953:head# 2024-03-30T13:50:59.095 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.095 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c813321:::benchmark_data_smithi043_119832_object24534:head# 2024-03-30T13:50:59.095 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.096 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c827e2b:::benchmark_data_smithi043_119832_object46505:head# 2024-03-30T13:50:59.096 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.096 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c830a28:::benchmark_data_smithi043_119832_object17577:head# 2024-03-30T13:50:59.096 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.096 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c8350b0:::benchmark_data_smithi043_119832_object62212:head# 2024-03-30T13:50:59.096 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.096 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c8378f1:::benchmark_data_smithi043_119832_object80902:head# 2024-03-30T13:50:59.097 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.097 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c83d78f:::benchmark_data_smithi043_119832_object52751:head# 2024-03-30T13:50:59.097 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.097 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c84c4f9:::benchmark_data_smithi043_119832_object44999:head# 2024-03-30T13:50:59.098 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.098 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c852903:::benchmark_data_smithi043_119832_object3714:head# 2024-03-30T13:50:59.098 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.099 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c867ae4:::benchmark_data_smithi043_119832_object51306:head# 2024-03-30T13:50:59.099 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.099 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c8680e7:::benchmark_data_smithi043_119832_object61303:head# 2024-03-30T13:50:59.099 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.099 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c87293c:::benchmark_data_smithi043_119832_object58024:head# 2024-03-30T13:50:59.099 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.099 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c89a441:::benchmark_data_smithi043_119832_object62877:head# 2024-03-30T13:50:59.099 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.100 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c89ae7b:::benchmark_data_smithi043_119832_object49619:head# 2024-03-30T13:50:59.100 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.100 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c8bb90d:::benchmark_data_smithi043_119832_object82350:head# 2024-03-30T13:50:59.100 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.100 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c8d4d65:::benchmark_data_smithi043_119832_object63957:head# 2024-03-30T13:50:59.100 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.100 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c8dedd5:::benchmark_data_smithi043_119832_object9468:head# 2024-03-30T13:50:59.100 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.101 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c906b7a:::benchmark_data_smithi043_119832_object34709:head# 2024-03-30T13:50:59.101 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.101 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c9168f6:::benchmark_data_smithi043_119832_object41099:head# 2024-03-30T13:50:59.102 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.102 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c922c0e:::benchmark_data_smithi043_119832_object63971:head# 2024-03-30T13:50:59.102 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.102 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c93096b:::benchmark_data_smithi043_119832_object39706:head# 2024-03-30T13:50:59.102 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.103 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c9583e2:::benchmark_data_smithi043_119832_object2215:head# 2024-03-30T13:50:59.103 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.103 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c95c636:::benchmark_data_smithi043_119832_object15087:head# 2024-03-30T13:50:59.103 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.103 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c95f85b:::benchmark_data_smithi043_119832_object38806:head# 2024-03-30T13:50:59.103 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.103 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c96bedd:::benchmark_data_smithi043_119832_object709:head# 2024-03-30T13:50:59.103 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.104 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c97c573:::benchmark_data_smithi043_119832_object51726:head# 2024-03-30T13:50:59.104 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.104 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c98e87e:::benchmark_data_smithi043_119832_object54029:head# 2024-03-30T13:50:59.105 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.105 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c98f68a:::benchmark_data_smithi043_119832_object26682:head# 2024-03-30T13:50:59.105 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.105 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c99d5ed:::benchmark_data_smithi043_119832_object68080:head# 2024-03-30T13:50:59.105 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.105 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c9a5840:::benchmark_data_smithi043_119832_object78861:head# 2024-03-30T13:50:59.105 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.106 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c9a9bdf:::benchmark_data_smithi043_119832_object66797:head# 2024-03-30T13:50:59.106 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.106 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c9b65a4:::benchmark_data_smithi043_119832_object43256:head# 2024-03-30T13:50:59.106 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.106 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c9d72b8:::benchmark_data_smithi043_119832_object80729:head# 2024-03-30T13:50:59.106 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.106 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c9ead6d:::benchmark_data_smithi043_119832_object18735:head# 2024-03-30T13:50:59.107 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.107 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c9edd70:::benchmark_data_smithi043_119832_object59080:head# 2024-03-30T13:50:59.107 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.108 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7c9fb839:::benchmark_data_smithi043_119832_object82515:head# 2024-03-30T13:50:59.108 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.108 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ca00092:::benchmark_data_smithi043_119832_object46207:head# 2024-03-30T13:50:59.108 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.108 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ca23d89:::benchmark_data_smithi043_119832_object75543:head# 2024-03-30T13:50:59.108 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.108 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ca32c3d:::benchmark_data_smithi043_119832_object86063:head# 2024-03-30T13:50:59.108 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.108 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ca3ec24:::benchmark_data_smithi043_119832_object15760:head# 2024-03-30T13:50:59.108 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.109 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ca3fa8c:::benchmark_data_smithi043_119832_object79945:head# 2024-03-30T13:50:59.110 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.110 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ca4afd9:::benchmark_data_smithi043_119832_object5167:head# 2024-03-30T13:50:59.110 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.110 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ca62115:::benchmark_data_smithi043_119832_object53267:head# 2024-03-30T13:50:59.110 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.110 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ca670c9:::benchmark_data_smithi043_119832_object72810:head# 2024-03-30T13:50:59.110 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.110 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ca69898:::benchmark_data_smithi043_119832_object44339:head# 2024-03-30T13:50:59.110 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.112 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ca751dc:::benchmark_data_smithi043_119832_object87876:head# 2024-03-30T13:50:59.112 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.112 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ca7dd7b:::benchmark_data_smithi043_119832_object47265:head# 2024-03-30T13:50:59.112 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.112 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ca8f68b:::benchmark_data_smithi043_119832_object84633:head# 2024-03-30T13:50:59.112 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.112 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ca98245:::benchmark_data_smithi043_119832_object46197:head# 2024-03-30T13:50:59.112 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.112 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ca9a3a7:::benchmark_data_smithi043_119832_object61391:head# 2024-03-30T13:50:59.112 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.113 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ca9abba:::benchmark_data_smithi043_119832_object43709:head# 2024-03-30T13:50:59.113 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.113 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ca9ae36:::benchmark_data_smithi043_119832_object44861:head# 2024-03-30T13:50:59.113 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.114 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ca9edc8:::benchmark_data_smithi043_119832_object13016:head# 2024-03-30T13:50:59.114 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.114 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cab3a17:::benchmark_data_smithi043_119832_object76560:head# 2024-03-30T13:50:59.114 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.115 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cab7cc2:::benchmark_data_smithi043_119832_object12942:head# 2024-03-30T13:50:59.115 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.115 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cabb3ca:::benchmark_data_smithi043_119832_object19359:head# 2024-03-30T13:50:59.115 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.115 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cabe792:::benchmark_data_smithi043_119832_object43604:head# 2024-03-30T13:50:59.115 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.115 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cad117c:::benchmark_data_smithi043_119832_object64644:head# 2024-03-30T13:50:59.115 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.115 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cadf752:::benchmark_data_smithi043_119832_object50504:head# 2024-03-30T13:50:59.115 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.115 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:59.119 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cb059e0:::benchmark_data_smithi043_119832_object55766:head# 2024-03-30T13:50:59.119 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.119 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cb0fbef:::benchmark_data_smithi043_119832_object12104:head# 2024-03-30T13:50:59.119 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.119 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cb15d97:::benchmark_data_smithi043_119832_object30973:head# 2024-03-30T13:50:59.119 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.119 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cb1f235:::benchmark_data_smithi043_119832_object86754:head# 2024-03-30T13:50:59.119 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.119 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cb2efda:::benchmark_data_smithi043_119832_object84313:head# 2024-03-30T13:50:59.119 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.119 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cb380a5:::benchmark_data_smithi043_119832_object21212:head# 2024-03-30T13:50:59.119 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.119 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cb3cf37:::benchmark_data_smithi043_119832_object44458:head# 2024-03-30T13:50:59.119 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.119 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cb459eb:::benchmark_data_smithi043_119832_object81051:head# 2024-03-30T13:50:59.120 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.120 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cb4ab99:::benchmark_data_smithi043_119832_object28774:head# 2024-03-30T13:50:59.120 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.120 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cb4d99d:::benchmark_data_smithi043_119832_object13431:head# 2024-03-30T13:50:59.120 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.120 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cb5654e:::benchmark_data_smithi043_119832_object36256:head# 2024-03-30T13:50:59.120 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.121 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cb71ada:::benchmark_data_smithi043_119832_object16306:head# 2024-03-30T13:50:59.121 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.121 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cb75d5b:::benchmark_data_smithi043_119832_object84027:head# 2024-03-30T13:50:59.121 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.121 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cb886d8:::benchmark_data_smithi043_119832_object31307:head# 2024-03-30T13:50:59.121 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.122 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cb88828:::benchmark_data_smithi043_119832_object10881:head# 2024-03-30T13:50:59.122 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.122 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cb8a83a:::benchmark_data_smithi043_119832_object46205:head# 2024-03-30T13:50:59.122 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.122 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cb935fb:::benchmark_data_smithi043_119832_object33779:head# 2024-03-30T13:50:59.122 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.123 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cba35b6:::benchmark_data_smithi043_119832_object4181:head# 2024-03-30T13:50:59.123 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.123 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cba855e:::benchmark_data_smithi043_119832_object12431:head# 2024-03-30T13:50:59.123 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.123 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cba9073:::benchmark_data_smithi043_119832_object31890:head# 2024-03-30T13:50:59.123 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.123 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cbacee0:::benchmark_data_smithi043_119832_object82340:head# 2024-03-30T13:50:59.123 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.123 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cbcbf4c:::benchmark_data_smithi043_119832_object12943:head# 2024-03-30T13:50:59.124 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.124 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cbd80a6:::benchmark_data_smithi043_119832_object63884:head# 2024-03-30T13:50:59.124 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.124 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cbe383e:::benchmark_data_smithi043_119832_object69846:head# 2024-03-30T13:50:59.124 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.125 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cbe46c7:::benchmark_data_smithi043_119832_object1228:head# 2024-03-30T13:50:59.125 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.125 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cbe7335:::benchmark_data_smithi043_119832_object9151:head# 2024-03-30T13:50:59.125 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.125 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cbef52b:::benchmark_data_smithi043_119832_object60087:head# 2024-03-30T13:50:59.125 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.125 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cbfd431:::benchmark_data_smithi043_119832_object48861:head# 2024-03-30T13:50:59.126 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.126 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cc195cc:::benchmark_data_smithi043_119832_object15400:head# 2024-03-30T13:50:59.126 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.126 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cc1c2df:::benchmark_data_smithi043_119832_object68476:head# 2024-03-30T13:50:59.127 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.127 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cc24130:::benchmark_data_smithi043_119832_object25093:head# 2024-03-30T13:50:59.127 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.127 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cc300ff:::benchmark_data_smithi043_119832_object18068:head# 2024-03-30T13:50:59.127 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.127 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cc3c3d0:::benchmark_data_smithi043_119832_object77827:head# 2024-03-30T13:50:59.127 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.128 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cc3fe7e:::benchmark_data_smithi043_119832_object29372:head# 2024-03-30T13:50:59.128 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.128 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cc49b66:::benchmark_data_smithi043_119832_object29681:head# 2024-03-30T13:50:59.128 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.128 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cc571ad:::benchmark_data_smithi043_119832_object20966:head# 2024-03-30T13:50:59.128 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.129 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cc5e623:::benchmark_data_smithi043_119832_object63196:head# 2024-03-30T13:50:59.129 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.129 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cc60f02:::benchmark_data_smithi043_119832_object7871:head# 2024-03-30T13:50:59.129 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.129 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cc64a2a:::benchmark_data_smithi043_119832_object248:head# 2024-03-30T13:50:59.129 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.130 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cc78cd2:::benchmark_data_smithi043_119832_object76150:head# 2024-03-30T13:50:59.130 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.130 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cc84426:::benchmark_data_smithi043_119832_object17314:head# 2024-03-30T13:50:59.130 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.131 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cc90965:::benchmark_data_smithi043_119832_object81788:head# 2024-03-30T13:50:59.131 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.131 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cc94a57:::benchmark_data_smithi043_119832_object23993:head# 2024-03-30T13:50:59.131 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.131 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cc9acd3:::benchmark_data_smithi043_119832_object41426:head# 2024-03-30T13:50:59.132 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.132 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cc9bcc2:::benchmark_data_smithi043_119832_object7286:head# 2024-03-30T13:50:59.132 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.132 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ccaadda:::benchmark_data_smithi043_119832_object70314:head# 2024-03-30T13:50:59.132 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.132 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ccc2e1a:::benchmark_data_smithi043_119832_object83412:head# 2024-03-30T13:50:59.133 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.133 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cccf63b:::benchmark_data_smithi043_119832_object1656:head# 2024-03-30T13:50:59.133 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.133 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cce2dfb:::benchmark_data_smithi043_119832_object2028:head# 2024-03-30T13:50:59.133 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.134 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cd05f37:::benchmark_data_smithi043_119832_object67116:head# 2024-03-30T13:50:59.134 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.134 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cd0ca71:::benchmark_data_smithi043_119832_object35682:head# 2024-03-30T13:50:59.134 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.134 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cd12e33:::benchmark_data_smithi043_119832_object8467:head# 2024-03-30T13:50:59.134 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.135 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cd1487a:::benchmark_data_smithi043_119832_object41327:head# 2024-03-30T13:50:59.135 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.135 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cd20b42:::benchmark_data_smithi043_119832_object2220:head# 2024-03-30T13:50:59.135 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.135 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cd264de:::benchmark_data_smithi043_119832_object71210:head# 2024-03-30T13:50:59.136 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.136 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cd26db8:::benchmark_data_smithi043_119832_object56866:head# 2024-03-30T13:50:59.136 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.136 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cd2f0c1:::benchmark_data_smithi043_119832_object36026:head# 2024-03-30T13:50:59.136 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.136 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cd301f0:::benchmark_data_smithi043_119832_object1878:head# 2024-03-30T13:50:59.136 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.137 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cd3cde5:::benchmark_data_smithi043_119832_object37278:head# 2024-03-30T13:50:59.137 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.137 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cd4fb83:::benchmark_data_smithi043_119832_object64275:head# 2024-03-30T13:50:59.137 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.138 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cd51735:::benchmark_data_smithi043_119832_object78990:head# 2024-03-30T13:50:59.138 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.138 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cd53322:::benchmark_data_smithi043_119832_object79127:head# 2024-03-30T13:50:59.139 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.139 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cd591fb:::benchmark_data_smithi043_119832_object66134:head# 2024-03-30T13:50:59.139 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.139 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cd6f3fc:::benchmark_data_smithi043_119832_object24823:head# 2024-03-30T13:50:59.139 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.140 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cd89d5e:::benchmark_data_smithi043_119832_object43357:head# 2024-03-30T13:50:59.140 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.140 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cd9364f:::benchmark_data_smithi043_119832_object39611:head# 2024-03-30T13:50:59.140 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.140 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cd974c6:::benchmark_data_smithi043_119832_object5871:head# 2024-03-30T13:50:59.140 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.140 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cd9d80f:::benchmark_data_smithi043_119832_object77233:head# 2024-03-30T13:50:59.140 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.141 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cda98a4:::benchmark_data_smithi043_119832_object82875:head# 2024-03-30T13:50:59.141 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.141 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cdaf3cd:::benchmark_data_smithi043_119832_object21240:head# 2024-03-30T13:50:59.141 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.142 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cdb9462:::benchmark_data_smithi043_119832_object75155:head# 2024-03-30T13:50:59.142 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.142 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cdbcbc2:::benchmark_data_smithi043_119832_object86336:head# 2024-03-30T13:50:59.142 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.142 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cdc4651:::benchmark_data_smithi043_119832_object10846:head# 2024-03-30T13:50:59.142 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.143 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cdc54a5:::benchmark_data_smithi043_119832_object60716:head# 2024-03-30T13:50:59.143 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.143 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cdc8bba:::benchmark_data_smithi043_119832_object17012:head# 2024-03-30T13:50:59.143 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.143 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cdcb445:::benchmark_data_smithi043_119832_object26188:head# 2024-03-30T13:50:59.143 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.143 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cdcd45c:::benchmark_data_smithi043_119832_object26544:head# 2024-03-30T13:50:59.143 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.144 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cddf80f:::benchmark_data_smithi043_119832_object27355:head# 2024-03-30T13:50:59.144 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.145 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ce01f1e:::benchmark_data_smithi043_119832_object66091:head# 2024-03-30T13:50:59.145 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.145 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ce02f3d:::benchmark_data_smithi043_119832_object20146:head# 2024-03-30T13:50:59.145 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.145 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ce05ba3:::benchmark_data_smithi043_119832_object18147:head# 2024-03-30T13:50:59.145 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.145 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ce12aaa:::benchmark_data_smithi043_119832_object21873:head# 2024-03-30T13:50:59.145 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.145 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ce19b69:::benchmark_data_smithi043_119832_object30481:head# 2024-03-30T13:50:59.145 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.146 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ce1e935:::benchmark_data_smithi043_119832_object79066:head# 2024-03-30T13:50:59.146 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.146 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ce21444:::benchmark_data_smithi043_119832_object80639:head# 2024-03-30T13:50:59.146 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.147 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ce28483:::benchmark_data_smithi043_119832_object10417:head# 2024-03-30T13:50:59.147 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.147 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ce3fcc2:::benchmark_data_smithi043_119832_object26782:head# 2024-03-30T13:50:59.147 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.147 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ce444f0:::benchmark_data_smithi043_119832_object77973:head# 2024-03-30T13:50:59.147 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.147 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ce589b1:::benchmark_data_smithi043_119832_object55393:head# 2024-03-30T13:50:59.147 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.148 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ce62561:::benchmark_data_smithi043_119832_object86276:head# 2024-03-30T13:50:59.148 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.148 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ce75942:::benchmark_data_smithi043_119832_object9925:head# 2024-03-30T13:50:59.148 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.149 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ce8b449:::benchmark_data_smithi043_119832_object66026:head# 2024-03-30T13:50:59.149 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.149 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ce95289:::benchmark_data_smithi043_119832_object44436:head# 2024-03-30T13:50:59.149 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.149 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ce96354:::benchmark_data_smithi043_119832_object51841:head# 2024-03-30T13:50:59.149 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.149 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cea19af:::benchmark_data_smithi043_119832_object69876:head# 2024-03-30T13:50:59.149 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.149 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ceb0bf2:::benchmark_data_smithi043_119832_object19875:head# 2024-03-30T13:50:59.150 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.150 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ceba4a6:::benchmark_data_smithi043_119832_object82297:head# 2024-03-30T13:50:59.150 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.151 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cec30ed:::benchmark_data_smithi043_119832_object54735:head# 2024-03-30T13:50:59.151 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.151 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ced3986:::benchmark_data_smithi043_119832_object86963:head# 2024-03-30T13:50:59.151 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.151 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ced4fb4:::benchmark_data_smithi043_119832_object43522:head# 2024-03-30T13:50:59.152 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.152 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cedc2c7:::benchmark_data_smithi043_119832_object56533:head# 2024-03-30T13:50:59.152 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.152 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cede28c:::benchmark_data_smithi043_119832_object81940:head# 2024-03-30T13:50:59.152 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.152 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cee3ed6:::benchmark_data_smithi043_119832_object35241:head# 2024-03-30T13:50:59.152 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.152 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cee53df:::benchmark_data_smithi043_119832_object68110:head# 2024-03-30T13:50:59.152 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.153 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cee6188:::benchmark_data_smithi043_119832_object65471:head# 2024-03-30T13:50:59.153 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.153 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ceea12c:::benchmark_data_smithi043_119832_object71838:head# 2024-03-30T13:50:59.153 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.153 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cef1b45:::benchmark_data_smithi043_119832_object6607:head# 2024-03-30T13:50:59.154 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.154 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cefcd19:::benchmark_data_smithi043_119832_object41167:head# 2024-03-30T13:50:59.154 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.154 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ceff10c:::benchmark_data_smithi043_119832_object13690:head# 2024-03-30T13:50:59.155 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.155 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cf0b053:::benchmark_data_smithi043_119832_object73707:head# 2024-03-30T13:50:59.155 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.155 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cf1043a:::benchmark_data_smithi043_119832_object76545:head# 2024-03-30T13:50:59.155 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.155 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cf24f7e:::benchmark_data_smithi043_119832_object59921:head# 2024-03-30T13:50:59.155 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.156 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cf2fee1:::benchmark_data_smithi043_119832_object42270:head# 2024-03-30T13:50:59.156 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.156 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cf326b8:::benchmark_data_smithi043_119832_object27288:head# 2024-03-30T13:50:59.156 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.156 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cf753cd:::benchmark_data_smithi043_119832_object73303:head# 2024-03-30T13:50:59.156 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:59.157 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:59.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:59.158 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cf76242:::benchmark_data_smithi043_119832_object6035:head# 2024-03-30T13:50:59.159 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.159 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cf7d338:::benchmark_data_smithi043_119832_object39460:head# 2024-03-30T13:50:59.159 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.159 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cfa6e1e:::benchmark_data_smithi043_119832_object51286:head# 2024-03-30T13:50:59.159 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.159 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cfb0aa5:::benchmark_data_smithi043_119832_object5139:head# 2024-03-30T13:50:59.159 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.159 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cfb995b:::benchmark_data_smithi043_119832_object58840:head# 2024-03-30T13:50:59.159 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.159 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cfcb234:::benchmark_data_smithi043_119832_object61236:head# 2024-03-30T13:50:59.159 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.159 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cfcbfb2:::benchmark_data_smithi043_119832_object19880:head# 2024-03-30T13:50:59.159 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.160 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cfef25a:::benchmark_data_smithi043_119832_object50483:head# 2024-03-30T13:50:59.160 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.160 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cff0411:::benchmark_data_smithi043_119832_object11922:head# 2024-03-30T13:50:59.160 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.160 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cff8c50:::benchmark_data_smithi043_119832_object44465:head# 2024-03-30T13:50:59.160 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.160 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7cffe7a5:::benchmark_data_smithi043_119832_object44744:head# 2024-03-30T13:50:59.161 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.161 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d00e610:::benchmark_data_smithi043_119832_object20987:head# 2024-03-30T13:50:59.161 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.161 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d022bdd:::benchmark_data_smithi043_119832_object60113:head# 2024-03-30T13:50:59.161 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.161 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d02d692:::benchmark_data_smithi043_119832_object75564:head# 2024-03-30T13:50:59.161 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.162 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d04f2fb:::benchmark_data_smithi043_119832_object24353:head# 2024-03-30T13:50:59.162 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.162 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d04fbae:::benchmark_data_smithi043_119832_object8016:head# 2024-03-30T13:50:59.162 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.163 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d053f1e:::benchmark_data_smithi043_119832_object76537:head# 2024-03-30T13:50:59.163 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.163 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d072d7e:::benchmark_data_smithi043_119832_object4309:head# 2024-03-30T13:50:59.163 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.163 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d07ea77:::benchmark_data_smithi043_119832_object60090:head# 2024-03-30T13:50:59.163 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.164 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d082da0:::benchmark_data_smithi043_119832_object31830:head# 2024-03-30T13:50:59.164 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.164 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d096f54:::benchmark_data_smithi043_119832_object22652:head# 2024-03-30T13:50:59.164 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.164 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d0a4824:::benchmark_data_smithi043_119832_object48986:head# 2024-03-30T13:50:59.164 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.164 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d0ab0ca:::benchmark_data_smithi043_119832_object4295:head# 2024-03-30T13:50:59.164 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.165 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d0b3b9d:::benchmark_data_smithi043_119832_object85193:head# 2024-03-30T13:50:59.165 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.165 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d0b85dd:::benchmark_data_smithi043_119832_object20644:head# 2024-03-30T13:50:59.165 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.165 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d0c6d1c:::benchmark_data_smithi043_119832_object57555:head# 2024-03-30T13:50:59.165 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.166 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d0c87a7:::benchmark_data_smithi043_119832_object2775:head# 2024-03-30T13:50:59.166 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.166 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d0c998d:::benchmark_data_smithi043_119832_object46651:head# 2024-03-30T13:50:59.166 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.166 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d11203c:::benchmark_data_smithi043_119832_object12208:head# 2024-03-30T13:50:59.167 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.167 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d1153d3:::benchmark_data_smithi043_119832_object79543:head# 2024-03-30T13:50:59.167 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.167 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d11bba1:::benchmark_data_smithi043_119832_object25637:head# 2024-03-30T13:50:59.167 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.168 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d11e406:::benchmark_data_smithi043_119832_object43030:head# 2024-03-30T13:50:59.168 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.168 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d12459e:::benchmark_data_smithi043_119832_object39449:head# 2024-03-30T13:50:59.168 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.168 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d135223:::benchmark_data_smithi043_119832_object87198:head# 2024-03-30T13:50:59.168 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.169 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d14898b:::benchmark_data_smithi043_119832_object15442:head# 2024-03-30T13:50:59.169 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.169 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d157fc9:::benchmark_data_smithi043_119832_object84017:head# 2024-03-30T13:50:59.169 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.169 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d16e521:::benchmark_data_smithi043_119832_object73709:head# 2024-03-30T13:50:59.169 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.170 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d171b32:::benchmark_data_smithi043_119832_object70965:head# 2024-03-30T13:50:59.170 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.170 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d17a5bd:::benchmark_data_smithi043_119832_object7713:head# 2024-03-30T13:50:59.170 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.170 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d17d547:::benchmark_data_smithi043_119832_object81913:head# 2024-03-30T13:50:59.170 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.171 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d1961f7:::benchmark_data_smithi043_119832_object54546:head# 2024-03-30T13:50:59.171 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.171 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d19a1d6:::benchmark_data_smithi043_119832_object65320:head# 2024-03-30T13:50:59.171 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.171 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d19c5d7:::benchmark_data_smithi043_119832_object44486:head# 2024-03-30T13:50:59.171 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.172 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d1afe48:::benchmark_data_smithi043_119832_object84323:head# 2024-03-30T13:50:59.172 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.172 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d1b0558:::benchmark_data_smithi043_119832_object77147:head# 2024-03-30T13:50:59.172 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.172 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d1b9f92:::benchmark_data_smithi043_119832_object41084:head# 2024-03-30T13:50:59.172 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.173 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d1c5232:::benchmark_data_smithi043_119832_object3530:head# 2024-03-30T13:50:59.173 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.173 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d1ca4b0:::benchmark_data_smithi043_119832_object23461:head# 2024-03-30T13:50:59.173 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.173 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d1d0f72:::benchmark_data_smithi043_119832_object24504:head# 2024-03-30T13:50:59.174 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.174 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d1d841d:::benchmark_data_smithi043_119832_object55497:head# 2024-03-30T13:50:59.174 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.174 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d1e2d08:::benchmark_data_smithi043_119832_object13963:head# 2024-03-30T13:50:59.174 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.175 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d1e62f4:::benchmark_data_smithi043_119832_object85021:head# 2024-03-30T13:50:59.175 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.175 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d21427a:::benchmark_data_smithi043_119832_object68719:head# 2024-03-30T13:50:59.175 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.175 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d21ff58:::benchmark_data_smithi043_119832_object66975:head# 2024-03-30T13:50:59.175 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.175 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d2253ab:::benchmark_data_smithi043_119832_object76782:head# 2024-03-30T13:50:59.175 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.176 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d23cedb:::benchmark_data_smithi043_119832_object33927:head# 2024-03-30T13:50:59.176 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.176 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d25aca1:::benchmark_data_smithi043_119832_object41348:head# 2024-03-30T13:50:59.176 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.177 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d2656db:::benchmark_data_smithi043_119832_object59098:head# 2024-03-30T13:50:59.177 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.177 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d278101:::benchmark_data_smithi043_119832_object86047:head# 2024-03-30T13:50:59.177 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.177 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d278650:::benchmark_data_smithi043_119832_object81334:head# 2024-03-30T13:50:59.177 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.178 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d282409:::benchmark_data_smithi043_119832_object3404:head# 2024-03-30T13:50:59.178 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.178 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d28f271:::benchmark_data_smithi043_119832_object32610:head# 2024-03-30T13:50:59.178 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.178 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d293b5a:::benchmark_data_smithi043_119832_object27546:head# 2024-03-30T13:50:59.178 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.179 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d2a23dc:::benchmark_data_smithi043_119832_object79181:head# 2024-03-30T13:50:59.179 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.179 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d2a79b2:::benchmark_data_smithi043_119832_object38126:head# 2024-03-30T13:50:59.179 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.179 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d2aacaf:::benchmark_data_smithi043_119832_object32709:head# 2024-03-30T13:50:59.179 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.180 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d2bb581:::benchmark_data_smithi043_119832_object29433:head# 2024-03-30T13:50:59.180 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.180 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d2bbe58:::benchmark_data_smithi043_119832_object64908:head# 2024-03-30T13:50:59.180 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.180 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d2bdf03:::benchmark_data_smithi043_119832_object83806:head# 2024-03-30T13:50:59.181 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.181 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d2beba0:::benchmark_data_smithi043_119832_object28591:head# 2024-03-30T13:50:59.181 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.181 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d2c32a0:::benchmark_data_smithi043_119832_object78498:head# 2024-03-30T13:50:59.181 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.182 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d2e364b:::benchmark_data_smithi043_119832_object43889:head# 2024-03-30T13:50:59.182 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.182 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d2e4fc6:::benchmark_data_smithi043_119832_object17519:head# 2024-03-30T13:50:59.182 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.182 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d2f1887:::benchmark_data_smithi043_119832_object20976:head# 2024-03-30T13:50:59.183 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.183 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d327795:::benchmark_data_smithi043_119832_object78362:head# 2024-03-30T13:50:59.183 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.183 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d32c076:::benchmark_data_smithi043_119832_object1989:head# 2024-03-30T13:50:59.183 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.183 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d334f86:::benchmark_data_smithi043_119832_object83277:head# 2024-03-30T13:50:59.183 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.184 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d34fea7:::benchmark_data_smithi043_119832_object46732:head# 2024-03-30T13:50:59.184 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.184 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d353eb8:::benchmark_data_smithi043_119832_object68831:head# 2024-03-30T13:50:59.184 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.184 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d378da0:::benchmark_data_smithi043_119832_object30945:head# 2024-03-30T13:50:59.184 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.185 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d388ed2:::benchmark_data_smithi043_119832_object88491:head# 2024-03-30T13:50:59.185 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.185 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d3acbba:::benchmark_data_smithi043_119832_object87479:head# 2024-03-30T13:50:59.185 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.185 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d3b0102:::benchmark_data_smithi043_119832_object64180:head# 2024-03-30T13:50:59.186 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.186 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d3b2df6:::benchmark_data_smithi043_119832_object69659:head# 2024-03-30T13:50:59.186 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.186 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d3c11a6:::benchmark_data_smithi043_119832_object4010:head# 2024-03-30T13:50:59.186 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.186 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d3c5806:::benchmark_data_smithi043_119832_object21406:head# 2024-03-30T13:50:59.187 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.187 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d3cf71c:::benchmark_data_smithi043_119832_object76575:head# 2024-03-30T13:50:59.187 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.187 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d3f7bf5:::benchmark_data_smithi043_119832_object13701:head# 2024-03-30T13:50:59.187 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.188 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d40e02b:::benchmark_data_smithi043_119832_object3847:head# 2024-03-30T13:50:59.188 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.188 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d410d54:::benchmark_data_smithi043_119832_object44923:head# 2024-03-30T13:50:59.188 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.188 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d42aa2a:::benchmark_data_smithi043_119832_object45482:head# 2024-03-30T13:50:59.188 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.189 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d43ce4e:::benchmark_data_smithi043_119832_object62170:head# 2024-03-30T13:50:59.189 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.189 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d453356:::benchmark_data_smithi043_119832_object68546:head# 2024-03-30T13:50:59.189 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.189 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d473a16:::benchmark_data_smithi043_119832_object15682:head# 2024-03-30T13:50:59.189 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.190 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d47815a:::benchmark_data_smithi043_119832_object66326:head# 2024-03-30T13:50:59.190 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.190 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d4886df:::benchmark_data_smithi043_119832_object18277:head# 2024-03-30T13:50:59.190 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.190 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d499364:::benchmark_data_smithi043_119832_object47166:head# 2024-03-30T13:50:59.190 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.191 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d4aeb70:::benchmark_data_smithi043_119832_object60390:head# 2024-03-30T13:50:59.191 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.191 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d4b1bee:::benchmark_data_smithi043_119832_object6566:head# 2024-03-30T13:50:59.191 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.192 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d4b2c5c:::benchmark_data_smithi043_119832_object21808:head# 2024-03-30T13:50:59.192 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.192 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d4b69b6:::benchmark_data_smithi043_119832_object69739:head# 2024-03-30T13:50:59.192 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.192 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d4bd048:::benchmark_data_smithi043_119832_object68651:head# 2024-03-30T13:50:59.192 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.193 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d4caf8b:::benchmark_data_smithi043_119832_object29307:head# 2024-03-30T13:50:59.193 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.193 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d4ce3c8:::benchmark_data_smithi043_119832_object76645:head# 2024-03-30T13:50:59.193 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.194 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d4d4636:::benchmark_data_smithi043_119832_object36605:head# 2024-03-30T13:50:59.194 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.194 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d4d8d6e:::benchmark_data_smithi043_119832_object74288:head# 2024-03-30T13:50:59.194 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.194 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d4e855e:::benchmark_data_smithi043_119832_object3626:head# 2024-03-30T13:50:59.194 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.195 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d4f3bcf:::benchmark_data_smithi043_119832_object63859:head# 2024-03-30T13:50:59.195 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.195 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d4fffd8:::benchmark_data_smithi043_119832_object75862:head# 2024-03-30T13:50:59.195 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.195 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d50dbf5:::benchmark_data_smithi043_119832_object73722:head# 2024-03-30T13:50:59.195 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.195 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d50f01c:::benchmark_data_smithi043_119832_object43788:head# 2024-03-30T13:50:59.195 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.196 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d511434:::benchmark_data_smithi043_119832_object16675:head# 2024-03-30T13:50:59.196 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.197 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d51425e:::benchmark_data_smithi043_119832_object70219:head# 2024-03-30T13:50:59.197 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.198 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d550096:::benchmark_data_smithi043_119832_object55318:head# 2024-03-30T13:50:59.198 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.198 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d55dfc6:::benchmark_data_smithi043_119832_object80838:head# 2024-03-30T13:50:59.198 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.199 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d562f61:::benchmark_data_smithi043_119832_object56843:head# 2024-03-30T13:50:59.199 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.199 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d5634e8:::benchmark_data_smithi043_119832_object88303:head# 2024-03-30T13:50:59.199 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.199 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d57e8e7:::benchmark_data_smithi043_119832_object73882:head# 2024-03-30T13:50:59.199 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.200 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d57fe5c:::benchmark_data_smithi043_119832_object27037:head# 2024-03-30T13:50:59.200 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.200 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d587c96:::benchmark_data_smithi043_119832_object46147:head# 2024-03-30T13:50:59.200 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.200 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d5945d8:::benchmark_data_smithi043_119832_object74695:head# 2024-03-30T13:50:59.200 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.200 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d59658b:::benchmark_data_smithi043_119832_object33396:head# 2024-03-30T13:50:59.201 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.201 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d59afac:::benchmark_data_smithi043_119832_object2299:head# 2024-03-30T13:50:59.202 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.202 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d59fb59:::benchmark_data_smithi043_119832_object49620:head# 2024-03-30T13:50:59.202 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.202 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d5af1b2:::benchmark_data_smithi043_119832_object4072:head# 2024-03-30T13:50:59.202 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.202 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d5c57df:::benchmark_data_smithi043_119832_object74446:head# 2024-03-30T13:50:59.202 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.203 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d5ce1f6:::benchmark_data_smithi043_119832_object72330:head# 2024-03-30T13:50:59.203 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.203 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d5f527d:::benchmark_data_smithi043_119832_object45966:head# 2024-03-30T13:50:59.203 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.203 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d617dfe:::benchmark_data_smithi043_119832_object47289:head# 2024-03-30T13:50:59.203 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.203 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d618b64:::benchmark_data_smithi043_119832_object40406:head# 2024-03-30T13:50:59.204 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.204 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d61a68c:::benchmark_data_smithi043_119832_object56350:head# 2024-03-30T13:50:59.205 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.205 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d625cf7:::benchmark_data_smithi043_119832_object44274:head# 2024-03-30T13:50:59.205 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.205 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d640b70:::benchmark_data_smithi043_119832_object51363:head# 2024-03-30T13:50:59.205 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.205 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d6436f3:::benchmark_data_smithi043_119832_object50579:head# 2024-03-30T13:50:59.205 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.206 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d64d67f:::benchmark_data_smithi043_119832_object27074:head# 2024-03-30T13:50:59.206 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.206 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d65add2:::benchmark_data_smithi043_119832_object5312:head# 2024-03-30T13:50:59.206 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.206 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d65ef66:::benchmark_data_smithi043_119832_object78966:head# 2024-03-30T13:50:59.207 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.207 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d667b19:::benchmark_data_smithi043_119832_object78354:head# 2024-03-30T13:50:59.207 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.207 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d667d79:::benchmark_data_smithi043_119832_object32401:head# 2024-03-30T13:50:59.207 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.208 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d668fff:::benchmark_data_smithi043_119832_object39796:head# 2024-03-30T13:50:59.208 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.208 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d669305:::benchmark_data_smithi043_119832_object60241:head# 2024-03-30T13:50:59.208 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.208 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d66d7e9:::benchmark_data_smithi043_119832_object51392:head# 2024-03-30T13:50:59.208 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.209 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d676411:::benchmark_data_smithi043_119832_object73134:head# 2024-03-30T13:50:59.209 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.209 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d680f39:::benchmark_data_smithi043_119832_object11672:head# 2024-03-30T13:50:59.209 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.210 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d69f76c:::benchmark_data_smithi043_119832_object83179:head# 2024-03-30T13:50:59.210 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.210 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d6aa67d:::benchmark_data_smithi043_119832_object19193:head# 2024-03-30T13:50:59.210 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.210 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d6aee25:::benchmark_data_smithi043_119832_object65010:head# 2024-03-30T13:50:59.210 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.210 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d6c2708:::benchmark_data_smithi043_119832_object84945:head# 2024-03-30T13:50:59.211 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.211 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d6c3119:::benchmark_data_smithi043_119832_object57592:head# 2024-03-30T13:50:59.211 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.212 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d6c5662:::benchmark_data_smithi043_119832_object17951:head# 2024-03-30T13:50:59.212 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.212 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d6c9ea1:::benchmark_data_smithi043_119832_object69564:head# 2024-03-30T13:50:59.212 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.212 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d6cf41f:::benchmark_data_smithi043_119832_object54302:head# 2024-03-30T13:50:59.212 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.212 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d6d5cbd:::benchmark_data_smithi043_119832_object28813:head# 2024-03-30T13:50:59.212 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.212 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d6da292:::benchmark_data_smithi043_119832_object1291:head# 2024-03-30T13:50:59.212 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.213 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d6ef02c:::benchmark_data_smithi043_119832_object57763:head# 2024-03-30T13:50:59.213 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.213 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d6f283f:::benchmark_data_smithi043_119832_object57570:head# 2024-03-30T13:50:59.214 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.214 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d6f9d44:::benchmark_data_smithi043_119832_object74816:head# 2024-03-30T13:50:59.214 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.214 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d6ff4ce:::benchmark_data_smithi043_119832_object42855:head# 2024-03-30T13:50:59.214 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.215 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d708bd7:::benchmark_data_smithi043_119832_object61807:head# 2024-03-30T13:50:59.215 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.215 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d70a859:::benchmark_data_smithi043_119832_object8629:head# 2024-03-30T13:50:59.215 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.215 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d70ace4:::benchmark_data_smithi043_119832_object53194:head# 2024-03-30T13:50:59.215 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.215 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d712312:::benchmark_data_smithi043_119832_object35265:head# 2024-03-30T13:50:59.215 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.216 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d71cc78:::benchmark_data_smithi043_119832_object57837:head# 2024-03-30T13:50:59.216 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.217 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d7238db:::benchmark_data_smithi043_119832_object29735:head# 2024-03-30T13:50:59.217 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.217 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d735c4b:::benchmark_data_smithi043_119832_object9359:head# 2024-03-30T13:50:59.217 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.217 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d75f162:::benchmark_data_smithi043_119832_object83282:head# 2024-03-30T13:50:59.217 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.217 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d760d52:::benchmark_data_smithi043_119832_object6601:head# 2024-03-30T13:50:59.217 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.217 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d7667ef:::benchmark_data_smithi043_119832_object88104:head# 2024-03-30T13:50:59.217 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.218 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d77475e:::benchmark_data_smithi043_119832_object12526:head# 2024-03-30T13:50:59.219 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.219 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d7911c3:::benchmark_data_smithi043_119832_object61777:head# 2024-03-30T13:50:59.219 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.219 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d79777e:::benchmark_data_smithi043_119832_object33120:head# 2024-03-30T13:50:59.219 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.219 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d79b328:::benchmark_data_smithi043_119832_object18354:head# 2024-03-30T13:50:59.219 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.219 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d7b964f:::benchmark_data_smithi043_119832_object21278:head# 2024-03-30T13:50:59.219 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.220 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d7d1a72:::benchmark_data_smithi043_119832_object44897:head# 2024-03-30T13:50:59.220 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.220 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d7d7178:::benchmark_data_smithi043_119832_object3534:head# 2024-03-30T13:50:59.220 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.221 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d7db848:::benchmark_data_smithi043_119832_object25584:head# 2024-03-30T13:50:59.221 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.221 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d7e1995:::benchmark_data_smithi043_119832_object3406:head# 2024-03-30T13:50:59.221 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.221 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d7fcd4a:::benchmark_data_smithi043_119832_object13911:head# 2024-03-30T13:50:59.221 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.222 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d808035:::benchmark_data_smithi043_119832_object53451:head# 2024-03-30T13:50:59.222 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.222 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d8081b7:::benchmark_data_smithi043_119832_object78543:head# 2024-03-30T13:50:59.222 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.222 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d80fb37:::benchmark_data_smithi043_119832_object80941:head# 2024-03-30T13:50:59.222 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.222 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d80fe64:::benchmark_data_smithi043_119832_object72877:head# 2024-03-30T13:50:59.222 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.223 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d814a40:::benchmark_data_smithi043_119832_object1364:head# 2024-03-30T13:50:59.223 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.224 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d8173f9:::benchmark_data_smithi043_119832_object41930:head# 2024-03-30T13:50:59.224 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.224 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d81741f:::benchmark_data_smithi043_119832_object68613:head# 2024-03-30T13:50:59.224 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.224 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d828dfb:::benchmark_data_smithi043_119832_object29232:head# 2024-03-30T13:50:59.224 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.224 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d8300b5:::benchmark_data_smithi043_119832_object66320:head# 2024-03-30T13:50:59.224 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.224 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d83a12b:::benchmark_data_smithi043_119832_object61824:head# 2024-03-30T13:50:59.224 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.225 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d842775:::benchmark_data_smithi043_119832_object16545:head# 2024-03-30T13:50:59.225 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.225 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d8678e9:::benchmark_data_smithi043_119832_object75627:head# 2024-03-30T13:50:59.225 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.225 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d88cff9:::benchmark_data_smithi043_119832_object26590:head# 2024-03-30T13:50:59.226 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.226 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d8a8288:::benchmark_data_smithi043_119832_object49568:head# 2024-03-30T13:50:59.226 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.227 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d8a9b97:::benchmark_data_smithi043_119832_object77763:head# 2024-03-30T13:50:59.227 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.227 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d8b79f1:::benchmark_data_smithi043_119832_object15866:head# 2024-03-30T13:50:59.227 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.227 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d8c4488:::benchmark_data_smithi043_119832_object76672:head# 2024-03-30T13:50:59.227 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.227 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d8e0928:::benchmark_data_smithi043_119832_object82298:head# 2024-03-30T13:50:59.227 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.227 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d8e8342:::benchmark_data_smithi043_119832_object61196:head# 2024-03-30T13:50:59.227 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.228 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d8e8528:::benchmark_data_smithi043_119832_object34335:head# 2024-03-30T13:50:59.228 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.228 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d906b05:::benchmark_data_smithi043_119832_object42003:head# 2024-03-30T13:50:59.229 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.229 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d910ed3:::benchmark_data_smithi043_119832_object26650:head# 2024-03-30T13:50:59.229 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.229 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d9221b6:::benchmark_data_smithi043_119832_object88391:head# 2024-03-30T13:50:59.229 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.230 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d93cc8b:::benchmark_data_smithi043_119832_object80282:head# 2024-03-30T13:50:59.230 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.230 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d948974:::benchmark_data_smithi043_119832_object74568:head# 2024-03-30T13:50:59.230 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.230 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d96627b:::benchmark_data_smithi043_119832_object38197:head# 2024-03-30T13:50:59.230 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.230 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d96cfbb:::benchmark_data_smithi043_119832_object12090:head# 2024-03-30T13:50:59.230 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.231 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d976876:::benchmark_data_smithi043_119832_object88423:head# 2024-03-30T13:50:59.231 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.231 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d97bcd0:::benchmark_data_smithi043_119832_object13542:head# 2024-03-30T13:50:59.231 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.232 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d99cbf2:::benchmark_data_smithi043_119832_object44610:head# 2024-03-30T13:50:59.232 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.232 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d9a333d:::benchmark_data_smithi043_119832_object33918:head# 2024-03-30T13:50:59.232 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.233 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d9a7b8e:::benchmark_data_smithi043_119832_object77792:head# 2024-03-30T13:50:59.233 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.233 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d9abf81:::benchmark_data_smithi043_119832_object37726:head# 2024-03-30T13:50:59.233 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.233 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d9b4437:::benchmark_data_smithi043_119832_object51043:head# 2024-03-30T13:50:59.233 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.233 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d9b7c53:::benchmark_data_smithi043_119832_object60920:head# 2024-03-30T13:50:59.233 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.234 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d9c3ff6:::benchmark_data_smithi043_119832_object18758:head# 2024-03-30T13:50:59.234 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.235 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d9cd66d:::benchmark_data_smithi043_119832_object79266:head# 2024-03-30T13:50:59.235 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.235 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d9d412f:::benchmark_data_smithi043_119832_object7708:head# 2024-03-30T13:50:59.235 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.235 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d9db571:::benchmark_data_smithi043_119832_object81068:head# 2024-03-30T13:50:59.235 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.235 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d9e2842:::benchmark_data_smithi043_119832_object27914:head# 2024-03-30T13:50:59.235 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.236 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d9fb062:::benchmark_data_smithi043_119832_object20548:head# 2024-03-30T13:50:59.236 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.236 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d9fba38:::benchmark_data_smithi043_119832_object33475:head# 2024-03-30T13:50:59.236 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.237 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d9fd889:::benchmark_data_smithi043_119832_object81518:head# 2024-03-30T13:50:59.237 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.237 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7d9fe21e:::benchmark_data_smithi043_119832_object80298:head# 2024-03-30T13:50:59.237 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.237 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7da0d397:::benchmark_data_smithi043_119832_object27173:head# 2024-03-30T13:50:59.237 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.238 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7da2c8e9:::benchmark_data_smithi043_119832_object60749:head# 2024-03-30T13:50:59.238 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.238 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7da2e5c1:::benchmark_data_smithi043_119832_object20735:head# 2024-03-30T13:50:59.238 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.238 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7da3231b:::benchmark_data_smithi043_119832_object48421:head# 2024-03-30T13:50:59.238 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.238 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7da38602:::benchmark_data_smithi043_119832_object52919:head# 2024-03-30T13:50:59.238 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.239 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7da46a7b:::benchmark_data_smithi043_119832_object3113:head# 2024-03-30T13:50:59.239 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.240 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7da5b86c:::benchmark_data_smithi043_119832_object63474:head# 2024-03-30T13:50:59.240 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.240 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7da69114:::benchmark_data_smithi043_119832_object19847:head# 2024-03-30T13:50:59.240 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.240 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7da724a6:::benchmark_data_smithi043_119832_object75855:head# 2024-03-30T13:50:59.240 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.240 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7da7c618:::benchmark_data_smithi043_119832_object19127:head# 2024-03-30T13:50:59.240 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.241 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7da7d371:::benchmark_data_smithi043_119832_object44727:head# 2024-03-30T13:50:59.241 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.241 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7da8c7c5:::benchmark_data_smithi043_119832_object79893:head# 2024-03-30T13:50:59.241 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.241 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7da9faed:::benchmark_data_smithi043_119832_object84943:head# 2024-03-30T13:50:59.241 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.241 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7daaac3e:::benchmark_data_smithi043_119832_object3472:head# 2024-03-30T13:50:59.241 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.243 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dab3e9f:::benchmark_data_smithi043_119832_object5645:head# 2024-03-30T13:50:59.243 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.243 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dadeed2:::benchmark_data_smithi043_119832_object82584:head# 2024-03-30T13:50:59.243 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.243 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7daf8857:::benchmark_data_smithi043_119832_object77236:head# 2024-03-30T13:50:59.243 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.243 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dafea86:::benchmark_data_smithi043_119832_object69826:head# 2024-03-30T13:50:59.243 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.243 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7db05f06:::benchmark_data_smithi043_119832_object71328:head# 2024-03-30T13:50:59.243 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.244 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7db0eaaa:::benchmark_data_smithi043_119832_object71739:head# 2024-03-30T13:50:59.244 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.244 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7db12ed8:::benchmark_data_smithi043_119832_object34659:head# 2024-03-30T13:50:59.244 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.244 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7db3f317:::benchmark_data_smithi043_119832_object68677:head# 2024-03-30T13:50:59.245 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.245 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7db4c447:::benchmark_data_smithi043_119832_object85867:head# 2024-03-30T13:50:59.245 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.246 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7db4f453:::benchmark_data_smithi043_119832_object881:head# 2024-03-30T13:50:59.246 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.246 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7db4fd88:::benchmark_data_smithi043_119832_object49657:head# 2024-03-30T13:50:59.246 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.246 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7db58523:::benchmark_data_smithi043_119832_object35130:head# 2024-03-30T13:50:59.246 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.246 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7db5cc95:::benchmark_data_smithi043_119832_object15124:head# 2024-03-30T13:50:59.246 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.246 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7db64696:::benchmark_data_smithi043_119832_object68832:head# 2024-03-30T13:50:59.246 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.247 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7db69982:::benchmark_data_smithi043_119832_object79843:head# 2024-03-30T13:50:59.247 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.247 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7db796fe:::benchmark_data_smithi043_119832_object70107:head# 2024-03-30T13:50:59.248 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.248 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7db7c6c9:::benchmark_data_smithi043_119832_object39806:head# 2024-03-30T13:50:59.248 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.248 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7db7d878:::benchmark_data_smithi043_119832_object37339:head# 2024-03-30T13:50:59.248 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.249 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7db92a18:::benchmark_data_smithi043_119832_object33167:head# 2024-03-30T13:50:59.249 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.249 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7db96452:::benchmark_data_smithi043_119832_object85663:head# 2024-03-30T13:50:59.249 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.249 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dba483f:::benchmark_data_smithi043_119832_object84477:head# 2024-03-30T13:50:59.249 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.249 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dbaa0dc:::benchmark_data_smithi043_119832_object31879:head# 2024-03-30T13:50:59.249 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.250 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dbb224e:::benchmark_data_smithi043_119832_object2590:head# 2024-03-30T13:50:59.250 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.250 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dbb67fe:::benchmark_data_smithi043_119832_object59487:head# 2024-03-30T13:50:59.250 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.251 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dbbbfc3:::benchmark_data_smithi043_119832_object61112:head# 2024-03-30T13:50:59.251 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.251 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dbbe545:::benchmark_data_smithi043_119832_object33473:head# 2024-03-30T13:50:59.251 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.251 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dbc74f8:::benchmark_data_smithi043_119832_object54354:head# 2024-03-30T13:50:59.251 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.252 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dbcf116:::benchmark_data_smithi043_119832_object55000:head# 2024-03-30T13:50:59.252 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.252 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dbd9ac4:::benchmark_data_smithi043_119832_object46824:head# 2024-03-30T13:50:59.252 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.252 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dbda748:::benchmark_data_smithi043_119832_object63783:head# 2024-03-30T13:50:59.252 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.252 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dbe87c9:::benchmark_data_smithi043_119832_object48626:head# 2024-03-30T13:50:59.252 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.253 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dbe8bbd:::benchmark_data_smithi043_119832_object41776:head# 2024-03-30T13:50:59.254 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.254 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dbe98cc:::benchmark_data_smithi043_119832_object40149:head# 2024-03-30T13:50:59.254 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.254 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dbec3b9:::benchmark_data_smithi043_119832_object60936:head# 2024-03-30T13:50:59.254 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.254 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dbeeb57:::benchmark_data_smithi043_119832_object36212:head# 2024-03-30T13:50:59.254 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.254 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dbf9fb6:::benchmark_data_smithi043_119832_object71932:head# 2024-03-30T13:50:59.254 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.255 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dc00703:::benchmark_data_smithi043_119832_object83495:head# 2024-03-30T13:50:59.255 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.255 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dc1f24a:::benchmark_data_smithi043_119832_object25222:head# 2024-03-30T13:50:59.255 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.256 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dc2bd83:::benchmark_data_smithi043_119832_object33572:head# 2024-03-30T13:50:59.256 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.256 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dc2d009:::benchmark_data_smithi043_119832_object55850:head# 2024-03-30T13:50:59.256 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.256 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dc45673:::benchmark_data_smithi043_119832_object40726:head# 2024-03-30T13:50:59.256 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.257 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dc5cc1e:::benchmark_data_smithi043_119832_object55742:head# 2024-03-30T13:50:59.257 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.257 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dc66d4c:::benchmark_data_smithi043_119832_object6058:head# 2024-03-30T13:50:59.257 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.257 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dc8312d:::benchmark_data_smithi043_119832_object47849:head# 2024-03-30T13:50:59.257 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.257 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dc849d8:::benchmark_data_smithi043_119832_object40314:head# 2024-03-30T13:50:59.257 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.257 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dc945a7:::benchmark_data_smithi043_119832_object41306:head# 2024-03-30T13:50:59.258 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.258 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:50:59.259 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dcb9a32:::benchmark_data_smithi043_119832_object52173:head# 2024-03-30T13:50:59.259 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.260 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dcc6160:::benchmark_data_smithi043_119832_object79918:head# 2024-03-30T13:50:59.260 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.260 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dcd4d15:::benchmark_data_smithi043_119832_object64052:head# 2024-03-30T13:50:59.260 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.260 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dce93ff:::benchmark_data_smithi043_119832_object57228:head# 2024-03-30T13:50:59.260 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.260 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dcf5ac7:::benchmark_data_smithi043_119832_object48946:head# 2024-03-30T13:50:59.260 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.260 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dcfd991:::benchmark_data_smithi043_119832_object25860:head# 2024-03-30T13:50:59.260 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.260 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd00e34:::benchmark_data_smithi043_119832_object18450:head# 2024-03-30T13:50:59.260 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.261 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd1f3ef:::benchmark_data_smithi043_119832_object36843:head# 2024-03-30T13:50:59.262 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.262 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd25ecf:::benchmark_data_smithi043_119832_object62891:head# 2024-03-30T13:50:59.262 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.262 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd2e710:::benchmark_data_smithi043_119832_object36776:head# 2024-03-30T13:50:59.262 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.262 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd31772:::benchmark_data_smithi043_119832_object56265:head# 2024-03-30T13:50:59.262 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.262 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd428fe:::benchmark_data_smithi043_119832_object13478:head# 2024-03-30T13:50:59.262 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.262 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd4b319:::benchmark_data_smithi043_119832_object64361:head# 2024-03-30T13:50:59.262 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.262 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd4c915:::benchmark_data_smithi043_119832_object71983:head# 2024-03-30T13:50:59.262 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.264 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd5182e:::benchmark_data_smithi043_119832_object13068:head# 2024-03-30T13:50:59.264 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.264 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd5fa66:::benchmark_data_smithi043_119832_object39041:head# 2024-03-30T13:50:59.264 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.264 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd72982:::benchmark_data_smithi043_119832_object52264:head# 2024-03-30T13:50:59.264 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.264 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd73ac0:::benchmark_data_smithi043_119832_object30853:head# 2024-03-30T13:50:59.264 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.265 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd73dae:::benchmark_data_smithi043_119832_object25311:head# 2024-03-30T13:50:59.265 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.265 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd758be:::benchmark_data_smithi043_119832_object71943:head# 2024-03-30T13:50:59.265 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.265 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd762c6:::benchmark_data_smithi043_119832_object67324:head# 2024-03-30T13:50:59.265 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.265 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd7c4df:::benchmark_data_smithi043_119832_object58408:head# 2024-03-30T13:50:59.265 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.266 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd7e086:::benchmark_data_smithi043_119832_object88085:head# 2024-03-30T13:50:59.266 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.266 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd7f0a0:::benchmark_data_smithi043_119832_object34069:head# 2024-03-30T13:50:59.266 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.266 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd83108:::benchmark_data_smithi043_119832_object6024:head# 2024-03-30T13:50:59.266 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.266 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd868d7:::benchmark_data_smithi043_119832_object81363:head# 2024-03-30T13:50:59.266 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.266 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd8aa84:::benchmark_data_smithi043_119832_object51692:head# 2024-03-30T13:50:59.266 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.266 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dd9eb91:::benchmark_data_smithi043_119832_object14391:head# 2024-03-30T13:50:59.266 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.267 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ddcc70b:::benchmark_data_smithi043_119832_object36775:head# 2024-03-30T13:50:59.267 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.267 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dde2507:::benchmark_data_smithi043_119832_object5862:head# 2024-03-30T13:50:59.267 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.267 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dde6f4e:::benchmark_data_smithi043_119832_object55526:head# 2024-03-30T13:50:59.268 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.268 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ddfed85:::benchmark_data_smithi043_119832_object85745:head# 2024-03-30T13:50:59.268 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.268 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7de110ff:::benchmark_data_smithi043_119832_object26708:head# 2024-03-30T13:50:59.268 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.269 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7de13348:::benchmark_data_smithi043_119832_object4929:head# 2024-03-30T13:50:59.269 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.269 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7de14cbf:::benchmark_data_smithi043_119832_object69173:head# 2024-03-30T13:50:59.269 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.269 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7de298ad:::benchmark_data_smithi043_119832_object85670:head# 2024-03-30T13:50:59.269 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.269 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7de2bc60:::benchmark_data_smithi043_119832_object87994:head# 2024-03-30T13:50:59.269 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.270 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7de3d1a8:::benchmark_data_smithi043_119832_object24567:head# 2024-03-30T13:50:59.270 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.270 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7de3ef8f:::benchmark_data_smithi043_119832_object13266:head# 2024-03-30T13:50:59.270 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.270 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7de52e66:::benchmark_data_smithi043_119832_object31705:head# 2024-03-30T13:50:59.270 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.271 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7de574be:::benchmark_data_smithi043_119832_object71288:head# 2024-03-30T13:50:59.271 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.271 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7de5ac2f:::benchmark_data_smithi043_119832_object61735:head# 2024-03-30T13:50:59.271 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.272 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7de5e97a:::benchmark_data_smithi043_119832_object57310:head# 2024-03-30T13:50:59.272 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.273 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7de753f9:::benchmark_data_smithi043_119832_object43391:head# 2024-03-30T13:50:59.273 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.273 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7de7845c:::benchmark_data_smithi043_119832_object35059:head# 2024-03-30T13:50:59.273 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.273 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7de7d175:::benchmark_data_smithi043_119832_object4644:head# 2024-03-30T13:50:59.273 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.273 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7de7ee93:::benchmark_data_smithi043_119832_object45626:head# 2024-03-30T13:50:59.273 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.274 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7de86c87:::benchmark_data_smithi043_119832_object42619:head# 2024-03-30T13:50:59.274 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.274 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7de88b73:::benchmark_data_smithi043_119832_object65554:head# 2024-03-30T13:50:59.274 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.274 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7de8c4b3:::benchmark_data_smithi043_119832_object73434:head# 2024-03-30T13:50:59.274 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.275 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7de8f2a6:::benchmark_data_smithi043_119832_object47971:head# 2024-03-30T13:50:59.275 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.275 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7de9012f:::benchmark_data_smithi043_119832_object66255:head# 2024-03-30T13:50:59.276 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.276 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7deaf0f9:::benchmark_data_smithi043_119832_object80593:head# 2024-03-30T13:50:59.276 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.276 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7deb0de7:::benchmark_data_smithi043_119832_object643:head# 2024-03-30T13:50:59.276 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.276 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dec04b9:::benchmark_data_smithi043_119832_object62604:head# 2024-03-30T13:50:59.276 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.277 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dec83ab:::benchmark_data_smithi043_119832_object49719:head# 2024-03-30T13:50:59.277 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.277 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7deca62a:::benchmark_data_smithi043_119832_object83360:head# 2024-03-30T13:50:59.277 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.277 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7deccbba:::benchmark_data_smithi043_119832_object85219:head# 2024-03-30T13:50:59.277 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.278 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ded9c49:::benchmark_data_smithi043_119832_object46764:head# 2024-03-30T13:50:59.278 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.279 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7deec61d:::benchmark_data_smithi043_119832_object12548:head# 2024-03-30T13:50:59.279 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.279 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7deeec61:::benchmark_data_smithi043_119832_object28803:head# 2024-03-30T13:50:59.279 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.279 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7def15f2:::benchmark_data_smithi043_119832_object44327:head# 2024-03-30T13:50:59.279 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.279 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7def8c6b:::benchmark_data_smithi043_119832_object14793:head# 2024-03-30T13:50:59.279 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.279 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7df00095:::benchmark_data_smithi043_119832_object48052:head# 2024-03-30T13:50:59.280 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.281 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7df0df5b:::benchmark_data_smithi043_119832_object66733:head# 2024-03-30T13:50:59.281 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.281 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7df1006e:::benchmark_data_smithi043_119832_object31900:head# 2024-03-30T13:50:59.281 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.281 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7df13cd7:::benchmark_data_smithi043_119832_object3328:head# 2024-03-30T13:50:59.281 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.281 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7df16327:::benchmark_data_smithi043_119832_object81041:head# 2024-03-30T13:50:59.281 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.281 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7df16578:::benchmark_data_smithi043_119832_object62647:head# 2024-03-30T13:50:59.281 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.283 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7df2c9fe:::benchmark_data_smithi043_119832_object29156:head# 2024-03-30T13:50:59.283 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.283 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7df2fe36:::benchmark_data_smithi043_119832_object6027:head# 2024-03-30T13:50:59.283 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.283 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7df31c9f:::benchmark_data_smithi043_119832_object86163:head# 2024-03-30T13:50:59.283 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.283 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7df3a787:::benchmark_data_smithi043_119832_object62946:head# 2024-03-30T13:50:59.283 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.283 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7df48d1a:::benchmark_data_smithi043_119832_object12876:head# 2024-03-30T13:50:59.283 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.284 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7df68789:::benchmark_data_smithi043_119832_object87816:head# 2024-03-30T13:50:59.284 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.285 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7df7b521:::benchmark_data_smithi043_119832_object9371:head# 2024-03-30T13:50:59.285 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.285 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7df7e0d1:::benchmark_data_smithi043_119832_object4536:head# 2024-03-30T13:50:59.285 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.285 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7df8ada7:::benchmark_data_smithi043_119832_object53306:head# 2024-03-30T13:50:59.285 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.285 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7df8ee4d:::benchmark_data_smithi043_119832_object85435:head# 2024-03-30T13:50:59.285 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.286 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7df901e3:::benchmark_data_smithi043_119832_object41500:head# 2024-03-30T13:50:59.286 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.286 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dfa3417:::benchmark_data_smithi043_119832_object38472:head# 2024-03-30T13:50:59.286 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.287 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dfa9e92:::benchmark_data_smithi043_119832_object56451:head# 2024-03-30T13:50:59.287 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.287 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dfc7089:::benchmark_data_smithi043_119832_object7880:head# 2024-03-30T13:50:59.287 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.287 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dfc7a95:::benchmark_data_smithi043_119832_object48879:head# 2024-03-30T13:50:59.287 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.288 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dfca142:::benchmark_data_smithi043_119832_object65680:head# 2024-03-30T13:50:59.288 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.288 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dfd13d6:::benchmark_data_smithi043_119832_object46700:head# 2024-03-30T13:50:59.288 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.288 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dfd39db:::benchmark_data_smithi043_119832_object18590:head# 2024-03-30T13:50:59.289 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.289 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dfddd99:::benchmark_data_smithi043_119832_object41165:head# 2024-03-30T13:50:59.289 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.289 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7dff9a43:::benchmark_data_smithi043_119832_object60685:head# 2024-03-30T13:50:59.289 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.290 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e006412:::benchmark_data_smithi043_119832_object50096:head# 2024-03-30T13:50:59.290 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.290 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e006ac8:::benchmark_data_smithi043_119832_object72589:head# 2024-03-30T13:50:59.290 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.290 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e00a23b:::benchmark_data_smithi043_119832_object70683:head# 2024-03-30T13:50:59.291 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.291 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e00d335:::benchmark_data_smithi043_119832_object67666:head# 2024-03-30T13:50:59.291 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.291 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e0130a9:::benchmark_data_smithi043_119832_object37357:head# 2024-03-30T13:50:59.291 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.291 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e01c0a5:::benchmark_data_smithi043_119832_object24976:head# 2024-03-30T13:50:59.291 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.292 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e01d508:::benchmark_data_smithi043_119832_object86196:head# 2024-03-30T13:50:59.292 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.292 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e0402a2:::benchmark_data_smithi043_119832_object12279:head# 2024-03-30T13:50:59.292 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.292 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e05a3d4:::benchmark_data_smithi043_119832_object39178:head# 2024-03-30T13:50:59.292 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.292 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e0766dc:::benchmark_data_smithi043_119832_object78358:head# 2024-03-30T13:50:59.293 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.293 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e07f41d:::benchmark_data_smithi043_119832_object11511:head# 2024-03-30T13:50:59.293 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:59.294 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:59.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:50:59.296 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e086090:::benchmark_data_smithi043_119832_object88003:head# 2024-03-30T13:50:59.296 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.296 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e08ce93:::benchmark_data_smithi043_119832_object83671:head# 2024-03-30T13:50:59.296 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.296 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e08e20a:::benchmark_data_smithi043_119832_object16665:head# 2024-03-30T13:50:59.296 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.296 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e0b924e:::benchmark_data_smithi043_119832_object34609:head# 2024-03-30T13:50:59.296 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.297 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e0d8542:::benchmark_data_smithi043_119832_object75831:head# 2024-03-30T13:50:59.297 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.297 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e0e616a:::benchmark_data_smithi043_119832_object57127:head# 2024-03-30T13:50:59.297 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.297 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e0f0329:::benchmark_data_smithi043_119832_object9602:head# 2024-03-30T13:50:59.297 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.297 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e115996:::benchmark_data_smithi043_119832_object70241:head# 2024-03-30T13:50:59.297 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.297 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e13b47d:::benchmark_data_smithi043_119832_object17394:head# 2024-03-30T13:50:59.297 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.297 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e13df1b:::benchmark_data_smithi043_119832_object39102:head# 2024-03-30T13:50:59.297 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.297 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e144382:::benchmark_data_smithi043_119832_object35463:head# 2024-03-30T13:50:59.297 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.299 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e145579:::benchmark_data_smithi043_119832_object79504:head# 2024-03-30T13:50:59.300 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.300 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e15e106:::benchmark_data_smithi043_119832_object9191:head# 2024-03-30T13:50:59.300 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.300 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e165238:::benchmark_data_smithi043_119832_object60472:head# 2024-03-30T13:50:59.300 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.300 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e168b92:::benchmark_data_smithi043_119832_object5579:head# 2024-03-30T13:50:59.300 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.300 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e16ad7c:::benchmark_data_smithi043_119832_object74413:head# 2024-03-30T13:50:59.300 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.300 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e16c966:::benchmark_data_smithi043_119832_object82172:head# 2024-03-30T13:50:59.300 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.300 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e173046:::benchmark_data_smithi043_119832_object14699:head# 2024-03-30T13:50:59.300 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.300 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e185945:::benchmark_data_smithi043_119832_object28833:head# 2024-03-30T13:50:59.301 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.301 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e19bda5:::benchmark_data_smithi043_119832_object43666:head# 2024-03-30T13:50:59.301 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.301 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e1ab598:::benchmark_data_smithi043_119832_object36903:head# 2024-03-30T13:50:59.301 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.302 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e1ae7a8:::benchmark_data_smithi043_119832_object945:head# 2024-03-30T13:50:59.303 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.303 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e1b216a:::benchmark_data_smithi043_119832_object5572:head# 2024-03-30T13:50:59.303 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.303 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e1bd9dc:::benchmark_data_smithi043_119832_object48241:head# 2024-03-30T13:50:59.303 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.303 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e1cca44:::benchmark_data_smithi043_119832_object48184:head# 2024-03-30T13:50:59.303 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.303 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e1d50ce:::benchmark_data_smithi043_119832_object42691:head# 2024-03-30T13:50:59.303 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.303 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e1dfb11:::benchmark_data_smithi043_119832_object73379:head# 2024-03-30T13:50:59.303 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.303 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e1e5d29:::benchmark_data_smithi043_119832_object15564:head# 2024-03-30T13:50:59.303 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.303 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e1f9aee:::benchmark_data_smithi043_119832_object26909:head# 2024-03-30T13:50:59.304 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.305 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e20b5de:::benchmark_data_smithi043_119832_object62372:head# 2024-03-30T13:50:59.305 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.305 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e20efae:::benchmark_data_smithi043_119832_object742:head# 2024-03-30T13:50:59.305 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.306 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e20f1ad:::benchmark_data_smithi043_119832_object26328:head# 2024-03-30T13:50:59.306 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.306 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e213bf3:::benchmark_data_smithi043_119832_object11250:head# 2024-03-30T13:50:59.306 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.306 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e215dbf:::benchmark_data_smithi043_119832_object57432:head# 2024-03-30T13:50:59.306 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.306 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e233bee:::benchmark_data_smithi043_119832_object37882:head# 2024-03-30T13:50:59.306 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.306 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e241828:::benchmark_data_smithi043_119832_object68367:head# 2024-03-30T13:50:59.306 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.306 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e24d74f:::benchmark_data_smithi043_119832_object75479:head# 2024-03-30T13:50:59.306 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.308 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e251e83:::benchmark_data_smithi043_119832_object79124:head# 2024-03-30T13:50:59.308 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.308 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e25570c:::benchmark_data_smithi043_119832_object39534:head# 2024-03-30T13:50:59.308 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.308 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e259382:::benchmark_data_smithi043_119832_object53637:head# 2024-03-30T13:50:59.308 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.308 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e25a238:::benchmark_data_smithi043_119832_object63428:head# 2024-03-30T13:50:59.308 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.308 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e25aa32:::benchmark_data_smithi043_119832_object35933:head# 2024-03-30T13:50:59.308 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.309 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e25deae:::benchmark_data_smithi043_119832_object69226:head# 2024-03-30T13:50:59.309 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.309 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e263521:::benchmark_data_smithi043_119832_object58973:head# 2024-03-30T13:50:59.309 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.310 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e273684:::benchmark_data_smithi043_119832_object51703:head# 2024-03-30T13:50:59.310 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.310 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e279bd8:::benchmark_data_smithi043_119832_object24092:head# 2024-03-30T13:50:59.310 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.310 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e28d467:::benchmark_data_smithi043_119832_object58205:head# 2024-03-30T13:50:59.310 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.310 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e2a5f08:::benchmark_data_smithi043_119832_object59294:head# 2024-03-30T13:50:59.310 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.310 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e2c6044:::benchmark_data_smithi043_119832_object20621:head# 2024-03-30T13:50:59.310 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.311 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e2db6db:::benchmark_data_smithi043_119832_object49122:head# 2024-03-30T13:50:59.311 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.311 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e2e578f:::benchmark_data_smithi043_119832_object56169:head# 2024-03-30T13:50:59.311 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.311 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e30bcd0:::benchmark_data_smithi043_119832_object6410:head# 2024-03-30T13:50:59.311 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.312 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e30caad:::benchmark_data_smithi043_119832_object44986:head# 2024-03-30T13:50:59.312 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.313 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e312672:::benchmark_data_smithi043_119832_object73193:head# 2024-03-30T13:50:59.313 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.313 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e324c31:::benchmark_data_smithi043_119832_object23486:head# 2024-03-30T13:50:59.313 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.313 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e330b80:::benchmark_data_smithi043_119832_object64245:head# 2024-03-30T13:50:59.313 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.313 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e330d36:::benchmark_data_smithi043_119832_object47854:head# 2024-03-30T13:50:59.313 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.313 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e33734c:::benchmark_data_smithi043_119832_object17495:head# 2024-03-30T13:50:59.313 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.315 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e34467a:::benchmark_data_smithi043_119832_object12086:head# 2024-03-30T13:50:59.315 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.315 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e345fc7:::benchmark_data_smithi043_119832_object13360:head# 2024-03-30T13:50:59.315 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.315 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e3550cc:::benchmark_data_smithi043_119832_object3563:head# 2024-03-30T13:50:59.315 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.315 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e3561f7:::benchmark_data_smithi043_119832_object54401:head# 2024-03-30T13:50:59.316 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.316 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e35a624:::benchmark_data_smithi043_119832_object55144:head# 2024-03-30T13:50:59.316 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.316 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e368dc3:::benchmark_data_smithi043_119832_object37495:head# 2024-03-30T13:50:59.316 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.316 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e384a20:::benchmark_data_smithi043_119832_object6548:head# 2024-03-30T13:50:59.316 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.317 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e3ab692:::benchmark_data_smithi043_119832_object10069:head# 2024-03-30T13:50:59.317 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.317 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e3afef8:::benchmark_data_smithi043_119832_object47643:head# 2024-03-30T13:50:59.317 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.317 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e3b5504:::benchmark_data_smithi043_119832_object48915:head# 2024-03-30T13:50:59.317 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.318 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e3d40e3:::benchmark_data_smithi043_119832_object41472:head# 2024-03-30T13:50:59.318 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.318 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e3df940:::benchmark_data_smithi043_119832_object36472:head# 2024-03-30T13:50:59.318 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.318 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e3e59a9:::benchmark_data_smithi043_119832_object35596:head# 2024-03-30T13:50:59.318 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.318 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e3f4726:::benchmark_data_smithi043_119832_object34646:head# 2024-03-30T13:50:59.318 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.319 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e3fdf14:::benchmark_data_smithi043_119832_object88071:head# 2024-03-30T13:50:59.319 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.319 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e42b110:::benchmark_data_smithi043_119832_object48581:head# 2024-03-30T13:50:59.319 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.319 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e42ef3a:::benchmark_data_smithi043_119832_object33130:head# 2024-03-30T13:50:59.319 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.319 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e43018f:::benchmark_data_smithi043_119832_object44090:head# 2024-03-30T13:50:59.319 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.319 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e44bf7f:::benchmark_data_smithi043_119832_object2905:head# 2024-03-30T13:50:59.319 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.321 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e452bba:::benchmark_data_smithi043_119832_object61224:head# 2024-03-30T13:50:59.321 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.321 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e4562f0:::benchmark_data_smithi043_119832_object44391:head# 2024-03-30T13:50:59.321 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.321 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e465512:::benchmark_data_smithi043_119832_object65840:head# 2024-03-30T13:50:59.321 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.321 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e46c397:::benchmark_data_smithi043_119832_object3265:head# 2024-03-30T13:50:59.321 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.321 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e475978:::benchmark_data_smithi043_119832_object64226:head# 2024-03-30T13:50:59.321 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.322 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e4942d7:::benchmark_data_smithi043_119832_object20870:head# 2024-03-30T13:50:59.322 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.322 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e4ac814:::benchmark_data_smithi043_119832_object49112:head# 2024-03-30T13:50:59.323 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.323 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e4bafe6:::benchmark_data_smithi043_119832_object21397:head# 2024-03-30T13:50:59.323 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.323 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e4be72e:::benchmark_data_smithi043_119832_object61184:head# 2024-03-30T13:50:59.323 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.323 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e4cfeb3:::benchmark_data_smithi043_119832_object36830:head# 2024-03-30T13:50:59.323 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.324 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e4d2980:::benchmark_data_smithi043_119832_object17106:head# 2024-03-30T13:50:59.324 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.324 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e4d62da:::benchmark_data_smithi043_119832_object2199:head# 2024-03-30T13:50:59.324 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.324 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e4d6c38:::benchmark_data_smithi043_119832_object48852:head# 2024-03-30T13:50:59.324 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.324 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e4dcd89:::benchmark_data_smithi043_119832_object71113:head# 2024-03-30T13:50:59.324 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.325 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e4fac57:::benchmark_data_smithi043_119832_object42952:head# 2024-03-30T13:50:59.325 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.326 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e504cfb:::benchmark_data_smithi043_119832_object51519:head# 2024-03-30T13:50:59.326 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.326 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e51e051:::benchmark_data_smithi043_119832_object33635:head# 2024-03-30T13:50:59.326 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.326 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e522467:::benchmark_data_smithi043_119832_object42110:head# 2024-03-30T13:50:59.326 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.327 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e52fa37:::benchmark_data_smithi043_119832_object20418:head# 2024-03-30T13:50:59.327 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.327 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e5425aa:::benchmark_data_smithi043_119832_object8994:head# 2024-03-30T13:50:59.327 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.327 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e54558b:::benchmark_data_smithi043_119832_object68735:head# 2024-03-30T13:50:59.327 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.328 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e554287:::benchmark_data_smithi043_119832_object42881:head# 2024-03-30T13:50:59.328 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.328 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e59b970:::benchmark_data_smithi043_119832_object1822:head# 2024-03-30T13:50:59.328 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.329 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e59e13c:::benchmark_data_smithi043_119832_object44243:head# 2024-03-30T13:50:59.329 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.329 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e59f6ec:::benchmark_data_smithi043_119832_object72898:head# 2024-03-30T13:50:59.329 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.329 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e5a68c1:::benchmark_data_smithi043_119832_object62515:head# 2024-03-30T13:50:59.329 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.329 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e5aa046:::benchmark_data_smithi043_119832_object59977:head# 2024-03-30T13:50:59.329 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.329 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e5ac30e:::benchmark_data_smithi043_119832_object83125:head# 2024-03-30T13:50:59.329 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.330 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e5aebf2:::benchmark_data_smithi043_119832_object699:head# 2024-03-30T13:50:59.330 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.330 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e5b1103:::benchmark_data_smithi043_119832_object32021:head# 2024-03-30T13:50:59.330 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.331 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e5b45ed:::benchmark_data_smithi043_119832_object54912:head# 2024-03-30T13:50:59.331 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.332 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e5d8b75:::benchmark_data_smithi043_119832_object8194:head# 2024-03-30T13:50:59.332 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.332 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e5decc0:::benchmark_data_smithi043_119832_object9318:head# 2024-03-30T13:50:59.332 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.333 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e5f9b76:::benchmark_data_smithi043_119832_object74941:head# 2024-03-30T13:50:59.333 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.333 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e5fb5f4:::benchmark_data_smithi043_119832_object79304:head# 2024-03-30T13:50:59.333 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.333 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e5fd69c:::benchmark_data_smithi043_119832_object84354:head# 2024-03-30T13:50:59.333 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.334 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e6043d9:::benchmark_data_smithi043_119832_object8273:head# 2024-03-30T13:50:59.335 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.335 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e61b5e4:::benchmark_data_smithi043_119832_object85763:head# 2024-03-30T13:50:59.335 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.335 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e620987:::benchmark_data_smithi043_119832_object31727:head# 2024-03-30T13:50:59.335 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.335 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e625534:::benchmark_data_smithi043_119832_object70068:head# 2024-03-30T13:50:59.335 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.336 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e625e32:::benchmark_data_smithi043_119832_object30443:head# 2024-03-30T13:50:59.336 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.336 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e63e7d1:::benchmark_data_smithi043_119832_object80116:head# 2024-03-30T13:50:59.336 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.336 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e66f301:::benchmark_data_smithi043_119832_object63341:head# 2024-03-30T13:50:59.337 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.337 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e6755d8:::benchmark_data_smithi043_119832_object13534:head# 2024-03-30T13:50:59.337 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.337 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e6a0e91:::benchmark_data_smithi043_119832_object34219:head# 2024-03-30T13:50:59.337 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.338 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e6ae25f:::benchmark_data_smithi043_119832_object9337:head# 2024-03-30T13:50:59.338 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.338 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e6b098b:::benchmark_data_smithi043_119832_object85308:head# 2024-03-30T13:50:59.338 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.338 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e6b4e85:::benchmark_data_smithi043_119832_object74582:head# 2024-03-30T13:50:59.338 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.339 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e6d2594:::benchmark_data_smithi043_119832_object69472:head# 2024-03-30T13:50:59.339 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.339 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e6d910e:::benchmark_data_smithi043_119832_object72691:head# 2024-03-30T13:50:59.339 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.339 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e6f306c:::benchmark_data_smithi043_119832_object79101:head# 2024-03-30T13:50:59.339 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.339 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e702471:::benchmark_data_smithi043_119832_object52793:head# 2024-03-30T13:50:59.340 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.341 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e70f6d2:::benchmark_data_smithi043_119832_object52617:head# 2024-03-30T13:50:59.341 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.341 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e711cf7:::benchmark_data_smithi043_119832_object76860:head# 2024-03-30T13:50:59.341 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.341 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e72e9bd:::benchmark_data_smithi043_119832_object62371:head# 2024-03-30T13:50:59.341 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.341 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e73551a:::benchmark_data_smithi043_119832_object43066:head# 2024-03-30T13:50:59.341 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.341 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e736822:::benchmark_data_smithi043_119832_object8916:head# 2024-03-30T13:50:59.341 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.343 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e73fb84:::benchmark_data_smithi043_119832_object36496:head# 2024-03-30T13:50:59.343 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.343 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e74d776:::benchmark_data_smithi043_119832_object83876:head# 2024-03-30T13:50:59.343 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.343 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e764b64:::benchmark_data_smithi043_119832_object44880:head# 2024-03-30T13:50:59.343 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.343 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e7701e0:::benchmark_data_smithi043_119832_object48188:head# 2024-03-30T13:50:59.343 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.343 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e78ce51:::benchmark_data_smithi043_119832_object31046:head# 2024-03-30T13:50:59.343 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.343 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e7a0bc0:::benchmark_data_smithi043_119832_object56436:head# 2024-03-30T13:50:59.344 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.345 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e7a488b:::benchmark_data_smithi043_119832_object12071:head# 2024-03-30T13:50:59.345 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.345 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e7ab437:::benchmark_data_smithi043_119832_object64393:head# 2024-03-30T13:50:59.345 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.345 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e7b1d5f:::benchmark_data_smithi043_119832_object29962:head# 2024-03-30T13:50:59.345 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.345 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e7b6269:::benchmark_data_smithi043_119832_object86342:head# 2024-03-30T13:50:59.345 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.345 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e7c5ba2:::benchmark_data_smithi043_119832_object40054:head# 2024-03-30T13:50:59.345 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.346 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e7f5e76:::benchmark_data_smithi043_119832_object54714:head# 2024-03-30T13:50:59.346 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.346 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e80158d:::benchmark_data_smithi043_119832_object4495:head# 2024-03-30T13:50:59.347 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.347 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e80add2:::benchmark_data_smithi043_119832_object73745:head# 2024-03-30T13:50:59.347 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.347 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e80e1d6:::benchmark_data_smithi043_119832_object87186:head# 2024-03-30T13:50:59.350 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.350 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e8129a8:::benchmark_data_smithi043_119832_object26995:head# 2024-03-30T13:50:59.350 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.350 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e815182:::benchmark_data_smithi043_119832_object78118:head# 2024-03-30T13:50:59.350 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.350 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e84733c:::benchmark_data_smithi043_119832_object88430:head# 2024-03-30T13:50:59.350 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.350 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e849338:::benchmark_data_smithi043_119832_object74188:head# 2024-03-30T13:50:59.350 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.350 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e84abed:::benchmark_data_smithi043_119832_object32146:head# 2024-03-30T13:50:59.350 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.350 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e855489:::benchmark_data_smithi043_119832_object36761:head# 2024-03-30T13:50:59.350 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.351 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e862b73:::benchmark_data_smithi043_119832_object36521:head# 2024-03-30T13:50:59.351 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.351 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e864e26:::benchmark_data_smithi043_119832_object67415:head# 2024-03-30T13:50:59.351 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.351 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e86ccc5:::benchmark_data_smithi043_119832_object30746:head# 2024-03-30T13:50:59.351 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.353 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e8704a3:::benchmark_data_smithi043_119832_object14180:head# 2024-03-30T13:50:59.353 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.353 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e88152f:::benchmark_data_smithi043_119832_object35552:head# 2024-03-30T13:50:59.353 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.353 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e882512:::benchmark_data_smithi043_119832_object52274:head# 2024-03-30T13:50:59.353 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.353 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e8837ea:::benchmark_data_smithi043_119832_object3793:head# 2024-03-30T13:50:59.353 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.353 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e89301c:::benchmark_data_smithi043_119832_object61643:head# 2024-03-30T13:50:59.353 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.353 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e89b271:::benchmark_data_smithi043_119832_object18481:head# 2024-03-30T13:50:59.353 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.354 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e89df90:::benchmark_data_smithi043_119832_object27556:head# 2024-03-30T13:50:59.354 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.354 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e8a1b6f:::benchmark_data_smithi043_119832_object27440:head# 2024-03-30T13:50:59.354 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.355 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e8a2064:::benchmark_data_smithi043_119832_object9590:head# 2024-03-30T13:50:59.356 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.356 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e8cd4b1:::benchmark_data_smithi043_119832_object58317:head# 2024-03-30T13:50:59.356 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.356 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e8d8b09:::benchmark_data_smithi043_119832_object26464:head# 2024-03-30T13:50:59.356 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.356 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e8efb42:::benchmark_data_smithi043_119832_object32135:head# 2024-03-30T13:50:59.356 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.356 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e8fa29f:::benchmark_data_smithi043_119832_object43530:head# 2024-03-30T13:50:59.356 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.356 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e8fb446:::benchmark_data_smithi043_119832_object19612:head# 2024-03-30T13:50:59.356 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.356 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e919669:::benchmark_data_smithi043_119832_object9287:head# 2024-03-30T13:50:59.356 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.356 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e92f307:::benchmark_data_smithi043_119832_object57150:head# 2024-03-30T13:50:59.357 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.358 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e93765b:::benchmark_data_smithi043_119832_object41706:head# 2024-03-30T13:50:59.358 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.358 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e93bda1:::benchmark_data_smithi043_119832_object70677:head# 2024-03-30T13:50:59.358 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.358 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e94bdb0:::benchmark_data_smithi043_119832_object82251:head# 2024-03-30T13:50:59.358 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.358 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e955002:::benchmark_data_smithi043_119832_object61770:head# 2024-03-30T13:50:59.358 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.359 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e9598de:::benchmark_data_smithi043_119832_object78947:head# 2024-03-30T13:50:59.359 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.359 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e9669d2:::benchmark_data_smithi043_119832_object39959:head# 2024-03-30T13:50:59.359 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.360 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e96cadd:::benchmark_data_smithi043_119832_object66811:head# 2024-03-30T13:50:59.360 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.360 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e97bbec:::benchmark_data_smithi043_119832_object30965:head# 2024-03-30T13:50:59.360 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.360 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e985363:::benchmark_data_smithi043_119832_object22995:head# 2024-03-30T13:50:59.361 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.361 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e986747:::benchmark_data_smithi043_119832_object26731:head# 2024-03-30T13:50:59.361 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.361 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e99ebe9:::benchmark_data_smithi043_119832_object31552:head# 2024-03-30T13:50:59.361 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.361 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e9c7de2:::benchmark_data_smithi043_119832_object38428:head# 2024-03-30T13:50:59.361 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.362 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e9cc098:::benchmark_data_smithi043_119832_object47135:head# 2024-03-30T13:50:59.363 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.363 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e9d103d:::benchmark_data_smithi043_119832_object3601:head# 2024-03-30T13:50:59.363 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.363 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e9d613a:::benchmark_data_smithi043_119832_object35398:head# 2024-03-30T13:50:59.363 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.363 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e9defb5:::benchmark_data_smithi043_119832_object15804:head# 2024-03-30T13:50:59.363 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.363 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e9ed10e:::benchmark_data_smithi043_119832_object52208:head# 2024-03-30T13:50:59.363 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.363 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7e9f19fc:::benchmark_data_smithi043_119832_object15660:head# 2024-03-30T13:50:59.363 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.365 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ea2bc5d:::benchmark_data_smithi043_119832_object9551:head# 2024-03-30T13:50:59.365 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.365 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ea40070:::benchmark_data_smithi043_119832_object10238:head# 2024-03-30T13:50:59.365 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.365 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ea47313:::benchmark_data_smithi043_119832_object63159:head# 2024-03-30T13:50:59.365 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.365 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ea488ed:::benchmark_data_smithi043_119832_object84669:head# 2024-03-30T13:50:59.365 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.365 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ea4d62c:::benchmark_data_smithi043_119832_object83977:head# 2024-03-30T13:50:59.366 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.366 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ea5b573:::benchmark_data_smithi043_119832_object48435:head# 2024-03-30T13:50:59.366 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.366 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ea6289b:::benchmark_data_smithi043_119832_object32229:head# 2024-03-30T13:50:59.366 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.367 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ea6408f:::benchmark_data_smithi043_119832_object46536:head# 2024-03-30T13:50:59.367 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.367 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ea68c09:::benchmark_data_smithi043_119832_object26381:head# 2024-03-30T13:50:59.368 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.368 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ea6c26f:::benchmark_data_smithi043_119832_object28870:head# 2024-03-30T13:50:59.368 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.368 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ea7027c:::benchmark_data_smithi043_119832_object62001:head# 2024-03-30T13:50:59.368 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.368 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ea74d88:::benchmark_data_smithi043_119832_object87046:head# 2024-03-30T13:50:59.368 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.368 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ea7e0a2:::benchmark_data_smithi043_119832_object85590:head# 2024-03-30T13:50:59.368 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.369 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ea7ea13:::benchmark_data_smithi043_119832_object15277:head# 2024-03-30T13:50:59.369 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.369 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ea80cd6:::benchmark_data_smithi043_119832_object9856:head# 2024-03-30T13:50:59.370 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.370 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ea9665c:::benchmark_data_smithi043_119832_object10047:head# 2024-03-30T13:50:59.370 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.370 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eaaa109:::benchmark_data_smithi043_119832_object31342:head# 2024-03-30T13:50:59.370 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.370 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eabe2d0:::benchmark_data_smithi043_119832_object72176:head# 2024-03-30T13:50:59.370 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.370 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eac6f2d:::benchmark_data_smithi043_119832_object15503:head# 2024-03-30T13:50:59.370 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.370 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ead9602:::benchmark_data_smithi043_119832_object83151:head# 2024-03-30T13:50:59.370 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.372 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eadc5e8:::benchmark_data_smithi043_119832_object81546:head# 2024-03-30T13:50:59.372 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.372 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eaee4cc:::benchmark_data_smithi043_119832_object55151:head# 2024-03-30T13:50:59.372 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.372 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eafbc03:::benchmark_data_smithi043_119832_object33895:head# 2024-03-30T13:50:59.372 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.372 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eb00248:::benchmark_data_smithi043_119832_object26047:head# 2024-03-30T13:50:59.372 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.372 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eb0679c:::benchmark_data_smithi043_119832_object15035:head# 2024-03-30T13:50:59.372 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.372 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eb0bc3e:::benchmark_data_smithi043_119832_object85731:head# 2024-03-30T13:50:59.372 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.374 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eb105a2:::benchmark_data_smithi043_119832_object78836:head# 2024-03-30T13:50:59.374 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.374 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eb20201:::benchmark_data_smithi043_119832_object16795:head# 2024-03-30T13:50:59.374 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.374 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eb2ee2f:::benchmark_data_smithi043_119832_object7408:head# 2024-03-30T13:50:59.374 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.374 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eb34524:::benchmark_data_smithi043_119832_object22889:head# 2024-03-30T13:50:59.374 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.374 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eb40550:::benchmark_data_smithi043_119832_object74339:head# 2024-03-30T13:50:59.375 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.375 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eb4adc8:::benchmark_data_smithi043_119832_object69594:head# 2024-03-30T13:50:59.375 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.376 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eb4bcfe:::benchmark_data_smithi043_119832_object4957:head# 2024-03-30T13:50:59.376 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.376 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eb52d62:::benchmark_data_smithi043_119832_object70051:head# 2024-03-30T13:50:59.376 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.376 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eb5b19f:::benchmark_data_smithi043_119832_object79982:head# 2024-03-30T13:50:59.376 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.376 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eb659b8:::benchmark_data_smithi043_119832_object66788:head# 2024-03-30T13:50:59.376 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.376 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eb68e5b:::benchmark_data_smithi043_119832_object48432:head# 2024-03-30T13:50:59.377 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.377 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eb85bc2:::benchmark_data_smithi043_119832_object8121:head# 2024-03-30T13:50:59.377 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.378 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eb89104:::benchmark_data_smithi043_119832_object46190:head# 2024-03-30T13:50:59.378 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.378 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eb93da8:::benchmark_data_smithi043_119832_object84528:head# 2024-03-30T13:50:59.378 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.378 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eb97976:::benchmark_data_smithi043_119832_object46391:head# 2024-03-30T13:50:59.378 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.378 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eb9beb5:::benchmark_data_smithi043_119832_object21499:head# 2024-03-30T13:50:59.378 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.379 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eba5b3a:::benchmark_data_smithi043_119832_object33699:head# 2024-03-30T13:50:59.380 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.380 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eba688f:::benchmark_data_smithi043_119832_object61247:head# 2024-03-30T13:50:59.380 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.380 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ebb6b2b:::benchmark_data_smithi043_119832_object24851:head# 2024-03-30T13:50:59.380 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.380 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ebc2bec:::benchmark_data_smithi043_119832_object33577:head# 2024-03-30T13:50:59.380 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.380 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ebc5af9:::benchmark_data_smithi043_119832_object32465:head# 2024-03-30T13:50:59.380 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.382 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ebcabab:::benchmark_data_smithi043_119832_object78969:head# 2024-03-30T13:50:59.382 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.382 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ebdc825:::benchmark_data_smithi043_119832_object61910:head# 2024-03-30T13:50:59.382 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.382 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ebdd97a:::benchmark_data_smithi043_119832_object83618:head# 2024-03-30T13:50:59.382 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.382 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ebe14e0:::benchmark_data_smithi043_119832_object12165:head# 2024-03-30T13:50:59.382 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.382 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ebe3f37:::benchmark_data_smithi043_119832_object38294:head# 2024-03-30T13:50:59.382 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.382 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ebe706c:::benchmark_data_smithi043_119832_object13017:head# 2024-03-30T13:50:59.382 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.384 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ebec910:::benchmark_data_smithi043_119832_object17935:head# 2024-03-30T13:50:59.384 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.384 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ebf1795:::benchmark_data_smithi043_119832_object18160:head# 2024-03-30T13:50:59.384 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.384 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ebf18c3:::benchmark_data_smithi043_119832_object22861:head# 2024-03-30T13:50:59.384 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.384 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ebf6ee8:::benchmark_data_smithi043_119832_object70331:head# 2024-03-30T13:50:59.384 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.385 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ebfc569:::benchmark_data_smithi043_119832_object48634:head# 2024-03-30T13:50:59.385 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.385 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ebfcc5f:::benchmark_data_smithi043_119832_object22282:head# 2024-03-30T13:50:59.385 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.386 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ec0d73f:::benchmark_data_smithi043_119832_object88149:head# 2024-03-30T13:50:59.387 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.387 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ec1f1d5:::benchmark_data_smithi043_119832_object49336:head# 2024-03-30T13:50:59.387 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.387 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ec293ff:::benchmark_data_smithi043_119832_object22216:head# 2024-03-30T13:50:59.387 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.387 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ec3149a:::benchmark_data_smithi043_119832_object56233:head# 2024-03-30T13:50:59.387 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.387 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ec397ec:::benchmark_data_smithi043_119832_object7202:head# 2024-03-30T13:50:59.387 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.387 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ec3fa31:::benchmark_data_smithi043_119832_object84954:head# 2024-03-30T13:50:59.387 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.387 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ec4b526:::benchmark_data_smithi043_119832_object86051:head# 2024-03-30T13:50:59.387 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.389 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ec5f8b0:::benchmark_data_smithi043_119832_object65685:head# 2024-03-30T13:50:59.389 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.389 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ec5ff88:::benchmark_data_smithi043_119832_object19798:head# 2024-03-30T13:50:59.389 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.389 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ec72c27:::benchmark_data_smithi043_119832_object59732:head# 2024-03-30T13:50:59.389 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.389 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ec7357e:::benchmark_data_smithi043_119832_object73794:head# 2024-03-30T13:50:59.390 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.390 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ec751fa:::benchmark_data_smithi043_119832_object63323:head# 2024-03-30T13:50:59.390 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.390 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ec77ce5:::benchmark_data_smithi043_119832_object15496:head# 2024-03-30T13:50:59.390 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.390 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ec77dcc:::benchmark_data_smithi043_119832_object51018:head# 2024-03-30T13:50:59.390 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.391 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ec9617f:::benchmark_data_smithi043_119832_object56515:head# 2024-03-30T13:50:59.391 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.391 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eca6de4:::benchmark_data_smithi043_119832_object32633:head# 2024-03-30T13:50:59.391 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.391 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ecafda7:::benchmark_data_smithi043_119832_object49152:head# 2024-03-30T13:50:59.391 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.392 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ecbb3d0:::benchmark_data_smithi043_119832_object21843:head# 2024-03-30T13:50:59.392 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.392 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ecbff9b:::benchmark_data_smithi043_119832_object25194:head# 2024-03-30T13:50:59.392 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.393 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ecca1d6:::benchmark_data_smithi043_119832_object67363:head# 2024-03-30T13:50:59.393 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.393 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ecd77c1:::benchmark_data_smithi043_119832_object73607:head# 2024-03-30T13:50:59.393 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.393 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ecde1f9:::benchmark_data_smithi043_119832_object54065:head# 2024-03-30T13:50:59.393 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.393 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ece2ec0:::benchmark_data_smithi043_119832_object6822:head# 2024-03-30T13:50:59.393 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.393 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ece5d4f:::benchmark_data_smithi043_119832_object73819:head# 2024-03-30T13:50:59.393 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.395 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ece65d4:::benchmark_data_smithi043_119832_object88287:head# 2024-03-30T13:50:59.395 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.395 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ecf9ed0:::benchmark_data_smithi043_119832_object32248:head# 2024-03-30T13:50:59.395 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.395 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ed09b0a:::benchmark_data_smithi043_119832_object69363:head# 2024-03-30T13:50:59.395 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.395 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ed0e19c:::benchmark_data_smithi043_119832_object50780:head# 2024-03-30T13:50:59.395 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.395 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ed1109a:::benchmark_data_smithi043_119832_object8137:head# 2024-03-30T13:50:59.395 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:50:59.397 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ed1dfb2:::benchmark_data_smithi043_119832_object6282:head# 2024-03-30T13:50:59.397 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.397 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ed1e1e4:::benchmark_data_smithi043_119832_object5471:head# 2024-03-30T13:50:59.398 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.398 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ed1f626:::benchmark_data_smithi043_119832_object4339:head# 2024-03-30T13:50:59.398 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.398 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ed41569:::benchmark_data_smithi043_119832_object39212:head# 2024-03-30T13:50:59.398 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.398 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ed42bed:::benchmark_data_smithi043_119832_object4734:head# 2024-03-30T13:50:59.398 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.398 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ed4b3c5:::benchmark_data_smithi043_119832_object22672:head# 2024-03-30T13:50:59.398 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.398 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ed4bd23:::benchmark_data_smithi043_119832_object88320:head# 2024-03-30T13:50:59.398 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.399 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ed66b19:::benchmark_data_smithi043_119832_object12957:head# 2024-03-30T13:50:59.399 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.399 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ed716d3:::benchmark_data_smithi043_119832_object61830:head# 2024-03-30T13:50:59.400 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.400 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eda912a:::benchmark_data_smithi043_119832_object5875:head# 2024-03-30T13:50:59.400 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.400 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7edb95de:::benchmark_data_smithi043_119832_object19215:head# 2024-03-30T13:50:59.400 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.400 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eddd353:::benchmark_data_smithi043_119832_object78022:head# 2024-03-30T13:50:59.400 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.401 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ede0d96:::benchmark_data_smithi043_119832_object30377:head# 2024-03-30T13:50:59.401 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.401 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ede122d:::benchmark_data_smithi043_119832_object18811:head# 2024-03-30T13:50:59.401 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.401 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ee06f7e:::benchmark_data_smithi043_119832_object31353:head# 2024-03-30T13:50:59.401 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.402 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ee19b4f:::benchmark_data_smithi043_119832_object337:head# 2024-03-30T13:50:59.402 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.402 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ee1dbe9:::benchmark_data_smithi043_119832_object66435:head# 2024-03-30T13:50:59.402 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.402 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ee57d9d:::benchmark_data_smithi043_119832_object66965:head# 2024-03-30T13:50:59.402 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.402 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ee637eb:::benchmark_data_smithi043_119832_object83592:head# 2024-03-30T13:50:59.402 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.403 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ee724cf:::benchmark_data_smithi043_119832_object67361:head# 2024-03-30T13:50:59.403 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.403 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ee83475:::benchmark_data_smithi043_119832_object18095:head# 2024-03-30T13:50:59.403 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.403 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ee8bdb3:::benchmark_data_smithi043_119832_object45060:head# 2024-03-30T13:50:59.404 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.404 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ee8cbe1:::benchmark_data_smithi043_119832_object68979:head# 2024-03-30T13:50:59.404 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.404 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eea1a79:::benchmark_data_smithi043_119832_object58589:head# 2024-03-30T13:50:59.404 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.405 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eea1c52:::benchmark_data_smithi043_119832_object45210:head# 2024-03-30T13:50:59.405 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.405 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eeabed5:::benchmark_data_smithi043_119832_object24999:head# 2024-03-30T13:50:59.405 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.405 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eeb8261:::benchmark_data_smithi043_119832_object34704:head# 2024-03-30T13:50:59.405 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.406 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eed281b:::benchmark_data_smithi043_119832_object2825:head# 2024-03-30T13:50:59.406 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.406 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eed2a0a:::benchmark_data_smithi043_119832_object80380:head# 2024-03-30T13:50:59.406 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.406 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eedf82c:::benchmark_data_smithi043_119832_object63982:head# 2024-03-30T13:50:59.406 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.407 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eee8338:::benchmark_data_smithi043_119832_object26908:head# 2024-03-30T13:50:59.407 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.407 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eefef26:::benchmark_data_smithi043_119832_object70250:head# 2024-03-30T13:50:59.407 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.407 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ef1c22d:::benchmark_data_smithi043_119832_object6914:head# 2024-03-30T13:50:59.408 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.408 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ef27761:::benchmark_data_smithi043_119832_object8557:head# 2024-03-30T13:50:59.408 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.408 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ef2d9f7:::benchmark_data_smithi043_119832_object66707:head# 2024-03-30T13:50:59.409 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.409 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ef45a81:::benchmark_data_smithi043_119832_object35376:head# 2024-03-30T13:50:59.409 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.409 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ef4b832:::benchmark_data_smithi043_119832_object1279:head# 2024-03-30T13:50:59.409 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.410 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ef54ee8:::benchmark_data_smithi043_119832_object47240:head# 2024-03-30T13:50:59.410 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.411 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ef581e1:::benchmark_data_smithi043_119832_object22681:head# 2024-03-30T13:50:59.411 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.411 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ef5dceb:::benchmark_data_smithi043_119832_object87992:head# 2024-03-30T13:50:59.411 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.411 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ef636c8:::benchmark_data_smithi043_119832_object40939:head# 2024-03-30T13:50:59.411 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.412 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ef67cd6:::benchmark_data_smithi043_119832_object88188:head# 2024-03-30T13:50:59.412 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.412 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ef6b3da:::benchmark_data_smithi043_119832_object25822:head# 2024-03-30T13:50:59.412 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.412 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ef74732:::benchmark_data_smithi043_119832_object74262:head# 2024-03-30T13:50:59.412 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.413 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ef83d87:::benchmark_data_smithi043_119832_object27071:head# 2024-03-30T13:50:59.413 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.413 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ef9961c:::benchmark_data_smithi043_119832_object76202:head# 2024-03-30T13:50:59.413 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.413 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7efa2645:::benchmark_data_smithi043_119832_object46228:head# 2024-03-30T13:50:59.413 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.414 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7efa3533:::benchmark_data_smithi043_119832_object40752:head# 2024-03-30T13:50:59.414 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.414 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7efa994e:::benchmark_data_smithi043_119832_object31574:head# 2024-03-30T13:50:59.414 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.414 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7efb8c12:::benchmark_data_smithi043_119832_object16773:head# 2024-03-30T13:50:59.414 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.415 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7efbb9bd:::benchmark_data_smithi043_119832_object4586:head# 2024-03-30T13:50:59.415 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.415 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7efd9c3f:::benchmark_data_smithi043_119832_object71963:head# 2024-03-30T13:50:59.415 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.415 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7efe62e6:::benchmark_data_smithi043_119832_object11982:head# 2024-03-30T13:50:59.415 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.416 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7eff1d63:::benchmark_data_smithi043_119832_object78837:head# 2024-03-30T13:50:59.417 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.417 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7efffc37:::benchmark_data_smithi043_119832_object83000:head# 2024-03-30T13:50:59.417 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.418 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f0239c7:::benchmark_data_smithi043_119832_object79526:head# 2024-03-30T13:50:59.418 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.418 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f028188:::benchmark_data_smithi043_119832_object52231:head# 2024-03-30T13:50:59.418 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.418 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f02b80d:::benchmark_data_smithi043_119832_object77144:head# 2024-03-30T13:50:59.418 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.419 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f043b09:::benchmark_data_smithi043_119832_object12755:head# 2024-03-30T13:50:59.419 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.419 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f046d65:::benchmark_data_smithi043_119832_object1061:head# 2024-03-30T13:50:59.419 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.419 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f048bf6:::benchmark_data_smithi043_119832_object48739:head# 2024-03-30T13:50:59.419 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.419 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f056f71:::benchmark_data_smithi043_119832_object46185:head# 2024-03-30T13:50:59.420 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.420 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f0597ab:::benchmark_data_smithi043_119832_object65615:head# 2024-03-30T13:50:59.420 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.420 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f05993e:::benchmark_data_smithi043_119832_object72067:head# 2024-03-30T13:50:59.420 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.421 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f05e151:::benchmark_data_smithi043_119832_object43444:head# 2024-03-30T13:50:59.421 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.421 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f068b74:::benchmark_data_smithi043_119832_object20110:head# 2024-03-30T13:50:59.422 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.422 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f0856d5:::benchmark_data_smithi043_119832_object63481:head# 2024-03-30T13:50:59.422 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.422 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f092b49:::benchmark_data_smithi043_119832_object79189:head# 2024-03-30T13:50:59.422 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.427 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f0a17d1:::benchmark_data_smithi043_119832_object70743:head# 2024-03-30T13:50:59.428 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.428 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f0abad2:::benchmark_data_smithi043_119832_object39271:head# 2024-03-30T13:50:59.428 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.428 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f0bd711:::benchmark_data_smithi043_119832_object37505:head# 2024-03-30T13:50:59.428 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.428 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f0bee31:::benchmark_data_smithi043_119832_object43037:head# 2024-03-30T13:50:59.428 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.428 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f0cb477:::benchmark_data_smithi043_119832_object39072:head# 2024-03-30T13:50:59.428 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.428 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f0cb7a9:::benchmark_data_smithi043_119832_object48024:head# 2024-03-30T13:50:59.429 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.429 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f0ce97d:::benchmark_data_smithi043_119832_object65523:head# 2024-03-30T13:50:59.429 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.429 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f0dc52a:::benchmark_data_smithi043_119832_object81157:head# 2024-03-30T13:50:59.429 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.429 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f0f19cc:::benchmark_data_smithi043_119832_object48392:head# 2024-03-30T13:50:59.429 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.429 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f11bdc9:::benchmark_data_smithi043_119832_object80405:head# 2024-03-30T13:50:59.429 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.429 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f11ce3a:::benchmark_data_smithi043_119832_object36979:head# 2024-03-30T13:50:59.429 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.430 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f12209d:::benchmark_data_smithi043_119832_object50524:head# 2024-03-30T13:50:59.430 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.430 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f14c170:::benchmark_data_smithi043_119832_object1539:head# 2024-03-30T13:50:59.430 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.430 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f16aba8:::benchmark_data_smithi043_119832_object63722:head# 2024-03-30T13:50:59.430 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.430 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f18bfb6:::benchmark_data_smithi043_119832_object55018:head# 2024-03-30T13:50:59.430 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.431 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f196a38:::benchmark_data_smithi043_119832_object52850:head# 2024-03-30T13:50:59.431 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.431 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f19acac:::benchmark_data_smithi043_119832_object9934:head# 2024-03-30T13:50:59.431 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.431 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f19cf5d:::benchmark_data_smithi043_119832_object70411:head# 2024-03-30T13:50:59.431 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.432 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f19f23b:::benchmark_data_smithi043_119832_object73139:head# 2024-03-30T13:50:59.432 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.432 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f1a92da:::benchmark_data_smithi043_119832_object47710:head# 2024-03-30T13:50:59.432 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.432 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f1abc01:::benchmark_data_smithi043_119832_object40876:head# 2024-03-30T13:50:59.432 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.433 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f1b3588:::benchmark_data_smithi043_119832_object35905:head# 2024-03-30T13:50:59.433 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.433 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f1c6b92:::benchmark_data_smithi043_119832_object63626:head# 2024-03-30T13:50:59.433 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.433 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f1c83b8:::benchmark_data_smithi043_119832_object5981:head# 2024-03-30T13:50:59.433 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.433 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f1cc229:::benchmark_data_smithi043_119832_object39404:head# 2024-03-30T13:50:59.433 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.433 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f1d1c9e:::benchmark_data_smithi043_119832_object44572:head# 2024-03-30T13:50:59.433 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.433 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f1d5672:::benchmark_data_smithi043_119832_object42518:head# 2024-03-30T13:50:59.434 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.434 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f1e2bc2:::benchmark_data_smithi043_119832_object7518:head# 2024-03-30T13:50:59.435 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.435 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f1eea10:::benchmark_data_smithi043_119832_object23997:head# 2024-03-30T13:50:59.435 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.435 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f1f0eb0:::benchmark_data_smithi043_119832_object71708:head# 2024-03-30T13:50:59.435 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.435 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f1f3808:::benchmark_data_smithi043_119832_object31216:head# 2024-03-30T13:50:59.435 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.435 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f1fa8d2:::benchmark_data_smithi043_119832_object56514:head# 2024-03-30T13:50:59.435 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.435 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f1fed64:::benchmark_data_smithi043_119832_object18987:head# 2024-03-30T13:50:59.436 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.436 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f213f01:::benchmark_data_smithi043_119832_object24195:head# 2024-03-30T13:50:59.436 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.436 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f21fe28:::benchmark_data_smithi043_119832_object86679:head# 2024-03-30T13:50:59.436 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.437 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f2400e3:::benchmark_data_smithi043_119832_object73963:head# 2024-03-30T13:50:59.437 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.437 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f26828c:::benchmark_data_smithi043_119832_object38748:head# 2024-03-30T13:50:59.437 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.437 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f274acd:::benchmark_data_smithi043_119832_object24605:head# 2024-03-30T13:50:59.437 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:59.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:50:59.442 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f275e2f:::benchmark_data_smithi043_119832_object8953:head# 2024-03-30T13:50:59.442 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.442 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f279b6f:::benchmark_data_smithi043_119832_object68402:head# 2024-03-30T13:50:59.442 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.442 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f27bdd7:::benchmark_data_smithi043_119832_object49499:head# 2024-03-30T13:50:59.442 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.442 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f281ed7:::benchmark_data_smithi043_119832_object53426:head# 2024-03-30T13:50:59.442 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.443 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f2843dc:::benchmark_data_smithi043_119832_object60625:head# 2024-03-30T13:50:59.443 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.443 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f28bf0e:::benchmark_data_smithi043_119832_object4892:head# 2024-03-30T13:50:59.443 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.443 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f28d3a6:::benchmark_data_smithi043_119832_object26297:head# 2024-03-30T13:50:59.443 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.443 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f28e369:::benchmark_data_smithi043_119832_object2612:head# 2024-03-30T13:50:59.443 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.443 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f29b501:::benchmark_data_smithi043_119832_object59657:head# 2024-03-30T13:50:59.443 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.443 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f29cf9e:::benchmark_data_smithi043_119832_object19467:head# 2024-03-30T13:50:59.444 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.444 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f2a5351:::benchmark_data_smithi043_119832_object28478:head# 2024-03-30T13:50:59.444 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.444 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f2bce33:::benchmark_data_smithi043_119832_object51301:head# 2024-03-30T13:50:59.444 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.444 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f2c94a9:::benchmark_data_smithi043_119832_object37936:head# 2024-03-30T13:50:59.444 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.444 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f2cbe63:::benchmark_data_smithi043_119832_object38153:head# 2024-03-30T13:50:59.444 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.444 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f2d02e8:::benchmark_data_smithi043_119832_object55115:head# 2024-03-30T13:50:59.444 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.449 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f2e2738:::benchmark_data_smithi043_119832_object52065:head# 2024-03-30T13:50:59.450 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.450 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f2ec152:::benchmark_data_smithi043_119832_object54296:head# 2024-03-30T13:50:59.450 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.450 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f2ee9f6:::benchmark_data_smithi043_119832_object69686:head# 2024-03-30T13:50:59.450 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.450 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f2f6fa1:::benchmark_data_smithi043_119832_object23713:head# 2024-03-30T13:50:59.450 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.450 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f303f25:::benchmark_data_smithi043_119832_object34245:head# 2024-03-30T13:50:59.450 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.450 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f3051b1:::benchmark_data_smithi043_119832_object12772:head# 2024-03-30T13:50:59.451 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.451 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f30680b:::benchmark_data_smithi043_119832_object24259:head# 2024-03-30T13:50:59.451 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.451 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f316b9f:::benchmark_data_smithi043_119832_object50455:head# 2024-03-30T13:50:59.451 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.451 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f31e9fa:::benchmark_data_smithi043_119832_object10262:head# 2024-03-30T13:50:59.451 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.451 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f335c87:::benchmark_data_smithi043_119832_object39701:head# 2024-03-30T13:50:59.451 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.451 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f336143:::benchmark_data_smithi043_119832_object26709:head# 2024-03-30T13:50:59.452 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.452 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f33692f:::benchmark_data_smithi043_119832_object65856:head# 2024-03-30T13:50:59.452 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.452 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f338c58:::benchmark_data_smithi043_119832_object106:head# 2024-03-30T13:50:59.452 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.452 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f33c7ee:::benchmark_data_smithi043_119832_object79190:head# 2024-03-30T13:50:59.452 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.452 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f34ec56:::benchmark_data_smithi043_119832_object85500:head# 2024-03-30T13:50:59.452 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.452 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f371887:::benchmark_data_smithi043_119832_object85185:head# 2024-03-30T13:50:59.452 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.453 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f372900:::benchmark_data_smithi043_119832_object3294:head# 2024-03-30T13:50:59.453 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.453 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f37482a:::benchmark_data_smithi043_119832_object52390:head# 2024-03-30T13:50:59.453 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.453 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f37851d:::benchmark_data_smithi043_119832_object57066:head# 2024-03-30T13:50:59.453 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.453 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f37fa85:::benchmark_data_smithi043_119832_object37185:head# 2024-03-30T13:50:59.453 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.453 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f387980:::benchmark_data_smithi043_119832_object64280:head# 2024-03-30T13:50:59.453 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.453 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f3981b9:::benchmark_data_smithi043_119832_object28504:head# 2024-03-30T13:50:59.454 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.459 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f3a5044:::benchmark_data_smithi043_119832_object52576:head# 2024-03-30T13:50:59.459 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.459 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f3ab1cf:::benchmark_data_smithi043_119832_object41066:head# 2024-03-30T13:50:59.459 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.459 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f3b6963:::benchmark_data_smithi043_119832_object73270:head# 2024-03-30T13:50:59.459 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.459 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f3c4998:::benchmark_data_smithi043_119832_object68083:head# 2024-03-30T13:50:59.460 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.460 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f3d67d7:::benchmark_data_smithi043_119832_object82509:head# 2024-03-30T13:50:59.460 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.460 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f40492c:::benchmark_data_smithi043_119832_object48244:head# 2024-03-30T13:50:59.460 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.460 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f405fd3:::benchmark_data_smithi043_119832_object35543:head# 2024-03-30T13:50:59.460 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.460 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f4060ca:::benchmark_data_smithi043_119832_object249:head# 2024-03-30T13:50:59.460 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.460 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f408acb:::benchmark_data_smithi043_119832_object57065:head# 2024-03-30T13:50:59.460 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.460 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f40bc4f:::benchmark_data_smithi043_119832_object27908:head# 2024-03-30T13:50:59.460 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.461 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f40f2d9:::benchmark_data_smithi043_119832_object35643:head# 2024-03-30T13:50:59.461 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.461 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f414536:::benchmark_data_smithi043_119832_object20943:head# 2024-03-30T13:50:59.461 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.461 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f42bc18:::benchmark_data_smithi043_119832_object32020:head# 2024-03-30T13:50:59.461 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.461 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f430a76:::benchmark_data_smithi043_119832_object83162:head# 2024-03-30T13:50:59.461 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.461 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f43829d:::benchmark_data_smithi043_119832_object86170:head# 2024-03-30T13:50:59.461 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.461 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f43d41e:::benchmark_data_smithi043_119832_object15349:head# 2024-03-30T13:50:59.462 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.462 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f4424f9:::benchmark_data_smithi043_119832_object84175:head# 2024-03-30T13:50:59.462 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.462 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f447587:::benchmark_data_smithi043_119832_object81156:head# 2024-03-30T13:50:59.462 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.462 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f44fc5c:::benchmark_data_smithi043_119832_object33150:head# 2024-03-30T13:50:59.462 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.462 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f4595fe:::benchmark_data_smithi043_119832_object77328:head# 2024-03-30T13:50:59.462 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.462 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f45a65a:::benchmark_data_smithi043_119832_object53390:head# 2024-03-30T13:50:59.462 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.462 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f46184e:::benchmark_data_smithi043_119832_object75366:head# 2024-03-30T13:50:59.463 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.463 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f46c0a0:::benchmark_data_smithi043_119832_object53363:head# 2024-03-30T13:50:59.463 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.463 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f474a59:::benchmark_data_smithi043_119832_object18178:head# 2024-03-30T13:50:59.463 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.463 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f4845af:::benchmark_data_smithi043_119832_object81696:head# 2024-03-30T13:50:59.463 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.463 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f48b4b9:::benchmark_data_smithi043_119832_object7320:head# 2024-03-30T13:50:59.463 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.463 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f48c5f7:::benchmark_data_smithi043_119832_object16793:head# 2024-03-30T13:50:59.463 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.468 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f49937e:::benchmark_data_smithi043_119832_object40839:head# 2024-03-30T13:50:59.469 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.469 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f49c277:::benchmark_data_smithi043_119832_object85241:head# 2024-03-30T13:50:59.469 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.469 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f49c522:::benchmark_data_smithi043_119832_object79086:head# 2024-03-30T13:50:59.469 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.469 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f4a2e65:::benchmark_data_smithi043_119832_object75226:head# 2024-03-30T13:50:59.469 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.469 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f4a900b:::benchmark_data_smithi043_119832_object11395:head# 2024-03-30T13:50:59.469 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.469 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f4b556f:::benchmark_data_smithi043_119832_object82130:head# 2024-03-30T13:50:59.469 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.469 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f4b70ea:::benchmark_data_smithi043_119832_object80714:head# 2024-03-30T13:50:59.470 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.470 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f4b728b:::benchmark_data_smithi043_119832_object65116:head# 2024-03-30T13:50:59.470 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.470 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f4bb8e3:::benchmark_data_smithi043_119832_object71081:head# 2024-03-30T13:50:59.470 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.470 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f4bf923:::benchmark_data_smithi043_119832_object45812:head# 2024-03-30T13:50:59.470 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.470 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f4c46a9:::benchmark_data_smithi043_119832_object61589:head# 2024-03-30T13:50:59.470 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.470 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f4c87ff:::benchmark_data_smithi043_119832_object42888:head# 2024-03-30T13:50:59.470 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.470 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f4caf50:::benchmark_data_smithi043_119832_object8978:head# 2024-03-30T13:50:59.471 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.471 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f4d0fae:::benchmark_data_smithi043_119832_object49906:head# 2024-03-30T13:50:59.471 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.471 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f4d33e3:::benchmark_data_smithi043_119832_object100:head# 2024-03-30T13:50:59.471 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.471 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f4de78d:::benchmark_data_smithi043_119832_object24815:head# 2024-03-30T13:50:59.471 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.471 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f4fdd51:::benchmark_data_smithi043_119832_object11593:head# 2024-03-30T13:50:59.471 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.471 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f4ffe66:::benchmark_data_smithi043_119832_object80368:head# 2024-03-30T13:50:59.472 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.472 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f50fb0e:::benchmark_data_smithi043_119832_object85690:head# 2024-03-30T13:50:59.472 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.472 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f51c4f6:::benchmark_data_smithi043_119832_object80545:head# 2024-03-30T13:50:59.472 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.472 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f52ed37:::benchmark_data_smithi043_119832_object77090:head# 2024-03-30T13:50:59.472 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.472 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f5324ad:::benchmark_data_smithi043_119832_object2764:head# 2024-03-30T13:50:59.472 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.472 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f5417b6:::benchmark_data_smithi043_119832_object24651:head# 2024-03-30T13:50:59.472 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.472 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f566775:::benchmark_data_smithi043_119832_object64857:head# 2024-03-30T13:50:59.472 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.473 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f56b124:::benchmark_data_smithi043_119832_object21400:head# 2024-03-30T13:50:59.473 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.473 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f574290:::benchmark_data_smithi043_119832_object13550:head# 2024-03-30T13:50:59.473 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.473 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f57df27:::benchmark_data_smithi043_119832_object5730:head# 2024-03-30T13:50:59.473 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.479 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f58c764:::benchmark_data_smithi043_119832_object74993:head# 2024-03-30T13:50:59.479 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.479 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f58d976:::benchmark_data_smithi043_119832_object21929:head# 2024-03-30T13:50:59.479 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.479 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f59f588:::benchmark_data_smithi043_119832_object58193:head# 2024-03-30T13:50:59.479 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.479 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f5a0652:::benchmark_data_smithi043_119832_object56354:head# 2024-03-30T13:50:59.479 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.479 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f5a39e6:::benchmark_data_smithi043_119832_object54160:head# 2024-03-30T13:50:59.479 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.480 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f5ad552:::benchmark_data_smithi043_119832_object57672:head# 2024-03-30T13:50:59.480 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.480 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f5b8b5e:::benchmark_data_smithi043_119832_object25123:head# 2024-03-30T13:50:59.480 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.480 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f5ba9c3:::benchmark_data_smithi043_119832_object26099:head# 2024-03-30T13:50:59.480 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.480 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f5befc8:::benchmark_data_smithi043_119832_object13154:head# 2024-03-30T13:50:59.480 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.480 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f5c010d:::benchmark_data_smithi043_119832_object76368:head# 2024-03-30T13:50:59.480 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.480 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f5db587:::benchmark_data_smithi043_119832_object32913:head# 2024-03-30T13:50:59.481 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.481 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f5e3511:::benchmark_data_smithi043_119832_object68295:head# 2024-03-30T13:50:59.481 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.481 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f5e75e3:::benchmark_data_smithi043_119832_object71867:head# 2024-03-30T13:50:59.481 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.481 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f5e7da9:::benchmark_data_smithi043_119832_object56122:head# 2024-03-30T13:50:59.481 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.481 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f5e8833:::benchmark_data_smithi043_119832_object24359:head# 2024-03-30T13:50:59.481 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.481 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f5ee467:::benchmark_data_smithi043_119832_object31689:head# 2024-03-30T13:50:59.481 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.481 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f5f70ef:::benchmark_data_smithi043_119832_object44429:head# 2024-03-30T13:50:59.482 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.482 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f60d3f5:::benchmark_data_smithi043_119832_object23919:head# 2024-03-30T13:50:59.482 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.482 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f61b40e:::benchmark_data_smithi043_119832_object46723:head# 2024-03-30T13:50:59.482 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.482 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f625325:::benchmark_data_smithi043_119832_object56203:head# 2024-03-30T13:50:59.482 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.482 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f6376b4:::benchmark_data_smithi043_119832_object65348:head# 2024-03-30T13:50:59.482 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.482 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f641230:::benchmark_data_smithi043_119832_object41578:head# 2024-03-30T13:50:59.482 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.482 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f6476dc:::benchmark_data_smithi043_119832_object785:head# 2024-03-30T13:50:59.483 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.483 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f6482eb:::benchmark_data_smithi043_119832_object21820:head# 2024-03-30T13:50:59.483 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.483 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f6574ce:::benchmark_data_smithi043_119832_object59497:head# 2024-03-30T13:50:59.483 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.483 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f6645ab:::benchmark_data_smithi043_119832_object66791:head# 2024-03-30T13:50:59.483 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.483 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f682197:::benchmark_data_smithi043_119832_object43994:head# 2024-03-30T13:50:59.483 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.483 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f68c434:::benchmark_data_smithi043_119832_object76872:head# 2024-03-30T13:50:59.483 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.484 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f691222:::benchmark_data_smithi043_119832_object2421:head# 2024-03-30T13:50:59.484 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.484 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f6a1b71:::benchmark_data_smithi043_119832_object63763:head# 2024-03-30T13:50:59.484 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.484 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f6aad42:::benchmark_data_smithi043_119832_object67511:head# 2024-03-30T13:50:59.484 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.487 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f6bfb2a:::benchmark_data_smithi043_119832_object61488:head# 2024-03-30T13:50:59.487 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.488 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f6c4632:::benchmark_data_smithi043_119832_object27813:head# 2024-03-30T13:50:59.488 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.488 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f6ce309:::benchmark_data_smithi043_119832_object24778:head# 2024-03-30T13:50:59.488 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.488 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f6da891:::benchmark_data_smithi043_119832_object80093:head# 2024-03-30T13:50:59.488 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.488 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f6dd18b:::benchmark_data_smithi043_119832_object80254:head# 2024-03-30T13:50:59.488 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.488 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f6ea7c0:::benchmark_data_smithi043_119832_object48401:head# 2024-03-30T13:50:59.488 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.488 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f6ebeef:::benchmark_data_smithi043_119832_object50082:head# 2024-03-30T13:50:59.489 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.489 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f6f0e2e:::benchmark_data_smithi043_119832_object26966:head# 2024-03-30T13:50:59.489 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.489 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f6f6999:::benchmark_data_smithi043_119832_object38712:head# 2024-03-30T13:50:59.489 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.489 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f6fe401:::benchmark_data_smithi043_119832_object36347:head# 2024-03-30T13:50:59.489 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.489 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f72a822:::benchmark_data_smithi043_119832_object31494:head# 2024-03-30T13:50:59.489 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.489 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f72efc9:::benchmark_data_smithi043_119832_object68935:head# 2024-03-30T13:50:59.489 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.489 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f72fbda:::benchmark_data_smithi043_119832_object70127:head# 2024-03-30T13:50:59.490 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.490 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f73322d:::benchmark_data_smithi043_119832_object33619:head# 2024-03-30T13:50:59.490 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.490 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f741329:::benchmark_data_smithi043_119832_object84551:head# 2024-03-30T13:50:59.490 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.490 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f758ea8:::benchmark_data_smithi043_119832_object63498:head# 2024-03-30T13:50:59.490 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.490 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f78569d:::benchmark_data_smithi043_119832_object55907:head# 2024-03-30T13:50:59.490 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.490 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f788ba1:::benchmark_data_smithi043_119832_object25182:head# 2024-03-30T13:50:59.490 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.491 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f792afd:::benchmark_data_smithi043_119832_object50190:head# 2024-03-30T13:50:59.491 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.491 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f7c03fc:::benchmark_data_smithi043_119832_object25052:head# 2024-03-30T13:50:59.491 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.491 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f7c9a09:::benchmark_data_smithi043_119832_object22667:head# 2024-03-30T13:50:59.491 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.491 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f7f06c1:::benchmark_data_smithi043_119832_object57129:head# 2024-03-30T13:50:59.491 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.491 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f802b0f:::benchmark_data_smithi043_119832_object5835:head# 2024-03-30T13:50:59.491 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.491 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f804daf:::benchmark_data_smithi043_119832_object17052:head# 2024-03-30T13:50:59.491 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.492 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f80e7dc:::benchmark_data_smithi043_119832_object54634:head# 2024-03-30T13:50:59.492 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.492 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f80ff64:::benchmark_data_smithi043_119832_object79024:head# 2024-03-30T13:50:59.492 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.492 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f818660:::benchmark_data_smithi043_119832_object26262:head# 2024-03-30T13:50:59.492 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.492 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f83feae:::benchmark_data_smithi043_119832_object44570:head# 2024-03-30T13:50:59.492 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.497 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f845bb5:::benchmark_data_smithi043_119832_object11819:head# 2024-03-30T13:50:59.497 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.497 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f848cf3:::benchmark_data_smithi043_119832_object81495:head# 2024-03-30T13:50:59.498 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.498 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f84df8d:::benchmark_data_smithi043_119832_object8010:head# 2024-03-30T13:50:59.498 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.498 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f8698d1:::benchmark_data_smithi043_119832_object42712:head# 2024-03-30T13:50:59.498 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.498 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f86ee0c:::benchmark_data_smithi043_119832_object28270:head# 2024-03-30T13:50:59.498 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.498 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f885536:::benchmark_data_smithi043_119832_object39606:head# 2024-03-30T13:50:59.498 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.498 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f885b25:::benchmark_data_smithi043_119832_object15274:head# 2024-03-30T13:50:59.498 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.499 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f895ab8:::benchmark_data_smithi043_119832_object6514:head# 2024-03-30T13:50:59.499 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.499 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f89aa75:::benchmark_data_smithi043_119832_object82126:head# 2024-03-30T13:50:59.499 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.499 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f8a47c8:::benchmark_data_smithi043_119832_object60885:head# 2024-03-30T13:50:59.499 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.499 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f8bfce9:::benchmark_data_smithi043_119832_object54620:head# 2024-03-30T13:50:59.499 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.499 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f8c5571:::benchmark_data_smithi043_119832_object16205:head# 2024-03-30T13:50:59.499 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.499 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f8c89a3:::benchmark_data_smithi043_119832_object73273:head# 2024-03-30T13:50:59.499 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.500 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f8d8a58:::benchmark_data_smithi043_119832_object48810:head# 2024-03-30T13:50:59.500 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.500 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f8e5f6d:::benchmark_data_smithi043_119832_object58119:head# 2024-03-30T13:50:59.500 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.500 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f8ea9c1:::benchmark_data_smithi043_119832_object49811:head# 2024-03-30T13:50:59.500 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.500 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f8ec9e6:::benchmark_data_smithi043_119832_object81635:head# 2024-03-30T13:50:59.500 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.500 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f8fa481:::benchmark_data_smithi043_119832_object4671:head# 2024-03-30T13:50:59.500 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.500 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f8fabe4:::benchmark_data_smithi043_119832_object58257:head# 2024-03-30T13:50:59.500 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.500 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f912092:::benchmark_data_smithi043_119832_object61327:head# 2024-03-30T13:50:59.501 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.501 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f9242f8:::benchmark_data_smithi043_119832_object6109:head# 2024-03-30T13:50:59.501 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.501 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f936238:::benchmark_data_smithi043_119832_object54004:head# 2024-03-30T13:50:59.501 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.501 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f93d410:::benchmark_data_smithi043_119832_object59057:head# 2024-03-30T13:50:59.501 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.501 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f9439ad:::benchmark_data_smithi043_119832_object58211:head# 2024-03-30T13:50:59.501 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.501 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f9446b9:::benchmark_data_smithi043_119832_object10429:head# 2024-03-30T13:50:59.501 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.501 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f94827a:::benchmark_data_smithi043_119832_object87655:head# 2024-03-30T13:50:59.502 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.502 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f950595:::benchmark_data_smithi043_119832_object18374:head# 2024-03-30T13:50:59.502 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.507 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f954685:::benchmark_data_smithi043_119832_object32425:head# 2024-03-30T13:50:59.507 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.507 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f957ab9:::benchmark_data_smithi043_119832_object17713:head# 2024-03-30T13:50:59.508 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.508 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f9584ff:::benchmark_data_smithi043_119832_object68969:head# 2024-03-30T13:50:59.508 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.508 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f95b118:::benchmark_data_smithi043_119832_object60232:head# 2024-03-30T13:50:59.508 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.508 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f966d22:::benchmark_data_smithi043_119832_object36930:head# 2024-03-30T13:50:59.508 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.508 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f96a716:::benchmark_data_smithi043_119832_object30175:head# 2024-03-30T13:50:59.508 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.508 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f979bc1:::benchmark_data_smithi043_119832_object71756:head# 2024-03-30T13:50:59.508 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.508 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f97a620:::benchmark_data_smithi043_119832_object8326:head# 2024-03-30T13:50:59.509 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.509 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f982d2b:::benchmark_data_smithi043_119832_object32657:head# 2024-03-30T13:50:59.509 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.509 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f9845ea:::benchmark_data_smithi043_119832_object81732:head# 2024-03-30T13:50:59.509 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.509 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f98b2bb:::benchmark_data_smithi043_119832_object23620:head# 2024-03-30T13:50:59.509 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.509 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f98c06b:::benchmark_data_smithi043_119832_object12106:head# 2024-03-30T13:50:59.509 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.509 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f995a21:::benchmark_data_smithi043_119832_object27932:head# 2024-03-30T13:50:59.509 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.510 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f997bdf:::benchmark_data_smithi043_119832_object20590:head# 2024-03-30T13:50:59.510 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.510 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f99aaaf:::benchmark_data_smithi043_119832_object71956:head# 2024-03-30T13:50:59.510 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.510 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f9a07e6:::benchmark_data_smithi043_119832_object31117:head# 2024-03-30T13:50:59.510 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.510 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f9a57f3:::benchmark_data_smithi043_119832_object4387:head# 2024-03-30T13:50:59.510 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.510 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f9bfdda:::benchmark_data_smithi043_119832_object32205:head# 2024-03-30T13:50:59.510 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.510 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f9c0aca:::benchmark_data_smithi043_119832_object60461:head# 2024-03-30T13:50:59.510 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.511 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f9d5bf7:::benchmark_data_smithi043_119832_object13495:head# 2024-03-30T13:50:59.511 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.511 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f9d5e89:::benchmark_data_smithi043_119832_object80087:head# 2024-03-30T13:50:59.511 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.511 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f9f8480:::benchmark_data_smithi043_119832_object15315:head# 2024-03-30T13:50:59.511 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.511 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7f9fd370:::benchmark_data_smithi043_119832_object23721:head# 2024-03-30T13:50:59.511 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.511 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fa09e59:::benchmark_data_smithi043_119832_object70632:head# 2024-03-30T13:50:59.511 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.511 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fa3a2bf:::benchmark_data_smithi043_119832_object51516:head# 2024-03-30T13:50:59.511 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.511 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fa60d26:::benchmark_data_smithi043_119832_object59373:head# 2024-03-30T13:50:59.512 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.512 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fa61604:::benchmark_data_smithi043_119832_object68901:head# 2024-03-30T13:50:59.512 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.515 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fa6ee67:::benchmark_data_smithi043_119832_object25848:head# 2024-03-30T13:50:59.515 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.515 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fa702d1:::benchmark_data_smithi043_119832_object57620:head# 2024-03-30T13:50:59.515 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.515 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fa742bc:::benchmark_data_smithi043_119832_object56656:head# 2024-03-30T13:50:59.515 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.515 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fa7c948:::benchmark_data_smithi043_119832_object81837:head# 2024-03-30T13:50:59.516 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.516 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fa9be6d:::benchmark_data_smithi043_119832_object74771:head# 2024-03-30T13:50:59.516 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.516 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fac7234:::benchmark_data_smithi043_119832_object87243:head# 2024-03-30T13:50:59.516 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.516 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7facf107:::benchmark_data_smithi043_119832_object21415:head# 2024-03-30T13:50:59.516 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.516 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fad2b4b:::benchmark_data_smithi043_119832_object53986:head# 2024-03-30T13:50:59.516 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.516 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fae13c9:::benchmark_data_smithi043_119832_object11410:head# 2024-03-30T13:50:59.516 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.516 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fae6204:::benchmark_data_smithi043_119832_object33195:head# 2024-03-30T13:50:59.517 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.517 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fae8215:::benchmark_data_smithi043_119832_object40127:head# 2024-03-30T13:50:59.517 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.517 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7faf7302:::benchmark_data_smithi043_119832_object78779:head# 2024-03-30T13:50:59.517 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.517 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fb02f60:::benchmark_data_smithi043_119832_object38961:head# 2024-03-30T13:50:59.517 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.517 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fb034a1:::benchmark_data_smithi043_119832_object80473:head# 2024-03-30T13:50:59.517 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.517 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fb0bf12:::benchmark_data_smithi043_119832_object10517:head# 2024-03-30T13:50:59.517 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.517 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fb0e64d:::benchmark_data_smithi043_119832_object24697:head# 2024-03-30T13:50:59.517 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.518 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fb34ef9:::benchmark_data_smithi043_119832_object37606:head# 2024-03-30T13:50:59.518 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.518 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fb37f10:::benchmark_data_smithi043_119832_object74704:head# 2024-03-30T13:50:59.518 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.518 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fb385ac:::benchmark_data_smithi043_119832_object13844:head# 2024-03-30T13:50:59.518 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.518 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fb3f820:::benchmark_data_smithi043_119832_object67777:head# 2024-03-30T13:50:59.518 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.520 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fb42e7d:::benchmark_data_smithi043_119832_object67621:head# 2024-03-30T13:50:59.520 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.520 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fb4bd49:::benchmark_data_smithi043_119832_object34237:head# 2024-03-30T13:50:59.520 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.520 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fb51911:::benchmark_data_smithi043_119832_object13048:head# 2024-03-30T13:50:59.520 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.520 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fb55044:::benchmark_data_smithi043_119832_object1805:head# 2024-03-30T13:50:59.520 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.520 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fb75641:::benchmark_data_smithi043_119832_object4784:head# 2024-03-30T13:50:59.520 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.521 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fb7f4c5:::benchmark_data_smithi043_119832_object23229:head# 2024-03-30T13:50:59.521 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.521 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fb8801c:::benchmark_data_smithi043_119832_object260:head# 2024-03-30T13:50:59.521 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.521 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fb934de:::benchmark_data_smithi043_119832_object55021:head# 2024-03-30T13:50:59.521 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.521 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fb9f599:::benchmark_data_smithi043_119832_object21314:head# 2024-03-30T13:50:59.521 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.521 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fb9fb3d:::benchmark_data_smithi043_119832_object56784:head# 2024-03-30T13:50:59.521 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.521 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fba0026:::benchmark_data_smithi043_119832_object30690:head# 2024-03-30T13:50:59.522 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.522 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fba6a7f:::benchmark_data_smithi043_119832_object43136:head# 2024-03-30T13:50:59.522 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.523 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fbaf62c:::benchmark_data_smithi043_119832_object41695:head# 2024-03-30T13:50:59.523 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.523 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fbbbe1b:::benchmark_data_smithi043_119832_object41933:head# 2024-03-30T13:50:59.524 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.524 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fbc4cc2:::benchmark_data_smithi043_119832_object59855:head# 2024-03-30T13:50:59.524 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.524 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fbc4d02:::benchmark_data_smithi043_119832_object75801:head# 2024-03-30T13:50:59.524 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.524 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fbc5230:::benchmark_data_smithi043_119832_object9601:head# 2024-03-30T13:50:59.524 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.524 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fbde701:::benchmark_data_smithi043_119832_object27265:head# 2024-03-30T13:50:59.524 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.524 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fbe2667:::benchmark_data_smithi043_119832_object49063:head# 2024-03-30T13:50:59.524 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.525 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fbe3c00:::benchmark_data_smithi043_119832_object68353:head# 2024-03-30T13:50:59.525 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.525 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fbe67b3:::benchmark_data_smithi043_119832_object41587:head# 2024-03-30T13:50:59.525 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.525 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fbf0eac:::benchmark_data_smithi043_119832_object76322:head# 2024-03-30T13:50:59.525 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.527 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fbf75b0:::benchmark_data_smithi043_119832_object15767:head# 2024-03-30T13:50:59.528 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.528 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fc14c20:::benchmark_data_smithi043_119832_object62497:head# 2024-03-30T13:50:59.528 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.528 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fc15035:::benchmark_data_smithi043_119832_object23054:head# 2024-03-30T13:50:59.528 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.528 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fc24dc6:::benchmark_data_smithi043_119832_object70019:head# 2024-03-30T13:50:59.528 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.528 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fc3e611:::benchmark_data_smithi043_119832_object50537:head# 2024-03-30T13:50:59.528 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.528 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fc44bf7:::benchmark_data_smithi043_119832_object85682:head# 2024-03-30T13:50:59.529 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.529 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fc4907d:::benchmark_data_smithi043_119832_object35668:head# 2024-03-30T13:50:59.529 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.529 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fc49dd1:::benchmark_data_smithi043_119832_object31979:head# 2024-03-30T13:50:59.529 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.529 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fc5ce96:::benchmark_data_smithi043_119832_object45676:head# 2024-03-30T13:50:59.529 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.529 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fc674b8:::benchmark_data_smithi043_119832_object4415:head# 2024-03-30T13:50:59.529 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.529 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fc70e34:::benchmark_data_smithi043_119832_object54596:head# 2024-03-30T13:50:59.529 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.529 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fc725ab:::benchmark_data_smithi043_119832_object74179:head# 2024-03-30T13:50:59.530 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.530 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fc77e0e:::benchmark_data_smithi043_119832_object63261:head# 2024-03-30T13:50:59.530 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.533 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fc85329:::benchmark_data_smithi043_119832_object74343:head# 2024-03-30T13:50:59.533 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.533 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fc93f08:::benchmark_data_smithi043_119832_object24606:head# 2024-03-30T13:50:59.533 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.533 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fc9ec67:::benchmark_data_smithi043_119832_object24233:head# 2024-03-30T13:50:59.533 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.534 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fca10c2:::benchmark_data_smithi043_119832_object74469:head# 2024-03-30T13:50:59.534 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.534 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fccaf26:::benchmark_data_smithi043_119832_object34388:head# 2024-03-30T13:50:59.534 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.534 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fcd12eb:::benchmark_data_smithi043_119832_object3829:head# 2024-03-30T13:50:59.534 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.534 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fcd9f52:::benchmark_data_smithi043_119832_object2534:head# 2024-03-30T13:50:59.534 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.534 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fcdbd49:::benchmark_data_smithi043_119832_object2967:head# 2024-03-30T13:50:59.534 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.534 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fcf1288:::benchmark_data_smithi043_119832_object65285:head# 2024-03-30T13:50:59.535 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.535 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fcf4bf7:::benchmark_data_smithi043_119832_object72628:head# 2024-03-30T13:50:59.535 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.535 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fd0a83e:::benchmark_data_smithi043_119832_object7452:head# 2024-03-30T13:50:59.535 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.535 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fd1026b:::benchmark_data_smithi043_119832_object74621:head# 2024-03-30T13:50:59.535 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.535 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fd1903b:::benchmark_data_smithi043_119832_object56987:head# 2024-03-30T13:50:59.535 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.535 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fd1d366:::benchmark_data_smithi043_119832_object42607:head# 2024-03-30T13:50:59.535 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.535 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fd26dd2:::benchmark_data_smithi043_119832_object52523:head# 2024-03-30T13:50:59.536 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.536 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fd31c9f:::benchmark_data_smithi043_119832_object70342:head# 2024-03-30T13:50:59.536 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.537 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fd353f5:::benchmark_data_smithi043_119832_object81097:head# 2024-03-30T13:50:59.537 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.537 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fd3ec15:::benchmark_data_smithi043_119832_object17089:head# 2024-03-30T13:50:59.537 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.537 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fd41518:::benchmark_data_smithi043_119832_object29239:head# 2024-03-30T13:50:59.538 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.538 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fd464e0:::benchmark_data_smithi043_119832_object24898:head# 2024-03-30T13:50:59.538 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.538 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fd4dbf3:::benchmark_data_smithi043_119832_object70521:head# 2024-03-30T13:50:59.538 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.538 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fd60723:::benchmark_data_smithi043_119832_object47874:head# 2024-03-30T13:50:59.538 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.538 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fd6b929:::benchmark_data_smithi043_119832_object3011:head# 2024-03-30T13:50:59.538 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.538 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fd70125:::benchmark_data_smithi043_119832_object18684:head# 2024-03-30T13:50:59.538 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.538 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fd70966:::benchmark_data_smithi043_119832_object43265:head# 2024-03-30T13:50:59.539 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.539 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fd70a8f:::benchmark_data_smithi043_119832_object55016:head# 2024-03-30T13:50:59.539 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.539 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fd71399:::benchmark_data_smithi043_119832_object26612:head# 2024-03-30T13:50:59.539 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.539 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:59.541 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fd82dfc:::benchmark_data_smithi043_119832_object65693:head# 2024-03-30T13:50:59.541 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.541 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fd847eb:::benchmark_data_smithi043_119832_object21611:head# 2024-03-30T13:50:59.541 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.541 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fda45a6:::benchmark_data_smithi043_119832_object57244:head# 2024-03-30T13:50:59.541 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.541 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fda83f7:::benchmark_data_smithi043_119832_object32865:head# 2024-03-30T13:50:59.541 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.541 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fda8671:::benchmark_data_smithi043_119832_object31590:head# 2024-03-30T13:50:59.542 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.542 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fdaf329:::benchmark_data_smithi043_119832_object22628:head# 2024-03-30T13:50:59.542 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.542 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fdd24e1:::benchmark_data_smithi043_119832_object8517:head# 2024-03-30T13:50:59.542 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.542 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fdd5397:::benchmark_data_smithi043_119832_object29943:head# 2024-03-30T13:50:59.542 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.542 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fddf818:::benchmark_data_smithi043_119832_object73307:head# 2024-03-30T13:50:59.542 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.542 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fde0a57:::benchmark_data_smithi043_119832_object42141:head# 2024-03-30T13:50:59.542 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.546 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fde44bf:::benchmark_data_smithi043_119832_object54044:head# 2024-03-30T13:50:59.546 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.546 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fde6978:::benchmark_data_smithi043_119832_object85112:head# 2024-03-30T13:50:59.546 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.546 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fdec3ba:::benchmark_data_smithi043_119832_object8236:head# 2024-03-30T13:50:59.546 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.546 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fdf014c:::benchmark_data_smithi043_119832_object31951:head# 2024-03-30T13:50:59.546 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.546 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fdf0514:::benchmark_data_smithi043_119832_object29878:head# 2024-03-30T13:50:59.547 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.547 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fdf497a:::benchmark_data_smithi043_119832_object45741:head# 2024-03-30T13:50:59.547 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.547 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fe009aa:::benchmark_data_smithi043_119832_object10627:head# 2024-03-30T13:50:59.547 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.547 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fe04cf6:::benchmark_data_smithi043_119832_object74735:head# 2024-03-30T13:50:59.547 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.547 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fe06c85:::benchmark_data_smithi043_119832_object2827:head# 2024-03-30T13:50:59.547 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.547 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fe10208:::benchmark_data_smithi043_119832_object57640:head# 2024-03-30T13:50:59.547 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.547 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fe284c5:::benchmark_data_smithi043_119832_object3250:head# 2024-03-30T13:50:59.548 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.548 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fe3689b:::benchmark_data_smithi043_119832_object17053:head# 2024-03-30T13:50:59.548 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.548 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fe3e8ee:::benchmark_data_smithi043_119832_object86784:head# 2024-03-30T13:50:59.548 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.548 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fe3ef99:::benchmark_data_smithi043_119832_object38104:head# 2024-03-30T13:50:59.548 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.553 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fe50325:::benchmark_data_smithi043_119832_object70766:head# 2024-03-30T13:50:59.553 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.553 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fe57b7e:::benchmark_data_smithi043_119832_object63879:head# 2024-03-30T13:50:59.553 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.553 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fe6d227:::benchmark_data_smithi043_119832_object34792:head# 2024-03-30T13:50:59.553 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.554 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fe71531:::benchmark_data_smithi043_119832_object40319:head# 2024-03-30T13:50:59.554 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.554 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fe7b112:::benchmark_data_smithi043_119832_object20083:head# 2024-03-30T13:50:59.554 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.554 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fe94e7b:::benchmark_data_smithi043_119832_object27740:head# 2024-03-30T13:50:59.554 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.554 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fe97aab:::benchmark_data_smithi043_119832_object24762:head# 2024-03-30T13:50:59.554 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.554 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fea1094:::benchmark_data_smithi043_119832_object4136:head# 2024-03-30T13:50:59.554 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.555 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7feaca24:::benchmark_data_smithi043_119832_object72982:head# 2024-03-30T13:50:59.555 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.555 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7feb3e28:::benchmark_data_smithi043_119832_object62850:head# 2024-03-30T13:50:59.555 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.555 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7feb7424:::benchmark_data_smithi043_119832_object77603:head# 2024-03-30T13:50:59.555 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.555 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fed17bd:::benchmark_data_smithi043_119832_object19587:head# 2024-03-30T13:50:59.555 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.555 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fedacbd:::benchmark_data_smithi043_119832_object64790:head# 2024-03-30T13:50:59.555 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.555 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ff0380d:::benchmark_data_smithi043_119832_object48251:head# 2024-03-30T13:50:59.556 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.556 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ff09a48:::benchmark_data_smithi043_119832_object62202:head# 2024-03-30T13:50:59.556 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.556 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ff09bd4:::benchmark_data_smithi043_119832_object75183:head# 2024-03-30T13:50:59.556 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.556 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ff0f2f2:::benchmark_data_smithi043_119832_object63851:head# 2024-03-30T13:50:59.556 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.556 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ff25ec9:::benchmark_data_smithi043_119832_object61193:head# 2024-03-30T13:50:59.556 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.556 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ff2835b:::benchmark_data_smithi043_119832_object51674:head# 2024-03-30T13:50:59.556 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.557 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ff5b365:::benchmark_data_smithi043_119832_object60971:head# 2024-03-30T13:50:59.557 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.561 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ff5cdef:::benchmark_data_smithi043_119832_object43195:head# 2024-03-30T13:50:59.561 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.561 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ff5f5a7:::benchmark_data_smithi043_119832_object12665:head# 2024-03-30T13:50:59.561 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.561 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ff65e19:::benchmark_data_smithi043_119832_object78720:head# 2024-03-30T13:50:59.561 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.561 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ff67e55:::benchmark_data_smithi043_119832_object69590:head# 2024-03-30T13:50:59.561 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.561 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ff7e9d9:::benchmark_data_smithi043_119832_object67507:head# 2024-03-30T13:50:59.562 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.562 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ff7ec5c:::benchmark_data_smithi043_119832_object23365:head# 2024-03-30T13:50:59.562 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.562 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ff82af1:::benchmark_data_smithi043_119832_object59820:head# 2024-03-30T13:50:59.562 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.562 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ff8a942:::benchmark_data_smithi043_119832_object5662:head# 2024-03-30T13:50:59.562 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.562 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ff9f6a0:::benchmark_data_smithi043_119832_object20126:head# 2024-03-30T13:50:59.562 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.562 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ffa3e16:::benchmark_data_smithi043_119832_object54434:head# 2024-03-30T13:50:59.562 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.563 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ffa446e:::benchmark_data_smithi043_119832_object26860:head# 2024-03-30T13:50:59.563 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.563 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ffa78be:::benchmark_data_smithi043_119832_object3462:head# 2024-03-30T13:50:59.563 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.563 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ffab638:::benchmark_data_smithi043_119832_object59912:head# 2024-03-30T13:50:59.563 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.563 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ffb743e:::benchmark_data_smithi043_119832_object15221:head# 2024-03-30T13:50:59.563 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.563 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ffc5789:::benchmark_data_smithi043_119832_object11843:head# 2024-03-30T13:50:59.563 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.563 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ffcfedd:::benchmark_data_smithi043_119832_object993:head# 2024-03-30T13:50:59.563 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.563 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7ffdd7d8:::benchmark_data_smithi043_119832_object83187:head# 2024-03-30T13:50:59.564 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.564 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fff5512:::benchmark_data_smithi043_119832_object23472:head# 2024-03-30T13:50:59.564 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.564 INFO:teuthology.orchestra.run.smithi043.stdout:Write #8:7fff689d:::benchmark_data_smithi043_119832_object6608:head# 2024-03-30T13:50:59.564 INFO:teuthology.orchestra.run.smithi043.stdout:snapset 0=[]:{} 2024-03-30T13:50:59.565 INFO:teuthology.orchestra.run.smithi043.stdout:write_pg epoch 166 info 8.e( v 166'5643 (0'0,166'5643] local-lis/les=147/148 n=5643 ec=147/147 lis/c=147/147 les/c/f=148/148/0 sis=147) 2024-03-30T13:50:59.576 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:59.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:59.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:59.615 INFO:teuthology.orchestra.run.smithi043.stdout:Import successful 2024-03-30T13:50:59.678 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:50:59.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:59.720 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:50:59.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:50:59.724 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:59 smithi043 ceph-mon[29963]: 4.e scrub starts 2024-03-30T13:50:59.724 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:59 smithi043 ceph-mon[29963]: 8.2 scrub ok 2024-03-30T13:50:59.724 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:59 smithi043 ceph-mon[29963]: 4.e scrub ok 2024-03-30T13:50:59.724 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:59 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:50:59.724 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:59 smithi043 ceph-mon[29963]: 6.7 scrub starts 2024-03-30T13:50:59.724 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:50:59 smithi043 ceph-mon[29963]: 6.7 scrub ok 2024-03-30T13:50:59.724 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:59 smithi043 ceph-mon[42099]: 4.e scrub starts 2024-03-30T13:50:59.724 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:59 smithi043 ceph-mon[42099]: 8.2 scrub ok 2024-03-30T13:50:59.724 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:59 smithi043 ceph-mon[42099]: 4.e scrub ok 2024-03-30T13:50:59.724 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:59 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:50:59.725 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:59 smithi043 ceph-mon[42099]: 6.7 scrub starts 2024-03-30T13:50:59.725 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:50:59 smithi043 ceph-mon[42099]: 6.7 scrub ok 2024-03-30T13:50:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:50:59.820 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:50:59.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:50:59.876 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:50:59.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:50:59.885 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:50:59.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:59 smithi113 ceph-mon[34341]: 4.e scrub starts 2024-03-30T13:50:59.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:59 smithi113 ceph-mon[34341]: 8.2 scrub ok 2024-03-30T13:50:59.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:59 smithi113 ceph-mon[34341]: 4.e scrub ok 2024-03-30T13:50:59.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:59 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:50:59.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:59 smithi113 ceph-mon[34341]: 6.7 scrub starts 2024-03-30T13:50:59.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:50:59 smithi113 ceph-mon[34341]: 6.7 scrub ok 2024-03-30T13:50:59.977 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:00.014 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:00.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:00.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:00.115 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:00.155 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:00.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:00.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:00.257 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:00.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:00.307 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:00.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:00.408 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:00.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:00.446 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:00.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:00.547 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:00.584 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:00.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:00.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:00.685 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:00.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:00.725 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:00.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:00.780 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:00 smithi043 ceph-mon[29963]: 4.1 scrub starts 2024-03-30T13:51:00.780 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:00 smithi043 ceph-mon[29963]: 4.1 scrub ok 2024-03-30T13:51:00.780 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:00 smithi043 ceph-mon[29963]: 4.3 scrub starts 2024-03-30T13:51:00.780 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:00 smithi043 ceph-mon[29963]: 4.3 scrub ok 2024-03-30T13:51:00.780 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:00 smithi043 ceph-mon[29963]: pgmap v330: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s; 41781/335190 objects degraded (12.465%) 2024-03-30T13:51:00.781 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:00 smithi043 ceph-mon[29963]: 6.b scrub starts 2024-03-30T13:51:00.781 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:00 smithi043 ceph-mon[29963]: 6.b scrub ok 2024-03-30T13:51:00.781 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:00 smithi043 ceph-mon[29963]: 8.9 scrub starts 2024-03-30T13:51:00.781 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:00 smithi043 ceph-mon[29963]: 4.2 scrub starts 2024-03-30T13:51:00.781 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:00 smithi043 ceph-mon[29963]: 4.2 scrub ok 2024-03-30T13:51:00.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:00 smithi043 ceph-mon[42099]: 4.1 scrub starts 2024-03-30T13:51:00.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:00 smithi043 ceph-mon[42099]: 4.1 scrub ok 2024-03-30T13:51:00.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:00 smithi043 ceph-mon[42099]: 4.3 scrub starts 2024-03-30T13:51:00.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:00 smithi043 ceph-mon[42099]: 4.3 scrub ok 2024-03-30T13:51:00.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:00 smithi043 ceph-mon[42099]: pgmap v330: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s; 41781/335190 objects degraded (12.465%) 2024-03-30T13:51:00.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:00 smithi043 ceph-mon[42099]: 6.b scrub starts 2024-03-30T13:51:00.782 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:00 smithi043 ceph-mon[42099]: 6.b scrub ok 2024-03-30T13:51:00.782 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:00 smithi043 ceph-mon[42099]: 8.9 scrub starts 2024-03-30T13:51:00.782 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:00 smithi043 ceph-mon[42099]: 4.2 scrub starts 2024-03-30T13:51:00.782 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:00 smithi043 ceph-mon[42099]: 4.2 scrub ok 2024-03-30T13:51:00.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:00.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:00.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:00.954 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:00 smithi113 ceph-mon[34341]: 4.1 scrub starts 2024-03-30T13:51:00.954 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:00 smithi113 ceph-mon[34341]: 4.1 scrub ok 2024-03-30T13:51:00.954 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:00 smithi113 ceph-mon[34341]: 4.3 scrub starts 2024-03-30T13:51:00.954 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:00 smithi113 ceph-mon[34341]: 4.3 scrub ok 2024-03-30T13:51:00.954 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:00 smithi113 ceph-mon[34341]: pgmap v330: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s; 41781/335190 objects degraded (12.465%) 2024-03-30T13:51:00.954 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:00 smithi113 ceph-mon[34341]: 6.b scrub starts 2024-03-30T13:51:00.954 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:00 smithi113 ceph-mon[34341]: 6.b scrub ok 2024-03-30T13:51:00.954 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:00 smithi113 ceph-mon[34341]: 8.9 scrub starts 2024-03-30T13:51:00.955 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:00 smithi113 ceph-mon[34341]: 4.2 scrub starts 2024-03-30T13:51:00.955 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:00 smithi113 ceph-mon[34341]: 4.2 scrub ok 2024-03-30T13:51:00.966 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:01.002 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:01.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:01.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:01.104 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:01.141 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:01.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:01.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:01.243 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:01.284 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:01.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:01.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:01.386 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:01.418 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:01.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:01.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:01.521 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:01.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:01.558 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:01.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:01.659 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:01.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:01.696 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:01.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:01.797 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:01.842 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:01.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:01.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:01.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:01 smithi043 ceph-mon[29963]: 5.15 scrub starts 2024-03-30T13:51:01.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:01 smithi043 ceph-mon[29963]: 5.15 scrub ok 2024-03-30T13:51:01.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:01 smithi043 ceph-mon[42099]: 5.15 scrub starts 2024-03-30T13:51:01.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:01 smithi043 ceph-mon[42099]: 5.15 scrub ok 2024-03-30T13:51:01.943 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:01.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:01 smithi113 ceph-mon[34341]: 5.15 scrub starts 2024-03-30T13:51:01.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:01 smithi113 ceph-mon[34341]: 5.15 scrub ok 2024-03-30T13:51:01.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:01.981 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:01.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:02.082 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:02.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:02.123 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:02.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:02.224 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:02.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:02.259 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:02.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:02.262 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/exp.8.e.2 2024-03-30T13:51:02.360 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:02.395 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:02.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:02.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:02.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:02.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:02.538 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:02.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:02.639 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:02.675 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:02.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:02.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:02.777 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:02.821 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:02.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:02.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:02.842 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:02 smithi043 ceph-mon[29963]: 3.3 scrub starts 2024-03-30T13:51:02.842 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:02 smithi043 ceph-mon[29963]: 3.3 scrub ok 2024-03-30T13:51:02.843 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:02 smithi043 ceph-mon[29963]: pgmap v331: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s; 41781/335190 objects degraded (12.465%) 2024-03-30T13:51:02.843 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:02 smithi043 ceph-mon[29963]: 8.9 scrub ok 2024-03-30T13:51:02.843 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:02 smithi043 ceph-mon[29963]: 4.1f scrub starts 2024-03-30T13:51:02.843 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:02 smithi043 ceph-mon[29963]: 4.1f scrub ok 2024-03-30T13:51:02.843 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:02 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:51:02.843 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:02 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:51:02.843 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:02 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 41781/335190 objects degraded (12.465%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:51:02.843 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:02 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1291035089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:51:02.843 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:02 smithi043 ceph-mon[42099]: 3.3 scrub starts 2024-03-30T13:51:02.843 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:02 smithi043 ceph-mon[42099]: 3.3 scrub ok 2024-03-30T13:51:02.843 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:02 smithi043 ceph-mon[42099]: pgmap v331: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s; 41781/335190 objects degraded (12.465%) 2024-03-30T13:51:02.844 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:02 smithi043 ceph-mon[42099]: 8.9 scrub ok 2024-03-30T13:51:02.844 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:02 smithi043 ceph-mon[42099]: 4.1f scrub starts 2024-03-30T13:51:02.844 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:02 smithi043 ceph-mon[42099]: 4.1f scrub ok 2024-03-30T13:51:02.844 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:02 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:51:02.844 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:02 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:51:02.844 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:02 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 41781/335190 objects degraded (12.465%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:51:02.844 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:02 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1291035089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:51:02.924 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:02.959 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:02.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:02.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:02.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:02 smithi113 ceph-mon[34341]: 3.3 scrub starts 2024-03-30T13:51:02.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:02 smithi113 ceph-mon[34341]: 3.3 scrub ok 2024-03-30T13:51:02.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:02 smithi113 ceph-mon[34341]: pgmap v331: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s; 41781/335190 objects degraded (12.465%) 2024-03-30T13:51:02.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:02 smithi113 ceph-mon[34341]: 8.9 scrub ok 2024-03-30T13:51:02.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:02 smithi113 ceph-mon[34341]: 4.1f scrub starts 2024-03-30T13:51:02.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:02 smithi113 ceph-mon[34341]: 4.1f scrub ok 2024-03-30T13:51:02.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:02 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:51:02.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:02 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:51:02.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:02 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 41781/335190 objects degraded (12.465%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:51:02.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:02 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1291035089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:51:02.973 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:51:02.973 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:51:02.973 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:51:02.973 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:51:02.973 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:51:02.973 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:51:02.973 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:51:02.973 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:51:02.973 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:51:02.973 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:51:02.974 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:51:02.974 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:51:02.974 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:51:02.974 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:51:02.974 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:51:03.061 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:03.095 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:03.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:03.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:03.197 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:03.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:03.236 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:03.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:03.336 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:03.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:03.361 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:03.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:03.462 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:03.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:03.499 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:03.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:03.600 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:03.633 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:03.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:03.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:03.735 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:03.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:03.761 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:03.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:03.862 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:03.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:03.898 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:03.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:03.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:03 smithi113 ceph-mon[34341]: 2.0 scrub starts 2024-03-30T13:51:03.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:03 smithi113 ceph-mon[34341]: 2.0 scrub ok 2024-03-30T13:51:03.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:03 smithi113 ceph-mon[34341]: osdmap e183: 8 total, 7 up, 8 in 2024-03-30T13:51:03.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:03 smithi113 ceph-mon[34341]: 5.1f scrub starts 2024-03-30T13:51:03.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:03 smithi113 ceph-mon[34341]: 5.1f scrub ok 2024-03-30T13:51:03.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:03 smithi113 ceph-mon[34341]: 3.19 scrub starts 2024-03-30T13:51:03.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:03 smithi113 ceph-mon[34341]: 3.19 scrub ok 2024-03-30T13:51:03.999 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:04.022 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:04.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:04.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:04.124 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:04.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:03 smithi043 ceph-mon[29963]: 2.0 scrub starts 2024-03-30T13:51:04.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:03 smithi043 ceph-mon[29963]: 2.0 scrub ok 2024-03-30T13:51:04.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:03 smithi043 ceph-mon[29963]: osdmap e183: 8 total, 7 up, 8 in 2024-03-30T13:51:04.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:03 smithi043 ceph-mon[29963]: 5.1f scrub starts 2024-03-30T13:51:04.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:03 smithi043 ceph-mon[29963]: 5.1f scrub ok 2024-03-30T13:51:04.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:03 smithi043 ceph-mon[29963]: 3.19 scrub starts 2024-03-30T13:51:04.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:03 smithi043 ceph-mon[29963]: 3.19 scrub ok 2024-03-30T13:51:04.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:03 smithi043 ceph-mon[42099]: 2.0 scrub starts 2024-03-30T13:51:04.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:03 smithi043 ceph-mon[42099]: 2.0 scrub ok 2024-03-30T13:51:04.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:03 smithi043 ceph-mon[42099]: osdmap e183: 8 total, 7 up, 8 in 2024-03-30T13:51:04.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:03 smithi043 ceph-mon[42099]: 5.1f scrub starts 2024-03-30T13:51:04.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:03 smithi043 ceph-mon[42099]: 5.1f scrub ok 2024-03-30T13:51:04.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:03 smithi043 ceph-mon[42099]: 3.19 scrub starts 2024-03-30T13:51:04.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:03 smithi043 ceph-mon[42099]: 3.19 scrub ok 2024-03-30T13:51:04.162 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:04.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:04.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:04.264 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:04.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:04.293 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:04.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:04.394 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:04.425 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:04.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:04.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:04.526 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:04.557 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:04.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:04.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:04.659 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:04.694 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:04.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:04.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:04.797 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:04.834 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:04.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:04.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:04.936 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:04.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:04.970 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:04.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:05.071 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:05.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:05.114 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:05.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:05.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:04 smithi043 ceph-mon[42099]: 1.0 scrub starts 2024-03-30T13:51:05.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:04 smithi043 ceph-mon[42099]: 1.0 scrub ok 2024-03-30T13:51:05.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:04 smithi043 ceph-mon[42099]: pgmap v333: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 806 op/s; 41781/335190 objects degraded (12.465%) 2024-03-30T13:51:05.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:04 smithi043 ceph-mon[42099]: 6.1b scrub starts 2024-03-30T13:51:05.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:04 smithi043 ceph-mon[42099]: 6.1b scrub ok 2024-03-30T13:51:05.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:04 smithi043 ceph-mon[42099]: 5.1b scrub starts 2024-03-30T13:51:05.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:04 smithi043 ceph-mon[42099]: 5.1b scrub ok 2024-03-30T13:51:05.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:04 smithi043 ceph-mon[42099]: 4.a scrub starts 2024-03-30T13:51:05.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:04 smithi043 ceph-mon[42099]: 4.a scrub ok 2024-03-30T13:51:05.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:04 smithi043 ceph-mon[29963]: 1.0 scrub starts 2024-03-30T13:51:05.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:04 smithi043 ceph-mon[29963]: 1.0 scrub ok 2024-03-30T13:51:05.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:04 smithi043 ceph-mon[29963]: pgmap v333: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 806 op/s; 41781/335190 objects degraded (12.465%) 2024-03-30T13:51:05.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:04 smithi043 ceph-mon[29963]: 6.1b scrub starts 2024-03-30T13:51:05.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:04 smithi043 ceph-mon[29963]: 6.1b scrub ok 2024-03-30T13:51:05.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:04 smithi043 ceph-mon[29963]: 5.1b scrub starts 2024-03-30T13:51:05.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:04 smithi043 ceph-mon[29963]: 5.1b scrub ok 2024-03-30T13:51:05.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:04 smithi043 ceph-mon[29963]: 4.a scrub starts 2024-03-30T13:51:05.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:04 smithi043 ceph-mon[29963]: 4.a scrub ok 2024-03-30T13:51:05.215 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:05.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:04 smithi113 ceph-mon[34341]: 1.0 scrub starts 2024-03-30T13:51:05.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:04 smithi113 ceph-mon[34341]: 1.0 scrub ok 2024-03-30T13:51:05.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:04 smithi113 ceph-mon[34341]: pgmap v333: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 806 op/s; 41781/335190 objects degraded (12.465%) 2024-03-30T13:51:05.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:04 smithi113 ceph-mon[34341]: 6.1b scrub starts 2024-03-30T13:51:05.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:04 smithi113 ceph-mon[34341]: 6.1b scrub ok 2024-03-30T13:51:05.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:04 smithi113 ceph-mon[34341]: 5.1b scrub starts 2024-03-30T13:51:05.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:04 smithi113 ceph-mon[34341]: 5.1b scrub ok 2024-03-30T13:51:05.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:04 smithi113 ceph-mon[34341]: 4.a scrub starts 2024-03-30T13:51:05.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:04 smithi113 ceph-mon[34341]: 4.a scrub ok 2024-03-30T13:51:05.252 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:05.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:05.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:05.354 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:05.393 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:05.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:05.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:05.494 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:05.520 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:05.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:05.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:05.622 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:05.653 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:05.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:05.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:05.755 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:05.787 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:05.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:05.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:05.889 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:05.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:05 smithi043 ceph-mon[29963]: 5.1d scrub starts 2024-03-30T13:51:05.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:05 smithi043 ceph-mon[29963]: 5.1d scrub ok 2024-03-30T13:51:05.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:05 smithi043 ceph-mon[42099]: 5.1d scrub starts 2024-03-30T13:51:05.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:05 smithi043 ceph-mon[42099]: 5.1d scrub ok 2024-03-30T13:51:05.923 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:05.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:05.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:06.026 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:06.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:06.058 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:06.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:06.159 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:06.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:06.195 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:06.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:06.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:05 smithi113 ceph-mon[34341]: 5.1d scrub starts 2024-03-30T13:51:06.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:05 smithi113 ceph-mon[34341]: 5.1d scrub ok 2024-03-30T13:51:06.296 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:06.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:06.330 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:06.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:06.398 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:51:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:51:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:51:06.431 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:06.468 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:06.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:06.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:06.569 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:06.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:06.598 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:06.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:06.699 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:06.722 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:06.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:06.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:06.824 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:06.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:06.858 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:06.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:06.959 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:06.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:06.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:07.095 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:07.130 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:07.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:07.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:07.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:06 smithi043 ceph-mon[29963]: 8.c scrub starts 2024-03-30T13:51:07.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:06 smithi043 ceph-mon[29963]: 6.1a scrub starts 2024-03-30T13:51:07.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:06 smithi043 ceph-mon[29963]: 6.1a scrub ok 2024-03-30T13:51:07.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:06 smithi043 ceph-mon[29963]: pgmap v334: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 39400/316014 objects degraded (12.468%) 2024-03-30T13:51:07.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:06 smithi043 ceph-mon[29963]: 6.1f scrub starts 2024-03-30T13:51:07.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:06 smithi043 ceph-mon[29963]: 6.1f scrub ok 2024-03-30T13:51:07.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:06 smithi043 ceph-mon[29963]: 6.a scrub starts 2024-03-30T13:51:07.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:06 smithi043 ceph-mon[29963]: 6.a scrub ok 2024-03-30T13:51:07.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:06 smithi043 ceph-mon[42099]: 8.c scrub starts 2024-03-30T13:51:07.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:06 smithi043 ceph-mon[42099]: 6.1a scrub starts 2024-03-30T13:51:07.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:06 smithi043 ceph-mon[42099]: 6.1a scrub ok 2024-03-30T13:51:07.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:06 smithi043 ceph-mon[42099]: pgmap v334: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 39400/316014 objects degraded (12.468%) 2024-03-30T13:51:07.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:06 smithi043 ceph-mon[42099]: 6.1f scrub starts 2024-03-30T13:51:07.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:06 smithi043 ceph-mon[42099]: 6.1f scrub ok 2024-03-30T13:51:07.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:06 smithi043 ceph-mon[42099]: 6.a scrub starts 2024-03-30T13:51:07.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:06 smithi043 ceph-mon[42099]: 6.a scrub ok 2024-03-30T13:51:07.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:06 smithi113 ceph-mon[34341]: 8.c scrub starts 2024-03-30T13:51:07.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:06 smithi113 ceph-mon[34341]: 6.1a scrub starts 2024-03-30T13:51:07.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:06 smithi113 ceph-mon[34341]: 6.1a scrub ok 2024-03-30T13:51:07.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:06 smithi113 ceph-mon[34341]: pgmap v334: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 39400/316014 objects degraded (12.468%) 2024-03-30T13:51:07.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:06 smithi113 ceph-mon[34341]: 6.1f scrub starts 2024-03-30T13:51:07.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:06 smithi113 ceph-mon[34341]: 6.1f scrub ok 2024-03-30T13:51:07.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:06 smithi113 ceph-mon[34341]: 6.a scrub starts 2024-03-30T13:51:07.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:06 smithi113 ceph-mon[34341]: 6.a scrub ok 2024-03-30T13:51:07.232 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:07.262 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:07.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:07.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:07.364 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:07.364 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-30T13:51:07.365 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:51:07.365 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-30T13:51:07.365 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-03-30T13:51:07.388 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:07.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:07.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:07.490 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:07.522 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:07.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:07.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:07.624 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:07.659 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:07.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:07.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:07.761 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:07.800 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:07.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:07.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:07.902 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:07.934 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:07.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:07.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:08.035 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:08.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:08.073 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:08.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:08.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:07 smithi043 ceph-mon[29963]: 5.1a scrub starts 2024-03-30T13:51:08.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:07 smithi043 ceph-mon[29963]: 5.1a scrub ok 2024-03-30T13:51:08.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:07 smithi043 ceph-mon[29963]: 4.18 scrub starts 2024-03-30T13:51:08.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:07 smithi043 ceph-mon[29963]: 4.18 scrub ok 2024-03-30T13:51:08.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:07 smithi043 ceph-mon[29963]: 3.8 scrub starts 2024-03-30T13:51:08.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:07 smithi043 ceph-mon[29963]: 3.8 scrub ok 2024-03-30T13:51:08.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:07 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 39400/316014 objects degraded (12.468%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:51:08.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:07 smithi043 ceph-mon[29963]: osdmap e184: 8 total, 7 up, 8 in 2024-03-30T13:51:08.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:07 smithi043 ceph-mon[42099]: 5.1a scrub starts 2024-03-30T13:51:08.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:07 smithi043 ceph-mon[42099]: 5.1a scrub ok 2024-03-30T13:51:08.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:07 smithi043 ceph-mon[42099]: 4.18 scrub starts 2024-03-30T13:51:08.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:07 smithi043 ceph-mon[42099]: 4.18 scrub ok 2024-03-30T13:51:08.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:07 smithi043 ceph-mon[42099]: 3.8 scrub starts 2024-03-30T13:51:08.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:07 smithi043 ceph-mon[42099]: 3.8 scrub ok 2024-03-30T13:51:08.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:07 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 39400/316014 objects degraded (12.468%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:51:08.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:07 smithi043 ceph-mon[42099]: osdmap e184: 8 total, 7 up, 8 in 2024-03-30T13:51:08.174 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:08.207 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:08.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:08.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:08.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:07 smithi113 ceph-mon[34341]: 5.1a scrub starts 2024-03-30T13:51:08.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:07 smithi113 ceph-mon[34341]: 5.1a scrub ok 2024-03-30T13:51:08.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:07 smithi113 ceph-mon[34341]: 4.18 scrub starts 2024-03-30T13:51:08.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:07 smithi113 ceph-mon[34341]: 4.18 scrub ok 2024-03-30T13:51:08.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:07 smithi113 ceph-mon[34341]: 3.8 scrub starts 2024-03-30T13:51:08.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:07 smithi113 ceph-mon[34341]: 3.8 scrub ok 2024-03-30T13:51:08.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:07 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 39400/316014 objects degraded (12.468%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T13:51:08.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:07 smithi113 ceph-mon[34341]: osdmap e184: 8 total, 7 up, 8 in 2024-03-30T13:51:08.309 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:08.339 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:08.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:08.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:08.440 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:08.476 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:08.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:08.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:08.578 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:08.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:08.613 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:08.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:08.714 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:08.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:08.755 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:08.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:08.856 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:08.893 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:08.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:08.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:08.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:08 smithi043 ceph-mon[29963]: pgmap v335: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 39400/316014 objects degraded (12.468%) 2024-03-30T13:51:08.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:08 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:51:08.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:08 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:51:08.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:08 smithi043 ceph-mon[29963]: 8.8 scrub starts 2024-03-30T13:51:08.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:08 smithi043 ceph-mon[29963]: 5.11 scrub starts 2024-03-30T13:51:08.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:08 smithi043 ceph-mon[29963]: 5.11 scrub ok 2024-03-30T13:51:08.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:08 smithi043 ceph-mon[29963]: osdmap e185: 8 total, 7 up, 8 in 2024-03-30T13:51:08.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:08 smithi043 ceph-mon[42099]: pgmap v335: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 39400/316014 objects degraded (12.468%) 2024-03-30T13:51:08.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:08 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:51:08.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:08 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:51:08.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:08 smithi043 ceph-mon[42099]: 8.8 scrub starts 2024-03-30T13:51:08.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:08 smithi043 ceph-mon[42099]: 5.11 scrub starts 2024-03-30T13:51:08.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:08 smithi043 ceph-mon[42099]: 5.11 scrub ok 2024-03-30T13:51:08.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:08 smithi043 ceph-mon[42099]: osdmap e185: 8 total, 7 up, 8 in 2024-03-30T13:51:08.995 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:09.030 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:09.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:09.132 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:09.177 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:09.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:09.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:09.221 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:51:08 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:51:09.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:08 smithi113 ceph-mon[34341]: pgmap v335: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 39400/316014 objects degraded (12.468%) 2024-03-30T13:51:09.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:08 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:51:09.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:08 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:51:09.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:08 smithi113 ceph-mon[34341]: 8.8 scrub starts 2024-03-30T13:51:09.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:08 smithi113 ceph-mon[34341]: 5.11 scrub starts 2024-03-30T13:51:09.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:08 smithi113 ceph-mon[34341]: 5.11 scrub ok 2024-03-30T13:51:09.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:08 smithi113 ceph-mon[34341]: osdmap e185: 8 total, 7 up, 8 in 2024-03-30T13:51:09.278 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:09.314 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:09.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:09.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:09.416 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:09.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:09.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:09.551 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:09.582 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:09.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:09.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:09.684 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:09.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:09.719 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:09.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:09.762 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:51:09.762 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:51:09.820 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:09.860 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:09.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:09.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:09.961 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:09.994 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:09.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:09.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:09.997 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:09 smithi043 ceph-mon[42099]: 8.c scrub ok 2024-03-30T13:51:09.997 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:09 smithi043 ceph-mon[42099]: 5.17 scrub starts 2024-03-30T13:51:09.997 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:09 smithi043 ceph-mon[42099]: 5.17 scrub ok 2024-03-30T13:51:09.997 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:09 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:51:09.997 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:09 smithi043 ceph-mon[42099]: 5.7 scrub starts 2024-03-30T13:51:09.997 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:09 smithi043 ceph-mon[42099]: 5.7 scrub ok 2024-03-30T13:51:09.997 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:09 smithi043 ceph-mon[42099]: 5.2 scrub starts 2024-03-30T13:51:09.997 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:09 smithi043 ceph-mon[42099]: 5.2 scrub ok 2024-03-30T13:51:09.997 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:09 smithi043 ceph-mon[29963]: 8.c scrub ok 2024-03-30T13:51:09.997 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:09 smithi043 ceph-mon[29963]: 5.17 scrub starts 2024-03-30T13:51:09.997 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:09 smithi043 ceph-mon[29963]: 5.17 scrub ok 2024-03-30T13:51:09.998 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:09 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:51:09.998 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:09 smithi043 ceph-mon[29963]: 5.7 scrub starts 2024-03-30T13:51:09.998 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:09 smithi043 ceph-mon[29963]: 5.7 scrub ok 2024-03-30T13:51:09.998 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:09 smithi043 ceph-mon[29963]: 5.2 scrub starts 2024-03-30T13:51:09.998 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:09 smithi043 ceph-mon[29963]: 5.2 scrub ok 2024-03-30T13:51:10.095 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:10.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:10.134 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:10.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:10.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:09 smithi113 ceph-mon[34341]: 8.c scrub ok 2024-03-30T13:51:10.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:09 smithi113 ceph-mon[34341]: 5.17 scrub starts 2024-03-30T13:51:10.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:09 smithi113 ceph-mon[34341]: 5.17 scrub ok 2024-03-30T13:51:10.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:09 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:51:10.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:09 smithi113 ceph-mon[34341]: 5.7 scrub starts 2024-03-30T13:51:10.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:09 smithi113 ceph-mon[34341]: 5.7 scrub ok 2024-03-30T13:51:10.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:09 smithi113 ceph-mon[34341]: 5.2 scrub starts 2024-03-30T13:51:10.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:09 smithi113 ceph-mon[34341]: 5.2 scrub ok 2024-03-30T13:51:10.235 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:10.267 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:10.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:10.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:10.369 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:10.406 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:10.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:10.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:10.443 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:51:10.508 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:10.545 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:10.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:10.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:10.647 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:10.683 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:10.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:10.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:10.785 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:10.805 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:51:10.805 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:51:10.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:10.830 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:10.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:10.846 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:10 smithi043 ceph-mon[29963]: 3.d scrub starts 2024-03-30T13:51:10.846 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:10 smithi043 ceph-mon[29963]: 3.d scrub ok 2024-03-30T13:51:10.846 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:10 smithi043 ceph-mon[29963]: 8.6 scrub starts 2024-03-30T13:51:10.846 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:10 smithi043 ceph-mon[29963]: 5.1 scrub starts 2024-03-30T13:51:10.846 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:10 smithi043 ceph-mon[29963]: 5.1 scrub ok 2024-03-30T13:51:10.846 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:10 smithi043 ceph-mon[29963]: pgmap v338: 148 pgs: 2 active+clean+scrubbing, 18 active+undersized+degraded, 26 active+undersized, 102 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.81k op/s; 37271/298581 objects degraded (12.483%) 2024-03-30T13:51:10.846 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:10 smithi043 ceph-mon[29963]: 8.8 scrub ok 2024-03-30T13:51:10.847 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:10 smithi043 ceph-mon[42099]: 3.d scrub starts 2024-03-30T13:51:10.847 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:10 smithi043 ceph-mon[42099]: 3.d scrub ok 2024-03-30T13:51:10.847 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:10 smithi043 ceph-mon[42099]: 8.6 scrub starts 2024-03-30T13:51:10.847 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:10 smithi043 ceph-mon[42099]: 5.1 scrub starts 2024-03-30T13:51:10.847 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:10 smithi043 ceph-mon[42099]: 5.1 scrub ok 2024-03-30T13:51:10.847 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:10 smithi043 ceph-mon[42099]: pgmap v338: 148 pgs: 2 active+clean+scrubbing, 18 active+undersized+degraded, 26 active+undersized, 102 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.81k op/s; 37271/298581 objects degraded (12.483%) 2024-03-30T13:51:10.847 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:10 smithi043 ceph-mon[42099]: 8.8 scrub ok 2024-03-30T13:51:10.932 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:10.971 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:10.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:10.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:11.072 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:11.108 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:11.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:11.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:11.210 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:11.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:10 smithi113 ceph-mon[34341]: 3.d scrub starts 2024-03-30T13:51:11.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:10 smithi113 ceph-mon[34341]: 3.d scrub ok 2024-03-30T13:51:11.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:10 smithi113 ceph-mon[34341]: 8.6 scrub starts 2024-03-30T13:51:11.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:10 smithi113 ceph-mon[34341]: 5.1 scrub starts 2024-03-30T13:51:11.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:10 smithi113 ceph-mon[34341]: 5.1 scrub ok 2024-03-30T13:51:11.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:10 smithi113 ceph-mon[34341]: pgmap v338: 148 pgs: 2 active+clean+scrubbing, 18 active+undersized+degraded, 26 active+undersized, 102 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.81k op/s; 37271/298581 objects degraded (12.483%) 2024-03-30T13:51:11.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:10 smithi113 ceph-mon[34341]: 8.8 scrub ok 2024-03-30T13:51:11.242 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:11.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:11.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:11.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:11.383 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:11.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:11.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:11.485 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:11.520 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:11.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:11.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:11.607 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-30T13:51:11.622 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:11.653 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:11.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:11.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:11.755 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:11.764 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:51:11.788 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-03-30T13:51:11.804 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:11.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:11.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:11.837 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:11 smithi043 ceph-mon[29963]: 6.c scrub starts 2024-03-30T13:51:11.837 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:11 smithi043 ceph-mon[29963]: 6.c scrub ok 2024-03-30T13:51:11.837 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:11 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3714729309' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:51:11.837 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:11 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:51:11.837 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:11 smithi043 ceph-mon[29963]: 6.19 scrub starts 2024-03-30T13:51:11.837 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:11 smithi043 ceph-mon[29963]: 6.19 scrub ok 2024-03-30T13:51:11.837 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:11 smithi043 ceph-mon[42099]: 6.c scrub starts 2024-03-30T13:51:11.838 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:11 smithi043 ceph-mon[42099]: 6.c scrub ok 2024-03-30T13:51:11.838 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:11 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3714729309' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:51:11.838 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:11 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:51:11.838 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:11 smithi043 ceph-mon[42099]: 6.19 scrub starts 2024-03-30T13:51:11.838 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:11 smithi043 ceph-mon[42099]: 6.19 scrub ok 2024-03-30T13:51:11.906 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:11.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:11.979 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:11.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:12.080 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:12.112 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:12.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:12.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:12.214 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:12.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:11 smithi113 ceph-mon[34341]: 6.c scrub starts 2024-03-30T13:51:12.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:11 smithi113 ceph-mon[34341]: 6.c scrub ok 2024-03-30T13:51:12.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:11 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3714729309' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:51:12.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:11 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:51:12.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:11 smithi113 ceph-mon[34341]: 6.19 scrub starts 2024-03-30T13:51:12.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:11 smithi113 ceph-mon[34341]: 6.19 scrub ok 2024-03-30T13:51:12.254 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:12.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:12.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:12.356 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:12.384 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:12.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:12.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:12.486 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:12.521 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:12.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:12.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:12.623 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:12.657 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:12.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:12.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:12.759 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:12.800 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:12.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:12.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:12.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:12 smithi043 ceph-mon[29963]: 8.6 scrub ok 2024-03-30T13:51:12.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:12 smithi043 ceph-mon[29963]: pgmap v339: 148 pgs: 2 active+clean+scrubbing, 18 active+undersized+degraded, 26 active+undersized, 102 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 37271/298581 objects degraded (12.483%) 2024-03-30T13:51:12.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:12 smithi043 ceph-mon[29963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:51:12.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:12 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:51:12.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:12 smithi043 ceph-mon[29963]: osdmap e186: 8 total, 7 up, 8 in 2024-03-30T13:51:12.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:12 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 37271/298581 objects degraded (12.483%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:12.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:12 smithi043 ceph-mon[42099]: 8.6 scrub ok 2024-03-30T13:51:12.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:12 smithi043 ceph-mon[42099]: pgmap v339: 148 pgs: 2 active+clean+scrubbing, 18 active+undersized+degraded, 26 active+undersized, 102 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 37271/298581 objects degraded (12.483%) 2024-03-30T13:51:12.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:12 smithi043 ceph-mon[42099]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:51:12.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:12 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:51:12.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:12 smithi043 ceph-mon[42099]: osdmap e186: 8 total, 7 up, 8 in 2024-03-30T13:51:12.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:12 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 37271/298581 objects degraded (12.483%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:12.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:12.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:12.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:13.035 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:13.070 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:13.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:13.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:13.172 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:13.206 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:13.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:13.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:13.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:12 smithi113 ceph-mon[34341]: 8.6 scrub ok 2024-03-30T13:51:13.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:12 smithi113 ceph-mon[34341]: pgmap v339: 148 pgs: 2 active+clean+scrubbing, 18 active+undersized+degraded, 26 active+undersized, 102 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 37271/298581 objects degraded (12.483%) 2024-03-30T13:51:13.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:12 smithi113 ceph-mon[34341]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:51:13.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:12 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:51:13.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:12 smithi113 ceph-mon[34341]: osdmap e186: 8 total, 7 up, 8 in 2024-03-30T13:51:13.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:12 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 37271/298581 objects degraded (12.483%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:13.308 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:13.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:13.332 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:13.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:13.433 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:13.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:13.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:13.573 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:13.606 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:13.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:13.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:13.707 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:13.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:13.743 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:13.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:13.844 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:13.882 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:13.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:13.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:13.984 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:14.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:14.029 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:14.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:14.130 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:14.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:13 smithi043 ceph-mon[29963]: 3.6 scrub starts 2024-03-30T13:51:14.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:13 smithi043 ceph-mon[29963]: 3.6 scrub ok 2024-03-30T13:51:14.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:13 smithi043 ceph-mon[29963]: 4.1b scrub starts 2024-03-30T13:51:14.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:13 smithi043 ceph-mon[29963]: 4.1b scrub ok 2024-03-30T13:51:14.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:13 smithi043 ceph-mon[29963]: 2.2 scrub starts 2024-03-30T13:51:14.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:13 smithi043 ceph-mon[29963]: 2.2 scrub ok 2024-03-30T13:51:14.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:13 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:51:14.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:13 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:51:14.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:13 smithi043 ceph-mon[29963]: 6.13 scrub starts 2024-03-30T13:51:14.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:13 smithi043 ceph-mon[42099]: 3.6 scrub starts 2024-03-30T13:51:14.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:13 smithi043 ceph-mon[42099]: 3.6 scrub ok 2024-03-30T13:51:14.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:13 smithi043 ceph-mon[42099]: 4.1b scrub starts 2024-03-30T13:51:14.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:13 smithi043 ceph-mon[42099]: 4.1b scrub ok 2024-03-30T13:51:14.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:13 smithi043 ceph-mon[42099]: 2.2 scrub starts 2024-03-30T13:51:14.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:13 smithi043 ceph-mon[42099]: 2.2 scrub ok 2024-03-30T13:51:14.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:13 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:51:14.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:13 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:51:14.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:13 smithi043 ceph-mon[42099]: 6.13 scrub starts 2024-03-30T13:51:14.170 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:14.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:14.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:14.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:13 smithi113 ceph-mon[34341]: 3.6 scrub starts 2024-03-30T13:51:14.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:13 smithi113 ceph-mon[34341]: 3.6 scrub ok 2024-03-30T13:51:14.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:13 smithi113 ceph-mon[34341]: 4.1b scrub starts 2024-03-30T13:51:14.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:13 smithi113 ceph-mon[34341]: 4.1b scrub ok 2024-03-30T13:51:14.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:13 smithi113 ceph-mon[34341]: 2.2 scrub starts 2024-03-30T13:51:14.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:13 smithi113 ceph-mon[34341]: 2.2 scrub ok 2024-03-30T13:51:14.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:13 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:51:14.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:13 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:51:14.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:13 smithi113 ceph-mon[34341]: 6.13 scrub starts 2024-03-30T13:51:14.272 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:14.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:14.308 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:14.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:14.409 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:14.441 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:14.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:14.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:14.543 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:14.575 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:14.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:14.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:14.677 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:14.712 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:14.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:14.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:14.814 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:14.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:14.852 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:14.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:14.953 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:14.976 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:14.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:14.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:15.078 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:15.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:15.114 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:15.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:15.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:14 smithi043 ceph-mon[29963]: 4.b scrub starts 2024-03-30T13:51:15.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:14 smithi043 ceph-mon[29963]: 4.b scrub ok 2024-03-30T13:51:15.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:14 smithi043 ceph-mon[29963]: pgmap v341: 148 pgs: 2 active+clean+scrubbing, 18 active+undersized+degraded, 26 active+undersized, 102 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 968 op/s; 37271/298581 objects degraded (12.483%) 2024-03-30T13:51:15.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:14 smithi043 ceph-mon[29963]: 6.13 scrub ok 2024-03-30T13:51:15.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:14 smithi043 ceph-mon[29963]: 8.b deep-scrub starts 2024-03-30T13:51:15.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:14 smithi043 ceph-mon[42099]: 4.b scrub starts 2024-03-30T13:51:15.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:14 smithi043 ceph-mon[42099]: 4.b scrub ok 2024-03-30T13:51:15.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:14 smithi043 ceph-mon[42099]: pgmap v341: 148 pgs: 2 active+clean+scrubbing, 18 active+undersized+degraded, 26 active+undersized, 102 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 968 op/s; 37271/298581 objects degraded (12.483%) 2024-03-30T13:51:15.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:14 smithi043 ceph-mon[42099]: 6.13 scrub ok 2024-03-30T13:51:15.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:14 smithi043 ceph-mon[42099]: 8.b deep-scrub starts 2024-03-30T13:51:15.215 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:15.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:14 smithi113 ceph-mon[34341]: 4.b scrub starts 2024-03-30T13:51:15.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:14 smithi113 ceph-mon[34341]: 4.b scrub ok 2024-03-30T13:51:15.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:14 smithi113 ceph-mon[34341]: pgmap v341: 148 pgs: 2 active+clean+scrubbing, 18 active+undersized+degraded, 26 active+undersized, 102 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 968 op/s; 37271/298581 objects degraded (12.483%) 2024-03-30T13:51:15.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:14 smithi113 ceph-mon[34341]: 6.13 scrub ok 2024-03-30T13:51:15.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:14 smithi113 ceph-mon[34341]: 8.b deep-scrub starts 2024-03-30T13:51:15.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:15.251 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:15.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:15.265 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T13:51:15.352 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:15.385 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:15.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:15.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:15.487 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:15.529 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:51:15.549 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:15.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:15.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:15.651 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:15.688 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:15.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:15.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:15.790 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:15.824 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:15.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:15.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:15.926 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:15.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:15.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:15.965 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:15 smithi043 ceph-mon[29963]: 6.5 scrub starts 2024-03-30T13:51:15.965 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:15 smithi043 ceph-mon[29963]: 6.5 scrub ok 2024-03-30T13:51:15.965 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:15 smithi043 ceph-mon[42099]: 6.5 scrub starts 2024-03-30T13:51:15.965 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:15 smithi043 ceph-mon[42099]: 6.5 scrub ok 2024-03-30T13:51:16.066 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:16.107 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:16.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:16.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:16.209 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:16.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:15 smithi113 ceph-mon[34341]: 6.5 scrub starts 2024-03-30T13:51:16.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:15 smithi113 ceph-mon[34341]: 6.5 scrub ok 2024-03-30T13:51:16.241 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:16.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:16.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:16.296 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:51:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:51:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:51:16.343 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:16.380 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:16.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:16.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:16.404 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:51:16.482 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:16.510 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:16.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:16.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:16.612 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:16.648 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:16.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:16.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:16.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:16.763 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:51:16.764 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:51:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 186, 2024-03-30T13:51:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "7ee932fa-ee9a-11ee-b647-cb9ed24678a4", 2024-03-30T13:51:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-03-30T13:37:07.346020+0000", 2024-03-30T13:51:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-03-30T13:51:11.765740+0000", 2024-03-30T13:51:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "last_up_change": "2024-03-30T13:50:07.489747+0000", 2024-03-30T13:51:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "last_in_change": "2024-03-30T13:42:12.831084+0000", 2024-03-30T13:51:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-30T13:51:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_num": 5799936, 2024-03-30T13:51:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_set": [ 2024-03-30T13:51:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "pglog_hardlimit", 2024-03-30T13:51:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "purged_snapdirs", 2024-03-30T13:51:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "recovery_deletes", 2024-03-30T13:51:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "sortbitwise" 2024-03-30T13:51:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:51:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_version": 18, 2024-03-30T13:51:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "full_ratio": 0.94999998807907104, 2024-03-30T13:51:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-30T13:51:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-30T13:51:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_snapshot": "", 2024-03-30T13:51:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_max": 8, 2024-03-30T13:51:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "max_osd": 8, 2024-03-30T13:51:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "require_min_compat_client": "luminous", 2024-03-30T13:51:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "min_compat_client": "luminous", 2024-03-30T13:51:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "require_osd_release": "squid", 2024-03-30T13:51:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_crimson": false, 2024-03-30T13:51:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "pools": [ 2024-03-30T13:51:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 1, 2024-03-30T13:51:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".mgr", 2024-03-30T13:51:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:40:27.658979+0000", 2024-03-30T13:51:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T13:51:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T13:51:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:51:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:51:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:51:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:51:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:51:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:51:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:51:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:51:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:51:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:51:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 1, 2024-03-30T13:51:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 1, 2024-03-30T13:51:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 1, 2024-03-30T13:51:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 1, 2024-03-30T13:51:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 1, 2024-03-30T13:51:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:51:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:51:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:51:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:51:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:51:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:51:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:51:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "21", 2024-03-30T13:51:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:51:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:51:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T13:51:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:51:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:51:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T13:51:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T13:51:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:51:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:51:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:51:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:51:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:51:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:51:16.770 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:51:16.770 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:51:16.770 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:51:16.770 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:51:16.770 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:51:16.770 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:51:16.770 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:51:16.770 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:51:16.770 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:51:16.770 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:51:16.770 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:51:16.770 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:51:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:51:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:51:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:51:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:51:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:51:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:51:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:51:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:51:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:51:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:51:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:51:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:51:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-03-30T13:51:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_max": 32, 2024-03-30T13:51:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_min": 1 2024-03-30T13:51:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:51:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": {} 2024-03-30T13:51:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:51:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:51:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 6.9800000190734863, 2024-03-30T13:51:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 6.9800000190734863, 2024-03-30T13:51:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.43000000715255737, 2024-03-30T13:51:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 3, 2024-03-30T13:51:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 3, 2024-03-30T13:51:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:51:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:51:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:51:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 2, 2024-03-30T13:51:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "datapool", 2024-03-30T13:51:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:42.989975+0000", 2024-03-30T13:51:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 8193, 2024-03-30T13:51:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T13:51:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:51:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:51:16.774 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:51:16.774 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:51:16.774 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:51:16.774 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:51:16.774 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:51:16.774 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:51:16.774 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:51:16.774 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:51:16.774 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 3, 2024-03-30T13:51:16.774 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 3, 2024-03-30T13:51:16.774 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 3, 2024-03-30T13:51:16.774 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 3, 2024-03-30T13:51:16.774 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 3, 2024-03-30T13:51:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:51:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:51:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:51:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:51:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:51:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:51:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:51:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "68", 2024-03-30T13:51:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:51:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:51:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T13:51:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:51:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:51:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 2, 2024-03-30T13:51:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 68, 2024-03-30T13:51:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:51:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:51:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:51:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:51:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:51:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:51:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:51:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:51:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:51:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:51:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:51:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:51:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:51:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:51:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:51:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:51:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:51:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:51:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:51:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:51:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:51:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:51:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:51:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:51:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:51:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:51:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:51:16.778 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:51:16.778 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:51:16.778 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:51:16.778 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T13:51:16.778 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:51:16.778 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd": {} 2024-03-30T13:51:16.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.778 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:51:16.778 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:51:16.778 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.3299999237060547, 2024-03-30T13:51:16.778 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.3299999237060547, 2024-03-30T13:51:16.778 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.86000001430511475, 2024-03-30T13:51:16.778 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2, 2024-03-30T13:51:16.779 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2, 2024-03-30T13:51:16.779 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:51:16.779 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:51:16.779 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:51:16.779 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.779 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 3, 2024-03-30T13:51:16.779 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".rgw.root", 2024-03-30T13:51:16.779 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:43.910306+0000", 2024-03-30T13:51:16.779 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T13:51:16.779 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T13:51:16.779 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:51:16.779 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:51:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:51:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:51:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:51:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:51:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:51:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:51:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:51:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:51:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T13:51:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T13:51:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T13:51:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T13:51:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T13:51:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:51:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:51:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:51:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:51:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:51:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:51:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:51:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "66", 2024-03-30T13:51:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:51:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:51:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T13:51:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:51:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:51:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T13:51:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T13:51:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:51:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:51:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:51:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:51:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:51:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:51:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:51:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:51:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:51:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:51:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:51:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:51:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:51:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:51:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:51:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:51:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:51:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:51:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:51:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:51:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:51:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:51:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:51:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:51:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:51:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:51:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:51:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:51:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:51:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:51:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T13:51:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:51:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T13:51:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:51:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:51:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.309999942779541, 2024-03-30T13:51:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.309999942779541, 2024-03-30T13:51:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1, 2024-03-30T13:51:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.309999942779541, 2024-03-30T13:51:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.309999942779541, 2024-03-30T13:51:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:51:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:51:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:51:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 4, 2024-03-30T13:51:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.log", 2024-03-30T13:51:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:45.633132+0000", 2024-03-30T13:51:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T13:51:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T13:51:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:51:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:51:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:51:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:51:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:51:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:51:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:51:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:51:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:51:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:51:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T13:51:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T13:51:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T13:51:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T13:51:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T13:51:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:51:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:51:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:51:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:51:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:51:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:51:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:51:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "68", 2024-03-30T13:51:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:51:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:51:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T13:51:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:51:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:51:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T13:51:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T13:51:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:51:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:51:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:51:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:51:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:51:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:51:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:51:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:51:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:51:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:51:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:51:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:51:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:51:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:51:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:51:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:51:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:51:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:51:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:51:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:51:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:51:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:51:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:51:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:51:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:51:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:51:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:51:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:51:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:51:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:51:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T13:51:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:51:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T13:51:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:51:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:51:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.9700000286102295, 2024-03-30T13:51:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.9700000286102295, 2024-03-30T13:51:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1, 2024-03-30T13:51:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-30T13:51:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-30T13:51:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:51:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:51:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:51:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 5, 2024-03-30T13:51:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.control", 2024-03-30T13:51:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:47.570084+0000", 2024-03-30T13:51:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T13:51:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T13:51:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:51:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:51:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:51:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:51:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:51:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:51:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:51:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:51:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:51:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:51:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T13:51:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T13:51:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T13:51:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T13:51:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T13:51:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:51:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:51:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:51:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:51:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:51:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:51:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:51:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "70", 2024-03-30T13:51:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:51:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:51:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T13:51:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:51:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:51:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T13:51:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T13:51:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:51:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:51:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:51:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:51:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:51:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:51:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:51:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:51:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:51:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:51:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:51:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:51:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:51:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:51:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:51:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:51:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:51:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:51:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:51:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:51:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:51:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:51:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:51:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:51:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:51:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:51:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:51:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:51:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:51:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:51:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T13:51:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:51:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T13:51:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:51:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:51:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.309999942779541, 2024-03-30T13:51:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.309999942779541, 2024-03-30T13:51:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1, 2024-03-30T13:51:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.309999942779541, 2024-03-30T13:51:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.309999942779541, 2024-03-30T13:51:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:51:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:51:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:51:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 6, 2024-03-30T13:51:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.meta", 2024-03-30T13:51:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:49.637189+0000", 2024-03-30T13:51:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T13:51:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T13:51:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:51:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:51:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:51:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:51:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:51:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:51:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:51:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:51:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:51:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:51:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T13:51:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T13:51:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T13:51:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T13:51:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T13:51:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:51:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:51:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:51:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:51:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:51:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:51:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:51:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "72", 2024-03-30T13:51:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:51:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:51:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T13:51:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:51:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:51:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T13:51:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T13:51:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:51:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:51:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:51:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:51:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:51:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:51:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:51:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:51:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:51:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:51:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:51:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:51:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:51:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:51:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:51:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:51:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:51:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:51:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:51:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:51:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:51:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:51:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:51:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:51:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:51:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:51:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:51:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:51:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:51:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:51:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-03-30T13:51:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_bias": 4 2024-03-30T13:51:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:51:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T13:51:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:51:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:51:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.75, 2024-03-30T13:51:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.75, 2024-03-30T13:51:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1, 2024-03-30T13:51:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.75, 2024-03-30T13:51:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.75, 2024-03-30T13:51:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:51:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:51:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:51:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 8, 2024-03-30T13:51:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "unique_pool_1", 2024-03-30T13:51:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:48:59.919716+0000", 2024-03-30T13:51:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T13:51:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T13:51:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:51:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:51:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:51:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:51:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:51:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:51:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:51:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:51:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:51:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:51:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 16, 2024-03-30T13:51:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 16, 2024-03-30T13:51:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 16, 2024-03-30T13:51:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 16, 2024-03-30T13:51:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 16, 2024-03-30T13:51:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:51:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:51:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:51:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:51:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:51:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:51:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:51:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "151", 2024-03-30T13:51:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:51:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:51:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T13:51:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:51:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:51:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T13:51:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T13:51:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:51:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:51:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:51:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:51:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:51:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:51:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:51:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:51:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:51:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:51:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:51:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:51:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:51:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:51:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:51:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:51:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:51:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:51:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:51:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:51:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:51:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:51:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:51:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:51:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:51:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:51:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:51:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:51:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:51:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:51:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T13:51:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:51:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "rados": {} 2024-03-30T13:51:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:51:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:51:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.75, 2024-03-30T13:51:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.75, 2024-03-30T13:51:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1, 2024-03-30T13:51:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.75, 2024-03-30T13:51:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.75, 2024-03-30T13:51:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:51:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:51:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:51:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:51:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:51:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-03-30T13:51:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "94ea083e-6a27-41db-83af-2f7d3cae1956", 2024-03-30T13:51:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:51:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:51:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:51:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:51:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:51:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:51:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 8, 2024-03-30T13:51:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 167, 2024-03-30T13:51:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:51:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:51:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:51:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6802", 2024-03-30T13:51:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:51:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6803", 2024-03-30T13:51:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:51:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:51:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6804", 2024-03-30T13:51:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:51:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6805", 2024-03-30T13:51:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:51:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:51:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6808", 2024-03-30T13:51:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:51:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6809", 2024-03-30T13:51:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:51:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.814 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:51:16.814 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.814 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.814 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.814 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6806", 2024-03-30T13:51:16.814 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:51:16.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.814 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:16.815 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:16.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:16.816 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.816 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6807", 2024-03-30T13:51:16.816 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:51:16.816 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.816 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.816 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.816 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6803/1144592572", 2024-03-30T13:51:16.816 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6805/1144592572", 2024-03-30T13:51:16.816 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6809/1144592572", 2024-03-30T13:51:16.816 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6807/1144592572", 2024-03-30T13:51:16.816 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:51:16.816 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:51:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:51:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-03-30T13:51:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a", 2024-03-30T13:51:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:51:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:51:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:51:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:51:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:51:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:51:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 13, 2024-03-30T13:51:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 167, 2024-03-30T13:51:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:51:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:51:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:51:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6810", 2024-03-30T13:51:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:51:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6811", 2024-03-30T13:51:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:51:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:51:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6812", 2024-03-30T13:51:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:51:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6813", 2024-03-30T13:51:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:51:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:51:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6816", 2024-03-30T13:51:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:51:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6817", 2024-03-30T13:51:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:51:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:51:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6814", 2024-03-30T13:51:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:51:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6815", 2024-03-30T13:51:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:51:16.822 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.822 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.822 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6811/3901165928", 2024-03-30T13:51:16.822 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6813/3901165928", 2024-03-30T13:51:16.822 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6817/3901165928", 2024-03-30T13:51:16.822 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6815/3901165928", 2024-03-30T13:51:16.822 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:51:16.822 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:51:16.822 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:51:16.822 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-03-30T13:51:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "6db6782e-5889-443e-bc6c-df47a1f93bc8", 2024-03-30T13:51:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 0, 2024-03-30T13:51:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:51:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:51:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 0, 2024-03-30T13:51:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:51:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:51:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 18, 2024-03-30T13:51:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 98, 2024-03-30T13:51:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 167, 2024-03-30T13:51:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:51:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:51:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6818", 2024-03-30T13:51:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:51:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6819", 2024-03-30T13:51:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:51:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:51:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6820", 2024-03-30T13:51:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:51:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6821", 2024-03-30T13:51:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:51:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:51:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.826 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.826 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6824", 2024-03-30T13:51:16.826 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:51:16.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.826 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.826 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6825", 2024-03-30T13:51:16.826 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:51:16.826 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.826 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.826 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:51:16.826 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6822", 2024-03-30T13:51:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:51:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6823", 2024-03-30T13:51:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:51:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6819/3415037523", 2024-03-30T13:51:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6821/3415037523", 2024-03-30T13:51:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6825/3415037523", 2024-03-30T13:51:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6823/3415037523", 2024-03-30T13:51:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:51:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: "exists" 2024-03-30T13:51:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-03-30T13:51:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "7048ff46-2cba-4d18-9835-95ecbddb2bb6", 2024-03-30T13:51:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:51:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:51:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:51:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:51:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:51:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:51:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 25, 2024-03-30T13:51:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 167, 2024-03-30T13:51:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:51:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:51:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:51:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6826", 2024-03-30T13:51:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:51:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6827", 2024-03-30T13:51:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:51:16.830 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.830 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.830 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:51:16.830 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.830 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.830 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.830 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6828", 2024-03-30T13:51:16.830 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:51:16.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.830 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.830 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.830 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6829", 2024-03-30T13:51:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:51:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:51:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6832", 2024-03-30T13:51:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:51:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6833", 2024-03-30T13:51:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:51:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:51:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6830", 2024-03-30T13:51:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:51:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6831", 2024-03-30T13:51:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:51:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6827/2933589376", 2024-03-30T13:51:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6829/2933589376", 2024-03-30T13:51:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6833/2933589376", 2024-03-30T13:51:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6831/2933589376", 2024-03-30T13:51:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:51:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:51:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:51:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-03-30T13:51:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "42da30e6-9eec-4ed5-8c73-9ce78aa8ff63", 2024-03-30T13:51:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:51:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:51:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:51:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:51:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:51:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:51:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 32, 2024-03-30T13:51:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 167, 2024-03-30T13:51:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:51:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:51:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:51:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6800", 2024-03-30T13:51:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:51:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6801", 2024-03-30T13:51:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:51:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:51:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6802", 2024-03-30T13:51:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:51:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6803", 2024-03-30T13:51:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:51:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:51:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6806", 2024-03-30T13:51:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:51:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6807", 2024-03-30T13:51:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:51:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:51:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6804", 2024-03-30T13:51:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:51:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6805", 2024-03-30T13:51:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:51:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6801/1945833081", 2024-03-30T13:51:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6803/1945833081", 2024-03-30T13:51:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6807/1945833081", 2024-03-30T13:51:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6805/1945833081", 2024-03-30T13:51:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:51:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:51:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:51:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-03-30T13:51:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "99a11b81-0df7-4b1d-946a-4297ba62e263", 2024-03-30T13:51:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:51:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:51:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:51:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:51:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:51:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:51:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 40, 2024-03-30T13:51:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 167, 2024-03-30T13:51:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:51:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:51:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:51:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6808", 2024-03-30T13:51:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:51:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6809", 2024-03-30T13:51:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:51:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:51:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6810", 2024-03-30T13:51:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:51:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6811", 2024-03-30T13:51:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:51:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:51:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6814", 2024-03-30T13:51:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:51:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6815", 2024-03-30T13:51:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:51:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:51:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6812", 2024-03-30T13:51:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:51:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6813", 2024-03-30T13:51:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:51:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6809/3426360325", 2024-03-30T13:51:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6811/3426360325", 2024-03-30T13:51:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6815/3426360325", 2024-03-30T13:51:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6813/3426360325", 2024-03-30T13:51:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:51:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:51:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:51:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-03-30T13:51:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "cafd0e26-78e9-4164-a43a-96ca8382fefe", 2024-03-30T13:51:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:51:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:51:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:51:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:51:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:51:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:51:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 49, 2024-03-30T13:51:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 147, 2024-03-30T13:51:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:51:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:51:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:51:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6816", 2024-03-30T13:51:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:51:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6817", 2024-03-30T13:51:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:51:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:51:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6818", 2024-03-30T13:51:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:51:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6819", 2024-03-30T13:51:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:51:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:51:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6822", 2024-03-30T13:51:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:51:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6823", 2024-03-30T13:51:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:51:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:51:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6820", 2024-03-30T13:51:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:51:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6821", 2024-03-30T13:51:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:51:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6817/1527017698", 2024-03-30T13:51:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6819/1527017698", 2024-03-30T13:51:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6823/1527017698", 2024-03-30T13:51:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6821/1527017698", 2024-03-30T13:51:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:51:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:51:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:51:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-03-30T13:51:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "e9d59662-9132-48f4-9e95-842a3cfbd79f", 2024-03-30T13:51:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:51:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:51:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:51:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:51:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:51:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:51:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 58, 2024-03-30T13:51:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 167, 2024-03-30T13:51:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:51:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:51:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:51:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.852 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.852 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6824", 2024-03-30T13:51:16.852 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:51:16.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.852 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.852 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6825", 2024-03-30T13:51:16.852 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:51:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:51:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6826", 2024-03-30T13:51:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:51:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6827", 2024-03-30T13:51:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:51:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:51:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6830", 2024-03-30T13:51:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:51:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6831", 2024-03-30T13:51:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:51:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:51:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:51:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:51:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6828", 2024-03-30T13:51:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:51:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:51:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6829", 2024-03-30T13:51:16.856 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:51:16.856 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.856 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.856 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6825/3182476015", 2024-03-30T13:51:16.856 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6827/3182476015", 2024-03-30T13:51:16.856 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6831/3182476015", 2024-03-30T13:51:16.856 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6829/3182476015", 2024-03-30T13:51:16.856 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:51:16.856 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:51:16.856 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:51:16.856 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.856 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.856 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:51:16.857 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_xinfo": [ 2024-03-30T13:51:16.857 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.857 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-03-30T13:51:16.857 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:51:16.857 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:51:16.857 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:51:16.857 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:51:16.857 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:51:16.857 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:51:07.544068+0000", 2024-03-30T13:51:16.857 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:51:16.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.857 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.857 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-03-30T13:51:16.857 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:51:16.857 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:51:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:51:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:51:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:51:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:51:02.271123+0000", 2024-03-30T13:51:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:51:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-03-30T13:51:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:51:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:51:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:51:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:51:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:51:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:49:03.444490+0000", 2024-03-30T13:51:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 166 2024-03-30T13:51:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-03-30T13:51:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:51:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:51:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:51:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:51:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:51:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:50:30.249679+0000", 2024-03-30T13:51:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:51:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-03-30T13:51:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:51:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:51:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:51:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:51:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:51:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:50:19.616009+0000", 2024-03-30T13:51:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:51:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-03-30T13:51:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:51:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:51:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:51:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:51:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:51:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:50:27.087978+0000", 2024-03-30T13:51:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:51:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-03-30T13:51:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:51:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:51:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:51:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:51:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:51:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:50:42.636212+0000", 2024-03-30T13:51:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:51:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-03-30T13:51:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:51:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:51:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:51:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:51:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:51:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:50:05.355621+0000", 2024-03-30T13:51:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:51:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:51:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap": [], 2024-03-30T13:51:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_items": [ 2024-03-30T13:51:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.b", 2024-03-30T13:51:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-03-30T13:51:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-03-30T13:51:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 5 2024-03-30T13:51:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.13", 2024-03-30T13:51:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-03-30T13:51:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-03-30T13:51:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-03-30T13:51:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.14", 2024-03-30T13:51:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-03-30T13:51:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-03-30T13:51:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-03-30T13:51:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:51:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_primaries": [], 2024-03-30T13:51:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_temp": [], 2024-03-30T13:51:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_temp": [], 2024-03-30T13:51:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklist": { 2024-03-30T13:51:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/538304234": "2024-03-31T13:43:27.253365+0000", 2024-03-30T13:51:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6801/3411723337": "2024-03-31T13:37:41.733515+0000", 2024-03-30T13:51:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3656823679": "2024-03-31T13:37:41.733515+0000", 2024-03-30T13:51:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2618817647": "2024-03-31T13:38:09.321740+0000", 2024-03-30T13:51:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/3411723337": "2024-03-31T13:37:41.733515+0000", 2024-03-30T13:51:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1206218784": "2024-03-31T13:37:41.733515+0000", 2024-03-30T13:51:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/886631486": "2024-03-31T13:37:41.733515+0000", 2024-03-30T13:51:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/1339921781": "2024-03-31T13:38:09.321740+0000", 2024-03-30T13:51:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6801/1339921781": "2024-03-31T13:38:09.321740+0000", 2024-03-30T13:51:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2794494528": "2024-03-31T13:38:09.321740+0000", 2024-03-30T13:51:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/564868778": "2024-03-31T13:38:09.321740+0000", 2024-03-30T13:51:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/3033707884": "2024-03-31T13:43:27.253365+0000", 2024-03-30T13:51:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3539398453": "2024-03-31T13:43:27.253365+0000", 2024-03-30T13:51:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2414364740": "2024-03-31T13:43:27.253365+0000", 2024-03-30T13:51:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6801/3033707884": "2024-03-31T13:43:27.253365+0000", 2024-03-30T13:51:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3923597429": "2024-03-31T13:43:27.253365+0000", 2024-03-30T13:51:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2524269526": "2024-03-31T13:43:27.253365+0000" 2024-03-30T13:51:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: "range_blocklist": {}, 2024-03-30T13:51:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profiles": { 2024-03-30T13:51:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: "default": { 2024-03-30T13:51:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: "crush-failure-domain": "osd", 2024-03-30T13:51:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: "k": "2", 2024-03-30T13:51:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: "m": "1", 2024-03-30T13:51:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: "plugin": "jerasure", 2024-03-30T13:51:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: "technique": "reed_sol_van" 2024-03-30T13:51:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps_queue": [], 2024-03-30T13:51:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: "new_removed_snaps": [], 2024-03-30T13:51:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: "new_purged_snaps": [], 2024-03-30T13:51:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_node_flags": {}, 2024-03-30T13:51:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: "device_class_flags": {}, 2024-03-30T13:51:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": { 2024-03-30T13:51:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_enabled": false, 2024-03-30T13:51:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_bucket_count": 0, 2024-03-30T13:51:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: "degraded_stretch_mode": 0, 2024-03-30T13:51:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: "recovering_stretch_mode": 0, 2024-03-30T13:51:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_bucket": 0 2024-03-30T13:51:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:16.867 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:51:16.916 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:16.948 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:16.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:16.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:17.050 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:17.081 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:17.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:17.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:17.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:16 smithi043 ceph-mon[29963]: 5.e deep-scrub starts 2024-03-30T13:51:17.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:16 smithi043 ceph-mon[29963]: 5.e deep-scrub ok 2024-03-30T13:51:17.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:16 smithi043 ceph-mon[29963]: 3.e scrub starts 2024-03-30T13:51:17.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:16 smithi043 ceph-mon[29963]: 3.e scrub ok 2024-03-30T13:51:17.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:16 smithi043 ceph-mon[29963]: pgmap v342: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s; 34863/278994 objects degraded (12.496%) 2024-03-30T13:51:17.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:16 smithi043 ceph-mon[29963]: 3.1c deep-scrub starts 2024-03-30T13:51:17.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:16 smithi043 ceph-mon[29963]: 3.1c deep-scrub ok 2024-03-30T13:51:17.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:16 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3867233745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T13:51:17.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:16 smithi043 ceph-mon[42099]: 5.e deep-scrub starts 2024-03-30T13:51:17.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:16 smithi043 ceph-mon[42099]: 5.e deep-scrub ok 2024-03-30T13:51:17.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:16 smithi043 ceph-mon[42099]: 3.e scrub starts 2024-03-30T13:51:17.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:16 smithi043 ceph-mon[42099]: 3.e scrub ok 2024-03-30T13:51:17.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:16 smithi043 ceph-mon[42099]: pgmap v342: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s; 34863/278994 objects degraded (12.496%) 2024-03-30T13:51:17.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:16 smithi043 ceph-mon[42099]: 3.1c deep-scrub starts 2024-03-30T13:51:17.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:16 smithi043 ceph-mon[42099]: 3.1c deep-scrub ok 2024-03-30T13:51:17.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:16 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3867233745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T13:51:17.182 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:17.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:17.221 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:17.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:16 smithi113 ceph-mon[34341]: 5.e deep-scrub starts 2024-03-30T13:51:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:16 smithi113 ceph-mon[34341]: 5.e deep-scrub ok 2024-03-30T13:51:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:16 smithi113 ceph-mon[34341]: 3.e scrub starts 2024-03-30T13:51:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:16 smithi113 ceph-mon[34341]: 3.e scrub ok 2024-03-30T13:51:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:16 smithi113 ceph-mon[34341]: pgmap v342: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s; 34863/278994 objects degraded (12.496%) 2024-03-30T13:51:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:16 smithi113 ceph-mon[34341]: 3.1c deep-scrub starts 2024-03-30T13:51:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:16 smithi113 ceph-mon[34341]: 3.1c deep-scrub ok 2024-03-30T13:51:17.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:16 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3867233745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T13:51:17.322 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:17.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:17.358 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:17.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:17.459 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:17.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:17.493 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:17.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:17.594 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:17.626 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:17.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:17.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:17.728 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:17.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:17.758 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:17.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:17.859 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:17.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:17.894 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:17.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:17.995 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:18.031 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:18.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:18.133 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:18.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:17 smithi043 ceph-mon[29963]: 4.1a scrub starts 2024-03-30T13:51:18.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:17 smithi043 ceph-mon[29963]: 4.1a scrub ok 2024-03-30T13:51:18.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:17 smithi043 ceph-mon[29963]: 4.0 scrub starts 2024-03-30T13:51:18.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:17 smithi043 ceph-mon[29963]: 4.0 scrub ok 2024-03-30T13:51:18.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:17 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 34863/278994 objects degraded (12.496%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:18.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:17 smithi043 ceph-mon[42099]: 4.1a scrub starts 2024-03-30T13:51:18.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:17 smithi043 ceph-mon[42099]: 4.1a scrub ok 2024-03-30T13:51:18.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:17 smithi043 ceph-mon[42099]: 4.0 scrub starts 2024-03-30T13:51:18.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:17 smithi043 ceph-mon[42099]: 4.0 scrub ok 2024-03-30T13:51:18.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:17 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 34863/278994 objects degraded (12.496%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:18.164 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:18.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:18.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:18.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:17 smithi113 ceph-mon[34341]: 4.1a scrub starts 2024-03-30T13:51:18.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:17 smithi113 ceph-mon[34341]: 4.1a scrub ok 2024-03-30T13:51:18.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:17 smithi113 ceph-mon[34341]: 4.0 scrub starts 2024-03-30T13:51:18.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:17 smithi113 ceph-mon[34341]: 4.0 scrub ok 2024-03-30T13:51:18.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:17 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 34863/278994 objects degraded (12.496%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:18.266 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:18.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:18.289 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:18.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:18.390 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:18.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:18.420 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:18.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:18.521 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:18.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:18.554 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:18.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:18.656 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:18.694 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:18.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:18.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:18.796 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:18.826 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:18.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:18.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:18.928 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:18.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:18.960 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:18.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:19.062 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:19.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:19.087 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:19.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:19.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:51:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:51:19.123 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:18 smithi113 ceph-mon[34341]: 5.3 scrub starts 2024-03-30T13:51:19.123 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:18 smithi113 ceph-mon[34341]: 5.3 scrub ok 2024-03-30T13:51:19.123 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:18 smithi113 ceph-mon[34341]: 6.15 scrub starts 2024-03-30T13:51:19.123 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:18 smithi113 ceph-mon[34341]: 6.15 scrub ok 2024-03-30T13:51:19.123 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:18 smithi113 ceph-mon[34341]: pgmap v343: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.37k op/s; 34863/278994 objects degraded (12.496%) 2024-03-30T13:51:19.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:18 smithi043 ceph-mon[29963]: 5.3 scrub starts 2024-03-30T13:51:19.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:18 smithi043 ceph-mon[29963]: 5.3 scrub ok 2024-03-30T13:51:19.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:18 smithi043 ceph-mon[29963]: 6.15 scrub starts 2024-03-30T13:51:19.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:18 smithi043 ceph-mon[29963]: 6.15 scrub ok 2024-03-30T13:51:19.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:18 smithi043 ceph-mon[29963]: pgmap v343: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.37k op/s; 34863/278994 objects degraded (12.496%) 2024-03-30T13:51:19.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:18 smithi043 ceph-mon[42099]: 5.3 scrub starts 2024-03-30T13:51:19.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:18 smithi043 ceph-mon[42099]: 5.3 scrub ok 2024-03-30T13:51:19.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:18 smithi043 ceph-mon[42099]: 6.15 scrub starts 2024-03-30T13:51:19.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:18 smithi043 ceph-mon[42099]: 6.15 scrub ok 2024-03-30T13:51:19.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:18 smithi043 ceph-mon[42099]: pgmap v343: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.37k op/s; 34863/278994 objects degraded (12.496%) 2024-03-30T13:51:19.188 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:19.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:19.223 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:19.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:19.324 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:19.356 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:19.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:19.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:19.458 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:19.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:19.493 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:19.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:19.594 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:19.626 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:19.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:19.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:19.728 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:19.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:19.754 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:19.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:19.855 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:19.887 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:19.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:19.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:19.989 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:20.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:20.026 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:20.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:20.127 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:20.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:19 smithi043 ceph-mon[29963]: 3.11 scrub starts 2024-03-30T13:51:20.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:19 smithi043 ceph-mon[29963]: 3.11 scrub ok 2024-03-30T13:51:20.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:19 smithi043 ceph-mon[29963]: 5.5 scrub starts 2024-03-30T13:51:20.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:19 smithi043 ceph-mon[29963]: 5.5 scrub ok 2024-03-30T13:51:20.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:19 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:51:20.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:19 smithi043 ceph-mon[29963]: 8.0 scrub starts 2024-03-30T13:51:20.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:19 smithi043 ceph-mon[29963]: 3.c scrub starts 2024-03-30T13:51:20.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:19 smithi043 ceph-mon[29963]: 3.c scrub ok 2024-03-30T13:51:20.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:19 smithi043 ceph-mon[42099]: 3.11 scrub starts 2024-03-30T13:51:20.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:19 smithi043 ceph-mon[42099]: 3.11 scrub ok 2024-03-30T13:51:20.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:19 smithi043 ceph-mon[42099]: 5.5 scrub starts 2024-03-30T13:51:20.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:19 smithi043 ceph-mon[42099]: 5.5 scrub ok 2024-03-30T13:51:20.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:19 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:51:20.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:19 smithi043 ceph-mon[42099]: 8.0 scrub starts 2024-03-30T13:51:20.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:19 smithi043 ceph-mon[42099]: 3.c scrub starts 2024-03-30T13:51:20.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:19 smithi043 ceph-mon[42099]: 3.c scrub ok 2024-03-30T13:51:20.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:20.165 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:20.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:20.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:19 smithi113 ceph-mon[34341]: 3.11 scrub starts 2024-03-30T13:51:20.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:19 smithi113 ceph-mon[34341]: 3.11 scrub ok 2024-03-30T13:51:20.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:19 smithi113 ceph-mon[34341]: 5.5 scrub starts 2024-03-30T13:51:20.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:19 smithi113 ceph-mon[34341]: 5.5 scrub ok 2024-03-30T13:51:20.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:19 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:51:20.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:19 smithi113 ceph-mon[34341]: 8.0 scrub starts 2024-03-30T13:51:20.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:19 smithi113 ceph-mon[34341]: 3.c scrub starts 2024-03-30T13:51:20.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:19 smithi113 ceph-mon[34341]: 3.c scrub ok 2024-03-30T13:51:20.266 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:20.308 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:20.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:20.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:20.410 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:20.453 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:20.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:20.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:20.555 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:20.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:20.594 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:20.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:20.695 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:20.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:20.729 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:20.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:20.830 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:20.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:20.871 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:20.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:20.972 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:21.004 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:21.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:21.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:21.106 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:21.143 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:21.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:21.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:21.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:20 smithi043 ceph-mon[29963]: 5.9 scrub starts 2024-03-30T13:51:21.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:20 smithi043 ceph-mon[29963]: 5.9 scrub ok 2024-03-30T13:51:21.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:20 smithi043 ceph-mon[29963]: pgmap v344: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 32640/261642 objects degraded (12.475%) 2024-03-30T13:51:21.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:20 smithi043 ceph-mon[29963]: 5.18 scrub starts 2024-03-30T13:51:21.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:20 smithi043 ceph-mon[29963]: 5.18 scrub ok 2024-03-30T13:51:21.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:20 smithi043 ceph-mon[29963]: 3.10 deep-scrub starts 2024-03-30T13:51:21.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:20 smithi043 ceph-mon[29963]: 3.10 deep-scrub ok 2024-03-30T13:51:21.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:20 smithi043 ceph-mon[29963]: 4.16 scrub starts 2024-03-30T13:51:21.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:20 smithi043 ceph-mon[29963]: 4.16 scrub ok 2024-03-30T13:51:21.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:20 smithi043 ceph-mon[29963]: 8.0 scrub ok 2024-03-30T13:51:21.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:20 smithi043 ceph-mon[42099]: 5.9 scrub starts 2024-03-30T13:51:21.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:20 smithi043 ceph-mon[42099]: 5.9 scrub ok 2024-03-30T13:51:21.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:20 smithi043 ceph-mon[42099]: pgmap v344: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 32640/261642 objects degraded (12.475%) 2024-03-30T13:51:21.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:20 smithi043 ceph-mon[42099]: 5.18 scrub starts 2024-03-30T13:51:21.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:20 smithi043 ceph-mon[42099]: 5.18 scrub ok 2024-03-30T13:51:21.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:20 smithi043 ceph-mon[42099]: 3.10 deep-scrub starts 2024-03-30T13:51:21.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:20 smithi043 ceph-mon[42099]: 3.10 deep-scrub ok 2024-03-30T13:51:21.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:20 smithi043 ceph-mon[42099]: 4.16 scrub starts 2024-03-30T13:51:21.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:20 smithi043 ceph-mon[42099]: 4.16 scrub ok 2024-03-30T13:51:21.150 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:20 smithi043 ceph-mon[42099]: 8.0 scrub ok 2024-03-30T13:51:21.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:20 smithi113 ceph-mon[34341]: 5.9 scrub starts 2024-03-30T13:51:21.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:20 smithi113 ceph-mon[34341]: 5.9 scrub ok 2024-03-30T13:51:21.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:20 smithi113 ceph-mon[34341]: pgmap v344: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 32640/261642 objects degraded (12.475%) 2024-03-30T13:51:21.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:20 smithi113 ceph-mon[34341]: 5.18 scrub starts 2024-03-30T13:51:21.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:20 smithi113 ceph-mon[34341]: 5.18 scrub ok 2024-03-30T13:51:21.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:20 smithi113 ceph-mon[34341]: 3.10 deep-scrub starts 2024-03-30T13:51:21.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:20 smithi113 ceph-mon[34341]: 3.10 deep-scrub ok 2024-03-30T13:51:21.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:20 smithi113 ceph-mon[34341]: 4.16 scrub starts 2024-03-30T13:51:21.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:20 smithi113 ceph-mon[34341]: 4.16 scrub ok 2024-03-30T13:51:21.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:20 smithi113 ceph-mon[34341]: 8.0 scrub ok 2024-03-30T13:51:21.245 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:21.277 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:21.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:21.343 INFO:tasks.thrashosds.thrasher:j is {'epoch': 186, 'fsid': '7ee932fa-ee9a-11ee-b647-cb9ed24678a4', 'created': '2024-03-30T13:37:07.346020+0000', 'modified': '2024-03-30T13:51:11.765740+0000', 'last_up_change': '2024-03-30T13:50:07.489747+0000', 'last_in_change': '2024-03-30T13:42:12.831084+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T13:40:27.658979+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.980000019073486, 'score_stable': 6.980000019073486, 'optimal_score': 0.4300000071525574, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-30T13:42:42.989975+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-30T13:42:43.910306+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.309999942779541, 'score_stable': 1.309999942779541, 'optimal_score': 1, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-30T13:42:45.633132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-30T13:42:47.570084+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.309999942779541, 'score_stable': 1.309999942779541, 'optimal_score': 1, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-30T13:42:49.637189+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-03-30T13:48:59.919716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '151', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '94ea083e-6a27-41db-83af-2f7d3cae1956', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 167, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6802', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6803', 'nonce': 1144592572}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6804', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6805', 'nonce': 1144592572}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6808', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6809', 'nonce': 1144592572}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6806', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6807', 'nonce': 1144592572}]}, 'public_addr': '172.21.15.43:6803/1144592572', 'cluster_addr': '172.21.15.43:6805/1144592572', 'heartbeat_back_addr': '172.21.15.43:6809/1144592572', 'heartbeat_front_addr': '172.21.15.43:6807/1144592572', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 167, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6810', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6811', 'nonce': 3901165928}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6812', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6813', 'nonce': 3901165928}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6816', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6817', 'nonce': 3901165928}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6814', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6815', 'nonce': 3901165928}]}, 'public_addr': '172.21.15.43:6811/3901165928', 'cluster_addr': '172.21.15.43:6813/3901165928', 'heartbeat_back_addr': '172.21.15.43:6817/3901165928', 'heartbeat_front_addr': '172.21.15.43:6815/3901165928', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6db6782e-5889-443e-bc6c-df47a1f93bc8', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 98, 'down_at': 167, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6818', 'nonce': 3415037523}, {'type': 'v1', 'addr': '172.21.15.43:6819', 'nonce': 3415037523}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6820', 'nonce': 3415037523}, {'type': 'v1', 'addr': '172.21.15.43:6821', 'nonce': 3415037523}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6824', 'nonce': 3415037523}, {'type': 'v1', 'addr': '172.21.15.43:6825', 'nonce': 3415037523}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6822', 'nonce': 3415037523}, {'type': 'v1', 'addr': '172.21.15.43:6823', 'nonce': 3415037523}]}, 'public_addr': '172.21.15.43:6819/3415037523', 'cluster_addr': '172.21.15.43:6821/3415037523', 'heartbeat_back_addr': '172.21.15.43:6825/3415037523', 'heartbeat_front_addr': '172.21.15.43:6823/3415037523', 'state': ['exists']}, {'osd': 3, 'uuid': '7048ff46-2cba-4d18-9835-95ecbddb2bb6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 167, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6826', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6827', 'nonce': 2933589376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6828', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6829', 'nonce': 2933589376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6832', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6833', 'nonce': 2933589376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6830', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6831', 'nonce': 2933589376}]}, 'public_addr': '172.21.15.43:6827/2933589376', 'cluster_addr': '172.21.15.43:6829/2933589376', 'heartbeat_back_addr': '172.21.15.43:6833/2933589376', 'heartbeat_front_addr': '172.21.15.43:6831/2933589376', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '42da30e6-9eec-4ed5-8c73-9ce78aa8ff63', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 167, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6800', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6801', 'nonce': 1945833081}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6802', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6803', 'nonce': 1945833081}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6806', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6807', 'nonce': 1945833081}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6804', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6805', 'nonce': 1945833081}]}, 'public_addr': '172.21.15.113:6801/1945833081', 'cluster_addr': '172.21.15.113:6803/1945833081', 'heartbeat_back_addr': '172.21.15.113:6807/1945833081', 'heartbeat_front_addr': '172.21.15.113:6805/1945833081', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '99a11b81-0df7-4b1d-946a-4297ba62e263', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 167, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6808', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6809', 'nonce': 3426360325}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6810', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6811', 'nonce': 3426360325}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6814', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6815', 'nonce': 3426360325}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6812', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6813', 'nonce': 3426360325}]}, 'public_addr': '172.21.15.113:6809/3426360325', 'cluster_addr': '172.21.15.113:6811/3426360325', 'heartbeat_back_addr': '172.21.15.113:6815/3426360325', 'heartbeat_front_addr': '172.21.15.113:6813/3426360325', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'cafd0e26-78e9-4164-a43a-96ca8382fefe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 147, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6816', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6817', 'nonce': 1527017698}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6818', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6819', 'nonce': 1527017698}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6822', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6823', 'nonce': 1527017698}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6820', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6821', 'nonce': 1527017698}]}, 'public_addr': '172.21.15.113:6817/1527017698', 'cluster_addr': '172.21.15.113:6819/1527017698', 'heartbeat_back_addr': '172.21.15.113:6823/1527017698', 'heartbeat_front_addr': '172.21.15.113:6821/1527017698', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e9d59662-9132-48f4-9e95-842a3cfbd79f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 167, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6824', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6825', 'nonce': 3182476015}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6826', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6827', 'nonce': 3182476015}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6830', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6831', 'nonce': 3182476015}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6828', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6829', 'nonce': 3182476015}]}, 'public_addr': '172.21.15.113:6825/3182476015', 'cluster_addr': '172.21.15.113:6827/3182476015', 'heartbeat_back_addr': '172.21.15.113:6831/3182476015', 'heartbeat_front_addr': '172.21.15.113:6829/3182476015', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:51:07.544068+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:51:02.271123+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:49:03.444490+0000', 'dead_epoch': 166}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:50:30.249679+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:50:19.616009+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:50:27.087978+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:50:42.636212+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:50:05.355621+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.b', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.43:0/538304234': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:6801/3411723337': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/3656823679': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/2618817647': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:6800/3411723337': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/1206218784': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/886631486': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:6800/1339921781': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:6801/1339921781': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:0/2794494528': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:0/564868778': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:6800/3033707884': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/3539398453': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/2414364740': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:6801/3033707884': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/3923597429': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/2524269526': '2024-03-31T13:43:27.253365+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-30T13:51:21.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T13:51:21.378 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:21.425 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:21.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:21.527 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:21.558 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:21.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:21.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:21.659 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:21.705 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:21.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:21.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:21.807 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:21.833 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:21.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:21.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:21.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:21 smithi043 ceph-mon[29963]: 5.10 scrub starts 2024-03-30T13:51:21.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:21 smithi043 ceph-mon[29963]: 5.10 scrub ok 2024-03-30T13:51:21.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:21 smithi043 ceph-mon[29963]: 4.1d scrub starts 2024-03-30T13:51:21.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:21 smithi043 ceph-mon[29963]: 4.1d scrub ok 2024-03-30T13:51:21.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:21 smithi043 ceph-mon[29963]: 5.f scrub starts 2024-03-30T13:51:21.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:21 smithi043 ceph-mon[29963]: 5.f scrub ok 2024-03-30T13:51:21.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:21 smithi043 ceph-mon[29963]: 3.a scrub starts 2024-03-30T13:51:21.900 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:21 smithi043 ceph-mon[29963]: 3.a scrub ok 2024-03-30T13:51:21.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:21 smithi043 ceph-mon[42099]: 5.10 scrub starts 2024-03-30T13:51:21.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:21 smithi043 ceph-mon[42099]: 5.10 scrub ok 2024-03-30T13:51:21.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:21 smithi043 ceph-mon[42099]: 4.1d scrub starts 2024-03-30T13:51:21.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:21 smithi043 ceph-mon[42099]: 4.1d scrub ok 2024-03-30T13:51:21.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:21 smithi043 ceph-mon[42099]: 5.f scrub starts 2024-03-30T13:51:21.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:21 smithi043 ceph-mon[42099]: 5.f scrub ok 2024-03-30T13:51:21.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:21 smithi043 ceph-mon[42099]: 3.a scrub starts 2024-03-30T13:51:21.900 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:21 smithi043 ceph-mon[42099]: 3.a scrub ok 2024-03-30T13:51:21.935 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:21.963 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:21.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:21.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:22.013 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:51:22.050 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:51:22.065 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:22.089 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:51:22.097 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:22.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:22.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:22.128 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:51:22.167 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:51:22.199 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:22.205 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:51:22.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:21 smithi113 ceph-mon[34341]: 5.10 scrub starts 2024-03-30T13:51:22.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:21 smithi113 ceph-mon[34341]: 5.10 scrub ok 2024-03-30T13:51:22.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:21 smithi113 ceph-mon[34341]: 4.1d scrub starts 2024-03-30T13:51:22.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:21 smithi113 ceph-mon[34341]: 4.1d scrub ok 2024-03-30T13:51:22.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:21 smithi113 ceph-mon[34341]: 5.f scrub starts 2024-03-30T13:51:22.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:21 smithi113 ceph-mon[34341]: 5.f scrub ok 2024-03-30T13:51:22.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:21 smithi113 ceph-mon[34341]: 3.a scrub starts 2024-03-30T13:51:22.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:21 smithi113 ceph-mon[34341]: 3.a scrub ok 2024-03-30T13:51:22.239 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:22.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:22.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:22.245 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:51:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:51:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:51:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:51:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:51:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:51:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:51:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:51:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:51:22.341 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:22.378 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:22.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:22.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:22.480 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:22.513 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:22.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:22.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:22.615 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:22.654 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:22.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:22.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:22.756 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:22.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:22.790 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:22.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:22.891 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:22.923 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:22.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:22.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:23.025 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:23.056 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:23.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:23.088 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:22 smithi043 ceph-mon[29963]: pgmap v345: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 32640/261642 objects degraded (12.475%) 2024-03-30T13:51:23.088 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:22 smithi043 ceph-mon[29963]: 4.11 scrub starts 2024-03-30T13:51:23.088 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:22 smithi043 ceph-mon[29963]: 4.11 scrub ok 2024-03-30T13:51:23.088 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:22 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/625592781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:51:23.088 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:22 smithi043 ceph-mon[29963]: 4.1e deep-scrub starts 2024-03-30T13:51:23.089 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:22 smithi043 ceph-mon[29963]: 4.1e deep-scrub ok 2024-03-30T13:51:23.089 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:22 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 32640/261642 objects degraded (12.475%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:23.089 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:22 smithi043 ceph-mon[42099]: pgmap v345: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 32640/261642 objects degraded (12.475%) 2024-03-30T13:51:23.089 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:22 smithi043 ceph-mon[42099]: 4.11 scrub starts 2024-03-30T13:51:23.089 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:22 smithi043 ceph-mon[42099]: 4.11 scrub ok 2024-03-30T13:51:23.089 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:22 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/625592781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:51:23.089 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:22 smithi043 ceph-mon[42099]: 4.1e deep-scrub starts 2024-03-30T13:51:23.089 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:22 smithi043 ceph-mon[42099]: 4.1e deep-scrub ok 2024-03-30T13:51:23.089 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:22 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 32640/261642 objects degraded (12.475%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:23.132 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:51:23.133 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:51:23.157 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:23.191 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:23.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:23.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:23.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:22 smithi113 ceph-mon[34341]: pgmap v345: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 32640/261642 objects degraded (12.475%) 2024-03-30T13:51:23.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:22 smithi113 ceph-mon[34341]: 4.11 scrub starts 2024-03-30T13:51:23.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:22 smithi113 ceph-mon[34341]: 4.11 scrub ok 2024-03-30T13:51:23.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:22 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/625592781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:51:23.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:22 smithi113 ceph-mon[34341]: 4.1e deep-scrub starts 2024-03-30T13:51:23.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:22 smithi113 ceph-mon[34341]: 4.1e deep-scrub ok 2024-03-30T13:51:23.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:22 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 32640/261642 objects degraded (12.475%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:23.292 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:23.325 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:23.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:23.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:23.427 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:23.464 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:23.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:23.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:23.566 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:23.601 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:23.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:23.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:23.663 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:51:23.703 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:23.738 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:51:23.755 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:23.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:23.857 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:23.891 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:23.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:23.966 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:23 smithi043 ceph-mon[29963]: 3.17 scrub starts 2024-03-30T13:51:23.966 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:23 smithi043 ceph-mon[29963]: 3.17 scrub ok 2024-03-30T13:51:23.966 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:23 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:51:23.966 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:23 smithi043 ceph-mon[42099]: 3.17 scrub starts 2024-03-30T13:51:23.966 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:23 smithi043 ceph-mon[42099]: 3.17 scrub ok 2024-03-30T13:51:23.966 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:23 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:51:23.993 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:24.025 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:24.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:24.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:24.126 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:24.162 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:24.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:24.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:24.220 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:23 smithi113 ceph-mon[34341]: 3.17 scrub starts 2024-03-30T13:51:24.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:23 smithi113 ceph-mon[34341]: 3.17 scrub ok 2024-03-30T13:51:24.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:23 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:51:24.263 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:24.299 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:24.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:24.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:24.401 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:24.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:24.430 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:24.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:24.531 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:24.563 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:24.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:24.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:24.665 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:24.692 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:24.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:24.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:24.793 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:24.834 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:24.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:24.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:24.935 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:24.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:24.972 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:24.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:25.031 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:51:25.031 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-03-30T13:51:25.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.031 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(osd.3.0:81552 8.b e186/147 8:d557640e:::benchmark_data_smithi043_119832_object66195:head v 186'12976, mlcod=186'12957)", 2024-03-30T13:51:25.031 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:51:24.547376+0000", 2024-03-30T13:51:25.031 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.019105217000000001, 2024-03-30T13:51:25.031 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.019136944999999999, 2024-03-30T13:51:25.031 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:51:25.031 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:51:25.031 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T13:51:25.031 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:51:25.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.031 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:51:25.032 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.547376+0000", 2024-03-30T13:51:25.032 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:51:25.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.032 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:51:25.032 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.547376+0000", 2024-03-30T13:51:25.032 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:51:25.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.032 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:51:25.032 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.547377+0000", 2024-03-30T13:51:25.032 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9899999999999999e-07 2024-03-30T13:51:25.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.032 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:51:25.032 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.547383+0000", 2024-03-30T13:51:25.033 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6780000000000003e-06 2024-03-30T13:51:25.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.033 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:51:25.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.547398+0000", 2024-03-30T13:51:25.033 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4513e-05 2024-03-30T13:51:25.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.033 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:51:25.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.547405+0000", 2024-03-30T13:51:25.033 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1160000000000001e-06 2024-03-30T13:51:25.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.033 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:51:25.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.548445+0000", 2024-03-30T13:51:25.034 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0010400559999999999 2024-03-30T13:51:25.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.034 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:51:25.034 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.548505+0000", 2024-03-30T13:51:25.034 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9930999999999999e-05 2024-03-30T13:51:25.034 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.034 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:25.034 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.034 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(osd.3.0:81553 8.b e186/147 8:d5579ad1:::benchmark_data_smithi043_119832_object61688:head v 186'12977, mlcod=186'12957)", 2024-03-30T13:51:25.034 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:51:24.548121+0000", 2024-03-30T13:51:25.034 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.018359929000000001, 2024-03-30T13:51:25.034 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.018465065999999999, 2024-03-30T13:51:25.035 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:51:25.035 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:51:25.035 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T13:51:25.035 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:51:25.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.035 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:51:25.035 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.548121+0000", 2024-03-30T13:51:25.035 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:51:25.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.035 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:51:25.035 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.548121+0000", 2024-03-30T13:51:25.035 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:51:25.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.036 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:51:25.036 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.548122+0000", 2024-03-30T13:51:25.036 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5099999999999999e-07 2024-03-30T13:51:25.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.036 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:51:25.036 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.548128+0000", 2024-03-30T13:51:25.036 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0809999999999999e-06 2024-03-30T13:51:25.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.036 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:51:25.036 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.548139+0000", 2024-03-30T13:51:25.036 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1069e-05 2024-03-30T13:51:25.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.037 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:51:25.037 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.548146+0000", 2024-03-30T13:51:25.037 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9670000000000004e-06 2024-03-30T13:51:25.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.037 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:51:25.037 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.551644+0000", 2024-03-30T13:51:25.037 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0034979870000000001 2024-03-30T13:51:25.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.037 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:51:25.037 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.551664+0000", 2024-03-30T13:51:25.037 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9826e-05 2024-03-30T13:51:25.038 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.038 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:25.038 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.038 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(osd.3.0:81554 8.b e186/147 8:d557cdf6:::benchmark_data_smithi043_119832_object97227:head v 186'12978, mlcod=186'12957)", 2024-03-30T13:51:25.038 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:51:24.550642+0000", 2024-03-30T13:51:25.038 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.015838782999999999, 2024-03-30T13:51:25.038 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.016014006000000001, 2024-03-30T13:51:25.038 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:51:25.038 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:51:25.038 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T13:51:25.038 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:51:25.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.038 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:51:25.039 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.550642+0000", 2024-03-30T13:51:25.039 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:51:25.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.039 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:51:25.039 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.550642+0000", 2024-03-30T13:51:25.039 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:51:25.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.039 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:51:25.039 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.550643+0000", 2024-03-30T13:51:25.039 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8299999999999996e-07 2024-03-30T13:51:25.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.039 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:51:25.040 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.550649+0000", 2024-03-30T13:51:25.040 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4829999999999999e-06 2024-03-30T13:51:25.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.040 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:51:25.040 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.550662+0000", 2024-03-30T13:51:25.040 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3760999999999999e-05 2024-03-30T13:51:25.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.040 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:51:25.040 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.550670+0000", 2024-03-30T13:51:25.040 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5320000000000001e-06 2024-03-30T13:51:25.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.040 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:51:25.041 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.554067+0000", 2024-03-30T13:51:25.041 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0033975120000000001 2024-03-30T13:51:25.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.041 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:51:25.041 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.554125+0000", 2024-03-30T13:51:25.041 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7269999999999999e-05 2024-03-30T13:51:25.041 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.041 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:25.041 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.041 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(osd.3.0:81555 8.b e186/147 8:d557d7b3:::benchmark_data_smithi043_119832_object96994:head v 186'12979, mlcod=186'12957)", 2024-03-30T13:51:25.041 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:51:24.553322+0000", 2024-03-30T13:51:25.042 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.013158670000000001, 2024-03-30T13:51:25.042 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.013399912, 2024-03-30T13:51:25.042 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:51:25.042 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:51:25.042 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T13:51:25.042 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:51:25.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.042 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:51:25.042 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.553322+0000", 2024-03-30T13:51:25.042 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:51:25.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.042 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:51:25.042 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.553322+0000", 2024-03-30T13:51:25.042 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:51:25.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.043 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:51:25.043 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.553323+0000", 2024-03-30T13:51:25.043 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8500000000000001e-07 2024-03-30T13:51:25.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.043 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:51:25.043 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.553329+0000", 2024-03-30T13:51:25.043 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8950000000000003e-06 2024-03-30T13:51:25.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.043 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:51:25.043 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.553339+0000", 2024-03-30T13:51:25.043 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.6399999999999992e-06 2024-03-30T13:51:25.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.044 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.044 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:51:25.044 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.553346+0000", 2024-03-30T13:51:25.044 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2060000000000001e-06 2024-03-30T13:51:25.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.044 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.044 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:51:25.044 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.556550+0000", 2024-03-30T13:51:25.044 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0032036650000000001 2024-03-30T13:51:25.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.044 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.044 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:51:25.044 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.556570+0000", 2024-03-30T13:51:25.044 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0085000000000002e-05 2024-03-30T13:51:25.045 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.045 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:25.045 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.045 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.045 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:179988 8.3 8:c2c88954:::benchmark_data_smithi043_119832_object49634:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-03-30T13:51:25.045 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:51:24.556622+0000", 2024-03-30T13:51:25.045 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0098586909999999993, 2024-03-30T13:51:25.045 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010167272, 2024-03-30T13:51:25.045 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:51:25.045 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:51:25.045 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-03-30T13:51:25.045 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:51:25.045 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:51:25.045 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:51:25.045 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 179988 2024-03-30T13:51:25.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.046 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:51:25.046 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.046 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:51:25.046 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.556622+0000", 2024-03-30T13:51:25.046 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:51:25.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.046 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.046 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:51:25.046 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.556622+0000", 2024-03-30T13:51:25.046 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:51:25.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.046 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.046 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:51:25.046 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.556624+0000", 2024-03-30T13:51:25.047 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.389e-06 2024-03-30T13:51:25.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.047 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.047 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:51:25.047 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.556631+0000", 2024-03-30T13:51:25.047 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8889999999999996e-06 2024-03-30T13:51:25.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.047 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.047 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:51:25.047 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.556643+0000", 2024-03-30T13:51:25.047 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2169e-05 2024-03-30T13:51:25.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.047 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.047 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:51:25.047 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.556652+0000", 2024-03-30T13:51:25.047 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6570000000000003e-06 2024-03-30T13:51:25.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.048 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:51:25.048 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.559081+0000", 2024-03-30T13:51:25.048 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00242924 2024-03-30T13:51:25.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.048 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:51:25.048 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.559538+0000", 2024-03-30T13:51:25.048 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00045767900000000001 2024-03-30T13:51:25.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.048 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 5,6", 2024-03-30T13:51:25.048 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.559992+0000", 2024-03-30T13:51:25.048 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00045304900000000002 2024-03-30T13:51:25.048 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.049 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:25.049 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.049 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.049 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(osd.3.0:81557 8.b e186/147 8:d55959e6:::benchmark_data_smithi043_119832_object96894:head v 186'12980, mlcod=186'12957)", 2024-03-30T13:51:25.049 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:51:24.557940+0000", 2024-03-30T13:51:25.049 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0085413099999999999, 2024-03-30T13:51:25.049 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0089379809999999994, 2024-03-30T13:51:25.049 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:51:25.049 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:51:25.049 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T13:51:25.049 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:51:25.049 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.050 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:51:25.050 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.557940+0000", 2024-03-30T13:51:25.050 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:51:25.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.050 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:51:25.050 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.557940+0000", 2024-03-30T13:51:25.050 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:51:25.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.050 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:51:25.050 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.557940+0000", 2024-03-30T13:51:25.051 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4000000000000001e-07 2024-03-30T13:51:25.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.051 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:51:25.051 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.557948+0000", 2024-03-30T13:51:25.051 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.661e-06 2024-03-30T13:51:25.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.051 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:51:25.051 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.557960+0000", 2024-03-30T13:51:25.051 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1678000000000001e-05 2024-03-30T13:51:25.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.052 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:51:25.052 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.557967+0000", 2024-03-30T13:51:25.052 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.092e-06 2024-03-30T13:51:25.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.052 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:51:25.052 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.559101+0000", 2024-03-30T13:51:25.052 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0011336650000000001 2024-03-30T13:51:25.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.052 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:51:25.052 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.559124+0000", 2024-03-30T13:51:25.052 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3842e-05 2024-03-30T13:51:25.052 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.053 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:25.053 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.053 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(osd.3.0:81558 8.b e186/147 8:d55bdee3:::benchmark_data_smithi043_119832_object122415:head v 186'12981, mlcod=186'12957)", 2024-03-30T13:51:25.053 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:51:24.562133+0000", 2024-03-30T13:51:25.053 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0043482950000000003, 2024-03-30T13:51:25.053 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0048277789999999999, 2024-03-30T13:51:25.053 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:51:25.053 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:51:25.053 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T13:51:25.053 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:51:25.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.053 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:51:25.053 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.562133+0000", 2024-03-30T13:51:25.054 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:51:25.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.054 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:51:25.054 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.562133+0000", 2024-03-30T13:51:25.054 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:51:25.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.054 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:51:25.054 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.562134+0000", 2024-03-30T13:51:25.054 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5799999999999998e-07 2024-03-30T13:51:25.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.054 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:51:25.055 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.562140+0000", 2024-03-30T13:51:25.055 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7139999999999996e-06 2024-03-30T13:51:25.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.055 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.055 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:51:25.055 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.562154+0000", 2024-03-30T13:51:25.055 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3565e-05 2024-03-30T13:51:25.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.055 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.055 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:51:25.055 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.562161+0000", 2024-03-30T13:51:25.055 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3359999999999997e-06 2024-03-30T13:51:25.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.055 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.055 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:51:25.056 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.562642+0000", 2024-03-30T13:51:25.056 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000480643 2024-03-30T13:51:25.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.056 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:51:25.056 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.562675+0000", 2024-03-30T13:51:25.056 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3482000000000002e-05 2024-03-30T13:51:25.056 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.056 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:25.056 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.057 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(osd.3.0:81559 8.b e186/147 8:d55e51a8:::benchmark_data_smithi043_119832_object88216:head v 186'12982, mlcod=186'12957)", 2024-03-30T13:51:25.057 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:51:24.564586+0000", 2024-03-30T13:51:25.057 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.001895571, 2024-03-30T13:51:25.057 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002438689, 2024-03-30T13:51:25.057 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:51:25.057 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:51:25.057 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T13:51:25.057 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:51:25.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.057 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:51:25.057 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.564586+0000", 2024-03-30T13:51:25.057 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:51:25.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.057 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:51:25.057 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.564586+0000", 2024-03-30T13:51:25.058 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:51:25.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.058 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:51:25.058 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.564586+0000", 2024-03-30T13:51:25.058 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9299999999999997e-07 2024-03-30T13:51:25.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.058 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:51:25.058 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.564595+0000", 2024-03-30T13:51:25.058 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3229999999999998e-06 2024-03-30T13:51:25.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.058 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:51:25.059 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.564609+0000", 2024-03-30T13:51:25.059 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4205e-05 2024-03-30T13:51:25.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.059 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:51:25.059 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.564617+0000", 2024-03-30T13:51:25.059 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4190000000000002e-06 2024-03-30T13:51:25.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.059 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:51:25.059 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.565448+0000", 2024-03-30T13:51:25.059 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00083042700000000001 2024-03-30T13:51:25.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.059 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:51:25.060 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.565472+0000", 2024-03-30T13:51:25.060 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3985000000000002e-05 2024-03-30T13:51:25.060 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.060 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:25.060 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.060 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop_reply(client.24977.0:179988 8.3 e186/147)", 2024-03-30T13:51:25.060 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:51:24.566596+0000", 2024-03-30T13:51:25.060 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 4294967295.9998856, 2024-03-30T13:51:25.060 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00048996700000000001, 2024-03-30T13:51:25.060 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:51:25.061 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:51:25.061 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "queued for pg", 2024-03-30T13:51:25.061 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:51:25.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.061 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:51:25.061 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.566596+0000", 2024-03-30T13:51:25.061 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:51:25.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.061 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:51:25.061 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.566596+0000", 2024-03-30T13:51:25.061 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:51:25.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.062 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:51:25.062 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.566597+0000", 2024-03-30T13:51:25.062 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0589999999999999e-06 2024-03-30T13:51:25.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.062 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:51:25.062 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.566603+0000", 2024-03-30T13:51:25.062 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6250000000000001e-06 2024-03-30T13:51:25.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.062 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:51:25.062 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.566620+0000", 2024-03-30T13:51:25.062 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6903999999999999e-05 2024-03-30T13:51:25.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.063 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:51:25.063 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.566629+0000", 2024-03-30T13:51:25.063 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4630000000000001e-06 2024-03-30T13:51:25.063 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.063 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:25.063 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.063 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop_reply(client.24977.0:179988 8.3 e186/147)", 2024-03-30T13:51:25.063 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:51:24.566922+0000", 2024-03-30T13:51:25.063 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 4294967295.9995594, 2024-03-30T13:51:25.063 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00021092800000000001, 2024-03-30T13:51:25.064 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:51:25.064 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:51:25.064 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "queued for pg", 2024-03-30T13:51:25.064 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:51:25.064 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.064 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:51:25.064 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.566922+0000", 2024-03-30T13:51:25.064 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:51:25.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.064 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.064 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:51:25.064 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.566922+0000", 2024-03-30T13:51:25.064 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:51:25.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.065 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.065 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:51:25.065 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.566922+0000", 2024-03-30T13:51:25.065 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6199999999999998e-07 2024-03-30T13:51:25.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.065 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.065 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:51:25.065 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.566926+0000", 2024-03-30T13:51:25.065 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8430000000000003e-06 2024-03-30T13:51:25.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.065 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.065 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:51:25.065 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.566936+0000", 2024-03-30T13:51:25.065 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.8020000000000003e-06 2024-03-30T13:51:25.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:51:25.066 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:51:25.066 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:51:25.066 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:24.566941+0000", 2024-03-30T13:51:25.066 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9980000000000003e-06 2024-03-30T13:51:25.066 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.066 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:51:25.066 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.066 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:51:25.066 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:51:25.066 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 10 2024-03-30T13:51:25.066 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:51:25.074 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:25.109 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:25.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:25.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:25.133 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:51:25.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:24 smithi043 ceph-mon[29963]: pgmap v346: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 876 B/s rd, 0 B/s wr, 1.05k op/s; 32640/261642 objects degraded (12.475%) 2024-03-30T13:51:25.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:24 smithi043 ceph-mon[29963]: 4.f scrub starts 2024-03-30T13:51:25.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:24 smithi043 ceph-mon[29963]: 4.f scrub ok 2024-03-30T13:51:25.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:24 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:51:25.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:24 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:51:25.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:24 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:51:25.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:24 smithi043 ceph-mon[42099]: pgmap v346: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 876 B/s rd, 0 B/s wr, 1.05k op/s; 32640/261642 objects degraded (12.475%) 2024-03-30T13:51:25.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:24 smithi043 ceph-mon[42099]: 4.f scrub starts 2024-03-30T13:51:25.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:24 smithi043 ceph-mon[42099]: 4.f scrub ok 2024-03-30T13:51:25.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:24 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:51:25.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:24 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:51:25.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:24 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:51:25.210 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:25.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:24 smithi113 ceph-mon[34341]: pgmap v346: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 876 B/s rd, 0 B/s wr, 1.05k op/s; 32640/261642 objects degraded (12.475%) 2024-03-30T13:51:25.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:24 smithi113 ceph-mon[34341]: 4.f scrub starts 2024-03-30T13:51:25.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:24 smithi113 ceph-mon[34341]: 4.f scrub ok 2024-03-30T13:51:25.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:24 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:51:25.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:24 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:51:25.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:24 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:51:25.240 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:25.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:25.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:25.342 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:25.371 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:25.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:25.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:25.473 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:25.501 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:25.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:25.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:25.603 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:25.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:25.638 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:25.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:25.738 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:25.771 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:25.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:25.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:25.839 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-03-30T13:51:25.873 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:25.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:25.902 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:25.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:26.003 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:26.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:26.062 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:26.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:26.099 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:25 smithi043 ceph-mon[29963]: 3.7 scrub starts 2024-03-30T13:51:26.099 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:25 smithi043 ceph-mon[29963]: 3.7 scrub ok 2024-03-30T13:51:26.099 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:25 smithi043 ceph-mon[29963]: 6.16 scrub starts 2024-03-30T13:51:26.099 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:25 smithi043 ceph-mon[29963]: 6.16 scrub ok 2024-03-30T13:51:26.099 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:25 smithi043 ceph-mon[42099]: 3.7 scrub starts 2024-03-30T13:51:26.100 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:25 smithi043 ceph-mon[42099]: 3.7 scrub ok 2024-03-30T13:51:26.100 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:25 smithi043 ceph-mon[42099]: 6.16 scrub starts 2024-03-30T13:51:26.100 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:25 smithi043 ceph-mon[42099]: 6.16 scrub ok 2024-03-30T13:51:26.163 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:26.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:26.204 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:26.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:26.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:25 smithi113 ceph-mon[34341]: 3.7 scrub starts 2024-03-30T13:51:26.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:25 smithi113 ceph-mon[34341]: 3.7 scrub ok 2024-03-30T13:51:26.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:25 smithi113 ceph-mon[34341]: 6.16 scrub starts 2024-03-30T13:51:26.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:25 smithi113 ceph-mon[34341]: 6.16 scrub ok 2024-03-30T13:51:26.305 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:26.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:26.345 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:26.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:26.397 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:51:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:51:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:51:26.446 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:26.480 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:26.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:26.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:26.582 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:26.618 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:26.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:26.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:26.720 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:26.753 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:26.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:26.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:26.855 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:26.887 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:26.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:26.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:26.989 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:27.029 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:27.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:27.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:27.131 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:27.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:26 smithi043 ceph-mon[29963]: pgmap v347: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s; 30533/245028 objects degraded (12.461%) 2024-03-30T13:51:27.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:26 smithi043 ceph-mon[29963]: 5.12 scrub starts 2024-03-30T13:51:27.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:26 smithi043 ceph-mon[29963]: 5.12 scrub ok 2024-03-30T13:51:27.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:26 smithi043 ceph-mon[42099]: pgmap v347: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s; 30533/245028 objects degraded (12.461%) 2024-03-30T13:51:27.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:26 smithi043 ceph-mon[42099]: 5.12 scrub starts 2024-03-30T13:51:27.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:26 smithi043 ceph-mon[42099]: 5.12 scrub ok 2024-03-30T13:51:27.159 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:27.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:27.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:27.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:26 smithi113 ceph-mon[34341]: pgmap v347: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s; 30533/245028 objects degraded (12.461%) 2024-03-30T13:51:27.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:26 smithi113 ceph-mon[34341]: 5.12 scrub starts 2024-03-30T13:51:27.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:26 smithi113 ceph-mon[34341]: 5.12 scrub ok 2024-03-30T13:51:27.261 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:27.299 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:27.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:27.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:27.401 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:27.433 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:27.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:27.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:27.535 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:27.570 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:27.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:27.672 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:27.706 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:27.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:27.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:27.808 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:27.838 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:27.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:27.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:27.873 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:51:27.915 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-03-30T13:51:27.939 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:27.983 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:27.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:27.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:28.031 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:27 smithi043 ceph-mon[29963]: 4.8 scrub starts 2024-03-30T13:51:28.031 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:27 smithi043 ceph-mon[29963]: 4.8 scrub ok 2024-03-30T13:51:28.031 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:27 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2779817602' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:51:28.031 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:27 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 30533/245028 objects degraded (12.461%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:28.031 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:27 smithi043 ceph-mon[42099]: 4.8 scrub starts 2024-03-30T13:51:28.032 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:27 smithi043 ceph-mon[42099]: 4.8 scrub ok 2024-03-30T13:51:28.032 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:27 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2779817602' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:51:28.032 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:27 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 30533/245028 objects degraded (12.461%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:28.085 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:28.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:28.118 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:28.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:28.219 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:28.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:27 smithi113 ceph-mon[34341]: 4.8 scrub starts 2024-03-30T13:51:28.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:27 smithi113 ceph-mon[34341]: 4.8 scrub ok 2024-03-30T13:51:28.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:27 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2779817602' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:51:28.222 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:27 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 30533/245028 objects degraded (12.461%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:28.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:28.252 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:28.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:28.353 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:28.389 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:28.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:28.490 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:28.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:28.527 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:28.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:28.628 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:28.674 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:28.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:28.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:28.776 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:28.808 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:28.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:28.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:28.821 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:51:28.910 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:28.944 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:28.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:28.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:28.990 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:28 smithi043 ceph-mon[29963]: pgmap v348: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 943 op/s; 30533/245028 objects degraded (12.461%) 2024-03-30T13:51:28.990 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:28 smithi043 ceph-mon[29963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:51:28.990 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:28 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2779817602' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:51:28.990 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:28 smithi043 ceph-mon[29963]: osdmap e187: 8 total, 7 up, 8 in 2024-03-30T13:51:28.990 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:51:28.990 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:51:28.991 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:28 smithi043 ceph-mon[42099]: pgmap v348: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 943 op/s; 30533/245028 objects degraded (12.461%) 2024-03-30T13:51:28.991 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:28 smithi043 ceph-mon[42099]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:51:28.991 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:28 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2779817602' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:51:28.991 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:28 smithi043 ceph-mon[42099]: osdmap e187: 8 total, 7 up, 8 in 2024-03-30T13:51:28.991 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:51:28.991 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:51:29.046 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:29.082 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:29.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:29.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:29.183 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:29.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:29.218 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:29.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:29.220 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:51:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:51:29.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:28 smithi113 ceph-mon[34341]: pgmap v348: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 943 op/s; 30533/245028 objects degraded (12.461%) 2024-03-30T13:51:29.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:28 smithi113 ceph-mon[34341]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:51:29.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:28 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2779817602' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:51:29.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:28 smithi113 ceph-mon[34341]: osdmap e187: 8 total, 7 up, 8 in 2024-03-30T13:51:29.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:51:29.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:51:29.303 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:51:29.316 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-30T13:51:29.319 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:29.348 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:29.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:29.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:29.450 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:29.488 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:29.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:29.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:29.590 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:29.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:29.629 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:29.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:29.730 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:29.769 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:29.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:29.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:29.871 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:29.902 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:29.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:29.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:30.004 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:30.040 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:30.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:30.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:30.141 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:30.163 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:29 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:51:30.164 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:29 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:51:30.178 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:30.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:30.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:30.220 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:29 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:51:30.280 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:30.312 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:30.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:30.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:30.413 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:30.447 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:30.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:30.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:30.549 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:30.583 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:30.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:30.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:30.685 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:30.717 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:30.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:30.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:30.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:30.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:30.856 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:30.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:30.956 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:30.995 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:30.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:30.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:31.096 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:31.134 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:31.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:31.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:31.220 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:30 smithi113 ceph-mon[34341]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:51:31.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:30 smithi113 ceph-mon[34341]: pgmap v350: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s; 28439/228216 objects degraded (12.461%) 2024-03-30T13:51:31.236 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:31.273 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:31.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:31.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:31.367 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:30 smithi043 ceph-mon[29963]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:51:31.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:30 smithi043 ceph-mon[29963]: pgmap v350: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s; 28439/228216 objects degraded (12.461%) 2024-03-30T13:51:31.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:30 smithi043 ceph-mon[42099]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:51:31.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:30 smithi043 ceph-mon[42099]: pgmap v350: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s; 28439/228216 objects degraded (12.461%) 2024-03-30T13:51:31.375 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:31.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:31.423 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:31.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:31.524 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:31.554 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:31.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:31.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:31.656 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:31.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:31.695 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:31.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:31.796 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:31.831 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:31.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:31.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:31.869 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-03-30T13:51:27.920932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5340273014,"num_objects":81676,"num_object_clones":0,"num_object_copies":245028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30533,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81676,"num_whiteouts":0,"num_read":1445,"num_read_kb":1291,"num_write":179841,"num_write_kb":8344926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1836860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86475,"ondisk_log_size":86475,"up":400,"acting":400,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17867452,"kb_used_data":15802044,"kb_used_omap":12,"kb_used_meta":2065203,"kb_avail":732093764,"statfs":{"total":767960285184,"available":749664014336,"internally_reserved":0,"allocated":16181293056,"data_stored":16177393176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":2114768432},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,1,4,1,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-741998592,"num_objects":-11322,"num_object_clones":0,"num_object_copies":-33966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11322,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11322,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.473770"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":276,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.773347+0000","last_change":"2024-03-30T13:51:05.234138+0000","last_active":"2024-03-30T13:51:11.773347+0000","last_peered":"2024-03-30T13:51:11.773347+0000","last_clean":"2024-03-30T13:51:11.773347+0000","last_became_active":"2024-03-30T13:42:51.329797+0000","last_became_peered":"2024-03-30T13:42:51.329797+0000","last_unstale":"2024-03-30T13:51:11.773347+0000","last_undegraded":"2024-03-30T13:51:11.773347+0000","last_fullsized":"2024-03-30T13:51:11.773347+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:05.234023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:05.234023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:22.013024+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":287,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.799017+0000","last_change":"2024-03-30T13:50:56.296909+0000","last_active":"2024-03-30T13:51:11.799017+0000","last_peered":"2024-03-30T13:51:11.799017+0000","last_clean":"2024-03-30T13:51:11.799017+0000","last_became_active":"2024-03-30T13:42:49.323268+0000","last_became_peered":"2024-03-30T13:42:49.323268+0000","last_unstale":"2024-03-30T13:51:11.799017+0000","last_undegraded":"2024-03-30T13:51:11.799017+0000","last_fullsized":"2024-03-30T13:51:11.799017+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:56.296785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:50:56.296785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:13.943731+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"65'1","reported_seq":287,"reported_epoch":186,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:51:11.787744+0000","last_change":"2024-03-30T13:50:08.513977+0000","last_active":"2024-03-30T13:51:11.787744+0000","last_peered":"2024-03-30T13:51:11.787744+0000","last_clean":"2024-03-30T13:50:05.499990+0000","last_became_active":"2024-03-30T13:50:08.513977+0000","last_became_peered":"2024-03-30T13:50:08.513977+0000","last_unstale":"2024-03-30T13:51:11.787744+0000","last_undegraded":"2024-03-30T13:50:08.505072+0000","last_fullsized":"2024-03-30T13:50:08.504945+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:48:19.930059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:48:19.930059+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":300,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.770989+0000","last_change":"2024-03-30T13:51:06.228575+0000","last_active":"2024-03-30T13:51:11.770989+0000","last_peered":"2024-03-30T13:51:11.770989+0000","last_clean":"2024-03-30T13:51:11.770989+0000","last_became_active":"2024-03-30T13:42:47.319638+0000","last_became_peered":"2024-03-30T13:42:47.319638+0000","last_unstale":"2024-03-30T13:51:11.770989+0000","last_undegraded":"2024-03-30T13:51:11.770989+0000","last_fullsized":"2024-03-30T13:51:11.770989+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:51:06.228451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:06.228451+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:13.739442+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":267,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.784107+0000","last_change":"2024-03-30T13:51:03.698577+0000","last_active":"2024-03-30T13:51:11.784107+0000","last_peered":"2024-03-30T13:51:11.784107+0000","last_clean":"2024-03-30T13:51:11.784107+0000","last_became_active":"2024-03-30T13:42:51.374403+0000","last_became_peered":"2024-03-30T13:42:51.374403+0000","last_unstale":"2024-03-30T13:51:11.784107+0000","last_undegraded":"2024-03-30T13:51:11.784107+0000","last_fullsized":"2024-03-30T13:51:11.784107+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:03.698501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:03.698501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:18.724603+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":280,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:20.149758+0000","last_change":"2024-03-30T13:51:20.149796+0000","last_active":"2024-03-30T13:51:20.149758+0000","last_peered":"2024-03-30T13:51:20.149758+0000","last_clean":"2024-03-30T13:51:20.149758+0000","last_became_active":"2024-03-30T13:42:49.327124+0000","last_became_peered":"2024-03-30T13:42:49.327124+0000","last_unstale":"2024-03-30T13:51:20.149758+0000","last_undegraded":"2024-03-30T13:51:20.149758+0000","last_fullsized":"2024-03-30T13:51:20.149758+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:20.149684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:20.149684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:20.873892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":283,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.782493+0000","last_change":"2024-03-30T13:50:08.512316+0000","last_active":"2024-03-30T13:51:11.782493+0000","last_peered":"2024-03-30T13:51:11.782493+0000","last_clean":"2024-03-30T13:50:05.504688+0000","last_became_active":"2024-03-30T13:50:08.512316+0000","last_became_peered":"2024-03-30T13:50:08.512316+0000","last_unstale":"2024-03-30T13:51:11.782493+0000","last_undegraded":"2024-03-30T13:51:11.782493+0000","last_fullsized":"2024-03-30T13:50:08.501439+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:20.499768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:20.499768+0000","last_clean_scrub_stamp":"2024-03-30T13:49:20.499768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"95'22","reported_seq":302,"reported_epoch":186,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:51:11.782931+0000","last_change":"2024-03-30T13:50:08.511099+0000","last_active":"2024-03-30T13:51:11.782931+0000","last_peered":"2024-03-30T13:51:11.782931+0000","last_clean":"2024-03-30T13:50:05.499201+0000","last_became_active":"2024-03-30T13:50:08.511099+0000","last_became_peered":"2024-03-30T13:50:08.511099+0000","last_unstale":"2024-03-30T13:51:11.782931+0000","last_undegraded":"2024-03-30T13:50:08.502733+0000","last_fullsized":"2024-03-30T13:50:08.502606+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'22","last_scrub_stamp":"2024-03-30T13:46:49.339954+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:29.151791+0000","last_clean_scrub_stamp":"2024-03-30T13:46:49.339954+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:13.663643+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":273,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.781808+0000","last_change":"2024-03-30T13:50:57.339042+0000","last_active":"2024-03-30T13:51:11.781808+0000","last_peered":"2024-03-30T13:51:11.781808+0000","last_clean":"2024-03-30T13:51:11.781808+0000","last_became_active":"2024-03-30T13:42:51.331196+0000","last_became_peered":"2024-03-30T13:42:51.331196+0000","last_unstale":"2024-03-30T13:51:11.781808+0000","last_undegraded":"2024-03-30T13:51:11.781808+0000","last_fullsized":"2024-03-30T13:51:11.781808+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:57.338783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:30.766332+0000","last_clean_scrub_stamp":"2024-03-30T13:50:57.338783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:21.578808+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":286,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.776155+0000","last_change":"2024-03-30T13:51:03.925694+0000","last_active":"2024-03-30T13:51:11.776155+0000","last_peered":"2024-03-30T13:51:11.776155+0000","last_clean":"2024-03-30T13:51:11.776155+0000","last_became_active":"2024-03-30T13:42:49.321811+0000","last_became_peered":"2024-03-30T13:42:49.321811+0000","last_unstale":"2024-03-30T13:51:11.776155+0000","last_undegraded":"2024-03-30T13:51:11.776155+0000","last_fullsized":"2024-03-30T13:51:11.776155+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:03.925590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:03.925590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:05.213109+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":286,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.776525+0000","last_change":"2024-03-30T13:50:12.645636+0000","last_active":"2024-03-30T13:51:11.776525+0000","last_peered":"2024-03-30T13:51:11.776525+0000","last_clean":"2024-03-30T13:51:11.776525+0000","last_became_active":"2024-03-30T13:42:45.319827+0000","last_became_peered":"2024-03-30T13:42:45.319827+0000","last_unstale":"2024-03-30T13:51:11.776525+0000","last_undegraded":"2024-03-30T13:51:11.776525+0000","last_fullsized":"2024-03-30T13:51:11.776525+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:12.645521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:37.513596+0000","last_clean_scrub_stamp":"2024-03-30T13:50:12.645521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:34.999707+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":308,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:16.153582+0000","last_change":"2024-03-30T13:51:16.153633+0000","last_active":"2024-03-30T13:51:16.153582+0000","last_peered":"2024-03-30T13:51:16.153582+0000","last_clean":"2024-03-30T13:51:16.153582+0000","last_became_active":"2024-03-30T13:42:47.310289+0000","last_became_peered":"2024-03-30T13:42:47.310289+0000","last_unstale":"2024-03-30T13:51:16.153582+0000","last_undegraded":"2024-03-30T13:51:16.153582+0000","last_fullsized":"2024-03-30T13:51:16.153582+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:51:16.153478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:16.153478+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:26.806078+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":282,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.775956+0000","last_change":"2024-03-30T13:51:10.988992+0000","last_active":"2024-03-30T13:51:11.775956+0000","last_peered":"2024-03-30T13:51:11.775956+0000","last_clean":"2024-03-30T13:51:11.775956+0000","last_became_active":"2024-03-30T13:42:51.325748+0000","last_became_peered":"2024-03-30T13:42:51.325748+0000","last_unstale":"2024-03-30T13:51:11.775956+0000","last_undegraded":"2024-03-30T13:51:11.775956+0000","last_fullsized":"2024-03-30T13:51:11.775956+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:10.988839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:10.988839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:27.596535+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":277,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.781338+0000","last_change":"2024-03-30T13:51:06.204404+0000","last_active":"2024-03-30T13:51:11.781338+0000","last_peered":"2024-03-30T13:51:11.781338+0000","last_clean":"2024-03-30T13:51:11.781338+0000","last_became_active":"2024-03-30T13:42:49.326027+0000","last_became_peered":"2024-03-30T13:42:49.326027+0000","last_unstale":"2024-03-30T13:51:11.781338+0000","last_undegraded":"2024-03-30T13:51:11.781338+0000","last_fullsized":"2024-03-30T13:51:11.781338+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:06.204282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:41.633284+0000","last_clean_scrub_stamp":"2024-03-30T13:51:06.204282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:21.468259+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"186'2","reported_seq":301,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:15.932151+0000","last_change":"2024-03-30T13:51:15.932194+0000","last_active":"2024-03-30T13:51:15.932151+0000","last_peered":"2024-03-30T13:51:15.932151+0000","last_clean":"2024-03-30T13:51:15.932151+0000","last_became_active":"2024-03-30T13:42:45.319975+0000","last_became_peered":"2024-03-30T13:42:45.319975+0000","last_unstale":"2024-03-30T13:51:15.932151+0000","last_undegraded":"2024-03-30T13:51:15.932151+0000","last_fullsized":"2024-03-30T13:51:15.932151+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-30T13:51:15.932076+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-30T13:51:15.932076+0000","last_clean_scrub_stamp":"2024-03-30T13:51:15.932076+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:34.376891+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":304,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:12.231648+0000","last_change":"2024-03-30T13:51:12.231695+0000","last_active":"2024-03-30T13:51:12.231648+0000","last_peered":"2024-03-30T13:51:12.231648+0000","last_clean":"2024-03-30T13:51:12.231648+0000","last_became_active":"2024-03-30T13:42:47.317352+0000","last_became_peered":"2024-03-30T13:42:47.317352+0000","last_unstale":"2024-03-30T13:51:12.231648+0000","last_undegraded":"2024-03-30T13:51:12.231648+0000","last_fullsized":"2024-03-30T13:51:12.231648+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-30T13:51:12.231574+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-30T13:43:50.960980+0000","last_clean_scrub_stamp":"2024-03-30T13:51:12.231574+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:13.364083+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":277,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.771673+0000","last_change":"2024-03-30T13:50:16.240962+0000","last_active":"2024-03-30T13:51:11.771673+0000","last_peered":"2024-03-30T13:51:11.771673+0000","last_clean":"2024-03-30T13:51:11.771673+0000","last_became_active":"2024-03-30T13:42:51.372011+0000","last_became_peered":"2024-03-30T13:42:51.372011+0000","last_unstale":"2024-03-30T13:51:11.771673+0000","last_undegraded":"2024-03-30T13:51:11.771673+0000","last_fullsized":"2024-03-30T13:51:11.771673+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:16.240830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:50:16.240830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:36.555476+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":287,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.783093+0000","last_change":"2024-03-30T13:51:05.319693+0000","last_active":"2024-03-30T13:51:11.783093+0000","last_peered":"2024-03-30T13:51:11.783093+0000","last_clean":"2024-03-30T13:51:11.783093+0000","last_became_active":"2024-03-30T13:42:49.322665+0000","last_became_peered":"2024-03-30T13:42:49.322665+0000","last_unstale":"2024-03-30T13:51:11.783093+0000","last_undegraded":"2024-03-30T13:51:11.783093+0000","last_fullsized":"2024-03-30T13:51:11.783093+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:05.319558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:30.465974+0000","last_clean_scrub_stamp":"2024-03-30T13:51:05.319558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:14.938754+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":285,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.786713+0000","last_change":"2024-03-30T13:50:56.342569+0000","last_active":"2024-03-30T13:51:11.786713+0000","last_peered":"2024-03-30T13:51:11.786713+0000","last_clean":"2024-03-30T13:51:11.786713+0000","last_became_active":"2024-03-30T13:42:45.317378+0000","last_became_peered":"2024-03-30T13:42:45.317378+0000","last_unstale":"2024-03-30T13:51:11.786713+0000","last_undegraded":"2024-03-30T13:51:11.786713+0000","last_fullsized":"2024-03-30T13:51:11.786713+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:56.342454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:43.088964+0000","last_clean_scrub_stamp":"2024-03-30T13:50:56.342454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:13.203972+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":103,"reported_epoch":186,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:51:11.782008+0000","last_change":"2024-03-30T13:50:08.508437+0000","last_active":"2024-03-30T13:51:11.782008+0000","last_peered":"2024-03-30T13:51:11.782008+0000","last_clean":"2024-03-30T13:50:05.490228+0000","last_became_active":"2024-03-30T13:50:08.508437+0000","last_became_peered":"2024-03-30T13:50:08.508437+0000","last_unstale":"2024-03-30T13:51:11.782008+0000","last_undegraded":"2024-03-30T13:50:08.506233+0000","last_fullsized":"2024-03-30T13:50:08.506116+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-30T13:49:08.415681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:49:08.415681+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:15.231634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":268,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.787837+0000","last_change":"2024-03-30T13:51:05.756377+0000","last_active":"2024-03-30T13:51:11.787837+0000","last_peered":"2024-03-30T13:51:11.787837+0000","last_clean":"2024-03-30T13:51:11.787837+0000","last_became_active":"2024-03-30T13:42:51.371378+0000","last_became_peered":"2024-03-30T13:42:51.371378+0000","last_unstale":"2024-03-30T13:51:11.787837+0000","last_undegraded":"2024-03-30T13:51:11.787837+0000","last_fullsized":"2024-03-30T13:51:11.787837+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:05.756146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:05.756146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:35.754284+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":275,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.772537+0000","last_change":"2024-03-30T13:50:08.506226+0000","last_active":"2024-03-30T13:51:11.772537+0000","last_peered":"2024-03-30T13:51:11.772537+0000","last_clean":"2024-03-30T13:50:05.490914+0000","last_became_active":"2024-03-30T13:50:08.506226+0000","last_became_peered":"2024-03-30T13:50:08.506226+0000","last_unstale":"2024-03-30T13:51:11.772537+0000","last_undegraded":"2024-03-30T13:51:11.772537+0000","last_fullsized":"2024-03-30T13:50:08.502611+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:25.104610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:48:25.104610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":283,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.773698+0000","last_change":"2024-03-30T13:50:08.508556+0000","last_active":"2024-03-30T13:51:11.773698+0000","last_peered":"2024-03-30T13:51:11.773698+0000","last_clean":"2024-03-30T13:50:07.261781+0000","last_became_active":"2024-03-30T13:50:08.508556+0000","last_became_peered":"2024-03-30T13:50:08.508556+0000","last_unstale":"2024-03-30T13:51:11.773698+0000","last_undegraded":"2024-03-30T13:51:11.773698+0000","last_fullsized":"2024-03-30T13:50:08.502793+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:07.261595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:50:07.261595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:21.269991+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"88'17","reported_seq":311,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:20.908735+0000","last_change":"2024-03-30T13:51:20.908809+0000","last_active":"2024-03-30T13:51:20.908735+0000","last_peered":"2024-03-30T13:51:20.908735+0000","last_clean":"2024-03-30T13:51:20.908735+0000","last_became_active":"2024-03-30T13:42:47.317881+0000","last_became_peered":"2024-03-30T13:42:47.317881+0000","last_unstale":"2024-03-30T13:51:20.908735+0000","last_undegraded":"2024-03-30T13:51:20.908735+0000","last_fullsized":"2024-03-30T13:51:20.908735+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'17","last_scrub_stamp":"2024-03-30T13:51:20.908641+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-03-30T13:43:52.044427+0000","last_clean_scrub_stamp":"2024-03-30T13:51:20.908641+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:35.127825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":269,"reported_epoch":186,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:51:11.782329+0000","last_change":"2024-03-30T13:50:08.517791+0000","last_active":"2024-03-30T13:51:11.782329+0000","last_peered":"2024-03-30T13:51:11.782329+0000","last_clean":"2024-03-30T13:50:05.495152+0000","last_became_active":"2024-03-30T13:50:08.517791+0000","last_became_peered":"2024-03-30T13:50:08.517791+0000","last_unstale":"2024-03-30T13:51:11.782329+0000","last_undegraded":"2024-03-30T13:50:08.511324+0000","last_fullsized":"2024-03-30T13:50:08.511207+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T13:48:25.249724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:25.249724+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:14.170881+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":401,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:15.464543+0000","last_change":"2024-03-30T13:51:03.111932+0000","last_active":"2024-03-30T13:51:15.464543+0000","last_peered":"2024-03-30T13:51:15.464543+0000","last_clean":"2024-03-30T13:51:15.464543+0000","last_became_active":"2024-03-30T13:42:49.323362+0000","last_became_peered":"2024-03-30T13:42:49.323362+0000","last_unstale":"2024-03-30T13:51:15.464543+0000","last_undegraded":"2024-03-30T13:51:15.464543+0000","last_fullsized":"2024-03-30T13:51:15.464543+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:51:03.111787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:03.111787+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:06.704581+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"65'1","reported_seq":296,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.799117+0000","last_change":"2024-03-30T13:51:03.380725+0000","last_active":"2024-03-30T13:51:11.799117+0000","last_peered":"2024-03-30T13:51:11.799117+0000","last_clean":"2024-03-30T13:51:11.799117+0000","last_became_active":"2024-03-30T13:42:45.307832+0000","last_became_peered":"2024-03-30T13:42:45.307832+0000","last_unstale":"2024-03-30T13:51:11.799117+0000","last_undegraded":"2024-03-30T13:51:11.799117+0000","last_fullsized":"2024-03-30T13:51:11.799117+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:51:03.380596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:03.380596+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:24.809963+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"153'15","reported_seq":310,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:22.253059+0000","last_change":"2024-03-30T13:51:22.253128+0000","last_active":"2024-03-30T13:51:22.253059+0000","last_peered":"2024-03-30T13:51:22.253059+0000","last_clean":"2024-03-30T13:51:22.253059+0000","last_became_active":"2024-03-30T13:42:47.500763+0000","last_became_peered":"2024-03-30T13:42:47.500763+0000","last_unstale":"2024-03-30T13:51:22.253059+0000","last_undegraded":"2024-03-30T13:51:22.253059+0000","last_fullsized":"2024-03-30T13:51:22.253059+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'15","last_scrub_stamp":"2024-03-30T13:51:22.252831+0000","last_deep_scrub":"153'15","last_deep_scrub_stamp":"2024-03-30T13:51:22.252831+0000","last_clean_scrub_stamp":"2024-03-30T13:51:22.252831+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:23.402370+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":275,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.782825+0000","last_change":"2024-03-30T13:50:12.388013+0000","last_active":"2024-03-30T13:51:11.782825+0000","last_peered":"2024-03-30T13:51:11.782825+0000","last_clean":"2024-03-30T13:51:11.782825+0000","last_became_active":"2024-03-30T13:42:51.325513+0000","last_became_peered":"2024-03-30T13:42:51.325513+0000","last_unstale":"2024-03-30T13:51:11.782825+0000","last_undegraded":"2024-03-30T13:51:11.782825+0000","last_fullsized":"2024-03-30T13:51:11.782825+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:12.387888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:50:12.387888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:27.186699+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":281,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.788483+0000","last_change":"2024-03-30T13:50:08.513281+0000","last_active":"2024-03-30T13:51:11.788483+0000","last_peered":"2024-03-30T13:51:11.788483+0000","last_clean":"2024-03-30T13:50:05.503977+0000","last_became_active":"2024-03-30T13:50:08.513281+0000","last_became_peered":"2024-03-30T13:50:08.513281+0000","last_unstale":"2024-03-30T13:51:11.788483+0000","last_undegraded":"2024-03-30T13:51:11.788483+0000","last_fullsized":"2024-03-30T13:50:08.504796+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:27.947416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:27.947416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":279,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.784266+0000","last_change":"2024-03-30T13:50:23.479815+0000","last_active":"2024-03-30T13:51:11.784266+0000","last_peered":"2024-03-30T13:51:11.784266+0000","last_clean":"2024-03-30T13:51:11.784266+0000","last_became_active":"2024-03-30T13:42:45.310165+0000","last_became_peered":"2024-03-30T13:42:45.310165+0000","last_unstale":"2024-03-30T13:51:11.784266+0000","last_undegraded":"2024-03-30T13:51:11.784266+0000","last_fullsized":"2024-03-30T13:51:11.784266+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:23.479531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:50:23.479531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:41.289396+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":309,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.778497+0000","last_change":"2024-03-30T13:51:02.103956+0000","last_active":"2024-03-30T13:51:11.778497+0000","last_peered":"2024-03-30T13:51:11.778497+0000","last_clean":"2024-03-30T13:51:11.778497+0000","last_became_active":"2024-03-30T13:42:47.324399+0000","last_became_peered":"2024-03-30T13:42:47.324399+0000","last_unstale":"2024-03-30T13:51:11.778497+0000","last_undegraded":"2024-03-30T13:51:11.778497+0000","last_fullsized":"2024-03-30T13:51:11.778497+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:51:02.103808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:02.103808+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:25.392277+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":268,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.781155+0000","last_change":"2024-03-30T13:50:08.512275+0000","last_active":"2024-03-30T13:51:11.781155+0000","last_peered":"2024-03-30T13:51:11.781155+0000","last_clean":"2024-03-30T13:50:05.495214+0000","last_became_active":"2024-03-30T13:50:08.512275+0000","last_became_peered":"2024-03-30T13:50:08.512275+0000","last_unstale":"2024-03-30T13:51:11.781155+0000","last_undegraded":"2024-03-30T13:51:11.781155+0000","last_fullsized":"2024-03-30T13:50:08.506150+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:23.264538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:23.264538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:14.170898+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":293,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.777751+0000","last_change":"2024-03-30T13:51:08.152625+0000","last_active":"2024-03-30T13:51:11.777751+0000","last_peered":"2024-03-30T13:51:11.777751+0000","last_clean":"2024-03-30T13:51:11.777751+0000","last_became_active":"2024-03-30T13:42:49.322799+0000","last_became_peered":"2024-03-30T13:42:49.322799+0000","last_unstale":"2024-03-30T13:51:11.777751+0000","last_undegraded":"2024-03-30T13:51:11.777751+0000","last_fullsized":"2024-03-30T13:51:11.777751+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:08.152508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:54.893653+0000","last_clean_scrub_stamp":"2024-03-30T13:51:08.152508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:14.770028+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":293,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:23.233239+0000","last_change":"2024-03-30T13:51:23.233277+0000","last_active":"2024-03-30T13:51:23.233239+0000","last_peered":"2024-03-30T13:51:23.233239+0000","last_clean":"2024-03-30T13:51:23.233239+0000","last_became_active":"2024-03-30T13:42:45.314302+0000","last_became_peered":"2024-03-30T13:42:45.314302+0000","last_unstale":"2024-03-30T13:51:23.233239+0000","last_undegraded":"2024-03-30T13:51:23.233239+0000","last_fullsized":"2024-03-30T13:51:23.233239+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:23.233143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:23.233143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:38.884648+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":281,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.784365+0000","last_change":"2024-03-30T13:49:25.632546+0000","last_active":"2024-03-30T13:51:11.784365+0000","last_peered":"2024-03-30T13:51:11.784365+0000","last_clean":"2024-03-30T13:51:11.784365+0000","last_became_active":"2024-03-30T13:42:47.321428+0000","last_became_peered":"2024-03-30T13:42:47.321428+0000","last_unstale":"2024-03-30T13:51:11.784365+0000","last_undegraded":"2024-03-30T13:51:11.784365+0000","last_fullsized":"2024-03-30T13:51:11.784365+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-30T13:49:25.632223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:49:25.632223+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":268,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:13.783491+0000","last_change":"2024-03-30T13:51:13.783533+0000","last_active":"2024-03-30T13:51:13.783491+0000","last_peered":"2024-03-30T13:51:13.783491+0000","last_clean":"2024-03-30T13:51:13.783491+0000","last_became_active":"2024-03-30T13:42:51.374304+0000","last_became_peered":"2024-03-30T13:42:51.374304+0000","last_unstale":"2024-03-30T13:51:13.783491+0000","last_undegraded":"2024-03-30T13:51:13.783491+0000","last_fullsized":"2024-03-30T13:51:13.783491+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:13.783403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:13.783403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:20.259203+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":285,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:20.297550+0000","last_change":"2024-03-30T13:51:20.297584+0000","last_active":"2024-03-30T13:51:20.297550+0000","last_peered":"2024-03-30T13:51:20.297550+0000","last_clean":"2024-03-30T13:51:20.297550+0000","last_became_active":"2024-03-30T13:42:49.324206+0000","last_became_peered":"2024-03-30T13:42:49.324206+0000","last_unstale":"2024-03-30T13:51:20.297550+0000","last_undegraded":"2024-03-30T13:51:20.297550+0000","last_fullsized":"2024-03-30T13:51:20.297550+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:20.297467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:42.616155+0000","last_clean_scrub_stamp":"2024-03-30T13:51:20.297467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:25.769571+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":286,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.777151+0000","last_change":"2024-03-30T13:50:17.703470+0000","last_active":"2024-03-30T13:51:11.777151+0000","last_peered":"2024-03-30T13:51:11.777151+0000","last_clean":"2024-03-30T13:51:11.777151+0000","last_became_active":"2024-03-30T13:42:45.319482+0000","last_became_peered":"2024-03-30T13:42:45.319482+0000","last_unstale":"2024-03-30T13:51:11.777151+0000","last_undegraded":"2024-03-30T13:51:11.777151+0000","last_fullsized":"2024-03-30T13:51:11.777151+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:17.703346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:25.494903+0000","last_clean_scrub_stamp":"2024-03-30T13:50:17.703346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:32.610667+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":302,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:21.862985+0000","last_change":"2024-03-30T13:51:21.863041+0000","last_active":"2024-03-30T13:51:21.862985+0000","last_peered":"2024-03-30T13:51:21.862985+0000","last_clean":"2024-03-30T13:51:21.862985+0000","last_became_active":"2024-03-30T13:42:47.500712+0000","last_became_peered":"2024-03-30T13:42:47.500712+0000","last_unstale":"2024-03-30T13:51:21.862985+0000","last_undegraded":"2024-03-30T13:51:21.862985+0000","last_fullsized":"2024-03-30T13:51:21.862985+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:51:21.862885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:21.862885+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:37.730856+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":273,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.781400+0000","last_change":"2024-03-30T13:50:14.255222+0000","last_active":"2024-03-30T13:51:11.781400+0000","last_peered":"2024-03-30T13:51:11.781400+0000","last_clean":"2024-03-30T13:51:11.781400+0000","last_became_active":"2024-03-30T13:42:51.333603+0000","last_became_peered":"2024-03-30T13:42:51.333603+0000","last_unstale":"2024-03-30T13:51:11.781400+0000","last_undegraded":"2024-03-30T13:51:11.781400+0000","last_fullsized":"2024-03-30T13:51:11.781400+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:14.254943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:28.314391+0000","last_clean_scrub_stamp":"2024-03-30T13:50:14.254943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:30.665848+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":271,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.784006+0000","last_change":"2024-03-30T13:50:13.446848+0000","last_active":"2024-03-30T13:51:11.784006+0000","last_peered":"2024-03-30T13:51:11.784006+0000","last_clean":"2024-03-30T13:51:11.784006+0000","last_became_active":"2024-03-30T13:42:49.325219+0000","last_became_peered":"2024-03-30T13:42:49.325219+0000","last_unstale":"2024-03-30T13:51:11.784006+0000","last_undegraded":"2024-03-30T13:51:11.784006+0000","last_fullsized":"2024-03-30T13:51:11.784006+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:13.446724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:50:13.446724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:33.481891+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":285,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.784208+0000","last_change":"2024-03-30T13:50:21.125840+0000","last_active":"2024-03-30T13:51:11.784208+0000","last_peered":"2024-03-30T13:51:11.784208+0000","last_clean":"2024-03-30T13:51:11.784208+0000","last_became_active":"2024-03-30T13:42:45.311071+0000","last_became_peered":"2024-03-30T13:42:45.311071+0000","last_unstale":"2024-03-30T13:51:11.784208+0000","last_undegraded":"2024-03-30T13:51:11.784208+0000","last_fullsized":"2024-03-30T13:51:11.784208+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:21.125709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:30.187218+0000","last_clean_scrub_stamp":"2024-03-30T13:50:21.125709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:45.343639+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":302,"reported_epoch":186,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:51:11.782428+0000","last_change":"2024-03-30T13:50:08.513156+0000","last_active":"2024-03-30T13:51:11.782428+0000","last_peered":"2024-03-30T13:51:11.782428+0000","last_clean":"2024-03-30T13:50:05.496678+0000","last_became_active":"2024-03-30T13:50:08.513156+0000","last_became_peered":"2024-03-30T13:50:08.513156+0000","last_unstale":"2024-03-30T13:51:11.782428+0000","last_undegraded":"2024-03-30T13:50:08.507726+0000","last_fullsized":"2024-03-30T13:50:08.507616+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:49:26.480540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:49:26.480540+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":267,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.782490+0000","last_change":"2024-03-30T13:49:24.599095+0000","last_active":"2024-03-30T13:51:11.782490+0000","last_peered":"2024-03-30T13:51:11.782490+0000","last_clean":"2024-03-30T13:51:11.782490+0000","last_became_active":"2024-03-30T13:42:51.334597+0000","last_became_peered":"2024-03-30T13:42:51.334597+0000","last_unstale":"2024-03-30T13:51:11.782490+0000","last_undegraded":"2024-03-30T13:51:11.782490+0000","last_fullsized":"2024-03-30T13:51:11.782490+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:24.598976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:24.598976+0000","last_clean_scrub_stamp":"2024-03-30T13:49:24.598976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":279,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.782230+0000","last_change":"2024-03-30T13:50:15.357882+0000","last_active":"2024-03-30T13:51:11.782230+0000","last_peered":"2024-03-30T13:51:11.782230+0000","last_clean":"2024-03-30T13:51:11.782230+0000","last_became_active":"2024-03-30T13:42:49.322843+0000","last_became_peered":"2024-03-30T13:42:49.322843+0000","last_unstale":"2024-03-30T13:51:11.782230+0000","last_undegraded":"2024-03-30T13:51:11.782230+0000","last_fullsized":"2024-03-30T13:51:11.782230+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:15.357758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:34.815783+0000","last_clean_scrub_stamp":"2024-03-30T13:50:15.357758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:25.366457+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":288,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.772324+0000","last_change":"2024-03-30T13:50:22.123301+0000","last_active":"2024-03-30T13:51:11.772324+0000","last_peered":"2024-03-30T13:51:11.772324+0000","last_clean":"2024-03-30T13:51:11.772324+0000","last_became_active":"2024-03-30T13:42:45.315864+0000","last_became_peered":"2024-03-30T13:42:45.315864+0000","last_unstale":"2024-03-30T13:51:11.772324+0000","last_undegraded":"2024-03-30T13:51:11.772324+0000","last_fullsized":"2024-03-30T13:51:11.772324+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:22.123086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:50:22.123086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:43.354388+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":316,"reported_epoch":186,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:51:11.772600+0000","last_change":"2024-03-30T13:50:08.513382+0000","last_active":"2024-03-30T13:51:11.772600+0000","last_peered":"2024-03-30T13:51:11.772600+0000","last_clean":"2024-03-30T13:50:05.491029+0000","last_became_active":"2024-03-30T13:50:08.513382+0000","last_became_peered":"2024-03-30T13:50:08.513382+0000","last_unstale":"2024-03-30T13:51:11.772600+0000","last_undegraded":"2024-03-30T13:50:08.503759+0000","last_fullsized":"2024-03-30T13:50:08.503643+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:48:22.068206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:22.068206+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":273,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.787828+0000","last_change":"2024-03-30T13:50:08.074731+0000","last_active":"2024-03-30T13:51:11.787828+0000","last_peered":"2024-03-30T13:51:11.787828+0000","last_clean":"2024-03-30T13:51:11.787828+0000","last_became_active":"2024-03-30T13:42:51.336162+0000","last_became_peered":"2024-03-30T13:42:51.336162+0000","last_unstale":"2024-03-30T13:51:11.787828+0000","last_undegraded":"2024-03-30T13:51:11.787828+0000","last_fullsized":"2024-03-30T13:51:11.787828+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:08.074599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:50:08.074599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:20.929340+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":390,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:15.464323+0000","last_change":"2024-03-30T13:51:01.033692+0000","last_active":"2024-03-30T13:51:15.464323+0000","last_peered":"2024-03-30T13:51:15.464323+0000","last_clean":"2024-03-30T13:51:15.464323+0000","last_became_active":"2024-03-30T13:42:49.321203+0000","last_became_peered":"2024-03-30T13:42:49.321203+0000","last_unstale":"2024-03-30T13:51:15.464323+0000","last_undegraded":"2024-03-30T13:51:15.464323+0000","last_fullsized":"2024-03-30T13:51:15.464323+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:51:01.033348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:01.033348+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:02.136442+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":280,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.781620+0000","last_change":"2024-03-30T13:50:08.511650+0000","last_active":"2024-03-30T13:51:11.781620+0000","last_peered":"2024-03-30T13:51:11.781620+0000","last_clean":"2024-03-30T13:50:05.495125+0000","last_became_active":"2024-03-30T13:50:08.511650+0000","last_became_peered":"2024-03-30T13:50:08.511650+0000","last_unstale":"2024-03-30T13:51:11.781620+0000","last_undegraded":"2024-03-30T13:51:11.781620+0000","last_fullsized":"2024-03-30T13:50:08.506728+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:07.618186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:49:07.618186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:14.170926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":303,"reported_epoch":186,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:51:11.783141+0000","last_change":"2024-03-30T13:50:08.514919+0000","last_active":"2024-03-30T13:51:11.783141+0000","last_peered":"2024-03-30T13:51:11.783141+0000","last_clean":"2024-03-30T13:50:05.508102+0000","last_became_active":"2024-03-30T13:50:08.514919+0000","last_became_peered":"2024-03-30T13:50:08.514919+0000","last_unstale":"2024-03-30T13:51:11.783141+0000","last_undegraded":"2024-03-30T13:50:08.503113+0000","last_fullsized":"2024-03-30T13:50:08.502993+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-30T13:48:21.678227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:21.678227+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:15.672439+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":271,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.771717+0000","last_change":"2024-03-30T13:50:08.514514+0000","last_active":"2024-03-30T13:51:11.771717+0000","last_peered":"2024-03-30T13:51:11.771717+0000","last_clean":"2024-03-30T13:50:05.490356+0000","last_became_active":"2024-03-30T13:50:08.514514+0000","last_became_peered":"2024-03-30T13:50:08.514514+0000","last_unstale":"2024-03-30T13:51:11.771717+0000","last_undegraded":"2024-03-30T13:51:11.771717+0000","last_fullsized":"2024-03-30T13:50:08.502291+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:30.238582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:30.238582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:14.241654+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"168'9","reported_seq":387,"reported_epoch":186,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:51:15.463184+0000","last_change":"2024-03-30T13:50:08.515013+0000","last_active":"2024-03-30T13:51:15.463184+0000","last_peered":"2024-03-30T13:51:15.463184+0000","last_clean":"2024-03-30T13:50:05.508192+0000","last_became_active":"2024-03-30T13:50:08.515013+0000","last_became_peered":"2024-03-30T13:50:08.515013+0000","last_unstale":"2024-03-30T13:51:15.463184+0000","last_undegraded":"2024-03-30T13:50:08.503461+0000","last_fullsized":"2024-03-30T13:50:08.503337+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:49:23.573435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:23.573435+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:23.836011+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":285,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.786132+0000","last_change":"2024-03-30T13:50:24.116308+0000","last_active":"2024-03-30T13:51:11.786132+0000","last_peered":"2024-03-30T13:51:11.786132+0000","last_clean":"2024-03-30T13:51:11.786132+0000","last_became_active":"2024-03-30T13:42:45.314585+0000","last_became_peered":"2024-03-30T13:42:45.314585+0000","last_unstale":"2024-03-30T13:51:11.786132+0000","last_undegraded":"2024-03-30T13:51:11.786132+0000","last_fullsized":"2024-03-30T13:51:11.786132+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:24.116187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:28.734080+0000","last_clean_scrub_stamp":"2024-03-30T13:50:24.116187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:43.756475+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":298,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.776729+0000","last_change":"2024-03-30T13:50:04.654720+0000","last_active":"2024-03-30T13:51:11.776729+0000","last_peered":"2024-03-30T13:51:11.776729+0000","last_clean":"2024-03-30T13:51:11.776729+0000","last_became_active":"2024-03-30T13:42:47.501559+0000","last_became_peered":"2024-03-30T13:42:47.501559+0000","last_unstale":"2024-03-30T13:51:11.776729+0000","last_undegraded":"2024-03-30T13:51:11.776729+0000","last_fullsized":"2024-03-30T13:51:11.776729+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:50:04.654608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:50:04.654608+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:28.190191+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":59,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.772832+0000","last_change":"2024-03-30T13:50:08.513016+0000","last_active":"2024-03-30T13:51:11.772832+0000","last_peered":"2024-03-30T13:51:11.772832+0000","last_clean":"2024-03-30T13:50:05.491001+0000","last_became_active":"2024-03-30T13:50:08.513016+0000","last_became_peered":"2024-03-30T13:50:08.513016+0000","last_unstale":"2024-03-30T13:51:11.772832+0000","last_undegraded":"2024-03-30T13:51:11.772832+0000","last_fullsized":"2024-03-30T13:50:08.502775+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:14.322254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:14.322254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":271,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.788417+0000","last_change":"2024-03-30T13:51:08.670117+0000","last_active":"2024-03-30T13:51:11.788417+0000","last_peered":"2024-03-30T13:51:11.788417+0000","last_clean":"2024-03-30T13:51:11.788417+0000","last_became_active":"2024-03-30T13:42:49.323222+0000","last_became_peered":"2024-03-30T13:42:49.323222+0000","last_unstale":"2024-03-30T13:51:11.788417+0000","last_undegraded":"2024-03-30T13:51:11.788417+0000","last_fullsized":"2024-03-30T13:51:11.788417+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:08.670035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:08.670035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:16.796672+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":293,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:18.239104+0000","last_change":"2024-03-30T13:51:18.239125+0000","last_active":"2024-03-30T13:51:18.239104+0000","last_peered":"2024-03-30T13:51:18.239104+0000","last_clean":"2024-03-30T13:51:18.239104+0000","last_became_active":"2024-03-30T13:42:45.313617+0000","last_became_peered":"2024-03-30T13:42:45.313617+0000","last_unstale":"2024-03-30T13:51:18.239104+0000","last_undegraded":"2024-03-30T13:51:18.239104+0000","last_fullsized":"2024-03-30T13:51:18.239104+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:18.239047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:18.239047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:47.096925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":305,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:20.179718+0000","last_change":"2024-03-30T13:51:20.179768+0000","last_active":"2024-03-30T13:51:20.179718+0000","last_peered":"2024-03-30T13:51:20.179718+0000","last_clean":"2024-03-30T13:51:20.179718+0000","last_became_active":"2024-03-30T13:42:47.501925+0000","last_became_peered":"2024-03-30T13:42:47.501925+0000","last_unstale":"2024-03-30T13:51:20.179718+0000","last_undegraded":"2024-03-30T13:51:20.179718+0000","last_fullsized":"2024-03-30T13:51:20.179718+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:51:20.179623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:20.179623+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:45.908958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"72'1","reported_seq":282,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:17.299768+0000","last_change":"2024-03-30T13:51:17.299808+0000","last_active":"2024-03-30T13:51:17.299768+0000","last_peered":"2024-03-30T13:51:17.299768+0000","last_clean":"2024-03-30T13:51:17.299768+0000","last_became_active":"2024-03-30T13:42:51.372023+0000","last_became_peered":"2024-03-30T13:42:51.372023+0000","last_unstale":"2024-03-30T13:51:17.299768+0000","last_undegraded":"2024-03-30T13:51:17.299768+0000","last_fullsized":"2024-03-30T13:51:17.299768+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T13:51:17.299690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:17.299690+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:20.658989+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":278,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.777206+0000","last_change":"2024-03-30T13:50:21.754632+0000","last_active":"2024-03-30T13:51:11.777206+0000","last_peered":"2024-03-30T13:51:11.777206+0000","last_clean":"2024-03-30T13:51:11.777206+0000","last_became_active":"2024-03-30T13:42:49.323924+0000","last_became_peered":"2024-03-30T13:42:49.323924+0000","last_unstale":"2024-03-30T13:51:11.777206+0000","last_undegraded":"2024-03-30T13:51:11.777206+0000","last_fullsized":"2024-03-30T13:51:11.777206+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:21.754451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:50:21.754451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:39.713177+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":301,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:20.175525+0000","last_change":"2024-03-30T13:51:20.175560+0000","last_active":"2024-03-30T13:51:20.175525+0000","last_peered":"2024-03-30T13:51:20.175525+0000","last_clean":"2024-03-30T13:51:20.175525+0000","last_became_active":"2024-03-30T13:42:45.318956+0000","last_became_peered":"2024-03-30T13:42:45.318956+0000","last_unstale":"2024-03-30T13:51:20.175525+0000","last_undegraded":"2024-03-30T13:51:20.175525+0000","last_fullsized":"2024-03-30T13:51:20.175525+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:20.175438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:51:20.175438+0000","last_clean_scrub_stamp":"2024-03-30T13:51:20.175438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:36.026962+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":284,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.788321+0000","last_change":"2024-03-30T13:50:21.411326+0000","last_active":"2024-03-30T13:51:11.788321+0000","last_peered":"2024-03-30T13:51:11.788321+0000","last_clean":"2024-03-30T13:51:11.788321+0000","last_became_active":"2024-03-30T13:42:47.501426+0000","last_became_peered":"2024-03-30T13:42:47.501426+0000","last_unstale":"2024-03-30T13:51:11.788321+0000","last_undegraded":"2024-03-30T13:51:11.788321+0000","last_fullsized":"2024-03-30T13:51:11.788321+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-03-30T13:50:21.411186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:50:21.411186+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:26.160791+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"186'11256","reported_seq":11350,"reported_epoch":186,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:51:23.486934+0000","last_change":"2024-03-30T13:50:08.578484+0000","last_active":"2024-03-30T13:51:23.486934+0000","last_peered":"2024-03-30T13:51:23.486934+0000","last_clean":"2024-03-30T13:50:07.502722+0000","last_became_active":"2024-03-30T13:50:08.524860+0000","last_became_peered":"2024-03-30T13:50:08.524860+0000","last_unstale":"2024-03-30T13:51:23.486934+0000","last_undegraded":"2024-03-30T13:50:08.516186+0000","last_fullsized":"2024-03-30T13:50:08.505072+0000","mapping_epoch":167,"log_start":"168'5800","ondisk_log_start":"168'5800","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:49:00.428679+0000","objects_scrubbed":0,"log_size":5456,"log_dups_size":0,"ondisk_log_size":5456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:26.075985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339607552,"num_objects":5182,"num_object_clones":0,"num_object_copies":15546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11256,"num_write_kb":526016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":5182}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":274,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.776852+0000","last_change":"2024-03-30T13:51:05.957160+0000","last_active":"2024-03-30T13:51:11.776852+0000","last_peered":"2024-03-30T13:51:11.776852+0000","last_clean":"2024-03-30T13:51:11.776852+0000","last_became_active":"2024-03-30T13:42:51.371360+0000","last_became_peered":"2024-03-30T13:42:51.371360+0000","last_unstale":"2024-03-30T13:51:11.776852+0000","last_undegraded":"2024-03-30T13:51:11.776852+0000","last_fullsized":"2024-03-30T13:51:11.776852+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:05.957043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:05.957043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:32.455311+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":297,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.782582+0000","last_change":"2024-03-30T13:50:24.087276+0000","last_active":"2024-03-30T13:51:11.782582+0000","last_peered":"2024-03-30T13:51:11.782582+0000","last_clean":"2024-03-30T13:51:11.782582+0000","last_became_active":"2024-03-30T13:42:45.318963+0000","last_became_peered":"2024-03-30T13:42:45.318963+0000","last_unstale":"2024-03-30T13:51:11.782582+0000","last_undegraded":"2024-03-30T13:51:11.782582+0000","last_fullsized":"2024-03-30T13:51:11.782582+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-30T13:50:24.087137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:50:24.087137+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:40.212530+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":307,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.776375+0000","last_change":"2024-03-30T13:50:15.706829+0000","last_active":"2024-03-30T13:51:11.776375+0000","last_peered":"2024-03-30T13:51:11.776375+0000","last_clean":"2024-03-30T13:51:11.776375+0000","last_became_active":"2024-03-30T13:42:47.501937+0000","last_became_peered":"2024-03-30T13:42:47.501937+0000","last_unstale":"2024-03-30T13:51:11.776375+0000","last_undegraded":"2024-03-30T13:51:11.776375+0000","last_fullsized":"2024-03-30T13:51:11.776375+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-30T13:50:15.706257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:50:15.706257+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:26.932938+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"72'8","reported_seq":397,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:19.286719+0000","last_change":"2024-03-30T13:51:19.286719+0000","last_active":"2024-03-30T13:51:19.286719+0000","last_peered":"2024-03-30T13:51:19.286719+0000","last_clean":"2024-03-30T13:51:19.286719+0000","last_became_active":"2024-03-30T13:42:49.322675+0000","last_became_peered":"2024-03-30T13:42:49.322675+0000","last_unstale":"2024-03-30T13:51:19.286719+0000","last_undegraded":"2024-03-30T13:51:19.286719+0000","last_fullsized":"2024-03-30T13:51:19.286719+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:51:19.286592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:19.286592+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:46.501686+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"186'13394","reported_seq":13689,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:23.535989+0000","last_change":"2024-03-30T13:50:57.613375+0000","last_active":"2024-03-30T13:51:23.535989+0000","last_peered":"2024-03-30T13:51:23.535989+0000","last_clean":"2024-03-30T13:51:23.535989+0000","last_became_active":"2024-03-30T13:49:01.523463+0000","last_became_peered":"2024-03-30T13:49:01.523463+0000","last_unstale":"2024-03-30T13:51:23.535989+0000","last_undegraded":"2024-03-30T13:51:23.535989+0000","last_fullsized":"2024-03-30T13:51:23.535989+0000","mapping_epoch":147,"log_start":"172'7101","ondisk_log_start":"172'7101","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'11418","last_scrub_stamp":"2024-03-30T13:50:57.613202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:50:57.613202+0000","objects_scrubbed":7237,"log_size":6293,"log_dups_size":0,"ondisk_log_size":6293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:01.709319+0000","scrub_duration":2538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337772544,"num_objects":5154,"num_object_clones":0,"num_object_copies":15462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11288,"num_write_kb":526144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":267,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.784200+0000","last_change":"2024-03-30T13:50:59.723370+0000","last_active":"2024-03-30T13:51:11.784200+0000","last_peered":"2024-03-30T13:51:11.784200+0000","last_clean":"2024-03-30T13:51:11.784200+0000","last_became_active":"2024-03-30T13:42:51.334672+0000","last_became_peered":"2024-03-30T13:42:51.334672+0000","last_unstale":"2024-03-30T13:51:11.784200+0000","last_undegraded":"2024-03-30T13:51:11.784200+0000","last_fullsized":"2024-03-30T13:51:11.784200+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:59.723249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:50:59.723249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:27.249728+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":293,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:15.247307+0000","last_change":"2024-03-30T13:51:15.247447+0000","last_active":"2024-03-30T13:51:15.247307+0000","last_peered":"2024-03-30T13:51:15.247307+0000","last_clean":"2024-03-30T13:51:15.247307+0000","last_became_active":"2024-03-30T13:42:45.312328+0000","last_became_peered":"2024-03-30T13:42:45.312328+0000","last_unstale":"2024-03-30T13:51:15.247307+0000","last_undegraded":"2024-03-30T13:51:15.247307+0000","last_fullsized":"2024-03-30T13:51:15.247307+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:15.246997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:15.246997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:35.377772+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":307,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.771661+0000","last_change":"2024-03-30T13:50:10.274074+0000","last_active":"2024-03-30T13:51:11.771661+0000","last_peered":"2024-03-30T13:51:11.771661+0000","last_clean":"2024-03-30T13:51:11.771661+0000","last_became_active":"2024-03-30T13:42:47.316187+0000","last_became_peered":"2024-03-30T13:42:47.316187+0000","last_unstale":"2024-03-30T13:51:11.771661+0000","last_undegraded":"2024-03-30T13:51:11.771661+0000","last_fullsized":"2024-03-30T13:51:11.771661+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-30T13:50:10.273940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:50:10.273940+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:38.877835+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":59,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.781909+0000","last_change":"2024-03-30T13:50:08.509701+0000","last_active":"2024-03-30T13:51:11.781909+0000","last_peered":"2024-03-30T13:51:11.781909+0000","last_clean":"2024-03-30T13:50:05.499237+0000","last_became_active":"2024-03-30T13:50:08.509701+0000","last_became_peered":"2024-03-30T13:50:08.509701+0000","last_unstale":"2024-03-30T13:51:11.781909+0000","last_undegraded":"2024-03-30T13:51:11.781909+0000","last_fullsized":"2024-03-30T13:50:08.501671+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:22.401087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:22.401087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"186'11214","reported_seq":11372,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:24.241881+0000","last_change":"2024-03-30T13:51:11.338320+0000","last_active":"2024-03-30T13:51:24.241881+0000","last_peered":"2024-03-30T13:51:24.241881+0000","last_clean":"2024-03-30T13:51:24.241881+0000","last_became_active":"2024-03-30T13:49:01.523984+0000","last_became_peered":"2024-03-30T13:49:01.523984+0000","last_unstale":"2024-03-30T13:51:24.241881+0000","last_undegraded":"2024-03-30T13:51:24.241881+0000","last_fullsized":"2024-03-30T13:51:24.241881+0000","mapping_epoch":147,"log_start":"166'5400","ondisk_log_start":"166'5400","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'10292","last_scrub_stamp":"2024-03-30T13:51:11.338007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:51:11.338007+0000","objects_scrubbed":6036,"log_size":5814,"log_dups_size":0,"ondisk_log_size":5814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:28.787160+0000","scrub_duration":2135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329383936,"num_objects":5026,"num_object_clones":0,"num_object_copies":15078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11214,"num_write_kb":519680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":276,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.781023+0000","last_change":"2024-03-30T13:50:08.510439+0000","last_active":"2024-03-30T13:51:11.781023+0000","last_peered":"2024-03-30T13:51:11.781023+0000","last_clean":"2024-03-30T13:50:05.494808+0000","last_became_active":"2024-03-30T13:50:08.510439+0000","last_became_peered":"2024-03-30T13:50:08.510439+0000","last_unstale":"2024-03-30T13:51:11.781023+0000","last_undegraded":"2024-03-30T13:51:11.781023+0000","last_fullsized":"2024-03-30T13:50:08.506960+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:31.860675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:31.860675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":59,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.786349+0000","last_change":"2024-03-30T13:50:08.509003+0000","last_active":"2024-03-30T13:51:11.786349+0000","last_peered":"2024-03-30T13:51:11.786349+0000","last_clean":"2024-03-30T13:50:05.499015+0000","last_became_active":"2024-03-30T13:50:08.509003+0000","last_became_peered":"2024-03-30T13:50:08.509003+0000","last_unstale":"2024-03-30T13:51:11.786349+0000","last_undegraded":"2024-03-30T13:51:11.786349+0000","last_fullsized":"2024-03-30T13:50:08.502979+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:19.432242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:19.432242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":285,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.784350+0000","last_change":"2024-03-30T13:51:09.173688+0000","last_active":"2024-03-30T13:51:11.784350+0000","last_peered":"2024-03-30T13:51:11.784350+0000","last_clean":"2024-03-30T13:51:11.784350+0000","last_became_active":"2024-03-30T13:42:45.315687+0000","last_became_peered":"2024-03-30T13:42:45.315687+0000","last_unstale":"2024-03-30T13:51:11.784350+0000","last_undegraded":"2024-03-30T13:51:11.784350+0000","last_fullsized":"2024-03-30T13:51:11.784350+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:09.173459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:09.173459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:33.137912+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"108'28","reported_seq":329,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.777726+0000","last_change":"2024-03-30T13:51:04.139113+0000","last_active":"2024-03-30T13:51:11.777726+0000","last_peered":"2024-03-30T13:51:11.777726+0000","last_clean":"2024-03-30T13:51:11.777726+0000","last_became_active":"2024-03-30T13:42:47.500898+0000","last_became_peered":"2024-03-30T13:42:47.500898+0000","last_unstale":"2024-03-30T13:51:11.777726+0000","last_undegraded":"2024-03-30T13:51:11.777726+0000","last_fullsized":"2024-03-30T13:51:11.777726+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'28","last_scrub_stamp":"2024-03-30T13:51:04.138964+0000","last_deep_scrub":"108'28","last_deep_scrub_stamp":"2024-03-30T13:45:49.140418+0000","last_clean_scrub_stamp":"2024-03-30T13:51:04.138964+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:06.081318+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"186'11157","reported_seq":11317,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:23.811638+0000","last_change":"2024-03-30T13:50:18.439070+0000","last_active":"2024-03-30T13:51:23.811638+0000","last_peered":"2024-03-30T13:51:23.811638+0000","last_clean":"2024-03-30T13:51:23.811638+0000","last_became_active":"2024-03-30T13:49:01.453528+0000","last_became_peered":"2024-03-30T13:49:01.453528+0000","last_unstale":"2024-03-30T13:51:23.811638+0000","last_undegraded":"2024-03-30T13:51:23.811638+0000","last_fullsized":"2024-03-30T13:51:23.811638+0000","mapping_epoch":147,"log_start":"169'6200","ondisk_log_start":"169'6200","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'6278","last_scrub_stamp":"2024-03-30T13:50:18.438748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:50:18.438748+0000","objects_scrubbed":6210,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:41.170909+0000","scrub_duration":2083,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323420160,"num_objects":4935,"num_object_clones":0,"num_object_copies":14805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11157,"num_write_kb":514944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":268,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.790399+0000","last_change":"2024-03-30T13:50:08.509476+0000","last_active":"2024-03-30T13:51:11.790399+0000","last_peered":"2024-03-30T13:51:11.790399+0000","last_clean":"2024-03-30T13:50:05.499348+0000","last_became_active":"2024-03-30T13:50:08.509476+0000","last_became_peered":"2024-03-30T13:50:08.509476+0000","last_unstale":"2024-03-30T13:51:11.790399+0000","last_undegraded":"2024-03-30T13:51:11.790399+0000","last_fullsized":"2024-03-30T13:50:08.503098+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:12.807605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:44.095338+0000","last_clean_scrub_stamp":"2024-03-30T13:49:12.807605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":59,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.773788+0000","last_change":"2024-03-30T13:50:08.507875+0000","last_active":"2024-03-30T13:51:11.773788+0000","last_peered":"2024-03-30T13:51:11.773788+0000","last_clean":"2024-03-30T13:50:05.490245+0000","last_became_active":"2024-03-30T13:50:08.507875+0000","last_became_peered":"2024-03-30T13:50:08.507875+0000","last_unstale":"2024-03-30T13:51:11.773788+0000","last_undegraded":"2024-03-30T13:51:11.773788+0000","last_fullsized":"2024-03-30T13:50:08.502700+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:13.399683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:28.225742+0000","last_clean_scrub_stamp":"2024-03-30T13:49:13.399683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:14.241636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":294,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:19.099365+0000","last_change":"2024-03-30T13:51:19.099448+0000","last_active":"2024-03-30T13:51:19.099365+0000","last_peered":"2024-03-30T13:51:19.099365+0000","last_clean":"2024-03-30T13:51:19.099365+0000","last_became_active":"2024-03-30T13:42:45.314767+0000","last_became_peered":"2024-03-30T13:42:45.314767+0000","last_unstale":"2024-03-30T13:51:19.099365+0000","last_undegraded":"2024-03-30T13:51:19.099365+0000","last_fullsized":"2024-03-30T13:51:19.099365+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:19.099205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:19.099205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:25.579479+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":321,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:13.148767+0000","last_change":"2024-03-30T13:51:13.148898+0000","last_active":"2024-03-30T13:51:13.148767+0000","last_peered":"2024-03-30T13:51:13.148767+0000","last_clean":"2024-03-30T13:51:13.148767+0000","last_became_active":"2024-03-30T13:43:11.446379+0000","last_became_peered":"2024-03-30T13:43:11.446379+0000","last_unstale":"2024-03-30T13:51:13.148767+0000","last_undegraded":"2024-03-30T13:51:13.148767+0000","last_fullsized":"2024-03-30T13:51:13.148767+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:51:13.148563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:13.148563+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:13.860133+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"186'11306","reported_seq":11467,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:23.497880+0000","last_change":"2024-03-30T13:51:09.898743+0000","last_active":"2024-03-30T13:51:23.497880+0000","last_peered":"2024-03-30T13:51:23.497880+0000","last_clean":"2024-03-30T13:51:23.497880+0000","last_became_active":"2024-03-30T13:49:01.523800+0000","last_became_peered":"2024-03-30T13:49:01.523800+0000","last_unstale":"2024-03-30T13:51:23.497880+0000","last_undegraded":"2024-03-30T13:51:23.497880+0000","last_fullsized":"2024-03-30T13:51:23.497880+0000","mapping_epoch":147,"log_start":"168'5900","ondisk_log_start":"168'5900","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'10246","last_scrub_stamp":"2024-03-30T13:51:09.898422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:51:09.898422+0000","objects_scrubbed":6175,"log_size":5406,"log_dups_size":0,"ondisk_log_size":5406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:31.561635+0000","scrub_duration":1956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330694656,"num_objects":5046,"num_object_clones":0,"num_object_copies":15138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11306,"num_write_kb":523264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":267,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.783225+0000","last_change":"2024-03-30T13:49:34.585368+0000","last_active":"2024-03-30T13:51:11.783225+0000","last_peered":"2024-03-30T13:51:11.783225+0000","last_clean":"2024-03-30T13:51:11.783225+0000","last_became_active":"2024-03-30T13:42:51.331054+0000","last_became_peered":"2024-03-30T13:42:51.331054+0000","last_unstale":"2024-03-30T13:51:11.783225+0000","last_undegraded":"2024-03-30T13:51:11.783225+0000","last_fullsized":"2024-03-30T13:51:11.783225+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:34.585241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:34.585241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":306,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:12.972555+0000","last_change":"2024-03-30T13:51:12.972656+0000","last_active":"2024-03-30T13:51:12.972555+0000","last_peered":"2024-03-30T13:51:12.972555+0000","last_clean":"2024-03-30T13:51:12.972555+0000","last_became_active":"2024-03-30T13:42:44.300789+0000","last_became_peered":"2024-03-30T13:42:44.300789+0000","last_unstale":"2024-03-30T13:51:12.972555+0000","last_undegraded":"2024-03-30T13:51:12.972555+0000","last_fullsized":"2024-03-30T13:51:12.972555+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-30T13:51:12.972422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:51:12.972422+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:25.492356+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.0021837800000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":289,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.772290+0000","last_change":"2024-03-30T13:51:01.240545+0000","last_active":"2024-03-30T13:51:11.772290+0000","last_peered":"2024-03-30T13:51:11.772290+0000","last_clean":"2024-03-30T13:51:11.772290+0000","last_became_active":"2024-03-30T13:42:45.310209+0000","last_became_peered":"2024-03-30T13:42:45.310209+0000","last_unstale":"2024-03-30T13:51:11.772290+0000","last_undegraded":"2024-03-30T13:51:11.772290+0000","last_fullsized":"2024-03-30T13:51:11.772290+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:01.240468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:01.240468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:27.407517+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"168'29","reported_seq":432,"reported_epoch":186,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:51:15.463686+0000","last_change":"2024-03-30T13:50:08.508727+0000","last_active":"2024-03-30T13:51:15.463686+0000","last_peered":"2024-03-30T13:51:15.463686+0000","last_clean":"2024-03-30T13:50:05.490389+0000","last_became_active":"2024-03-30T13:50:08.508727+0000","last_became_peered":"2024-03-30T13:50:08.508727+0000","last_unstale":"2024-03-30T13:51:15.463686+0000","last_undegraded":"2024-03-30T13:50:08.506113+0000","last_fullsized":"2024-03-30T13:50:08.505993+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-03-30T13:48:27.377242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:27.377242+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:25.067246+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":286,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:18.299261+0000","last_change":"2024-03-30T13:51:18.299446+0000","last_active":"2024-03-30T13:51:18.299261+0000","last_peered":"2024-03-30T13:51:18.299261+0000","last_clean":"2024-03-30T13:51:18.299261+0000","last_became_active":"2024-03-30T13:42:49.322154+0000","last_became_peered":"2024-03-30T13:42:49.322154+0000","last_unstale":"2024-03-30T13:51:18.299261+0000","last_undegraded":"2024-03-30T13:51:18.299261+0000","last_fullsized":"2024-03-30T13:51:18.299261+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:18.299184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:18.862784+0000","last_clean_scrub_stamp":"2024-03-30T13:51:18.299184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:21.412798+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"186'12826","reported_seq":12983,"reported_epoch":186,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-30T13:51:23.886409+0000","last_change":"2024-03-30T13:51:14.761728+0000","last_active":"2024-03-30T13:51:23.886409+0000","last_peered":"2024-03-30T13:51:23.886409+0000","last_clean":"2024-03-30T13:51:23.886409+0000","last_became_active":"2024-03-30T13:49:01.451971+0000","last_became_peered":"2024-03-30T13:49:01.451971+0000","last_unstale":"2024-03-30T13:51:23.886409+0000","last_undegraded":"2024-03-30T13:51:23.886409+0000","last_fullsized":"2024-03-30T13:51:23.886409+0000","mapping_epoch":147,"log_start":"178'7901","ondisk_log_start":"178'7901","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:49:00.428679+0000","objects_scrubbed":1686,"log_size":4925,"log_dups_size":0,"ondisk_log_size":4925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 9s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332660736,"num_objects":5076,"num_object_clones":0,"num_object_copies":15228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11140,"num_write_kb":518912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":274,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:14.276209+0000","last_change":"2024-03-30T13:51:14.276390+0000","last_active":"2024-03-30T13:51:14.276209+0000","last_peered":"2024-03-30T13:51:14.276209+0000","last_clean":"2024-03-30T13:51:14.276209+0000","last_became_active":"2024-03-30T13:42:51.372320+0000","last_became_peered":"2024-03-30T13:42:51.372320+0000","last_unstale":"2024-03-30T13:51:14.276209+0000","last_undegraded":"2024-03-30T13:51:14.276209+0000","last_fullsized":"2024-03-30T13:51:14.276209+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:14.275893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:14.275893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:28.628361+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":291,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.794185+0000","last_change":"2024-03-30T13:50:08.512561+0000","last_active":"2024-03-30T13:51:11.794185+0000","last_peered":"2024-03-30T13:51:11.794185+0000","last_clean":"2024-03-30T13:50:05.489457+0000","last_became_active":"2024-03-30T13:50:08.512561+0000","last_became_peered":"2024-03-30T13:50:08.512561+0000","last_unstale":"2024-03-30T13:51:11.794185+0000","last_undegraded":"2024-03-30T13:51:11.794185+0000","last_fullsized":"2024-03-30T13:50:08.506886+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:16.464234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:59.810189+0000","last_clean_scrub_stamp":"2024-03-30T13:49:16.464234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:15.231618+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":59,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.782721+0000","last_change":"2024-03-30T13:50:08.511337+0000","last_active":"2024-03-30T13:51:11.782721+0000","last_peered":"2024-03-30T13:51:11.782721+0000","last_clean":"2024-03-30T13:50:05.498892+0000","last_became_active":"2024-03-30T13:50:08.511337+0000","last_became_peered":"2024-03-30T13:50:08.511337+0000","last_unstale":"2024-03-30T13:51:11.782721+0000","last_undegraded":"2024-03-30T13:51:11.782721+0000","last_fullsized":"2024-03-30T13:50:08.501456+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:25.398232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:51.919203+0000","last_clean_scrub_stamp":"2024-03-30T13:49:25.398232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"72'13","reported_seq":318,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.802477+0000","last_change":"2024-03-30T13:50:55.258399+0000","last_active":"2024-03-30T13:51:11.802477+0000","last_peered":"2024-03-30T13:51:11.802477+0000","last_clean":"2024-03-30T13:51:11.802477+0000","last_became_active":"2024-03-30T13:42:47.320541+0000","last_became_peered":"2024-03-30T13:42:47.320541+0000","last_unstale":"2024-03-30T13:51:11.802477+0000","last_undegraded":"2024-03-30T13:51:11.802477+0000","last_fullsized":"2024-03-30T13:51:11.802477+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'13","last_scrub_stamp":"2024-03-30T13:50:55.258141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:50:55.258141+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:15.076428+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":59,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.776975+0000","last_change":"2024-03-30T13:50:08.514353+0000","last_active":"2024-03-30T13:51:11.776975+0000","last_peered":"2024-03-30T13:51:11.776975+0000","last_clean":"2024-03-30T13:50:05.492654+0000","last_became_active":"2024-03-30T13:50:08.514353+0000","last_became_peered":"2024-03-30T13:50:08.514353+0000","last_unstale":"2024-03-30T13:51:11.776975+0000","last_undegraded":"2024-03-30T13:51:11.776975+0000","last_fullsized":"2024-03-30T13:50:08.505968+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:28.373277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:28.373277+0000","last_clean_scrub_stamp":"2024-03-30T13:49:28.373277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"186'11026","reported_seq":11191,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:23.522130+0000","last_change":"2024-03-30T13:51:02.091636+0000","last_active":"2024-03-30T13:51:23.522130+0000","last_peered":"2024-03-30T13:51:23.522130+0000","last_clean":"2024-03-30T13:51:23.522130+0000","last_became_active":"2024-03-30T13:49:01.450928+0000","last_became_peered":"2024-03-30T13:49:01.450928+0000","last_unstale":"2024-03-30T13:51:23.522130+0000","last_undegraded":"2024-03-30T13:51:23.522130+0000","last_fullsized":"2024-03-30T13:51:23.522130+0000","mapping_epoch":147,"log_start":"161'4700","ondisk_log_start":"161'4700","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'9448","last_scrub_stamp":"2024-03-30T13:51:02.091497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:51:02.091497+0000","objects_scrubbed":6714,"log_size":6326,"log_dups_size":0,"ondisk_log_size":6326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:19.157876+0000","scrub_duration":1970,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331218944,"num_objects":5054,"num_object_clones":0,"num_object_copies":15162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11026,"num_write_kb":514560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":282,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.776621+0000","last_change":"2024-03-30T13:50:59.155093+0000","last_active":"2024-03-30T13:51:11.776621+0000","last_peered":"2024-03-30T13:51:11.776621+0000","last_clean":"2024-03-30T13:51:11.776621+0000","last_became_active":"2024-03-30T13:42:51.331081+0000","last_became_peered":"2024-03-30T13:42:51.331081+0000","last_unstale":"2024-03-30T13:51:11.776621+0000","last_undegraded":"2024-03-30T13:51:11.776621+0000","last_fullsized":"2024-03-30T13:51:11.776621+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:59.154972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:12.433166+0000","last_clean_scrub_stamp":"2024-03-30T13:50:59.154972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:25.335507+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":539,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.781336+0000","last_change":"2024-03-30T13:51:03.160585+0000","last_active":"2024-03-30T13:51:11.781336+0000","last_peered":"2024-03-30T13:51:11.781336+0000","last_clean":"2024-03-30T13:51:11.781336+0000","last_became_active":"2024-03-30T13:42:34.988246+0000","last_became_peered":"2024-03-30T13:42:34.988246+0000","last_unstale":"2024-03-30T13:51:11.781336+0000","last_undegraded":"2024-03-30T13:51:11.781336+0000","last_fullsized":"2024-03-30T13:51:11.781336+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:51:03.160454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:51:03.160454+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:15.406710+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":280,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.783453+0000","last_change":"2024-03-30T13:50:05.402032+0000","last_active":"2024-03-30T13:51:11.783453+0000","last_peered":"2024-03-30T13:51:11.783453+0000","last_clean":"2024-03-30T13:51:11.783453+0000","last_became_active":"2024-03-30T13:42:45.309954+0000","last_became_peered":"2024-03-30T13:42:45.309954+0000","last_unstale":"2024-03-30T13:51:11.783453+0000","last_undegraded":"2024-03-30T13:51:11.783453+0000","last_fullsized":"2024-03-30T13:51:11.783453+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:05.401914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:50:05.401914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":416,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:15.464291+0000","last_change":"2024-03-30T13:50:04.749219+0000","last_active":"2024-03-30T13:51:15.464291+0000","last_peered":"2024-03-30T13:51:15.464291+0000","last_clean":"2024-03-30T13:51:15.464291+0000","last_became_active":"2024-03-30T13:42:47.502338+0000","last_became_peered":"2024-03-30T13:42:47.502338+0000","last_unstale":"2024-03-30T13:51:15.464291+0000","last_undegraded":"2024-03-30T13:51:15.464291+0000","last_fullsized":"2024-03-30T13:51:15.464291+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'16","last_scrub_stamp":"2024-03-30T13:50:04.748944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:50:04.748944+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:30.953018+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":281,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.782546+0000","last_change":"2024-03-30T13:50:08.517677+0000","last_active":"2024-03-30T13:51:11.782546+0000","last_peered":"2024-03-30T13:51:11.782546+0000","last_clean":"2024-03-30T13:50:05.495178+0000","last_became_active":"2024-03-30T13:50:08.517677+0000","last_became_peered":"2024-03-30T13:50:08.517677+0000","last_unstale":"2024-03-30T13:51:11.782546+0000","last_undegraded":"2024-03-30T13:51:11.782546+0000","last_fullsized":"2024-03-30T13:50:08.511555+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:25.787524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:25.787524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:14.170912+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"186'11292","reported_seq":11384,"reported_epoch":186,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:51:23.739568+0000","last_change":"2024-03-30T13:50:08.514509+0000","last_active":"2024-03-30T13:51:23.739568+0000","last_peered":"2024-03-30T13:51:23.739568+0000","last_clean":"2024-03-30T13:50:07.503192+0000","last_became_active":"2024-03-30T13:50:08.514509+0000","last_became_peered":"2024-03-30T13:50:08.514509+0000","last_unstale":"2024-03-30T13:51:23.739568+0000","last_undegraded":"2024-03-30T13:50:08.507026+0000","last_fullsized":"2024-03-30T13:50:08.506909+0000","mapping_epoch":167,"log_start":"169'6300","ondisk_log_start":"169'6300","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:49:00.428679+0000","objects_scrubbed":0,"log_size":4992,"log_dups_size":0,"ondisk_log_size":4992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:28.168170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339476480,"num_objects":5180,"num_object_clones":0,"num_object_copies":15540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11292,"num_write_kb":527104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5180}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":276,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.781887+0000","last_change":"2024-03-30T13:50:10.359099+0000","last_active":"2024-03-30T13:51:11.781887+0000","last_peered":"2024-03-30T13:51:11.781887+0000","last_clean":"2024-03-30T13:51:11.781887+0000","last_became_active":"2024-03-30T13:42:51.335188+0000","last_became_peered":"2024-03-30T13:42:51.335188+0000","last_unstale":"2024-03-30T13:51:11.781887+0000","last_undegraded":"2024-03-30T13:51:11.781887+0000","last_fullsized":"2024-03-30T13:51:11.781887+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:10.358990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:50:10.358990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:28.941367+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":286,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.790029+0000","last_change":"2024-03-30T13:50:23.121798+0000","last_active":"2024-03-30T13:51:11.790029+0000","last_peered":"2024-03-30T13:51:11.790029+0000","last_clean":"2024-03-30T13:51:11.790029+0000","last_became_active":"2024-03-30T13:42:45.307905+0000","last_became_peered":"2024-03-30T13:42:45.307905+0000","last_unstale":"2024-03-30T13:51:11.790029+0000","last_undegraded":"2024-03-30T13:51:11.790029+0000","last_fullsized":"2024-03-30T13:51:11.790029+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:23.121692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:50:23.121692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:31.992593+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":809,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:22.417595+0000","last_change":"2024-03-30T13:51:02.143007+0000","last_active":"2024-03-30T13:51:22.417595+0000","last_peered":"2024-03-30T13:51:22.417595+0000","last_clean":"2024-03-30T13:51:22.417595+0000","last_became_active":"2024-03-30T13:42:44.302343+0000","last_became_peered":"2024-03-30T13:42:44.302343+0000","last_unstale":"2024-03-30T13:51:22.417595+0000","last_undegraded":"2024-03-30T13:51:22.417595+0000","last_fullsized":"2024-03-30T13:51:22.417595+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-30T13:51:02.142275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:51:02.142275+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:03.353373+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00126391,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":511,"num_read_kb":506,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"93'18","reported_seq":308,"reported_epoch":186,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:51:11.787929+0000","last_change":"2024-03-30T13:50:08.510256+0000","last_active":"2024-03-30T13:51:11.787929+0000","last_peered":"2024-03-30T13:51:11.787929+0000","last_clean":"2024-03-30T13:50:05.503358+0000","last_became_active":"2024-03-30T13:50:08.510256+0000","last_became_peered":"2024-03-30T13:50:08.510256+0000","last_unstale":"2024-03-30T13:51:11.787929+0000","last_undegraded":"2024-03-30T13:50:08.504536+0000","last_fullsized":"2024-03-30T13:50:08.504398+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'18","last_scrub_stamp":"2024-03-30T13:49:13.774982+0000","last_deep_scrub":"93'18","last_deep_scrub_stamp":"2024-03-30T13:47:28.872529+0000","last_clean_scrub_stamp":"2024-03-30T13:49:13.774982+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":278,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.777103+0000","last_change":"2024-03-30T13:51:08.978000+0000","last_active":"2024-03-30T13:51:11.777103+0000","last_peered":"2024-03-30T13:51:11.777103+0000","last_clean":"2024-03-30T13:51:11.777103+0000","last_became_active":"2024-03-30T13:42:49.322700+0000","last_became_peered":"2024-03-30T13:42:49.322700+0000","last_unstale":"2024-03-30T13:51:11.777103+0000","last_undegraded":"2024-03-30T13:51:11.777103+0000","last_fullsized":"2024-03-30T13:51:11.777103+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:08.977897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:08.977897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:36.468589+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"186'11283","reported_seq":11374,"reported_epoch":186,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:51:23.843158+0000","last_change":"2024-03-30T13:50:08.509534+0000","last_active":"2024-03-30T13:51:23.843158+0000","last_peered":"2024-03-30T13:51:23.843158+0000","last_clean":"2024-03-30T13:50:07.503222+0000","last_became_active":"2024-03-30T13:50:08.509534+0000","last_became_peered":"2024-03-30T13:50:08.509534+0000","last_unstale":"2024-03-30T13:51:23.843158+0000","last_undegraded":"2024-03-30T13:50:08.503986+0000","last_fullsized":"2024-03-30T13:50:08.503847+0000","mapping_epoch":167,"log_start":"170'6400","ondisk_log_start":"170'6400","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:49:00.428679+0000","objects_scrubbed":0,"log_size":4883,"log_dups_size":0,"ondisk_log_size":4883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331284480,"num_objects":5055,"num_object_clones":0,"num_object_copies":15165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11283,"num_write_kb":522816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":5055}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":279,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.798800+0000","last_change":"2024-03-30T13:50:08.512648+0000","last_active":"2024-03-30T13:51:11.798800+0000","last_peered":"2024-03-30T13:51:11.798800+0000","last_clean":"2024-03-30T13:50:05.489180+0000","last_became_active":"2024-03-30T13:50:08.512648+0000","last_became_peered":"2024-03-30T13:50:08.512648+0000","last_unstale":"2024-03-30T13:51:11.798800+0000","last_undegraded":"2024-03-30T13:51:11.798800+0000","last_fullsized":"2024-03-30T13:50:08.505931+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:21.527712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:21.527712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:15.231602+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":291,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.782208+0000","last_change":"2024-03-30T13:50:08.517734+0000","last_active":"2024-03-30T13:51:11.782208+0000","last_peered":"2024-03-30T13:51:11.782208+0000","last_clean":"2024-03-30T13:50:05.490307+0000","last_became_active":"2024-03-30T13:50:08.517734+0000","last_became_peered":"2024-03-30T13:50:08.517734+0000","last_unstale":"2024-03-30T13:51:11.782208+0000","last_undegraded":"2024-03-30T13:51:11.782208+0000","last_fullsized":"2024-03-30T13:50:08.507406+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:19.525007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:45.975163+0000","last_clean_scrub_stamp":"2024-03-30T13:49:19.525007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":318,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.786795+0000","last_change":"2024-03-30T13:50:59.350614+0000","last_active":"2024-03-30T13:51:11.786795+0000","last_peered":"2024-03-30T13:51:11.786795+0000","last_clean":"2024-03-30T13:51:11.786795+0000","last_became_active":"2024-03-30T13:42:47.501272+0000","last_became_peered":"2024-03-30T13:42:47.501272+0000","last_unstale":"2024-03-30T13:51:11.786795+0000","last_undegraded":"2024-03-30T13:51:11.786795+0000","last_fullsized":"2024-03-30T13:51:11.786795+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-30T13:50:59.350459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:50:59.350459+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:09.076654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":294,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.777833+0000","last_change":"2024-03-30T13:51:09.196116+0000","last_active":"2024-03-30T13:51:11.777833+0000","last_peered":"2024-03-30T13:51:11.777833+0000","last_clean":"2024-03-30T13:51:11.777833+0000","last_became_active":"2024-03-30T13:42:49.321520+0000","last_became_peered":"2024-03-30T13:42:49.321520+0000","last_unstale":"2024-03-30T13:51:11.777833+0000","last_undegraded":"2024-03-30T13:51:11.777833+0000","last_fullsized":"2024-03-30T13:51:11.777833+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:09.196024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:51.900305+0000","last_clean_scrub_stamp":"2024-03-30T13:51:09.196024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:37.584924+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"186'11335","reported_seq":11425,"reported_epoch":186,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:51:24.262457+0000","last_change":"2024-03-30T13:50:08.513477+0000","last_active":"2024-03-30T13:51:24.262457+0000","last_peered":"2024-03-30T13:51:24.262457+0000","last_clean":"2024-03-30T13:50:07.482534+0000","last_became_active":"2024-03-30T13:50:08.513477+0000","last_became_peered":"2024-03-30T13:50:08.513477+0000","last_unstale":"2024-03-30T13:51:24.262457+0000","last_undegraded":"2024-03-30T13:50:08.504066+0000","last_fullsized":"2024-03-30T13:50:08.503954+0000","mapping_epoch":167,"log_start":"166'5500","ondisk_log_start":"166'5500","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:49:00.428679+0000","objects_scrubbed":0,"log_size":5835,"log_dups_size":0,"ondisk_log_size":5835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:28.226936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333119488,"num_objects":5083,"num_object_clones":0,"num_object_copies":15249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11335,"num_write_kb":525376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":5083}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":277,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.781501+0000","last_change":"2024-03-30T13:50:08.510025+0000","last_active":"2024-03-30T13:51:11.781501+0000","last_peered":"2024-03-30T13:51:11.781501+0000","last_clean":"2024-03-30T13:50:05.498930+0000","last_became_active":"2024-03-30T13:50:08.510025+0000","last_became_peered":"2024-03-30T13:50:08.510025+0000","last_unstale":"2024-03-30T13:51:11.781501+0000","last_undegraded":"2024-03-30T13:51:11.781501+0000","last_fullsized":"2024-03-30T13:50:08.502478+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:28.946303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:41.142941+0000","last_clean_scrub_stamp":"2024-03-30T13:49:28.946303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":290,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.776614+0000","last_change":"2024-03-30T13:50:08.514920+0000","last_active":"2024-03-30T13:51:11.776614+0000","last_peered":"2024-03-30T13:51:11.776614+0000","last_clean":"2024-03-30T13:50:05.493549+0000","last_became_active":"2024-03-30T13:50:08.514920+0000","last_became_peered":"2024-03-30T13:50:08.514920+0000","last_unstale":"2024-03-30T13:51:11.776614+0000","last_undegraded":"2024-03-30T13:51:11.776614+0000","last_fullsized":"2024-03-30T13:50:08.504625+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:08.543531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:55.458779+0000","last_clean_scrub_stamp":"2024-03-30T13:49:08.543531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:13.959106+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"109'15","reported_seq":312,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.782325+0000","last_change":"2024-03-30T13:51:00.279278+0000","last_active":"2024-03-30T13:51:11.782325+0000","last_peered":"2024-03-30T13:51:11.782325+0000","last_clean":"2024-03-30T13:51:11.782325+0000","last_became_active":"2024-03-30T13:42:47.321433+0000","last_became_peered":"2024-03-30T13:42:47.321433+0000","last_unstale":"2024-03-30T13:51:11.782325+0000","last_undegraded":"2024-03-30T13:51:11.782325+0000","last_fullsized":"2024-03-30T13:51:11.782325+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'15","last_scrub_stamp":"2024-03-30T13:51:00.279148+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-03-30T13:45:50.945837+0000","last_clean_scrub_stamp":"2024-03-30T13:51:00.279148+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:03.034933+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":398,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:17.257600+0000","last_change":"2024-03-30T13:51:17.257600+0000","last_active":"2024-03-30T13:51:17.257600+0000","last_peered":"2024-03-30T13:51:17.257600+0000","last_clean":"2024-03-30T13:51:17.257600+0000","last_became_active":"2024-03-30T13:42:49.325533+0000","last_became_peered":"2024-03-30T13:42:49.325533+0000","last_unstale":"2024-03-30T13:51:17.257600+0000","last_undegraded":"2024-03-30T13:51:17.257600+0000","last_fullsized":"2024-03-30T13:51:17.257600+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:51:17.257464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:17.257464+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:41.745294+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"186'11028","reported_seq":11121,"reported_epoch":186,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:51:23.470538+0000","last_change":"2024-03-30T13:50:08.515012+0000","last_active":"2024-03-30T13:51:23.470538+0000","last_peered":"2024-03-30T13:51:23.470538+0000","last_clean":"2024-03-30T13:50:07.483605+0000","last_became_active":"2024-03-30T13:50:08.515012+0000","last_became_peered":"2024-03-30T13:50:08.515012+0000","last_unstale":"2024-03-30T13:51:23.470538+0000","last_undegraded":"2024-03-30T13:50:08.506241+0000","last_fullsized":"2024-03-30T13:50:08.506127+0000","mapping_epoch":167,"log_start":"168'5600","ondisk_log_start":"168'5600","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:49:00.428679+0000","objects_scrubbed":0,"log_size":5428,"log_dups_size":0,"ondisk_log_size":5428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:26.075957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327483416,"num_objects":4998,"num_object_clones":0,"num_object_copies":14994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4998,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11028,"num_write_kb":512769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":4998}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":269,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.781901+0000","last_change":"2024-03-30T13:50:08.513975+0000","last_active":"2024-03-30T13:51:11.781901+0000","last_peered":"2024-03-30T13:51:11.781901+0000","last_clean":"2024-03-30T13:50:05.495405+0000","last_became_active":"2024-03-30T13:50:08.513975+0000","last_became_peered":"2024-03-30T13:50:08.513975+0000","last_unstale":"2024-03-30T13:51:11.781901+0000","last_undegraded":"2024-03-30T13:51:11.781901+0000","last_fullsized":"2024-03-30T13:50:08.507347+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:21.782290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:31.900908+0000","last_clean_scrub_stamp":"2024-03-30T13:49:21.782290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":295,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:12.127429+0000","last_change":"2024-03-30T13:51:12.127429+0000","last_active":"2024-03-30T13:51:12.127429+0000","last_peered":"2024-03-30T13:51:12.127429+0000","last_clean":"2024-03-30T13:51:12.127429+0000","last_became_active":"2024-03-30T13:42:45.309886+0000","last_became_peered":"2024-03-30T13:42:45.309886+0000","last_unstale":"2024-03-30T13:51:12.127429+0000","last_undegraded":"2024-03-30T13:51:12.127429+0000","last_fullsized":"2024-03-30T13:51:12.127429+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:51:12.127305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:12.127305+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:34.020036+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"91'21","reported_seq":321,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.771465+0000","last_change":"2024-03-30T13:50:59.187414+0000","last_active":"2024-03-30T13:51:11.771465+0000","last_peered":"2024-03-30T13:51:11.771465+0000","last_clean":"2024-03-30T13:51:11.771465+0000","last_became_active":"2024-03-30T13:42:47.314184+0000","last_became_peered":"2024-03-30T13:42:47.314184+0000","last_unstale":"2024-03-30T13:51:11.771465+0000","last_undegraded":"2024-03-30T13:51:11.771465+0000","last_fullsized":"2024-03-30T13:51:11.771465+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'21","last_scrub_stamp":"2024-03-30T13:50:59.187324+0000","last_deep_scrub":"91'21","last_deep_scrub_stamp":"2024-03-30T13:44:05.917064+0000","last_clean_scrub_stamp":"2024-03-30T13:50:59.187324+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:13.439993+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":384,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:15.463959+0000","last_change":"2024-03-30T13:50:57.701974+0000","last_active":"2024-03-30T13:51:15.463959+0000","last_peered":"2024-03-30T13:51:15.463959+0000","last_clean":"2024-03-30T13:51:15.463959+0000","last_became_active":"2024-03-30T13:42:49.321423+0000","last_became_peered":"2024-03-30T13:42:49.321423+0000","last_unstale":"2024-03-30T13:51:15.463959+0000","last_undegraded":"2024-03-30T13:51:15.463959+0000","last_fullsized":"2024-03-30T13:51:15.463959+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:50:57.701778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:50:57.701778+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:01.616640+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"186'11118","reported_seq":11280,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:24.253723+0000","last_change":"2024-03-30T13:51:07.821191+0000","last_active":"2024-03-30T13:51:24.253723+0000","last_peered":"2024-03-30T13:51:24.253723+0000","last_clean":"2024-03-30T13:51:24.253723+0000","last_became_active":"2024-03-30T13:49:01.523003+0000","last_became_peered":"2024-03-30T13:49:01.523003+0000","last_unstale":"2024-03-30T13:51:24.253723+0000","last_undegraded":"2024-03-30T13:51:24.253723+0000","last_fullsized":"2024-03-30T13:51:24.253723+0000","mapping_epoch":147,"log_start":"165'5300","ondisk_log_start":"165'5300","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'9942","last_scrub_stamp":"2024-03-30T13:51:07.820988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:51:07.820988+0000","objects_scrubbed":6427,"log_size":5818,"log_dups_size":0,"ondisk_log_size":5818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:15.476236+0000","scrub_duration":2640,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338296832,"num_objects":5162,"num_object_clones":0,"num_object_copies":15486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11118,"num_write_kb":520960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":272,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.773507+0000","last_change":"2024-03-30T13:50:08.504411+0000","last_active":"2024-03-30T13:51:11.773507+0000","last_peered":"2024-03-30T13:51:11.773507+0000","last_clean":"2024-03-30T13:50:05.489944+0000","last_became_active":"2024-03-30T13:50:08.504411+0000","last_became_peered":"2024-03-30T13:50:08.504411+0000","last_unstale":"2024-03-30T13:51:11.773507+0000","last_undegraded":"2024-03-30T13:51:11.773507+0000","last_fullsized":"2024-03-30T13:50:08.501823+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:10.216387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:10.216387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:17.222191+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":279,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.788752+0000","last_change":"2024-03-30T13:49:36.535267+0000","last_active":"2024-03-30T13:51:11.788752+0000","last_peered":"2024-03-30T13:51:11.788752+0000","last_clean":"2024-03-30T13:51:11.788752+0000","last_became_active":"2024-03-30T13:42:45.314533+0000","last_became_peered":"2024-03-30T13:42:45.314533+0000","last_unstale":"2024-03-30T13:51:11.788752+0000","last_undegraded":"2024-03-30T13:51:11.788752+0000","last_fullsized":"2024-03-30T13:51:11.788752+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:36.534524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:24.171036+0000","last_clean_scrub_stamp":"2024-03-30T13:49:36.534524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":316,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:16.834690+0000","last_change":"2024-03-30T13:51:16.834761+0000","last_active":"2024-03-30T13:51:16.834690+0000","last_peered":"2024-03-30T13:51:16.834690+0000","last_clean":"2024-03-30T13:51:16.834690+0000","last_became_active":"2024-03-30T13:42:47.502708+0000","last_became_peered":"2024-03-30T13:42:47.502708+0000","last_unstale":"2024-03-30T13:51:16.834690+0000","last_undegraded":"2024-03-30T13:51:16.834690+0000","last_fullsized":"2024-03-30T13:51:16.834690+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-03-30T13:51:16.834525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:16.834525+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:36.215406+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":281,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.771308+0000","last_change":"2024-03-30T13:51:09.247429+0000","last_active":"2024-03-30T13:51:11.771308+0000","last_peered":"2024-03-30T13:51:11.771308+0000","last_clean":"2024-03-30T13:51:11.771308+0000","last_became_active":"2024-03-30T13:42:49.324022+0000","last_became_peered":"2024-03-30T13:42:49.324022+0000","last_unstale":"2024-03-30T13:51:11.771308+0000","last_undegraded":"2024-03-30T13:51:11.771308+0000","last_fullsized":"2024-03-30T13:51:11.771308+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:09.247282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:09.247282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:39.099111+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"186'11264","reported_seq":11424,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:23.781852+0000","last_change":"2024-03-30T13:50:20.738998+0000","last_active":"2024-03-30T13:51:23.781852+0000","last_peered":"2024-03-30T13:51:23.781852+0000","last_clean":"2024-03-30T13:51:23.781852+0000","last_became_active":"2024-03-30T13:49:01.453308+0000","last_became_peered":"2024-03-30T13:49:01.453308+0000","last_unstale":"2024-03-30T13:51:23.781852+0000","last_undegraded":"2024-03-30T13:51:23.781852+0000","last_fullsized":"2024-03-30T13:51:23.781852+0000","mapping_epoch":147,"log_start":"170'6300","ondisk_log_start":"170'6300","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'6443","last_scrub_stamp":"2024-03-30T13:50:20.738584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:50:20.738584+0000","objects_scrubbed":6365,"log_size":4964,"log_dups_size":0,"ondisk_log_size":4964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:32.295223+0000","scrub_duration":2346,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334495744,"num_objects":5104,"num_object_clones":0,"num_object_copies":15312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11264,"num_write_kb":523776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":274,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.776504+0000","last_change":"2024-03-30T13:50:22.770010+0000","last_active":"2024-03-30T13:51:11.776504+0000","last_peered":"2024-03-30T13:51:11.776504+0000","last_clean":"2024-03-30T13:51:11.776504+0000","last_became_active":"2024-03-30T13:42:51.329026+0000","last_became_peered":"2024-03-30T13:42:51.329026+0000","last_unstale":"2024-03-30T13:51:11.776504+0000","last_undegraded":"2024-03-30T13:51:11.776504+0000","last_fullsized":"2024-03-30T13:51:11.776504+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:22.769749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:50:22.769749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:40.108723+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"186'9","reported_seq":401,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:15.462867+0000","last_change":"2024-03-30T13:51:15.146386+0000","last_active":"2024-03-30T13:51:15.462867+0000","last_peered":"2024-03-30T13:51:15.462867+0000","last_clean":"2024-03-30T13:51:15.462867+0000","last_became_active":"2024-03-30T13:42:49.321790+0000","last_became_peered":"2024-03-30T13:42:49.321790+0000","last_unstale":"2024-03-30T13:51:15.462867+0000","last_undegraded":"2024-03-30T13:51:15.462867+0000","last_fullsized":"2024-03-30T13:51:15.462867+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-30T13:51:15.146255+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-30T13:51:15.146255+0000","last_clean_scrub_stamp":"2024-03-30T13:51:15.146255+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:35.921845+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":279,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.784469+0000","last_change":"2024-03-30T13:51:06.667727+0000","last_active":"2024-03-30T13:51:11.784469+0000","last_peered":"2024-03-30T13:51:11.784469+0000","last_clean":"2024-03-30T13:51:11.784469+0000","last_became_active":"2024-03-30T13:42:45.315319+0000","last_became_peered":"2024-03-30T13:42:45.315319+0000","last_unstale":"2024-03-30T13:51:11.784469+0000","last_undegraded":"2024-03-30T13:51:11.784469+0000","last_fullsized":"2024-03-30T13:51:11.784469+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:06.667621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:06.667621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:23.994913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"95'22","reported_seq":315,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.802291+0000","last_change":"2024-03-30T13:50:11.391175+0000","last_active":"2024-03-30T13:51:11.802291+0000","last_peered":"2024-03-30T13:51:11.802291+0000","last_clean":"2024-03-30T13:51:11.802291+0000","last_became_active":"2024-03-30T13:42:47.324350+0000","last_became_peered":"2024-03-30T13:42:47.324350+0000","last_unstale":"2024-03-30T13:51:11.802291+0000","last_undegraded":"2024-03-30T13:51:11.802291+0000","last_fullsized":"2024-03-30T13:51:11.802291+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'22","last_scrub_stamp":"2024-03-30T13:50:11.390813+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:28.831527+0000","last_clean_scrub_stamp":"2024-03-30T13:50:11.390813+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:23.783612+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"186'11240","reported_seq":11408,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:23.487260+0000","last_change":"2024-03-30T13:50:58.228849+0000","last_active":"2024-03-30T13:51:23.487260+0000","last_peered":"2024-03-30T13:51:23.487260+0000","last_clean":"2024-03-30T13:51:23.487260+0000","last_became_active":"2024-03-30T13:49:01.450057+0000","last_became_peered":"2024-03-30T13:49:01.450057+0000","last_unstale":"2024-03-30T13:51:23.487260+0000","last_undegraded":"2024-03-30T13:51:23.487260+0000","last_fullsized":"2024-03-30T13:51:23.487260+0000","mapping_epoch":147,"log_start":"170'6400","ondisk_log_start":"170'6400","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'9381","last_scrub_stamp":"2024-03-30T13:50:58.228675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:50:58.228675+0000","objects_scrubbed":7120,"log_size":4840,"log_dups_size":0,"ondisk_log_size":4840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:17.135717+0000","scrub_duration":1990,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340656128,"num_objects":5198,"num_object_clones":0,"num_object_copies":15594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11240,"num_write_kb":526016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":267,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.782945+0000","last_change":"2024-03-30T13:51:10.683723+0000","last_active":"2024-03-30T13:51:11.782945+0000","last_peered":"2024-03-30T13:51:11.782945+0000","last_clean":"2024-03-30T13:51:11.782945+0000","last_became_active":"2024-03-30T13:42:51.371369+0000","last_became_peered":"2024-03-30T13:42:51.371369+0000","last_unstale":"2024-03-30T13:51:11.782945+0000","last_undegraded":"2024-03-30T13:51:11.782945+0000","last_fullsized":"2024-03-30T13:51:11.782945+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:10.683596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:10.683596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:11.668157+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":286,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:21.087806+0000","last_change":"2024-03-30T13:51:21.087839+0000","last_active":"2024-03-30T13:51:21.087806+0000","last_peered":"2024-03-30T13:51:21.087806+0000","last_clean":"2024-03-30T13:51:21.087806+0000","last_became_active":"2024-03-30T13:42:49.323319+0000","last_became_peered":"2024-03-30T13:42:49.323319+0000","last_unstale":"2024-03-30T13:51:21.087806+0000","last_undegraded":"2024-03-30T13:51:21.087806+0000","last_fullsized":"2024-03-30T13:51:21.087806+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:21.087732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:52.603772+0000","last_clean_scrub_stamp":"2024-03-30T13:51:21.087732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:26.664698+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":283,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.771984+0000","last_change":"2024-03-30T13:50:08.509331+0000","last_active":"2024-03-30T13:51:11.771984+0000","last_peered":"2024-03-30T13:51:11.771984+0000","last_clean":"2024-03-30T13:50:05.489382+0000","last_became_active":"2024-03-30T13:50:08.509331+0000","last_became_peered":"2024-03-30T13:50:08.509331+0000","last_unstale":"2024-03-30T13:51:11.771984+0000","last_undegraded":"2024-03-30T13:51:11.771984+0000","last_fullsized":"2024-03-30T13:50:08.501285+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:15.107879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:48:15.107879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"72'11","reported_seq":311,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.771572+0000","last_change":"2024-03-30T13:50:58.241795+0000","last_active":"2024-03-30T13:51:11.771572+0000","last_peered":"2024-03-30T13:51:11.771572+0000","last_clean":"2024-03-30T13:51:11.771572+0000","last_became_active":"2024-03-30T13:42:47.315667+0000","last_became_peered":"2024-03-30T13:42:47.315667+0000","last_unstale":"2024-03-30T13:51:11.771572+0000","last_undegraded":"2024-03-30T13:51:11.771572+0000","last_fullsized":"2024-03-30T13:51:11.771572+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:50:58.241649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:50:58.241649+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:00.294790+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"186'11009","reported_seq":11101,"reported_epoch":186,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:51:23.831169+0000","last_change":"2024-03-30T13:50:08.511322+0000","last_active":"2024-03-30T13:51:23.831169+0000","last_peered":"2024-03-30T13:51:23.831169+0000","last_clean":"2024-03-30T13:50:07.457630+0000","last_became_active":"2024-03-30T13:50:08.511322+0000","last_became_peered":"2024-03-30T13:50:08.511322+0000","last_unstale":"2024-03-30T13:51:23.831169+0000","last_undegraded":"2024-03-30T13:50:08.505556+0000","last_fullsized":"2024-03-30T13:50:08.505411+0000","mapping_epoch":167,"log_start":"169'6000","ondisk_log_start":"169'6000","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:49:00.428679+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:25.072357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326303744,"num_objects":4979,"num_object_clones":0,"num_object_copies":14937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11009,"num_write_kb":511616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":4979}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":59,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.787609+0000","last_change":"2024-03-30T13:50:08.511676+0000","last_active":"2024-03-30T13:51:11.787609+0000","last_peered":"2024-03-30T13:51:11.787609+0000","last_clean":"2024-03-30T13:50:05.504989+0000","last_became_active":"2024-03-30T13:50:08.511676+0000","last_became_peered":"2024-03-30T13:50:08.511676+0000","last_unstale":"2024-03-30T13:51:11.787609+0000","last_undegraded":"2024-03-30T13:51:11.787609+0000","last_fullsized":"2024-03-30T13:50:08.504157+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:09.357378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:09.357378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:15.672422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":287,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.782577+0000","last_change":"2024-03-30T13:50:57.267042+0000","last_active":"2024-03-30T13:51:11.782577+0000","last_peered":"2024-03-30T13:51:11.782577+0000","last_clean":"2024-03-30T13:51:11.782577+0000","last_became_active":"2024-03-30T13:42:49.321360+0000","last_became_peered":"2024-03-30T13:42:49.321360+0000","last_unstale":"2024-03-30T13:51:11.782577+0000","last_undegraded":"2024-03-30T13:51:11.782577+0000","last_fullsized":"2024-03-30T13:51:11.782577+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:57.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:50:57.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:59.315622+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":301,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:21.196540+0000","last_change":"2024-03-30T13:51:21.196578+0000","last_active":"2024-03-30T13:51:21.196540+0000","last_peered":"2024-03-30T13:51:21.196540+0000","last_clean":"2024-03-30T13:51:21.196540+0000","last_became_active":"2024-03-30T13:42:45.315251+0000","last_became_peered":"2024-03-30T13:42:45.315251+0000","last_unstale":"2024-03-30T13:51:21.196540+0000","last_undegraded":"2024-03-30T13:51:21.196540+0000","last_fullsized":"2024-03-30T13:51:21.196540+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:21.196457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:21.196457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:24.534984+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":307,"reported_epoch":186,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:51:11.773075+0000","last_change":"2024-03-30T13:50:08.508030+0000","last_active":"2024-03-30T13:51:11.773075+0000","last_peered":"2024-03-30T13:51:11.773075+0000","last_clean":"2024-03-30T13:50:05.491159+0000","last_became_active":"2024-03-30T13:50:08.508030+0000","last_became_peered":"2024-03-30T13:50:08.508030+0000","last_unstale":"2024-03-30T13:51:11.773075+0000","last_undegraded":"2024-03-30T13:50:08.502958+0000","last_fullsized":"2024-03-30T13:50:08.502815+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-03-30T13:48:27.096261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:27.096261+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"186'11270","reported_seq":11423,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:23.874924+0000","last_change":"2024-03-30T13:51:20.782899+0000","last_active":"2024-03-30T13:51:23.874924+0000","last_peered":"2024-03-30T13:51:23.874924+0000","last_clean":"2024-03-30T13:51:23.874924+0000","last_became_active":"2024-03-30T13:49:01.450507+0000","last_became_peered":"2024-03-30T13:49:01.450507+0000","last_unstale":"2024-03-30T13:51:23.874924+0000","last_undegraded":"2024-03-30T13:51:23.874924+0000","last_fullsized":"2024-03-30T13:51:23.874924+0000","mapping_epoch":147,"log_start":"170'6400","ondisk_log_start":"170'6400","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'11079","last_scrub_stamp":"2024-03-30T13:51:20.782636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:51:20.782636+0000","objects_scrubbed":5500,"log_size":4870,"log_dups_size":0,"ondisk_log_size":4870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:47.960026+0000","scrub_duration":1945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343801856,"num_objects":5246,"num_object_clones":0,"num_object_copies":15738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11270,"num_write_kb":528512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":271,"reported_epoch":186,"state":"active+undersized","last_fresh":"2024-03-30T13:51:11.771846+0000","last_change":"2024-03-30T13:50:08.511010+0000","last_active":"2024-03-30T13:51:11.771846+0000","last_peered":"2024-03-30T13:51:11.771846+0000","last_clean":"2024-03-30T13:50:05.490535+0000","last_became_active":"2024-03-30T13:50:08.511010+0000","last_became_peered":"2024-03-30T13:50:08.511010+0000","last_unstale":"2024-03-30T13:51:11.771846+0000","last_undegraded":"2024-03-30T13:51:11.771846+0000","last_fullsized":"2024-03-30T13:50:08.503118+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:06.154805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:06.154805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"168'10","reported_seq":105,"reported_epoch":186,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:51:15.462446+0000","last_change":"2024-03-30T13:50:08.526935+0000","last_active":"2024-03-30T13:51:15.462446+0000","last_peered":"2024-03-30T13:51:15.462446+0000","last_clean":"2024-03-30T13:50:05.495636+0000","last_became_active":"2024-03-30T13:50:08.526935+0000","last_became_peered":"2024-03-30T13:50:08.526935+0000","last_unstale":"2024-03-30T13:51:15.462446+0000","last_undegraded":"2024-03-30T13:50:08.509321+0000","last_fullsized":"2024-03-30T13:50:08.509210+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:49:36.397622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:36.397622+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":279,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.783944+0000","last_change":"2024-03-30T13:49:33.573694+0000","last_active":"2024-03-30T13:51:11.783944+0000","last_peered":"2024-03-30T13:51:11.783944+0000","last_clean":"2024-03-30T13:51:11.783944+0000","last_became_active":"2024-03-30T13:42:45.309265+0000","last_became_peered":"2024-03-30T13:42:45.309265+0000","last_unstale":"2024-03-30T13:51:11.783944+0000","last_undegraded":"2024-03-30T13:51:11.783944+0000","last_fullsized":"2024-03-30T13:51:11.783944+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:33.573581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:49:33.573581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":299,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-30T13:51:11.772449+0000","last_change":"2024-03-30T13:50:13.222840+0000","last_active":"2024-03-30T13:51:11.772449+0000","last_peered":"2024-03-30T13:51:11.772449+0000","last_clean":"2024-03-30T13:51:11.772449+0000","last_became_active":"2024-03-30T13:42:47.313026+0000","last_became_peered":"2024-03-30T13:42:47.313026+0000","last_unstale":"2024-03-30T13:51:11.772449+0000","last_undegraded":"2024-03-30T13:51:11.772449+0000","last_fullsized":"2024-03-30T13:51:11.772449+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-30T13:50:13.219342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:50:13.219342+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:51:28.600898+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5339676696,"num_objects":81478,"num_object_clones":0,"num_object_copies":244434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30477,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81478,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179226,"num_write_kb":8342465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16105086976,"data_stored":16105078832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85816,"ondisk_log_size":85816,"up":42,"acting":42,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":511,"num_read_kb":506,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":88,"acting":88,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":447,"ondisk_log_size":447,"up":88,"acting":88,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":87,"acting":87,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":84,"acting":84,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103277,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1899020,"kb_used_data":1638788,"kb_used_omap":1,"kb_used_meta":260222,"kb_avail":91846132,"statfs":{"total":95995035648,"available":94050439168,"internally_reserved":0,"allocated":1678118912,"data_stored":1677616757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":266467786},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397617,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1861172,"kb_used_data":1635564,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":91883980,"statfs":{"total":95995035648,"available":94089195520,"internally_reserved":0,"allocated":1674817536,"data_stored":1674338881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":230947270},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574987,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2583848,"kb_used_data":2300000,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":91161304,"statfs":{"total":95995035648,"available":93349175296,"internally_reserved":0,"allocated":2355200000,"data_stored":2354669503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738512,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2267788,"kb_used_data":1947844,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":91477364,"statfs":{"total":95995035648,"available":93672820736,"internally_reserved":0,"allocated":1994592256,"data_stored":1994090681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411446,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2213288,"kb_used_data":2015520,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":91531864,"statfs":{"total":95995035648,"available":93728628736,"internally_reserved":0,"allocated":2063892480,"data_stored":2063509110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182529,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2244368,"kb_used_data":1956360,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":91500784,"statfs":{"total":95995035648,"available":93696802816,"internally_reserved":0,"allocated":2003312640,"data_stored":2002805647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":294910407},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,1,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953595,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1912464,"kb_used_data":1660488,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":91832688,"statfs":{"total":95995035648,"available":94036672512,"internally_reserved":0,"allocated":1700339712,"data_stored":1699832844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691958,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2885504,"kb_used_data":2647480,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":90859648,"statfs":{"total":95995035648,"available":93040279552,"internally_reserved":0,"allocated":2711019520,"data_stored":2710529753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1991507968,"data_stored":1991507968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2331967488,"data_stored":2331967488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2068250624,"data_stored":2068250624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2000224256,"data_stored":2000224256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1689583616,"data_stored":1689583616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2691960832,"data_stored":2691956760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1661206528,"data_stored":1661206528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1670385664,"data_stored":1670381592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:51:31.876 INFO:tasks.thrashosds.thrasher:Setting 8.3 to [7, 6, 3, 2, 4, 1] 2024-03-30T13:51:31.876 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '8.3', '7', '6', '3', '2', '4', '1'] 2024-03-30T13:51:31.876 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pg-upmap-items 8.3 7 6 3 2 4 1 2024-03-30T13:51:31.933 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:31.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:31.969 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:31.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:32.070 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:32.106 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:32.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:32.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:32.136 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T13:51:32.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:31 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:51:32.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:31 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:51:32.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:31 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:51:32.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:31 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:51:32.207 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:32.220 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:31 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:51:32.220 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:31 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:51:32.244 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:32.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:32.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:32.346 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:32.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:32.420 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:32.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:32.521 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:32.560 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:32.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:32.661 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:32.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:32.699 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:32.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:32.800 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:32.841 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:32.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:32.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:32.932 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:51:32.943 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:32.974 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:32.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:32.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:33.034 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:32 smithi043 ceph-mon[29963]: pgmap v351: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s; 28439/228216 objects degraded (12.461%) 2024-03-30T13:51:33.034 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:32 smithi043 ceph-mon[29963]: osdmap e188: 8 total, 7 up, 8 in 2024-03-30T13:51:33.034 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:32 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:51:33.034 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:32 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:51:33.034 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:32 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 28439/228216 objects degraded (12.461%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:33.034 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:51:33.035 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:32 smithi043 ceph-mon[42099]: pgmap v351: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s; 28439/228216 objects degraded (12.461%) 2024-03-30T13:51:33.035 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:32 smithi043 ceph-mon[42099]: osdmap e188: 8 total, 7 up, 8 in 2024-03-30T13:51:33.035 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:32 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:51:33.035 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:32 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:51:33.035 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:32 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 28439/228216 objects degraded (12.461%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:33.070 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:51:33.076 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:33.108 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:33.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:33.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:33.125 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:51:33.166 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:51:33.201 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:51:33.210 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:33.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:32 smithi113 ceph-mon[34341]: pgmap v351: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s; 28439/228216 objects degraded (12.461%) 2024-03-30T13:51:33.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:32 smithi113 ceph-mon[34341]: osdmap e188: 8 total, 7 up, 8 in 2024-03-30T13:51:33.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:32 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:51:33.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:32 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:51:33.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:32 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 28439/228216 objects degraded (12.461%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:33.241 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:51:33.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:33.244 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:33.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:33.279 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:51:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:51:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:51:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:51:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:51:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:51:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:51:33.280 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:51:33.280 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:51:33.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:33.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:33.380 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:33.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:33.481 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:33.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:33.524 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:33.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:33.625 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:33.662 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:33.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:33.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:33.708 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:51:33.708 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:51:33.764 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:33.799 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:33.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:33.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:33.901 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:33.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:33.936 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:33.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:34.037 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:34.073 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:34.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:34.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:34.175 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:34.209 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:34.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:34.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:34.220 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:33 smithi113 ceph-mon[34341]: osdmap e189: 8 total, 7 up, 8 in 2024-03-30T13:51:34.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:33 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2749849593' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:51:34.311 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:34.348 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:34.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:34.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:34.398 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:33 smithi043 ceph-mon[29963]: osdmap e189: 8 total, 7 up, 8 in 2024-03-30T13:51:34.398 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:33 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2749849593' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:51:34.398 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:33 smithi043 ceph-mon[42099]: osdmap e189: 8 total, 7 up, 8 in 2024-03-30T13:51:34.398 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:33 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2749849593' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:51:34.450 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:34.487 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:34.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:34.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:34.589 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:34.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:34.625 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:34.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:34.726 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:34.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:34.757 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:34.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:34.858 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:34.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:34.892 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:34.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:34.993 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:35.025 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:35.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:35.127 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:35.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:34 smithi043 ceph-mon[29963]: pgmap v354: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 934 op/s; 28439/228216 objects degraded (12.461%) 2024-03-30T13:51:35.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:34 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:51:35.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:34 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:51:35.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:34 smithi043 ceph-mon[29963]: osdmap e190: 8 total, 7 up, 8 in 2024-03-30T13:51:35.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:34 smithi043 ceph-mon[42099]: pgmap v354: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 934 op/s; 28439/228216 objects degraded (12.461%) 2024-03-30T13:51:35.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:34 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:51:35.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:34 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:51:35.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:34 smithi043 ceph-mon[42099]: osdmap e190: 8 total, 7 up, 8 in 2024-03-30T13:51:35.160 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:35.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:35.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:35.220 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:34 smithi113 ceph-mon[34341]: pgmap v354: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 934 op/s; 28439/228216 objects degraded (12.461%) 2024-03-30T13:51:35.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:34 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:51:35.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:34 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:51:35.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:34 smithi113 ceph-mon[34341]: osdmap e190: 8 total, 7 up, 8 in 2024-03-30T13:51:35.262 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:35.295 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:35.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:35.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:35.397 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:35.435 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:35.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:35.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:35.537 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:35.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:35.576 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:35.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:35.677 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:35.711 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:35.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:35.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:35.812 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:35.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:35.850 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:35.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:35.951 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:35.988 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:35.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:35.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:36.090 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:36.119 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:36.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:36.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:36.221 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:36.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:36.259 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:36.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:36.360 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:36.374 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:51:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:51:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:51:36.397 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:36.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:36.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:36.420 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:51:36.421 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:51:36.499 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:36.538 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:36.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:36.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:36.640 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:36.673 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:36.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:36.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:36.775 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:36.807 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:36.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:36.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:36.840 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:51:36.882 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-03-30T13:51:36.908 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:36.966 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:36.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:36.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:37.067 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:37.102 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:37.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:37.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:37.132 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:36 smithi043 ceph-mon[29963]: pgmap v356: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 993 op/s; 26189/210339 objects degraded (12.451%) 2024-03-30T13:51:37.133 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:36 smithi043 ceph-mon[42099]: pgmap v356: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 993 op/s; 26189/210339 objects degraded (12.451%) 2024-03-30T13:51:37.204 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:37.220 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:36 smithi113 ceph-mon[34341]: pgmap v356: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 993 op/s; 26189/210339 objects degraded (12.451%) 2024-03-30T13:51:37.237 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:37.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:37.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:37.341 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:37.376 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:37.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:37.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:37.477 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:37.513 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:37.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:37.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:37.615 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:37.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:37.656 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:37.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:37.758 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:37.794 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:37.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:37.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:37.896 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:37.933 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:37.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:37.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:37.969 INFO:teuthology.orchestra.run.smithi043.stderr:set 8.3 pg_upmap_items mapping to [7->6,3->2,4->1] 2024-03-30T13:51:38.035 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:38.071 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:38.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:38.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:38.173 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:38.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:38.208 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:38.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:38.220 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:37 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 26189/210339 objects degraded (12.451%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:38.220 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:37 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/4292384699' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.3", "id": [7, 6, 3, 2, 4, 1]}]: dispatch 2024-03-30T13:51:38.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:37 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.3", "id": [7, 6, 3, 2, 4, 1]}]: dispatch 2024-03-30T13:51:38.309 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:38.344 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:38.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:38.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:38.355 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:37 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 26189/210339 objects degraded (12.451%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:38.355 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:37 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4292384699' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.3", "id": [7, 6, 3, 2, 4, 1]}]: dispatch 2024-03-30T13:51:38.355 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:37 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.3", "id": [7, 6, 3, 2, 4, 1]}]: dispatch 2024-03-30T13:51:38.357 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:37 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 26189/210339 objects degraded (12.451%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:38.357 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:37 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/4292384699' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.3", "id": [7, 6, 3, 2, 4, 1]}]: dispatch 2024-03-30T13:51:38.357 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:37 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.3", "id": [7, 6, 3, 2, 4, 1]}]: dispatch 2024-03-30T13:51:38.422 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:51:38.446 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:38.515 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:38.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:38.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:38.617 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:38.646 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:38.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:38.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:38.748 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:38.780 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:38.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:38.882 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:38.913 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:38.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:38.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:38.997 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:38 smithi043 ceph-mon[29963]: pgmap v357: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 993 op/s; 26189/210339 objects degraded (12.451%) 2024-03-30T13:51:38.997 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:38 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.3", "id": [7, 6, 3, 2, 4, 1]}]': finished 2024-03-30T13:51:38.997 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:38 smithi043 ceph-mon[29963]: osdmap e191: 8 total, 7 up, 8 in 2024-03-30T13:51:38.997 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:38 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:51:38.997 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:38 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:51:38.997 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:38 smithi043 ceph-mon[42099]: pgmap v357: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 993 op/s; 26189/210339 objects degraded (12.451%) 2024-03-30T13:51:38.997 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:38 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.3", "id": [7, 6, 3, 2, 4, 1]}]': finished 2024-03-30T13:51:38.997 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:38 smithi043 ceph-mon[42099]: osdmap e191: 8 total, 7 up, 8 in 2024-03-30T13:51:38.997 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:38 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:51:38.998 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:38 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:51:39.015 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:39.052 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:39.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:39.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:39.154 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:39.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:39.187 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:39.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:39.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:38 smithi113 ceph-mon[34341]: pgmap v357: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 993 op/s; 26189/210339 objects degraded (12.451%) 2024-03-30T13:51:39.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:38 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.3", "id": [7, 6, 3, 2, 4, 1]}]': finished 2024-03-30T13:51:39.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:38 smithi113 ceph-mon[34341]: osdmap e191: 8 total, 7 up, 8 in 2024-03-30T13:51:39.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:38 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:51:39.221 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:38 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:51:39.221 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:51:38 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:51:39.289 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:39.329 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:39.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:39.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:39.431 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:39.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:39.470 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:39.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:39.571 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:39.582 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:51:39.601 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:39.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:39.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:39.661 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:51:39.703 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:39.736 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:39.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:39.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:39.838 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:39.868 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:39.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:39.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:39.970 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:40.001 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:40.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:40.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:40.103 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:40.136 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:40.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:40.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:40.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:39 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:51:40.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:39 smithi043 ceph-mon[29963]: osdmap e192: 8 total, 7 up, 8 in 2024-03-30T13:51:40.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:39 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:51:40.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:39 smithi043 ceph-mon[42099]: osdmap e192: 8 total, 7 up, 8 in 2024-03-30T13:51:40.238 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:40.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:40.282 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:40.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:40.383 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:40.419 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:40.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:40.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:40.470 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:39 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:51:40.471 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:39 smithi113 ceph-mon[34341]: osdmap e192: 8 total, 7 up, 8 in 2024-03-30T13:51:40.520 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:40.554 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:40.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:40.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:40.656 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:40.690 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:40.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:40.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:40.792 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:40.825 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:40.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:40.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:40.927 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:40.959 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:40.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:40.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-03-30T13:51:41.061 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:41.091 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:41.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:41.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:41.193 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:41.216 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:41.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:41.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:41.318 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:41.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:41.352 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:41.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:41.397 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:40 smithi043 ceph-mon[29963]: pgmap v360: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.97k op/s; 23981/192711 objects degraded (12.444%) 2024-03-30T13:51:41.398 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:40 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1752029670' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:51:41.398 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:40 smithi043 ceph-mon[29963]: 8.b deep-scrub ok 2024-03-30T13:51:41.398 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:40 smithi043 ceph-mon[42099]: pgmap v360: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.97k op/s; 23981/192711 objects degraded (12.444%) 2024-03-30T13:51:41.398 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:40 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1752029670' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:51:41.398 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:40 smithi043 ceph-mon[42099]: 8.b deep-scrub ok 2024-03-30T13:51:41.453 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:41.470 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:40 smithi113 ceph-mon[34341]: pgmap v360: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.97k op/s; 23981/192711 objects degraded (12.444%) 2024-03-30T13:51:41.470 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:40 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1752029670' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:51:41.471 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:40 smithi113 ceph-mon[34341]: 8.b deep-scrub ok 2024-03-30T13:51:41.486 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:41.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:41.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:41.588 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:41.625 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:41.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:41.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:41.727 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:41.750 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:41.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:41.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:41.852 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:41.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:41.881 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:41.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:41.982 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:42.022 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:42.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:42.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:42.123 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:42.147 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:42.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:42.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:42.249 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:42.284 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:42.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:42.386 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:42.398 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:41 smithi043 ceph-mon[29963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:51:42.398 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:41 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1752029670' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:51:42.398 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:41 smithi043 ceph-mon[29963]: osdmap e193: 8 total, 7 up, 8 in 2024-03-30T13:51:42.398 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:41 smithi043 ceph-mon[42099]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:51:42.398 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:41 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1752029670' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:51:42.398 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:41 smithi043 ceph-mon[42099]: osdmap e193: 8 total, 7 up, 8 in 2024-03-30T13:51:42.423 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:42.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:42.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:42.470 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:41 smithi113 ceph-mon[34341]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:51:42.470 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:41 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1752029670' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:51:42.470 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:41 smithi113 ceph-mon[34341]: osdmap e193: 8 total, 7 up, 8 in 2024-03-30T13:51:42.525 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:42.561 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:42.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:42.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:42.662 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:42.700 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:42.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:42.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:42.801 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:42.839 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:42.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:42.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:42.941 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:42.964 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:42.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:42.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:43.066 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:43.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:43.101 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:43.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:43.202 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:43.237 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:43.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:43.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:43.339 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:43.373 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:43.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:43.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:43.398 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:42 smithi043 ceph-mon[29963]: pgmap v362: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 979 op/s; 23981/192711 objects degraded (12.444%) 2024-03-30T13:51:43.398 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:42 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 23981/192711 objects degraded (12.444%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:43.398 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:42 smithi043 ceph-mon[29963]: osdmap e194: 8 total, 7 up, 8 in 2024-03-30T13:51:43.398 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:42 smithi043 ceph-mon[42099]: pgmap v362: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 979 op/s; 23981/192711 objects degraded (12.444%) 2024-03-30T13:51:43.398 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:42 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 23981/192711 objects degraded (12.444%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:43.398 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:42 smithi043 ceph-mon[42099]: osdmap e194: 8 total, 7 up, 8 in 2024-03-30T13:51:43.470 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:42 smithi113 ceph-mon[34341]: pgmap v362: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 979 op/s; 23981/192711 objects degraded (12.444%) 2024-03-30T13:51:43.471 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:42 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 23981/192711 objects degraded (12.444%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:43.471 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:42 smithi113 ceph-mon[34341]: osdmap e194: 8 total, 7 up, 8 in 2024-03-30T13:51:43.475 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:43.498 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:43.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:43.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:43.600 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:43.633 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:43.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:43.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:43.735 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:43.769 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:43.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:43.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:43.870 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:43.906 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:43.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:43.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:44.008 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:44.045 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:44.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:44.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:44.146 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:44.178 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:44.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:44.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:44.279 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:44.311 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:44.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:44.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:44.320 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-30T13:51:44.321 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:51:44.321 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-03-30T13:51:44.321 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-30T13:51:44.321 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-30T13:51:44.321 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-03-30T13:51:44.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:43 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:51:44.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:43 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:51:44.413 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:44.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:44.452 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:44.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:44.470 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:43 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:51:44.553 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:44.585 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:44.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:44.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:44.688 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:44.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:44.718 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:44.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:44.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:44.854 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:44.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:44.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:44.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:44.989 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:44.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:44.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:45.091 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:45.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:45.118 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:45.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:45.219 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:45.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:45.257 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:45.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:45.358 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:45.381 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:45.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:45.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:45.397 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:44 smithi043 ceph-mon[29963]: pgmap v364: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 930 B/s rd, 0 B/s wr, 1.07k op/s; 23981/192711 objects degraded (12.444%) 2024-03-30T13:51:45.397 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:44 smithi043 ceph-mon[42099]: pgmap v364: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 930 B/s rd, 0 B/s wr, 1.07k op/s; 23981/192711 objects degraded (12.444%) 2024-03-30T13:51:45.470 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:44 smithi113 ceph-mon[34341]: pgmap v364: 148 pgs: 1 active+clean+scrubbing+deep, 18 active+undersized+degraded, 26 active+undersized, 103 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 930 B/s rd, 0 B/s wr, 1.07k op/s; 23981/192711 objects degraded (12.444%) 2024-03-30T13:51:45.483 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:45.518 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:45.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:45.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:45.620 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:45.659 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:45.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:45.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:45.761 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:45.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:45.795 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:45.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:45.899 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:45.922 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:45.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:45.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:46.024 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:46.059 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:46.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:46.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:46.161 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:46.185 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:46.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:46.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:46.286 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:46.319 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:46.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:46.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:46.397 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:46 smithi043 ceph-mon[29963]: pgmap v365: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 788 B/s rd, 0 B/s wr, 1.26k op/s; 21740/174927 objects degraded (12.428%) 2024-03-30T13:51:46.398 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:51:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:51:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:51:46.398 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:46 smithi043 ceph-mon[42099]: pgmap v365: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 788 B/s rd, 0 B/s wr, 1.26k op/s; 21740/174927 objects degraded (12.428%) 2024-03-30T13:51:46.420 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:46.446 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:46.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:46.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:46.470 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:46 smithi113 ceph-mon[34341]: pgmap v365: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 788 B/s rd, 0 B/s wr, 1.26k op/s; 21740/174927 objects degraded (12.428%) 2024-03-30T13:51:46.548 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:46.578 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:46.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:46.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:46.680 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:46.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:46.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:46.814 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:46.840 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:46.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:46.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:46.942 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:46.977 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:46.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:46.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:47.079 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:47.102 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:47.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:47.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:47.204 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:47.236 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:47.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:47.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:47.338 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:47.362 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:47.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:47.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:47.463 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:47.499 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:47.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:47.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:47.600 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:47.635 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:47.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:47.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:47.736 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:47.760 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:47.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:47.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:47.862 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:47.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:47 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 21740/174927 objects degraded (12.428%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:47.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:47 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 21740/174927 objects degraded (12.428%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:47.898 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:47.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:47.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:47.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:47 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 21740/174927 objects degraded (12.428%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:48.000 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:48.024 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:48.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:48.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:48.126 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:48.163 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:48.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:48.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:48.264 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:48.300 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:48.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:48.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:48.401 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:48.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:48.439 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:48.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:48.540 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:48.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:48.580 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:48.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:48.681 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:48.712 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:48.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:48.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:48.787 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:48 smithi113 ceph-mon[34341]: pgmap v366: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 741 op/s; 21740/174927 objects degraded (12.428%) 2024-03-30T13:51:48.814 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:48.849 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:48.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:48.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:48.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:48 smithi043 ceph-mon[29963]: pgmap v366: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 741 op/s; 21740/174927 objects degraded (12.428%) 2024-03-30T13:51:48.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:48 smithi043 ceph-mon[42099]: pgmap v366: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 741 op/s; 21740/174927 objects degraded (12.428%) 2024-03-30T13:51:48.951 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:48.985 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:48.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:48.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:49.087 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:49.125 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:49.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:49.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:49.220 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:51:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:51:49.227 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:49.270 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:49.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:49.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:49.372 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:49.411 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:49.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:49.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:49.513 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:49.549 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:49.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:49.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:49.651 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:49.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:49.685 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:49.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:49.786 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:49.813 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:49 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:51:49.814 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:49 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:51:49.819 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:49.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:49.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:49.921 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:49.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:49.956 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:49.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:49.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:49 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:51:50.057 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:50.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:50.094 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:50.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:50.195 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:50.229 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:50.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:50.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:50.331 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:50.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:50.367 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:50.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:50.468 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:50.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:50.507 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:50.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:50.608 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:50.643 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:50.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:50.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:50.745 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:50.770 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:50 smithi043 ceph-mon[29963]: pgmap v367: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 19604/157506 objects degraded (12.447%) 2024-03-30T13:51:50.771 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:50 smithi043 ceph-mon[42099]: pgmap v367: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 19604/157506 objects degraded (12.447%) 2024-03-30T13:51:50.778 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:50.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:50.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:50.879 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:50.912 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:50.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:50.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:50.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:50 smithi113 ceph-mon[34341]: pgmap v367: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 19604/157506 objects degraded (12.447%) 2024-03-30T13:51:51.014 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:51.046 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:51.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:51.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:51.107 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:51:51.107 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:51:51.148 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:51.181 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:51.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:51.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:51.253 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T13:51:51.283 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:51.337 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:51.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:51.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:51.375 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:51:51.422 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:51:51.439 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:51.466 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:51:51.498 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:51.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:51.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:51.519 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:51:51.563 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:51:51.601 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:51.607 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:51:51.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:51.637 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:51.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:51.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:51 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2383688965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:51:51.642 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:51:51.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:51 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2383688965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:51:51.644 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:51:51.644 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:51:51.644 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:51:51.645 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:51:51.645 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:51:51.645 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:51:51.645 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:51:51.645 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:51:51.737 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:51.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:51.781 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:51.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:51.882 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:51.916 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:51.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:51.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:51.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:51 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2383688965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:51:52.019 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:52.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:52.055 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:52.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:52.155 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:52.199 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:52.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:52.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:52.301 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:52.333 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:52.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:52.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:52.435 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:52.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:52.475 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:52.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:52.576 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:52.608 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:52.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:52.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:52.657 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:51:52.699 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-30T13:51:52.710 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:52.785 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:52.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:52.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:52.801 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:52 smithi043 ceph-mon[29963]: pgmap v368: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 19604/157506 objects degraded (12.447%) 2024-03-30T13:51:52.802 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:52 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 19604/157506 objects degraded (12.447%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:52.802 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:52 smithi043 ceph-mon[42099]: pgmap v368: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 19604/157506 objects degraded (12.447%) 2024-03-30T13:51:52.802 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:52 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 19604/157506 objects degraded (12.447%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:52.888 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:52.924 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:52.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:52.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:52 smithi113 ceph-mon[34341]: pgmap v368: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 19604/157506 objects degraded (12.447%) 2024-03-30T13:51:52.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:52 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 19604/157506 objects degraded (12.447%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:53.026 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:53.063 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:53.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:53.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:53.164 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:53.202 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:53.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:53.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:53.307 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:53.344 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:53.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:53.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:53.446 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:53.482 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:53.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:53.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:53.549 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:51:53.550 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:51:53.583 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:53.621 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:53.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:53.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:53.723 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:53.758 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:53.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:53.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:53.860 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:53.896 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:53.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:53.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:53.998 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:54.028 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:54.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:54.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:54.129 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:54.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:54.164 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:54.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:54.265 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:54.303 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:54.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:54.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:54.405 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:54.439 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:54.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:54.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:54.541 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:54.576 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:54.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:54.612 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:51:54.612 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:51:54.677 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:54.732 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:54.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:54.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:54.795 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:54 smithi043 ceph-mon[29963]: pgmap v369: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 929 B/s rd, 0 B/s wr, 1.06k op/s; 19604/157506 objects degraded (12.447%) 2024-03-30T13:51:54.795 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:54 smithi043 ceph-mon[42099]: pgmap v369: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 929 B/s rd, 0 B/s wr, 1.06k op/s; 19604/157506 objects degraded (12.447%) 2024-03-30T13:51:54.835 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:54.870 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:54.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:54.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:54.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:54 smithi113 ceph-mon[34341]: pgmap v369: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 929 B/s rd, 0 B/s wr, 1.06k op/s; 19604/157506 objects degraded (12.447%) 2024-03-30T13:51:54.972 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:55.004 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:55.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:55.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:55.105 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:55.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:55.141 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:55.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:55.242 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:55.280 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:55.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:55.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:55.382 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:55.419 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:55.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:55.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:55.521 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:55.551 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:51:55.558 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:55.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:55.660 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:55.693 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:55.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:55.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:55.795 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:55.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:55.842 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:55.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:55.943 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:55.976 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:55.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:55.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:56.078 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:56.114 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:56.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:56.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:56.147 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:51:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:51:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:51:56.216 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:56.256 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:56.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:56.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:56.358 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:56.395 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:56.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:56.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:56.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:56.532 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:56.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:56.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:56.634 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:56.647 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:56 smithi043 ceph-mon[42099]: pgmap v370: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s; 16940/136491 objects degraded (12.411%) 2024-03-30T13:51:56.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:56 smithi043 ceph-mon[29963]: pgmap v370: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s; 16940/136491 objects degraded (12.411%) 2024-03-30T13:51:56.669 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:56.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:56.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:56.770 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:56.804 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:56.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:56.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:56.906 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:56.935 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:56.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:56.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:56.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:56 smithi113 ceph-mon[34341]: pgmap v370: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s; 16940/136491 objects degraded (12.411%) 2024-03-30T13:51:57.037 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:57.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:57.072 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:57.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:57.173 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:57.210 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:57.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:57.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:57.312 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:57.345 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:57.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:57.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:57.447 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:57.478 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:57.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:57.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:57.525 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:51:57.526 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-30T13:51:57.526 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:51:57.579 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:57.612 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:57.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:57.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:57.647 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:57 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 16940/136491 objects degraded (12.411%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:57.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:57 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 16940/136491 objects degraded (12.411%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:57.714 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:57.753 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:57.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:57.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:57.854 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:57.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:57.890 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:57.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:57.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:57 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 16940/136491 objects degraded (12.411%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:51:57.992 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:58.026 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:58.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:58.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:58.128 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:58.165 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:58.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:58.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:58.267 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:58.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:58.298 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:58.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:58.400 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:58.434 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:58.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:58.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:58.535 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:58.576 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T13:51:58.577 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:58.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:58.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:51:58.679 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:51:58.712 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:58.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:58.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:51:58.814 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:51:58.851 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:58.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:58.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:51:58.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:58 smithi043 ceph-mon[29963]: pgmap v371: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.07k op/s; 16940/136491 objects degraded (12.411%) 2024-03-30T13:51:58.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:58 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:51:58.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:58 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/10642851' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:51:58.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:58 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:51:58.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:58 smithi043 ceph-mon[42099]: pgmap v371: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.07k op/s; 16940/136491 objects degraded (12.411%) 2024-03-30T13:51:58.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:58 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:51:58.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:58 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/10642851' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:51:58.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:58 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:51:58.954 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:51:58.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:58 smithi113 ceph-mon[34341]: pgmap v371: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.07k op/s; 16940/136491 objects degraded (12.411%) 2024-03-30T13:51:58.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:58 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:51:58.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:58 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/10642851' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:51:58.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:58 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:51:58.971 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:51:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:51:58.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:58.990 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:58.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:51:59.090 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:59.125 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:59.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:59.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:59.227 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:59.263 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:59.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:59.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:59.365 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:51:59.401 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:59.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:59.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:51:59.503 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:51:59.540 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:59.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:59.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:51:59.642 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:59.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:59 smithi043 ceph-mon[29963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:51:59.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:59 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:51:59.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:59 smithi043 ceph-mon[29963]: osdmap e195: 8 total, 7 up, 8 in 2024-03-30T13:51:59.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:59 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:51:59.648 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:59 smithi043 ceph-mon[29963]: 3.b scrub starts 2024-03-30T13:51:59.648 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:51:59 smithi043 ceph-mon[29963]: 3.b scrub ok 2024-03-30T13:51:59.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:59 smithi043 ceph-mon[42099]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:51:59.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:59 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:51:59.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:59 smithi043 ceph-mon[42099]: osdmap e195: 8 total, 7 up, 8 in 2024-03-30T13:51:59.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:59 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:51:59.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:59 smithi043 ceph-mon[42099]: 3.b scrub starts 2024-03-30T13:51:59.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:51:59 smithi043 ceph-mon[42099]: 3.b scrub ok 2024-03-30T13:51:59.678 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:59.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:59.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:59.779 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:51:59.811 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:51:59.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:59.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:51:59.913 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:51:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:51:59.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:59 smithi113 ceph-mon[34341]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:51:59.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:59 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:51:59.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:59 smithi113 ceph-mon[34341]: osdmap e195: 8 total, 7 up, 8 in 2024-03-30T13:51:59.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:59 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:51:59.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:59 smithi113 ceph-mon[34341]: 3.b scrub starts 2024-03-30T13:51:59.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:51:59 smithi113 ceph-mon[34341]: 3.b scrub ok 2024-03-30T13:51:59.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:51:59.985 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:51:59.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:00.005 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:52:00.005 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-03-30T13:52:00.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:00.005 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:224123 8.f 8:fc1a812b:::benchmark_data_smithi043_119832_object93769:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-03-30T13:52:00.005 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:52:00.004771+0000", 2024-03-30T13:52:00.005 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.00017889100000000001, 2024-03-30T13:52:00.005 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00021242599999999999, 2024-03-30T13:52:00.005 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:00.005 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:00.006 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-03-30T13:52:00.006 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:52:00.006 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:52:00.006 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:00.006 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 224123 2024-03-30T13:52:00.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:00.006 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:00.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:00.006 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:00.006 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:00.004771+0000", 2024-03-30T13:52:00.006 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:00.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:00.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:00.007 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:00.007 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:00.004771+0000", 2024-03-30T13:52:00.007 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:00.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:00.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:00.007 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:00.007 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:00.004772+0000", 2024-03-30T13:52:00.007 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.347e-06 2024-03-30T13:52:00.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:00.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:00.007 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:00.007 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:00.004778+0000", 2024-03-30T13:52:00.007 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7309999999999998e-06 2024-03-30T13:52:00.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:00.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:00.008 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:00.008 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:00.004792+0000", 2024-03-30T13:52:00.008 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4110999999999999e-05 2024-03-30T13:52:00.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:00.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:00.008 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:00.008 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:00.004801+0000", 2024-03-30T13:52:00.008 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6000000000000007e-06 2024-03-30T13:52:00.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:00.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:00.008 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:00.008 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:00.004852+0000", 2024-03-30T13:52:00.008 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1390000000000001e-05 2024-03-30T13:52:00.008 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:00.008 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:00.009 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:00.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:00.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:00.009 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.24977.0:224122 8.c e195/147 8:3c71e50e:::benchmark_data_smithi043_119832_object93768:head v 195'13909, mlcod=195'13906)", 2024-03-30T13:52:00.009 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:52:00.004912+0000", 2024-03-30T13:52:00.009 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 3.7837999999999999e-05, 2024-03-30T13:52:00.009 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015546099999999999, 2024-03-30T13:52:00.009 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:00.009 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:00.009 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T13:52:00.009 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:00.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:00.009 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:00.009 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:00.004912+0000", 2024-03-30T13:52:00.010 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:00.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:00.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:00.010 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:00.010 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:00.004912+0000", 2024-03-30T13:52:00.010 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:00.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:00.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:00.010 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:00.010 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:00.004912+0000", 2024-03-30T13:52:00.010 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6300000000000005e-07 2024-03-30T13:52:00.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:00.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:00.011 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:00.011 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:00.004918+0000", 2024-03-30T13:52:00.011 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6520000000000003e-06 2024-03-30T13:52:00.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:00.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:00.011 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:00.011 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:00.004926+0000", 2024-03-30T13:52:00.011 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2139999999999996e-06 2024-03-30T13:52:00.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:00.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:00.011 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:00.011 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:00.004932+0000", 2024-03-30T13:52:00.011 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0070000000000003e-06 2024-03-30T13:52:00.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:00.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:00.012 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:00.012 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:00.004982+0000", 2024-03-30T13:52:00.012 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0114999999999997e-05 2024-03-30T13:52:00.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:00.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:00.012 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:00.012 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:00.005002+0000", 2024-03-30T13:52:00.012 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9667999999999999e-05 2024-03-30T13:52:00.012 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:00.012 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:00.012 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:00.012 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:00.012 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:52:00.012 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 2 2024-03-30T13:52:00.012 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:52:00.086 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:00.121 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:00.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:00.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:00.223 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:00.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:00.269 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:00.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:00.370 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:00.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:00.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:00.515 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:00.552 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:00.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:00.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:00.654 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:00.690 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:00.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:00.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:00.791 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:00.825 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:00.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:00.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:00.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:00 smithi043 ceph-mon[29963]: 4.c scrub starts 2024-03-30T13:52:00.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:00 smithi043 ceph-mon[29963]: 4.c scrub ok 2024-03-30T13:52:00.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:00 smithi043 ceph-mon[29963]: pgmap v373: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s; 14296/115218 objects degraded (12.408%) 2024-03-30T13:52:00.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:00 smithi043 ceph-mon[29963]: 6.1d scrub starts 2024-03-30T13:52:00.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:00 smithi043 ceph-mon[29963]: 6.1d scrub ok 2024-03-30T13:52:00.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:00 smithi043 ceph-mon[42099]: 4.c scrub starts 2024-03-30T13:52:00.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:00 smithi043 ceph-mon[42099]: 4.c scrub ok 2024-03-30T13:52:00.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:00 smithi043 ceph-mon[42099]: pgmap v373: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s; 14296/115218 objects degraded (12.408%) 2024-03-30T13:52:00.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:00 smithi043 ceph-mon[42099]: 6.1d scrub starts 2024-03-30T13:52:00.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:00 smithi043 ceph-mon[42099]: 6.1d scrub ok 2024-03-30T13:52:00.928 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:00.960 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-03-30T13:52:00.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:00.962 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:00.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:00.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:00 smithi113 ceph-mon[34341]: 4.c scrub starts 2024-03-30T13:52:00.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:00 smithi113 ceph-mon[34341]: 4.c scrub ok 2024-03-30T13:52:00.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:00 smithi113 ceph-mon[34341]: pgmap v373: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s; 14296/115218 objects degraded (12.408%) 2024-03-30T13:52:00.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:00 smithi113 ceph-mon[34341]: 6.1d scrub starts 2024-03-30T13:52:00.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:00 smithi113 ceph-mon[34341]: 6.1d scrub ok 2024-03-30T13:52:01.063 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:01.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:01.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:01.207 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:01.246 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:01.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:01.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:01.348 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:01.387 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:01.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:01.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:01.489 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:01.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:01.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:01.627 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:01.664 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:01.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:01.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:01.756 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T13:52:01.766 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:01.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:01.798 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:01.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:01.899 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:01.933 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:01.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:01.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:02.035 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:02.070 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:02.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:02.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:02.172 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:02.204 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:02.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:02.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:02.222 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:02.306 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:02.347 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:02.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:02.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:02.449 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:02.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:02.487 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:02.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:02.586 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:52:02.588 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:02.621 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:52:02.621 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:02 smithi043 ceph-mon[29963]: 4.5 scrub starts 2024-03-30T13:52:02.622 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:02 smithi043 ceph-mon[29963]: 4.5 scrub ok 2024-03-30T13:52:02.622 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:02 smithi043 ceph-mon[29963]: pgmap v374: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s; 14296/115218 objects degraded (12.408%) 2024-03-30T13:52:02.622 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:02 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 14296/115218 objects degraded (12.408%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:02.622 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:02 smithi043 ceph-mon[29963]: osdmap e196: 8 total, 7 up, 8 in 2024-03-30T13:52:02.622 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:02 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3231094708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:52:02.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:02 smithi043 ceph-mon[42099]: 4.5 scrub starts 2024-03-30T13:52:02.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:02 smithi043 ceph-mon[42099]: 4.5 scrub ok 2024-03-30T13:52:02.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:02 smithi043 ceph-mon[42099]: pgmap v374: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s; 14296/115218 objects degraded (12.408%) 2024-03-30T13:52:02.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:02 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 14296/115218 objects degraded (12.408%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:02.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:02 smithi043 ceph-mon[42099]: osdmap e196: 8 total, 7 up, 8 in 2024-03-30T13:52:02.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:02 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3231094708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:52:02.625 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:02.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:02.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:02.656 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:52:02.698 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:52:02.727 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:02.741 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:52:02.760 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:02.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:02.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:02.790 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:52:02.838 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:52:02.838 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:52:02.838 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:52:02.838 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:52:02.838 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:52:02.838 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:52:02.838 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:52:02.838 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:52:02.838 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:52:02.862 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:02.896 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:02.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:02.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:02.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:02 smithi113 ceph-mon[34341]: 4.5 scrub starts 2024-03-30T13:52:02.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:02 smithi113 ceph-mon[34341]: 4.5 scrub ok 2024-03-30T13:52:02.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:02 smithi113 ceph-mon[34341]: pgmap v374: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s; 14296/115218 objects degraded (12.408%) 2024-03-30T13:52:02.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:02 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 14296/115218 objects degraded (12.408%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:02.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:02 smithi113 ceph-mon[34341]: osdmap e196: 8 total, 7 up, 8 in 2024-03-30T13:52:02.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:02 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3231094708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:52:02.998 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:03.032 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:03.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:03.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:03.134 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:03.169 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:03.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:03.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:03.271 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:03.306 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:03.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:03.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:03.408 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:03.443 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:03.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:03.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:03.494 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:03.545 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:03.585 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:03.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:03.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:03.687 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:03.722 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:03.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:03.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:03.824 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:03.858 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:03.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:03.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:03.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:03 smithi043 ceph-mon[29963]: 8.5 scrub starts 2024-03-30T13:52:03.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:03 smithi043 ceph-mon[29963]: 8.5 scrub ok 2024-03-30T13:52:03.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:03 smithi043 ceph-mon[29963]: 4.15 scrub starts 2024-03-30T13:52:03.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:03 smithi043 ceph-mon[29963]: 4.15 scrub ok 2024-03-30T13:52:03.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:03 smithi043 ceph-mon[29963]: 6.4 scrub starts 2024-03-30T13:52:03.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:03 smithi043 ceph-mon[29963]: 6.4 scrub ok 2024-03-30T13:52:03.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:03 smithi043 ceph-mon[42099]: 8.5 scrub starts 2024-03-30T13:52:03.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:03 smithi043 ceph-mon[42099]: 8.5 scrub ok 2024-03-30T13:52:03.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:03 smithi043 ceph-mon[42099]: 4.15 scrub starts 2024-03-30T13:52:03.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:03 smithi043 ceph-mon[42099]: 4.15 scrub ok 2024-03-30T13:52:03.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:03 smithi043 ceph-mon[42099]: 6.4 scrub starts 2024-03-30T13:52:03.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:03 smithi043 ceph-mon[42099]: 6.4 scrub ok 2024-03-30T13:52:03.959 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:03.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:03 smithi113 ceph-mon[34341]: 8.5 scrub starts 2024-03-30T13:52:03.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:03 smithi113 ceph-mon[34341]: 8.5 scrub ok 2024-03-30T13:52:03.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:03 smithi113 ceph-mon[34341]: 4.15 scrub starts 2024-03-30T13:52:03.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:03 smithi113 ceph-mon[34341]: 4.15 scrub ok 2024-03-30T13:52:03.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:03 smithi113 ceph-mon[34341]: 6.4 scrub starts 2024-03-30T13:52:03.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:03 smithi113 ceph-mon[34341]: 6.4 scrub ok 2024-03-30T13:52:03.992 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:03.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:03.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:04.094 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:04.095 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:52:04.096 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:52:04.172 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:04.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:04.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:04.274 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:04.308 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:04.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:04.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:04.410 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:04.433 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-30T13:52:04.433 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:52:04.433 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-30T13:52:04.434 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T13:52:04.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:04.448 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:04.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:04.549 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:04.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:04.580 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:04.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:04.681 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:04.718 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:04.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:04.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:04.819 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:04.854 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:04.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:04.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:04.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:04 smithi043 ceph-mon[29963]: 6.10 scrub starts 2024-03-30T13:52:04.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:04 smithi043 ceph-mon[29963]: 6.11 scrub starts 2024-03-30T13:52:04.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:04 smithi043 ceph-mon[29963]: 6.10 scrub ok 2024-03-30T13:52:04.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:04 smithi043 ceph-mon[29963]: 6.11 scrub ok 2024-03-30T13:52:04.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:04 smithi043 ceph-mon[29963]: pgmap v376: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 886 op/s; 14296/115218 objects degraded (12.408%) 2024-03-30T13:52:04.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:04 smithi043 ceph-mon[29963]: 3.16 scrub starts 2024-03-30T13:52:04.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:04 smithi043 ceph-mon[29963]: 3.16 scrub ok 2024-03-30T13:52:04.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:04 smithi043 ceph-mon[29963]: 8.3 scrub starts 2024-03-30T13:52:04.883 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:04 smithi043 ceph-mon[42099]: 6.10 scrub starts 2024-03-30T13:52:04.883 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:04 smithi043 ceph-mon[42099]: 6.11 scrub starts 2024-03-30T13:52:04.883 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:04 smithi043 ceph-mon[42099]: 6.10 scrub ok 2024-03-30T13:52:04.883 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:04 smithi043 ceph-mon[42099]: 6.11 scrub ok 2024-03-30T13:52:04.883 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:04 smithi043 ceph-mon[42099]: pgmap v376: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 886 op/s; 14296/115218 objects degraded (12.408%) 2024-03-30T13:52:04.883 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:04 smithi043 ceph-mon[42099]: 3.16 scrub starts 2024-03-30T13:52:04.883 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:04 smithi043 ceph-mon[42099]: 3.16 scrub ok 2024-03-30T13:52:04.883 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:04 smithi043 ceph-mon[42099]: 8.3 scrub starts 2024-03-30T13:52:04.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:04.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:04 smithi113 ceph-mon[34341]: 6.10 scrub starts 2024-03-30T13:52:04.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:04 smithi113 ceph-mon[34341]: 6.11 scrub starts 2024-03-30T13:52:04.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:04 smithi113 ceph-mon[34341]: 6.10 scrub ok 2024-03-30T13:52:04.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:04 smithi113 ceph-mon[34341]: 6.11 scrub ok 2024-03-30T13:52:04.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:04 smithi113 ceph-mon[34341]: pgmap v376: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 886 op/s; 14296/115218 objects degraded (12.408%) 2024-03-30T13:52:04.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:04 smithi113 ceph-mon[34341]: 3.16 scrub starts 2024-03-30T13:52:04.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:04 smithi113 ceph-mon[34341]: 3.16 scrub ok 2024-03-30T13:52:04.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:04 smithi113 ceph-mon[34341]: 8.3 scrub starts 2024-03-30T13:52:05.003 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:05.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:05.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:05.104 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:05.143 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:05.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:05.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:05.244 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:05.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:05.281 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:05.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:05.382 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:05.415 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:05.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:05.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:05.517 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:05.549 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:05.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:05.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:05.651 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:05.686 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:05.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:05.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:05.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:05 smithi043 ceph-mon[29963]: 4.10 scrub starts 2024-03-30T13:52:05.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:05 smithi043 ceph-mon[29963]: 4.10 scrub ok 2024-03-30T13:52:05.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:05 smithi043 ceph-mon[29963]: 8.3 scrub ok 2024-03-30T13:52:05.741 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:05 smithi043 ceph-mon[42099]: 4.10 scrub starts 2024-03-30T13:52:05.741 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:05 smithi043 ceph-mon[42099]: 4.10 scrub ok 2024-03-30T13:52:05.742 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:05 smithi043 ceph-mon[42099]: 8.3 scrub ok 2024-03-30T13:52:05.787 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:05.822 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:05.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:05.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:05.924 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:05.958 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:05.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:05.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:05.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:05 smithi113 ceph-mon[34341]: 4.10 scrub starts 2024-03-30T13:52:05.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:05 smithi113 ceph-mon[34341]: 4.10 scrub ok 2024-03-30T13:52:05.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:05 smithi113 ceph-mon[34341]: 8.3 scrub ok 2024-03-30T13:52:06.060 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:06.092 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:06.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:06.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:06.096 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:52:06.194 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:06.225 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:06.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:06.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:06.318 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:52:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:52:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:52:06.327 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:06.329 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:06.359 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:06.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:06.416 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:52:06.416 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:52:06.461 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:06.497 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:06.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:06.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:06.599 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:06.635 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:06.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:06.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:06.736 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:06.774 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:06.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:06.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:06.844 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:06 smithi043 ceph-mon[29963]: pgmap v377: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.9 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s; 11617/94008 objects degraded (12.357%) 2024-03-30T13:52:06.845 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:06 smithi043 ceph-mon[29963]: 4.a scrub starts 2024-03-30T13:52:06.845 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:06 smithi043 ceph-mon[29963]: 4.a scrub ok 2024-03-30T13:52:06.845 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:06 smithi043 ceph-mon[42099]: pgmap v377: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.9 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s; 11617/94008 objects degraded (12.357%) 2024-03-30T13:52:06.845 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:06 smithi043 ceph-mon[42099]: 4.a scrub starts 2024-03-30T13:52:06.845 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:06 smithi043 ceph-mon[42099]: 4.a scrub ok 2024-03-30T13:52:06.877 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:06.915 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:06.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:06.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:06.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:06 smithi113 ceph-mon[34341]: pgmap v377: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.9 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s; 11617/94008 objects degraded (12.357%) 2024-03-30T13:52:06.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:06 smithi113 ceph-mon[34341]: 4.a scrub starts 2024-03-30T13:52:06.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:06 smithi113 ceph-mon[34341]: 4.a scrub ok 2024-03-30T13:52:07.017 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:07.049 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:07.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:07.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:07.151 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:07.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:07.190 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:07.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:07.291 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:07.327 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:07.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:07.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:07.429 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:07.464 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:07.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:07.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:07.565 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:07.606 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:07.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:07.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:07.644 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-03-30T13:52:07.707 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:07.743 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:07.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:07.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:07.845 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:07.880 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:07.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:07.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:07.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:07 smithi043 ceph-mon[29963]: 3.1d scrub starts 2024-03-30T13:52:07.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:07 smithi043 ceph-mon[29963]: 3.1d scrub ok 2024-03-30T13:52:07.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:07 smithi043 ceph-mon[29963]: 5.1f scrub starts 2024-03-30T13:52:07.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:07 smithi043 ceph-mon[29963]: 5.1f scrub ok 2024-03-30T13:52:07.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:07 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 11617/94008 objects degraded (12.357%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:07.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:07 smithi043 ceph-mon[42099]: 3.1d scrub starts 2024-03-30T13:52:07.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:07 smithi043 ceph-mon[42099]: 3.1d scrub ok 2024-03-30T13:52:07.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:07 smithi043 ceph-mon[42099]: 5.1f scrub starts 2024-03-30T13:52:07.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:07 smithi043 ceph-mon[42099]: 5.1f scrub ok 2024-03-30T13:52:07.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:07 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 11617/94008 objects degraded (12.357%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:07.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:07 smithi113 ceph-mon[34341]: 3.1d scrub starts 2024-03-30T13:52:07.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:07 smithi113 ceph-mon[34341]: 3.1d scrub ok 2024-03-30T13:52:07.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:07 smithi113 ceph-mon[34341]: 5.1f scrub starts 2024-03-30T13:52:07.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:07 smithi113 ceph-mon[34341]: 5.1f scrub ok 2024-03-30T13:52:07.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:07 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 11617/94008 objects degraded (12.357%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:07.982 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:08.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:08.019 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:08.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:08.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:08.157 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:08.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:08.258 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:08.288 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:08.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:08.389 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:08.446 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:08.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:08.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:08.548 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:08.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:08.596 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:08.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:08.696 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:08.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:08.735 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:08.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:08.836 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:08.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:08.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:08.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:08 smithi043 ceph-mon[29963]: pgmap v378: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.9 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 574 B/s rd, 0 B/s wr, 1.38k op/s; 11617/94008 objects degraded (12.357%) 2024-03-30T13:52:08.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:08 smithi043 ceph-mon[42099]: pgmap v378: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.9 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 574 B/s rd, 0 B/s wr, 1.38k op/s; 11617/94008 objects degraded (12.357%) 2024-03-30T13:52:08.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:08 smithi113 ceph-mon[34341]: pgmap v378: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.9 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 574 B/s rd, 0 B/s wr, 1.38k op/s; 11617/94008 objects degraded (12.357%) 2024-03-30T13:52:08.971 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:52:08 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:52:08.971 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:09.006 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:09.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:09.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:09.108 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:09.141 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:09.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:09.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:09.242 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:09.244 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:09.297 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:09.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:09.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:09.399 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:09.435 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:09.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:09.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:09.537 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:09.568 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:09.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:09.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-03-30T13:52:09.670 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:09.707 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:09.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:09.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:09.809 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:09.843 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:09.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:09.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:09.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[29963]: 6.1e scrub starts 2024-03-30T13:52:09.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[29963]: 6.1e scrub ok 2024-03-30T13:52:09.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[29963]: 3.1 scrub starts 2024-03-30T13:52:09.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[29963]: 3.1 scrub ok 2024-03-30T13:52:09.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[29963]: 6.6 scrub starts 2024-03-30T13:52:09.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[29963]: 6.6 scrub ok 2024-03-30T13:52:09.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1716993899' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:52:09.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:52:09.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[29963]: 5.16 scrub starts 2024-03-30T13:52:09.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[29963]: 5.16 scrub ok 2024-03-30T13:52:09.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[29963]: 8.7 scrub starts 2024-03-30T13:52:09.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[42099]: 6.1e scrub starts 2024-03-30T13:52:09.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[42099]: 6.1e scrub ok 2024-03-30T13:52:09.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[42099]: 3.1 scrub starts 2024-03-30T13:52:09.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[42099]: 3.1 scrub ok 2024-03-30T13:52:09.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[42099]: 6.6 scrub starts 2024-03-30T13:52:09.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[42099]: 6.6 scrub ok 2024-03-30T13:52:09.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1716993899' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:52:09.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:52:09.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[42099]: 5.16 scrub starts 2024-03-30T13:52:09.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[42099]: 5.16 scrub ok 2024-03-30T13:52:09.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:09 smithi043 ceph-mon[42099]: 8.7 scrub starts 2024-03-30T13:52:09.945 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:09.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:09 smithi113 ceph-mon[34341]: 6.1e scrub starts 2024-03-30T13:52:09.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:09 smithi113 ceph-mon[34341]: 6.1e scrub ok 2024-03-30T13:52:09.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:09 smithi113 ceph-mon[34341]: 3.1 scrub starts 2024-03-30T13:52:09.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:09 smithi113 ceph-mon[34341]: 3.1 scrub ok 2024-03-30T13:52:09.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:09 smithi113 ceph-mon[34341]: 6.6 scrub starts 2024-03-30T13:52:09.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:09 smithi113 ceph-mon[34341]: 6.6 scrub ok 2024-03-30T13:52:09.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:09 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1716993899' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:52:09.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:09 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:52:09.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:09 smithi113 ceph-mon[34341]: 5.16 scrub starts 2024-03-30T13:52:09.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:09 smithi113 ceph-mon[34341]: 5.16 scrub ok 2024-03-30T13:52:09.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:09 smithi113 ceph-mon[34341]: 8.7 scrub starts 2024-03-30T13:52:09.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:09.979 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:09.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:10.080 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:10.117 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:10.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:10.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:10.218 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:10.256 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:10.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:10.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:10.358 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:10.397 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:10.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:10.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:10.498 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:10.534 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:10.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:10.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:10.636 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:10.670 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:10.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:10.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:10.772 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:10.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:10.810 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:10.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:10.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:10 smithi043 ceph-mon[42099]: 3.12 scrub starts 2024-03-30T13:52:10.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:10 smithi043 ceph-mon[42099]: 3.12 scrub ok 2024-03-30T13:52:10.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:10 smithi043 ceph-mon[42099]: pgmap v379: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.40k op/s; 9058/73122 objects degraded (12.388%) 2024-03-30T13:52:10.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:10 smithi043 ceph-mon[42099]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:52:10.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:10 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1716993899' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:52:10.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:10 smithi043 ceph-mon[42099]: osdmap e197: 8 total, 7 up, 8 in 2024-03-30T13:52:10.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:10 smithi043 ceph-mon[42099]: 3.f scrub starts 2024-03-30T13:52:10.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:10 smithi043 ceph-mon[42099]: 3.f scrub ok 2024-03-30T13:52:10.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:10 smithi043 ceph-mon[42099]: 8.7 scrub ok 2024-03-30T13:52:10.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:10 smithi043 ceph-mon[29963]: 3.12 scrub starts 2024-03-30T13:52:10.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:10 smithi043 ceph-mon[29963]: 3.12 scrub ok 2024-03-30T13:52:10.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:10 smithi043 ceph-mon[29963]: pgmap v379: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.40k op/s; 9058/73122 objects degraded (12.388%) 2024-03-30T13:52:10.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:10 smithi043 ceph-mon[29963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:52:10.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:10 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1716993899' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:52:10.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:10 smithi043 ceph-mon[29963]: osdmap e197: 8 total, 7 up, 8 in 2024-03-30T13:52:10.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:10 smithi043 ceph-mon[29963]: 3.f scrub starts 2024-03-30T13:52:10.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:10 smithi043 ceph-mon[29963]: 3.f scrub ok 2024-03-30T13:52:10.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:10 smithi043 ceph-mon[29963]: 8.7 scrub ok 2024-03-30T13:52:10.911 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:10.944 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:10.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:10.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:10.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:10 smithi113 ceph-mon[34341]: 3.12 scrub starts 2024-03-30T13:52:10.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:10 smithi113 ceph-mon[34341]: 3.12 scrub ok 2024-03-30T13:52:10.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:10 smithi113 ceph-mon[34341]: pgmap v379: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.40k op/s; 9058/73122 objects degraded (12.388%) 2024-03-30T13:52:10.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:10 smithi113 ceph-mon[34341]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:52:10.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:10 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1716993899' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:52:10.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:10 smithi113 ceph-mon[34341]: osdmap e197: 8 total, 7 up, 8 in 2024-03-30T13:52:10.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:10 smithi113 ceph-mon[34341]: 3.f scrub starts 2024-03-30T13:52:10.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:10 smithi113 ceph-mon[34341]: 3.f scrub ok 2024-03-30T13:52:10.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:10 smithi113 ceph-mon[34341]: 8.7 scrub ok 2024-03-30T13:52:11.046 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:11.078 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:11.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:11.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:11.179 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:11.211 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:11.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:11.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:11.313 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:11.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:11.346 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:11.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:11.446 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:11.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:11.486 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:11.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:11.587 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:11.622 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:11.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:11.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:11.724 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:11.756 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:11.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:11.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:11.858 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:11.891 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:11.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:11.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:11.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:11 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:52:11.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:11 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:52:11.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:11 smithi043 ceph-mon[29963]: osdmap e198: 8 total, 7 up, 8 in 2024-03-30T13:52:11.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:11 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:52:11.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:11 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:52:11.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:11 smithi043 ceph-mon[42099]: osdmap e198: 8 total, 7 up, 8 in 2024-03-30T13:52:11.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:11 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:52:11.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:11 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:52:11.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:11 smithi113 ceph-mon[34341]: osdmap e198: 8 total, 7 up, 8 in 2024-03-30T13:52:11.993 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:12.026 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:12.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:12.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:12.128 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:12.160 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:12.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:12.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:12.262 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:12.293 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:12.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:12.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:12.395 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:12.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:12.439 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:12.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:12.541 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:12.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:12.575 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:12.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:12.618 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:52:12.626 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-30T13:52:12.676 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:12.711 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:12.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:12.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:12.788 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:12 smithi043 ceph-mon[29963]: pgmap v382: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.75k op/s; 9058/73122 objects degraded (12.388%) 2024-03-30T13:52:12.788 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:12 smithi043 ceph-mon[29963]: 3.15 deep-scrub starts 2024-03-30T13:52:12.788 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:12 smithi043 ceph-mon[29963]: 3.15 deep-scrub ok 2024-03-30T13:52:12.788 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:12 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:52:12.789 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:12 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:52:12.789 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:12 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 9058/73122 objects degraded (12.388%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:12.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:12 smithi043 ceph-mon[42099]: pgmap v382: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.75k op/s; 9058/73122 objects degraded (12.388%) 2024-03-30T13:52:12.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:12 smithi043 ceph-mon[42099]: 3.15 deep-scrub starts 2024-03-30T13:52:12.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:12 smithi043 ceph-mon[42099]: 3.15 deep-scrub ok 2024-03-30T13:52:12.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:12 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:52:12.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:12 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:52:12.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:12 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 9058/73122 objects degraded (12.388%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:12.813 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:12.849 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:12.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:12.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:12.951 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:12.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:12 smithi113 ceph-mon[34341]: pgmap v382: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.75k op/s; 9058/73122 objects degraded (12.388%) 2024-03-30T13:52:12.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:12 smithi113 ceph-mon[34341]: 3.15 deep-scrub starts 2024-03-30T13:52:12.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:12 smithi113 ceph-mon[34341]: 3.15 deep-scrub ok 2024-03-30T13:52:12.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:12 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:52:12.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:12 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:52:12.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:12 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 9058/73122 objects degraded (12.388%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:12.988 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:12.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:12.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:13.090 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:13.125 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:13.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:13.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:13.227 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:13.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:13.263 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:13.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:13.364 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:13.401 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:13.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:13.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:13.503 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:13.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:13.541 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:13.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:13.642 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:14.613 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:13 smithi113 ceph-mon[34341]: 4.9 scrub starts 2024-03-30T13:52:14.613 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:13 smithi113 ceph-mon[34341]: 4.9 scrub ok 2024-03-30T13:52:14.613 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:13 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:52:14.613 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:13 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:52:14.613 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:13 smithi113 ceph-mon[34341]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:52:14.613 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:13 smithi113 ceph-mon[34341]: osdmap e199: 8 total, 7 up, 8 in 2024-03-30T13:52:14.613 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:13 smithi113 ceph-mon[34341]: 5.c scrub starts 2024-03-30T13:52:14.613 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:13 smithi113 ceph-mon[34341]: 5.c scrub ok 2024-03-30T13:52:14.613 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:13 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:52:14.613 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:13 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:52:14.614 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T13:52:14.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:13 smithi043 ceph-mon[29963]: 4.9 scrub starts 2024-03-30T13:52:14.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:13 smithi043 ceph-mon[29963]: 4.9 scrub ok 2024-03-30T13:52:14.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:13 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:52:14.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:13 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:52:14.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:13 smithi043 ceph-mon[29963]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:52:14.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:13 smithi043 ceph-mon[29963]: osdmap e199: 8 total, 7 up, 8 in 2024-03-30T13:52:14.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:13 smithi043 ceph-mon[29963]: 5.c scrub starts 2024-03-30T13:52:14.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:13 smithi043 ceph-mon[29963]: 5.c scrub ok 2024-03-30T13:52:14.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:13 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:52:14.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:13 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:52:14.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:13 smithi043 ceph-mon[42099]: 4.9 scrub starts 2024-03-30T13:52:14.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:13 smithi043 ceph-mon[42099]: 4.9 scrub ok 2024-03-30T13:52:14.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:13 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:52:14.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:13 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:52:14.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:13 smithi043 ceph-mon[42099]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:52:14.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:13 smithi043 ceph-mon[42099]: osdmap e199: 8 total, 7 up, 8 in 2024-03-30T13:52:14.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:13 smithi043 ceph-mon[42099]: 5.c scrub starts 2024-03-30T13:52:14.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:13 smithi043 ceph-mon[42099]: 5.c scrub ok 2024-03-30T13:52:14.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:13 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:52:14.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:13 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:52:14.617 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2024-03-30T13:52:11.475617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1584929142,"num_objects":24374,"num_object_clones":0,"num_object_copies":73122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24374,"num_whiteouts":0,"num_read":1490,"num_read_kb":1336,"num_write":237143,"num_write_kb":8344926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1836860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86396,"ondisk_log_size":86396,"up":400,"acting":400,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8422788,"kb_used_data":6219844,"kb_used_omap":12,"kb_used_meta":2202675,"kb_avail":741538428,"statfs":{"total":767960285184,"available":759335350272,"internally_reserved":0,"allocated":6369120256,"data_stored":6364929085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":2255539760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-919601152,"num_objects":-14032,"num_object_clones":0,"num_object_copies":-42096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-5238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-14032,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":14032,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005470"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":296,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.463387+0000","last_change":"2024-03-30T13:51:05.234138+0000","last_active":"2024-03-30T13:52:02.463387+0000","last_peered":"2024-03-30T13:52:02.463387+0000","last_clean":"2024-03-30T13:52:02.463387+0000","last_became_active":"2024-03-30T13:42:51.329797+0000","last_became_peered":"2024-03-30T13:42:51.329797+0000","last_unstale":"2024-03-30T13:52:02.463387+0000","last_undegraded":"2024-03-30T13:52:02.463387+0000","last_fullsized":"2024-03-30T13:52:02.463387+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:05.234023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:05.234023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:22.013024+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":307,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.474665+0000","last_change":"2024-03-30T13:50:56.296909+0000","last_active":"2024-03-30T13:52:02.474665+0000","last_peered":"2024-03-30T13:52:02.474665+0000","last_clean":"2024-03-30T13:52:02.474665+0000","last_became_active":"2024-03-30T13:42:49.323268+0000","last_became_peered":"2024-03-30T13:42:49.323268+0000","last_unstale":"2024-03-30T13:52:02.474665+0000","last_undegraded":"2024-03-30T13:52:02.474665+0000","last_fullsized":"2024-03-30T13:52:02.474665+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:56.296785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:50:56.296785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:13.943731+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"65'1","reported_seq":307,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:02.461274+0000","last_change":"2024-03-30T13:50:08.513977+0000","last_active":"2024-03-30T13:52:02.461274+0000","last_peered":"2024-03-30T13:52:02.461274+0000","last_clean":"2024-03-30T13:50:05.499990+0000","last_became_active":"2024-03-30T13:50:08.513977+0000","last_became_peered":"2024-03-30T13:50:08.513977+0000","last_unstale":"2024-03-30T13:52:02.461274+0000","last_undegraded":"2024-03-30T13:50:08.505072+0000","last_fullsized":"2024-03-30T13:50:08.504945+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:48:19.930059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:48:19.930059+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":320,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.462665+0000","last_change":"2024-03-30T13:51:06.228575+0000","last_active":"2024-03-30T13:52:02.462665+0000","last_peered":"2024-03-30T13:52:02.462665+0000","last_clean":"2024-03-30T13:52:02.462665+0000","last_became_active":"2024-03-30T13:42:47.319638+0000","last_became_peered":"2024-03-30T13:42:47.319638+0000","last_unstale":"2024-03-30T13:52:02.462665+0000","last_undegraded":"2024-03-30T13:52:02.462665+0000","last_fullsized":"2024-03-30T13:52:02.462665+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:51:06.228451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:06.228451+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:13.739442+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":287,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.469089+0000","last_change":"2024-03-30T13:51:03.698577+0000","last_active":"2024-03-30T13:52:02.469089+0000","last_peered":"2024-03-30T13:52:02.469089+0000","last_clean":"2024-03-30T13:52:02.469089+0000","last_became_active":"2024-03-30T13:42:51.374403+0000","last_became_peered":"2024-03-30T13:42:51.374403+0000","last_unstale":"2024-03-30T13:52:02.469089+0000","last_undegraded":"2024-03-30T13:52:02.469089+0000","last_fullsized":"2024-03-30T13:52:02.469089+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:03.698501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:03.698501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:18.724603+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":300,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.462775+0000","last_change":"2024-03-30T13:51:20.149796+0000","last_active":"2024-03-30T13:52:02.462775+0000","last_peered":"2024-03-30T13:52:02.462775+0000","last_clean":"2024-03-30T13:52:02.462775+0000","last_became_active":"2024-03-30T13:42:49.327124+0000","last_became_peered":"2024-03-30T13:42:49.327124+0000","last_unstale":"2024-03-30T13:52:02.462775+0000","last_undegraded":"2024-03-30T13:52:02.462775+0000","last_fullsized":"2024-03-30T13:52:02.462775+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:20.149684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:20.149684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:20.873892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":303,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.460969+0000","last_change":"2024-03-30T13:50:08.512316+0000","last_active":"2024-03-30T13:52:02.460969+0000","last_peered":"2024-03-30T13:52:02.460969+0000","last_clean":"2024-03-30T13:50:05.504688+0000","last_became_active":"2024-03-30T13:50:08.512316+0000","last_became_peered":"2024-03-30T13:50:08.512316+0000","last_unstale":"2024-03-30T13:52:02.460969+0000","last_undegraded":"2024-03-30T13:52:02.460969+0000","last_fullsized":"2024-03-30T13:50:08.501439+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:20.499768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:20.499768+0000","last_clean_scrub_stamp":"2024-03-30T13:49:20.499768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:03.933880+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"95'22","reported_seq":322,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:02.461624+0000","last_change":"2024-03-30T13:50:08.511099+0000","last_active":"2024-03-30T13:52:02.461624+0000","last_peered":"2024-03-30T13:52:02.461624+0000","last_clean":"2024-03-30T13:50:05.499201+0000","last_became_active":"2024-03-30T13:50:08.511099+0000","last_became_peered":"2024-03-30T13:50:08.511099+0000","last_unstale":"2024-03-30T13:52:02.461624+0000","last_undegraded":"2024-03-30T13:50:08.502733+0000","last_fullsized":"2024-03-30T13:50:08.502606+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'22","last_scrub_stamp":"2024-03-30T13:46:49.339954+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:29.151791+0000","last_clean_scrub_stamp":"2024-03-30T13:46:49.339954+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:03.929484+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":293,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.462294+0000","last_change":"2024-03-30T13:50:57.339042+0000","last_active":"2024-03-30T13:52:02.462294+0000","last_peered":"2024-03-30T13:52:02.462294+0000","last_clean":"2024-03-30T13:52:02.462294+0000","last_became_active":"2024-03-30T13:42:51.331196+0000","last_became_peered":"2024-03-30T13:42:51.331196+0000","last_unstale":"2024-03-30T13:52:02.462294+0000","last_undegraded":"2024-03-30T13:52:02.462294+0000","last_fullsized":"2024-03-30T13:52:02.462294+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:57.338783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:30.766332+0000","last_clean_scrub_stamp":"2024-03-30T13:50:57.338783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:21.578808+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":306,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.469838+0000","last_change":"2024-03-30T13:51:03.925694+0000","last_active":"2024-03-30T13:52:02.469838+0000","last_peered":"2024-03-30T13:52:02.469838+0000","last_clean":"2024-03-30T13:52:02.469838+0000","last_became_active":"2024-03-30T13:42:49.321811+0000","last_became_peered":"2024-03-30T13:42:49.321811+0000","last_unstale":"2024-03-30T13:52:02.469838+0000","last_undegraded":"2024-03-30T13:52:02.469838+0000","last_fullsized":"2024-03-30T13:52:02.469838+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:03.925590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:03.925590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:05.213109+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":314,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:07.129217+0000","last_change":"2024-03-30T13:52:07.129258+0000","last_active":"2024-03-30T13:52:07.129217+0000","last_peered":"2024-03-30T13:52:07.129217+0000","last_clean":"2024-03-30T13:52:07.129217+0000","last_became_active":"2024-03-30T13:42:45.319827+0000","last_became_peered":"2024-03-30T13:42:45.319827+0000","last_unstale":"2024-03-30T13:52:07.129217+0000","last_undegraded":"2024-03-30T13:52:07.129217+0000","last_fullsized":"2024-03-30T13:52:07.129217+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:07.129142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:37.513596+0000","last_clean_scrub_stamp":"2024-03-30T13:52:07.129142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:13.605042+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":328,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.463641+0000","last_change":"2024-03-30T13:51:16.153633+0000","last_active":"2024-03-30T13:52:02.463641+0000","last_peered":"2024-03-30T13:52:02.463641+0000","last_clean":"2024-03-30T13:52:02.463641+0000","last_became_active":"2024-03-30T13:42:47.310289+0000","last_became_peered":"2024-03-30T13:42:47.310289+0000","last_unstale":"2024-03-30T13:52:02.463641+0000","last_undegraded":"2024-03-30T13:52:02.463641+0000","last_fullsized":"2024-03-30T13:52:02.463641+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:51:16.153478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:16.153478+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:26.806078+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":302,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.470158+0000","last_change":"2024-03-30T13:51:10.988992+0000","last_active":"2024-03-30T13:52:02.470158+0000","last_peered":"2024-03-30T13:52:02.470158+0000","last_clean":"2024-03-30T13:52:02.470158+0000","last_became_active":"2024-03-30T13:42:51.325748+0000","last_became_peered":"2024-03-30T13:42:51.325748+0000","last_unstale":"2024-03-30T13:52:02.470158+0000","last_undegraded":"2024-03-30T13:52:02.470158+0000","last_fullsized":"2024-03-30T13:52:02.470158+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:10.988839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:10.988839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:27.596535+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":297,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.481134+0000","last_change":"2024-03-30T13:51:06.204404+0000","last_active":"2024-03-30T13:52:02.481134+0000","last_peered":"2024-03-30T13:52:02.481134+0000","last_clean":"2024-03-30T13:52:02.481134+0000","last_became_active":"2024-03-30T13:42:49.326027+0000","last_became_peered":"2024-03-30T13:42:49.326027+0000","last_unstale":"2024-03-30T13:52:02.481134+0000","last_undegraded":"2024-03-30T13:52:02.481134+0000","last_fullsized":"2024-03-30T13:52:02.481134+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:06.204282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:41.633284+0000","last_clean_scrub_stamp":"2024-03-30T13:51:06.204282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:21.468259+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"186'2","reported_seq":321,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.473519+0000","last_change":"2024-03-30T13:51:15.932194+0000","last_active":"2024-03-30T13:52:02.473519+0000","last_peered":"2024-03-30T13:52:02.473519+0000","last_clean":"2024-03-30T13:52:02.473519+0000","last_became_active":"2024-03-30T13:42:45.319975+0000","last_became_peered":"2024-03-30T13:42:45.319975+0000","last_unstale":"2024-03-30T13:52:02.473519+0000","last_undegraded":"2024-03-30T13:52:02.473519+0000","last_fullsized":"2024-03-30T13:52:02.473519+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-30T13:51:15.932076+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-30T13:51:15.932076+0000","last_clean_scrub_stamp":"2024-03-30T13:51:15.932076+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:34.376891+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"88'7","reported_seq":324,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.466010+0000","last_change":"2024-03-30T13:51:12.231695+0000","last_active":"2024-03-30T13:52:02.466010+0000","last_peered":"2024-03-30T13:52:02.466010+0000","last_clean":"2024-03-30T13:52:02.466010+0000","last_became_active":"2024-03-30T13:42:47.317352+0000","last_became_peered":"2024-03-30T13:42:47.317352+0000","last_unstale":"2024-03-30T13:52:02.466010+0000","last_undegraded":"2024-03-30T13:52:02.466010+0000","last_fullsized":"2024-03-30T13:52:02.466010+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-30T13:51:12.231574+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-30T13:43:50.960980+0000","last_clean_scrub_stamp":"2024-03-30T13:51:12.231574+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:13.364083+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":305,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:07.996808+0000","last_change":"2024-03-30T13:52:07.996845+0000","last_active":"2024-03-30T13:52:07.996808+0000","last_peered":"2024-03-30T13:52:07.996808+0000","last_clean":"2024-03-30T13:52:07.996808+0000","last_became_active":"2024-03-30T13:42:51.372011+0000","last_became_peered":"2024-03-30T13:42:51.372011+0000","last_unstale":"2024-03-30T13:52:07.996808+0000","last_undegraded":"2024-03-30T13:52:07.996808+0000","last_fullsized":"2024-03-30T13:52:07.996808+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:07.996724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:07.996724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:16.479878+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":307,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.475161+0000","last_change":"2024-03-30T13:51:05.319693+0000","last_active":"2024-03-30T13:52:02.475161+0000","last_peered":"2024-03-30T13:52:02.475161+0000","last_clean":"2024-03-30T13:52:02.475161+0000","last_became_active":"2024-03-30T13:42:49.322665+0000","last_became_peered":"2024-03-30T13:42:49.322665+0000","last_unstale":"2024-03-30T13:52:02.475161+0000","last_undegraded":"2024-03-30T13:52:02.475161+0000","last_fullsized":"2024-03-30T13:52:02.475161+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:05.319558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:30.465974+0000","last_clean_scrub_stamp":"2024-03-30T13:51:05.319558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:14.938754+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":305,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.461416+0000","last_change":"2024-03-30T13:50:56.342569+0000","last_active":"2024-03-30T13:52:02.461416+0000","last_peered":"2024-03-30T13:52:02.461416+0000","last_clean":"2024-03-30T13:52:02.461416+0000","last_became_active":"2024-03-30T13:42:45.317378+0000","last_became_peered":"2024-03-30T13:42:45.317378+0000","last_unstale":"2024-03-30T13:52:02.461416+0000","last_undegraded":"2024-03-30T13:52:02.461416+0000","last_fullsized":"2024-03-30T13:52:02.461416+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:56.342454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:43.088964+0000","last_clean_scrub_stamp":"2024-03-30T13:50:56.342454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:13.203972+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":123,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:02.472527+0000","last_change":"2024-03-30T13:50:08.508437+0000","last_active":"2024-03-30T13:52:02.472527+0000","last_peered":"2024-03-30T13:52:02.472527+0000","last_clean":"2024-03-30T13:50:05.490228+0000","last_became_active":"2024-03-30T13:50:08.508437+0000","last_became_peered":"2024-03-30T13:50:08.508437+0000","last_unstale":"2024-03-30T13:52:02.472527+0000","last_undegraded":"2024-03-30T13:50:08.506233+0000","last_fullsized":"2024-03-30T13:50:08.506116+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-30T13:49:08.415681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:49:08.415681+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:05.360528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":288,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.462817+0000","last_change":"2024-03-30T13:51:05.756377+0000","last_active":"2024-03-30T13:52:02.462817+0000","last_peered":"2024-03-30T13:52:02.462817+0000","last_clean":"2024-03-30T13:52:02.462817+0000","last_became_active":"2024-03-30T13:42:51.371378+0000","last_became_peered":"2024-03-30T13:42:51.371378+0000","last_unstale":"2024-03-30T13:52:02.462817+0000","last_undegraded":"2024-03-30T13:52:02.462817+0000","last_fullsized":"2024-03-30T13:52:02.462817+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:05.756146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:05.756146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:35.754284+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":295,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.464441+0000","last_change":"2024-03-30T13:50:08.506226+0000","last_active":"2024-03-30T13:52:02.464441+0000","last_peered":"2024-03-30T13:52:02.464441+0000","last_clean":"2024-03-30T13:50:05.490914+0000","last_became_active":"2024-03-30T13:50:08.506226+0000","last_became_peered":"2024-03-30T13:50:08.506226+0000","last_unstale":"2024-03-30T13:52:02.464441+0000","last_undegraded":"2024-03-30T13:52:02.464441+0000","last_fullsized":"2024-03-30T13:50:08.502611+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:25.104610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:48:25.104610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:04.016136+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":303,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.463677+0000","last_change":"2024-03-30T13:50:08.508556+0000","last_active":"2024-03-30T13:52:02.463677+0000","last_peered":"2024-03-30T13:52:02.463677+0000","last_clean":"2024-03-30T13:50:07.261781+0000","last_became_active":"2024-03-30T13:50:08.508556+0000","last_became_peered":"2024-03-30T13:50:08.508556+0000","last_unstale":"2024-03-30T13:52:02.463677+0000","last_undegraded":"2024-03-30T13:52:02.463677+0000","last_fullsized":"2024-03-30T13:50:08.502793+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:07.261595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:50:07.261595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:04.016200+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"88'17","reported_seq":331,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.462099+0000","last_change":"2024-03-30T13:51:20.908809+0000","last_active":"2024-03-30T13:52:02.462099+0000","last_peered":"2024-03-30T13:52:02.462099+0000","last_clean":"2024-03-30T13:52:02.462099+0000","last_became_active":"2024-03-30T13:42:47.317881+0000","last_became_peered":"2024-03-30T13:42:47.317881+0000","last_unstale":"2024-03-30T13:52:02.462099+0000","last_undegraded":"2024-03-30T13:52:02.462099+0000","last_fullsized":"2024-03-30T13:52:02.462099+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'17","last_scrub_stamp":"2024-03-30T13:51:20.908641+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-03-30T13:43:52.044427+0000","last_clean_scrub_stamp":"2024-03-30T13:51:20.908641+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:35.127825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":289,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:02.473064+0000","last_change":"2024-03-30T13:50:08.517791+0000","last_active":"2024-03-30T13:52:02.473064+0000","last_peered":"2024-03-30T13:52:02.473064+0000","last_clean":"2024-03-30T13:50:05.495152+0000","last_became_active":"2024-03-30T13:50:08.517791+0000","last_became_peered":"2024-03-30T13:50:08.517791+0000","last_unstale":"2024-03-30T13:52:02.473064+0000","last_undegraded":"2024-03-30T13:50:08.511324+0000","last_fullsized":"2024-03-30T13:50:08.511207+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T13:48:25.249724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:25.249724+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":438,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:07.177311+0000","last_change":"2024-03-30T13:52:07.177353+0000","last_active":"2024-03-30T13:52:07.177311+0000","last_peered":"2024-03-30T13:52:07.177311+0000","last_clean":"2024-03-30T13:52:07.177311+0000","last_became_active":"2024-03-30T13:42:49.323362+0000","last_became_peered":"2024-03-30T13:42:49.323362+0000","last_unstale":"2024-03-30T13:52:07.177311+0000","last_undegraded":"2024-03-30T13:52:07.177311+0000","last_fullsized":"2024-03-30T13:52:07.177311+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:52:07.177214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:07.177214+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:33.858714+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"65'1","reported_seq":316,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.474757+0000","last_change":"2024-03-30T13:51:03.380725+0000","last_active":"2024-03-30T13:52:02.474757+0000","last_peered":"2024-03-30T13:52:02.474757+0000","last_clean":"2024-03-30T13:52:02.474757+0000","last_became_active":"2024-03-30T13:42:45.307832+0000","last_became_peered":"2024-03-30T13:42:45.307832+0000","last_unstale":"2024-03-30T13:52:02.474757+0000","last_undegraded":"2024-03-30T13:52:02.474757+0000","last_fullsized":"2024-03-30T13:52:02.474757+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:51:03.380596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:03.380596+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:24.809963+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"153'15","reported_seq":330,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.465326+0000","last_change":"2024-03-30T13:51:22.253128+0000","last_active":"2024-03-30T13:52:02.465326+0000","last_peered":"2024-03-30T13:52:02.465326+0000","last_clean":"2024-03-30T13:52:02.465326+0000","last_became_active":"2024-03-30T13:42:47.500763+0000","last_became_peered":"2024-03-30T13:42:47.500763+0000","last_unstale":"2024-03-30T13:52:02.465326+0000","last_undegraded":"2024-03-30T13:52:02.465326+0000","last_fullsized":"2024-03-30T13:52:02.465326+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'15","last_scrub_stamp":"2024-03-30T13:51:22.252831+0000","last_deep_scrub":"153'15","last_deep_scrub_stamp":"2024-03-30T13:51:22.252831+0000","last_clean_scrub_stamp":"2024-03-30T13:51:22.252831+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:23.402370+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":303,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.474898+0000","last_change":"2024-03-30T13:52:00.433677+0000","last_active":"2024-03-30T13:52:02.474898+0000","last_peered":"2024-03-30T13:52:02.474898+0000","last_clean":"2024-03-30T13:52:02.474898+0000","last_became_active":"2024-03-30T13:42:51.325513+0000","last_became_peered":"2024-03-30T13:42:51.325513+0000","last_unstale":"2024-03-30T13:52:02.474898+0000","last_undegraded":"2024-03-30T13:52:02.474898+0000","last_fullsized":"2024-03-30T13:52:02.474898+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:00.432456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:00.432456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:05.621703+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":301,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.465585+0000","last_change":"2024-03-30T13:50:08.513281+0000","last_active":"2024-03-30T13:52:02.465585+0000","last_peered":"2024-03-30T13:52:02.465585+0000","last_clean":"2024-03-30T13:50:05.503977+0000","last_became_active":"2024-03-30T13:50:08.513281+0000","last_became_peered":"2024-03-30T13:50:08.513281+0000","last_unstale":"2024-03-30T13:52:02.465585+0000","last_undegraded":"2024-03-30T13:52:02.465585+0000","last_fullsized":"2024-03-30T13:50:08.504796+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:27.947416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:27.947416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":299,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.472089+0000","last_change":"2024-03-30T13:50:23.479815+0000","last_active":"2024-03-30T13:52:02.472089+0000","last_peered":"2024-03-30T13:52:02.472089+0000","last_clean":"2024-03-30T13:52:02.472089+0000","last_became_active":"2024-03-30T13:42:45.310165+0000","last_became_peered":"2024-03-30T13:42:45.310165+0000","last_unstale":"2024-03-30T13:52:02.472089+0000","last_undegraded":"2024-03-30T13:52:02.472089+0000","last_fullsized":"2024-03-30T13:52:02.472089+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:23.479531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:50:23.479531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":329,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.484217+0000","last_change":"2024-03-30T13:51:02.103956+0000","last_active":"2024-03-30T13:52:02.484217+0000","last_peered":"2024-03-30T13:52:02.484217+0000","last_clean":"2024-03-30T13:52:02.484217+0000","last_became_active":"2024-03-30T13:42:47.324399+0000","last_became_peered":"2024-03-30T13:42:47.324399+0000","last_unstale":"2024-03-30T13:52:02.484217+0000","last_undegraded":"2024-03-30T13:52:02.484217+0000","last_fullsized":"2024-03-30T13:52:02.484217+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:51:02.103808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:02.103808+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:25.392277+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":288,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.480947+0000","last_change":"2024-03-30T13:50:08.512275+0000","last_active":"2024-03-30T13:52:02.480947+0000","last_peered":"2024-03-30T13:52:02.480947+0000","last_clean":"2024-03-30T13:50:05.495214+0000","last_became_active":"2024-03-30T13:50:08.512275+0000","last_became_peered":"2024-03-30T13:50:08.512275+0000","last_unstale":"2024-03-30T13:52:02.480947+0000","last_undegraded":"2024-03-30T13:52:02.480947+0000","last_fullsized":"2024-03-30T13:50:08.506150+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:23.264538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:23.264538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":314,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.482210+0000","last_change":"2024-03-30T13:51:08.152625+0000","last_active":"2024-03-30T13:52:02.482210+0000","last_peered":"2024-03-30T13:52:02.482210+0000","last_clean":"2024-03-30T13:52:02.482210+0000","last_became_active":"2024-03-30T13:42:49.322799+0000","last_became_peered":"2024-03-30T13:42:49.322799+0000","last_unstale":"2024-03-30T13:52:02.482210+0000","last_undegraded":"2024-03-30T13:52:02.482210+0000","last_fullsized":"2024-03-30T13:52:02.482210+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:08.152508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:54.893653+0000","last_clean_scrub_stamp":"2024-03-30T13:51:08.152508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:14.770028+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":313,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.461553+0000","last_change":"2024-03-30T13:51:23.233277+0000","last_active":"2024-03-30T13:52:02.461553+0000","last_peered":"2024-03-30T13:52:02.461553+0000","last_clean":"2024-03-30T13:52:02.461553+0000","last_became_active":"2024-03-30T13:42:45.314302+0000","last_became_peered":"2024-03-30T13:42:45.314302+0000","last_unstale":"2024-03-30T13:52:02.461553+0000","last_undegraded":"2024-03-30T13:52:02.461553+0000","last_fullsized":"2024-03-30T13:52:02.461553+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:23.233143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:23.233143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:38.884648+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":310,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:03.989310+0000","last_change":"2024-03-30T13:52:03.989310+0000","last_active":"2024-03-30T13:52:03.989310+0000","last_peered":"2024-03-30T13:52:03.989310+0000","last_clean":"2024-03-30T13:52:03.989310+0000","last_became_active":"2024-03-30T13:42:47.321428+0000","last_became_peered":"2024-03-30T13:42:47.321428+0000","last_unstale":"2024-03-30T13:52:03.989310+0000","last_undegraded":"2024-03-30T13:52:03.989310+0000","last_fullsized":"2024-03-30T13:52:03.989310+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-30T13:52:03.989075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:03.989075+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:07.727442+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":288,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.469275+0000","last_change":"2024-03-30T13:51:13.783533+0000","last_active":"2024-03-30T13:52:02.469275+0000","last_peered":"2024-03-30T13:52:02.469275+0000","last_clean":"2024-03-30T13:52:02.469275+0000","last_became_active":"2024-03-30T13:42:51.374304+0000","last_became_peered":"2024-03-30T13:42:51.374304+0000","last_unstale":"2024-03-30T13:52:02.469275+0000","last_undegraded":"2024-03-30T13:52:02.469275+0000","last_fullsized":"2024-03-30T13:52:02.469275+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:13.783403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:13.783403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:20.259203+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":306,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.472680+0000","last_change":"2024-03-30T13:51:20.297584+0000","last_active":"2024-03-30T13:52:02.472680+0000","last_peered":"2024-03-30T13:52:02.472680+0000","last_clean":"2024-03-30T13:52:02.472680+0000","last_became_active":"2024-03-30T13:42:49.324206+0000","last_became_peered":"2024-03-30T13:42:49.324206+0000","last_unstale":"2024-03-30T13:52:02.472680+0000","last_undegraded":"2024-03-30T13:52:02.472680+0000","last_fullsized":"2024-03-30T13:52:02.472680+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:20.297467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:42.616155+0000","last_clean_scrub_stamp":"2024-03-30T13:51:20.297467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:25.769571+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":314,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:04.222665+0000","last_change":"2024-03-30T13:52:04.222697+0000","last_active":"2024-03-30T13:52:04.222665+0000","last_peered":"2024-03-30T13:52:04.222665+0000","last_clean":"2024-03-30T13:52:04.222665+0000","last_became_active":"2024-03-30T13:42:45.319482+0000","last_became_peered":"2024-03-30T13:42:45.319482+0000","last_unstale":"2024-03-30T13:52:04.222665+0000","last_undegraded":"2024-03-30T13:52:04.222665+0000","last_fullsized":"2024-03-30T13:52:04.222665+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:04.222591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:25.494903+0000","last_clean_scrub_stamp":"2024-03-30T13:52:04.222591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:24.498670+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":323,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.461564+0000","last_change":"2024-03-30T13:51:21.863041+0000","last_active":"2024-03-30T13:52:02.461564+0000","last_peered":"2024-03-30T13:52:02.461564+0000","last_clean":"2024-03-30T13:52:02.461564+0000","last_became_active":"2024-03-30T13:42:47.500712+0000","last_became_peered":"2024-03-30T13:42:47.500712+0000","last_unstale":"2024-03-30T13:52:02.461564+0000","last_undegraded":"2024-03-30T13:52:02.461564+0000","last_fullsized":"2024-03-30T13:52:02.461564+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:51:21.862885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:21.862885+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:37.730856+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":301,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:03.006939+0000","last_change":"2024-03-30T13:52:03.006976+0000","last_active":"2024-03-30T13:52:03.006939+0000","last_peered":"2024-03-30T13:52:03.006939+0000","last_clean":"2024-03-30T13:52:03.006939+0000","last_became_active":"2024-03-30T13:42:51.333603+0000","last_became_peered":"2024-03-30T13:42:51.333603+0000","last_unstale":"2024-03-30T13:52:03.006939+0000","last_undegraded":"2024-03-30T13:52:03.006939+0000","last_fullsized":"2024-03-30T13:52:03.006939+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:03.006855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:28.314391+0000","last_clean_scrub_stamp":"2024-03-30T13:52:03.006855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:25.254259+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":291,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.468992+0000","last_change":"2024-03-30T13:50:13.446848+0000","last_active":"2024-03-30T13:52:02.468992+0000","last_peered":"2024-03-30T13:52:02.468992+0000","last_clean":"2024-03-30T13:52:02.468992+0000","last_became_active":"2024-03-30T13:42:49.325219+0000","last_became_peered":"2024-03-30T13:42:49.325219+0000","last_unstale":"2024-03-30T13:52:02.468992+0000","last_undegraded":"2024-03-30T13:52:02.468992+0000","last_fullsized":"2024-03-30T13:52:02.468992+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:13.446724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:50:13.446724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":305,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.477111+0000","last_change":"2024-03-30T13:50:21.125840+0000","last_active":"2024-03-30T13:52:02.477111+0000","last_peered":"2024-03-30T13:52:02.477111+0000","last_clean":"2024-03-30T13:52:02.477111+0000","last_became_active":"2024-03-30T13:42:45.311071+0000","last_became_peered":"2024-03-30T13:42:45.311071+0000","last_unstale":"2024-03-30T13:52:02.477111+0000","last_undegraded":"2024-03-30T13:52:02.477111+0000","last_fullsized":"2024-03-30T13:52:02.477111+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:21.125709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:30.187218+0000","last_clean_scrub_stamp":"2024-03-30T13:50:21.125709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":322,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:02.474031+0000","last_change":"2024-03-30T13:50:08.513156+0000","last_active":"2024-03-30T13:52:02.474031+0000","last_peered":"2024-03-30T13:52:02.474031+0000","last_clean":"2024-03-30T13:50:05.496678+0000","last_became_active":"2024-03-30T13:50:08.513156+0000","last_became_peered":"2024-03-30T13:50:08.513156+0000","last_unstale":"2024-03-30T13:52:02.474031+0000","last_undegraded":"2024-03-30T13:50:08.507726+0000","last_fullsized":"2024-03-30T13:50:08.507616+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:49:26.480540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:49:26.480540+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:05.360558+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":295,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:03.011958+0000","last_change":"2024-03-30T13:52:03.011993+0000","last_active":"2024-03-30T13:52:03.011958+0000","last_peered":"2024-03-30T13:52:03.011958+0000","last_clean":"2024-03-30T13:52:03.011958+0000","last_became_active":"2024-03-30T13:42:51.334597+0000","last_became_peered":"2024-03-30T13:42:51.334597+0000","last_unstale":"2024-03-30T13:52:03.011958+0000","last_undegraded":"2024-03-30T13:52:03.011958+0000","last_fullsized":"2024-03-30T13:52:03.011958+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:03.011875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:24.598976+0000","last_clean_scrub_stamp":"2024-03-30T13:52:03.011875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:18.152197+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":307,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.473827+0000","last_change":"2024-03-30T13:51:26.193937+0000","last_active":"2024-03-30T13:52:02.473827+0000","last_peered":"2024-03-30T13:52:02.473827+0000","last_clean":"2024-03-30T13:52:02.473827+0000","last_became_active":"2024-03-30T13:42:49.322843+0000","last_became_peered":"2024-03-30T13:42:49.322843+0000","last_unstale":"2024-03-30T13:52:02.473827+0000","last_undegraded":"2024-03-30T13:52:02.473827+0000","last_fullsized":"2024-03-30T13:52:02.473827+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:26.193818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:34.815783+0000","last_clean_scrub_stamp":"2024-03-30T13:51:26.193818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:26.501190+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":309,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.463964+0000","last_change":"2024-03-30T13:50:22.123301+0000","last_active":"2024-03-30T13:52:02.463964+0000","last_peered":"2024-03-30T13:52:02.463964+0000","last_clean":"2024-03-30T13:52:02.463964+0000","last_became_active":"2024-03-30T13:42:45.315864+0000","last_became_peered":"2024-03-30T13:42:45.315864+0000","last_unstale":"2024-03-30T13:52:02.463964+0000","last_undegraded":"2024-03-30T13:52:02.463964+0000","last_fullsized":"2024-03-30T13:52:02.463964+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:22.123086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:50:22.123086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":337,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:02.466279+0000","last_change":"2024-03-30T13:50:08.513382+0000","last_active":"2024-03-30T13:52:02.466279+0000","last_peered":"2024-03-30T13:52:02.466279+0000","last_clean":"2024-03-30T13:50:05.491029+0000","last_became_active":"2024-03-30T13:50:08.513382+0000","last_became_peered":"2024-03-30T13:50:08.513382+0000","last_unstale":"2024-03-30T13:52:02.466279+0000","last_undegraded":"2024-03-30T13:50:08.503759+0000","last_fullsized":"2024-03-30T13:50:08.503643+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:48:22.068206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:22.068206+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:12.975244+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":301,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.464909+0000","last_change":"2024-03-30T13:51:25.374768+0000","last_active":"2024-03-30T13:52:02.464909+0000","last_peered":"2024-03-30T13:52:02.464909+0000","last_clean":"2024-03-30T13:52:02.464909+0000","last_became_active":"2024-03-30T13:42:51.336162+0000","last_became_peered":"2024-03-30T13:42:51.336162+0000","last_unstale":"2024-03-30T13:52:02.464909+0000","last_undegraded":"2024-03-30T13:52:02.464909+0000","last_fullsized":"2024-03-30T13:52:02.464909+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:25.374633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:25.374633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:29.366747+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":419,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.471303+0000","last_change":"2024-03-30T13:51:01.033692+0000","last_active":"2024-03-30T13:52:02.471303+0000","last_peered":"2024-03-30T13:52:02.471303+0000","last_clean":"2024-03-30T13:52:02.471303+0000","last_became_active":"2024-03-30T13:42:49.321203+0000","last_became_peered":"2024-03-30T13:42:49.321203+0000","last_unstale":"2024-03-30T13:52:02.471303+0000","last_undegraded":"2024-03-30T13:52:02.471303+0000","last_fullsized":"2024-03-30T13:52:02.471303+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:51:01.033348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:01.033348+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":301,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.472745+0000","last_change":"2024-03-30T13:50:08.511650+0000","last_active":"2024-03-30T13:52:02.472745+0000","last_peered":"2024-03-30T13:52:02.472745+0000","last_clean":"2024-03-30T13:50:05.495125+0000","last_became_active":"2024-03-30T13:50:08.511650+0000","last_became_peered":"2024-03-30T13:50:08.511650+0000","last_unstale":"2024-03-30T13:52:02.472745+0000","last_undegraded":"2024-03-30T13:52:02.472745+0000","last_fullsized":"2024-03-30T13:50:08.506728+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:07.618186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:49:07.618186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":324,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:02.470338+0000","last_change":"2024-03-30T13:50:08.514919+0000","last_active":"2024-03-30T13:52:02.470338+0000","last_peered":"2024-03-30T13:52:02.470338+0000","last_clean":"2024-03-30T13:50:05.508102+0000","last_became_active":"2024-03-30T13:50:08.514919+0000","last_became_peered":"2024-03-30T13:50:08.514919+0000","last_unstale":"2024-03-30T13:52:02.470338+0000","last_undegraded":"2024-03-30T13:50:08.503113+0000","last_fullsized":"2024-03-30T13:50:08.502993+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-30T13:48:21.678227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:21.678227+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":291,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.463926+0000","last_change":"2024-03-30T13:50:08.514514+0000","last_active":"2024-03-30T13:52:02.463926+0000","last_peered":"2024-03-30T13:52:02.463926+0000","last_clean":"2024-03-30T13:50:05.490356+0000","last_became_active":"2024-03-30T13:50:08.514514+0000","last_became_peered":"2024-03-30T13:50:08.514514+0000","last_unstale":"2024-03-30T13:52:02.463926+0000","last_undegraded":"2024-03-30T13:52:02.463926+0000","last_fullsized":"2024-03-30T13:50:08.502291+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:30.238582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:30.238582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:04.016188+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"168'9","reported_seq":416,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:02.470562+0000","last_change":"2024-03-30T13:50:08.515013+0000","last_active":"2024-03-30T13:52:02.470562+0000","last_peered":"2024-03-30T13:52:02.470562+0000","last_clean":"2024-03-30T13:50:05.508192+0000","last_became_active":"2024-03-30T13:50:08.515013+0000","last_became_peered":"2024-03-30T13:50:08.515013+0000","last_unstale":"2024-03-30T13:52:02.470562+0000","last_undegraded":"2024-03-30T13:50:08.503461+0000","last_fullsized":"2024-03-30T13:50:08.503337+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:49:23.573435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:23.573435+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:08.958903+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":314,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:09.007570+0000","last_change":"2024-03-30T13:52:09.007605+0000","last_active":"2024-03-30T13:52:09.007570+0000","last_peered":"2024-03-30T13:52:09.007570+0000","last_clean":"2024-03-30T13:52:09.007570+0000","last_became_active":"2024-03-30T13:42:45.314585+0000","last_became_peered":"2024-03-30T13:42:45.314585+0000","last_unstale":"2024-03-30T13:52:09.007570+0000","last_undegraded":"2024-03-30T13:52:09.007570+0000","last_fullsized":"2024-03-30T13:52:09.007570+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:09.007490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:28.734080+0000","last_clean_scrub_stamp":"2024-03-30T13:52:09.007490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:32.345744+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":326,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:03.227770+0000","last_change":"2024-03-30T13:52:03.227854+0000","last_active":"2024-03-30T13:52:03.227770+0000","last_peered":"2024-03-30T13:52:03.227770+0000","last_clean":"2024-03-30T13:52:03.227770+0000","last_became_active":"2024-03-30T13:42:47.501559+0000","last_became_peered":"2024-03-30T13:42:47.501559+0000","last_unstale":"2024-03-30T13:52:03.227770+0000","last_undegraded":"2024-03-30T13:52:03.227770+0000","last_fullsized":"2024-03-30T13:52:03.227770+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:52:03.227628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:03.227628+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:28.935372+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":79,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.464627+0000","last_change":"2024-03-30T13:50:08.513016+0000","last_active":"2024-03-30T13:52:02.464627+0000","last_peered":"2024-03-30T13:52:02.464627+0000","last_clean":"2024-03-30T13:50:05.491001+0000","last_became_active":"2024-03-30T13:50:08.513016+0000","last_became_peered":"2024-03-30T13:50:08.513016+0000","last_unstale":"2024-03-30T13:52:02.464627+0000","last_undegraded":"2024-03-30T13:52:02.464627+0000","last_fullsized":"2024-03-30T13:50:08.502775+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:14.322254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:14.322254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:04.016095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":291,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.463222+0000","last_change":"2024-03-30T13:51:08.670117+0000","last_active":"2024-03-30T13:52:02.463222+0000","last_peered":"2024-03-30T13:52:02.463222+0000","last_clean":"2024-03-30T13:52:02.463222+0000","last_became_active":"2024-03-30T13:42:49.323222+0000","last_became_peered":"2024-03-30T13:42:49.323222+0000","last_unstale":"2024-03-30T13:52:02.463222+0000","last_undegraded":"2024-03-30T13:52:02.463222+0000","last_fullsized":"2024-03-30T13:52:02.463222+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:08.670035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:08.670035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:16.796672+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":314,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.472339+0000","last_change":"2024-03-30T13:51:18.239125+0000","last_active":"2024-03-30T13:52:02.472339+0000","last_peered":"2024-03-30T13:52:02.472339+0000","last_clean":"2024-03-30T13:52:02.472339+0000","last_became_active":"2024-03-30T13:42:45.313617+0000","last_became_peered":"2024-03-30T13:42:45.313617+0000","last_unstale":"2024-03-30T13:52:02.472339+0000","last_undegraded":"2024-03-30T13:52:02.472339+0000","last_fullsized":"2024-03-30T13:52:02.472339+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:18.239047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:18.239047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:47.096925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":325,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.465424+0000","last_change":"2024-03-30T13:51:20.179768+0000","last_active":"2024-03-30T13:52:02.465424+0000","last_peered":"2024-03-30T13:52:02.465424+0000","last_clean":"2024-03-30T13:52:02.465424+0000","last_became_active":"2024-03-30T13:42:47.501925+0000","last_became_peered":"2024-03-30T13:42:47.501925+0000","last_unstale":"2024-03-30T13:52:02.465424+0000","last_undegraded":"2024-03-30T13:52:02.465424+0000","last_fullsized":"2024-03-30T13:52:02.465424+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:51:20.179623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:20.179623+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:45.908958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"72'1","reported_seq":302,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.476929+0000","last_change":"2024-03-30T13:51:17.299808+0000","last_active":"2024-03-30T13:52:02.476929+0000","last_peered":"2024-03-30T13:52:02.476929+0000","last_clean":"2024-03-30T13:52:02.476929+0000","last_became_active":"2024-03-30T13:42:51.372023+0000","last_became_peered":"2024-03-30T13:42:51.372023+0000","last_unstale":"2024-03-30T13:52:02.476929+0000","last_undegraded":"2024-03-30T13:52:02.476929+0000","last_fullsized":"2024-03-30T13:52:02.476929+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T13:51:17.299690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:17.299690+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:20.658989+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":298,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.470039+0000","last_change":"2024-03-30T13:50:21.754632+0000","last_active":"2024-03-30T13:52:02.470039+0000","last_peered":"2024-03-30T13:52:02.470039+0000","last_clean":"2024-03-30T13:52:02.470039+0000","last_became_active":"2024-03-30T13:42:49.323924+0000","last_became_peered":"2024-03-30T13:42:49.323924+0000","last_unstale":"2024-03-30T13:52:02.470039+0000","last_undegraded":"2024-03-30T13:52:02.470039+0000","last_fullsized":"2024-03-30T13:52:02.470039+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:21.754451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:50:21.754451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:03.115866+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":322,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.480405+0000","last_change":"2024-03-30T13:51:20.175560+0000","last_active":"2024-03-30T13:52:02.480405+0000","last_peered":"2024-03-30T13:52:02.480405+0000","last_clean":"2024-03-30T13:52:02.480405+0000","last_became_active":"2024-03-30T13:42:45.318956+0000","last_became_peered":"2024-03-30T13:42:45.318956+0000","last_unstale":"2024-03-30T13:52:02.480405+0000","last_undegraded":"2024-03-30T13:52:02.480405+0000","last_fullsized":"2024-03-30T13:52:02.480405+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:20.175438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:51:20.175438+0000","last_clean_scrub_stamp":"2024-03-30T13:51:20.175438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:36.026962+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":304,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.463127+0000","last_change":"2024-03-30T13:50:21.411326+0000","last_active":"2024-03-30T13:52:02.463127+0000","last_peered":"2024-03-30T13:52:02.463127+0000","last_clean":"2024-03-30T13:52:02.463127+0000","last_became_active":"2024-03-30T13:42:47.501426+0000","last_became_peered":"2024-03-30T13:42:47.501426+0000","last_unstale":"2024-03-30T13:52:02.463127+0000","last_undegraded":"2024-03-30T13:52:02.463127+0000","last_fullsized":"2024-03-30T13:52:02.463127+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-03-30T13:50:21.411186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:50:21.411186+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"196'14913","reported_seq":15027,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:08.552689+0000","last_change":"2024-03-30T13:50:08.578484+0000","last_active":"2024-03-30T13:52:08.552689+0000","last_peered":"2024-03-30T13:52:08.552689+0000","last_clean":"2024-03-30T13:50:07.502722+0000","last_became_active":"2024-03-30T13:50:08.524860+0000","last_became_peered":"2024-03-30T13:50:08.524860+0000","last_unstale":"2024-03-30T13:52:08.552689+0000","last_undegraded":"2024-03-30T13:50:08.516186+0000","last_fullsized":"2024-03-30T13:50:08.505072+0000","mapping_epoch":167,"log_start":"182'9500","ondisk_log_start":"182'9500","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:49:00.428679+0000","objects_scrubbed":0,"log_size":5413,"log_dups_size":0,"ondisk_log_size":5413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99942400,"num_objects":1525,"num_object_clones":0,"num_object_copies":4575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14913,"num_write_kb":526016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":1525}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":295,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.470713+0000","last_change":"2024-03-30T13:51:05.957160+0000","last_active":"2024-03-30T13:52:02.470713+0000","last_peered":"2024-03-30T13:52:02.470713+0000","last_clean":"2024-03-30T13:52:02.470713+0000","last_became_active":"2024-03-30T13:42:51.371360+0000","last_became_peered":"2024-03-30T13:42:51.371360+0000","last_unstale":"2024-03-30T13:52:02.470713+0000","last_undegraded":"2024-03-30T13:52:02.470713+0000","last_fullsized":"2024-03-30T13:52:02.470713+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:05.957043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:05.957043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:32.455311+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":318,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.477148+0000","last_change":"2024-03-30T13:50:24.087276+0000","last_active":"2024-03-30T13:52:02.477148+0000","last_peered":"2024-03-30T13:52:02.477148+0000","last_clean":"2024-03-30T13:52:02.477148+0000","last_became_active":"2024-03-30T13:42:45.318963+0000","last_became_peered":"2024-03-30T13:42:45.318963+0000","last_unstale":"2024-03-30T13:52:02.477148+0000","last_undegraded":"2024-03-30T13:52:02.477148+0000","last_fullsized":"2024-03-30T13:52:02.477148+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-30T13:50:24.087137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:50:24.087137+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":336,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.472433+0000","last_change":"2024-03-30T13:51:27.133579+0000","last_active":"2024-03-30T13:52:02.472433+0000","last_peered":"2024-03-30T13:52:02.472433+0000","last_clean":"2024-03-30T13:52:02.472433+0000","last_became_active":"2024-03-30T13:42:47.501937+0000","last_became_peered":"2024-03-30T13:42:47.501937+0000","last_unstale":"2024-03-30T13:52:02.472433+0000","last_undegraded":"2024-03-30T13:52:02.472433+0000","last_fullsized":"2024-03-30T13:52:02.472433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-30T13:51:27.133454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:27.133454+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:39.077427+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"72'8","reported_seq":427,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.472529+0000","last_change":"2024-03-30T13:51:19.286719+0000","last_active":"2024-03-30T13:52:02.472529+0000","last_peered":"2024-03-30T13:52:02.472529+0000","last_clean":"2024-03-30T13:52:02.472529+0000","last_became_active":"2024-03-30T13:42:49.322675+0000","last_became_peered":"2024-03-30T13:42:49.322675+0000","last_unstale":"2024-03-30T13:52:02.472529+0000","last_undegraded":"2024-03-30T13:52:02.472529+0000","last_fullsized":"2024-03-30T13:52:02.472529+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:51:19.286592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:19.286592+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:46.501686+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"196'17010","reported_seq":17353,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:08.563932+0000","last_change":"2024-03-30T13:52:02.969134+0000","last_active":"2024-03-30T13:52:08.563932+0000","last_peered":"2024-03-30T13:52:08.563932+0000","last_clean":"2024-03-30T13:52:08.563932+0000","last_became_active":"2024-03-30T13:49:01.523463+0000","last_became_peered":"2024-03-30T13:49:01.523463+0000","last_unstale":"2024-03-30T13:52:08.563932+0000","last_undegraded":"2024-03-30T13:52:08.563932+0000","last_fullsized":"2024-03-30T13:52:08.563932+0000","mapping_epoch":147,"log_start":"180'10701","ondisk_log_start":"180'10701","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'16529","last_scrub_stamp":"2024-03-30T13:52:02.968906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:52:02.968906+0000","objects_scrubbed":2043,"log_size":6309,"log_dups_size":0,"ondisk_log_size":6309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:13.745032+0000","scrub_duration":613,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100794368,"num_objects":1538,"num_object_clones":0,"num_object_copies":4614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14904,"num_write_kb":526144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":288,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.469185+0000","last_change":"2024-03-30T13:50:59.723370+0000","last_active":"2024-03-30T13:52:02.469185+0000","last_peered":"2024-03-30T13:52:02.469185+0000","last_clean":"2024-03-30T13:52:02.469185+0000","last_became_active":"2024-03-30T13:42:51.334672+0000","last_became_peered":"2024-03-30T13:42:51.334672+0000","last_unstale":"2024-03-30T13:52:02.469185+0000","last_undegraded":"2024-03-30T13:52:02.469185+0000","last_fullsized":"2024-03-30T13:52:02.469185+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:59.723249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:50:59.723249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:27.249728+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":314,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.477554+0000","last_change":"2024-03-30T13:51:15.247447+0000","last_active":"2024-03-30T13:52:02.477554+0000","last_peered":"2024-03-30T13:52:02.477554+0000","last_clean":"2024-03-30T13:52:02.477554+0000","last_became_active":"2024-03-30T13:42:45.312328+0000","last_became_peered":"2024-03-30T13:42:45.312328+0000","last_unstale":"2024-03-30T13:52:02.477554+0000","last_undegraded":"2024-03-30T13:52:02.477554+0000","last_fullsized":"2024-03-30T13:52:02.477554+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:15.246997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:15.246997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:35.377772+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":328,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.463212+0000","last_change":"2024-03-30T13:50:10.274074+0000","last_active":"2024-03-30T13:52:02.463212+0000","last_peered":"2024-03-30T13:52:02.463212+0000","last_clean":"2024-03-30T13:52:02.463212+0000","last_became_active":"2024-03-30T13:42:47.316187+0000","last_became_peered":"2024-03-30T13:42:47.316187+0000","last_unstale":"2024-03-30T13:52:02.463212+0000","last_undegraded":"2024-03-30T13:52:02.463212+0000","last_fullsized":"2024-03-30T13:52:02.463212+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-30T13:50:10.273940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:50:10.273940+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":79,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.462386+0000","last_change":"2024-03-30T13:50:08.509701+0000","last_active":"2024-03-30T13:52:02.462386+0000","last_peered":"2024-03-30T13:52:02.462386+0000","last_clean":"2024-03-30T13:50:05.499237+0000","last_became_active":"2024-03-30T13:50:08.509701+0000","last_became_peered":"2024-03-30T13:50:08.509701+0000","last_unstale":"2024-03-30T13:52:02.462386+0000","last_undegraded":"2024-03-30T13:52:02.462386+0000","last_fullsized":"2024-03-30T13:50:08.501671+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:22.401087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:22.401087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"196'14734","reported_seq":14912,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:09.326697+0000","last_change":"2024-03-30T13:51:11.338320+0000","last_active":"2024-03-30T13:52:09.326697+0000","last_peered":"2024-03-30T13:52:09.326697+0000","last_clean":"2024-03-30T13:52:09.326697+0000","last_became_active":"2024-03-30T13:49:01.523984+0000","last_became_peered":"2024-03-30T13:49:01.523984+0000","last_unstale":"2024-03-30T13:52:09.326697+0000","last_undegraded":"2024-03-30T13:52:09.326697+0000","last_fullsized":"2024-03-30T13:52:09.326697+0000","mapping_epoch":147,"log_start":"180'8900","ondisk_log_start":"180'8900","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'10292","last_scrub_stamp":"2024-03-30T13:51:11.338007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:51:11.338007+0000","objects_scrubbed":6036,"log_size":5834,"log_dups_size":0,"ondisk_log_size":5834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:28.787160+0000","scrub_duration":2135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98697216,"num_objects":1506,"num_object_clones":0,"num_object_copies":4518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14734,"num_write_kb":519680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":297,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.472017+0000","last_change":"2024-03-30T13:50:08.510439+0000","last_active":"2024-03-30T13:52:02.472017+0000","last_peered":"2024-03-30T13:52:02.472017+0000","last_clean":"2024-03-30T13:50:05.494808+0000","last_became_active":"2024-03-30T13:50:08.510439+0000","last_became_peered":"2024-03-30T13:50:08.510439+0000","last_unstale":"2024-03-30T13:52:02.472017+0000","last_undegraded":"2024-03-30T13:52:02.472017+0000","last_fullsized":"2024-03-30T13:50:08.506960+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:31.860675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:31.860675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":79,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.460978+0000","last_change":"2024-03-30T13:50:08.509003+0000","last_active":"2024-03-30T13:52:02.460978+0000","last_peered":"2024-03-30T13:52:02.460978+0000","last_clean":"2024-03-30T13:50:05.499015+0000","last_became_active":"2024-03-30T13:50:08.509003+0000","last_became_peered":"2024-03-30T13:50:08.509003+0000","last_unstale":"2024-03-30T13:52:02.460978+0000","last_undegraded":"2024-03-30T13:52:02.460978+0000","last_fullsized":"2024-03-30T13:50:08.502979+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:19.432242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:19.432242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":306,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.477509+0000","last_change":"2024-03-30T13:51:09.173688+0000","last_active":"2024-03-30T13:52:02.477509+0000","last_peered":"2024-03-30T13:52:02.477509+0000","last_clean":"2024-03-30T13:52:02.477509+0000","last_became_active":"2024-03-30T13:42:45.315687+0000","last_became_peered":"2024-03-30T13:42:45.315687+0000","last_unstale":"2024-03-30T13:52:02.477509+0000","last_undegraded":"2024-03-30T13:52:02.477509+0000","last_fullsized":"2024-03-30T13:52:02.477509+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:09.173459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:09.173459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:33.137912+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"108'28","reported_seq":358,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:06.210746+0000","last_change":"2024-03-30T13:52:06.210819+0000","last_active":"2024-03-30T13:52:06.210746+0000","last_peered":"2024-03-30T13:52:06.210746+0000","last_clean":"2024-03-30T13:52:06.210746+0000","last_became_active":"2024-03-30T13:42:47.500898+0000","last_became_peered":"2024-03-30T13:42:47.500898+0000","last_unstale":"2024-03-30T13:52:06.210746+0000","last_undegraded":"2024-03-30T13:52:06.210746+0000","last_fullsized":"2024-03-30T13:52:06.210746+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'28","last_scrub_stamp":"2024-03-30T13:52:06.210663+0000","last_deep_scrub":"108'28","last_deep_scrub_stamp":"2024-03-30T13:45:49.140418+0000","last_clean_scrub_stamp":"2024-03-30T13:52:06.210663+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:20.718042+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"196'14650","reported_seq":14830,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:08.884483+0000","last_change":"2024-03-30T13:50:18.439070+0000","last_active":"2024-03-30T13:52:08.884483+0000","last_peered":"2024-03-30T13:52:08.884483+0000","last_clean":"2024-03-30T13:52:08.884483+0000","last_became_active":"2024-03-30T13:49:01.453528+0000","last_became_peered":"2024-03-30T13:49:01.453528+0000","last_unstale":"2024-03-30T13:52:08.884483+0000","last_undegraded":"2024-03-30T13:52:08.884483+0000","last_fullsized":"2024-03-30T13:52:08.884483+0000","mapping_epoch":147,"log_start":"183'9700","ondisk_log_start":"183'9700","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'6278","last_scrub_stamp":"2024-03-30T13:50:18.438748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:50:18.438748+0000","objects_scrubbed":6210,"log_size":4950,"log_dups_size":0,"ondisk_log_size":4950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2083,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94502912,"num_objects":1442,"num_object_clones":0,"num_object_copies":4326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14650,"num_write_kb":514944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":289,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.462607+0000","last_change":"2024-03-30T13:50:08.509476+0000","last_active":"2024-03-30T13:52:02.462607+0000","last_peered":"2024-03-30T13:52:02.462607+0000","last_clean":"2024-03-30T13:50:05.499348+0000","last_became_active":"2024-03-30T13:50:08.509476+0000","last_became_peered":"2024-03-30T13:50:08.509476+0000","last_unstale":"2024-03-30T13:52:02.462607+0000","last_undegraded":"2024-03-30T13:52:02.462607+0000","last_fullsized":"2024-03-30T13:50:08.503098+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:12.807605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:44.095338+0000","last_clean_scrub_stamp":"2024-03-30T13:49:12.807605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":79,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.463819+0000","last_change":"2024-03-30T13:50:08.507875+0000","last_active":"2024-03-30T13:52:02.463819+0000","last_peered":"2024-03-30T13:52:02.463819+0000","last_clean":"2024-03-30T13:50:05.490245+0000","last_became_active":"2024-03-30T13:50:08.507875+0000","last_became_peered":"2024-03-30T13:50:08.507875+0000","last_unstale":"2024-03-30T13:52:02.463819+0000","last_undegraded":"2024-03-30T13:52:02.463819+0000","last_fullsized":"2024-03-30T13:50:08.502700+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:13.399683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:28.225742+0000","last_clean_scrub_stamp":"2024-03-30T13:49:13.399683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:04.016176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":315,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.473612+0000","last_change":"2024-03-30T13:51:19.099448+0000","last_active":"2024-03-30T13:52:02.473612+0000","last_peered":"2024-03-30T13:52:02.473612+0000","last_clean":"2024-03-30T13:52:02.473612+0000","last_became_active":"2024-03-30T13:42:45.314767+0000","last_became_peered":"2024-03-30T13:42:45.314767+0000","last_unstale":"2024-03-30T13:52:02.473612+0000","last_undegraded":"2024-03-30T13:52:02.473612+0000","last_fullsized":"2024-03-30T13:52:02.473612+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:19.099205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:19.099205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:25.579479+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":342,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.461237+0000","last_change":"2024-03-30T13:51:13.148898+0000","last_active":"2024-03-30T13:52:02.461237+0000","last_peered":"2024-03-30T13:52:02.461237+0000","last_clean":"2024-03-30T13:52:02.461237+0000","last_became_active":"2024-03-30T13:43:11.446379+0000","last_became_peered":"2024-03-30T13:43:11.446379+0000","last_unstale":"2024-03-30T13:52:02.461237+0000","last_undegraded":"2024-03-30T13:52:02.461237+0000","last_fullsized":"2024-03-30T13:52:02.461237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:51:13.148563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:13.148563+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:13.860133+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"196'14833","reported_seq":15014,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:08.535033+0000","last_change":"2024-03-30T13:51:09.898743+0000","last_active":"2024-03-30T13:52:08.535033+0000","last_peered":"2024-03-30T13:52:08.535033+0000","last_clean":"2024-03-30T13:52:08.535033+0000","last_became_active":"2024-03-30T13:49:01.523800+0000","last_became_peered":"2024-03-30T13:49:01.523800+0000","last_unstale":"2024-03-30T13:52:08.535033+0000","last_undegraded":"2024-03-30T13:52:08.535033+0000","last_fullsized":"2024-03-30T13:52:08.535033+0000","mapping_epoch":147,"log_start":"182'9400","ondisk_log_start":"182'9400","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'10246","last_scrub_stamp":"2024-03-30T13:51:09.898422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:51:09.898422+0000","objects_scrubbed":6175,"log_size":5433,"log_dups_size":0,"ondisk_log_size":5433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:31.561635+0000","scrub_duration":1956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99549184,"num_objects":1519,"num_object_clones":0,"num_object_copies":4557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14833,"num_write_kb":523264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":296,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:08.043849+0000","last_change":"2024-03-30T13:52:08.043894+0000","last_active":"2024-03-30T13:52:08.043849+0000","last_peered":"2024-03-30T13:52:08.043849+0000","last_clean":"2024-03-30T13:52:08.043849+0000","last_became_active":"2024-03-30T13:42:51.331054+0000","last_became_peered":"2024-03-30T13:42:51.331054+0000","last_unstale":"2024-03-30T13:52:08.043849+0000","last_undegraded":"2024-03-30T13:52:08.043849+0000","last_fullsized":"2024-03-30T13:52:08.043849+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:08.043774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:08.043774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:37.608884+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":326,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.470497+0000","last_change":"2024-03-30T13:51:12.972656+0000","last_active":"2024-03-30T13:52:02.470497+0000","last_peered":"2024-03-30T13:52:02.470497+0000","last_clean":"2024-03-30T13:52:02.470497+0000","last_became_active":"2024-03-30T13:42:44.300789+0000","last_became_peered":"2024-03-30T13:42:44.300789+0000","last_unstale":"2024-03-30T13:52:02.470497+0000","last_undegraded":"2024-03-30T13:52:02.470497+0000","last_fullsized":"2024-03-30T13:52:02.470497+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-30T13:51:12.972422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:51:12.972422+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:25.492356+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.0021837800000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":309,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.465865+0000","last_change":"2024-03-30T13:51:01.240545+0000","last_active":"2024-03-30T13:52:02.465865+0000","last_peered":"2024-03-30T13:52:02.465865+0000","last_clean":"2024-03-30T13:52:02.465865+0000","last_became_active":"2024-03-30T13:42:45.310209+0000","last_became_peered":"2024-03-30T13:42:45.310209+0000","last_unstale":"2024-03-30T13:52:02.465865+0000","last_undegraded":"2024-03-30T13:52:02.465865+0000","last_fullsized":"2024-03-30T13:52:02.465865+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:01.240468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:01.240468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:27.407517+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"168'29","reported_seq":461,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:02.473422+0000","last_change":"2024-03-30T13:50:08.508727+0000","last_active":"2024-03-30T13:52:02.473422+0000","last_peered":"2024-03-30T13:52:02.473422+0000","last_clean":"2024-03-30T13:50:05.490389+0000","last_became_active":"2024-03-30T13:50:08.508727+0000","last_became_peered":"2024-03-30T13:50:08.508727+0000","last_unstale":"2024-03-30T13:52:02.473422+0000","last_undegraded":"2024-03-30T13:50:08.506113+0000","last_fullsized":"2024-03-30T13:50:08.505993+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-03-30T13:48:27.377242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:27.377242+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":306,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.461874+0000","last_change":"2024-03-30T13:51:18.299446+0000","last_active":"2024-03-30T13:52:02.461874+0000","last_peered":"2024-03-30T13:52:02.461874+0000","last_clean":"2024-03-30T13:52:02.461874+0000","last_became_active":"2024-03-30T13:42:49.322154+0000","last_became_peered":"2024-03-30T13:42:49.322154+0000","last_unstale":"2024-03-30T13:52:02.461874+0000","last_undegraded":"2024-03-30T13:52:02.461874+0000","last_fullsized":"2024-03-30T13:52:02.461874+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:18.299184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:18.862784+0000","last_clean_scrub_stamp":"2024-03-30T13:51:18.299184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:21.412798+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"196'19398","reported_seq":19706,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:09.047011+0000","last_change":"2024-03-30T13:51:40.320525+0000","last_active":"2024-03-30T13:52:09.047011+0000","last_peered":"2024-03-30T13:52:09.047011+0000","last_clean":"2024-03-30T13:52:09.047011+0000","last_became_active":"2024-03-30T13:49:01.451971+0000","last_became_peered":"2024-03-30T13:49:01.451971+0000","last_unstale":"2024-03-30T13:52:09.047011+0000","last_undegraded":"2024-03-30T13:52:09.047011+0000","last_fullsized":"2024-03-30T13:52:09.047011+0000","mapping_epoch":147,"log_start":"187'14502","ondisk_log_start":"187'14502","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'17048","last_scrub_stamp":"2024-03-30T13:51:40.320351+0000","last_deep_scrub":"192'17048","last_deep_scrub_stamp":"2024-03-30T13:51:40.320351+0000","last_clean_scrub_stamp":"2024-03-30T13:51:40.320351+0000","objects_scrubbed":4706,"log_size":4896,"log_dups_size":0,"ondisk_log_size":4896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":26,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:53.544019+0000","scrub_duration":25558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99876864,"num_objects":1524,"num_object_clones":0,"num_object_copies":4572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14692,"num_write_kb":518912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":294,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.477666+0000","last_change":"2024-03-30T13:51:14.276390+0000","last_active":"2024-03-30T13:52:02.477666+0000","last_peered":"2024-03-30T13:52:02.477666+0000","last_clean":"2024-03-30T13:52:02.477666+0000","last_became_active":"2024-03-30T13:42:51.372320+0000","last_became_peered":"2024-03-30T13:42:51.372320+0000","last_unstale":"2024-03-30T13:52:02.477666+0000","last_undegraded":"2024-03-30T13:52:02.477666+0000","last_fullsized":"2024-03-30T13:52:02.477666+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:14.275893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:14.275893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:28.628361+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":311,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.476182+0000","last_change":"2024-03-30T13:50:08.512561+0000","last_active":"2024-03-30T13:52:02.476182+0000","last_peered":"2024-03-30T13:52:02.476182+0000","last_clean":"2024-03-30T13:50:05.489457+0000","last_became_active":"2024-03-30T13:50:08.512561+0000","last_became_peered":"2024-03-30T13:50:08.512561+0000","last_unstale":"2024-03-30T13:52:02.476182+0000","last_undegraded":"2024-03-30T13:52:02.476182+0000","last_fullsized":"2024-03-30T13:50:08.506886+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:16.464234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:59.810189+0000","last_clean_scrub_stamp":"2024-03-30T13:49:16.464234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:05.360512+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":80,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.461308+0000","last_change":"2024-03-30T13:50:08.511337+0000","last_active":"2024-03-30T13:52:02.461308+0000","last_peered":"2024-03-30T13:52:02.461308+0000","last_clean":"2024-03-30T13:50:05.498892+0000","last_became_active":"2024-03-30T13:50:08.511337+0000","last_became_peered":"2024-03-30T13:50:08.511337+0000","last_unstale":"2024-03-30T13:52:02.461308+0000","last_undegraded":"2024-03-30T13:52:02.461308+0000","last_fullsized":"2024-03-30T13:50:08.501456+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:25.398232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:51.919203+0000","last_clean_scrub_stamp":"2024-03-30T13:49:25.398232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"72'13","reported_seq":339,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.479115+0000","last_change":"2024-03-30T13:50:55.258399+0000","last_active":"2024-03-30T13:52:02.479115+0000","last_peered":"2024-03-30T13:52:02.479115+0000","last_clean":"2024-03-30T13:52:02.479115+0000","last_became_active":"2024-03-30T13:42:47.320541+0000","last_became_peered":"2024-03-30T13:42:47.320541+0000","last_unstale":"2024-03-30T13:52:02.479115+0000","last_undegraded":"2024-03-30T13:52:02.479115+0000","last_fullsized":"2024-03-30T13:52:02.479115+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'13","last_scrub_stamp":"2024-03-30T13:50:55.258141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:50:55.258141+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:15.076428+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":79,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.469760+0000","last_change":"2024-03-30T13:50:08.514353+0000","last_active":"2024-03-30T13:52:02.469760+0000","last_peered":"2024-03-30T13:52:02.469760+0000","last_clean":"2024-03-30T13:50:05.492654+0000","last_became_active":"2024-03-30T13:50:08.514353+0000","last_became_peered":"2024-03-30T13:50:08.514353+0000","last_unstale":"2024-03-30T13:52:02.469760+0000","last_undegraded":"2024-03-30T13:52:02.469760+0000","last_fullsized":"2024-03-30T13:50:08.505968+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:28.373277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:28.373277+0000","last_clean_scrub_stamp":"2024-03-30T13:49:28.373277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:03.115754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"196'14640","reported_seq":14825,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:08.563326+0000","last_change":"2024-03-30T13:51:02.091636+0000","last_active":"2024-03-30T13:52:08.563326+0000","last_peered":"2024-03-30T13:52:08.563326+0000","last_clean":"2024-03-30T13:52:08.563326+0000","last_became_active":"2024-03-30T13:49:01.450928+0000","last_became_peered":"2024-03-30T13:49:01.450928+0000","last_unstale":"2024-03-30T13:52:08.563326+0000","last_undegraded":"2024-03-30T13:52:08.563326+0000","last_fullsized":"2024-03-30T13:52:08.563326+0000","mapping_epoch":147,"log_start":"180'8300","ondisk_log_start":"180'8300","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'9448","last_scrub_stamp":"2024-03-30T13:51:02.091497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:51:02.091497+0000","objects_scrubbed":6714,"log_size":6340,"log_dups_size":0,"ondisk_log_size":6340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:19.157876+0000","scrub_duration":1970,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94371840,"num_objects":1440,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14640,"num_write_kb":514560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":303,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.469849+0000","last_change":"2024-03-30T13:50:59.155093+0000","last_active":"2024-03-30T13:52:02.469849+0000","last_peered":"2024-03-30T13:52:02.469849+0000","last_clean":"2024-03-30T13:52:02.469849+0000","last_became_active":"2024-03-30T13:42:51.331081+0000","last_became_peered":"2024-03-30T13:42:51.331081+0000","last_unstale":"2024-03-30T13:52:02.469849+0000","last_undegraded":"2024-03-30T13:52:02.469849+0000","last_fullsized":"2024-03-30T13:52:02.469849+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:59.154972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:12.433166+0000","last_clean_scrub_stamp":"2024-03-30T13:50:59.154972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:25.335507+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":559,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.472346+0000","last_change":"2024-03-30T13:51:03.160585+0000","last_active":"2024-03-30T13:52:02.472346+0000","last_peered":"2024-03-30T13:52:02.472346+0000","last_clean":"2024-03-30T13:52:02.472346+0000","last_became_active":"2024-03-30T13:42:34.988246+0000","last_became_peered":"2024-03-30T13:42:34.988246+0000","last_unstale":"2024-03-30T13:52:02.472346+0000","last_undegraded":"2024-03-30T13:52:02.472346+0000","last_fullsized":"2024-03-30T13:52:02.472346+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:51:03.160454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:51:03.160454+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:15.406710+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":300,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.460984+0000","last_change":"2024-03-30T13:50:05.402032+0000","last_active":"2024-03-30T13:52:02.460984+0000","last_peered":"2024-03-30T13:52:02.460984+0000","last_clean":"2024-03-30T13:52:02.460984+0000","last_became_active":"2024-03-30T13:42:45.309954+0000","last_became_peered":"2024-03-30T13:42:45.309954+0000","last_unstale":"2024-03-30T13:52:02.460984+0000","last_undegraded":"2024-03-30T13:52:02.460984+0000","last_fullsized":"2024-03-30T13:52:02.460984+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:05.401914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:50:05.401914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":453,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.484100+0000","last_change":"2024-03-30T13:52:01.369622+0000","last_active":"2024-03-30T13:52:02.484100+0000","last_peered":"2024-03-30T13:52:02.484100+0000","last_clean":"2024-03-30T13:52:02.484100+0000","last_became_active":"2024-03-30T13:42:47.502338+0000","last_became_peered":"2024-03-30T13:42:47.502338+0000","last_unstale":"2024-03-30T13:52:02.484100+0000","last_undegraded":"2024-03-30T13:52:02.484100+0000","last_fullsized":"2024-03-30T13:52:02.484100+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'16","last_scrub_stamp":"2024-03-30T13:52:01.369487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:01.369487+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:15.847088+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":301,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.476084+0000","last_change":"2024-03-30T13:50:08.517677+0000","last_active":"2024-03-30T13:52:02.476084+0000","last_peered":"2024-03-30T13:52:02.476084+0000","last_clean":"2024-03-30T13:50:05.495178+0000","last_became_active":"2024-03-30T13:50:08.517677+0000","last_became_peered":"2024-03-30T13:50:08.517677+0000","last_unstale":"2024-03-30T13:52:02.476084+0000","last_undegraded":"2024-03-30T13:52:02.476084+0000","last_fullsized":"2024-03-30T13:50:08.511555+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:25.787524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:25.787524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"196'14934","reported_seq":15046,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:08.913671+0000","last_change":"2024-03-30T13:50:08.514509+0000","last_active":"2024-03-30T13:52:08.913671+0000","last_peered":"2024-03-30T13:52:08.913671+0000","last_clean":"2024-03-30T13:50:07.503192+0000","last_became_active":"2024-03-30T13:50:08.514509+0000","last_became_peered":"2024-03-30T13:50:08.514509+0000","last_unstale":"2024-03-30T13:52:08.913671+0000","last_undegraded":"2024-03-30T13:50:08.507026+0000","last_fullsized":"2024-03-30T13:50:08.506909+0000","mapping_epoch":167,"log_start":"183'10000","ondisk_log_start":"183'10000","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:49:00.428679+0000","objects_scrubbed":0,"log_size":4934,"log_dups_size":0,"ondisk_log_size":4934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100794368,"num_objects":1538,"num_object_clones":0,"num_object_copies":4614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14934,"num_write_kb":527104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1538}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":304,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:03.346048+0000","last_change":"2024-03-30T13:52:03.346086+0000","last_active":"2024-03-30T13:52:03.346048+0000","last_peered":"2024-03-30T13:52:03.346048+0000","last_clean":"2024-03-30T13:52:03.346048+0000","last_became_active":"2024-03-30T13:42:51.335188+0000","last_became_peered":"2024-03-30T13:42:51.335188+0000","last_unstale":"2024-03-30T13:52:03.346048+0000","last_undegraded":"2024-03-30T13:52:03.346048+0000","last_fullsized":"2024-03-30T13:52:03.346048+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:03.345965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:03.345965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:06.636168+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":314,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:08.002782+0000","last_change":"2024-03-30T13:52:08.002816+0000","last_active":"2024-03-30T13:52:08.002782+0000","last_peered":"2024-03-30T13:52:08.002782+0000","last_clean":"2024-03-30T13:52:08.002782+0000","last_became_active":"2024-03-30T13:42:45.307905+0000","last_became_peered":"2024-03-30T13:42:45.307905+0000","last_unstale":"2024-03-30T13:52:08.002782+0000","last_undegraded":"2024-03-30T13:52:08.002782+0000","last_fullsized":"2024-03-30T13:52:08.002782+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:08.002706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:08.002706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:12.846644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":874,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:07.500317+0000","last_change":"2024-03-30T13:51:02.143007+0000","last_active":"2024-03-30T13:52:07.500317+0000","last_peered":"2024-03-30T13:52:07.500317+0000","last_clean":"2024-03-30T13:52:07.500317+0000","last_became_active":"2024-03-30T13:42:44.302343+0000","last_became_peered":"2024-03-30T13:42:44.302343+0000","last_unstale":"2024-03-30T13:52:07.500317+0000","last_undegraded":"2024-03-30T13:52:07.500317+0000","last_fullsized":"2024-03-30T13:52:07.500317+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-30T13:51:02.142275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:51:02.142275+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00126391,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":556,"num_read_kb":551,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"93'18","reported_seq":329,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:02.465014+0000","last_change":"2024-03-30T13:50:08.510256+0000","last_active":"2024-03-30T13:52:02.465014+0000","last_peered":"2024-03-30T13:52:02.465014+0000","last_clean":"2024-03-30T13:50:05.503358+0000","last_became_active":"2024-03-30T13:50:08.510256+0000","last_became_peered":"2024-03-30T13:50:08.510256+0000","last_unstale":"2024-03-30T13:52:02.465014+0000","last_undegraded":"2024-03-30T13:50:08.504536+0000","last_fullsized":"2024-03-30T13:50:08.504398+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'18","last_scrub_stamp":"2024-03-30T13:49:13.774982+0000","last_deep_scrub":"93'18","last_deep_scrub_stamp":"2024-03-30T13:47:28.872529+0000","last_clean_scrub_stamp":"2024-03-30T13:49:13.774982+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":299,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.470249+0000","last_change":"2024-03-30T13:51:08.978000+0000","last_active":"2024-03-30T13:52:02.470249+0000","last_peered":"2024-03-30T13:52:02.470249+0000","last_clean":"2024-03-30T13:52:02.470249+0000","last_became_active":"2024-03-30T13:42:49.322700+0000","last_became_peered":"2024-03-30T13:42:49.322700+0000","last_unstale":"2024-03-30T13:52:02.470249+0000","last_undegraded":"2024-03-30T13:52:02.470249+0000","last_fullsized":"2024-03-30T13:52:02.470249+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:08.977897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:08.977897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:36.468589+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"196'14832","reported_seq":14942,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:09.066851+0000","last_change":"2024-03-30T13:50:08.509534+0000","last_active":"2024-03-30T13:52:09.066851+0000","last_peered":"2024-03-30T13:52:09.066851+0000","last_clean":"2024-03-30T13:50:07.503222+0000","last_became_active":"2024-03-30T13:50:08.509534+0000","last_became_peered":"2024-03-30T13:50:08.509534+0000","last_unstale":"2024-03-30T13:52:09.066851+0000","last_undegraded":"2024-03-30T13:50:08.503986+0000","last_fullsized":"2024-03-30T13:50:08.503847+0000","mapping_epoch":167,"log_start":"183'9900","ondisk_log_start":"183'9900","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:49:00.428679+0000","objects_scrubbed":0,"log_size":4932,"log_dups_size":0,"ondisk_log_size":4932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98697216,"num_objects":1506,"num_object_clones":0,"num_object_copies":4518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14832,"num_write_kb":522816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1506}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":299,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.474437+0000","last_change":"2024-03-30T13:50:08.512648+0000","last_active":"2024-03-30T13:52:02.474437+0000","last_peered":"2024-03-30T13:52:02.474437+0000","last_clean":"2024-03-30T13:50:05.489180+0000","last_became_active":"2024-03-30T13:50:08.512648+0000","last_became_peered":"2024-03-30T13:50:08.512648+0000","last_unstale":"2024-03-30T13:52:02.474437+0000","last_undegraded":"2024-03-30T13:52:02.474437+0000","last_fullsized":"2024-03-30T13:50:08.505931+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:21.527712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:21.527712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:05.360498+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":311,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.472743+0000","last_change":"2024-03-30T13:50:08.517734+0000","last_active":"2024-03-30T13:52:02.472743+0000","last_peered":"2024-03-30T13:52:02.472743+0000","last_clean":"2024-03-30T13:50:05.490307+0000","last_became_active":"2024-03-30T13:50:08.517734+0000","last_became_peered":"2024-03-30T13:50:08.517734+0000","last_unstale":"2024-03-30T13:52:02.472743+0000","last_undegraded":"2024-03-30T13:52:02.472743+0000","last_fullsized":"2024-03-30T13:50:08.507406+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:19.525007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:45.975163+0000","last_clean_scrub_stamp":"2024-03-30T13:49:19.525007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:05.360542+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":338,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.461521+0000","last_change":"2024-03-30T13:50:59.350614+0000","last_active":"2024-03-30T13:52:02.461521+0000","last_peered":"2024-03-30T13:52:02.461521+0000","last_clean":"2024-03-30T13:52:02.461521+0000","last_became_active":"2024-03-30T13:42:47.501272+0000","last_became_peered":"2024-03-30T13:42:47.501272+0000","last_unstale":"2024-03-30T13:52:02.461521+0000","last_undegraded":"2024-03-30T13:52:02.461521+0000","last_fullsized":"2024-03-30T13:52:02.461521+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-30T13:50:59.350459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:50:59.350459+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:09.076654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":314,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.483286+0000","last_change":"2024-03-30T13:51:09.196116+0000","last_active":"2024-03-30T13:52:02.483286+0000","last_peered":"2024-03-30T13:52:02.483286+0000","last_clean":"2024-03-30T13:52:02.483286+0000","last_became_active":"2024-03-30T13:42:49.321520+0000","last_became_peered":"2024-03-30T13:42:49.321520+0000","last_unstale":"2024-03-30T13:52:02.483286+0000","last_undegraded":"2024-03-30T13:52:02.483286+0000","last_fullsized":"2024-03-30T13:52:02.483286+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:09.196024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:51.900305+0000","last_clean_scrub_stamp":"2024-03-30T13:51:09.196024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:37.584924+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"196'14988","reported_seq":15098,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:09.314345+0000","last_change":"2024-03-30T13:50:08.513477+0000","last_active":"2024-03-30T13:52:09.314345+0000","last_peered":"2024-03-30T13:52:09.314345+0000","last_clean":"2024-03-30T13:50:07.482534+0000","last_became_active":"2024-03-30T13:50:08.513477+0000","last_became_peered":"2024-03-30T13:50:08.513477+0000","last_unstale":"2024-03-30T13:52:09.314345+0000","last_undegraded":"2024-03-30T13:50:08.504066+0000","last_fullsized":"2024-03-30T13:50:08.503954+0000","mapping_epoch":167,"log_start":"181'9200","ondisk_log_start":"181'9200","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:49:00.428679+0000","objects_scrubbed":0,"log_size":5788,"log_dups_size":0,"ondisk_log_size":5788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:12.997829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93716480,"num_objects":1430,"num_object_clones":0,"num_object_copies":4290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14988,"num_write_kb":525376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":1430}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":297,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.462036+0000","last_change":"2024-03-30T13:50:08.510025+0000","last_active":"2024-03-30T13:52:02.462036+0000","last_peered":"2024-03-30T13:52:02.462036+0000","last_clean":"2024-03-30T13:50:05.498930+0000","last_became_active":"2024-03-30T13:50:08.510025+0000","last_became_peered":"2024-03-30T13:50:08.510025+0000","last_unstale":"2024-03-30T13:52:02.462036+0000","last_undegraded":"2024-03-30T13:52:02.462036+0000","last_fullsized":"2024-03-30T13:50:08.502478+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:28.946303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:41.142941+0000","last_clean_scrub_stamp":"2024-03-30T13:49:28.946303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":310,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.471353+0000","last_change":"2024-03-30T13:50:08.514920+0000","last_active":"2024-03-30T13:52:02.471353+0000","last_peered":"2024-03-30T13:52:02.471353+0000","last_clean":"2024-03-30T13:50:05.493549+0000","last_became_active":"2024-03-30T13:50:08.514920+0000","last_became_peered":"2024-03-30T13:50:08.514920+0000","last_unstale":"2024-03-30T13:52:02.471353+0000","last_undegraded":"2024-03-30T13:52:02.471353+0000","last_fullsized":"2024-03-30T13:50:08.504625+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:08.543531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:55.458779+0000","last_clean_scrub_stamp":"2024-03-30T13:49:08.543531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:03.115711+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"109'15","reported_seq":332,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.473932+0000","last_change":"2024-03-30T13:51:00.279278+0000","last_active":"2024-03-30T13:52:02.473932+0000","last_peered":"2024-03-30T13:52:02.473932+0000","last_clean":"2024-03-30T13:52:02.473932+0000","last_became_active":"2024-03-30T13:42:47.321433+0000","last_became_peered":"2024-03-30T13:42:47.321433+0000","last_unstale":"2024-03-30T13:52:02.473932+0000","last_undegraded":"2024-03-30T13:52:02.473932+0000","last_fullsized":"2024-03-30T13:52:02.473932+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'15","last_scrub_stamp":"2024-03-30T13:51:00.279148+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-03-30T13:45:50.945837+0000","last_clean_scrub_stamp":"2024-03-30T13:51:00.279148+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:03.034933+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":427,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.461677+0000","last_change":"2024-03-30T13:51:17.257600+0000","last_active":"2024-03-30T13:52:02.461677+0000","last_peered":"2024-03-30T13:52:02.461677+0000","last_clean":"2024-03-30T13:52:02.461677+0000","last_became_active":"2024-03-30T13:42:49.325533+0000","last_became_peered":"2024-03-30T13:42:49.325533+0000","last_unstale":"2024-03-30T13:52:02.461677+0000","last_undegraded":"2024-03-30T13:52:02.461677+0000","last_fullsized":"2024-03-30T13:52:02.461677+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:51:17.257464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:17.257464+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:41.745294+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"196'14513","reported_seq":14626,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:08.542385+0000","last_change":"2024-03-30T13:50:08.515012+0000","last_active":"2024-03-30T13:52:08.542385+0000","last_peered":"2024-03-30T13:52:08.542385+0000","last_clean":"2024-03-30T13:50:07.483605+0000","last_became_active":"2024-03-30T13:50:08.515012+0000","last_became_peered":"2024-03-30T13:50:08.515012+0000","last_unstale":"2024-03-30T13:52:08.542385+0000","last_undegraded":"2024-03-30T13:50:08.506241+0000","last_fullsized":"2024-03-30T13:50:08.506127+0000","mapping_epoch":167,"log_start":"182'9100","ondisk_log_start":"182'9100","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:49:00.428679+0000","objects_scrubbed":0,"log_size":5413,"log_dups_size":0,"ondisk_log_size":5413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99090456,"num_objects":1513,"num_object_clones":0,"num_object_copies":4539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14513,"num_write_kb":512769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":1513}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":289,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.472925+0000","last_change":"2024-03-30T13:50:08.513975+0000","last_active":"2024-03-30T13:52:02.472925+0000","last_peered":"2024-03-30T13:52:02.472925+0000","last_clean":"2024-03-30T13:50:05.495405+0000","last_became_active":"2024-03-30T13:50:08.513975+0000","last_became_peered":"2024-03-30T13:50:08.513975+0000","last_unstale":"2024-03-30T13:52:02.472925+0000","last_undegraded":"2024-03-30T13:52:02.472925+0000","last_fullsized":"2024-03-30T13:50:08.507347+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:21.782290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:31.900908+0000","last_clean_scrub_stamp":"2024-03-30T13:49:21.782290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":315,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.465227+0000","last_change":"2024-03-30T13:51:12.127429+0000","last_active":"2024-03-30T13:52:02.465227+0000","last_peered":"2024-03-30T13:52:02.465227+0000","last_clean":"2024-03-30T13:52:02.465227+0000","last_became_active":"2024-03-30T13:42:45.309886+0000","last_became_peered":"2024-03-30T13:42:45.309886+0000","last_unstale":"2024-03-30T13:52:02.465227+0000","last_undegraded":"2024-03-30T13:52:02.465227+0000","last_fullsized":"2024-03-30T13:52:02.465227+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:51:12.127305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:12.127305+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:34.020036+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"91'21","reported_seq":341,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.463045+0000","last_change":"2024-03-30T13:50:59.187414+0000","last_active":"2024-03-30T13:52:02.463045+0000","last_peered":"2024-03-30T13:52:02.463045+0000","last_clean":"2024-03-30T13:52:02.463045+0000","last_became_active":"2024-03-30T13:42:47.314184+0000","last_became_peered":"2024-03-30T13:42:47.314184+0000","last_unstale":"2024-03-30T13:52:02.463045+0000","last_undegraded":"2024-03-30T13:52:02.463045+0000","last_fullsized":"2024-03-30T13:52:02.463045+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'21","last_scrub_stamp":"2024-03-30T13:50:59.187324+0000","last_deep_scrub":"91'21","last_deep_scrub_stamp":"2024-03-30T13:44:05.917064+0000","last_clean_scrub_stamp":"2024-03-30T13:50:59.187324+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:13.439993+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":413,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.473192+0000","last_change":"2024-03-30T13:50:57.701974+0000","last_active":"2024-03-30T13:52:02.473192+0000","last_peered":"2024-03-30T13:52:02.473192+0000","last_clean":"2024-03-30T13:52:02.473192+0000","last_became_active":"2024-03-30T13:42:49.321423+0000","last_became_peered":"2024-03-30T13:42:49.321423+0000","last_unstale":"2024-03-30T13:52:02.473192+0000","last_undegraded":"2024-03-30T13:52:02.473192+0000","last_fullsized":"2024-03-30T13:52:02.473192+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:50:57.701778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:50:57.701778+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"196'14750","reported_seq":14930,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:09.317938+0000","last_change":"2024-03-30T13:51:07.821191+0000","last_active":"2024-03-30T13:52:09.317938+0000","last_peered":"2024-03-30T13:52:09.317938+0000","last_clean":"2024-03-30T13:52:09.317938+0000","last_became_active":"2024-03-30T13:49:01.523003+0000","last_became_peered":"2024-03-30T13:49:01.523003+0000","last_unstale":"2024-03-30T13:52:09.317938+0000","last_undegraded":"2024-03-30T13:52:09.317938+0000","last_fullsized":"2024-03-30T13:52:09.317938+0000","mapping_epoch":147,"log_start":"180'8900","ondisk_log_start":"180'8900","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'9942","last_scrub_stamp":"2024-03-30T13:51:07.820988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:51:07.820988+0000","objects_scrubbed":6427,"log_size":5850,"log_dups_size":0,"ondisk_log_size":5850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:15.476236+0000","scrub_duration":2640,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100270080,"num_objects":1530,"num_object_clones":0,"num_object_copies":4590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14750,"num_write_kb":520960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":292,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.463492+0000","last_change":"2024-03-30T13:50:08.504411+0000","last_active":"2024-03-30T13:52:02.463492+0000","last_peered":"2024-03-30T13:52:02.463492+0000","last_clean":"2024-03-30T13:50:05.489944+0000","last_became_active":"2024-03-30T13:50:08.504411+0000","last_became_peered":"2024-03-30T13:50:08.504411+0000","last_unstale":"2024-03-30T13:52:02.463492+0000","last_undegraded":"2024-03-30T13:52:02.463492+0000","last_fullsized":"2024-03-30T13:50:08.501823+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:10.216387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:10.216387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:04.016077+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":308,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.462973+0000","last_change":"2024-03-30T13:51:24.984126+0000","last_active":"2024-03-30T13:52:02.462973+0000","last_peered":"2024-03-30T13:52:02.462973+0000","last_clean":"2024-03-30T13:52:02.462973+0000","last_became_active":"2024-03-30T13:42:45.314533+0000","last_became_peered":"2024-03-30T13:42:45.314533+0000","last_unstale":"2024-03-30T13:52:02.462973+0000","last_undegraded":"2024-03-30T13:52:02.462973+0000","last_fullsized":"2024-03-30T13:52:02.462973+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:24.983992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:24.171036+0000","last_clean_scrub_stamp":"2024-03-30T13:51:24.983992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:52.450102+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":336,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.473043+0000","last_change":"2024-03-30T13:51:16.834761+0000","last_active":"2024-03-30T13:52:02.473043+0000","last_peered":"2024-03-30T13:52:02.473043+0000","last_clean":"2024-03-30T13:52:02.473043+0000","last_became_active":"2024-03-30T13:42:47.502708+0000","last_became_peered":"2024-03-30T13:42:47.502708+0000","last_unstale":"2024-03-30T13:52:02.473043+0000","last_undegraded":"2024-03-30T13:52:02.473043+0000","last_fullsized":"2024-03-30T13:52:02.473043+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-03-30T13:51:16.834525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:16.834525+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:36.215406+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":301,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.462901+0000","last_change":"2024-03-30T13:51:09.247429+0000","last_active":"2024-03-30T13:52:02.462901+0000","last_peered":"2024-03-30T13:52:02.462901+0000","last_clean":"2024-03-30T13:52:02.462901+0000","last_became_active":"2024-03-30T13:42:49.324022+0000","last_became_peered":"2024-03-30T13:42:49.324022+0000","last_unstale":"2024-03-30T13:52:02.462901+0000","last_undegraded":"2024-03-30T13:52:02.462901+0000","last_fullsized":"2024-03-30T13:52:02.462901+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:09.247282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:09.247282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:39.099111+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"196'14808","reported_seq":15014,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:08.901604+0000","last_change":"2024-03-30T13:52:04.931308+0000","last_active":"2024-03-30T13:52:08.901604+0000","last_peered":"2024-03-30T13:52:08.901604+0000","last_clean":"2024-03-30T13:52:08.901604+0000","last_became_active":"2024-03-30T13:49:01.453308+0000","last_became_peered":"2024-03-30T13:49:01.453308+0000","last_unstale":"2024-03-30T13:52:08.901604+0000","last_undegraded":"2024-03-30T13:52:08.901604+0000","last_fullsized":"2024-03-30T13:52:08.901604+0000","mapping_epoch":147,"log_start":"183'9800","ondisk_log_start":"183'9800","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'14440","last_scrub_stamp":"2024-03-30T13:52:04.930721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:52:04.930721+0000","objects_scrubbed":1945,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:18.160723+0000","scrub_duration":618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102236160,"num_objects":1560,"num_object_clones":0,"num_object_copies":4680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14808,"num_write_kb":523776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":295,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.471080+0000","last_change":"2024-03-30T13:50:22.770010+0000","last_active":"2024-03-30T13:52:02.471080+0000","last_peered":"2024-03-30T13:52:02.471080+0000","last_clean":"2024-03-30T13:52:02.471080+0000","last_became_active":"2024-03-30T13:42:51.329026+0000","last_became_peered":"2024-03-30T13:42:51.329026+0000","last_unstale":"2024-03-30T13:52:02.471080+0000","last_undegraded":"2024-03-30T13:52:02.471080+0000","last_fullsized":"2024-03-30T13:52:02.471080+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:22.769749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:50:22.769749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:03.115892+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"186'9","reported_seq":431,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.464193+0000","last_change":"2024-03-30T13:51:15.146386+0000","last_active":"2024-03-30T13:52:02.464193+0000","last_peered":"2024-03-30T13:52:02.464193+0000","last_clean":"2024-03-30T13:52:02.464193+0000","last_became_active":"2024-03-30T13:42:49.321790+0000","last_became_peered":"2024-03-30T13:42:49.321790+0000","last_unstale":"2024-03-30T13:52:02.464193+0000","last_undegraded":"2024-03-30T13:52:02.464193+0000","last_fullsized":"2024-03-30T13:52:02.464193+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-30T13:51:15.146255+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-30T13:51:15.146255+0000","last_clean_scrub_stamp":"2024-03-30T13:51:15.146255+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:35.921845+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":300,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.472870+0000","last_change":"2024-03-30T13:51:06.667727+0000","last_active":"2024-03-30T13:52:02.472870+0000","last_peered":"2024-03-30T13:52:02.472870+0000","last_clean":"2024-03-30T13:52:02.472870+0000","last_became_active":"2024-03-30T13:42:45.315319+0000","last_became_peered":"2024-03-30T13:42:45.315319+0000","last_unstale":"2024-03-30T13:52:02.472870+0000","last_undegraded":"2024-03-30T13:52:02.472870+0000","last_fullsized":"2024-03-30T13:52:02.472870+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:06.667621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:06.667621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:23.994913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"95'22","reported_seq":344,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.478958+0000","last_change":"2024-03-30T13:51:24.149732+0000","last_active":"2024-03-30T13:52:02.478958+0000","last_peered":"2024-03-30T13:52:02.478958+0000","last_clean":"2024-03-30T13:52:02.478958+0000","last_became_active":"2024-03-30T13:42:47.324350+0000","last_became_peered":"2024-03-30T13:42:47.324350+0000","last_unstale":"2024-03-30T13:52:02.478958+0000","last_undegraded":"2024-03-30T13:52:02.478958+0000","last_fullsized":"2024-03-30T13:52:02.478958+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'22","last_scrub_stamp":"2024-03-30T13:51:24.149237+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:28.831527+0000","last_clean_scrub_stamp":"2024-03-30T13:51:24.149237+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:47.643117+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"196'14893","reported_seq":15082,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:08.493355+0000","last_change":"2024-03-30T13:50:58.228849+0000","last_active":"2024-03-30T13:52:08.493355+0000","last_peered":"2024-03-30T13:52:08.493355+0000","last_clean":"2024-03-30T13:52:08.493355+0000","last_became_active":"2024-03-30T13:49:01.450057+0000","last_became_peered":"2024-03-30T13:49:01.450057+0000","last_unstale":"2024-03-30T13:52:08.493355+0000","last_undegraded":"2024-03-30T13:52:08.493355+0000","last_fullsized":"2024-03-30T13:52:08.493355+0000","mapping_epoch":147,"log_start":"184'10100","ondisk_log_start":"184'10100","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'9381","last_scrub_stamp":"2024-03-30T13:50:58.228675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:50:58.228675+0000","objects_scrubbed":7120,"log_size":4793,"log_dups_size":0,"ondisk_log_size":4793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:17.135717+0000","scrub_duration":1990,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101253120,"num_objects":1545,"num_object_clones":0,"num_object_copies":4635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14893,"num_write_kb":526016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":288,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.470060+0000","last_change":"2024-03-30T13:51:10.683723+0000","last_active":"2024-03-30T13:52:02.470060+0000","last_peered":"2024-03-30T13:52:02.470060+0000","last_clean":"2024-03-30T13:52:02.470060+0000","last_became_active":"2024-03-30T13:42:51.371369+0000","last_became_peered":"2024-03-30T13:42:51.371369+0000","last_unstale":"2024-03-30T13:52:02.470060+0000","last_undegraded":"2024-03-30T13:52:02.470060+0000","last_fullsized":"2024-03-30T13:52:02.470060+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:10.683596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:10.683596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:11.668157+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":307,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.470157+0000","last_change":"2024-03-30T13:51:21.087839+0000","last_active":"2024-03-30T13:52:02.470157+0000","last_peered":"2024-03-30T13:52:02.470157+0000","last_clean":"2024-03-30T13:52:02.470157+0000","last_became_active":"2024-03-30T13:42:49.323319+0000","last_became_peered":"2024-03-30T13:42:49.323319+0000","last_unstale":"2024-03-30T13:52:02.470157+0000","last_undegraded":"2024-03-30T13:52:02.470157+0000","last_fullsized":"2024-03-30T13:52:02.470157+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:21.087732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:52.603772+0000","last_clean_scrub_stamp":"2024-03-30T13:51:21.087732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:26.664698+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":304,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.462953+0000","last_change":"2024-03-30T13:50:08.509331+0000","last_active":"2024-03-30T13:52:02.462953+0000","last_peered":"2024-03-30T13:52:02.462953+0000","last_clean":"2024-03-30T13:50:05.489382+0000","last_became_active":"2024-03-30T13:50:08.509331+0000","last_became_peered":"2024-03-30T13:50:08.509331+0000","last_unstale":"2024-03-30T13:52:02.462953+0000","last_undegraded":"2024-03-30T13:52:02.462953+0000","last_fullsized":"2024-03-30T13:50:08.501285+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:15.107879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:48:15.107879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:04.016108+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"72'11","reported_seq":332,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.463912+0000","last_change":"2024-03-30T13:50:58.241795+0000","last_active":"2024-03-30T13:52:02.463912+0000","last_peered":"2024-03-30T13:52:02.463912+0000","last_clean":"2024-03-30T13:52:02.463912+0000","last_became_active":"2024-03-30T13:42:47.315667+0000","last_became_peered":"2024-03-30T13:42:47.315667+0000","last_unstale":"2024-03-30T13:52:02.463912+0000","last_undegraded":"2024-03-30T13:52:02.463912+0000","last_fullsized":"2024-03-30T13:52:02.463912+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:50:58.241649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:50:58.241649+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"196'14498","reported_seq":14610,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:08.900786+0000","last_change":"2024-03-30T13:50:08.511322+0000","last_active":"2024-03-30T13:52:08.900786+0000","last_peered":"2024-03-30T13:52:08.900786+0000","last_clean":"2024-03-30T13:50:07.457630+0000","last_became_active":"2024-03-30T13:50:08.511322+0000","last_became_peered":"2024-03-30T13:50:08.511322+0000","last_unstale":"2024-03-30T13:52:08.900786+0000","last_undegraded":"2024-03-30T13:50:08.505556+0000","last_fullsized":"2024-03-30T13:50:08.505411+0000","mapping_epoch":167,"log_start":"183'9500","ondisk_log_start":"183'9500","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:49:00.428679+0000","objects_scrubbed":0,"log_size":4998,"log_dups_size":0,"ondisk_log_size":4998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97648640,"num_objects":1490,"num_object_clones":0,"num_object_copies":4470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14498,"num_write_kb":511616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1490}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":79,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.462634+0000","last_change":"2024-03-30T13:50:08.511676+0000","last_active":"2024-03-30T13:52:02.462634+0000","last_peered":"2024-03-30T13:52:02.462634+0000","last_clean":"2024-03-30T13:50:05.504989+0000","last_became_active":"2024-03-30T13:50:08.511676+0000","last_became_peered":"2024-03-30T13:50:08.511676+0000","last_unstale":"2024-03-30T13:52:02.462634+0000","last_undegraded":"2024-03-30T13:52:02.462634+0000","last_fullsized":"2024-03-30T13:50:08.504157+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:09.357378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:09.357378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:03.929500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":308,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.473523+0000","last_change":"2024-03-30T13:50:57.267042+0000","last_active":"2024-03-30T13:52:02.473523+0000","last_peered":"2024-03-30T13:52:02.473523+0000","last_clean":"2024-03-30T13:52:02.473523+0000","last_became_active":"2024-03-30T13:42:49.321360+0000","last_became_peered":"2024-03-30T13:42:49.321360+0000","last_unstale":"2024-03-30T13:52:02.473523+0000","last_undegraded":"2024-03-30T13:52:02.473523+0000","last_fullsized":"2024-03-30T13:52:02.473523+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:57.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:50:57.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":322,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.483049+0000","last_change":"2024-03-30T13:51:21.196578+0000","last_active":"2024-03-30T13:52:02.483049+0000","last_peered":"2024-03-30T13:52:02.483049+0000","last_clean":"2024-03-30T13:52:02.483049+0000","last_became_active":"2024-03-30T13:42:45.315251+0000","last_became_peered":"2024-03-30T13:42:45.315251+0000","last_unstale":"2024-03-30T13:52:02.483049+0000","last_undegraded":"2024-03-30T13:52:02.483049+0000","last_fullsized":"2024-03-30T13:52:02.483049+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:21.196457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:21.196457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:24.534984+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":328,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:02.464573+0000","last_change":"2024-03-30T13:50:08.508030+0000","last_active":"2024-03-30T13:52:02.464573+0000","last_peered":"2024-03-30T13:52:02.464573+0000","last_clean":"2024-03-30T13:50:05.491159+0000","last_became_active":"2024-03-30T13:50:08.508030+0000","last_became_peered":"2024-03-30T13:50:08.508030+0000","last_unstale":"2024-03-30T13:52:02.464573+0000","last_undegraded":"2024-03-30T13:50:08.502958+0000","last_fullsized":"2024-03-30T13:50:08.502815+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-03-30T13:48:27.096261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:27.096261+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:04.016150+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"196'14946","reported_seq":15118,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:09.059770+0000","last_change":"2024-03-30T13:51:20.782899+0000","last_active":"2024-03-30T13:52:09.059770+0000","last_peered":"2024-03-30T13:52:09.059770+0000","last_clean":"2024-03-30T13:52:09.059770+0000","last_became_active":"2024-03-30T13:49:01.450507+0000","last_became_peered":"2024-03-30T13:49:01.450507+0000","last_unstale":"2024-03-30T13:52:09.059770+0000","last_undegraded":"2024-03-30T13:52:09.059770+0000","last_fullsized":"2024-03-30T13:52:09.059770+0000","mapping_epoch":147,"log_start":"183'10100","ondisk_log_start":"183'10100","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'11079","last_scrub_stamp":"2024-03-30T13:51:20.782636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:00.428679+0000","last_clean_scrub_stamp":"2024-03-30T13:51:20.782636+0000","objects_scrubbed":5500,"log_size":4846,"log_dups_size":0,"ondisk_log_size":4846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:47.960026+0000","scrub_duration":1945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102891520,"num_objects":1570,"num_object_clones":0,"num_object_copies":4710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14946,"num_write_kb":528512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":292,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-30T13:52:02.464143+0000","last_change":"2024-03-30T13:50:08.511010+0000","last_active":"2024-03-30T13:52:02.464143+0000","last_peered":"2024-03-30T13:52:02.464143+0000","last_clean":"2024-03-30T13:50:05.490535+0000","last_became_active":"2024-03-30T13:50:08.511010+0000","last_became_peered":"2024-03-30T13:50:08.511010+0000","last_unstale":"2024-03-30T13:52:02.464143+0000","last_undegraded":"2024-03-30T13:52:02.464143+0000","last_fullsized":"2024-03-30T13:50:08.503118+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:06.154805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:06.154805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:04.016163+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"168'10","reported_seq":134,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:02.477400+0000","last_change":"2024-03-30T13:50:08.526935+0000","last_active":"2024-03-30T13:52:02.477400+0000","last_peered":"2024-03-30T13:52:02.477400+0000","last_clean":"2024-03-30T13:50:05.495636+0000","last_became_active":"2024-03-30T13:50:08.526935+0000","last_became_peered":"2024-03-30T13:50:08.526935+0000","last_unstale":"2024-03-30T13:52:02.477400+0000","last_undegraded":"2024-03-30T13:50:08.509321+0000","last_fullsized":"2024-03-30T13:50:08.509210+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:49:36.397622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:36.397622+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":308,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.468894+0000","last_change":"2024-03-30T13:51:58.939856+0000","last_active":"2024-03-30T13:52:02.468894+0000","last_peered":"2024-03-30T13:52:02.468894+0000","last_clean":"2024-03-30T13:52:02.468894+0000","last_became_active":"2024-03-30T13:42:45.309265+0000","last_became_peered":"2024-03-30T13:42:45.309265+0000","last_unstale":"2024-03-30T13:52:02.468894+0000","last_undegraded":"2024-03-30T13:52:02.468894+0000","last_fullsized":"2024-03-30T13:52:02.468894+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:58.939556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:58.939556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:03.912680+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":328,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:52:02.464359+0000","last_change":"2024-03-30T13:51:59.022459+0000","last_active":"2024-03-30T13:52:02.464359+0000","last_peered":"2024-03-30T13:52:02.464359+0000","last_clean":"2024-03-30T13:52:02.464359+0000","last_became_active":"2024-03-30T13:42:47.313026+0000","last_became_peered":"2024-03-30T13:42:47.313026+0000","last_unstale":"2024-03-30T13:52:02.464359+0000","last_undegraded":"2024-03-30T13:52:02.464359+0000","last_fullsized":"2024-03-30T13:52:02.464359+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-30T13:51:59.022304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:59.022304+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:13.924563+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":1584332824,"num_objects":24176,"num_object_clones":0,"num_object_copies":72528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24176,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":236528,"num_write_kb":8342465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6254698496,"data_stored":6254690352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85737,"ondisk_log_size":85737,"up":42,"acting":42,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":556,"num_read_kb":551,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":88,"acting":88,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":447,"ondisk_log_size":447,"up":88,"acting":88,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":87,"acting":87,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":84,"acting":84,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103286,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":768004,"kb_used_data":524412,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":92977148,"statfs":{"total":95995035648,"available":95208599552,"internally_reserved":0,"allocated":536997888,"data_stored":536454120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":249428426},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397626,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":723372,"kb_used_data":497828,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":93021780,"statfs":{"total":95995035648,"available":95254302720,"internally_reserved":0,"allocated":509775872,"data_stored":509255604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":230947270},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574996,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1050080,"kb_used_data":707800,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92695072,"statfs":{"total":95995035648,"available":94919753728,"internally_reserved":0,"allocated":724787200,"data_stored":724215090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738521,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":912324,"kb_used_data":574332,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92832828,"statfs":{"total":95995035648,"available":95060815872,"internally_reserved":0,"allocated":588115968,"data_stored":587572780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411446,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2213288,"kb_used_data":2015520,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":91531864,"statfs":{"total":95995035648,"available":93728628736,"internally_reserved":0,"allocated":2063892480,"data_stored":2063509110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182538,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902280,"kb_used_data":582336,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92842872,"statfs":{"total":95995035648,"available":95071100928,"internally_reserved":0,"allocated":596312064,"data_stored":595763458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":327612871},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953604,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":777928,"kb_used_data":507904,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92967224,"statfs":{"total":95995035648,"available":95198437376,"internally_reserved":0,"allocated":520093696,"data_stored":519545215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691967,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1075512,"kb_used_data":809712,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92669640,"statfs":{"total":95995035648,"available":94893711360,"internally_reserved":0,"allocated":829145088,"data_stored":828613708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":575537152,"data_stored":575537152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":700383232,"data_stored":700383232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2068250624,"data_stored":2068250624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593494016,"data_stored":593494016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":502267904,"data_stored":502267904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":807604224,"data_stored":807600152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":498597888,"data_stored":498597888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":508563456,"data_stored":508559384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:52:14.623 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-30T13:52:14.623 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool set .mgr pg_num 11 2024-03-30T13:52:14.644 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:14.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:14.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:14.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:14 smithi043 ceph-mon[29963]: 3.2 deep-scrub starts 2024-03-30T13:52:14.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:14 smithi043 ceph-mon[29963]: 3.2 deep-scrub ok 2024-03-30T13:52:14.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:14 smithi043 ceph-mon[29963]: pgmap v384: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s; 9058/73122 objects degraded (12.388%) 2024-03-30T13:52:14.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:14 smithi043 ceph-mon[29963]: 6.d scrub starts 2024-03-30T13:52:14.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:14 smithi043 ceph-mon[29963]: 6.d scrub ok 2024-03-30T13:52:14.697 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:14 smithi043 ceph-mon[42099]: 3.2 deep-scrub starts 2024-03-30T13:52:14.697 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:14 smithi043 ceph-mon[42099]: 3.2 deep-scrub ok 2024-03-30T13:52:14.697 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:14 smithi043 ceph-mon[42099]: pgmap v384: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s; 9058/73122 objects degraded (12.388%) 2024-03-30T13:52:14.697 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:14 smithi043 ceph-mon[42099]: 6.d scrub starts 2024-03-30T13:52:14.697 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:14 smithi043 ceph-mon[42099]: 6.d scrub ok 2024-03-30T13:52:14.747 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:14.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:14.799 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:14.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:14.900 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:14.933 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:14.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:14.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:14.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:14 smithi113 ceph-mon[34341]: 3.2 deep-scrub starts 2024-03-30T13:52:14.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:14 smithi113 ceph-mon[34341]: 3.2 deep-scrub ok 2024-03-30T13:52:14.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:14 smithi113 ceph-mon[34341]: pgmap v384: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s; 9058/73122 objects degraded (12.388%) 2024-03-30T13:52:14.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:14 smithi113 ceph-mon[34341]: 6.d scrub starts 2024-03-30T13:52:14.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:14 smithi113 ceph-mon[34341]: 6.d scrub ok 2024-03-30T13:52:15.036 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:15.061 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:15.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:15.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:15.163 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:15.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:15.205 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:15.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:15.306 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:15.339 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:15.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:15.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:15.441 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:15.476 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:15.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:15.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:15.577 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:15.601 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:15.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:15.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:15.702 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:15.734 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:15.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:15.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:15.737 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:15 smithi043 ceph-mon[29963]: 3.14 scrub starts 2024-03-30T13:52:15.737 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:15 smithi043 ceph-mon[29963]: 3.14 scrub ok 2024-03-30T13:52:15.738 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:15 smithi043 ceph-mon[29963]: 4.17 scrub starts 2024-03-30T13:52:15.738 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:15 smithi043 ceph-mon[29963]: 4.17 scrub ok 2024-03-30T13:52:15.738 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:15 smithi043 ceph-mon[29963]: 4.2 scrub starts 2024-03-30T13:52:15.738 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:15 smithi043 ceph-mon[29963]: 4.2 scrub ok 2024-03-30T13:52:15.738 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:15 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2228412063' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:52:15.738 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:15 smithi043 ceph-mon[42099]: 3.14 scrub starts 2024-03-30T13:52:15.738 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:15 smithi043 ceph-mon[42099]: 3.14 scrub ok 2024-03-30T13:52:15.738 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:15 smithi043 ceph-mon[42099]: 4.17 scrub starts 2024-03-30T13:52:15.738 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:15 smithi043 ceph-mon[42099]: 4.17 scrub ok 2024-03-30T13:52:15.738 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:15 smithi043 ceph-mon[42099]: 4.2 scrub starts 2024-03-30T13:52:15.738 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:15 smithi043 ceph-mon[42099]: 4.2 scrub ok 2024-03-30T13:52:15.739 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:15 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2228412063' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:52:15.836 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:15.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:15.863 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:15.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:15.963 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:15.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:15 smithi113 ceph-mon[34341]: 3.14 scrub starts 2024-03-30T13:52:15.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:15 smithi113 ceph-mon[34341]: 3.14 scrub ok 2024-03-30T13:52:15.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:15 smithi113 ceph-mon[34341]: 4.17 scrub starts 2024-03-30T13:52:15.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:15 smithi113 ceph-mon[34341]: 4.17 scrub ok 2024-03-30T13:52:15.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:15 smithi113 ceph-mon[34341]: 4.2 scrub starts 2024-03-30T13:52:15.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:15 smithi113 ceph-mon[34341]: 4.2 scrub ok 2024-03-30T13:52:15.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:15 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2228412063' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:52:15.995 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:15.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:15.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:16.096 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:16.132 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:16.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:16.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:16.234 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:16.258 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:16.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:16.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:16.359 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:16.393 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:16.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:16.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:16.397 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:52:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:52:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:52:16.495 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:16.535 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:16.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:16.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:16.637 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:16.670 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:16.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:16.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:16.771 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:16.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:16.809 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:16.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:16.871 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:52:16.872 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:52:16.872 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:52:16.872 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:52:16.872 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:52:16.872 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:52:16.872 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:52:16.872 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:52:16.873 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:52:16.873 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:52:16.873 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:52:16.873 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:52:16.873 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:52:16.873 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:52:16.873 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:52:16.910 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:16.938 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:16.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:16.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:16.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:16 smithi113 ceph-mon[34341]: 4.3 scrub starts 2024-03-30T13:52:16.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:16 smithi113 ceph-mon[34341]: 4.3 scrub ok 2024-03-30T13:52:16.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:16 smithi113 ceph-mon[34341]: pgmap v385: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.1 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s; 6433/52188 objects degraded (12.327%) 2024-03-30T13:52:16.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:16 smithi113 ceph-mon[34341]: 5.13 scrub starts 2024-03-30T13:52:16.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:16 smithi113 ceph-mon[34341]: 5.13 scrub ok 2024-03-30T13:52:16.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:16 smithi113 ceph-mon[34341]: 5.15 scrub starts 2024-03-30T13:52:16.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:16 smithi113 ceph-mon[34341]: 5.15 scrub ok 2024-03-30T13:52:17.040 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:17.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:17.077 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:17.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:17.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:16 smithi043 ceph-mon[29963]: 4.3 scrub starts 2024-03-30T13:52:17.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:16 smithi043 ceph-mon[29963]: 4.3 scrub ok 2024-03-30T13:52:17.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:16 smithi043 ceph-mon[29963]: pgmap v385: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.1 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s; 6433/52188 objects degraded (12.327%) 2024-03-30T13:52:17.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:16 smithi043 ceph-mon[29963]: 5.13 scrub starts 2024-03-30T13:52:17.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:16 smithi043 ceph-mon[29963]: 5.13 scrub ok 2024-03-30T13:52:17.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:16 smithi043 ceph-mon[29963]: 5.15 scrub starts 2024-03-30T13:52:17.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:16 smithi043 ceph-mon[29963]: 5.15 scrub ok 2024-03-30T13:52:17.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:16 smithi043 ceph-mon[42099]: 4.3 scrub starts 2024-03-30T13:52:17.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:16 smithi043 ceph-mon[42099]: 4.3 scrub ok 2024-03-30T13:52:17.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:16 smithi043 ceph-mon[42099]: pgmap v385: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.1 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s; 6433/52188 objects degraded (12.327%) 2024-03-30T13:52:17.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:16 smithi043 ceph-mon[42099]: 5.13 scrub starts 2024-03-30T13:52:17.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:16 smithi043 ceph-mon[42099]: 5.13 scrub ok 2024-03-30T13:52:17.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:16 smithi043 ceph-mon[42099]: 5.15 scrub starts 2024-03-30T13:52:17.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:16 smithi043 ceph-mon[42099]: 5.15 scrub ok 2024-03-30T13:52:17.178 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:17.201 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:17.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:17.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:17.303 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:17.339 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:17.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:17.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:17.441 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:17.465 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:17.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:17.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:17.567 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:17.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:17.607 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:17.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:17.710 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:17.748 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:17.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:17.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:17.849 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:17.884 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:17.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:17.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:17.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:17 smithi113 ceph-mon[34341]: 3.1b scrub starts 2024-03-30T13:52:17.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:17 smithi113 ceph-mon[34341]: 3.1b scrub ok 2024-03-30T13:52:17.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:17 smithi113 ceph-mon[34341]: 5.11 scrub starts 2024-03-30T13:52:17.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:17 smithi113 ceph-mon[34341]: 5.11 scrub ok 2024-03-30T13:52:17.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:17 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 6433/52188 objects degraded (12.327%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:17.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:17 smithi113 ceph-mon[34341]: osdmap e200: 8 total, 7 up, 8 in 2024-03-30T13:52:17.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:17 smithi113 ceph-mon[34341]: 5.19 deep-scrub starts 2024-03-30T13:52:17.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:17 smithi113 ceph-mon[34341]: 5.19 deep-scrub ok 2024-03-30T13:52:17.986 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:18.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:18.025 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:18.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:18.126 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:18.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:17 smithi043 ceph-mon[29963]: 3.1b scrub starts 2024-03-30T13:52:18.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:17 smithi043 ceph-mon[29963]: 3.1b scrub ok 2024-03-30T13:52:18.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:17 smithi043 ceph-mon[29963]: 5.11 scrub starts 2024-03-30T13:52:18.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:17 smithi043 ceph-mon[29963]: 5.11 scrub ok 2024-03-30T13:52:18.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:17 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 6433/52188 objects degraded (12.327%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:18.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:17 smithi043 ceph-mon[29963]: osdmap e200: 8 total, 7 up, 8 in 2024-03-30T13:52:18.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:17 smithi043 ceph-mon[29963]: 5.19 deep-scrub starts 2024-03-30T13:52:18.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:17 smithi043 ceph-mon[29963]: 5.19 deep-scrub ok 2024-03-30T13:52:18.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:17 smithi043 ceph-mon[42099]: 3.1b scrub starts 2024-03-30T13:52:18.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:17 smithi043 ceph-mon[42099]: 3.1b scrub ok 2024-03-30T13:52:18.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:17 smithi043 ceph-mon[42099]: 5.11 scrub starts 2024-03-30T13:52:18.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:17 smithi043 ceph-mon[42099]: 5.11 scrub ok 2024-03-30T13:52:18.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:17 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 6433/52188 objects degraded (12.327%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:18.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:17 smithi043 ceph-mon[42099]: osdmap e200: 8 total, 7 up, 8 in 2024-03-30T13:52:18.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:17 smithi043 ceph-mon[42099]: 5.19 deep-scrub starts 2024-03-30T13:52:18.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:17 smithi043 ceph-mon[42099]: 5.19 deep-scrub ok 2024-03-30T13:52:18.157 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:18.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:18.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:18.258 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:18.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:18.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:18.393 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:18.416 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:18.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:18.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:18.518 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:18.551 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:18.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:18.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:18.652 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:18.677 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:18.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:18.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:18.778 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:18.820 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:18.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:18.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:18.922 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:18.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:18.952 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:18.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:18.970 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:52:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:52:18.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:18 smithi113 ceph-mon[34341]: 3.18 scrub starts 2024-03-30T13:52:18.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:18 smithi113 ceph-mon[34341]: 3.18 scrub ok 2024-03-30T13:52:18.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:18 smithi113 ceph-mon[34341]: 4.b scrub starts 2024-03-30T13:52:18.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:18 smithi113 ceph-mon[34341]: 4.b scrub ok 2024-03-30T13:52:18.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:18 smithi113 ceph-mon[34341]: 4.e scrub starts 2024-03-30T13:52:18.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:18 smithi113 ceph-mon[34341]: 4.e scrub ok 2024-03-30T13:52:18.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:18 smithi113 ceph-mon[34341]: pgmap v387: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.1 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 1.02k op/s; 6433/52188 objects degraded (12.327%) 2024-03-30T13:52:18.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:18 smithi113 ceph-mon[34341]: 5.0 scrub starts 2024-03-30T13:52:18.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:18 smithi113 ceph-mon[34341]: 5.0 scrub ok 2024-03-30T13:52:19.053 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:19.076 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:19.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:19.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:19.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:18 smithi043 ceph-mon[29963]: 3.18 scrub starts 2024-03-30T13:52:19.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:18 smithi043 ceph-mon[29963]: 3.18 scrub ok 2024-03-30T13:52:19.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:18 smithi043 ceph-mon[29963]: 4.b scrub starts 2024-03-30T13:52:19.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:18 smithi043 ceph-mon[29963]: 4.b scrub ok 2024-03-30T13:52:19.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:18 smithi043 ceph-mon[29963]: 4.e scrub starts 2024-03-30T13:52:19.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:18 smithi043 ceph-mon[29963]: 4.e scrub ok 2024-03-30T13:52:19.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:18 smithi043 ceph-mon[29963]: pgmap v387: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.1 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 1.02k op/s; 6433/52188 objects degraded (12.327%) 2024-03-30T13:52:19.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:18 smithi043 ceph-mon[29963]: 5.0 scrub starts 2024-03-30T13:52:19.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:18 smithi043 ceph-mon[29963]: 5.0 scrub ok 2024-03-30T13:52:19.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:18 smithi043 ceph-mon[42099]: 3.18 scrub starts 2024-03-30T13:52:19.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:18 smithi043 ceph-mon[42099]: 3.18 scrub ok 2024-03-30T13:52:19.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:18 smithi043 ceph-mon[42099]: 4.b scrub starts 2024-03-30T13:52:19.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:18 smithi043 ceph-mon[42099]: 4.b scrub ok 2024-03-30T13:52:19.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:18 smithi043 ceph-mon[42099]: 4.e scrub starts 2024-03-30T13:52:19.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:18 smithi043 ceph-mon[42099]: 4.e scrub ok 2024-03-30T13:52:19.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:18 smithi043 ceph-mon[42099]: pgmap v387: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 1.1 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 1.02k op/s; 6433/52188 objects degraded (12.327%) 2024-03-30T13:52:19.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:18 smithi043 ceph-mon[42099]: 5.0 scrub starts 2024-03-30T13:52:19.149 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:18 smithi043 ceph-mon[42099]: 5.0 scrub ok 2024-03-30T13:52:19.178 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:19.214 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:19.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:19.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:19.316 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:19.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:19.344 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:19.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:19.445 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:19.476 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:19.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:19.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:19.578 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:19.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:19.614 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:19.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:19.715 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:19.738 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:19.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:19.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:19.839 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:19.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:19.871 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:19.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:19.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:19 smithi113 ceph-mon[34341]: 2.0 scrub starts 2024-03-30T13:52:19.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:19 smithi113 ceph-mon[34341]: 2.0 scrub ok 2024-03-30T13:52:19.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:19 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:52:19.972 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:19.995 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:19.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:19.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:20.096 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:20.132 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:20.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:20.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:20.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:19 smithi043 ceph-mon[29963]: 2.0 scrub starts 2024-03-30T13:52:20.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:19 smithi043 ceph-mon[29963]: 2.0 scrub ok 2024-03-30T13:52:20.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:19 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:52:20.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:19 smithi043 ceph-mon[42099]: 2.0 scrub starts 2024-03-30T13:52:20.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:19 smithi043 ceph-mon[42099]: 2.0 scrub ok 2024-03-30T13:52:20.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:19 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:52:20.234 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:20.269 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:20.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:20.370 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:20.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:20.408 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:20.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:20.509 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:20.542 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:20.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:20.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:20.644 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:20.673 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:20.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:20.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:20.775 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:20.809 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:20.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:20.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:20.911 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:20.943 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:20.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:20.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:20.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:20 smithi113 ceph-mon[34341]: pgmap v388: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 710 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 4258/34626 objects degraded (12.297%) 2024-03-30T13:52:20.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:20 smithi113 ceph-mon[34341]: 5.1d scrub starts 2024-03-30T13:52:20.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:20 smithi113 ceph-mon[34341]: 5.1d scrub ok 2024-03-30T13:52:20.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:20 smithi113 ceph-mon[34341]: 5.1b deep-scrub starts 2024-03-30T13:52:20.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:20 smithi113 ceph-mon[34341]: 5.1b deep-scrub ok 2024-03-30T13:52:21.013 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:20 smithi043 ceph-mon[29963]: pgmap v388: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 710 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 4258/34626 objects degraded (12.297%) 2024-03-30T13:52:21.013 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:20 smithi043 ceph-mon[29963]: 5.1d scrub starts 2024-03-30T13:52:21.013 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:20 smithi043 ceph-mon[29963]: 5.1d scrub ok 2024-03-30T13:52:21.013 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:20 smithi043 ceph-mon[29963]: 5.1b deep-scrub starts 2024-03-30T13:52:21.013 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:20 smithi043 ceph-mon[29963]: 5.1b deep-scrub ok 2024-03-30T13:52:21.013 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:20 smithi043 ceph-mon[42099]: pgmap v388: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 710 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 4258/34626 objects degraded (12.297%) 2024-03-30T13:52:21.013 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:20 smithi043 ceph-mon[42099]: 5.1d scrub starts 2024-03-30T13:52:21.013 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:20 smithi043 ceph-mon[42099]: 5.1d scrub ok 2024-03-30T13:52:21.013 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:20 smithi043 ceph-mon[42099]: 5.1b deep-scrub starts 2024-03-30T13:52:21.013 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:20 smithi043 ceph-mon[42099]: 5.1b deep-scrub ok 2024-03-30T13:52:21.045 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:21.081 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:21.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:21.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:21.183 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:21.206 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:21.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:21.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:21.308 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:21.349 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:21.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:21.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:21.451 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:21.490 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:21.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:21.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:21.591 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:21.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:21.627 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:21.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:21.728 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:21.762 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:21.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:21.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:21.787 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:52:21.787 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-03-30T13:52:21.787 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-03-30T13:52:21.787 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-03-30T13:52:21.788 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.788 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(osd.4.0:11273 7.1 e123/98 7:8560b898:::benchmark_data_smithi043_86989_object43301:head v 123'11159, mlcod=123'11152)", 2024-03-30T13:52:21.788 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:46:55.519084+0000", 2024-03-30T13:52:21.788 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 325.491284372, 2024-03-30T13:52:21.788 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.143160654, 2024-03-30T13:52:21.788 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:21.788 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:21.788 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:21.788 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:21.788 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.788 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:21.788 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.519084+0000", 2024-03-30T13:52:21.788 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.789 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.789 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:21.789 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.519084+0000", 2024-03-30T13:52:21.789 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.789 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.789 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:21.789 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.519085+0000", 2024-03-30T13:52:21.789 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4600000000000005e-07 2024-03-30T13:52:21.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.789 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.789 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:21.789 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.519090+0000", 2024-03-30T13:52:21.790 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3900000000000001e-06 2024-03-30T13:52:21.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.790 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.790 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:21.790 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.519098+0000", 2024-03-30T13:52:21.790 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4440000000000008e-06 2024-03-30T13:52:21.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.790 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.790 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:21.790 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.519105+0000", 2024-03-30T13:52:21.790 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1260000000000003e-06 2024-03-30T13:52:21.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.791 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.791 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:21.791 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.650245+0000", 2024-03-30T13:52:21.791 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.131140794 2024-03-30T13:52:21.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.791 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.791 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:21.791 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.650271+0000", 2024-03-30T13:52:21.791 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5627e-05 2024-03-30T13:52:21.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.791 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.791 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:52:21.791 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.662204+0000", 2024-03-30T13:52:21.791 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011933365 2024-03-30T13:52:21.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.792 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.792 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:52:21.792 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:46:55.662245+0000", 2024-03-30T13:52:21.792 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0361999999999997e-05 2024-03-30T13:52:21.792 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.792 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.792 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.792 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.792 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:31276 8.b 8:d05106bb:::benchmark_data_smithi043_119832_object31275:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-30T13:52:21.792 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:49:27.602579+0000", 2024-03-30T13:52:21.792 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 173.40778932699999, 2024-03-30T13:52:21.793 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17785183600000001, 2024-03-30T13:52:21.793 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:21.793 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:21.793 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:21.793 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:52:21.793 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:52:21.793 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:21.793 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 31276 2024-03-30T13:52:21.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.793 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:21.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.793 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:21.793 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.602579+0000", 2024-03-30T13:52:21.793 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.794 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.794 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.794 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:21.794 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.602579+0000", 2024-03-30T13:52:21.794 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.794 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.794 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.794 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:21.794 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.602581+0000", 2024-03-30T13:52:21.794 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.79e-06 2024-03-30T13:52:21.794 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.794 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.794 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:21.795 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.602606+0000", 2024-03-30T13:52:21.795 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5021e-05 2024-03-30T13:52:21.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.795 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.795 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:21.795 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.602618+0000", 2024-03-30T13:52:21.795 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2568000000000001e-05 2024-03-30T13:52:21.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.795 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.795 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:21.795 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.602627+0000", 2024-03-30T13:52:21.795 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6859999999999995e-06 2024-03-30T13:52:21.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.795 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.796 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:21.796 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.602688+0000", 2024-03-30T13:52:21.796 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1011999999999998e-05 2024-03-30T13:52:21.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.796 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.796 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:21.796 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.603023+0000", 2024-03-30T13:52:21.796 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000334937 2024-03-30T13:52:21.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.796 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.796 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:52:21.796 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.603224+0000", 2024-03-30T13:52:21.796 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000201227 2024-03-30T13:52:21.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.796 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.797 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.797 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.605455+0000", 2024-03-30T13:52:21.797 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022307920000000001 2024-03-30T13:52:21.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.797 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:52:21.797 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.605467+0000", 2024-03-30T13:52:21.797 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2232000000000001e-05 2024-03-30T13:52:21.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.797 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.797 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780257+0000", 2024-03-30T13:52:21.797 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.174789747 2024-03-30T13:52:21.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.798 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:52:21.798 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780377+0000", 2024-03-30T13:52:21.798 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012008800000000001 2024-03-30T13:52:21.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.798 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:52:21.798 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780431+0000", 2024-03-30T13:52:21.798 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3736000000000002e-05 2024-03-30T13:52:21.798 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.798 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.798 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.799 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:31289 8.b 8:d1f0aa81:::benchmark_data_smithi043_119832_object31288:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-30T13:52:21.799 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:49:27.605999+0000", 2024-03-30T13:52:21.799 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 173.40436894699999, 2024-03-30T13:52:21.799 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17461035799999999, 2024-03-30T13:52:21.799 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:21.799 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:21.799 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:21.799 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:52:21.799 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:52:21.799 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:21.799 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 31289 2024-03-30T13:52:21.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.799 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:21.799 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.800 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:21.800 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.605999+0000", 2024-03-30T13:52:21.800 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.800 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:21.800 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.605999+0000", 2024-03-30T13:52:21.800 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.800 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:21.800 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.606001+0000", 2024-03-30T13:52:21.800 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.567e-06 2024-03-30T13:52:21.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.801 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:21.801 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.606020+0000", 2024-03-30T13:52:21.801 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8974000000000002e-05 2024-03-30T13:52:21.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.801 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:21.801 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.606034+0000", 2024-03-30T13:52:21.801 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.403e-05 2024-03-30T13:52:21.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.801 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:21.801 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.606042+0000", 2024-03-30T13:52:21.801 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8720000000000002e-06 2024-03-30T13:52:21.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.802 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.802 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:21.802 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.606114+0000", 2024-03-30T13:52:21.802 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2198000000000002e-05 2024-03-30T13:52:21.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.802 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.802 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:21.802 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.606316+0000", 2024-03-30T13:52:21.802 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00020206900000000001 2024-03-30T13:52:21.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.802 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.802 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:52:21.803 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.606465+0000", 2024-03-30T13:52:21.803 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014919400000000001 2024-03-30T13:52:21.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.803 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:52:21.803 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.608103+0000", 2024-03-30T13:52:21.803 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0016373049999999999 2024-03-30T13:52:21.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.803 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.803 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.610843+0000", 2024-03-30T13:52:21.803 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0027407769999999998 2024-03-30T13:52:21.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.804 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.804 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780498+0000", 2024-03-30T13:52:21.804 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.16965456300000001 2024-03-30T13:52:21.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.804 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:52:21.804 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780565+0000", 2024-03-30T13:52:21.804 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6587999999999993e-05 2024-03-30T13:52:21.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.804 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:52:21.804 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780610+0000", 2024-03-30T13:52:21.804 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5221e-05 2024-03-30T13:52:21.804 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.805 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.805 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.805 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:31302 8.b 8:db19a970:::benchmark_data_smithi043_119832_object31301:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-30T13:52:21.805 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:49:27.614761+0000", 2024-03-30T13:52:21.805 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 173.39560786499999, 2024-03-30T13:52:21.805 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.16602650699999999, 2024-03-30T13:52:21.805 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:21.805 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:21.805 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:21.805 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:52:21.805 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:52:21.806 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:21.806 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 31302 2024-03-30T13:52:21.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.806 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:21.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.806 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:21.806 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.614761+0000", 2024-03-30T13:52:21.806 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.806 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:21.806 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.614761+0000", 2024-03-30T13:52:21.806 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.807 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:21.807 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.614762+0000", 2024-03-30T13:52:21.807 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3999999999999999e-06 2024-03-30T13:52:21.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.807 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:21.807 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.614796+0000", 2024-03-30T13:52:21.807 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4400000000000003e-05 2024-03-30T13:52:21.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.807 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:21.808 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.614809+0000", 2024-03-30T13:52:21.808 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2699e-05 2024-03-30T13:52:21.808 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.808 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:21.808 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.614817+0000", 2024-03-30T13:52:21.808 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5299999999999999e-06 2024-03-30T13:52:21.808 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.808 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:21.808 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.615011+0000", 2024-03-30T13:52:21.808 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00019433699999999999 2024-03-30T13:52:21.808 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.809 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:21.809 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.615129+0000", 2024-03-30T13:52:21.809 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011833900000000001 2024-03-30T13:52:21.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.809 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.809 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:52:21.809 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.615547+0000", 2024-03-30T13:52:21.809 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00041808699999999998 2024-03-30T13:52:21.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.809 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.809 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:52:21.809 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.617305+0000", 2024-03-30T13:52:21.809 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0017581070000000001 2024-03-30T13:52:21.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.809 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.810 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.810 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.618095+0000", 2024-03-30T13:52:21.810 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00078949499999999998 2024-03-30T13:52:21.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.810 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.810 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.810 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780674+0000", 2024-03-30T13:52:21.810 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.16257946000000001 2024-03-30T13:52:21.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.810 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.810 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:52:21.810 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780743+0000", 2024-03-30T13:52:21.810 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8725000000000004e-05 2024-03-30T13:52:21.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.810 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.811 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:52:21.811 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780787+0000", 2024-03-30T13:52:21.811 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3927999999999997e-05 2024-03-30T13:52:21.811 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.811 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.811 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.811 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.811 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:31311 8.b 8:de8ee756:::benchmark_data_smithi043_119832_object31310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-30T13:52:21.811 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:49:27.618998+0000", 2024-03-30T13:52:21.811 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 173.39137013999999, 2024-03-30T13:52:21.811 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.16195649300000001, 2024-03-30T13:52:21.811 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:21.812 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:21.812 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:21.812 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:52:21.812 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:52:21.812 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:21.812 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 31311 2024-03-30T13:52:21.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.812 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:21.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.812 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:21.812 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.618998+0000", 2024-03-30T13:52:21.812 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.618998+0000", 2024-03-30T13:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.619000+0000", 2024-03-30T13:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5969999999999999e-06 2024-03-30T13:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.619022+0000", 2024-03-30T13:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2180000000000001e-05 2024-03-30T13:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.619036+0000", 2024-03-30T13:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3726999999999999e-05 2024-03-30T13:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.619044+0000", 2024-03-30T13:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2609999999999993e-06 2024-03-30T13:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.619105+0000", 2024-03-30T13:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0843e-05 2024-03-30T13:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.619236+0000", 2024-03-30T13:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013116500000000001 2024-03-30T13:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.619385+0000", 2024-03-30T13:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014934799999999999 2024-03-30T13:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.620675+0000", 2024-03-30T13:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0012892890000000001 2024-03-30T13:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.621289+0000", 2024-03-30T13:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00061401499999999998 2024-03-30T13:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780836+0000", 2024-03-30T13:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.15954684799999999 2024-03-30T13:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780917+0000", 2024-03-30T13:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1414000000000004e-05 2024-03-30T13:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780955+0000", 2024-03-30T13:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7805999999999997e-05 2024-03-30T13:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:180578 8.b 8:da5cea62:::benchmark_data_smithi043_119832_object50224:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-03-30T13:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:51:25.080942+0000", 2024-03-30T13:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 55.929426876000001, 2024-03-30T13:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.148373793, 2024-03-30T13:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 180578 2024-03-30T13:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.819 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:21.819 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:25.080942+0000", 2024-03-30T13:52:21.819 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.819 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.819 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:21.819 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:25.080942+0000", 2024-03-30T13:52:21.819 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.819 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.819 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:21.819 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:25.080943+0000", 2024-03-30T13:52:21.819 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.127e-06 2024-03-30T13:52:21.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.820 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.820 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:21.820 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:25.080947+0000", 2024-03-30T13:52:21.820 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7489999999999998e-06 2024-03-30T13:52:21.820 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.820 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.820 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:21.820 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:25.080969+0000", 2024-03-30T13:52:21.820 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1118999999999999e-05 2024-03-30T13:52:21.820 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.820 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.820 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:21.820 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:25.080976+0000", 2024-03-30T13:52:21.820 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3560000000000004e-06 2024-03-30T13:52:21.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.821 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.821 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:21.821 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:25.098749+0000", 2024-03-30T13:52:21.821 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.017773557999999998 2024-03-30T13:52:21.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.821 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.821 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:21.821 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:25.098881+0000", 2024-03-30T13:52:21.821 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013135600000000001 2024-03-30T13:52:21.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.821 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.821 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:52:21.821 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:25.099004+0000", 2024-03-30T13:52:21.822 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012339600000000001 2024-03-30T13:52:21.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.822 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.822 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:52:21.822 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:25.101277+0000", 2024-03-30T13:52:21.822 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00227247 2024-03-30T13:52:21.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.822 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.822 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.822 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:25.106249+0000", 2024-03-30T13:52:21.822 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0049723629999999996 2024-03-30T13:52:21.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.822 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.823 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.823 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:25.229201+0000", 2024-03-30T13:52:21.823 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.122952284 2024-03-30T13:52:21.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.823 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.823 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:52:21.823 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:25.229256+0000", 2024-03-30T13:52:21.823 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5016000000000003e-05 2024-03-30T13:52:21.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.823 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.823 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:52:21.823 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:25.229315+0000", 2024-03-30T13:52:21.823 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8999000000000002e-05 2024-03-30T13:52:21.823 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.823 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.824 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.824 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.824 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:181650 8.b 8:da4a27be:::benchmark_data_smithi043_119832_object51296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-03-30T13:52:21.824 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:51:26.014158+0000", 2024-03-30T13:52:21.824 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 54.996210634000001, 2024-03-30T13:52:21.824 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13740359199999999, 2024-03-30T13:52:21.824 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:21.824 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:21.824 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:21.824 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:52:21.824 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:52:21.824 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:21.825 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 181650 2024-03-30T13:52:21.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.825 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:21.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.825 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:21.825 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.014158+0000", 2024-03-30T13:52:21.825 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.825 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:21.825 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.014158+0000", 2024-03-30T13:52:21.825 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.826 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:21.826 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.014159+0000", 2024-03-30T13:52:21.826 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.68e-06 2024-03-30T13:52:21.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.826 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:21.826 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.014166+0000", 2024-03-30T13:52:21.826 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3419999999999996e-06 2024-03-30T13:52:21.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.826 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:21.826 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.014183+0000", 2024-03-30T13:52:21.827 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7198e-05 2024-03-30T13:52:21.827 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.827 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.827 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:21.827 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.014193+0000", 2024-03-30T13:52:21.827 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5529999999999998e-06 2024-03-30T13:52:21.827 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.827 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.827 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:21.827 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.019669+0000", 2024-03-30T13:52:21.827 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0054768689999999997 2024-03-30T13:52:21.827 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.827 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.827 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:21.828 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.019805+0000", 2024-03-30T13:52:21.828 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000136005 2024-03-30T13:52:21.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.828 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:52:21.828 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.019945+0000", 2024-03-30T13:52:21.828 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013926100000000001 2024-03-30T13:52:21.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.828 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:52:21.828 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.023214+0000", 2024-03-30T13:52:21.828 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0032694220000000001 2024-03-30T13:52:21.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.829 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.829 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.028896+0000", 2024-03-30T13:52:21.829 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0056819779999999999 2024-03-30T13:52:21.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.829 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.829 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.151460+0000", 2024-03-30T13:52:21.829 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.12256368400000001 2024-03-30T13:52:21.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.829 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:52:21.830 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.151520+0000", 2024-03-30T13:52:21.830 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0081000000000003e-05 2024-03-30T13:52:21.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.830 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.830 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:52:21.830 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.151561+0000", 2024-03-30T13:52:21.830 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1519e-05 2024-03-30T13:52:21.830 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.830 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.830 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.830 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.830 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:182543 8.b 8:dfd3032c:::benchmark_data_smithi043_119832_object52189:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-03-30T13:52:21.830 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:51:26.787103+0000", 2024-03-30T13:52:21.831 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 54.223265822999998, 2024-03-30T13:52:21.831 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14079040300000001, 2024-03-30T13:52:21.831 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:21.831 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:21.831 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:21.831 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:52:21.831 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:52:21.831 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:21.831 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 182543 2024-03-30T13:52:21.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.831 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:21.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.831 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:21.831 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.787103+0000", 2024-03-30T13:52:21.832 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.832 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:21.832 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.787103+0000", 2024-03-30T13:52:21.832 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.832 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:21.832 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.787104+0000", 2024-03-30T13:52:21.832 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1200000000000001e-06 2024-03-30T13:52:21.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.833 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:21.833 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.787108+0000", 2024-03-30T13:52:21.833 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7319999999999998e-06 2024-03-30T13:52:21.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.833 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:21.833 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.787120+0000", 2024-03-30T13:52:21.833 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1946e-05 2024-03-30T13:52:21.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.833 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:21.833 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.787128+0000", 2024-03-30T13:52:21.833 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8690000000000004e-06 2024-03-30T13:52:21.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.834 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:21.834 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.817026+0000", 2024-03-30T13:52:21.834 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.029897489999999999 2024-03-30T13:52:21.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.834 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:21.834 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.817166+0000", 2024-03-30T13:52:21.834 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014073199999999999 2024-03-30T13:52:21.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.834 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:52:21.835 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.817304+0000", 2024-03-30T13:52:21.835 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013771700000000001 2024-03-30T13:52:21.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.835 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.835 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.823452+0000", 2024-03-30T13:52:21.835 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0061476170000000002 2024-03-30T13:52:21.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.835 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:52:21.835 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.828464+0000", 2024-03-30T13:52:21.835 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0050120590000000001 2024-03-30T13:52:21.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.836 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.836 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.927781+0000", 2024-03-30T13:52:21.836 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.099317180000000005 2024-03-30T13:52:21.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.836 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:52:21.836 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.927834+0000", 2024-03-30T13:52:21.836 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2562000000000003e-05 2024-03-30T13:52:21.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.836 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:52:21.836 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:26.927893+0000", 2024-03-30T13:52:21.836 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9379000000000002e-05 2024-03-30T13:52:21.836 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.837 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.837 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.837 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:183777 8.b 8:dffce38c:::benchmark_data_smithi043_119832_object53423:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-03-30T13:52:21.837 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:51:27.849665+0000", 2024-03-30T13:52:21.837 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 53.160703804999997, 2024-03-30T13:52:21.837 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.139413963, 2024-03-30T13:52:21.837 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:21.837 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:21.837 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:21.837 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:52:21.837 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:52:21.837 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:21.838 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 183777 2024-03-30T13:52:21.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.838 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:21.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.838 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:21.838 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:27.849665+0000", 2024-03-30T13:52:21.838 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.838 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:21.838 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:27.849665+0000", 2024-03-30T13:52:21.838 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.839 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:21.839 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:27.849666+0000", 2024-03-30T13:52:21.839 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.61e-06 2024-03-30T13:52:21.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.839 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:21.839 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:27.849673+0000", 2024-03-30T13:52:21.839 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4570000000000002e-06 2024-03-30T13:52:21.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.839 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:21.839 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:27.849689+0000", 2024-03-30T13:52:21.839 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6447999999999999e-05 2024-03-30T13:52:21.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.840 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:21.840 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:27.849697+0000", 2024-03-30T13:52:21.840 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1289999999999996e-06 2024-03-30T13:52:21.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.840 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:21.840 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:27.875456+0000", 2024-03-30T13:52:21.840 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.025758878999999998 2024-03-30T13:52:21.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.840 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:21.840 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:27.875579+0000", 2024-03-30T13:52:21.841 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012277699999999999 2024-03-30T13:52:21.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.841 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:52:21.841 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:27.875693+0000", 2024-03-30T13:52:21.841 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000114448 2024-03-30T13:52:21.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.841 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.841 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:27.879198+0000", 2024-03-30T13:52:21.841 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0035044210000000002 2024-03-30T13:52:21.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.841 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.842 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:27.908802+0000", 2024-03-30T13:52:21.842 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.029603846999999999 2024-03-30T13:52:21.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.842 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.842 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:52:21.842 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:27.989008+0000", 2024-03-30T13:52:21.842 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.080206361000000004 2024-03-30T13:52:21.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.842 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.842 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:52:21.842 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:27.989042+0000", 2024-03-30T13:52:21.842 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3624999999999997e-05 2024-03-30T13:52:21.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.842 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.843 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:52:21.843 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:27.989079+0000", 2024-03-30T13:52:21.843 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6961000000000001e-05 2024-03-30T13:52:21.843 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.843 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.843 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.843 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:186524 8.b 8:d08afbcc:::benchmark_data_smithi043_119832_object56170:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-03-30T13:52:21.843 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:51:30.200072+0000", 2024-03-30T13:52:21.843 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 50.810295977999999, 2024-03-30T13:52:21.843 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14062374899999999, 2024-03-30T13:52:21.843 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:21.844 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:21.844 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:21.844 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:52:21.844 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:52:21.844 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:21.844 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 186524 2024-03-30T13:52:21.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.844 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:21.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.844 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:21.844 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:30.200072+0000", 2024-03-30T13:52:21.844 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.845 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:21.845 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:30.200072+0000", 2024-03-30T13:52:21.845 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.845 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:21.845 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:30.200074+0000", 2024-03-30T13:52:21.845 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4470000000000001e-06 2024-03-30T13:52:21.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.845 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:21.845 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:30.200080+0000", 2024-03-30T13:52:21.846 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4400000000000002e-06 2024-03-30T13:52:21.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.846 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.846 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:21.846 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:30.200097+0000", 2024-03-30T13:52:21.846 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6455000000000001e-05 2024-03-30T13:52:21.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.846 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.846 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:21.846 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:30.200170+0000", 2024-03-30T13:52:21.846 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3726999999999994e-05 2024-03-30T13:52:21.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.846 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.847 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:21.847 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:30.240480+0000", 2024-03-30T13:52:21.847 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.040309640000000001 2024-03-30T13:52:21.847 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.847 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.847 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:21.847 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:30.240589+0000", 2024-03-30T13:52:21.847 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000108795 2024-03-30T13:52:21.847 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.847 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.847 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:52:21.847 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:30.240709+0000", 2024-03-30T13:52:21.847 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000120005 2024-03-30T13:52:21.847 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.848 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:52:21.848 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:30.246331+0000", 2024-03-30T13:52:21.848 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0056220269999999999 2024-03-30T13:52:21.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.848 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.848 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:30.255938+0000", 2024-03-30T13:52:21.848 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0096067080000000003 2024-03-30T13:52:21.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.848 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.848 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:30.340587+0000", 2024-03-30T13:52:21.848 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.084649242999999999 2024-03-30T13:52:21.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.849 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:52:21.849 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:30.340640+0000", 2024-03-30T13:52:21.849 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2828000000000003e-05 2024-03-30T13:52:21.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.849 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:52:21.849 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:51:30.340696+0000", 2024-03-30T13:52:21.849 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6434000000000002e-05 2024-03-30T13:52:21.849 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.849 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.849 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.850 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.850 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:247649 8.0 8:052dadc5:::benchmark_data_smithi043_119832_object117295:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-03-30T13:52:21.850 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:52:17.381399+0000", 2024-03-30T13:52:21.850 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 3.6289697859999999, 2024-03-30T13:52:21.850 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14601703699999999, 2024-03-30T13:52:21.850 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:21.850 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:21.850 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:21.850 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:52:21.850 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:52:21.850 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:21.850 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 247649 2024-03-30T13:52:21.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.851 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:21.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.851 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:21.851 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.381399+0000", 2024-03-30T13:52:21.851 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.851 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:21.851 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.381399+0000", 2024-03-30T13:52:21.851 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.851 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:21.852 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.381400+0000", 2024-03-30T13:52:21.852 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3239999999999999e-06 2024-03-30T13:52:21.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.852 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:21.852 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.381405+0000", 2024-03-30T13:52:21.852 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0300000000000001e-06 2024-03-30T13:52:21.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.852 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:21.852 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.381417+0000", 2024-03-30T13:52:21.852 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1800999999999999e-05 2024-03-30T13:52:21.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.853 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:21.853 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.381425+0000", 2024-03-30T13:52:21.853 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9659999999999996e-06 2024-03-30T13:52:21.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.853 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.853 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:21.853 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.381480+0000", 2024-03-30T13:52:21.853 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5588999999999999e-05 2024-03-30T13:52:21.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.853 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.853 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:21.853 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.381555+0000", 2024-03-30T13:52:21.853 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4913000000000006e-05 2024-03-30T13:52:21.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.854 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,5", 2024-03-30T13:52:21.854 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.381621+0000", 2024-03-30T13:52:21.854 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5370000000000006e-05 2024-03-30T13:52:21.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.854 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:52:21.854 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.384359+0000", 2024-03-30T13:52:21.854 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0027381240000000002 2024-03-30T13:52:21.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.854 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.855 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.393047+0000", 2024-03-30T13:52:21.855 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0086885380000000009 2024-03-30T13:52:21.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.855 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.855 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.527318+0000", 2024-03-30T13:52:21.855 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.134270793 2024-03-30T13:52:21.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.855 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:52:21.855 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.527372+0000", 2024-03-30T13:52:21.855 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4301000000000001e-05 2024-03-30T13:52:21.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.856 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:52:21.856 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.527416+0000", 2024-03-30T13:52:21.856 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3288000000000003e-05 2024-03-30T13:52:21.856 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.856 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.856 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.856 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.856 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:247656 8.0 8:03934271:::benchmark_data_smithi043_119832_object117302:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-03-30T13:52:21.856 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:52:17.384794+0000", 2024-03-30T13:52:21.856 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 3.62557449, 2024-03-30T13:52:21.856 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.142746661, 2024-03-30T13:52:21.856 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:21.857 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:21.857 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:21.857 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:52:21.857 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:52:21.857 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:21.857 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 247656 2024-03-30T13:52:21.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.857 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:21.857 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.857 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:21.857 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.384794+0000", 2024-03-30T13:52:21.857 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.857 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.857 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:21.858 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.384794+0000", 2024-03-30T13:52:21.858 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.858 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.858 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:21.858 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.384796+0000", 2024-03-30T13:52:21.858 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6950000000000001e-06 2024-03-30T13:52:21.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.858 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.858 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:21.858 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.384801+0000", 2024-03-30T13:52:21.858 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3460000000000001e-06 2024-03-30T13:52:21.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.859 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:21.859 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.384813+0000", 2024-03-30T13:52:21.859 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2364000000000001e-05 2024-03-30T13:52:21.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.859 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:21.859 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.384821+0000", 2024-03-30T13:52:21.859 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9640000000000003e-06 2024-03-30T13:52:21.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.859 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:21.859 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.384883+0000", 2024-03-30T13:52:21.859 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1830999999999998e-05 2024-03-30T13:52:21.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.860 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:21.860 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.385002+0000", 2024-03-30T13:52:21.860 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011925 2024-03-30T13:52:21.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.860 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,5", 2024-03-30T13:52:21.860 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.385111+0000", 2024-03-30T13:52:21.860 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000108533 2024-03-30T13:52:21.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.860 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:52:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.387925+0000", 2024-03-30T13:52:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002814418 2024-03-30T13:52:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.397728+0000", 2024-03-30T13:52:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0098026599999999995 2024-03-30T13:52:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.527455+0000", 2024-03-30T13:52:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.12972661499999999 2024-03-30T13:52:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:52:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.527502+0000", 2024-03-30T13:52:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7262999999999998e-05 2024-03-30T13:52:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:52:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.527541+0000", 2024-03-30T13:52:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8722e-05 2024-03-30T13:52:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:247664 8.0 8:0adf3677:::benchmark_data_smithi043_119832_object117310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-03-30T13:52:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:52:17.393085+0000", 2024-03-30T13:52:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 3.6172832599999998, 2024-03-30T13:52:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13758706500000001, 2024-03-30T13:52:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:52:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:52:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 247664 2024-03-30T13:52:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.393085+0000", 2024-03-30T13:52:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.393085+0000", 2024-03-30T13:52:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.393087+0000", 2024-03-30T13:52:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4640000000000001e-06 2024-03-30T13:52:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.393091+0000", 2024-03-30T13:52:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2899999999999996e-06 2024-03-30T13:52:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.393102+0000", 2024-03-30T13:52:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1279e-05 2024-03-30T13:52:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:21.866 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.393110+0000", 2024-03-30T13:52:21.866 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9489999999999996e-06 2024-03-30T13:52:21.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.866 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.866 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:21.866 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.393138+0000", 2024-03-30T13:52:21.866 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7874000000000001e-05 2024-03-30T13:52:21.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.866 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.866 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:21.866 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.393212+0000", 2024-03-30T13:52:21.866 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3937999999999996e-05 2024-03-30T13:52:21.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.867 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,5", 2024-03-30T13:52:21.867 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.393280+0000", 2024-03-30T13:52:21.867 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8287000000000005e-05 2024-03-30T13:52:21.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.867 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:52:21.867 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.395325+0000", 2024-03-30T13:52:21.867 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002044391 2024-03-30T13:52:21.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.867 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.867 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.399226+0000", 2024-03-30T13:52:21.867 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0039016860000000001 2024-03-30T13:52:21.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.868 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.868 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.868 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.530572+0000", 2024-03-30T13:52:21.868 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.131346148 2024-03-30T13:52:21.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.868 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.868 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:52:21.868 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.530629+0000", 2024-03-30T13:52:21.868 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6359e-05 2024-03-30T13:52:21.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.868 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.868 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:52:21.869 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.530672+0000", 2024-03-30T13:52:21.869 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3399999999999998e-05 2024-03-30T13:52:21.869 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.869 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.869 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.869 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.869 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.869 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:247667 8.f 8:f9c5d2a0:::benchmark_data_smithi043_119832_object117313:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-03-30T13:52:21.869 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:52:17.394714+0000", 2024-03-30T13:52:21.869 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 3.6156547450000001, 2024-03-30T13:52:21.869 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13829195999999999, 2024-03-30T13:52:21.869 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:21.869 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:21.869 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:21.870 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:52:21.870 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:52:21.870 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:21.870 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 247667 2024-03-30T13:52:21.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.870 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:21.870 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.870 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:21.870 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.394714+0000", 2024-03-30T13:52:21.870 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.870 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.870 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:21.871 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.394714+0000", 2024-03-30T13:52:21.871 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.871 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.871 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:21.871 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.394714+0000", 2024-03-30T13:52:21.871 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8199999999999999e-07 2024-03-30T13:52:21.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.871 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.871 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:21.871 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.394719+0000", 2024-03-30T13:52:21.871 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4479999999999996e-06 2024-03-30T13:52:21.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.872 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.872 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:21.872 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.394727+0000", 2024-03-30T13:52:21.872 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6260000000000001e-06 2024-03-30T13:52:21.872 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.872 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.872 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:21.872 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.394733+0000", 2024-03-30T13:52:21.872 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5269999999999998e-06 2024-03-30T13:52:21.872 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.872 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.872 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.394775+0000", 2024-03-30T13:52:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1792000000000002e-05 2024-03-30T13:52:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.394849+0000", 2024-03-30T13:52:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4494999999999997e-05 2024-03-30T13:52:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1", 2024-03-30T13:52:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.394925+0000", 2024-03-30T13:52:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5900000000000002e-05 2024-03-30T13:52:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:52:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.397359+0000", 2024-03-30T13:52:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024343540000000001 2024-03-30T13:52:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.532851+0000", 2024-03-30T13:52:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.135491212 2024-03-30T13:52:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:52:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.532911+0000", 2024-03-30T13:52:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0390000000000003e-05 2024-03-30T13:52:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:52:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.533006+0000", 2024-03-30T13:52:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4534e-05 2024-03-30T13:52:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:247668 8.b 8:d5b1e705:::benchmark_data_smithi043_119832_object117314:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-03-30T13:52:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:52:17.395678+0000", 2024-03-30T13:52:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 3.6146907219999997, 2024-03-30T13:52:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.15649107600000001, 2024-03-30T13:52:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:52:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:52:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 247668 2024-03-30T13:52:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.395678+0000", 2024-03-30T13:52:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.395678+0000", 2024-03-30T13:52:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.395678+0000", 2024-03-30T13:52:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2799999999999995e-07 2024-03-30T13:52:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.395682+0000", 2024-03-30T13:52:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8539999999999999e-06 2024-03-30T13:52:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.395691+0000", 2024-03-30T13:52:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5159999999999994e-06 2024-03-30T13:52:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.395696+0000", 2024-03-30T13:52:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5690000000000004e-06 2024-03-30T13:52:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.395745+0000", 2024-03-30T13:52:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8381999999999999e-05 2024-03-30T13:52:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.395855+0000", 2024-03-30T13:52:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011047700000000001 2024-03-30T13:52:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:52:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.395974+0000", 2024-03-30T13:52:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000118937 2024-03-30T13:52:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:52:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.398454+0000", 2024-03-30T13:52:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002480323 2024-03-30T13:52:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.398867+0000", 2024-03-30T13:52:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000412631 2024-03-30T13:52:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.552060+0000", 2024-03-30T13:52:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.15319280699999999 2024-03-30T13:52:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:52:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.552118+0000", 2024-03-30T13:52:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8072000000000002e-05 2024-03-30T13:52:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:52:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.552169+0000", 2024-03-30T13:52:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0779999999999998e-05 2024-03-30T13:52:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:247669 8.0 8:058a409c:::benchmark_data_smithi043_119832_object117315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-03-30T13:52:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:52:17.395717+0000", 2024-03-30T13:52:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 3.6146511329999997, 2024-03-30T13:52:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14735190500000001, 2024-03-30T13:52:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:52:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:52:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 247669 2024-03-30T13:52:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.395717+0000", 2024-03-30T13:52:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.395717+0000", 2024-03-30T13:52:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.395718+0000", 2024-03-30T13:52:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.22e-07 2024-03-30T13:52:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.395720+0000", 2024-03-30T13:52:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4559999999999999e-06 2024-03-30T13:52:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.395725+0000", 2024-03-30T13:52:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9840000000000001e-06 2024-03-30T13:52:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.395729+0000", 2024-03-30T13:52:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9779999999999999e-06 2024-03-30T13:52:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.395763+0000", 2024-03-30T13:52:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4140999999999997e-05 2024-03-30T13:52:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.395830+0000", 2024-03-30T13:52:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6897999999999995e-05 2024-03-30T13:52:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,5", 2024-03-30T13:52:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.395919+0000", 2024-03-30T13:52:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.9295999999999993e-05 2024-03-30T13:52:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:52:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.397806+0000", 2024-03-30T13:52:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0018870009999999999 2024-03-30T13:52:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.399315+0000", 2024-03-30T13:52:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001508487 2024-03-30T13:52:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.542396+0000", 2024-03-30T13:52:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14308143600000001 2024-03-30T13:52:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:52:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.543027+0000", 2024-03-30T13:52:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00063089299999999999 2024-03-30T13:52:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:52:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:17.543069+0000", 2024-03-30T13:52:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2012999999999999e-05 2024-03-30T13:52:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:249822 8.0 8:09f5cd2e:::benchmark_data_smithi043_119832_object119468:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:52:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:52:19.316253+0000", 2024-03-30T13:52:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 1.694115155, 2024-03-30T13:52:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.141323806, 2024-03-30T13:52:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:52:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:52:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 249822 2024-03-30T13:52:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.316253+0000", 2024-03-30T13:52:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.316253+0000", 2024-03-30T13:52:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.316254+0000", 2024-03-30T13:52:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7199999999999997e-07 2024-03-30T13:52:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.316260+0000", 2024-03-30T13:52:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3399999999999997e-06 2024-03-30T13:52:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.316271+0000", 2024-03-30T13:52:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1653e-05 2024-03-30T13:52:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.316279+0000", 2024-03-30T13:52:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4529999999999996e-06 2024-03-30T13:52:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.317582+0000", 2024-03-30T13:52:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0013034329999999999 2024-03-30T13:52:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.317695+0000", 2024-03-30T13:52:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000113292 2024-03-30T13:52:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,5", 2024-03-30T13:52:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.317787+0000", 2024-03-30T13:52:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1692000000000002e-05 2024-03-30T13:52:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:52:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.326229+0000", 2024-03-30T13:52:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0084416690000000006 2024-03-30T13:52:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.328608+0000", 2024-03-30T13:52:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0023795700000000001 2024-03-30T13:52:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.457442+0000", 2024-03-30T13:52:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.12883370399999999 2024-03-30T13:52:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:52:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.457526+0000", 2024-03-30T13:52:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3813000000000005e-05 2024-03-30T13:52:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:52:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.457577+0000", 2024-03-30T13:52:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1214999999999997e-05 2024-03-30T13:52:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:249823 8.0 8:0dd1cbc6:::benchmark_data_smithi043_119832_object119469:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:52:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:52:19.316295+0000", 2024-03-30T13:52:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 1.6940736699999999, 2024-03-30T13:52:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14597875799999999, 2024-03-30T13:52:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:52:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:52:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 249823 2024-03-30T13:52:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.316295+0000", 2024-03-30T13:52:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.316295+0000", 2024-03-30T13:52:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.316295+0000", 2024-03-30T13:52:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2100000000000002e-07 2024-03-30T13:52:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.316298+0000", 2024-03-30T13:52:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.791e-06 2024-03-30T13:52:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.316304+0000", 2024-03-30T13:52:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0900000000000001e-06 2024-03-30T13:52:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.316309+0000", 2024-03-30T13:52:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2820000000000004e-06 2024-03-30T13:52:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.320300+0000", 2024-03-30T13:52:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003990777 2024-03-30T13:52:21.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.898 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:21.898 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.320407+0000", 2024-03-30T13:52:21.898 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000107038 2024-03-30T13:52:21.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.898 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,5", 2024-03-30T13:52:21.898 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.320523+0000", 2024-03-30T13:52:21.898 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000115386 2024-03-30T13:52:21.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.898 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:52:21.898 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.326234+0000", 2024-03-30T13:52:21.899 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0057114080000000003 2024-03-30T13:52:21.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.899 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.899 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.328662+0000", 2024-03-30T13:52:21.899 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024277489999999999 2024-03-30T13:52:21.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.899 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.899 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.462163+0000", 2024-03-30T13:52:21.899 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.133501329 2024-03-30T13:52:21.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.899 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:52:21.899 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.462219+0000", 2024-03-30T13:52:21.900 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6283000000000003e-05 2024-03-30T13:52:21.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.900 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.900 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:52:21.900 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.462273+0000", 2024-03-30T13:52:21.900 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4203999999999998e-05 2024-03-30T13:52:21.900 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.900 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.900 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.900 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.900 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:249828 8.0 8:053edacb:::benchmark_data_smithi043_119832_object119474:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:52:21.900 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:52:19.319881+0000", 2024-03-30T13:52:21.900 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 1.6904878540000001, 2024-03-30T13:52:21.901 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14255368800000001, 2024-03-30T13:52:21.901 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:21.901 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:21.901 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:21.901 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:52:21.901 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:52:21.901 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:21.901 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 249828 2024-03-30T13:52:21.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.901 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:21.901 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.901 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:21.901 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.319881+0000", 2024-03-30T13:52:21.901 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.901 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.902 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:21.902 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.319881+0000", 2024-03-30T13:52:21.902 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.902 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.902 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:21.902 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.319882+0000", 2024-03-30T13:52:21.902 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.24e-06 2024-03-30T13:52:21.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.902 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.902 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:21.902 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.319888+0000", 2024-03-30T13:52:21.902 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8440000000000003e-06 2024-03-30T13:52:21.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.903 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.903 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:21.903 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.319900+0000", 2024-03-30T13:52:21.903 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1936000000000001e-05 2024-03-30T13:52:21.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.903 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.903 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:21.903 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.319908+0000", 2024-03-30T13:52:21.903 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1980000000000001e-06 2024-03-30T13:52:21.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.903 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.903 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:21.903 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.323365+0000", 2024-03-30T13:52:21.904 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0034576469999999999 2024-03-30T13:52:21.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.904 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:21.904 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.323491+0000", 2024-03-30T13:52:21.904 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000125341 2024-03-30T13:52:21.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.904 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,5", 2024-03-30T13:52:21.904 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.323609+0000", 2024-03-30T13:52:21.904 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011789 2024-03-30T13:52:21.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.904 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.904 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.328696+0000", 2024-03-30T13:52:21.905 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0050871249999999996 2024-03-30T13:52:21.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.905 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:52:21.905 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.448795+0000", 2024-03-30T13:52:21.905 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.12009971899999999 2024-03-30T13:52:21.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.905 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.905 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.462312+0000", 2024-03-30T13:52:21.905 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.013516947 2024-03-30T13:52:21.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.905 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:52:21.905 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.462361+0000", 2024-03-30T13:52:21.906 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8553000000000001e-05 2024-03-30T13:52:21.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.906 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:52:21.906 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.462434+0000", 2024-03-30T13:52:21.906 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3248000000000001e-05 2024-03-30T13:52:21.906 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.906 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.906 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.906 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:249830 8.b 8:d2e496ea:::benchmark_data_smithi043_119832_object119476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:52:21.906 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:52:19.319985+0000", 2024-03-30T13:52:21.906 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 1.6903838850000001, 2024-03-30T13:52:21.907 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13727057400000001, 2024-03-30T13:52:21.907 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:52:21.907 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:52:21.907 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:21.907 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:52:21.907 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:52:21.907 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:21.907 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 249830 2024-03-30T13:52:21.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.907 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:52:21.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.907 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:52:21.907 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.319985+0000", 2024-03-30T13:52:21.907 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.908 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:52:21.908 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.319985+0000", 2024-03-30T13:52:21.908 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:52:21.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.908 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:52:21.908 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.319985+0000", 2024-03-30T13:52:21.908 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4799999999999998e-07 2024-03-30T13:52:21.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.908 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:52:21.908 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.319988+0000", 2024-03-30T13:52:21.909 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.1719999999999999e-06 2024-03-30T13:52:21.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.909 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:52:21.909 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.319995+0000", 2024-03-30T13:52:21.909 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4790000000000001e-06 2024-03-30T13:52:21.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.909 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:52:21.909 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.319999+0000", 2024-03-30T13:52:21.909 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.216e-06 2024-03-30T13:52:21.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.909 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:52:21.909 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.320045+0000", 2024-03-30T13:52:21.910 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6360000000000003e-05 2024-03-30T13:52:21.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.910 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.910 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:52:21.910 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.320198+0000", 2024-03-30T13:52:21.910 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015254999999999999 2024-03-30T13:52:21.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.910 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.910 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-03-30T13:52:21.910 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.320285+0000", 2024-03-30T13:52:21.910 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7028999999999996e-05 2024-03-30T13:52:21.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.910 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.911 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:52:21.911 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.323089+0000", 2024-03-30T13:52:21.911 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002804305 2024-03-30T13:52:21.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.911 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.911 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.323808+0000", 2024-03-30T13:52:21.911 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00071830899999999998 2024-03-30T13:52:21.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.911 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:21.911 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.457145+0000", 2024-03-30T13:52:21.911 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13333732100000001 2024-03-30T13:52:21.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.912 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.912 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:52:21.912 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.457206+0000", 2024-03-30T13:52:21.912 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0618999999999997e-05 2024-03-30T13:52:21.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:52:21.912 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:52:21.912 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:52:21.912 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.457255+0000", 2024-03-30T13:52:21.912 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9565999999999999e-05 2024-03-30T13:52:21.912 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.912 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.912 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.912 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:52:21.912 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:52:21.913 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:52:21.913 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:21.944 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:21.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:21.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:21.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:21 smithi113 ceph-mon[34341]: 4.1b scrub starts 2024-03-30T13:52:21.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:21 smithi113 ceph-mon[34341]: 4.1b scrub ok 2024-03-30T13:52:21.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:21 smithi113 ceph-mon[34341]: 8.c scrub starts 2024-03-30T13:52:21.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:21 smithi113 ceph-mon[34341]: 8.9 scrub starts 2024-03-30T13:52:21.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:21 smithi113 ceph-mon[34341]: 8.9 scrub ok 2024-03-30T13:52:22.046 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:22.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:22.083 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:22.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:22.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:21 smithi043 ceph-mon[29963]: 4.1b scrub starts 2024-03-30T13:52:22.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:21 smithi043 ceph-mon[29963]: 4.1b scrub ok 2024-03-30T13:52:22.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:21 smithi043 ceph-mon[29963]: 8.c scrub starts 2024-03-30T13:52:22.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:21 smithi043 ceph-mon[29963]: 8.9 scrub starts 2024-03-30T13:52:22.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:21 smithi043 ceph-mon[29963]: 8.9 scrub ok 2024-03-30T13:52:22.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:21 smithi043 ceph-mon[42099]: 4.1b scrub starts 2024-03-30T13:52:22.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:21 smithi043 ceph-mon[42099]: 4.1b scrub ok 2024-03-30T13:52:22.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:21 smithi043 ceph-mon[42099]: 8.c scrub starts 2024-03-30T13:52:22.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:21 smithi043 ceph-mon[42099]: 8.9 scrub starts 2024-03-30T13:52:22.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:21 smithi043 ceph-mon[42099]: 8.9 scrub ok 2024-03-30T13:52:22.184 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:22.216 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:22.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:22.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:22.318 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:22.351 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:22.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:22.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:22.453 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:22.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:22.491 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:22.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:22.592 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:22.626 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:22.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:22.727 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:22.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:22.763 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:22.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:22.864 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:22.894 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:22.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:22.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:22.933 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:22 smithi043 ceph-mon[29963]: 6.c deep-scrub starts 2024-03-30T13:52:22.933 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:22 smithi043 ceph-mon[29963]: 6.c deep-scrub ok 2024-03-30T13:52:22.933 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:22 smithi043 ceph-mon[29963]: 8.c scrub ok 2024-03-30T13:52:22.933 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:22 smithi043 ceph-mon[29963]: pgmap v389: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 710 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.45k op/s; 4258/34626 objects degraded (12.297%) 2024-03-30T13:52:22.933 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:22 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 4258/34626 objects degraded (12.297%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:22.933 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:22 smithi043 ceph-mon[42099]: 6.c deep-scrub starts 2024-03-30T13:52:22.933 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:22 smithi043 ceph-mon[42099]: 6.c deep-scrub ok 2024-03-30T13:52:22.933 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:22 smithi043 ceph-mon[42099]: 8.c scrub ok 2024-03-30T13:52:22.933 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:22 smithi043 ceph-mon[42099]: pgmap v389: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 710 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.45k op/s; 4258/34626 objects degraded (12.297%) 2024-03-30T13:52:22.934 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:22 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 4258/34626 objects degraded (12.297%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:22.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:22 smithi113 ceph-mon[34341]: 6.c deep-scrub starts 2024-03-30T13:52:22.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:22 smithi113 ceph-mon[34341]: 6.c deep-scrub ok 2024-03-30T13:52:22.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:22 smithi113 ceph-mon[34341]: 8.c scrub ok 2024-03-30T13:52:22.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:22 smithi113 ceph-mon[34341]: pgmap v389: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 710 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.45k op/s; 4258/34626 objects degraded (12.297%) 2024-03-30T13:52:22.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:22 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 4258/34626 objects degraded (12.297%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:22.996 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:23.027 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:23.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:23.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:23.129 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:23.165 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:23.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:23.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:23.267 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:23.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:23.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:23.402 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:23.438 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:23.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:23.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:23.525 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-30T13:52:23.539 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:23.579 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:23.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:23.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:23.681 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:23.724 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:23.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:23.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:23.733 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:23 smithi043 ceph-mon[42099]: 5.17 scrub starts 2024-03-30T13:52:23.734 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:23 smithi043 ceph-mon[42099]: 5.17 scrub ok 2024-03-30T13:52:23.734 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:23 smithi043 ceph-mon[42099]: 5.5 scrub starts 2024-03-30T13:52:23.734 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:23 smithi043 ceph-mon[42099]: 5.5 scrub ok 2024-03-30T13:52:23.734 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:23 smithi043 ceph-mon[29963]: 5.17 scrub starts 2024-03-30T13:52:23.734 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:23 smithi043 ceph-mon[29963]: 5.17 scrub ok 2024-03-30T13:52:23.734 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:23 smithi043 ceph-mon[29963]: 5.5 scrub starts 2024-03-30T13:52:23.734 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:23 smithi043 ceph-mon[29963]: 5.5 scrub ok 2024-03-30T13:52:23.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:23.859 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:23.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:23.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:23.937 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:52:23.960 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:23.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:23 smithi113 ceph-mon[34341]: 5.17 scrub starts 2024-03-30T13:52:23.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:23 smithi113 ceph-mon[34341]: 5.17 scrub ok 2024-03-30T13:52:23.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:23 smithi113 ceph-mon[34341]: 5.5 scrub starts 2024-03-30T13:52:23.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:23 smithi113 ceph-mon[34341]: 5.5 scrub ok 2024-03-30T13:52:23.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:23.992 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:23.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:24.052 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:52:24.052 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:52:24.066 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:24.093 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:24.132 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:24.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:24.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:24.234 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:24.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:24.279 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:24.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:24.380 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:24.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:24.415 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:24.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:24.516 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:24.550 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:24.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:24.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:24.652 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:24.689 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:24.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:24.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:24.782 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:24.790 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:24.825 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:24.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:24.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:24.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:24 smithi043 ceph-mon[29963]: 6.1b scrub starts 2024-03-30T13:52:24.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:24 smithi043 ceph-mon[29963]: 6.1b scrub ok 2024-03-30T13:52:24.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:24 smithi043 ceph-mon[29963]: pgmap v390: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 710 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 4258/34626 objects degraded (12.297%) 2024-03-30T13:52:24.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:24 smithi043 ceph-mon[42099]: 6.1b scrub starts 2024-03-30T13:52:24.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:24 smithi043 ceph-mon[42099]: 6.1b scrub ok 2024-03-30T13:52:24.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:24 smithi043 ceph-mon[42099]: pgmap v390: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 710 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 4258/34626 objects degraded (12.297%) 2024-03-30T13:52:24.927 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:24.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:24.964 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:24.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:24.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:24 smithi113 ceph-mon[34341]: 6.1b scrub starts 2024-03-30T13:52:24.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:24 smithi113 ceph-mon[34341]: 6.1b scrub ok 2024-03-30T13:52:24.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:24 smithi113 ceph-mon[34341]: pgmap v390: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 710 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 4258/34626 objects degraded (12.297%) 2024-03-30T13:52:25.065 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:25.099 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:25.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:25.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:25.201 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:25.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:25.256 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:25.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:25.357 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:25.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:25.395 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:25.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:25.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:25.531 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:25.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:25.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:25.633 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:25.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:25.689 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:25.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:25.790 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:25.827 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:25.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:25.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:25.930 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:25.956 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:25 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:52:25.956 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:25 smithi113 ceph-mon[34341]: 3.a scrub starts 2024-03-30T13:52:25.957 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:25 smithi113 ceph-mon[34341]: 3.a scrub ok 2024-03-30T13:52:25.967 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:25.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:25.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:26.046 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:52:26.046 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [ 2024-03-30T13:52:26.046 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.047 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:258031 8.2 8:42d9c5a4:::benchmark_data_smithi043_119832_object127677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:52:26.047 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:26.039784+0000", 2024-03-30T13:52:26.047 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.0059974169999999997, 2024-03-30T13:52:26.047 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0060336390000000004, 2024-03-30T13:52:26.047 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:26.047 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:26.047 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "waiting for sub ops", 2024-03-30T13:52:26.047 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:26.047 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:52:26.047 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:26.047 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 258031 2024-03-30T13:52:26.047 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.047 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:26.048 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.048 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:26.048 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.039784+0000", 2024-03-30T13:52:26.048 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:26.048 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.048 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.048 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:26.048 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.039784+0000", 2024-03-30T13:52:26.048 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:26.048 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.048 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.048 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:26.049 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.039786+0000", 2024-03-30T13:52:26.049 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.8309999999999999e-06 2024-03-30T13:52:26.049 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.049 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.049 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:26.049 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.039794+0000", 2024-03-30T13:52:26.049 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6200000000000005e-06 2024-03-30T13:52:26.049 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.049 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.049 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:26.049 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.039817+0000", 2024-03-30T13:52:26.049 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.3217000000000001e-05 2024-03-30T13:52:26.049 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.050 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.050 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:26.050 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.039828+0000", 2024-03-30T13:52:26.050 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1123e-05 2024-03-30T13:52:26.050 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.050 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.050 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:26.050 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.039894+0000", 2024-03-30T13:52:26.050 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6031999999999994e-05 2024-03-30T13:52:26.050 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.050 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.050 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:26.050 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.040025+0000", 2024-03-30T13:52:26.050 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013099499999999999 2024-03-30T13:52:26.050 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.051 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.051 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,5", 2024-03-30T13:52:26.051 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.040152+0000", 2024-03-30T13:52:26.051 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012631599999999999 2024-03-30T13:52:26.051 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.051 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.051 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:26.051 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.045795+0000", 2024-03-30T13:52:26.051 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0056429180000000002 2024-03-30T13:52:26.051 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:26.051 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:26.051 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:26.051 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.051 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.051 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop(client.24977.0:258035 8.e e200/167 8:7f44b2ce:::benchmark_data_smithi043_119832_object127681:head v 200'16262, mlcod=200'16258)", 2024-03-30T13:52:26.052 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:26.041038+0000", 2024-03-30T13:52:26.052 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.004742802, 2024-03-30T13:52:26.052 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0048988740000000001, 2024-03-30T13:52:26.052 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:26.052 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:26.052 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:52:26.052 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:26.052 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.052 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:26.052 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.041038+0000", 2024-03-30T13:52:26.052 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:26.052 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.052 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.052 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:26.052 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.041038+0000", 2024-03-30T13:52:26.052 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:26.052 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.053 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.053 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:26.053 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.041039+0000", 2024-03-30T13:52:26.053 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0210000000000001e-06 2024-03-30T13:52:26.053 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.053 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.053 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:26.053 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.041046+0000", 2024-03-30T13:52:26.053 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.4320000000000004e-06 2024-03-30T13:52:26.053 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.053 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.053 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:26.053 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.041060+0000", 2024-03-30T13:52:26.053 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3817e-05 2024-03-30T13:52:26.053 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.053 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.054 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:26.054 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.041067+0000", 2024-03-30T13:52:26.054 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.3459999999999996e-06 2024-03-30T13:52:26.054 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.054 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.054 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:26.054 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.041338+0000", 2024-03-30T13:52:26.054 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00027106 2024-03-30T13:52:26.054 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.054 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.054 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:26.054 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.041385+0000", 2024-03-30T13:52:26.054 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7241999999999999e-05 2024-03-30T13:52:26.054 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:26.055 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:26.055 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:26.055 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.055 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.055 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop(client.24977.0:258037 8.5 e200/147 8:a1ec1715:::benchmark_data_smithi043_119832_object127683:head v 200'18376, mlcod=200'18374)", 2024-03-30T13:52:26.055 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:26.041101+0000", 2024-03-30T13:52:26.055 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.0046802370000000003, 2024-03-30T13:52:26.055 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0049100819999999996, 2024-03-30T13:52:26.055 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:26.055 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:26.055 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:52:26.055 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:26.055 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.055 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:26.055 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.041101+0000", 2024-03-30T13:52:26.056 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:26.056 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.056 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.056 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:26.056 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.041101+0000", 2024-03-30T13:52:26.056 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:26.056 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.056 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.056 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:26.056 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.041101+0000", 2024-03-30T13:52:26.056 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.0400000000000002e-07 2024-03-30T13:52:26.056 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.056 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.056 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:26.056 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.041106+0000", 2024-03-30T13:52:26.056 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6750000000000001e-06 2024-03-30T13:52:26.056 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.057 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.057 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:26.057 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.041113+0000", 2024-03-30T13:52:26.057 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8109999999999997e-06 2024-03-30T13:52:26.057 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.057 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.057 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:26.057 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.041118+0000", 2024-03-30T13:52:26.057 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.006e-06 2024-03-30T13:52:26.057 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.057 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.057 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:26.058 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.042163+0000", 2024-03-30T13:52:26.058 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0010448440000000001 2024-03-30T13:52:26.058 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.058 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.058 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:26.058 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.042184+0000", 2024-03-30T13:52:26.058 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.1758000000000001e-05 2024-03-30T13:52:26.058 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:26.058 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:26.058 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:26.058 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.058 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.058 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:258042 8.2 8.b51ce982 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:52:26.058 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:26.043453+0000", 2024-03-30T13:52:26.059 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.0023277609999999998, 2024-03-30T13:52:26.059 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.002626729, 2024-03-30T13:52:26.059 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:26.059 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:26.059 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "queued for pg", 2024-03-30T13:52:26.059 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:26.059 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:52:26.059 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:26.059 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 258042 2024-03-30T13:52:26.059 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.059 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:26.059 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.059 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:26.059 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.043453+0000", 2024-03-30T13:52:26.059 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:26.060 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.060 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.060 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:26.060 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.043453+0000", 2024-03-30T13:52:26.060 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:26.060 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.060 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.060 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:26.060 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.043455+0000", 2024-03-30T13:52:26.060 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.305e-06 2024-03-30T13:52:26.060 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.060 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.060 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:26.061 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.043460+0000", 2024-03-30T13:52:26.061 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6579999999999999e-06 2024-03-30T13:52:26.061 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.061 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.061 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:26.061 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.043475+0000", 2024-03-30T13:52:26.061 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4462e-05 2024-03-30T13:52:26.061 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.061 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.061 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:26.062 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.043483+0000", 2024-03-30T13:52:26.062 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6349999999999995e-06 2024-03-30T13:52:26.062 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:26.062 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:26.062 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:26.062 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.062 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.062 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop_reply(client.24977.0:258031 8.2 e200/147)", 2024-03-30T13:52:26.062 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:26.044230+0000", 2024-03-30T13:52:26.062 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.001550832, 2024-03-30T13:52:26.062 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001908187, 2024-03-30T13:52:26.062 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:26.062 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:26.062 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "queued for pg", 2024-03-30T13:52:26.063 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:26.063 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.063 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:26.063 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.044230+0000", 2024-03-30T13:52:26.063 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:26.063 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.063 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.063 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:26.063 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.044230+0000", 2024-03-30T13:52:26.063 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:26.063 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.063 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.064 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:26.064 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.044231+0000", 2024-03-30T13:52:26.064 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.61e-07 2024-03-30T13:52:26.064 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.064 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.064 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:26.064 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.044236+0000", 2024-03-30T13:52:26.064 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8300000000000003e-06 2024-03-30T13:52:26.064 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.064 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.064 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:26.064 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.044248+0000", 2024-03-30T13:52:26.064 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2303999999999999e-05 2024-03-30T13:52:26.065 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:26.065 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:26.065 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:26.065 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.044255+0000", 2024-03-30T13:52:26.065 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.516e-06 2024-03-30T13:52:26.065 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:26.065 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:26.065 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:26.065 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:26.065 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-03-30T13:52:26.065 INFO:teuthology.orchestra.run.smithi113.stdout: "num_ops": 5 2024-03-30T13:52:26.065 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:52:26.066 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:52:26.066 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:25 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:52:26.067 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:25 smithi043 ceph-mon[29963]: 3.a scrub starts 2024-03-30T13:52:26.067 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:25 smithi043 ceph-mon[29963]: 3.a scrub ok 2024-03-30T13:52:26.067 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:25 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:52:26.067 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:25 smithi043 ceph-mon[42099]: 3.a scrub starts 2024-03-30T13:52:26.067 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:25 smithi043 ceph-mon[42099]: 3.a scrub ok 2024-03-30T13:52:26.069 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:26.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:26.172 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:26.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:26.273 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:26.298 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:26.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:26.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:26.397 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:52:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:52:26.399 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:26.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:26.433 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:26.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:26.534 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:26.567 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:26.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:26.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:26.669 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:26.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:26.709 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:26.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:26.810 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:26.847 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:26.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:26.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:26.894 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-03-30T13:52:26.949 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:26.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:26 smithi113 ceph-mon[34341]: pgmap v391: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 334 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 2075/16587 objects degraded (12.510%) 2024-03-30T13:52:26.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:26 smithi113 ceph-mon[34341]: 4.7 scrub starts 2024-03-30T13:52:26.972 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:26 smithi113 ceph-mon[34341]: 4.7 scrub ok 2024-03-30T13:52:26.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:26 smithi113 ceph-mon[34341]: 1.0 scrub starts 2024-03-30T13:52:26.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:26 smithi113 ceph-mon[34341]: 1.0 scrub ok 2024-03-30T13:52:26.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:26 smithi113 ceph-mon[34341]: 4.1f scrub starts 2024-03-30T13:52:26.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:26 smithi113 ceph-mon[34341]: 4.1f scrub ok 2024-03-30T13:52:26.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:26 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:52:26.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:26 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:52:26.973 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:26 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:52:27.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:27.001 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:27.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:27.102 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:27.128 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-mon[29963]: pgmap v391: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 334 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 2075/16587 objects degraded (12.510%) 2024-03-30T13:52:27.128 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-mon[29963]: 4.7 scrub starts 2024-03-30T13:52:27.128 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-mon[29963]: 4.7 scrub ok 2024-03-30T13:52:27.128 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-mon[29963]: 1.0 scrub starts 2024-03-30T13:52:27.128 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-mon[29963]: 1.0 scrub ok 2024-03-30T13:52:27.128 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-mon[29963]: 4.1f scrub starts 2024-03-30T13:52:27.128 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-mon[29963]: 4.1f scrub ok 2024-03-30T13:52:27.128 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:52:27.128 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:52:27.128 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:52:27.128 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-mon[42099]: pgmap v391: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 334 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 2075/16587 objects degraded (12.510%) 2024-03-30T13:52:27.129 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-mon[42099]: 4.7 scrub starts 2024-03-30T13:52:27.129 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-mon[42099]: 4.7 scrub ok 2024-03-30T13:52:27.129 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-mon[42099]: 1.0 scrub starts 2024-03-30T13:52:27.129 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-mon[42099]: 1.0 scrub ok 2024-03-30T13:52:27.129 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-mon[42099]: 4.1f scrub starts 2024-03-30T13:52:27.129 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-mon[42099]: 4.1f scrub ok 2024-03-30T13:52:27.129 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:52:27.129 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:52:27.129 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:26 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:52:27.139 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:27.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:27.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:27.240 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:27.278 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:27.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:27.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:27.286 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:52:27.380 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:27.418 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:27.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:27.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:27.520 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:27.554 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:27.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:27.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:27.656 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:27.694 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:27.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:27.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:27.796 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:27.829 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:27.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:27.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:27.931 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:27.965 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:27.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:27.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:27.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:27 smithi113 ceph-mon[34341]: 6.13 scrub starts 2024-03-30T13:52:27.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:27 smithi113 ceph-mon[34341]: 6.13 scrub ok 2024-03-30T13:52:27.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:27 smithi113 ceph-mon[34341]: 6.18 deep-scrub starts 2024-03-30T13:52:27.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:27 smithi113 ceph-mon[34341]: 3.8 scrub starts 2024-03-30T13:52:27.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:27 smithi113 ceph-mon[34341]: 3.8 scrub ok 2024-03-30T13:52:27.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:27 smithi113 ceph-mon[34341]: 6.18 deep-scrub ok 2024-03-30T13:52:27.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:27 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 2075/16587 objects degraded (12.510%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:28.067 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:28.101 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:28.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:28.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:27 smithi043 ceph-mon[29963]: 6.13 scrub starts 2024-03-30T13:52:28.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:27 smithi043 ceph-mon[29963]: 6.13 scrub ok 2024-03-30T13:52:28.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:27 smithi043 ceph-mon[29963]: 6.18 deep-scrub starts 2024-03-30T13:52:28.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:27 smithi043 ceph-mon[29963]: 3.8 scrub starts 2024-03-30T13:52:28.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:27 smithi043 ceph-mon[29963]: 3.8 scrub ok 2024-03-30T13:52:28.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:27 smithi043 ceph-mon[29963]: 6.18 deep-scrub ok 2024-03-30T13:52:28.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:27 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 2075/16587 objects degraded (12.510%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:28.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:27 smithi043 ceph-mon[42099]: 6.13 scrub starts 2024-03-30T13:52:28.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:27 smithi043 ceph-mon[42099]: 6.13 scrub ok 2024-03-30T13:52:28.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:27 smithi043 ceph-mon[42099]: 6.18 deep-scrub starts 2024-03-30T13:52:28.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:27 smithi043 ceph-mon[42099]: 3.8 scrub starts 2024-03-30T13:52:28.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:27 smithi043 ceph-mon[42099]: 3.8 scrub ok 2024-03-30T13:52:28.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:27 smithi043 ceph-mon[42099]: 6.18 deep-scrub ok 2024-03-30T13:52:28.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:27 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 2075/16587 objects degraded (12.510%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:28.199 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Removed 130351 objects 2024-03-30T13:52:28.199 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Clean up completed and total clean up time :103.212 2024-03-30T13:52:28.203 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:28.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:28.226 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:28.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:28.327 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:28.350 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:28.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:28.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:28.452 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:28.476 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:28.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:28.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:28.578 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:28.603 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:28.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:28.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:28.705 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:28.731 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:28.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:28.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:28.833 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:28.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[29963]: pgmap v392: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 334 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1020 B/s rd, 0 B/s wr, 1.18k op/s; 2075/16587 objects degraded (12.510%) 2024-03-30T13:52:28.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[29963]: 6.15 scrub starts 2024-03-30T13:52:28.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[29963]: 6.15 scrub ok 2024-03-30T13:52:28.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[29963]: 6.b scrub starts 2024-03-30T13:52:28.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[29963]: 6.7 scrub starts 2024-03-30T13:52:28.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[29963]: 6.7 scrub ok 2024-03-30T13:52:28.839 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:52:28.839 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2251071288' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-30T13:52:28.839 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-30T13:52:28.839 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[29963]: 3.19 scrub starts 2024-03-30T13:52:28.839 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[29963]: 3.19 scrub ok 2024-03-30T13:52:28.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[42099]: pgmap v392: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 334 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1020 B/s rd, 0 B/s wr, 1.18k op/s; 2075/16587 objects degraded (12.510%) 2024-03-30T13:52:28.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[42099]: 6.15 scrub starts 2024-03-30T13:52:28.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[42099]: 6.15 scrub ok 2024-03-30T13:52:28.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[42099]: 6.b scrub starts 2024-03-30T13:52:28.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[42099]: 6.7 scrub starts 2024-03-30T13:52:28.839 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[42099]: 6.7 scrub ok 2024-03-30T13:52:28.840 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:52:28.840 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2251071288' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-30T13:52:28.840 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-30T13:52:28.840 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[42099]: 3.19 scrub starts 2024-03-30T13:52:28.840 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:28 smithi043 ceph-mon[42099]: 3.19 scrub ok 2024-03-30T13:52:28.844 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:28.857 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:28.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:28.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:28.958 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:28.970 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:52:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:52:28.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:28 smithi113 ceph-mon[34341]: pgmap v392: 148 pgs: 18 active+undersized+degraded, 26 active+undersized, 104 active+clean; 334 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1020 B/s rd, 0 B/s wr, 1.18k op/s; 2075/16587 objects degraded (12.510%) 2024-03-30T13:52:28.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:28 smithi113 ceph-mon[34341]: 6.15 scrub starts 2024-03-30T13:52:28.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:28 smithi113 ceph-mon[34341]: 6.15 scrub ok 2024-03-30T13:52:28.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:28 smithi113 ceph-mon[34341]: 6.b scrub starts 2024-03-30T13:52:28.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:28 smithi113 ceph-mon[34341]: 6.7 scrub starts 2024-03-30T13:52:28.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:28 smithi113 ceph-mon[34341]: 6.7 scrub ok 2024-03-30T13:52:28.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:52:28.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:28 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2251071288' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-30T13:52:28.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:28 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-30T13:52:28.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:28 smithi113 ceph-mon[34341]: 3.19 scrub starts 2024-03-30T13:52:28.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:28 smithi113 ceph-mon[34341]: 3.19 scrub ok 2024-03-30T13:52:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:28.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:28.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:29.083 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:29.109 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:29.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:29.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:29.211 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:29.237 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:29.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:29.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:29.338 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:29.365 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:29.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:29.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:29.467 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:29.471 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-03-30T13:52:29.472 INFO:teuthology.orchestra.run.smithi043.stderr:set pool 1 pg_num to 11 2024-03-30T13:52:29.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:29.495 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:29.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:29.596 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:29.624 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:29.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:29.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:29.726 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:29.751 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:29.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:29.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:29.853 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:29.879 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:29.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:29.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:29 smithi043 ceph-mon[42099]: 6.b scrub ok 2024-03-30T13:52:29.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:29 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:52:29.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:29 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2035199575' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:52:29.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:29 smithi043 ceph-mon[42099]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:52:29.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:29 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-30T13:52:29.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:29 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2035199575' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:52:29.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:29 smithi043 ceph-mon[42099]: osdmap e201: 8 total, 7 up, 8 in 2024-03-30T13:52:29.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:29 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-30T13:52:29.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:29 smithi043 ceph-mon[29963]: 6.b scrub ok 2024-03-30T13:52:29.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:29 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:52:29.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:29 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2035199575' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:52:29.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:29 smithi043 ceph-mon[29963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:52:29.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:29 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-30T13:52:29.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:29 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2035199575' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:52:29.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:29 smithi043 ceph-mon[29963]: osdmap e201: 8 total, 7 up, 8 in 2024-03-30T13:52:29.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:29 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-30T13:52:29.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:29 smithi113 ceph-mon[34341]: 6.b scrub ok 2024-03-30T13:52:29.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:29 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:52:29.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:29 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2035199575' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:52:29.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:29 smithi113 ceph-mon[34341]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:52:29.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:29 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-30T13:52:29.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:29 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2035199575' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:52:29.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:29 smithi113 ceph-mon[34341]: osdmap e201: 8 total, 7 up, 8 in 2024-03-30T13:52:29.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:29 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-30T13:52:29.980 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:30.006 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:30.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:30.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:30.108 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:30.134 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:30.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:30.236 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:30.264 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:30.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:30.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:30.327 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-03-30T13:52:30.327 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-03-30T13:52:30.366 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:30.396 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:30.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:30.497 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:30.524 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:30.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:30.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:30.626 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:30.649 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:30.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:30.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:30.672 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:52:30.672 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [], 2024-03-30T13:52:30.672 INFO:teuthology.orchestra.run.smithi113.stdout: "complaint_time": 30, 2024-03-30T13:52:30.672 INFO:teuthology.orchestra.run.smithi113.stdout: "num_blocked_ops": 0 2024-03-30T13:52:30.672 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:52:30.752 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:30.775 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:30.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:30.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:30.877 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:30.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:30.902 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:30.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:30.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:30 smithi113 ceph-mon[34341]: pgmap v394: 148 pgs: 15 active+undersized+degraded, 29 active+undersized, 104 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 56/594 objects degraded (9.428%) 2024-03-30T13:52:30.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:30 smithi113 ceph-mon[34341]: 5.1a scrub starts 2024-03-30T13:52:30.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-30T13:52:30.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:30 smithi113 ceph-mon[34341]: osdmap e202: 8 total, 7 up, 8 in 2024-03-30T13:52:31.003 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:31.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:31.028 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:31.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:31.062 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:30 smithi043 ceph-mon[29963]: pgmap v394: 148 pgs: 15 active+undersized+degraded, 29 active+undersized, 104 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 56/594 objects degraded (9.428%) 2024-03-30T13:52:31.062 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:30 smithi043 ceph-mon[29963]: 5.1a scrub starts 2024-03-30T13:52:31.062 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-30T13:52:31.062 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:30 smithi043 ceph-mon[29963]: osdmap e202: 8 total, 7 up, 8 in 2024-03-30T13:52:31.062 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:30 smithi043 ceph-mon[42099]: pgmap v394: 148 pgs: 15 active+undersized+degraded, 29 active+undersized, 104 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 56/594 objects degraded (9.428%) 2024-03-30T13:52:31.062 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:30 smithi043 ceph-mon[42099]: 5.1a scrub starts 2024-03-30T13:52:31.062 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-30T13:52:31.063 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:30 smithi043 ceph-mon[42099]: osdmap e202: 8 total, 7 up, 8 in 2024-03-30T13:52:31.128 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:31.151 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:31.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:31.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:31.253 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:31.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:31.279 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:31.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:31.365 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-03-30T13:52:31.380 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:31.431 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:31.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:31.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:31.533 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:31.560 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:31.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:31.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:31.612 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:52:31.662 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:31.685 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:31.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:31.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:31.787 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:31.810 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:31.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:31.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:31.912 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:31.936 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:31.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:31.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:32.038 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:32.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:32.063 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:32.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:32.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:32.190 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:32.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:32.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:32.222 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:32.228 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:52:32.249 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:52:32.270 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:52:32.290 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:52:32.293 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:32.311 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:52:32.316 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:32.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:32.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:32.331 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:52:32.352 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:52:32.353 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:52:32.353 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:52:32.353 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:52:32.353 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:52:32.353 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:52:32.353 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:52:32.353 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:52:32.353 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:52:32.418 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:32.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:32.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:32.543 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:32.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:32.567 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:32.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:32.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:32 smithi043 ceph-mon[29963]: osdmap e203: 8 total, 7 up, 8 in 2024-03-30T13:52:32.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:32 smithi043 ceph-mon[29963]: pgmap v397: 158 pgs: 10 unknown, 15 active+undersized+degraded, 29 active+undersized, 104 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 888 op/s; 56/594 objects degraded (9.428%) 2024-03-30T13:52:32.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:32 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-30T13:52:32.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:32 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3473893594' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:52:32.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:32 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 56/594 objects degraded (9.428%), 15 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:32.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:32 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-30T13:52:32.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:32 smithi043 ceph-mon[29963]: osdmap e204: 8 total, 7 up, 8 in 2024-03-30T13:52:32.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:32 smithi043 ceph-mon[42099]: osdmap e203: 8 total, 7 up, 8 in 2024-03-30T13:52:32.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:32 smithi043 ceph-mon[42099]: pgmap v397: 158 pgs: 10 unknown, 15 active+undersized+degraded, 29 active+undersized, 104 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 888 op/s; 56/594 objects degraded (9.428%) 2024-03-30T13:52:32.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:32 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-30T13:52:32.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:32 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3473893594' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:52:32.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:32 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 56/594 objects degraded (9.428%), 15 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:32.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:32 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-30T13:52:32.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:32 smithi043 ceph-mon[42099]: osdmap e204: 8 total, 7 up, 8 in 2024-03-30T13:52:32.668 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:32.692 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:32.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:32.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:32.780 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:32 smithi113 ceph-mon[34341]: osdmap e203: 8 total, 7 up, 8 in 2024-03-30T13:52:32.780 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:32 smithi113 ceph-mon[34341]: pgmap v397: 158 pgs: 10 unknown, 15 active+undersized+degraded, 29 active+undersized, 104 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 888 op/s; 56/594 objects degraded (9.428%) 2024-03-30T13:52:32.780 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:32 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-30T13:52:32.780 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:32 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3473893594' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:52:32.780 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:32 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 56/594 objects degraded (9.428%), 15 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:32.780 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:32 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-30T13:52:32.780 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:32 smithi113 ceph-mon[34341]: osdmap e204: 8 total, 7 up, 8 in 2024-03-30T13:52:32.784 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T13:52:32.793 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:32.820 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:32.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:32.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:32.922 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:32.949 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:32.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:32.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:33.051 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:33.074 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:33.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:33.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:33.176 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:33.199 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:33.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:33.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:33.301 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:33.325 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:33.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:33.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:33.426 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:33.450 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:33.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:33.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:33.551 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:33.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:33.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:33.591 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:52:33.591 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:52:33.635 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:52:33.635 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:52:33.678 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:33.703 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:33.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:33.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:33.805 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:33.829 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:33.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:33.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:33.931 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:33.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:33.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:34.056 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:34.080 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:34.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:34.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:34.127 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:34.183 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:34.207 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:34.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:34.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:34.309 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:34.320 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-30T13:52:34.334 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:34.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:34.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:34.436 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:34.459 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:34.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:34.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:34.561 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:34.562 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:52:34.586 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:34.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:34.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:34.636 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:34 smithi043 ceph-mon[29963]: osdmap e205: 8 total, 7 up, 8 in 2024-03-30T13:52:34.636 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:34 smithi043 ceph-mon[29963]: pgmap v400: 158 pgs: 10 unknown, 15 active+undersized+degraded, 29 active+undersized, 104 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 56/594 objects degraded (9.428%) 2024-03-30T13:52:34.637 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:34 smithi043 ceph-mon[42099]: osdmap e205: 8 total, 7 up, 8 in 2024-03-30T13:52:34.637 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:34 smithi043 ceph-mon[42099]: pgmap v400: 158 pgs: 10 unknown, 15 active+undersized+degraded, 29 active+undersized, 104 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 56/594 objects degraded (9.428%) 2024-03-30T13:52:34.683 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:34 smithi113 ceph-mon[34341]: osdmap e205: 8 total, 7 up, 8 in 2024-03-30T13:52:34.683 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:34 smithi113 ceph-mon[34341]: pgmap v400: 158 pgs: 10 unknown, 15 active+undersized+degraded, 29 active+undersized, 104 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 56/594 objects degraded (9.428%) 2024-03-30T13:52:34.688 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:34.711 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:34.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:34.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:34.813 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:34.837 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:34.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:34.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:34.939 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:34.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:34.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:35.065 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:35.088 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:35.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:35.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:35.190 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:35.214 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:35.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:35.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:35.316 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:35.328 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-30T13:52:35.328 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:52:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:35.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:35.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:35.441 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:35.465 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:35.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:35.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:35.493 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'unique_pool_1' removed 2024-03-30T13:52:35.567 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:35.603 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:35.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:35.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:35.637 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:52:35.705 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:35.719 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:35 smithi113 ceph-mon[34341]: osdmap e206: 8 total, 7 up, 8 in 2024-03-30T13:52:35.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:35 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/577147822' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T13:52:35.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:35 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T13:52:35.728 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:35.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:35.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:35.742 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:35 smithi043 ceph-mon[29963]: osdmap e206: 8 total, 7 up, 8 in 2024-03-30T13:52:35.742 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:35 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/577147822' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T13:52:35.742 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:35 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T13:52:35.742 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:35 smithi043 ceph-mon[42099]: osdmap e206: 8 total, 7 up, 8 in 2024-03-30T13:52:35.743 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:35 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/577147822' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T13:52:35.743 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:35 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T13:52:35.830 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:35.855 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:35.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:35.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:35.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:35.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:35.982 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:35.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:36.083 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:36.107 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:36.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:36.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:36.209 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:36.233 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:36.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:36.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:36.315 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:52:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:52:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:52:36.320 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:52:36.320 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:52:36.336 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:36.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:36.360 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:36.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:36.461 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:36.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:36.491 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:36.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:36.592 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:36.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:36.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:36.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:36 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T13:52:36.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:36 smithi043 ceph-mon[29963]: osdmap e207: 8 total, 7 up, 8 in 2024-03-30T13:52:36.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:36 smithi043 ceph-mon[29963]: pgmap v403: 142 pgs: 10 remapped+peering, 12 active+undersized+degraded, 26 active+undersized, 94 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 56/594 objects degraded (9.428%) 2024-03-30T13:52:36.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:36 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T13:52:36.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:36 smithi043 ceph-mon[42099]: osdmap e207: 8 total, 7 up, 8 in 2024-03-30T13:52:36.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:36 smithi043 ceph-mon[42099]: pgmap v403: 142 pgs: 10 remapped+peering, 12 active+undersized+degraded, 26 active+undersized, 94 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 56/594 objects degraded (9.428%) 2024-03-30T13:52:36.719 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:36.742 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:36.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:36.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:36.844 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:36.867 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:36.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:36.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:36.909 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-30T13:52:36.910 INFO:tasks.radosbench:Beginning radosbench... 2024-03-30T13:52:36.910 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-03-30T13:52:36.910 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool create unique_pool_2 16 2024-03-30T13:52:36.969 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:36.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:36 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T13:52:36.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:36 smithi113 ceph-mon[34341]: osdmap e207: 8 total, 7 up, 8 in 2024-03-30T13:52:36.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:36 smithi113 ceph-mon[34341]: pgmap v403: 142 pgs: 10 remapped+peering, 12 active+undersized+degraded, 26 active+undersized, 94 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 56/594 objects degraded (9.428%) 2024-03-30T13:52:37.017 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:37.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:37.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:37.032 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:37.096 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-03-30T13:52:37.119 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:37.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:37.162 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:37.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:37.263 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:37.287 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:37.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:37.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:37.336 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:52:37.389 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:37.412 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:37.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:37.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:37.499 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-03-30T13:52:37.514 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:37.537 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:37.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:37.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:37.639 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:37.663 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:37.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:37.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:37.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:37 smithi113 ceph-mon[34341]: Health check failed: Reduced data availability: 8 pgs peering (PG_AVAILABILITY) 2024-03-30T13:52:37.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:37 smithi113 ceph-mon[34341]: osdmap e208: 8 total, 7 up, 8 in 2024-03-30T13:52:37.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:37 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2265415222' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:52:37.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:37 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:52:37.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:37 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 56/594 objects degraded (9.428%), 12 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:37.738 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:37 smithi043 ceph-mon[29963]: Health check failed: Reduced data availability: 8 pgs peering (PG_AVAILABILITY) 2024-03-30T13:52:37.738 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:37 smithi043 ceph-mon[29963]: osdmap e208: 8 total, 7 up, 8 in 2024-03-30T13:52:37.738 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:37 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2265415222' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:52:37.738 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:37 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:52:37.739 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:37 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 56/594 objects degraded (9.428%), 12 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:37.739 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:37 smithi043 ceph-mon[42099]: Health check failed: Reduced data availability: 8 pgs peering (PG_AVAILABILITY) 2024-03-30T13:52:37.739 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:37 smithi043 ceph-mon[42099]: osdmap e208: 8 total, 7 up, 8 in 2024-03-30T13:52:37.739 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:37 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2265415222' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:52:37.739 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:37 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:52:37.739 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:37 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 56/594 objects degraded (9.428%), 12 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:37.765 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:37.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:37.789 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:37.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:37.890 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:37.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:37.914 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:37.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:37.948 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:38.016 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:38.042 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:38.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:38.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:38.144 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:38.168 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:38.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:38.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:38.270 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:38.294 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:38.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:38.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:38.396 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:38.422 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:38.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:38.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:38.524 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:38.550 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:38.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:38.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:38.652 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:38.678 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:38.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:38.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:38.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:38 smithi043 ceph-mon[29963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:52:38.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:38 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:52:38.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:38 smithi043 ceph-mon[29963]: pgmap v406: 142 pgs: 10 remapped+peering, 12 active+undersized+degraded, 26 active+undersized, 94 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 56/594 objects degraded (9.428%) 2024-03-30T13:52:38.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:38 smithi043 ceph-mon[29963]: osdmap e209: 8 total, 7 up, 8 in 2024-03-30T13:52:38.709 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:38 smithi043 ceph-mon[42099]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:52:38.709 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:38 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:52:38.709 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:38 smithi043 ceph-mon[42099]: pgmap v406: 142 pgs: 10 remapped+peering, 12 active+undersized+degraded, 26 active+undersized, 94 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 56/594 objects degraded (9.428%) 2024-03-30T13:52:38.709 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:38 smithi043 ceph-mon[42099]: osdmap e209: 8 total, 7 up, 8 in 2024-03-30T13:52:38.780 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:38.804 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:38.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:38.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:38.834 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:38 smithi113 ceph-mon[34341]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:52:38.834 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:38 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:52:38.834 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:38 smithi113 ceph-mon[34341]: pgmap v406: 142 pgs: 10 remapped+peering, 12 active+undersized+degraded, 26 active+undersized, 94 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 56/594 objects degraded (9.428%) 2024-03-30T13:52:38.834 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:38 smithi113 ceph-mon[34341]: osdmap e209: 8 total, 7 up, 8 in 2024-03-30T13:52:38.905 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:38.933 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:38.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:38.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:39.035 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:39.058 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:39.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:39.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:39.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:52:38 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:52:39.160 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:39.184 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:39.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:39.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:39.286 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:39.312 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:39.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:39.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:39.414 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:39.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:39.439 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:39.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:39.503 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:52:39.503 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:52:39.539 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:39.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:39.564 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:39.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:39.665 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:39.688 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:39.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:39.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:39.719 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:39 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:52:39.743 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:39 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:52:39.744 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:39 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:52:39.790 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:39.814 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:39.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:39.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:39.916 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:39.940 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:39.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:39.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:40.042 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:40.066 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:40.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:40.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:40.096 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-03-30T13:52:40.167 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:40.190 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:40.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:40.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:40.275 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:52:40.292 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:40.295 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:52:40.314 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:52:40.316 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:40.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:40.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:40.335 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:52:40.355 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:52:40.376 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:52:40.377 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:52:40.398 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:52:40.398 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:52:40.398 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:52:40.399 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:52:40.399 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:52:40.399 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:52:40.399 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:52:40.399 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:52:40.399 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:52:40.417 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:40.442 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:40.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:40.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:40.452 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T13:52:40.544 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:40.567 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:40.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:40.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:40.669 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:40.692 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:40.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:40.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:40.780 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:40 smithi113 ceph-mon[34341]: pgmap v407: 142 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 26 active+undersized, 99 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 611 B/s rd, 0 op/s; 56/594 objects degraded (9.428%); 1/594 objects misplaced (0.168%); 102 KiB/s, 0 objects/s recovering 2024-03-30T13:52:40.780 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:40 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3324017258' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:52:40.795 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:40.816 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:40 smithi043 ceph-mon[29963]: pgmap v407: 142 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 26 active+undersized, 99 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 611 B/s rd, 0 op/s; 56/594 objects degraded (9.428%); 1/594 objects misplaced (0.168%); 102 KiB/s, 0 objects/s recovering 2024-03-30T13:52:40.816 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:40 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3324017258' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:52:40.816 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:40 smithi043 ceph-mon[42099]: pgmap v407: 142 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 26 active+undersized, 99 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 611 B/s rd, 0 op/s; 56/594 objects degraded (9.428%); 1/594 objects misplaced (0.168%); 102 KiB/s, 0 objects/s recovering 2024-03-30T13:52:40.816 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:40 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3324017258' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:52:40.819 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:40.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:40.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:40.921 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:40.946 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:40.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:40.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:41.048 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:41.071 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:41.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:41.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:41.173 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:41.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:41.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:41.299 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:41.322 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:41.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:41.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:41.424 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:41.452 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:41.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:41.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:41.473 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:52:41.474 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:52:41.553 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:41.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:41.580 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:41.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:41.681 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:41.698 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:41 smithi043 ceph-mon[29963]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs peering) 2024-03-30T13:52:41.699 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:41 smithi043 ceph-mon[42099]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs peering) 2024-03-30T13:52:41.705 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:41.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:41.806 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:41.834 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:41.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:41.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:41.936 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:41.959 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:41.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:41.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:41.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:41 smithi113 ceph-mon[34341]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs peering) 2024-03-30T13:52:42.061 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:42.087 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:42.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:42.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:42.188 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:42.211 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:42.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:42.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:42.313 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:42.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:42.341 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:42.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:42.442 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:42.467 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:42.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:42.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:42.513 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'unique_pool_2' created 2024-03-30T13:52:42.569 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:42.572 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:42 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:52:42.572 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:42 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:52:42.572 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:42 smithi113 ceph-mon[34341]: pgmap v408: 142 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 26 active+undersized, 99 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 680 B/s rd, 0 op/s; 56/594 objects degraded (9.428%); 1/594 objects misplaced (0.168%); 96 KiB/s, 0 objects/s recovering 2024-03-30T13:52:42.572 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:42 smithi113 ceph-mon[34341]: osdmap e210: 8 total, 7 up, 8 in 2024-03-30T13:52:42.572 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:42 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2025786194' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-30T13:52:42.573 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:52:42.573 INFO:teuthology.orchestra.run.smithi113.stdout: "size": 20, 2024-03-30T13:52:42.573 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 600, 2024-03-30T13:52:42.573 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [ 2024-03-30T13:52:42.573 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.573 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:22329 7.0 7:05055661:::benchmark_data_smithi043_86989_object22328:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-30T13:52:42.573 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:45:09.136749+0000", 2024-03-30T13:52:42.573 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 453.43394777100002, 2024-03-30T13:52:42.573 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.079349797999999999, 2024-03-30T13:52:42.573 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:42.573 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:42.573 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:42.574 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:42.574 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:52:42.574 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:52:42.574 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 22329 2024-03-30T13:52:42.574 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.574 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:42.574 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.574 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:42.574 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.136749+0000", 2024-03-30T13:52:42.574 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.574 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.574 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.574 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:42.575 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.136749+0000", 2024-03-30T13:52:42.575 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.575 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.575 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.575 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:42.575 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.136750+0000", 2024-03-30T13:52:42.575 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0640000000000001e-06 2024-03-30T13:52:42.575 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.575 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.575 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:42.575 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.136810+0000", 2024-03-30T13:52:42.575 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9842000000000001e-05 2024-03-30T13:52:42.575 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.136820+0000", 2024-03-30T13:52:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0505e-05 2024-03-30T13:52:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.136826+0000", 2024-03-30T13:52:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0750000000000003e-06 2024-03-30T13:52:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.136864+0000", 2024-03-30T13:52:42.577 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.7468000000000001e-05 2024-03-30T13:52:42.577 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.577 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.577 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:42.577 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.136943+0000", 2024-03-30T13:52:42.577 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.9567999999999994e-05 2024-03-30T13:52:42.577 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.577 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.577 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:52:42.577 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.137061+0000", 2024-03-30T13:52:42.577 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000117378 2024-03-30T13:52:42.577 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.578 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.578 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:52:42.578 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.137761+0000", 2024-03-30T13:52:42.578 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00070024400000000004 2024-03-30T13:52:42.578 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.578 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.578 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.578 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.138365+0000", 2024-03-30T13:52:42.578 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00060360399999999999 2024-03-30T13:52:42.578 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.578 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.578 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.579 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.215982+0000", 2024-03-30T13:52:42.579 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.077617647999999997 2024-03-30T13:52:42.579 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.579 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.579 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:52:42.579 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.216053+0000", 2024-03-30T13:52:42.579 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.0603000000000001e-05 2024-03-30T13:52:42.579 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.579 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.579 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:52:42.579 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:09.216099+0000", 2024-03-30T13:52:42.579 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.5799e-05 2024-03-30T13:52:42.580 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.580 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.580 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.580 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.580 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.580 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:61995 7.0 7:0437627d:::benchmark_data_smithi043_86989_object61994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-30T13:52:42.580 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:45:33.954329+0000", 2024-03-30T13:52:42.580 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 428.61636794200001, 2024-03-30T13:52:42.580 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.090862954999999995, 2024-03-30T13:52:42.580 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:42.580 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:42.580 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:42.581 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:42.581 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:52:42.581 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:52:42.581 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 61995 2024-03-30T13:52:42.581 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.581 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:42.581 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.581 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:42.581 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.954329+0000", 2024-03-30T13:52:42.581 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.581 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.581 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.581 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:42.582 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.954329+0000", 2024-03-30T13:52:42.582 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.582 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.582 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.582 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:42.582 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.954330+0000", 2024-03-30T13:52:42.582 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.1200000000000001e-07 2024-03-30T13:52:42.582 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.582 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.582 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:42.582 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.954380+0000", 2024-03-30T13:52:42.582 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0637999999999999e-05 2024-03-30T13:52:42.582 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.582 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.583 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:42.583 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.954390+0000", 2024-03-30T13:52:42.583 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.645e-06 2024-03-30T13:52:42.583 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.583 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.583 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:42.583 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.954395+0000", 2024-03-30T13:52:42.583 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3419999999999998e-06 2024-03-30T13:52:42.583 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.583 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.583 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:42.583 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.954442+0000", 2024-03-30T13:52:42.583 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7296999999999998e-05 2024-03-30T13:52:42.583 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.584 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.584 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:42.584 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.954618+0000", 2024-03-30T13:52:42.584 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00017529199999999999 2024-03-30T13:52:42.584 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.584 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.584 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:52:42.584 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.954763+0000", 2024-03-30T13:52:42.584 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000145196 2024-03-30T13:52:42.584 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.584 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.584 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:52:42.584 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.957069+0000", 2024-03-30T13:52:42.584 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00230583 2024-03-30T13:52:42.584 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.585 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.585 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.585 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.959940+0000", 2024-03-30T13:52:42.585 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0028708969999999999 2024-03-30T13:52:42.585 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.585 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.585 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.585 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:34.045064+0000", 2024-03-30T13:52:42.585 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.085124209000000006 2024-03-30T13:52:42.585 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.585 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.585 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:52:42.585 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:34.045153+0000", 2024-03-30T13:52:42.585 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.8701000000000003e-05 2024-03-30T13:52:42.586 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.586 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.586 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:52:42.586 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:34.045192+0000", 2024-03-30T13:52:42.586 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.8995999999999997e-05 2024-03-30T13:52:42.586 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.586 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.586 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.586 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.586 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.586 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:62003 7.0 7:0229d695:::benchmark_data_smithi043_86989_object62002:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-30T13:52:42.586 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:45:33.965096+0000", 2024-03-30T13:52:42.586 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 428.60560068299998, 2024-03-30T13:52:42.587 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.086717765000000002, 2024-03-30T13:52:42.587 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:42.587 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:42.587 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:42.587 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:42.587 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:52:42.587 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:52:42.587 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 62003 2024-03-30T13:52:42.587 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.587 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:42.587 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.587 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:42.587 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.965096+0000", 2024-03-30T13:52:42.588 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.588 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.588 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.588 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:42.588 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.965096+0000", 2024-03-30T13:52:42.588 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.588 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.588 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.588 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:42.588 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.965097+0000", 2024-03-30T13:52:42.588 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1820000000000001e-06 2024-03-30T13:52:42.588 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.588 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.588 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:42.589 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.965152+0000", 2024-03-30T13:52:42.589 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4993e-05 2024-03-30T13:52:42.589 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.589 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.589 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:42.589 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.965164+0000", 2024-03-30T13:52:42.589 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1474999999999999e-05 2024-03-30T13:52:42.589 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.589 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.589 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:42.589 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.965169+0000", 2024-03-30T13:52:42.589 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8690000000000001e-06 2024-03-30T13:52:42.589 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.590 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.590 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:42.590 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.965211+0000", 2024-03-30T13:52:42.590 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1322000000000002e-05 2024-03-30T13:52:42.590 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.590 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.590 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:42.590 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.965305+0000", 2024-03-30T13:52:42.590 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4408999999999997e-05 2024-03-30T13:52:42.590 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.590 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.590 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:52:42.590 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.965394+0000", 2024-03-30T13:52:42.591 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.8945999999999998e-05 2024-03-30T13:52:42.591 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.591 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.591 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:52:42.591 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.967037+0000", 2024-03-30T13:52:42.591 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001642527 2024-03-30T13:52:42.591 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.591 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.591 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.591 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:33.967294+0000", 2024-03-30T13:52:42.591 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000257457 2024-03-30T13:52:42.591 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.591 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.591 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.592 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:34.051715+0000", 2024-03-30T13:52:42.592 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.084420766999999994 2024-03-30T13:52:42.592 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.592 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.592 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:52:42.592 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:34.051766+0000", 2024-03-30T13:52:42.592 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1371e-05 2024-03-30T13:52:42.592 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.592 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.592 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:52:42.592 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:45:34.051814+0000", 2024-03-30T13:52:42.592 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7447000000000002e-05 2024-03-30T13:52:42.592 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.593 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.593 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.593 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.593 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.593 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:146027 7.0 7:0624f1a7:::benchmark_data_smithi043_86989_object23689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-30T13:52:42.593 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:55.859056+0000", 2024-03-30T13:52:42.593 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 346.71164043800002, 2024-03-30T13:52:42.593 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.079134809, 2024-03-30T13:52:42.593 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:42.593 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:42.593 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:42.593 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:42.593 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:52:42.594 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:52:42.594 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 146027 2024-03-30T13:52:42.594 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.594 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:42.594 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.594 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:42.594 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.859056+0000", 2024-03-30T13:52:42.594 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.594 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.594 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.594 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:42.595 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.859056+0000", 2024-03-30T13:52:42.595 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.595 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.595 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.595 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:42.595 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.859058+0000", 2024-03-30T13:52:42.595 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.57e-06 2024-03-30T13:52:42.595 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.595 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.595 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:42.595 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.859065+0000", 2024-03-30T13:52:42.596 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.6210000000000004e-06 2024-03-30T13:52:42.596 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.596 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.596 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:42.596 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.859079+0000", 2024-03-30T13:52:42.596 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3961000000000001e-05 2024-03-30T13:52:42.596 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.596 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.596 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:42.596 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.859088+0000", 2024-03-30T13:52:42.596 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.9709999999999993e-06 2024-03-30T13:52:42.596 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.596 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.596 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:42.596 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.859158+0000", 2024-03-30T13:52:42.597 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.0240999999999994e-05 2024-03-30T13:52:42.597 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.597 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.597 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:42.597 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.859302+0000", 2024-03-30T13:52:42.597 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00014327200000000001 2024-03-30T13:52:42.597 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.597 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.597 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:52:42.597 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.859432+0000", 2024-03-30T13:52:42.597 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000129932 2024-03-30T13:52:42.597 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.597 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.597 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.598 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.862282+0000", 2024-03-30T13:52:42.598 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0028500969999999998 2024-03-30T13:52:42.598 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.598 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.598 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:52:42.598 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.862313+0000", 2024-03-30T13:52:42.598 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.1152000000000003e-05 2024-03-30T13:52:42.598 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.598 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.598 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.598 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.938112+0000", 2024-03-30T13:52:42.598 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.075798692000000001 2024-03-30T13:52:42.598 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.599 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.599 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:52:42.599 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.938149+0000", 2024-03-30T13:52:42.599 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.7839000000000001e-05 2024-03-30T13:52:42.599 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.599 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.599 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:52:42.599 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.938191+0000", 2024-03-30T13:52:42.599 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1461000000000002e-05 2024-03-30T13:52:42.599 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.599 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.599 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.599 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.599 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.599 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:237116 7.0 7:058289d1:::benchmark_data_smithi043_86989_object114778:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-30T13:52:42.600 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:48:30.937033+0000", 2024-03-30T13:52:42.600 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 251.63366334299999, 2024-03-30T13:52:42.600 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.084083505000000003, 2024-03-30T13:52:42.600 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:42.600 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:42.600 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:42.600 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:42.600 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:52:42.600 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:52:42.600 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 237116 2024-03-30T13:52:42.600 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.600 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:42.600 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.600 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:42.601 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:48:30.937033+0000", 2024-03-30T13:52:42.601 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.601 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.601 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.601 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:42.601 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:48:30.937033+0000", 2024-03-30T13:52:42.601 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.601 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.601 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.601 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:42.602 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:48:30.937035+0000", 2024-03-30T13:52:42.602 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4980000000000001e-06 2024-03-30T13:52:42.602 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.602 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.602 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:42.602 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:48:30.937041+0000", 2024-03-30T13:52:42.603 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.336e-06 2024-03-30T13:52:42.603 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.603 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.603 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:42.603 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:48:30.937056+0000", 2024-03-30T13:52:42.603 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5302999999999999e-05 2024-03-30T13:52:42.603 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.603 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.603 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:42.603 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:48:30.937065+0000", 2024-03-30T13:52:42.603 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.4079999999999998e-06 2024-03-30T13:52:42.603 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.603 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.604 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:42.604 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:48:30.937775+0000", 2024-03-30T13:52:42.604 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00071025099999999996 2024-03-30T13:52:42.604 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.604 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.604 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:42.604 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:48:30.937889+0000", 2024-03-30T13:52:42.604 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000113677 2024-03-30T13:52:42.604 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.604 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.604 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:52:42.604 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:48:30.938085+0000", 2024-03-30T13:52:42.604 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00019643199999999999 2024-03-30T13:52:42.604 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.604 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.605 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.605 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:48:30.946298+0000", 2024-03-30T13:52:42.605 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0082127439999999993 2024-03-30T13:52:42.605 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.605 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.605 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:52:42.605 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:48:30.946558+0000", 2024-03-30T13:52:42.605 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00025976500000000002 2024-03-30T13:52:42.605 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.605 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.605 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.605 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:48:31.020848+0000", 2024-03-30T13:52:42.605 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.074290371999999993 2024-03-30T13:52:42.605 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.605 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.606 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:52:42.606 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:48:31.020913+0000", 2024-03-30T13:52:42.606 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.4615000000000004e-05 2024-03-30T13:52:42.606 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.606 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.606 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:52:42.606 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:48:31.021117+0000", 2024-03-30T13:52:42.606 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00020410400000000001 2024-03-30T13:52:42.606 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.606 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.606 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.606 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.606 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.606 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:28080 8.8 8:113ed673:::benchmark_data_smithi043_119832_object28079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-30T13:52:42.607 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:49:25.863547+0000", 2024-03-30T13:52:42.607 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 196.70714926299999, 2024-03-30T13:52:42.607 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.10021473, 2024-03-30T13:52:42.607 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:42.607 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:42.607 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:42.607 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:42.607 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:52:42.607 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:42.607 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 28080 2024-03-30T13:52:42.607 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.607 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:42.607 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.608 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:42.608 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863547+0000", 2024-03-30T13:52:42.608 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.608 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.608 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.608 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:42.608 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863547+0000", 2024-03-30T13:52:42.608 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.608 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.608 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.608 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:42.608 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863548+0000", 2024-03-30T13:52:42.608 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.8800000000000001e-07 2024-03-30T13:52:42.608 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.609 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.609 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:42.609 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863574+0000", 2024-03-30T13:52:42.609 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.5948999999999998e-05 2024-03-30T13:52:42.609 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.609 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.609 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:42.609 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863585+0000", 2024-03-30T13:52:42.609 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0411000000000001e-05 2024-03-30T13:52:42.609 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.609 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.609 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:42.609 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863594+0000", 2024-03-30T13:52:42.609 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.0720000000000006e-06 2024-03-30T13:52:42.609 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.610 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.610 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:42.610 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863644+0000", 2024-03-30T13:52:42.610 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0078999999999999e-05 2024-03-30T13:52:42.610 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.610 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.610 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:42.610 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863905+0000", 2024-03-30T13:52:42.610 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00026154999999999999 2024-03-30T13:52:42.610 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.610 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.610 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:52:42.611 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.864048+0000", 2024-03-30T13:52:42.611 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000142731 2024-03-30T13:52:42.611 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.611 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.611 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:52:42.611 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.866137+0000", 2024-03-30T13:52:42.611 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0020888 2024-03-30T13:52:42.611 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.611 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.611 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.611 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.866834+0000", 2024-03-30T13:52:42.611 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00069740999999999998 2024-03-30T13:52:42.611 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.611 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.612 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.612 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.963689+0000", 2024-03-30T13:52:42.612 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.096855105999999996 2024-03-30T13:52:42.612 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.612 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.612 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:52:42.612 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.963729+0000", 2024-03-30T13:52:42.612 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.9792000000000001e-05 2024-03-30T13:52:42.612 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.612 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.612 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:52:42.612 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.963762+0000", 2024-03-30T13:52:42.612 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2941999999999997e-05 2024-03-30T13:52:42.612 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.612 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.613 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.613 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.613 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.613 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:28082 8.8 8:1b8ed0ff:::benchmark_data_smithi043_119832_object28081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-30T13:52:42.613 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:49:25.863684+0000", 2024-03-30T13:52:42.613 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 196.70701233299999, 2024-03-30T13:52:42.613 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.10475061099999999, 2024-03-30T13:52:42.613 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:42.613 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:42.613 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:42.613 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:42.613 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:52:42.613 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:42.613 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 28082 2024-03-30T13:52:42.614 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.614 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:42.614 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.614 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:42.614 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863684+0000", 2024-03-30T13:52:42.614 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.614 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.614 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.614 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:42.614 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863684+0000", 2024-03-30T13:52:42.614 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.614 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.614 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.614 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:42.615 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863685+0000", 2024-03-30T13:52:42.615 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1500000000000004e-07 2024-03-30T13:52:42.615 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.615 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.615 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:42.615 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863720+0000", 2024-03-30T13:52:42.615 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.5271999999999997e-05 2024-03-30T13:52:42.615 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.615 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.615 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:42.615 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863728+0000", 2024-03-30T13:52:42.615 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.2269999999999993e-06 2024-03-30T13:52:42.615 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.615 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.616 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:42.616 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863735+0000", 2024-03-30T13:52:42.616 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6749999999999996e-06 2024-03-30T13:52:42.616 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.616 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.616 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:42.616 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.865132+0000", 2024-03-30T13:52:42.616 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0013967999999999999 2024-03-30T13:52:42.616 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.616 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.616 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:42.616 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.865264+0000", 2024-03-30T13:52:42.616 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013255799999999999 2024-03-30T13:52:42.616 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.616 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.617 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:52:42.617 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.865398+0000", 2024-03-30T13:52:42.617 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000133449 2024-03-30T13:52:42.617 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.617 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.617 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:52:42.617 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.866258+0000", 2024-03-30T13:52:42.617 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00086043299999999997 2024-03-30T13:52:42.617 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.617 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.617 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.617 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.866918+0000", 2024-03-30T13:52:42.617 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000660055 2024-03-30T13:52:42.617 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.618 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.618 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.618 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.968343+0000", 2024-03-30T13:52:42.618 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.101424903 2024-03-30T13:52:42.618 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.618 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.618 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:52:42.618 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.968402+0000", 2024-03-30T13:52:42.618 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8968000000000002e-05 2024-03-30T13:52:42.618 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.618 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.618 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:52:42.618 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.968435+0000", 2024-03-30T13:52:42.618 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2656e-05 2024-03-30T13:52:42.619 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.619 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.619 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.619 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.619 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.619 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:31312 8.4 8:22ccd403:::benchmark_data_smithi043_119832_object31311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-30T13:52:42.619 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:49:27.619219+0000", 2024-03-30T13:52:42.619 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 194.95147791400001, 2024-03-30T13:52:42.619 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.16068914500000001, 2024-03-30T13:52:42.619 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:42.619 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:42.619 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:42.619 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:42.619 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:52:42.619 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:42.620 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 31312 2024-03-30T13:52:42.620 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.620 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:42.620 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.620 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:42.620 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.619219+0000", 2024-03-30T13:52:42.620 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.620 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.620 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.620 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:42.620 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.619219+0000", 2024-03-30T13:52:42.620 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.620 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.620 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.621 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:42.621 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.619219+0000", 2024-03-30T13:52:42.621 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8500000000000001e-07 2024-03-30T13:52:42.621 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.621 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.621 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:42.621 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.619288+0000", 2024-03-30T13:52:42.621 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8913999999999998e-05 2024-03-30T13:52:42.621 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.621 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.621 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:42.621 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.619298+0000", 2024-03-30T13:52:42.621 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.3409999999999992e-06 2024-03-30T13:52:42.621 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.621 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.622 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:42.622 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.619303+0000", 2024-03-30T13:52:42.622 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3990000000000003e-06 2024-03-30T13:52:42.622 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.622 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.622 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:42.622 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.619344+0000", 2024-03-30T13:52:42.622 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.0772000000000002e-05 2024-03-30T13:52:42.622 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.622 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.622 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:42.622 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.619423+0000", 2024-03-30T13:52:42.622 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.9049999999999997e-05 2024-03-30T13:52:42.622 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.623 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.623 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 2,4", 2024-03-30T13:52:42.623 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.619545+0000", 2024-03-30T13:52:42.623 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012253500000000001 2024-03-30T13:52:42.623 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.623 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.623 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:52:42.623 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.621662+0000", 2024-03-30T13:52:42.623 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0021171610000000002 2024-03-30T13:52:42.623 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.623 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.623 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.623 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.622183+0000", 2024-03-30T13:52:42.623 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00052070800000000004 2024-03-30T13:52:42.624 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.624 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.624 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.624 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.779799+0000", 2024-03-30T13:52:42.624 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15761560099999999 2024-03-30T13:52:42.624 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.624 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.624 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:52:42.624 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.779861+0000", 2024-03-30T13:52:42.624 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.2067000000000001e-05 2024-03-30T13:52:42.624 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.624 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.624 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:52:42.625 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.779908+0000", 2024-03-30T13:52:42.625 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6912000000000001e-05 2024-03-30T13:52:42.625 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.625 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.625 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.625 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.625 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.625 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:31315 8.4 8:2d5d1e8e:::benchmark_data_smithi043_119832_object31314:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-30T13:52:42.625 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:49:27.620358+0000", 2024-03-30T13:52:42.625 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 194.95033810499999, 2024-03-30T13:52:42.625 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15976934100000001, 2024-03-30T13:52:42.625 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:42.625 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:42.625 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:42.626 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:42.626 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:52:42.626 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:42.626 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 31315 2024-03-30T13:52:42.626 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.626 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:42.626 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.626 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:42.626 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.620358+0000", 2024-03-30T13:52:42.626 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.626 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.626 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.626 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:42.626 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.620358+0000", 2024-03-30T13:52:42.627 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.627 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.627 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.627 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:42.627 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.620359+0000", 2024-03-30T13:52:42.627 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.7800000000000001e-07 2024-03-30T13:52:42.627 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.627 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.627 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:42.627 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.620419+0000", 2024-03-30T13:52:42.627 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9920000000000002e-05 2024-03-30T13:52:42.627 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.627 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.627 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:42.628 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.620429+0000", 2024-03-30T13:52:42.628 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.8099999999999992e-06 2024-03-30T13:52:42.628 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.628 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.628 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:42.628 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.620436+0000", 2024-03-30T13:52:42.628 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.3819999999999998e-06 2024-03-30T13:52:42.628 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.628 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.628 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:42.628 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.620510+0000", 2024-03-30T13:52:42.628 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.3495e-05 2024-03-30T13:52:42.628 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.628 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.629 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:42.629 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.620644+0000", 2024-03-30T13:52:42.629 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013414199999999999 2024-03-30T13:52:42.629 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.629 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.629 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 2,4", 2024-03-30T13:52:42.629 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.620778+0000", 2024-03-30T13:52:42.629 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013415700000000001 2024-03-30T13:52:42.629 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.629 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.629 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:52:42.629 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.621756+0000", 2024-03-30T13:52:42.629 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00097763800000000007 2024-03-30T13:52:42.630 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.630 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.630 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.630 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.622314+0000", 2024-03-30T13:52:42.630 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00055802399999999995 2024-03-30T13:52:42.630 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.630 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.630 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.630 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.779972+0000", 2024-03-30T13:52:42.630 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15765843800000001 2024-03-30T13:52:42.630 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.630 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.630 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:52:42.631 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.780086+0000", 2024-03-30T13:52:42.631 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011418599999999999 2024-03-30T13:52:42.631 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.631 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.631 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:52:42.631 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.780128+0000", 2024-03-30T13:52:42.631 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1471000000000003e-05 2024-03-30T13:52:42.631 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.631 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.631 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.631 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.631 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.631 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:31317 8.4 8:244a887a:::benchmark_data_smithi043_119832_object31316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-30T13:52:42.631 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:49:27.624141+0000", 2024-03-30T13:52:42.631 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 194.946555093, 2024-03-30T13:52:42.632 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15630417999999999, 2024-03-30T13:52:42.632 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:42.632 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:42.632 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:42.632 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:42.632 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:52:42.632 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:42.632 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 31317 2024-03-30T13:52:42.632 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.632 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:42.632 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.632 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:42.632 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.624141+0000", 2024-03-30T13:52:42.632 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.632 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.633 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.633 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:42.633 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.624141+0000", 2024-03-30T13:52:42.633 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.633 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.633 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.633 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:42.633 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.624142+0000", 2024-03-30T13:52:42.633 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.4700000000000001e-07 2024-03-30T13:52:42.633 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.633 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.633 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:42.633 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.624205+0000", 2024-03-30T13:52:42.634 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.2424999999999999e-05 2024-03-30T13:52:42.634 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.634 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.634 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:42.634 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.624215+0000", 2024-03-30T13:52:42.634 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.9629999999999993e-06 2024-03-30T13:52:42.634 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.634 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.634 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:42.634 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.624231+0000", 2024-03-30T13:52:42.634 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.6421000000000001e-05 2024-03-30T13:52:42.634 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.634 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.635 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:42.635 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.624274+0000", 2024-03-30T13:52:42.635 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.2549000000000002e-05 2024-03-30T13:52:42.635 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.635 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.635 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:42.635 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.624360+0000", 2024-03-30T13:52:42.635 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6681000000000006e-05 2024-03-30T13:52:42.635 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.635 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.635 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 2,4", 2024-03-30T13:52:42.635 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.624446+0000", 2024-03-30T13:52:42.635 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6037000000000003e-05 2024-03-30T13:52:42.635 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.635 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.636 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:52:42.636 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.625196+0000", 2024-03-30T13:52:42.636 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00074996299999999998 2024-03-30T13:52:42.636 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.636 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.636 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.636 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.625529+0000", 2024-03-30T13:52:42.636 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00033245200000000001 2024-03-30T13:52:42.636 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.636 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.636 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.636 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.780307+0000", 2024-03-30T13:52:42.636 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.154778215 2024-03-30T13:52:42.636 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.636 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.780407+0000", 2024-03-30T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000100058 2024-03-30T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.780446+0000", 2024-03-30T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.8668999999999999e-05 2024-03-30T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:72355 8.8 8:11951a93:::benchmark_data_smithi043_119832_object72354:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-30T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:49:53.882694+0000", 2024-03-30T13:52:42.638 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 168.688002778, 2024-03-30T13:52:42.638 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.082963059000000006, 2024-03-30T13:52:42.638 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:42.638 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:42.638 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:42.638 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:42.638 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:52:42.638 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:42.638 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 72355 2024-03-30T13:52:42.638 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.638 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:42.638 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.638 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:42.638 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:53.882694+0000", 2024-03-30T13:52:42.639 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.639 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.639 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.639 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:42.639 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:53.882694+0000", 2024-03-30T13:52:42.639 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.639 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.639 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.639 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:42.639 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:53.882694+0000", 2024-03-30T13:52:42.639 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1399999999999997e-07 2024-03-30T13:52:42.639 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:53.882754+0000", 2024-03-30T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9549999999999997e-05 2024-03-30T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:53.882763+0000", 2024-03-30T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.1980000000000007e-06 2024-03-30T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:53.882770+0000", 2024-03-30T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3899999999999998e-06 2024-03-30T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:53.882806+0000", 2024-03-30T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.6137999999999999e-05 2024-03-30T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:53.882888+0000", 2024-03-30T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.2188999999999995e-05 2024-03-30T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:53.882976+0000", 2024-03-30T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.7837999999999995e-05 2024-03-30T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.642 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.642 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:52:42.642 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:53.884916+0000", 2024-03-30T13:52:42.642 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001940422 2024-03-30T13:52:42.642 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.642 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.642 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:42.643 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:42.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:42.644 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:53.885050+0000", 2024-03-30T13:52:42.644 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013402800000000001 2024-03-30T13:52:42.644 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.644 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.645 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.645 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:53.965548+0000", 2024-03-30T13:52:42.645 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.080498346999999998 2024-03-30T13:52:42.645 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.645 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.645 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:52:42.645 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:53.965613+0000", 2024-03-30T13:52:42.645 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5007999999999998e-05 2024-03-30T13:52:42.645 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.645 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.645 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:52:42.645 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:53.965657+0000", 2024-03-30T13:52:42.645 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3337000000000002e-05 2024-03-30T13:52:42.645 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.646 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.646 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.646 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.646 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.646 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:101728 8.8 8:1ec65c71:::benchmark_data_smithi043_119832_object101727:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-03-30T13:52:42.646 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:19.190830+0000", 2024-03-30T13:52:42.646 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 143.37986645999999, 2024-03-30T13:52:42.646 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.079937380000000002, 2024-03-30T13:52:42.646 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:42.646 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:42.646 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:42.646 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:42.646 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:52:42.647 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:42.647 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 101728 2024-03-30T13:52:42.647 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.647 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:42.647 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.647 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:42.647 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:19.190830+0000", 2024-03-30T13:52:42.647 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.647 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.647 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.647 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:42.647 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:19.190830+0000", 2024-03-30T13:52:42.647 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.648 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.648 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.648 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:42.648 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:19.190831+0000", 2024-03-30T13:52:42.648 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.404e-06 2024-03-30T13:52:42.648 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.648 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.648 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:42.648 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:19.190916+0000", 2024-03-30T13:52:42.648 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.4016000000000003e-05 2024-03-30T13:52:42.648 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.648 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.648 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:42.648 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:19.190927+0000", 2024-03-30T13:52:42.649 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1701e-05 2024-03-30T13:52:42.649 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.649 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.649 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:42.649 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:19.190937+0000", 2024-03-30T13:52:42.649 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0210000000000001e-05 2024-03-30T13:52:42.649 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.649 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.649 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:42.649 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:19.190995+0000", 2024-03-30T13:52:42.649 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7277000000000001e-05 2024-03-30T13:52:42.649 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.649 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.649 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:42.650 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:19.191111+0000", 2024-03-30T13:52:42.650 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000116146 2024-03-30T13:52:42.650 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.650 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.650 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:52:42.650 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:19.191230+0000", 2024-03-30T13:52:42.650 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0001187 2024-03-30T13:52:42.650 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.650 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.650 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:52:42.650 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:19.194071+0000", 2024-03-30T13:52:42.650 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0028411970000000002 2024-03-30T13:52:42.650 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.650 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.651 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.651 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:19.194451+0000", 2024-03-30T13:52:42.651 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00037988300000000002 2024-03-30T13:52:42.651 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.651 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.651 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.651 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:19.270532+0000", 2024-03-30T13:52:42.651 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.076081160999999994 2024-03-30T13:52:42.651 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.651 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.651 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:52:42.652 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:19.270715+0000", 2024-03-30T13:52:42.652 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000183661 2024-03-30T13:52:42.652 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.652 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.652 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:52:42.652 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:19.270767+0000", 2024-03-30T13:52:42.652 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2024000000000002e-05 2024-03-30T13:52:42.652 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.652 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.652 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.652 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.652 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.653 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:246734 8.8 8:1d180adf:::benchmark_data_smithi043_119832_object116380:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-03-30T13:52:42.653 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:16.536012+0000", 2024-03-30T13:52:42.653 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 26.034685011000001, 2024-03-30T13:52:42.653 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.078730381000000002, 2024-03-30T13:52:42.653 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:42.653 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:42.653 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:42.653 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:42.653 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:52:42.653 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:42.653 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 246734 2024-03-30T13:52:42.653 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.653 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:42.654 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.654 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:42.654 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:16.536012+0000", 2024-03-30T13:52:42.654 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.654 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.654 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.654 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:42.654 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:16.536012+0000", 2024-03-30T13:52:42.654 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.654 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.654 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.655 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:42.655 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:16.536013+0000", 2024-03-30T13:52:42.655 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.573e-06 2024-03-30T13:52:42.655 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.655 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.655 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:42.655 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:16.536019+0000", 2024-03-30T13:52:42.655 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6080000000000003e-06 2024-03-30T13:52:42.655 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.655 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.655 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:42.655 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:16.536033+0000", 2024-03-30T13:52:42.656 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4401e-05 2024-03-30T13:52:42.656 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.656 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.656 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:42.656 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:16.536041+0000", 2024-03-30T13:52:42.656 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.9640000000000003e-06 2024-03-30T13:52:42.656 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.656 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.656 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:42.656 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:16.536095+0000", 2024-03-30T13:52:42.656 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4199000000000001e-05 2024-03-30T13:52:42.656 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.657 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.657 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:42.657 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:16.536246+0000", 2024-03-30T13:52:42.657 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00015091200000000001 2024-03-30T13:52:42.657 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.657 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.657 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:52:42.657 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:16.536359+0000", 2024-03-30T13:52:42.657 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000112633 2024-03-30T13:52:42.657 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.657 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.657 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.657 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:16.539296+0000", 2024-03-30T13:52:42.658 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0029375909999999998 2024-03-30T13:52:42.658 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.658 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.658 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:52:42.658 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:16.539317+0000", 2024-03-30T13:52:42.658 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.1050999999999999e-05 2024-03-30T13:52:42.658 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.658 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.658 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.658 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:16.614641+0000", 2024-03-30T13:52:42.658 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.075323940000000006 2024-03-30T13:52:42.659 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.659 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.659 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:52:42.659 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:16.614695+0000", 2024-03-30T13:52:42.659 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3990999999999999e-05 2024-03-30T13:52:42.659 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.659 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.659 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:52:42.659 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:16.614742+0000", 2024-03-30T13:52:42.659 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6517999999999997e-05 2024-03-30T13:52:42.660 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.660 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.660 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.660 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.660 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.660 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:249827 8.8 8:1adcec11:::benchmark_data_smithi043_119832_object119473:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:52:42.660 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:19.321904+0000", 2024-03-30T13:52:42.660 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 23.248792073000001, 2024-03-30T13:52:42.660 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.127091228, 2024-03-30T13:52:42.660 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:42.660 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:42.661 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:42.661 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:42.661 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:52:42.661 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:42.661 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 249827 2024-03-30T13:52:42.661 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.661 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:42.661 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.661 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:42.661 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.321904+0000", 2024-03-30T13:52:42.661 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.662 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.662 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.662 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:42.662 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.321904+0000", 2024-03-30T13:52:42.662 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.662 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.662 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.662 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:42.662 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.321905+0000", 2024-03-30T13:52:42.662 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0109999999999999e-06 2024-03-30T13:52:42.662 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.663 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.663 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:42.663 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.321911+0000", 2024-03-30T13:52:42.663 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8810000000000001e-06 2024-03-30T13:52:42.663 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.663 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.663 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:42.663 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.321922+0000", 2024-03-30T13:52:42.663 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0659000000000001e-05 2024-03-30T13:52:42.663 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.663 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.664 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:42.664 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.321929+0000", 2024-03-30T13:52:42.664 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.08e-06 2024-03-30T13:52:42.664 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.664 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.664 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:42.664 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.322098+0000", 2024-03-30T13:52:42.664 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00016901000000000001 2024-03-30T13:52:42.664 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.664 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.665 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:42.665 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.322244+0000", 2024-03-30T13:52:42.665 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00014547799999999999 2024-03-30T13:52:42.665 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.665 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.665 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:52:42.665 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.322367+0000", 2024-03-30T13:52:42.665 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012317299999999999 2024-03-30T13:52:42.665 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.665 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.665 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.665 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.328056+0000", 2024-03-30T13:52:42.666 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0056895740000000002 2024-03-30T13:52:42.666 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.666 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.666 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:52:42.666 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.328217+0000", 2024-03-30T13:52:42.666 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000161101 2024-03-30T13:52:42.666 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.666 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.666 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.666 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.448923+0000", 2024-03-30T13:52:42.666 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.12070563199999999 2024-03-30T13:52:42.666 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.667 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.667 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:52:42.667 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.448958+0000", 2024-03-30T13:52:42.667 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.5289e-05 2024-03-30T13:52:42.667 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.667 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.667 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:52:42.667 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.448996+0000", 2024-03-30T13:52:42.667 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.7339999999999998e-05 2024-03-30T13:52:42.667 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.667 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.668 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.668 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.668 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.668 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:249832 8.8 8:176df1f4:::benchmark_data_smithi043_119832_object119478:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:52:42.668 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:19.322778+0000", 2024-03-30T13:52:42.668 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 23.247918748, 2024-03-30T13:52:42.668 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.12633915500000001, 2024-03-30T13:52:42.668 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:42.668 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:42.668 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:42.668 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:42.668 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:52:42.668 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:42.669 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 249832 2024-03-30T13:52:42.669 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.669 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:42.669 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.669 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:42.669 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.322778+0000", 2024-03-30T13:52:42.669 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.669 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.669 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.669 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:42.670 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.322778+0000", 2024-03-30T13:52:42.670 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.670 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.670 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.670 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:42.670 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.322779+0000", 2024-03-30T13:52:42.670 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.1200000000000002e-07 2024-03-30T13:52:42.670 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.670 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.670 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:42.670 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.322783+0000", 2024-03-30T13:52:42.670 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6020000000000002e-06 2024-03-30T13:52:42.671 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.671 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.671 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:42.671 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.322795+0000", 2024-03-30T13:52:42.671 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1806e-05 2024-03-30T13:52:42.671 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.671 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.671 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:42.671 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.322802+0000", 2024-03-30T13:52:42.671 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.4240000000000004e-06 2024-03-30T13:52:42.671 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.672 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.672 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:42.672 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.325035+0000", 2024-03-30T13:52:42.672 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0022325980000000001 2024-03-30T13:52:42.672 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.672 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.672 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:42.672 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.325133+0000", 2024-03-30T13:52:42.672 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.8256000000000003e-05 2024-03-30T13:52:42.672 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.672 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.672 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:52:42.673 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.325242+0000", 2024-03-30T13:52:42.673 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00010881 2024-03-30T13:52:42.673 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.673 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.673 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:52:42.673 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.328269+0000", 2024-03-30T13:52:42.673 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.003026782 2024-03-30T13:52:42.673 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.673 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.673 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.674 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.328961+0000", 2024-03-30T13:52:42.674 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00069232300000000001 2024-03-30T13:52:42.674 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.674 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.674 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.674 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.449044+0000", 2024-03-30T13:52:42.674 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.12008284 2024-03-30T13:52:42.674 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.674 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.674 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:52:42.674 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.449090+0000", 2024-03-30T13:52:42.674 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6076999999999999e-05 2024-03-30T13:52:42.675 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.675 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.675 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:52:42.675 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.449117+0000", 2024-03-30T13:52:42.675 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.6825000000000002e-05 2024-03-30T13:52:42.675 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.675 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.675 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.675 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.675 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.675 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:250368 8.8 8:1d8b7580:::benchmark_data_smithi043_119832_object120014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:52:42.675 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:19.958219+0000", 2024-03-30T13:52:42.676 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 22.612477943000002, 2024-03-30T13:52:42.676 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.108421089, 2024-03-30T13:52:42.676 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:42.676 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:42.676 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:42.676 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:42.676 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:52:42.676 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:42.676 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 250368 2024-03-30T13:52:42.676 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.676 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:42.676 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.677 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:42.677 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.958219+0000", 2024-03-30T13:52:42.677 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.677 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.677 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.677 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:42.677 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.958219+0000", 2024-03-30T13:52:42.677 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.677 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.677 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.677 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:42.677 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.958220+0000", 2024-03-30T13:52:42.678 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2950000000000001e-06 2024-03-30T13:52:42.678 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.678 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.678 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:42.678 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.958232+0000", 2024-03-30T13:52:42.678 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1756000000000001e-05 2024-03-30T13:52:42.678 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.678 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.678 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:42.678 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.958245+0000", 2024-03-30T13:52:42.678 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3329000000000001e-05 2024-03-30T13:52:42.678 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.679 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.679 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:42.679 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.958254+0000", 2024-03-30T13:52:42.679 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.9290000000000004e-06 2024-03-30T13:52:42.679 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.679 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.679 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:42.679 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.958313+0000", 2024-03-30T13:52:42.679 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8714000000000001e-05 2024-03-30T13:52:42.679 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.679 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.679 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:42.680 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.958460+0000", 2024-03-30T13:52:42.680 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00014767299999999999 2024-03-30T13:52:42.680 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.680 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.680 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:52:42.680 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.958580+0000", 2024-03-30T13:52:42.680 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000119349 2024-03-30T13:52:42.680 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.680 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.680 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:52:42.680 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.964242+0000", 2024-03-30T13:52:42.680 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0056619929999999997 2024-03-30T13:52:42.681 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.681 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.681 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.681 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.967163+0000", 2024-03-30T13:52:42.681 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0029217069999999999 2024-03-30T13:52:42.681 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.681 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.681 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.681 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:20.066562+0000", 2024-03-30T13:52:42.681 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.099398891000000003 2024-03-30T13:52:42.681 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.681 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.682 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:52:42.682 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:20.066603+0000", 2024-03-30T13:52:42.682 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.0794000000000003e-05 2024-03-30T13:52:42.682 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.682 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.682 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:52:42.682 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:20.066640+0000", 2024-03-30T13:52:42.682 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.6659000000000003e-05 2024-03-30T13:52:42.682 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.682 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.682 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.682 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.683 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.683 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:253241 8.8 8:11f260d4:::benchmark_data_smithi043_119832_object122887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:52:42.683 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:22.334195+0000", 2024-03-30T13:52:42.683 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 20.236501863000001, 2024-03-30T13:52:42.683 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.081960973000000006, 2024-03-30T13:52:42.683 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:42.683 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:42.683 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:42.683 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:42.683 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:52:42.683 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:42.683 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 253241 2024-03-30T13:52:42.683 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.684 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:42.684 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.684 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:42.684 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:22.334195+0000", 2024-03-30T13:52:42.684 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.684 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.684 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.684 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:42.684 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:22.334195+0000", 2024-03-30T13:52:42.684 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.684 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.684 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.685 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:42.685 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:22.334196+0000", 2024-03-30T13:52:42.685 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.358e-06 2024-03-30T13:52:42.685 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.685 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.685 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:42.685 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:22.334202+0000", 2024-03-30T13:52:42.685 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5860000000000004e-06 2024-03-30T13:52:42.685 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.685 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.685 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:42.685 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:22.334213+0000", 2024-03-30T13:52:42.685 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1576000000000001e-05 2024-03-30T13:52:42.686 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.686 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.686 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:42.686 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:22.334229+0000", 2024-03-30T13:52:42.686 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5644000000000001e-05 2024-03-30T13:52:42.686 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.686 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.686 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:42.686 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:22.334602+0000", 2024-03-30T13:52:42.686 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00037344500000000003 2024-03-30T13:52:42.686 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.686 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.687 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:42.687 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:22.334735+0000", 2024-03-30T13:52:42.687 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000132621 2024-03-30T13:52:42.687 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.687 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.687 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:52:42.687 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:22.334857+0000", 2024-03-30T13:52:42.687 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012219100000000001 2024-03-30T13:52:42.687 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.687 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.687 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.687 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:22.340202+0000", 2024-03-30T13:52:42.688 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0053444709999999999 2024-03-30T13:52:42.688 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.688 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.688 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:52:42.688 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:22.343079+0000", 2024-03-30T13:52:42.688 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0028770850000000001 2024-03-30T13:52:42.688 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.688 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.688 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.688 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:22.416052+0000", 2024-03-30T13:52:42.688 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.072973113000000006 2024-03-30T13:52:42.688 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.688 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.688 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:52:42.689 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:22.416107+0000", 2024-03-30T13:52:42.689 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.507e-05 2024-03-30T13:52:42.689 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.689 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.689 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:52:42.689 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:22.416156+0000", 2024-03-30T13:52:42.689 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8813000000000002e-05 2024-03-30T13:52:42.689 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.689 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.689 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.689 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.689 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.689 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:258235 8.8 8:16116686:::benchmark_data_smithi043_119832_object127881:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:52:42.690 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:26.226138+0000", 2024-03-30T13:52:42.690 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 16.344558935999999, 2024-03-30T13:52:42.690 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.088397372000000002, 2024-03-30T13:52:42.690 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:42.690 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:42.690 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:42.690 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:42.690 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:52:42.690 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:42.690 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 258235 2024-03-30T13:52:42.690 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.690 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:42.691 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.691 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:42.691 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.226138+0000", 2024-03-30T13:52:42.691 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.691 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.691 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.691 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:42.691 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.226138+0000", 2024-03-30T13:52:42.691 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.691 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.691 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.691 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:42.691 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.226139+0000", 2024-03-30T13:52:42.692 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2049999999999999e-06 2024-03-30T13:52:42.692 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.692 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.692 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:42.692 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.226146+0000", 2024-03-30T13:52:42.692 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.0589999999999997e-06 2024-03-30T13:52:42.692 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.692 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.692 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:42.692 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.226162+0000", 2024-03-30T13:52:42.692 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5704000000000001e-05 2024-03-30T13:52:42.692 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.692 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.693 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:42.693 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.226171+0000", 2024-03-30T13:52:42.693 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.8379999999999997e-06 2024-03-30T13:52:42.693 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.693 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.693 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:42.693 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.226336+0000", 2024-03-30T13:52:42.693 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000164251 2024-03-30T13:52:42.693 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.693 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.693 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:42.693 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.226486+0000", 2024-03-30T13:52:42.693 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000150445 2024-03-30T13:52:42.693 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.694 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.694 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:52:42.694 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.226605+0000", 2024-03-30T13:52:42.694 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000119187 2024-03-30T13:52:42.694 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.694 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.694 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.694 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.234878+0000", 2024-03-30T13:52:42.694 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0082726220000000003 2024-03-30T13:52:42.694 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.694 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.695 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:52:42.695 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.234892+0000", 2024-03-30T13:52:42.695 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4048e-05 2024-03-30T13:52:42.695 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.695 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.695 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.695 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.314428+0000", 2024-03-30T13:52:42.695 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.079535884000000001 2024-03-30T13:52:42.695 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.695 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.695 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:52:42.695 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.314484+0000", 2024-03-30T13:52:42.695 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5729999999999997e-05 2024-03-30T13:52:42.696 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.696 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.696 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:52:42.696 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.314535+0000", 2024-03-30T13:52:42.696 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1399e-05 2024-03-30T13:52:42.696 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.696 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.696 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.696 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.696 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.696 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:258236 8.8 8:16e99abd:::benchmark_data_smithi043_119832_object127882:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:52:42.696 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:26.226198+0000", 2024-03-30T13:52:42.696 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 16.344498828999999, 2024-03-30T13:52:42.697 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.095556452, 2024-03-30T13:52:42.697 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:42.697 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:42.697 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:42.697 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:42.697 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:52:42.697 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:42.697 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 258236 2024-03-30T13:52:42.697 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.697 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:42.697 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.697 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:42.697 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.226198+0000", 2024-03-30T13:52:42.698 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.698 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.698 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.698 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:42.698 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.226198+0000", 2024-03-30T13:52:42.698 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.698 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.698 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.698 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:42.698 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.226198+0000", 2024-03-30T13:52:42.698 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4300000000000003e-07 2024-03-30T13:52:42.699 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.699 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.699 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:42.699 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.226202+0000", 2024-03-30T13:52:42.699 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.4829999999999999e-06 2024-03-30T13:52:42.699 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.699 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.699 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:42.699 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.226214+0000", 2024-03-30T13:52:42.699 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2662e-05 2024-03-30T13:52:42.699 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.699 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.700 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:42.700 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.226230+0000", 2024-03-30T13:52:42.700 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5752000000000001e-05 2024-03-30T13:52:42.700 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.700 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.700 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:42.700 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.229053+0000", 2024-03-30T13:52:42.700 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0028223599999999999 2024-03-30T13:52:42.700 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.700 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.700 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:42.700 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.229178+0000", 2024-03-30T13:52:42.700 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000124993 2024-03-30T13:52:42.701 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.701 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.701 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:52:42.701 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.229305+0000", 2024-03-30T13:52:42.701 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000127458 2024-03-30T13:52:42.701 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.701 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.701 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:52:42.701 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.234898+0000", 2024-03-30T13:52:42.701 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0055933110000000001 2024-03-30T13:52:42.701 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.701 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.701 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.702 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.234932+0000", 2024-03-30T13:52:42.702 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.3609000000000003e-05 2024-03-30T13:52:42.702 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.702 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.702 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.702 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.321648+0000", 2024-03-30T13:52:42.702 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.086715875999999997 2024-03-30T13:52:42.702 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.702 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.702 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:52:42.702 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.321708+0000", 2024-03-30T13:52:42.702 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9796000000000001e-05 2024-03-30T13:52:42.703 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.703 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.703 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:52:42.703 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.321754+0000", 2024-03-30T13:52:42.703 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6609e-05 2024-03-30T13:52:42.703 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.703 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.703 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.703 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.703 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.703 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:258239 8.8 8:1e00a959:::benchmark_data_smithi043_119832_object127885:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:52:42.703 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:26.227202+0000", 2024-03-30T13:52:42.704 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 16.343494087, 2024-03-30T13:52:42.704 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.094674865999999996, 2024-03-30T13:52:42.704 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:52:42.704 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:52:42.704 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:52:42.704 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:52:42.704 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:52:42.704 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:52:42.704 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 258239 2024-03-30T13:52:42.704 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.704 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:52:42.704 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.705 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:52:42.705 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.227202+0000", 2024-03-30T13:52:42.705 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.705 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.705 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.705 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:52:42.705 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.227202+0000", 2024-03-30T13:52:42.705 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:52:42.705 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.705 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.705 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:52:42.705 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.227203+0000", 2024-03-30T13:52:42.705 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8100000000000002e-07 2024-03-30T13:52:42.706 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.706 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.706 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:52:42.706 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.227208+0000", 2024-03-30T13:52:42.706 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9389999999999996e-06 2024-03-30T13:52:42.706 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.706 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.706 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:52:42.706 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.227230+0000", 2024-03-30T13:52:42.706 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.196e-05 2024-03-30T13:52:42.706 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.706 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.706 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:52:42.707 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.227238+0000", 2024-03-30T13:52:42.707 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.1780000000000003e-06 2024-03-30T13:52:42.707 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.707 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.707 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:52:42.707 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.231806+0000", 2024-03-30T13:52:42.707 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0045678979999999999 2024-03-30T13:52:42.707 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.707 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.707 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:52:42.707 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.231906+0000", 2024-03-30T13:52:42.707 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.9446999999999998e-05 2024-03-30T13:52:42.708 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.708 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.708 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:52:42.708 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.232033+0000", 2024-03-30T13:52:42.708 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012780800000000001 2024-03-30T13:52:42.708 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.708 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.708 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:52:42.708 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.234947+0000", 2024-03-30T13:52:42.708 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0029136259999999999 2024-03-30T13:52:42.708 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.708 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.708 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.709 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.235859+0000", 2024-03-30T13:52:42.709 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00091205699999999997 2024-03-30T13:52:42.709 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.709 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.709 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:52:42.709 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.321794+0000", 2024-03-30T13:52:42.709 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.085935046000000001 2024-03-30T13:52:42.709 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.709 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.709 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:52:42.709 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.321840+0000", 2024-03-30T13:52:42.709 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6239999999999998e-05 2024-03-30T13:52:42.709 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:52:42.710 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:52:42.710 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:52:42.710 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:26.321877+0000", 2024-03-30T13:52:42.710 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.6885999999999999e-05 2024-03-30T13:52:42.710 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.710 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.710 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.710 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:52:42.710 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:52:42.710 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:52:42.745 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:42.770 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:42.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:42.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:42.871 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:42.872 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:42 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:52:42.873 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:42 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:52:42.873 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:42 smithi043 ceph-mon[29963]: pgmap v408: 142 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 26 active+undersized, 99 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 680 B/s rd, 0 op/s; 56/594 objects degraded (9.428%); 1/594 objects misplaced (0.168%); 96 KiB/s, 0 objects/s recovering 2024-03-30T13:52:42.873 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:42 smithi043 ceph-mon[29963]: osdmap e210: 8 total, 7 up, 8 in 2024-03-30T13:52:42.873 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:42 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2025786194' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-30T13:52:42.873 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:42 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:52:42.873 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:42 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:52:42.873 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:42 smithi043 ceph-mon[42099]: pgmap v408: 142 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 26 active+undersized, 99 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 680 B/s rd, 0 op/s; 56/594 objects degraded (9.428%); 1/594 objects misplaced (0.168%); 96 KiB/s, 0 objects/s recovering 2024-03-30T13:52:42.873 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:42 smithi043 ceph-mon[42099]: osdmap e210: 8 total, 7 up, 8 in 2024-03-30T13:52:42.873 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:42 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2025786194' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-30T13:52:42.897 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:42.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:42.999 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:43.022 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:43.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:43.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:43.124 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:43.148 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:43.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:43.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:43.250 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:43.274 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:43.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:43.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:43.376 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:43.399 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:43.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:43.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:43.444 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-30T13:52:43.474 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:52:43.501 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:43.530 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:43.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:43.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:43.631 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:43.656 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:43.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:43.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:43.696 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:52:43.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:43 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2025786194' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-30T13:52:43.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:43 smithi113 ceph-mon[34341]: osdmap e211: 8 total, 7 up, 8 in 2024-03-30T13:52:43.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:43 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:52:43.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:43 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:52:43.757 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:43.782 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:43.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:43.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:43.883 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:43.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:43 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2025786194' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-30T13:52:43.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:43 smithi043 ceph-mon[29963]: osdmap e211: 8 total, 7 up, 8 in 2024-03-30T13:52:43.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:43 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:52:43.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:43 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:52:43.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:43 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2025786194' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-30T13:52:43.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:43 smithi043 ceph-mon[42099]: osdmap e211: 8 total, 7 up, 8 in 2024-03-30T13:52:43.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:43 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:52:43.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:43 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:52:43.907 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:43.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:43.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:44.009 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:44.022 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-03-30T13:52:44.032 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:44.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:44.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:44.133 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:44.158 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:44.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:44.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:44.167 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:44.261 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:44.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:44.286 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:44.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:44.387 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:44.411 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:44.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:44.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:44.513 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:44.527 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T13:52:44.537 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:44.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:44.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:44.567 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:44 smithi043 ceph-mon[29963]: pgmap v411: 158 pgs: 16 unknown, 5 active+clean+remapped, 12 active+undersized+degraded, 26 active+undersized, 99 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 56/594 objects degraded (9.428%); 1/594 objects misplaced (0.168%); 96 KiB/s, 0 objects/s recovering 2024-03-30T13:52:44.567 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:44 smithi043 ceph-mon[29963]: osdmap e212: 8 total, 7 up, 8 in 2024-03-30T13:52:44.567 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:44 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1941552714' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:52:44.567 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:44 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:52:44.567 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:44 smithi043 ceph-mon[29963]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:52:44.568 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:44 smithi043 ceph-mon[42099]: pgmap v411: 158 pgs: 16 unknown, 5 active+clean+remapped, 12 active+undersized+degraded, 26 active+undersized, 99 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 56/594 objects degraded (9.428%); 1/594 objects misplaced (0.168%); 96 KiB/s, 0 objects/s recovering 2024-03-30T13:52:44.568 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:44 smithi043 ceph-mon[42099]: osdmap e212: 8 total, 7 up, 8 in 2024-03-30T13:52:44.568 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:44 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1941552714' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:52:44.568 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:44 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:52:44.568 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:44 smithi043 ceph-mon[42099]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:52:44.639 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:44.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:44.776 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:44.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:44.877 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:44.883 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:44 smithi113 ceph-mon[34341]: pgmap v411: 158 pgs: 16 unknown, 5 active+clean+remapped, 12 active+undersized+degraded, 26 active+undersized, 99 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 56/594 objects degraded (9.428%); 1/594 objects misplaced (0.168%); 96 KiB/s, 0 objects/s recovering 2024-03-30T13:52:44.883 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:44 smithi113 ceph-mon[34341]: osdmap e212: 8 total, 7 up, 8 in 2024-03-30T13:52:44.883 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:44 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1941552714' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:52:44.883 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:44 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:52:44.883 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:44 smithi113 ceph-mon[34341]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:52:44.900 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:44.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:44.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:45.003 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:45.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:45.028 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:45.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:45.129 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:45.154 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:45.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:45.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:45.256 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:45.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:45.282 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:45.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:45.382 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:45.408 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:45.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:45.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:45.424 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:52:45.424 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [], 2024-03-30T13:52:45.424 INFO:teuthology.orchestra.run.smithi113.stdout: "num_ops": 0 2024-03-30T13:52:45.424 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:52:45.510 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:45.536 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:45.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:45.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:45.638 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:45.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:45 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:52:45.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:45 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:52:45.647 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:45 smithi043 ceph-mon[42099]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:52:45.647 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:45 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:52:45.647 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:45 smithi043 ceph-mon[42099]: osdmap e213: 8 total, 7 up, 8 in 2024-03-30T13:52:45.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:45 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:52:45.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:45 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:52:45.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:45 smithi043 ceph-mon[29963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:52:45.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:45 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:52:45.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:45 smithi043 ceph-mon[29963]: osdmap e213: 8 total, 7 up, 8 in 2024-03-30T13:52:45.663 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:45.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:45.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:45.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:45 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:52:45.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:45 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:52:45.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:45 smithi113 ceph-mon[34341]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:52:45.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:45 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:52:45.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:45 smithi113 ceph-mon[34341]: osdmap e213: 8 total, 7 up, 8 in 2024-03-30T13:52:45.764 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:45.788 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:45.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:45.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:45.890 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:45.913 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:45.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:46.015 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:46.040 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:46.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:46.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:46.044 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-03-30T13:52:46.142 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:46.166 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:46.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:46.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:46.268 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:46.293 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:46.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:46.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:46.323 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:52:46.395 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:46.396 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:52:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:52:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:52:46.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:46.419 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:46.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:46.458 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:46.520 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:46.545 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:46.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:46.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:46.647 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:46.671 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:46.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:46.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:46.675 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:52:46.695 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:52:46.695 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:46 smithi043 ceph-mon[29963]: 4.1 scrub starts 2024-03-30T13:52:46.695 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:46 smithi043 ceph-mon[29963]: 4.1 scrub ok 2024-03-30T13:52:46.695 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:46 smithi043 ceph-mon[29963]: pgmap v414: 158 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 29 active+undersized, 112 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 56/594 objects degraded (9.428%); 1/594 objects misplaced (0.168%) 2024-03-30T13:52:46.695 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:46 smithi043 ceph-mon[29963]: 3.10 scrub starts 2024-03-30T13:52:46.695 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:46 smithi043 ceph-mon[29963]: 3.10 scrub ok 2024-03-30T13:52:46.695 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:46 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:52:46.695 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:46 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:52:46.696 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:46 smithi043 ceph-mon[42099]: 4.1 scrub starts 2024-03-30T13:52:46.696 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:46 smithi043 ceph-mon[42099]: 4.1 scrub ok 2024-03-30T13:52:46.696 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:46 smithi043 ceph-mon[42099]: pgmap v414: 158 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 29 active+undersized, 112 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 56/594 objects degraded (9.428%); 1/594 objects misplaced (0.168%) 2024-03-30T13:52:46.696 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:46 smithi043 ceph-mon[42099]: 3.10 scrub starts 2024-03-30T13:52:46.696 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:46 smithi043 ceph-mon[42099]: 3.10 scrub ok 2024-03-30T13:52:46.696 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:46 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:52:46.696 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:46 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:52:46.704 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:46 smithi113 ceph-mon[34341]: 4.1 scrub starts 2024-03-30T13:52:46.704 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:46 smithi113 ceph-mon[34341]: 4.1 scrub ok 2024-03-30T13:52:46.704 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:46 smithi113 ceph-mon[34341]: pgmap v414: 158 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 29 active+undersized, 112 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 56/594 objects degraded (9.428%); 1/594 objects misplaced (0.168%) 2024-03-30T13:52:46.704 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:46 smithi113 ceph-mon[34341]: 3.10 scrub starts 2024-03-30T13:52:46.704 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:46 smithi113 ceph-mon[34341]: 3.10 scrub ok 2024-03-30T13:52:46.704 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:46 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:52:46.704 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:46 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:52:46.714 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:52:46.735 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:52:46.754 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:52:46.773 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:46.774 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:52:46.794 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:52:46.795 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:52:46.795 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:52:46.795 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:52:46.795 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:52:46.795 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:52:46.795 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:52:46.795 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:52:46.795 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:52:46.796 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:46.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:46.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:46.898 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:46.922 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:46.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:46.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:47.024 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:47.048 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:47.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:47.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:47.150 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:47.173 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:47.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:47.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:47.275 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:47.300 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:47.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:47.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:47.347 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T13:52:47.402 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:47.426 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:47.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:47.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:47.528 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:47.552 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:47.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:47.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:47.589 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:47 smithi113 ceph-mon[34341]: osdmap e214: 8 total, 7 up, 8 in 2024-03-30T13:52:47.589 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:47 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3830565038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:52:47.589 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:47 smithi113 ceph-mon[34341]: 5.12 scrub starts 2024-03-30T13:52:47.589 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:47 smithi113 ceph-mon[34341]: 5.12 scrub ok 2024-03-30T13:52:47.590 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:47 smithi113 ceph-mon[34341]: 1.2 scrub starts 2024-03-30T13:52:47.590 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:47 smithi113 ceph-mon[34341]: 1.2 scrub ok 2024-03-30T13:52:47.590 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:47 smithi113 ceph-mon[34341]: 3.6 scrub starts 2024-03-30T13:52:47.590 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:47 smithi113 ceph-mon[34341]: 3.6 scrub ok 2024-03-30T13:52:47.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:47 smithi043 ceph-mon[29963]: osdmap e214: 8 total, 7 up, 8 in 2024-03-30T13:52:47.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:47 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3830565038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:52:47.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:47 smithi043 ceph-mon[29963]: 5.12 scrub starts 2024-03-30T13:52:47.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:47 smithi043 ceph-mon[29963]: 5.12 scrub ok 2024-03-30T13:52:47.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:47 smithi043 ceph-mon[29963]: 1.2 scrub starts 2024-03-30T13:52:47.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:47 smithi043 ceph-mon[29963]: 1.2 scrub ok 2024-03-30T13:52:47.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:47 smithi043 ceph-mon[29963]: 3.6 scrub starts 2024-03-30T13:52:47.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:47 smithi043 ceph-mon[29963]: 3.6 scrub ok 2024-03-30T13:52:47.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:47 smithi043 ceph-mon[42099]: osdmap e214: 8 total, 7 up, 8 in 2024-03-30T13:52:47.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:47 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3830565038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:52:47.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:47 smithi043 ceph-mon[42099]: 5.12 scrub starts 2024-03-30T13:52:47.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:47 smithi043 ceph-mon[42099]: 5.12 scrub ok 2024-03-30T13:52:47.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:47 smithi043 ceph-mon[42099]: 1.2 scrub starts 2024-03-30T13:52:47.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:47 smithi043 ceph-mon[42099]: 1.2 scrub ok 2024-03-30T13:52:47.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:47 smithi043 ceph-mon[42099]: 3.6 scrub starts 2024-03-30T13:52:47.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:47 smithi043 ceph-mon[42099]: 3.6 scrub ok 2024-03-30T13:52:47.654 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:47.678 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:47.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:47.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:47.779 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:47.804 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:47.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:47.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:47.906 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:47.930 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:47.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:47.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:48.032 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:48.043 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:52:48.043 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:52:48.056 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:48.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:48.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:48.157 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:48.160 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:52:48.160 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:52:48.182 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:48.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:48.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:48.284 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:48.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:48.308 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:48.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:48.409 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:48.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:48.434 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:48.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:48.535 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:48.559 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:48.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:48.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:48.661 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:48.685 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:48.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:48.787 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:48.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:48.811 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:48.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:48.814 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[42099]: pgmap v416: 158 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 29 active+undersized, 112 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 56/594 objects degraded (9.428%); 1/594 objects misplaced (0.168%) 2024-03-30T13:52:48.814 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[42099]: 1.3 scrub starts 2024-03-30T13:52:48.814 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[42099]: 1.3 scrub ok 2024-03-30T13:52:48.814 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[42099]: 1.6 scrub starts 2024-03-30T13:52:48.815 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[42099]: 1.6 scrub ok 2024-03-30T13:52:48.815 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[42099]: 1.8 scrub starts 2024-03-30T13:52:48.815 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[42099]: 3.17 scrub starts 2024-03-30T13:52:48.815 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[42099]: 1.8 scrub ok 2024-03-30T13:52:48.815 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[42099]: 3.17 scrub ok 2024-03-30T13:52:48.815 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[42099]: 1.0 scrub starts 2024-03-30T13:52:48.815 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[42099]: 1.0 scrub ok 2024-03-30T13:52:48.815 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[29963]: pgmap v416: 158 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 29 active+undersized, 112 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 56/594 objects degraded (9.428%); 1/594 objects misplaced (0.168%) 2024-03-30T13:52:48.815 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[29963]: 1.3 scrub starts 2024-03-30T13:52:48.815 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[29963]: 1.3 scrub ok 2024-03-30T13:52:48.815 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[29963]: 1.6 scrub starts 2024-03-30T13:52:48.815 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[29963]: 1.6 scrub ok 2024-03-30T13:52:48.816 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[29963]: 1.8 scrub starts 2024-03-30T13:52:48.816 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[29963]: 3.17 scrub starts 2024-03-30T13:52:48.816 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[29963]: 1.8 scrub ok 2024-03-30T13:52:48.816 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[29963]: 3.17 scrub ok 2024-03-30T13:52:48.816 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[29963]: 1.0 scrub starts 2024-03-30T13:52:48.816 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:48 smithi043 ceph-mon[29963]: 1.0 scrub ok 2024-03-30T13:52:48.844 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:48 smithi113 ceph-mon[34341]: pgmap v416: 158 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 29 active+undersized, 112 active+clean; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 56/594 objects degraded (9.428%); 1/594 objects misplaced (0.168%) 2024-03-30T13:52:48.845 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:48 smithi113 ceph-mon[34341]: 1.3 scrub starts 2024-03-30T13:52:48.845 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:48 smithi113 ceph-mon[34341]: 1.3 scrub ok 2024-03-30T13:52:48.845 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:48 smithi113 ceph-mon[34341]: 1.6 scrub starts 2024-03-30T13:52:48.845 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:48 smithi113 ceph-mon[34341]: 1.6 scrub ok 2024-03-30T13:52:48.845 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:48 smithi113 ceph-mon[34341]: 1.8 scrub starts 2024-03-30T13:52:48.845 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:48 smithi113 ceph-mon[34341]: 3.17 scrub starts 2024-03-30T13:52:48.845 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:48 smithi113 ceph-mon[34341]: 1.8 scrub ok 2024-03-30T13:52:48.845 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:48 smithi113 ceph-mon[34341]: 3.17 scrub ok 2024-03-30T13:52:48.845 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:48 smithi113 ceph-mon[34341]: 1.0 scrub starts 2024-03-30T13:52:48.845 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:48 smithi113 ceph-mon[34341]: 1.0 scrub ok 2024-03-30T13:52:48.873 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-03-30T13:52:48.913 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:48.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:48.939 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:48.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:49.040 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:49.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:49.064 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:49.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:49.125 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:49.155 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:52:49.166 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:49.190 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:49.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:49.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:49.220 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:52:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:52:49.292 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:49.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:49.317 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:49.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:49.418 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:49.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:49.443 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:49.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:49.543 INFO:teuthology.orchestra.run.smithi043.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-03-30T13:52:49.544 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:49.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:49.569 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:49.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:49.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[29963]: 1.5 scrub starts 2024-03-30T13:52:49.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[29963]: 1.5 scrub ok 2024-03-30T13:52:49.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:52:49.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[29963]: 1.4 scrub starts 2024-03-30T13:52:49.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[29963]: 1.4 scrub ok 2024-03-30T13:52:49.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[29963]: 1.1 scrub starts 2024-03-30T13:52:49.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[29963]: 1.1 scrub ok 2024-03-30T13:52:49.648 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[29963]: 1.7 scrub starts 2024-03-30T13:52:49.648 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[29963]: 1.7 scrub ok 2024-03-30T13:52:49.648 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[29963]: 1.9 scrub starts 2024-03-30T13:52:49.648 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[29963]: 1.9 scrub ok 2024-03-30T13:52:49.648 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/217324121' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:52:49.648 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:52:49.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[42099]: 1.5 scrub starts 2024-03-30T13:52:49.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[42099]: 1.5 scrub ok 2024-03-30T13:52:49.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:52:49.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[42099]: 1.4 scrub starts 2024-03-30T13:52:49.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[42099]: 1.4 scrub ok 2024-03-30T13:52:49.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[42099]: 1.1 scrub starts 2024-03-30T13:52:49.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[42099]: 1.1 scrub ok 2024-03-30T13:52:49.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[42099]: 1.7 scrub starts 2024-03-30T13:52:49.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[42099]: 1.7 scrub ok 2024-03-30T13:52:49.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[42099]: 1.9 scrub starts 2024-03-30T13:52:49.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[42099]: 1.9 scrub ok 2024-03-30T13:52:49.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/217324121' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:52:49.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:49 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:52:49.671 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:49.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:49.695 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:49.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:49.796 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:49.820 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:49.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:49.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:49.921 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:49.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:49.946 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:49.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:49.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:49 smithi113 ceph-mon[34341]: 1.5 scrub starts 2024-03-30T13:52:49.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:49 smithi113 ceph-mon[34341]: 1.5 scrub ok 2024-03-30T13:52:49.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:49 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:52:49.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:49 smithi113 ceph-mon[34341]: 1.4 scrub starts 2024-03-30T13:52:49.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:49 smithi113 ceph-mon[34341]: 1.4 scrub ok 2024-03-30T13:52:49.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:49 smithi113 ceph-mon[34341]: 1.1 scrub starts 2024-03-30T13:52:49.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:49 smithi113 ceph-mon[34341]: 1.1 scrub ok 2024-03-30T13:52:49.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:49 smithi113 ceph-mon[34341]: 1.7 scrub starts 2024-03-30T13:52:49.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:49 smithi113 ceph-mon[34341]: 1.7 scrub ok 2024-03-30T13:52:49.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:49 smithi113 ceph-mon[34341]: 1.9 scrub starts 2024-03-30T13:52:49.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:49 smithi113 ceph-mon[34341]: 1.9 scrub ok 2024-03-30T13:52:49.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:49 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/217324121' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:52:49.971 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:49 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:52:50.044 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:52:50.047 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:50.112 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:50.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:50.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:50.214 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:50.238 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:50.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:50.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:50.340 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:50.365 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:50.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:50.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:50.467 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:50.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:50.493 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:50.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:50.536 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:50 smithi113 ceph-mon[34341]: pgmap v417: 158 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 29 active+undersized, 112 active+clean; 792 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 56/597 objects degraded (9.380%); 2/597 objects misplaced (0.335%) 2024-03-30T13:52:50.536 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:50 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 56/597 objects degraded (9.380%), 12 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:50.536 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:50 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-30T13:52:50.537 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:50 smithi113 ceph-mon[34341]: osdmap e215: 8 total, 7 up, 8 in 2024-03-30T13:52:50.594 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:50.619 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:50.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:50.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:50.720 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:50.746 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:50.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:50.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:50.816 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:50 smithi113 ceph-mon[34341]: 1.a scrub starts 2024-03-30T13:52:50.816 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:50 smithi113 ceph-mon[34341]: 1.a scrub ok 2024-03-30T13:52:50.840 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:52:50.840 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:52:50.848 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:50.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:50.874 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:50.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:50.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:50 smithi043 ceph-mon[29963]: pgmap v417: 158 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 29 active+undersized, 112 active+clean; 792 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 56/597 objects degraded (9.380%); 2/597 objects misplaced (0.335%) 2024-03-30T13:52:50.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:50 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 56/597 objects degraded (9.380%), 12 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:50.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:50 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-30T13:52:50.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:50 smithi043 ceph-mon[29963]: osdmap e215: 8 total, 7 up, 8 in 2024-03-30T13:52:50.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:50 smithi043 ceph-mon[29963]: 1.a scrub starts 2024-03-30T13:52:50.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:50 smithi043 ceph-mon[29963]: 1.a scrub ok 2024-03-30T13:52:50.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:50 smithi043 ceph-mon[42099]: pgmap v417: 158 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 29 active+undersized, 112 active+clean; 792 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 56/597 objects degraded (9.380%); 2/597 objects misplaced (0.335%) 2024-03-30T13:52:50.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:50 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 56/597 objects degraded (9.380%), 12 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:50.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:50 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-30T13:52:50.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:50 smithi043 ceph-mon[42099]: osdmap e215: 8 total, 7 up, 8 in 2024-03-30T13:52:50.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:50 smithi043 ceph-mon[42099]: 1.a scrub starts 2024-03-30T13:52:50.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:50 smithi043 ceph-mon[42099]: 1.a scrub ok 2024-03-30T13:52:50.975 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:51.001 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:51.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:51.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:51.103 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:51.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:51.128 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:51.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:51.229 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:51.252 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:51.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:51.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:51.354 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:51.378 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:51.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:51.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:51.480 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:51.505 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:51.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:51.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:51.530 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-30T13:52:51.533 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:51.607 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:51.616 DEBUG:teuthology.orchestra.run.smithi043:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-03-30T13:52:51.617 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-30T13:52:51.617 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-30T13:52:51.617 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T13:52:51.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:51.631 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:51.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:51.632 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-30T13:52:51.632 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-30T13:52:51.685 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:hints = 1 2024-03-30T13:52:51.685 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-30T13:52:51.685 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Object prefix: benchmark_data_smithi043_145506 2024-03-30T13:52:51.685 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T13:52:51.685 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 0 0 0 0 0 0 - 0 2024-03-30T13:52:51.732 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:51.774 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:51.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:51.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:51.807 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:51 smithi043 ceph-mon[29963]: 4.1d scrub starts 2024-03-30T13:52:51.808 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:51 smithi043 ceph-mon[29963]: 4.1d scrub ok 2024-03-30T13:52:51.808 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:51 smithi043 ceph-mon[29963]: 5.1a scrub starts 2024-03-30T13:52:51.808 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:51 smithi043 ceph-mon[29963]: 5.1a scrub ok 2024-03-30T13:52:51.809 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:51 smithi043 ceph-mon[42099]: 4.1d scrub starts 2024-03-30T13:52:51.809 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:51 smithi043 ceph-mon[42099]: 4.1d scrub ok 2024-03-30T13:52:51.809 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:51 smithi043 ceph-mon[42099]: 5.1a scrub starts 2024-03-30T13:52:51.809 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:51 smithi043 ceph-mon[42099]: 5.1a scrub ok 2024-03-30T13:52:51.818 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:52:51.876 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:51.924 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:51.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:51.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:51.953 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:51 smithi113 ceph-mon[34341]: 4.1d scrub starts 2024-03-30T13:52:51.953 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:51 smithi113 ceph-mon[34341]: 4.1d scrub ok 2024-03-30T13:52:51.953 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:51 smithi113 ceph-mon[34341]: 5.1a scrub starts 2024-03-30T13:52:51.953 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:51 smithi113 ceph-mon[34341]: 5.1a scrub ok 2024-03-30T13:52:52.027 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:52.054 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:52.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:52.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:52.155 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:52.193 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:52.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:52.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:52.294 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:52.333 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:52.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:52.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:52.435 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:52.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:52.473 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:52.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:52.478 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-03-30T13:52:52.574 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:52.613 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:52.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:52.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:52.685 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 1 15 3497 3482 217.608 217.625 0.00364124 0.0045801 2024-03-30T13:52:52.714 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:52.719 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:52 smithi113 ceph-mon[34341]: pgmap v419: 158 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 29 active+undersized, 112 active+clean; 792 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 56/597 objects degraded (9.380%); 2/597 objects misplaced (0.335%) 2024-03-30T13:52:52.719 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:52 smithi113 ceph-mon[34341]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T13:52:52.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:52 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3772348065' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:52:52.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:52 smithi113 ceph-mon[34341]: 6.1f scrub starts 2024-03-30T13:52:52.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:52 smithi113 ceph-mon[34341]: 2.2 scrub starts 2024-03-30T13:52:52.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:52 smithi113 ceph-mon[34341]: 2.2 scrub ok 2024-03-30T13:52:52.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:52 smithi113 ceph-mon[34341]: 6.1f scrub ok 2024-03-30T13:52:52.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:52 smithi113 ceph-mon[34341]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:52:52.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:52 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3772348065' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:52:52.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:52 smithi113 ceph-mon[34341]: osdmap e216: 8 total, 7 up, 8 in 2024-03-30T13:52:52.750 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:52.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:52.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:52.753 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:52 smithi043 ceph-mon[29963]: pgmap v419: 158 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 29 active+undersized, 112 active+clean; 792 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 56/597 objects degraded (9.380%); 2/597 objects misplaced (0.335%) 2024-03-30T13:52:52.753 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:52 smithi043 ceph-mon[29963]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T13:52:52.753 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:52 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3772348065' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:52:52.753 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:52 smithi043 ceph-mon[29963]: 6.1f scrub starts 2024-03-30T13:52:52.753 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:52 smithi043 ceph-mon[29963]: 2.2 scrub starts 2024-03-30T13:52:52.753 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:52 smithi043 ceph-mon[29963]: 2.2 scrub ok 2024-03-30T13:52:52.753 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:52 smithi043 ceph-mon[29963]: 6.1f scrub ok 2024-03-30T13:52:52.753 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:52 smithi043 ceph-mon[29963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:52:52.754 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:52 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3772348065' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:52:52.754 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:52 smithi043 ceph-mon[29963]: osdmap e216: 8 total, 7 up, 8 in 2024-03-30T13:52:52.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:52 smithi043 ceph-mon[42099]: pgmap v419: 158 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 29 active+undersized, 112 active+clean; 792 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 56/597 objects degraded (9.380%); 2/597 objects misplaced (0.335%) 2024-03-30T13:52:52.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:52 smithi043 ceph-mon[42099]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T13:52:52.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:52 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3772348065' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:52:52.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:52 smithi043 ceph-mon[42099]: 6.1f scrub starts 2024-03-30T13:52:52.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:52 smithi043 ceph-mon[42099]: 2.2 scrub starts 2024-03-30T13:52:52.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:52 smithi043 ceph-mon[42099]: 2.2 scrub ok 2024-03-30T13:52:52.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:52 smithi043 ceph-mon[42099]: 6.1f scrub ok 2024-03-30T13:52:52.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:52 smithi043 ceph-mon[42099]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:52:52.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:52 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3772348065' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:52:52.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:52 smithi043 ceph-mon[42099]: osdmap e216: 8 total, 7 up, 8 in 2024-03-30T13:52:52.852 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:52.899 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:52.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:52.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:53.001 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:53.037 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:53.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:53.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:53.138 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:53.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:53.184 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:53.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:53.285 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:53.321 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:53.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:53.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:53.423 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:53.461 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:53.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:53.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:53.562 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:53.600 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:53.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:53.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:53.671 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:53 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:52:53.671 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:53 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:52:53.671 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:53 smithi113 ceph-mon[34341]: 4.f scrub starts 2024-03-30T13:52:53.671 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:53 smithi113 ceph-mon[34341]: 4.f scrub ok 2024-03-30T13:52:53.671 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:53 smithi113 ceph-mon[34341]: 5.2 deep-scrub starts 2024-03-30T13:52:53.672 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:53 smithi113 ceph-mon[34341]: 5.2 deep-scrub ok 2024-03-30T13:52:53.672 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:53 smithi113 ceph-mon[34341]: 4.0 deep-scrub starts 2024-03-30T13:52:53.672 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:53 smithi113 ceph-mon[34341]: 4.0 deep-scrub ok 2024-03-30T13:52:53.672 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:53 smithi113 ceph-mon[34341]: osdmap e217: 8 total, 7 up, 8 in 2024-03-30T13:52:53.685 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 2 15 6941 6926 216.42 215.25 0.00615842 0.00461099 2024-03-30T13:52:53.702 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:53.741 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:53.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:53.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:53.787 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:52:53.787 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:52:53.843 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:53.882 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:53.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:53.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:53.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:53 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:52:53.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:53 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:52:53.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:53 smithi043 ceph-mon[29963]: 4.f scrub starts 2024-03-30T13:52:53.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:53 smithi043 ceph-mon[29963]: 4.f scrub ok 2024-03-30T13:52:53.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:53 smithi043 ceph-mon[29963]: 5.2 deep-scrub starts 2024-03-30T13:52:53.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:53 smithi043 ceph-mon[29963]: 5.2 deep-scrub ok 2024-03-30T13:52:53.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:53 smithi043 ceph-mon[29963]: 4.0 deep-scrub starts 2024-03-30T13:52:53.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:53 smithi043 ceph-mon[29963]: 4.0 deep-scrub ok 2024-03-30T13:52:53.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:53 smithi043 ceph-mon[29963]: osdmap e217: 8 total, 7 up, 8 in 2024-03-30T13:52:53.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:53 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:52:53.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:53 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:52:53.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:53 smithi043 ceph-mon[42099]: 4.f scrub starts 2024-03-30T13:52:53.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:53 smithi043 ceph-mon[42099]: 4.f scrub ok 2024-03-30T13:52:53.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:53 smithi043 ceph-mon[42099]: 5.2 deep-scrub starts 2024-03-30T13:52:53.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:53 smithi043 ceph-mon[42099]: 5.2 deep-scrub ok 2024-03-30T13:52:53.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:53 smithi043 ceph-mon[42099]: 4.0 deep-scrub starts 2024-03-30T13:52:53.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:53 smithi043 ceph-mon[42099]: 4.0 deep-scrub ok 2024-03-30T13:52:53.898 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:53 smithi043 ceph-mon[42099]: osdmap e217: 8 total, 7 up, 8 in 2024-03-30T13:52:53.985 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:54.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:54.023 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:54.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:54.124 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:54.159 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:54.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:54.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:54.261 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:54.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:54.301 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:54.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:54.316 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:54.402 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:54.444 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:52:54.447 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:54.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:54.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:54.486 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:52:54.529 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:52:54.549 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:54.560 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-03-30T13:52:54.569 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:52:54.586 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:54.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:54.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:54.610 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:52:54.653 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:52:54.685 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 3 16 10188 10172 211.899 202.875 0.00862987 0.0047098 2024-03-30T13:52:54.687 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:54.696 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:52:54.696 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:52:54.697 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:52:54.697 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:52:54.697 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:52:54.697 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:52:54.697 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:52:54.697 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:52:54.697 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:52:54.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:54.730 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:54.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:54.809 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:54 smithi043 ceph-mon[29963]: pgmap v422: 158 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 29 active+undersized, 112 active+clean; 792 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 56/597 objects degraded (9.380%); 2/597 objects misplaced (0.335%) 2024-03-30T13:52:54.809 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:54 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4000382396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:52:54.810 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:54 smithi043 ceph-mon[42099]: pgmap v422: 158 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 29 active+undersized, 112 active+clean; 792 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 56/597 objects degraded (9.380%); 2/597 objects misplaced (0.335%) 2024-03-30T13:52:54.810 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:54 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/4000382396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:52:54.831 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:54.871 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:54.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:54.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:54.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:54 smithi113 ceph-mon[34341]: pgmap v422: 158 pgs: 5 active+clean+remapped, 12 active+undersized+degraded, 29 active+undersized, 112 active+clean; 792 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 56/597 objects degraded (9.380%); 2/597 objects misplaced (0.335%) 2024-03-30T13:52:54.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:54 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/4000382396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:52:54.972 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:54.990 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:52:55.011 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:55.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:55.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:55.112 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:55.150 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:55.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:55.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:55.252 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:55.292 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:55.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:55.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:55.393 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:55.432 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:55.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:55.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:55.534 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:55.574 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:55.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:55.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:55.621 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T13:52:55.646 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:55 smithi043 ceph-mon[29963]: 4.1e scrub starts 2024-03-30T13:52:55.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:55 smithi043 ceph-mon[29963]: 4.1e scrub ok 2024-03-30T13:52:55.647 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:55 smithi043 ceph-mon[42099]: 4.1e scrub starts 2024-03-30T13:52:55.647 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:55 smithi043 ceph-mon[42099]: 4.1e scrub ok 2024-03-30T13:52:55.677 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:55.685 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 4 16 13317 13301 207.811 195.562 0.00343382 0.0048042 2024-03-30T13:52:55.721 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:55.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:55.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:55.823 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:55.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:55.873 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:55.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:55.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:55 smithi113 ceph-mon[34341]: 4.1e scrub starts 2024-03-30T13:52:55.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:55 smithi113 ceph-mon[34341]: 4.1e scrub ok 2024-03-30T13:52:55.974 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:56.011 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:56.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:56.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:56.113 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:56.148 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:56.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:56.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:56.250 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:56.285 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:56.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:56.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:56.387 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:56.388 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:52:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:52:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:52:56.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:56.430 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:56.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:56.441 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:52:56.441 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:52:56.532 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:56.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:56.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:56.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:56 smithi043 ceph-mon[29963]: pgmap v423: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 484 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 54 KiB/s rd, 80 MiB/s wr, 1.39k op/s; 1463/23799 objects degraded (6.147%); 1/23799 objects misplaced (0.004%) 2024-03-30T13:52:56.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:56 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 1463/23799 objects degraded (6.147%), 15 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:56.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:56 smithi043 ceph-mon[42099]: pgmap v423: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 484 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 54 KiB/s rd, 80 MiB/s wr, 1.39k op/s; 1463/23799 objects degraded (6.147%); 1/23799 objects misplaced (0.004%) 2024-03-30T13:52:56.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:56 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 1463/23799 objects degraded (6.147%), 15 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:56.677 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:56.687 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 5 15 16135 16120 201.409 176.188 0.00847401 0.00495691 2024-03-30T13:52:56.722 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:56.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:56.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:56.824 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:56.861 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:56.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:56.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:56.894 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:56 smithi113 ceph-mon[34341]: pgmap v423: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 484 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 54 KiB/s rd, 80 MiB/s wr, 1.39k op/s; 1463/23799 objects degraded (6.147%); 1/23799 objects misplaced (0.004%) 2024-03-30T13:52:56.894 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:56 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 1463/23799 objects degraded (6.147%), 15 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-30T13:52:56.963 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:57.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:57.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:57.050 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:57.104 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:57.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:57.149 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:57.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:57.250 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:57.276 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:52:57.277 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [], 2024-03-30T13:52:57.277 INFO:teuthology.orchestra.run.smithi113.stdout: "complaint_time": 30, 2024-03-30T13:52:57.277 INFO:teuthology.orchestra.run.smithi113.stdout: "num_blocked_ops": 0 2024-03-30T13:52:57.277 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:52:57.291 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:57.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:57.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:57.393 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:57.430 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:57.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:57.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:57.461 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:52:57.464 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-30T13:52:57.531 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:52:57.571 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:57.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:57.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:52:57.673 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:57.687 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 6 15 18789 18774 195.486 165.875 0.00714262 0.00510956 2024-03-30T13:52:57.710 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:57.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:57.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:57.812 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:57.846 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:57.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:57.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:57.881 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:57 smithi043 ceph-mon[42099]: 4.18 scrub starts 2024-03-30T13:52:57.882 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:57 smithi043 ceph-mon[42099]: 4.18 scrub ok 2024-03-30T13:52:57.882 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:57 smithi043 ceph-mon[42099]: 5.10 scrub starts 2024-03-30T13:52:57.882 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:57 smithi043 ceph-mon[42099]: 5.10 scrub ok 2024-03-30T13:52:57.882 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:57 smithi043 ceph-mon[42099]: 6.16 deep-scrub starts 2024-03-30T13:52:57.882 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:57 smithi043 ceph-mon[42099]: 6.16 deep-scrub ok 2024-03-30T13:52:57.882 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:57 smithi043 ceph-mon[42099]: osdmap e218: 8 total, 7 up, 8 in 2024-03-30T13:52:57.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:57 smithi043 ceph-mon[29963]: 4.18 scrub starts 2024-03-30T13:52:57.883 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:57 smithi043 ceph-mon[29963]: 4.18 scrub ok 2024-03-30T13:52:57.883 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:57 smithi043 ceph-mon[29963]: 5.10 scrub starts 2024-03-30T13:52:57.883 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:57 smithi043 ceph-mon[29963]: 5.10 scrub ok 2024-03-30T13:52:57.883 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:57 smithi043 ceph-mon[29963]: 6.16 deep-scrub starts 2024-03-30T13:52:57.883 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:57 smithi043 ceph-mon[29963]: 6.16 deep-scrub ok 2024-03-30T13:52:57.883 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:57 smithi043 ceph-mon[29963]: osdmap e218: 8 total, 7 up, 8 in 2024-03-30T13:52:57.944 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:57 smithi113 ceph-mon[34341]: 4.18 scrub starts 2024-03-30T13:52:57.944 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:57 smithi113 ceph-mon[34341]: 4.18 scrub ok 2024-03-30T13:52:57.944 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:57 smithi113 ceph-mon[34341]: 5.10 scrub starts 2024-03-30T13:52:57.944 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:57 smithi113 ceph-mon[34341]: 5.10 scrub ok 2024-03-30T13:52:57.944 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:57 smithi113 ceph-mon[34341]: 6.16 deep-scrub starts 2024-03-30T13:52:57.944 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:57 smithi113 ceph-mon[34341]: 6.16 deep-scrub ok 2024-03-30T13:52:57.944 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:57 smithi113 ceph-mon[34341]: osdmap e218: 8 total, 7 up, 8 in 2024-03-30T13:52:57.948 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:57.989 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:57.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:57.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:57.999 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-03-30T13:52:58.091 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:58.127 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:58.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:58.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:58.229 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:58.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:58.268 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:58.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:58.370 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:58.410 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:58.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:58.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:58.442 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:52:58.451 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:52:58.512 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:58.563 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:58.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:58.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:58.665 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:58.687 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 7 15 21496 21481 191.727 169.188 0.00767737 0.00520933 2024-03-30T13:52:58.699 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:58.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:58.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:58.762 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:58 smithi043 ceph-mon[29963]: from='client.25226 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:52:58.762 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:58 smithi043 ceph-mon[29963]: pgmap v425: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 484 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 54 KiB/s rd, 80 MiB/s wr, 1.39k op/s; 1463/23799 objects degraded (6.147%); 1/23799 objects misplaced (0.004%) 2024-03-30T13:52:58.762 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:58 smithi043 ceph-mon[29963]: 5.3 deep-scrub starts 2024-03-30T13:52:58.762 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:58 smithi043 ceph-mon[29963]: 5.3 deep-scrub ok 2024-03-30T13:52:58.762 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:58 smithi043 ceph-mon[29963]: 4.11 scrub starts 2024-03-30T13:52:58.762 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:58 smithi043 ceph-mon[29963]: 4.11 scrub ok 2024-03-30T13:52:58.762 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:58 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:52:58.763 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:58 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:52:58.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:58 smithi043 ceph-mon[42099]: from='client.25226 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:52:58.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:58 smithi043 ceph-mon[42099]: pgmap v425: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 484 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 54 KiB/s rd, 80 MiB/s wr, 1.39k op/s; 1463/23799 objects degraded (6.147%); 1/23799 objects misplaced (0.004%) 2024-03-30T13:52:58.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:58 smithi043 ceph-mon[42099]: 5.3 deep-scrub starts 2024-03-30T13:52:58.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:58 smithi043 ceph-mon[42099]: 5.3 deep-scrub ok 2024-03-30T13:52:58.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:58 smithi043 ceph-mon[42099]: 4.11 scrub starts 2024-03-30T13:52:58.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:58 smithi043 ceph-mon[42099]: 4.11 scrub ok 2024-03-30T13:52:58.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:58 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:52:58.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:58 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:52:58.801 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:58.824 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2024-03-30T13:52:55.499582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":507517278,"num_objects":7933,"num_object_clones":0,"num_object_copies":23799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1463,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":7933,"num_whiteouts":0,"num_read":1750,"num_read_kb":1502,"num_write":8457,"num_write_kb":495085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9965,"ondisk_log_size":9965,"up":428,"acting":433,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":477,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5436596,"kb_used_data":3068620,"kb_used_omap":12,"kb_used_meta":2367859,"kb_avail":744524620,"statfs":{"total":767960285184,"available":762393210880,"internally_reserved":0,"allocated":3142266880,"data_stored":3137487757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":2424688176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":506706281,"num_objects":7734,"num_object_clones":0,"num_object_copies":23202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":-1,"num_objects_unfound":0,"num_objects_dirty":7734,"num_whiteouts":0,"num_read":361,"num_read_kb":326,"num_write":7956,"num_write_kb":494162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-644037,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002981"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":339,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.475515+0000","last_change":"2024-03-30T13:51:05.234138+0000","last_active":"2024-03-30T13:52:53.475515+0000","last_peered":"2024-03-30T13:52:53.475515+0000","last_clean":"2024-03-30T13:52:53.475515+0000","last_became_active":"2024-03-30T13:42:51.329797+0000","last_became_peered":"2024-03-30T13:42:51.329797+0000","last_unstale":"2024-03-30T13:52:53.475515+0000","last_undegraded":"2024-03-30T13:52:53.475515+0000","last_fullsized":"2024-03-30T13:52:53.475515+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:05.234023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:05.234023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":358,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.481533+0000","last_change":"2024-03-30T13:52:17.548701+0000","last_active":"2024-03-30T13:52:53.481533+0000","last_peered":"2024-03-30T13:52:53.481533+0000","last_clean":"2024-03-30T13:52:53.481533+0000","last_became_active":"2024-03-30T13:42:49.323268+0000","last_became_peered":"2024-03-30T13:42:49.323268+0000","last_unstale":"2024-03-30T13:52:53.481533+0000","last_undegraded":"2024-03-30T13:52:53.481533+0000","last_fullsized":"2024-03-30T13:52:53.481533+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:17.548567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:17.548567+0000","last_clean_scrub_stamp":"2024-03-30T13:52:17.548567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:34.274579+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"65'1","reported_seq":349,"reported_epoch":217,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:53.479871+0000","last_change":"2024-03-30T13:50:08.513977+0000","last_active":"2024-03-30T13:52:53.479871+0000","last_peered":"2024-03-30T13:52:53.479871+0000","last_clean":"2024-03-30T13:50:05.499990+0000","last_became_active":"2024-03-30T13:50:08.513977+0000","last_became_peered":"2024-03-30T13:50:08.513977+0000","last_unstale":"2024-03-30T13:52:53.479871+0000","last_undegraded":"2024-03-30T13:50:08.505072+0000","last_fullsized":"2024-03-30T13:50:08.504945+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:48:19.930059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:48:19.930059+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"216'15","reported_seq":372,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.475437+0000","last_change":"2024-03-30T13:51:06.228575+0000","last_active":"2024-03-30T13:52:53.475437+0000","last_peered":"2024-03-30T13:52:53.475437+0000","last_clean":"2024-03-30T13:52:53.475437+0000","last_became_active":"2024-03-30T13:42:47.319638+0000","last_became_peered":"2024-03-30T13:42:47.319638+0000","last_unstale":"2024-03-30T13:52:53.475437+0000","last_undegraded":"2024-03-30T13:52:53.475437+0000","last_fullsized":"2024-03-30T13:52:53.475437+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:51:06.228451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:06.228451+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":338,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.478100+0000","last_change":"2024-03-30T13:52:23.133952+0000","last_active":"2024-03-30T13:52:53.478100+0000","last_peered":"2024-03-30T13:52:53.478100+0000","last_clean":"2024-03-30T13:52:53.478100+0000","last_became_active":"2024-03-30T13:42:51.374403+0000","last_became_peered":"2024-03-30T13:42:51.374403+0000","last_unstale":"2024-03-30T13:52:53.478100+0000","last_undegraded":"2024-03-30T13:52:53.478100+0000","last_fullsized":"2024-03-30T13:52:53.478100+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:23.133743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:23.133743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:47.986000+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":343,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.475542+0000","last_change":"2024-03-30T13:51:20.149796+0000","last_active":"2024-03-30T13:52:53.475542+0000","last_peered":"2024-03-30T13:52:53.475542+0000","last_clean":"2024-03-30T13:52:53.475542+0000","last_became_active":"2024-03-30T13:42:49.327124+0000","last_became_peered":"2024-03-30T13:42:49.327124+0000","last_unstale":"2024-03-30T13:52:53.475542+0000","last_undegraded":"2024-03-30T13:52:53.475542+0000","last_fullsized":"2024-03-30T13:52:53.475542+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:20.149684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:20.149684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":346,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.478618+0000","last_change":"2024-03-30T13:50:08.512316+0000","last_active":"2024-03-30T13:52:53.478618+0000","last_peered":"2024-03-30T13:52:53.478618+0000","last_clean":"2024-03-30T13:50:05.504688+0000","last_became_active":"2024-03-30T13:50:08.512316+0000","last_became_peered":"2024-03-30T13:50:08.512316+0000","last_unstale":"2024-03-30T13:52:53.478618+0000","last_undegraded":"2024-03-30T13:52:53.478618+0000","last_fullsized":"2024-03-30T13:50:08.501439+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:20.499768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:20.499768+0000","last_clean_scrub_stamp":"2024-03-30T13:49:20.499768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:55.942945+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"217'34","reported_seq":383,"reported_epoch":217,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:53.565918+0000","last_change":"2024-03-30T13:50:08.511099+0000","last_active":"2024-03-30T13:52:53.565918+0000","last_peered":"2024-03-30T13:52:53.565918+0000","last_clean":"2024-03-30T13:50:05.499201+0000","last_became_active":"2024-03-30T13:50:08.511099+0000","last_became_peered":"2024-03-30T13:50:08.511099+0000","last_unstale":"2024-03-30T13:52:53.565918+0000","last_undegraded":"2024-03-30T13:50:08.502733+0000","last_fullsized":"2024-03-30T13:50:08.502606+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'22","last_scrub_stamp":"2024-03-30T13:46:49.339954+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:29.151791+0000","last_clean_scrub_stamp":"2024-03-30T13:46:49.339954+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:55.942909+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":344,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.479396+0000","last_change":"2024-03-30T13:52:27.061489+0000","last_active":"2024-03-30T13:52:53.479396+0000","last_peered":"2024-03-30T13:52:53.479396+0000","last_clean":"2024-03-30T13:52:53.479396+0000","last_became_active":"2024-03-30T13:42:51.331196+0000","last_became_peered":"2024-03-30T13:42:51.331196+0000","last_unstale":"2024-03-30T13:52:53.479396+0000","last_undegraded":"2024-03-30T13:52:53.479396+0000","last_fullsized":"2024-03-30T13:52:53.479396+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:27.061281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:27.061281+0000","last_clean_scrub_stamp":"2024-03-30T13:52:27.061281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:55.346730+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":357,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.483041+0000","last_change":"2024-03-30T13:52:20.069944+0000","last_active":"2024-03-30T13:52:53.483041+0000","last_peered":"2024-03-30T13:52:53.483041+0000","last_clean":"2024-03-30T13:52:53.483041+0000","last_became_active":"2024-03-30T13:42:49.321811+0000","last_became_peered":"2024-03-30T13:42:49.321811+0000","last_unstale":"2024-03-30T13:52:53.483041+0000","last_undegraded":"2024-03-30T13:52:53.483041+0000","last_fullsized":"2024-03-30T13:52:53.483041+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:20.069864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:20.069864+0000","last_clean_scrub_stamp":"2024-03-30T13:52:20.069864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:27.802831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":357,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.483410+0000","last_change":"2024-03-30T13:52:07.129258+0000","last_active":"2024-03-30T13:52:53.483410+0000","last_peered":"2024-03-30T13:52:53.483410+0000","last_clean":"2024-03-30T13:52:53.483410+0000","last_became_active":"2024-03-30T13:42:45.319827+0000","last_became_peered":"2024-03-30T13:42:45.319827+0000","last_unstale":"2024-03-30T13:52:53.483410+0000","last_undegraded":"2024-03-30T13:52:53.483410+0000","last_fullsized":"2024-03-30T13:52:53.483410+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:07.129142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:37.513596+0000","last_clean_scrub_stamp":"2024-03-30T13:52:07.129142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:13.605042+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"216'15","reported_seq":380,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.475723+0000","last_change":"2024-03-30T13:51:16.153633+0000","last_active":"2024-03-30T13:52:53.475723+0000","last_peered":"2024-03-30T13:52:53.475723+0000","last_clean":"2024-03-30T13:52:53.475723+0000","last_became_active":"2024-03-30T13:42:47.310289+0000","last_became_peered":"2024-03-30T13:42:47.310289+0000","last_unstale":"2024-03-30T13:52:53.475723+0000","last_undegraded":"2024-03-30T13:52:53.475723+0000","last_fullsized":"2024-03-30T13:52:53.475723+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:51:16.153478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:16.153478+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":345,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.481627+0000","last_change":"2024-03-30T13:51:10.988992+0000","last_active":"2024-03-30T13:52:53.481627+0000","last_peered":"2024-03-30T13:52:53.481627+0000","last_clean":"2024-03-30T13:52:53.481627+0000","last_became_active":"2024-03-30T13:42:51.325748+0000","last_became_peered":"2024-03-30T13:42:51.325748+0000","last_unstale":"2024-03-30T13:52:53.481627+0000","last_undegraded":"2024-03-30T13:52:53.481627+0000","last_fullsized":"2024-03-30T13:52:53.481627+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:10.988839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:10.988839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":351,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.480456+0000","last_change":"2024-03-30T13:52:50.953670+0000","last_active":"2024-03-30T13:52:53.480456+0000","last_peered":"2024-03-30T13:52:53.480456+0000","last_clean":"2024-03-30T13:52:53.480456+0000","last_became_active":"2024-03-30T13:42:49.326027+0000","last_became_peered":"2024-03-30T13:42:49.326027+0000","last_unstale":"2024-03-30T13:52:53.480456+0000","last_undegraded":"2024-03-30T13:52:53.480456+0000","last_fullsized":"2024-03-30T13:52:53.480456+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:50.953533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:41.633284+0000","last_clean_scrub_stamp":"2024-03-30T13:52:50.953533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:59.091477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"186'2","reported_seq":364,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.483191+0000","last_change":"2024-03-30T13:51:15.932194+0000","last_active":"2024-03-30T13:52:53.483191+0000","last_peered":"2024-03-30T13:52:53.483191+0000","last_clean":"2024-03-30T13:52:53.483191+0000","last_became_active":"2024-03-30T13:42:45.319975+0000","last_became_peered":"2024-03-30T13:42:45.319975+0000","last_unstale":"2024-03-30T13:52:53.483191+0000","last_undegraded":"2024-03-30T13:52:53.483191+0000","last_fullsized":"2024-03-30T13:52:53.483191+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-30T13:51:15.932076+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-30T13:51:15.932076+0000","last_clean_scrub_stamp":"2024-03-30T13:51:15.932076+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"216'9","reported_seq":378,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.478141+0000","last_change":"2024-03-30T13:52:19.931416+0000","last_active":"2024-03-30T13:52:53.478141+0000","last_peered":"2024-03-30T13:52:53.478141+0000","last_clean":"2024-03-30T13:52:53.478141+0000","last_became_active":"2024-03-30T13:42:47.317352+0000","last_became_peered":"2024-03-30T13:42:47.317352+0000","last_unstale":"2024-03-30T13:52:53.478141+0000","last_undegraded":"2024-03-30T13:52:53.478141+0000","last_fullsized":"2024-03-30T13:52:53.478141+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-30T13:52:19.931197+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-30T13:43:50.960980+0000","last_clean_scrub_stamp":"2024-03-30T13:52:19.931197+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:47.014671+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":348,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.478472+0000","last_change":"2024-03-30T13:52:07.996845+0000","last_active":"2024-03-30T13:52:53.478472+0000","last_peered":"2024-03-30T13:52:53.478472+0000","last_clean":"2024-03-30T13:52:53.478472+0000","last_became_active":"2024-03-30T13:42:51.372011+0000","last_became_peered":"2024-03-30T13:42:51.372011+0000","last_unstale":"2024-03-30T13:52:53.478472+0000","last_undegraded":"2024-03-30T13:52:53.478472+0000","last_fullsized":"2024-03-30T13:52:53.478472+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:07.996724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:07.996724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:16.479878+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":358,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.482589+0000","last_change":"2024-03-30T13:52:19.544522+0000","last_active":"2024-03-30T13:52:53.482589+0000","last_peered":"2024-03-30T13:52:53.482589+0000","last_clean":"2024-03-30T13:52:53.482589+0000","last_became_active":"2024-03-30T13:42:49.322665+0000","last_became_peered":"2024-03-30T13:42:49.322665+0000","last_unstale":"2024-03-30T13:52:53.482589+0000","last_undegraded":"2024-03-30T13:52:53.482589+0000","last_fullsized":"2024-03-30T13:52:53.482589+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:19.544305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:30.465974+0000","last_clean_scrub_stamp":"2024-03-30T13:52:19.544305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:28.946233+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":356,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.480772+0000","last_change":"2024-03-30T13:52:16.009455+0000","last_active":"2024-03-30T13:52:53.480772+0000","last_peered":"2024-03-30T13:52:53.480772+0000","last_clean":"2024-03-30T13:52:53.480772+0000","last_became_active":"2024-03-30T13:42:45.317378+0000","last_became_peered":"2024-03-30T13:42:45.317378+0000","last_unstale":"2024-03-30T13:52:53.480772+0000","last_undegraded":"2024-03-30T13:52:53.480772+0000","last_fullsized":"2024-03-30T13:52:53.480772+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:16.009354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:43.088964+0000","last_clean_scrub_stamp":"2024-03-30T13:52:16.009354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:37.935631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"216'27","reported_seq":184,"reported_epoch":217,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:53.482064+0000","last_change":"2024-03-30T13:50:08.508437+0000","last_active":"2024-03-30T13:52:53.482064+0000","last_peered":"2024-03-30T13:52:53.482064+0000","last_clean":"2024-03-30T13:50:05.490228+0000","last_became_active":"2024-03-30T13:50:08.508437+0000","last_became_peered":"2024-03-30T13:50:08.508437+0000","last_unstale":"2024-03-30T13:52:53.482064+0000","last_undegraded":"2024-03-30T13:50:08.506233+0000","last_fullsized":"2024-03-30T13:50:08.506116+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-30T13:49:08.415681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:49:08.415681+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:57.611614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":338,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.481396+0000","last_change":"2024-03-30T13:52:52.054425+0000","last_active":"2024-03-30T13:52:53.481396+0000","last_peered":"2024-03-30T13:52:53.481396+0000","last_clean":"2024-03-30T13:52:53.481396+0000","last_became_active":"2024-03-30T13:42:51.371378+0000","last_became_peered":"2024-03-30T13:42:51.371378+0000","last_unstale":"2024-03-30T13:52:53.481396+0000","last_undegraded":"2024-03-30T13:52:53.481396+0000","last_fullsized":"2024-03-30T13:52:53.481396+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:52.054302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:52.054302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:14.226766+0000","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":338,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.477145+0000","last_change":"2024-03-30T13:50:08.506226+0000","last_active":"2024-03-30T13:52:53.477145+0000","last_peered":"2024-03-30T13:52:53.477145+0000","last_clean":"2024-03-30T13:50:05.490914+0000","last_became_active":"2024-03-30T13:50:08.506226+0000","last_became_peered":"2024-03-30T13:50:08.506226+0000","last_unstale":"2024-03-30T13:52:53.477145+0000","last_undegraded":"2024-03-30T13:52:53.477145+0000","last_fullsized":"2024-03-30T13:50:08.502611+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:25.104610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:48:25.104610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":346,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.475824+0000","last_change":"2024-03-30T13:50:08.508556+0000","last_active":"2024-03-30T13:52:53.475824+0000","last_peered":"2024-03-30T13:52:53.475824+0000","last_clean":"2024-03-30T13:50:07.261781+0000","last_became_active":"2024-03-30T13:50:08.508556+0000","last_became_peered":"2024-03-30T13:50:08.508556+0000","last_unstale":"2024-03-30T13:52:53.475824+0000","last_undegraded":"2024-03-30T13:52:53.475824+0000","last_fullsized":"2024-03-30T13:50:08.502793+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:07.261595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:50:07.261595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"217'23","reported_seq":391,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.576075+0000","last_change":"2024-03-30T13:52:50.945958+0000","last_active":"2024-03-30T13:52:53.576075+0000","last_peered":"2024-03-30T13:52:53.576075+0000","last_clean":"2024-03-30T13:52:53.576075+0000","last_became_active":"2024-03-30T13:42:47.317881+0000","last_became_peered":"2024-03-30T13:42:47.317881+0000","last_unstale":"2024-03-30T13:52:53.576075+0000","last_undegraded":"2024-03-30T13:52:53.576075+0000","last_fullsized":"2024-03-30T13:52:53.576075+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'17","last_scrub_stamp":"2024-03-30T13:52:50.945844+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-03-30T13:43:52.044427+0000","last_clean_scrub_stamp":"2024-03-30T13:52:50.945844+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:04.119247+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":332,"reported_epoch":217,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:53.482050+0000","last_change":"2024-03-30T13:50:08.517791+0000","last_active":"2024-03-30T13:52:53.482050+0000","last_peered":"2024-03-30T13:52:53.482050+0000","last_clean":"2024-03-30T13:50:05.495152+0000","last_became_active":"2024-03-30T13:50:08.517791+0000","last_became_peered":"2024-03-30T13:50:08.517791+0000","last_unstale":"2024-03-30T13:52:53.482050+0000","last_undegraded":"2024-03-30T13:50:08.511324+0000","last_fullsized":"2024-03-30T13:50:08.511207+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T13:48:25.249724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:25.249724+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:55.950567+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":489,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.483924+0000","last_change":"2024-03-30T13:52:07.177353+0000","last_active":"2024-03-30T13:52:53.483924+0000","last_peered":"2024-03-30T13:52:53.483924+0000","last_clean":"2024-03-30T13:52:53.483924+0000","last_became_active":"2024-03-30T13:42:49.323362+0000","last_became_peered":"2024-03-30T13:42:49.323362+0000","last_unstale":"2024-03-30T13:52:53.483924+0000","last_undegraded":"2024-03-30T13:52:53.483924+0000","last_fullsized":"2024-03-30T13:52:53.483924+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:52:07.177214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:07.177214+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:33.858714+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"65'1","reported_seq":367,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.481640+0000","last_change":"2024-03-30T13:52:28.566954+0000","last_active":"2024-03-30T13:52:53.481640+0000","last_peered":"2024-03-30T13:52:53.481640+0000","last_clean":"2024-03-30T13:52:53.481640+0000","last_became_active":"2024-03-30T13:42:45.307832+0000","last_became_peered":"2024-03-30T13:42:45.307832+0000","last_unstale":"2024-03-30T13:52:53.481640+0000","last_undegraded":"2024-03-30T13:52:53.481640+0000","last_fullsized":"2024-03-30T13:52:53.481640+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:52:28.566847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:28.566847+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:57.125000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"217'25","reported_seq":388,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.536242+0000","last_change":"2024-03-30T13:51:22.253128+0000","last_active":"2024-03-30T13:52:53.536242+0000","last_peered":"2024-03-30T13:52:53.536242+0000","last_clean":"2024-03-30T13:52:53.536242+0000","last_became_active":"2024-03-30T13:42:47.500763+0000","last_became_peered":"2024-03-30T13:42:47.500763+0000","last_unstale":"2024-03-30T13:52:53.536242+0000","last_undegraded":"2024-03-30T13:52:53.536242+0000","last_fullsized":"2024-03-30T13:52:53.536242+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'15","last_scrub_stamp":"2024-03-30T13:51:22.252831+0000","last_deep_scrub":"153'15","last_deep_scrub_stamp":"2024-03-30T13:51:22.252831+0000","last_clean_scrub_stamp":"2024-03-30T13:51:22.252831+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":346,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.482900+0000","last_change":"2024-03-30T13:52:00.433677+0000","last_active":"2024-03-30T13:52:53.482900+0000","last_peered":"2024-03-30T13:52:53.482900+0000","last_clean":"2024-03-30T13:52:53.482900+0000","last_became_active":"2024-03-30T13:42:51.325513+0000","last_became_peered":"2024-03-30T13:42:51.325513+0000","last_unstale":"2024-03-30T13:52:53.482900+0000","last_undegraded":"2024-03-30T13:52:53.482900+0000","last_fullsized":"2024-03-30T13:52:53.482900+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:00.432456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:00.432456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:05.621703+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":344,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.481555+0000","last_change":"2024-03-30T13:50:08.513281+0000","last_active":"2024-03-30T13:52:53.481555+0000","last_peered":"2024-03-30T13:52:53.481555+0000","last_clean":"2024-03-30T13:50:05.503977+0000","last_became_active":"2024-03-30T13:50:08.513281+0000","last_became_peered":"2024-03-30T13:50:08.513281+0000","last_unstale":"2024-03-30T13:52:53.481555+0000","last_undegraded":"2024-03-30T13:52:53.481555+0000","last_fullsized":"2024-03-30T13:50:08.504796+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:27.947416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:27.947416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":350,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.480143+0000","last_change":"2024-03-30T13:52:17.021134+0000","last_active":"2024-03-30T13:52:53.480143+0000","last_peered":"2024-03-30T13:52:53.480143+0000","last_clean":"2024-03-30T13:52:53.480143+0000","last_became_active":"2024-03-30T13:42:45.310165+0000","last_became_peered":"2024-03-30T13:42:45.310165+0000","last_unstale":"2024-03-30T13:52:53.480143+0000","last_undegraded":"2024-03-30T13:52:53.480143+0000","last_fullsized":"2024-03-30T13:52:53.480143+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:17.020887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:17.020887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:39.014261+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"217'19","reported_seq":391,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.553098+0000","last_change":"2024-03-30T13:52:26.096182+0000","last_active":"2024-03-30T13:52:53.553098+0000","last_peered":"2024-03-30T13:52:53.553098+0000","last_clean":"2024-03-30T13:52:53.553098+0000","last_became_active":"2024-03-30T13:42:47.324399+0000","last_became_peered":"2024-03-30T13:42:47.324399+0000","last_unstale":"2024-03-30T13:52:53.553098+0000","last_undegraded":"2024-03-30T13:52:53.553098+0000","last_fullsized":"2024-03-30T13:52:53.553098+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:52:26.096051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:26.096051+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:28.828248+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":331,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.480164+0000","last_change":"2024-03-30T13:50:08.512275+0000","last_active":"2024-03-30T13:52:53.480164+0000","last_peered":"2024-03-30T13:52:53.480164+0000","last_clean":"2024-03-30T13:50:05.495214+0000","last_became_active":"2024-03-30T13:50:08.512275+0000","last_became_peered":"2024-03-30T13:50:08.512275+0000","last_unstale":"2024-03-30T13:52:53.480164+0000","last_undegraded":"2024-03-30T13:52:53.480164+0000","last_fullsized":"2024-03-30T13:50:08.506150+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:23.264538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:23.264538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:55.950583+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":364,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.483163+0000","last_change":"2024-03-30T13:52:17.109093+0000","last_active":"2024-03-30T13:52:53.483163+0000","last_peered":"2024-03-30T13:52:53.483163+0000","last_clean":"2024-03-30T13:52:53.483163+0000","last_became_active":"2024-03-30T13:42:49.322799+0000","last_became_peered":"2024-03-30T13:42:49.322799+0000","last_unstale":"2024-03-30T13:52:53.483163+0000","last_undegraded":"2024-03-30T13:52:53.483163+0000","last_fullsized":"2024-03-30T13:52:53.483163+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:17.108986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:54.893653+0000","last_clean_scrub_stamp":"2024-03-30T13:52:17.108986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:23.056030+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":364,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.480207+0000","last_change":"2024-03-30T13:52:47.933155+0000","last_active":"2024-03-30T13:52:53.480207+0000","last_peered":"2024-03-30T13:52:53.480207+0000","last_clean":"2024-03-30T13:52:53.480207+0000","last_became_active":"2024-03-30T13:42:45.314302+0000","last_became_peered":"2024-03-30T13:42:45.314302+0000","last_unstale":"2024-03-30T13:52:53.480207+0000","last_undegraded":"2024-03-30T13:52:53.480207+0000","last_fullsized":"2024-03-30T13:52:53.480207+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:47.933046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:47.933046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:08.447668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"216'8","reported_seq":358,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.480236+0000","last_change":"2024-03-30T13:52:03.989310+0000","last_active":"2024-03-30T13:52:53.480236+0000","last_peered":"2024-03-30T13:52:53.480236+0000","last_clean":"2024-03-30T13:52:53.480236+0000","last_became_active":"2024-03-30T13:42:47.321428+0000","last_became_peered":"2024-03-30T13:42:47.321428+0000","last_unstale":"2024-03-30T13:52:53.480236+0000","last_undegraded":"2024-03-30T13:52:53.480236+0000","last_fullsized":"2024-03-30T13:52:53.480236+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-30T13:52:03.989075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:03.989075+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:07.727442+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":339,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.480904+0000","last_change":"2024-03-30T13:52:26.149264+0000","last_active":"2024-03-30T13:52:53.480904+0000","last_peered":"2024-03-30T13:52:53.480904+0000","last_clean":"2024-03-30T13:52:53.480904+0000","last_became_active":"2024-03-30T13:42:51.374304+0000","last_became_peered":"2024-03-30T13:42:51.374304+0000","last_unstale":"2024-03-30T13:52:53.480904+0000","last_undegraded":"2024-03-30T13:52:53.480904+0000","last_fullsized":"2024-03-30T13:52:53.480904+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:26.149132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:26.149132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:55.020422+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":348,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.480212+0000","last_change":"2024-03-30T13:51:20.297584+0000","last_active":"2024-03-30T13:52:53.480212+0000","last_peered":"2024-03-30T13:52:53.480212+0000","last_clean":"2024-03-30T13:52:53.480212+0000","last_became_active":"2024-03-30T13:42:49.324206+0000","last_became_peered":"2024-03-30T13:42:49.324206+0000","last_unstale":"2024-03-30T13:52:53.480212+0000","last_undegraded":"2024-03-30T13:52:53.480212+0000","last_fullsized":"2024-03-30T13:52:53.480212+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:20.297467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:42.616155+0000","last_clean_scrub_stamp":"2024-03-30T13:51:20.297467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":357,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.484204+0000","last_change":"2024-03-30T13:52:04.222697+0000","last_active":"2024-03-30T13:52:53.484204+0000","last_peered":"2024-03-30T13:52:53.484204+0000","last_clean":"2024-03-30T13:52:53.484204+0000","last_became_active":"2024-03-30T13:42:45.319482+0000","last_became_peered":"2024-03-30T13:42:45.319482+0000","last_unstale":"2024-03-30T13:52:53.484204+0000","last_undegraded":"2024-03-30T13:52:53.484204+0000","last_fullsized":"2024-03-30T13:52:53.484204+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:04.222591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:25.494903+0000","last_clean_scrub_stamp":"2024-03-30T13:52:04.222591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:24.498670+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"217'19","reported_seq":377,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.544275+0000","last_change":"2024-03-30T13:51:21.863041+0000","last_active":"2024-03-30T13:52:53.544275+0000","last_peered":"2024-03-30T13:52:53.544275+0000","last_clean":"2024-03-30T13:52:53.544275+0000","last_became_active":"2024-03-30T13:42:47.500712+0000","last_became_peered":"2024-03-30T13:42:47.500712+0000","last_unstale":"2024-03-30T13:52:53.544275+0000","last_undegraded":"2024-03-30T13:52:53.544275+0000","last_fullsized":"2024-03-30T13:52:53.544275+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:51:21.862885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:21.862885+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":344,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.478980+0000","last_change":"2024-03-30T13:52:03.006976+0000","last_active":"2024-03-30T13:52:53.478980+0000","last_peered":"2024-03-30T13:52:53.478980+0000","last_clean":"2024-03-30T13:52:53.478980+0000","last_became_active":"2024-03-30T13:42:51.333603+0000","last_became_peered":"2024-03-30T13:42:51.333603+0000","last_unstale":"2024-03-30T13:52:53.478980+0000","last_undegraded":"2024-03-30T13:52:53.478980+0000","last_fullsized":"2024-03-30T13:52:53.478980+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:03.006855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:28.314391+0000","last_clean_scrub_stamp":"2024-03-30T13:52:03.006855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:25.254259+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":342,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.477983+0000","last_change":"2024-03-30T13:52:16.009739+0000","last_active":"2024-03-30T13:52:53.477983+0000","last_peered":"2024-03-30T13:52:53.477983+0000","last_clean":"2024-03-30T13:52:53.477983+0000","last_became_active":"2024-03-30T13:42:49.325219+0000","last_became_peered":"2024-03-30T13:42:49.325219+0000","last_unstale":"2024-03-30T13:52:53.477983+0000","last_undegraded":"2024-03-30T13:52:53.477983+0000","last_fullsized":"2024-03-30T13:52:53.477983+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:16.009499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:16.009499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:44.528058+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":356,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.481935+0000","last_change":"2024-03-30T13:52:11.701415+0000","last_active":"2024-03-30T13:52:53.481935+0000","last_peered":"2024-03-30T13:52:53.481935+0000","last_clean":"2024-03-30T13:52:53.481935+0000","last_became_active":"2024-03-30T13:42:45.311071+0000","last_became_peered":"2024-03-30T13:42:45.311071+0000","last_unstale":"2024-03-30T13:52:53.481935+0000","last_undegraded":"2024-03-30T13:52:53.481935+0000","last_fullsized":"2024-03-30T13:52:53.481935+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:11.701294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:11.701294+0000","last_clean_scrub_stamp":"2024-03-30T13:52:11.701294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:25.127453+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"216'15","reported_seq":374,"reported_epoch":217,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:53.481968+0000","last_change":"2024-03-30T13:50:08.513156+0000","last_active":"2024-03-30T13:52:53.481968+0000","last_peered":"2024-03-30T13:52:53.481968+0000","last_clean":"2024-03-30T13:50:05.496678+0000","last_became_active":"2024-03-30T13:50:08.513156+0000","last_became_peered":"2024-03-30T13:50:08.513156+0000","last_unstale":"2024-03-30T13:52:53.481968+0000","last_undegraded":"2024-03-30T13:50:08.507726+0000","last_fullsized":"2024-03-30T13:50:08.507616+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:49:26.480540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:49:26.480540+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:57.611645+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":338,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.478229+0000","last_change":"2024-03-30T13:52:03.011993+0000","last_active":"2024-03-30T13:52:53.478229+0000","last_peered":"2024-03-30T13:52:53.478229+0000","last_clean":"2024-03-30T13:52:53.478229+0000","last_became_active":"2024-03-30T13:42:51.334597+0000","last_became_peered":"2024-03-30T13:42:51.334597+0000","last_unstale":"2024-03-30T13:52:53.478229+0000","last_undegraded":"2024-03-30T13:52:53.478229+0000","last_fullsized":"2024-03-30T13:52:53.478229+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:03.011875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:24.598976+0000","last_clean_scrub_stamp":"2024-03-30T13:52:03.011875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:18.152197+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":358,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.481830+0000","last_change":"2024-03-30T13:52:46.670567+0000","last_active":"2024-03-30T13:52:53.481830+0000","last_peered":"2024-03-30T13:52:53.481830+0000","last_clean":"2024-03-30T13:52:53.481830+0000","last_became_active":"2024-03-30T13:42:49.322843+0000","last_became_peered":"2024-03-30T13:42:49.322843+0000","last_unstale":"2024-03-30T13:52:53.481830+0000","last_undegraded":"2024-03-30T13:52:53.481830+0000","last_fullsized":"2024-03-30T13:52:53.481830+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:46.670446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:34.815783+0000","last_clean_scrub_stamp":"2024-03-30T13:52:46.670446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:06.853571+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":359,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.476820+0000","last_change":"2024-03-30T13:52:14.015548+0000","last_active":"2024-03-30T13:52:53.476820+0000","last_peered":"2024-03-30T13:52:53.476820+0000","last_clean":"2024-03-30T13:52:53.476820+0000","last_became_active":"2024-03-30T13:42:45.315864+0000","last_became_peered":"2024-03-30T13:42:45.315864+0000","last_unstale":"2024-03-30T13:52:53.476820+0000","last_undegraded":"2024-03-30T13:52:53.476820+0000","last_fullsized":"2024-03-30T13:52:53.476820+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:14.015356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:14.015356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:42.047492+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"216'19","reported_seq":391,"reported_epoch":217,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:53.478307+0000","last_change":"2024-03-30T13:50:08.513382+0000","last_active":"2024-03-30T13:52:53.478307+0000","last_peered":"2024-03-30T13:52:53.478307+0000","last_clean":"2024-03-30T13:50:05.491029+0000","last_became_active":"2024-03-30T13:50:08.513382+0000","last_became_peered":"2024-03-30T13:50:08.513382+0000","last_unstale":"2024-03-30T13:52:53.478307+0000","last_undegraded":"2024-03-30T13:50:08.503759+0000","last_fullsized":"2024-03-30T13:50:08.503643+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:48:22.068206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:22.068206+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":344,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.479676+0000","last_change":"2024-03-30T13:51:25.374768+0000","last_active":"2024-03-30T13:52:53.479676+0000","last_peered":"2024-03-30T13:52:53.479676+0000","last_clean":"2024-03-30T13:52:53.479676+0000","last_became_active":"2024-03-30T13:42:51.336162+0000","last_became_peered":"2024-03-30T13:42:51.336162+0000","last_unstale":"2024-03-30T13:52:53.479676+0000","last_undegraded":"2024-03-30T13:52:53.479676+0000","last_fullsized":"2024-03-30T13:52:53.479676+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:25.374633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:25.374633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":479,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.483503+0000","last_change":"2024-03-30T13:52:16.103854+0000","last_active":"2024-03-30T13:52:53.483503+0000","last_peered":"2024-03-30T13:52:53.483503+0000","last_clean":"2024-03-30T13:52:53.483503+0000","last_became_active":"2024-03-30T13:42:49.321203+0000","last_became_peered":"2024-03-30T13:42:49.321203+0000","last_unstale":"2024-03-30T13:52:53.483503+0000","last_undegraded":"2024-03-30T13:52:53.483503+0000","last_fullsized":"2024-03-30T13:52:53.483503+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:52:16.103727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:16.103727+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:25.804953+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":343,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.480381+0000","last_change":"2024-03-30T13:50:08.511650+0000","last_active":"2024-03-30T13:52:53.480381+0000","last_peered":"2024-03-30T13:52:53.480381+0000","last_clean":"2024-03-30T13:50:05.495125+0000","last_became_active":"2024-03-30T13:50:08.511650+0000","last_became_peered":"2024-03-30T13:50:08.511650+0000","last_unstale":"2024-03-30T13:52:53.480381+0000","last_undegraded":"2024-03-30T13:52:53.480381+0000","last_fullsized":"2024-03-30T13:50:08.506728+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:07.618186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:49:07.618186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:55.950609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"217'20","reported_seq":381,"reported_epoch":217,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:53.481832+0000","last_change":"2024-03-30T13:50:08.514919+0000","last_active":"2024-03-30T13:52:53.481832+0000","last_peered":"2024-03-30T13:52:53.481832+0000","last_clean":"2024-03-30T13:50:05.508102+0000","last_became_active":"2024-03-30T13:50:08.514919+0000","last_became_peered":"2024-03-30T13:50:08.514919+0000","last_unstale":"2024-03-30T13:52:53.481832+0000","last_undegraded":"2024-03-30T13:50:08.503113+0000","last_fullsized":"2024-03-30T13:50:08.502993+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-30T13:48:21.678227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:21.678227+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:55.942929+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":334,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.478497+0000","last_change":"2024-03-30T13:50:08.514514+0000","last_active":"2024-03-30T13:52:53.478497+0000","last_peered":"2024-03-30T13:52:53.478497+0000","last_clean":"2024-03-30T13:50:05.490356+0000","last_became_active":"2024-03-30T13:50:08.514514+0000","last_became_peered":"2024-03-30T13:50:08.514514+0000","last_unstale":"2024-03-30T13:52:53.478497+0000","last_undegraded":"2024-03-30T13:52:53.478497+0000","last_fullsized":"2024-03-30T13:50:08.502291+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:30.238582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:30.238582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"168'9","reported_seq":468,"reported_epoch":217,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:53.478821+0000","last_change":"2024-03-30T13:50:08.515013+0000","last_active":"2024-03-30T13:52:53.478821+0000","last_peered":"2024-03-30T13:52:53.478821+0000","last_clean":"2024-03-30T13:50:05.508192+0000","last_became_active":"2024-03-30T13:50:08.515013+0000","last_became_peered":"2024-03-30T13:50:08.515013+0000","last_unstale":"2024-03-30T13:52:53.478821+0000","last_undegraded":"2024-03-30T13:50:08.503461+0000","last_fullsized":"2024-03-30T13:50:08.503337+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:49:23.573435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:23.573435+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:55.942938+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":356,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.481135+0000","last_change":"2024-03-30T13:52:09.007605+0000","last_active":"2024-03-30T13:52:53.481135+0000","last_peered":"2024-03-30T13:52:53.481135+0000","last_clean":"2024-03-30T13:52:53.481135+0000","last_became_active":"2024-03-30T13:42:45.314585+0000","last_became_peered":"2024-03-30T13:42:45.314585+0000","last_unstale":"2024-03-30T13:52:53.481135+0000","last_undegraded":"2024-03-30T13:52:53.481135+0000","last_fullsized":"2024-03-30T13:52:53.481135+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:09.007490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:28.734080+0000","last_clean_scrub_stamp":"2024-03-30T13:52:09.007490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:32.345744+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"217'15","reported_seq":378,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.491079+0000","last_change":"2024-03-30T13:52:03.227854+0000","last_active":"2024-03-30T13:52:53.491079+0000","last_peered":"2024-03-30T13:52:53.491079+0000","last_clean":"2024-03-30T13:52:53.491079+0000","last_became_active":"2024-03-30T13:42:47.501559+0000","last_became_peered":"2024-03-30T13:42:47.501559+0000","last_unstale":"2024-03-30T13:52:53.491079+0000","last_undegraded":"2024-03-30T13:52:53.491079+0000","last_fullsized":"2024-03-30T13:52:53.491079+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:52:03.227628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:03.227628+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:28.935372+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":122,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.477245+0000","last_change":"2024-03-30T13:50:08.513016+0000","last_active":"2024-03-30T13:52:53.477245+0000","last_peered":"2024-03-30T13:52:53.477245+0000","last_clean":"2024-03-30T13:50:05.491001+0000","last_became_active":"2024-03-30T13:50:08.513016+0000","last_became_peered":"2024-03-30T13:50:08.513016+0000","last_unstale":"2024-03-30T13:52:53.477245+0000","last_undegraded":"2024-03-30T13:52:53.477245+0000","last_fullsized":"2024-03-30T13:50:08.502775+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:14.322254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:14.322254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":342,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.481854+0000","last_change":"2024-03-30T13:52:22.106904+0000","last_active":"2024-03-30T13:52:53.481854+0000","last_peered":"2024-03-30T13:52:53.481854+0000","last_clean":"2024-03-30T13:52:53.481854+0000","last_became_active":"2024-03-30T13:42:49.323222+0000","last_became_peered":"2024-03-30T13:42:49.323222+0000","last_unstale":"2024-03-30T13:52:53.481854+0000","last_undegraded":"2024-03-30T13:52:53.481854+0000","last_fullsized":"2024-03-30T13:52:53.481854+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:22.106782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:22.106782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:38.428203+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":356,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.480800+0000","last_change":"2024-03-30T13:51:18.239125+0000","last_active":"2024-03-30T13:52:53.480800+0000","last_peered":"2024-03-30T13:52:53.480800+0000","last_clean":"2024-03-30T13:52:53.480800+0000","last_became_active":"2024-03-30T13:42:45.313617+0000","last_became_peered":"2024-03-30T13:42:45.313617+0000","last_unstale":"2024-03-30T13:52:53.480800+0000","last_undegraded":"2024-03-30T13:52:53.480800+0000","last_fullsized":"2024-03-30T13:52:53.480800+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:18.239047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:18.239047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"216'15","reported_seq":377,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.480537+0000","last_change":"2024-03-30T13:51:20.179768+0000","last_active":"2024-03-30T13:52:53.480537+0000","last_peered":"2024-03-30T13:52:53.480537+0000","last_clean":"2024-03-30T13:52:53.480537+0000","last_became_active":"2024-03-30T13:42:47.501925+0000","last_became_peered":"2024-03-30T13:42:47.501925+0000","last_unstale":"2024-03-30T13:52:53.480537+0000","last_undegraded":"2024-03-30T13:52:53.480537+0000","last_fullsized":"2024-03-30T13:52:53.480537+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:51:20.179623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:20.179623+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"72'1","reported_seq":353,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.481835+0000","last_change":"2024-03-30T13:52:27.892001+0000","last_active":"2024-03-30T13:52:53.481835+0000","last_peered":"2024-03-30T13:52:53.481835+0000","last_clean":"2024-03-30T13:52:53.481835+0000","last_became_active":"2024-03-30T13:42:51.372023+0000","last_became_peered":"2024-03-30T13:42:51.372023+0000","last_unstale":"2024-03-30T13:52:53.481835+0000","last_undegraded":"2024-03-30T13:52:53.481835+0000","last_fullsized":"2024-03-30T13:52:53.481835+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T13:52:27.891771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:27.891771+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:40.661790+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":349,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.484373+0000","last_change":"2024-03-30T13:52:09.123923+0000","last_active":"2024-03-30T13:52:53.484373+0000","last_peered":"2024-03-30T13:52:53.484373+0000","last_clean":"2024-03-30T13:52:53.484373+0000","last_became_active":"2024-03-30T13:42:49.323924+0000","last_became_peered":"2024-03-30T13:42:49.323924+0000","last_unstale":"2024-03-30T13:52:53.484373+0000","last_undegraded":"2024-03-30T13:52:53.484373+0000","last_fullsized":"2024-03-30T13:52:53.484373+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:09.123811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:09.123811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:12.531734+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":372,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.482708+0000","last_change":"2024-03-30T13:52:45.998803+0000","last_active":"2024-03-30T13:52:53.482708+0000","last_peered":"2024-03-30T13:52:53.482708+0000","last_clean":"2024-03-30T13:52:53.482708+0000","last_became_active":"2024-03-30T13:42:45.318956+0000","last_became_peered":"2024-03-30T13:42:45.318956+0000","last_unstale":"2024-03-30T13:52:53.482708+0000","last_undegraded":"2024-03-30T13:52:53.482708+0000","last_fullsized":"2024-03-30T13:52:53.482708+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:45.998724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:51:20.175438+0000","last_clean_scrub_stamp":"2024-03-30T13:52:45.998724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:54.534778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"216'12","reported_seq":364,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.481757+0000","last_change":"2024-03-30T13:52:14.027664+0000","last_active":"2024-03-30T13:52:53.481757+0000","last_peered":"2024-03-30T13:52:53.481757+0000","last_clean":"2024-03-30T13:52:53.481757+0000","last_became_active":"2024-03-30T13:42:47.501426+0000","last_became_peered":"2024-03-30T13:42:47.501426+0000","last_unstale":"2024-03-30T13:52:53.481757+0000","last_undegraded":"2024-03-30T13:52:53.481757+0000","last_fullsized":"2024-03-30T13:52:53.481757+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-03-30T13:52:14.027491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:14.027491+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:24.045049+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"217'626","reported_seq":647,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:54.446940+0000","last_change":"2024-03-30T13:52:43.533497+0000","last_active":"2024-03-30T13:52:54.446940+0000","last_peered":"2024-03-30T13:52:54.446940+0000","last_clean":"2024-03-30T13:52:54.446940+0000","last_became_active":"2024-03-30T13:52:43.533198+0000","last_became_peered":"2024-03-30T13:52:43.533198+0000","last_unstale":"2024-03-30T13:52:54.446940+0000","last_undegraded":"2024-03-30T13:52:54.446940+0000","last_fullsized":"2024-03-30T13:52:54.446940+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:01.980653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41025536,"num_objects":626,"num_object_clones":0,"num_object_copies":1878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":626,"num_write_kb":40064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":337,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.482201+0000","last_change":"2024-03-30T13:51:05.957160+0000","last_active":"2024-03-30T13:52:53.482201+0000","last_peered":"2024-03-30T13:52:53.482201+0000","last_clean":"2024-03-30T13:52:53.482201+0000","last_became_active":"2024-03-30T13:42:51.371360+0000","last_became_peered":"2024-03-30T13:42:51.371360+0000","last_unstale":"2024-03-30T13:52:53.482201+0000","last_undegraded":"2024-03-30T13:52:53.482201+0000","last_fullsized":"2024-03-30T13:52:53.482201+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:05.957043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:05.957043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":368,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.480795+0000","last_change":"2024-03-30T13:52:09.680728+0000","last_active":"2024-03-30T13:52:53.480795+0000","last_peered":"2024-03-30T13:52:53.480795+0000","last_clean":"2024-03-30T13:52:53.480795+0000","last_became_active":"2024-03-30T13:42:45.318963+0000","last_became_peered":"2024-03-30T13:42:45.318963+0000","last_unstale":"2024-03-30T13:52:53.480795+0000","last_undegraded":"2024-03-30T13:52:53.480795+0000","last_fullsized":"2024-03-30T13:52:53.480795+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-30T13:52:09.680567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:09.680567+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:15.944959+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"216'27","reported_seq":396,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.481731+0000","last_change":"2024-03-30T13:51:27.133579+0000","last_active":"2024-03-30T13:52:53.481731+0000","last_peered":"2024-03-30T13:52:53.481731+0000","last_clean":"2024-03-30T13:52:53.481731+0000","last_became_active":"2024-03-30T13:42:47.501937+0000","last_became_peered":"2024-03-30T13:42:47.501937+0000","last_unstale":"2024-03-30T13:52:53.481731+0000","last_undegraded":"2024-03-30T13:52:53.481731+0000","last_fullsized":"2024-03-30T13:52:53.481731+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-30T13:51:27.133454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:27.133454+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"72'8","reported_seq":478,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.481026+0000","last_change":"2024-03-30T13:51:19.286719+0000","last_active":"2024-03-30T13:52:53.481026+0000","last_peered":"2024-03-30T13:52:53.481026+0000","last_clean":"2024-03-30T13:52:53.481026+0000","last_became_active":"2024-03-30T13:42:49.322675+0000","last_became_peered":"2024-03-30T13:42:49.322675+0000","last_unstale":"2024-03-30T13:52:53.481026+0000","last_undegraded":"2024-03-30T13:52:53.481026+0000","last_fullsized":"2024-03-30T13:52:53.481026+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:51:19.286592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:19.286592+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"217'481","reported_seq":503,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.986924+0000","last_change":"2024-03-30T13:52:43.529140+0000","last_active":"2024-03-30T13:52:53.986924+0000","last_peered":"2024-03-30T13:52:53.986924+0000","last_clean":"2024-03-30T13:52:53.986924+0000","last_became_active":"2024-03-30T13:52:43.528506+0000","last_became_peered":"2024-03-30T13:52:43.528506+0000","last_unstale":"2024-03-30T13:52:53.986924+0000","last_undegraded":"2024-03-30T13:52:53.986924+0000","last_fullsized":"2024-03-30T13:52:53.986924+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:58.679998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31522816,"num_objects":481,"num_object_clones":0,"num_object_copies":1443,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":481,"num_write_kb":30784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":338,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.478143+0000","last_change":"2024-03-30T13:52:28.078697+0000","last_active":"2024-03-30T13:52:53.478143+0000","last_peered":"2024-03-30T13:52:53.478143+0000","last_clean":"2024-03-30T13:52:53.478143+0000","last_became_active":"2024-03-30T13:42:51.334672+0000","last_became_peered":"2024-03-30T13:42:51.334672+0000","last_unstale":"2024-03-30T13:52:53.478143+0000","last_undegraded":"2024-03-30T13:52:53.478143+0000","last_fullsized":"2024-03-30T13:52:53.478143+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:28.078616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:28.078616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:36.312112+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":356,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.480902+0000","last_change":"2024-03-30T13:51:15.247447+0000","last_active":"2024-03-30T13:52:53.480902+0000","last_peered":"2024-03-30T13:52:53.480902+0000","last_clean":"2024-03-30T13:52:53.480902+0000","last_became_active":"2024-03-30T13:42:45.312328+0000","last_became_peered":"2024-03-30T13:42:45.312328+0000","last_unstale":"2024-03-30T13:52:53.480902+0000","last_undegraded":"2024-03-30T13:52:53.480902+0000","last_fullsized":"2024-03-30T13:52:53.480902+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:15.246997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:15.246997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"216'18","reported_seq":387,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.476062+0000","last_change":"2024-03-30T13:52:11.994083+0000","last_active":"2024-03-30T13:52:53.476062+0000","last_peered":"2024-03-30T13:52:53.476062+0000","last_clean":"2024-03-30T13:52:53.476062+0000","last_became_active":"2024-03-30T13:42:47.316187+0000","last_became_peered":"2024-03-30T13:42:47.316187+0000","last_unstale":"2024-03-30T13:52:53.476062+0000","last_undegraded":"2024-03-30T13:52:53.476062+0000","last_fullsized":"2024-03-30T13:52:53.476062+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-30T13:52:11.993946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:11.993946+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:23.918405+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":122,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.479494+0000","last_change":"2024-03-30T13:50:08.509701+0000","last_active":"2024-03-30T13:52:53.479494+0000","last_peered":"2024-03-30T13:52:53.479494+0000","last_clean":"2024-03-30T13:50:05.499237+0000","last_became_active":"2024-03-30T13:50:08.509701+0000","last_became_peered":"2024-03-30T13:50:08.509701+0000","last_unstale":"2024-03-30T13:52:53.479494+0000","last_undegraded":"2024-03-30T13:52:53.479494+0000","last_fullsized":"2024-03-30T13:50:08.501671+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:22.401087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:22.401087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"217'448","reported_seq":470,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.568666+0000","last_change":"2024-03-30T13:52:43.533623+0000","last_active":"2024-03-30T13:52:53.568666+0000","last_peered":"2024-03-30T13:52:53.568666+0000","last_clean":"2024-03-30T13:52:53.568666+0000","last_became_active":"2024-03-30T13:52:43.533072+0000","last_became_peered":"2024-03-30T13:52:43.533072+0000","last_unstale":"2024-03-30T13:52:53.568666+0000","last_undegraded":"2024-03-30T13:52:53.568666+0000","last_fullsized":"2024-03-30T13:52:53.568666+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:44.194020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":448,"num_object_clones":0,"num_object_copies":1344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":448,"num_write_kb":28672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":339,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.479846+0000","last_change":"2024-03-30T13:50:08.510439+0000","last_active":"2024-03-30T13:52:53.479846+0000","last_peered":"2024-03-30T13:52:53.479846+0000","last_clean":"2024-03-30T13:50:05.494808+0000","last_became_active":"2024-03-30T13:50:08.510439+0000","last_became_peered":"2024-03-30T13:50:08.510439+0000","last_unstale":"2024-03-30T13:52:53.479846+0000","last_undegraded":"2024-03-30T13:52:53.479846+0000","last_fullsized":"2024-03-30T13:50:08.506960+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:31.860675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:31.860675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:55.950622+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":122,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.480338+0000","last_change":"2024-03-30T13:50:08.509003+0000","last_active":"2024-03-30T13:52:53.480338+0000","last_peered":"2024-03-30T13:52:53.480338+0000","last_clean":"2024-03-30T13:50:05.499015+0000","last_became_active":"2024-03-30T13:50:08.509003+0000","last_became_peered":"2024-03-30T13:50:08.509003+0000","last_unstale":"2024-03-30T13:52:53.480338+0000","last_undegraded":"2024-03-30T13:52:53.480338+0000","last_fullsized":"2024-03-30T13:50:08.502979+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:19.432242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:19.432242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":348,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.482279+0000","last_change":"2024-03-30T13:51:09.173688+0000","last_active":"2024-03-30T13:52:53.482279+0000","last_peered":"2024-03-30T13:52:53.482279+0000","last_clean":"2024-03-30T13:52:53.482279+0000","last_became_active":"2024-03-30T13:42:45.315687+0000","last_became_peered":"2024-03-30T13:42:45.315687+0000","last_unstale":"2024-03-30T13:52:53.482279+0000","last_undegraded":"2024-03-30T13:52:53.482279+0000","last_fullsized":"2024-03-30T13:52:53.482279+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:09.173459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:09.173459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"217'44","reported_seq":424,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.592722+0000","last_change":"2024-03-30T13:52:06.210819+0000","last_active":"2024-03-30T13:52:53.592722+0000","last_peered":"2024-03-30T13:52:53.592722+0000","last_clean":"2024-03-30T13:52:53.592722+0000","last_became_active":"2024-03-30T13:42:47.500898+0000","last_became_peered":"2024-03-30T13:42:47.500898+0000","last_unstale":"2024-03-30T13:52:53.592722+0000","last_undegraded":"2024-03-30T13:52:53.592722+0000","last_fullsized":"2024-03-30T13:52:53.592722+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'28","last_scrub_stamp":"2024-03-30T13:52:06.210663+0000","last_deep_scrub":"108'28","last_deep_scrub_stamp":"2024-03-30T13:45:49.140418+0000","last_clean_scrub_stamp":"2024-03-30T13:52:06.210663+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:20.718042+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"217'381","reported_seq":400,"reported_epoch":217,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:53.565372+0000","last_change":"2024-03-30T13:52:51.700195+0000","last_active":"2024-03-30T13:52:53.565372+0000","last_peered":"2024-03-30T13:52:53.565372+0000","last_clean":"2024-03-30T13:52:42.503419+0000","last_became_active":"2024-03-30T13:52:43.531451+0000","last_became_peered":"2024-03-30T13:52:43.531451+0000","last_unstale":"2024-03-30T13:52:53.565372+0000","last_undegraded":"2024-03-30T13:52:51.691582+0000","last_fullsized":"2024-03-30T13:52:43.527839+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:09.802561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24969216,"num_objects":381,"num_object_clones":0,"num_object_copies":1143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":381,"num_write_kb":24384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":381}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":331,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.480499+0000","last_change":"2024-03-30T13:50:08.509476+0000","last_active":"2024-03-30T13:52:53.480499+0000","last_peered":"2024-03-30T13:52:53.480499+0000","last_clean":"2024-03-30T13:50:05.499348+0000","last_became_active":"2024-03-30T13:50:08.509476+0000","last_became_peered":"2024-03-30T13:50:08.509476+0000","last_unstale":"2024-03-30T13:52:53.480499+0000","last_undegraded":"2024-03-30T13:52:53.480499+0000","last_fullsized":"2024-03-30T13:50:08.503098+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:12.807605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:44.095338+0000","last_clean_scrub_stamp":"2024-03-30T13:49:12.807605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":122,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.477732+0000","last_change":"2024-03-30T13:50:08.507875+0000","last_active":"2024-03-30T13:52:53.477732+0000","last_peered":"2024-03-30T13:52:53.477732+0000","last_clean":"2024-03-30T13:50:05.490245+0000","last_became_active":"2024-03-30T13:50:08.507875+0000","last_became_peered":"2024-03-30T13:50:08.507875+0000","last_unstale":"2024-03-30T13:52:53.477732+0000","last_undegraded":"2024-03-30T13:52:53.477732+0000","last_fullsized":"2024-03-30T13:50:08.502700+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:13.399683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:28.225742+0000","last_clean_scrub_stamp":"2024-03-30T13:49:13.399683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":357,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.483739+0000","last_change":"2024-03-30T13:51:19.099448+0000","last_active":"2024-03-30T13:52:53.483739+0000","last_peered":"2024-03-30T13:52:53.483739+0000","last_clean":"2024-03-30T13:52:53.483739+0000","last_became_active":"2024-03-30T13:42:45.314767+0000","last_became_peered":"2024-03-30T13:42:45.314767+0000","last_unstale":"2024-03-30T13:52:53.483739+0000","last_undegraded":"2024-03-30T13:52:53.483739+0000","last_fullsized":"2024-03-30T13:52:53.483739+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:19.099205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:19.099205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"216'15","reported_seq":401,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.480612+0000","last_change":"2024-03-30T13:52:17.023987+0000","last_active":"2024-03-30T13:52:53.480612+0000","last_peered":"2024-03-30T13:52:53.480612+0000","last_clean":"2024-03-30T13:52:53.480612+0000","last_became_active":"2024-03-30T13:43:11.446379+0000","last_became_peered":"2024-03-30T13:43:11.446379+0000","last_unstale":"2024-03-30T13:52:53.480612+0000","last_undegraded":"2024-03-30T13:52:53.480612+0000","last_fullsized":"2024-03-30T13:52:53.480612+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:52:17.023838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:17.023838+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:39.125999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"217'614","reported_seq":635,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:54.457348+0000","last_change":"2024-03-30T13:52:43.532691+0000","last_active":"2024-03-30T13:52:54.457348+0000","last_peered":"2024-03-30T13:52:54.457348+0000","last_clean":"2024-03-30T13:52:54.457348+0000","last_became_active":"2024-03-30T13:52:43.532257+0000","last_became_peered":"2024-03-30T13:52:43.532257+0000","last_unstale":"2024-03-30T13:52:54.457348+0000","last_undegraded":"2024-03-30T13:52:54.457348+0000","last_fullsized":"2024-03-30T13:52:54.457348+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":614,"log_dups_size":0,"ondisk_log_size":614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:08.937201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40239104,"num_objects":614,"num_object_clones":0,"num_object_copies":1842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":614,"num_write_kb":39296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":337,"reported_epoch":217,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:52:53.481764+0000","last_change":"2024-03-30T13:52:48.895015+0000","last_active":"2024-03-30T13:52:53.481764+0000","last_peered":"2024-03-30T13:52:53.481764+0000","last_clean":"2024-03-30T13:52:53.481764+0000","last_became_active":"2024-03-30T13:52:36.491153+0000","last_became_peered":"2024-03-30T13:52:36.491153+0000","last_unstale":"2024-03-30T13:52:53.481764+0000","last_undegraded":"2024-03-30T13:52:53.481764+0000","last_fullsized":"2024-03-30T13:52:53.481764+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:48.894763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:48.894763+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:51.949336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0,6],"avail_no_missing":["5","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":338,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.480578+0000","last_change":"2024-03-30T13:52:08.043894+0000","last_active":"2024-03-30T13:52:53.480578+0000","last_peered":"2024-03-30T13:52:53.480578+0000","last_clean":"2024-03-30T13:52:53.480578+0000","last_became_active":"2024-03-30T13:42:51.331054+0000","last_became_peered":"2024-03-30T13:42:51.331054+0000","last_unstale":"2024-03-30T13:52:53.480578+0000","last_undegraded":"2024-03-30T13:52:53.480578+0000","last_fullsized":"2024-03-30T13:52:53.480578+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:08.043774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:08.043774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:37.608884+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":376,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.482047+0000","last_change":"2024-03-30T13:52:51.932892+0000","last_active":"2024-03-30T13:52:53.482047+0000","last_peered":"2024-03-30T13:52:53.482047+0000","last_clean":"2024-03-30T13:52:53.482047+0000","last_became_active":"2024-03-30T13:42:44.300789+0000","last_became_peered":"2024-03-30T13:42:44.300789+0000","last_unstale":"2024-03-30T13:52:53.482047+0000","last_undegraded":"2024-03-30T13:52:53.482047+0000","last_fullsized":"2024-03-30T13:52:53.482047+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-30T13:52:51.932750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:52:51.932750+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:00.199754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021837800000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":351,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.477951+0000","last_change":"2024-03-30T13:51:01.240545+0000","last_active":"2024-03-30T13:52:53.477951+0000","last_peered":"2024-03-30T13:52:53.477951+0000","last_clean":"2024-03-30T13:52:53.477951+0000","last_became_active":"2024-03-30T13:42:45.310209+0000","last_became_peered":"2024-03-30T13:42:45.310209+0000","last_unstale":"2024-03-30T13:52:53.477951+0000","last_undegraded":"2024-03-30T13:52:53.477951+0000","last_fullsized":"2024-03-30T13:52:53.477951+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:01.240468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:01.240468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"217'41","reported_seq":530,"reported_epoch":217,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:53.508879+0000","last_change":"2024-03-30T13:50:08.508727+0000","last_active":"2024-03-30T13:52:53.508879+0000","last_peered":"2024-03-30T13:52:53.508879+0000","last_clean":"2024-03-30T13:50:05.490389+0000","last_became_active":"2024-03-30T13:50:08.508727+0000","last_became_peered":"2024-03-30T13:50:08.508727+0000","last_unstale":"2024-03-30T13:52:53.508879+0000","last_undegraded":"2024-03-30T13:50:08.506113+0000","last_fullsized":"2024-03-30T13:50:08.505993+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-03-30T13:48:27.377242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:27.377242+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:57.611567+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":356,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.479729+0000","last_change":"2024-03-30T13:52:23.006053+0000","last_active":"2024-03-30T13:52:53.479729+0000","last_peered":"2024-03-30T13:52:53.479729+0000","last_clean":"2024-03-30T13:52:53.479729+0000","last_became_active":"2024-03-30T13:42:49.322154+0000","last_became_peered":"2024-03-30T13:42:49.322154+0000","last_unstale":"2024-03-30T13:52:53.479729+0000","last_undegraded":"2024-03-30T13:52:53.479729+0000","last_fullsized":"2024-03-30T13:52:53.479729+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:23.005788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:18.862784+0000","last_clean_scrub_stamp":"2024-03-30T13:52:23.005788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:32.489896+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"217'415","reported_seq":436,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.575211+0000","last_change":"2024-03-30T13:52:43.535018+0000","last_active":"2024-03-30T13:52:53.575211+0000","last_peered":"2024-03-30T13:52:53.575211+0000","last_clean":"2024-03-30T13:52:53.575211+0000","last_became_active":"2024-03-30T13:52:43.534662+0000","last_became_peered":"2024-03-30T13:52:43.534662+0000","last_unstale":"2024-03-30T13:52:53.575211+0000","last_undegraded":"2024-03-30T13:52:53.575211+0000","last_fullsized":"2024-03-30T13:52:53.575211+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:01.323985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27197440,"num_objects":415,"num_object_clones":0,"num_object_copies":1245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":415,"num_write_kb":26560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":634,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.480358+0000","last_change":"2024-03-30T13:52:46.926489+0000","last_active":"2024-03-30T13:52:53.480358+0000","last_peered":"2024-03-30T13:52:53.480358+0000","last_clean":"2024-03-30T13:52:53.480358+0000","last_became_active":"2024-03-30T13:52:36.483883+0000","last_became_peered":"2024-03-30T13:52:36.483883+0000","last_unstale":"2024-03-30T13:52:53.480358+0000","last_undegraded":"2024-03-30T13:52:53.480358+0000","last_fullsized":"2024-03-30T13:52:53.480358+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:46.926407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:46.926407+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:57.492343+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":336,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.482421+0000","last_change":"2024-03-30T13:51:14.276390+0000","last_active":"2024-03-30T13:52:53.482421+0000","last_peered":"2024-03-30T13:52:53.482421+0000","last_clean":"2024-03-30T13:52:53.482421+0000","last_became_active":"2024-03-30T13:42:51.372320+0000","last_became_peered":"2024-03-30T13:42:51.372320+0000","last_unstale":"2024-03-30T13:52:53.482421+0000","last_undegraded":"2024-03-30T13:52:53.482421+0000","last_fullsized":"2024-03-30T13:52:53.482421+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:14.275893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:14.275893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":354,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.483490+0000","last_change":"2024-03-30T13:50:08.512561+0000","last_active":"2024-03-30T13:52:53.483490+0000","last_peered":"2024-03-30T13:52:53.483490+0000","last_clean":"2024-03-30T13:50:05.489457+0000","last_became_active":"2024-03-30T13:50:08.512561+0000","last_became_peered":"2024-03-30T13:50:08.512561+0000","last_unstale":"2024-03-30T13:52:53.483490+0000","last_undegraded":"2024-03-30T13:52:53.483490+0000","last_fullsized":"2024-03-30T13:50:08.506886+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:16.464234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:59.810189+0000","last_clean_scrub_stamp":"2024-03-30T13:49:16.464234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:57.611598+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":122,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.478406+0000","last_change":"2024-03-30T13:50:08.511337+0000","last_active":"2024-03-30T13:52:53.478406+0000","last_peered":"2024-03-30T13:52:53.478406+0000","last_clean":"2024-03-30T13:50:05.498892+0000","last_became_active":"2024-03-30T13:50:08.511337+0000","last_became_peered":"2024-03-30T13:50:08.511337+0000","last_unstale":"2024-03-30T13:52:53.478406+0000","last_undegraded":"2024-03-30T13:52:53.478406+0000","last_fullsized":"2024-03-30T13:50:08.501456+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:25.398232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:51.919203+0000","last_clean_scrub_stamp":"2024-03-30T13:49:25.398232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:55.942953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"216'17","reported_seq":395,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.482446+0000","last_change":"2024-03-30T13:52:25.542204+0000","last_active":"2024-03-30T13:52:53.482446+0000","last_peered":"2024-03-30T13:52:53.482446+0000","last_clean":"2024-03-30T13:52:53.482446+0000","last_became_active":"2024-03-30T13:42:47.320541+0000","last_became_peered":"2024-03-30T13:42:47.320541+0000","last_unstale":"2024-03-30T13:52:53.482446+0000","last_undegraded":"2024-03-30T13:52:53.482446+0000","last_fullsized":"2024-03-30T13:52:53.482446+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'13","last_scrub_stamp":"2024-03-30T13:52:25.542111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:25.542111+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:37.758143+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":122,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.484199+0000","last_change":"2024-03-30T13:50:08.514353+0000","last_active":"2024-03-30T13:52:53.484199+0000","last_peered":"2024-03-30T13:52:53.484199+0000","last_clean":"2024-03-30T13:50:05.492654+0000","last_became_active":"2024-03-30T13:50:08.514353+0000","last_became_peered":"2024-03-30T13:50:08.514353+0000","last_unstale":"2024-03-30T13:52:53.484199+0000","last_undegraded":"2024-03-30T13:52:53.484199+0000","last_fullsized":"2024-03-30T13:50:08.505968+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:28.373277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:28.373277+0000","last_clean_scrub_stamp":"2024-03-30T13:49:28.373277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:56.928070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"217'520","reported_seq":542,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:54.079497+0000","last_change":"2024-03-30T13:52:43.532265+0000","last_active":"2024-03-30T13:52:54.079497+0000","last_peered":"2024-03-30T13:52:54.079497+0000","last_clean":"2024-03-30T13:52:54.079497+0000","last_became_active":"2024-03-30T13:52:43.531214+0000","last_became_peered":"2024-03-30T13:52:43.531214+0000","last_unstale":"2024-03-30T13:52:54.079497+0000","last_undegraded":"2024-03-30T13:52:54.079497+0000","last_fullsized":"2024-03-30T13:52:54.079497+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:09.794874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34078720,"num_objects":520,"num_object_clones":0,"num_object_copies":1560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":520,"num_write_kb":33280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":353,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.483935+0000","last_change":"2024-03-30T13:52:28.079141+0000","last_active":"2024-03-30T13:52:53.483935+0000","last_peered":"2024-03-30T13:52:53.483935+0000","last_clean":"2024-03-30T13:52:53.483935+0000","last_became_active":"2024-03-30T13:42:51.331081+0000","last_became_peered":"2024-03-30T13:42:51.331081+0000","last_unstale":"2024-03-30T13:52:53.483935+0000","last_undegraded":"2024-03-30T13:52:53.483935+0000","last_fullsized":"2024-03-30T13:52:53.483935+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:28.079029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:12.433166+0000","last_clean_scrub_stamp":"2024-03-30T13:52:28.079029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:54.062037+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":627,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.479947+0000","last_change":"2024-03-30T13:52:47.943890+0000","last_active":"2024-03-30T13:52:53.479947+0000","last_peered":"2024-03-30T13:52:53.479947+0000","last_clean":"2024-03-30T13:52:53.479947+0000","last_became_active":"2024-03-30T13:52:31.482884+0000","last_became_peered":"2024-03-30T13:52:31.482884+0000","last_unstale":"2024-03-30T13:52:53.479947+0000","last_undegraded":"2024-03-30T13:52:53.479947+0000","last_fullsized":"2024-03-30T13:52:53.479947+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:47.943814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:47.943814+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:55.844094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":350,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.478179+0000","last_change":"2024-03-30T13:52:12.975404+0000","last_active":"2024-03-30T13:52:53.478179+0000","last_peered":"2024-03-30T13:52:53.478179+0000","last_clean":"2024-03-30T13:52:53.478179+0000","last_became_active":"2024-03-30T13:42:45.309954+0000","last_became_peered":"2024-03-30T13:42:45.309954+0000","last_unstale":"2024-03-30T13:52:53.478179+0000","last_undegraded":"2024-03-30T13:52:53.478179+0000","last_fullsized":"2024-03-30T13:52:53.478179+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:12.975288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:12.975288+0000","last_clean_scrub_stamp":"2024-03-30T13:52:12.975288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:40.450910+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"217'28","reported_seq":522,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.518151+0000","last_change":"2024-03-30T13:52:01.369622+0000","last_active":"2024-03-30T13:52:53.518151+0000","last_peered":"2024-03-30T13:52:53.518151+0000","last_clean":"2024-03-30T13:52:53.518151+0000","last_became_active":"2024-03-30T13:42:47.502338+0000","last_became_peered":"2024-03-30T13:42:47.502338+0000","last_unstale":"2024-03-30T13:52:53.518151+0000","last_undegraded":"2024-03-30T13:52:53.518151+0000","last_fullsized":"2024-03-30T13:52:53.518151+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'16","last_scrub_stamp":"2024-03-30T13:52:01.369487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:01.369487+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:15.847088+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":343,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.482142+0000","last_change":"2024-03-30T13:50:08.517677+0000","last_active":"2024-03-30T13:52:53.482142+0000","last_peered":"2024-03-30T13:52:53.482142+0000","last_clean":"2024-03-30T13:50:05.495178+0000","last_became_active":"2024-03-30T13:50:08.517677+0000","last_became_peered":"2024-03-30T13:50:08.517677+0000","last_unstale":"2024-03-30T13:52:53.482142+0000","last_undegraded":"2024-03-30T13:52:53.482142+0000","last_fullsized":"2024-03-30T13:50:08.511555+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:25.787524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:25.787524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:55.950596+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"217'581","reported_seq":603,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:54.447322+0000","last_change":"2024-03-30T13:52:43.606337+0000","last_active":"2024-03-30T13:52:54.447322+0000","last_peered":"2024-03-30T13:52:54.447322+0000","last_clean":"2024-03-30T13:52:54.447322+0000","last_became_active":"2024-03-30T13:52:43.605967+0000","last_became_peered":"2024-03-30T13:52:43.605967+0000","last_unstale":"2024-03-30T13:52:54.447322+0000","last_undegraded":"2024-03-30T13:52:54.447322+0000","last_fullsized":"2024-03-30T13:52:54.447322+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":581,"log_dups_size":0,"ondisk_log_size":581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:44.033460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38076416,"num_objects":581,"num_object_clones":0,"num_object_copies":1743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":581,"num_write_kb":37184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":337,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.482323+0000","last_change":"2024-03-30T13:52:47.632199+0000","last_active":"2024-03-30T13:52:53.482323+0000","last_peered":"2024-03-30T13:52:53.482323+0000","last_clean":"2024-03-30T13:52:53.482323+0000","last_became_active":"2024-03-30T13:52:36.489806+0000","last_became_peered":"2024-03-30T13:52:36.489806+0000","last_unstale":"2024-03-30T13:52:53.482323+0000","last_undegraded":"2024-03-30T13:52:53.482323+0000","last_fullsized":"2024-03-30T13:52:53.482323+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:47.632117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:47.632117+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:17.252927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":346,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.481754+0000","last_change":"2024-03-30T13:52:03.346086+0000","last_active":"2024-03-30T13:52:53.481754+0000","last_peered":"2024-03-30T13:52:53.481754+0000","last_clean":"2024-03-30T13:52:53.481754+0000","last_became_active":"2024-03-30T13:42:51.335188+0000","last_became_peered":"2024-03-30T13:42:51.335188+0000","last_unstale":"2024-03-30T13:52:53.481754+0000","last_undegraded":"2024-03-30T13:52:53.481754+0000","last_fullsized":"2024-03-30T13:52:53.481754+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:03.345965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:03.345965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:06.636168+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":356,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.480199+0000","last_change":"2024-03-30T13:52:08.002816+0000","last_active":"2024-03-30T13:52:53.480199+0000","last_peered":"2024-03-30T13:52:53.480199+0000","last_clean":"2024-03-30T13:52:53.480199+0000","last_became_active":"2024-03-30T13:42:45.307905+0000","last_became_peered":"2024-03-30T13:42:45.307905+0000","last_unstale":"2024-03-30T13:52:53.480199+0000","last_undegraded":"2024-03-30T13:52:53.480199+0000","last_fullsized":"2024-03-30T13:52:53.480199+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:08.002706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:08.002706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:12.846644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":969,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.479877+0000","last_change":"2024-03-30T13:52:18.785508+0000","last_active":"2024-03-30T13:52:53.479877+0000","last_peered":"2024-03-30T13:52:53.479877+0000","last_clean":"2024-03-30T13:52:53.479877+0000","last_became_active":"2024-03-30T13:42:44.302343+0000","last_became_peered":"2024-03-30T13:42:44.302343+0000","last_unstale":"2024-03-30T13:52:53.479877+0000","last_undegraded":"2024-03-30T13:52:53.479877+0000","last_fullsized":"2024-03-30T13:52:53.479877+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-30T13:52:18.785378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:52:18.785378+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:28.985258+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00126391,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":600,"num_read_kb":595,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"216'30","reported_seq":389,"reported_epoch":217,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:53.479992+0000","last_change":"2024-03-30T13:50:08.510256+0000","last_active":"2024-03-30T13:52:53.479992+0000","last_peered":"2024-03-30T13:52:53.479992+0000","last_clean":"2024-03-30T13:50:05.503358+0000","last_became_active":"2024-03-30T13:50:08.510256+0000","last_became_peered":"2024-03-30T13:50:08.510256+0000","last_unstale":"2024-03-30T13:52:53.479992+0000","last_undegraded":"2024-03-30T13:50:08.504536+0000","last_fullsized":"2024-03-30T13:50:08.504398+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'18","last_scrub_stamp":"2024-03-30T13:49:13.774982+0000","last_deep_scrub":"93'18","last_deep_scrub_stamp":"2024-03-30T13:47:28.872529+0000","last_clean_scrub_stamp":"2024-03-30T13:49:13.774982+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":341,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.484190+0000","last_change":"2024-03-30T13:51:08.978000+0000","last_active":"2024-03-30T13:52:53.484190+0000","last_peered":"2024-03-30T13:52:53.484190+0000","last_clean":"2024-03-30T13:52:53.484190+0000","last_became_active":"2024-03-30T13:42:49.322700+0000","last_became_peered":"2024-03-30T13:42:49.322700+0000","last_unstale":"2024-03-30T13:52:53.484190+0000","last_undegraded":"2024-03-30T13:52:53.484190+0000","last_fullsized":"2024-03-30T13:52:53.484190+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:08.977897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:08.977897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"217'405","reported_seq":427,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.547497+0000","last_change":"2024-03-30T13:52:43.534865+0000","last_active":"2024-03-30T13:52:53.547497+0000","last_peered":"2024-03-30T13:52:53.547497+0000","last_clean":"2024-03-30T13:52:53.547497+0000","last_became_active":"2024-03-30T13:52:43.534488+0000","last_became_peered":"2024-03-30T13:52:43.534488+0000","last_unstale":"2024-03-30T13:52:53.547497+0000","last_undegraded":"2024-03-30T13:52:53.547497+0000","last_fullsized":"2024-03-30T13:52:53.547497+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:48.583426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26542080,"num_objects":405,"num_object_clones":0,"num_object_copies":1215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":405,"num_write_kb":25920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":342,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.483449+0000","last_change":"2024-03-30T13:52:47.924323+0000","last_active":"2024-03-30T13:52:53.483449+0000","last_peered":"2024-03-30T13:52:53.483449+0000","last_clean":"2024-03-30T13:52:53.483449+0000","last_became_active":"2024-03-30T13:52:36.483298+0000","last_became_peered":"2024-03-30T13:52:36.483298+0000","last_unstale":"2024-03-30T13:52:53.483449+0000","last_undegraded":"2024-03-30T13:52:53.483449+0000","last_fullsized":"2024-03-30T13:52:53.483449+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:47.924212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:47.924212+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:12.215142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":341,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.481501+0000","last_change":"2024-03-30T13:50:08.512648+0000","last_active":"2024-03-30T13:52:53.481501+0000","last_peered":"2024-03-30T13:52:53.481501+0000","last_clean":"2024-03-30T13:50:05.489180+0000","last_became_active":"2024-03-30T13:50:08.512648+0000","last_became_peered":"2024-03-30T13:50:08.512648+0000","last_unstale":"2024-03-30T13:52:53.481501+0000","last_undegraded":"2024-03-30T13:52:53.481501+0000","last_fullsized":"2024-03-30T13:50:08.505931+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:21.527712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:21.527712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:57.611585+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":353,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.482251+0000","last_change":"2024-03-30T13:50:08.517734+0000","last_active":"2024-03-30T13:52:53.482251+0000","last_peered":"2024-03-30T13:52:53.482251+0000","last_clean":"2024-03-30T13:50:05.490307+0000","last_became_active":"2024-03-30T13:50:08.517734+0000","last_became_peered":"2024-03-30T13:50:08.517734+0000","last_unstale":"2024-03-30T13:52:53.482251+0000","last_undegraded":"2024-03-30T13:52:53.482251+0000","last_fullsized":"2024-03-30T13:50:08.507406+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:19.525007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:45.975163+0000","last_clean_scrub_stamp":"2024-03-30T13:49:19.525007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:57.611629+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"216'29","reported_seq":403,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.480919+0000","last_change":"2024-03-30T13:52:15.020713+0000","last_active":"2024-03-30T13:52:53.480919+0000","last_peered":"2024-03-30T13:52:53.480919+0000","last_clean":"2024-03-30T13:52:53.480919+0000","last_became_active":"2024-03-30T13:42:47.501272+0000","last_became_peered":"2024-03-30T13:42:47.501272+0000","last_unstale":"2024-03-30T13:52:53.480919+0000","last_undegraded":"2024-03-30T13:52:53.480919+0000","last_fullsized":"2024-03-30T13:52:53.480919+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-30T13:52:15.020534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:15.020534+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:34.088122+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":364,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.483670+0000","last_change":"2024-03-30T13:52:52.887173+0000","last_active":"2024-03-30T13:52:53.483670+0000","last_peered":"2024-03-30T13:52:53.483670+0000","last_clean":"2024-03-30T13:52:53.483670+0000","last_became_active":"2024-03-30T13:42:49.321520+0000","last_became_peered":"2024-03-30T13:42:49.321520+0000","last_unstale":"2024-03-30T13:52:53.483670+0000","last_undegraded":"2024-03-30T13:52:53.483670+0000","last_fullsized":"2024-03-30T13:52:53.483670+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:52.887014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:52.887014+0000","last_clean_scrub_stamp":"2024-03-30T13:52:52.887014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:54.009340+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"217'416","reported_seq":438,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.568827+0000","last_change":"2024-03-30T13:52:43.531145+0000","last_active":"2024-03-30T13:52:53.568827+0000","last_peered":"2024-03-30T13:52:53.568827+0000","last_clean":"2024-03-30T13:52:53.568827+0000","last_became_active":"2024-03-30T13:52:43.530855+0000","last_became_peered":"2024-03-30T13:52:43.530855+0000","last_unstale":"2024-03-30T13:52:53.568827+0000","last_undegraded":"2024-03-30T13:52:53.568827+0000","last_fullsized":"2024-03-30T13:52:53.568827+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:57.477559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27262976,"num_objects":416,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":26624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":342,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.483821+0000","last_change":"2024-03-30T13:52:48.950030+0000","last_active":"2024-03-30T13:52:53.483821+0000","last_peered":"2024-03-30T13:52:53.483821+0000","last_clean":"2024-03-30T13:52:53.483821+0000","last_became_active":"2024-03-30T13:52:36.484212+0000","last_became_peered":"2024-03-30T13:52:36.484212+0000","last_unstale":"2024-03-30T13:52:53.483821+0000","last_undegraded":"2024-03-30T13:52:53.483821+0000","last_fullsized":"2024-03-30T13:52:53.483821+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:48.949941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:48.949941+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:55.333122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":339,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.479186+0000","last_change":"2024-03-30T13:50:08.510025+0000","last_active":"2024-03-30T13:52:53.479186+0000","last_peered":"2024-03-30T13:52:53.479186+0000","last_clean":"2024-03-30T13:50:05.498930+0000","last_became_active":"2024-03-30T13:50:08.510025+0000","last_became_peered":"2024-03-30T13:50:08.510025+0000","last_unstale":"2024-03-30T13:52:53.479186+0000","last_undegraded":"2024-03-30T13:52:53.479186+0000","last_fullsized":"2024-03-30T13:50:08.502478+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:28.946303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:41.142941+0000","last_clean_scrub_stamp":"2024-03-30T13:49:28.946303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":352,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.483511+0000","last_change":"2024-03-30T13:50:08.514920+0000","last_active":"2024-03-30T13:52:53.483511+0000","last_peered":"2024-03-30T13:52:53.483511+0000","last_clean":"2024-03-30T13:50:05.493549+0000","last_became_active":"2024-03-30T13:50:08.514920+0000","last_became_peered":"2024-03-30T13:50:08.514920+0000","last_unstale":"2024-03-30T13:52:53.483511+0000","last_undegraded":"2024-03-30T13:52:53.483511+0000","last_fullsized":"2024-03-30T13:50:08.504625+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:08.543531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:55.458779+0000","last_clean_scrub_stamp":"2024-03-30T13:49:08.543531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:56.928052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"217'25","reported_seq":397,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.501556+0000","last_change":"2024-03-30T13:52:15.457401+0000","last_active":"2024-03-30T13:52:53.501556+0000","last_peered":"2024-03-30T13:52:53.501556+0000","last_clean":"2024-03-30T13:52:53.501556+0000","last_became_active":"2024-03-30T13:42:47.321433+0000","last_became_peered":"2024-03-30T13:42:47.321433+0000","last_unstale":"2024-03-30T13:52:53.501556+0000","last_undegraded":"2024-03-30T13:52:53.501556+0000","last_fullsized":"2024-03-30T13:52:53.501556+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'15","last_scrub_stamp":"2024-03-30T13:52:15.457275+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-03-30T13:45:50.945837+0000","last_clean_scrub_stamp":"2024-03-30T13:52:15.457275+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:43.387403+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":478,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.481059+0000","last_change":"2024-03-30T13:51:17.257600+0000","last_active":"2024-03-30T13:52:53.481059+0000","last_peered":"2024-03-30T13:52:53.481059+0000","last_clean":"2024-03-30T13:52:53.481059+0000","last_became_active":"2024-03-30T13:42:49.325533+0000","last_became_peered":"2024-03-30T13:42:49.325533+0000","last_unstale":"2024-03-30T13:52:53.481059+0000","last_undegraded":"2024-03-30T13:52:53.481059+0000","last_fullsized":"2024-03-30T13:52:53.481059+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:51:17.257464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:17.257464+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"217'506","reported_seq":528,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:54.069158+0000","last_change":"2024-03-30T13:52:43.534654+0000","last_active":"2024-03-30T13:52:54.069158+0000","last_peered":"2024-03-30T13:52:54.069158+0000","last_clean":"2024-03-30T13:52:54.069158+0000","last_became_active":"2024-03-30T13:52:43.534310+0000","last_became_peered":"2024-03-30T13:52:43.534310+0000","last_unstale":"2024-03-30T13:52:54.069158+0000","last_undegraded":"2024-03-30T13:52:54.069158+0000","last_fullsized":"2024-03-30T13:52:54.069158+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:04.651523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33161216,"num_objects":506,"num_object_clones":0,"num_object_copies":1518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":506,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":337,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.478307+0000","last_change":"2024-03-30T13:52:48.866197+0000","last_active":"2024-03-30T13:52:53.478307+0000","last_peered":"2024-03-30T13:52:53.478307+0000","last_clean":"2024-03-30T13:52:53.478307+0000","last_became_active":"2024-03-30T13:52:36.490002+0000","last_became_peered":"2024-03-30T13:52:36.490002+0000","last_unstale":"2024-03-30T13:52:53.478307+0000","last_undegraded":"2024-03-30T13:52:53.478307+0000","last_fullsized":"2024-03-30T13:52:53.478307+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:48.866100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:48.866100+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:55.075246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":331,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.480662+0000","last_change":"2024-03-30T13:50:08.513975+0000","last_active":"2024-03-30T13:52:53.480662+0000","last_peered":"2024-03-30T13:52:53.480662+0000","last_clean":"2024-03-30T13:50:05.495405+0000","last_became_active":"2024-03-30T13:50:08.513975+0000","last_became_peered":"2024-03-30T13:50:08.513975+0000","last_unstale":"2024-03-30T13:52:53.480662+0000","last_undegraded":"2024-03-30T13:52:53.480662+0000","last_fullsized":"2024-03-30T13:50:08.507347+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:21.782290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:31.900908+0000","last_clean_scrub_stamp":"2024-03-30T13:49:21.782290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:55.950635+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":365,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.480315+0000","last_change":"2024-03-30T13:52:46.959816+0000","last_active":"2024-03-30T13:52:53.480315+0000","last_peered":"2024-03-30T13:52:53.480315+0000","last_clean":"2024-03-30T13:52:53.480315+0000","last_became_active":"2024-03-30T13:42:45.309886+0000","last_became_peered":"2024-03-30T13:42:45.309886+0000","last_unstale":"2024-03-30T13:52:53.480315+0000","last_undegraded":"2024-03-30T13:52:53.480315+0000","last_fullsized":"2024-03-30T13:52:53.480315+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:52:46.959694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:46.959694+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:57.276541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"217'35","reported_seq":412,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.539415+0000","last_change":"2024-03-30T13:52:44.926022+0000","last_active":"2024-03-30T13:52:53.539415+0000","last_peered":"2024-03-30T13:52:53.539415+0000","last_clean":"2024-03-30T13:52:53.539415+0000","last_became_active":"2024-03-30T13:42:47.314184+0000","last_became_peered":"2024-03-30T13:42:47.314184+0000","last_unstale":"2024-03-30T13:52:53.539415+0000","last_undegraded":"2024-03-30T13:52:53.539415+0000","last_fullsized":"2024-03-30T13:52:53.539415+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'21","last_scrub_stamp":"2024-03-30T13:52:44.925926+0000","last_deep_scrub":"91'21","last_deep_scrub_stamp":"2024-03-30T13:44:05.917064+0000","last_clean_scrub_stamp":"2024-03-30T13:52:44.925926+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:03.961017+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":473,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.480560+0000","last_change":"2024-03-30T13:52:17.989527+0000","last_active":"2024-03-30T13:52:53.480560+0000","last_peered":"2024-03-30T13:52:53.480560+0000","last_clean":"2024-03-30T13:52:53.480560+0000","last_became_active":"2024-03-30T13:42:49.321423+0000","last_became_peered":"2024-03-30T13:42:49.321423+0000","last_unstale":"2024-03-30T13:52:53.480560+0000","last_undegraded":"2024-03-30T13:52:53.480560+0000","last_fullsized":"2024-03-30T13:52:53.480560+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:52:17.989349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:17.989349+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:25.123045+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"217'415","reported_seq":437,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.545149+0000","last_change":"2024-03-30T13:52:43.535271+0000","last_active":"2024-03-30T13:52:53.545149+0000","last_peered":"2024-03-30T13:52:53.545149+0000","last_clean":"2024-03-30T13:52:53.545149+0000","last_became_active":"2024-03-30T13:52:43.534924+0000","last_became_peered":"2024-03-30T13:52:43.534924+0000","last_unstale":"2024-03-30T13:52:53.545149+0000","last_undegraded":"2024-03-30T13:52:53.545149+0000","last_fullsized":"2024-03-30T13:52:53.545149+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:51.634862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27197440,"num_objects":415,"num_object_clones":0,"num_object_copies":1245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":415,"num_write_kb":26560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":337,"reported_epoch":217,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:52:53.476833+0000","last_change":"2024-03-30T13:52:47.990769+0000","last_active":"2024-03-30T13:52:53.476833+0000","last_peered":"2024-03-30T13:52:53.476833+0000","last_clean":"2024-03-30T13:52:53.476833+0000","last_became_active":"2024-03-30T13:52:36.490909+0000","last_became_peered":"2024-03-30T13:52:36.490909+0000","last_unstale":"2024-03-30T13:52:53.476833+0000","last_undegraded":"2024-03-30T13:52:53.476833+0000","last_fullsized":"2024-03-30T13:52:53.476833+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:47.990635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:47.990635+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:17.956945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,0],"avail_no_missing":["4","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":334,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.475572+0000","last_change":"2024-03-30T13:50:08.504411+0000","last_active":"2024-03-30T13:52:53.475572+0000","last_peered":"2024-03-30T13:52:53.475572+0000","last_clean":"2024-03-30T13:50:05.489944+0000","last_became_active":"2024-03-30T13:50:08.504411+0000","last_became_peered":"2024-03-30T13:50:08.504411+0000","last_unstale":"2024-03-30T13:52:53.475572+0000","last_undegraded":"2024-03-30T13:52:53.475572+0000","last_fullsized":"2024-03-30T13:50:08.501823+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:10.216387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:10.216387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":350,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.481605+0000","last_change":"2024-03-30T13:51:24.984126+0000","last_active":"2024-03-30T13:52:53.481605+0000","last_peered":"2024-03-30T13:52:53.481605+0000","last_clean":"2024-03-30T13:52:53.481605+0000","last_became_active":"2024-03-30T13:42:45.314533+0000","last_became_peered":"2024-03-30T13:42:45.314533+0000","last_unstale":"2024-03-30T13:52:53.481605+0000","last_undegraded":"2024-03-30T13:52:53.481605+0000","last_fullsized":"2024-03-30T13:52:53.481605+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:24.983992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:24.171036+0000","last_clean_scrub_stamp":"2024-03-30T13:51:24.983992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"216'35","reported_seq":410,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.480419+0000","last_change":"2024-03-30T13:52:52.973764+0000","last_active":"2024-03-30T13:52:53.480419+0000","last_peered":"2024-03-30T13:52:53.480419+0000","last_clean":"2024-03-30T13:52:53.480419+0000","last_became_active":"2024-03-30T13:42:47.502708+0000","last_became_peered":"2024-03-30T13:42:47.502708+0000","last_unstale":"2024-03-30T13:52:53.480419+0000","last_undegraded":"2024-03-30T13:52:53.480419+0000","last_fullsized":"2024-03-30T13:52:53.480419+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'29","last_scrub_stamp":"2024-03-30T13:52:52.973602+0000","last_deep_scrub":"216'29","last_deep_scrub_stamp":"2024-03-30T13:52:52.973602+0000","last_clean_scrub_stamp":"2024-03-30T13:52:52.973602+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:09.953239+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":343,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.475619+0000","last_change":"2024-03-30T13:51:09.247429+0000","last_active":"2024-03-30T13:52:53.475619+0000","last_peered":"2024-03-30T13:52:53.475619+0000","last_clean":"2024-03-30T13:52:53.475619+0000","last_became_active":"2024-03-30T13:42:49.324022+0000","last_became_peered":"2024-03-30T13:42:49.324022+0000","last_unstale":"2024-03-30T13:52:53.475619+0000","last_undegraded":"2024-03-30T13:52:53.475619+0000","last_fullsized":"2024-03-30T13:52:53.475619+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:09.247282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:09.247282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"217'511","reported_seq":530,"reported_epoch":217,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:54.078314+0000","last_change":"2024-03-30T13:52:51.714810+0000","last_active":"2024-03-30T13:52:54.078314+0000","last_peered":"2024-03-30T13:52:54.078314+0000","last_clean":"2024-03-30T13:52:42.503419+0000","last_became_active":"2024-03-30T13:52:43.528398+0000","last_became_peered":"2024-03-30T13:52:43.528398+0000","last_unstale":"2024-03-30T13:52:54.078314+0000","last_undegraded":"2024-03-30T13:52:51.712396+0000","last_fullsized":"2024-03-30T13:52:43.524380+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:58.393360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33488896,"num_objects":511,"num_object_clones":0,"num_object_copies":1533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":511,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":511,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":511}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":337,"reported_epoch":217,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:52:53.481755+0000","last_change":"2024-03-30T13:52:49.876112+0000","last_active":"2024-03-30T13:52:53.481755+0000","last_peered":"2024-03-30T13:52:53.481755+0000","last_clean":"2024-03-30T13:52:53.481755+0000","last_became_active":"2024-03-30T13:52:36.492521+0000","last_became_peered":"2024-03-30T13:52:36.492521+0000","last_unstale":"2024-03-30T13:52:53.481755+0000","last_undegraded":"2024-03-30T13:52:53.481755+0000","last_fullsized":"2024-03-30T13:52:53.481755+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:49.875971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:49.875971+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:03.217204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6,0],"avail_no_missing":["5","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":345,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.483137+0000","last_change":"2024-03-30T13:52:14.144206+0000","last_active":"2024-03-30T13:52:53.483137+0000","last_peered":"2024-03-30T13:52:53.483137+0000","last_clean":"2024-03-30T13:52:53.483137+0000","last_became_active":"2024-03-30T13:42:51.329026+0000","last_became_peered":"2024-03-30T13:42:51.329026+0000","last_unstale":"2024-03-30T13:52:53.483137+0000","last_undegraded":"2024-03-30T13:52:53.483137+0000","last_fullsized":"2024-03-30T13:52:53.483137+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:14.144050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:14.144050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:32.689890+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"186'9","reported_seq":482,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.478435+0000","last_change":"2024-03-30T13:51:15.146386+0000","last_active":"2024-03-30T13:52:53.478435+0000","last_peered":"2024-03-30T13:52:53.478435+0000","last_clean":"2024-03-30T13:52:53.478435+0000","last_became_active":"2024-03-30T13:42:49.321790+0000","last_became_peered":"2024-03-30T13:42:49.321790+0000","last_unstale":"2024-03-30T13:52:53.478435+0000","last_undegraded":"2024-03-30T13:52:53.478435+0000","last_fullsized":"2024-03-30T13:52:53.478435+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-30T13:51:15.146255+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-30T13:51:15.146255+0000","last_clean_scrub_stamp":"2024-03-30T13:51:15.146255+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":350,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.480331+0000","last_change":"2024-03-30T13:52:27.058941+0000","last_active":"2024-03-30T13:52:53.480331+0000","last_peered":"2024-03-30T13:52:53.480331+0000","last_clean":"2024-03-30T13:52:53.480331+0000","last_became_active":"2024-03-30T13:42:45.315319+0000","last_became_peered":"2024-03-30T13:42:45.315319+0000","last_unstale":"2024-03-30T13:52:53.480331+0000","last_undegraded":"2024-03-30T13:52:53.480331+0000","last_fullsized":"2024-03-30T13:52:53.480331+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:27.058807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:27.058807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:52.862721+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"217'34","reported_seq":412,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.598817+0000","last_change":"2024-03-30T13:52:52.619670+0000","last_active":"2024-03-30T13:52:53.598817+0000","last_peered":"2024-03-30T13:52:53.598817+0000","last_clean":"2024-03-30T13:52:53.598817+0000","last_became_active":"2024-03-30T13:42:47.324350+0000","last_became_peered":"2024-03-30T13:42:47.324350+0000","last_unstale":"2024-03-30T13:52:53.598817+0000","last_undegraded":"2024-03-30T13:52:53.598817+0000","last_fullsized":"2024-03-30T13:52:53.598817+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'22","last_scrub_stamp":"2024-03-30T13:52:52.619482+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:28.831527+0000","last_clean_scrub_stamp":"2024-03-30T13:52:52.619482+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:18.476091+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"217'396","reported_seq":418,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.530794+0000","last_change":"2024-03-30T13:52:43.532021+0000","last_active":"2024-03-30T13:52:53.530794+0000","last_peered":"2024-03-30T13:52:53.530794+0000","last_clean":"2024-03-30T13:52:53.530794+0000","last_became_active":"2024-03-30T13:52:43.531741+0000","last_became_peered":"2024-03-30T13:52:43.531741+0000","last_unstale":"2024-03-30T13:52:53.530794+0000","last_undegraded":"2024-03-30T13:52:53.530794+0000","last_fullsized":"2024-03-30T13:52:53.530794+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":396,"log_dups_size":0,"ondisk_log_size":396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:10.441483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25952256,"num_objects":396,"num_object_clones":0,"num_object_copies":1188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":396,"num_write_kb":25344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":338,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.478501+0000","last_change":"2024-03-30T13:52:21.082083+0000","last_active":"2024-03-30T13:52:53.478501+0000","last_peered":"2024-03-30T13:52:53.478501+0000","last_clean":"2024-03-30T13:52:53.478501+0000","last_became_active":"2024-03-30T13:42:51.371369+0000","last_became_peered":"2024-03-30T13:42:51.371369+0000","last_unstale":"2024-03-30T13:52:53.478501+0000","last_undegraded":"2024-03-30T13:52:53.478501+0000","last_fullsized":"2024-03-30T13:52:53.478501+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:21.081938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:21.081938+0000","last_clean_scrub_stamp":"2024-03-30T13:52:21.081938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:49.566123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":349,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.484148+0000","last_change":"2024-03-30T13:51:21.087839+0000","last_active":"2024-03-30T13:52:53.484148+0000","last_peered":"2024-03-30T13:52:53.484148+0000","last_clean":"2024-03-30T13:52:53.484148+0000","last_became_active":"2024-03-30T13:42:49.323319+0000","last_became_peered":"2024-03-30T13:42:49.323319+0000","last_unstale":"2024-03-30T13:52:53.484148+0000","last_undegraded":"2024-03-30T13:52:53.484148+0000","last_fullsized":"2024-03-30T13:52:53.484148+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:21.087732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:52.603772+0000","last_clean_scrub_stamp":"2024-03-30T13:51:21.087732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":346,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.476213+0000","last_change":"2024-03-30T13:50:08.509331+0000","last_active":"2024-03-30T13:52:53.476213+0000","last_peered":"2024-03-30T13:52:53.476213+0000","last_clean":"2024-03-30T13:50:05.489382+0000","last_became_active":"2024-03-30T13:50:08.509331+0000","last_became_peered":"2024-03-30T13:50:08.509331+0000","last_unstale":"2024-03-30T13:52:53.476213+0000","last_undegraded":"2024-03-30T13:52:53.476213+0000","last_fullsized":"2024-03-30T13:50:08.501285+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:15.107879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:48:15.107879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"217'19","reported_seq":394,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.529003+0000","last_change":"2024-03-30T13:52:17.057583+0000","last_active":"2024-03-30T13:52:53.529003+0000","last_peered":"2024-03-30T13:52:53.529003+0000","last_clean":"2024-03-30T13:52:53.529003+0000","last_became_active":"2024-03-30T13:42:47.315667+0000","last_became_peered":"2024-03-30T13:42:47.315667+0000","last_unstale":"2024-03-30T13:52:53.529003+0000","last_undegraded":"2024-03-30T13:52:53.529003+0000","last_fullsized":"2024-03-30T13:52:53.529003+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:52:17.057433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:17.057433+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:18.989398+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"217'515","reported_seq":534,"reported_epoch":217,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:54.009984+0000","last_change":"2024-03-30T13:52:51.696416+0000","last_active":"2024-03-30T13:52:54.009984+0000","last_peered":"2024-03-30T13:52:54.009984+0000","last_clean":"2024-03-30T13:52:42.503419+0000","last_became_active":"2024-03-30T13:52:43.605206+0000","last_became_peered":"2024-03-30T13:52:43.605206+0000","last_unstale":"2024-03-30T13:52:54.009984+0000","last_undegraded":"2024-03-30T13:52:51.691832+0000","last_fullsized":"2024-03-30T13:52:43.519471+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:00.616672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33751040,"num_objects":515,"num_object_clones":0,"num_object_copies":1545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":515,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":515,"num_write_kb":32960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":515}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":337,"reported_epoch":217,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:52:53.481465+0000","last_change":"2024-03-30T13:52:47.932358+0000","last_active":"2024-03-30T13:52:53.481465+0000","last_peered":"2024-03-30T13:52:53.481465+0000","last_clean":"2024-03-30T13:52:53.481465+0000","last_became_active":"2024-03-30T13:52:36.489659+0000","last_became_peered":"2024-03-30T13:52:36.489659+0000","last_unstale":"2024-03-30T13:52:53.481465+0000","last_undegraded":"2024-03-30T13:52:53.481465+0000","last_fullsized":"2024-03-30T13:52:53.481465+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:47.932192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:47.932192+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:05.571488+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3,0],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":122,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.481213+0000","last_change":"2024-03-30T13:50:08.511676+0000","last_active":"2024-03-30T13:52:53.481213+0000","last_peered":"2024-03-30T13:52:53.481213+0000","last_clean":"2024-03-30T13:50:05.504989+0000","last_became_active":"2024-03-30T13:50:08.511676+0000","last_became_peered":"2024-03-30T13:50:08.511676+0000","last_unstale":"2024-03-30T13:52:53.481213+0000","last_undegraded":"2024-03-30T13:52:53.481213+0000","last_fullsized":"2024-03-30T13:50:08.504157+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:09.357378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:09.357378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:55.942919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":358,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.482450+0000","last_change":"2024-03-30T13:52:13.440265+0000","last_active":"2024-03-30T13:52:53.482450+0000","last_peered":"2024-03-30T13:52:53.482450+0000","last_clean":"2024-03-30T13:52:53.482450+0000","last_became_active":"2024-03-30T13:42:49.321360+0000","last_became_peered":"2024-03-30T13:42:49.321360+0000","last_unstale":"2024-03-30T13:52:53.482450+0000","last_undegraded":"2024-03-30T13:52:53.482450+0000","last_fullsized":"2024-03-30T13:52:53.482450+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:13.440150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:13.440150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:16.505795+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":372,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.483433+0000","last_change":"2024-03-30T13:52:25.063121+0000","last_active":"2024-03-30T13:52:53.483433+0000","last_peered":"2024-03-30T13:52:53.483433+0000","last_clean":"2024-03-30T13:52:53.483433+0000","last_became_active":"2024-03-30T13:42:45.315251+0000","last_became_peered":"2024-03-30T13:42:45.315251+0000","last_unstale":"2024-03-30T13:52:53.483433+0000","last_undegraded":"2024-03-30T13:52:53.483433+0000","last_fullsized":"2024-03-30T13:52:53.483433+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:25.062771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:25.062771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:36.905655+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"217'31","reported_seq":391,"reported_epoch":217,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:53.523736+0000","last_change":"2024-03-30T13:50:08.508030+0000","last_active":"2024-03-30T13:52:53.523736+0000","last_peered":"2024-03-30T13:52:53.523736+0000","last_clean":"2024-03-30T13:50:05.491159+0000","last_became_active":"2024-03-30T13:50:08.508030+0000","last_became_peered":"2024-03-30T13:50:08.508030+0000","last_unstale":"2024-03-30T13:52:53.523736+0000","last_undegraded":"2024-03-30T13:50:08.502958+0000","last_fullsized":"2024-03-30T13:50:08.502815+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-03-30T13:48:27.096261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:27.096261+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"217'505","reported_seq":527,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:54.080317+0000","last_change":"2024-03-30T13:52:43.606247+0000","last_active":"2024-03-30T13:52:54.080317+0000","last_peered":"2024-03-30T13:52:54.080317+0000","last_clean":"2024-03-30T13:52:54.080317+0000","last_became_active":"2024-03-30T13:52:43.605826+0000","last_became_peered":"2024-03-30T13:52:43.605826+0000","last_unstale":"2024-03-30T13:52:54.080317+0000","last_undegraded":"2024-03-30T13:52:54.080317+0000","last_fullsized":"2024-03-30T13:52:54.080317+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":505,"log_dups_size":0,"ondisk_log_size":505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:09.355401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33095680,"num_objects":505,"num_object_clones":0,"num_object_copies":1515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":505,"num_write_kb":32320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":634,"reported_epoch":217,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:52:53.480036+0000","last_change":"2024-03-30T13:52:48.985777+0000","last_active":"2024-03-30T13:52:53.480036+0000","last_peered":"2024-03-30T13:52:53.480036+0000","last_clean":"2024-03-30T13:52:53.480036+0000","last_became_active":"2024-03-30T13:52:36.484417+0000","last_became_peered":"2024-03-30T13:52:36.484417+0000","last_unstale":"2024-03-30T13:52:53.480036+0000","last_undegraded":"2024-03-30T13:52:53.480036+0000","last_fullsized":"2024-03-30T13:52:53.480036+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:48.985609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:48.985609+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:02.243193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,0],"avail_no_missing":["7","0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":334,"reported_epoch":217,"state":"active+undersized","last_fresh":"2024-03-30T13:52:53.478523+0000","last_change":"2024-03-30T13:50:08.511010+0000","last_active":"2024-03-30T13:52:53.478523+0000","last_peered":"2024-03-30T13:52:53.478523+0000","last_clean":"2024-03-30T13:50:05.490535+0000","last_became_active":"2024-03-30T13:50:08.511010+0000","last_became_peered":"2024-03-30T13:50:08.511010+0000","last_unstale":"2024-03-30T13:52:53.478523+0000","last_undegraded":"2024-03-30T13:52:53.478523+0000","last_fullsized":"2024-03-30T13:50:08.503118+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:06.154805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:06.154805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"168'10","reported_seq":186,"reported_epoch":217,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:52:53.482078+0000","last_change":"2024-03-30T13:50:08.526935+0000","last_active":"2024-03-30T13:52:53.482078+0000","last_peered":"2024-03-30T13:52:53.482078+0000","last_clean":"2024-03-30T13:50:05.495636+0000","last_became_active":"2024-03-30T13:50:08.526935+0000","last_became_peered":"2024-03-30T13:50:08.526935+0000","last_unstale":"2024-03-30T13:52:53.482078+0000","last_undegraded":"2024-03-30T13:50:08.509321+0000","last_fullsized":"2024-03-30T13:50:08.509210+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:49:36.397622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:36.397622+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:52:55.950650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":350,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.477892+0000","last_change":"2024-03-30T13:51:58.939856+0000","last_active":"2024-03-30T13:52:53.477892+0000","last_peered":"2024-03-30T13:52:53.477892+0000","last_clean":"2024-03-30T13:52:53.477892+0000","last_became_active":"2024-03-30T13:42:45.309265+0000","last_became_peered":"2024-03-30T13:42:45.309265+0000","last_unstale":"2024-03-30T13:52:53.477892+0000","last_undegraded":"2024-03-30T13:52:53.477892+0000","last_fullsized":"2024-03-30T13:52:53.477892+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:58.939556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:58.939556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:03.912680+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"216'20","reported_seq":385,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:52:53.476969+0000","last_change":"2024-03-30T13:51:59.022459+0000","last_active":"2024-03-30T13:52:53.476969+0000","last_peered":"2024-03-30T13:52:53.476969+0000","last_clean":"2024-03-30T13:52:53.476969+0000","last_became_active":"2024-03-30T13:42:47.313026+0000","last_became_peered":"2024-03-30T13:42:47.313026+0000","last_unstale":"2024-03-30T13:52:53.476969+0000","last_undegraded":"2024-03-30T13:52:53.476969+0000","last_fullsized":"2024-03-30T13:52:53.476969+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-30T13:51:59.022304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:59.022304+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:53:13.924563+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":506920960,"num_objects":7735,"num_object_clones":0,"num_object_copies":23205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7735,"num_write_kb":495040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1369374720,"data_stored":1369374720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7735,"ondisk_log_size":7735,"up":45,"acting":45,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":600,"num_read_kb":595,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":28,"acting":33,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":88,"acting":88,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1126,"num_read_kb":883,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":738,"ondisk_log_size":738,"up":88,"acting":88,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":87,"acting":87,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":84,"acting":84,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103295,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":431992,"kb_used_data":162024,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":93313160,"statfs":{"total":95995035648,"available":95552675840,"internally_reserved":0,"allocated":165912576,"data_stored":165292976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":276429258},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397635,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":294816,"kb_used_data":51220,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":93450336,"statfs":{"total":95995035648,"available":95693144064,"internally_reserved":0,"allocated":52449280,"data_stored":51849108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":249428422},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575005,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":608552,"kb_used_data":244184,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93136600,"statfs":{"total":95995035648,"available":95371878400,"internally_reserved":0,"allocated":250044416,"data_stored":249384882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738530,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":492492,"kb_used_data":128124,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93252660,"statfs":{"total":95995035648,"available":95490723840,"internally_reserved":0,"allocated":131198976,"data_stored":130568364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411446,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2213288,"kb_used_data":2015520,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":91531864,"statfs":{"total":95995035648,"available":93728628736,"internally_reserved":0,"allocated":2063892480,"data_stored":2063509110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182547,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":605268,"kb_used_data":253380,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93139884,"statfs":{"total":95995035648,"available":95375241216,"internally_reserved":0,"allocated":259461120,"data_stored":258821538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":360315335},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953613,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":339196,"kb_used_data":51188,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":93405956,"statfs":{"total":95995035648,"available":95647698944,"internally_reserved":0,"allocated":52416512,"data_stored":51784575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691976,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":450992,"kb_used_data":162980,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":93294160,"statfs":{"total":95995035648,"available":95533219840,"internally_reserved":0,"allocated":166891520,"data_stored":166277204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":158400512,"data_stored":158400512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292945920,"data_stored":292945920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270204928,"data_stored":270204928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":77987840,"data_stored":77987840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":245628928,"data_stored":245628928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81526784,"data_stored":81526784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":242679808,"data_stored":242679808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:52:58.831 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-03-30T13:52:58.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:58.843 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:58.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:58.855 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:58 smithi113 ceph-mon[34341]: from='client.25226 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:52:58.855 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:58 smithi113 ceph-mon[34341]: pgmap v425: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 484 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 54 KiB/s rd, 80 MiB/s wr, 1.39k op/s; 1463/23799 objects degraded (6.147%); 1/23799 objects misplaced (0.004%) 2024-03-30T13:52:58.855 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:58 smithi113 ceph-mon[34341]: 5.3 deep-scrub starts 2024-03-30T13:52:58.855 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:58 smithi113 ceph-mon[34341]: 5.3 deep-scrub ok 2024-03-30T13:52:58.855 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:58 smithi113 ceph-mon[34341]: 4.11 scrub starts 2024-03-30T13:52:58.855 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:58 smithi113 ceph-mon[34341]: 4.11 scrub ok 2024-03-30T13:52:58.856 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:58 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:52:58.856 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:58 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:52:58.944 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:52:58.985 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:58.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:58.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:52:59.086 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:59.122 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:59.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:59.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:59.198 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:52:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:52:59.224 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:59.261 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:59.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:59.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:59.363 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:52:59.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:59.414 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:59.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:52:59.456 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:59.516 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:52:59.539 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:59.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:59.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:52:59.640 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:52:59.646 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:52:59 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:52:59.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:52:59 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:52:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:52:59.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:59.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:52:59.687 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 8 16 23581 23565 184.043 130.25 0.00371453 0.00542715 2024-03-30T13:52:59.780 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:52:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:59.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:52:59.919 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:52:59.957 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:52:59.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:52:59.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:52:59.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:52:59 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:53:00.059 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:00.109 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:00.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:00.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:00.211 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:00.252 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:00.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:00.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:00.354 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:00.391 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:00.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:00.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:00.493 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:00.528 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:00.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:00.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:00.630 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:00.646 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:00 smithi043 ceph-mon[29963]: pgmap v426: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 1.3 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 47 KiB/s rd, 195 MiB/s wr, 3.21k op/s; 4059/66522 objects degraded (6.102%); 1/66522 objects misplaced (0.002%) 2024-03-30T13:53:00.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:00 smithi043 ceph-mon[29963]: 4.16 deep-scrub starts 2024-03-30T13:53:00.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:00 smithi043 ceph-mon[29963]: 4.16 deep-scrub ok 2024-03-30T13:53:00.647 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:00 smithi043 ceph-mon[42099]: pgmap v426: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 1.3 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 47 KiB/s rd, 195 MiB/s wr, 3.21k op/s; 4059/66522 objects degraded (6.102%); 1/66522 objects misplaced (0.002%) 2024-03-30T13:53:00.647 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:00 smithi043 ceph-mon[42099]: 4.16 deep-scrub starts 2024-03-30T13:53:00.647 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:00 smithi043 ceph-mon[42099]: 4.16 deep-scrub ok 2024-03-30T13:53:00.653 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:00.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:00.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:00.687 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 9 16 25945 25929 180.01 147.75 0.00254052 0.00553976 2024-03-30T13:53:00.719 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:00 smithi113 ceph-mon[34341]: pgmap v426: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 1.3 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 47 KiB/s rd, 195 MiB/s wr, 3.21k op/s; 4059/66522 objects degraded (6.102%); 1/66522 objects misplaced (0.002%) 2024-03-30T13:53:00.719 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:00 smithi113 ceph-mon[34341]: 4.16 deep-scrub starts 2024-03-30T13:53:00.720 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:00 smithi113 ceph-mon[34341]: 4.16 deep-scrub ok 2024-03-30T13:53:00.755 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:00.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:00.782 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:00.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:00.883 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:00.920 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:00.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:00.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:01.022 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:01.057 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:01.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:01.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:01.159 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:01.197 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:01.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:01.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:01.298 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:01.329 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:01.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:01.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:01.431 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:01.466 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:01.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:01.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:01.568 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:01.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:01.604 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:01.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:01.688 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 10 16 27903 27887 174.246 122.375 0.00460369 0.00568594 2024-03-30T13:53:01.705 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:01.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:01.731 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:01.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:01.826 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:01 smithi043 ceph-mon[42099]: 3.7 deep-scrub starts 2024-03-30T13:53:01.826 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:01 smithi043 ceph-mon[42099]: 3.7 deep-scrub ok 2024-03-30T13:53:01.827 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:01 smithi043 ceph-mon[29963]: 3.7 deep-scrub starts 2024-03-30T13:53:01.827 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:01 smithi043 ceph-mon[29963]: 3.7 deep-scrub ok 2024-03-30T13:53:01.831 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:01.864 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:01.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:01.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:01.966 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:01.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:01 smithi113 ceph-mon[34341]: 3.7 deep-scrub starts 2024-03-30T13:53:01.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:01 smithi113 ceph-mon[34341]: 3.7 deep-scrub ok 2024-03-30T13:53:02.001 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:02.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:02.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:02.103 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:02.126 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:02.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:02.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:02.228 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:02.264 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:02.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:02.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:02.366 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:02.404 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:02.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:02.504 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:02.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:02.543 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:02.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:02.595 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-03-30T13:53:02.644 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:02.667 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:02.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:02.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:02.687 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 11 16 29699 29683 168.61 112.25 0.00283547 0.00589424 2024-03-30T13:53:02.769 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:02.782 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:53:02.782 INFO:teuthology.orchestra.run.smithi113.stdout: "size": 20, 2024-03-30T13:53:02.782 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 600, 2024-03-30T13:53:02.782 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [ 2024-03-30T13:53:02.782 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.782 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:145825 7.5 7:a688f4b0:::benchmark_data_smithi043_86989_object23487:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-30T13:53:02.782 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:55.513638+0000", 2024-03-30T13:53:02.782 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 366.16183967799998, 2024-03-30T13:53:02.782 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.155179499, 2024-03-30T13:53:02.782 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:53:02.783 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:53:02.783 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:02.783 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:53:02.783 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:53:02.783 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:53:02.783 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 145825 2024-03-30T13:53:02.783 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.783 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:53:02.783 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.783 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:53:02.783 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.513638+0000", 2024-03-30T13:53:02.783 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.783 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.783 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.784 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:53:02.784 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.513638+0000", 2024-03-30T13:53:02.784 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.784 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.784 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.784 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:53:02.784 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.513639+0000", 2024-03-30T13:53:02.784 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1790000000000001e-06 2024-03-30T13:53:02.784 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.784 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.784 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:53:02.784 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.513644+0000", 2024-03-30T13:53:02.784 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2460000000000003e-06 2024-03-30T13:53:02.784 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.785 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.785 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:53:02.785 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.513656+0000", 2024-03-30T13:53:02.785 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1983e-05 2024-03-30T13:53:02.785 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.785 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.785 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:53:02.785 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.513663+0000", 2024-03-30T13:53:02.785 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8700000000000003e-06 2024-03-30T13:53:02.785 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.785 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.785 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.785 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.513718+0000", 2024-03-30T13:53:02.786 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4533000000000001e-05 2024-03-30T13:53:02.786 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.786 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.786 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:53:02.786 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.513852+0000", 2024-03-30T13:53:02.786 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013423100000000001 2024-03-30T13:53:02.786 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.786 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.786 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:53:02.786 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.513986+0000", 2024-03-30T13:53:02.786 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000133397 2024-03-30T13:53:02.786 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.786 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.786 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:53:02.786 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.516904+0000", 2024-03-30T13:53:02.787 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0029180730000000002 2024-03-30T13:53:02.787 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.787 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.787 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.787 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.517328+0000", 2024-03-30T13:53:02.787 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00042470099999999998 2024-03-30T13:53:02.787 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.787 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.787 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.787 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.668714+0000", 2024-03-30T13:53:02.787 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15138600599999999 2024-03-30T13:53:02.787 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.787 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.787 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:53:02.787 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.668767+0000", 2024-03-30T13:53:02.788 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2853999999999999e-05 2024-03-30T13:53:02.788 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.788 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.788 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:53:02.788 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.668818+0000", 2024-03-30T13:53:02.788 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0426000000000001e-05 2024-03-30T13:53:02.788 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.788 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.788 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.788 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.788 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.788 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:184866 7.5 7:a99b6bae:::benchmark_data_smithi043_86989_object62528:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-30T13:53:02.788 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:36.958864+0000", 2024-03-30T13:53:02.788 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 324.716614155, 2024-03-30T13:53:02.788 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15833154399999999, 2024-03-30T13:53:02.789 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:53:02.789 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:53:02.789 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:02.789 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:53:02.789 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:53:02.789 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:53:02.789 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 184866 2024-03-30T13:53:02.789 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.789 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:53:02.789 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.789 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:53:02.789 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.958864+0000", 2024-03-30T13:53:02.789 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.790 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.790 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.790 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:53:02.790 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.958864+0000", 2024-03-30T13:53:02.790 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.790 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.790 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.790 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:53:02.790 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.958865+0000", 2024-03-30T13:53:02.790 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.345e-06 2024-03-30T13:53:02.790 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.790 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.790 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:53:02.791 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.958870+0000", 2024-03-30T13:53:02.791 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8049999999999997e-06 2024-03-30T13:53:02.791 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.791 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.791 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:53:02.791 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.958881+0000", 2024-03-30T13:53:02.791 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1724e-05 2024-03-30T13:53:02.791 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.791 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.791 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:53:02.791 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.958889+0000", 2024-03-30T13:53:02.791 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.5109999999999997e-06 2024-03-30T13:53:02.791 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.792 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.792 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.792 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.009980+0000", 2024-03-30T13:53:02.792 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.051090871000000003 2024-03-30T13:53:02.792 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.792 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.792 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:53:02.792 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.010096+0000", 2024-03-30T13:53:02.792 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011582300000000001 2024-03-30T13:53:02.792 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.792 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.792 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:53:02.792 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.010173+0000", 2024-03-30T13:53:02.792 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.7374999999999997e-05 2024-03-30T13:53:02.792 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.793 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.793 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:53:02.793 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.015369+0000", 2024-03-30T13:53:02.793 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0051959789999999999 2024-03-30T13:53:02.793 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.793 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.793 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.793 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.048919+0000", 2024-03-30T13:53:02.793 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.033550096000000001 2024-03-30T13:53:02.793 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.793 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.793 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.793 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.117068+0000", 2024-03-30T13:53:02.793 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.068148743999999997 2024-03-30T13:53:02.793 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.794 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.794 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:53:02.794 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.117131+0000", 2024-03-30T13:53:02.794 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3268000000000005e-05 2024-03-30T13:53:02.794 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.794 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.794 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:53:02.794 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.117195+0000", 2024-03-30T13:53:02.794 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.4003000000000004e-05 2024-03-30T13:53:02.794 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.794 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.794 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.794 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.795 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.795 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:184867 7.5 7:a079a26e:::benchmark_data_smithi043_86989_object62529:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-30T13:53:02.795 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:36.961802+0000", 2024-03-30T13:53:02.795 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 324.71367547, 2024-03-30T13:53:02.795 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15550554899999999, 2024-03-30T13:53:02.795 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:53:02.795 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:53:02.795 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:02.795 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:53:02.795 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:53:02.795 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:53:02.795 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 184867 2024-03-30T13:53:02.795 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.795 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:53:02.796 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.796 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:53:02.796 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.961802+0000", 2024-03-30T13:53:02.796 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.796 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.796 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.796 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:53:02.796 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.961802+0000", 2024-03-30T13:53:02.796 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.796 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.796 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.796 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:53:02.796 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.961804+0000", 2024-03-30T13:53:02.797 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3170000000000001e-06 2024-03-30T13:53:02.797 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.797 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.797 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:53:02.797 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.961807+0000", 2024-03-30T13:53:02.797 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.6909999999999999e-06 2024-03-30T13:53:02.797 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.797 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.797 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:53:02.797 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.961817+0000", 2024-03-30T13:53:02.797 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.5349999999999993e-06 2024-03-30T13:53:02.797 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.797 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.797 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:53:02.798 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:36.961824+0000", 2024-03-30T13:53:02.798 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.7340000000000002e-06 2024-03-30T13:53:02.798 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.798 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.798 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.798 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.012549+0000", 2024-03-30T13:53:02.798 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.050725943000000003 2024-03-30T13:53:02.798 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.798 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.798 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:53:02.798 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.012671+0000", 2024-03-30T13:53:02.798 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000121777 2024-03-30T13:53:02.798 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.799 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.799 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:53:02.799 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.012803+0000", 2024-03-30T13:53:02.799 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013154 2024-03-30T13:53:02.799 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.799 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.799 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:53:02.799 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.015537+0000", 2024-03-30T13:53:02.799 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.002733709 2024-03-30T13:53:02.799 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.799 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.799 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.799 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.058704+0000", 2024-03-30T13:53:02.800 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.043167533000000001 2024-03-30T13:53:02.800 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.800 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.800 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.800 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.117242+0000", 2024-03-30T13:53:02.800 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.058537759000000002 2024-03-30T13:53:02.800 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.800 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.800 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:53:02.800 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.117276+0000", 2024-03-30T13:53:02.800 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.4131000000000002e-05 2024-03-30T13:53:02.800 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.801 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.801 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:53:02.801 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:37.117308+0000", 2024-03-30T13:53:02.801 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.188e-05 2024-03-30T13:53:02.801 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.801 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.801 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.801 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.801 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.801 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:186577 7.5 7:a290ff28:::benchmark_data_smithi043_86989_object64239:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-30T13:53:02.801 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:38.859881+0000", 2024-03-30T13:53:02.801 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 322.81559670600001, 2024-03-30T13:53:02.801 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.16084641199999999, 2024-03-30T13:53:02.802 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:53:02.802 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:53:02.802 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:02.802 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:53:02.802 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:53:02.802 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:53:02.802 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 186577 2024-03-30T13:53:02.802 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.802 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:53:02.802 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.802 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:53:02.802 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.859881+0000", 2024-03-30T13:53:02.802 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.802 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.803 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.803 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:53:02.803 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.859881+0000", 2024-03-30T13:53:02.803 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.803 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.803 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.803 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:53:02.803 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.859882+0000", 2024-03-30T13:53:02.803 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.119e-06 2024-03-30T13:53:02.803 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.803 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.803 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:53:02.803 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.859887+0000", 2024-03-30T13:53:02.804 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0499999999999999e-06 2024-03-30T13:53:02.804 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.804 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.804 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:53:02.804 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.859898+0000", 2024-03-30T13:53:02.804 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0665e-05 2024-03-30T13:53:02.804 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.804 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.804 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:53:02.804 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.859907+0000", 2024-03-30T13:53:02.804 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6799999999999999e-06 2024-03-30T13:53:02.804 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.804 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.804 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.805 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.900822+0000", 2024-03-30T13:53:02.805 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.040915002999999998 2024-03-30T13:53:02.805 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.805 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.805 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:53:02.805 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.900937+0000", 2024-03-30T13:53:02.805 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000115683 2024-03-30T13:53:02.805 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.805 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.805 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:53:02.805 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.901067+0000", 2024-03-30T13:53:02.805 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000130242 2024-03-30T13:53:02.805 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.805 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.805 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:53:02.806 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.905083+0000", 2024-03-30T13:53:02.806 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0040151739999999998 2024-03-30T13:53:02.806 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.806 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.806 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.806 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:38.943163+0000", 2024-03-30T13:53:02.806 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.038080073999999998 2024-03-30T13:53:02.806 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.806 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.806 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.806 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.020632+0000", 2024-03-30T13:53:02.806 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.077468843999999995 2024-03-30T13:53:02.807 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.807 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.807 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:53:02.807 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.020669+0000", 2024-03-30T13:53:02.807 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.7403000000000001e-05 2024-03-30T13:53:02.807 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.807 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.807 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:53:02.807 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.020727+0000", 2024-03-30T13:53:02.807 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8474999999999998e-05 2024-03-30T13:53:02.807 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.807 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.808 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.808 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.808 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.808 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:186884 7.5 7:a73a07c4:::benchmark_data_smithi043_86989_object64546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-30T13:53:02.808 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.217916+0000", 2024-03-30T13:53:02.808 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 322.45756171300002, 2024-03-30T13:53:02.808 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.22441243499999999, 2024-03-30T13:53:02.808 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:53:02.808 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:53:02.808 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:02.808 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:53:02.808 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:53:02.809 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:53:02.809 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 186884 2024-03-30T13:53:02.809 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.809 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:53:02.809 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.809 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:53:02.809 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.217916+0000", 2024-03-30T13:53:02.809 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.809 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.809 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.809 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:53:02.809 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.217916+0000", 2024-03-30T13:53:02.809 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.810 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.810 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.810 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:53:02.810 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.217917+0000", 2024-03-30T13:53:02.810 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.0100000000000003e-07 2024-03-30T13:53:02.810 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.810 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.810 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:53:02.810 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.217921+0000", 2024-03-30T13:53:02.810 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.997e-06 2024-03-30T13:53:02.810 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.810 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.811 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:53:02.811 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.217930+0000", 2024-03-30T13:53:02.811 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.5289999999999997e-06 2024-03-30T13:53:02.811 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.811 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.811 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:53:02.811 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.217936+0000", 2024-03-30T13:53:02.811 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1420000000000001e-06 2024-03-30T13:53:02.811 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.811 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.811 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.811 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.217979+0000", 2024-03-30T13:53:02.811 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.2970999999999999e-05 2024-03-30T13:53:02.812 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.812 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.812 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:53:02.812 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.218077+0000", 2024-03-30T13:53:02.812 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.8084999999999993e-05 2024-03-30T13:53:02.812 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.812 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.812 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:53:02.812 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.218150+0000", 2024-03-30T13:53:02.812 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.3424999999999996e-05 2024-03-30T13:53:02.812 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.812 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.812 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:53:02.812 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.222912+0000", 2024-03-30T13:53:02.813 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.004761927 2024-03-30T13:53:02.813 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.813 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.813 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.813 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.241362+0000", 2024-03-30T13:53:02.813 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.018450139000000001 2024-03-30T13:53:02.813 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.813 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.813 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.813 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442214+0000", 2024-03-30T13:53:02.813 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.200851737 2024-03-30T13:53:02.813 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.813 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.814 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:53:02.814 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442271+0000", 2024-03-30T13:53:02.814 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6712999999999997e-05 2024-03-30T13:53:02.814 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.814 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.814 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:53:02.814 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442328+0000", 2024-03-30T13:53:02.814 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7868999999999998e-05 2024-03-30T13:53:02.814 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.814 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.814 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.814 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.814 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.815 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:186886 7.5 7:a2cfd65b:::benchmark_data_smithi043_86989_object64548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-30T13:53:02.815 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.219014+0000", 2024-03-30T13:53:02.815 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 322.45646357499999, 2024-03-30T13:53:02.815 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.22343902400000001, 2024-03-30T13:53:02.815 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:53:02.815 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:53:02.815 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:02.815 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:53:02.815 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:53:02.815 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:53:02.815 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 186886 2024-03-30T13:53:02.815 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.815 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:53:02.816 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.816 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:53:02.816 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.219014+0000", 2024-03-30T13:53:02.816 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.816 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.816 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.816 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:53:02.816 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.219014+0000", 2024-03-30T13:53:02.816 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.816 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.816 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.816 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:53:02.816 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.219015+0000", 2024-03-30T13:53:02.816 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.06e-07 2024-03-30T13:53:02.817 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.817 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.817 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:53:02.817 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.219019+0000", 2024-03-30T13:53:02.817 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.464e-06 2024-03-30T13:53:02.817 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.817 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.817 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:53:02.817 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.219028+0000", 2024-03-30T13:53:02.817 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.9260000000000006e-06 2024-03-30T13:53:02.817 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.817 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.817 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:53:02.817 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.219035+0000", 2024-03-30T13:53:02.818 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6660000000000002e-06 2024-03-30T13:53:02.818 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.818 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.818 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.818 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.220296+0000", 2024-03-30T13:53:02.818 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0012610950000000001 2024-03-30T13:53:02.818 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.818 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.818 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:53:02.818 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.220412+0000", 2024-03-30T13:53:02.818 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000116413 2024-03-30T13:53:02.818 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.818 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.819 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:53:02.819 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.220542+0000", 2024-03-30T13:53:02.819 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012981700000000001 2024-03-30T13:53:02.819 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.819 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.819 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:53:02.819 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.223017+0000", 2024-03-30T13:53:02.819 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0024748909999999999 2024-03-30T13:53:02.819 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.819 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.819 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.819 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.241404+0000", 2024-03-30T13:53:02.819 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.018386848000000001 2024-03-30T13:53:02.819 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.820 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.820 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.820 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442373+0000", 2024-03-30T13:53:02.820 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.20096920300000001 2024-03-30T13:53:02.820 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.820 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.820 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:53:02.820 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442406+0000", 2024-03-30T13:53:02.820 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2446000000000001e-05 2024-03-30T13:53:02.820 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.820 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.820 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:53:02.820 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442453+0000", 2024-03-30T13:53:02.820 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7549000000000002e-05 2024-03-30T13:53:02.820 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.820 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.821 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.821 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.821 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.821 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:186896 7.5 7:a006897a:::benchmark_data_smithi043_86989_object64558:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-30T13:53:02.821 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.232075+0000", 2024-03-30T13:53:02.821 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 322.44340231500001, 2024-03-30T13:53:02.821 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.21065776999999999, 2024-03-30T13:53:02.821 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:53:02.821 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:53:02.821 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:02.821 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:53:02.821 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:53:02.821 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:53:02.821 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 186896 2024-03-30T13:53:02.821 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.821 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:53:02.822 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.822 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:53:02.822 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.232075+0000", 2024-03-30T13:53:02.822 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.822 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.822 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.822 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:53:02.822 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.232075+0000", 2024-03-30T13:53:02.822 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.822 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.822 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.822 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:53:02.822 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.232076+0000", 2024-03-30T13:53:02.822 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0410000000000001e-06 2024-03-30T13:53:02.822 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.822 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.823 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:53:02.823 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.232081+0000", 2024-03-30T13:53:02.823 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.2980000000000002e-06 2024-03-30T13:53:02.823 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.823 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.823 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:53:02.823 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.232090+0000", 2024-03-30T13:53:02.823 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.952e-06 2024-03-30T13:53:02.823 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.823 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.823 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:53:02.823 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.232095+0000", 2024-03-30T13:53:02.823 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2560000000000002e-06 2024-03-30T13:53:02.823 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.824 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.824 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.824 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.232144+0000", 2024-03-30T13:53:02.824 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9450999999999998e-05 2024-03-30T13:53:02.824 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.824 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.824 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:53:02.824 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.232248+0000", 2024-03-30T13:53:02.824 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000103526 2024-03-30T13:53:02.824 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.824 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.824 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:53:02.824 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.232333+0000", 2024-03-30T13:53:02.824 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5550000000000006e-05 2024-03-30T13:53:02.824 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.825 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.825 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:53:02.825 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.234450+0000", 2024-03-30T13:53:02.825 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0021169990000000001 2024-03-30T13:53:02.825 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.825 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.825 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.825 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.241435+0000", 2024-03-30T13:53:02.825 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0069840379999999997 2024-03-30T13:53:02.825 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.825 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.825 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.825 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442614+0000", 2024-03-30T13:53:02.826 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.20117961100000001 2024-03-30T13:53:02.826 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.826 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.826 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:53:02.826 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442675+0000", 2024-03-30T13:53:02.826 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0401e-05 2024-03-30T13:53:02.826 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.826 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.826 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:53:02.826 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442733+0000", 2024-03-30T13:53:02.826 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8647000000000003e-05 2024-03-30T13:53:02.826 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.826 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.827 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.827 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.827 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.827 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:186905 7.5 7:ab740115:::benchmark_data_smithi043_86989_object64567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-30T13:53:02.827 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.242589+0000", 2024-03-30T13:53:02.827 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 322.432889042, 2024-03-30T13:53:02.827 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.200289468, 2024-03-30T13:53:02.827 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:53:02.827 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:53:02.827 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:02.827 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:53:02.827 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:53:02.827 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:53:02.827 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 186905 2024-03-30T13:53:02.828 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.828 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:53:02.828 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.828 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:53:02.828 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.242589+0000", 2024-03-30T13:53:02.828 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.828 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.828 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.828 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:53:02.828 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.242589+0000", 2024-03-30T13:53:02.828 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.828 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.828 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.828 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:53:02.828 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.242590+0000", 2024-03-30T13:53:02.829 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.091e-06 2024-03-30T13:53:02.829 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.829 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.829 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:53:02.829 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.242595+0000", 2024-03-30T13:53:02.829 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8840000000000002e-06 2024-03-30T13:53:02.829 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.829 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.829 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:53:02.829 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.242606+0000", 2024-03-30T13:53:02.829 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1384e-05 2024-03-30T13:53:02.829 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.829 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.830 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:53:02.830 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.242612+0000", 2024-03-30T13:53:02.830 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7699999999999998e-06 2024-03-30T13:53:02.830 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.830 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.830 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.830 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.242660+0000", 2024-03-30T13:53:02.830 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8554999999999998e-05 2024-03-30T13:53:02.830 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.830 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.830 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:53:02.830 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.242789+0000", 2024-03-30T13:53:02.830 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000128918 2024-03-30T13:53:02.830 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.831 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.831 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:53:02.831 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.242917+0000", 2024-03-30T13:53:02.831 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012774100000000001 2024-03-30T13:53:02.831 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.831 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.831 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:53:02.831 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245127+0000", 2024-03-30T13:53:02.831 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0022098980000000001 2024-03-30T13:53:02.831 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.831 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.831 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.831 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.249020+0000", 2024-03-30T13:53:02.832 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0038933969999999998 2024-03-30T13:53:02.832 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.832 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.832 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.832 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442788+0000", 2024-03-30T13:53:02.832 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.19376775800000001 2024-03-30T13:53:02.832 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.832 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.832 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:53:02.832 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442835+0000", 2024-03-30T13:53:02.832 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6813000000000001e-05 2024-03-30T13:53:02.832 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.832 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.832 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:53:02.833 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442878+0000", 2024-03-30T13:53:02.833 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3259e-05 2024-03-30T13:53:02.833 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.833 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.833 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.833 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.833 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.833 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:186909 7.5 7:ac8b9ff7:::benchmark_data_smithi043_86989_object64571:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-30T13:53:02.833 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.245217+0000", 2024-03-30T13:53:02.833 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 322.43026044499999, 2024-03-30T13:53:02.833 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.19782831200000001, 2024-03-30T13:53:02.833 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:53:02.833 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:53:02.833 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:02.834 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:53:02.834 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:53:02.834 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:53:02.834 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 186909 2024-03-30T13:53:02.834 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.834 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:53:02.834 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.834 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:53:02.834 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245217+0000", 2024-03-30T13:53:02.834 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.834 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.834 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.834 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:53:02.835 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245217+0000", 2024-03-30T13:53:02.835 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.835 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.835 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.835 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:53:02.835 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245228+0000", 2024-03-30T13:53:02.835 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.042e-05 2024-03-30T13:53:02.835 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.835 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.835 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:53:02.835 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245232+0000", 2024-03-30T13:53:02.835 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7550000000000002e-06 2024-03-30T13:53:02.835 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.836 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.836 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:53:02.836 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245243+0000", 2024-03-30T13:53:02.836 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0342e-05 2024-03-30T13:53:02.836 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.836 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.836 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:53:02.836 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245248+0000", 2024-03-30T13:53:02.836 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.592e-06 2024-03-30T13:53:02.836 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.836 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.836 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.836 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245284+0000", 2024-03-30T13:53:02.836 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.5901000000000001e-05 2024-03-30T13:53:02.837 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.837 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.837 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:53:02.837 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245362+0000", 2024-03-30T13:53:02.837 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.7622999999999999e-05 2024-03-30T13:53:02.837 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.837 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.837 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:53:02.837 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245437+0000", 2024-03-30T13:53:02.837 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.4944999999999995e-05 2024-03-30T13:53:02.837 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.837 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.837 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:53:02.837 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.248971+0000", 2024-03-30T13:53:02.838 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0035339450000000001 2024-03-30T13:53:02.838 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.838 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.838 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.838 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.251184+0000", 2024-03-30T13:53:02.838 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.002213095 2024-03-30T13:53:02.838 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.838 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.838 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.838 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442940+0000", 2024-03-30T13:53:02.838 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.19175640099999999 2024-03-30T13:53:02.838 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.838 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.839 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:53:02.839 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442991+0000", 2024-03-30T13:53:02.839 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0535999999999999e-05 2024-03-30T13:53:02.839 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.839 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.839 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:53:02.839 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443046+0000", 2024-03-30T13:53:02.839 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4756999999999998e-05 2024-03-30T13:53:02.839 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.839 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.839 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.839 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.839 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.839 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:186910 7.5 7:a7b0943c:::benchmark_data_smithi043_86989_object64572:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-30T13:53:02.840 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.245266+0000", 2024-03-30T13:53:02.840 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 322.43021206999998, 2024-03-30T13:53:02.840 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.19788240700000001, 2024-03-30T13:53:02.840 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:53:02.840 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:53:02.840 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:02.840 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:53:02.840 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:53:02.840 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:53:02.841 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 186910 2024-03-30T13:53:02.842 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.842 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:53:02.842 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.842 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:53:02.842 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245266+0000", 2024-03-30T13:53:02.842 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.842 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.842 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.842 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:53:02.842 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245266+0000", 2024-03-30T13:53:02.842 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.842 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.843 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.843 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:53:02.843 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245266+0000", 2024-03-30T13:53:02.843 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.96e-07 2024-03-30T13:53:02.843 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.843 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.843 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:53:02.843 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245268+0000", 2024-03-30T13:53:02.843 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.9700000000000002e-06 2024-03-30T13:53:02.843 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.843 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.843 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:53:02.843 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245273+0000", 2024-03-30T13:53:02.844 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1279999999999999e-06 2024-03-30T13:53:02.844 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.844 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.844 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:53:02.844 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.245277+0000", 2024-03-30T13:53:02.844 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.0559999999999998e-06 2024-03-30T13:53:02.844 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.844 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.844 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.844 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.247136+0000", 2024-03-30T13:53:02.844 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0018592789999999999 2024-03-30T13:53:02.844 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.845 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.845 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:53:02.845 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.247232+0000", 2024-03-30T13:53:02.845 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.5748000000000006e-05 2024-03-30T13:53:02.845 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.845 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.845 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:53:02.845 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.247318+0000", 2024-03-30T13:53:02.845 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5373999999999999e-05 2024-03-30T13:53:02.845 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.845 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.845 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:53:02.845 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.249088+0000", 2024-03-30T13:53:02.845 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001770766 2024-03-30T13:53:02.845 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.846 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.846 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.846 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.252085+0000", 2024-03-30T13:53:02.846 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0029968500000000001 2024-03-30T13:53:02.846 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.846 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.846 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.846 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443081+0000", 2024-03-30T13:53:02.846 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.19099596399999999 2024-03-30T13:53:02.846 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.846 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.846 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:53:02.846 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443110+0000", 2024-03-30T13:53:02.847 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.8690000000000001e-05 2024-03-30T13:53:02.847 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.847 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.847 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:53:02.847 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443148+0000", 2024-03-30T13:53:02.847 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.8186000000000003e-05 2024-03-30T13:53:02.847 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.847 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.847 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.847 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.847 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.847 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:186926 7.5 7:a7188cf2:::benchmark_data_smithi043_86989_object64588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-30T13:53:02.847 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.267148+0000", 2024-03-30T13:53:02.847 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 322.40833017099999, 2024-03-30T13:53:02.847 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.176160026, 2024-03-30T13:53:02.848 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:53:02.848 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:53:02.848 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:02.848 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:53:02.848 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:53:02.848 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:53:02.848 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 186926 2024-03-30T13:53:02.848 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.848 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:53:02.848 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.848 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:53:02.848 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.267148+0000", 2024-03-30T13:53:02.848 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.848 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.848 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.849 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:53:02.849 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.267148+0000", 2024-03-30T13:53:02.849 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.849 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.849 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.849 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:53:02.849 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.267149+0000", 2024-03-30T13:53:02.849 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1990000000000001e-06 2024-03-30T13:53:02.849 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.849 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.849 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:53:02.849 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.267153+0000", 2024-03-30T13:53:02.849 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6619999999999996e-06 2024-03-30T13:53:02.849 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.850 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.850 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:53:02.850 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.267164+0000", 2024-03-30T13:53:02.850 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0678e-05 2024-03-30T13:53:02.850 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.850 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.850 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:53:02.850 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.267171+0000", 2024-03-30T13:53:02.850 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.9999999999999999e-06 2024-03-30T13:53:02.850 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.850 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.850 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.850 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.267208+0000", 2024-03-30T13:53:02.850 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.6990000000000003e-05 2024-03-30T13:53:02.851 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.851 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.851 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:53:02.851 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.267317+0000", 2024-03-30T13:53:02.851 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000109308 2024-03-30T13:53:02.851 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.851 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.851 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:53:02.851 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.267412+0000", 2024-03-30T13:53:02.851 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4116000000000005e-05 2024-03-30T13:53:02.851 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.851 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.851 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:53:02.851 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.269599+0000", 2024-03-30T13:53:02.851 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.002187128 2024-03-30T13:53:02.852 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.852 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.852 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.852 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.270659+0000", 2024-03-30T13:53:02.852 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0010600189999999999 2024-03-30T13:53:02.852 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.852 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.852 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.852 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443194+0000", 2024-03-30T13:53:02.852 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.172535569 2024-03-30T13:53:02.852 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.852 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.852 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:53:02.852 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443253+0000", 2024-03-30T13:53:02.852 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8888000000000003e-05 2024-03-30T13:53:02.853 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.853 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.853 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:53:02.853 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443308+0000", 2024-03-30T13:53:02.853 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4469000000000003e-05 2024-03-30T13:53:02.853 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.853 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.853 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.853 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.853 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.853 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop_reply(client.14817.0:186884 7.5 e131/98 ondisk, result = 0)", 2024-03-30T13:53:02.853 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.288030+0000", 2024-03-30T13:53:02.853 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 322.38744777300002, 2024-03-30T13:53:02.853 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.154306057, 2024-03-30T13:53:02.854 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:53:02.854 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:53:02.854 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:53:02.854 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:53:02.854 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.854 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:53:02.854 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.288030+0000", 2024-03-30T13:53:02.854 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.854 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.854 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.854 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:53:02.854 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.288030+0000", 2024-03-30T13:53:02.855 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.855 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.855 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.855 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:53:02.855 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.288031+0000", 2024-03-30T13:53:02.855 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6400000000000002e-07 2024-03-30T13:53:02.855 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.855 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.855 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:53:02.855 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.288035+0000", 2024-03-30T13:53:02.855 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3930000000000001e-06 2024-03-30T13:53:02.855 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.855 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.855 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:53:02.856 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.288046+0000", 2024-03-30T13:53:02.856 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0811e-05 2024-03-30T13:53:02.856 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.856 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.856 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:53:02.856 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.288051+0000", 2024-03-30T13:53:02.856 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2270000000000001e-06 2024-03-30T13:53:02.856 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.856 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.856 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.856 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442192+0000", 2024-03-30T13:53:02.856 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.154141159 2024-03-30T13:53:02.856 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.857 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.857 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:53:02.857 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442201+0000", 2024-03-30T13:53:02.857 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.3249999999999997e-06 2024-03-30T13:53:02.857 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.857 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.857 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:53:02.857 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442336+0000", 2024-03-30T13:53:02.857 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000134578 2024-03-30T13:53:02.857 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.857 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.857 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.857 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.857 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.857 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop_reply(client.14817.0:186896 7.5 e131/98 ondisk, result = 0)", 2024-03-30T13:53:02.858 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.291749+0000", 2024-03-30T13:53:02.858 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 322.38372871500002, 2024-03-30T13:53:02.858 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15099047900000001, 2024-03-30T13:53:02.858 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:53:02.858 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:53:02.858 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:53:02.858 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:53:02.858 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.858 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:53:02.858 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291749+0000", 2024-03-30T13:53:02.858 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.858 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.858 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.859 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:53:02.859 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291749+0000", 2024-03-30T13:53:02.859 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.859 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.859 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.859 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:53:02.859 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291749+0000", 2024-03-30T13:53:02.859 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.8799999999999998e-07 2024-03-30T13:53:02.859 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.859 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.859 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:53:02.859 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291752+0000", 2024-03-30T13:53:02.859 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.9179999999999998e-06 2024-03-30T13:53:02.859 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.860 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.860 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:53:02.860 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291759+0000", 2024-03-30T13:53:02.860 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5690000000000001e-06 2024-03-30T13:53:02.860 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.860 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.860 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:53:02.860 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291764+0000", 2024-03-30T13:53:02.860 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9030000000000003e-06 2024-03-30T13:53:02.860 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.860 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.860 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.860 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442598+0000", 2024-03-30T13:53:02.860 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15083411999999999 2024-03-30T13:53:02.861 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.861 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.861 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:53:02.861 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442607+0000", 2024-03-30T13:53:02.861 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.3000000000000007e-06 2024-03-30T13:53:02.861 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.861 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.861 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:53:02.861 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442740+0000", 2024-03-30T13:53:02.861 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013228099999999999 2024-03-30T13:53:02.861 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.861 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.861 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.861 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.861 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.862 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop_reply(client.14817.0:186905 7.5 e131/98 ondisk, result = 0)", 2024-03-30T13:53:02.862 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.291773+0000", 2024-03-30T13:53:02.862 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 322.38370488700002, 2024-03-30T13:53:02.862 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15111850800000001, 2024-03-30T13:53:02.862 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:53:02.862 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:53:02.862 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:53:02.862 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:53:02.862 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.862 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:53:02.862 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291773+0000", 2024-03-30T13:53:02.862 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.862 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.863 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.863 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:53:02.863 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291773+0000", 2024-03-30T13:53:02.863 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.863 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.863 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.863 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:53:02.863 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291773+0000", 2024-03-30T13:53:02.863 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.28e-07 2024-03-30T13:53:02.863 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.863 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.863 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:53:02.863 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291775+0000", 2024-03-30T13:53:02.863 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.26e-06 2024-03-30T13:53:02.864 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.864 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.864 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:53:02.864 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291781+0000", 2024-03-30T13:53:02.864 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6330000000000001e-06 2024-03-30T13:53:02.864 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.864 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.864 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:53:02.864 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291788+0000", 2024-03-30T13:53:02.864 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1849999999999998e-06 2024-03-30T13:53:02.864 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.864 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.864 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.864 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442773+0000", 2024-03-30T13:53:02.864 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15098513 2024-03-30T13:53:02.865 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.865 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.865 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:53:02.865 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442782+0000", 2024-03-30T13:53:02.865 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5839999999999995e-06 2024-03-30T13:53:02.865 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.865 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.865 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:53:02.865 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442891+0000", 2024-03-30T13:53:02.865 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00010948800000000001 2024-03-30T13:53:02.865 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.865 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.866 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.866 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.866 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.866 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop_reply(client.14817.0:186909 7.5 e131/98 ondisk, result = 0)", 2024-03-30T13:53:02.866 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.291831+0000", 2024-03-30T13:53:02.866 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 322.38364718999998, 2024-03-30T13:53:02.866 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15122132399999999, 2024-03-30T13:53:02.866 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:53:02.866 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:53:02.866 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:53:02.866 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:53:02.866 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.867 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:53:02.867 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291831+0000", 2024-03-30T13:53:02.867 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.867 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.867 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.867 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:53:02.867 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291831+0000", 2024-03-30T13:53:02.867 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.867 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.867 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.867 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:53:02.867 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291831+0000", 2024-03-30T13:53:02.867 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.41e-07 2024-03-30T13:53:02.867 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.868 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.868 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:53:02.868 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291834+0000", 2024-03-30T13:53:02.868 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.7099999999999999e-06 2024-03-30T13:53:02.868 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.868 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.868 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:53:02.868 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291841+0000", 2024-03-30T13:53:02.868 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.43e-06 2024-03-30T13:53:02.868 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.868 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.868 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:53:02.868 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291847+0000", 2024-03-30T13:53:02.868 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5280000000000003e-06 2024-03-30T13:53:02.869 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.869 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.869 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.869 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442927+0000", 2024-03-30T13:53:02.869 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15108020599999999 2024-03-30T13:53:02.869 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.869 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.869 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:53:02.869 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.442934+0000", 2024-03-30T13:53:02.869 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.6869999999999994e-06 2024-03-30T13:53:02.869 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.869 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.869 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:53:02.870 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443052+0000", 2024-03-30T13:53:02.870 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000117422 2024-03-30T13:53:02.870 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.870 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.870 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.870 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.870 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.870 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop_reply(client.14817.0:186910 7.5 e131/98 ondisk, result = 0)", 2024-03-30T13:53:02.870 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.291937+0000", 2024-03-30T13:53:02.870 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 322.38354059400001, 2024-03-30T13:53:02.870 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.151214919, 2024-03-30T13:53:02.870 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:53:02.870 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:53:02.871 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:53:02.871 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:53:02.871 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.871 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:53:02.871 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291937+0000", 2024-03-30T13:53:02.871 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.871 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.871 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.871 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:53:02.871 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291937+0000", 2024-03-30T13:53:02.871 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.871 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.871 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.871 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:53:02.872 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291937+0000", 2024-03-30T13:53:02.872 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.9400000000000001e-07 2024-03-30T13:53:02.872 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.872 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.872 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:53:02.872 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291940+0000", 2024-03-30T13:53:02.872 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.4269999999999998e-06 2024-03-30T13:53:02.872 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.872 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.872 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:53:02.872 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291947+0000", 2024-03-30T13:53:02.872 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1019999999999999e-06 2024-03-30T13:53:02.872 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.873 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.873 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:53:02.873 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.291952+0000", 2024-03-30T13:53:02.873 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6750000000000001e-06 2024-03-30T13:53:02.873 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.873 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.873 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.873 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443074+0000", 2024-03-30T13:53:02.873 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15112245299999999 2024-03-30T13:53:02.873 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.873 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.873 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:53:02.873 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443078+0000", 2024-03-30T13:53:02.873 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.9840000000000003e-06 2024-03-30T13:53:02.874 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.874 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.874 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:53:02.874 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.443152+0000", 2024-03-30T13:53:02.874 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.3984000000000002e-05 2024-03-30T13:53:02.874 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.874 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.874 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.874 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.874 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.874 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:186951 7.5 7:a7e7bfd7:::benchmark_data_smithi043_86989_object64613:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-30T13:53:02.874 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.292317+0000", 2024-03-30T13:53:02.875 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 322.38316113799999, 2024-03-30T13:53:02.875 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.181689726, 2024-03-30T13:53:02.875 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:53:02.875 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:53:02.875 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:02.875 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:53:02.875 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:53:02.875 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:53:02.875 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 186951 2024-03-30T13:53:02.875 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.875 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:53:02.875 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.875 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:53:02.875 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.292317+0000", 2024-03-30T13:53:02.876 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.876 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.876 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.876 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:53:02.876 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.292317+0000", 2024-03-30T13:53:02.876 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.876 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.876 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.876 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:53:02.876 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.292318+0000", 2024-03-30T13:53:02.876 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.624e-06 2024-03-30T13:53:02.876 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.876 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.876 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:53:02.876 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.292323+0000", 2024-03-30T13:53:02.877 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.4150000000000001e-06 2024-03-30T13:53:02.877 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.877 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.877 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:53:02.877 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.292334+0000", 2024-03-30T13:53:02.877 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1192000000000001e-05 2024-03-30T13:53:02.877 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.877 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.877 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:53:02.877 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.292342+0000", 2024-03-30T13:53:02.877 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.7540000000000006e-06 2024-03-30T13:53:02.877 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.877 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.878 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.878 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.446331+0000", 2024-03-30T13:53:02.878 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.153989391 2024-03-30T13:53:02.878 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.878 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.878 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:53:02.878 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.446450+0000", 2024-03-30T13:53:02.878 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000118868 2024-03-30T13:53:02.878 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.878 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.878 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:53:02.878 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.446530+0000", 2024-03-30T13:53:02.878 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.0203000000000004e-05 2024-03-30T13:53:02.878 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.879 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.879 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:53:02.879 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.450029+0000", 2024-03-30T13:53:02.879 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00349863 2024-03-30T13:53:02.879 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.879 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.879 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.879 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.464588+0000", 2024-03-30T13:53:02.879 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.014559631999999999 2024-03-30T13:53:02.879 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.879 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.879 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.879 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.473921+0000", 2024-03-30T13:53:02.879 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0093331649999999992 2024-03-30T13:53:02.880 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.880 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.880 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:53:02.880 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.473961+0000", 2024-03-30T13:53:02.880 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.9443000000000002e-05 2024-03-30T13:53:02.880 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.880 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.880 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:53:02.880 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.474006+0000", 2024-03-30T13:53:02.880 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.5408999999999998e-05 2024-03-30T13:53:02.880 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.880 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.880 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.880 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.881 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.881 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:186962 7.5 7:a78141fa:::benchmark_data_smithi043_86989_object64624:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-30T13:53:02.881 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:39.303756+0000", 2024-03-30T13:53:02.881 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 322.37172148600001, 2024-03-30T13:53:02.881 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.17041271799999999, 2024-03-30T13:53:02.881 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:53:02.881 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:53:02.881 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:02.881 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:53:02.881 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:53:02.881 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:53:02.881 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 186962 2024-03-30T13:53:02.881 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.881 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:53:02.881 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.882 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:53:02.882 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.303756+0000", 2024-03-30T13:53:02.882 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.882 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.882 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.882 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:53:02.882 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.303756+0000", 2024-03-30T13:53:02.882 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.882 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.882 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.882 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:53:02.882 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.303758+0000", 2024-03-30T13:53:02.882 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3790000000000001e-06 2024-03-30T13:53:02.882 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.883 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.883 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:53:02.883 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.303762+0000", 2024-03-30T13:53:02.883 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7160000000000002e-06 2024-03-30T13:53:02.883 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.883 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.883 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:53:02.883 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.303774+0000", 2024-03-30T13:53:02.883 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1916000000000001e-05 2024-03-30T13:53:02.883 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.883 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.883 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:53:02.883 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.303783+0000", 2024-03-30T13:53:02.883 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.7760000000000003e-06 2024-03-30T13:53:02.884 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.884 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.884 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.884 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.448546+0000", 2024-03-30T13:53:02.884 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.14476262400000001 2024-03-30T13:53:02.884 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.884 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.884 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:53:02.884 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.448623+0000", 2024-03-30T13:53:02.884 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.7118000000000003e-05 2024-03-30T13:53:02.884 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.884 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.884 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:53:02.884 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.448692+0000", 2024-03-30T13:53:02.885 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.9321999999999998e-05 2024-03-30T13:53:02.885 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.885 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.885 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:53:02.885 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.454432+0000", 2024-03-30T13:53:02.885 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0057398090000000002 2024-03-30T13:53:02.885 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.885 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.885 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.885 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.464644+0000", 2024-03-30T13:53:02.885 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.010211981 2024-03-30T13:53:02.885 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.885 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.885 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.885 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.474058+0000", 2024-03-30T13:53:02.886 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0094136930000000008 2024-03-30T13:53:02.886 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.886 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.886 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:53:02.886 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.474113+0000", 2024-03-30T13:53:02.886 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5714999999999998e-05 2024-03-30T13:53:02.886 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.886 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.886 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:53:02.886 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:39.474169+0000", 2024-03-30T13:53:02.886 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5668999999999998e-05 2024-03-30T13:53:02.886 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.886 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.886 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.887 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.887 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.887 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:190128 7.5 7:ad64c2ee:::benchmark_data_smithi043_86989_object67790:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-30T13:53:02.887 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:47:43.187981+0000", 2024-03-30T13:53:02.887 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 318.487496927, 2024-03-30T13:53:02.887 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.156478691, 2024-03-30T13:53:02.887 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:53:02.887 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:53:02.887 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:02.887 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:53:02.887 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:53:02.887 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:53:02.887 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 190128 2024-03-30T13:53:02.887 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.888 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:53:02.888 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.888 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:53:02.888 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:43.187981+0000", 2024-03-30T13:53:02.888 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.888 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.888 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.888 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:53:02.888 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:43.187981+0000", 2024-03-30T13:53:02.888 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.888 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.888 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.888 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:53:02.888 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:43.187982+0000", 2024-03-30T13:53:02.888 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0899999999999999e-06 2024-03-30T13:53:02.889 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.889 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.889 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:53:02.889 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:43.187987+0000", 2024-03-30T13:53:02.889 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7360000000000001e-06 2024-03-30T13:53:02.889 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.889 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.889 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:53:02.889 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:43.187998+0000", 2024-03-30T13:53:02.889 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0892e-05 2024-03-30T13:53:02.889 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.889 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.889 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:53:02.889 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:43.188004+0000", 2024-03-30T13:53:02.889 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3849999999999999e-06 2024-03-30T13:53:02.890 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.890 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.890 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.890 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:43.210360+0000", 2024-03-30T13:53:02.890 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.022356279999999999 2024-03-30T13:53:02.890 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.890 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.890 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:53:02.890 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:43.210493+0000", 2024-03-30T13:53:02.890 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013317599999999999 2024-03-30T13:53:02.890 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.890 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.890 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2", 2024-03-30T13:53:02.890 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:43.210621+0000", 2024-03-30T13:53:02.891 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012767900000000001 2024-03-30T13:53:02.891 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.891 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.891 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:53:02.891 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:43.212181+0000", 2024-03-30T13:53:02.891 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001559749 2024-03-30T13:53:02.891 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.891 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.891 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.891 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:43.270311+0000", 2024-03-30T13:53:02.891 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.058130573999999997 2024-03-30T13:53:02.891 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.891 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.891 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:02.891 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:43.344384+0000", 2024-03-30T13:53:02.891 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.074073103000000001 2024-03-30T13:53:02.892 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.892 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.892 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:53:02.892 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:43.344419+0000", 2024-03-30T13:53:02.892 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.4425999999999999e-05 2024-03-30T13:53:02.892 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.892 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.892 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:53:02.892 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:47:43.344460+0000", 2024-03-30T13:53:02.892 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.0601e-05 2024-03-30T13:53:02.892 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.892 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.892 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.893 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.893 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.893 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14451.0:1770 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94764809514240 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-03-30T13:53:02.893 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:00.450514+0000", 2024-03-30T13:53:02.893 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 181.22496328700001, 2024-03-30T13:53:02.893 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0126963689999999, 2024-03-30T13:53:02.893 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:53:02.893 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:53:02.893 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:53:02.893 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:53:02.893 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14451", 2024-03-30T13:53:02.893 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T13:53:02.893 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 1770 2024-03-30T13:53:02.893 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.893 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:53:02.893 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.894 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:53:02.894 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.450514+0000", 2024-03-30T13:53:02.894 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.894 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.894 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.894 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:53:02.894 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.450514+0000", 2024-03-30T13:53:02.894 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:53:02.894 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.894 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.894 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:53:02.894 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.450516+0000", 2024-03-30T13:53:02.894 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3239999999999999e-06 2024-03-30T13:53:02.894 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.895 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.895 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:53:02.895 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.450522+0000", 2024-03-30T13:53:02.895 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.2349999999999996e-06 2024-03-30T13:53:02.895 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.895 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.895 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:53:02.895 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.450532+0000", 2024-03-30T13:53:02.895 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0258000000000001e-05 2024-03-30T13:53:02.895 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.895 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.895 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:53:02.895 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.450541+0000", 2024-03-30T13:53:02.895 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.1239999999999994e-06 2024-03-30T13:53:02.895 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.896 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.896 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.896 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.456718+0000", 2024-03-30T13:53:02.896 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0061767339999999997 2024-03-30T13:53:02.896 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.896 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.896 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T13:53:02.896 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:00.456727+0000", 2024-03-30T13:53:02.896 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.1190000000000003e-06 2024-03-30T13:53:02.896 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.896 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.896 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:53:02.896 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:01.460930+0000", 2024-03-30T13:53:02.896 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0042028730000001 2024-03-30T13:53:02.897 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.897 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.897 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:53:02.897 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:01.461105+0000", 2024-03-30T13:53:02.897 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00017471199999999999 2024-03-30T13:53:02.897 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:53:02.897 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:53:02.897 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:53:02.897 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:01.463211+0000", 2024-03-30T13:53:02.897 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0021059899999999999 2024-03-30T13:53:02.897 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.897 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.897 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.897 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:53:02.898 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:53:02.898 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:53:02.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:02 smithi043 ceph-mon[29963]: pgmap v427: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 1.3 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 41 KiB/s rd, 171 MiB/s wr, 2.81k op/s; 4059/66522 objects degraded (6.102%); 1/66522 objects misplaced (0.002%) 2024-03-30T13:53:02.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:02 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1474261311' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:53:02.899 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:02 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 4059/66522 objects degraded (6.102%), 15 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:02.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:02 smithi043 ceph-mon[42099]: pgmap v427: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 1.3 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 41 KiB/s rd, 171 MiB/s wr, 2.81k op/s; 4059/66522 objects degraded (6.102%); 1/66522 objects misplaced (0.002%) 2024-03-30T13:53:02.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:02 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1474261311' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:53:02.899 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:02 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 4059/66522 objects degraded (6.102%), 15 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:02.900 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:02.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:02.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:02 smithi113 ceph-mon[34341]: pgmap v427: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 1.3 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 41 KiB/s rd, 171 MiB/s wr, 2.81k op/s; 4059/66522 objects degraded (6.102%); 1/66522 objects misplaced (0.002%) 2024-03-30T13:53:02.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:02 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1474261311' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:53:02.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:02 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 4059/66522 objects degraded (6.102%), 15 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:03.001 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:03.040 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:03.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:03.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:03.142 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:03.177 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:03.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:03.278 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:03.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:03.317 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:03.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:03.418 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:03.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:03.454 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:03.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:03.555 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:03.591 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:03.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:03.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:03.688 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 12 16 31444 31428 163.648 109.062 0.00689309 0.0061018 2024-03-30T13:53:03.693 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:03.716 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:03.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:03.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:03.818 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:03.832 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-30T13:53:03.832 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:53:03.832 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T13:53:03.832 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd out 5 2024-03-30T13:53:03.854 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:03.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:03.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:03.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:03 smithi043 ceph-mon[42099]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:53:03.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:03 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1474261311' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:53:03.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:03 smithi043 ceph-mon[42099]: osdmap e219: 8 total, 7 up, 8 in 2024-03-30T13:53:03.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:03 smithi043 ceph-mon[29963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:53:03.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:03 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1474261311' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:53:03.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:03 smithi043 ceph-mon[29963]: osdmap e219: 8 total, 7 up, 8 in 2024-03-30T13:53:03.956 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:03.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:03 smithi113 ceph-mon[34341]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:53:03.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:03 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1474261311' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:53:03.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:03 smithi113 ceph-mon[34341]: osdmap e219: 8 total, 7 up, 8 in 2024-03-30T13:53:03.982 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:03.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:03.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:04.084 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:04.112 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:04.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:04.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:04.213 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:04.249 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:04.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:04.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:04.351 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:04.388 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:04.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:04.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:04.490 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:04.531 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:04.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:04.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:04.633 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:04.668 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:04.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:04.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:04.688 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 13 16 33084 33068 158.944 102.5 0.0063402 0.00628561 2024-03-30T13:53:04.770 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:04.806 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:04.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:04.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:04.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:04 smithi043 ceph-mon[29963]: pgmap v429: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 1.3 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 111 MiB/s wr, 1.78k op/s; 4059/66522 objects degraded (6.102%); 1/66522 objects misplaced (0.002%) 2024-03-30T13:53:04.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:04 smithi043 ceph-mon[42099]: pgmap v429: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 1.3 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 111 MiB/s wr, 1.78k op/s; 4059/66522 objects degraded (6.102%); 1/66522 objects misplaced (0.002%) 2024-03-30T13:53:04.909 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:04.946 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:04.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:04.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:04.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:04 smithi113 ceph-mon[34341]: pgmap v429: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 1.3 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 111 MiB/s wr, 1.78k op/s; 4059/66522 objects degraded (6.102%); 1/66522 objects misplaced (0.002%) 2024-03-30T13:53:05.048 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:05.081 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:05.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:05.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:05.183 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:05.221 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:05.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:05.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:05.324 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:05.361 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:05.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:05.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:05.463 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:05.500 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:05.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:05.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:05.602 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:05.639 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:05.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:05.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:05.688 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 14 16 35236 35220 157.197 134.5 0.00360169 0.00635582 2024-03-30T13:53:05.741 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:05.793 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-30T13:53:05.807 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:05.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:05.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:05.908 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:05.944 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:05.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:06.046 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:06.075 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:06.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:06.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:06.100 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:53:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:53:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:53:06.177 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:06.219 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:06.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:06.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:06.321 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:06.357 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:06.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:06.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:06.459 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:06.494 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:06.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:06.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:06.596 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:06.643 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:06.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:06.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:06.691 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 15 16 37323 37307 155.386 130.438 0.00980313 0.00643038 2024-03-30T13:53:06.746 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:06.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:06.800 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:06.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:06.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:06 smithi043 ceph-mon[42099]: pgmap v430: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 1.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 187 MiB/s wr, 2.99k op/s; 5950/95697 objects degraded (6.218%); 1/95697 objects misplaced (0.001%) 2024-03-30T13:53:06.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:06 smithi043 ceph-mon[29963]: pgmap v430: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 1.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 187 MiB/s wr, 2.99k op/s; 5950/95697 objects degraded (6.218%); 1/95697 objects misplaced (0.001%) 2024-03-30T13:53:06.900 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:06.937 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:06.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:06.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:06.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:06 smithi113 ceph-mon[34341]: pgmap v430: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 1.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 187 MiB/s wr, 2.99k op/s; 5950/95697 objects degraded (6.218%); 1/95697 objects misplaced (0.001%) 2024-03-30T13:53:07.039 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:07.074 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:07.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:07.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:07.176 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:07.214 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:07.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:07.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:07.316 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:07.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:07.349 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:07.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:07.450 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:07.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:07.483 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:07.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:07.584 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:07.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:07.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:07.690 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 16 16 39140 39124 152.772 113.562 0.00433165 0.00654118 2024-03-30T13:53:07.717 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:53:07.726 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:07.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:07.780 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:07.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:07.853 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:07 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 5950/95697 objects degraded (6.218%), 15 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:07.853 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:07 smithi043 ceph-mon[29963]: osdmap e220: 8 total, 7 up, 8 in 2024-03-30T13:53:07.853 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:07 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 5950/95697 objects degraded (6.218%), 15 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:07.853 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:07 smithi043 ceph-mon[42099]: osdmap e220: 8 total, 7 up, 8 in 2024-03-30T13:53:07.881 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:07.916 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:07.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:07.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:07.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:07 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 5950/95697 objects degraded (6.218%), 15 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:07.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:07 smithi113 ceph-mon[34341]: osdmap e220: 8 total, 7 up, 8 in 2024-03-30T13:53:08.007 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:53:08.019 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:08.048 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:08.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:08.052 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:53:08.092 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:53:08.134 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:53:08.150 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:08.176 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:53:08.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:08.197 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:08.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:08.220 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:53:08.274 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:53:08.298 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:08.313 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:53:08.313 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:53:08.313 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:53:08.313 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:53:08.313 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:53:08.313 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:53:08.313 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:53:08.314 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:53:08.314 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:53:08.338 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:08.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:08.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:08.440 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:08.473 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:08.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:08.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:08.575 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:08.608 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:08.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:08.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:08.659 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T13:53:08.691 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 17 16 41026 41010 150.715 117.875 0.00321332 0.00663031 2024-03-30T13:53:08.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:08 smithi043 ceph-mon[29963]: pgmap v432: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 1.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s; 5950/95697 objects degraded (6.218%); 1/95697 objects misplaced (0.001%) 2024-03-30T13:53:08.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:08 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3680622147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:53:08.703 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:08 smithi043 ceph-mon[42099]: pgmap v432: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 1.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s; 5950/95697 objects degraded (6.218%); 1/95697 objects misplaced (0.001%) 2024-03-30T13:53:08.704 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:08 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3680622147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:53:08.710 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:08.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:08.748 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:08.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:08.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:08.866 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:08 smithi113 ceph-mon[34341]: pgmap v432: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 1.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s; 5950/95697 objects degraded (6.218%); 1/95697 objects misplaced (0.001%) 2024-03-30T13:53:08.866 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:08 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3680622147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:53:08.896 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:08.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:08.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:08.998 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:09.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:09.041 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:09.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:09.142 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:09.178 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:09.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:09.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:09.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:53:08 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:53:09.280 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:09.318 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:09.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:09.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:09.420 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:09.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:09.458 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:09.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:09.559 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:09.594 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:09.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:09.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:09.691 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 18 15 43094 43079 149.525 129.312 0.00941586 0.00668246 2024-03-30T13:53:09.692 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:53:09.692 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:53:09.696 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:09.730 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:09.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:09.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:09.831 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:09.882 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:09.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:09.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:09.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:09 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:53:09.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:09 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:53:09.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:09 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:53:09.984 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:10.019 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:10.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:10.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:10.121 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:10.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:10.161 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:10.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:10.262 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:10.297 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:10.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:10.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:10.399 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:10.433 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:10.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:10.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:10.535 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:10.572 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:10.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:10.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:10.674 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:10.691 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 19 15 45145 45130 148.402 128.188 0.00866713 0.00673381 2024-03-30T13:53:10.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:10.712 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:10.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:10.813 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:10.852 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:10.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:10.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:10.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:10 smithi043 ceph-mon[29963]: pgmap v433: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 152 MiB/s wr, 2.44k op/s; 7780/125019 objects degraded (6.223%); 1/125019 objects misplaced (0.001%) 2024-03-30T13:53:10.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:10 smithi043 ceph-mon[42099]: pgmap v433: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 152 MiB/s wr, 2.44k op/s; 7780/125019 objects degraded (6.223%); 1/125019 objects misplaced (0.001%) 2024-03-30T13:53:10.954 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:10.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:10 smithi113 ceph-mon[34341]: pgmap v433: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 152 MiB/s wr, 2.44k op/s; 7780/125019 objects degraded (6.223%); 1/125019 objects misplaced (0.001%) 2024-03-30T13:53:10.991 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:10.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:10.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:11.093 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:11.130 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:11.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:11.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:11.232 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:11.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:11.267 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:11.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:11.368 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:11.404 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:11.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:11.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:11.446 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:53:11.505 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:11.540 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:11.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:11.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:11.642 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:11.676 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:11.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:11.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:11.691 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-03-30T13:53:11.693731+0000 min lat: 0.00127384 max lat: 0.160828 avg lat: 0.00680589 2024-03-30T13:53:11.691 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T13:53:11.692 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 20 16 47017 47001 146.829 116.938 0.00439941 0.00680589 2024-03-30T13:53:11.693 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:53:11.778 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:11.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:11.839 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:11.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:11.940 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:11.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:11.998 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:11.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:12.099 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:12.130 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:53:12.131 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:53:12.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:12.151 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:12.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:12.252 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:12.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:12.287 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:12.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:12.388 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:12.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:12.441 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:12.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:12.542 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:12.582 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:12.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:12.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:12.618 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:12 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a[29940]: 2024-03-30T13:53:12.456+0000 7ff6065d3700 -1 mon.a@0(leader).osd e220 definitely_dead 0 2024-03-30T13:53:12.646 INFO:teuthology.orchestra.run.smithi043.stderr:marked out osd.5. 2024-03-30T13:53:12.684 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:12.691 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 21 16 48576 48560 144.477 97.4375 0.0038399 0.00690283 2024-03-30T13:53:12.708 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:12.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:12.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:12.809 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:12.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:12.834 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:12.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:12.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:12 smithi043 ceph-mon[29963]: pgmap v434: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 137 MiB/s wr, 2.19k op/s; 7780/125019 objects degraded (6.223%); 1/125019 objects misplaced (0.001%) 2024-03-30T13:53:12.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:12 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1163705070' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-30T13:53:12.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:12 smithi043 ceph-mon[29963]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-30T13:53:12.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:12 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 7780/125019 objects degraded (6.223%), 15 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:12.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:12 smithi043 ceph-mon[42099]: pgmap v434: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 137 MiB/s wr, 2.19k op/s; 7780/125019 objects degraded (6.223%); 1/125019 objects misplaced (0.001%) 2024-03-30T13:53:12.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:12 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1163705070' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-30T13:53:12.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:12 smithi043 ceph-mon[42099]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-30T13:53:12.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:12 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 7780/125019 objects degraded (6.223%), 15 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:12.935 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:12.960 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:12.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:12.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:12.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:12 smithi113 ceph-mon[34341]: pgmap v434: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 137 MiB/s wr, 2.19k op/s; 7780/125019 objects degraded (6.223%); 1/125019 objects misplaced (0.001%) 2024-03-30T13:53:12.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:12 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1163705070' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-30T13:53:12.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:12 smithi113 ceph-mon[34341]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-30T13:53:12.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:12 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 7780/125019 objects degraded (6.223%), 15 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:13.061 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:13.085 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:13.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:13.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:13.187 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:13.214 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:13.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:13.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:13.316 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:13.343 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:13.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:13.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:13.444 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:13.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:13.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:13.573 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:13.600 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:13.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:13.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:13.691 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 22 16 48576 48560 137.911 0 - 0.00690283 2024-03-30T13:53:13.702 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:13.727 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:13.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:13.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:13.828 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:13.853 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:13.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:13.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:13.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:13 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1163705070' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-30T13:53:13.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:13 smithi043 ceph-mon[29963]: osdmap e221: 8 total, 7 up, 7 in 2024-03-30T13:53:13.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:13 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:53:13.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:13 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:53:13.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:13 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1163705070' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-30T13:53:13.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:13 smithi043 ceph-mon[42099]: osdmap e221: 8 total, 7 up, 7 in 2024-03-30T13:53:13.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:13 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:53:13.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:13 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:53:13.954 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:13.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:13 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1163705070' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-30T13:53:13.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:13 smithi113 ceph-mon[34341]: osdmap e221: 8 total, 7 up, 7 in 2024-03-30T13:53:13.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:13 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:53:13.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:13 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:53:13.978 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:13.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:13.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:14.080 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:14.106 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:14.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:14.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:14.151 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-03-30T13:53:14.208 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:14.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:14.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:14.334 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:14.357 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:14.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:14.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:14.459 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:14.482 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:14.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:14.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:14.584 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:14.607 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:14.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:14.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:14.691 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 23 16 48583 48567 131.935 0.21875 0.0136504 0.00715186 2024-03-30T13:53:14.709 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:14.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:14.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:14.848 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:14 smithi043 ceph-mon[29963]: pgmap v436: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 2.5 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s; 7780/125019 objects degraded (6.223%); 1/125019 objects misplaced (0.001%) 2024-03-30T13:53:14.848 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:14 smithi043 ceph-mon[29963]: osdmap e222: 8 total, 7 up, 7 in 2024-03-30T13:53:14.848 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:14 smithi043 ceph-mon[42099]: pgmap v436: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 2.5 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s; 7780/125019 objects degraded (6.223%); 1/125019 objects misplaced (0.001%) 2024-03-30T13:53:14.848 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:14 smithi043 ceph-mon[42099]: osdmap e222: 8 total, 7 up, 7 in 2024-03-30T13:53:14.864 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:14.899 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:14.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:14.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:14.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:14 smithi113 ceph-mon[34341]: pgmap v436: 158 pgs: 5 active+clean+remapped, 15 active+undersized+degraded, 26 active+undersized, 112 active+clean; 2.5 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s; 7780/125019 objects degraded (6.223%); 1/125019 objects misplaced (0.001%) 2024-03-30T13:53:14.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:14 smithi113 ceph-mon[34341]: osdmap e222: 8 total, 7 up, 7 in 2024-03-30T13:53:15.001 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:15.032 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:15.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:15.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:15.133 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:15.157 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:15.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:15.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:15.191 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:53:15.258 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:15.283 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:15.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:15.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:15.385 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:15.409 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:15.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:15.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:15.510 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:15.534 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:15.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:15.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:15.636 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:15.695 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 24 16 48629 48613 126.559 2.875 0.0114591 0.00734469 2024-03-30T13:53:15.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:15.720 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:15.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:15.736 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:15 smithi043 ceph-mon[29963]: osdmap e223: 8 total, 7 up, 7 in 2024-03-30T13:53:15.736 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:15 smithi043 ceph-mon[42099]: osdmap e223: 8 total, 7 up, 7 in 2024-03-30T13:53:15.750 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:53:15.821 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:15.845 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:15.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:15.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:15.947 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:15.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:15 smithi113 ceph-mon[34341]: osdmap e223: 8 total, 7 up, 7 in 2024-03-30T13:53:15.971 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:15.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:15.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:16.073 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:16.112 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:16.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:16.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:16.220 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:16.254 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:16.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:16.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:16.269 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:53:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:53:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:53:16.356 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:16.382 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:16.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:16.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:16.484 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:16.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:16.510 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:16.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:16.611 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:16.643 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:16.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:16.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:16.674 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-03-30T13:53:16.692 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 25 16 48629 48613 121.498 0 - 0.00734469 2024-03-30T13:53:16.745 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:16.780 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:16.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:16.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:16.882 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:16.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:16.908 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:16.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:16.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:16 smithi113 ceph-mon[34341]: pgmap v439: 158 pgs: 5 unknown, 83 active+clean, 20 active+undersized, 11 active+undersized+degraded, 17 peering, 2 active+clean+remapped, 20 remapped+peering; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 5.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 6103/146283 objects degraded (4.172%); 26 KiB/s, 4 objects/s recovering 2024-03-30T13:53:16.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:16 smithi113 ceph-mon[34341]: Health check failed: Reduced data availability: 8 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-30T13:53:16.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:16 smithi113 ceph-mon[34341]: osdmap e224: 8 total, 7 up, 7 in 2024-03-30T13:53:16.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:16 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/947348848' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:53:16.970 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:16 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:53:17.009 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:17.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:17.035 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:17.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:17.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:16 smithi043 ceph-mon[29963]: pgmap v439: 158 pgs: 5 unknown, 83 active+clean, 20 active+undersized, 11 active+undersized+degraded, 17 peering, 2 active+clean+remapped, 20 remapped+peering; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 5.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 6103/146283 objects degraded (4.172%); 26 KiB/s, 4 objects/s recovering 2024-03-30T13:53:17.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:16 smithi043 ceph-mon[29963]: Health check failed: Reduced data availability: 8 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-30T13:53:17.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:16 smithi043 ceph-mon[29963]: osdmap e224: 8 total, 7 up, 7 in 2024-03-30T13:53:17.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:16 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/947348848' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:53:17.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:16 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:53:17.122 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:16 smithi043 ceph-mon[42099]: pgmap v439: 158 pgs: 5 unknown, 83 active+clean, 20 active+undersized, 11 active+undersized+degraded, 17 peering, 2 active+clean+remapped, 20 remapped+peering; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 5.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 6103/146283 objects degraded (4.172%); 26 KiB/s, 4 objects/s recovering 2024-03-30T13:53:17.122 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:16 smithi043 ceph-mon[42099]: Health check failed: Reduced data availability: 8 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-30T13:53:17.122 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:16 smithi043 ceph-mon[42099]: osdmap e224: 8 total, 7 up, 7 in 2024-03-30T13:53:17.122 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:16 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/947348848' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:53:17.122 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:16 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:53:17.136 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:17.168 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:17.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:17.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:17.270 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:17.294 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:17.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:17.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:17.396 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:17.421 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:17.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:17.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:17.522 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:17.547 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:17.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:17.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:17.648 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:17.683 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:17.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:17.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:17.692 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 26 16 48629 48613 116.825 0 - 0.00734469 2024-03-30T13:53:17.784 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:17.808 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:17.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:17.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:17.862 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:17 smithi043 ceph-mon[29963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:53:17.862 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:17 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:53:17.862 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:17 smithi043 ceph-mon[29963]: osdmap e225: 8 total, 7 up, 7 in 2024-03-30T13:53:17.862 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:17 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 6103/146283 objects degraded (4.172%), 11 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:17.862 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:17 smithi043 ceph-mon[29963]: osdmap e226: 8 total, 7 up, 7 in 2024-03-30T13:53:17.862 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:17 smithi043 ceph-mon[42099]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:53:17.862 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:17 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:53:17.862 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:17 smithi043 ceph-mon[42099]: osdmap e225: 8 total, 7 up, 7 in 2024-03-30T13:53:17.863 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:17 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 6103/146283 objects degraded (4.172%), 11 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:17.863 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:17 smithi043 ceph-mon[42099]: osdmap e226: 8 total, 7 up, 7 in 2024-03-30T13:53:17.910 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:17.935 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:17.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:17.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:17.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:17 smithi113 ceph-mon[34341]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:53:17.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:17 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:53:17.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:17 smithi113 ceph-mon[34341]: osdmap e225: 8 total, 7 up, 7 in 2024-03-30T13:53:17.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:17 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 6103/146283 objects degraded (4.172%), 11 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:17.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:17 smithi113 ceph-mon[34341]: osdmap e226: 8 total, 7 up, 7 in 2024-03-30T13:53:18.037 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:18.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:18.073 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:18.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:18.174 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:18.206 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:18.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:18.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:18.308 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:18.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:18.344 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:18.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:18.445 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:18.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:18.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:18.579 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:18.604 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:18.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:18.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:18.692 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 27 16 48719 48703 112.707 1.875 0.0034801 0.0081935 2024-03-30T13:53:18.705 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:18.730 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:18.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:18.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:18.831 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:18.855 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:18.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:18.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:18.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:18 smithi043 ceph-mon[29963]: pgmap v443: 158 pgs: 5 unknown, 83 active+clean, 20 active+undersized, 11 active+undersized+degraded, 17 peering, 2 active+clean+remapped, 20 remapped+peering; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 7.0 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 6103/146283 objects degraded (4.172%); 41 KiB/s, 6 objects/s recovering 2024-03-30T13:53:18.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:18 smithi043 ceph-mon[42099]: pgmap v443: 158 pgs: 5 unknown, 83 active+clean, 20 active+undersized, 11 active+undersized+degraded, 17 peering, 2 active+clean+remapped, 20 remapped+peering; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 7.0 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 6103/146283 objects degraded (4.172%); 41 KiB/s, 6 objects/s recovering 2024-03-30T13:53:18.958 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:18.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:18 smithi113 ceph-mon[34341]: pgmap v443: 158 pgs: 5 unknown, 83 active+clean, 20 active+undersized, 11 active+undersized+degraded, 17 peering, 2 active+clean+remapped, 20 remapped+peering; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 7.0 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 6103/146283 objects degraded (4.172%); 41 KiB/s, 6 objects/s recovering 2024-03-30T13:53:18.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:53:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:53:18.982 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:18.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:18.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:19.084 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:19.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:19.109 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:19.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:19.210 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:19.236 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:19.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:19.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:19.338 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:19.377 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:19.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:19.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:19.479 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:19.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:19.512 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:19.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:19.613 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:19.644 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:19.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:19.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:19.692 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 28 16 48719 48703 108.682 0 - 0.0081935 2024-03-30T13:53:19.746 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:19.782 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:19.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:19.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:19.884 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:19.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:19 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:53:19.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:19 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:53:19.914 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:19.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:19.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:19.916 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T13:53:19.948 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:53:19.948 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:53:19.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:19 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:53:20.016 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:20.048 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:20.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:20.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:20.150 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:20.179 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:20.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:20.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:20.185 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-30T13:53:20.185 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:53:20.185 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 4, 6, 7] 2024-03-30T13:53:20.185 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd out 7 2024-03-30T13:53:20.280 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:20.281 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:53:20.282 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:53:20.282 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:53:20.282 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:53:20.288 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:53:20.312 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:20.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:20.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:20.335 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:53:20.373 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:53:20.373 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:53:20.373 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:53:20.373 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:53:20.373 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:53:20.373 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:53:20.373 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:53:20.373 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:53:20.373 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:53:20.414 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:20.446 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:20.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:20.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:20.548 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:20.608 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:20.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:20.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:20.692 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 29 15 49420 49405 106.448 21.9375 0.0220023 0.0093876 2024-03-30T13:53:20.711 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:20.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:20.755 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:20.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:20.856 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:20.889 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:20.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:20.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:20 smithi043 ceph-mon[29963]: pgmap v444: 158 pgs: 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 11 active+undersized+degraded, 20 active+clean+remapped, 8 remapped+peering; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 12 KiB/s rd, 1.5 MiB/s wr, 38 op/s; 6105/146628 objects degraded (4.164%); 16737/146628 objects misplaced (11.415%); 23 MiB/s, 375 objects/s recovering 2024-03-30T13:53:20.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:20 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1108680588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:53:20.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:20 smithi043 ceph-mon[42099]: pgmap v444: 158 pgs: 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 11 active+undersized+degraded, 20 active+clean+remapped, 8 remapped+peering; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 12 KiB/s rd, 1.5 MiB/s wr, 38 op/s; 6105/146628 objects degraded (4.164%); 16737/146628 objects misplaced (11.415%); 23 MiB/s, 375 objects/s recovering 2024-03-30T13:53:20.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:20 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1108680588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:53:20.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:20 smithi113 ceph-mon[34341]: pgmap v444: 158 pgs: 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 11 active+undersized+degraded, 20 active+clean+remapped, 8 remapped+peering; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 12 KiB/s rd, 1.5 MiB/s wr, 38 op/s; 6105/146628 objects degraded (4.164%); 16737/146628 objects misplaced (11.415%); 23 MiB/s, 375 objects/s recovering 2024-03-30T13:53:20.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:20 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1108680588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:53:20.991 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:21.034 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:21.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:21.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:21.136 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:21.172 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:21.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:21.274 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:21.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:21.311 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:21.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:21.412 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:21.457 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:21.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:21.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:21.559 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:21.597 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:21.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:21.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:21.692 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 30 16 50505 50489 105.158 67.75 0.00801618 0.00950265 2024-03-30T13:53:21.699 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:21.749 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:21.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:21.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:21.752 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-03-30T13:53:21.852 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:21.903 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:21.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:21.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:21.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:21 smithi113 ceph-mon[34341]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-30T13:53:22.005 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:22.036 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:22.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:22.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:21 smithi043 ceph-mon[29963]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-30T13:53:22.056 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:21 smithi043 ceph-mon[42099]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-30T13:53:22.111 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:53:22.112 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:53:22.137 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:22.173 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:53:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:53:22.225 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:22.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:22.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:22.260 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:53:22.330 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:22.385 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:22.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:22.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:22.486 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:22.543 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:22.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:22.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:22.645 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:22.678 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:22.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:22.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:22.692 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 31 16 51412 51396 103.594 56.6875 0.0163841 0.00964637 2024-03-30T13:53:22.780 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:22.813 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:22.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:22.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:22 smithi043 ceph-mon[29963]: pgmap v445: 158 pgs: 1 activating+undersized+degraded+remapped, 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 9.7 KiB/s rd, 1.5 MiB/s wr, 35 op/s; 12106/146712 objects degraded (8.252%); 22565/146712 objects misplaced (15.380%); 20 MiB/s, 329 objects/s recovering 2024-03-30T13:53:22.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:22 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 12106/146712 objects degraded (8.252%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:22.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:22 smithi043 ceph-mon[42099]: pgmap v445: 158 pgs: 1 activating+undersized+degraded+remapped, 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 9.7 KiB/s rd, 1.5 MiB/s wr, 35 op/s; 12106/146712 objects degraded (8.252%); 22565/146712 objects misplaced (15.380%); 20 MiB/s, 329 objects/s recovering 2024-03-30T13:53:22.897 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:22 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 12106/146712 objects degraded (8.252%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:22.915 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:22.948 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:22.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:22.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:23.050 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:23.103 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:23.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:23.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:23.205 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:23.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:22 smithi113 ceph-mon[34341]: pgmap v445: 158 pgs: 1 activating+undersized+degraded+remapped, 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 9.7 KiB/s rd, 1.5 MiB/s wr, 35 op/s; 12106/146712 objects degraded (8.252%); 22565/146712 objects misplaced (15.380%); 20 MiB/s, 329 objects/s recovering 2024-03-30T13:53:23.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:22 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 12106/146712 objects degraded (8.252%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:23.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:23.236 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:23.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:23.337 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:23.383 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:23.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:23.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:23.484 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:23.518 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:23.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:23.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:23.620 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:23.651 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:23.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:23.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:23.692 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 32 16 52459 52443 102.402 65.4375 0.00711861 0.00975851 2024-03-30T13:53:23.753 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:23.785 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:23.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:23.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:23.888 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:23.917 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:23.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:23.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:24.019 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:24.050 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:24.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:24.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:24.113 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:53:24.152 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:24.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:24.210 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:24.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:24.311 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:24.367 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:24.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:24.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:24.469 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:24.499 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:24.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:24.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:24.601 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:24.644 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:24.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:24.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:24.692 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 33 16 53457 53441 101.189 62.375 0.0232218 0.00987704 2024-03-30T13:53:24.746 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:24.792 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:24.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:24.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:24.894 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:24.924 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:24.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:24.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:25.024 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:24 smithi043 ceph-mon[29963]: pgmap v446: 158 pgs: 1 activating+undersized+degraded+remapped, 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 7.4 KiB/s rd, 1.2 MiB/s wr, 27 op/s; 12106/146712 objects degraded (8.252%); 22565/146712 objects misplaced (15.380%); 16 MiB/s, 251 objects/s recovering 2024-03-30T13:53:25.025 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:24 smithi043 ceph-mon[42099]: pgmap v446: 158 pgs: 1 activating+undersized+degraded+remapped, 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 7.4 KiB/s rd, 1.2 MiB/s wr, 27 op/s; 12106/146712 objects degraded (8.252%); 22565/146712 objects misplaced (15.380%); 16 MiB/s, 251 objects/s recovering 2024-03-30T13:53:25.025 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:25.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:25.083 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:25.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:25.184 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:25.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:24 smithi113 ceph-mon[34341]: pgmap v446: 158 pgs: 1 activating+undersized+degraded+remapped, 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 7.4 KiB/s rd, 1.2 MiB/s wr, 27 op/s; 12106/146712 objects degraded (8.252%); 22565/146712 objects misplaced (15.380%); 16 MiB/s, 251 objects/s recovering 2024-03-30T13:53:25.223 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:25.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:25.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:25.325 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:25.358 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:25.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:25.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:25.460 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:25.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:25.540 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:25.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:25.641 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:25.673 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:25.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:25.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:25.693 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 34 16 54447 54431 100.033 61.875 0.0130163 0.00999024 2024-03-30T13:53:25.774 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:25.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:25.821 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:25.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:25.921 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:25.946 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:25.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:25.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:26.047 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:26.079 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:26.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:26.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:26.181 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:26.204 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:26.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:26.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:26.306 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:26.337 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:26.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:26.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:26.396 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:53:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:53:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:53:26.439 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:26.463 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:26.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:26.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:26.565 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:26.598 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:26.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:26.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:26.693 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 35 16 55226 55210 98.5659 48.6875 0.00792404 0.010139 2024-03-30T13:53:26.700 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:26.752 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:26.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:26.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:26.854 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:26.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:26.918 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:26.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:27.019 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:27.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:27.056 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:27.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:27.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:26 smithi043 ceph-mon[29963]: pgmap v447: 158 pgs: 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 7.1 KiB/s rd, 31 MiB/s wr, 510 op/s; 12970/159603 objects degraded (8.126%); 21908/159603 objects misplaced (13.727%); 24 MiB/s, 391 objects/s recovering 2024-03-30T13:53:27.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:26 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:27.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:26 smithi043 ceph-mon[42099]: pgmap v447: 158 pgs: 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 7.1 KiB/s rd, 31 MiB/s wr, 510 op/s; 12970/159603 objects degraded (8.126%); 21908/159603 objects misplaced (13.727%); 24 MiB/s, 391 objects/s recovering 2024-03-30T13:53:27.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:26 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:27.157 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:27.191 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:27.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:27.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:27.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:26 smithi113 ceph-mon[34341]: pgmap v447: 158 pgs: 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 7.1 KiB/s rd, 31 MiB/s wr, 510 op/s; 12970/159603 objects degraded (8.126%); 21908/159603 objects misplaced (13.727%); 24 MiB/s, 391 objects/s recovering 2024-03-30T13:53:27.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:26 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:27.293 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:27.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:27.343 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:27.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:27.444 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:27.475 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:27.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:27.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:27.577 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:27.607 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:27.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:27.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:27.693 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 36 16 56069 56053 97.2914 52.6875 0.00503956 0.010257 2024-03-30T13:53:27.709 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:27.733 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:27.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:27.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:27.835 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:27.865 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:27.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:27.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:27.967 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:28.008 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:28.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:28.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:28.110 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:28.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:27 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 12970/159603 objects degraded (8.126%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:28.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:27 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 12970/159603 objects degraded (8.126%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:28.152 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:28.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:28.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:28.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:27 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 12970/159603 objects degraded (8.126%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:28.254 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:28.281 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:28.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:28.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:28.383 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:28.429 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:28.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:28.530 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:28.559 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:28.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:28.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:28.661 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:28.693 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 37 16 56931 56915 96.118 53.875 0.0091813 0.0103986 2024-03-30T13:53:28.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:28.694 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:28.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:28.795 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:28.832 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:28.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:28.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:28.934 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:28.960 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:28.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:28.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:29.061 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:29.092 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:29.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:29.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:29.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:28 smithi043 ceph-mon[29963]: pgmap v448: 158 pgs: 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 6.3 KiB/s rd, 28 MiB/s wr, 450 op/s; 12970/159603 objects degraded (8.126%); 21908/159603 objects misplaced (13.727%); 21 MiB/s, 345 objects/s recovering 2024-03-30T13:53:29.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:53:29.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:28 smithi043 ceph-mon[42099]: pgmap v448: 158 pgs: 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 6.3 KiB/s rd, 28 MiB/s wr, 450 op/s; 12970/159603 objects degraded (8.126%); 21908/159603 objects misplaced (13.727%); 21 MiB/s, 345 objects/s recovering 2024-03-30T13:53:29.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:53:29.193 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:29.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:28 smithi113 ceph-mon[34341]: pgmap v448: 158 pgs: 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 6.3 KiB/s rd, 28 MiB/s wr, 450 op/s; 12970/159603 objects degraded (8.126%); 21908/159603 objects misplaced (13.727%); 21 MiB/s, 345 objects/s recovering 2024-03-30T13:53:29.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:53:29.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:53:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:53:29.241 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:29.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:29.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:29.342 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:29.374 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:29.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:29.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:29.476 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:29.507 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:29.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:29.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:29.608 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:29.632 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:29.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:29.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:29.693 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 38 16 57816 57800 95.0442 55.3125 0.00872743 0.0105149 2024-03-30T13:53:29.734 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:29.765 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:29.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:29.866 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:29.895 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:29.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:29.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:29.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:30.042 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:30.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:30.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:30.143 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:30.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:29 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:53:30.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:29 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:53:30.173 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:30.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:30.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:30.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:29 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:53:30.275 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:30.300 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:30.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:30.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:30.402 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:30.444 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:30.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:30.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:30.546 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:30.576 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:30.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:30.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:30.677 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:30.693 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 39 16 58543 58527 93.7722 45.4375 0.0293154 0.0106566 2024-03-30T13:53:30.720 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:30.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:30.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:30.822 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:30.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:30.864 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:30.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:30.965 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:30.995 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:30.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:30.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:31.097 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:31.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:31.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:31.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:30 smithi043 ceph-mon[29963]: pgmap v449: 158 pgs: 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 5.7 KiB/s rd, 40 MiB/s wr, 638 op/s; 13250/169050 objects degraded (7.838%); 21246/169050 objects misplaced (12.568%); 24 MiB/s, 392 objects/s recovering 2024-03-30T13:53:31.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:30 smithi043 ceph-mon[42099]: pgmap v449: 158 pgs: 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 5.7 KiB/s rd, 40 MiB/s wr, 638 op/s; 13250/169050 objects degraded (7.838%); 21246/169050 objects misplaced (12.568%); 24 MiB/s, 392 objects/s recovering 2024-03-30T13:53:31.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:30 smithi113 ceph-mon[34341]: pgmap v449: 158 pgs: 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 5.7 KiB/s rd, 40 MiB/s wr, 638 op/s; 13250/169050 objects degraded (7.838%); 21246/169050 objects misplaced (12.568%); 24 MiB/s, 392 objects/s recovering 2024-03-30T13:53:31.239 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:31.270 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:31.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:31.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:31.372 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:31.398 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:31.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:31.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:31.500 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:31.529 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:31.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:31.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:31.631 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:31.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:31.665 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:31.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:31.693 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-03-30T13:53:31.695822+0000 min lat: 0.00127384 max lat: 7.25374 avg lat: 0.010763 2024-03-30T13:53:31.693 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T13:53:31.693 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 40 16 59420 59404 92.7982 54.8125 0.0031408 0.010763 2024-03-30T13:53:31.766 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:31.806 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:31.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:31.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:31.908 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:31.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:31.955 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:31.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:32.056 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:32.085 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:32.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:32.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:32.187 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:32.210 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:32.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:32.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:32.312 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:32.345 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:32.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:32.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:32.446 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:32.485 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:32.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:32.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:32.587 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:32.616 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:32.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:32.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:32.693 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 41 16 60223 60207 91.7591 50.1875 0.0110853 0.0108915 2024-03-30T13:53:32.718 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:32.742 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:32.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:32.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:32.844 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:32.888 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:32.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:32.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:32.990 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:33.021 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:33.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:33.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:33.123 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:33.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:32 smithi043 ceph-mon[29963]: pgmap v450: 158 pgs: 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 46 MiB/s wr, 733 op/s; 13758/173001 objects degraded (7.953%); 21341/173001 objects misplaced (12.336%); 16 MiB/s, 255 objects/s recovering 2024-03-30T13:53:33.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:32 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:33.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:32 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:33.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:32 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:53:33.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:32 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 13758/173001 objects degraded (7.953%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:33.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:32 smithi043 ceph-mon[42099]: pgmap v450: 158 pgs: 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 46 MiB/s wr, 733 op/s; 13758/173001 objects degraded (7.953%); 21341/173001 objects misplaced (12.336%); 16 MiB/s, 255 objects/s recovering 2024-03-30T13:53:33.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:32 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:33.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:32 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:33.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:32 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:53:33.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:32 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 13758/173001 objects degraded (7.953%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:33.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:33.165 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:33.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:33.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:32 smithi113 ceph-mon[34341]: pgmap v450: 158 pgs: 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 46 MiB/s wr, 733 op/s; 13758/173001 objects degraded (7.953%); 21341/173001 objects misplaced (12.336%); 16 MiB/s, 255 objects/s recovering 2024-03-30T13:53:33.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:32 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:33.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:32 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:33.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:32 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:53:33.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:32 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 13758/173001 objects degraded (7.953%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:33.266 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:33.295 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:33.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:33.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:33.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:33.419 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:33.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:33.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:33.521 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:33.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:33.562 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:33.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:33.663 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:33.689 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:33.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:33.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:33.693 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 42 16 60973 60957 90.6904 46.875 0.00619425 0.0110166 2024-03-30T13:53:33.792 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:33.815 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:33.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:33.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:33.916 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:33.945 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:33.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:33.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:34.047 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:34.081 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:34.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:34.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:34.183 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:34.216 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:34.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:34.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:34.317 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:34.346 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:34.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:34.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:34.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:33 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a[29940]: 2024-03-30T13:53:33.832+0000 7ff6065d3700 -1 mon.a@0(leader).osd e227 definitely_dead 0 2024-03-30T13:53:34.449 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:34.471 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:34.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:34.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:34.573 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:34.603 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:34.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:34.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:34.658 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:53:34.693 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 43 16 61214 61198 88.9318 15.0625 0.00321214 0.0110634 2024-03-30T13:53:34.705 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:34.733 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:34.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:34.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:34.834 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:34.844 INFO:teuthology.orchestra.run.smithi043.stderr:marked out osd.7. 2024-03-30T13:53:34.860 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:34 smithi043 ceph-mon[29963]: pgmap v451: 158 pgs: 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 46 MiB/s wr, 730 op/s; 13758/173001 objects degraded (7.953%); 21341/173001 objects misplaced (12.336%); 15 MiB/s, 243 objects/s recovering 2024-03-30T13:53:34.860 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:34 smithi043 ceph-mon[29963]: osdmap e227: 8 total, 7 up, 7 in 2024-03-30T13:53:34.860 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:34 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3803932373' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-30T13:53:34.860 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:34 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-30T13:53:34.860 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:34 smithi043 ceph-mon[29963]: Client client.admin marked osd.7 out, while it was still marked up 2024-03-30T13:53:34.860 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:34 smithi043 ceph-mon[42099]: pgmap v451: 158 pgs: 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 46 MiB/s wr, 730 op/s; 13758/173001 objects degraded (7.953%); 21341/173001 objects misplaced (12.336%); 15 MiB/s, 243 objects/s recovering 2024-03-30T13:53:34.860 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:34 smithi043 ceph-mon[42099]: osdmap e227: 8 total, 7 up, 7 in 2024-03-30T13:53:34.860 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:34 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3803932373' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-30T13:53:34.860 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:34 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-30T13:53:34.860 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:34 smithi043 ceph-mon[42099]: Client client.admin marked osd.7 out, while it was still marked up 2024-03-30T13:53:34.866 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:34.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:34.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:34.968 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:34.991 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:34.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:34.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:35.093 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:35.117 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:35.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:35.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:35.219 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:35.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:34 smithi113 ceph-mon[34341]: pgmap v451: 158 pgs: 95 active+clean, 4 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 46 MiB/s wr, 730 op/s; 13758/173001 objects degraded (7.953%); 21341/173001 objects misplaced (12.336%); 15 MiB/s, 243 objects/s recovering 2024-03-30T13:53:35.220 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:34 smithi113 ceph-mon[34341]: osdmap e227: 8 total, 7 up, 7 in 2024-03-30T13:53:35.220 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:34 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3803932373' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-30T13:53:35.220 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:34 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-30T13:53:35.220 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:34 smithi113 ceph-mon[34341]: Client client.admin marked osd.7 out, while it was still marked up 2024-03-30T13:53:35.243 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:35.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:35.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:35.345 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:35.368 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:35.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:35.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:35.470 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:35.493 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:35.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:35.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:35.595 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:35.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:35.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:35.693 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 44 16 61214 61198 86.9109 0 - 0.0110634 2024-03-30T13:53:35.721 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:35.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:35.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:35.846 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:35.877 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:35.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:35.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:35.979 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:36.003 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:36.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:36.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:36.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:35 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:53:36.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:35 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:53:36.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:35 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:53:36.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:35 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:53:36.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:35 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-03-30T13:53:36.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:35 smithi043 ceph-mon[29963]: osdmap e228: 8 total, 7 up, 6 in 2024-03-30T13:53:36.093 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:35 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:53:36.093 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:35 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:53:36.093 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:35 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:53:36.093 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:35 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:53:36.093 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:35 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-03-30T13:53:36.093 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:35 smithi043 ceph-mon[42099]: osdmap e228: 8 total, 7 up, 6 in 2024-03-30T13:53:36.104 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:36.129 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:36.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:36.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:36.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:35 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:53:36.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:35 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:53:36.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:35 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:53:36.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:35 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:53:36.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:35 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-03-30T13:53:36.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:35 smithi113 ceph-mon[34341]: osdmap e228: 8 total, 7 up, 6 in 2024-03-30T13:53:36.231 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:36.256 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:36.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:36.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:36.358 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:36.383 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:36.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:36.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:36.396 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:53:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:53:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:53:36.485 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:36.509 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:36.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:36.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:36.611 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:36.634 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:36.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:36.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:36.693 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 45 16 61214 61198 84.9798 0 - 0.0110634 2024-03-30T13:53:36.735 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:36.758 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:36.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:36.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:36.860 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:36.885 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:36.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:36.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:36.987 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:37.011 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:37.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:37.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:37.112 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:37.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:36 smithi043 ceph-mon[29963]: pgmap v454: 158 pgs: 95 active+clean, 3 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 14440/183966 objects degraded (7.849%); 20143/183966 objects misplaced (10.949%); 26 MiB/s, 408 objects/s recovering 2024-03-30T13:53:37.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:36 smithi043 ceph-mon[29963]: osdmap e229: 8 total, 7 up, 6 in 2024-03-30T13:53:37.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:36 smithi043 ceph-mon[42099]: pgmap v454: 158 pgs: 95 active+clean, 3 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 14440/183966 objects degraded (7.849%); 20143/183966 objects misplaced (10.949%); 26 MiB/s, 408 objects/s recovering 2024-03-30T13:53:37.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:36 smithi043 ceph-mon[42099]: osdmap e229: 8 total, 7 up, 6 in 2024-03-30T13:53:37.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:37.155 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:37.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:37.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:36 smithi113 ceph-mon[34341]: pgmap v454: 158 pgs: 95 active+clean, 3 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 14440/183966 objects degraded (7.849%); 20143/183966 objects misplaced (10.949%); 26 MiB/s, 408 objects/s recovering 2024-03-30T13:53:37.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:36 smithi113 ceph-mon[34341]: osdmap e229: 8 total, 7 up, 6 in 2024-03-30T13:53:37.256 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:37.281 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:37.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:37.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:37.383 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:37.422 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:37.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:37.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:37.524 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:37.550 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:37.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:37.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:37.651 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:37.678 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:37.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:37.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:37.693 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 46 16 61383 61367 83.3622 3.52083 0.0168747 0.0119869 2024-03-30T13:53:37.780 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:37.824 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:37.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:37.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:37.926 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:37.949 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:37.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:37.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:38.050 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:38.076 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:38.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:38.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:38.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:37 smithi043 ceph-mon[29963]: osdmap e230: 8 total, 7 up, 6 in 2024-03-30T13:53:38.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:37 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 14440/183966 objects degraded (7.849%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:38.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:37 smithi043 ceph-mon[29963]: osdmap e231: 8 total, 7 up, 6 in 2024-03-30T13:53:38.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:37 smithi043 ceph-mon[42099]: osdmap e230: 8 total, 7 up, 6 in 2024-03-30T13:53:38.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:37 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 14440/183966 objects degraded (7.849%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:38.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:37 smithi043 ceph-mon[42099]: osdmap e231: 8 total, 7 up, 6 in 2024-03-30T13:53:38.178 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:38.204 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:38.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:38.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:38.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:37 smithi113 ceph-mon[34341]: osdmap e230: 8 total, 7 up, 6 in 2024-03-30T13:53:38.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:37 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 14440/183966 objects degraded (7.849%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:38.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:37 smithi113 ceph-mon[34341]: osdmap e231: 8 total, 7 up, 6 in 2024-03-30T13:53:38.306 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:38.347 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:38.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:38.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:38.448 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:38.475 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:38.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:38.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:38.577 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:38.602 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:38.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:38.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:38.694 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 47 16 61887 61871 82.2588 31.5 0.0221509 0.0121486 2024-03-30T13:53:38.704 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:38.744 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:38.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:38.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:38.846 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:38.871 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:38.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:38.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:38.973 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:38.999 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:39.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:39.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:39.101 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:39.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:38 smithi043 ceph-mon[29963]: pgmap v458: 158 pgs: 95 active+clean, 3 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 45 MiB/s wr, 717 op/s; 14440/183966 objects degraded (7.849%); 20143/183966 objects misplaced (10.949%); 14 MiB/s, 225 objects/s recovering 2024-03-30T13:53:39.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:38 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:53:39.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:38 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:53:39.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:38 smithi043 ceph-mon[29963]: osdmap e232: 8 total, 7 up, 6 in 2024-03-30T13:53:39.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:38 smithi043 ceph-mon[42099]: pgmap v458: 158 pgs: 95 active+clean, 3 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 45 MiB/s wr, 717 op/s; 14440/183966 objects degraded (7.849%); 20143/183966 objects misplaced (10.949%); 14 MiB/s, 225 objects/s recovering 2024-03-30T13:53:39.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:38 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:53:39.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:38 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:53:39.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:38 smithi043 ceph-mon[42099]: osdmap e232: 8 total, 7 up, 6 in 2024-03-30T13:53:39.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:39.149 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:39.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:39.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:38 smithi113 ceph-mon[34341]: pgmap v458: 158 pgs: 95 active+clean, 3 active+recovering+undersized+remapped, 20 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 11 active+undersized+degraded, 26 active+clean+remapped; 3.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 45 MiB/s wr, 717 op/s; 14440/183966 objects degraded (7.849%); 20143/183966 objects misplaced (10.949%); 14 MiB/s, 225 objects/s recovering 2024-03-30T13:53:39.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:38 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:53:39.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:38 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:53:39.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:38 smithi113 ceph-mon[34341]: osdmap e232: 8 total, 7 up, 6 in 2024-03-30T13:53:39.220 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:53:38 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:53:39.250 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:39.277 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:39.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:39.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:39.378 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:39.432 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:39.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:39.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:39.534 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:39.557 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:39.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:39.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:39.659 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:39.685 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:39.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:39.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:39.694 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 48 16 62389 62373 81.1987 31.375 0.00537912 0.012302 2024-03-30T13:53:39.787 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:39.832 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:39.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:39.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:39.934 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:39.957 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:39.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:39.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:40.059 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:40.105 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:40.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:40.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:40.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:39 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:53:40.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:39 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:53:40.207 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:40.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:39 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:53:40.252 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:40.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:40.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:40.354 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:40.377 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:40.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:40.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:40.479 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:40.506 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:40.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:40.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:40.608 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:40.631 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:40.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:40.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:40.694 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 49 16 62832 62816 80.1067 27.6875 0.172368 0.0124492 2024-03-30T13:53:40.733 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:40.760 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:40.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:40.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:40.862 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:40.887 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:40.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:40.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:40.989 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:41.014 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:41.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:41.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:41.115 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:41.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:40 smithi043 ceph-mon[29963]: pgmap v460: 158 pgs: 86 active+clean, 5 active+recovering+undersized+remapped, 14 active+undersized, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 9 active+undersized+degraded, 38 active+clean+remapped; 3.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 4.5 KiB/s rd, 7.7 MiB/s wr, 128 op/s; 25928/185436 objects degraded (13.982%); 29505/185436 objects misplaced (15.911%); 4.8 MiB/s, 84 objects/s recovering 2024-03-30T13:53:41.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:40 smithi043 ceph-mon[42099]: pgmap v460: 158 pgs: 86 active+clean, 5 active+recovering+undersized+remapped, 14 active+undersized, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 9 active+undersized+degraded, 38 active+clean+remapped; 3.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 4.5 KiB/s rd, 7.7 MiB/s wr, 128 op/s; 25928/185436 objects degraded (13.982%); 29505/185436 objects misplaced (15.911%); 4.8 MiB/s, 84 objects/s recovering 2024-03-30T13:53:41.154 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:41.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:41.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:41.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:40 smithi113 ceph-mon[34341]: pgmap v460: 158 pgs: 86 active+clean, 5 active+recovering+undersized+remapped, 14 active+undersized, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 9 active+undersized+degraded, 38 active+clean+remapped; 3.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 4.5 KiB/s rd, 7.7 MiB/s wr, 128 op/s; 25928/185436 objects degraded (13.982%); 29505/185436 objects misplaced (15.911%); 4.8 MiB/s, 84 objects/s recovering 2024-03-30T13:53:41.256 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:41.297 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:41.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:41.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:41.399 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:41.426 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:41.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:41.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:41.528 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:41.552 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:41.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:41.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:41.654 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:41.679 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:41.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:41.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:41.694 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 50 16 63356 63340 79.1596 32.75 0.0164113 0.0126251 2024-03-30T13:53:41.781 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:41.810 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:41.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:41.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:41.912 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:41.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:41.957 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:41.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:42.058 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:42.083 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:42.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:42.184 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:42.225 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:42.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:42.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:42.327 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:42.350 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:42.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:42.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:42.452 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:42.493 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:42.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:42.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:42.595 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:42.622 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:42.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:42.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:42.694 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 51 16 63935 63919 78.317 36.1875 0.00828821 0.012761 2024-03-30T13:53:42.724 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:42.748 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:42.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:42.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:42.849 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:42.875 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:42.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:42.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:42.882 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T13:53:42.977 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:43.001 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:43.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:43.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:43.103 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:43.142 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:43.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:43.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:43.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:42 smithi043 ceph-mon[29963]: pgmap v461: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 3.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 13 MiB/s wr, 214 op/s; 41886/187479 objects degraded (22.342%); 28642/187479 objects misplaced (15.277%); 10 MiB/s, 173 objects/s recovering 2024-03-30T13:53:43.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:42 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1008527491' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:53:43.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:42 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:53:43.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:42 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 41886/187479 objects degraded (22.342%), 16 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:43.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:42 smithi043 ceph-mon[42099]: pgmap v461: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 3.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 13 MiB/s wr, 214 op/s; 41886/187479 objects degraded (22.342%); 28642/187479 objects misplaced (15.277%); 10 MiB/s, 173 objects/s recovering 2024-03-30T13:53:43.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:42 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1008527491' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:53:43.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:42 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:53:43.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:42 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 41886/187479 objects degraded (22.342%), 16 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:43.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:42 smithi113 ceph-mon[34341]: pgmap v461: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 3.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 13 MiB/s wr, 214 op/s; 41886/187479 objects degraded (22.342%); 28642/187479 objects misplaced (15.277%); 10 MiB/s, 173 objects/s recovering 2024-03-30T13:53:43.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:42 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1008527491' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:53:43.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:42 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:53:43.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:42 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 41886/187479 objects degraded (22.342%), 16 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:43.244 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:43.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:43.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:43.387 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:43.415 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:43.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:43.516 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:43.543 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:43.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:43.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:43.645 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:43.693 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:43.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:43.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:43.694 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 52 16 64482 64466 77.4684 34.1875 0.0418017 0.0129013 2024-03-30T13:53:43.795 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:43.821 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:43.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:43.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:43.923 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:43.964 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:43.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:43.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:44.065 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:44.094 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:44.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:44.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:44.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:43 smithi043 ceph-mon[29963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:53:44.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:43 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:53:44.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:43 smithi043 ceph-mon[29963]: osdmap e233: 8 total, 7 up, 6 in 2024-03-30T13:53:44.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:43 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:53:44.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:43 smithi043 ceph-mon[42099]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:53:44.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:43 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:53:44.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:43 smithi043 ceph-mon[42099]: osdmap e233: 8 total, 7 up, 6 in 2024-03-30T13:53:44.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:43 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:53:44.196 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:44.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:43 smithi113 ceph-mon[34341]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:53:44.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:43 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:53:44.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:43 smithi113 ceph-mon[34341]: osdmap e233: 8 total, 7 up, 6 in 2024-03-30T13:53:44.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:43 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:53:44.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:44.237 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:44.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:44.338 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:44.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:44.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:44.489 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:44.515 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:44.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:44.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:44.617 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:44.644 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:44.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:44.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:44.694 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 53 16 65076 65060 76.7072 37.125 0.0125919 0.0130291 2024-03-30T13:53:44.746 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:44.794 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:44.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:44.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:44.895 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:44.921 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:44.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:44.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:45.023 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:45.050 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:45.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:45.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:45.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:44 smithi043 ceph-mon[29963]: pgmap v463: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 3.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 12 MiB/s wr, 202 op/s; 41886/187479 objects degraded (22.342%); 28642/187479 objects misplaced (15.277%); 9.8 MiB/s, 163 objects/s recovering 2024-03-30T13:53:45.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:44 smithi043 ceph-mon[42099]: pgmap v463: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 3.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 12 MiB/s wr, 202 op/s; 41886/187479 objects degraded (22.342%); 28642/187479 objects misplaced (15.277%); 9.8 MiB/s, 163 objects/s recovering 2024-03-30T13:53:45.152 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:45.177 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:45.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:45.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:45.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:44 smithi113 ceph-mon[34341]: pgmap v463: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 3.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 12 MiB/s wr, 202 op/s; 41886/187479 objects degraded (22.342%); 28642/187479 objects misplaced (15.277%); 9.8 MiB/s, 163 objects/s recovering 2024-03-30T13:53:45.279 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:45.302 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:45.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:45.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:45.404 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:45.432 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:45.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:45.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:45.534 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:45.578 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:45.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:45.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:45.680 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:45.694 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 54 16 65661 65645 75.9638 36.5625 0.0133924 0.0131562 2024-03-30T13:53:45.706 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:45.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:45.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:45.808 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:45.859 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:45.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:45.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:45.960 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:45.988 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:45.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:45.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:46.090 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:46.136 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:46.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:46.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:46.238 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:46.265 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:46.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:46.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:46.367 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:46.391 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:46.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:46.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:46.396 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:53:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:53:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:53:46.493 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:46.520 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:46.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:46.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:46.622 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:46.689 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:46.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:46.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:46.694 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 55 16 66250 66234 75.2519 36.8125 0.0163905 0.0132808 2024-03-30T13:53:46.791 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:46.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:46.820 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:46.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:46.921 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:46.950 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:46.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:46.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:46.996 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:46 smithi043 ceph-mon[29963]: pgmap v464: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.0 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 30 MiB/s wr, 482 op/s; 42442/195417 objects degraded (21.719%); 28154/195417 objects misplaced (14.407%); 15 MiB/s, 244 objects/s recovering 2024-03-30T13:53:46.997 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:46 smithi043 ceph-mon[42099]: pgmap v464: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.0 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 30 MiB/s wr, 482 op/s; 42442/195417 objects degraded (21.719%); 28154/195417 objects misplaced (14.407%); 15 MiB/s, 244 objects/s recovering 2024-03-30T13:53:47.052 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:47.116 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:47.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:47.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:47.218 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:47.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:46 smithi113 ceph-mon[34341]: pgmap v464: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.0 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 30 MiB/s wr, 482 op/s; 42442/195417 objects degraded (21.719%); 28154/195417 objects misplaced (14.407%); 15 MiB/s, 244 objects/s recovering 2024-03-30T13:53:47.267 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:47.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:47.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:47.369 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:47.409 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:47.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:47.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:47.511 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:47.536 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:47.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:47.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:47.638 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:47.666 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:47.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:47.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:47.694 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 56 16 66845 66829 74.5722 37.1875 0.0110463 0.013403 2024-03-30T13:53:47.768 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:47.796 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:47.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:47.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:47.898 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:47.931 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:53:47.931 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-03-30T13:53:47.931 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-03-30T13:53:47.931 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-03-30T13:53:47.931 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.931 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:31281 8.e 8:78ec8ffe:::benchmark_data_smithi043_119832_object31280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-30T13:53:47.931 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:49:27.603455+0000", 2024-03-30T13:53:47.931 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 259.60764964200001, 2024-03-30T13:53:47.932 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.177000824, 2024-03-30T13:53:47.932 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:53:47.932 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:53:47.932 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:47.932 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:53:47.932 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:53:47.932 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:53:47.932 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 31281 2024-03-30T13:53:47.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.932 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:53:47.932 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.932 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:53:47.933 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.603455+0000", 2024-03-30T13:53:47.933 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:47.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.933 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:53:47.933 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.603455+0000", 2024-03-30T13:53:47.933 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:47.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.933 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:53:47.933 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.603456+0000", 2024-03-30T13:53:47.934 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.61e-07 2024-03-30T13:53:47.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.934 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.934 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:53:47.934 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.603582+0000", 2024-03-30T13:53:47.934 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012653499999999999 2024-03-30T13:53:47.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.934 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.934 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:53:47.934 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.603593+0000", 2024-03-30T13:53:47.934 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0295000000000001e-05 2024-03-30T13:53:47.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.935 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:53:47.935 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.603601+0000", 2024-03-30T13:53:47.935 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1149999999999994e-06 2024-03-30T13:53:47.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.935 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:47.935 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.603822+0000", 2024-03-30T13:53:47.935 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000221253 2024-03-30T13:53:47.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.935 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:53:47.936 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.604003+0000", 2024-03-30T13:53:47.936 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000180784 2024-03-30T13:53:47.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.936 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,4", 2024-03-30T13:53:47.936 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.604182+0000", 2024-03-30T13:53:47.936 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000179288 2024-03-30T13:53:47.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.936 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:53:47.936 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.607465+0000", 2024-03-30T13:53:47.936 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0032827609999999999 2024-03-30T13:53:47.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.937 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:47.937 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.610200+0000", 2024-03-30T13:53:47.937 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0027350090000000001 2024-03-30T13:53:47.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.937 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:47.937 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780338+0000", 2024-03-30T13:53:47.937 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.170137546 2024-03-30T13:53:47.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.938 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:53:47.938 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780403+0000", 2024-03-30T13:53:47.938 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5133000000000001e-05 2024-03-30T13:53:47.938 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.938 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.938 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:53:47.938 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780456+0000", 2024-03-30T13:53:47.938 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3443999999999999e-05 2024-03-30T13:53:47.938 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:47.938 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:47.938 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:47.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.939 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:31288 8.e 8:77d8c65d:::benchmark_data_smithi043_119832_object31287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-30T13:53:47.939 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:49:27.604854+0000", 2024-03-30T13:53:47.939 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 259.60625042100003, 2024-03-30T13:53:47.939 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.175763427, 2024-03-30T13:53:47.939 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:53:47.939 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:53:47.939 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:47.939 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:53:47.939 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:53:47.939 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:53:47.939 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 31288 2024-03-30T13:53:47.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.940 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:53:47.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.940 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:53:47.940 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.604854+0000", 2024-03-30T13:53:47.940 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:47.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.940 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:53:47.940 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.604854+0000", 2024-03-30T13:53:47.940 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:47.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.941 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:53:47.941 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.604856+0000", 2024-03-30T13:53:47.941 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0920000000000001e-06 2024-03-30T13:53:47.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.941 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:53:47.941 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.604915+0000", 2024-03-30T13:53:47.941 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9020000000000001e-05 2024-03-30T13:53:47.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.941 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:53:47.942 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.604932+0000", 2024-03-30T13:53:47.942 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7187999999999999e-05 2024-03-30T13:53:47.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.942 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:53:47.942 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.604941+0000", 2024-03-30T13:53:47.942 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0159999999999997e-06 2024-03-30T13:53:47.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.942 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:47.942 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.605523+0000", 2024-03-30T13:53:47.942 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00058246899999999996 2024-03-30T13:53:47.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.943 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:53:47.943 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.605644+0000", 2024-03-30T13:53:47.943 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012092200000000001 2024-03-30T13:53:47.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.943 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,4", 2024-03-30T13:53:47.943 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.605969+0000", 2024-03-30T13:53:47.943 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00032491600000000002 2024-03-30T13:53:47.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.944 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:53:47.944 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.608743+0000", 2024-03-30T13:53:47.944 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0027743469999999999 2024-03-30T13:53:47.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.944 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:47.944 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.610790+0000", 2024-03-30T13:53:47.944 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0020461899999999998 2024-03-30T13:53:47.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.944 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:47.944 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780518+0000", 2024-03-30T13:53:47.945 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.16972848500000001 2024-03-30T13:53:47.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.945 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:53:47.945 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780577+0000", 2024-03-30T13:53:47.945 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8403999999999999e-05 2024-03-30T13:53:47.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.945 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:53:47.945 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:49:27.780618+0000", 2024-03-30T13:53:47.945 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1378000000000003e-05 2024-03-30T13:53:47.945 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:47.946 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:47.946 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:47.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.946 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24977.0:250163 8.c 8:3dc0bff9:::benchmark_data_smithi043_119832_object119809:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:53:47.946 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:52:19.661343+0000", 2024-03-30T13:53:47.946 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 87.549762125000001, 2024-03-30T13:53:47.946 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17045159900000001, 2024-03-30T13:53:47.946 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:53:47.946 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:53:47.946 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:47.946 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:53:47.947 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24977", 2024-03-30T13:53:47.947 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:53:47.947 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 250163 2024-03-30T13:53:47.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.947 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:53:47.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.947 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:53:47.947 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.661343+0000", 2024-03-30T13:53:47.947 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:47.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.948 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:53:47.948 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.661343+0000", 2024-03-30T13:53:47.948 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:47.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.948 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:53:47.948 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.661344+0000", 2024-03-30T13:53:47.948 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5799999999999998e-07 2024-03-30T13:53:47.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.948 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:53:47.948 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.661348+0000", 2024-03-30T13:53:47.949 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3050000000000003e-06 2024-03-30T13:53:47.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.949 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:53:47.949 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.661357+0000", 2024-03-30T13:53:47.949 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.816e-06 2024-03-30T13:53:47.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.949 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:53:47.949 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.661362+0000", 2024-03-30T13:53:47.949 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5690000000000004e-06 2024-03-30T13:53:47.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.950 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:47.950 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.662206+0000", 2024-03-30T13:53:47.950 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00084358199999999995 2024-03-30T13:53:47.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.950 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:53:47.950 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.662279+0000", 2024-03-30T13:53:47.950 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3246999999999999e-05 2024-03-30T13:53:47.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.950 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:53:47.950 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.662354+0000", 2024-03-30T13:53:47.951 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4503000000000001e-05 2024-03-30T13:53:47.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.951 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:53:47.951 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.664761+0000", 2024-03-30T13:53:47.951 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024073160000000001 2024-03-30T13:53:47.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.951 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:47.951 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.665636+0000", 2024-03-30T13:53:47.951 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00087550300000000002 2024-03-30T13:53:47.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.952 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:47.952 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.669095+0000", 2024-03-30T13:53:47.952 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0034584310000000001 2024-03-30T13:53:47.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.952 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:53:47.952 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.831742+0000", 2024-03-30T13:53:47.952 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.16264714 2024-03-30T13:53:47.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.952 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:53:47.952 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.831794+0000", 2024-03-30T13:53:47.953 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2329e-05 2024-03-30T13:53:47.953 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:47.953 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:47.953 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:47.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.953 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.24977.0:250171 8.2 e200/147 8:413c74b9:::benchmark_data_smithi043_119832_object119817:head v 200'15760, mlcod=200'15758)", 2024-03-30T13:53:47.953 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:52:19.669233+0000", 2024-03-30T13:53:47.953 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 87.541872354999995, 2024-03-30T13:53:47.953 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17043029800000001, 2024-03-30T13:53:47.953 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:53:47.953 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:53:47.953 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:47.954 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:53:47.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.954 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:53:47.954 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.669233+0000", 2024-03-30T13:53:47.954 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:47.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.954 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:53:47.954 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.669233+0000", 2024-03-30T13:53:47.954 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:47.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.955 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:53:47.955 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.669233+0000", 2024-03-30T13:53:47.955 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8199999999999999e-07 2024-03-30T13:53:47.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.955 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:53:47.955 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.669239+0000", 2024-03-30T13:53:47.955 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2339999999999999e-06 2024-03-30T13:53:47.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.955 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:53:47.955 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.831047+0000", 2024-03-30T13:53:47.956 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.161807905 2024-03-30T13:53:47.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.956 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:53:47.956 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.831069+0000", 2024-03-30T13:53:47.956 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1662e-05 2024-03-30T13:53:47.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.956 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:47.956 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.831422+0000", 2024-03-30T13:53:47.956 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00035306299999999998 2024-03-30T13:53:47.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.957 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:53:47.957 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.831443+0000", 2024-03-30T13:53:47.957 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1121999999999999e-05 2024-03-30T13:53:47.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.957 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:53:47.957 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.839620+0000", 2024-03-30T13:53:47.957 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0081766740000000001 2024-03-30T13:53:47.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.957 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:53:47.957 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:52:19.839663+0000", 2024-03-30T13:53:47.958 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2956e-05 2024-03-30T13:53:47.958 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:47.958 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:47.958 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:47.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.958 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:56045 9.9 9:9d125b1d:::benchmark_data_smithi043_145506_object56044:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-30T13:53:47.958 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:27.623224+0000", 2024-03-30T13:53:47.958 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 19.587880570999999, 2024-03-30T13:53:47.958 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.19522012399999999, 2024-03-30T13:53:47.958 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:53:47.958 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:53:47.958 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:47.959 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:53:47.959 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:53:47.959 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:53:47.959 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 56045 2024-03-30T13:53:47.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.959 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:53:47.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:53:47.959 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.623224+0000", 2024-03-30T13:53:47.959 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:47.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:53:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.623224+0000", 2024-03-30T13:53:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:53:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.623225+0000", 2024-03-30T13:53:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1400000000000001e-06 2024-03-30T13:53:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:53:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.623247+0000", 2024-03-30T13:53:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1265000000000001e-05 2024-03-30T13:53:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:53:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.623254+0000", 2024-03-30T13:53:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1350000000000003e-06 2024-03-30T13:53:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:53:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.623259+0000", 2024-03-30T13:53:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5659999999999997e-06 2024-03-30T13:53:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.630735+0000", 2024-03-30T13:53:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0074759639999999999 2024-03-30T13:53:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:53:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.630909+0000", 2024-03-30T13:53:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017374400000000001 2024-03-30T13:53:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:53:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.631087+0000", 2024-03-30T13:53:47.963 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017782799999999999 2024-03-30T13:53:47.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.963 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:53:47.963 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.635848+0000", 2024-03-30T13:53:47.963 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0047612590000000003 2024-03-30T13:53:47.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.963 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:47.963 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638798+0000", 2024-03-30T13:53:47.963 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0029492820000000001 2024-03-30T13:53:47.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.964 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:47.964 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.816562+0000", 2024-03-30T13:53:47.964 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.177764483 2024-03-30T13:53:47.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.964 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:53:47.964 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.816647+0000", 2024-03-30T13:53:47.964 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5346000000000006e-05 2024-03-30T13:53:47.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.965 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:53:47.965 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.818444+0000", 2024-03-30T13:53:47.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001797112 2024-03-30T13:53:47.965 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:47.965 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:47.965 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:47.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.965 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:56049 9.9 9:9a2cdab1:::benchmark_data_smithi043_145506_object56048:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-30T13:53:47.965 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:27.626166+0000", 2024-03-30T13:53:47.965 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 19.584939127999998, 2024-03-30T13:53:47.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.194835114, 2024-03-30T13:53:47.966 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:53:47.966 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:53:47.966 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:47.966 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:53:47.966 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:53:47.966 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:53:47.966 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 56049 2024-03-30T13:53:47.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.966 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:53:47.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.966 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:53:47.966 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626166+0000", 2024-03-30T13:53:47.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:47.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.967 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:53:47.967 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626166+0000", 2024-03-30T13:53:47.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:47.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.967 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:53:47.967 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626168+0000", 2024-03-30T13:53:47.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9290000000000001e-06 2024-03-30T13:53:47.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.968 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:53:47.968 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626196+0000", 2024-03-30T13:53:47.968 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.8492000000000001e-05 2024-03-30T13:53:47.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.968 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:53:47.968 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626210+0000", 2024-03-30T13:53:47.968 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3713999999999999e-05 2024-03-30T13:53:47.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.968 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:53:47.968 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626220+0000", 2024-03-30T13:53:47.968 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0210999999999999e-05 2024-03-30T13:53:47.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.969 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:47.969 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.633332+0000", 2024-03-30T13:53:47.969 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0071117130000000004 2024-03-30T13:53:47.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.969 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:53:47.969 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.633484+0000", 2024-03-30T13:53:47.969 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015213600000000001 2024-03-30T13:53:47.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.969 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:53:47.970 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.633660+0000", 2024-03-30T13:53:47.970 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017599299999999999 2024-03-30T13:53:47.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.970 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:47.970 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638845+0000", 2024-03-30T13:53:47.970 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.005185182 2024-03-30T13:53:47.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.970 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:53:47.970 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.788979+0000", 2024-03-30T13:53:47.970 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.150133868 2024-03-30T13:53:47.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.971 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:47.971 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.818506+0000", 2024-03-30T13:53:47.971 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.029527241999999999 2024-03-30T13:53:47.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.971 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:53:47.971 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.818764+0000", 2024-03-30T13:53:47.971 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00025784799999999997 2024-03-30T13:53:47.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.971 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:53:47.972 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.821001+0000", 2024-03-30T13:53:47.972 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022367860000000002 2024-03-30T13:53:47.972 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:47.972 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:47.972 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:47.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.972 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:56054 9.9 9:94de70d2:::benchmark_data_smithi043_145506_object56053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-30T13:53:47.972 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:27.626948+0000", 2024-03-30T13:53:47.972 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 19.584157056999999, 2024-03-30T13:53:47.972 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.19715785699999999, 2024-03-30T13:53:47.972 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:53:47.973 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:53:47.973 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:47.973 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:53:47.973 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:53:47.973 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:53:47.973 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 56054 2024-03-30T13:53:47.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.973 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:53:47.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.973 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:53:47.973 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626948+0000", 2024-03-30T13:53:47.973 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:47.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.974 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:53:47.974 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626948+0000", 2024-03-30T13:53:47.974 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:47.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.974 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:53:47.974 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626949+0000", 2024-03-30T13:53:47.974 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1060000000000001e-06 2024-03-30T13:53:47.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.974 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:53:47.975 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626971+0000", 2024-03-30T13:53:47.975 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1872e-05 2024-03-30T13:53:47.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.975 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.975 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:53:47.975 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626983+0000", 2024-03-30T13:53:47.975 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2349e-05 2024-03-30T13:53:47.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.975 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.975 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:53:47.975 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626992+0000", 2024-03-30T13:53:47.975 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3639999999999999e-06 2024-03-30T13:53:47.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.976 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:47.976 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.636090+0000", 2024-03-30T13:53:47.976 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0090986650000000006 2024-03-30T13:53:47.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.976 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:53:47.976 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.636249+0000", 2024-03-30T13:53:47.976 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015851 2024-03-30T13:53:47.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.977 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:53:47.977 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.636486+0000", 2024-03-30T13:53:47.977 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000236958 2024-03-30T13:53:47.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.977 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:47.977 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638903+0000", 2024-03-30T13:53:47.977 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024170289999999998 2024-03-30T13:53:47.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.977 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:53:47.978 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.788986+0000", 2024-03-30T13:53:47.978 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.150083314 2024-03-30T13:53:47.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.978 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:47.978 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.821064+0000", 2024-03-30T13:53:47.978 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.032077556 2024-03-30T13:53:47.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.978 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:53:47.978 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.821770+0000", 2024-03-30T13:53:47.978 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00070655000000000002 2024-03-30T13:53:47.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.979 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:53:47.979 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.824106+0000", 2024-03-30T13:53:47.979 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0023355839999999999 2024-03-30T13:53:47.979 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:47.979 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:47.979 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:47.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.979 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:56056 9.9 9:90e814ba:::benchmark_data_smithi043_145506_object56055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-30T13:53:47.980 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:27.628943+0000", 2024-03-30T13:53:47.980 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 19.582162328999999, 2024-03-30T13:53:47.980 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.20046303700000001, 2024-03-30T13:53:47.980 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:53:47.980 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:53:47.980 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:47.980 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:53:47.980 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:53:47.980 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:53:47.980 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 56056 2024-03-30T13:53:47.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.980 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:53:47.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.981 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:53:47.981 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.628943+0000", 2024-03-30T13:53:47.981 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:47.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.981 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:53:47.981 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.628943+0000", 2024-03-30T13:53:47.981 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:47.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.981 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:53:47.982 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.628944+0000", 2024-03-30T13:53:47.982 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0300000000000001e-06 2024-03-30T13:53:47.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.982 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:53:47.982 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.628963+0000", 2024-03-30T13:53:47.982 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9885999999999999e-05 2024-03-30T13:53:47.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.982 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:53:47.982 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.628972+0000", 2024-03-30T13:53:47.983 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2689999999999999e-06 2024-03-30T13:53:47.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.983 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:53:47.983 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.628978+0000", 2024-03-30T13:53:47.983 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8989999999999998e-06 2024-03-30T13:53:47.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.983 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:47.983 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638945+0000", 2024-03-30T13:53:47.983 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0099676839999999992 2024-03-30T13:53:47.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.984 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:53:47.984 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.639093+0000", 2024-03-30T13:53:47.984 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014789 2024-03-30T13:53:47.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.984 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:53:47.984 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.639261+0000", 2024-03-30T13:53:47.984 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016739500000000001 2024-03-30T13:53:47.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.984 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:47.985 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.788960+0000", 2024-03-30T13:53:47.985 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.149699096 2024-03-30T13:53:47.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.985 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:53:47.985 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.788993+0000", 2024-03-30T13:53:47.985 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.2978999999999997e-05 2024-03-30T13:53:47.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.985 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:47.986 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.824162+0000", 2024-03-30T13:53:47.986 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.035169259000000001 2024-03-30T13:53:47.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.986 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:53:47.986 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.824379+0000", 2024-03-30T13:53:47.986 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00021719600000000001 2024-03-30T13:53:47.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.986 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:53:47.986 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.829406+0000", 2024-03-30T13:53:47.986 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0050264539999999996 2024-03-30T13:53:47.987 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:47.987 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:47.987 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:47.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.987 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:56058 9.5 9:aae71768:::benchmark_data_smithi043_145506_object56057:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-30T13:53:47.987 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:27.634637+0000", 2024-03-30T13:53:47.987 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 19.576467496999999, 2024-03-30T13:53:47.987 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.16738136000000001, 2024-03-30T13:53:47.987 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:53:47.987 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:53:47.988 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:47.988 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:53:47.988 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:53:47.988 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:53:47.988 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 56058 2024-03-30T13:53:47.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.988 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:53:47.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.988 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:53:47.988 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.634637+0000", 2024-03-30T13:53:47.988 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:47.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.989 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:53:47.989 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.634637+0000", 2024-03-30T13:53:47.989 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:47.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.989 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:53:47.989 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.634639+0000", 2024-03-30T13:53:47.989 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.505e-06 2024-03-30T13:53:47.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.989 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:53:47.989 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.634665+0000", 2024-03-30T13:53:47.989 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5894999999999999e-05 2024-03-30T13:53:47.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.990 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:53:47.990 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.634677+0000", 2024-03-30T13:53:47.990 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2542e-05 2024-03-30T13:53:47.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.990 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:53:47.990 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.634685+0000", 2024-03-30T13:53:47.990 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6399999999999997e-06 2024-03-30T13:53:47.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.991 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:47.991 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638878+0000", 2024-03-30T13:53:47.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0041928529999999999 2024-03-30T13:53:47.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.991 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:53:47.991 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.639024+0000", 2024-03-30T13:53:47.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014612499999999999 2024-03-30T13:53:47.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.991 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,7", 2024-03-30T13:53:47.991 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.639195+0000", 2024-03-30T13:53:47.992 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017087100000000001 2024-03-30T13:53:47.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.992 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:47.992 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.799158+0000", 2024-03-30T13:53:47.992 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.159962982 2024-03-30T13:53:47.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.992 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:53:47.992 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.799172+0000", 2024-03-30T13:53:47.992 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4525000000000001e-05 2024-03-30T13:53:47.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.993 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:47.993 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.799214+0000", 2024-03-30T13:53:47.993 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1526999999999997e-05 2024-03-30T13:53:47.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.993 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:53:47.993 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.799273+0000", 2024-03-30T13:53:47.993 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9425000000000001e-05 2024-03-30T13:53:47.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.993 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:53:47.994 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.802019+0000", 2024-03-30T13:53:47.994 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0027454699999999999 2024-03-30T13:53:47.994 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:47.994 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:47.994 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:47.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.994 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:56060 9.9 9:969aa2d0:::benchmark_data_smithi043_145506_object56059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-30T13:53:47.994 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:27.634703+0000", 2024-03-30T13:53:47.994 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 19.576401747999999, 2024-03-30T13:53:47.994 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.20106921799999999, 2024-03-30T13:53:47.994 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:53:47.995 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:53:47.995 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:47.995 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:53:47.995 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:53:47.995 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:53:47.995 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 56060 2024-03-30T13:53:47.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.995 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:53:47.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.995 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:53:47.995 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.634703+0000", 2024-03-30T13:53:47.995 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:47.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.996 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:53:47.996 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.634703+0000", 2024-03-30T13:53:47.996 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:47.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.996 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:53:47.996 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.634704+0000", 2024-03-30T13:53:47.996 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2e-07 2024-03-30T13:53:47.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.997 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:53:47.997 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.634723+0000", 2024-03-30T13:53:47.997 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9125000000000001e-05 2024-03-30T13:53:47.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.997 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:53:47.997 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.634784+0000", 2024-03-30T13:53:47.997 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0797000000000001e-05 2024-03-30T13:53:47.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.997 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:53:47.997 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.634790+0000", 2024-03-30T13:53:47.998 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3010000000000003e-06 2024-03-30T13:53:47.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.998 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:47.998 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.641402+0000", 2024-03-30T13:53:47.998 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0066118039999999998 2024-03-30T13:53:47.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.998 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:53:47.998 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.641547+0000", 2024-03-30T13:53:47.998 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014569 2024-03-30T13:53:47.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.999 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:53:47.999 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.641656+0000", 2024-03-30T13:53:47.999 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000108305 2024-03-30T13:53:47.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.999 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:53:47.999 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.788999+0000", 2024-03-30T13:53:47.999 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14734349499999999 2024-03-30T13:53:47.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:47.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:47.999 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:47.999 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.802260+0000", 2024-03-30T13:53:48.000 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.013260625999999999 2024-03-30T13:53:48.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.000 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:48.000 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.829466+0000", 2024-03-30T13:53:48.000 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.027206096999999999 2024-03-30T13:53:48.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.000 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:53:48.000 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.829525+0000", 2024-03-30T13:53:48.000 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8798999999999997e-05 2024-03-30T13:53:48.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.001 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:53:48.001 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.835772+0000", 2024-03-30T13:53:48.001 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.006247659 2024-03-30T13:53:48.001 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.001 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:48.001 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.001 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:56064 9.5 9:aa87c190:::benchmark_data_smithi043_145506_object56063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-30T13:53:48.001 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:27.638259+0000", 2024-03-30T13:53:48.001 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 19.572846332000001, 2024-03-30T13:53:48.002 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.18948137900000001, 2024-03-30T13:53:48.002 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:53:48.002 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:53:48.002 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:48.002 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:53:48.002 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:53:48.002 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:53:48.002 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 56064 2024-03-30T13:53:48.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.002 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:53:48.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.002 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:53:48.003 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638259+0000", 2024-03-30T13:53:48.003 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:48.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.003 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:53:48.003 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638259+0000", 2024-03-30T13:53:48.003 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:48.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.003 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:53:48.003 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638259+0000", 2024-03-30T13:53:48.003 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1500000000000005e-07 2024-03-30T13:53:48.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.004 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:53:48.004 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638280+0000", 2024-03-30T13:53:48.004 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1359999999999999e-05 2024-03-30T13:53:48.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.004 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:53:48.004 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638288+0000", 2024-03-30T13:53:48.004 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.571e-06 2024-03-30T13:53:48.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.004 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:53:48.005 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638294+0000", 2024-03-30T13:53:48.005 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.981e-06 2024-03-30T13:53:48.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.005 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:48.005 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.796790+0000", 2024-03-30T13:53:48.005 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.15849614300000001 2024-03-30T13:53:48.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.005 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:53:48.005 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.796977+0000", 2024-03-30T13:53:48.005 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018698400000000001 2024-03-30T13:53:48.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.006 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,7", 2024-03-30T13:53:48.006 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.797148+0000", 2024-03-30T13:53:48.006 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000170755 2024-03-30T13:53:48.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.006 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:53:48.006 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.810226+0000", 2024-03-30T13:53:48.006 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.013078485000000001 2024-03-30T13:53:48.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.006 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:48.006 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.810298+0000", 2024-03-30T13:53:48.007 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1335999999999996e-05 2024-03-30T13:53:48.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.007 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:48.007 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.823041+0000", 2024-03-30T13:53:48.007 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.012742974000000001 2024-03-30T13:53:48.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.007 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:53:48.007 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.823104+0000", 2024-03-30T13:53:48.007 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3792000000000002e-05 2024-03-30T13:53:48.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.008 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:53:48.008 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.827740+0000", 2024-03-30T13:53:48.008 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0046354830000000001 2024-03-30T13:53:48.008 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.008 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:48.008 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.008 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61013 9.5 9:abc7156e:::benchmark_data_smithi043_145506_object61012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-30T13:53:48.008 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:33.735921+0000", 2024-03-30T13:53:48.008 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 13.475184199999999, 2024-03-30T13:53:48.008 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.19262681400000001, 2024-03-30T13:53:48.009 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:53:48.009 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:53:48.009 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:48.009 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:53:48.009 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:53:48.009 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:53:48.009 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61013 2024-03-30T13:53:48.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.009 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:53:48.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.009 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:53:48.009 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.735921+0000", 2024-03-30T13:53:48.009 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:48.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.010 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:53:48.010 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.735921+0000", 2024-03-30T13:53:48.010 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:48.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.010 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:53:48.010 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.735922+0000", 2024-03-30T13:53:48.010 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3030000000000001e-06 2024-03-30T13:53:48.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.011 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:53:48.011 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.735947+0000", 2024-03-30T13:53:48.011 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5089999999999999e-05 2024-03-30T13:53:48.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.011 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:53:48.011 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.735961+0000", 2024-03-30T13:53:48.011 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3869999999999999e-05 2024-03-30T13:53:48.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.011 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:53:48.011 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.735970+0000", 2024-03-30T13:53:48.012 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0580000000000004e-06 2024-03-30T13:53:48.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.012 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:48.012 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.738486+0000", 2024-03-30T13:53:48.012 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0025157740000000001 2024-03-30T13:53:48.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.012 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:53:48.012 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.738632+0000", 2024-03-30T13:53:48.012 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000145983 2024-03-30T13:53:48.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.013 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,7", 2024-03-30T13:53:48.013 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.738791+0000", 2024-03-30T13:53:48.013 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015888999999999999 2024-03-30T13:53:48.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.013 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:53:48.013 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.741638+0000", 2024-03-30T13:53:48.013 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0028475689999999999 2024-03-30T13:53:48.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.013 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:48.013 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.744616+0000", 2024-03-30T13:53:48.014 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0029772689999999998 2024-03-30T13:53:48.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.014 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:48.014 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.926765+0000", 2024-03-30T13:53:48.014 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.182149847 2024-03-30T13:53:48.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.014 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:53:48.014 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.926825+0000", 2024-03-30T13:53:48.014 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0127000000000002e-05 2024-03-30T13:53:48.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.015 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:53:48.015 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.928548+0000", 2024-03-30T13:53:48.015 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0017220340000000001 2024-03-30T13:53:48.015 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.015 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:48.015 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.015 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14451.0:2653 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94764809514240 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-03-30T13:53:48.015 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:35.846463+0000", 2024-03-30T13:53:48.015 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 11.364641812, 2024-03-30T13:53:48.015 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.741426648, 2024-03-30T13:53:48.016 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:53:48.016 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:53:48.016 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T13:53:48.016 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:53:48.016 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14451", 2024-03-30T13:53:48.016 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T13:53:48.016 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 2653 2024-03-30T13:53:48.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.016 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:53:48.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.016 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:53:48.016 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.846463+0000", 2024-03-30T13:53:48.017 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:48.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.017 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:53:48.017 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.846463+0000", 2024-03-30T13:53:48.017 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:48.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.017 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:53:48.017 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.846464+0000", 2024-03-30T13:53:48.017 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.322e-06 2024-03-30T13:53:48.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.018 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:53:48.018 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.846470+0000", 2024-03-30T13:53:48.018 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8449999999999999e-06 2024-03-30T13:53:48.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.018 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:53:48.018 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.847878+0000", 2024-03-30T13:53:48.018 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0014079959999999999 2024-03-30T13:53:48.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.018 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:53:48.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.847890+0000", 2024-03-30T13:53:48.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1647999999999999e-05 2024-03-30T13:53:48.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.019 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:48.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.854592+0000", 2024-03-30T13:53:48.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.006701762 2024-03-30T13:53:48.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.019 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:53:48.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.854603+0000", 2024-03-30T13:53:48.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1771e-05 2024-03-30T13:53:48.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:48.020 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.557525+0000", 2024-03-30T13:53:48.020 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.702921425 2024-03-30T13:53:48.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for degraded object", 2024-03-30T13:53:48.020 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.559277+0000", 2024-03-30T13:53:48.020 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001752219 2024-03-30T13:53:48.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:48.020 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.562914+0000", 2024-03-30T13:53:48.021 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003636996 2024-03-30T13:53:48.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.021 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:53:48.021 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.563000+0000", 2024-03-30T13:53:48.021 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5944000000000002e-05 2024-03-30T13:53:48.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.021 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:53:48.021 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.587890+0000", 2024-03-30T13:53:48.021 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.024889720000000001 2024-03-30T13:53:48.021 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.022 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:48.022 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.022 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61235 9.5 9:a316fbe6:::benchmark_data_smithi043_145506_object61234:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e230)", 2024-03-30T13:53:48.022 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:37.375080+0000", 2024-03-30T13:53:48.022 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 9.8360250600000008, 2024-03-30T13:53:48.022 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.21255780299999999, 2024-03-30T13:53:48.022 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:53:48.022 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:53:48.022 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:48.022 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:53:48.023 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:53:48.023 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:53:48.023 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61235 2024-03-30T13:53:48.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.023 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:53:48.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.023 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:53:48.023 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.375080+0000", 2024-03-30T13:53:48.023 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:48.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.023 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:53:48.023 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.375080+0000", 2024-03-30T13:53:48.024 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:48.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.024 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:53:48.024 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.375081+0000", 2024-03-30T13:53:48.024 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.482e-06 2024-03-30T13:53:48.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.024 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:53:48.024 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.375104+0000", 2024-03-30T13:53:48.024 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3110000000000001e-05 2024-03-30T13:53:48.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.025 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:53:48.025 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.375116+0000", 2024-03-30T13:53:48.025 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.137e-05 2024-03-30T13:53:48.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.025 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:53:48.025 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.375125+0000", 2024-03-30T13:53:48.025 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5869999999999999e-06 2024-03-30T13:53:48.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.025 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:48.025 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.559403+0000", 2024-03-30T13:53:48.025 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.18427763699999999 2024-03-30T13:53:48.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.026 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:53:48.026 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.559556+0000", 2024-03-30T13:53:48.026 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015315199999999999 2024-03-30T13:53:48.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.026 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:53:48.026 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.559772+0000", 2024-03-30T13:53:48.026 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000216135 2024-03-30T13:53:48.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.027 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:53:48.027 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.569599+0000", 2024-03-30T13:53:48.027 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0098271369999999997 2024-03-30T13:53:48.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.027 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:48.027 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.583927+0000", 2024-03-30T13:53:48.027 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.014327555 2024-03-30T13:53:48.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.028 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:48.028 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.584054+0000", 2024-03-30T13:53:48.028 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012727300000000001 2024-03-30T13:53:48.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.028 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:53:48.028 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.584116+0000", 2024-03-30T13:53:48.028 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1396999999999995e-05 2024-03-30T13:53:48.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.028 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:53:48.028 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.587638+0000", 2024-03-30T13:53:48.029 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0035219679999999999 2024-03-30T13:53:48.029 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.029 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:48.029 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.029 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61236 9.5 9:ae283ce4:::benchmark_data_smithi043_145506_object61235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e230)", 2024-03-30T13:53:48.029 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:37.384596+0000", 2024-03-30T13:53:48.029 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 9.8265090050000001, 2024-03-30T13:53:48.029 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.21038131199999999, 2024-03-30T13:53:48.029 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:53:48.029 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:53:48.029 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:48.030 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:53:48.030 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:53:48.030 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:53:48.030 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61236 2024-03-30T13:53:48.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.030 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:53:48.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.030 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:53:48.030 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.384596+0000", 2024-03-30T13:53:48.030 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:48.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.030 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:53:48.031 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.384596+0000", 2024-03-30T13:53:48.031 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:48.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.031 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:53:48.031 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.384597+0000", 2024-03-30T13:53:48.031 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4810000000000001e-06 2024-03-30T13:53:48.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.031 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:53:48.031 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.384619+0000", 2024-03-30T13:53:48.031 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1508000000000001e-05 2024-03-30T13:53:48.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.031 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:53:48.032 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.384630+0000", 2024-03-30T13:53:48.032 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1474000000000001e-05 2024-03-30T13:53:48.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.032 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:53:48.032 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.384639+0000", 2024-03-30T13:53:48.032 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3259999999999996e-06 2024-03-30T13:53:48.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.032 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:48.032 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.563297+0000", 2024-03-30T13:53:48.032 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.178658075 2024-03-30T13:53:48.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.032 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:53:48.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.563421+0000", 2024-03-30T13:53:48.033 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000123733 2024-03-30T13:53:48.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.033 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:53:48.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.563588+0000", 2024-03-30T13:53:48.033 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016715400000000001 2024-03-30T13:53:48.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.033 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:53:48.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.569613+0000", 2024-03-30T13:53:48.033 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0060258239999999999 2024-03-30T13:53:48.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.034 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:48.034 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.583971+0000", 2024-03-30T13:53:48.034 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.014358006 2024-03-30T13:53:48.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.034 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:48.034 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.587872+0000", 2024-03-30T13:53:48.034 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0039002189999999999 2024-03-30T13:53:48.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.034 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:53:48.034 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.587934+0000", 2024-03-30T13:53:48.034 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2324999999999997e-05 2024-03-30T13:53:48.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.035 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:53:48.035 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.594977+0000", 2024-03-30T13:53:48.035 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0070431870000000002 2024-03-30T13:53:48.035 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.035 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:48.035 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.035 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:62353 9.9 9:98d23096:::benchmark_data_smithi043_145506_object62352:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-30T13:53:48.035 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:39.492713+0000", 2024-03-30T13:53:48.035 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 7.7183922589999998, 2024-03-30T13:53:48.035 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17202758300000001, 2024-03-30T13:53:48.035 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:53:48.036 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:53:48.036 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:48.036 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:53:48.036 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:53:48.036 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:53:48.036 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 62353 2024-03-30T13:53:48.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.036 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:53:48.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.036 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:53:48.036 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.492713+0000", 2024-03-30T13:53:48.036 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:48.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.036 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:53:48.037 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.492713+0000", 2024-03-30T13:53:48.037 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:48.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.037 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:53:48.037 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.492713+0000", 2024-03-30T13:53:48.037 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8899999999999998e-07 2024-03-30T13:53:48.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.037 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:53:48.037 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.492735+0000", 2024-03-30T13:53:48.037 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.19e-05 2024-03-30T13:53:48.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.037 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:53:48.038 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.492745+0000", 2024-03-30T13:53:48.038 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.8500000000000006e-06 2024-03-30T13:53:48.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.038 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:53:48.038 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.492753+0000", 2024-03-30T13:53:48.038 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8220000000000007e-06 2024-03-30T13:53:48.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.038 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:48.038 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.495086+0000", 2024-03-30T13:53:48.038 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0023325070000000002 2024-03-30T13:53:48.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.039 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:53:48.039 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.495266+0000", 2024-03-30T13:53:48.039 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000180192 2024-03-30T13:53:48.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.039 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,6", 2024-03-30T13:53:48.039 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.495472+0000", 2024-03-30T13:53:48.039 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00020590100000000001 2024-03-30T13:53:48.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.039 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:53:48.039 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.508368+0000", 2024-03-30T13:53:48.039 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.012895973999999999 2024-03-30T13:53:48.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.040 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:48.040 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.508434+0000", 2024-03-30T13:53:48.040 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6865000000000004e-05 2024-03-30T13:53:48.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.040 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:48.040 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.658198+0000", 2024-03-30T13:53:48.040 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.149763323 2024-03-30T13:53:48.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.040 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:53:48.040 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.658318+0000", 2024-03-30T13:53:48.041 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000119898 2024-03-30T13:53:48.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.041 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:53:48.041 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.664740+0000", 2024-03-30T13:53:48.041 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0064225619999999997 2024-03-30T13:53:48.041 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.041 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:48.041 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.041 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:62357 9.9 9:9a6084bc:::benchmark_data_smithi043_145506_object62356:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-30T13:53:48.041 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:39.499871+0000", 2024-03-30T13:53:48.041 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 7.7112339739999998, 2024-03-30T13:53:48.041 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.18574223100000001, 2024-03-30T13:53:48.042 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:53:48.042 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:53:48.042 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:48.042 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:53:48.042 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:53:48.042 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:53:48.042 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 62357 2024-03-30T13:53:48.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.042 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:53:48.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.042 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:53:48.042 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.499871+0000", 2024-03-30T13:53:48.042 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:48.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.043 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:53:48.043 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.499871+0000", 2024-03-30T13:53:48.043 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:48.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.043 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:53:48.043 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.499873+0000", 2024-03-30T13:53:48.043 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.688e-06 2024-03-30T13:53:48.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.043 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:53:48.043 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.499922+0000", 2024-03-30T13:53:48.043 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9441999999999999e-05 2024-03-30T13:53:48.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.044 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.044 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:53:48.044 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.499932+0000", 2024-03-30T13:53:48.044 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.01e-05 2024-03-30T13:53:48.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.044 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.044 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:53:48.044 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.499939+0000", 2024-03-30T13:53:48.044 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6579999999999996e-06 2024-03-30T13:53:48.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.044 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.044 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:48.044 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.508463+0000", 2024-03-30T13:53:48.044 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0085245210000000002 2024-03-30T13:53:48.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.045 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.045 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:53:48.045 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.508580+0000", 2024-03-30T13:53:48.045 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000116755 2024-03-30T13:53:48.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.045 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.045 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,6", 2024-03-30T13:53:48.045 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.508720+0000", 2024-03-30T13:53:48.045 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014024999999999999 2024-03-30T13:53:48.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.045 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.045 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:48.045 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.654170+0000", 2024-03-30T13:53:48.045 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.145449936 2024-03-30T13:53:48.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.046 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.046 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:53:48.046 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.654221+0000", 2024-03-30T13:53:48.046 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0223999999999999e-05 2024-03-30T13:53:48.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.046 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.046 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:48.046 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.681398+0000", 2024-03-30T13:53:48.046 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.027177903 2024-03-30T13:53:48.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.046 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.046 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:53:48.046 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.681511+0000", 2024-03-30T13:53:48.047 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000112841 2024-03-30T13:53:48.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.047 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.047 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:53:48.047 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.685613+0000", 2024-03-30T13:53:48.047 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0041019129999999996 2024-03-30T13:53:48.047 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.047 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:48.047 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.047 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.047 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:62905 9.9 9:96a8e6b4:::benchmark_data_smithi043_145506_object62904:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-30T13:53:48.047 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:40.797229+0000", 2024-03-30T13:53:48.047 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 6.4138758490000001, 2024-03-30T13:53:48.047 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17372147199999999, 2024-03-30T13:53:48.048 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:53:48.048 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:53:48.048 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:48.048 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:53:48.048 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:53:48.048 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:53:48.048 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 62905 2024-03-30T13:53:48.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.048 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:53:48.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.048 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:53:48.048 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.797229+0000", 2024-03-30T13:53:48.048 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:48.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.049 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.049 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:53:48.049 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.797229+0000", 2024-03-30T13:53:48.049 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:48.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.049 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.049 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:53:48.049 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.797230+0000", 2024-03-30T13:53:48.049 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.16e-07 2024-03-30T13:53:48.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.049 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.049 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:53:48.049 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.797251+0000", 2024-03-30T13:53:48.049 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1393e-05 2024-03-30T13:53:48.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.050 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:53:48.050 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.797263+0000", 2024-03-30T13:53:48.050 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1506e-05 2024-03-30T13:53:48.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.050 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:53:48.050 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.797270+0000", 2024-03-30T13:53:48.050 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6129999999999998e-06 2024-03-30T13:53:48.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.050 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:48.050 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.797337+0000", 2024-03-30T13:53:48.051 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6089999999999999e-05 2024-03-30T13:53:48.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.051 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:53:48.051 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.797494+0000", 2024-03-30T13:53:48.051 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000157606 2024-03-30T13:53:48.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.051 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,6", 2024-03-30T13:53:48.051 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.797676+0000", 2024-03-30T13:53:48.051 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018169700000000001 2024-03-30T13:53:48.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.051 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:53:48.051 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.803150+0000", 2024-03-30T13:53:48.052 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0054741929999999996 2024-03-30T13:53:48.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.052 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:48.052 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.806734+0000", 2024-03-30T13:53:48.052 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0035834299999999999 2024-03-30T13:53:48.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.052 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:48.052 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.818176+0000", 2024-03-30T13:53:48.052 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011442511000000001 2024-03-30T13:53:48.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.052 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:53:48.052 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.818227+0000", 2024-03-30T13:53:48.053 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1187999999999998e-05 2024-03-30T13:53:48.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.053 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:53:48.053 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.970951+0000", 2024-03-30T13:53:48.053 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.15272332899999999 2024-03-30T13:53:48.053 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.053 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:48.053 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.053 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:62906 9.9 9:9cbd581b:::benchmark_data_smithi043_145506_object62905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-30T13:53:48.053 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:40.799969+0000", 2024-03-30T13:53:48.053 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 6.4111355589999999, 2024-03-30T13:53:48.053 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17861787100000001, 2024-03-30T13:53:48.054 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:53:48.054 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:53:48.054 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:48.054 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:53:48.054 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:53:48.054 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:53:48.054 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 62906 2024-03-30T13:53:48.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.054 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:53:48.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.054 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:53:48.054 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.799969+0000", 2024-03-30T13:53:48.054 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:48.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.055 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.055 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:53:48.055 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.799969+0000", 2024-03-30T13:53:48.055 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:48.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.055 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.055 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:53:48.055 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.799971+0000", 2024-03-30T13:53:48.055 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6470000000000001e-06 2024-03-30T13:53:48.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.055 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.055 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:53:48.055 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.799993+0000", 2024-03-30T13:53:48.055 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1851000000000001e-05 2024-03-30T13:53:48.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.056 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:53:48.056 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.800005+0000", 2024-03-30T13:53:48.056 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2471e-05 2024-03-30T13:53:48.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.056 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:53:48.056 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.800014+0000", 2024-03-30T13:53:48.056 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3760000000000008e-06 2024-03-30T13:53:48.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.056 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:48.056 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.800077+0000", 2024-03-30T13:53:48.057 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2972999999999995e-05 2024-03-30T13:53:48.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.057 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:53:48.057 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.800250+0000", 2024-03-30T13:53:48.057 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017310000000000001 2024-03-30T13:53:48.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.057 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,6", 2024-03-30T13:53:48.057 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.800447+0000", 2024-03-30T13:53:48.057 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00019700499999999999 2024-03-30T13:53:48.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.057 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:53:48.057 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.804124+0000", 2024-03-30T13:53:48.058 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003677699 2024-03-30T13:53:48.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.058 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:48.058 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.806781+0000", 2024-03-30T13:53:48.058 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026568709999999999 2024-03-30T13:53:48.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.058 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:53:48.058 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.972322+0000", 2024-03-30T13:53:48.058 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.165540771 2024-03-30T13:53:48.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.058 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:53:48.059 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.972508+0000", 2024-03-30T13:53:48.059 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000186217 2024-03-30T13:53:48.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.059 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:53:48.059 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.978587+0000", 2024-03-30T13:53:48.059 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00607889 2024-03-30T13:53:48.059 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.059 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:48.059 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.059 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.15354.0:62913 9.e e232/229 9:75aa2c64:::benchmark_data_smithi043_145506_object62912:head v 232'3938, mlcod=216'215)", 2024-03-30T13:53:48.059 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:40.811116+0000", 2024-03-30T13:53:48.060 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 6.3999889830000001, 2024-03-30T13:53:48.060 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17777310900000001, 2024-03-30T13:53:48.060 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:53:48.060 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:53:48.060 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:53:48.060 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:53:48.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.060 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:53:48.060 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.811116+0000", 2024-03-30T13:53:48.060 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:48.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.060 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:53:48.060 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.811116+0000", 2024-03-30T13:53:48.060 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:53:48.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.061 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:53:48.061 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.811117+0000", 2024-03-30T13:53:48.061 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6199999999999997e-07 2024-03-30T13:53:48.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.061 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:53:48.061 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.811271+0000", 2024-03-30T13:53:48.061 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000154458 2024-03-30T13:53:48.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.061 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:53:48.061 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.811282+0000", 2024-03-30T13:53:48.061 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0723000000000001e-05 2024-03-30T13:53:48.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.062 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:53:48.062 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.811289+0000", 2024-03-30T13:53:48.062 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8889999999999996e-06 2024-03-30T13:53:48.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.062 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:53:48.062 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.812914+0000", 2024-03-30T13:53:48.062 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001625381 2024-03-30T13:53:48.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.062 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:53:48.062 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.812944+0000", 2024-03-30T13:53:48.063 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.0351000000000001e-05 2024-03-30T13:53:48.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.063 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:53:48.063 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.988846+0000", 2024-03-30T13:53:48.063 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.175901218 2024-03-30T13:53:48.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:53:48.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:53:48.063 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:53:48.063 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.988889+0000", 2024-03-30T13:53:48.063 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3427000000000003e-05 2024-03-30T13:53:48.063 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.063 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:48.063 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.063 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:53:48.063 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:53:48.064 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:53:48.064 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:47 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 42442/195417 objects degraded (21.719%), 16 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:48.064 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:47 smithi043 ceph-mon[29963]: osdmap e234: 8 total, 7 up, 6 in 2024-03-30T13:53:48.064 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:47 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 42442/195417 objects degraded (21.719%), 16 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:48.064 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:47 smithi043 ceph-mon[42099]: osdmap e234: 8 total, 7 up, 6 in 2024-03-30T13:53:48.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:48.065 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:48.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:48.166 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:48.194 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:48.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:48.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:48.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:47 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 42442/195417 objects degraded (21.719%), 16 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:48.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:47 smithi113 ceph-mon[34341]: osdmap e234: 8 total, 7 up, 6 in 2024-03-30T13:53:48.295 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:48.322 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:48.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:48.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:48.424 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:48.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:48.467 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:48.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:48.567 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:48.594 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:48.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:48.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:48.694 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 57 16 67538 67522 74.0238 43.3125 0.0227912 0.0135008 2024-03-30T13:53:48.696 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:48.741 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-30T13:53:48.753 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:48.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:48.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:48.855 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:48.881 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:48.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:48.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:48.983 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:49.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:49.043 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:49.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:49.136 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:48 smithi043 ceph-mon[29963]: pgmap v466: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.0 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 26 MiB/s wr, 419 op/s; 42442/195417 objects degraded (21.719%); 28154/195417 objects misplaced (14.407%); 13 MiB/s, 201 objects/s recovering 2024-03-30T13:53:49.136 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:48 smithi043 ceph-mon[42099]: pgmap v466: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.0 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 26 MiB/s wr, 419 op/s; 42442/195417 objects degraded (21.719%); 28154/195417 objects misplaced (14.407%); 13 MiB/s, 201 objects/s recovering 2024-03-30T13:53:49.145 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:49.171 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:49.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:49.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:49.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:48 smithi113 ceph-mon[34341]: pgmap v466: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.0 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 26 MiB/s wr, 419 op/s; 42442/195417 objects degraded (21.719%); 28154/195417 objects misplaced (14.407%); 13 MiB/s, 201 objects/s recovering 2024-03-30T13:53:49.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:53:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:53:49.273 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:49.298 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:49.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:49.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:49.399 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:49.427 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:49.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:49.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:49.529 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:49.585 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:49.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:49.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:49.686 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:49.695 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 58 15 68121 68106 73.3756 36.5 0.0207384 0.0136212 2024-03-30T13:53:49.713 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:49.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:49.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:49.814 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:49.870 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:49.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:49.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:49.972 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:50.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:50.020 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:50.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:50.121 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:50.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:49 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:53:50.124 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:49 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:53:50.174 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:50.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:50.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:49 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:53:50.234 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:53:50.276 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:50.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:50.304 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:50.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:50.405 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:50.454 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:50.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:50.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:50.556 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:50.583 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:50.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:50.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:50.684 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:50.696 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 59 16 68770 68754 72.8182 40.5 0.0389196 0.0137255 2024-03-30T13:53:50.713 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:50.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:50.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:50.815 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:50.879 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:50.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:50.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:50.980 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:51.006 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:51.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:51.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:51.108 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:51.114 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:53:51.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:51.158 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:51.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:51.181 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:53:51.182 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:50 smithi043 ceph-mon[29963]: pgmap v467: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 32 MiB/s wr, 508 op/s; 42653/199674 objects degraded (21.361%); 27820/199674 objects misplaced (13.933%); 12 MiB/s, 187 objects/s recovering 2024-03-30T13:53:51.182 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:50 smithi043 ceph-mon[42099]: pgmap v467: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 32 MiB/s wr, 508 op/s; 42653/199674 objects degraded (21.361%); 27820/199674 objects misplaced (13.933%); 12 MiB/s, 187 objects/s recovering 2024-03-30T13:53:51.192 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T13:53:51.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:50 smithi113 ceph-mon[34341]: pgmap v467: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 32 MiB/s wr, 508 op/s; 42653/199674 objects degraded (21.361%); 27820/199674 objects misplaced (13.933%); 12 MiB/s, 187 objects/s recovering 2024-03-30T13:53:51.232 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:53:51.259 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:51.284 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:53:51.317 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:51.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:51.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:51.330 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:53:51.372 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:53:51.418 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:53:51.418 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:53:51.418 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:53:51.418 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:53:51.418 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:53:51.418 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:53:51.418 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:53:51.418 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:53:51.418 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:53:51.419 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:51.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:51.447 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:51.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:51.548 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:51.576 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:51.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:51.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:51.678 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:51.697 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-03-30T13:53:51.699495+0000 min lat: 0.00127384 max lat: 7.25374 avg lat: 0.0138416 2024-03-30T13:53:51.697 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T13:53:51.697 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 60 16 69353 69337 72.211 36.4375 0.0128759 0.0138416 2024-03-30T13:53:51.703 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:51.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:51.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:51.805 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:51.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:51.841 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:51.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:51.942 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:51.982 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:51.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:51.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:52.084 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:52.110 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:52.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:52.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:52.212 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:52.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:51 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/4168834397' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:53:52.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:52.242 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:52.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:52.342 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:52.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:52.369 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:52.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:52.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:51 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4168834397' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:53:52.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:51 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/4168834397' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:53:52.471 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:52.484 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6] out_osds: [5, 7] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-30T13:53:52.484 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:53:52.484 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-30T13:53:52.484 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T13:53:52.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:52.520 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:52.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:52.621 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:52.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:52.651 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:52.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:52.697 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 61 16 69935 69919 71.6235 36.375 0.0150538 0.0139547 2024-03-30T13:53:52.753 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:52.778 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:52.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:52.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:52.880 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:52.905 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:52.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:52.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:53.008 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:53.058 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:53.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:53.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:53.160 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:53.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:53.198 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:53.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:53.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:52 smithi113 ceph-mon[34341]: pgmap v468: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 42 MiB/s wr, 665 op/s; 43029/204732 objects degraded (21.017%); 27757/204732 objects misplaced (13.558%); 14 MiB/s, 224 objects/s recovering 2024-03-30T13:53:53.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:52 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 43029/204732 objects degraded (21.017%), 16 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:53.301 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:53.328 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:53.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:53.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:53.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:52 smithi043 ceph-mon[29963]: pgmap v468: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 42 MiB/s wr, 665 op/s; 43029/204732 objects degraded (21.017%); 27757/204732 objects misplaced (13.558%); 14 MiB/s, 224 objects/s recovering 2024-03-30T13:53:53.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:52 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 43029/204732 objects degraded (21.017%), 16 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:53.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:52 smithi043 ceph-mon[42099]: pgmap v468: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 42 MiB/s wr, 665 op/s; 43029/204732 objects degraded (21.017%); 27757/204732 objects misplaced (13.558%); 14 MiB/s, 224 objects/s recovering 2024-03-30T13:53:53.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:52 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 43029/204732 objects degraded (21.017%), 16 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:53.430 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:53.457 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:53.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:53.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:53.559 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:53.593 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:53.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:53.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:53.694 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:53.697 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 62 16 70482 70466 71.0197 34.1875 0.0275759 0.014073 2024-03-30T13:53:53.722 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:53.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:53.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:53.823 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:53.849 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:53.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:53.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:53.950 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:53.991 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:53.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:53.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:54.093 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:54.117 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:54.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:54.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:54.219 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:54.246 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:54.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:54.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:54.348 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:54.375 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:54.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:54.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:54.477 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:54.503 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:54.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:54.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:54.605 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:54.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:54.670 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:54.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:54.697 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 63 16 71076 71060 70.4817 37.125 0.00356439 0.014181 2024-03-30T13:53:54.771 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:54.799 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:54.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:54.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:54.900 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:54.945 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:54.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:54.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:55.047 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:55.072 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:55.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:55.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:55.127 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:53:55.128 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:53:55.168 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:53:55.174 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:55.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:54 smithi113 ceph-mon[34341]: pgmap v469: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 36 MiB/s wr, 575 op/s; 43029/204732 objects degraded (21.017%); 27757/204732 objects misplaced (13.558%); 12 MiB/s, 194 objects/s recovering 2024-03-30T13:53:55.235 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:54 smithi043 ceph-mon[29963]: pgmap v469: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 36 MiB/s wr, 575 op/s; 43029/204732 objects degraded (21.017%); 27757/204732 objects misplaced (13.558%); 12 MiB/s, 194 objects/s recovering 2024-03-30T13:53:55.236 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:54 smithi043 ceph-mon[42099]: pgmap v469: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 36 MiB/s wr, 575 op/s; 43029/204732 objects degraded (21.017%); 27757/204732 objects misplaced (13.558%); 12 MiB/s, 194 objects/s recovering 2024-03-30T13:53:55.244 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:55.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:55.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:55.346 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:55.373 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:55.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:55.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:55.475 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:55.500 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:55.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:55.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:55.601 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:55.642 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:55.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:55.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:55.697 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 64 16 71714 71698 70.0035 39.875 0.0546203 0.0142775 2024-03-30T13:53:55.744 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:55.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:55.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:55.895 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:55.922 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:55.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:55.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:56.024 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:56.049 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:56.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:56.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:56.150 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:56.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:56.180 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:56.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:56.280 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:56.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:56.333 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:56.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:56.396 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:53:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:53:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:53:56.434 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:56.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:56.475 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:56.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:56.576 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:56.595 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:53:56.595 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:53:56.601 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:56.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:56.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:56.697 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 65 16 72356 72340 69.5438 40.125 0.025327 0.0143717 2024-03-30T13:53:56.703 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:56.730 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:56.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:56.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:56.832 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:56.877 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:56.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:56.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:56.979 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:57.007 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:57.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:57.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:57.110 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:57.128 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:53:57.139 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:57.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:57.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:57.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:56 smithi113 ceph-mon[34341]: pgmap v470: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.3 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 38 MiB/s wr, 606 op/s; 43579/213585 objects degraded (20.404%); 27247/213585 objects misplaced (12.757%); 13 MiB/s, 202 objects/s recovering 2024-03-30T13:53:57.241 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:57.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:57.298 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:57.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:57.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:56 smithi043 ceph-mon[29963]: pgmap v470: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.3 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 38 MiB/s wr, 606 op/s; 43579/213585 objects degraded (20.404%); 27247/213585 objects misplaced (12.757%); 13 MiB/s, 202 objects/s recovering 2024-03-30T13:53:57.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:56 smithi043 ceph-mon[42099]: pgmap v470: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.3 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 38 MiB/s wr, 606 op/s; 43579/213585 objects degraded (20.404%); 27247/213585 objects misplaced (12.757%); 13 MiB/s, 202 objects/s recovering 2024-03-30T13:53:57.399 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:57.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:57.479 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:57.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:57.580 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:57.607 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:57.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:57.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:57.697 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 66 16 72966 72950 69.0678 38.125 0.0188874 0.0144725 2024-03-30T13:53:57.708 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:57.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:57.759 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:57.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:57.860 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:57.904 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:57.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:57.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:58.006 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:53:58.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:58.055 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:58.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:53:58.086 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-03-30T13:53:58.156 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:58.183 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:58.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:58.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:58.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:57 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 43579/213585 objects degraded (20.404%), 16 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:58.285 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:58.312 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:58.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:58.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:58.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:57 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 43579/213585 objects degraded (20.404%), 16 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:58.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:57 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 43579/213585 objects degraded (20.404%), 16 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:53:58.413 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:58.440 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:58.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:58.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:58.541 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:58.567 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:58.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:58.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:58.640 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:53:58.668 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:53:58.694 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:58.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:58.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:53:58.697 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 67 16 73564 73548 68.5947 37.375 0.0163429 0.0145692 2024-03-30T13:53:58.797 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:53:58.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:58.870 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:58.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:53:58.970 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:53:58.995 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:58.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:58.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:53:59.097 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:59.124 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:59.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:59.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:59.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:53:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:53:59.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:58 smithi113 ceph-mon[34341]: pgmap v471: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.3 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1020 B/s rd, 38 MiB/s wr, 604 op/s; 43579/213585 objects degraded (20.404%); 27247/213585 objects misplaced (12.757%); 13 MiB/s, 201 objects/s recovering 2024-03-30T13:53:59.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:58 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:53:59.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:58 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:53:59.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:58 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:53:59.226 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:59.271 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:59.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:59.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:59.373 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:53:59.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:58 smithi043 ceph-mon[29963]: pgmap v471: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.3 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1020 B/s rd, 38 MiB/s wr, 604 op/s; 43579/213585 objects degraded (20.404%); 27247/213585 objects misplaced (12.757%); 13 MiB/s, 201 objects/s recovering 2024-03-30T13:53:59.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:58 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:53:59.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:58 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:53:59.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:58 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:53:59.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:58 smithi043 ceph-mon[42099]: pgmap v471: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.3 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1020 B/s rd, 38 MiB/s wr, 604 op/s; 43579/213585 objects degraded (20.404%); 27247/213585 objects misplaced (12.757%); 13 MiB/s, 201 objects/s recovering 2024-03-30T13:53:59.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:58 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:53:59.397 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:58 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:53:59.397 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:58 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:53:59.417 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:53:59.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:59.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:53:59.518 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:59.544 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:59.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:59.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:59.646 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:53:59.671 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:59.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:59.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:53:59.697 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 68 16 74122 74106 68.0988 34.875 0.00386467 0.0146767 2024-03-30T13:53:59.773 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:59.799 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:59.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:59.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:59.901 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:53:59.929 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:53:59.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:53:59.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:53:59.972 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-03-30T13:54:00.030 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:00.059 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:00.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:00.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:00.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:59 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:54:00.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:59 smithi043 ceph-mon[29963]: osdmap e235: 8 total, 7 up, 6 in 2024-03-30T13:54:00.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:53:59 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2325762588' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:54:00.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:59 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:54:00.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:59 smithi043 ceph-mon[42099]: osdmap e235: 8 total, 7 up, 6 in 2024-03-30T13:54:00.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:53:59 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2325762588' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:54:00.161 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:00.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:00.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:00.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:59 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:54:00.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:59 smithi113 ceph-mon[34341]: osdmap e235: 8 total, 7 up, 6 in 2024-03-30T13:54:00.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:53:59 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2325762588' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:54:00.310 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:00.347 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:00.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:00.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:00.449 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:00.476 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:00.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:00.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:00.578 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:00.620 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:00.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:00.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:00.697 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 69 16 74715 74699 67.649 37.0625 0.0264014 0.0147762 2024-03-30T13:54:00.723 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:00.750 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:00.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:00.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:00.851 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:00.878 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:00.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:00.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:00.980 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:01.003 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:01.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:01.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:01.105 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:01.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:00 smithi043 ceph-mon[29963]: pgmap v473: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 37 MiB/s wr, 599 op/s; 43760/217632 objects degraded (20.107%); 26852/217632 objects misplaced (12.338%); 13 MiB/s, 207 objects/s recovering 2024-03-30T13:54:01.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:01 smithi043 ceph-mon[29963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:54:01.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:01 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2325762588' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:54:01.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:01 smithi043 ceph-mon[29963]: osdmap e236: 8 total, 7 up, 6 in 2024-03-30T13:54:01.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:00 smithi043 ceph-mon[42099]: pgmap v473: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 37 MiB/s wr, 599 op/s; 43760/217632 objects degraded (20.107%); 26852/217632 objects misplaced (12.338%); 13 MiB/s, 207 objects/s recovering 2024-03-30T13:54:01.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:01 smithi043 ceph-mon[42099]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:54:01.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:01 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2325762588' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:54:01.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:01 smithi043 ceph-mon[42099]: osdmap e236: 8 total, 7 up, 6 in 2024-03-30T13:54:01.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:01.152 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:01.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:01.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:00 smithi113 ceph-mon[34341]: pgmap v473: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 37 MiB/s wr, 599 op/s; 43760/217632 objects degraded (20.107%); 26852/217632 objects misplaced (12.338%); 13 MiB/s, 207 objects/s recovering 2024-03-30T13:54:01.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:00 smithi113 ceph-mon[34341]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:54:01.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:00 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2325762588' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:54:01.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:00 smithi113 ceph-mon[34341]: osdmap e236: 8 total, 7 up, 6 in 2024-03-30T13:54:01.253 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:01.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:01.295 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:01.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:01.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:01.446 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:01.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:01.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:01.547 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:01.577 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:01.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:01.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:01.678 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:01.699 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 70 16 75319 75303 67.22 37.75 0.024098 0.0148688 2024-03-30T13:54:01.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:01.719 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:01.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:01.820 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:01.849 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:01.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:01.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:01.951 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:01.996 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:01.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:02.098 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:02.138 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:02.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:02.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:02.240 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:02.267 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:02.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:02.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:02.369 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:02.396 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:02.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:02.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:02.498 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:02.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:02.527 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:02.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:02.628 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:02.657 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:02.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:02.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:02.699 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 71 16 75969 75953 66.8454 40.625 0.0367217 0.0149537 2024-03-30T13:54:02.759 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:02.786 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:02.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:02.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:02.888 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:02.912 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:02.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:02.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:03.013 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:03.039 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:03.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:03.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:03.140 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:03.177 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:03.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:03.278 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:03.305 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:03.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:03.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:03.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:02 smithi043 ceph-mon[29963]: pgmap v475: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 750 op/s; 44127/222729 objects degraded (19.812%); 26592/222729 objects misplaced (11.939%); 18 MiB/s, 280 objects/s recovering 2024-03-30T13:54:03.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:02 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 44127/222729 objects degraded (19.812%), 16 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:03.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:02 smithi043 ceph-mon[42099]: pgmap v475: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 750 op/s; 44127/222729 objects degraded (19.812%); 26592/222729 objects misplaced (11.939%); 18 MiB/s, 280 objects/s recovering 2024-03-30T13:54:03.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:02 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 44127/222729 objects degraded (19.812%), 16 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:03.407 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:03.443 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:03.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:03.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:03.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:02 smithi113 ceph-mon[34341]: pgmap v475: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 750 op/s; 44127/222729 objects degraded (19.812%); 26592/222729 objects misplaced (11.939%); 18 MiB/s, 280 objects/s recovering 2024-03-30T13:54:03.469 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:02 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 44127/222729 objects degraded (19.812%), 16 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:03.545 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:03.569 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:03.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:03.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:03.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:03.695 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:03.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:03.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:03.699 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 72 16 76278 76262 66.1853 19.3125 0.00555693 0.0150101 2024-03-30T13:54:03.797 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:03.821 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:03.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:03.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:03.923 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:03.947 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:03.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:03.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:04.049 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:04.086 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:04.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:04.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:04.188 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:04.216 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:04.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:04.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:04.318 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:04.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:04.365 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:04.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:04.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:03 smithi043 ceph-mon[29963]: osdmap e237: 8 total, 7 up, 6 in 2024-03-30T13:54:04.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:03 smithi043 ceph-mon[42099]: osdmap e237: 8 total, 7 up, 6 in 2024-03-30T13:54:04.465 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:04.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:03 smithi113 ceph-mon[34341]: osdmap e237: 8 total, 7 up, 6 in 2024-03-30T13:54:04.491 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:04.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:04.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:04.592 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:04.617 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:04.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:04.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:04.700 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 73 16 76654 76638 65.6006 23.5 0.00992104 0.015236 2024-03-30T13:54:04.718 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:04.743 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:04.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:04.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:04.845 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:04.869 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:04.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:04.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:04.971 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:05.007 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:05.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:05.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:05.109 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:05.141 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:05.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:05.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:05.243 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:05.286 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:05.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:05.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:05.387 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:05.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:04 smithi043 ceph-mon[29963]: pgmap v477: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 32 MiB/s wr, 508 op/s; 44127/222729 objects degraded (19.812%); 26592/222729 objects misplaced (11.939%); 13 MiB/s, 200 objects/s recovering 2024-03-30T13:54:05.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:04 smithi043 ceph-mon[29963]: osdmap e238: 8 total, 7 up, 6 in 2024-03-30T13:54:05.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:04 smithi043 ceph-mon[42099]: pgmap v477: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 32 MiB/s wr, 508 op/s; 44127/222729 objects degraded (19.812%); 26592/222729 objects misplaced (11.939%); 13 MiB/s, 200 objects/s recovering 2024-03-30T13:54:05.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:04 smithi043 ceph-mon[42099]: osdmap e238: 8 total, 7 up, 6 in 2024-03-30T13:54:05.413 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:05.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:05.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:05.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:04 smithi113 ceph-mon[34341]: pgmap v477: 158 pgs: 81 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 45 active+clean+remapped; 4.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 32 MiB/s wr, 508 op/s; 44127/222729 objects degraded (19.812%); 26592/222729 objects misplaced (11.939%); 13 MiB/s, 200 objects/s recovering 2024-03-30T13:54:05.469 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:04 smithi113 ceph-mon[34341]: osdmap e238: 8 total, 7 up, 6 in 2024-03-30T13:54:05.515 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:05.541 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:05.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:05.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:05.643 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:05.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:05.689 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:05.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:05.700 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 74 16 77247 77231 65.2149 37.0625 0.00491529 0.0153224 2024-03-30T13:54:05.790 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:05.817 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:05.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:05.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:05.918 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:05.959 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:05.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:05.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:06.060 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:06.088 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:06.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:06.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:06.190 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:06.216 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:06.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:06.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:06.318 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:06.367 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:06.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:06.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:06.396 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:54:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:54:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:54:06.419 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:54:06.470 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:06.494 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:06.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:06.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:06.596 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:06.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:06.650 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:06.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:06.700 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 75 16 77777 77761 64.7871 33.125 0.00548414 0.0154271 2024-03-30T13:54:06.751 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:06.777 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:06.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:06.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:06.878 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:06.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:06.926 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:06.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:07.027 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:07.054 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:07.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:07.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:07.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:06 smithi043 ceph-mon[29963]: pgmap v479: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 45 MiB/s wr, 716 op/s; 33061/230508 objects degraded (14.343%); 37372/230508 objects misplaced (16.213%); 20 MiB/s, 313 objects/s recovering 2024-03-30T13:54:07.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:06 smithi043 ceph-mon[42099]: pgmap v479: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 45 MiB/s wr, 716 op/s; 33061/230508 objects degraded (14.343%); 37372/230508 objects misplaced (16.213%); 20 MiB/s, 313 objects/s recovering 2024-03-30T13:54:07.156 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:07.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:07.209 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:07.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:07.310 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:07.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:07.358 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:07.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:07.459 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:07.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:06 smithi113 ceph-mon[34341]: pgmap v479: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 45 MiB/s wr, 716 op/s; 33061/230508 objects degraded (14.343%); 37372/230508 objects misplaced (16.213%); 20 MiB/s, 313 objects/s recovering 2024-03-30T13:54:07.511 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:07.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:07.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:07.615 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:07.642 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:07.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:07.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:07.700 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 76 16 78407 78391 64.4527 39.375 0.0143353 0.015509 2024-03-30T13:54:07.743 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:07.787 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:07.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:07.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:07.889 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:07.916 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:07.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:07.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:08.017 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:08.080 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:08.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:08.131 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:54:08.182 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:08.238 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:08.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:08.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:08.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:08 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:54:08.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:08 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:54:08.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:08 smithi043 ceph-mon[42099]: osdmap e239: 8 total, 7 up, 6 in 2024-03-30T13:54:08.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:08 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 33061/230508 objects degraded (14.343%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:08.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:08 smithi043 ceph-mon[42099]: osdmap e240: 8 total, 7 up, 6 in 2024-03-30T13:54:08.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:08 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:54:08.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:08 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:54:08.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:08 smithi043 ceph-mon[29963]: osdmap e239: 8 total, 7 up, 6 in 2024-03-30T13:54:08.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:08 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 33061/230508 objects degraded (14.343%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:08.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:08 smithi043 ceph-mon[29963]: osdmap e240: 8 total, 7 up, 6 in 2024-03-30T13:54:08.340 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:08.365 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:08.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:08.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:08.467 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:08.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:08 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:54:08.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:08 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:54:08.469 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:08 smithi113 ceph-mon[34341]: osdmap e239: 8 total, 7 up, 6 in 2024-03-30T13:54:08.469 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:08 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 33061/230508 objects degraded (14.343%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:08.469 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:08 smithi113 ceph-mon[34341]: osdmap e240: 8 total, 7 up, 6 in 2024-03-30T13:54:08.494 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:08.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:08.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:08.596 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:08.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:08.638 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:08.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:08.700 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 77 16 79011 78995 64.1057 37.75 0.00737246 0.0155921 2024-03-30T13:54:08.740 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:08.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:08.812 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:08.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:08.913 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:08.939 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:08.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:08.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:09.041 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:09.069 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:09.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:09.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:09.171 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:09.190 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:09 smithi043 ceph-mon[29963]: pgmap v482: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 36 MiB/s wr, 571 op/s; 33061/230508 objects degraded (14.343%); 37372/230508 objects misplaced (16.213%); 17 MiB/s, 278 objects/s recovering 2024-03-30T13:54:09.191 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:09 smithi043 ceph-mon[42099]: pgmap v482: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 36 MiB/s wr, 571 op/s; 33061/230508 objects degraded (14.343%); 37372/230508 objects misplaced (16.213%); 17 MiB/s, 278 objects/s recovering 2024-03-30T13:54:09.197 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:09.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:09.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:09.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:09 smithi113 ceph-mon[34341]: pgmap v482: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 36 MiB/s wr, 571 op/s; 33061/230508 objects degraded (14.343%); 37372/230508 objects misplaced (16.213%); 17 MiB/s, 278 objects/s recovering 2024-03-30T13:54:09.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:54:08 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:54:09.298 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:09.343 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:09.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:09.443 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:09.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:09.471 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:09.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:09.571 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:09.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:09.629 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:09.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:09.700 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 78 16 79585 79569 63.7438 35.875 0.0388048 0.0156789 2024-03-30T13:54:09.709 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:54:09.730 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:09.757 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:09.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:09.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:09.762 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:54:09.818 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:54:09.859 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:09.864 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:54:09.931 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:09.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:09.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:09.936 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:54:09.995 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:54:10.034 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:10.045 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:54:10.046 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:54:10.046 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:54:10.047 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:54:10.047 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:54:10.047 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:54:10.047 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:54:10.047 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:54:10.047 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:54:10.068 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T13:54:10.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:10.083 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:10.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:10.107 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:10 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:54:10.107 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:10 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:54:10.107 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:10 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:54:10.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:10 smithi043 ceph-mon[29963]: osdmap e241: 8 total, 7 up, 6 in 2024-03-30T13:54:10.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:10 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/507792968' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:54:10.108 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:10 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:54:10.108 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:10 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:54:10.108 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:10 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:54:10.108 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:10 smithi043 ceph-mon[42099]: osdmap e241: 8 total, 7 up, 6 in 2024-03-30T13:54:10.108 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:10 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/507792968' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:54:10.184 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:10.210 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:54:10.220 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-30T13:54:10.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:10.252 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:10.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:10.353 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:10.381 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:10.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:10.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:10.469 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:10 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:54:10.469 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:10 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:54:10.469 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:10 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:54:10.469 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:10 smithi113 ceph-mon[34341]: osdmap e241: 8 total, 7 up, 6 in 2024-03-30T13:54:10.469 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:10 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/507792968' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:54:10.482 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:10.508 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:10.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:10.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:10.609 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:10.660 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:10.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:10.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:10.702 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 79 16 80121 80105 63.3596 33.5 0.0146194 0.0157763 2024-03-30T13:54:10.763 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:10.812 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:10.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:10.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:10.913 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:10.956 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:10.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:10.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:11.058 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:11.085 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:11.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:11.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:11.187 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:11.215 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:11.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:11.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:11.317 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:11.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:11.363 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:11.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:11.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:11 smithi043 ceph-mon[29963]: pgmap v484: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 922 B/s rd, 29 MiB/s wr, 461 op/s; 33061/230958 objects degraded (14.315%); 36923/230958 objects misplaced (15.987%); 15 MiB/s, 240 objects/s recovering 2024-03-30T13:54:11.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:11 smithi043 ceph-mon[29963]: from='client.25277 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:54:11.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:11 smithi043 ceph-mon[42099]: pgmap v484: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 922 B/s rd, 29 MiB/s wr, 461 op/s; 33061/230958 objects degraded (14.315%); 36923/230958 objects misplaced (15.987%); 15 MiB/s, 240 objects/s recovering 2024-03-30T13:54:11.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:11 smithi043 ceph-mon[42099]: from='client.25277 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:54:11.464 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:11.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:11 smithi113 ceph-mon[34341]: pgmap v484: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 922 B/s rd, 29 MiB/s wr, 461 op/s; 33061/230958 objects degraded (14.315%); 36923/230958 objects misplaced (15.987%); 15 MiB/s, 240 objects/s recovering 2024-03-30T13:54:11.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:11 smithi113 ceph-mon[34341]: from='client.25277 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:54:11.489 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:11.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:11.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:11.591 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:11.623 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:11.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:11.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:11.702 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-03-30T13:54:11.705271+0000 min lat: 0.00127384 max lat: 7.25374 avg lat: 0.0158595 2024-03-30T13:54:11.702 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T13:54:11.702 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 80 16 80702 80686 63.0215 36.3125 0.0144295 0.0158595 2024-03-30T13:54:11.725 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:11.774 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:11.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:11.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:11.876 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:11.904 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:11.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:11.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:12.006 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:12.033 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:12.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:12.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:12.134 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:12.161 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:12.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:12.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:12.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:12.305 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:12.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:12.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:12.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:12 smithi043 ceph-mon[29963]: pgmap v485: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 31 MiB/s wr, 491 op/s; 33433/239343 objects degraded (13.969%); 37019/239343 objects misplaced (15.467%); 11 MiB/s, 168 objects/s recovering 2024-03-30T13:54:12.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:12 smithi043 ceph-mon[42099]: pgmap v485: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 31 MiB/s wr, 491 op/s; 33433/239343 objects degraded (13.969%); 37019/239343 objects misplaced (15.467%); 11 MiB/s, 168 objects/s recovering 2024-03-30T13:54:12.407 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:12.433 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:12.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:12.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:12.469 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:12 smithi113 ceph-mon[34341]: pgmap v485: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 31 MiB/s wr, 491 op/s; 33433/239343 objects degraded (13.969%); 37019/239343 objects misplaced (15.467%); 11 MiB/s, 168 objects/s recovering 2024-03-30T13:54:12.534 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:12.561 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:12.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:12.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:12.663 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:12.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:12.693 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:12.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:12.702 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 81 15 81286 81271 62.6949 36.5625 0.0104973 0.0159429 2024-03-30T13:54:12.795 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:12.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:12.829 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:12.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:12.930 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:12.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:12.962 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:12.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:13.063 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:13.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:13.091 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:13.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:13.192 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:13.245 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:13.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:13.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:13.350 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:13.374 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:13.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:13.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:13.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:13 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 33433/239343 objects degraded (13.969%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:13.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:13 smithi043 ceph-mon[29963]: osdmap e242: 8 total, 7 up, 6 in 2024-03-30T13:54:13.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:13 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 33433/239343 objects degraded (13.969%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:13.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:13 smithi043 ceph-mon[42099]: osdmap e242: 8 total, 7 up, 6 in 2024-03-30T13:54:13.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:13 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 33433/239343 objects degraded (13.969%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:13.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:13 smithi113 ceph-mon[34341]: osdmap e242: 8 total, 7 up, 6 in 2024-03-30T13:54:13.476 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:13.516 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:13.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:13.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:13.618 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:13.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:13.645 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:13.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:13.702 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 82 16 81903 81887 62.3998 38.5 0.0387506 0.0160191 2024-03-30T13:54:13.746 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:13.796 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:13.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:13.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:13.898 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:13.924 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:13.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:13.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:14.026 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:14.053 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:14.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:14.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:14.155 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:14.204 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:14.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:14.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:14.306 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:14.332 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:14.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:14.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:14.434 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:14.460 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:14.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:14.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:14.562 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:14.588 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:14.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:14.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:14.690 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:14.702 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 83 16 82462 82446 62.0689 34.9375 0.0481551 0.0161032 2024-03-30T13:54:14.716 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:14.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:14.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:14.818 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:14.843 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:14.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:14.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:14 smithi043 ceph-mon[29963]: pgmap v487: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 848 B/s rd, 30 MiB/s wr, 488 op/s; 33433/239343 objects degraded (13.969%); 37019/239343 objects misplaced (15.467%); 10 MiB/s, 167 objects/s recovering 2024-03-30T13:54:14.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:14 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:54:14.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:14 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:54:14.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:14 smithi043 ceph-mon[42099]: pgmap v487: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 848 B/s rd, 30 MiB/s wr, 488 op/s; 33433/239343 objects degraded (13.969%); 37019/239343 objects misplaced (15.467%); 10 MiB/s, 167 objects/s recovering 2024-03-30T13:54:14.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:14 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:54:14.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:14 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:54:14.945 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:14.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:14 smithi113 ceph-mon[34341]: pgmap v487: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 848 B/s rd, 30 MiB/s wr, 488 op/s; 33433/239343 objects degraded (13.969%); 37019/239343 objects misplaced (15.467%); 10 MiB/s, 167 objects/s recovering 2024-03-30T13:54:14.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:14 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:54:14.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:14 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:54:14.974 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:14.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:14.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:15.076 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:15.102 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:15.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:15.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:15.203 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:15.250 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:15.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:15.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:15.352 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:15.377 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:15.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:15.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:15.479 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:15.508 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:15.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:15.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:15.609 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:15.635 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:15.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:15.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:15.702 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 84 16 83147 83131 61.8397 42.8125 0.011291 0.0161632 2024-03-30T13:54:15.737 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:15.764 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:15.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:15.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:15.866 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:15.892 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:15.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:15.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:15.994 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:16.017 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:16.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:16.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:16.119 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:16.146 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:16.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:16.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:16.248 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:16.275 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:16.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:16.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:16.377 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:16.396 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:54:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:54:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:54:16.403 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:16.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:16.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:16.505 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:16.531 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:16.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:16.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:16.633 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:16.659 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:16.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:16.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:16.702 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 85 16 83761 83745 61.5636 38.375 0.0130649 0.0162363 2024-03-30T13:54:16.761 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:16.784 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:16.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:16.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:16.886 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:16.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:16 smithi043 ceph-mon[29963]: pgmap v488: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 736 op/s; 33820/248160 objects degraded (13.628%); 36679/248160 objects misplaced (14.780%); 15 MiB/s, 247 objects/s recovering 2024-03-30T13:54:16.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:16 smithi043 ceph-mon[42099]: pgmap v488: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 736 op/s; 33820/248160 objects degraded (13.628%); 36679/248160 objects misplaced (14.780%); 15 MiB/s, 247 objects/s recovering 2024-03-30T13:54:16.912 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:16.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:16.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:16.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:16 smithi113 ceph-mon[34341]: pgmap v488: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 736 op/s; 33820/248160 objects degraded (13.628%); 36679/248160 objects misplaced (14.780%); 15 MiB/s, 247 objects/s recovering 2024-03-30T13:54:17.014 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:17.042 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:17.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:17.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:17.144 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:17.188 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:17.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:17.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:17.290 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:17.313 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:17.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:17.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:17.415 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:17.466 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:17.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:17.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:17.567 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:17.614 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:17.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:17.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:17.703 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 86 16 84335 84319 61.2649 35.875 0.0196719 0.016316 2024-03-30T13:54:17.716 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:17.760 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:17.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:17.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:17.867 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:17.893 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:17.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:17.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:17.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:17 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 33820/248160 objects degraded (13.628%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:17.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:17 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 33820/248160 objects degraded (13.628%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:17.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:17 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 33820/248160 objects degraded (13.628%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:17.995 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:18.041 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:18.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:18.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:18.143 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:18.170 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:18.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:18.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:18.272 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:18.323 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:18.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:18.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:18.425 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:18.493 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:54:18.494 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:54:18.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:18.518 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:18.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:18.612 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":484,"stamp":"2024-03-30T13:54:09.536418+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5032968480,"num_objects":76986,"num_object_clones":0,"num_object_copies":230958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33061,"num_objects_misplaced":36923,"num_objects_unfound":0,"num_objects_dirty":76986,"num_whiteouts":0,"num_read":1282,"num_read_kb":1086,"num_write":77607,"num_write_kb":4915729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13671,"num_bytes_recovered":889835768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59622,"ondisk_log_size":59622,"up":406,"acting":443,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":451,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":14515436,"kb_used_data":12344496,"kb_used_omap":9,"kb_used_meta":2170870,"kb_avail":547955476,"statfs":{"total":575970213888,"available":561106407424,"internally_reserved":0,"allocated":12640763904,"data_stored":12636457157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":2222971553},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":167510016,"num_objects":2556,"num_object_clones":0,"num_object_copies":7668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-11066,"num_objects_misplaced":10714,"num_objects_unfound":0,"num_objects_dirty":2556,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2556,"num_write_kb":163584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1333,"num_bytes_recovered":87359488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.550213"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":397,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.503039+0000","last_change":"2024-03-30T13:53:35.862417+0000","last_active":"2024-03-30T13:54:07.503039+0000","last_peered":"2024-03-30T13:54:07.503039+0000","last_clean":"2024-03-30T13:54:07.503039+0000","last_became_active":"2024-03-30T13:53:35.861853+0000","last_became_peered":"2024-03-30T13:53:35.861853+0000","last_unstale":"2024-03-30T13:54:07.503039+0000","last_undegraded":"2024-03-30T13:54:07.503039+0000","last_fullsized":"2024-03-30T13:54:07.503039+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:05.234023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:05.234023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":410,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:04.018073+0000","last_change":"2024-03-30T13:53:36.861076+0000","last_active":"2024-03-30T13:54:04.018073+0000","last_peered":"2024-03-30T13:54:04.018073+0000","last_clean":"2024-03-30T13:54:04.018073+0000","last_became_active":"2024-03-30T13:53:36.860312+0000","last_became_peered":"2024-03-30T13:53:36.860312+0000","last_unstale":"2024-03-30T13:54:04.018073+0000","last_undegraded":"2024-03-30T13:54:04.018073+0000","last_fullsized":"2024-03-30T13:54:04.018073+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:17.548567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:17.548567+0000","last_clean_scrub_stamp":"2024-03-30T13:52:17.548567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6,7],"avail_no_missing":["1","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"65'1","reported_seq":401,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:03.997000+0000","last_change":"2024-03-30T13:53:14.675272+0000","last_active":"2024-03-30T13:54:03.997000+0000","last_peered":"2024-03-30T13:54:03.997000+0000","last_clean":"2024-03-30T13:54:03.997000+0000","last_became_active":"2024-03-30T13:53:14.657693+0000","last_became_peered":"2024-03-30T13:53:14.657693+0000","last_unstale":"2024-03-30T13:54:03.997000+0000","last_undegraded":"2024-03-30T13:54:03.997000+0000","last_fullsized":"2024-03-30T13:54:03.997000+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:48:19.930059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:48:19.930059+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,5],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"216'15","reported_seq":426,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.502548+0000","last_change":"2024-03-30T13:52:56.028286+0000","last_active":"2024-03-30T13:54:07.502548+0000","last_peered":"2024-03-30T13:54:07.502548+0000","last_clean":"2024-03-30T13:54:07.502548+0000","last_became_active":"2024-03-30T13:42:47.319638+0000","last_became_peered":"2024-03-30T13:42:47.319638+0000","last_unstale":"2024-03-30T13:54:07.502548+0000","last_undegraded":"2024-03-30T13:54:07.502548+0000","last_fullsized":"2024-03-30T13:54:07.502548+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'15","last_scrub_stamp":"2024-03-30T13:52:56.028141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:56.028141+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":386,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.999973+0000","last_change":"2024-03-30T13:53:35.863312+0000","last_active":"2024-03-30T13:54:03.999973+0000","last_peered":"2024-03-30T13:54:03.999973+0000","last_clean":"2024-03-30T13:54:03.999973+0000","last_became_active":"2024-03-30T13:53:35.862085+0000","last_became_peered":"2024-03-30T13:53:35.862085+0000","last_unstale":"2024-03-30T13:54:03.999973+0000","last_undegraded":"2024-03-30T13:54:03.999973+0000","last_fullsized":"2024-03-30T13:54:03.999973+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:23.133743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:23.133743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":389,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.502639+0000","last_change":"2024-03-30T13:51:20.149796+0000","last_active":"2024-03-30T13:54:07.502639+0000","last_peered":"2024-03-30T13:54:07.502639+0000","last_clean":"2024-03-30T13:54:07.502639+0000","last_became_active":"2024-03-30T13:42:49.327124+0000","last_became_peered":"2024-03-30T13:42:49.327124+0000","last_unstale":"2024-03-30T13:54:07.502639+0000","last_undegraded":"2024-03-30T13:54:07.502639+0000","last_fullsized":"2024-03-30T13:54:07.502639+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:20.149684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:20.149684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":388,"reported_epoch":238,"state":"active+undersized","last_fresh":"2024-03-30T13:54:04.002174+0000","last_change":"2024-03-30T13:50:08.512316+0000","last_active":"2024-03-30T13:54:04.002174+0000","last_peered":"2024-03-30T13:54:04.002174+0000","last_clean":"2024-03-30T13:50:05.504688+0000","last_became_active":"2024-03-30T13:50:08.512316+0000","last_became_peered":"2024-03-30T13:50:08.512316+0000","last_unstale":"2024-03-30T13:54:04.002174+0000","last_undegraded":"2024-03-30T13:54:04.002174+0000","last_fullsized":"2024-03-30T13:50:08.501439+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:20.499768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:20.499768+0000","last_clean_scrub_stamp":"2024-03-30T13:49:20.499768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"217'34","reported_seq":425,"reported_epoch":238,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:54:04.000174+0000","last_change":"2024-03-30T13:50:08.511099+0000","last_active":"2024-03-30T13:54:04.000174+0000","last_peered":"2024-03-30T13:54:04.000174+0000","last_clean":"2024-03-30T13:50:05.499201+0000","last_became_active":"2024-03-30T13:50:08.511099+0000","last_became_peered":"2024-03-30T13:50:08.511099+0000","last_unstale":"2024-03-30T13:54:04.000174+0000","last_undegraded":"2024-03-30T13:50:08.502733+0000","last_fullsized":"2024-03-30T13:50:08.502606+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'22","last_scrub_stamp":"2024-03-30T13:46:49.339954+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:29.151791+0000","last_clean_scrub_stamp":"2024-03-30T13:46:49.339954+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":391,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:03.987585+0000","last_change":"2024-03-30T13:53:36.858395+0000","last_active":"2024-03-30T13:54:03.987585+0000","last_peered":"2024-03-30T13:54:03.987585+0000","last_clean":"2024-03-30T13:54:03.987585+0000","last_became_active":"2024-03-30T13:53:36.855710+0000","last_became_peered":"2024-03-30T13:53:36.855710+0000","last_unstale":"2024-03-30T13:54:03.987585+0000","last_undegraded":"2024-03-30T13:54:03.987585+0000","last_fullsized":"2024-03-30T13:54:03.987585+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:27.061281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:27.061281+0000","last_clean_scrub_stamp":"2024-03-30T13:52:27.061281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":50,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.994848+0000","last_change":"2024-03-30T13:53:35.857026+0000","last_active":"2024-03-30T13:54:03.994848+0000","last_peered":"2024-03-30T13:54:03.994848+0000","last_clean":"2024-03-30T13:54:03.994848+0000","last_became_active":"2024-03-30T13:53:35.856684+0000","last_became_peered":"2024-03-30T13:53:35.856684+0000","last_unstale":"2024-03-30T13:54:03.994848+0000","last_undegraded":"2024-03-30T13:54:03.994848+0000","last_fullsized":"2024-03-30T13:54:03.994848+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:20.069864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:20.069864+0000","last_clean_scrub_stamp":"2024-03-30T13:52:20.069864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":48,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.503501+0000","last_change":"2024-03-30T13:53:13.645288+0000","last_active":"2024-03-30T13:54:07.503501+0000","last_peered":"2024-03-30T13:54:07.503501+0000","last_clean":"2024-03-30T13:54:07.503501+0000","last_became_active":"2024-03-30T13:53:13.644821+0000","last_became_peered":"2024-03-30T13:53:13.644821+0000","last_unstale":"2024-03-30T13:54:07.503501+0000","last_undegraded":"2024-03-30T13:54:07.503501+0000","last_fullsized":"2024-03-30T13:54:07.503501+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:07.129142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:37.513596+0000","last_clean_scrub_stamp":"2024-03-30T13:52:07.129142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"216'15","reported_seq":426,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.503944+0000","last_change":"2024-03-30T13:51:16.153633+0000","last_active":"2024-03-30T13:54:07.503944+0000","last_peered":"2024-03-30T13:54:07.503944+0000","last_clean":"2024-03-30T13:54:07.503944+0000","last_became_active":"2024-03-30T13:42:47.310289+0000","last_became_peered":"2024-03-30T13:42:47.310289+0000","last_unstale":"2024-03-30T13:54:07.503944+0000","last_undegraded":"2024-03-30T13:54:07.503944+0000","last_fullsized":"2024-03-30T13:54:07.503944+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:51:16.153478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:16.153478+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":44,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:04.018451+0000","last_change":"2024-03-30T13:53:14.658585+0000","last_active":"2024-03-30T13:54:04.018451+0000","last_peered":"2024-03-30T13:54:04.018451+0000","last_clean":"2024-03-30T13:54:04.018451+0000","last_became_active":"2024-03-30T13:53:14.658167+0000","last_became_peered":"2024-03-30T13:53:14.658167+0000","last_unstale":"2024-03-30T13:54:04.018451+0000","last_undegraded":"2024-03-30T13:54:04.018451+0000","last_fullsized":"2024-03-30T13:54:04.018451+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:10.988839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:10.988839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3,5],"avail_no_missing":["1","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":34,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.504221+0000","last_change":"2024-03-30T13:53:35.862161+0000","last_active":"2024-03-30T13:54:07.504221+0000","last_peered":"2024-03-30T13:54:07.504221+0000","last_clean":"2024-03-30T13:54:07.504221+0000","last_became_active":"2024-03-30T13:53:35.861405+0000","last_became_peered":"2024-03-30T13:53:35.861405+0000","last_unstale":"2024-03-30T13:54:07.504221+0000","last_undegraded":"2024-03-30T13:54:07.504221+0000","last_fullsized":"2024-03-30T13:54:07.504221+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:50.953533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:41.633284+0000","last_clean_scrub_stamp":"2024-03-30T13:52:50.953533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"186'2","reported_seq":347,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.503977+0000","last_change":"2024-03-30T13:53:13.664011+0000","last_active":"2024-03-30T13:54:07.503977+0000","last_peered":"2024-03-30T13:54:07.503977+0000","last_clean":"2024-03-30T13:54:07.503977+0000","last_became_active":"2024-03-30T13:53:13.646659+0000","last_became_peered":"2024-03-30T13:53:13.646659+0000","last_unstale":"2024-03-30T13:54:07.503977+0000","last_undegraded":"2024-03-30T13:54:07.503977+0000","last_fullsized":"2024-03-30T13:54:07.503977+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-30T13:51:15.932076+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-30T13:51:15.932076+0000","last_clean_scrub_stamp":"2024-03-30T13:51:15.932076+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"216'9","reported_seq":424,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.503840+0000","last_change":"2024-03-30T13:52:19.931416+0000","last_active":"2024-03-30T13:54:07.503840+0000","last_peered":"2024-03-30T13:54:07.503840+0000","last_clean":"2024-03-30T13:54:07.503840+0000","last_became_active":"2024-03-30T13:42:47.317352+0000","last_became_peered":"2024-03-30T13:42:47.317352+0000","last_unstale":"2024-03-30T13:54:07.503840+0000","last_undegraded":"2024-03-30T13:54:07.503840+0000","last_fullsized":"2024-03-30T13:54:07.503840+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-30T13:52:19.931197+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-30T13:43:50.960980+0000","last_clean_scrub_stamp":"2024-03-30T13:52:19.931197+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":400,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.504191+0000","last_change":"2024-03-30T13:53:13.655592+0000","last_active":"2024-03-30T13:54:07.504191+0000","last_peered":"2024-03-30T13:54:07.504191+0000","last_clean":"2024-03-30T13:54:07.504191+0000","last_became_active":"2024-03-30T13:53:13.654803+0000","last_became_peered":"2024-03-30T13:53:13.654803+0000","last_unstale":"2024-03-30T13:54:07.504191+0000","last_undegraded":"2024-03-30T13:54:07.504191+0000","last_fullsized":"2024-03-30T13:54:07.504191+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:07.996724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:07.996724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":400,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.024742+0000","last_change":"2024-03-30T13:52:19.544522+0000","last_active":"2024-03-30T13:54:04.024742+0000","last_peered":"2024-03-30T13:54:04.024742+0000","last_clean":"2024-03-30T13:54:04.024742+0000","last_became_active":"2024-03-30T13:42:49.322665+0000","last_became_peered":"2024-03-30T13:42:49.322665+0000","last_unstale":"2024-03-30T13:54:04.024742+0000","last_undegraded":"2024-03-30T13:54:04.024742+0000","last_fullsized":"2024-03-30T13:54:04.024742+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:19.544305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:30.465974+0000","last_clean_scrub_stamp":"2024-03-30T13:52:19.544305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":404,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.995213+0000","last_change":"2024-03-30T13:53:35.861421+0000","last_active":"2024-03-30T13:54:03.995213+0000","last_peered":"2024-03-30T13:54:03.995213+0000","last_clean":"2024-03-30T13:54:03.995213+0000","last_became_active":"2024-03-30T13:53:35.861043+0000","last_became_peered":"2024-03-30T13:53:35.861043+0000","last_unstale":"2024-03-30T13:54:03.995213+0000","last_undegraded":"2024-03-30T13:54:03.995213+0000","last_fullsized":"2024-03-30T13:54:03.995213+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:16.009354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:43.088964+0000","last_clean_scrub_stamp":"2024-03-30T13:52:16.009354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"216'27","reported_seq":226,"reported_epoch":238,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:54:04.021814+0000","last_change":"2024-03-30T13:50:08.508437+0000","last_active":"2024-03-30T13:54:04.021814+0000","last_peered":"2024-03-30T13:54:04.021814+0000","last_clean":"2024-03-30T13:50:05.490228+0000","last_became_active":"2024-03-30T13:50:08.508437+0000","last_became_peered":"2024-03-30T13:50:08.508437+0000","last_unstale":"2024-03-30T13:54:04.021814+0000","last_undegraded":"2024-03-30T13:50:08.506233+0000","last_fullsized":"2024-03-30T13:50:08.506116+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-30T13:49:08.415681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:49:08.415681+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":387,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.002503+0000","last_change":"2024-03-30T13:53:13.665423+0000","last_active":"2024-03-30T13:54:04.002503+0000","last_peered":"2024-03-30T13:54:04.002503+0000","last_clean":"2024-03-30T13:54:04.002503+0000","last_became_active":"2024-03-30T13:53:13.664873+0000","last_became_peered":"2024-03-30T13:53:13.664873+0000","last_unstale":"2024-03-30T13:54:04.002503+0000","last_undegraded":"2024-03-30T13:54:04.002503+0000","last_fullsized":"2024-03-30T13:54:04.002503+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:52.054302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:52.054302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":384,"reported_epoch":240,"state":"active+undersized","last_fresh":"2024-03-30T13:54:07.504506+0000","last_change":"2024-03-30T13:50:08.506226+0000","last_active":"2024-03-30T13:54:07.504506+0000","last_peered":"2024-03-30T13:54:07.504506+0000","last_clean":"2024-03-30T13:50:05.490914+0000","last_became_active":"2024-03-30T13:50:08.506226+0000","last_became_peered":"2024-03-30T13:50:08.506226+0000","last_unstale":"2024-03-30T13:54:07.504506+0000","last_undegraded":"2024-03-30T13:54:07.504506+0000","last_fullsized":"2024-03-30T13:50:08.502611+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:25.104610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:48:25.104610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":392,"reported_epoch":240,"state":"active+undersized","last_fresh":"2024-03-30T13:54:07.503992+0000","last_change":"2024-03-30T13:50:08.508556+0000","last_active":"2024-03-30T13:54:07.503992+0000","last_peered":"2024-03-30T13:54:07.503992+0000","last_clean":"2024-03-30T13:50:07.261781+0000","last_became_active":"2024-03-30T13:50:08.508556+0000","last_became_peered":"2024-03-30T13:50:08.508556+0000","last_unstale":"2024-03-30T13:54:07.503992+0000","last_undegraded":"2024-03-30T13:54:07.503992+0000","last_fullsized":"2024-03-30T13:50:08.502793+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:07.261595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:50:07.261595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"217'23","reported_seq":433,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.000347+0000","last_change":"2024-03-30T13:52:50.945958+0000","last_active":"2024-03-30T13:54:04.000347+0000","last_peered":"2024-03-30T13:54:04.000347+0000","last_clean":"2024-03-30T13:54:04.000347+0000","last_became_active":"2024-03-30T13:42:47.317881+0000","last_became_peered":"2024-03-30T13:42:47.317881+0000","last_unstale":"2024-03-30T13:54:04.000347+0000","last_undegraded":"2024-03-30T13:54:04.000347+0000","last_fullsized":"2024-03-30T13:54:04.000347+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'17","last_scrub_stamp":"2024-03-30T13:52:50.945844+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-03-30T13:43:52.044427+0000","last_clean_scrub_stamp":"2024-03-30T13:52:50.945844+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:04.119247+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":372,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:03.993708+0000","last_change":"2024-03-30T13:53:37.211479+0000","last_active":"2024-03-30T13:54:03.993708+0000","last_peered":"2024-03-30T13:54:03.993708+0000","last_clean":"2024-03-30T13:54:03.993708+0000","last_became_active":"2024-03-30T13:53:37.109227+0000","last_became_peered":"2024-03-30T13:53:37.109227+0000","last_unstale":"2024-03-30T13:54:03.993708+0000","last_undegraded":"2024-03-30T13:54:03.993708+0000","last_fullsized":"2024-03-30T13:54:03.993708+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T13:48:25.249724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:25.249724+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,7],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"230'10","reported_seq":557,"reported_epoch":240,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:07.520682+0000","last_change":"2024-03-30T13:53:36.854785+0000","last_active":"2024-03-30T13:54:07.520682+0000","last_peered":"2024-03-30T13:54:07.520682+0000","last_clean":"2024-03-30T13:54:07.520682+0000","last_became_active":"2024-03-30T13:53:36.852879+0000","last_became_peered":"2024-03-30T13:53:36.852879+0000","last_unstale":"2024-03-30T13:54:07.520682+0000","last_undegraded":"2024-03-30T13:54:07.520682+0000","last_fullsized":"2024-03-30T13:54:07.520682+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:52:07.177214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:07.177214+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,7],"avail_no_missing":["6","4","7"],"object_location_counts":[{"shards":"4,6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"65'1","reported_seq":409,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.017955+0000","last_change":"2024-03-30T13:52:28.566954+0000","last_active":"2024-03-30T13:54:04.017955+0000","last_peered":"2024-03-30T13:54:04.017955+0000","last_clean":"2024-03-30T13:54:04.017955+0000","last_became_active":"2024-03-30T13:42:45.307832+0000","last_became_peered":"2024-03-30T13:42:45.307832+0000","last_unstale":"2024-03-30T13:54:04.017955+0000","last_undegraded":"2024-03-30T13:54:04.017955+0000","last_fullsized":"2024-03-30T13:54:04.017955+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:52:28.566847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:28.566847+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"217'25","reported_seq":456,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.998070+0000","last_change":"2024-03-30T13:53:35.974113+0000","last_active":"2024-03-30T13:54:03.998070+0000","last_peered":"2024-03-30T13:54:03.998070+0000","last_clean":"2024-03-30T13:54:03.998070+0000","last_became_active":"2024-03-30T13:53:35.858156+0000","last_became_peered":"2024-03-30T13:53:35.858156+0000","last_unstale":"2024-03-30T13:54:03.998070+0000","last_undegraded":"2024-03-30T13:54:03.998070+0000","last_fullsized":"2024-03-30T13:54:03.998070+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'25","last_scrub_stamp":"2024-03-30T13:52:54.925758+0000","last_deep_scrub":"153'15","last_deep_scrub_stamp":"2024-03-30T13:51:22.252831+0000","last_clean_scrub_stamp":"2024-03-30T13:52:54.925758+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:12.129951+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":393,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:04.024309+0000","last_change":"2024-03-30T13:53:14.660108+0000","last_active":"2024-03-30T13:54:04.024309+0000","last_peered":"2024-03-30T13:54:04.024309+0000","last_clean":"2024-03-30T13:54:04.024309+0000","last_became_active":"2024-03-30T13:53:14.659650+0000","last_became_peered":"2024-03-30T13:53:14.659650+0000","last_unstale":"2024-03-30T13:54:04.024309+0000","last_undegraded":"2024-03-30T13:54:04.024309+0000","last_fullsized":"2024-03-30T13:54:04.024309+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:00.432456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:00.432456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,5],"avail_no_missing":["1","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":391,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:03.998243+0000","last_change":"2024-03-30T13:53:36.860961+0000","last_active":"2024-03-30T13:54:03.998243+0000","last_peered":"2024-03-30T13:54:03.998243+0000","last_clean":"2024-03-30T13:54:03.998243+0000","last_became_active":"2024-03-30T13:53:36.857751+0000","last_became_peered":"2024-03-30T13:53:36.857751+0000","last_unstale":"2024-03-30T13:54:03.998243+0000","last_undegraded":"2024-03-30T13:54:03.998243+0000","last_fullsized":"2024-03-30T13:54:03.998243+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:27.947416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:27.947416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":392,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.994299+0000","last_change":"2024-03-30T13:52:17.021134+0000","last_active":"2024-03-30T13:54:03.994299+0000","last_peered":"2024-03-30T13:54:03.994299+0000","last_clean":"2024-03-30T13:54:03.994299+0000","last_became_active":"2024-03-30T13:42:45.310165+0000","last_became_peered":"2024-03-30T13:42:45.310165+0000","last_unstale":"2024-03-30T13:54:03.994299+0000","last_undegraded":"2024-03-30T13:54:03.994299+0000","last_fullsized":"2024-03-30T13:54:03.994299+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:17.020887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:17.020887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"217'19","reported_seq":456,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.521089+0000","last_change":"2024-03-30T13:53:13.673522+0000","last_active":"2024-03-30T13:54:07.521089+0000","last_peered":"2024-03-30T13:54:07.521089+0000","last_clean":"2024-03-30T13:54:07.521089+0000","last_became_active":"2024-03-30T13:53:13.650706+0000","last_became_peered":"2024-03-30T13:53:13.650706+0000","last_unstale":"2024-03-30T13:54:07.521089+0000","last_undegraded":"2024-03-30T13:54:07.521089+0000","last_fullsized":"2024-03-30T13:54:07.521089+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:52:26.096051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:26.096051+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":259,"reported_epoch":240,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:07.502800+0000","last_change":"2024-03-30T13:53:36.864671+0000","last_active":"2024-03-30T13:54:07.502800+0000","last_peered":"2024-03-30T13:54:07.502800+0000","last_clean":"2024-03-30T13:54:07.502800+0000","last_became_active":"2024-03-30T13:53:36.863938+0000","last_became_peered":"2024-03-30T13:53:36.863938+0000","last_unstale":"2024-03-30T13:54:07.502800+0000","last_undegraded":"2024-03-30T13:54:07.502800+0000","last_fullsized":"2024-03-30T13:54:07.502800+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:23.264538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:23.264538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3,7],"avail_no_missing":["4","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":416,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.520331+0000","last_change":"2024-03-30T13:53:35.862789+0000","last_active":"2024-03-30T13:54:07.520331+0000","last_peered":"2024-03-30T13:54:07.520331+0000","last_clean":"2024-03-30T13:54:07.520331+0000","last_became_active":"2024-03-30T13:53:35.862467+0000","last_became_peered":"2024-03-30T13:53:35.862467+0000","last_unstale":"2024-03-30T13:54:07.520331+0000","last_undegraded":"2024-03-30T13:54:07.520331+0000","last_fullsized":"2024-03-30T13:54:07.520331+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:17.108986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:54.893653+0000","last_clean_scrub_stamp":"2024-03-30T13:52:17.108986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":412,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.997352+0000","last_change":"2024-03-30T13:53:13.661456+0000","last_active":"2024-03-30T13:54:03.997352+0000","last_peered":"2024-03-30T13:54:03.997352+0000","last_clean":"2024-03-30T13:54:03.997352+0000","last_became_active":"2024-03-30T13:53:13.661128+0000","last_became_peered":"2024-03-30T13:53:13.661128+0000","last_unstale":"2024-03-30T13:54:03.997352+0000","last_undegraded":"2024-03-30T13:54:03.997352+0000","last_fullsized":"2024-03-30T13:54:03.997352+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:47.933046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:47.933046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"216'8","reported_seq":400,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.994373+0000","last_change":"2024-03-30T13:52:03.989310+0000","last_active":"2024-03-30T13:54:03.994373+0000","last_peered":"2024-03-30T13:54:03.994373+0000","last_clean":"2024-03-30T13:54:03.994373+0000","last_became_active":"2024-03-30T13:42:47.321428+0000","last_became_peered":"2024-03-30T13:42:47.321428+0000","last_unstale":"2024-03-30T13:54:03.994373+0000","last_undegraded":"2024-03-30T13:54:03.994373+0000","last_fullsized":"2024-03-30T13:54:03.994373+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-30T13:52:03.989075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:03.989075+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":381,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.000142+0000","last_change":"2024-03-30T13:52:26.149264+0000","last_active":"2024-03-30T13:54:04.000142+0000","last_peered":"2024-03-30T13:54:04.000142+0000","last_clean":"2024-03-30T13:54:04.000142+0000","last_became_active":"2024-03-30T13:42:51.374304+0000","last_became_peered":"2024-03-30T13:42:51.374304+0000","last_unstale":"2024-03-30T13:54:04.000142+0000","last_undegraded":"2024-03-30T13:54:04.000142+0000","last_fullsized":"2024-03-30T13:54:04.000142+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:26.149132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:26.149132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":34,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.502449+0000","last_change":"2024-03-30T13:53:35.854347+0000","last_active":"2024-03-30T13:54:07.502449+0000","last_peered":"2024-03-30T13:54:07.502449+0000","last_clean":"2024-03-30T13:54:07.502449+0000","last_became_active":"2024-03-30T13:53:35.854002+0000","last_became_peered":"2024-03-30T13:53:35.854002+0000","last_unstale":"2024-03-30T13:54:07.502449+0000","last_undegraded":"2024-03-30T13:54:07.502449+0000","last_fullsized":"2024-03-30T13:54:07.502449+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:56.878297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:42.616155+0000","last_clean_scrub_stamp":"2024-03-30T13:52:56.878297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:13.302043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":32,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:04.021618+0000","last_change":"2024-03-30T13:53:36.870170+0000","last_active":"2024-03-30T13:54:04.021618+0000","last_peered":"2024-03-30T13:54:04.021618+0000","last_clean":"2024-03-30T13:54:04.021618+0000","last_became_active":"2024-03-30T13:53:36.869434+0000","last_became_peered":"2024-03-30T13:53:36.869434+0000","last_unstale":"2024-03-30T13:54:04.021618+0000","last_undegraded":"2024-03-30T13:54:04.021618+0000","last_fullsized":"2024-03-30T13:54:04.021618+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:04.222591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:25.494903+0000","last_clean_scrub_stamp":"2024-03-30T13:52:04.222591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3,7],"avail_no_missing":["1","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"217'19","reported_seq":445,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.000060+0000","last_change":"2024-03-30T13:53:36.012242+0000","last_active":"2024-03-30T13:54:04.000060+0000","last_peered":"2024-03-30T13:54:04.000060+0000","last_clean":"2024-03-30T13:54:04.000060+0000","last_became_active":"2024-03-30T13:53:35.863880+0000","last_became_peered":"2024-03-30T13:53:35.863880+0000","last_unstale":"2024-03-30T13:54:04.000060+0000","last_undegraded":"2024-03-30T13:54:04.000060+0000","last_fullsized":"2024-03-30T13:54:04.000060+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'19","last_scrub_stamp":"2024-03-30T13:52:57.970377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:57.970377+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":392,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.987332+0000","last_change":"2024-03-30T13:53:13.650823+0000","last_active":"2024-03-30T13:54:03.987332+0000","last_peered":"2024-03-30T13:54:03.987332+0000","last_clean":"2024-03-30T13:54:03.987332+0000","last_became_active":"2024-03-30T13:53:13.650411+0000","last_became_peered":"2024-03-30T13:53:13.650411+0000","last_unstale":"2024-03-30T13:54:03.987332+0000","last_undegraded":"2024-03-30T13:54:03.987332+0000","last_fullsized":"2024-03-30T13:54:03.987332+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:03.006855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:28.314391+0000","last_clean_scrub_stamp":"2024-03-30T13:52:03.006855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":384,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.999890+0000","last_change":"2024-03-30T13:52:16.009739+0000","last_active":"2024-03-30T13:54:03.999890+0000","last_peered":"2024-03-30T13:54:03.999890+0000","last_clean":"2024-03-30T13:54:03.999890+0000","last_became_active":"2024-03-30T13:42:49.325219+0000","last_became_peered":"2024-03-30T13:42:49.325219+0000","last_unstale":"2024-03-30T13:54:03.999890+0000","last_undegraded":"2024-03-30T13:54:03.999890+0000","last_fullsized":"2024-03-30T13:54:03.999890+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:16.009499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:16.009499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":30,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.000205+0000","last_change":"2024-03-30T13:53:35.859545+0000","last_active":"2024-03-30T13:54:04.000205+0000","last_peered":"2024-03-30T13:54:04.000205+0000","last_clean":"2024-03-30T13:54:04.000205+0000","last_became_active":"2024-03-30T13:53:35.859253+0000","last_became_peered":"2024-03-30T13:53:35.859253+0000","last_unstale":"2024-03-30T13:54:04.000205+0000","last_undegraded":"2024-03-30T13:54:04.000205+0000","last_fullsized":"2024-03-30T13:54:04.000205+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:11.701294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:11.701294+0000","last_clean_scrub_stamp":"2024-03-30T13:52:11.701294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"216'15","reported_seq":414,"reported_epoch":238,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:54:04.024812+0000","last_change":"2024-03-30T13:50:08.513156+0000","last_active":"2024-03-30T13:54:04.024812+0000","last_peered":"2024-03-30T13:54:04.024812+0000","last_clean":"2024-03-30T13:50:05.496678+0000","last_became_active":"2024-03-30T13:50:08.513156+0000","last_became_peered":"2024-03-30T13:50:08.513156+0000","last_unstale":"2024-03-30T13:54:04.024812+0000","last_undegraded":"2024-03-30T13:50:08.507726+0000","last_fullsized":"2024-03-30T13:50:08.507616+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:49:26.480540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:49:26.480540+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":385,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:03.999257+0000","last_change":"2024-03-30T13:53:14.826350+0000","last_active":"2024-03-30T13:54:03.999257+0000","last_peered":"2024-03-30T13:54:03.999257+0000","last_clean":"2024-03-30T13:54:03.999257+0000","last_became_active":"2024-03-30T13:53:14.822437+0000","last_became_peered":"2024-03-30T13:53:14.822437+0000","last_unstale":"2024-03-30T13:54:03.999257+0000","last_undegraded":"2024-03-30T13:54:03.999257+0000","last_fullsized":"2024-03-30T13:54:03.999257+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:03.011875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:24.598976+0000","last_clean_scrub_stamp":"2024-03-30T13:52:03.011875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0,5],"avail_no_missing":["3","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":410,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.024595+0000","last_change":"2024-03-30T13:53:35.871876+0000","last_active":"2024-03-30T13:54:04.024595+0000","last_peered":"2024-03-30T13:54:04.024595+0000","last_clean":"2024-03-30T13:54:04.024595+0000","last_became_active":"2024-03-30T13:53:35.870924+0000","last_became_peered":"2024-03-30T13:53:35.870924+0000","last_unstale":"2024-03-30T13:54:04.024595+0000","last_undegraded":"2024-03-30T13:54:04.024595+0000","last_fullsized":"2024-03-30T13:54:04.024595+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:46.670446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:34.815783+0000","last_clean_scrub_stamp":"2024-03-30T13:52:46.670446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:08.196457+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":410,"reported_epoch":240,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:07.504220+0000","last_change":"2024-03-30T13:53:36.849109+0000","last_active":"2024-03-30T13:54:07.504220+0000","last_peered":"2024-03-30T13:54:07.504220+0000","last_clean":"2024-03-30T13:54:07.504220+0000","last_became_active":"2024-03-30T13:53:36.848656+0000","last_became_peered":"2024-03-30T13:53:36.848656+0000","last_unstale":"2024-03-30T13:54:07.504220+0000","last_undegraded":"2024-03-30T13:54:07.504220+0000","last_fullsized":"2024-03-30T13:54:07.504220+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:14.015356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:14.015356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,7],"avail_no_missing":["4","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"216'19","reported_seq":437,"reported_epoch":240,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:54:07.503573+0000","last_change":"2024-03-30T13:50:08.513382+0000","last_active":"2024-03-30T13:54:07.503573+0000","last_peered":"2024-03-30T13:54:07.503573+0000","last_clean":"2024-03-30T13:50:05.491029+0000","last_became_active":"2024-03-30T13:50:08.513382+0000","last_became_peered":"2024-03-30T13:50:08.513382+0000","last_unstale":"2024-03-30T13:54:07.503573+0000","last_undegraded":"2024-03-30T13:50:08.503759+0000","last_fullsized":"2024-03-30T13:50:08.503643+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:48:22.068206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:22.068206+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":400,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.997833+0000","last_change":"2024-03-30T13:53:35.856379+0000","last_active":"2024-03-30T13:54:03.997833+0000","last_peered":"2024-03-30T13:54:03.997833+0000","last_clean":"2024-03-30T13:54:03.997833+0000","last_became_active":"2024-03-30T13:53:35.855996+0000","last_became_peered":"2024-03-30T13:53:35.855996+0000","last_unstale":"2024-03-30T13:54:03.997833+0000","last_undegraded":"2024-03-30T13:54:03.997833+0000","last_fullsized":"2024-03-30T13:54:03.997833+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:56.942611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:56.942611+0000","last_clean_scrub_stamp":"2024-03-30T13:52:56.942611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:08.829333+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"222'9","reported_seq":84,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.024662+0000","last_change":"2024-03-30T13:53:13.664599+0000","last_active":"2024-03-30T13:54:04.024662+0000","last_peered":"2024-03-30T13:54:04.024662+0000","last_clean":"2024-03-30T13:54:04.024662+0000","last_became_active":"2024-03-30T13:53:13.651996+0000","last_became_peered":"2024-03-30T13:53:13.651996+0000","last_unstale":"2024-03-30T13:54:04.024662+0000","last_undegraded":"2024-03-30T13:54:04.024662+0000","last_fullsized":"2024-03-30T13:54:04.024662+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:52:16.103727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:16.103727+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":271,"reported_epoch":240,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:07.503385+0000","last_change":"2024-03-30T13:53:36.850388+0000","last_active":"2024-03-30T13:54:07.503385+0000","last_peered":"2024-03-30T13:54:07.503385+0000","last_clean":"2024-03-30T13:54:07.503385+0000","last_became_active":"2024-03-30T13:53:36.849971+0000","last_became_peered":"2024-03-30T13:53:36.849971+0000","last_unstale":"2024-03-30T13:54:07.503385+0000","last_undegraded":"2024-03-30T13:54:07.503385+0000","last_fullsized":"2024-03-30T13:54:07.503385+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:07.618186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:49:07.618186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,7],"avail_no_missing":["4","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"217'20","reported_seq":440,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:04.001165+0000","last_change":"2024-03-30T13:53:37.435184+0000","last_active":"2024-03-30T13:54:04.001165+0000","last_peered":"2024-03-30T13:54:04.001165+0000","last_clean":"2024-03-30T13:54:04.001165+0000","last_became_active":"2024-03-30T13:53:37.110194+0000","last_became_peered":"2024-03-30T13:53:37.110194+0000","last_unstale":"2024-03-30T13:54:04.001165+0000","last_undegraded":"2024-03-30T13:54:04.001165+0000","last_fullsized":"2024-03-30T13:54:04.001165+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-30T13:48:21.678227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:21.678227+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":385,"reported_epoch":240,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:07.503899+0000","last_change":"2024-03-30T13:53:14.659456+0000","last_active":"2024-03-30T13:54:07.503899+0000","last_peered":"2024-03-30T13:54:07.503899+0000","last_clean":"2024-03-30T13:54:07.503899+0000","last_became_active":"2024-03-30T13:53:14.658940+0000","last_became_peered":"2024-03-30T13:53:14.658940+0000","last_unstale":"2024-03-30T13:54:07.503899+0000","last_undegraded":"2024-03-30T13:54:07.503899+0000","last_fullsized":"2024-03-30T13:54:07.503899+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:30.238582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:30.238582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,5],"avail_no_missing":["4","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"230'11","reported_seq":534,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:04.001426+0000","last_change":"2024-03-30T13:53:37.356715+0000","last_active":"2024-03-30T13:54:04.001426+0000","last_peered":"2024-03-30T13:54:04.001426+0000","last_clean":"2024-03-30T13:54:04.001426+0000","last_became_active":"2024-03-30T13:53:37.127826+0000","last_became_peered":"2024-03-30T13:53:37.127826+0000","last_unstale":"2024-03-30T13:54:04.001426+0000","last_undegraded":"2024-03-30T13:54:04.001426+0000","last_fullsized":"2024-03-30T13:54:04.001426+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:49:23.573435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:23.573435+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":404,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.993560+0000","last_change":"2024-03-30T13:53:35.857771+0000","last_active":"2024-03-30T13:54:03.993560+0000","last_peered":"2024-03-30T13:54:03.993560+0000","last_clean":"2024-03-30T13:54:03.993560+0000","last_became_active":"2024-03-30T13:53:35.857456+0000","last_became_peered":"2024-03-30T13:53:35.857456+0000","last_unstale":"2024-03-30T13:54:03.993560+0000","last_undegraded":"2024-03-30T13:54:03.993560+0000","last_fullsized":"2024-03-30T13:54:03.993560+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:09.007490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:28.734080+0000","last_clean_scrub_stamp":"2024-03-30T13:52:09.007490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"217'15","reported_seq":416,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.000442+0000","last_change":"2024-03-30T13:53:35.896795+0000","last_active":"2024-03-30T13:54:04.000442+0000","last_peered":"2024-03-30T13:54:04.000442+0000","last_clean":"2024-03-30T13:54:04.000442+0000","last_became_active":"2024-03-30T13:53:35.865891+0000","last_became_peered":"2024-03-30T13:53:35.865891+0000","last_unstale":"2024-03-30T13:54:04.000442+0000","last_undegraded":"2024-03-30T13:54:04.000442+0000","last_fullsized":"2024-03-30T13:54:04.000442+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:52:03.227628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:03.227628+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":173,"reported_epoch":240,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:07.504672+0000","last_change":"2024-03-30T13:53:36.857499+0000","last_active":"2024-03-30T13:54:07.504672+0000","last_peered":"2024-03-30T13:54:07.504672+0000","last_clean":"2024-03-30T13:54:07.504672+0000","last_became_active":"2024-03-30T13:53:36.857026+0000","last_became_peered":"2024-03-30T13:53:36.857026+0000","last_unstale":"2024-03-30T13:54:07.504672+0000","last_undegraded":"2024-03-30T13:54:07.504672+0000","last_fullsized":"2024-03-30T13:54:07.504672+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:14.322254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:14.322254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,7],"avail_no_missing":["4","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":390,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.003099+0000","last_change":"2024-03-30T13:53:35.876084+0000","last_active":"2024-03-30T13:54:04.003099+0000","last_peered":"2024-03-30T13:54:04.003099+0000","last_clean":"2024-03-30T13:54:04.003099+0000","last_became_active":"2024-03-30T13:53:35.875783+0000","last_became_peered":"2024-03-30T13:53:35.875783+0000","last_unstale":"2024-03-30T13:54:04.003099+0000","last_undegraded":"2024-03-30T13:54:04.003099+0000","last_fullsized":"2024-03-30T13:54:04.003099+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:22.106782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:22.106782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":34,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.503037+0000","last_change":"2024-03-30T13:53:35.858594+0000","last_active":"2024-03-30T13:54:07.503037+0000","last_peered":"2024-03-30T13:54:07.503037+0000","last_clean":"2024-03-30T13:54:07.503037+0000","last_became_active":"2024-03-30T13:53:35.858261+0000","last_became_peered":"2024-03-30T13:53:35.858261+0000","last_unstale":"2024-03-30T13:54:07.503037+0000","last_undegraded":"2024-03-30T13:54:07.503037+0000","last_fullsized":"2024-03-30T13:54:07.503037+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:18.239047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:18.239047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"218'16","reported_seq":433,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:03.998126+0000","last_change":"2024-03-30T13:53:36.861902+0000","last_active":"2024-03-30T13:54:03.998126+0000","last_peered":"2024-03-30T13:54:03.998126+0000","last_clean":"2024-03-30T13:54:03.998126+0000","last_became_active":"2024-03-30T13:53:36.861455+0000","last_became_peered":"2024-03-30T13:53:36.861455+0000","last_unstale":"2024-03-30T13:54:03.998126+0000","last_undegraded":"2024-03-30T13:54:03.998126+0000","last_fullsized":"2024-03-30T13:54:03.998126+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'16","last_scrub_stamp":"2024-03-30T13:52:59.883027+0000","last_deep_scrub":"218'16","last_deep_scrub_stamp":"2024-03-30T13:52:59.883027+0000","last_clean_scrub_stamp":"2024-03-30T13:52:59.883027+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:14.792396+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3,7],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"72'1","reported_seq":50,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.521079+0000","last_change":"2024-03-30T13:53:35.870213+0000","last_active":"2024-03-30T13:54:07.521079+0000","last_peered":"2024-03-30T13:54:07.521079+0000","last_clean":"2024-03-30T13:54:07.521079+0000","last_became_active":"2024-03-30T13:53:35.864200+0000","last_became_peered":"2024-03-30T13:53:35.864200+0000","last_unstale":"2024-03-30T13:54:07.521079+0000","last_undegraded":"2024-03-30T13:54:07.521079+0000","last_fullsized":"2024-03-30T13:54:07.521079+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T13:52:27.891771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:27.891771+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":44,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.001164+0000","last_change":"2024-03-30T13:53:13.661413+0000","last_active":"2024-03-30T13:54:04.001164+0000","last_peered":"2024-03-30T13:54:04.001164+0000","last_clean":"2024-03-30T13:54:04.001164+0000","last_became_active":"2024-03-30T13:53:13.661052+0000","last_became_peered":"2024-03-30T13:53:13.661052+0000","last_unstale":"2024-03-30T13:54:04.001164+0000","last_undegraded":"2024-03-30T13:54:04.001164+0000","last_fullsized":"2024-03-30T13:54:04.001164+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:09.123811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:09.123811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":424,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.520181+0000","last_change":"2024-03-30T13:53:13.658240+0000","last_active":"2024-03-30T13:54:07.520181+0000","last_peered":"2024-03-30T13:54:07.520181+0000","last_clean":"2024-03-30T13:54:07.520181+0000","last_became_active":"2024-03-30T13:53:13.657810+0000","last_became_peered":"2024-03-30T13:53:13.657810+0000","last_unstale":"2024-03-30T13:54:07.520181+0000","last_undegraded":"2024-03-30T13:54:07.520181+0000","last_fullsized":"2024-03-30T13:54:07.520181+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:45.998724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:51:20.175438+0000","last_clean_scrub_stamp":"2024-03-30T13:52:45.998724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:08.872950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"216'12","reported_seq":411,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:04.002876+0000","last_change":"2024-03-30T13:53:37.104181+0000","last_active":"2024-03-30T13:54:04.002876+0000","last_peered":"2024-03-30T13:54:04.002876+0000","last_clean":"2024-03-30T13:54:04.002876+0000","last_became_active":"2024-03-30T13:53:37.098834+0000","last_became_peered":"2024-03-30T13:53:37.098834+0000","last_unstale":"2024-03-30T13:54:04.002876+0000","last_undegraded":"2024-03-30T13:54:04.002876+0000","last_fullsized":"2024-03-30T13:54:04.002876+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-03-30T13:52:14.027491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:14.027491+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"238'4806","reported_seq":6847,"reported_epoch":238,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-30T13:54:04.675781+0000","last_change":"2024-03-30T13:53:37.555513+0000","last_active":"2024-03-30T13:54:04.675781+0000","last_peered":"2024-03-30T13:54:04.675781+0000","last_clean":"2024-03-30T13:53:12.631731+0000","last_became_active":"2024-03-30T13:53:37.539916+0000","last_became_peered":"2024-03-30T13:53:37.539916+0000","last_unstale":"2024-03-30T13:54:04.675781+0000","last_undegraded":"2024-03-30T13:53:37.277057+0000","last_fullsized":"2024-03-30T13:53:36.839152+0000","mapping_epoch":229,"log_start":"217'900","ondisk_log_start":"217'900","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3906,"log_dups_size":0,"ondisk_log_size":3906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314966016,"num_objects":4806,"num_object_clones":0,"num_object_copies":14418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4806,"num_write_kb":307584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":985,"num_bytes_recovered":64552960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,1","objects":916},{"shards":"0,1,5,7","objects":985},{"shards":"0,1,7","objects":791},{"shards":"0,5,7","objects":2114}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":48,"reported_epoch":240,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:07.520965+0000","last_change":"2024-03-30T13:53:14.655924+0000","last_active":"2024-03-30T13:54:07.520965+0000","last_peered":"2024-03-30T13:54:07.520965+0000","last_clean":"2024-03-30T13:54:07.520965+0000","last_became_active":"2024-03-30T13:53:14.655517+0000","last_became_peered":"2024-03-30T13:53:14.655517+0000","last_unstale":"2024-03-30T13:54:07.520965+0000","last_undegraded":"2024-03-30T13:54:07.520965+0000","last_fullsized":"2024-03-30T13:54:07.520965+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:05.957043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:05.957043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0,5],"avail_no_missing":["6","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":49,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.503551+0000","last_change":"2024-03-30T13:53:36.015472+0000","last_active":"2024-03-30T13:54:07.503551+0000","last_peered":"2024-03-30T13:54:07.503551+0000","last_clean":"2024-03-30T13:54:07.503551+0000","last_became_active":"2024-03-30T13:53:35.874521+0000","last_became_peered":"2024-03-30T13:53:35.874521+0000","last_unstale":"2024-03-30T13:54:07.503551+0000","last_undegraded":"2024-03-30T13:54:07.503551+0000","last_fullsized":"2024-03-30T13:54:07.503551+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-30T13:52:09.680567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:09.680567+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"216'27","reported_seq":442,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.520354+0000","last_change":"2024-03-30T13:53:35.901313+0000","last_active":"2024-03-30T13:54:07.520354+0000","last_peered":"2024-03-30T13:54:07.520354+0000","last_clean":"2024-03-30T13:54:07.520354+0000","last_became_active":"2024-03-30T13:53:35.887732+0000","last_became_peered":"2024-03-30T13:53:35.887732+0000","last_unstale":"2024-03-30T13:54:07.520354+0000","last_undegraded":"2024-03-30T13:54:07.520354+0000","last_fullsized":"2024-03-30T13:54:07.520354+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-30T13:51:27.133454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:27.133454+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"229'9","reported_seq":70,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.520118+0000","last_change":"2024-03-30T13:53:35.877180+0000","last_active":"2024-03-30T13:54:07.520118+0000","last_peered":"2024-03-30T13:54:07.520118+0000","last_clean":"2024-03-30T13:54:07.520118+0000","last_became_active":"2024-03-30T13:53:35.862126+0000","last_became_peered":"2024-03-30T13:53:35.862126+0000","last_unstale":"2024-03-30T13:54:07.520118+0000","last_undegraded":"2024-03-30T13:54:07.520118+0000","last_fullsized":"2024-03-30T13:54:07.520118+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:51:19.286592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:19.286592+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"238'4553","reported_seq":4617,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.879678+0000","last_change":"2024-03-30T13:52:43.529140+0000","last_active":"2024-03-30T13:54:04.879678+0000","last_peered":"2024-03-30T13:54:04.879678+0000","last_clean":"2024-03-30T13:54:04.879678+0000","last_became_active":"2024-03-30T13:52:43.528506+0000","last_became_peered":"2024-03-30T13:52:43.528506+0000","last_unstale":"2024-03-30T13:54:04.879678+0000","last_undegraded":"2024-03-30T13:54:04.879678+0000","last_fullsized":"2024-03-30T13:54:04.879678+0000","mapping_epoch":211,"log_start":"218'1149","ondisk_log_start":"218'1149","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3404,"log_dups_size":0,"ondisk_log_size":3404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298385408,"num_objects":4553,"num_object_clones":0,"num_object_copies":13659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4553,"num_write_kb":291392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":386,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.000059+0000","last_change":"2024-03-30T13:53:35.863569+0000","last_active":"2024-03-30T13:54:04.000059+0000","last_peered":"2024-03-30T13:54:04.000059+0000","last_clean":"2024-03-30T13:54:04.000059+0000","last_became_active":"2024-03-30T13:53:35.862056+0000","last_became_peered":"2024-03-30T13:53:35.862056+0000","last_unstale":"2024-03-30T13:54:04.000059+0000","last_undegraded":"2024-03-30T13:54:04.000059+0000","last_fullsized":"2024-03-30T13:54:04.000059+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:28.078616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:28.078616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":34,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.503157+0000","last_change":"2024-03-30T13:53:35.861728+0000","last_active":"2024-03-30T13:54:07.503157+0000","last_peered":"2024-03-30T13:54:07.503157+0000","last_clean":"2024-03-30T13:54:07.503157+0000","last_became_active":"2024-03-30T13:53:35.861370+0000","last_became_peered":"2024-03-30T13:53:35.861370+0000","last_unstale":"2024-03-30T13:54:07.503157+0000","last_undegraded":"2024-03-30T13:54:07.503157+0000","last_fullsized":"2024-03-30T13:54:07.503157+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:15.246997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:15.246997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"216'18","reported_seq":434,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.503126+0000","last_change":"2024-03-30T13:52:11.994083+0000","last_active":"2024-03-30T13:54:07.503126+0000","last_peered":"2024-03-30T13:54:07.503126+0000","last_clean":"2024-03-30T13:54:07.503126+0000","last_became_active":"2024-03-30T13:42:47.316187+0000","last_became_peered":"2024-03-30T13:42:47.316187+0000","last_unstale":"2024-03-30T13:54:07.503126+0000","last_undegraded":"2024-03-30T13:54:07.503126+0000","last_fullsized":"2024-03-30T13:54:07.503126+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-30T13:52:11.993946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:11.993946+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":164,"reported_epoch":238,"state":"active+undersized","last_fresh":"2024-03-30T13:54:03.987588+0000","last_change":"2024-03-30T13:50:08.509701+0000","last_active":"2024-03-30T13:54:03.987588+0000","last_peered":"2024-03-30T13:54:03.987588+0000","last_clean":"2024-03-30T13:50:05.499237+0000","last_became_active":"2024-03-30T13:50:08.509701+0000","last_became_peered":"2024-03-30T13:50:08.509701+0000","last_unstale":"2024-03-30T13:54:03.987588+0000","last_undegraded":"2024-03-30T13:54:03.987588+0000","last_fullsized":"2024-03-30T13:50:08.501671+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:22.401087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:22.401087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"238'4834","reported_seq":4909,"reported_epoch":238,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-30T13:54:04.483170+0000","last_change":"2024-03-30T13:53:37.280997+0000","last_active":"2024-03-30T13:54:04.483170+0000","last_peered":"2024-03-30T13:54:04.483170+0000","last_clean":"2024-03-30T13:53:34.134847+0000","last_became_active":"2024-03-30T13:53:37.270473+0000","last_became_peered":"2024-03-30T13:53:37.270473+0000","last_unstale":"2024-03-30T13:54:04.483170+0000","last_undegraded":"2024-03-30T13:53:37.090209+0000","last_fullsized":"2024-03-30T13:53:36.840043+0000","mapping_epoch":229,"log_start":"218'1229","ondisk_log_start":"218'1229","created":211,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3605,"log_dups_size":0,"ondisk_log_size":3605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316801024,"num_objects":4834,"num_object_clones":0,"num_object_copies":14502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3858,"num_objects_misplaced":4834,"num_objects_unfound":0,"num_objects_dirty":4834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4834,"num_write_kb":309376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":3858},{"shards":"3,7","objects":976}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":263,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:03.994144+0000","last_change":"2024-03-30T13:53:36.863779+0000","last_active":"2024-03-30T13:54:03.994144+0000","last_peered":"2024-03-30T13:54:03.994144+0000","last_clean":"2024-03-30T13:54:03.994144+0000","last_became_active":"2024-03-30T13:53:36.862437+0000","last_became_peered":"2024-03-30T13:53:36.862437+0000","last_unstale":"2024-03-30T13:54:03.994144+0000","last_undegraded":"2024-03-30T13:54:03.994144+0000","last_fullsized":"2024-03-30T13:54:03.994144+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:31.860675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:31.860675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,7],"avail_no_missing":["3","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":174,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:03.994849+0000","last_change":"2024-03-30T13:53:36.854424+0000","last_active":"2024-03-30T13:54:03.994849+0000","last_peered":"2024-03-30T13:54:03.994849+0000","last_clean":"2024-03-30T13:54:03.994849+0000","last_became_active":"2024-03-30T13:53:36.850279+0000","last_became_peered":"2024-03-30T13:53:36.850279+0000","last_unstale":"2024-03-30T13:54:03.994849+0000","last_undegraded":"2024-03-30T13:54:03.994849+0000","last_fullsized":"2024-03-30T13:54:03.994849+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:19.432242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:19.432242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":389,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.520944+0000","last_change":"2024-03-30T13:53:35.853307+0000","last_active":"2024-03-30T13:54:07.520944+0000","last_peered":"2024-03-30T13:54:07.520944+0000","last_clean":"2024-03-30T13:54:07.520944+0000","last_became_active":"2024-03-30T13:53:35.852959+0000","last_became_peered":"2024-03-30T13:53:35.852959+0000","last_unstale":"2024-03-30T13:54:07.520944+0000","last_undegraded":"2024-03-30T13:54:07.520944+0000","last_fullsized":"2024-03-30T13:54:07.520944+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:09.173459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:09.173459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"217'44","reported_seq":497,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.521238+0000","last_change":"2024-03-30T13:53:36.006059+0000","last_active":"2024-03-30T13:54:07.521238+0000","last_peered":"2024-03-30T13:54:07.521238+0000","last_clean":"2024-03-30T13:54:07.521238+0000","last_became_active":"2024-03-30T13:53:35.862052+0000","last_became_peered":"2024-03-30T13:53:35.862052+0000","last_unstale":"2024-03-30T13:54:07.521238+0000","last_undegraded":"2024-03-30T13:54:07.521238+0000","last_fullsized":"2024-03-30T13:54:07.521238+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'28","last_scrub_stamp":"2024-03-30T13:52:06.210663+0000","last_deep_scrub":"108'28","last_deep_scrub_stamp":"2024-03-30T13:45:49.140418+0000","last_clean_scrub_stamp":"2024-03-30T13:52:06.210663+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"238'4745","reported_seq":10730,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:04.712268+0000","last_change":"2024-03-30T13:54:04.031799+0000","last_active":"2024-03-30T13:54:04.712268+0000","last_peered":"2024-03-30T13:54:04.712268+0000","last_clean":"2024-03-30T13:54:04.712268+0000","last_became_active":"2024-03-30T13:54:04.026120+0000","last_became_peered":"2024-03-30T13:54:04.026120+0000","last_unstale":"2024-03-30T13:54:04.712268+0000","last_undegraded":"2024-03-30T13:54:04.712268+0000","last_fullsized":"2024-03-30T13:54:04.712268+0000","mapping_epoch":237,"log_start":"218'1320","ondisk_log_start":"218'1320","created":211,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310968320,"num_objects":4745,"num_object_clones":0,"num_object_copies":14235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4745,"num_objects_unfound":0,"num_objects_dirty":4745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4745,"num_write_kb":303680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2957,"num_bytes_recovered":193789952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6,5],"avail_no_missing":["1","5","6"],"object_location_counts":[{"shards":"1,5,6","objects":4745}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":379,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:03.994620+0000","last_change":"2024-03-30T13:53:37.362559+0000","last_active":"2024-03-30T13:54:03.994620+0000","last_peered":"2024-03-30T13:54:03.994620+0000","last_clean":"2024-03-30T13:54:03.994620+0000","last_became_active":"2024-03-30T13:53:37.359258+0000","last_became_peered":"2024-03-30T13:53:37.359258+0000","last_unstale":"2024-03-30T13:54:03.994620+0000","last_undegraded":"2024-03-30T13:54:03.994620+0000","last_fullsized":"2024-03-30T13:54:03.994620+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:12.807605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:44.095338+0000","last_clean_scrub_stamp":"2024-03-30T13:49:12.807605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3,7],"avail_no_missing":["0","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":168,"reported_epoch":240,"state":"active+undersized","last_fresh":"2024-03-30T13:54:07.504201+0000","last_change":"2024-03-30T13:50:08.507875+0000","last_active":"2024-03-30T13:54:07.504201+0000","last_peered":"2024-03-30T13:54:07.504201+0000","last_clean":"2024-03-30T13:50:05.490245+0000","last_became_active":"2024-03-30T13:50:08.507875+0000","last_became_peered":"2024-03-30T13:50:08.507875+0000","last_unstale":"2024-03-30T13:54:07.504201+0000","last_undegraded":"2024-03-30T13:54:07.504201+0000","last_fullsized":"2024-03-30T13:50:08.502700+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:13.399683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:28.225742+0000","last_clean_scrub_stamp":"2024-03-30T13:49:13.399683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":44,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.000909+0000","last_change":"2024-03-30T13:53:13.666944+0000","last_active":"2024-03-30T13:54:04.000909+0000","last_peered":"2024-03-30T13:54:04.000909+0000","last_clean":"2024-03-30T13:54:04.000909+0000","last_became_active":"2024-03-30T13:53:13.666227+0000","last_became_peered":"2024-03-30T13:53:13.666227+0000","last_unstale":"2024-03-30T13:54:04.000909+0000","last_undegraded":"2024-03-30T13:54:04.000909+0000","last_fullsized":"2024-03-30T13:54:04.000909+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:19.099205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:19.099205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"216'15","reported_seq":460,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.995072+0000","last_change":"2024-03-30T13:53:13.672815+0000","last_active":"2024-03-30T13:54:03.995072+0000","last_peered":"2024-03-30T13:54:03.995072+0000","last_clean":"2024-03-30T13:54:03.995072+0000","last_became_active":"2024-03-30T13:53:13.656465+0000","last_became_peered":"2024-03-30T13:53:13.656465+0000","last_unstale":"2024-03-30T13:54:03.995072+0000","last_undegraded":"2024-03-30T13:54:03.995072+0000","last_fullsized":"2024-03-30T13:54:03.995072+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:52:17.023838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:17.023838+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"238'4876","reported_seq":6932,"reported_epoch":238,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-30T13:54:04.569587+0000","last_change":"2024-03-30T13:53:37.457202+0000","last_active":"2024-03-30T13:54:04.569587+0000","last_peered":"2024-03-30T13:54:04.569587+0000","last_clean":"2024-03-30T13:53:12.627470+0000","last_became_active":"2024-03-30T13:53:37.366928+0000","last_became_peered":"2024-03-30T13:53:37.366928+0000","last_unstale":"2024-03-30T13:54:04.569587+0000","last_undegraded":"2024-03-30T13:53:37.269836+0000","last_fullsized":"2024-03-30T13:53:36.839916+0000","mapping_epoch":229,"log_start":"217'1000","ondisk_log_start":"217'1000","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3876,"log_dups_size":0,"ondisk_log_size":3876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319553536,"num_objects":4876,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6003,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4876,"num_write_kb":312064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":992,"num_bytes_recovered":65011712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,3","objects":984},{"shards":"0,3,5,7","objects":992},{"shards":"0,3,7","objects":789},{"shards":"0,5,7","objects":2111}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":349,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:03.987580+0000","last_change":"2024-03-30T13:53:16.691600+0000","last_active":"2024-03-30T13:54:03.987580+0000","last_peered":"2024-03-30T13:54:03.987580+0000","last_clean":"2024-03-30T13:54:03.987580+0000","last_became_active":"2024-03-30T13:53:16.690961+0000","last_became_peered":"2024-03-30T13:53:16.690961+0000","last_unstale":"2024-03-30T13:54:03.987580+0000","last_undegraded":"2024-03-30T13:54:03.987580+0000","last_fullsized":"2024-03-30T13:54:03.987580+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:48.894763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:48.894763+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:05.154403+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4,5],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":387,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.001324+0000","last_change":"2024-03-30T13:53:35.855546+0000","last_active":"2024-03-30T13:54:04.001324+0000","last_peered":"2024-03-30T13:54:04.001324+0000","last_clean":"2024-03-30T13:54:04.001324+0000","last_became_active":"2024-03-30T13:53:35.855186+0000","last_became_peered":"2024-03-30T13:53:35.855186+0000","last_unstale":"2024-03-30T13:54:04.001324+0000","last_undegraded":"2024-03-30T13:54:04.001324+0000","last_fullsized":"2024-03-30T13:54:04.001324+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:08.043774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:08.043774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":71,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.024241+0000","last_change":"2024-03-30T13:53:13.683544+0000","last_active":"2024-03-30T13:54:04.024241+0000","last_peered":"2024-03-30T13:54:04.024241+0000","last_clean":"2024-03-30T13:54:04.024241+0000","last_became_active":"2024-03-30T13:53:13.655879+0000","last_became_peered":"2024-03-30T13:53:13.655879+0000","last_unstale":"2024-03-30T13:54:04.024241+0000","last_undegraded":"2024-03-30T13:54:04.024241+0000","last_fullsized":"2024-03-30T13:54:04.024241+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-30T13:52:51.932750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:52:51.932750+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:09.628273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":398,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.503164+0000","last_change":"2024-03-30T13:51:01.240545+0000","last_active":"2024-03-30T13:54:07.503164+0000","last_peered":"2024-03-30T13:54:07.503164+0000","last_clean":"2024-03-30T13:54:07.503164+0000","last_became_active":"2024-03-30T13:42:45.310209+0000","last_became_peered":"2024-03-30T13:42:45.310209+0000","last_unstale":"2024-03-30T13:54:07.503164+0000","last_undegraded":"2024-03-30T13:54:07.503164+0000","last_fullsized":"2024-03-30T13:54:07.503164+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:01.240468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:01.240468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"217'41","reported_seq":587,"reported_epoch":238,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:54:04.022590+0000","last_change":"2024-03-30T13:50:08.508727+0000","last_active":"2024-03-30T13:54:04.022590+0000","last_peered":"2024-03-30T13:54:04.022590+0000","last_clean":"2024-03-30T13:50:05.490389+0000","last_became_active":"2024-03-30T13:50:08.508727+0000","last_became_peered":"2024-03-30T13:50:08.508727+0000","last_unstale":"2024-03-30T13:54:04.022590+0000","last_undegraded":"2024-03-30T13:50:08.506113+0000","last_fullsized":"2024-03-30T13:50:08.505993+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-03-30T13:48:27.377242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:27.377242+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":399,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.996348+0000","last_change":"2024-03-30T13:52:23.006053+0000","last_active":"2024-03-30T13:54:03.996348+0000","last_peered":"2024-03-30T13:54:03.996348+0000","last_clean":"2024-03-30T13:54:03.996348+0000","last_became_active":"2024-03-30T13:42:49.322154+0000","last_became_peered":"2024-03-30T13:42:49.322154+0000","last_unstale":"2024-03-30T13:54:03.996348+0000","last_undegraded":"2024-03-30T13:54:03.996348+0000","last_fullsized":"2024-03-30T13:54:03.996348+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:23.005788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:18.862784+0000","last_clean_scrub_stamp":"2024-03-30T13:52:23.005788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"238'4792","reported_seq":4996,"reported_epoch":238,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-30T13:54:04.807333+0000","last_change":"2024-03-30T13:54:03.000107+0000","last_active":"2024-03-30T13:54:04.807333+0000","last_peered":"2024-03-30T13:54:04.807333+0000","last_clean":"2024-03-30T13:53:12.641871+0000","last_became_active":"2024-03-30T13:53:37.228167+0000","last_became_peered":"2024-03-30T13:53:37.228167+0000","last_unstale":"2024-03-30T13:54:04.807333+0000","last_undegraded":"2024-03-30T13:54:04.807333+0000","last_fullsized":"2024-03-30T13:53:36.850042+0000","mapping_epoch":229,"log_start":"218'1361","ondisk_log_start":"218'1361","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3431,"log_dups_size":0,"ondisk_log_size":3431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314048512,"num_objects":4792,"num_object_clones":0,"num_object_copies":14376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6738,"num_objects_unfound":0,"num_objects_dirty":4792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4792,"num_write_kb":306688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":2883584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,3","objects":1041},{"shards":"1,3,5,7","objects":1},{"shards":"1,3,7","objects":763},{"shards":"1,5,7","objects":2987}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":324,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:03.993664+0000","last_change":"2024-03-30T13:53:36.851601+0000","last_active":"2024-03-30T13:54:03.993664+0000","last_peered":"2024-03-30T13:54:03.993664+0000","last_clean":"2024-03-30T13:54:03.993664+0000","last_became_active":"2024-03-30T13:53:36.851158+0000","last_became_peered":"2024-03-30T13:53:36.851158+0000","last_unstale":"2024-03-30T13:54:03.993664+0000","last_undegraded":"2024-03-30T13:54:03.993664+0000","last_fullsized":"2024-03-30T13:54:03.993664+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:46.926407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:46.926407+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:04.128927+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4,7],"avail_no_missing":["0","4","7"],"object_location_counts":[{"shards":"0,4,7","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":33,"reported_epoch":240,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:07.521070+0000","last_change":"2024-03-30T13:53:36.861043+0000","last_active":"2024-03-30T13:54:07.521070+0000","last_peered":"2024-03-30T13:54:07.521070+0000","last_clean":"2024-03-30T13:54:07.521070+0000","last_became_active":"2024-03-30T13:53:36.860597+0000","last_became_peered":"2024-03-30T13:53:36.860597+0000","last_unstale":"2024-03-30T13:54:07.521070+0000","last_undegraded":"2024-03-30T13:54:07.521070+0000","last_fullsized":"2024-03-30T13:54:07.521070+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:14.275893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:14.275893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3,7],"avail_no_missing":["6","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":396,"reported_epoch":238,"state":"active+undersized","last_fresh":"2024-03-30T13:54:04.021524+0000","last_change":"2024-03-30T13:50:08.512561+0000","last_active":"2024-03-30T13:54:04.021524+0000","last_peered":"2024-03-30T13:54:04.021524+0000","last_clean":"2024-03-30T13:50:05.489457+0000","last_became_active":"2024-03-30T13:50:08.512561+0000","last_became_peered":"2024-03-30T13:50:08.512561+0000","last_unstale":"2024-03-30T13:54:04.021524+0000","last_undegraded":"2024-03-30T13:54:04.021524+0000","last_fullsized":"2024-03-30T13:50:08.506886+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:16.464234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:59.810189+0000","last_clean_scrub_stamp":"2024-03-30T13:49:16.464234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":164,"reported_epoch":238,"state":"active+undersized","last_fresh":"2024-03-30T13:54:03.999638+0000","last_change":"2024-03-30T13:50:08.511337+0000","last_active":"2024-03-30T13:54:03.999638+0000","last_peered":"2024-03-30T13:54:03.999638+0000","last_clean":"2024-03-30T13:50:05.498892+0000","last_became_active":"2024-03-30T13:50:08.511337+0000","last_became_peered":"2024-03-30T13:50:08.511337+0000","last_unstale":"2024-03-30T13:54:03.999638+0000","last_undegraded":"2024-03-30T13:54:03.999638+0000","last_fullsized":"2024-03-30T13:50:08.501456+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:25.398232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:51.919203+0000","last_clean_scrub_stamp":"2024-03-30T13:49:25.398232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"216'17","reported_seq":474,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.023560+0000","last_change":"2024-03-30T13:53:36.011804+0000","last_active":"2024-03-30T13:54:04.023560+0000","last_peered":"2024-03-30T13:54:04.023560+0000","last_clean":"2024-03-30T13:54:04.023560+0000","last_became_active":"2024-03-30T13:53:35.867159+0000","last_became_peered":"2024-03-30T13:53:35.867159+0000","last_unstale":"2024-03-30T13:54:04.023560+0000","last_undegraded":"2024-03-30T13:54:04.023560+0000","last_fullsized":"2024-03-30T13:54:04.023560+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'13","last_scrub_stamp":"2024-03-30T13:52:25.542111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:25.542111+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":53,"reported_epoch":240,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:07.502894+0000","last_change":"2024-03-30T13:53:36.864869+0000","last_active":"2024-03-30T13:54:07.502894+0000","last_peered":"2024-03-30T13:54:07.502894+0000","last_clean":"2024-03-30T13:54:07.502894+0000","last_became_active":"2024-03-30T13:53:36.864365+0000","last_became_peered":"2024-03-30T13:53:36.864365+0000","last_unstale":"2024-03-30T13:54:07.502894+0000","last_undegraded":"2024-03-30T13:54:07.502894+0000","last_fullsized":"2024-03-30T13:54:07.502894+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:28.373277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:28.373277+0000","last_clean_scrub_stamp":"2024-03-30T13:49:28.373277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,7],"avail_no_missing":["4","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.8","version":"238'4821","reported_seq":8366,"reported_epoch":238,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-30T13:54:04.525513+0000","last_change":"2024-03-30T13:53:36.086138+0000","last_active":"2024-03-30T13:54:04.525513+0000","last_peered":"2024-03-30T13:54:04.525513+0000","last_clean":"2024-03-30T13:53:12.628594+0000","last_became_active":"2024-03-30T13:53:36.071150+0000","last_became_peered":"2024-03-30T13:53:36.071150+0000","last_unstale":"2024-03-30T13:54:04.525513+0000","last_undegraded":"2024-03-30T13:54:04.525513+0000","last_fullsized":"2024-03-30T13:53:35.835566+0000","mapping_epoch":228,"log_start":"218'1227","ondisk_log_start":"218'1227","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3594,"log_dups_size":0,"ondisk_log_size":3594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315949056,"num_objects":4821,"num_object_clones":0,"num_object_copies":14463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3177,"num_objects_unfound":0,"num_objects_dirty":4821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4821,"num_write_kb":308544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1737,"num_bytes_recovered":113836032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0],"avail_no_missing":["3","0","5"],"object_location_counts":[{"shards":"0,3","objects":1644},{"shards":"0,3,5","objects":2009},{"shards":"0,3,5,7","objects":380},{"shards":"0,3,7","objects":788}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":45,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.003186+0000","last_change":"2024-03-30T13:53:13.660514+0000","last_active":"2024-03-30T13:54:04.003186+0000","last_peered":"2024-03-30T13:54:04.003186+0000","last_clean":"2024-03-30T13:54:04.003186+0000","last_became_active":"2024-03-30T13:53:13.660154+0000","last_became_peered":"2024-03-30T13:53:13.660154+0000","last_unstale":"2024-03-30T13:54:04.003186+0000","last_undegraded":"2024-03-30T13:54:04.003186+0000","last_fullsized":"2024-03-30T13:54:04.003186+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:28.079029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:12.433166+0000","last_clean_scrub_stamp":"2024-03-30T13:52:28.079029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":328,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.987476+0000","last_change":"2024-03-30T13:53:38.545959+0000","last_active":"2024-03-30T13:54:03.987476+0000","last_peered":"2024-03-30T13:54:03.987476+0000","last_clean":"2024-03-30T13:54:03.987476+0000","last_became_active":"2024-03-30T13:53:38.545543+0000","last_became_peered":"2024-03-30T13:53:38.545543+0000","last_unstale":"2024-03-30T13:54:03.987476+0000","last_undegraded":"2024-03-30T13:54:03.987476+0000","last_fullsized":"2024-03-30T13:54:03.987476+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:47.943814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:47.943814+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:05.154427+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":399,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.994096+0000","last_change":"2024-03-30T13:53:13.662546+0000","last_active":"2024-03-30T13:54:03.994096+0000","last_peered":"2024-03-30T13:54:03.994096+0000","last_clean":"2024-03-30T13:54:03.994096+0000","last_became_active":"2024-03-30T13:53:13.662164+0000","last_became_peered":"2024-03-30T13:53:13.662164+0000","last_unstale":"2024-03-30T13:54:03.994096+0000","last_undegraded":"2024-03-30T13:54:03.994096+0000","last_fullsized":"2024-03-30T13:54:03.994096+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:12.975288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:12.975288+0000","last_clean_scrub_stamp":"2024-03-30T13:52:12.975288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"229'29","reported_seq":608,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.520938+0000","last_change":"2024-03-30T13:53:36.017011+0000","last_active":"2024-03-30T13:54:07.520938+0000","last_peered":"2024-03-30T13:54:07.520938+0000","last_clean":"2024-03-30T13:54:07.520938+0000","last_became_active":"2024-03-30T13:53:35.864416+0000","last_became_peered":"2024-03-30T13:53:35.864416+0000","last_unstale":"2024-03-30T13:54:07.520938+0000","last_undegraded":"2024-03-30T13:54:07.520938+0000","last_fullsized":"2024-03-30T13:54:07.520938+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'16","last_scrub_stamp":"2024-03-30T13:52:01.369487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:01.369487+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":379,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:03.995960+0000","last_change":"2024-03-30T13:53:36.855679+0000","last_active":"2024-03-30T13:54:03.995960+0000","last_peered":"2024-03-30T13:54:03.995960+0000","last_clean":"2024-03-30T13:54:03.995960+0000","last_became_active":"2024-03-30T13:53:36.853892+0000","last_became_peered":"2024-03-30T13:53:36.853892+0000","last_unstale":"2024-03-30T13:54:03.995960+0000","last_undegraded":"2024-03-30T13:54:03.995960+0000","last_fullsized":"2024-03-30T13:54:03.995960+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:25.787524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:25.787524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"238'4907","reported_seq":4976,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:04.562427+0000","last_change":"2024-03-30T13:53:14.659142+0000","last_active":"2024-03-30T13:54:04.562427+0000","last_peered":"2024-03-30T13:54:04.562427+0000","last_clean":"2024-03-30T13:54:04.562427+0000","last_became_active":"2024-03-30T13:53:14.651477+0000","last_became_peered":"2024-03-30T13:53:14.651477+0000","last_unstale":"2024-03-30T13:54:04.562427+0000","last_undegraded":"2024-03-30T13:54:04.562427+0000","last_fullsized":"2024-03-30T13:54:04.562427+0000","mapping_epoch":222,"log_start":"217'1056","ondisk_log_start":"217'1056","created":211,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3851,"log_dups_size":0,"ondisk_log_size":3851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:06.562226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321585152,"num_objects":4907,"num_object_clones":0,"num_object_copies":14721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4907,"num_objects_unfound":0,"num_objects_dirty":4907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4907,"num_write_kb":314048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,5],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":4907}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":379,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.019108+0000","last_change":"2024-03-30T13:52:47.632199+0000","last_active":"2024-03-30T13:54:04.019108+0000","last_peered":"2024-03-30T13:54:04.019108+0000","last_clean":"2024-03-30T13:54:04.019108+0000","last_became_active":"2024-03-30T13:52:36.489806+0000","last_became_peered":"2024-03-30T13:52:36.489806+0000","last_unstale":"2024-03-30T13:54:04.019108+0000","last_undegraded":"2024-03-30T13:54:04.019108+0000","last_fullsized":"2024-03-30T13:54:04.019108+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:47.632117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:47.632117+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:17.252927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":395,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.021568+0000","last_change":"2024-03-30T13:53:13.664555+0000","last_active":"2024-03-30T13:54:04.021568+0000","last_peered":"2024-03-30T13:54:04.021568+0000","last_clean":"2024-03-30T13:54:04.021568+0000","last_became_active":"2024-03-30T13:53:13.664207+0000","last_became_peered":"2024-03-30T13:53:13.664207+0000","last_unstale":"2024-03-30T13:54:04.021568+0000","last_undegraded":"2024-03-30T13:54:04.021568+0000","last_fullsized":"2024-03-30T13:54:04.021568+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:03.345965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:03.345965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":399,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.994280+0000","last_change":"2024-03-30T13:52:08.002816+0000","last_active":"2024-03-30T13:54:03.994280+0000","last_peered":"2024-03-30T13:54:03.994280+0000","last_clean":"2024-03-30T13:54:03.994280+0000","last_became_active":"2024-03-30T13:42:45.307905+0000","last_became_peered":"2024-03-30T13:42:45.307905+0000","last_unstale":"2024-03-30T13:54:03.994280+0000","last_undegraded":"2024-03-30T13:54:03.994280+0000","last_fullsized":"2024-03-30T13:54:03.994280+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:08.002706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:08.002706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":108,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.021412+0000","last_change":"2024-03-30T13:53:36.056309+0000","last_active":"2024-03-30T13:54:04.021412+0000","last_peered":"2024-03-30T13:54:04.021412+0000","last_clean":"2024-03-30T13:54:04.021412+0000","last_became_active":"2024-03-30T13:53:35.889071+0000","last_became_peered":"2024-03-30T13:53:35.889071+0000","last_unstale":"2024-03-30T13:54:04.021412+0000","last_undegraded":"2024-03-30T13:54:04.021412+0000","last_fullsized":"2024-03-30T13:54:04.021412+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-30T13:52:18.785378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:52:18.785378+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00126391,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":30,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"216'30","reported_seq":432,"reported_epoch":238,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:54:03.997940+0000","last_change":"2024-03-30T13:50:08.510256+0000","last_active":"2024-03-30T13:54:03.997940+0000","last_peered":"2024-03-30T13:54:03.997940+0000","last_clean":"2024-03-30T13:50:05.503358+0000","last_became_active":"2024-03-30T13:50:08.510256+0000","last_became_peered":"2024-03-30T13:50:08.510256+0000","last_unstale":"2024-03-30T13:54:03.997940+0000","last_undegraded":"2024-03-30T13:50:08.504536+0000","last_fullsized":"2024-03-30T13:50:08.504398+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'18","last_scrub_stamp":"2024-03-30T13:49:13.774982+0000","last_deep_scrub":"93'18","last_deep_scrub_stamp":"2024-03-30T13:47:28.872529+0000","last_clean_scrub_stamp":"2024-03-30T13:49:13.774982+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":44,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.023724+0000","last_change":"2024-03-30T13:53:13.649717+0000","last_active":"2024-03-30T13:54:04.023724+0000","last_peered":"2024-03-30T13:54:04.023724+0000","last_clean":"2024-03-30T13:54:04.023724+0000","last_became_active":"2024-03-30T13:53:13.649288+0000","last_became_peered":"2024-03-30T13:53:13.649288+0000","last_unstale":"2024-03-30T13:54:04.023724+0000","last_undegraded":"2024-03-30T13:54:04.023724+0000","last_fullsized":"2024-03-30T13:54:04.023724+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:08.977897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:08.977897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"240'4933","reported_seq":9189,"reported_epoch":240,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-30T13:54:08.563516+0000","last_change":"2024-03-30T13:53:37.492272+0000","last_active":"2024-03-30T13:54:08.563516+0000","last_peered":"2024-03-30T13:54:08.563516+0000","last_clean":"2024-03-30T13:53:34.158152+0000","last_became_active":"2024-03-30T13:53:37.431480+0000","last_became_peered":"2024-03-30T13:53:37.431480+0000","last_unstale":"2024-03-30T13:54:08.563516+0000","last_undegraded":"2024-03-30T13:54:08.563516+0000","last_fullsized":"2024-03-30T13:53:36.842297+0000","mapping_epoch":229,"log_start":"217'1100","ondisk_log_start":"217'1100","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3833,"log_dups_size":0,"ondisk_log_size":3833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323289088,"num_objects":4933,"num_object_clones":0,"num_object_copies":14799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1746,"num_objects_unfound":0,"num_objects_dirty":4933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4933,"num_write_kb":315712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2092,"num_bytes_recovered":137101312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1],"avail_no_missing":["4","1","7"],"object_location_counts":[{"shards":"1,4","objects":3187},{"shards":"1,4,7","objects":1746}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":388,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.520856+0000","last_change":"2024-03-30T13:52:47.924323+0000","last_active":"2024-03-30T13:54:07.520856+0000","last_peered":"2024-03-30T13:54:07.520856+0000","last_clean":"2024-03-30T13:54:07.520856+0000","last_became_active":"2024-03-30T13:52:36.483298+0000","last_became_peered":"2024-03-30T13:52:36.483298+0000","last_unstale":"2024-03-30T13:54:07.520856+0000","last_undegraded":"2024-03-30T13:54:07.520856+0000","last_fullsized":"2024-03-30T13:54:07.520856+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:47.924212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:47.924212+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:12.215142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":384,"reported_epoch":238,"state":"active+undersized","last_fresh":"2024-03-30T13:54:04.018275+0000","last_change":"2024-03-30T13:50:08.512648+0000","last_active":"2024-03-30T13:54:04.018275+0000","last_peered":"2024-03-30T13:54:04.018275+0000","last_clean":"2024-03-30T13:50:05.489180+0000","last_became_active":"2024-03-30T13:50:08.512648+0000","last_became_peered":"2024-03-30T13:50:08.512648+0000","last_unstale":"2024-03-30T13:54:04.018275+0000","last_undegraded":"2024-03-30T13:54:04.018275+0000","last_fullsized":"2024-03-30T13:50:08.505931+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:21.527712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:21.527712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":401,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:04.022375+0000","last_change":"2024-03-30T13:53:14.658635+0000","last_active":"2024-03-30T13:54:04.022375+0000","last_peered":"2024-03-30T13:54:04.022375+0000","last_clean":"2024-03-30T13:54:04.022375+0000","last_became_active":"2024-03-30T13:53:14.658165+0000","last_became_peered":"2024-03-30T13:53:14.658165+0000","last_unstale":"2024-03-30T13:54:04.022375+0000","last_undegraded":"2024-03-30T13:54:04.022375+0000","last_fullsized":"2024-03-30T13:54:04.022375+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:19.525007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:45.975163+0000","last_clean_scrub_stamp":"2024-03-30T13:49:19.525007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0,5],"avail_no_missing":["1","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"216'29","reported_seq":475,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:03.995307+0000","last_change":"2024-03-30T13:53:36.858207+0000","last_active":"2024-03-30T13:54:03.995307+0000","last_peered":"2024-03-30T13:54:03.995307+0000","last_clean":"2024-03-30T13:54:03.995307+0000","last_became_active":"2024-03-30T13:53:36.857573+0000","last_became_peered":"2024-03-30T13:53:36.857573+0000","last_unstale":"2024-03-30T13:54:03.995307+0000","last_undegraded":"2024-03-30T13:54:03.995307+0000","last_fullsized":"2024-03-30T13:54:03.995307+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-30T13:52:15.020534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:15.020534+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[{"shards":"0,1,7","objects":8}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":416,"reported_epoch":240,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:07.521267+0000","last_change":"2024-03-30T13:53:14.654792+0000","last_active":"2024-03-30T13:54:07.521267+0000","last_peered":"2024-03-30T13:54:07.521267+0000","last_clean":"2024-03-30T13:54:07.521267+0000","last_became_active":"2024-03-30T13:53:14.654346+0000","last_became_peered":"2024-03-30T13:53:14.654346+0000","last_unstale":"2024-03-30T13:54:07.521267+0000","last_undegraded":"2024-03-30T13:54:07.521267+0000","last_fullsized":"2024-03-30T13:54:07.521267+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:52.887014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:52.887014+0000","last_clean_scrub_stamp":"2024-03-30T13:52:52.887014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:07.858502+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0,5],"avail_no_missing":["6","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"238'4736","reported_seq":4805,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:04.888413+0000","last_change":"2024-03-30T13:53:14.666961+0000","last_active":"2024-03-30T13:54:04.888413+0000","last_peered":"2024-03-30T13:54:04.888413+0000","last_clean":"2024-03-30T13:54:04.888413+0000","last_became_active":"2024-03-30T13:53:14.661730+0000","last_became_peered":"2024-03-30T13:53:14.661730+0000","last_unstale":"2024-03-30T13:54:04.888413+0000","last_undegraded":"2024-03-30T13:54:04.888413+0000","last_fullsized":"2024-03-30T13:54:04.888413+0000","mapping_epoch":222,"log_start":"218'1356","ondisk_log_start":"218'1356","created":211,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3380,"log_dups_size":0,"ondisk_log_size":3380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:12.128234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":4736,"num_object_clones":0,"num_object_copies":14208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4736,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4736,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3,5],"avail_no_missing":["1","3","5"],"object_location_counts":[{"shards":"1,3,5","objects":4736}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":393,"reported_epoch":240,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:07.520797+0000","last_change":"2024-03-30T13:53:37.108143+0000","last_active":"2024-03-30T13:54:07.520797+0000","last_peered":"2024-03-30T13:54:07.520797+0000","last_clean":"2024-03-30T13:54:07.520797+0000","last_became_active":"2024-03-30T13:53:37.107659+0000","last_became_peered":"2024-03-30T13:53:37.107659+0000","last_unstale":"2024-03-30T13:54:07.520797+0000","last_undegraded":"2024-03-30T13:54:07.520797+0000","last_fullsized":"2024-03-30T13:54:07.520797+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:48.949941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:48.949941+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3,7],"avail_no_missing":["6","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":382,"reported_epoch":238,"state":"active+undersized","last_fresh":"2024-03-30T13:54:03.987341+0000","last_change":"2024-03-30T13:50:08.510025+0000","last_active":"2024-03-30T13:54:03.987341+0000","last_peered":"2024-03-30T13:54:03.987341+0000","last_clean":"2024-03-30T13:50:05.498930+0000","last_became_active":"2024-03-30T13:50:08.510025+0000","last_became_peered":"2024-03-30T13:50:08.510025+0000","last_unstale":"2024-03-30T13:54:03.987341+0000","last_undegraded":"2024-03-30T13:54:03.987341+0000","last_fullsized":"2024-03-30T13:50:08.502478+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:28.946303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:41.142941+0000","last_clean_scrub_stamp":"2024-03-30T13:49:28.946303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":290,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:04.000021+0000","last_change":"2024-03-30T13:53:14.651231+0000","last_active":"2024-03-30T13:54:04.000021+0000","last_peered":"2024-03-30T13:54:04.000021+0000","last_clean":"2024-03-30T13:54:04.000021+0000","last_became_active":"2024-03-30T13:53:14.650772+0000","last_became_peered":"2024-03-30T13:53:14.650772+0000","last_unstale":"2024-03-30T13:54:04.000021+0000","last_undegraded":"2024-03-30T13:54:04.000021+0000","last_fullsized":"2024-03-30T13:54:04.000021+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:08.543531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:55.458779+0000","last_clean_scrub_stamp":"2024-03-30T13:49:08.543531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,5],"avail_no_missing":["3","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"217'25","reported_seq":456,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.024699+0000","last_change":"2024-03-30T13:53:13.688776+0000","last_active":"2024-03-30T13:54:04.024699+0000","last_peered":"2024-03-30T13:54:04.024699+0000","last_clean":"2024-03-30T13:54:04.024699+0000","last_became_active":"2024-03-30T13:53:13.657632+0000","last_became_peered":"2024-03-30T13:53:13.657632+0000","last_unstale":"2024-03-30T13:54:04.024699+0000","last_undegraded":"2024-03-30T13:54:04.024699+0000","last_fullsized":"2024-03-30T13:54:04.024699+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'15","last_scrub_stamp":"2024-03-30T13:52:15.457275+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-03-30T13:45:50.945837+0000","last_clean_scrub_stamp":"2024-03-30T13:52:15.457275+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"222'10","reported_seq":555,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.995317+0000","last_change":"2024-03-30T13:53:13.669795+0000","last_active":"2024-03-30T13:54:03.995317+0000","last_peered":"2024-03-30T13:54:03.995317+0000","last_clean":"2024-03-30T13:54:03.995317+0000","last_became_active":"2024-03-30T13:53:13.658557+0000","last_became_peered":"2024-03-30T13:53:13.658557+0000","last_unstale":"2024-03-30T13:54:03.995317+0000","last_undegraded":"2024-03-30T13:54:03.995317+0000","last_fullsized":"2024-03-30T13:54:03.995317+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'9","last_scrub_stamp":"2024-03-30T13:52:57.873554+0000","last_deep_scrub":"218'9","last_deep_scrub_stamp":"2024-03-30T13:52:57.873554+0000","last_clean_scrub_stamp":"2024-03-30T13:52:57.873554+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"238'4706","reported_seq":4773,"reported_epoch":238,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-30T13:54:04.545975+0000","last_change":"2024-03-30T13:53:37.391308+0000","last_active":"2024-03-30T13:54:04.545975+0000","last_peered":"2024-03-30T13:54:04.545975+0000","last_clean":"2024-03-30T13:53:34.242680+0000","last_became_active":"2024-03-30T13:53:37.374630+0000","last_became_peered":"2024-03-30T13:53:37.374630+0000","last_unstale":"2024-03-30T13:54:04.545975+0000","last_undegraded":"2024-03-30T13:53:37.099819+0000","last_fullsized":"2024-03-30T13:53:36.841150+0000","mapping_epoch":229,"log_start":"218'1138","ondisk_log_start":"218'1138","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3568,"log_dups_size":0,"ondisk_log_size":3568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308412416,"num_objects":4706,"num_object_clones":0,"num_object_copies":14118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3767,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4706,"num_write_kb":301184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4","objects":939},{"shards":"3,4,7","objects":3767}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":405,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.000668+0000","last_change":"2024-03-30T13:53:38.553854+0000","last_active":"2024-03-30T13:54:04.000668+0000","last_peered":"2024-03-30T13:54:04.000668+0000","last_clean":"2024-03-30T13:54:04.000668+0000","last_became_active":"2024-03-30T13:53:38.553065+0000","last_became_peered":"2024-03-30T13:53:38.553065+0000","last_unstale":"2024-03-30T13:54:04.000668+0000","last_undegraded":"2024-03-30T13:54:04.000668+0000","last_fullsized":"2024-03-30T13:54:04.000668+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:48.866100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:48.866100+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:16.865938+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":259,"reported_epoch":240,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:07.520810+0000","last_change":"2024-03-30T13:53:36.853679+0000","last_active":"2024-03-30T13:54:07.520810+0000","last_peered":"2024-03-30T13:54:07.520810+0000","last_clean":"2024-03-30T13:54:07.520810+0000","last_became_active":"2024-03-30T13:53:36.852004+0000","last_became_peered":"2024-03-30T13:53:36.852004+0000","last_unstale":"2024-03-30T13:54:07.520810+0000","last_undegraded":"2024-03-30T13:54:07.520810+0000","last_fullsized":"2024-03-30T13:54:07.520810+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:21.782290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:31.900908+0000","last_clean_scrub_stamp":"2024-03-30T13:49:21.782290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1,7],"avail_no_missing":["6","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":408,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.997945+0000","last_change":"2024-03-30T13:52:46.959816+0000","last_active":"2024-03-30T13:54:03.997945+0000","last_peered":"2024-03-30T13:54:03.997945+0000","last_clean":"2024-03-30T13:54:03.997945+0000","last_became_active":"2024-03-30T13:42:45.309886+0000","last_became_peered":"2024-03-30T13:42:45.309886+0000","last_unstale":"2024-03-30T13:54:03.997945+0000","last_undegraded":"2024-03-30T13:54:03.997945+0000","last_fullsized":"2024-03-30T13:54:03.997945+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:52:46.959694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:46.959694+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"217'35","reported_seq":481,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.502895+0000","last_change":"2024-03-30T13:53:13.687444+0000","last_active":"2024-03-30T13:54:07.502895+0000","last_peered":"2024-03-30T13:54:07.502895+0000","last_clean":"2024-03-30T13:54:07.502895+0000","last_became_active":"2024-03-30T13:53:13.652175+0000","last_became_peered":"2024-03-30T13:53:13.652175+0000","last_unstale":"2024-03-30T13:54:07.502895+0000","last_undegraded":"2024-03-30T13:54:07.502895+0000","last_fullsized":"2024-03-30T13:54:07.502895+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'21","last_scrub_stamp":"2024-03-30T13:52:44.925926+0000","last_deep_scrub":"91'21","last_deep_scrub_stamp":"2024-03-30T13:44:05.917064+0000","last_clean_scrub_stamp":"2024-03-30T13:52:44.925926+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":529,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.998718+0000","last_change":"2024-03-30T13:52:17.989527+0000","last_active":"2024-03-30T13:54:03.998718+0000","last_peered":"2024-03-30T13:54:03.998718+0000","last_clean":"2024-03-30T13:54:03.998718+0000","last_became_active":"2024-03-30T13:42:49.321423+0000","last_became_peered":"2024-03-30T13:42:49.321423+0000","last_unstale":"2024-03-30T13:54:03.998718+0000","last_undegraded":"2024-03-30T13:54:03.998718+0000","last_fullsized":"2024-03-30T13:54:03.998718+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:52:17.989349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:17.989349+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"238'4820","reported_seq":12609,"reported_epoch":238,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-30T13:54:04.900288+0000","last_change":"2024-03-30T13:53:37.314806+0000","last_active":"2024-03-30T13:54:04.900288+0000","last_peered":"2024-03-30T13:54:04.900288+0000","last_clean":"2024-03-30T13:53:12.623816+0000","last_became_active":"2024-03-30T13:53:37.058168+0000","last_became_peered":"2024-03-30T13:53:37.058168+0000","last_unstale":"2024-03-30T13:54:04.900288+0000","last_undegraded":"2024-03-30T13:54:04.900288+0000","last_fullsized":"2024-03-30T13:53:36.844928+0000","mapping_epoch":229,"log_start":"218'1401","ondisk_log_start":"218'1401","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3419,"log_dups_size":0,"ondisk_log_size":3419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315883520,"num_objects":4820,"num_object_clones":0,"num_object_copies":14460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3009,"num_objects_unfound":0,"num_objects_dirty":4820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4820,"num_write_kb":308480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3859,"num_bytes_recovered":252903424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4],"avail_no_missing":["1","4","5","7"],"object_location_counts":[{"shards":"1,4","objects":1811},{"shards":"1,4,5,7","objects":2209},{"shards":"1,4,7","objects":800}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":383,"reported_epoch":240,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:07.503275+0000","last_change":"2024-03-30T13:52:47.990769+0000","last_active":"2024-03-30T13:54:07.503275+0000","last_peered":"2024-03-30T13:54:07.503275+0000","last_clean":"2024-03-30T13:54:07.503275+0000","last_became_active":"2024-03-30T13:52:36.490909+0000","last_became_peered":"2024-03-30T13:52:36.490909+0000","last_unstale":"2024-03-30T13:54:07.503275+0000","last_undegraded":"2024-03-30T13:54:07.503275+0000","last_fullsized":"2024-03-30T13:54:07.503275+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:47.990635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:47.990635+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:17.956945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,0],"avail_no_missing":["4","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":381,"reported_epoch":240,"state":"active+undersized","last_fresh":"2024-03-30T13:54:07.503829+0000","last_change":"2024-03-30T13:50:08.504411+0000","last_active":"2024-03-30T13:54:07.503829+0000","last_peered":"2024-03-30T13:54:07.503829+0000","last_clean":"2024-03-30T13:50:05.489944+0000","last_became_active":"2024-03-30T13:50:08.504411+0000","last_became_peered":"2024-03-30T13:50:08.504411+0000","last_unstale":"2024-03-30T13:54:07.503829+0000","last_undegraded":"2024-03-30T13:54:07.503829+0000","last_fullsized":"2024-03-30T13:50:08.501823+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:10.216387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:10.216387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":407,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.002715+0000","last_change":"2024-03-30T13:53:35.851353+0000","last_active":"2024-03-30T13:54:04.002715+0000","last_peered":"2024-03-30T13:54:04.002715+0000","last_clean":"2024-03-30T13:54:04.002715+0000","last_became_active":"2024-03-30T13:53:35.851006+0000","last_became_peered":"2024-03-30T13:53:35.851006+0000","last_unstale":"2024-03-30T13:54:04.002715+0000","last_undegraded":"2024-03-30T13:54:04.002715+0000","last_fullsized":"2024-03-30T13:54:04.002715+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:53:01.031216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:53:01.031216+0000","last_clean_scrub_stamp":"2024-03-30T13:53:01.031216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:17.312078+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"216'35","reported_seq":476,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.998589+0000","last_change":"2024-03-30T13:53:36.081652+0000","last_active":"2024-03-30T13:54:03.998589+0000","last_peered":"2024-03-30T13:54:03.998589+0000","last_clean":"2024-03-30T13:54:03.998589+0000","last_became_active":"2024-03-30T13:53:35.888652+0000","last_became_peered":"2024-03-30T13:53:35.888652+0000","last_unstale":"2024-03-30T13:54:03.998589+0000","last_undegraded":"2024-03-30T13:54:03.998589+0000","last_fullsized":"2024-03-30T13:54:03.998589+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'29","last_scrub_stamp":"2024-03-30T13:52:52.973602+0000","last_deep_scrub":"216'29","last_deep_scrub_stamp":"2024-03-30T13:52:52.973602+0000","last_clean_scrub_stamp":"2024-03-30T13:52:52.973602+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:13.561845+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":396,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.502558+0000","last_change":"2024-03-30T13:53:35.863472+0000","last_active":"2024-03-30T13:54:07.502558+0000","last_peered":"2024-03-30T13:54:07.502558+0000","last_clean":"2024-03-30T13:54:07.502558+0000","last_became_active":"2024-03-30T13:53:35.863124+0000","last_became_peered":"2024-03-30T13:53:35.863124+0000","last_unstale":"2024-03-30T13:54:07.502558+0000","last_undegraded":"2024-03-30T13:54:07.502558+0000","last_fullsized":"2024-03-30T13:54:07.502558+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:09.247282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:09.247282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"238'4793","reported_seq":4854,"reported_epoch":238,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:54:04.547551+0000","last_change":"2024-03-30T13:52:51.714810+0000","last_active":"2024-03-30T13:54:04.547551+0000","last_peered":"2024-03-30T13:54:04.547551+0000","last_clean":"2024-03-30T13:52:42.503419+0000","last_became_active":"2024-03-30T13:52:43.528398+0000","last_became_peered":"2024-03-30T13:52:43.528398+0000","last_unstale":"2024-03-30T13:54:04.547551+0000","last_undegraded":"2024-03-30T13:52:51.712396+0000","last_fullsized":"2024-03-30T13:52:43.524380+0000","mapping_epoch":211,"log_start":"218'1247","ondisk_log_start":"218'1247","created":211,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3546,"log_dups_size":0,"ondisk_log_size":3546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314114048,"num_objects":4793,"num_object_clones":0,"num_object_copies":14379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4793,"num_write_kb":306752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":4793}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":345,"reported_epoch":240,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:07.520517+0000","last_change":"2024-03-30T13:53:16.673293+0000","last_active":"2024-03-30T13:54:07.520517+0000","last_peered":"2024-03-30T13:54:07.520517+0000","last_clean":"2024-03-30T13:54:07.520517+0000","last_became_active":"2024-03-30T13:53:16.672717+0000","last_became_peered":"2024-03-30T13:53:16.672717+0000","last_unstale":"2024-03-30T13:54:07.520517+0000","last_undegraded":"2024-03-30T13:54:07.520517+0000","last_fullsized":"2024-03-30T13:54:07.520517+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:49.875971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:49.875971+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:08.477169+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,5],"avail_no_missing":["6","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":44,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:04.024035+0000","last_change":"2024-03-30T13:53:14.661063+0000","last_active":"2024-03-30T13:54:04.024035+0000","last_peered":"2024-03-30T13:54:04.024035+0000","last_clean":"2024-03-30T13:54:04.024035+0000","last_became_active":"2024-03-30T13:53:14.660483+0000","last_became_peered":"2024-03-30T13:53:14.660483+0000","last_unstale":"2024-03-30T13:54:04.024035+0000","last_undegraded":"2024-03-30T13:54:04.024035+0000","last_fullsized":"2024-03-30T13:54:04.024035+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:14.144050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:14.144050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0,5],"avail_no_missing":["1","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"223'11","reported_seq":551,"reported_epoch":240,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:07.504437+0000","last_change":"2024-03-30T13:53:14.651471+0000","last_active":"2024-03-30T13:54:07.504437+0000","last_peered":"2024-03-30T13:54:07.504437+0000","last_clean":"2024-03-30T13:54:07.504437+0000","last_became_active":"2024-03-30T13:53:14.650289+0000","last_became_peered":"2024-03-30T13:53:14.650289+0000","last_unstale":"2024-03-30T13:54:07.504437+0000","last_undegraded":"2024-03-30T13:54:07.504437+0000","last_fullsized":"2024-03-30T13:54:07.504437+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-30T13:51:15.146255+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-30T13:51:15.146255+0000","last_clean_scrub_stamp":"2024-03-30T13:51:15.146255+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,5],"avail_no_missing":["4","0","5"],"object_location_counts":[{"shards":"0,4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":399,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.994451+0000","last_change":"2024-03-30T13:53:35.887442+0000","last_active":"2024-03-30T13:54:03.994451+0000","last_peered":"2024-03-30T13:54:03.994451+0000","last_clean":"2024-03-30T13:54:03.994451+0000","last_became_active":"2024-03-30T13:53:35.886750+0000","last_became_peered":"2024-03-30T13:53:35.886750+0000","last_unstale":"2024-03-30T13:54:03.994451+0000","last_undegraded":"2024-03-30T13:54:03.994451+0000","last_fullsized":"2024-03-30T13:54:03.994451+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:27.058807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:27.058807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"217'34","reported_seq":454,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.023356+0000","last_change":"2024-03-30T13:52:52.619670+0000","last_active":"2024-03-30T13:54:04.023356+0000","last_peered":"2024-03-30T13:54:04.023356+0000","last_clean":"2024-03-30T13:54:04.023356+0000","last_became_active":"2024-03-30T13:42:47.324350+0000","last_became_peered":"2024-03-30T13:42:47.324350+0000","last_unstale":"2024-03-30T13:54:04.023356+0000","last_undegraded":"2024-03-30T13:54:04.023356+0000","last_fullsized":"2024-03-30T13:54:04.023356+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'22","last_scrub_stamp":"2024-03-30T13:52:52.619482+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:28.831527+0000","last_clean_scrub_stamp":"2024-03-30T13:52:52.619482+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:18.476091+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"238'4789","reported_seq":4856,"reported_epoch":238,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-30T13:54:04.534794+0000","last_change":"2024-03-30T13:53:37.276439+0000","last_active":"2024-03-30T13:54:04.534794+0000","last_peered":"2024-03-30T13:54:04.534794+0000","last_clean":"2024-03-30T13:53:34.056415+0000","last_became_active":"2024-03-30T13:53:37.267573+0000","last_became_peered":"2024-03-30T13:53:37.267573+0000","last_unstale":"2024-03-30T13:54:04.534794+0000","last_undegraded":"2024-03-30T13:53:37.097392+0000","last_fullsized":"2024-03-30T13:53:36.839832+0000","mapping_epoch":229,"log_start":"218'1175","ondisk_log_start":"218'1175","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3614,"log_dups_size":0,"ondisk_log_size":3614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313851904,"num_objects":4789,"num_object_clones":0,"num_object_copies":14367,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3804,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4789,"num_write_kb":306496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3","objects":985},{"shards":"1,3,7","objects":3804}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":392,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.000819+0000","last_change":"2024-03-30T13:53:35.875435+0000","last_active":"2024-03-30T13:54:04.000819+0000","last_peered":"2024-03-30T13:54:04.000819+0000","last_clean":"2024-03-30T13:54:04.000819+0000","last_became_active":"2024-03-30T13:53:35.875118+0000","last_became_peered":"2024-03-30T13:53:35.875118+0000","last_unstale":"2024-03-30T13:54:04.000819+0000","last_undegraded":"2024-03-30T13:54:04.000819+0000","last_fullsized":"2024-03-30T13:54:04.000819+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:21.081938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:21.081938+0000","last_clean_scrub_stamp":"2024-03-30T13:52:21.081938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":47,"reported_epoch":240,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:07.503762+0000","last_change":"2024-03-30T13:53:14.656429+0000","last_active":"2024-03-30T13:54:07.503762+0000","last_peered":"2024-03-30T13:54:07.503762+0000","last_clean":"2024-03-30T13:54:07.503762+0000","last_became_active":"2024-03-30T13:53:14.655908+0000","last_became_peered":"2024-03-30T13:53:14.655908+0000","last_unstale":"2024-03-30T13:54:07.503762+0000","last_undegraded":"2024-03-30T13:54:07.503762+0000","last_fullsized":"2024-03-30T13:54:07.503762+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:21.087732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:52.603772+0000","last_clean_scrub_stamp":"2024-03-30T13:51:21.087732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,5],"avail_no_missing":["4","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":398,"reported_epoch":240,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:07.502651+0000","last_change":"2024-03-30T13:53:36.852955+0000","last_active":"2024-03-30T13:54:07.502651+0000","last_peered":"2024-03-30T13:54:07.502651+0000","last_clean":"2024-03-30T13:54:07.502651+0000","last_became_active":"2024-03-30T13:53:36.852500+0000","last_became_peered":"2024-03-30T13:53:36.852500+0000","last_unstale":"2024-03-30T13:54:07.502651+0000","last_undegraded":"2024-03-30T13:54:07.502651+0000","last_fullsized":"2024-03-30T13:54:07.502651+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:15.107879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:48:15.107879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,7],"avail_no_missing":["4","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"217'19","reported_seq":441,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.503332+0000","last_change":"2024-03-30T13:52:17.057583+0000","last_active":"2024-03-30T13:54:07.503332+0000","last_peered":"2024-03-30T13:54:07.503332+0000","last_clean":"2024-03-30T13:54:07.503332+0000","last_became_active":"2024-03-30T13:42:47.315667+0000","last_became_peered":"2024-03-30T13:42:47.315667+0000","last_unstale":"2024-03-30T13:54:07.503332+0000","last_undegraded":"2024-03-30T13:54:07.503332+0000","last_fullsized":"2024-03-30T13:54:07.503332+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:52:17.057433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:17.057433+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"238'4785","reported_seq":4846,"reported_epoch":238,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:54:04.812110+0000","last_change":"2024-03-30T13:52:51.696416+0000","last_active":"2024-03-30T13:54:04.812110+0000","last_peered":"2024-03-30T13:54:04.812110+0000","last_clean":"2024-03-30T13:52:42.503419+0000","last_became_active":"2024-03-30T13:52:43.605206+0000","last_became_peered":"2024-03-30T13:52:43.605206+0000","last_unstale":"2024-03-30T13:54:04.812110+0000","last_undegraded":"2024-03-30T13:52:51.691832+0000","last_fullsized":"2024-03-30T13:52:43.519471+0000","mapping_epoch":211,"log_start":"218'1395","ondisk_log_start":"218'1395","created":211,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313589760,"num_objects":4785,"num_object_clones":0,"num_object_copies":14355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4785,"num_write_kb":306240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":4785}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":348,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:03.993921+0000","last_change":"2024-03-30T13:53:16.672844+0000","last_active":"2024-03-30T13:54:03.993921+0000","last_peered":"2024-03-30T13:54:03.993921+0000","last_clean":"2024-03-30T13:54:03.993921+0000","last_became_active":"2024-03-30T13:53:16.671377+0000","last_became_peered":"2024-03-30T13:53:16.671377+0000","last_unstale":"2024-03-30T13:54:03.993921+0000","last_undegraded":"2024-03-30T13:54:03.993921+0000","last_fullsized":"2024-03-30T13:54:03.993921+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:47.932192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:47.932192+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:05.237103+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":423697,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,5],"avail_no_missing":["3","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":164,"reported_epoch":238,"state":"active+undersized","last_fresh":"2024-03-30T13:54:04.002300+0000","last_change":"2024-03-30T13:50:08.511676+0000","last_active":"2024-03-30T13:54:04.002300+0000","last_peered":"2024-03-30T13:54:04.002300+0000","last_clean":"2024-03-30T13:50:05.504989+0000","last_became_active":"2024-03-30T13:50:08.511676+0000","last_became_peered":"2024-03-30T13:50:08.511676+0000","last_unstale":"2024-03-30T13:54:04.002300+0000","last_undegraded":"2024-03-30T13:54:04.002300+0000","last_fullsized":"2024-03-30T13:50:08.504157+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:09.357378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:09.357378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":401,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:04.023990+0000","last_change":"2024-03-30T13:52:13.440265+0000","last_active":"2024-03-30T13:54:04.023990+0000","last_peered":"2024-03-30T13:54:04.023990+0000","last_clean":"2024-03-30T13:54:04.023990+0000","last_became_active":"2024-03-30T13:42:49.321360+0000","last_became_peered":"2024-03-30T13:42:49.321360+0000","last_unstale":"2024-03-30T13:54:04.023990+0000","last_undegraded":"2024-03-30T13:54:04.023990+0000","last_fullsized":"2024-03-30T13:54:04.023990+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:13.440150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:13.440150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":419,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.521036+0000","last_change":"2024-03-30T13:52:25.063121+0000","last_active":"2024-03-30T13:54:07.521036+0000","last_peered":"2024-03-30T13:54:07.521036+0000","last_clean":"2024-03-30T13:54:07.521036+0000","last_became_active":"2024-03-30T13:42:45.315251+0000","last_became_peered":"2024-03-30T13:42:45.315251+0000","last_unstale":"2024-03-30T13:54:07.521036+0000","last_undegraded":"2024-03-30T13:54:07.521036+0000","last_fullsized":"2024-03-30T13:54:07.521036+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:25.062771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:25.062771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"217'31","reported_seq":438,"reported_epoch":240,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:54:07.503550+0000","last_change":"2024-03-30T13:50:08.508030+0000","last_active":"2024-03-30T13:54:07.503550+0000","last_peered":"2024-03-30T13:54:07.503550+0000","last_clean":"2024-03-30T13:50:05.491159+0000","last_became_active":"2024-03-30T13:50:08.508030+0000","last_became_peered":"2024-03-30T13:50:08.508030+0000","last_unstale":"2024-03-30T13:54:07.503550+0000","last_undegraded":"2024-03-30T13:50:08.502958+0000","last_fullsized":"2024-03-30T13:50:08.502815+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-03-30T13:48:27.096261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:27.096261+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"238'4891","reported_seq":6723,"reported_epoch":238,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-30T13:54:04.534460+0000","last_change":"2024-03-30T13:53:37.405292+0000","last_active":"2024-03-30T13:54:04.534460+0000","last_peered":"2024-03-30T13:54:04.534460+0000","last_clean":"2024-03-30T13:53:34.131594+0000","last_became_active":"2024-03-30T13:53:37.251062+0000","last_became_peered":"2024-03-30T13:53:37.251062+0000","last_unstale":"2024-03-30T13:54:04.534460+0000","last_undegraded":"2024-03-30T13:54:04.534460+0000","last_fullsized":"2024-03-30T13:53:36.839134+0000","mapping_epoch":229,"log_start":"218'1356","ondisk_log_start":"218'1356","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3535,"log_dups_size":0,"ondisk_log_size":3535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320536576,"num_objects":4891,"num_object_clones":0,"num_object_copies":14673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3003,"num_objects_unfound":0,"num_objects_dirty":4891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4891,"num_write_kb":313024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":882,"num_bytes_recovered":57802752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6],"avail_no_missing":["3","6","7"],"object_location_counts":[{"shards":"3,6","objects":1888},{"shards":"3,6,7","objects":3003}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":623,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:03.999029+0000","last_change":"2024-03-30T13:53:38.550111+0000","last_active":"2024-03-30T13:54:03.999029+0000","last_peered":"2024-03-30T13:54:03.999029+0000","last_clean":"2024-03-30T13:54:03.999029+0000","last_became_active":"2024-03-30T13:53:38.543500+0000","last_became_peered":"2024-03-30T13:53:38.543500+0000","last_unstale":"2024-03-30T13:54:03.999029+0000","last_undegraded":"2024-03-30T13:54:03.999029+0000","last_fullsized":"2024-03-30T13:54:03.999029+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:48.985609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:48.985609+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:54:05.237131+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,7],"avail_no_missing":["3","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":381,"reported_epoch":240,"state":"active+undersized","last_fresh":"2024-03-30T13:54:07.504643+0000","last_change":"2024-03-30T13:50:08.511010+0000","last_active":"2024-03-30T13:54:07.504643+0000","last_peered":"2024-03-30T13:54:07.504643+0000","last_clean":"2024-03-30T13:50:05.490535+0000","last_became_active":"2024-03-30T13:50:08.511010+0000","last_became_peered":"2024-03-30T13:50:08.511010+0000","last_unstale":"2024-03-30T13:54:07.504643+0000","last_undegraded":"2024-03-30T13:54:07.504643+0000","last_fullsized":"2024-03-30T13:50:08.503118+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:06.154805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:06.154805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"231'13","reported_seq":236,"reported_epoch":238,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:54:03.995693+0000","last_change":"2024-03-30T13:53:37.600947+0000","last_active":"2024-03-30T13:54:03.995693+0000","last_peered":"2024-03-30T13:54:03.995693+0000","last_clean":"2024-03-30T13:54:03.995693+0000","last_became_active":"2024-03-30T13:53:37.518434+0000","last_became_peered":"2024-03-30T13:53:37.518434+0000","last_unstale":"2024-03-30T13:54:03.995693+0000","last_undegraded":"2024-03-30T13:54:03.995693+0000","last_fullsized":"2024-03-30T13:54:03.995693+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:49:36.397622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:36.397622+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3,7],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":393,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:54:03.999839+0000","last_change":"2024-03-30T13:51:58.939856+0000","last_active":"2024-03-30T13:54:03.999839+0000","last_peered":"2024-03-30T13:54:03.999839+0000","last_clean":"2024-03-30T13:54:03.999839+0000","last_became_active":"2024-03-30T13:42:45.309265+0000","last_became_peered":"2024-03-30T13:42:45.309265+0000","last_unstale":"2024-03-30T13:54:03.999839+0000","last_undegraded":"2024-03-30T13:54:03.999839+0000","last_fullsized":"2024-03-30T13:54:03.999839+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:58.939556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:58.939556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"216'20","reported_seq":432,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-30T13:54:07.504477+0000","last_change":"2024-03-30T13:51:59.022459+0000","last_active":"2024-03-30T13:54:07.504477+0000","last_peered":"2024-03-30T13:54:07.504477+0000","last_clean":"2024-03-30T13:54:07.504477+0000","last_became_active":"2024-03-30T13:42:47.313026+0000","last_became_peered":"2024-03-30T13:42:47.313026+0000","last_unstale":"2024-03-30T13:54:07.504477+0000","last_undegraded":"2024-03-30T13:54:07.504477+0000","last_fullsized":"2024-03-30T13:54:07.504477+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-30T13:51:59.022304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:59.022304+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":5032312832,"num_objects":76787,"num_object_clones":0,"num_object_copies":230361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33014,"num_objects_misplaced":36895,"num_objects_unfound":0,"num_objects_dirty":76787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76787,"num_write_kb":4914368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13549,"num_bytes_recovered":887947264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15304556544,"data_stored":15304556544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57377,"ondisk_log_size":57377,"up":42,"acting":36,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":30,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":649698,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":109,"num_read_kb":161,"num_write":98,"num_write_kb":1316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1884104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":26,"acting":33,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":5246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":86,"acting":93,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47,"num_objects_misplaced":20,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":2244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":12206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":740,"ondisk_log_size":740,"up":85,"acting":89,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":82,"acting":93,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":77,"acting":91,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":249108103310,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397650,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1947608,"kb_used_data":1631824,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":91797544,"statfs":{"total":95995035648,"available":94000685056,"internally_reserved":0,"allocated":1670987776,"data_stored":1670229801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":323353030},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575019,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3511796,"kb_used_data":3061352,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":90233356,"statfs":{"total":95995035648,"available":92398956544,"internally_reserved":0,"allocated":3134824448,"data_stored":3134012962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738544,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1872932,"kb_used_data":1472468,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":91872220,"statfs":{"total":95995035648,"available":94077153280,"internally_reserved":0,"allocated":1507807232,"data_stored":1507025698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411446,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2213288,"kb_used_data":2015520,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":91531864,"statfs":{"total":95995035648,"available":93728628736,"internally_reserved":0,"allocated":2063892480,"data_stored":2063509110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182561,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3261028,"kb_used_data":2782812,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":90484124,"statfs":{"total":95995035648,"available":92655742976,"internally_reserved":0,"allocated":2849599488,"data_stored":2848810101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":489683399},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953628,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1708784,"kb_used_data":1380520,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92036368,"statfs":{"total":95995035648,"available":94245240832,"internally_reserved":0,"allocated":1413652480,"data_stored":1412869485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":171798691991,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":2778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1514209280,"data_stored":1514209280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3158179840,"data_stored":3158179840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2855862272,"data_stored":2855862272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1416626176,"data_stored":1416626176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2228486144,"data_stored":2228486144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1671364608,"data_stored":1671364608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2459828224,"data_stored":2459828224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:54:18.620 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool set .mgr pgp_num 11 2024-03-30T13:54:18.620 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:18.683 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:18.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:18.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:18.703 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 87 16 84965 84949 61.0128 39.375 0.016576 0.0163819 2024-03-30T13:54:18.786 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:18.811 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:18.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:18.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:18.876 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:18 smithi043 ceph-mon[29963]: pgmap v489: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 42 MiB/s wr, 673 op/s; 33820/248160 objects degraded (13.628%); 36679/248160 objects misplaced (14.780%); 11 MiB/s, 179 objects/s recovering 2024-03-30T13:54:18.877 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:18 smithi043 ceph-mon[42099]: pgmap v489: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 42 MiB/s wr, 673 op/s; 33820/248160 objects degraded (13.628%); 36679/248160 objects misplaced (14.780%); 11 MiB/s, 179 objects/s recovering 2024-03-30T13:54:18.913 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:18.937 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:18 smithi113 ceph-mon[34341]: pgmap v489: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 42 MiB/s wr, 673 op/s; 33820/248160 objects degraded (13.628%); 36679/248160 objects misplaced (14.780%); 11 MiB/s, 179 objects/s recovering 2024-03-30T13:54:18.940 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:18.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:18.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:19.041 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:19.093 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:19.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:19.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:19.136 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:54:19.195 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:19.218 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:54:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:54:19.261 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:54:19.261 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T13:54:19.261 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-03-30T13:54:19.261 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-03-30T13:54:19.261 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:54:19.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:19.271 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:19.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:19.373 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:19.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:19.429 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:19.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:19.530 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:19.557 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:19.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:19.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:19.659 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:19.685 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:19.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:19.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:19.704 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 88 16 85559 85543 60.7414 37.125 0.0137676 0.0164576 2024-03-30T13:54:19.713 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:19 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:54:19.713 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:19 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:54:19.787 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:19.830 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:19.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:19.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:19.932 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:19.959 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:19.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:19.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:19.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:19 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:54:20.061 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:20.119 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:20.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:20.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:20.220 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:20.248 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:20.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:20.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:20.350 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:20.391 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:20.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:20.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:20.492 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:20.495 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:54:20.518 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:20.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:20.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:20.620 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:20.647 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:20.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:20.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:20.703 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 89 16 86190 86174 60.502 39.4375 0.0264763 0.0165214 2024-03-30T13:54:20.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:20.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:20.797 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:20.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:20.829 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:20 smithi043 ceph-mon[29963]: pgmap v490: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 37 MiB/s wr, 594 op/s; 33820/248775 objects degraded (13.595%); 36209/248775 objects misplaced (14.555%); 13 MiB/s, 200 objects/s recovering 2024-03-30T13:54:20.830 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:20 smithi043 ceph-mon[42099]: pgmap v490: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 37 MiB/s wr, 594 op/s; 33820/248775 objects degraded (13.595%); 36209/248775 objects misplaced (14.555%); 13 MiB/s, 200 objects/s recovering 2024-03-30T13:54:20.898 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:20.944 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:20.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:20.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:20.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:20 smithi113 ceph-mon[34341]: pgmap v490: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 37 MiB/s wr, 594 op/s; 33820/248775 objects degraded (13.595%); 36209/248775 objects misplaced (14.555%); 13 MiB/s, 200 objects/s recovering 2024-03-30T13:54:21.046 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:21.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:21.092 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:21.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:21.193 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:21.248 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:21.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:21.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:21.350 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:21.377 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:21.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:21.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:21.479 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:21.505 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:21.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:21.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:21.606 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:21.633 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:21.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:21.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:21.704 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 90 2 86813 86811 60.2721 39.8125 0.0225772 0.0165848 2024-03-30T13:54:21.738 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:21.762 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:21.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:21.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:21.864 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:21.896 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:21.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:21.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:21.926 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-03-30T13:54:21.998 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:22.043 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:22.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:22.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:22.144 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:22.168 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:22.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:22.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:22.270 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:22.294 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:22.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:22.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:22.361 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:54:22.362 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:54:22.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:22.447 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:22.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:22.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:22.468 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:54:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=cleanup t=2024-03-30T13:54:22.119854023Z level=info msg="Completed cleanup jobs" duration=1.925345ms 2024-03-30T13:54:22.468 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:54:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=sqlstore.transactions t=2024-03-30T13:54:22.128256198Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-30T13:54:22.549 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:22.573 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:22.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:22.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:22.674 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:22.698 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:22.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:22.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:22.704 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Total time run: 90.021 2024-03-30T13:54:22.704 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Total writes made: 86813 2024-03-30T13:54:22.704 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Write size: 65536 2024-03-30T13:54:22.704 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Object size: 65536 2024-03-30T13:54:22.704 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Bandwidth (MB/sec): 60.2728 2024-03-30T13:54:22.704 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev Bandwidth: 52.3469 2024-03-30T13:54:22.704 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max bandwidth (MB/sec): 217.625 2024-03-30T13:54:22.704 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min bandwidth (MB/sec): 0 2024-03-30T13:54:22.704 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Average IOPS: 964 2024-03-30T13:54:22.704 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev IOPS: 837.561 2024-03-30T13:54:22.705 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max IOPS: 3482 2024-03-30T13:54:22.705 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min IOPS: 0 2024-03-30T13:54:22.705 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Average Latency(s): 0.0165851 2024-03-30T13:54:22.705 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev Latency(s): 0.0885074 2024-03-30T13:54:22.705 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max latency(s): 7.25374 2024-03-30T13:54:22.705 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min latency(s): 0.00127384 2024-03-30T13:54:22.725 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Cleaning up (deleting benchmark objects) 2024-03-30T13:54:22.800 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:22.826 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:22.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:22.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:22.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:22 smithi043 ceph-mon[29963]: pgmap v491: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 38 MiB/s wr, 605 op/s; 34199/257475 objects degraded (13.282%); 36299/257475 objects misplaced (14.098%); 13 MiB/s, 203 objects/s recovering 2024-03-30T13:54:22.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:22 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 34199/257475 objects degraded (13.282%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:22.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:22 smithi043 ceph-mon[42099]: pgmap v491: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 38 MiB/s wr, 605 op/s; 34199/257475 objects degraded (13.282%); 36299/257475 objects misplaced (14.098%); 13 MiB/s, 203 objects/s recovering 2024-03-30T13:54:22.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:22 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 34199/257475 objects degraded (13.282%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:22.928 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:22.953 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:22.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:22.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:22.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:22 smithi113 ceph-mon[34341]: pgmap v491: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 38 MiB/s wr, 605 op/s; 34199/257475 objects degraded (13.282%); 36299/257475 objects misplaced (14.098%); 13 MiB/s, 203 objects/s recovering 2024-03-30T13:54:22.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:22 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 34199/257475 objects degraded (13.282%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:23.056 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:23.083 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:23.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:23.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:23.185 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:23.234 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:23.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:23.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:23.336 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:23.364 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:23.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:23.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:23.466 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:23.491 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:23.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:23.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:23.593 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:23.635 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:23.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:23.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-03-30T13:54:23.737 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:23.764 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:23.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:23.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:23.866 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:23.908 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:23.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:23.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:23.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:23 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:54:23.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:23 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:54:23.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:23 smithi113 ceph-mon[34341]: osdmap e243: 8 total, 7 up, 6 in 2024-03-30T13:54:23.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:23 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1016702328' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:54:24.010 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:24.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:24.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:24.112 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:23 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:54:24.112 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:23 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:54:24.112 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:23 smithi043 ceph-mon[29963]: osdmap e243: 8 total, 7 up, 6 in 2024-03-30T13:54:24.112 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:23 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1016702328' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:54:24.112 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:23 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:54:24.113 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:23 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:54:24.113 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:23 smithi043 ceph-mon[42099]: osdmap e243: 8 total, 7 up, 6 in 2024-03-30T13:54:24.113 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:23 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1016702328' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:54:24.150 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:24.174 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:24.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:24.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:24.276 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:24.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:24.324 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:24.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:24.425 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:24.449 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:24.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:24.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:24.551 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:24.600 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:24.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:24.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:24.702 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:24.727 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:24.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:24.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:24.829 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:24.854 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:24.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:24.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:24.956 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:24.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:24 smithi113 ceph-mon[34341]: pgmap v493: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 38 MiB/s wr, 605 op/s; 34199/257475 objects degraded (13.282%); 36299/257475 objects misplaced (14.098%); 13 MiB/s, 203 objects/s recovering 2024-03-30T13:54:24.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:24 smithi113 ceph-mon[34341]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:54:24.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:24 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1016702328' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:54:24.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:24 smithi113 ceph-mon[34341]: osdmap e244: 8 total, 7 up, 6 in 2024-03-30T13:54:24.976 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:24 smithi043 ceph-mon[29963]: pgmap v493: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 38 MiB/s wr, 605 op/s; 34199/257475 objects degraded (13.282%); 36299/257475 objects misplaced (14.098%); 13 MiB/s, 203 objects/s recovering 2024-03-30T13:54:24.976 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:24 smithi043 ceph-mon[29963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:54:24.976 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:24 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1016702328' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:54:24.976 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:24 smithi043 ceph-mon[29963]: osdmap e244: 8 total, 7 up, 6 in 2024-03-30T13:54:24.976 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:24 smithi043 ceph-mon[42099]: pgmap v493: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 38 MiB/s wr, 605 op/s; 34199/257475 objects degraded (13.282%); 36299/257475 objects misplaced (14.098%); 13 MiB/s, 203 objects/s recovering 2024-03-30T13:54:24.976 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:24 smithi043 ceph-mon[42099]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:54:24.977 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:24 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1016702328' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:54:24.977 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:24 smithi043 ceph-mon[42099]: osdmap e244: 8 total, 7 up, 6 in 2024-03-30T13:54:24.982 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:24.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:24.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:25.083 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:25.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:25.131 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:25.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:25.232 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:25.259 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:25.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:25.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:25.361 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:25.386 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:25.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:25.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:25.488 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:25.527 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:25.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:25.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:25.629 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:25.665 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:25.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:25.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:25.767 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:25.814 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:25.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:25.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:25.916 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:25.957 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:25.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:25.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:26.059 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:26.101 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:26.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:26.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:26.202 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:26.226 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:26.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:26.328 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:26.351 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:26.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:26.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:26.395 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:54:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:54:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:54:26.453 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:26.484 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:26.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:26.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:26.585 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:26.615 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:26.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:26.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:26.717 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:26.744 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:26.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:26.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:26.846 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:26.886 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:26.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:26.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:26.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:26 smithi113 ceph-mon[34341]: pgmap v495: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.4 KiB/s rd, 34 MiB/s wr, 699 op/s; 34192/257145 objects degraded (13.297%); 34599/257145 objects misplaced (13.455%); 20 MiB/s, 320 objects/s recovering 2024-03-30T13:54:26.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:26 smithi113 ceph-mon[34341]: osdmap e245: 8 total, 7 up, 6 in 2024-03-30T13:54:26.988 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:27.013 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:27.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:27.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:27.115 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:27.139 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:27.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:27.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:27.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:26 smithi043 ceph-mon[29963]: pgmap v495: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.4 KiB/s rd, 34 MiB/s wr, 699 op/s; 34192/257145 objects degraded (13.297%); 34599/257145 objects misplaced (13.455%); 20 MiB/s, 320 objects/s recovering 2024-03-30T13:54:27.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:26 smithi043 ceph-mon[29963]: osdmap e245: 8 total, 7 up, 6 in 2024-03-30T13:54:27.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:26 smithi043 ceph-mon[42099]: pgmap v495: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.4 KiB/s rd, 34 MiB/s wr, 699 op/s; 34192/257145 objects degraded (13.297%); 34599/257145 objects misplaced (13.455%); 20 MiB/s, 320 objects/s recovering 2024-03-30T13:54:27.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:26 smithi043 ceph-mon[42099]: osdmap e245: 8 total, 7 up, 6 in 2024-03-30T13:54:27.241 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:27.264 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:27.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:27.366 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:27.391 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:27.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:27.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:27.493 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:27.518 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:27.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:27.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:27.620 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:27.656 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:27.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:27.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:27.758 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:27.798 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:27.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:27.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:27.900 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:27.923 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:27.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:27.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:27.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:27 smithi113 ceph-mon[34341]: osdmap e246: 8 total, 7 up, 6 in 2024-03-30T13:54:27.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:27 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 34192/257145 objects degraded (13.297%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:28.025 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:28.051 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:28.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:28.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:28.134 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:27 smithi043 ceph-mon[29963]: osdmap e246: 8 total, 7 up, 6 in 2024-03-30T13:54:28.134 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:27 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 34192/257145 objects degraded (13.297%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:28.135 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:27 smithi043 ceph-mon[42099]: osdmap e246: 8 total, 7 up, 6 in 2024-03-30T13:54:28.135 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:27 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 34192/257145 objects degraded (13.297%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:28.153 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:28.187 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:28.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:28.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:28.289 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:28.337 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:28.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:28.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:28.438 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:28.464 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:28.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:28.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:28.566 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:28.592 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:28.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:28.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:28.698 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:28.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:28.722 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:28.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:28.823 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:28.850 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:28.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:28.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:28.952 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:28.968 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:54:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:54:28.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:28 smithi113 ceph-mon[34341]: pgmap v498: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 15 MiB/s wr, 510 op/s; 34192/257145 objects degraded (13.297%); 34599/257145 objects misplaced (13.455%); 19 MiB/s, 306 objects/s recovering 2024-03-30T13:54:28.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:54:28.969 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:54:28.977 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:28.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:28.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:29.078 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:29.104 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:29.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:29.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:29.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:28 smithi043 ceph-mon[29963]: pgmap v498: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 15 MiB/s wr, 510 op/s; 34192/257145 objects degraded (13.297%); 34599/257145 objects misplaced (13.455%); 19 MiB/s, 306 objects/s recovering 2024-03-30T13:54:29.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:54:29.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:54:29.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:28 smithi043 ceph-mon[42099]: pgmap v498: 158 pgs: 81 active+clean, 5 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 15 MiB/s wr, 510 op/s; 34192/257145 objects degraded (13.297%); 34599/257145 objects misplaced (13.455%); 19 MiB/s, 306 objects/s recovering 2024-03-30T13:54:29.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:54:29.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:54:29.205 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:29.231 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:29.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:29.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:29.333 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:29.356 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:29.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:29.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:29.458 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:29.483 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:29.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:29.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:29.585 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:29.611 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:29.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:29.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:29.713 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:29.737 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:29.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:29.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:29.839 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:29.877 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:29.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:29.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:29 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:54:29.979 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:30.004 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:30.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:30.106 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:30.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:29 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:54:30.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:29 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:54:30.150 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:30.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:30.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:30.253 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:30.278 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:30.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:30.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:30.380 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:30.403 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:54:30.404 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:30.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:30.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:30.506 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:30.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:30.560 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:30.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:30.661 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:30.685 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:30.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:30.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:30.787 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:30.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:30.832 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:30.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:30.933 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:30.958 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:30.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:30.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:30.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:30 smithi113 ceph-mon[34341]: pgmap v499: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 12 MiB/s wr, 441 op/s; 34192/256674 objects degraded (13.321%); 34384/256674 objects misplaced (13.396%); 18 MiB/s, 285 objects/s recovering 2024-03-30T13:54:31.061 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:31.097 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:31.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:31.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:31.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:30 smithi043 ceph-mon[29963]: pgmap v499: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 12 MiB/s wr, 441 op/s; 34192/256674 objects degraded (13.321%); 34384/256674 objects misplaced (13.396%); 18 MiB/s, 285 objects/s recovering 2024-03-30T13:54:31.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:30 smithi043 ceph-mon[42099]: pgmap v499: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 12 MiB/s wr, 441 op/s; 34192/256674 objects degraded (13.321%); 34384/256674 objects misplaced (13.396%); 18 MiB/s, 285 objects/s recovering 2024-03-30T13:54:31.199 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:31.241 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:31.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:31.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:31.343 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:31.367 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:31.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:31.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:31.469 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:31.493 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:31.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:31.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:31.595 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:31.620 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:31.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:31.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:31.722 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:31.766 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:31.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:31.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:31.868 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:31.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:31.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:31.993 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:32.035 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:32.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:32.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:32.136 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:32.180 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:32.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:32.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:32.282 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:32.306 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:32.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:32.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:32.408 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:32.462 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:32.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:32.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:32.563 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:32.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:32.621 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:32.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:32.722 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:32.749 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:32.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:32.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:32.850 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:32.880 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:32.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:32.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:32.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:32 smithi113 ceph-mon[34341]: pgmap v500: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.4 KiB/s rd, 8.5 MiB/s wr, 632 op/s; 28107/249267 objects degraded (11.276%); 38464/249267 objects misplaced (15.431%); 15 MiB/s, 245 objects/s recovering 2024-03-30T13:54:32.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:32 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:54:32.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:32 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 28107/249267 objects degraded (11.276%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:32.982 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:33.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:33.036 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:33.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:33.137 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:33.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:32 smithi043 ceph-mon[29963]: pgmap v500: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.4 KiB/s rd, 8.5 MiB/s wr, 632 op/s; 28107/249267 objects degraded (11.276%); 38464/249267 objects misplaced (15.431%); 15 MiB/s, 245 objects/s recovering 2024-03-30T13:54:33.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:32 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:54:33.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:32 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 28107/249267 objects degraded (11.276%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:33.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:32 smithi043 ceph-mon[42099]: pgmap v500: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.4 KiB/s rd, 8.5 MiB/s wr, 632 op/s; 28107/249267 objects degraded (11.276%); 38464/249267 objects misplaced (15.431%); 15 MiB/s, 245 objects/s recovering 2024-03-30T13:54:33.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:32 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:54:33.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:32 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 28107/249267 objects degraded (11.276%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:33.163 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:33.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:33.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:33.265 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:33.292 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:33.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:33.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:33.394 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:33.422 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:33.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:33.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:33.523 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:33.549 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:33.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:33.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:33.651 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:33.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:33.675 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:33.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:33.776 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:33.804 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:33.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:33.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:33.905 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:33.933 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:33.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:33.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:34.035 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:34.060 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:34.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:34.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:34.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:33 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:34.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:33 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:34.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:33 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:34.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:33 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:34.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:33 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:54:34.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:33 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:34.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:33 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:34.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:33 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:34.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:33 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:34.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:33 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:54:34.162 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:34.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:34.187 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:34.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:34.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:33 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:34.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:33 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:34.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:33 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:34.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:33 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:34.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:33 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:54:34.288 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:34.313 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:34.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:34.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:34.415 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:34.440 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:34.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:34.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:34.542 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:34.567 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:34.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:34.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:34.669 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:34.695 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:34.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:34.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:34.797 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:34.833 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:34.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:34.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:34.935 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:34.960 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:34.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:34.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:35.062 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:35.089 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:35.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:35.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:35.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:34 smithi043 ceph-mon[29963]: Adjusting osd_memory_target on smithi113 to 3248M 2024-03-30T13:54:35.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:34 smithi043 ceph-mon[29963]: pgmap v501: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 328 op/s; 28107/249267 objects degraded (11.276%); 38464/249267 objects misplaced (15.431%); 8.3 MiB/s, 132 objects/s recovering 2024-03-30T13:54:35.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:34 smithi043 ceph-mon[42099]: Adjusting osd_memory_target on smithi113 to 3248M 2024-03-30T13:54:35.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:34 smithi043 ceph-mon[42099]: pgmap v501: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 328 op/s; 28107/249267 objects degraded (11.276%); 38464/249267 objects misplaced (15.431%); 8.3 MiB/s, 132 objects/s recovering 2024-03-30T13:54:35.190 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:35.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:34 smithi113 ceph-mon[34341]: Adjusting osd_memory_target on smithi113 to 3248M 2024-03-30T13:54:35.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:34 smithi113 ceph-mon[34341]: pgmap v501: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 5.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 328 op/s; 28107/249267 objects degraded (11.276%); 38464/249267 objects misplaced (15.431%); 8.3 MiB/s, 132 objects/s recovering 2024-03-30T13:54:35.230 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:35.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:35.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:35.332 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:35.380 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:35.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:35.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:35.482 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:35.523 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:35.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:35.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:35.625 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:35.688 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:35.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:35.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:35.789 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:35.813 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:35.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:35.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:35.915 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:35.941 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:35.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:35.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:36.043 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:36.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:36.093 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:36.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:36.194 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:36.218 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:36.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:36.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:36.319 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:36.344 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:36.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:36.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:36.395 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:54:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:54:36.446 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:36.472 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:36.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:36.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:36.574 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:36.620 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:36.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:36.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:36.722 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:36.773 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:36.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:36.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:36.875 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:36.901 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:36.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:36.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:37.002 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:37.026 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:37.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:37.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:37.128 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:37.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-mon[29963]: pgmap v502: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.9 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 558 op/s; 27918/240669 objects degraded (11.600%); 36682/240669 objects misplaced (15.242%); 11 MiB/s, 179 objects/s recovering 2024-03-30T13:54:37.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:37.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:37.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:37.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:37.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-mon[29963]: Adjusting osd_memory_target on smithi043 to 3249M 2024-03-30T13:54:37.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:54:37.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:37.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:54:37.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:54:37.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-mon[42099]: pgmap v502: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.9 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 558 op/s; 27918/240669 objects degraded (11.600%); 36682/240669 objects misplaced (15.242%); 11 MiB/s, 179 objects/s recovering 2024-03-30T13:54:37.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:37.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:37.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:37.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:37.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-mon[42099]: Adjusting osd_memory_target on smithi043 to 3249M 2024-03-30T13:54:37.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:54:37.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:37.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:54:37.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:36 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:54:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:37.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:37.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:37.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:36 smithi113 ceph-mon[34341]: pgmap v502: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.9 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 558 op/s; 27918/240669 objects degraded (11.600%); 36682/240669 objects misplaced (15.242%); 11 MiB/s, 179 objects/s recovering 2024-03-30T13:54:37.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:36 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:37.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:36 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:37.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:36 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:37.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:36 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:37.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:36 smithi113 ceph-mon[34341]: Adjusting osd_memory_target on smithi043 to 3249M 2024-03-30T13:54:37.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:36 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:54:37.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:36 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:37.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:36 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:54:37.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:36 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:54:37.270 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:37.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:37.313 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:37.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:37.414 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:37.437 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:37.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:37.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:37.539 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:37.564 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:37.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:37.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:37.666 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:37.690 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:37.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:37.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:37.791 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:37.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:37.831 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:37.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:37.932 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:37.977 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:37.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:37.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:38.033 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:37 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 27918/240669 objects degraded (11.600%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:38.033 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:37 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 27918/240669 objects degraded (11.600%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:38.079 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:38.126 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:38.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:38.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:38.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:37 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 27918/240669 objects degraded (11.600%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:38.228 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:38.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:38.272 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:38.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:38.373 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:38.400 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:38.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:38.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:38.504 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:38.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:38.553 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:38.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:38.654 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:38.680 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:38.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:38.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:38.785 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:38.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:38.826 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:38.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:38.927 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:38.953 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:38.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:38.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:39.058 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:39.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:39.085 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:39.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:39.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:38 smithi043 ceph-mon[29963]: pgmap v503: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.9 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 942 B/s rd, 0 B/s wr, 506 op/s; 27918/240669 objects degraded (11.600%); 36682/240669 objects misplaced (15.242%); 10 MiB/s, 163 objects/s recovering 2024-03-30T13:54:39.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:38 smithi043 ceph-mon[42099]: pgmap v503: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.9 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 942 B/s rd, 0 B/s wr, 506 op/s; 27918/240669 objects degraded (11.600%); 36682/240669 objects misplaced (15.242%); 10 MiB/s, 163 objects/s recovering 2024-03-30T13:54:39.186 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:39.213 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:39.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:39.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:39.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:38 smithi113 ceph-mon[34341]: pgmap v503: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.9 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 942 B/s rd, 0 B/s wr, 506 op/s; 27918/240669 objects degraded (11.600%); 36682/240669 objects misplaced (15.242%); 10 MiB/s, 163 objects/s recovering 2024-03-30T13:54:39.218 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:54:38 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:54:39.315 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:39.354 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:39.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:39.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:39.456 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:39.483 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:39.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:39.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:39.585 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:39.608 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:39.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:39.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:39.710 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:39.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:39.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:39.853 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:39.876 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:39.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:39.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:39.978 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:40.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:40.022 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:40.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:40.123 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:40.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:39 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:54:40.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:39 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:54:40.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:40.164 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:40.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:40.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:39 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:54:40.265 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:40.290 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:40.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:40.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:40.392 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:40.415 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:40.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:40.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:40.517 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:40.544 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:40.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:40.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:40.646 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:40.670 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:40.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:40.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:40.772 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:40.815 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:40.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:40.916 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:40.940 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:40.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:40.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:41.041 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:41.066 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:41.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:41.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:41.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:40 smithi043 ceph-mon[29963]: pgmap v504: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.9 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 473 op/s; 27918/240105 objects degraded (11.627%); 36682/240105 objects misplaced (15.277%); 9.2 MiB/s, 147 objects/s recovering 2024-03-30T13:54:41.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:40 smithi043 ceph-mon[42099]: pgmap v504: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.9 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 473 op/s; 27918/240105 objects degraded (11.627%); 36682/240105 objects misplaced (15.277%); 9.2 MiB/s, 147 objects/s recovering 2024-03-30T13:54:41.169 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:41.192 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:41.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:41.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:41.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:40 smithi113 ceph-mon[34341]: pgmap v504: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.9 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 473 op/s; 27918/240105 objects degraded (11.627%); 36682/240105 objects misplaced (15.277%); 9.2 MiB/s, 147 objects/s recovering 2024-03-30T13:54:41.295 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:41.340 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:41.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:41.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:41.443 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:41.467 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:41.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:41.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:41.569 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:41.595 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:41.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:41.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:41.697 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:41.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:41.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:41.851 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:41.875 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:41.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:41.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:41.977 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:42.020 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:42.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:42.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:42.122 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:42.163 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:42.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:42.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:42.265 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:42.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:42.311 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:42.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:42.412 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:42.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:42.439 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:42.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:42.540 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:42.567 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:42.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:42.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:42.668 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:42.695 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:42.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:42.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:42.797 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:42.823 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:42.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:42.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:42.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:42 smithi043 ceph-mon[29963]: pgmap v505: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 677 op/s; 27773/232320 objects degraded (11.955%); 34984/232320 objects misplaced (15.059%); 12 MiB/s, 186 objects/s recovering 2024-03-30T13:54:42.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:42 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 27773/232320 objects degraded (11.955%), 14 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:42.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:42 smithi043 ceph-mon[42099]: pgmap v505: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 677 op/s; 27773/232320 objects degraded (11.955%); 34984/232320 objects misplaced (15.059%); 12 MiB/s, 186 objects/s recovering 2024-03-30T13:54:42.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:42 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 27773/232320 objects degraded (11.955%), 14 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:42.925 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:42.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:42.973 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:42.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:43.074 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:43.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:43.147 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:43.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:43.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:42 smithi113 ceph-mon[34341]: pgmap v505: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 677 op/s; 27773/232320 objects degraded (11.955%); 34984/232320 objects misplaced (15.059%); 12 MiB/s, 186 objects/s recovering 2024-03-30T13:54:43.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:42 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 27773/232320 objects degraded (11.955%), 14 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:43.248 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:43.273 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:43.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:43.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:43.374 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:43.410 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T13:54:43.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:43.428 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:43.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:43.529 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:43.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:43.618 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:43.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:43.719 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:43.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:43.773 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:43.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:43.874 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:43.900 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:43.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:43.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:44.002 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:44.027 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:44.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:44.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:44.129 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:44.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:43 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:54:44.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:43 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:54:44.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:43 smithi043 ceph-mon[29963]: osdmap e247: 8 total, 7 up, 6 in 2024-03-30T13:54:44.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:43 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:54:44.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:43 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:54:44.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:43 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:54:44.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:43 smithi043 ceph-mon[42099]: osdmap e247: 8 total, 7 up, 6 in 2024-03-30T13:54:44.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:43 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:54:44.155 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:44.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:44.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:44.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:43 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:54:44.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:43 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:54:44.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:43 smithi113 ceph-mon[34341]: osdmap e247: 8 total, 7 up, 6 in 2024-03-30T13:54:44.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:43 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:54:44.257 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:44.283 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:44.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:44.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:44.385 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:44.442 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:44.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:44.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:44.544 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:44.588 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:44.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:44.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:44.690 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:44.715 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:44.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:44.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:44.796 INFO:teuthology.orchestra.run.smithi043.stderr:set pool 1 pgp_num to 11 2024-03-30T13:54:44.817 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:44.897 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:44.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:44.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:44.999 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:45.026 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:45.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:45.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:45.128 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:45.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:44 smithi043 ceph-mon[42099]: pgmap v507: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 565 op/s; 27773/232320 objects degraded (11.955%); 34984/232320 objects misplaced (15.059%); 8.7 MiB/s, 139 objects/s recovering 2024-03-30T13:54:45.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:44 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1034007692' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-30T13:54:45.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:44 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-30T13:54:45.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:44 smithi043 ceph-mon[29963]: pgmap v507: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 565 op/s; 27773/232320 objects degraded (11.955%); 34984/232320 objects misplaced (15.059%); 8.7 MiB/s, 139 objects/s recovering 2024-03-30T13:54:45.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:44 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1034007692' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-30T13:54:45.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:44 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-30T13:54:45.172 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:45.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:45.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:45.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:44 smithi113 ceph-mon[34341]: pgmap v507: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 565 op/s; 27773/232320 objects degraded (11.955%); 34984/232320 objects misplaced (15.059%); 8.7 MiB/s, 139 objects/s recovering 2024-03-30T13:54:45.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:44 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1034007692' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-30T13:54:45.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:44 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-30T13:54:45.274 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:45.299 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:45.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:45.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:45.401 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:45.425 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:45.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:45.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:45.527 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:45.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:45.586 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:45.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:45.687 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:45.712 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:45.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:45.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:45.814 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:45.839 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:45.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:45.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:45.940 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:45.981 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:45.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:45.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:45.994 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:54:45.994 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:45 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-30T13:54:45.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:45 smithi043 ceph-mon[29963]: osdmap e248: 8 total, 7 up, 6 in 2024-03-30T13:54:45.995 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:45 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-30T13:54:45.995 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:45 smithi043 ceph-mon[42099]: osdmap e248: 8 total, 7 up, 6 in 2024-03-30T13:54:46.053 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:54:46.083 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:46.103 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:54:46.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:46.134 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:46.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:46.145 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:54:46.187 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:54:46.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:45 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-30T13:54:46.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:45 smithi113 ceph-mon[34341]: osdmap e248: 8 total, 7 up, 6 in 2024-03-30T13:54:46.235 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:46.243 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:54:46.248 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:54:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:54:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:54:46.291 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:46.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:46.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:46.294 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:54:46.294 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:54:46.294 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:54:46.294 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:54:46.295 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:54:46.295 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:54:46.295 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:54:46.295 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:54:46.295 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:54:46.394 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:46.420 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:46.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:46.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:46.522 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:46.547 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:46.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:46.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:46.649 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:46.673 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:46.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:46.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:46.775 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:46.818 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:46.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:46.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:46.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:46 smithi043 ceph-mon[29963]: pgmap v509: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 739 op/s; 27566/222948 objects degraded (12.364%); 33029/222948 objects misplaced (14.815%); 12 MiB/s, 185 objects/s recovering 2024-03-30T13:54:46.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:46 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1448198501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:54:46.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:46 smithi043 ceph-mon[42099]: pgmap v509: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 739 op/s; 27566/222948 objects degraded (12.364%); 33029/222948 objects misplaced (14.815%); 12 MiB/s, 185 objects/s recovering 2024-03-30T13:54:46.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:46 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1448198501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:54:46.920 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:46.946 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:46.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:46.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:47.049 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:47.060 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:54:47.060 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:54:47.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:47.075 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:47.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:47.176 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:47.202 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:47.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:47.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:47.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:46 smithi113 ceph-mon[34341]: pgmap v509: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 739 op/s; 27566/222948 objects degraded (12.364%); 33029/222948 objects misplaced (14.815%); 12 MiB/s, 185 objects/s recovering 2024-03-30T13:54:47.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:46 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1448198501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:54:47.304 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:47.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:47.365 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:47.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:47.466 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:47.504 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:47.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:47.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:47.606 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:47.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:47.630 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:47.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:47.731 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:47.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:47.769 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:47.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:47.870 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:47.895 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:47.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:47.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:47.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:48.040 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:48.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:48.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:48.142 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:48.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:47 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 27566/222948 objects degraded (12.364%), 14 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:48.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:47 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 27566/222948 objects degraded (12.364%), 14 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:48.166 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:48.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:48.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:48.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:47 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 27566/222948 objects degraded (12.364%), 14 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:48.268 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:48.294 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:48.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:48.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:48.396 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:48.421 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:48.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:48.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:48.523 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:48.550 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:48.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:48.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:48.652 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:48.699 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:48.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:48.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:48.801 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:48.827 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:48.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:48.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:48.929 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:48.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:48.957 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:48.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:49.058 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:49.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:49.082 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:49.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:49.119 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:48 smithi113 ceph-mon[34341]: pgmap v510: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 715 op/s; 27566/222948 objects degraded (12.364%); 33029/222948 objects misplaced (14.815%); 12 MiB/s, 185 objects/s recovering 2024-03-30T13:54:49.119 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:54:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:54:49.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:48 smithi043 ceph-mon[29963]: pgmap v510: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 715 op/s; 27566/222948 objects degraded (12.364%); 33029/222948 objects misplaced (14.815%); 12 MiB/s, 185 objects/s recovering 2024-03-30T13:54:49.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:48 smithi043 ceph-mon[42099]: pgmap v510: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 715 op/s; 27566/222948 objects degraded (12.364%); 33029/222948 objects misplaced (14.815%); 12 MiB/s, 185 objects/s recovering 2024-03-30T13:54:49.184 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:49.228 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:49.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:49.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:49.330 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:49.355 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:49.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:49.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:49.457 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:49.482 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:49.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:49.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:49.584 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:49.632 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:49.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:49.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:49.659 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:54:49.686 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:54:49.687 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:54:49.734 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:49.761 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:49.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:49.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:49.863 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:49.887 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:49.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:49.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:49.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:49 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:54:49.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:49 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:54:49.990 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:50.015 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:50.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:50.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:50.117 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:50.169 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:50.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:50.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:50.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:49 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:54:50.242 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-30T13:54:50.271 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:50.296 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:50.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:50.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:50.398 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:50.423 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:50.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:50.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:50.525 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:50.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:50.607 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:50.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:50.708 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:50.733 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:50.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:50.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:50.835 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:50.861 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:50.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:50.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:50.963 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:50.988 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:50.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:50.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:51.090 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:51.114 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:51.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:51.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:51.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:50 smithi043 ceph-mon[29963]: pgmap v511: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 416 op/s; 27566/222327 objects degraded (12.399%); 33029/222327 objects misplaced (14.856%); 6.3 MiB/s, 101 objects/s recovering 2024-03-30T13:54:51.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:50 smithi043 ceph-mon[42099]: pgmap v511: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 416 op/s; 27566/222327 objects degraded (12.399%); 33029/222327 objects misplaced (14.856%); 6.3 MiB/s, 101 objects/s recovering 2024-03-30T13:54:51.216 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:51.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:50 smithi113 ceph-mon[34341]: pgmap v511: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 416 op/s; 27566/222327 objects degraded (12.399%); 33029/222327 objects misplaced (14.856%); 6.3 MiB/s, 101 objects/s recovering 2024-03-30T13:54:51.241 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:51.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:51.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:51.343 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:51.369 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:51.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:51.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:51.471 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:51.516 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:51.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:51.595 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:54:51.618 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:51.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:51.679 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:51.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:51.688 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:54:51.780 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6] out_osds: [5, 7] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-30T13:54:51.781 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:54:51.781 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-30T13:54:51.781 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-30T13:54:51.781 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-30T13:54:51.781 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-03-30T13:54:51.782 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:51.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:51.840 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:51.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:51.941 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:51.966 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:51.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:51.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:52.068 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:52.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:52.125 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:52.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:52.226 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:52.291 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:52.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:52.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:52.393 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:52.446 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:52.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:52.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:52.547 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:52.572 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:52.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:52.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:52.674 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:52.716 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:52.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:52.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:52.818 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:52.873 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:52.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:52.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:52.975 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:53.002 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:53.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:53.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:53.104 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:53.117 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:54:53.129 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:53.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:53.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:53.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:52 smithi043 ceph-mon[29963]: pgmap v512: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 701 op/s; 27341/213831 objects degraded (12.786%); 30684/213831 objects misplaced (14.350%); 14 MiB/s, 221 objects/s recovering 2024-03-30T13:54:53.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:52 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 27341/213831 objects degraded (12.786%), 14 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:53.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:52 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4274339492' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:54:53.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:52 smithi043 ceph-mon[42099]: pgmap v512: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 701 op/s; 27341/213831 objects degraded (12.786%); 30684/213831 objects misplaced (14.350%); 14 MiB/s, 221 objects/s recovering 2024-03-30T13:54:53.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:52 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 27341/213831 objects degraded (12.786%), 14 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:53.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:52 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/4274339492' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:54:53.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:52 smithi113 ceph-mon[34341]: pgmap v512: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 701 op/s; 27341/213831 objects degraded (12.786%); 30684/213831 objects misplaced (14.350%); 14 MiB/s, 221 objects/s recovering 2024-03-30T13:54:53.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:52 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 27341/213831 objects degraded (12.786%), 14 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:53.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:52 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/4274339492' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:54:53.231 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:53.256 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:53.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:53.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:53.358 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:53.382 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:53.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:53.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:53.484 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:53.525 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:53.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:53.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:53.530 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-03-30T13:54:53.627 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:53.653 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:53.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:53.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:53.755 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:53.795 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:53.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:53.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:53.897 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:53.923 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:53.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:53.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:54.024 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:54.050 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:54.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:54.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:54.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:53 smithi043 ceph-mon[29963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:54:54.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:53 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4274339492' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:54:54.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:53 smithi043 ceph-mon[29963]: osdmap e249: 8 total, 7 up, 6 in 2024-03-30T13:54:54.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:53 smithi043 ceph-mon[42099]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:54:54.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:53 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/4274339492' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:54:54.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:53 smithi043 ceph-mon[42099]: osdmap e249: 8 total, 7 up, 6 in 2024-03-30T13:54:54.152 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:54.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:54.180 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:54.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:54.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:53 smithi113 ceph-mon[34341]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:54:54.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:53 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/4274339492' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:54:54.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:53 smithi113 ceph-mon[34341]: osdmap e249: 8 total, 7 up, 6 in 2024-03-30T13:54:54.281 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:54.324 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:54.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:54.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:54.426 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:54.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:54.468 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:54.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:54.569 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:54.595 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:54.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:54.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:54.697 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:54.721 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:54.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:54.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:54.822 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:54.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:54.861 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:54.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:54.962 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:54.988 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:54.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:54.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:55.090 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:55.115 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:55.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:55.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:55.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:54 smithi043 ceph-mon[29963]: pgmap v514: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 680 op/s; 27341/213831 objects degraded (12.786%); 30684/213831 objects misplaced (14.350%); 14 MiB/s, 222 objects/s recovering 2024-03-30T13:54:55.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:54 smithi043 ceph-mon[42099]: pgmap v514: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 680 op/s; 27341/213831 objects degraded (12.786%); 30684/213831 objects misplaced (14.350%); 14 MiB/s, 222 objects/s recovering 2024-03-30T13:54:55.217 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:55.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:54 smithi113 ceph-mon[34341]: pgmap v514: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 680 op/s; 27341/213831 objects degraded (12.786%); 30684/213831 objects misplaced (14.350%); 14 MiB/s, 222 objects/s recovering 2024-03-30T13:54:55.246 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:55.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:55.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:55.348 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:55.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:55.406 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:55.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:55.507 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:55.553 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:55.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:55.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:55.655 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:55.680 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:55.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:55.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:55.782 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:55.807 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:55.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:55.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:55.909 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:55.947 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:55.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:55.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:56.049 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:56.099 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:56.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:56.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:56.201 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:54:56.236 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:56.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:56.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:54:56.338 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:56.365 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:56.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:56.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:56.395 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:54:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:54:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:54:56.467 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:56.492 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:56.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:56.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:56.594 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:56.620 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:56.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:56.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:56.722 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:56.749 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:56.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:56.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:56.851 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:56.877 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:56.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:56.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:56.979 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:57.004 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:57.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:57.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:57.106 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:57.131 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:57.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:57.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:57.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:56 smithi043 ceph-mon[29963]: pgmap v515: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 583 op/s; 26640/205479 objects degraded (12.965%); 28579/205479 objects misplaced (13.908%); 16 MiB/s, 251 objects/s recovering 2024-03-30T13:54:57.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:56 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:54:57.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:56 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:54:57.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:56 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:54:57.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:56 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:54:57.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:56 smithi043 ceph-mon[29963]: osdmap e250: 8 total, 7 up, 6 in 2024-03-30T13:54:57.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:56 smithi043 ceph-mon[42099]: pgmap v515: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 583 op/s; 26640/205479 objects degraded (12.965%); 28579/205479 objects misplaced (13.908%); 16 MiB/s, 251 objects/s recovering 2024-03-30T13:54:57.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:56 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:54:57.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:56 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:54:57.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:56 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:54:57.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:56 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:54:57.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:56 smithi043 ceph-mon[42099]: osdmap e250: 8 total, 7 up, 6 in 2024-03-30T13:54:57.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:56 smithi113 ceph-mon[34341]: pgmap v515: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 583 op/s; 26640/205479 objects degraded (12.965%); 28579/205479 objects misplaced (13.908%); 16 MiB/s, 251 objects/s recovering 2024-03-30T13:54:57.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:56 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:54:57.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:56 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:54:57.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:56 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:54:57.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:56 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:54:57.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:56 smithi113 ceph-mon[34341]: osdmap e250: 8 total, 7 up, 6 in 2024-03-30T13:54:57.233 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:57.260 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:57.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:57.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:57.362 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:57.388 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:57.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:57.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:57.490 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:57.518 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:57.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:57.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:57.619 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:57.643 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:57.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:57.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:57.745 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:57.771 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:57.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:57.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:57.872 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:57.899 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:57.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:57.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:58.000 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:58.025 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:58.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:58.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:58.127 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:58.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:57 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 26640/205479 objects degraded (12.965%), 14 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:58.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:57 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 26640/205479 objects degraded (12.965%), 14 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:58.153 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:58.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:58.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:58.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:57 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 26640/205479 objects degraded (12.965%), 14 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-30T13:54:58.255 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:58.280 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:58.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:58.382 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:58.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:58.424 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:58.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:58.525 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:58.549 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:58.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:58.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:58.651 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:58.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:58.675 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:58.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:58.776 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:54:58.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:58.802 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:58.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:54:58.903 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:58.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:58.941 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:58.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:59.042 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:54:59.068 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:59.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:59.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:54:59.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:58 smithi043 ceph-mon[29963]: pgmap v517: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 702 op/s; 26640/205479 objects degraded (12.965%); 28579/205479 objects misplaced (13.908%); 20 MiB/s, 313 objects/s recovering 2024-03-30T13:54:59.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:58 smithi043 ceph-mon[29963]: osdmap e251: 8 total, 7 up, 6 in 2024-03-30T13:54:59.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:58 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:54:59.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:58 smithi043 ceph-mon[42099]: pgmap v517: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 702 op/s; 26640/205479 objects degraded (12.965%); 28579/205479 objects misplaced (13.908%); 20 MiB/s, 313 objects/s recovering 2024-03-30T13:54:59.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:58 smithi043 ceph-mon[42099]: osdmap e251: 8 total, 7 up, 6 in 2024-03-30T13:54:59.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:58 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:54:59.171 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:59.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:59.195 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:59.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:59.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:58 smithi113 ceph-mon[34341]: pgmap v517: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 702 op/s; 26640/205479 objects degraded (12.965%); 28579/205479 objects misplaced (13.908%); 20 MiB/s, 313 objects/s recovering 2024-03-30T13:54:59.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:58 smithi113 ceph-mon[34341]: osdmap e251: 8 total, 7 up, 6 in 2024-03-30T13:54:59.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:58 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:54:59.218 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:54:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:54:59.296 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:54:59.333 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:59.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:59.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:54:59.435 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:59.470 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:54:59.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:59.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:54:59.572 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:54:59.601 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:59.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:59.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:54:59.702 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:59.726 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:59.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:59.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:59.828 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:54:59.854 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:54:59.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:54:59.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:54:59.956 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:54:59.999 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:00.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:00.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:00.102 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:00.128 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:00.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:00.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:00.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:59 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:55:00.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:54:59 smithi043 ceph-mon[29963]: osdmap e252: 8 total, 7 up, 6 in 2024-03-30T13:55:00.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:59 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:55:00.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:54:59 smithi043 ceph-mon[42099]: osdmap e252: 8 total, 7 up, 6 in 2024-03-30T13:55:00.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:59 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:55:00.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:54:59 smithi113 ceph-mon[34341]: osdmap e252: 8 total, 7 up, 6 in 2024-03-30T13:55:00.229 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:00.269 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:00.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:00.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:00.371 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:00.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:00.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:00.513 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:00.541 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:00.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:00.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:00.643 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:00.669 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:00.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:00.771 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:00.797 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:00.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:00.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:00.898 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:00.943 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:00.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:00.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:01.044 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:01.070 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:01.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:01.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:01.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:00 smithi043 ceph-mon[29963]: pgmap v520: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 496 op/s; 26640/204903 objects degraded (13.001%); 28579/204903 objects misplaced (13.948%); 14 MiB/s, 228 objects/s recovering 2024-03-30T13:55:01.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:00 smithi043 ceph-mon[42099]: pgmap v520: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 496 op/s; 26640/204903 objects degraded (13.001%); 28579/204903 objects misplaced (13.948%); 14 MiB/s, 228 objects/s recovering 2024-03-30T13:55:01.172 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:01.197 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:01.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:01.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:01.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:00 smithi113 ceph-mon[34341]: pgmap v520: 158 pgs: 82 active+clean, 4 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 496 op/s; 26640/204903 objects degraded (13.001%); 28579/204903 objects misplaced (13.948%); 14 MiB/s, 228 objects/s recovering 2024-03-30T13:55:01.271 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:55:01.299 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:01.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:01.366 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:01.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:01.467 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:01.516 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:01.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:01.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:01.618 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:01.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:01.642 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:01.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:01.744 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:01.787 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:01.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:01.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:01.889 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:01.913 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:01.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:01.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:02.014 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:02.040 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:02.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:02.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:02.142 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:02.169 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:02.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:02.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:02.271 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:02.296 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:02.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:02.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:02.398 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:02.423 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:02.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:02.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:02.525 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:02.551 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:02.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:02.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:02.653 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:02.676 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:02.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:02.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:02.778 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:02.804 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:02.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:02.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:02.906 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:02.954 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:02.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:02.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:03.057 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:03.101 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:03.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:03.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:03.203 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:03.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:02 smithi113 ceph-mon[34341]: pgmap v521: 158 pgs: 83 active+clean, 3 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 430 op/s; 23443/197733 objects degraded (11.856%); 28704/197733 objects misplaced (14.517%); 17 MiB/s, 276 objects/s recovering 2024-03-30T13:55:03.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:02 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 23443/197733 objects degraded (11.856%), 14 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:03.229 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:03.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:03.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:03.325 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T13:55:03.331 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:03.361 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:02 smithi043 ceph-mon[29963]: pgmap v521: 158 pgs: 83 active+clean, 3 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 430 op/s; 23443/197733 objects degraded (11.856%); 28704/197733 objects misplaced (14.517%); 17 MiB/s, 276 objects/s recovering 2024-03-30T13:55:03.361 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:02 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 23443/197733 objects degraded (11.856%), 14 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:03.361 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:02 smithi043 ceph-mon[42099]: pgmap v521: 158 pgs: 83 active+clean, 3 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 430 op/s; 23443/197733 objects degraded (11.856%); 28704/197733 objects misplaced (14.517%); 17 MiB/s, 276 objects/s recovering 2024-03-30T13:55:03.361 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:02 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 23443/197733 objects degraded (11.856%), 14 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:03.395 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:03.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:03.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:03.497 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:03.521 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:03.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:03.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:03.623 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:03.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:03.655 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:03.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:03.756 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:03.780 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:03.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:03.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:03.882 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:03.906 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:03.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:03.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:04.008 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:04.045 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:04.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:04.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:04.146 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:04.170 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:04.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:04.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:04.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:03 smithi113 ceph-mon[34341]: osdmap e253: 8 total, 7 up, 6 in 2024-03-30T13:55:04.271 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:04.295 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:04.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:04.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:04.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:03 smithi043 ceph-mon[29963]: osdmap e253: 8 total, 7 up, 6 in 2024-03-30T13:55:04.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:03 smithi043 ceph-mon[42099]: osdmap e253: 8 total, 7 up, 6 in 2024-03-30T13:55:04.397 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:04.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:04.450 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:04.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:04.551 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:04.597 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:04.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:04.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:04.699 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:04.744 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:04.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:04.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:04.845 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:04.872 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:04.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:04.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:04.974 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:04.999 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:05.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:05.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:05.101 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:05.145 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:05.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:05.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:05.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:04 smithi113 ceph-mon[34341]: pgmap v523: 158 pgs: 83 active+clean, 3 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 430 op/s; 23443/197733 objects degraded (11.856%); 28704/197733 objects misplaced (14.517%); 17 MiB/s, 276 objects/s recovering 2024-03-30T13:55:05.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:04 smithi113 ceph-mon[34341]: osdmap e254: 8 total, 7 up, 6 in 2024-03-30T13:55:05.247 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:05.284 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:05.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:05.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:05.386 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:05.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:04 smithi043 ceph-mon[29963]: pgmap v523: 158 pgs: 83 active+clean, 3 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 430 op/s; 23443/197733 objects degraded (11.856%); 28704/197733 objects misplaced (14.517%); 17 MiB/s, 276 objects/s recovering 2024-03-30T13:55:05.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:04 smithi043 ceph-mon[29963]: osdmap e254: 8 total, 7 up, 6 in 2024-03-30T13:55:05.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:04 smithi043 ceph-mon[42099]: pgmap v523: 158 pgs: 83 active+clean, 3 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 4.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 430 op/s; 23443/197733 objects degraded (11.856%); 28704/197733 objects misplaced (14.517%); 17 MiB/s, 276 objects/s recovering 2024-03-30T13:55:05.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:04 smithi043 ceph-mon[42099]: osdmap e254: 8 total, 7 up, 6 in 2024-03-30T13:55:05.429 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:05.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:05.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:05.531 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:05.557 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:05.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:05.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:05.659 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:05.684 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:05.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:05.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:05.785 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:05.811 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:05.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:05.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:05.913 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:05.955 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:05.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:05.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:06.057 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:06.079 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:05 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:55:06.079 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:05 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:55:06.079 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:05 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:55:06.079 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:05 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:55:06.082 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:06.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:06.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:06.184 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:06.208 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:06.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:06.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:06.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:05 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:55:06.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:05 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:55:06.310 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:06.353 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:06.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:06.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:06.395 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:55:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:55:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:55:06.455 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:06.479 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:06.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:06.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:06.523 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:55:06.524 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:55:06.581 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:06.626 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:06.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:06.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:06.728 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:06.785 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:06.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:06.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:06.887 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:06.912 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:06.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:06.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:07.014 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:07.039 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:07.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:07.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:07.141 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:07.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:06 smithi043 ceph-mon[29963]: pgmap v525: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 786 op/s; 18867/190689 objects degraded (9.894%); 30680/190689 objects misplaced (16.089%); 30 MiB/s, 473 objects/s recovering 2024-03-30T13:55:07.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:06 smithi043 ceph-mon[29963]: osdmap e255: 8 total, 7 up, 6 in 2024-03-30T13:55:07.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:06 smithi043 ceph-mon[42099]: pgmap v525: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 786 op/s; 18867/190689 objects degraded (9.894%); 30680/190689 objects misplaced (16.089%); 30 MiB/s, 473 objects/s recovering 2024-03-30T13:55:07.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:06 smithi043 ceph-mon[42099]: osdmap e255: 8 total, 7 up, 6 in 2024-03-30T13:55:07.184 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:07.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:07.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:07.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:06 smithi113 ceph-mon[34341]: pgmap v525: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 786 op/s; 18867/190689 objects degraded (9.894%); 30680/190689 objects misplaced (16.089%); 30 MiB/s, 473 objects/s recovering 2024-03-30T13:55:07.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:06 smithi113 ceph-mon[34341]: osdmap e255: 8 total, 7 up, 6 in 2024-03-30T13:55:07.286 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:07.312 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:07.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:07.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:07.414 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:07.439 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:07.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:07.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:07.541 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:07.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:07.606 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:07.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:07.707 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:07.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:07.762 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:07.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:07.863 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:07.889 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:07.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:07.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:07.991 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:08.016 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-03-30T13:55:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:08.055 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:08.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:08.156 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:08.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:07 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 18867/190689 objects degraded (9.894%), 14 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:08.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:07 smithi113 ceph-mon[34341]: osdmap e256: 8 total, 7 up, 6 in 2024-03-30T13:55:08.228 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:07 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 18867/190689 objects degraded (9.894%), 14 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:08.228 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:07 smithi043 ceph-mon[29963]: osdmap e256: 8 total, 7 up, 6 in 2024-03-30T13:55:08.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:07 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 18867/190689 objects degraded (9.894%), 14 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:08.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:07 smithi043 ceph-mon[42099]: osdmap e256: 8 total, 7 up, 6 in 2024-03-30T13:55:08.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:08.240 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:08.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:08.341 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:08.367 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:08.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:08.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:08.468 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:08.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:08.495 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:08.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:08.596 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:08.621 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:08.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:08.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:08.723 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:08.747 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:08.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:08.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:08.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:08.899 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:08.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:08.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:09.001 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:09.034 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:09.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:09.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:09.136 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:09.162 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:09.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:09.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:09.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:08 smithi113 ceph-mon[34341]: pgmap v528: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 504 op/s; 18867/190689 objects degraded (9.894%); 30680/190689 objects misplaced (16.089%); 16 MiB/s, 257 objects/s recovering 2024-03-30T13:55:09.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:08 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:55:09.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:08 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:55:09.218 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:55:08 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:55:09.263 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:09.288 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:09.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:09.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:09.390 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:09.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:08 smithi043 ceph-mon[29963]: pgmap v528: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 504 op/s; 18867/190689 objects degraded (9.894%); 30680/190689 objects misplaced (16.089%); 16 MiB/s, 257 objects/s recovering 2024-03-30T13:55:09.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:08 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:55:09.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:08 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:55:09.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:08 smithi043 ceph-mon[42099]: pgmap v528: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 504 op/s; 18867/190689 objects degraded (9.894%); 30680/190689 objects misplaced (16.089%); 16 MiB/s, 257 objects/s recovering 2024-03-30T13:55:09.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:08 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:55:09.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:08 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:55:09.414 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:09.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:09.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:09.516 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:09.541 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:09.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:09.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:09.643 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:09.669 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:09.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:09.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:09.771 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:09.795 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:09.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:09.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:09.897 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:09.922 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:09.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:10.024 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:10.049 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:10.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:10.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:10.151 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:10.176 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:10.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:10.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:10.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:09 smithi113 ceph-mon[34341]: osdmap e257: 8 total, 7 up, 6 in 2024-03-30T13:55:10.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:09 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:55:10.278 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:10.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:10.304 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:10.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:10.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:09 smithi043 ceph-mon[29963]: osdmap e257: 8 total, 7 up, 6 in 2024-03-30T13:55:10.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:10 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:55:10.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:09 smithi043 ceph-mon[42099]: osdmap e257: 8 total, 7 up, 6 in 2024-03-30T13:55:10.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:10 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:55:10.404 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:10.429 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:10.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:10.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:10.531 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:10.557 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:10.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:10.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:10.658 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:10.683 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:10.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:10.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:10.785 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:10.829 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:10.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:10.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:10.931 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:10.980 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:10.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:10.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:11.082 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:11.105 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:11.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:11.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:11.207 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:11.232 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:11.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:11.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:11.334 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:11.381 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:11.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:11.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:11.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:11 smithi043 ceph-mon[29963]: pgmap v530: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 905 B/s rd, 0 B/s wr, 425 op/s; 18867/190158 objects degraded (9.922%); 30680/190158 objects misplaced (16.134%); 13 MiB/s, 212 objects/s recovering 2024-03-30T13:55:11.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:11 smithi043 ceph-mon[42099]: pgmap v530: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 905 B/s rd, 0 B/s wr, 425 op/s; 18867/190158 objects degraded (9.922%); 30680/190158 objects misplaced (16.134%); 13 MiB/s, 212 objects/s recovering 2024-03-30T13:55:11.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:11 smithi113 ceph-mon[34341]: pgmap v530: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 905 B/s rd, 0 B/s wr, 425 op/s; 18867/190158 objects degraded (9.922%); 30680/190158 objects misplaced (16.134%); 13 MiB/s, 212 objects/s recovering 2024-03-30T13:55:11.484 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:11.508 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:11.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:11.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:11.610 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:11.634 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:11.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:11.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:11.736 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:11.761 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:11.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:11.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:11.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:11.904 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:11.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:11.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:12.006 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:12.051 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:12.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:12.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:12.153 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:12.178 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:12.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:12.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:12.280 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:12.306 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:12.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:12.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:12.408 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:12.431 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:12.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:12.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:12.533 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:12.573 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:12.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:12.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:12.674 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:12.699 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:12.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:12.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:12.801 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:12.825 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:12.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:12.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:12.927 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:12.968 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:12.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:12.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:13.069 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:13.098 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:13.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:13.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:13.200 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:13.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:13.239 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:13.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:13.340 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:13.380 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:13.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:13.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:13.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:13 smithi043 ceph-mon[29963]: pgmap v531: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 532 op/s; 18333/181110 objects degraded (10.123%); 28849/181110 objects misplaced (15.929%); 8.0 MiB/s, 128 objects/s recovering 2024-03-30T13:55:13.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:13 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 18333/181110 objects degraded (10.123%), 14 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:13.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:13 smithi043 ceph-mon[29963]: osdmap e258: 8 total, 7 up, 6 in 2024-03-30T13:55:13.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:13 smithi043 ceph-mon[42099]: pgmap v531: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 532 op/s; 18333/181110 objects degraded (10.123%); 28849/181110 objects misplaced (15.929%); 8.0 MiB/s, 128 objects/s recovering 2024-03-30T13:55:13.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:13 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 18333/181110 objects degraded (10.123%), 14 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:13.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:13 smithi043 ceph-mon[42099]: osdmap e258: 8 total, 7 up, 6 in 2024-03-30T13:55:13.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:13 smithi113 ceph-mon[34341]: pgmap v531: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 532 op/s; 18333/181110 objects degraded (10.123%); 28849/181110 objects misplaced (15.929%); 8.0 MiB/s, 128 objects/s recovering 2024-03-30T13:55:13.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:13 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 18333/181110 objects degraded (10.123%), 14 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:13.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:13 smithi113 ceph-mon[34341]: osdmap e258: 8 total, 7 up, 6 in 2024-03-30T13:55:13.481 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:13.505 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:13.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:13.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:13.607 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:13.644 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:13.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:13.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:13.746 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:13.770 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:13.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:13.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:13.872 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:13.897 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:13.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:13.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:13.998 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:14.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:14.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:14.149 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:14.174 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:14.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:14.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:14.276 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:14.322 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:14.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:14.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:14.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:14 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:55:14.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:14 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:55:14.424 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:14.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:14.467 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:14.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:14.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:14 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:55:14.568 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:14.593 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:14.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:14.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:14.695 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:14.721 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:14.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:14.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:14.823 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:14.862 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:14.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:14.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:14.964 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:14.990 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:14.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:14.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:15.092 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:15.127 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:15.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:15.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:15.229 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:15.254 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:15.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:15.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:15.356 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:15.382 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:15.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:15.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:15.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:15 smithi043 ceph-mon[29963]: pgmap v533: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 846 B/s rd, 0 B/s wr, 529 op/s; 18333/181110 objects degraded (10.123%); 28849/181110 objects misplaced (15.929%); 8.0 MiB/s, 127 objects/s recovering 2024-03-30T13:55:15.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:15 smithi043 ceph-mon[42099]: pgmap v533: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 846 B/s rd, 0 B/s wr, 529 op/s; 18333/181110 objects degraded (10.123%); 28849/181110 objects misplaced (15.929%); 8.0 MiB/s, 127 objects/s recovering 2024-03-30T13:55:15.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:15 smithi113 ceph-mon[34341]: pgmap v533: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 846 B/s rd, 0 B/s wr, 529 op/s; 18333/181110 objects degraded (10.123%); 28849/181110 objects misplaced (15.929%); 8.0 MiB/s, 127 objects/s recovering 2024-03-30T13:55:15.484 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:15.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:15.510 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:15.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:15.611 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:15.637 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:15.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:15.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:15.738 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:15.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:15.775 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:15.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:15.876 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:15.901 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:15.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:15.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:16.003 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:16.036 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:16.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:16.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:16.138 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:16.163 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:16.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:16.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:16.265 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:16.309 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:16.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:16.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:16.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:16 smithi043 ceph-mon[29963]: pgmap v534: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 622 op/s; 17944/175752 objects degraded (10.210%); 27584/175752 objects misplaced (15.695%); 12 MiB/s, 191 objects/s recovering 2024-03-30T13:55:16.395 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:55:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:55:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:55:16.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:16 smithi043 ceph-mon[42099]: pgmap v534: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 622 op/s; 17944/175752 objects degraded (10.210%); 27584/175752 objects misplaced (15.695%); 12 MiB/s, 191 objects/s recovering 2024-03-30T13:55:16.411 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:16.419 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:55:16.419 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-30T13:55:16.419 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:55:16.436 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:16.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:16.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:16.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:16 smithi113 ceph-mon[34341]: pgmap v534: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 622 op/s; 17944/175752 objects degraded (10.210%); 27584/175752 objects misplaced (15.695%); 12 MiB/s, 191 objects/s recovering 2024-03-30T13:55:16.538 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:16.577 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:16.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:16.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:16.679 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:16.706 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:16.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:16.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:16.807 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:16.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:16.936 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:16.961 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:16.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:16.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:17.062 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:17.101 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:17.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:17.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:17.203 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:17.229 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:17.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:17.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:17.331 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:17.354 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:17.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:17.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:17.456 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:17.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:17.503 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:17.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:17.604 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:17.628 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:17.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:17.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:17.730 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:17.754 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:17.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:17.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:17.856 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:17.879 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:17.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:17.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:17.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:17 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 17944/175752 objects degraded (10.210%), 14 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:17.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:17 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 17944/175752 objects degraded (10.210%), 14 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:17.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:17 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 17944/175752 objects degraded (10.210%), 14 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:17.981 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:18.006 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:18.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:18.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:18.108 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:18.131 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:18.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:18.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:18.233 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:18.260 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:18.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:18.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:18.361 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:18.404 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:18.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:18.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:18.506 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:18.546 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:18.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:18.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:18.622 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:55:18.648 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:18.673 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:18.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:18.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:18.774 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:18.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:18.828 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:18.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:18.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:18 smithi043 ceph-mon[29963]: pgmap v535: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 685 op/s; 17742/172458 objects degraded (10.288%); 26809/172458 objects misplaced (15.545%); 13 MiB/s, 202 objects/s recovering 2024-03-30T13:55:18.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:18 smithi043 ceph-mon[42099]: pgmap v535: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 685 op/s; 17742/172458 objects degraded (10.288%); 26809/172458 objects misplaced (15.545%); 13 MiB/s, 202 objects/s recovering 2024-03-30T13:55:18.930 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:18.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:18 smithi113 ceph-mon[34341]: pgmap v535: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 685 op/s; 17742/172458 objects degraded (10.288%); 26809/172458 objects misplaced (15.545%); 13 MiB/s, 202 objects/s recovering 2024-03-30T13:55:18.971 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:18.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:18.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:19.073 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:19.114 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:19.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:19.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:19.216 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:19.241 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:19.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:19.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:19.342 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:19.369 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:19.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:19.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:19.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:55:18 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:55:19.486 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:19.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:19.541 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:19.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:19.642 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:19.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:19.674 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:19.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:19.775 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:19.801 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:19.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:19.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:19 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:55:19.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:19 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:55:19.903 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:19.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:19.949 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:19.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:19.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:19 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:55:20.049 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:20.079 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:20.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:20.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:20.181 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:20.224 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:20.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:20.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:20.325 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:20.351 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:20.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:20.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:20.453 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:20.478 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:20.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:20.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:20.580 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:20.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:20.608 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:20.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:20.709 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:20.753 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:20.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:20.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:20.854 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:20.891 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:20.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:20.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:20.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:20 smithi043 ceph-mon[29963]: pgmap v536: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 609 op/s; 17742/171891 objects degraded (10.322%); 26809/171891 objects misplaced (15.597%); 11 MiB/s, 174 objects/s recovering 2024-03-30T13:55:20.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:20 smithi043 ceph-mon[42099]: pgmap v536: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 609 op/s; 17742/171891 objects degraded (10.322%); 26809/171891 objects misplaced (15.597%); 11 MiB/s, 174 objects/s recovering 2024-03-30T13:55:20.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:20 smithi113 ceph-mon[34341]: pgmap v536: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 609 op/s; 17742/171891 objects degraded (10.322%); 26809/171891 objects misplaced (15.597%); 11 MiB/s, 174 objects/s recovering 2024-03-30T13:55:20.993 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:21.031 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:21.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:21.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:21.132 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:21.158 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:21.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:21.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:21.260 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:21.286 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:21.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:21.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:21.388 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:21.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:21.414 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:21.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:21.515 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:21.538 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:21.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:21.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:21.640 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:21.667 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:21.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:21.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:21.768 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:21.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:21.792 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:21.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:21.893 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:21.918 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:21.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:21.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:22.021 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:22.046 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:22.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:22.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:22.148 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:22.175 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:22.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:22.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:22.277 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:22.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:22.308 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:22.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:22.409 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:22.448 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:22.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:22.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:22.550 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:22.581 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:22.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:22.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:22.682 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:22.710 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:22.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:22.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:22.812 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:22.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:22.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:22.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:22 smithi043 ceph-mon[29963]: pgmap v537: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 584 op/s; 17206/163596 objects degraded (10.517%); 24678/163596 objects misplaced (15.085%); 12 MiB/s, 194 objects/s recovering 2024-03-30T13:55:22.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:22 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 17206/163596 objects degraded (10.517%), 14 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:22.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:22 smithi043 ceph-mon[42099]: pgmap v537: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 584 op/s; 17206/163596 objects degraded (10.517%); 24678/163596 objects misplaced (15.085%); 12 MiB/s, 194 objects/s recovering 2024-03-30T13:55:22.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:22 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 17206/163596 objects degraded (10.517%), 14 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:22.938 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:22.962 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:22.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:22.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:22.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:22 smithi113 ceph-mon[34341]: pgmap v537: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 584 op/s; 17206/163596 objects degraded (10.517%); 24678/163596 objects misplaced (15.085%); 12 MiB/s, 194 objects/s recovering 2024-03-30T13:55:22.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:22 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 17206/163596 objects degraded (10.517%), 14 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:23.064 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:23.089 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:23.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:23.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:23.190 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:23.214 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:23.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:23.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:23.316 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:23.339 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:23.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:23.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:23.441 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:23.476 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:23.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:23.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:23.578 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:23.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:23.619 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:23.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:23.720 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:23.743 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:23.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:23.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:23.845 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:23.885 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:23.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:23.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:23.987 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:24.013 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:24.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:24.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:24.115 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:24.140 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:24.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:24.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:24.242 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:24.288 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:24.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:24.390 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:24.413 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:24.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:24.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:24.515 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:24.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:24.567 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:24.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:24.668 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:24.694 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:24.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:24.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:24.796 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:24.819 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:24.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:24.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:24.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:24 smithi043 ceph-mon[42099]: pgmap v538: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 927 B/s rd, 0 B/s wr, 529 op/s; 17206/163596 objects degraded (10.517%); 24678/163596 objects misplaced (15.085%); 11 MiB/s, 175 objects/s recovering 2024-03-30T13:55:24.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:24 smithi043 ceph-mon[29963]: pgmap v538: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 927 B/s rd, 0 B/s wr, 529 op/s; 17206/163596 objects degraded (10.517%); 24678/163596 objects misplaced (15.085%); 11 MiB/s, 175 objects/s recovering 2024-03-30T13:55:24.921 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:24.967 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:24.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:24 smithi113 ceph-mon[34341]: pgmap v538: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 927 B/s rd, 0 B/s wr, 529 op/s; 17206/163596 objects degraded (10.517%); 24678/163596 objects misplaced (15.085%); 11 MiB/s, 175 objects/s recovering 2024-03-30T13:55:24.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:24.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:25.070 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:25.096 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:25.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:25.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:25.198 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:25.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:25.240 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:25.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:25.341 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:25.365 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:25.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:25.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:25.467 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:25.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:25.512 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:25.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:25.613 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:25.638 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:25.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:25.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:25.740 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:25.765 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:25.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:25.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:25.867 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:25.907 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:25.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:25.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:26.009 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:26.015 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:55:26.033 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:26.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:26.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:26.135 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:26.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:26.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:26.293 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:26.316 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:26.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:26.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:26.395 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:55:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:55:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:55:26.418 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:26.461 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:26.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:26.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:26.562 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:26.587 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:26.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:26.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:26.688 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:26.715 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:26.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:26.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:26.817 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:26.844 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:26.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:26.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:26.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:26 smithi043 ceph-mon[29963]: pgmap v539: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 627 op/s; 16860/158550 objects degraded (10.634%); 23465/158550 objects misplaced (14.800%); 14 MiB/s, 223 objects/s recovering 2024-03-30T13:55:26.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:26 smithi043 ceph-mon[42099]: pgmap v539: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 627 op/s; 16860/158550 objects degraded (10.634%); 23465/158550 objects misplaced (14.800%); 14 MiB/s, 223 objects/s recovering 2024-03-30T13:55:26.945 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:26.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:26 smithi113 ceph-mon[34341]: pgmap v539: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 627 op/s; 16860/158550 objects degraded (10.634%); 23465/158550 objects misplaced (14.800%); 14 MiB/s, 223 objects/s recovering 2024-03-30T13:55:26.981 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:26.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:27.084 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:27.109 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:27.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:27.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:27.211 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:27.240 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:27.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:27.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:27.342 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:27.378 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:27.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:27.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:27.480 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:27.506 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:27.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:27.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:27.608 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:27.632 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:27.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:27.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:27.734 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:27.760 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:27.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:27.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:27.862 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:27.889 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:27.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:27.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:27.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:27 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 16860/158550 objects degraded (10.634%), 14 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:27.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:27 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 16860/158550 objects degraded (10.634%), 14 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:27.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:27 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 16860/158550 objects degraded (10.634%), 14 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:27.991 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:28.034 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:28.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:28.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:28.136 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:28.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:28.175 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:28.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:28.276 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:28.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:28.328 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:28.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:28.429 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:28.453 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:28.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:28.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:28.555 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:28.599 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:28.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:28.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:28.701 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:28.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:28.746 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:28.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:28.847 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:28.887 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:28.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:28.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:28.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:28 smithi043 ceph-mon[29963]: pgmap v540: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 560 op/s; 16695/155598 objects degraded (10.730%); 22838/155598 objects misplaced (14.678%); 11 MiB/s, 172 objects/s recovering 2024-03-30T13:55:28.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:28 smithi043 ceph-mon[42099]: pgmap v540: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 560 op/s; 16695/155598 objects degraded (10.730%); 22838/155598 objects misplaced (14.678%); 11 MiB/s, 172 objects/s recovering 2024-03-30T13:55:28.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:28 smithi113 ceph-mon[34341]: pgmap v540: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 560 op/s; 16695/155598 objects degraded (10.730%); 22838/155598 objects misplaced (14.678%); 11 MiB/s, 172 objects/s recovering 2024-03-30T13:55:28.989 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:29.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:29.039 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:29.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:29.140 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:29.164 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:29.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:29.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:29.266 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:29.302 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:29.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:29.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:29.404 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:29.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:29.431 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:29.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:29.467 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:55:29 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:55:29.532 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:29.557 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:29.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:29.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:29.659 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:29.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:29.685 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:29.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:29.786 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:29.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:29.812 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:29.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:29.848 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:29 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:55:29.848 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:29 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:55:29.849 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:29 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:55:29.849 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:29 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:55:29.912 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:29.952 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:29.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:29.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:29.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:29 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:55:29.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:29 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:55:30.054 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:30.090 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:30.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:30.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:30.192 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:30.216 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:30.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:30.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:30.319 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:30.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:30.353 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:30.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:30.455 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:30.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:30.494 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:30.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:30.595 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:30.618 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:30.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:30.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:30.624 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:55:30.666 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:30 smithi043 ceph-mon[29963]: pgmap v541: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 485 op/s; 16695/154998 objects degraded (10.771%); 22838/154998 objects misplaced (14.734%); 9.7 MiB/s, 155 objects/s recovering 2024-03-30T13:55:30.666 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:30 smithi043 ceph-mon[29963]: osdmap e259: 8 total, 7 up, 6 in 2024-03-30T13:55:30.667 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:30 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/58707326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:55:30.667 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:30 smithi043 ceph-mon[42099]: pgmap v541: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 485 op/s; 16695/154998 objects degraded (10.771%); 22838/154998 objects misplaced (14.734%); 9.7 MiB/s, 155 objects/s recovering 2024-03-30T13:55:30.667 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:30 smithi043 ceph-mon[42099]: osdmap e259: 8 total, 7 up, 6 in 2024-03-30T13:55:30.667 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:30 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/58707326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:55:30.670 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:55:30.720 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:30.734 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:55:30.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:30.756 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:30.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:30.777 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:55:30.816 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:55:30.854 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:55:30.857 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:30.898 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:30.898 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:55:30.898 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:55:30.899 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:55:30.899 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:55:30.899 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:55:30.899 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:55:30.899 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:55:30.899 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:55:30.899 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:55:30.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:30.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:30.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:30 smithi113 ceph-mon[34341]: pgmap v541: 158 pgs: 83 active+clean, 2 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 485 op/s; 16695/154998 objects degraded (10.771%); 22838/154998 objects misplaced (14.734%); 9.7 MiB/s, 155 objects/s recovering 2024-03-30T13:55:30.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:30 smithi113 ceph-mon[34341]: osdmap e259: 8 total, 7 up, 6 in 2024-03-30T13:55:30.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:30 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/58707326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:55:31.000 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:31.002 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6] out_osds: [5, 7] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-30T13:55:31.002 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:55:31.003 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_2 2024-03-30T13:55:31.003 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T13:55:31.025 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:31.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:31.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:31.127 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:31.152 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:31.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:31.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:31.254 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:31.278 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:31.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:31.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:31.380 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:31.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:31.406 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:31.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:31.507 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:31.532 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:31.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:31.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:31.634 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:31.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:31 smithi043 ceph-mon[29963]: osdmap e260: 8 total, 7 up, 6 in 2024-03-30T13:55:31.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:31 smithi043 ceph-mon[42099]: osdmap e260: 8 total, 7 up, 6 in 2024-03-30T13:55:31.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:31.683 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:31.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:31.782 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:55:31.782 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:55:31.785 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:31.810 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:31.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:31.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:31.912 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:31.958 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:31.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:31.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:31.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:31 smithi113 ceph-mon[34341]: osdmap e260: 8 total, 7 up, 6 in 2024-03-30T13:55:32.060 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:32.086 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:32.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:32.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:32.188 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:32.213 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:32.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:32.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:32.314 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:32.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:32.341 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:32.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:32.442 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:32.467 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:32.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:32.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:32.569 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:32.595 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:32.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:32.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:32.697 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:32.757 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:32.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:32.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:32.858 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:32.872 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:32 smithi043 ceph-mon[29963]: pgmap v544: 158 pgs: 1 peering, 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 681 op/s; 14763/147261 objects degraded (10.025%); 21795/147261 objects misplaced (14.800%); 17 MiB/s, 271 objects/s recovering 2024-03-30T13:55:32.872 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:32 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 14763/147261 objects degraded (10.025%), 14 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:32.872 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:32 smithi043 ceph-mon[42099]: pgmap v544: 158 pgs: 1 peering, 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 681 op/s; 14763/147261 objects degraded (10.025%); 21795/147261 objects misplaced (14.800%); 17 MiB/s, 271 objects/s recovering 2024-03-30T13:55:32.872 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:32 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 14763/147261 objects degraded (10.025%), 14 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:32.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:32.910 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:32.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:32.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:32 smithi113 ceph-mon[34341]: pgmap v544: 158 pgs: 1 peering, 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 681 op/s; 14763/147261 objects degraded (10.025%); 21795/147261 objects misplaced (14.800%); 17 MiB/s, 271 objects/s recovering 2024-03-30T13:55:32.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:32 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 14763/147261 objects degraded (10.025%), 14 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:33.010 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:33.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:33.036 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:33.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:33.137 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:33.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:33.164 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:33.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:33.230 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:55:33.265 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:33.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:33.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:33.425 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:33.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:33.452 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:33.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:33.553 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:33.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:33.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:33.655 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T13:55:33.677 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:33.703 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:33.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:33.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:33.784 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:55:33.805 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:33.828 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:33.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:33.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:33.930 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:33.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:33 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1996689363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:55:33.971 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:33.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:33.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:34.074 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:34.141 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:34.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:34.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:34.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:33 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1996689363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:55:34.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:33 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1996689363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:55:34.244 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:34.270 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:34.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:34.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:34.372 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:34.415 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:34.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:34.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:34.517 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:34.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:34.559 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:34.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:34.660 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:34.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:34.703 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:34.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:34.804 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:34.856 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:34.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:34.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:34.958 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:34.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:34 smithi113 ceph-mon[34341]: pgmap v545: 158 pgs: 1 peering, 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 471 op/s; 14763/147261 objects degraded (10.025%); 21795/147261 objects misplaced (14.800%); 11 MiB/s, 179 objects/s recovering 2024-03-30T13:55:34.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:34 smithi113 ceph-mon[34341]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:55:34.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:34 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1996689363' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:55:34.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:34 smithi113 ceph-mon[34341]: osdmap e261: 8 total, 7 up, 6 in 2024-03-30T13:55:34.981 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:34.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:34.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:35.084 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:35.109 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:35.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:35.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:35.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:34 smithi043 ceph-mon[29963]: pgmap v545: 158 pgs: 1 peering, 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 471 op/s; 14763/147261 objects degraded (10.025%); 21795/147261 objects misplaced (14.800%); 11 MiB/s, 179 objects/s recovering 2024-03-30T13:55:35.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:34 smithi043 ceph-mon[29963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:55:35.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:34 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1996689363' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:55:35.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:34 smithi043 ceph-mon[29963]: osdmap e261: 8 total, 7 up, 6 in 2024-03-30T13:55:35.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:34 smithi043 ceph-mon[42099]: pgmap v545: 158 pgs: 1 peering, 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 1 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 3.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 471 op/s; 14763/147261 objects degraded (10.025%); 21795/147261 objects misplaced (14.800%); 11 MiB/s, 179 objects/s recovering 2024-03-30T13:55:35.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:34 smithi043 ceph-mon[42099]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:55:35.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:34 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1996689363' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:55:35.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:34 smithi043 ceph-mon[42099]: osdmap e261: 8 total, 7 up, 6 in 2024-03-30T13:55:35.210 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:35.276 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:35.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:35.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:35.378 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:35.404 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:35.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:35.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:35.505 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:35.535 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:35.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:35.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:35.637 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:35.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:35.664 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:35.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:35.765 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:35.790 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:35.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:35.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:35.893 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:35.917 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:35.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:35.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:36.018 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:36.043 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:36.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:36.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:36.145 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:36.170 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:36.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:36.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:36.272 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:36.295 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:36.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:36.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:36.397 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:36.398 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:55:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:55:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:55:36.474 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:36.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:36.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:36.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:36.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:36.639 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:36.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:36.740 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:36.821 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:36.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:36.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:36.923 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:36.949 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:36.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:36.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:36.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:36 smithi113 ceph-mon[34341]: pgmap v547: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 745 op/s; 14391/141594 objects degraded (10.164%); 20481/141594 objects misplaced (14.465%); 22 MiB/s, 345 objects/s recovering 2024-03-30T13:55:36.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:36 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:55:37.051 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:37.077 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:37.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:37.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:37.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:36 smithi043 ceph-mon[29963]: pgmap v547: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 745 op/s; 14391/141594 objects degraded (10.164%); 20481/141594 objects misplaced (14.465%); 22 MiB/s, 345 objects/s recovering 2024-03-30T13:55:37.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:36 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:55:37.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:36 smithi043 ceph-mon[42099]: pgmap v547: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 745 op/s; 14391/141594 objects degraded (10.164%); 20481/141594 objects misplaced (14.465%); 22 MiB/s, 345 objects/s recovering 2024-03-30T13:55:37.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:36 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:55:37.179 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:37.201 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:37.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:37.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:37.303 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:37.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:37.354 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:37.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:37.455 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:37.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:37.481 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:37.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:37.582 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:37.636 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:37.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:37.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:37.738 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:37.782 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:37.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:37.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:37.884 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:37.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:37 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:55:37.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:37 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:55:37.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:37 smithi043 ceph-mon[29963]: osdmap e262: 8 total, 7 up, 6 in 2024-03-30T13:55:37.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:37 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:37.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:37 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:55:37.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:37 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:55:37.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:37 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 14391/141594 objects degraded (10.164%), 14 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:37.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:37 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:55:37.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:37 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:55:37.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:37 smithi043 ceph-mon[42099]: osdmap e262: 8 total, 7 up, 6 in 2024-03-30T13:55:37.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:37 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:37.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:37 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:55:37.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:37 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:55:37.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:37 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 14391/141594 objects degraded (10.164%), 14 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:37.908 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:37.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:37.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:37.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:37 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:55:37.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:37 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:55:37.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:37 smithi113 ceph-mon[34341]: osdmap e262: 8 total, 7 up, 6 in 2024-03-30T13:55:37.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:37 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:37.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:37 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:55:37.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:37 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:55:37.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:37 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 14391/141594 objects degraded (10.164%), 14 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:38.011 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:38.052 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:38.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:38.153 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:38.178 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:38.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:38.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:38.280 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:38.321 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:38.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:38.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:38.423 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:38.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:38.463 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:38.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:38.564 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:38.586 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:55:38.586 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T13:55:38.586 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-03-30T13:55:38.586 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-03-30T13:55:38.587 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:55:38.589 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:38.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:38.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:38.690 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:38.729 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:38 smithi043 ceph-mon[29963]: pgmap v549: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 588 op/s; 14199/137961 objects degraded (10.292%); 19747/137961 objects misplaced (14.313%); 11 MiB/s, 173 objects/s recovering 2024-03-30T13:55:38.729 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:38 smithi043 ceph-mon[42099]: pgmap v549: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 588 op/s; 14199/137961 objects degraded (10.292%); 19747/137961 objects misplaced (14.313%); 11 MiB/s, 173 objects/s recovering 2024-03-30T13:55:38.748 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:38.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:38.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:38.850 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:38.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:38.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:38.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:38 smithi113 ceph-mon[34341]: pgmap v549: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 588 op/s; 14199/137961 objects degraded (10.292%); 19747/137961 objects misplaced (14.313%); 11 MiB/s, 173 objects/s recovering 2024-03-30T13:55:39.000 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:39.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:39.047 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:39.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:39.148 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:39.173 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:39.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:39.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:39.275 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:39.302 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:39.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:39.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:39.404 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:39.430 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:39.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:39.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:39.467 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:55:39 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:55:39.532 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:39.573 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:39.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:39.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:39.675 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:39.713 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:39.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:39.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:39.815 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:39.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:39.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:39.966 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:39.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:39 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:55:39.986 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:39 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:55:39.987 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:39 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:55:40.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:40.011 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:40.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:40.112 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:40.137 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:40.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:40.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:40.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T13:55:40.239 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:40.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:40.265 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:40.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:40.366 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:40.453 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:40.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:40.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:40.557 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:40.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:40.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:40.724 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:40.749 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:40.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:40.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:40.851 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:40.877 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:40.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:40.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:40.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:40 smithi043 ceph-mon[29963]: pgmap v550: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 410 op/s; 14199/137415 objects degraded (10.333%); 19747/137415 objects misplaced (14.370%); 7.5 MiB/s, 120 objects/s recovering 2024-03-30T13:55:40.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:40 smithi043 ceph-mon[42099]: pgmap v550: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 410 op/s; 14199/137415 objects degraded (10.333%); 19747/137415 objects misplaced (14.370%); 7.5 MiB/s, 120 objects/s recovering 2024-03-30T13:55:40.957 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:55:40.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:40 smithi113 ceph-mon[34341]: pgmap v550: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 410 op/s; 14199/137415 objects degraded (10.333%); 19747/137415 objects misplaced (14.370%); 7.5 MiB/s, 120 objects/s recovering 2024-03-30T13:55:40.979 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:41.000 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-03-30T13:55:41.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:41.055 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:41.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:41.156 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:41.181 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:41.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:41.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:41.283 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:41.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:41.310 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:41.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:41.411 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:41.435 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:41.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:41.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:41.516 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:55:41.536 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:41.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:41.619 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:41.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:41.720 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:41.745 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:41.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:41.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:41.847 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:41.871 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:41.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:41.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:41.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:42.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:42.016 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:42.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:42.050 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:55:42.063 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-30T13:55:42.117 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:42.175 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:42.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:42.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:42.277 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:42.331 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:42.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:42.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:42.432 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:42.458 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:42.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:42.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:42.560 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:42.589 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:42.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:42.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:42.691 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:42.717 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:42.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:42.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:42.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:42.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:42.859 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:42.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:42.960 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:42.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:42 smithi113 ceph-mon[34341]: pgmap v551: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 771 op/s; 13641/128742 objects degraded (10.596%); 17461/128742 objects misplaced (13.563%); 16 MiB/s, 256 objects/s recovering 2024-03-30T13:55:42.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:42 smithi113 ceph-mon[34341]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:55:42.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:42 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 13641/128742 objects degraded (10.596%), 14 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:42.986 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:42.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:43.088 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:43.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:43.140 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:43.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:43.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:42 smithi043 ceph-mon[29963]: pgmap v551: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 771 op/s; 13641/128742 objects degraded (10.596%); 17461/128742 objects misplaced (13.563%); 16 MiB/s, 256 objects/s recovering 2024-03-30T13:55:43.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:42 smithi043 ceph-mon[29963]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:55:43.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:42 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 13641/128742 objects degraded (10.596%), 14 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:43.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:42 smithi043 ceph-mon[42099]: pgmap v551: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 771 op/s; 13641/128742 objects degraded (10.596%); 17461/128742 objects misplaced (13.563%); 16 MiB/s, 256 objects/s recovering 2024-03-30T13:55:43.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:42 smithi043 ceph-mon[42099]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:55:43.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:42 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 13641/128742 objects degraded (10.596%), 14 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:43.241 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:43.268 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:43.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:43.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:43.370 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:43.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:43.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:43.512 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:43.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:43.559 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:43.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:43.660 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:43.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:43.706 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:43.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:43.807 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:43.854 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:43.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:43.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:43.956 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:43.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:43 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:55:43.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:43 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:55:43.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:43 smithi113 ceph-mon[34341]: osdmap e263: 8 total, 7 up, 6 in 2024-03-30T13:55:43.980 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:43.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:43.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:44.082 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:44.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:44.141 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:44.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:44.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:43 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:55:44.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:43 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:55:44.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:43 smithi043 ceph-mon[29963]: osdmap e263: 8 total, 7 up, 6 in 2024-03-30T13:55:44.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:43 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:55:44.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:43 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:55:44.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:43 smithi043 ceph-mon[42099]: osdmap e263: 8 total, 7 up, 6 in 2024-03-30T13:55:44.242 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:44.288 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:44.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:44.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:44.389 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:44.414 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:44.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:44.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:44.516 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:44.543 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:44.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:44.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:44.645 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:44.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:44.672 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:44.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:44.773 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:44.800 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:44.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:44.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:44.902 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:44.926 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:44.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:44.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:44.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:44 smithi113 ceph-mon[34341]: pgmap v553: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 536 op/s; 13641/128742 objects degraded (10.596%); 17461/128742 objects misplaced (13.563%); 9.8 MiB/s, 156 objects/s recovering 2024-03-30T13:55:44.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:44 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:55:45.028 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:45.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:45.056 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:45.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:45.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:44 smithi043 ceph-mon[29963]: pgmap v553: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 536 op/s; 13641/128742 objects degraded (10.596%); 17461/128742 objects misplaced (13.563%); 9.8 MiB/s, 156 objects/s recovering 2024-03-30T13:55:45.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:44 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:55:45.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:44 smithi043 ceph-mon[42099]: pgmap v553: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 536 op/s; 13641/128742 objects degraded (10.596%); 17461/128742 objects misplaced (13.563%); 9.8 MiB/s, 156 objects/s recovering 2024-03-30T13:55:45.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:44 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:55:45.157 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:45.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:45.198 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:45.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:45.299 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:45.340 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:45.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:45.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:45.442 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:45.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:45.466 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:45.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:45.567 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:45.591 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:45.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:45.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:45.693 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:45.718 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:45.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:45.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:45.821 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:45.847 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:45.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:45.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:45.948 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:45.973 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:45.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:45.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:46.075 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:46.115 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:46.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:46.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:46.217 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:46.242 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:46.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:46.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:46.343 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:46.369 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:46.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:46.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:46.394 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:55:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:55:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:55:46.471 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:46.514 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:46.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:46.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:46.616 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:46.641 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:46.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:46.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:46.743 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:46.770 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:46.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:46.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:46.872 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:46.898 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:46.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:46.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:46.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:46 smithi113 ceph-mon[34341]: pgmap v554: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 573 B/s rd, 0 B/s wr, 437 op/s; 13641/126243 objects degraded (10.805%); 16462/126243 objects misplaced (13.040%); 10 MiB/s, 165 objects/s recovering 2024-03-30T13:55:47.000 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:47.025 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:47.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:47.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:47.127 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:47.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:46 smithi043 ceph-mon[29963]: pgmap v554: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 573 B/s rd, 0 B/s wr, 437 op/s; 13641/126243 objects degraded (10.805%); 16462/126243 objects misplaced (13.040%); 10 MiB/s, 165 objects/s recovering 2024-03-30T13:55:47.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:46 smithi043 ceph-mon[42099]: pgmap v554: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 573 B/s rd, 0 B/s wr, 437 op/s; 13641/126243 objects degraded (10.805%); 16462/126243 objects misplaced (13.040%); 10 MiB/s, 165 objects/s recovering 2024-03-30T13:55:47.153 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:47.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:47.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:47.255 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:47.300 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:47.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:47.401 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:47.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:47.427 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:47.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:47.529 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:47.580 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:47.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:47.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:47.681 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:47.709 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:47.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:47.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:47.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:47.841 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:47.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:47.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:47.942 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:47.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:47 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 13641/126243 objects degraded (10.805%), 14 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:47.968 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:47 smithi113 ceph-mon[34341]: osdmap e264: 8 total, 7 up, 6 in 2024-03-30T13:55:47.989 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:47.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:47.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:48.091 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:48.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:47 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 13641/126243 objects degraded (10.805%), 14 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:48.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:47 smithi043 ceph-mon[29963]: osdmap e264: 8 total, 7 up, 6 in 2024-03-30T13:55:48.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:47 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 13641/126243 objects degraded (10.805%), 14 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:48.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:47 smithi043 ceph-mon[42099]: osdmap e264: 8 total, 7 up, 6 in 2024-03-30T13:55:48.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:48.147 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:48.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:48.248 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:48.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:48.274 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:48.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:48.375 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:48.418 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:48.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:48.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:48.520 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:48.546 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:48.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:48.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:48.648 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:48.693 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:48.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:48.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:48.795 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:48.820 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:48.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:48.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:48.922 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:48.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:48.966 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:48.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:49.027 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:48 smithi113 ceph-mon[34341]: pgmap v556: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.4 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 760 op/s; 13122/119193 objects degraded (11.009%); 15124/119193 objects misplaced (12.689%); 17 MiB/s, 270 objects/s recovering 2024-03-30T13:55:49.067 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:49.094 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:49.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:49.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:49.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:48 smithi043 ceph-mon[29963]: pgmap v556: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.4 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 760 op/s; 13122/119193 objects degraded (11.009%); 15124/119193 objects misplaced (12.689%); 17 MiB/s, 270 objects/s recovering 2024-03-30T13:55:49.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:48 smithi043 ceph-mon[42099]: pgmap v556: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.4 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 760 op/s; 13122/119193 objects degraded (11.009%); 15124/119193 objects misplaced (12.689%); 17 MiB/s, 270 objects/s recovering 2024-03-30T13:55:49.196 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:49.240 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:49.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:49.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:49.341 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:49.367 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:49.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:49.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:49.467 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:55:49 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:55:49.469 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:49.495 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:49.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:49.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:49.597 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:49.620 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:49.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:49.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:49.722 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:49.771 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:49.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:49.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:49.874 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:49.916 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:49.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:49.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:50.017 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:50.044 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:50.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:50.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:50.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:49 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:55:50.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:49 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:55:50.146 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:50.194 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:50.227 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:49 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:55:50.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:50.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:50.329 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:50.373 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:50.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:50.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:50.475 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:50.501 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:50.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:50.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:50.605 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:50.650 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:50.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:50.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:50.751 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:50.777 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:50.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:50.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:50.878 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:50.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:50.905 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:50.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:51.006 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:51.031 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:51.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:51.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:51.133 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:51.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:50 smithi043 ceph-mon[29963]: pgmap v557: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.4 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 427 op/s; 13122/118497 objects degraded (11.074%); 15124/118497 objects misplaced (12.763%); 8.4 MiB/s, 134 objects/s recovering 2024-03-30T13:55:51.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:50 smithi043 ceph-mon[42099]: pgmap v557: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.4 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 427 op/s; 13122/118497 objects degraded (11.074%); 15124/118497 objects misplaced (12.763%); 8.4 MiB/s, 134 objects/s recovering 2024-03-30T13:55:51.160 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:51.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:51.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:51.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:50 smithi113 ceph-mon[34341]: pgmap v557: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.4 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 427 op/s; 13122/118497 objects degraded (11.074%); 15124/118497 objects misplaced (12.763%); 8.4 MiB/s, 134 objects/s recovering 2024-03-30T13:55:51.262 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:51.307 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:51.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:51.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:51.409 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:51.456 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:51.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:51.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:51.558 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:51.604 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:51.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:51.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:51.706 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:51.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:51.748 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:51.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:51.849 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:51.877 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:51.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:51.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:51.979 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:52.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:52.008 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:52.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:52.109 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:52.135 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:52.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:52.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:52.237 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:52.288 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:52.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:52.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:52.390 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:52.415 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:52.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:52.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:52.517 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:52.565 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:52.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:52.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:52.667 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:52.684 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:55:52.734 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:52.735 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":551,"stamp":"2024-03-30T13:55:41.585082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2799960376,"num_objects":42914,"num_object_clones":0,"num_object_copies":128742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13641,"num_objects_misplaced":17461,"num_objects_unfound":0,"num_objects_dirty":42914,"num_whiteouts":0,"num_read":1378,"num_read_kb":1182,"num_write":131733,"num_write_kb":5557394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33339,"num_bytes_recovered":2178601208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59448,"ondisk_log_size":59448,"up":406,"acting":447,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":451,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":11441536,"kb_used_data":8847052,"kb_used_omap":9,"kb_used_meta":2594294,"kb_avail":551029376,"statfs":{"total":575970213888,"available":564254081024,"internally_reserved":0,"allocated":9059381248,"data_stored":9054300153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":2656557729},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,5,7,1,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-404553728,"num_objects":-6173,"num_object_clones":0,"num_object_copies":-18519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1122,"num_objects_misplaced":-4334,"num_objects_unfound":0,"num_objects_dirty":-6173,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6173,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2052,"num_bytes_recovered":134414336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.009260"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":442,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.658843+0000","last_change":"2024-03-30T13:53:35.862417+0000","last_active":"2024-03-30T13:55:36.658843+0000","last_peered":"2024-03-30T13:55:36.658843+0000","last_clean":"2024-03-30T13:55:36.658843+0000","last_became_active":"2024-03-30T13:53:35.861853+0000","last_became_peered":"2024-03-30T13:53:35.861853+0000","last_unstale":"2024-03-30T13:55:36.658843+0000","last_undegraded":"2024-03-30T13:55:36.658843+0000","last_fullsized":"2024-03-30T13:55:36.658843+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:05.234023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:05.234023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":459,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.701134+0000","last_change":"2024-03-30T13:53:36.861076+0000","last_active":"2024-03-30T13:55:36.701134+0000","last_peered":"2024-03-30T13:55:36.701134+0000","last_clean":"2024-03-30T13:55:36.701134+0000","last_became_active":"2024-03-30T13:53:36.860312+0000","last_became_peered":"2024-03-30T13:53:36.860312+0000","last_unstale":"2024-03-30T13:55:36.701134+0000","last_undegraded":"2024-03-30T13:55:36.701134+0000","last_fullsized":"2024-03-30T13:55:36.701134+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:17.548567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:17.548567+0000","last_clean_scrub_stamp":"2024-03-30T13:52:17.548567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6,7],"avail_no_missing":["1","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"65'1","reported_seq":450,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.696438+0000","last_change":"2024-03-30T13:53:14.675272+0000","last_active":"2024-03-30T13:55:36.696438+0000","last_peered":"2024-03-30T13:55:36.696438+0000","last_clean":"2024-03-30T13:55:36.696438+0000","last_became_active":"2024-03-30T13:53:14.657693+0000","last_became_peered":"2024-03-30T13:53:14.657693+0000","last_unstale":"2024-03-30T13:55:36.696438+0000","last_undegraded":"2024-03-30T13:55:36.696438+0000","last_fullsized":"2024-03-30T13:55:36.696438+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:48:19.930059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:48:19.930059+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,5],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"216'15","reported_seq":471,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.658531+0000","last_change":"2024-03-30T13:52:56.028286+0000","last_active":"2024-03-30T13:55:36.658531+0000","last_peered":"2024-03-30T13:55:36.658531+0000","last_clean":"2024-03-30T13:55:36.658531+0000","last_became_active":"2024-03-30T13:42:47.319638+0000","last_became_peered":"2024-03-30T13:42:47.319638+0000","last_unstale":"2024-03-30T13:55:36.658531+0000","last_undegraded":"2024-03-30T13:55:36.658531+0000","last_fullsized":"2024-03-30T13:55:36.658531+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'15","last_scrub_stamp":"2024-03-30T13:52:56.028141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:56.028141+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":435,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.743735+0000","last_change":"2024-03-30T13:53:35.863312+0000","last_active":"2024-03-30T13:55:36.743735+0000","last_peered":"2024-03-30T13:55:36.743735+0000","last_clean":"2024-03-30T13:55:36.743735+0000","last_became_active":"2024-03-30T13:53:35.862085+0000","last_became_peered":"2024-03-30T13:53:35.862085+0000","last_unstale":"2024-03-30T13:55:36.743735+0000","last_undegraded":"2024-03-30T13:55:36.743735+0000","last_fullsized":"2024-03-30T13:55:36.743735+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:23.133743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:23.133743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":434,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.658619+0000","last_change":"2024-03-30T13:51:20.149796+0000","last_active":"2024-03-30T13:55:36.658619+0000","last_peered":"2024-03-30T13:55:36.658619+0000","last_clean":"2024-03-30T13:55:36.658619+0000","last_became_active":"2024-03-30T13:42:49.327124+0000","last_became_peered":"2024-03-30T13:42:49.327124+0000","last_unstale":"2024-03-30T13:55:36.658619+0000","last_undegraded":"2024-03-30T13:55:36.658619+0000","last_fullsized":"2024-03-30T13:55:36.658619+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:20.149684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:20.149684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":437,"reported_epoch":262,"state":"active+undersized","last_fresh":"2024-03-30T13:55:36.729795+0000","last_change":"2024-03-30T13:50:08.512316+0000","last_active":"2024-03-30T13:55:36.729795+0000","last_peered":"2024-03-30T13:55:36.729795+0000","last_clean":"2024-03-30T13:50:05.504688+0000","last_became_active":"2024-03-30T13:50:08.512316+0000","last_became_peered":"2024-03-30T13:50:08.512316+0000","last_unstale":"2024-03-30T13:55:36.729795+0000","last_undegraded":"2024-03-30T13:55:36.729795+0000","last_fullsized":"2024-03-30T13:50:08.501439+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:20.499768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:20.499768+0000","last_clean_scrub_stamp":"2024-03-30T13:49:20.499768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"217'34","reported_seq":474,"reported_epoch":262,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:55:36.725082+0000","last_change":"2024-03-30T13:50:08.511099+0000","last_active":"2024-03-30T13:55:36.725082+0000","last_peered":"2024-03-30T13:55:36.725082+0000","last_clean":"2024-03-30T13:50:05.499201+0000","last_became_active":"2024-03-30T13:50:08.511099+0000","last_became_peered":"2024-03-30T13:50:08.511099+0000","last_unstale":"2024-03-30T13:55:36.725082+0000","last_undegraded":"2024-03-30T13:50:08.502733+0000","last_fullsized":"2024-03-30T13:50:08.502606+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'22","last_scrub_stamp":"2024-03-30T13:46:49.339954+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:29.151791+0000","last_clean_scrub_stamp":"2024-03-30T13:46:49.339954+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":440,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.692685+0000","last_change":"2024-03-30T13:53:36.858395+0000","last_active":"2024-03-30T13:55:36.692685+0000","last_peered":"2024-03-30T13:55:36.692685+0000","last_clean":"2024-03-30T13:55:36.692685+0000","last_became_active":"2024-03-30T13:53:36.855710+0000","last_became_peered":"2024-03-30T13:53:36.855710+0000","last_unstale":"2024-03-30T13:55:36.692685+0000","last_undegraded":"2024-03-30T13:55:36.692685+0000","last_fullsized":"2024-03-30T13:55:36.692685+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:27.061281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:27.061281+0000","last_clean_scrub_stamp":"2024-03-30T13:52:27.061281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":98,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.695631+0000","last_change":"2024-03-30T13:53:35.857026+0000","last_active":"2024-03-30T13:55:36.695631+0000","last_peered":"2024-03-30T13:55:36.695631+0000","last_clean":"2024-03-30T13:55:36.695631+0000","last_became_active":"2024-03-30T13:53:35.856684+0000","last_became_peered":"2024-03-30T13:53:35.856684+0000","last_unstale":"2024-03-30T13:55:36.695631+0000","last_undegraded":"2024-03-30T13:55:36.695631+0000","last_fullsized":"2024-03-30T13:55:36.695631+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:20.069864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:20.069864+0000","last_clean_scrub_stamp":"2024-03-30T13:52:20.069864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":92,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.659885+0000","last_change":"2024-03-30T13:53:13.645288+0000","last_active":"2024-03-30T13:55:36.659885+0000","last_peered":"2024-03-30T13:55:36.659885+0000","last_clean":"2024-03-30T13:55:36.659885+0000","last_became_active":"2024-03-30T13:53:13.644821+0000","last_became_peered":"2024-03-30T13:53:13.644821+0000","last_unstale":"2024-03-30T13:55:36.659885+0000","last_undegraded":"2024-03-30T13:55:36.659885+0000","last_fullsized":"2024-03-30T13:55:36.659885+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:07.129142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:37.513596+0000","last_clean_scrub_stamp":"2024-03-30T13:52:07.129142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"216'15","reported_seq":471,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.659018+0000","last_change":"2024-03-30T13:51:16.153633+0000","last_active":"2024-03-30T13:55:36.659018+0000","last_peered":"2024-03-30T13:55:36.659018+0000","last_clean":"2024-03-30T13:55:36.659018+0000","last_became_active":"2024-03-30T13:42:47.310289+0000","last_became_peered":"2024-03-30T13:42:47.310289+0000","last_unstale":"2024-03-30T13:55:36.659018+0000","last_undegraded":"2024-03-30T13:55:36.659018+0000","last_fullsized":"2024-03-30T13:55:36.659018+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:51:16.153478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:16.153478+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":93,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.701592+0000","last_change":"2024-03-30T13:53:14.658585+0000","last_active":"2024-03-30T13:55:36.701592+0000","last_peered":"2024-03-30T13:55:36.701592+0000","last_clean":"2024-03-30T13:55:36.701592+0000","last_became_active":"2024-03-30T13:53:14.658167+0000","last_became_peered":"2024-03-30T13:53:14.658167+0000","last_unstale":"2024-03-30T13:55:36.701592+0000","last_undegraded":"2024-03-30T13:55:36.701592+0000","last_fullsized":"2024-03-30T13:55:36.701592+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:10.988839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:10.988839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3,5],"avail_no_missing":["1","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":78,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.659324+0000","last_change":"2024-03-30T13:53:35.862161+0000","last_active":"2024-03-30T13:55:36.659324+0000","last_peered":"2024-03-30T13:55:36.659324+0000","last_clean":"2024-03-30T13:55:36.659324+0000","last_became_active":"2024-03-30T13:53:35.861405+0000","last_became_peered":"2024-03-30T13:53:35.861405+0000","last_unstale":"2024-03-30T13:55:36.659324+0000","last_undegraded":"2024-03-30T13:55:36.659324+0000","last_fullsized":"2024-03-30T13:55:36.659324+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:50.953533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:41.633284+0000","last_clean_scrub_stamp":"2024-03-30T13:52:50.953533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"186'2","reported_seq":391,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.659333+0000","last_change":"2024-03-30T13:53:13.664011+0000","last_active":"2024-03-30T13:55:36.659333+0000","last_peered":"2024-03-30T13:55:36.659333+0000","last_clean":"2024-03-30T13:55:36.659333+0000","last_became_active":"2024-03-30T13:53:13.646659+0000","last_became_peered":"2024-03-30T13:53:13.646659+0000","last_unstale":"2024-03-30T13:55:36.659333+0000","last_undegraded":"2024-03-30T13:55:36.659333+0000","last_fullsized":"2024-03-30T13:55:36.659333+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-30T13:51:15.932076+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-30T13:51:15.932076+0000","last_clean_scrub_stamp":"2024-03-30T13:51:15.932076+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"216'9","reported_seq":469,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.658922+0000","last_change":"2024-03-30T13:52:19.931416+0000","last_active":"2024-03-30T13:55:36.658922+0000","last_peered":"2024-03-30T13:55:36.658922+0000","last_clean":"2024-03-30T13:55:36.658922+0000","last_became_active":"2024-03-30T13:42:47.317352+0000","last_became_peered":"2024-03-30T13:42:47.317352+0000","last_unstale":"2024-03-30T13:55:36.658922+0000","last_undegraded":"2024-03-30T13:55:36.658922+0000","last_fullsized":"2024-03-30T13:55:36.658922+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-30T13:52:19.931197+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-30T13:43:50.960980+0000","last_clean_scrub_stamp":"2024-03-30T13:52:19.931197+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":445,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.660477+0000","last_change":"2024-03-30T13:53:13.655592+0000","last_active":"2024-03-30T13:55:36.660477+0000","last_peered":"2024-03-30T13:55:36.660477+0000","last_clean":"2024-03-30T13:55:36.660477+0000","last_became_active":"2024-03-30T13:53:13.654803+0000","last_became_peered":"2024-03-30T13:53:13.654803+0000","last_unstale":"2024-03-30T13:55:36.660477+0000","last_undegraded":"2024-03-30T13:55:36.660477+0000","last_fullsized":"2024-03-30T13:55:36.660477+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:07.996724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:07.996724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":449,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.711602+0000","last_change":"2024-03-30T13:52:19.544522+0000","last_active":"2024-03-30T13:55:36.711602+0000","last_peered":"2024-03-30T13:55:36.711602+0000","last_clean":"2024-03-30T13:55:36.711602+0000","last_became_active":"2024-03-30T13:42:49.322665+0000","last_became_peered":"2024-03-30T13:42:49.322665+0000","last_unstale":"2024-03-30T13:55:36.711602+0000","last_undegraded":"2024-03-30T13:55:36.711602+0000","last_fullsized":"2024-03-30T13:55:36.711602+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:19.544305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:30.465974+0000","last_clean_scrub_stamp":"2024-03-30T13:52:19.544305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":453,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.695924+0000","last_change":"2024-03-30T13:53:35.861421+0000","last_active":"2024-03-30T13:55:36.695924+0000","last_peered":"2024-03-30T13:55:36.695924+0000","last_clean":"2024-03-30T13:55:36.695924+0000","last_became_active":"2024-03-30T13:53:35.861043+0000","last_became_peered":"2024-03-30T13:53:35.861043+0000","last_unstale":"2024-03-30T13:55:36.695924+0000","last_undegraded":"2024-03-30T13:55:36.695924+0000","last_fullsized":"2024-03-30T13:55:36.695924+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:16.009354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:43.088964+0000","last_clean_scrub_stamp":"2024-03-30T13:52:16.009354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"216'27","reported_seq":275,"reported_epoch":262,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:55:36.705792+0000","last_change":"2024-03-30T13:50:08.508437+0000","last_active":"2024-03-30T13:55:36.705792+0000","last_peered":"2024-03-30T13:55:36.705792+0000","last_clean":"2024-03-30T13:50:05.490228+0000","last_became_active":"2024-03-30T13:50:08.508437+0000","last_became_peered":"2024-03-30T13:50:08.508437+0000","last_unstale":"2024-03-30T13:55:36.705792+0000","last_undegraded":"2024-03-30T13:50:08.506233+0000","last_fullsized":"2024-03-30T13:50:08.506116+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-30T13:49:08.415681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:49:08.415681+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":436,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.739367+0000","last_change":"2024-03-30T13:53:13.665423+0000","last_active":"2024-03-30T13:55:36.739367+0000","last_peered":"2024-03-30T13:55:36.739367+0000","last_clean":"2024-03-30T13:55:36.739367+0000","last_became_active":"2024-03-30T13:53:13.664873+0000","last_became_peered":"2024-03-30T13:53:13.664873+0000","last_unstale":"2024-03-30T13:55:36.739367+0000","last_undegraded":"2024-03-30T13:55:36.739367+0000","last_fullsized":"2024-03-30T13:55:36.739367+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:52.054302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:52.054302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":429,"reported_epoch":262,"state":"active+undersized","last_fresh":"2024-03-30T13:55:36.659881+0000","last_change":"2024-03-30T13:50:08.506226+0000","last_active":"2024-03-30T13:55:36.659881+0000","last_peered":"2024-03-30T13:55:36.659881+0000","last_clean":"2024-03-30T13:50:05.490914+0000","last_became_active":"2024-03-30T13:50:08.506226+0000","last_became_peered":"2024-03-30T13:50:08.506226+0000","last_unstale":"2024-03-30T13:55:36.659881+0000","last_undegraded":"2024-03-30T13:55:36.659881+0000","last_fullsized":"2024-03-30T13:50:08.502611+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:25.104610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:48:25.104610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":437,"reported_epoch":262,"state":"active+undersized","last_fresh":"2024-03-30T13:55:36.659129+0000","last_change":"2024-03-30T13:50:08.508556+0000","last_active":"2024-03-30T13:55:36.659129+0000","last_peered":"2024-03-30T13:55:36.659129+0000","last_clean":"2024-03-30T13:50:07.261781+0000","last_became_active":"2024-03-30T13:50:08.508556+0000","last_became_peered":"2024-03-30T13:50:08.508556+0000","last_unstale":"2024-03-30T13:55:36.659129+0000","last_undegraded":"2024-03-30T13:55:36.659129+0000","last_fullsized":"2024-03-30T13:50:08.502793+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:07.261595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:50:07.261595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"217'23","reported_seq":482,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.730873+0000","last_change":"2024-03-30T13:52:50.945958+0000","last_active":"2024-03-30T13:55:36.730873+0000","last_peered":"2024-03-30T13:55:36.730873+0000","last_clean":"2024-03-30T13:55:36.730873+0000","last_became_active":"2024-03-30T13:42:47.317881+0000","last_became_peered":"2024-03-30T13:42:47.317881+0000","last_unstale":"2024-03-30T13:55:36.730873+0000","last_undegraded":"2024-03-30T13:55:36.730873+0000","last_fullsized":"2024-03-30T13:55:36.730873+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'17","last_scrub_stamp":"2024-03-30T13:52:50.945844+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-03-30T13:43:52.044427+0000","last_clean_scrub_stamp":"2024-03-30T13:52:50.945844+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":420,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.729584+0000","last_change":"2024-03-30T13:53:37.211479+0000","last_active":"2024-03-30T13:55:36.729584+0000","last_peered":"2024-03-30T13:55:36.729584+0000","last_clean":"2024-03-30T13:55:36.729584+0000","last_became_active":"2024-03-30T13:53:37.109227+0000","last_became_peered":"2024-03-30T13:53:37.109227+0000","last_unstale":"2024-03-30T13:55:36.729584+0000","last_undegraded":"2024-03-30T13:55:36.729584+0000","last_fullsized":"2024-03-30T13:55:36.729584+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T13:48:25.249724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:25.249724+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,7],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"230'10","reported_seq":620,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.677476+0000","last_change":"2024-03-30T13:53:36.854785+0000","last_active":"2024-03-30T13:55:36.677476+0000","last_peered":"2024-03-30T13:55:36.677476+0000","last_clean":"2024-03-30T13:55:36.677476+0000","last_became_active":"2024-03-30T13:53:36.852879+0000","last_became_peered":"2024-03-30T13:53:36.852879+0000","last_unstale":"2024-03-30T13:55:36.677476+0000","last_undegraded":"2024-03-30T13:55:36.677476+0000","last_fullsized":"2024-03-30T13:55:36.677476+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:52:07.177214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:07.177214+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,7],"avail_no_missing":["6","4","7"],"object_location_counts":[{"shards":"4,6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"65'1","reported_seq":458,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.700928+0000","last_change":"2024-03-30T13:52:28.566954+0000","last_active":"2024-03-30T13:55:36.700928+0000","last_peered":"2024-03-30T13:55:36.700928+0000","last_clean":"2024-03-30T13:55:36.700928+0000","last_became_active":"2024-03-30T13:42:45.307832+0000","last_became_peered":"2024-03-30T13:42:45.307832+0000","last_unstale":"2024-03-30T13:55:36.700928+0000","last_undegraded":"2024-03-30T13:55:36.700928+0000","last_fullsized":"2024-03-30T13:55:36.700928+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:52:28.566847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:28.566847+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"217'25","reported_seq":505,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.695022+0000","last_change":"2024-03-30T13:53:35.974113+0000","last_active":"2024-03-30T13:55:36.695022+0000","last_peered":"2024-03-30T13:55:36.695022+0000","last_clean":"2024-03-30T13:55:36.695022+0000","last_became_active":"2024-03-30T13:53:35.858156+0000","last_became_peered":"2024-03-30T13:53:35.858156+0000","last_unstale":"2024-03-30T13:55:36.695022+0000","last_undegraded":"2024-03-30T13:55:36.695022+0000","last_fullsized":"2024-03-30T13:55:36.695022+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'25","last_scrub_stamp":"2024-03-30T13:52:54.925758+0000","last_deep_scrub":"153'15","last_deep_scrub_stamp":"2024-03-30T13:51:22.252831+0000","last_clean_scrub_stamp":"2024-03-30T13:52:54.925758+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":442,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.711421+0000","last_change":"2024-03-30T13:53:14.660108+0000","last_active":"2024-03-30T13:55:36.711421+0000","last_peered":"2024-03-30T13:55:36.711421+0000","last_clean":"2024-03-30T13:55:36.711421+0000","last_became_active":"2024-03-30T13:53:14.659650+0000","last_became_peered":"2024-03-30T13:53:14.659650+0000","last_unstale":"2024-03-30T13:55:36.711421+0000","last_undegraded":"2024-03-30T13:55:36.711421+0000","last_fullsized":"2024-03-30T13:55:36.711421+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:00.432456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:00.432456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,5],"avail_no_missing":["1","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":440,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.695428+0000","last_change":"2024-03-30T13:53:36.860961+0000","last_active":"2024-03-30T13:55:36.695428+0000","last_peered":"2024-03-30T13:55:36.695428+0000","last_clean":"2024-03-30T13:55:36.695428+0000","last_became_active":"2024-03-30T13:53:36.857751+0000","last_became_peered":"2024-03-30T13:53:36.857751+0000","last_unstale":"2024-03-30T13:55:36.695428+0000","last_undegraded":"2024-03-30T13:55:36.695428+0000","last_fullsized":"2024-03-30T13:55:36.695428+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:27.947416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:27.947416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":441,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.722815+0000","last_change":"2024-03-30T13:52:17.021134+0000","last_active":"2024-03-30T13:55:36.722815+0000","last_peered":"2024-03-30T13:55:36.722815+0000","last_clean":"2024-03-30T13:55:36.722815+0000","last_became_active":"2024-03-30T13:42:45.310165+0000","last_became_peered":"2024-03-30T13:42:45.310165+0000","last_unstale":"2024-03-30T13:55:36.722815+0000","last_undegraded":"2024-03-30T13:55:36.722815+0000","last_fullsized":"2024-03-30T13:55:36.722815+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:17.020887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:17.020887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"217'19","reported_seq":501,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.677719+0000","last_change":"2024-03-30T13:53:13.673522+0000","last_active":"2024-03-30T13:55:36.677719+0000","last_peered":"2024-03-30T13:55:36.677719+0000","last_clean":"2024-03-30T13:55:36.677719+0000","last_became_active":"2024-03-30T13:53:13.650706+0000","last_became_peered":"2024-03-30T13:53:13.650706+0000","last_unstale":"2024-03-30T13:55:36.677719+0000","last_undegraded":"2024-03-30T13:55:36.677719+0000","last_fullsized":"2024-03-30T13:55:36.677719+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:52:26.096051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:26.096051+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":303,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.658766+0000","last_change":"2024-03-30T13:53:36.864671+0000","last_active":"2024-03-30T13:55:36.658766+0000","last_peered":"2024-03-30T13:55:36.658766+0000","last_clean":"2024-03-30T13:55:36.658766+0000","last_became_active":"2024-03-30T13:53:36.863938+0000","last_became_peered":"2024-03-30T13:53:36.863938+0000","last_unstale":"2024-03-30T13:55:36.658766+0000","last_undegraded":"2024-03-30T13:55:36.658766+0000","last_fullsized":"2024-03-30T13:55:36.658766+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:23.264538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:23.264538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3,7],"avail_no_missing":["4","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":461,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.677170+0000","last_change":"2024-03-30T13:53:35.862789+0000","last_active":"2024-03-30T13:55:36.677170+0000","last_peered":"2024-03-30T13:55:36.677170+0000","last_clean":"2024-03-30T13:55:36.677170+0000","last_became_active":"2024-03-30T13:53:35.862467+0000","last_became_peered":"2024-03-30T13:53:35.862467+0000","last_unstale":"2024-03-30T13:55:36.677170+0000","last_undegraded":"2024-03-30T13:55:36.677170+0000","last_fullsized":"2024-03-30T13:55:36.677170+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:17.108986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:54.893653+0000","last_clean_scrub_stamp":"2024-03-30T13:52:17.108986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":461,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.696491+0000","last_change":"2024-03-30T13:53:13.661456+0000","last_active":"2024-03-30T13:55:36.696491+0000","last_peered":"2024-03-30T13:55:36.696491+0000","last_clean":"2024-03-30T13:55:36.696491+0000","last_became_active":"2024-03-30T13:53:13.661128+0000","last_became_peered":"2024-03-30T13:53:13.661128+0000","last_unstale":"2024-03-30T13:55:36.696491+0000","last_undegraded":"2024-03-30T13:55:36.696491+0000","last_fullsized":"2024-03-30T13:55:36.696491+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:47.933046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:47.933046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"216'8","reported_seq":449,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.722911+0000","last_change":"2024-03-30T13:52:03.989310+0000","last_active":"2024-03-30T13:55:36.722911+0000","last_peered":"2024-03-30T13:55:36.722911+0000","last_clean":"2024-03-30T13:55:36.722911+0000","last_became_active":"2024-03-30T13:42:47.321428+0000","last_became_peered":"2024-03-30T13:42:47.321428+0000","last_unstale":"2024-03-30T13:55:36.722911+0000","last_undegraded":"2024-03-30T13:55:36.722911+0000","last_fullsized":"2024-03-30T13:55:36.722911+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-30T13:52:03.989075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:03.989075+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":430,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.738242+0000","last_change":"2024-03-30T13:52:26.149264+0000","last_active":"2024-03-30T13:55:36.738242+0000","last_peered":"2024-03-30T13:55:36.738242+0000","last_clean":"2024-03-30T13:55:36.738242+0000","last_became_active":"2024-03-30T13:42:51.374304+0000","last_became_peered":"2024-03-30T13:42:51.374304+0000","last_unstale":"2024-03-30T13:55:36.738242+0000","last_undegraded":"2024-03-30T13:55:36.738242+0000","last_fullsized":"2024-03-30T13:55:36.738242+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:26.149132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:26.149132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":78,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.658508+0000","last_change":"2024-03-30T13:53:35.854347+0000","last_active":"2024-03-30T13:55:36.658508+0000","last_peered":"2024-03-30T13:55:36.658508+0000","last_clean":"2024-03-30T13:55:36.658508+0000","last_became_active":"2024-03-30T13:53:35.854002+0000","last_became_peered":"2024-03-30T13:53:35.854002+0000","last_unstale":"2024-03-30T13:55:36.658508+0000","last_undegraded":"2024-03-30T13:55:36.658508+0000","last_fullsized":"2024-03-30T13:55:36.658508+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:56.878297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:42.616155+0000","last_clean_scrub_stamp":"2024-03-30T13:52:56.878297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":80,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.707527+0000","last_change":"2024-03-30T13:53:36.870170+0000","last_active":"2024-03-30T13:55:36.707527+0000","last_peered":"2024-03-30T13:55:36.707527+0000","last_clean":"2024-03-30T13:55:36.707527+0000","last_became_active":"2024-03-30T13:53:36.869434+0000","last_became_peered":"2024-03-30T13:53:36.869434+0000","last_unstale":"2024-03-30T13:55:36.707527+0000","last_undegraded":"2024-03-30T13:55:36.707527+0000","last_fullsized":"2024-03-30T13:55:36.707527+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:04.222591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:25.494903+0000","last_clean_scrub_stamp":"2024-03-30T13:52:04.222591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3,7],"avail_no_missing":["1","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"217'19","reported_seq":494,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.724980+0000","last_change":"2024-03-30T13:53:36.012242+0000","last_active":"2024-03-30T13:55:36.724980+0000","last_peered":"2024-03-30T13:55:36.724980+0000","last_clean":"2024-03-30T13:55:36.724980+0000","last_became_active":"2024-03-30T13:53:35.863880+0000","last_became_peered":"2024-03-30T13:53:35.863880+0000","last_unstale":"2024-03-30T13:55:36.724980+0000","last_undegraded":"2024-03-30T13:55:36.724980+0000","last_fullsized":"2024-03-30T13:55:36.724980+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'19","last_scrub_stamp":"2024-03-30T13:52:57.970377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:57.970377+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":441,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.692270+0000","last_change":"2024-03-30T13:53:13.650823+0000","last_active":"2024-03-30T13:55:36.692270+0000","last_peered":"2024-03-30T13:55:36.692270+0000","last_clean":"2024-03-30T13:55:36.692270+0000","last_became_active":"2024-03-30T13:53:13.650411+0000","last_became_peered":"2024-03-30T13:53:13.650411+0000","last_unstale":"2024-03-30T13:55:36.692270+0000","last_undegraded":"2024-03-30T13:55:36.692270+0000","last_fullsized":"2024-03-30T13:55:36.692270+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:03.006855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:28.314391+0000","last_clean_scrub_stamp":"2024-03-30T13:52:03.006855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":433,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.738035+0000","last_change":"2024-03-30T13:52:16.009739+0000","last_active":"2024-03-30T13:55:36.738035+0000","last_peered":"2024-03-30T13:55:36.738035+0000","last_clean":"2024-03-30T13:55:36.738035+0000","last_became_active":"2024-03-30T13:42:49.325219+0000","last_became_peered":"2024-03-30T13:42:49.325219+0000","last_unstale":"2024-03-30T13:55:36.738035+0000","last_undegraded":"2024-03-30T13:55:36.738035+0000","last_fullsized":"2024-03-30T13:55:36.738035+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:16.009499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:16.009499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":78,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.730708+0000","last_change":"2024-03-30T13:53:35.859545+0000","last_active":"2024-03-30T13:55:36.730708+0000","last_peered":"2024-03-30T13:55:36.730708+0000","last_clean":"2024-03-30T13:55:36.730708+0000","last_became_active":"2024-03-30T13:53:35.859253+0000","last_became_peered":"2024-03-30T13:53:35.859253+0000","last_unstale":"2024-03-30T13:55:36.730708+0000","last_undegraded":"2024-03-30T13:55:36.730708+0000","last_fullsized":"2024-03-30T13:55:36.730708+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:11.701294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:11.701294+0000","last_clean_scrub_stamp":"2024-03-30T13:52:11.701294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"216'15","reported_seq":463,"reported_epoch":262,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:55:36.712480+0000","last_change":"2024-03-30T13:50:08.513156+0000","last_active":"2024-03-30T13:55:36.712480+0000","last_peered":"2024-03-30T13:55:36.712480+0000","last_clean":"2024-03-30T13:50:05.496678+0000","last_became_active":"2024-03-30T13:50:08.513156+0000","last_became_peered":"2024-03-30T13:50:08.513156+0000","last_unstale":"2024-03-30T13:55:36.712480+0000","last_undegraded":"2024-03-30T13:50:08.507726+0000","last_fullsized":"2024-03-30T13:50:08.507616+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:49:26.480540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:49:26.480540+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":434,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.723153+0000","last_change":"2024-03-30T13:53:14.826350+0000","last_active":"2024-03-30T13:55:36.723153+0000","last_peered":"2024-03-30T13:55:36.723153+0000","last_clean":"2024-03-30T13:55:36.723153+0000","last_became_active":"2024-03-30T13:53:14.822437+0000","last_became_peered":"2024-03-30T13:53:14.822437+0000","last_unstale":"2024-03-30T13:55:36.723153+0000","last_undegraded":"2024-03-30T13:55:36.723153+0000","last_fullsized":"2024-03-30T13:55:36.723153+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:03.011875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:24.598976+0000","last_clean_scrub_stamp":"2024-03-30T13:52:03.011875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0,5],"avail_no_missing":["3","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":459,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.712206+0000","last_change":"2024-03-30T13:53:35.871876+0000","last_active":"2024-03-30T13:55:36.712206+0000","last_peered":"2024-03-30T13:55:36.712206+0000","last_clean":"2024-03-30T13:55:36.712206+0000","last_became_active":"2024-03-30T13:53:35.870924+0000","last_became_peered":"2024-03-30T13:53:35.870924+0000","last_unstale":"2024-03-30T13:55:36.712206+0000","last_undegraded":"2024-03-30T13:55:36.712206+0000","last_fullsized":"2024-03-30T13:55:36.712206+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:46.670446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:34.815783+0000","last_clean_scrub_stamp":"2024-03-30T13:52:46.670446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":455,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.659467+0000","last_change":"2024-03-30T13:53:36.849109+0000","last_active":"2024-03-30T13:55:36.659467+0000","last_peered":"2024-03-30T13:55:36.659467+0000","last_clean":"2024-03-30T13:55:36.659467+0000","last_became_active":"2024-03-30T13:53:36.848656+0000","last_became_peered":"2024-03-30T13:53:36.848656+0000","last_unstale":"2024-03-30T13:55:36.659467+0000","last_undegraded":"2024-03-30T13:55:36.659467+0000","last_fullsized":"2024-03-30T13:55:36.659467+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:14.015356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:14.015356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,7],"avail_no_missing":["4","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"216'19","reported_seq":482,"reported_epoch":262,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:55:36.660031+0000","last_change":"2024-03-30T13:50:08.513382+0000","last_active":"2024-03-30T13:55:36.660031+0000","last_peered":"2024-03-30T13:55:36.660031+0000","last_clean":"2024-03-30T13:50:05.491029+0000","last_became_active":"2024-03-30T13:50:08.513382+0000","last_became_peered":"2024-03-30T13:50:08.513382+0000","last_unstale":"2024-03-30T13:55:36.660031+0000","last_undegraded":"2024-03-30T13:50:08.503759+0000","last_fullsized":"2024-03-30T13:50:08.503643+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:48:22.068206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:22.068206+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":449,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.694602+0000","last_change":"2024-03-30T13:53:35.856379+0000","last_active":"2024-03-30T13:55:36.694602+0000","last_peered":"2024-03-30T13:55:36.694602+0000","last_clean":"2024-03-30T13:55:36.694602+0000","last_became_active":"2024-03-30T13:53:35.855996+0000","last_became_peered":"2024-03-30T13:53:35.855996+0000","last_unstale":"2024-03-30T13:55:36.694602+0000","last_undegraded":"2024-03-30T13:55:36.694602+0000","last_fullsized":"2024-03-30T13:55:36.694602+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:56.942611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:56.942611+0000","last_clean_scrub_stamp":"2024-03-30T13:52:56.942611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"222'9","reported_seq":153,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.711644+0000","last_change":"2024-03-30T13:53:13.664599+0000","last_active":"2024-03-30T13:55:36.711644+0000","last_peered":"2024-03-30T13:55:36.711644+0000","last_clean":"2024-03-30T13:55:36.711644+0000","last_became_active":"2024-03-30T13:53:13.651996+0000","last_became_peered":"2024-03-30T13:53:13.651996+0000","last_unstale":"2024-03-30T13:55:36.711644+0000","last_undegraded":"2024-03-30T13:55:36.711644+0000","last_fullsized":"2024-03-30T13:55:36.711644+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:52:16.103727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:16.103727+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":315,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.659026+0000","last_change":"2024-03-30T13:53:36.850388+0000","last_active":"2024-03-30T13:55:36.659026+0000","last_peered":"2024-03-30T13:55:36.659026+0000","last_clean":"2024-03-30T13:55:36.659026+0000","last_became_active":"2024-03-30T13:53:36.849971+0000","last_became_peered":"2024-03-30T13:53:36.849971+0000","last_unstale":"2024-03-30T13:55:36.659026+0000","last_undegraded":"2024-03-30T13:55:36.659026+0000","last_fullsized":"2024-03-30T13:55:36.659026+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:07.618186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:49:07.618186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,7],"avail_no_missing":["4","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"217'20","reported_seq":489,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.735292+0000","last_change":"2024-03-30T13:53:37.435184+0000","last_active":"2024-03-30T13:55:36.735292+0000","last_peered":"2024-03-30T13:55:36.735292+0000","last_clean":"2024-03-30T13:55:36.735292+0000","last_became_active":"2024-03-30T13:53:37.110194+0000","last_became_peered":"2024-03-30T13:53:37.110194+0000","last_unstale":"2024-03-30T13:55:36.735292+0000","last_undegraded":"2024-03-30T13:55:36.735292+0000","last_fullsized":"2024-03-30T13:55:36.735292+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-30T13:48:21.678227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:21.678227+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":430,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.659761+0000","last_change":"2024-03-30T13:53:14.659456+0000","last_active":"2024-03-30T13:55:36.659761+0000","last_peered":"2024-03-30T13:55:36.659761+0000","last_clean":"2024-03-30T13:55:36.659761+0000","last_became_active":"2024-03-30T13:53:14.658940+0000","last_became_peered":"2024-03-30T13:53:14.658940+0000","last_unstale":"2024-03-30T13:55:36.659761+0000","last_undegraded":"2024-03-30T13:55:36.659761+0000","last_fullsized":"2024-03-30T13:55:36.659761+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:30.238582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:30.238582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,5],"avail_no_missing":["4","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"230'11","reported_seq":603,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.735560+0000","last_change":"2024-03-30T13:53:37.356715+0000","last_active":"2024-03-30T13:55:36.735560+0000","last_peered":"2024-03-30T13:55:36.735560+0000","last_clean":"2024-03-30T13:55:36.735560+0000","last_became_active":"2024-03-30T13:53:37.127826+0000","last_became_peered":"2024-03-30T13:53:37.127826+0000","last_unstale":"2024-03-30T13:55:36.735560+0000","last_undegraded":"2024-03-30T13:55:36.735560+0000","last_fullsized":"2024-03-30T13:55:36.735560+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:49:23.573435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:23.573435+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":453,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.693476+0000","last_change":"2024-03-30T13:53:35.857771+0000","last_active":"2024-03-30T13:55:36.693476+0000","last_peered":"2024-03-30T13:55:36.693476+0000","last_clean":"2024-03-30T13:55:36.693476+0000","last_became_active":"2024-03-30T13:53:35.857456+0000","last_became_peered":"2024-03-30T13:53:35.857456+0000","last_unstale":"2024-03-30T13:55:36.693476+0000","last_undegraded":"2024-03-30T13:55:36.693476+0000","last_fullsized":"2024-03-30T13:55:36.693476+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:09.007490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:28.734080+0000","last_clean_scrub_stamp":"2024-03-30T13:52:09.007490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"217'15","reported_seq":464,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.730981+0000","last_change":"2024-03-30T13:53:35.896795+0000","last_active":"2024-03-30T13:55:36.730981+0000","last_peered":"2024-03-30T13:55:36.730981+0000","last_clean":"2024-03-30T13:55:36.730981+0000","last_became_active":"2024-03-30T13:53:35.865891+0000","last_became_peered":"2024-03-30T13:53:35.865891+0000","last_unstale":"2024-03-30T13:55:36.730981+0000","last_undegraded":"2024-03-30T13:55:36.730981+0000","last_fullsized":"2024-03-30T13:55:36.730981+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:52:03.227628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:03.227628+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":218,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.660045+0000","last_change":"2024-03-30T13:53:36.857499+0000","last_active":"2024-03-30T13:55:36.660045+0000","last_peered":"2024-03-30T13:55:36.660045+0000","last_clean":"2024-03-30T13:55:36.660045+0000","last_became_active":"2024-03-30T13:53:36.857026+0000","last_became_peered":"2024-03-30T13:53:36.857026+0000","last_unstale":"2024-03-30T13:55:36.660045+0000","last_undegraded":"2024-03-30T13:55:36.660045+0000","last_fullsized":"2024-03-30T13:55:36.660045+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:14.322254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:14.322254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,7],"avail_no_missing":["4","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":439,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.739956+0000","last_change":"2024-03-30T13:53:35.876084+0000","last_active":"2024-03-30T13:55:36.739956+0000","last_peered":"2024-03-30T13:55:36.739956+0000","last_clean":"2024-03-30T13:55:36.739956+0000","last_became_active":"2024-03-30T13:53:35.875783+0000","last_became_peered":"2024-03-30T13:53:35.875783+0000","last_unstale":"2024-03-30T13:55:36.739956+0000","last_undegraded":"2024-03-30T13:55:36.739956+0000","last_fullsized":"2024-03-30T13:55:36.739956+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:22.106782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:22.106782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":78,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.659259+0000","last_change":"2024-03-30T13:53:35.858594+0000","last_active":"2024-03-30T13:55:36.659259+0000","last_peered":"2024-03-30T13:55:36.659259+0000","last_clean":"2024-03-30T13:55:36.659259+0000","last_became_active":"2024-03-30T13:53:35.858261+0000","last_became_peered":"2024-03-30T13:53:35.858261+0000","last_unstale":"2024-03-30T13:55:36.659259+0000","last_undegraded":"2024-03-30T13:55:36.659259+0000","last_fullsized":"2024-03-30T13:55:36.659259+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:18.239047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:18.239047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"218'16","reported_seq":482,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.695129+0000","last_change":"2024-03-30T13:53:36.861902+0000","last_active":"2024-03-30T13:55:36.695129+0000","last_peered":"2024-03-30T13:55:36.695129+0000","last_clean":"2024-03-30T13:55:36.695129+0000","last_became_active":"2024-03-30T13:53:36.861455+0000","last_became_peered":"2024-03-30T13:53:36.861455+0000","last_unstale":"2024-03-30T13:55:36.695129+0000","last_undegraded":"2024-03-30T13:55:36.695129+0000","last_fullsized":"2024-03-30T13:55:36.695129+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'16","last_scrub_stamp":"2024-03-30T13:52:59.883027+0000","last_deep_scrub":"218'16","last_deep_scrub_stamp":"2024-03-30T13:52:59.883027+0000","last_clean_scrub_stamp":"2024-03-30T13:52:59.883027+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3,7],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"72'1","reported_seq":94,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.677679+0000","last_change":"2024-03-30T13:53:35.870213+0000","last_active":"2024-03-30T13:55:36.677679+0000","last_peered":"2024-03-30T13:55:36.677679+0000","last_clean":"2024-03-30T13:55:36.677679+0000","last_became_active":"2024-03-30T13:53:35.864200+0000","last_became_peered":"2024-03-30T13:53:35.864200+0000","last_unstale":"2024-03-30T13:55:36.677679+0000","last_undegraded":"2024-03-30T13:55:36.677679+0000","last_fullsized":"2024-03-30T13:55:36.677679+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T13:52:27.891771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:27.891771+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":93,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.730003+0000","last_change":"2024-03-30T13:53:13.661413+0000","last_active":"2024-03-30T13:55:36.730003+0000","last_peered":"2024-03-30T13:55:36.730003+0000","last_clean":"2024-03-30T13:55:36.730003+0000","last_became_active":"2024-03-30T13:53:13.661052+0000","last_became_peered":"2024-03-30T13:53:13.661052+0000","last_unstale":"2024-03-30T13:55:36.730003+0000","last_undegraded":"2024-03-30T13:55:36.730003+0000","last_fullsized":"2024-03-30T13:55:36.730003+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:09.123811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:09.123811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":469,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.676764+0000","last_change":"2024-03-30T13:53:13.658240+0000","last_active":"2024-03-30T13:55:36.676764+0000","last_peered":"2024-03-30T13:55:36.676764+0000","last_clean":"2024-03-30T13:55:36.676764+0000","last_became_active":"2024-03-30T13:53:13.657810+0000","last_became_peered":"2024-03-30T13:53:13.657810+0000","last_unstale":"2024-03-30T13:55:36.676764+0000","last_undegraded":"2024-03-30T13:55:36.676764+0000","last_fullsized":"2024-03-30T13:55:36.676764+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:45.998724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:51:20.175438+0000","last_clean_scrub_stamp":"2024-03-30T13:52:45.998724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"216'12","reported_seq":460,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.739730+0000","last_change":"2024-03-30T13:53:37.104181+0000","last_active":"2024-03-30T13:55:36.739730+0000","last_peered":"2024-03-30T13:55:36.739730+0000","last_clean":"2024-03-30T13:55:36.739730+0000","last_became_active":"2024-03-30T13:53:37.098834+0000","last_became_peered":"2024-03-30T13:53:37.098834+0000","last_unstale":"2024-03-30T13:55:36.739730+0000","last_undegraded":"2024-03-30T13:55:36.739730+0000","last_fullsized":"2024-03-30T13:55:36.739730+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-03-30T13:52:14.027491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:14.027491+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"262'8244","reported_seq":15812,"reported_epoch":262,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-30T13:55:40.239919+0000","last_change":"2024-03-30T13:54:25.237953+0000","last_active":"2024-03-30T13:55:40.239919+0000","last_peered":"2024-03-30T13:55:40.239919+0000","last_clean":"2024-03-30T13:53:12.631731+0000","last_became_active":"2024-03-30T13:53:37.539916+0000","last_became_peered":"2024-03-30T13:53:37.539916+0000","last_unstale":"2024-03-30T13:55:40.239919+0000","last_undegraded":"2024-03-30T13:53:37.277057+0000","last_fullsized":"2024-03-30T13:53:36.839152+0000","mapping_epoch":229,"log_start":"234'4300","ondisk_log_start":"234'4300","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3944,"log_dups_size":0,"ondisk_log_size":3944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":2624,"num_object_clones":0,"num_object_copies":7872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":985,"num_objects_misplaced":1369,"num_objects_unfound":0,"num_objects_dirty":2624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8244,"num_write_kb":347776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2810,"num_bytes_recovered":184156160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,1","objects":559},{"shards":"0,1,7","objects":791},{"shards":"0,5,7","objects":289}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":93,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.677165+0000","last_change":"2024-03-30T13:53:14.655924+0000","last_active":"2024-03-30T13:55:36.677165+0000","last_peered":"2024-03-30T13:55:36.677165+0000","last_clean":"2024-03-30T13:55:36.677165+0000","last_became_active":"2024-03-30T13:53:14.655517+0000","last_became_peered":"2024-03-30T13:53:14.655517+0000","last_unstale":"2024-03-30T13:55:36.677165+0000","last_undegraded":"2024-03-30T13:55:36.677165+0000","last_fullsized":"2024-03-30T13:55:36.677165+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:05.957043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:05.957043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0,5],"avail_no_missing":["6","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":93,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.659395+0000","last_change":"2024-03-30T13:53:36.015472+0000","last_active":"2024-03-30T13:55:36.659395+0000","last_peered":"2024-03-30T13:55:36.659395+0000","last_clean":"2024-03-30T13:55:36.659395+0000","last_became_active":"2024-03-30T13:53:35.874521+0000","last_became_peered":"2024-03-30T13:53:35.874521+0000","last_unstale":"2024-03-30T13:55:36.659395+0000","last_undegraded":"2024-03-30T13:55:36.659395+0000","last_fullsized":"2024-03-30T13:55:36.659395+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-30T13:52:09.680567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:09.680567+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"216'27","reported_seq":486,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.676843+0000","last_change":"2024-03-30T13:53:35.901313+0000","last_active":"2024-03-30T13:55:36.676843+0000","last_peered":"2024-03-30T13:55:36.676843+0000","last_clean":"2024-03-30T13:55:36.676843+0000","last_became_active":"2024-03-30T13:53:35.887732+0000","last_became_peered":"2024-03-30T13:53:35.887732+0000","last_unstale":"2024-03-30T13:55:36.676843+0000","last_undegraded":"2024-03-30T13:55:36.676843+0000","last_fullsized":"2024-03-30T13:55:36.676843+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-30T13:51:27.133454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:27.133454+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"229'9","reported_seq":132,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.677005+0000","last_change":"2024-03-30T13:53:35.877180+0000","last_active":"2024-03-30T13:55:36.677005+0000","last_peered":"2024-03-30T13:55:36.677005+0000","last_clean":"2024-03-30T13:55:36.677005+0000","last_became_active":"2024-03-30T13:53:35.862126+0000","last_became_peered":"2024-03-30T13:53:35.862126+0000","last_unstale":"2024-03-30T13:55:36.677005+0000","last_undegraded":"2024-03-30T13:55:36.677005+0000","last_fullsized":"2024-03-30T13:55:36.677005+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:51:19.286592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:19.286592+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"262'7811","reported_seq":7923,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:40.695472+0000","last_change":"2024-03-30T13:52:43.529140+0000","last_active":"2024-03-30T13:55:40.695472+0000","last_peered":"2024-03-30T13:55:40.695472+0000","last_clean":"2024-03-30T13:55:40.695472+0000","last_became_active":"2024-03-30T13:52:43.528506+0000","last_became_peered":"2024-03-30T13:52:43.528506+0000","last_unstale":"2024-03-30T13:55:40.695472+0000","last_undegraded":"2024-03-30T13:55:40.695472+0000","last_fullsized":"2024-03-30T13:55:40.695472+0000","mapping_epoch":211,"log_start":"236'4449","ondisk_log_start":"236'4449","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3362,"log_dups_size":0,"ondisk_log_size":3362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164691968,"num_objects":2513,"num_object_clones":0,"num_object_copies":7539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7811,"num_write_kb":330368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":435,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.738146+0000","last_change":"2024-03-30T13:53:35.863569+0000","last_active":"2024-03-30T13:55:36.738146+0000","last_peered":"2024-03-30T13:55:36.738146+0000","last_clean":"2024-03-30T13:55:36.738146+0000","last_became_active":"2024-03-30T13:53:35.862056+0000","last_became_peered":"2024-03-30T13:53:35.862056+0000","last_unstale":"2024-03-30T13:55:36.738146+0000","last_undegraded":"2024-03-30T13:55:36.738146+0000","last_fullsized":"2024-03-30T13:55:36.738146+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:28.078616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:28.078616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":78,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.659841+0000","last_change":"2024-03-30T13:53:35.861728+0000","last_active":"2024-03-30T13:55:36.659841+0000","last_peered":"2024-03-30T13:55:36.659841+0000","last_clean":"2024-03-30T13:55:36.659841+0000","last_became_active":"2024-03-30T13:53:35.861370+0000","last_became_peered":"2024-03-30T13:53:35.861370+0000","last_unstale":"2024-03-30T13:55:36.659841+0000","last_undegraded":"2024-03-30T13:55:36.659841+0000","last_fullsized":"2024-03-30T13:55:36.659841+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:15.246997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:15.246997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"216'18","reported_seq":478,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.659362+0000","last_change":"2024-03-30T13:52:11.994083+0000","last_active":"2024-03-30T13:55:36.659362+0000","last_peered":"2024-03-30T13:55:36.659362+0000","last_clean":"2024-03-30T13:55:36.659362+0000","last_became_active":"2024-03-30T13:42:47.316187+0000","last_became_peered":"2024-03-30T13:42:47.316187+0000","last_unstale":"2024-03-30T13:55:36.659362+0000","last_undegraded":"2024-03-30T13:55:36.659362+0000","last_fullsized":"2024-03-30T13:55:36.659362+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-30T13:52:11.993946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:11.993946+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":213,"reported_epoch":262,"state":"active+undersized","last_fresh":"2024-03-30T13:55:36.692790+0000","last_change":"2024-03-30T13:50:08.509701+0000","last_active":"2024-03-30T13:55:36.692790+0000","last_peered":"2024-03-30T13:55:36.692790+0000","last_clean":"2024-03-30T13:50:05.499237+0000","last_became_active":"2024-03-30T13:50:08.509701+0000","last_became_peered":"2024-03-30T13:50:08.509701+0000","last_unstale":"2024-03-30T13:55:36.692790+0000","last_undegraded":"2024-03-30T13:55:36.692790+0000","last_fullsized":"2024-03-30T13:50:08.501671+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:22.401087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:22.401087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"262'8216","reported_seq":8898,"reported_epoch":262,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-30T13:55:40.562181+0000","last_change":"2024-03-30T13:55:29.420130+0000","last_active":"2024-03-30T13:55:40.562181+0000","last_peered":"2024-03-30T13:55:40.562181+0000","last_clean":"2024-03-30T13:53:34.134847+0000","last_became_active":"2024-03-30T13:53:37.270473+0000","last_became_peered":"2024-03-30T13:53:37.270473+0000","last_unstale":"2024-03-30T13:55:40.562181+0000","last_undegraded":"2024-03-30T13:53:37.090209+0000","last_fullsized":"2024-03-30T13:53:36.840043+0000","mapping_epoch":229,"log_start":"234'4629","ondisk_log_start":"234'4629","created":211,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3587,"log_dups_size":0,"ondisk_log_size":3587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173539328,"num_objects":2648,"num_object_clones":0,"num_object_copies":7944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2648,"num_objects_misplaced":3579,"num_objects_unfound":0,"num_objects_dirty":2648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8216,"num_write_kb":347648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":18350080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":1073},{"shards":"3,7","objects":-931}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":311,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.722625+0000","last_change":"2024-03-30T13:53:36.863779+0000","last_active":"2024-03-30T13:55:36.722625+0000","last_peered":"2024-03-30T13:55:36.722625+0000","last_clean":"2024-03-30T13:55:36.722625+0000","last_became_active":"2024-03-30T13:53:36.862437+0000","last_became_peered":"2024-03-30T13:53:36.862437+0000","last_unstale":"2024-03-30T13:55:36.722625+0000","last_undegraded":"2024-03-30T13:55:36.722625+0000","last_fullsized":"2024-03-30T13:55:36.722625+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:31.860675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:31.860675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,7],"avail_no_missing":["3","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":223,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.695656+0000","last_change":"2024-03-30T13:53:36.854424+0000","last_active":"2024-03-30T13:55:36.695656+0000","last_peered":"2024-03-30T13:55:36.695656+0000","last_clean":"2024-03-30T13:55:36.695656+0000","last_became_active":"2024-03-30T13:53:36.850279+0000","last_became_peered":"2024-03-30T13:53:36.850279+0000","last_unstale":"2024-03-30T13:55:36.695656+0000","last_undegraded":"2024-03-30T13:55:36.695656+0000","last_fullsized":"2024-03-30T13:55:36.695656+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:19.432242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:19.432242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":433,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.677481+0000","last_change":"2024-03-30T13:53:35.853307+0000","last_active":"2024-03-30T13:55:36.677481+0000","last_peered":"2024-03-30T13:55:36.677481+0000","last_clean":"2024-03-30T13:55:36.677481+0000","last_became_active":"2024-03-30T13:53:35.852959+0000","last_became_peered":"2024-03-30T13:53:35.852959+0000","last_unstale":"2024-03-30T13:55:36.677481+0000","last_undegraded":"2024-03-30T13:55:36.677481+0000","last_fullsized":"2024-03-30T13:55:36.677481+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:09.173459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:09.173459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"217'44","reported_seq":541,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.677520+0000","last_change":"2024-03-30T13:53:36.006059+0000","last_active":"2024-03-30T13:55:36.677520+0000","last_peered":"2024-03-30T13:55:36.677520+0000","last_clean":"2024-03-30T13:55:36.677520+0000","last_became_active":"2024-03-30T13:53:35.862052+0000","last_became_peered":"2024-03-30T13:53:35.862052+0000","last_unstale":"2024-03-30T13:55:36.677520+0000","last_undegraded":"2024-03-30T13:55:36.677520+0000","last_fullsized":"2024-03-30T13:55:36.677520+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'28","last_scrub_stamp":"2024-03-30T13:52:06.210663+0000","last_deep_scrub":"108'28","last_deep_scrub_stamp":"2024-03-30T13:45:49.140418+0000","last_clean_scrub_stamp":"2024-03-30T13:52:06.210663+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"262'8100","reported_seq":14134,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:40.659330+0000","last_change":"2024-03-30T13:54:04.031799+0000","last_active":"2024-03-30T13:55:40.659330+0000","last_peered":"2024-03-30T13:55:40.659330+0000","last_clean":"2024-03-30T13:55:40.659330+0000","last_became_active":"2024-03-30T13:54:04.026120+0000","last_became_peered":"2024-03-30T13:54:04.026120+0000","last_unstale":"2024-03-30T13:55:40.659330+0000","last_undegraded":"2024-03-30T13:55:40.659330+0000","last_fullsized":"2024-03-30T13:55:40.659330+0000","mapping_epoch":237,"log_start":"238'4720","ondisk_log_start":"238'4720","created":211,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3380,"log_dups_size":0,"ondisk_log_size":3380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176553984,"num_objects":2694,"num_object_clones":0,"num_object_copies":8082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2694,"num_objects_unfound":0,"num_objects_dirty":2694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8100,"num_write_kb":345408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2957,"num_bytes_recovered":193789952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6,5],"avail_no_missing":["1","5","6"],"object_location_counts":[{"shards":"1,5,6","objects":2694}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":427,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.693794+0000","last_change":"2024-03-30T13:53:37.362559+0000","last_active":"2024-03-30T13:55:36.693794+0000","last_peered":"2024-03-30T13:55:36.693794+0000","last_clean":"2024-03-30T13:55:36.693794+0000","last_became_active":"2024-03-30T13:53:37.359258+0000","last_became_peered":"2024-03-30T13:53:37.359258+0000","last_unstale":"2024-03-30T13:55:36.693794+0000","last_undegraded":"2024-03-30T13:55:36.693794+0000","last_fullsized":"2024-03-30T13:55:36.693794+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:12.807605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:44.095338+0000","last_clean_scrub_stamp":"2024-03-30T13:49:12.807605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3,7],"avail_no_missing":["0","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":213,"reported_epoch":262,"state":"active+undersized","last_fresh":"2024-03-30T13:55:36.659273+0000","last_change":"2024-03-30T13:50:08.507875+0000","last_active":"2024-03-30T13:55:36.659273+0000","last_peered":"2024-03-30T13:55:36.659273+0000","last_clean":"2024-03-30T13:50:05.490245+0000","last_became_active":"2024-03-30T13:50:08.507875+0000","last_became_peered":"2024-03-30T13:50:08.507875+0000","last_unstale":"2024-03-30T13:55:36.659273+0000","last_undegraded":"2024-03-30T13:55:36.659273+0000","last_fullsized":"2024-03-30T13:50:08.502700+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:13.399683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:28.225742+0000","last_clean_scrub_stamp":"2024-03-30T13:49:13.399683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":93,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.735205+0000","last_change":"2024-03-30T13:53:13.666944+0000","last_active":"2024-03-30T13:55:36.735205+0000","last_peered":"2024-03-30T13:55:36.735205+0000","last_clean":"2024-03-30T13:55:36.735205+0000","last_became_active":"2024-03-30T13:53:13.666227+0000","last_became_peered":"2024-03-30T13:53:13.666227+0000","last_unstale":"2024-03-30T13:55:36.735205+0000","last_undegraded":"2024-03-30T13:55:36.735205+0000","last_fullsized":"2024-03-30T13:55:36.735205+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:19.099205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:19.099205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"216'15","reported_seq":508,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.695779+0000","last_change":"2024-03-30T13:53:13.672815+0000","last_active":"2024-03-30T13:55:36.695779+0000","last_peered":"2024-03-30T13:55:36.695779+0000","last_clean":"2024-03-30T13:55:36.695779+0000","last_became_active":"2024-03-30T13:53:13.656465+0000","last_became_peered":"2024-03-30T13:53:13.656465+0000","last_unstale":"2024-03-30T13:55:36.695779+0000","last_undegraded":"2024-03-30T13:55:36.695779+0000","last_fullsized":"2024-03-30T13:55:36.695779+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:52:17.023838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:17.023838+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"262'8368","reported_seq":16019,"reported_epoch":262,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-30T13:55:40.228681+0000","last_change":"2024-03-30T13:55:02.841588+0000","last_active":"2024-03-30T13:55:40.228681+0000","last_peered":"2024-03-30T13:55:40.228681+0000","last_clean":"2024-03-30T13:53:12.627470+0000","last_became_active":"2024-03-30T13:53:37.366928+0000","last_became_peered":"2024-03-30T13:53:37.366928+0000","last_unstale":"2024-03-30T13:55:40.228681+0000","last_undegraded":"2024-03-30T13:53:37.269836+0000","last_fullsized":"2024-03-30T13:53:36.839916+0000","mapping_epoch":229,"log_start":"234'4500","ondisk_log_start":"234'4500","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3868,"log_dups_size":0,"ondisk_log_size":3868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176291840,"num_objects":2690,"num_object_clones":0,"num_object_copies":8070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":991,"num_objects_misplaced":1315,"num_objects_unfound":0,"num_objects_dirty":2690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8368,"num_write_kb":353856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2840,"num_bytes_recovered":186122240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,3","objects":646},{"shards":"0,3,5,7","objects":1},{"shards":"0,3,7","objects":789},{"shards":"0,5,6,7","objects":1},{"shards":"0,5,7","objects":262}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":399,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.693098+0000","last_change":"2024-03-30T13:53:16.691600+0000","last_active":"2024-03-30T13:55:36.693098+0000","last_peered":"2024-03-30T13:55:36.693098+0000","last_clean":"2024-03-30T13:55:36.693098+0000","last_became_active":"2024-03-30T13:53:16.690961+0000","last_became_peered":"2024-03-30T13:53:16.690961+0000","last_unstale":"2024-03-30T13:55:36.693098+0000","last_undegraded":"2024-03-30T13:55:36.693098+0000","last_fullsized":"2024-03-30T13:55:36.693098+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:48.894763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:48.894763+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:39.634044+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4,5],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":435,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.730111+0000","last_change":"2024-03-30T13:53:35.855546+0000","last_active":"2024-03-30T13:55:36.730111+0000","last_peered":"2024-03-30T13:55:36.730111+0000","last_clean":"2024-03-30T13:55:36.730111+0000","last_became_active":"2024-03-30T13:53:35.855186+0000","last_became_peered":"2024-03-30T13:53:35.855186+0000","last_unstale":"2024-03-30T13:55:36.730111+0000","last_undegraded":"2024-03-30T13:55:36.730111+0000","last_fullsized":"2024-03-30T13:55:36.730111+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:08.043774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:08.043774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":120,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.711886+0000","last_change":"2024-03-30T13:53:13.683544+0000","last_active":"2024-03-30T13:55:36.711886+0000","last_peered":"2024-03-30T13:55:36.711886+0000","last_clean":"2024-03-30T13:55:36.711886+0000","last_became_active":"2024-03-30T13:53:13.655879+0000","last_became_peered":"2024-03-30T13:53:13.655879+0000","last_unstale":"2024-03-30T13:55:36.711886+0000","last_undegraded":"2024-03-30T13:55:36.711886+0000","last_fullsized":"2024-03-30T13:55:36.711886+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-30T13:52:51.932750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:52:51.932750+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":443,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.658713+0000","last_change":"2024-03-30T13:51:01.240545+0000","last_active":"2024-03-30T13:55:36.658713+0000","last_peered":"2024-03-30T13:55:36.658713+0000","last_clean":"2024-03-30T13:55:36.658713+0000","last_became_active":"2024-03-30T13:42:45.310209+0000","last_became_peered":"2024-03-30T13:42:45.310209+0000","last_unstale":"2024-03-30T13:55:36.658713+0000","last_undegraded":"2024-03-30T13:55:36.658713+0000","last_fullsized":"2024-03-30T13:55:36.658713+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:01.240468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:01.240468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"217'41","reported_seq":656,"reported_epoch":262,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:55:36.706333+0000","last_change":"2024-03-30T13:50:08.508727+0000","last_active":"2024-03-30T13:55:36.706333+0000","last_peered":"2024-03-30T13:55:36.706333+0000","last_clean":"2024-03-30T13:50:05.490389+0000","last_became_active":"2024-03-30T13:50:08.508727+0000","last_became_peered":"2024-03-30T13:50:08.508727+0000","last_unstale":"2024-03-30T13:55:36.706333+0000","last_undegraded":"2024-03-30T13:50:08.506113+0000","last_fullsized":"2024-03-30T13:50:08.505993+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-03-30T13:48:27.377242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:27.377242+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":448,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.695036+0000","last_change":"2024-03-30T13:52:23.006053+0000","last_active":"2024-03-30T13:55:36.695036+0000","last_peered":"2024-03-30T13:55:36.695036+0000","last_clean":"2024-03-30T13:55:36.695036+0000","last_became_active":"2024-03-30T13:42:49.322154+0000","last_became_peered":"2024-03-30T13:42:49.322154+0000","last_unstale":"2024-03-30T13:55:36.695036+0000","last_undegraded":"2024-03-30T13:55:36.695036+0000","last_fullsized":"2024-03-30T13:55:36.695036+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:23.005788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:18.862784+0000","last_clean_scrub_stamp":"2024-03-30T13:52:23.005788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"262'8199","reported_seq":16750,"reported_epoch":262,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-30T13:55:40.704261+0000","last_change":"2024-03-30T13:54:03.000107+0000","last_active":"2024-03-30T13:55:40.704261+0000","last_peered":"2024-03-30T13:55:40.704261+0000","last_clean":"2024-03-30T13:53:12.641871+0000","last_became_active":"2024-03-30T13:53:37.228167+0000","last_became_peered":"2024-03-30T13:53:37.228167+0000","last_unstale":"2024-03-30T13:55:40.704261+0000","last_undegraded":"2024-03-30T13:55:40.704261+0000","last_fullsized":"2024-03-30T13:53:36.850042+0000","mapping_epoch":229,"log_start":"238'4861","ondisk_log_start":"238'4861","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3338,"log_dups_size":0,"ondisk_log_size":3338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176226304,"num_objects":2689,"num_object_clones":0,"num_object_copies":8067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1206,"num_objects_unfound":0,"num_objects_dirty":2689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8199,"num_write_kb":348416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2810,"num_bytes_recovered":184156160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,3","objects":1704},{"shards":"1,3,5,7","objects":1},{"shards":"1,3,7","objects":763},{"shards":"1,5,7","objects":221}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":373,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.693593+0000","last_change":"2024-03-30T13:53:36.851601+0000","last_active":"2024-03-30T13:55:36.693593+0000","last_peered":"2024-03-30T13:55:36.693593+0000","last_clean":"2024-03-30T13:55:36.693593+0000","last_became_active":"2024-03-30T13:53:36.851158+0000","last_became_peered":"2024-03-30T13:53:36.851158+0000","last_unstale":"2024-03-30T13:55:36.693593+0000","last_undegraded":"2024-03-30T13:55:36.693593+0000","last_fullsized":"2024-03-30T13:55:36.693593+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:46.926407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:46.926407+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:39.634002+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4,7],"avail_no_missing":["0","4","7"],"object_location_counts":[{"shards":"0,4,7","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":77,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.677579+0000","last_change":"2024-03-30T13:53:36.861043+0000","last_active":"2024-03-30T13:55:36.677579+0000","last_peered":"2024-03-30T13:55:36.677579+0000","last_clean":"2024-03-30T13:55:36.677579+0000","last_became_active":"2024-03-30T13:53:36.860597+0000","last_became_peered":"2024-03-30T13:53:36.860597+0000","last_unstale":"2024-03-30T13:55:36.677579+0000","last_undegraded":"2024-03-30T13:55:36.677579+0000","last_fullsized":"2024-03-30T13:55:36.677579+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:14.275893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:51:14.275893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3,7],"avail_no_missing":["6","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":445,"reported_epoch":262,"state":"active+undersized","last_fresh":"2024-03-30T13:55:36.701324+0000","last_change":"2024-03-30T13:50:08.512561+0000","last_active":"2024-03-30T13:55:36.701324+0000","last_peered":"2024-03-30T13:55:36.701324+0000","last_clean":"2024-03-30T13:50:05.489457+0000","last_became_active":"2024-03-30T13:50:08.512561+0000","last_became_peered":"2024-03-30T13:50:08.512561+0000","last_unstale":"2024-03-30T13:55:36.701324+0000","last_undegraded":"2024-03-30T13:55:36.701324+0000","last_fullsized":"2024-03-30T13:50:08.506886+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:16.464234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:59.810189+0000","last_clean_scrub_stamp":"2024-03-30T13:49:16.464234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":213,"reported_epoch":262,"state":"active+undersized","last_fresh":"2024-03-30T13:55:36.724597+0000","last_change":"2024-03-30T13:50:08.511337+0000","last_active":"2024-03-30T13:55:36.724597+0000","last_peered":"2024-03-30T13:55:36.724597+0000","last_clean":"2024-03-30T13:50:05.498892+0000","last_became_active":"2024-03-30T13:50:08.511337+0000","last_became_peered":"2024-03-30T13:50:08.511337+0000","last_unstale":"2024-03-30T13:55:36.724597+0000","last_undegraded":"2024-03-30T13:55:36.724597+0000","last_fullsized":"2024-03-30T13:50:08.501456+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:25.398232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:51.919203+0000","last_clean_scrub_stamp":"2024-03-30T13:49:25.398232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"216'17","reported_seq":522,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.718158+0000","last_change":"2024-03-30T13:53:36.011804+0000","last_active":"2024-03-30T13:55:36.718158+0000","last_peered":"2024-03-30T13:55:36.718158+0000","last_clean":"2024-03-30T13:55:36.718158+0000","last_became_active":"2024-03-30T13:53:35.867159+0000","last_became_peered":"2024-03-30T13:53:35.867159+0000","last_unstale":"2024-03-30T13:55:36.718158+0000","last_undegraded":"2024-03-30T13:55:36.718158+0000","last_fullsized":"2024-03-30T13:55:36.718158+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'13","last_scrub_stamp":"2024-03-30T13:52:25.542111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:25.542111+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":97,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.659553+0000","last_change":"2024-03-30T13:53:36.864869+0000","last_active":"2024-03-30T13:55:36.659553+0000","last_peered":"2024-03-30T13:55:36.659553+0000","last_clean":"2024-03-30T13:55:36.659553+0000","last_became_active":"2024-03-30T13:53:36.864365+0000","last_became_peered":"2024-03-30T13:53:36.864365+0000","last_unstale":"2024-03-30T13:55:36.659553+0000","last_undegraded":"2024-03-30T13:55:36.659553+0000","last_fullsized":"2024-03-30T13:55:36.659553+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:28.373277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:28.373277+0000","last_clean_scrub_stamp":"2024-03-30T13:49:28.373277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,7],"avail_no_missing":["4","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.8","version":"262'8263","reported_seq":18219,"reported_epoch":262,"state":"active+recovery_wait","last_fresh":"2024-03-30T13:55:40.561369+0000","last_change":"2024-03-30T13:55:30.706805+0000","last_active":"2024-03-30T13:55:40.561369+0000","last_peered":"2024-03-30T13:55:40.561369+0000","last_clean":"2024-03-30T13:53:12.628594+0000","last_became_active":"2024-03-30T13:55:30.660683+0000","last_became_peered":"2024-03-30T13:55:30.660683+0000","last_unstale":"2024-03-30T13:55:40.561369+0000","last_undegraded":"2024-03-30T13:55:40.561369+0000","last_fullsized":"2024-03-30T13:55:40.561369+0000","mapping_epoch":259,"log_start":"236'4727","ondisk_log_start":"236'4727","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175439872,"num_objects":2677,"num_object_clones":0,"num_object_copies":8031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8263,"num_write_kb":350080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4915,"num_bytes_recovered":322043904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":["3","0","1","5"],"object_location_counts":[{"shards":"0,1,3","objects":2677}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":94,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.740044+0000","last_change":"2024-03-30T13:53:13.660514+0000","last_active":"2024-03-30T13:55:36.740044+0000","last_peered":"2024-03-30T13:55:36.740044+0000","last_clean":"2024-03-30T13:55:36.740044+0000","last_became_active":"2024-03-30T13:53:13.660154+0000","last_became_peered":"2024-03-30T13:53:13.660154+0000","last_unstale":"2024-03-30T13:55:36.740044+0000","last_undegraded":"2024-03-30T13:55:36.740044+0000","last_fullsized":"2024-03-30T13:55:36.740044+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:28.079029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:12.433166+0000","last_clean_scrub_stamp":"2024-03-30T13:52:28.079029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":378,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.692836+0000","last_change":"2024-03-30T13:53:38.545959+0000","last_active":"2024-03-30T13:55:36.692836+0000","last_peered":"2024-03-30T13:55:36.692836+0000","last_clean":"2024-03-30T13:55:36.692836+0000","last_became_active":"2024-03-30T13:53:38.545543+0000","last_became_peered":"2024-03-30T13:53:38.545543+0000","last_unstale":"2024-03-30T13:55:36.692836+0000","last_undegraded":"2024-03-30T13:55:36.692836+0000","last_fullsized":"2024-03-30T13:55:36.692836+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:47.943814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:47.943814+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":448,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.726026+0000","last_change":"2024-03-30T13:53:13.662546+0000","last_active":"2024-03-30T13:55:36.726026+0000","last_peered":"2024-03-30T13:55:36.726026+0000","last_clean":"2024-03-30T13:55:36.726026+0000","last_became_active":"2024-03-30T13:53:13.662164+0000","last_became_peered":"2024-03-30T13:53:13.662164+0000","last_unstale":"2024-03-30T13:55:36.726026+0000","last_undegraded":"2024-03-30T13:55:36.726026+0000","last_fullsized":"2024-03-30T13:55:36.726026+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:12.975288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:12.975288+0000","last_clean_scrub_stamp":"2024-03-30T13:52:12.975288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"229'29","reported_seq":671,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.677540+0000","last_change":"2024-03-30T13:53:36.017011+0000","last_active":"2024-03-30T13:55:36.677540+0000","last_peered":"2024-03-30T13:55:36.677540+0000","last_clean":"2024-03-30T13:55:36.677540+0000","last_became_active":"2024-03-30T13:53:35.864416+0000","last_became_peered":"2024-03-30T13:53:35.864416+0000","last_unstale":"2024-03-30T13:55:36.677540+0000","last_undegraded":"2024-03-30T13:55:36.677540+0000","last_fullsized":"2024-03-30T13:55:36.677540+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'16","last_scrub_stamp":"2024-03-30T13:52:01.369487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:01.369487+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":427,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.694236+0000","last_change":"2024-03-30T13:53:36.855679+0000","last_active":"2024-03-30T13:55:36.694236+0000","last_peered":"2024-03-30T13:55:36.694236+0000","last_clean":"2024-03-30T13:55:36.694236+0000","last_became_active":"2024-03-30T13:53:36.853892+0000","last_became_peered":"2024-03-30T13:53:36.853892+0000","last_unstale":"2024-03-30T13:55:36.694236+0000","last_undegraded":"2024-03-30T13:55:36.694236+0000","last_fullsized":"2024-03-30T13:55:36.694236+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:25.787524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:25.787524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"262'8386","reported_seq":8504,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:40.206038+0000","last_change":"2024-03-30T13:53:14.659142+0000","last_active":"2024-03-30T13:55:40.206038+0000","last_peered":"2024-03-30T13:55:40.206038+0000","last_clean":"2024-03-30T13:55:40.206038+0000","last_became_active":"2024-03-30T13:53:14.651477+0000","last_became_peered":"2024-03-30T13:53:14.651477+0000","last_unstale":"2024-03-30T13:55:40.206038+0000","last_undegraded":"2024-03-30T13:55:40.206038+0000","last_fullsized":"2024-03-30T13:55:40.206038+0000","mapping_epoch":222,"log_start":"234'4456","ondisk_log_start":"234'4456","created":211,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3930,"log_dups_size":0,"ondisk_log_size":3930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178913280,"num_objects":2730,"num_object_clones":0,"num_object_copies":8190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2730,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8386,"num_write_kb":355712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,5],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":2730}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":429,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.705590+0000","last_change":"2024-03-30T13:52:47.632199+0000","last_active":"2024-03-30T13:55:36.705590+0000","last_peered":"2024-03-30T13:55:36.705590+0000","last_clean":"2024-03-30T13:55:36.705590+0000","last_became_active":"2024-03-30T13:52:36.489806+0000","last_became_peered":"2024-03-30T13:52:36.489806+0000","last_unstale":"2024-03-30T13:55:36.705590+0000","last_undegraded":"2024-03-30T13:55:36.705590+0000","last_fullsized":"2024-03-30T13:55:36.705590+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:47.632117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:47.632117+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":444,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.705636+0000","last_change":"2024-03-30T13:53:13.664555+0000","last_active":"2024-03-30T13:55:36.705636+0000","last_peered":"2024-03-30T13:55:36.705636+0000","last_clean":"2024-03-30T13:55:36.705636+0000","last_became_active":"2024-03-30T13:53:13.664207+0000","last_became_peered":"2024-03-30T13:53:13.664207+0000","last_unstale":"2024-03-30T13:55:36.705636+0000","last_undegraded":"2024-03-30T13:55:36.705636+0000","last_fullsized":"2024-03-30T13:55:36.705636+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:03.345965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:03.345965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":448,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.693327+0000","last_change":"2024-03-30T13:52:08.002816+0000","last_active":"2024-03-30T13:55:36.693327+0000","last_peered":"2024-03-30T13:55:36.693327+0000","last_clean":"2024-03-30T13:55:36.693327+0000","last_became_active":"2024-03-30T13:42:45.307905+0000","last_became_peered":"2024-03-30T13:42:45.307905+0000","last_unstale":"2024-03-30T13:55:36.693327+0000","last_undegraded":"2024-03-30T13:55:36.693327+0000","last_fullsized":"2024-03-30T13:55:36.693327+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:08.002706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:08.002706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":252,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:40.140666+0000","last_change":"2024-03-30T13:53:36.056309+0000","last_active":"2024-03-30T13:55:40.140666+0000","last_peered":"2024-03-30T13:55:40.140666+0000","last_clean":"2024-03-30T13:55:40.140666+0000","last_became_active":"2024-03-30T13:53:35.889071+0000","last_became_peered":"2024-03-30T13:53:35.889071+0000","last_unstale":"2024-03-30T13:55:40.140666+0000","last_undegraded":"2024-03-30T13:55:40.140666+0000","last_fullsized":"2024-03-30T13:55:40.140666+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-30T13:52:18.785378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:52:18.785378+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00126391,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":130,"num_read_kb":125,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"216'30","reported_seq":480,"reported_epoch":262,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:55:36.694697+0000","last_change":"2024-03-30T13:50:08.510256+0000","last_active":"2024-03-30T13:55:36.694697+0000","last_peered":"2024-03-30T13:55:36.694697+0000","last_clean":"2024-03-30T13:50:05.503358+0000","last_became_active":"2024-03-30T13:50:08.510256+0000","last_became_peered":"2024-03-30T13:50:08.510256+0000","last_unstale":"2024-03-30T13:55:36.694697+0000","last_undegraded":"2024-03-30T13:50:08.504536+0000","last_fullsized":"2024-03-30T13:50:08.504398+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'18","last_scrub_stamp":"2024-03-30T13:49:13.774982+0000","last_deep_scrub":"93'18","last_deep_scrub_stamp":"2024-03-30T13:47:28.872529+0000","last_clean_scrub_stamp":"2024-03-30T13:49:13.774982+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":93,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.718397+0000","last_change":"2024-03-30T13:53:13.649717+0000","last_active":"2024-03-30T13:55:36.718397+0000","last_peered":"2024-03-30T13:55:36.718397+0000","last_clean":"2024-03-30T13:55:36.718397+0000","last_became_active":"2024-03-30T13:53:13.649288+0000","last_became_peered":"2024-03-30T13:53:13.649288+0000","last_unstale":"2024-03-30T13:55:36.718397+0000","last_undegraded":"2024-03-30T13:55:36.718397+0000","last_fullsized":"2024-03-30T13:55:36.718397+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:08.977897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:08.977897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"262'8067","reported_seq":15866,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:38.561990+0000","last_change":"2024-03-30T13:54:26.711248+0000","last_active":"2024-03-30T13:55:38.561990+0000","last_peered":"2024-03-30T13:55:38.561990+0000","last_clean":"2024-03-30T13:55:38.561990+0000","last_became_active":"2024-03-30T13:54:26.705825+0000","last_became_peered":"2024-03-30T13:54:26.705825+0000","last_unstale":"2024-03-30T13:55:38.561990+0000","last_undegraded":"2024-03-30T13:55:38.561990+0000","last_fullsized":"2024-03-30T13:55:38.561990+0000","mapping_epoch":245,"log_start":"234'4200","ondisk_log_start":"234'4200","created":211,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3867,"log_dups_size":0,"ondisk_log_size":3867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181338112,"num_objects":2767,"num_object_clones":0,"num_object_copies":8301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8067,"num_write_kb":346688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3838,"num_bytes_recovered":251527168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":434,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.677366+0000","last_change":"2024-03-30T13:52:47.924323+0000","last_active":"2024-03-30T13:55:36.677366+0000","last_peered":"2024-03-30T13:55:36.677366+0000","last_clean":"2024-03-30T13:55:36.677366+0000","last_became_active":"2024-03-30T13:52:36.483298+0000","last_became_peered":"2024-03-30T13:52:36.483298+0000","last_unstale":"2024-03-30T13:55:36.677366+0000","last_undegraded":"2024-03-30T13:55:36.677366+0000","last_fullsized":"2024-03-30T13:55:36.677366+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:47.924212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:47.924212+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":433,"reported_epoch":262,"state":"active+undersized","last_fresh":"2024-03-30T13:55:36.701325+0000","last_change":"2024-03-30T13:50:08.512648+0000","last_active":"2024-03-30T13:55:36.701325+0000","last_peered":"2024-03-30T13:55:36.701325+0000","last_clean":"2024-03-30T13:50:05.489180+0000","last_became_active":"2024-03-30T13:50:08.512648+0000","last_became_peered":"2024-03-30T13:50:08.512648+0000","last_unstale":"2024-03-30T13:55:36.701325+0000","last_undegraded":"2024-03-30T13:55:36.701325+0000","last_fullsized":"2024-03-30T13:50:08.505931+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:21.527712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:21.527712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":450,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.706003+0000","last_change":"2024-03-30T13:53:14.658635+0000","last_active":"2024-03-30T13:55:36.706003+0000","last_peered":"2024-03-30T13:55:36.706003+0000","last_clean":"2024-03-30T13:55:36.706003+0000","last_became_active":"2024-03-30T13:53:14.658165+0000","last_became_peered":"2024-03-30T13:53:14.658165+0000","last_unstale":"2024-03-30T13:55:36.706003+0000","last_undegraded":"2024-03-30T13:55:36.706003+0000","last_fullsized":"2024-03-30T13:55:36.706003+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:19.525007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:45.975163+0000","last_clean_scrub_stamp":"2024-03-30T13:49:19.525007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0,5],"avail_no_missing":["1","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"216'29","reported_seq":524,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.695932+0000","last_change":"2024-03-30T13:53:36.858207+0000","last_active":"2024-03-30T13:55:36.695932+0000","last_peered":"2024-03-30T13:55:36.695932+0000","last_clean":"2024-03-30T13:55:36.695932+0000","last_became_active":"2024-03-30T13:53:36.857573+0000","last_became_peered":"2024-03-30T13:53:36.857573+0000","last_unstale":"2024-03-30T13:55:36.695932+0000","last_undegraded":"2024-03-30T13:55:36.695932+0000","last_fullsized":"2024-03-30T13:55:36.695932+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-30T13:52:15.020534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:15.020534+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[{"shards":"0,1,7","objects":8}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":461,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.677744+0000","last_change":"2024-03-30T13:53:14.654792+0000","last_active":"2024-03-30T13:55:36.677744+0000","last_peered":"2024-03-30T13:55:36.677744+0000","last_clean":"2024-03-30T13:55:36.677744+0000","last_became_active":"2024-03-30T13:53:14.654346+0000","last_became_peered":"2024-03-30T13:53:14.654346+0000","last_unstale":"2024-03-30T13:55:36.677744+0000","last_undegraded":"2024-03-30T13:55:36.677744+0000","last_fullsized":"2024-03-30T13:55:36.677744+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:52.887014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:52.887014+0000","last_clean_scrub_stamp":"2024-03-30T13:52:52.887014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0,5],"avail_no_missing":["6","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"262'8085","reported_seq":8203,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:40.632443+0000","last_change":"2024-03-30T13:53:14.666961+0000","last_active":"2024-03-30T13:55:40.632443+0000","last_peered":"2024-03-30T13:55:40.632443+0000","last_clean":"2024-03-30T13:55:40.632443+0000","last_became_active":"2024-03-30T13:53:14.661730+0000","last_became_peered":"2024-03-30T13:53:14.661730+0000","last_unstale":"2024-03-30T13:55:40.632443+0000","last_undegraded":"2024-03-30T13:55:40.632443+0000","last_fullsized":"2024-03-30T13:55:40.632443+0000","mapping_epoch":222,"log_start":"236'4656","ondisk_log_start":"236'4656","created":211,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3429,"log_dups_size":0,"ondisk_log_size":3429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172163072,"num_objects":2627,"num_object_clones":0,"num_object_copies":7881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2627,"num_objects_unfound":0,"num_objects_dirty":2627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8085,"num_write_kb":342784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3,5],"avail_no_missing":["1","3","5"],"object_location_counts":[{"shards":"1,3,5","objects":2627}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":439,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.677529+0000","last_change":"2024-03-30T13:53:37.108143+0000","last_active":"2024-03-30T13:55:36.677529+0000","last_peered":"2024-03-30T13:55:36.677529+0000","last_clean":"2024-03-30T13:55:36.677529+0000","last_became_active":"2024-03-30T13:53:37.107659+0000","last_became_peered":"2024-03-30T13:53:37.107659+0000","last_unstale":"2024-03-30T13:55:36.677529+0000","last_undegraded":"2024-03-30T13:55:36.677529+0000","last_fullsized":"2024-03-30T13:55:36.677529+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:48.949941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:48.949941+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3,7],"avail_no_missing":["6","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":431,"reported_epoch":262,"state":"active+undersized","last_fresh":"2024-03-30T13:55:36.692489+0000","last_change":"2024-03-30T13:50:08.510025+0000","last_active":"2024-03-30T13:55:36.692489+0000","last_peered":"2024-03-30T13:55:36.692489+0000","last_clean":"2024-03-30T13:50:05.498930+0000","last_became_active":"2024-03-30T13:50:08.510025+0000","last_became_peered":"2024-03-30T13:50:08.510025+0000","last_unstale":"2024-03-30T13:55:36.692489+0000","last_undegraded":"2024-03-30T13:55:36.692489+0000","last_fullsized":"2024-03-30T13:50:08.502478+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:28.946303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:41.142941+0000","last_clean_scrub_stamp":"2024-03-30T13:49:28.946303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":339,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.730507+0000","last_change":"2024-03-30T13:53:14.651231+0000","last_active":"2024-03-30T13:55:36.730507+0000","last_peered":"2024-03-30T13:55:36.730507+0000","last_clean":"2024-03-30T13:55:36.730507+0000","last_became_active":"2024-03-30T13:53:14.650772+0000","last_became_peered":"2024-03-30T13:53:14.650772+0000","last_unstale":"2024-03-30T13:55:36.730507+0000","last_undegraded":"2024-03-30T13:55:36.730507+0000","last_fullsized":"2024-03-30T13:55:36.730507+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:08.543531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:55.458779+0000","last_clean_scrub_stamp":"2024-03-30T13:49:08.543531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,5],"avail_no_missing":["3","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"217'25","reported_seq":505,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.712379+0000","last_change":"2024-03-30T13:53:13.688776+0000","last_active":"2024-03-30T13:55:36.712379+0000","last_peered":"2024-03-30T13:55:36.712379+0000","last_clean":"2024-03-30T13:55:36.712379+0000","last_became_active":"2024-03-30T13:53:13.657632+0000","last_became_peered":"2024-03-30T13:53:13.657632+0000","last_unstale":"2024-03-30T13:55:36.712379+0000","last_undegraded":"2024-03-30T13:55:36.712379+0000","last_fullsized":"2024-03-30T13:55:36.712379+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'15","last_scrub_stamp":"2024-03-30T13:52:15.457275+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-03-30T13:45:50.945837+0000","last_clean_scrub_stamp":"2024-03-30T13:52:15.457275+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"222'10","reported_seq":623,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.696015+0000","last_change":"2024-03-30T13:53:13.669795+0000","last_active":"2024-03-30T13:55:36.696015+0000","last_peered":"2024-03-30T13:55:36.696015+0000","last_clean":"2024-03-30T13:55:36.696015+0000","last_became_active":"2024-03-30T13:53:13.658557+0000","last_became_peered":"2024-03-30T13:53:13.658557+0000","last_unstale":"2024-03-30T13:55:36.696015+0000","last_undegraded":"2024-03-30T13:55:36.696015+0000","last_fullsized":"2024-03-30T13:55:36.696015+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'9","last_scrub_stamp":"2024-03-30T13:52:57.873554+0000","last_deep_scrub":"218'9","last_deep_scrub_stamp":"2024-03-30T13:52:57.873554+0000","last_clean_scrub_stamp":"2024-03-30T13:52:57.873554+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"262'8015","reported_seq":8765,"reported_epoch":262,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-30T13:55:40.548732+0000","last_change":"2024-03-30T13:55:29.426190+0000","last_active":"2024-03-30T13:55:40.548732+0000","last_peered":"2024-03-30T13:55:40.548732+0000","last_clean":"2024-03-30T13:53:34.242680+0000","last_became_active":"2024-03-30T13:53:37.374630+0000","last_became_peered":"2024-03-30T13:53:37.374630+0000","last_unstale":"2024-03-30T13:55:40.548732+0000","last_undegraded":"2024-03-30T13:53:37.099819+0000","last_fullsized":"2024-03-30T13:53:36.841150+0000","mapping_epoch":229,"log_start":"234'4438","ondisk_log_start":"234'4438","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3577,"log_dups_size":0,"ondisk_log_size":3577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169410560,"num_objects":2585,"num_object_clones":0,"num_object_copies":7755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2400,"num_objects_misplaced":1050,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8015,"num_write_kb":339200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":317,"num_bytes_recovered":20774912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4","objects":-865},{"shards":"3,4,7","objects":1050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":455,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.734347+0000","last_change":"2024-03-30T13:53:38.553854+0000","last_active":"2024-03-30T13:55:36.734347+0000","last_peered":"2024-03-30T13:55:36.734347+0000","last_clean":"2024-03-30T13:55:36.734347+0000","last_became_active":"2024-03-30T13:53:38.553065+0000","last_became_peered":"2024-03-30T13:53:38.553065+0000","last_unstale":"2024-03-30T13:55:36.734347+0000","last_undegraded":"2024-03-30T13:55:36.734347+0000","last_fullsized":"2024-03-30T13:55:36.734347+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:48.866100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:48.866100+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":303,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.677886+0000","last_change":"2024-03-30T13:53:36.853679+0000","last_active":"2024-03-30T13:55:36.677886+0000","last_peered":"2024-03-30T13:55:36.677886+0000","last_clean":"2024-03-30T13:55:36.677886+0000","last_became_active":"2024-03-30T13:53:36.852004+0000","last_became_peered":"2024-03-30T13:53:36.852004+0000","last_unstale":"2024-03-30T13:55:36.677886+0000","last_undegraded":"2024-03-30T13:55:36.677886+0000","last_fullsized":"2024-03-30T13:55:36.677886+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:21.782290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:31.900908+0000","last_clean_scrub_stamp":"2024-03-30T13:49:21.782290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1,7],"avail_no_missing":["6","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":456,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.694919+0000","last_change":"2024-03-30T13:52:46.959816+0000","last_active":"2024-03-30T13:55:36.694919+0000","last_peered":"2024-03-30T13:55:36.694919+0000","last_clean":"2024-03-30T13:55:36.694919+0000","last_became_active":"2024-03-30T13:42:45.309886+0000","last_became_peered":"2024-03-30T13:42:45.309886+0000","last_unstale":"2024-03-30T13:55:36.694919+0000","last_undegraded":"2024-03-30T13:55:36.694919+0000","last_fullsized":"2024-03-30T13:55:36.694919+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:52:46.959694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:46.959694+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"217'35","reported_seq":526,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.659062+0000","last_change":"2024-03-30T13:53:13.687444+0000","last_active":"2024-03-30T13:55:36.659062+0000","last_peered":"2024-03-30T13:55:36.659062+0000","last_clean":"2024-03-30T13:55:36.659062+0000","last_became_active":"2024-03-30T13:53:13.652175+0000","last_became_peered":"2024-03-30T13:53:13.652175+0000","last_unstale":"2024-03-30T13:55:36.659062+0000","last_undegraded":"2024-03-30T13:55:36.659062+0000","last_fullsized":"2024-03-30T13:55:36.659062+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'21","last_scrub_stamp":"2024-03-30T13:52:44.925926+0000","last_deep_scrub":"91'21","last_deep_scrub_stamp":"2024-03-30T13:44:05.917064+0000","last_clean_scrub_stamp":"2024-03-30T13:52:44.925926+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":598,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.723230+0000","last_change":"2024-03-30T13:52:17.989527+0000","last_active":"2024-03-30T13:55:36.723230+0000","last_peered":"2024-03-30T13:55:36.723230+0000","last_clean":"2024-03-30T13:55:36.723230+0000","last_became_active":"2024-03-30T13:42:49.321423+0000","last_became_peered":"2024-03-30T13:42:49.321423+0000","last_unstale":"2024-03-30T13:55:36.723230+0000","last_undegraded":"2024-03-30T13:55:36.723230+0000","last_fullsized":"2024-03-30T13:55:36.723230+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:52:17.989349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:17.989349+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"262'8251","reported_seq":22115,"reported_epoch":262,"state":"active+recovery_wait","last_fresh":"2024-03-30T13:55:40.685989+0000","last_change":"2024-03-30T13:55:04.299728+0000","last_active":"2024-03-30T13:55:40.685989+0000","last_peered":"2024-03-30T13:55:40.685989+0000","last_clean":"2024-03-30T13:53:12.623816+0000","last_became_active":"2024-03-30T13:55:04.226781+0000","last_became_peered":"2024-03-30T13:55:04.226781+0000","last_unstale":"2024-03-30T13:55:40.685989+0000","last_undegraded":"2024-03-30T13:55:40.685989+0000","last_fullsized":"2024-03-30T13:55:40.685989+0000","mapping_epoch":253,"log_start":"238'4901","ondisk_log_start":"238'4901","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176947224,"num_objects":2701,"num_object_clones":0,"num_object_copies":8103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2701,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8251,"num_write_kb":350401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6870,"num_bytes_recovered":450101248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":["1","4","5","6","7"],"object_location_counts":[{"shards":"1,4,6","objects":2701}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":429,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.659425+0000","last_change":"2024-03-30T13:52:47.990769+0000","last_active":"2024-03-30T13:55:36.659425+0000","last_peered":"2024-03-30T13:55:36.659425+0000","last_clean":"2024-03-30T13:55:36.659425+0000","last_became_active":"2024-03-30T13:52:36.490909+0000","last_became_peered":"2024-03-30T13:52:36.490909+0000","last_unstale":"2024-03-30T13:55:36.659425+0000","last_undegraded":"2024-03-30T13:55:36.659425+0000","last_fullsized":"2024-03-30T13:55:36.659425+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:47.990635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:47.990635+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,0],"avail_no_missing":["4","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":426,"reported_epoch":262,"state":"active+undersized","last_fresh":"2024-03-30T13:55:36.658944+0000","last_change":"2024-03-30T13:50:08.504411+0000","last_active":"2024-03-30T13:55:36.658944+0000","last_peered":"2024-03-30T13:55:36.658944+0000","last_clean":"2024-03-30T13:50:05.489944+0000","last_became_active":"2024-03-30T13:50:08.504411+0000","last_became_peered":"2024-03-30T13:50:08.504411+0000","last_unstale":"2024-03-30T13:55:36.658944+0000","last_undegraded":"2024-03-30T13:55:36.658944+0000","last_fullsized":"2024-03-30T13:50:08.501823+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:10.216387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:10.216387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":455,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.739580+0000","last_change":"2024-03-30T13:53:35.851353+0000","last_active":"2024-03-30T13:55:36.739580+0000","last_peered":"2024-03-30T13:55:36.739580+0000","last_clean":"2024-03-30T13:55:36.739580+0000","last_became_active":"2024-03-30T13:53:35.851006+0000","last_became_peered":"2024-03-30T13:53:35.851006+0000","last_unstale":"2024-03-30T13:55:36.739580+0000","last_undegraded":"2024-03-30T13:55:36.739580+0000","last_fullsized":"2024-03-30T13:55:36.739580+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:53:01.031216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:53:01.031216+0000","last_clean_scrub_stamp":"2024-03-30T13:53:01.031216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"216'35","reported_seq":525,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.723089+0000","last_change":"2024-03-30T13:53:36.081652+0000","last_active":"2024-03-30T13:55:36.723089+0000","last_peered":"2024-03-30T13:55:36.723089+0000","last_clean":"2024-03-30T13:55:36.723089+0000","last_became_active":"2024-03-30T13:53:35.888652+0000","last_became_peered":"2024-03-30T13:53:35.888652+0000","last_unstale":"2024-03-30T13:55:36.723089+0000","last_undegraded":"2024-03-30T13:55:36.723089+0000","last_fullsized":"2024-03-30T13:55:36.723089+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'29","last_scrub_stamp":"2024-03-30T13:52:52.973602+0000","last_deep_scrub":"216'29","last_deep_scrub_stamp":"2024-03-30T13:52:52.973602+0000","last_clean_scrub_stamp":"2024-03-30T13:52:52.973602+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":441,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.659534+0000","last_change":"2024-03-30T13:53:35.863472+0000","last_active":"2024-03-30T13:55:36.659534+0000","last_peered":"2024-03-30T13:55:36.659534+0000","last_clean":"2024-03-30T13:55:36.659534+0000","last_became_active":"2024-03-30T13:53:35.863124+0000","last_became_peered":"2024-03-30T13:53:35.863124+0000","last_unstale":"2024-03-30T13:55:36.659534+0000","last_undegraded":"2024-03-30T13:55:36.659534+0000","last_fullsized":"2024-03-30T13:55:36.659534+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:09.247282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:51:09.247282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"262'8187","reported_seq":8296,"reported_epoch":262,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:55:40.531811+0000","last_change":"2024-03-30T13:52:51.714810+0000","last_active":"2024-03-30T13:55:40.531811+0000","last_peered":"2024-03-30T13:55:40.531811+0000","last_clean":"2024-03-30T13:52:42.503419+0000","last_became_active":"2024-03-30T13:52:43.528398+0000","last_became_peered":"2024-03-30T13:52:43.528398+0000","last_unstale":"2024-03-30T13:55:40.531811+0000","last_undegraded":"2024-03-30T13:52:51.712396+0000","last_fullsized":"2024-03-30T13:52:43.524380+0000","mapping_epoch":211,"log_start":"236'4647","ondisk_log_start":"236'4647","created":211,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3540,"log_dups_size":0,"ondisk_log_size":3540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176488448,"num_objects":2693,"num_object_clones":0,"num_object_copies":8079,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8187,"num_write_kb":348160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":2693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":391,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.676956+0000","last_change":"2024-03-30T13:53:16.673293+0000","last_active":"2024-03-30T13:55:36.676956+0000","last_peered":"2024-03-30T13:55:36.676956+0000","last_clean":"2024-03-30T13:55:36.676956+0000","last_became_active":"2024-03-30T13:53:16.672717+0000","last_became_peered":"2024-03-30T13:53:16.672717+0000","last_unstale":"2024-03-30T13:55:36.676956+0000","last_undegraded":"2024-03-30T13:55:36.676956+0000","last_fullsized":"2024-03-30T13:55:36.676956+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:49.875971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:49.875971+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:40.389484+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,5],"avail_no_missing":["6","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":93,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.711228+0000","last_change":"2024-03-30T13:53:14.661063+0000","last_active":"2024-03-30T13:55:36.711228+0000","last_peered":"2024-03-30T13:55:36.711228+0000","last_clean":"2024-03-30T13:55:36.711228+0000","last_became_active":"2024-03-30T13:53:14.660483+0000","last_became_peered":"2024-03-30T13:53:14.660483+0000","last_unstale":"2024-03-30T13:55:36.711228+0000","last_undegraded":"2024-03-30T13:55:36.711228+0000","last_fullsized":"2024-03-30T13:55:36.711228+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:14.144050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:14.144050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0,5],"avail_no_missing":["1","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"223'11","reported_seq":613,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.660560+0000","last_change":"2024-03-30T13:53:14.651471+0000","last_active":"2024-03-30T13:55:36.660560+0000","last_peered":"2024-03-30T13:55:36.660560+0000","last_clean":"2024-03-30T13:55:36.660560+0000","last_became_active":"2024-03-30T13:53:14.650289+0000","last_became_peered":"2024-03-30T13:53:14.650289+0000","last_unstale":"2024-03-30T13:55:36.660560+0000","last_undegraded":"2024-03-30T13:55:36.660560+0000","last_fullsized":"2024-03-30T13:55:36.660560+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-30T13:51:15.146255+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-30T13:51:15.146255+0000","last_clean_scrub_stamp":"2024-03-30T13:51:15.146255+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,5],"avail_no_missing":["4","0","5"],"object_location_counts":[{"shards":"0,4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":447,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.723003+0000","last_change":"2024-03-30T13:53:35.887442+0000","last_active":"2024-03-30T13:55:36.723003+0000","last_peered":"2024-03-30T13:55:36.723003+0000","last_clean":"2024-03-30T13:55:36.723003+0000","last_became_active":"2024-03-30T13:53:35.886750+0000","last_became_peered":"2024-03-30T13:53:35.886750+0000","last_unstale":"2024-03-30T13:55:36.723003+0000","last_undegraded":"2024-03-30T13:55:36.723003+0000","last_fullsized":"2024-03-30T13:55:36.723003+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:27.058807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:27.058807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"217'34","reported_seq":503,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.717900+0000","last_change":"2024-03-30T13:52:52.619670+0000","last_active":"2024-03-30T13:55:36.717900+0000","last_peered":"2024-03-30T13:55:36.717900+0000","last_clean":"2024-03-30T13:55:36.717900+0000","last_became_active":"2024-03-30T13:42:47.324350+0000","last_became_peered":"2024-03-30T13:42:47.324350+0000","last_unstale":"2024-03-30T13:55:36.717900+0000","last_undegraded":"2024-03-30T13:55:36.717900+0000","last_fullsized":"2024-03-30T13:55:36.717900+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'22","last_scrub_stamp":"2024-03-30T13:52:52.619482+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:28.831527+0000","last_clean_scrub_stamp":"2024-03-30T13:52:52.619482+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"262'8180","reported_seq":11686,"reported_epoch":262,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-30T13:55:40.562528+0000","last_change":"2024-03-30T13:54:57.567600+0000","last_active":"2024-03-30T13:55:40.562528+0000","last_peered":"2024-03-30T13:55:40.562528+0000","last_clean":"2024-03-30T13:53:34.056415+0000","last_became_active":"2024-03-30T13:53:37.267573+0000","last_became_peered":"2024-03-30T13:53:37.267573+0000","last_unstale":"2024-03-30T13:55:40.562528+0000","last_undegraded":"2024-03-30T13:53:37.097392+0000","last_fullsized":"2024-03-30T13:53:36.839832+0000","mapping_epoch":229,"log_start":"234'4575","ondisk_log_start":"234'4575","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3605,"log_dups_size":0,"ondisk_log_size":3605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175243264,"num_objects":2674,"num_object_clones":0,"num_object_copies":8022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1246,"num_objects_misplaced":863,"num_objects_unfound":0,"num_objects_dirty":2674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8180,"num_write_kb":347328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1695,"num_bytes_recovered":111083520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3","objects":565},{"shards":"1,3,7","objects":863}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":441,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.734814+0000","last_change":"2024-03-30T13:53:35.875435+0000","last_active":"2024-03-30T13:55:36.734814+0000","last_peered":"2024-03-30T13:55:36.734814+0000","last_clean":"2024-03-30T13:55:36.734814+0000","last_became_active":"2024-03-30T13:53:35.875118+0000","last_became_peered":"2024-03-30T13:53:35.875118+0000","last_unstale":"2024-03-30T13:55:36.734814+0000","last_undegraded":"2024-03-30T13:55:36.734814+0000","last_fullsized":"2024-03-30T13:55:36.734814+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:21.081938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:21.081938+0000","last_clean_scrub_stamp":"2024-03-30T13:52:21.081938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":92,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.659558+0000","last_change":"2024-03-30T13:53:14.656429+0000","last_active":"2024-03-30T13:55:36.659558+0000","last_peered":"2024-03-30T13:55:36.659558+0000","last_clean":"2024-03-30T13:55:36.659558+0000","last_became_active":"2024-03-30T13:53:14.655908+0000","last_became_peered":"2024-03-30T13:53:14.655908+0000","last_unstale":"2024-03-30T13:55:36.659558+0000","last_undegraded":"2024-03-30T13:55:36.659558+0000","last_fullsized":"2024-03-30T13:55:36.659558+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:21.087732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:52.603772+0000","last_clean_scrub_stamp":"2024-03-30T13:51:21.087732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,5],"avail_no_missing":["4","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":442,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.658751+0000","last_change":"2024-03-30T13:53:36.852955+0000","last_active":"2024-03-30T13:55:36.658751+0000","last_peered":"2024-03-30T13:55:36.658751+0000","last_clean":"2024-03-30T13:55:36.658751+0000","last_became_active":"2024-03-30T13:53:36.852500+0000","last_became_peered":"2024-03-30T13:53:36.852500+0000","last_unstale":"2024-03-30T13:55:36.658751+0000","last_undegraded":"2024-03-30T13:55:36.658751+0000","last_fullsized":"2024-03-30T13:55:36.658751+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:15.107879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:48:15.107879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,7],"avail_no_missing":["4","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"217'19","reported_seq":485,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.660390+0000","last_change":"2024-03-30T13:52:17.057583+0000","last_active":"2024-03-30T13:55:36.660390+0000","last_peered":"2024-03-30T13:55:36.660390+0000","last_clean":"2024-03-30T13:55:36.660390+0000","last_became_active":"2024-03-30T13:42:47.315667+0000","last_became_peered":"2024-03-30T13:42:47.315667+0000","last_unstale":"2024-03-30T13:55:36.660390+0000","last_undegraded":"2024-03-30T13:55:36.660390+0000","last_fullsized":"2024-03-30T13:55:36.660390+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:52:17.057433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:17.057433+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"262'8229","reported_seq":8338,"reported_epoch":262,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:55:40.671500+0000","last_change":"2024-03-30T13:52:51.696416+0000","last_active":"2024-03-30T13:55:40.671500+0000","last_peered":"2024-03-30T13:55:40.671500+0000","last_clean":"2024-03-30T13:52:42.503419+0000","last_became_active":"2024-03-30T13:52:43.605206+0000","last_became_peered":"2024-03-30T13:52:43.605206+0000","last_unstale":"2024-03-30T13:55:40.671500+0000","last_undegraded":"2024-03-30T13:52:51.691832+0000","last_fullsized":"2024-03-30T13:52:43.519471+0000","mapping_epoch":211,"log_start":"240'4895","ondisk_log_start":"240'4895","created":211,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3334,"log_dups_size":0,"ondisk_log_size":3334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172425216,"num_objects":2631,"num_object_clones":0,"num_object_copies":7893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8229,"num_write_kb":347520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":2631}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":398,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.722349+0000","last_change":"2024-03-30T13:53:16.672844+0000","last_active":"2024-03-30T13:55:36.722349+0000","last_peered":"2024-03-30T13:55:36.722349+0000","last_clean":"2024-03-30T13:55:36.722349+0000","last_became_active":"2024-03-30T13:53:16.671377+0000","last_became_peered":"2024-03-30T13:53:16.671377+0000","last_unstale":"2024-03-30T13:55:36.722349+0000","last_undegraded":"2024-03-30T13:55:36.722349+0000","last_fullsized":"2024-03-30T13:55:36.722349+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:47.932192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:47.932192+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:39.052419+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":423697,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,5],"avail_no_missing":["3","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":213,"reported_epoch":262,"state":"active+undersized","last_fresh":"2024-03-30T13:55:36.729079+0000","last_change":"2024-03-30T13:50:08.511676+0000","last_active":"2024-03-30T13:55:36.729079+0000","last_peered":"2024-03-30T13:55:36.729079+0000","last_clean":"2024-03-30T13:50:05.504989+0000","last_became_active":"2024-03-30T13:50:08.511676+0000","last_became_peered":"2024-03-30T13:50:08.511676+0000","last_unstale":"2024-03-30T13:55:36.729079+0000","last_undegraded":"2024-03-30T13:55:36.729079+0000","last_fullsized":"2024-03-30T13:50:08.504157+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:09.357378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:09.357378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":449,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.706424+0000","last_change":"2024-03-30T13:52:13.440265+0000","last_active":"2024-03-30T13:55:36.706424+0000","last_peered":"2024-03-30T13:55:36.706424+0000","last_clean":"2024-03-30T13:55:36.706424+0000","last_became_active":"2024-03-30T13:42:49.321360+0000","last_became_peered":"2024-03-30T13:42:49.321360+0000","last_unstale":"2024-03-30T13:55:36.706424+0000","last_undegraded":"2024-03-30T13:55:36.706424+0000","last_fullsized":"2024-03-30T13:55:36.706424+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:13.440150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:13.440150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":463,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.677294+0000","last_change":"2024-03-30T13:52:25.063121+0000","last_active":"2024-03-30T13:55:36.677294+0000","last_peered":"2024-03-30T13:55:36.677294+0000","last_clean":"2024-03-30T13:55:36.677294+0000","last_became_active":"2024-03-30T13:42:45.315251+0000","last_became_peered":"2024-03-30T13:42:45.315251+0000","last_unstale":"2024-03-30T13:55:36.677294+0000","last_undegraded":"2024-03-30T13:55:36.677294+0000","last_fullsized":"2024-03-30T13:55:36.677294+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:25.062771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:25.062771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"217'31","reported_seq":482,"reported_epoch":262,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:55:36.659985+0000","last_change":"2024-03-30T13:50:08.508030+0000","last_active":"2024-03-30T13:55:36.659985+0000","last_peered":"2024-03-30T13:55:36.659985+0000","last_clean":"2024-03-30T13:50:05.491159+0000","last_became_active":"2024-03-30T13:50:08.508030+0000","last_became_peered":"2024-03-30T13:50:08.508030+0000","last_unstale":"2024-03-30T13:55:36.659985+0000","last_undegraded":"2024-03-30T13:50:08.502958+0000","last_fullsized":"2024-03-30T13:50:08.502815+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-03-30T13:48:27.096261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:27.096261+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"262'8312","reported_seq":16206,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:40.572750+0000","last_change":"2024-03-30T13:54:59.552555+0000","last_active":"2024-03-30T13:55:40.572750+0000","last_peered":"2024-03-30T13:55:40.572750+0000","last_clean":"2024-03-30T13:55:40.572750+0000","last_became_active":"2024-03-30T13:54:59.538170+0000","last_became_peered":"2024-03-30T13:54:59.538170+0000","last_unstale":"2024-03-30T13:55:40.572750+0000","last_undegraded":"2024-03-30T13:55:40.572750+0000","last_fullsized":"2024-03-30T13:55:40.572750+0000","mapping_epoch":251,"log_start":"236'4756","ondisk_log_start":"236'4756","created":211,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:42.503419+0000","last_clean_scrub_stamp":"2024-03-30T13:52:42.503419+0000","objects_scrubbed":0,"log_size":3556,"log_dups_size":0,"ondisk_log_size":3556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181665792,"num_objects":2772,"num_object_clones":0,"num_object_copies":8316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8312,"num_write_kb":354688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3885,"num_bytes_recovered":254607360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":672,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.722974+0000","last_change":"2024-03-30T13:53:38.550111+0000","last_active":"2024-03-30T13:55:36.722974+0000","last_peered":"2024-03-30T13:55:36.722974+0000","last_clean":"2024-03-30T13:55:36.722974+0000","last_became_active":"2024-03-30T13:53:38.543500+0000","last_became_peered":"2024-03-30T13:53:38.543500+0000","last_unstale":"2024-03-30T13:55:36.722974+0000","last_undegraded":"2024-03-30T13:55:36.722974+0000","last_fullsized":"2024-03-30T13:55:36.722974+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:48.985609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:48.985609+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:55:39.052443+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,7],"avail_no_missing":["3","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":425,"reported_epoch":262,"state":"active+undersized","last_fresh":"2024-03-30T13:55:36.660561+0000","last_change":"2024-03-30T13:50:08.511010+0000","last_active":"2024-03-30T13:55:36.660561+0000","last_peered":"2024-03-30T13:55:36.660561+0000","last_clean":"2024-03-30T13:50:05.490535+0000","last_became_active":"2024-03-30T13:50:08.511010+0000","last_became_peered":"2024-03-30T13:50:08.511010+0000","last_unstale":"2024-03-30T13:55:36.660561+0000","last_undegraded":"2024-03-30T13:55:36.660561+0000","last_fullsized":"2024-03-30T13:50:08.503118+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:06.154805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:06.154805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"231'13","reported_seq":303,"reported_epoch":262,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:55:36.694683+0000","last_change":"2024-03-30T13:53:37.600947+0000","last_active":"2024-03-30T13:55:36.694683+0000","last_peered":"2024-03-30T13:55:36.694683+0000","last_clean":"2024-03-30T13:55:36.694683+0000","last_became_active":"2024-03-30T13:53:37.518434+0000","last_became_peered":"2024-03-30T13:53:37.518434+0000","last_unstale":"2024-03-30T13:55:36.694683+0000","last_undegraded":"2024-03-30T13:55:36.694683+0000","last_fullsized":"2024-03-30T13:55:36.694683+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:49:36.397622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:36.397622+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3,7],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":441,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.738028+0000","last_change":"2024-03-30T13:51:58.939856+0000","last_active":"2024-03-30T13:55:36.738028+0000","last_peered":"2024-03-30T13:55:36.738028+0000","last_clean":"2024-03-30T13:55:36.738028+0000","last_became_active":"2024-03-30T13:42:45.309265+0000","last_became_peered":"2024-03-30T13:42:45.309265+0000","last_unstale":"2024-03-30T13:55:36.738028+0000","last_undegraded":"2024-03-30T13:55:36.738028+0000","last_fullsized":"2024-03-30T13:55:36.738028+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:51:58.939556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:51:58.939556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"216'20","reported_seq":476,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-30T13:55:36.659780+0000","last_change":"2024-03-30T13:51:59.022459+0000","last_active":"2024-03-30T13:55:36.659780+0000","last_peered":"2024-03-30T13:55:36.659780+0000","last_clean":"2024-03-30T13:55:36.659780+0000","last_became_active":"2024-03-30T13:42:47.313026+0000","last_became_peered":"2024-03-30T13:42:47.313026+0000","last_unstale":"2024-03-30T13:55:36.659780+0000","last_undegraded":"2024-03-30T13:55:36.659780+0000","last_fullsized":"2024-03-30T13:55:36.659780+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-30T13:51:59.022304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:59.022304+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2799304728,"num_objects":42715,"num_object_clones":0,"num_object_copies":128145,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":17433,"num_objects_unfound":0,"num_objects_dirty":42715,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130913,"num_write_kb":5556033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33217,"num_bytes_recovered":2176712704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10541477888,"data_stored":10541465672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57203,"ondisk_log_size":57203,"up":42,"acting":40,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":125,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":649698,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":109,"num_read_kb":161,"num_write":98,"num_write_kb":1316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1884104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":26,"acting":33,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":5246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":86,"acting":93,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47,"num_objects_misplaced":20,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":2244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":12206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":740,"ondisk_log_size":740,"up":85,"acting":89,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":82,"acting":93,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":77,"acting":91,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":249108103328,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397668,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1637628,"kb_used_data":1223284,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92107524,"statfs":{"total":95995035648,"available":94318104576,"internally_reserved":0,"allocated":1252642816,"data_stored":1251737411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":424278470},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575038,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2557764,"kb_used_data":1962940,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":91187388,"statfs":{"total":95995035648,"available":93375885312,"internally_reserved":0,"allocated":2010050560,"data_stored":2009072202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738563,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1421084,"kb_used_data":970644,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":92324068,"statfs":{"total":95995035648,"available":94539845632,"internally_reserved":0,"allocated":993939456,"data_stored":993001614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,0,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411446,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2213288,"kb_used_data":2015520,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":91531864,"statfs":{"total":95995035648,"available":93728628736,"internally_reserved":0,"allocated":2063892480,"data_stored":2063509110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182580,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2364196,"kb_used_data":1783132,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":91380956,"statfs":{"total":95995035648,"available":93574098944,"internally_reserved":0,"allocated":1825927168,"data_stored":1824981473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":594868679},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953646,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1247576,"kb_used_data":891532,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92497576,"statfs":{"total":95995035648,"available":94717517824,"internally_reserved":0,"allocated":912928768,"data_stored":911998343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":171798692009,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":2778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":980811776,"data_stored":980811776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1996951552,"data_stored":1996947480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1811087360,"data_stored":1811087360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":897126400,"data_stored":897122328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1800601600,"data_stored":1800601600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1248071680,"data_stored":1248067608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1806827520,"data_stored":1806827520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:55:52.743 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool set unique_pool_2 pgp_num 16 2024-03-30T13:55:52.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:52.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:52.845 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:52.872 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:52.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:52.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:52.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:52 smithi043 ceph-mon[29963]: pgmap v558: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 778 op/s; 12559/107979 objects degraded (11.631%); 12661/107979 objects misplaced (11.725%); 18 MiB/s, 281 objects/s recovering 2024-03-30T13:55:52.933 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:52 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 12559/107979 objects degraded (11.631%), 14 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:52.933 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:52 smithi043 ceph-mon[42099]: pgmap v558: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 778 op/s; 12559/107979 objects degraded (11.631%); 12661/107979 objects misplaced (11.725%); 18 MiB/s, 281 objects/s recovering 2024-03-30T13:55:52.933 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:52 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 12559/107979 objects degraded (11.631%), 14 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:52.974 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:53.002 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:53.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:53.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:53.103 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:53.128 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:53.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:53.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:53.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:52 smithi113 ceph-mon[34341]: pgmap v558: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 778 op/s; 12559/107979 objects degraded (11.631%); 12661/107979 objects misplaced (11.725%); 18 MiB/s, 281 objects/s recovering 2024-03-30T13:55:53.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:52 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 12559/107979 objects degraded (11.631%), 14 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:53.230 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:53.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:53.256 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:53.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:53.357 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:53.425 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:53.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:53.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:53.527 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:53.552 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:53.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:53.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:53.653 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:53.679 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:53.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:53.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:53.782 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:53.807 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:53.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:53.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:53.909 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:53.934 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:53.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:53.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:54.036 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:54.078 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:54.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:54.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:54.180 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:54.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:54.205 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:54.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:54.306 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:54.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:54.331 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:54.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:54.432 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:54.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:54.464 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:54.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:54.566 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:54.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:54.590 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:54.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:54.691 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:54.723 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:54.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:54.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:54.767 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-03-30T13:55:54.826 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:54.850 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:54.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:54.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:54.858 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:55:54.859 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:55:54.859 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:55:54.859 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:55:54.859 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:55:54.859 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:55:54.859 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:55:54.859 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:55:54.859 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:55:54.859 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:55:54.859 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:55:54.859 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:55:54.860 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:55:54.860 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:55:54.860 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:55:54.952 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:55.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:55.002 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:55.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:55.103 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:55.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:55.130 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:55.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:55.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:54 smithi043 ceph-mon[42099]: pgmap v559: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 692 op/s; 12559/107979 objects degraded (11.631%); 12661/107979 objects misplaced (11.725%); 16 MiB/s, 250 objects/s recovering 2024-03-30T13:55:55.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:54 smithi043 ceph-mon[42099]: osdmap e265: 8 total, 7 up, 6 in 2024-03-30T13:55:55.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:54 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1571940451' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:55:55.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:54 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3281617190' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:55:55.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:54 smithi043 ceph-mon[29963]: pgmap v559: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 692 op/s; 12559/107979 objects degraded (11.631%); 12661/107979 objects misplaced (11.725%); 16 MiB/s, 250 objects/s recovering 2024-03-30T13:55:55.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:54 smithi043 ceph-mon[29963]: osdmap e265: 8 total, 7 up, 6 in 2024-03-30T13:55:55.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:54 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1571940451' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:55:55.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:54 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3281617190' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:55:55.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:54 smithi113 ceph-mon[34341]: pgmap v559: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 692 op/s; 12559/107979 objects degraded (11.631%); 12661/107979 objects misplaced (11.725%); 16 MiB/s, 250 objects/s recovering 2024-03-30T13:55:55.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:54 smithi113 ceph-mon[34341]: osdmap e265: 8 total, 7 up, 6 in 2024-03-30T13:55:55.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:54 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1571940451' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:55:55.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:54 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3281617190' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:55:55.231 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:55.256 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:55.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:55.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:55.358 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:55.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:55.392 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:55.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:55.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:55.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:55.538 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:55.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:55.639 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:55.667 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:55.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:55.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:55.769 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:55.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:55.797 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:55.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:55.898 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:55.924 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:55.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:55.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:56.026 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:56.052 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:56.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:56.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:56.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:55 smithi043 ceph-mon[29963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:55:56.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:55 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1571940451' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:55:56.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:55 smithi043 ceph-mon[29963]: osdmap e266: 8 total, 7 up, 6 in 2024-03-30T13:55:56.093 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:55 smithi043 ceph-mon[42099]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:55:56.093 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:55 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1571940451' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:55:56.093 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:55 smithi043 ceph-mon[42099]: osdmap e266: 8 total, 7 up, 6 in 2024-03-30T13:55:56.154 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:56.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:56.201 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:56.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:56.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:55 smithi113 ceph-mon[34341]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:55:56.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:55 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1571940451' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:55:56.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:55 smithi113 ceph-mon[34341]: osdmap e266: 8 total, 7 up, 6 in 2024-03-30T13:55:56.302 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:56.329 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:56.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:56.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:56.394 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:55:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:55:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:55:56.431 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:56.457 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:56.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:56.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:56.559 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:56.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:56.586 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:56.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:56.687 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:56.727 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:56.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:56.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:56.829 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:56.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:56.875 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:56.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:56.976 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:57.002 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:57.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:57.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:57.104 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:57.131 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:57.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:57.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:57.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:56 smithi043 ceph-mon[29963]: pgmap v562: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.1 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 580 op/s; 12595/105264 objects degraded (11.965%); 11708/105264 objects misplaced (11.123%); 17 MiB/s, 265 objects/s recovering 2024-03-30T13:55:57.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:56 smithi043 ceph-mon[42099]: pgmap v562: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.1 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 580 op/s; 12595/105264 objects degraded (11.965%); 11708/105264 objects misplaced (11.123%); 17 MiB/s, 265 objects/s recovering 2024-03-30T13:55:57.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:56 smithi113 ceph-mon[34341]: pgmap v562: 158 pgs: 83 active+clean, 1 active+recovering+undersized+remapped, 12 active+undersized, 2 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.1 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 580 op/s; 12595/105264 objects degraded (11.965%); 11708/105264 objects misplaced (11.123%); 17 MiB/s, 265 objects/s recovering 2024-03-30T13:55:57.233 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:57.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:57.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:57.373 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:55:57.400 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:57.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:57.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:55:57.502 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:57.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:57.530 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:57.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:57.631 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:57.658 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:57.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:57.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:57.760 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:57.783 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:57.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:57.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:57.885 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:57.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:57.927 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:57.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:58.028 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:58.071 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:58.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:58.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:58.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:57 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 12595/105264 objects degraded (11.965%), 14 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:58.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:57 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 12595/105264 objects degraded (11.965%), 14 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:58.173 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:58.212 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:58.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:58.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:58.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:57 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 12595/105264 objects degraded (11.965%), 14 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:55:58.314 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:58.352 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:58.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:58.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:58.454 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:58.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:58.481 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:58.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:58.582 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:58.621 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:58.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:58.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:58.723 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:55:58.747 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:58.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:58.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:55:58.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:55:58.890 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:58.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:58.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:55:58.992 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:59.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:59.016 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:59.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:59.117 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:59.144 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:59.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:58 smithi043 ceph-mon[42099]: pgmap v563: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.0 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 868 op/s; 11967/97671 objects degraded (12.252%); 10165/97671 objects misplaced (10.407%); 25 MiB/s, 403 objects/s recovering 2024-03-30T13:55:59.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:58 smithi043 ceph-mon[29963]: pgmap v563: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.0 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 868 op/s; 11967/97671 objects degraded (12.252%); 10165/97671 objects misplaced (10.407%); 25 MiB/s, 403 objects/s recovering 2024-03-30T13:55:59.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:59.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:59.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:58 smithi113 ceph-mon[34341]: pgmap v563: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.0 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 868 op/s; 11967/97671 objects degraded (12.252%); 10165/97671 objects misplaced (10.407%); 25 MiB/s, 403 objects/s recovering 2024-03-30T13:55:59.218 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:55:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:55:59.246 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:55:59.273 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:59.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:59.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:55:59.374 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:59.417 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:59.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:59.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:59.519 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:55:59.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:59.562 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:55:59.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:55:59.663 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:55:59.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:59.687 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:59.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:55:59.788 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:59.814 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:59.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:59.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:55:59.916 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:55:59.941 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:55:59.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:55:59.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:00.043 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:00.072 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:00.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:00.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:59 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:56:00.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:59 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:56:00.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:55:59 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:56:00.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:59 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:56:00.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:59 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:56:00.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:55:59 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:56:00.174 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:00.197 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:00.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:00.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:00.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:59 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:56:00.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:59 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:56:00.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:55:59 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:56:00.299 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:00.327 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:00.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:00.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:00.429 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:00.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:00.454 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:00.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:00.555 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:00.582 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:00.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:00.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:00.684 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:00.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:00.719 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:00.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:00.821 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:00.844 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:00.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:00.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:00.946 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:00.989 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:00.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:00.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:01.091 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:01.122 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:01.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:01.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:01.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:00 smithi043 ceph-mon[29963]: pgmap v564: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.0 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 455 op/s; 11967/97050 objects degraded (12.331%); 10165/97050 objects misplaced (10.474%); 14 MiB/s, 225 objects/s recovering 2024-03-30T13:56:01.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:00 smithi043 ceph-mon[42099]: pgmap v564: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.0 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 455 op/s; 11967/97050 objects degraded (12.331%); 10165/97050 objects misplaced (10.474%); 14 MiB/s, 225 objects/s recovering 2024-03-30T13:56:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:00 smithi113 ceph-mon[34341]: pgmap v564: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 2.0 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 455 op/s; 11967/97050 objects degraded (12.331%); 10165/97050 objects misplaced (10.474%); 14 MiB/s, 225 objects/s recovering 2024-03-30T13:56:01.224 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:01.251 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:01.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:01.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:01.353 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:01.395 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:01.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:01.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:01.497 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:01.523 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:01.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:01.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:01.625 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:01.666 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:01.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:01.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:01.768 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:01.816 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:01.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:01.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:01.918 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:01.944 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:01.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:01.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:02.046 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:02.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:02.081 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:02.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:02.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:01 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:56:02.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:01 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:56:02.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:01 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:56:02.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:01 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:56:02.182 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:02.205 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:02.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:02.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:02.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:01 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:56:02.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:01 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:56:02.307 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:02.333 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:02.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:02.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:02.435 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:02.462 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:02.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:02.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:02.564 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:02.609 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:02.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:02.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:02.713 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:02.740 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:02.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:02.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:02.842 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:02.868 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:02.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:02.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:02.970 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:03.012 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:03.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:03.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:03.114 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:03.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:03.140 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:03.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:03.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:02 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:56:03.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:02 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:56:03.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:02 smithi043 ceph-mon[42099]: pgmap v565: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.7 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s; 10550/85974 objects degraded (12.271%); 8713/85974 objects misplaced (10.134%); 19 MiB/s, 401 objects/s recovering 2024-03-30T13:56:03.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:02 smithi043 ceph-mon[42099]: osdmap e267: 8 total, 7 up, 6 in 2024-03-30T13:56:03.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:02 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 10550/85974 objects degraded (12.271%), 14 pgs degraded, 26 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:03.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:02 smithi043 ceph-mon[42099]: osdmap e268: 8 total, 7 up, 6 in 2024-03-30T13:56:03.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:02 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:56:03.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:02 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:56:03.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:02 smithi043 ceph-mon[29963]: pgmap v565: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.7 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s; 10550/85974 objects degraded (12.271%); 8713/85974 objects misplaced (10.134%); 19 MiB/s, 401 objects/s recovering 2024-03-30T13:56:03.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:02 smithi043 ceph-mon[29963]: osdmap e267: 8 total, 7 up, 6 in 2024-03-30T13:56:03.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:02 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 10550/85974 objects degraded (12.271%), 14 pgs degraded, 26 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:03.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:02 smithi043 ceph-mon[29963]: osdmap e268: 8 total, 7 up, 6 in 2024-03-30T13:56:03.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:02 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:56:03.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:02 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:56:03.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:02 smithi113 ceph-mon[34341]: pgmap v565: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.7 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s; 10550/85974 objects degraded (12.271%); 8713/85974 objects misplaced (10.134%); 19 MiB/s, 401 objects/s recovering 2024-03-30T13:56:03.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:02 smithi113 ceph-mon[34341]: osdmap e267: 8 total, 7 up, 6 in 2024-03-30T13:56:03.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:02 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 10550/85974 objects degraded (12.271%), 14 pgs degraded, 26 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:03.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:02 smithi113 ceph-mon[34341]: osdmap e268: 8 total, 7 up, 6 in 2024-03-30T13:56:03.241 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:03.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:03.294 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:03.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:03.395 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:03.419 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:03.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:03.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:03.520 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:03.546 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:03.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:03.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:03.648 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:03.690 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:03.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:03.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:03.792 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:03.832 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:03.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:03.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:03.934 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:03.962 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:03.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:03.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:04.064 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:04.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:04.092 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:04.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:04.193 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:04.220 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:04.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:04.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:04.322 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:04.346 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:04.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:04.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:04.400 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:56:04.400 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:56:04.448 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:04.475 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:04.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:04.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:04.577 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:04.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:04.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:04.719 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:04.747 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:04.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:04.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:04.849 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:04.875 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:04.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:04.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:04.977 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:05.020 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:05.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:05.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:05.122 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:05.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:04 smithi043 ceph-mon[29963]: pgmap v568: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.7 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 804 op/s; 10550/85974 objects degraded (12.271%); 8713/85974 objects misplaced (10.134%); 13 MiB/s, 314 objects/s recovering 2024-03-30T13:56:05.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:04 smithi043 ceph-mon[42099]: pgmap v568: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.7 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 804 op/s; 10550/85974 objects degraded (12.271%); 8713/85974 objects misplaced (10.134%); 13 MiB/s, 314 objects/s recovering 2024-03-30T13:56:05.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:05.164 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:05.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:05.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:04 smithi113 ceph-mon[34341]: pgmap v568: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+recovery_wait, 5 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.7 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 804 op/s; 10550/85974 objects degraded (12.271%); 8713/85974 objects misplaced (10.134%); 13 MiB/s, 314 objects/s recovering 2024-03-30T13:56:05.265 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:05.288 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:05.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:05.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:05.390 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:05.416 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:05.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:05.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:05.518 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:05.544 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:05.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:05.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:05.645 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:05.669 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:05.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:05.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:05.771 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:05.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:05.798 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:05.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:05.899 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:05.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:05.926 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:05.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:06.027 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:06.050 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:06.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:06.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:06.152 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:06.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:06.187 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:06.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:06.288 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:06.314 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:06.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:06.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:06.394 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:56:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:56:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:56:06.416 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:06.442 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:06.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:06.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:06.544 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:06.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:06.568 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:06.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:06.669 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:06.696 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:06.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:06.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:06.798 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:06.821 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:06.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:06.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:06.923 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:06.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:06.948 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:06.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:07.049 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:07.076 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:07.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:07.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:07.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:06 smithi043 ceph-mon[29963]: pgmap v569: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+undersized+remapped, 1 active+recovery_wait, 4 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.7 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 612 op/s; 9738/82989 objects degraded (11.734%); 8359/82989 objects misplaced (10.072%); 5.9 MiB/s, 291 objects/s recovering 2024-03-30T13:56:07.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:06 smithi043 ceph-mon[29963]: osdmap e269: 8 total, 7 up, 6 in 2024-03-30T13:56:07.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:06 smithi043 ceph-mon[42099]: pgmap v569: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+undersized+remapped, 1 active+recovery_wait, 4 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.7 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 612 op/s; 9738/82989 objects degraded (11.734%); 8359/82989 objects misplaced (10.072%); 5.9 MiB/s, 291 objects/s recovering 2024-03-30T13:56:07.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:06 smithi043 ceph-mon[42099]: osdmap e269: 8 total, 7 up, 6 in 2024-03-30T13:56:07.178 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:07.204 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:07.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:07.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:07.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:06 smithi113 ceph-mon[34341]: pgmap v569: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+undersized+remapped, 1 active+recovery_wait, 4 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.7 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 612 op/s; 9738/82989 objects degraded (11.734%); 8359/82989 objects misplaced (10.072%); 5.9 MiB/s, 291 objects/s recovering 2024-03-30T13:56:07.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:06 smithi113 ceph-mon[34341]: osdmap e269: 8 total, 7 up, 6 in 2024-03-30T13:56:07.306 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:07.333 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:07.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:07.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:07.435 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:07.463 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:07.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:07.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:07.565 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:07.588 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:07.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:07.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:07.690 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:07.714 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:07.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:07.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:07.816 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:07.839 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:07.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:07.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:07.941 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:07.964 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:07.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:07.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:08.066 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:08.093 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:08.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:08.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:08.195 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:08.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:08.219 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:08.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:08.320 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:08.344 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:08.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:08.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:08.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:07 smithi043 ceph-mon[29963]: osdmap e270: 8 total, 7 up, 6 in 2024-03-30T13:56:08.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:07 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 9738/82989 objects degraded (11.734%), 13 pgs degraded, 26 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:08.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:07 smithi043 ceph-mon[42099]: osdmap e270: 8 total, 7 up, 6 in 2024-03-30T13:56:08.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:07 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 9738/82989 objects degraded (11.734%), 13 pgs degraded, 26 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:08.445 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:08.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:07 smithi113 ceph-mon[34341]: osdmap e270: 8 total, 7 up, 6 in 2024-03-30T13:56:08.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:07 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 9738/82989 objects degraded (11.734%), 13 pgs degraded, 26 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:08.472 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:08.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:08.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:08.574 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:08.606 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:08.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:08.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:08.708 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:08.750 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:08.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:08.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:08.852 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:08.877 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:08.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:08.980 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:09.004 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:09.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:09.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:09.106 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:09.129 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:09.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:09.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:09.231 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:09.258 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:09.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:09.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:09.360 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:09.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:09 smithi043 ceph-mon[29963]: pgmap v572: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+recovery_wait, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.5 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 889 B/s rd, 0 B/s wr, 708 op/s; 8430/73749 objects degraded (11.431%); 7562/73749 objects misplaced (10.254%); 3.1 MiB/s, 302 objects/s recovering 2024-03-30T13:56:09.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:09 smithi043 ceph-mon[29963]: osdmap e271: 8 total, 7 up, 6 in 2024-03-30T13:56:09.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:09 smithi043 ceph-mon[42099]: pgmap v572: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+recovery_wait, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.5 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 889 B/s rd, 0 B/s wr, 708 op/s; 8430/73749 objects degraded (11.431%); 7562/73749 objects misplaced (10.254%); 3.1 MiB/s, 302 objects/s recovering 2024-03-30T13:56:09.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:09 smithi043 ceph-mon[42099]: osdmap e271: 8 total, 7 up, 6 in 2024-03-30T13:56:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:09.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:09.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:09.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:09 smithi113 ceph-mon[34341]: pgmap v572: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+recovery_wait, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.5 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 889 B/s rd, 0 B/s wr, 708 op/s; 8430/73749 objects degraded (11.431%); 7562/73749 objects misplaced (10.254%); 3.1 MiB/s, 302 objects/s recovering 2024-03-30T13:56:09.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:09 smithi113 ceph-mon[34341]: osdmap e271: 8 total, 7 up, 6 in 2024-03-30T13:56:09.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:56:09 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:56:09.499 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:09.523 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:09.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:09.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:09.625 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:09.653 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:09.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:09.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:09.755 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:09.782 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:09.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:09.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:09.884 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:09.913 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:09.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:09.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:10.015 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:10.056 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:10.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:10.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:10.158 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:10.200 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:10.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:10.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:10.302 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:10.325 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:10.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:10.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:10.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:10 smithi043 ceph-mon[29963]: 1.a deep-scrub starts 2024-03-30T13:56:10.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:10 smithi043 ceph-mon[29963]: 1.a deep-scrub ok 2024-03-30T13:56:10.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:10 smithi043 ceph-mon[42099]: 1.a deep-scrub starts 2024-03-30T13:56:10.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:10 smithi043 ceph-mon[42099]: 1.a deep-scrub ok 2024-03-30T13:56:10.427 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:10.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:10.468 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:10.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:10.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:10 smithi113 ceph-mon[34341]: 1.a deep-scrub starts 2024-03-30T13:56:10.469 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:10 smithi113 ceph-mon[34341]: 1.a deep-scrub ok 2024-03-30T13:56:10.569 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:10.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:10.610 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:10.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:10.711 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:10.754 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:10.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:10.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:10.856 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:10.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:10.902 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:10.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:11.003 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:11.030 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:11.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:11.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:11.133 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:11.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:11.174 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:11.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:11.275 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:11.302 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:11.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:11.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:11.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:11 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:56:11.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:11 smithi043 ceph-mon[29963]: pgmap v574: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+recovery_wait, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.5 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 703 op/s; 8430/73254 objects degraded (11.508%); 7562/73254 objects misplaced (10.323%); 3.0 MiB/s, 289 objects/s recovering 2024-03-30T13:56:11.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:11 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:56:11.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:11 smithi043 ceph-mon[42099]: pgmap v574: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+recovery_wait, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.5 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 703 op/s; 8430/73254 objects degraded (11.508%); 7562/73254 objects misplaced (10.323%); 3.0 MiB/s, 289 objects/s recovering 2024-03-30T13:56:11.403 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:11.429 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:11.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:11.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:11.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:11 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:56:11.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:11 smithi113 ceph-mon[34341]: pgmap v574: 158 pgs: 85 active+clean, 12 active+undersized, 1 active+recovery_wait, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.5 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 703 op/s; 8430/73254 objects degraded (11.508%); 7562/73254 objects misplaced (10.323%); 3.0 MiB/s, 289 objects/s recovering 2024-03-30T13:56:11.531 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:11.575 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:11.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:11.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:11.676 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:11.701 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:11.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:11.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:11.803 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:11.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:11.845 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:11.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:11.947 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:11.974 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:11.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:11.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:12.076 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:12.103 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:12.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:12.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:12.205 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:12.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:12.245 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:12.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:12.346 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:12.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:12.376 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:12.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:12.477 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:12.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:12.504 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:12.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:12.605 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:12.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:12.633 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:12.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:12.734 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:12.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:12.775 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:12.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:12.877 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:12.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:12.902 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:12.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:13.004 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:13.047 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:13.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:13.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:13.149 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:13.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:13.176 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:13.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:13.278 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:13.317 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:13.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:13.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:13.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:13 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:56:13.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:13 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:56:13.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:13 smithi043 ceph-mon[29963]: pgmap v575: 158 pgs: 88 active+clean, 12 active+undersized, 3 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.3 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s; 6814/64635 objects degraded (10.542%); 6776/64635 objects misplaced (10.483%); 1.8 MiB/s, 339 objects/s recovering 2024-03-30T13:56:13.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:13 smithi043 ceph-mon[29963]: osdmap e272: 8 total, 7 up, 6 in 2024-03-30T13:56:13.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:13 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 6814/64635 objects degraded (10.542%), 12 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:13.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:13 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:56:13.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:13 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:56:13.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:13 smithi043 ceph-mon[42099]: pgmap v575: 158 pgs: 88 active+clean, 12 active+undersized, 3 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.3 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s; 6814/64635 objects degraded (10.542%); 6776/64635 objects misplaced (10.483%); 1.8 MiB/s, 339 objects/s recovering 2024-03-30T13:56:13.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:13 smithi043 ceph-mon[42099]: osdmap e272: 8 total, 7 up, 6 in 2024-03-30T13:56:13.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:13 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 6814/64635 objects degraded (10.542%), 12 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:13.419 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:13.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:13.458 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:13.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:13.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:13 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:56:13.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:13 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:56:13.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:13 smithi113 ceph-mon[34341]: pgmap v575: 158 pgs: 88 active+clean, 12 active+undersized, 3 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.3 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s; 6814/64635 objects degraded (10.542%); 6776/64635 objects misplaced (10.483%); 1.8 MiB/s, 339 objects/s recovering 2024-03-30T13:56:13.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:13 smithi113 ceph-mon[34341]: osdmap e272: 8 total, 7 up, 6 in 2024-03-30T13:56:13.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:13 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 6814/64635 objects degraded (10.542%), 12 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:13.559 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:13.584 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:13.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:13.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:13.686 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:13.731 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:13.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:13.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:13.833 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:13.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:13.859 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:13.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:13.960 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:13.987 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:13.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:13.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:14.089 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:14.114 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:14.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:14.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:14.165 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:56:14.166 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:56:14.216 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:14.260 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:14.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:14.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:14.361 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:14.387 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:14.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:14.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:14.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:14 smithi043 ceph-mon[29963]: 6.3 scrub starts 2024-03-30T13:56:14.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:14 smithi043 ceph-mon[29963]: 6.3 scrub ok 2024-03-30T13:56:14.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:14 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:56:14.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:14 smithi043 ceph-mon[42099]: 6.3 scrub starts 2024-03-30T13:56:14.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:14 smithi043 ceph-mon[42099]: 6.3 scrub ok 2024-03-30T13:56:14.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:14 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:56:14.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:14 smithi113 ceph-mon[34341]: 6.3 scrub starts 2024-03-30T13:56:14.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:14 smithi113 ceph-mon[34341]: 6.3 scrub ok 2024-03-30T13:56:14.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:14 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:56:14.490 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:14.516 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:14.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:14.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:14.618 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:14.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:14.659 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:14.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:14.760 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:14.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:14.810 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:14.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:14.911 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:14.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:14.948 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:14.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:15.049 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:15.075 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:15.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:15.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:15.177 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:15.221 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:15.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:15.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:15.323 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:15.366 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:15.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:15.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:15.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:15 smithi043 ceph-mon[29963]: pgmap v577: 158 pgs: 88 active+clean, 12 active+undersized, 3 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.3 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 457 op/s; 6814/64635 objects degraded (10.542%); 6776/64635 objects misplaced (10.483%); 0 B/s, 184 objects/s recovering 2024-03-30T13:56:15.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:15 smithi043 ceph-mon[42099]: pgmap v577: 158 pgs: 88 active+clean, 12 active+undersized, 3 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.3 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 457 op/s; 6814/64635 objects degraded (10.542%); 6776/64635 objects misplaced (10.483%); 0 B/s, 184 objects/s recovering 2024-03-30T13:56:15.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:15 smithi113 ceph-mon[34341]: pgmap v577: 158 pgs: 88 active+clean, 12 active+undersized, 3 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.3 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 457 op/s; 6814/64635 objects degraded (10.542%); 6776/64635 objects misplaced (10.483%); 0 B/s, 184 objects/s recovering 2024-03-30T13:56:15.469 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:15.494 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:15.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:15.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:15.596 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:15.621 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:15.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:15.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:15.723 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:15.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:15.759 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:15.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:15.861 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:15.886 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:15.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:15.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:15.988 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:16.013 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:16.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:16.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:16.115 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:16.149 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:16.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:16.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:16.167 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:56:16.252 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:16.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:16.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:16.315 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:16 smithi043 ceph-mon[29963]: 6.5 scrub starts 2024-03-30T13:56:16.316 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:16 smithi043 ceph-mon[29963]: 6.5 scrub ok 2024-03-30T13:56:16.316 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:16 smithi043 ceph-mon[42099]: 6.5 scrub starts 2024-03-30T13:56:16.316 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:16 smithi043 ceph-mon[42099]: 6.5 scrub ok 2024-03-30T13:56:16.316 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:56:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:56:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:56:16.393 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:16.420 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:16.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:16.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:16.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:16 smithi113 ceph-mon[34341]: 6.5 scrub starts 2024-03-30T13:56:16.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:16 smithi113 ceph-mon[34341]: 6.5 scrub ok 2024-03-30T13:56:16.522 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:16.548 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:16.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:16.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:16.650 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:16.677 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:16.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:16.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:16.779 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:16.804 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:16.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:16.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:16.906 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:16.931 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:16.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:17.032 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:17.066 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:17.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:17.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:17.087 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:56:17.121 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-30T13:56:17.140 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T13:56:17.168 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:17.196 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:17.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:17.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:17.298 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:17.355 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:17.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:17.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:17.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:17 smithi043 ceph-mon[29963]: 6.a scrub starts 2024-03-30T13:56:17.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:17 smithi043 ceph-mon[29963]: 6.a scrub ok 2024-03-30T13:56:17.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:17 smithi043 ceph-mon[29963]: pgmap v578: 158 pgs: 88 active+clean, 12 active+undersized, 3 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.2 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 537 op/s; 6463/60804 objects degraded (10.629%); 6441/60804 objects misplaced (10.593%); 0 B/s, 196 objects/s recovering 2024-03-30T13:56:17.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:17 smithi043 ceph-mon[29963]: osdmap e273: 8 total, 7 up, 6 in 2024-03-30T13:56:17.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:17 smithi043 ceph-mon[42099]: 6.a scrub starts 2024-03-30T13:56:17.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:17 smithi043 ceph-mon[42099]: 6.a scrub ok 2024-03-30T13:56:17.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:17 smithi043 ceph-mon[42099]: pgmap v578: 158 pgs: 88 active+clean, 12 active+undersized, 3 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.2 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 537 op/s; 6463/60804 objects degraded (10.629%); 6441/60804 objects misplaced (10.593%); 0 B/s, 196 objects/s recovering 2024-03-30T13:56:17.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:17 smithi043 ceph-mon[42099]: osdmap e273: 8 total, 7 up, 6 in 2024-03-30T13:56:17.421 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:17.457 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:17.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:17 smithi113 ceph-mon[34341]: 6.a scrub starts 2024-03-30T13:56:17.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:17 smithi113 ceph-mon[34341]: 6.a scrub ok 2024-03-30T13:56:17.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:17 smithi113 ceph-mon[34341]: pgmap v578: 158 pgs: 88 active+clean, 12 active+undersized, 3 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1.2 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 537 op/s; 6463/60804 objects degraded (10.629%); 6441/60804 objects misplaced (10.593%); 0 B/s, 196 objects/s recovering 2024-03-30T13:56:17.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:17 smithi113 ceph-mon[34341]: osdmap e273: 8 total, 7 up, 6 in 2024-03-30T13:56:17.527 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:17.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:17.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:17.629 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:17.674 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:17.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:17.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:17.776 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:17.814 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:17.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:17.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:17.916 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:17.942 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:17.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:17.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:17.973 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:56:18.044 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:18.051 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:18 smithi043 ceph-mon[29963]: osdmap e274: 8 total, 7 up, 6 in 2024-03-30T13:56:18.051 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:18 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:56:18.051 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:18 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:56:18.051 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:18 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 6463/60804 objects degraded (10.629%), 12 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:18.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:18.097 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:18.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:18.106 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:18 smithi113 ceph-mon[34341]: osdmap e274: 8 total, 7 up, 6 in 2024-03-30T13:56:18.107 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:18 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:56:18.107 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:18 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:56:18.107 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:18 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 6463/60804 objects degraded (10.629%), 12 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:18.197 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:18.242 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:18.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:18.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:18.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:18.384 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:18.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:18.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:18.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:18 smithi043 ceph-mon[42099]: osdmap e274: 8 total, 7 up, 6 in 2024-03-30T13:56:18.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:18 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:56:18.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:18 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:56:18.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:18 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 6463/60804 objects degraded (10.629%), 12 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:18.486 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:18.515 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:18.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:18.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:18.617 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:18.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:18.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:18.739 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:56:18.756 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:18.782 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:18.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:18.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:18.884 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:18.916 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:18.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:18.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:19.018 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:19.044 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:19.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:19.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:19.146 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:19.184 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:19 smithi113 ceph-mon[34341]: 3.d deep-scrub starts 2024-03-30T13:56:19.185 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:19 smithi113 ceph-mon[34341]: 3.d deep-scrub ok 2024-03-30T13:56:19.185 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:19 smithi113 ceph-mon[34341]: pgmap v581: 158 pgs: 88 active+clean, 12 active+undersized, 3 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1018 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 845 op/s; 5341/49419 objects degraded (10.808%); 5640/49419 objects misplaced (11.413%); 0 B/s, 116 objects/s recovering 2024-03-30T13:56:19.185 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:19 smithi113 ceph-mon[34341]: osdmap e275: 8 total, 7 up, 6 in 2024-03-30T13:56:19.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:56:19 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:56:19.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:19.190 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:19.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:19.291 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:19.308 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:19 smithi043 ceph-mon[29963]: 3.d deep-scrub starts 2024-03-30T13:56:19.308 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:19 smithi043 ceph-mon[29963]: 3.d deep-scrub ok 2024-03-30T13:56:19.308 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:19 smithi043 ceph-mon[29963]: pgmap v581: 158 pgs: 88 active+clean, 12 active+undersized, 3 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1018 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 845 op/s; 5341/49419 objects degraded (10.808%); 5640/49419 objects misplaced (11.413%); 0 B/s, 116 objects/s recovering 2024-03-30T13:56:19.308 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:19 smithi043 ceph-mon[29963]: osdmap e275: 8 total, 7 up, 6 in 2024-03-30T13:56:19.308 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:19 smithi043 ceph-mon[42099]: 3.d deep-scrub starts 2024-03-30T13:56:19.309 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:19 smithi043 ceph-mon[42099]: 3.d deep-scrub ok 2024-03-30T13:56:19.309 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:19 smithi043 ceph-mon[42099]: pgmap v581: 158 pgs: 88 active+clean, 12 active+undersized, 3 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1018 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 845 op/s; 5341/49419 objects degraded (10.808%); 5640/49419 objects misplaced (11.413%); 0 B/s, 116 objects/s recovering 2024-03-30T13:56:19.309 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:19 smithi043 ceph-mon[42099]: osdmap e275: 8 total, 7 up, 6 in 2024-03-30T13:56:19.320 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:19.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:19.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:19.422 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:19.448 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:19.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:19.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:19.479 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:56:19.479 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:56:19.550 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:19.576 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:19.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:19.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:19.678 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:19.705 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:19.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:19.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:19.808 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:19.834 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:19.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:19.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:19.936 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:19.963 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:19.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:19.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:20.065 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:20.095 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:20.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:20.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:20.199 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:20.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:20.246 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:20.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:20.282 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-03-30T13:56:20.347 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:20.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:20.381 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:20.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:20.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:20 smithi043 ceph-mon[29963]: 5.9 scrub starts 2024-03-30T13:56:20.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:20 smithi043 ceph-mon[29963]: 5.9 scrub ok 2024-03-30T13:56:20.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:20 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:56:20.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:20 smithi043 ceph-mon[29963]: pgmap v583: 158 pgs: 88 active+clean, 12 active+undersized, 3 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1000 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 891 op/s; 5341/48594 objects degraded (10.991%); 5640/48594 objects misplaced (11.606%); 0 B/s, 116 objects/s recovering 2024-03-30T13:56:20.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:20 smithi043 ceph-mon[42099]: 5.9 scrub starts 2024-03-30T13:56:20.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:20 smithi043 ceph-mon[42099]: 5.9 scrub ok 2024-03-30T13:56:20.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:20 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:56:20.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:20 smithi043 ceph-mon[42099]: pgmap v583: 158 pgs: 88 active+clean, 12 active+undersized, 3 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1000 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 891 op/s; 5341/48594 objects degraded (10.991%); 5640/48594 objects misplaced (11.606%); 0 B/s, 116 objects/s recovering 2024-03-30T13:56:20.409 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:20 smithi113 ceph-mon[34341]: 5.9 scrub starts 2024-03-30T13:56:20.410 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:20 smithi113 ceph-mon[34341]: 5.9 scrub ok 2024-03-30T13:56:20.410 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:20 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:56:20.410 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:20 smithi113 ceph-mon[34341]: pgmap v583: 158 pgs: 88 active+clean, 12 active+undersized, 3 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 1000 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 891 op/s; 5341/48594 objects degraded (10.991%); 5640/48594 objects misplaced (11.606%); 0 B/s, 116 objects/s recovering 2024-03-30T13:56:20.482 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:20.515 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:20.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:20.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:20.613 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:20.617 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:20.651 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:20.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:20.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:20.753 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:20.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:20.784 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:20.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:20.885 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:20.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:20.952 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:20.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:21.053 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:21.103 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:21.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:21.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:21.205 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:21.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:21.236 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:21.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:21.337 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:21.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:21.368 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:21.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:21.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:21 smithi043 ceph-mon[29963]: 4.8 scrub starts 2024-03-30T13:56:21.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:21 smithi043 ceph-mon[29963]: 4.8 scrub ok 2024-03-30T13:56:21.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:21 smithi043 ceph-mon[42099]: 4.8 scrub starts 2024-03-30T13:56:21.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:21 smithi043 ceph-mon[42099]: 4.8 scrub ok 2024-03-30T13:56:21.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:21 smithi113 ceph-mon[34341]: 4.8 scrub starts 2024-03-30T13:56:21.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:21 smithi113 ceph-mon[34341]: 4.8 scrub ok 2024-03-30T13:56:21.469 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:21.507 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:21.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:21.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:21.609 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:21.634 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:21.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:21.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:21.735 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:21.760 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:21.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:21.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:21.861 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:21.903 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:21.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:21.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:22.005 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:22.046 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:22.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:22.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:22.148 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:22.193 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:22.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:22.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:22.294 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:22.322 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:22.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:22.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:22.351 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:22 smithi113 ceph-mon[34341]: pgmap v584: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 715 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.44k op/s; 4057/34905 objects degraded (11.623%); 4481/34905 objects misplaced (12.838%); 0 B/s, 165 objects/s recovering 2024-03-30T13:56:22.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:22 smithi043 ceph-mon[29963]: pgmap v584: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 715 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.44k op/s; 4057/34905 objects degraded (11.623%); 4481/34905 objects misplaced (12.838%); 0 B/s, 165 objects/s recovering 2024-03-30T13:56:22.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:22 smithi043 ceph-mon[42099]: pgmap v584: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 715 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.44k op/s; 4057/34905 objects degraded (11.623%); 4481/34905 objects misplaced (12.838%); 0 B/s, 165 objects/s recovering 2024-03-30T13:56:22.424 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:22.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:22.451 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:22.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:22.552 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:22.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:22.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:22.699 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:22.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:22.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:22.841 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:22.871 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:22.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:22.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:22.881 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:56:22.881 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:56:22.973 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:23.002 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:23.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:23.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:23.104 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:23.131 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:23.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:23.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:23.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:23 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 4057/34905 objects degraded (11.623%), 11 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:23.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:23 smithi043 ceph-mon[29963]: osdmap e276: 8 total, 7 up, 6 in 2024-03-30T13:56:23.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:23 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 4057/34905 objects degraded (11.623%), 11 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:23.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:23 smithi043 ceph-mon[42099]: osdmap e276: 8 total, 7 up, 6 in 2024-03-30T13:56:23.233 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:23.274 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:23.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:23.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:23.376 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:23.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:23.407 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:23.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:23.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:23 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 4057/34905 objects degraded (11.623%), 11 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:23.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:23 smithi113 ceph-mon[34341]: osdmap e276: 8 total, 7 up, 6 in 2024-03-30T13:56:23.508 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:23.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:23.546 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:23.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:23.598 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-03-30T13:56:23.648 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:23.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:23.676 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:23.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:23.778 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:23.819 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:23.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:23.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:23.883 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:23.922 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:23.945 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:23.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:23.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:24.046 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:24.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:24.074 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:24.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:24.175 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:24.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:24.215 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:24.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:24.272 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:24 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:56:24.272 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:24 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:56:24.272 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:24 smithi113 ceph-mon[34341]: pgmap v586: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 715 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 780 B/s rd, 0 B/s wr, 737 op/s; 4057/34905 objects degraded (11.623%); 4481/34905 objects misplaced (12.838%); 0 B/s, 97 objects/s recovering 2024-03-30T13:56:24.316 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:24.341 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:24.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:24.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:24.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:24 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:56:24.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:24 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:56:24.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:24 smithi043 ceph-mon[29963]: pgmap v586: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 715 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 780 B/s rd, 0 B/s wr, 737 op/s; 4057/34905 objects degraded (11.623%); 4481/34905 objects misplaced (12.838%); 0 B/s, 97 objects/s recovering 2024-03-30T13:56:24.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:24 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:56:24.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:24 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:56:24.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:24 smithi043 ceph-mon[42099]: pgmap v586: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 715 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 780 B/s rd, 0 B/s wr, 737 op/s; 4057/34905 objects degraded (11.623%); 4481/34905 objects misplaced (12.838%); 0 B/s, 97 objects/s recovering 2024-03-30T13:56:24.444 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:24.485 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:24.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:24.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:24.587 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:24.628 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:24.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:24.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:24.730 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:24.772 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:24.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:24.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:24.874 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:24.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:24.915 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:24.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:25.016 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:25.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:25.046 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:25.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:25.121 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-03-30T13:56:25.148 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:25.193 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:25.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:25.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:25.257 INFO:teuthology.orchestra.run.smithi043.stderr:set pool 9 pgp_num to 16 2024-03-30T13:56:25.296 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:25.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:25.335 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:25.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:25.341 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:25 smithi113 ceph-mon[34341]: osdmap e277: 8 total, 7 up, 6 in 2024-03-30T13:56:25.341 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:25 smithi113 ceph-mon[34341]: 4.a scrub starts 2024-03-30T13:56:25.342 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:25 smithi113 ceph-mon[34341]: 4.a scrub ok 2024-03-30T13:56:25.342 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:25 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1208483983' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:56:25.342 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:25 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:56:25.342 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:25 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3157583540' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-30T13:56:25.342 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:25 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-30T13:56:25.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:25 smithi043 ceph-mon[29963]: osdmap e277: 8 total, 7 up, 6 in 2024-03-30T13:56:25.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:25 smithi043 ceph-mon[29963]: 4.a scrub starts 2024-03-30T13:56:25.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:25 smithi043 ceph-mon[29963]: 4.a scrub ok 2024-03-30T13:56:25.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:25 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1208483983' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:56:25.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:25 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:56:25.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:25 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3157583540' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-30T13:56:25.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:25 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-30T13:56:25.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:25 smithi043 ceph-mon[42099]: osdmap e277: 8 total, 7 up, 6 in 2024-03-30T13:56:25.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:25 smithi043 ceph-mon[42099]: 4.a scrub starts 2024-03-30T13:56:25.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:25 smithi043 ceph-mon[42099]: 4.a scrub ok 2024-03-30T13:56:25.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:25 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1208483983' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:56:25.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:25 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:56:25.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:25 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3157583540' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-30T13:56:25.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:25 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-30T13:56:25.436 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:25.476 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:25.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:25.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:25.578 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:25.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:25.608 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:25.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:25.635 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:56:25.635 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:56:25.709 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:25.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:25.733 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:25.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:25.834 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:25.860 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:25.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:25.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:25.962 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:25.984 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:25.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:25.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:26.086 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:26.113 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:26.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:26.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:26.215 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:26.241 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:26.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:26.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:26.343 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:26.368 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:26.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:26.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:26.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:26 smithi043 ceph-mon[29963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:56:26.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:26 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:56:26.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:26 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-03-30T13:56:26.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:26 smithi043 ceph-mon[29963]: osdmap e278: 8 total, 7 up, 6 in 2024-03-30T13:56:26.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:26 smithi043 ceph-mon[29963]: pgmap v589: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 634 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 978 op/s; 4057/30981 objects degraded (13.095%); 4157/30981 objects misplaced (13.418%); 0 B/s, 107 objects/s recovering 2024-03-30T13:56:26.395 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:56:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:56:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:56:26.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:26 smithi043 ceph-mon[42099]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:56:26.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:26 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:56:26.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:26 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-03-30T13:56:26.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:26 smithi043 ceph-mon[42099]: osdmap e278: 8 total, 7 up, 6 in 2024-03-30T13:56:26.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:26 smithi043 ceph-mon[42099]: pgmap v589: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 634 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 978 op/s; 4057/30981 objects degraded (13.095%); 4157/30981 objects misplaced (13.418%); 0 B/s, 107 objects/s recovering 2024-03-30T13:56:26.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:26 smithi113 ceph-mon[34341]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:56:26.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:26 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:56:26.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:26 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-03-30T13:56:26.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:26 smithi113 ceph-mon[34341]: osdmap e278: 8 total, 7 up, 6 in 2024-03-30T13:56:26.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:26 smithi113 ceph-mon[34341]: pgmap v589: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 634 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 978 op/s; 4057/30981 objects degraded (13.095%); 4157/30981 objects misplaced (13.418%); 0 B/s, 107 objects/s recovering 2024-03-30T13:56:26.470 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:26.496 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:26.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:26.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:26.598 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:26.623 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:26.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:26.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:26.712 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-30T13:56:26.725 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:26.782 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:26.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:26.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:26.885 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:26.914 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:26.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:26.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:26.993 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:27.016 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:27.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:27.060 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:27.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:27.161 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:27.187 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:27.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:27.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:27.289 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:27.316 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:27.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:27.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:27.417 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:27.443 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:27.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:27.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:27.545 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:27.587 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:27.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:27.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:27.689 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:27.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:27.729 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:27.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:27.827 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:27 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 4057/30981 objects degraded (13.095%), 11 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:27.830 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:27.868 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:27.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:27.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:27.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:27 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 4057/30981 objects degraded (13.095%), 11 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:27.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:27 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 4057/30981 objects degraded (13.095%), 11 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:27.970 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:27.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:27.996 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:27.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:28.097 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:28.126 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:28.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:28.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:28.228 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:28.256 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:28.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:28.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:28.358 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:28.382 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:28.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:28.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:28.483 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:28.510 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:28.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:28.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:28.612 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:28.656 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:28.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:28.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:28.758 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:28.782 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:28.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:28.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:28.831 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:28 smithi113 ceph-mon[34341]: pgmap v590: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 396 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 851 op/s; 2870/19587 objects degraded (14.653%); 3384/19587 objects misplaced (17.277%); 0 B/s, 62 objects/s recovering 2024-03-30T13:56:28.884 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:28.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:28 smithi043 ceph-mon[42099]: pgmap v590: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 396 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 851 op/s; 2870/19587 objects degraded (14.653%); 3384/19587 objects misplaced (17.277%); 0 B/s, 62 objects/s recovering 2024-03-30T13:56:28.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:28 smithi043 ceph-mon[29963]: pgmap v590: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 396 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 851 op/s; 2870/19587 objects degraded (14.653%); 3384/19587 objects misplaced (17.277%); 0 B/s, 62 objects/s recovering 2024-03-30T13:56:28.921 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:28.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:28.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:29.022 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:29.052 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:29.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:29.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:29.123 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:56:29.123 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:56:29.124 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:56:29 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:56:29.154 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:29.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:29.179 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:29.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:29.280 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:29.306 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:29.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:29.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:29.408 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:29.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:29.435 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:29.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:29.538 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:29.573 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:29.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:29.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:29.675 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:29.702 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:29.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:29.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:29.804 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:29.832 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-03-30T13:56:29.849 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:29.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:29.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:29.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:29 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:56:29.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:29 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:56:29.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:29 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:56:29.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:29 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:56:29.951 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:29.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:29.975 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:29.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:30.076 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:30.109 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:30.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:30.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:30.120 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:30.211 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:30.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:29 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:56:30.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:29 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:56:30.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:30.252 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:30.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:30.353 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:30.395 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:30.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:30.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:30.498 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:30.524 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:30.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:30.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:30.626 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:30.654 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:30.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:30.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:30.756 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:30.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:30.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:30.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:30.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:30.930 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:30.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:31.031 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:31.059 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:31.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:31.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:31.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:30 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:56:31.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:30 smithi043 ceph-mon[42099]: pgmap v591: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 378 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 725 B/s rd, 0 B/s wr, 765 op/s; 2870/18699 objects degraded (15.348%); 3384/18699 objects misplaced (18.097%); 0 B/s, 53 objects/s recovering 2024-03-30T13:56:31.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:30 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:56:31.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:30 smithi043 ceph-mon[29963]: pgmap v591: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 378 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 725 B/s rd, 0 B/s wr, 765 op/s; 2870/18699 objects degraded (15.348%); 3384/18699 objects misplaced (18.097%); 0 B/s, 53 objects/s recovering 2024-03-30T13:56:31.161 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:31.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:31.187 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:31.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:31.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:30 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:56:31.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:30 smithi113 ceph-mon[34341]: pgmap v591: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 378 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 725 B/s rd, 0 B/s wr, 765 op/s; 2870/18699 objects degraded (15.348%); 3384/18699 objects misplaced (18.097%); 0 B/s, 53 objects/s recovering 2024-03-30T13:56:31.288 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:31.328 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:31.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:31.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:31.430 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:31.457 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:31.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:31.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:31.559 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:31.600 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:31.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:31.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:31.698 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Removed 86813 objects 2024-03-30T13:56:31.699 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Clean up completed and total clean up time :128.974 2024-03-30T13:56:31.703 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:31.729 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:31.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:31.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:31.831 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:31.856 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:31.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:31.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:31.958 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:31.983 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:31.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:31.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:32.084 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:32.108 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:32.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:32.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:32.209 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:32.233 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:32.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:32.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:32.335 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:32.358 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:32.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:32.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:32.460 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:32.464 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:56:32.464 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:56:32.483 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:32.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:32.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:32.584 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:32.608 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:32.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:32.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:32.710 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:32.733 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:32.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:32.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:32.835 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:32.860 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:32.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:32.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:32.961 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:32.985 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:32.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:32.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:33.056 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:32 smithi043 ceph-mon[42099]: pgmap v592: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 73 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 1669/4053 objects degraded (41.179%); 2225/4053 objects misplaced (54.898%); 0 B/s, 97 objects/s recovering 2024-03-30T13:56:33.056 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:32 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 1669/4053 objects degraded (41.179%), 11 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:33.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:32 smithi043 ceph-mon[29963]: pgmap v592: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 73 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 1669/4053 objects degraded (41.179%); 2225/4053 objects misplaced (54.898%); 0 B/s, 97 objects/s recovering 2024-03-30T13:56:33.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:32 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 1669/4053 objects degraded (41.179%), 11 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:33.087 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:33.112 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:33.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:33.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:33.135 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:32 smithi113 ceph-mon[34341]: pgmap v592: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 73 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 1669/4053 objects degraded (41.179%); 2225/4053 objects misplaced (54.898%); 0 B/s, 97 objects/s recovering 2024-03-30T13:56:33.135 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:32 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 1669/4053 objects degraded (41.179%), 11 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:33.161 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-03-30T13:56:33.214 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:33.267 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:33.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:33.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:33.368 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:33.393 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:33.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:33.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:33.396 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:56:33.422 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:56:33.442 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:33.444 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:56:33.471 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:56:33.495 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:33.497 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:56:33.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:33.519 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:33.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:33.522 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:56:33.554 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:56:33.554 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:56:33.554 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:56:33.554 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:56:33.554 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:56:33.554 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:56:33.554 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:56:33.554 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:56:33.555 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:56:33.620 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:33.643 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:33.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:33.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:33.745 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:33.774 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:33.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:33.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:33.875 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:33.899 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:33.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:33.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:34.001 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:34.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:34.025 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:34.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:34.126 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:34.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:33 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1895888385' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:56:34.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:33 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1895888385' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:56:34.150 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:34.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:34.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:34.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:33 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1895888385' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:56:34.252 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:34.278 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:34.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:34.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:34.380 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:34.406 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:34.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:34.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:34.508 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:34.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:34.532 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:34.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:34.634 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:34.659 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:34.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:34.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:34.761 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:34.788 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:34.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:34.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:34.890 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:34.916 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:34.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:34.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:35.018 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:35.029 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:56:35.029 INFO:teuthology.orchestra.run.smithi113.stdout: "size": 20, 2024-03-30T13:56:35.029 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 600, 2024-03-30T13:56:35.030 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [ 2024-03-30T13:56:35.030 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.030 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:28080 8.8 8:113ed673:::benchmark_data_smithi043_119832_object28079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-30T13:56:35.030 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:49:25.863547+0000", 2024-03-30T13:56:35.030 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 429.16791647500003, 2024-03-30T13:56:35.030 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.10021473, 2024-03-30T13:56:35.030 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:35.030 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:35.030 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:35.030 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:35.030 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:35.030 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:35.031 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 28080 2024-03-30T13:56:35.031 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.031 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:35.031 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.031 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:35.031 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863547+0000", 2024-03-30T13:56:35.031 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.031 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.031 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.031 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:35.031 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863547+0000", 2024-03-30T13:56:35.031 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.031 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.031 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.031 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:35.032 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863548+0000", 2024-03-30T13:56:35.032 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.8800000000000001e-07 2024-03-30T13:56:35.032 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.032 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.032 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:35.032 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863574+0000", 2024-03-30T13:56:35.032 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.5948999999999998e-05 2024-03-30T13:56:35.032 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.032 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.032 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:35.032 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863585+0000", 2024-03-30T13:56:35.032 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0411000000000001e-05 2024-03-30T13:56:35.032 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.032 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.033 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:35.033 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863594+0000", 2024-03-30T13:56:35.033 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.0720000000000006e-06 2024-03-30T13:56:35.033 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.033 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.033 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:35.033 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863644+0000", 2024-03-30T13:56:35.033 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0078999999999999e-05 2024-03-30T13:56:35.033 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.033 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.033 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:35.033 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863905+0000", 2024-03-30T13:56:35.033 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00026154999999999999 2024-03-30T13:56:35.034 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.034 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.034 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:56:35.034 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.864048+0000", 2024-03-30T13:56:35.034 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000142731 2024-03-30T13:56:35.034 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.034 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.034 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:35.034 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.866137+0000", 2024-03-30T13:56:35.034 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0020888 2024-03-30T13:56:35.034 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.034 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.034 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.034 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.866834+0000", 2024-03-30T13:56:35.035 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00069740999999999998 2024-03-30T13:56:35.035 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.035 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.035 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.035 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.963689+0000", 2024-03-30T13:56:35.035 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.096855105999999996 2024-03-30T13:56:35.035 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.035 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.035 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:35.035 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.963729+0000", 2024-03-30T13:56:35.035 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.9792000000000001e-05 2024-03-30T13:56:35.035 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.035 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.036 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:35.036 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.963762+0000", 2024-03-30T13:56:35.036 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2941999999999997e-05 2024-03-30T13:56:35.036 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.036 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.036 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.036 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.036 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.036 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:28082 8.8 8:1b8ed0ff:::benchmark_data_smithi043_119832_object28081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-30T13:56:35.036 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:49:25.863684+0000", 2024-03-30T13:56:35.036 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 429.16777954499997, 2024-03-30T13:56:35.036 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.10475061099999999, 2024-03-30T13:56:35.036 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:35.036 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:35.037 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:35.037 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:35.037 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:35.037 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:35.037 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 28082 2024-03-30T13:56:35.037 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.037 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:35.037 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.037 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:35.037 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863684+0000", 2024-03-30T13:56:35.037 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.037 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.037 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.038 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:35.038 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863684+0000", 2024-03-30T13:56:35.038 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.038 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.038 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.038 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:35.038 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863685+0000", 2024-03-30T13:56:35.038 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1500000000000004e-07 2024-03-30T13:56:35.038 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.038 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.038 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:35.038 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863720+0000", 2024-03-30T13:56:35.038 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.5271999999999997e-05 2024-03-30T13:56:35.039 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.039 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.039 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:35.039 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863728+0000", 2024-03-30T13:56:35.039 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.2269999999999993e-06 2024-03-30T13:56:35.039 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.039 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.039 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:35.039 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.863735+0000", 2024-03-30T13:56:35.039 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6749999999999996e-06 2024-03-30T13:56:35.039 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.039 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.039 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:35.040 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.865132+0000", 2024-03-30T13:56:35.040 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0013967999999999999 2024-03-30T13:56:35.040 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.040 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.040 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:35.040 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.865264+0000", 2024-03-30T13:56:35.040 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013255799999999999 2024-03-30T13:56:35.040 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.040 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.040 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:56:35.040 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.865398+0000", 2024-03-30T13:56:35.040 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000133449 2024-03-30T13:56:35.040 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.040 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.041 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:35.041 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.866258+0000", 2024-03-30T13:56:35.041 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00086043299999999997 2024-03-30T13:56:35.041 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.041 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.041 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.041 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.866918+0000", 2024-03-30T13:56:35.041 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000660055 2024-03-30T13:56:35.041 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.041 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.041 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.041 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.968343+0000", 2024-03-30T13:56:35.041 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.101424903 2024-03-30T13:56:35.042 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.042 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.042 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:35.042 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.968402+0000", 2024-03-30T13:56:35.042 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8968000000000002e-05 2024-03-30T13:56:35.042 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.042 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.042 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:35.042 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:25.968435+0000", 2024-03-30T13:56:35.042 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2656e-05 2024-03-30T13:56:35.042 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.042 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.042 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.042 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.043 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.043 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:31312 8.4 8:22ccd403:::benchmark_data_smithi043_119832_object31311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-30T13:56:35.043 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:49:27.619219+0000", 2024-03-30T13:56:35.043 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 427.41224512600002, 2024-03-30T13:56:35.043 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.16068914500000001, 2024-03-30T13:56:35.043 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:35.043 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:35.043 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:35.044 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:35.044 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:35.044 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:35.044 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 31312 2024-03-30T13:56:35.044 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.044 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:35.045 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.045 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:35.045 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.619219+0000", 2024-03-30T13:56:35.045 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.045 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.045 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.045 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:35.045 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.619219+0000", 2024-03-30T13:56:35.045 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.045 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.045 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.045 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:35.045 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.619219+0000", 2024-03-30T13:56:35.045 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8500000000000001e-07 2024-03-30T13:56:35.046 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.046 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.046 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:35.046 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.619288+0000", 2024-03-30T13:56:35.046 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8913999999999998e-05 2024-03-30T13:56:35.046 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.046 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.046 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:35.046 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.619298+0000", 2024-03-30T13:56:35.046 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.3409999999999992e-06 2024-03-30T13:56:35.046 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.046 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.046 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:35.047 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.619303+0000", 2024-03-30T13:56:35.047 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3990000000000003e-06 2024-03-30T13:56:35.047 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.047 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.047 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:35.047 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.619344+0000", 2024-03-30T13:56:35.047 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.0772000000000002e-05 2024-03-30T13:56:35.047 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.047 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.047 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:35.047 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.619423+0000", 2024-03-30T13:56:35.047 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.9049999999999997e-05 2024-03-30T13:56:35.047 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.047 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.048 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 2,4", 2024-03-30T13:56:35.048 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.619545+0000", 2024-03-30T13:56:35.048 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012253500000000001 2024-03-30T13:56:35.048 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.048 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.048 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:35.048 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.621662+0000", 2024-03-30T13:56:35.048 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0021171610000000002 2024-03-30T13:56:35.048 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.048 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.048 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.048 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.622183+0000", 2024-03-30T13:56:35.048 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00052070800000000004 2024-03-30T13:56:35.049 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.049 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.049 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.049 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.779799+0000", 2024-03-30T13:56:35.049 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15761560099999999 2024-03-30T13:56:35.049 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.049 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.049 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:35.049 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.779861+0000", 2024-03-30T13:56:35.049 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.2067000000000001e-05 2024-03-30T13:56:35.049 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.049 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.049 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:35.049 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.779908+0000", 2024-03-30T13:56:35.050 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6912000000000001e-05 2024-03-30T13:56:35.050 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.050 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.050 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.050 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.050 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.050 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:31315 8.4 8:2d5d1e8e:::benchmark_data_smithi043_119832_object31314:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-30T13:56:35.050 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:49:27.620358+0000", 2024-03-30T13:56:35.050 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 427.41110531700002, 2024-03-30T13:56:35.050 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15976934100000001, 2024-03-30T13:56:35.050 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:35.050 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:35.050 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:35.051 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:35.051 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:35.051 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:35.051 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 31315 2024-03-30T13:56:35.051 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.051 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:35.051 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.051 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:35.051 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.620358+0000", 2024-03-30T13:56:35.051 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.051 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.051 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.051 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:35.052 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.620358+0000", 2024-03-30T13:56:35.052 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.052 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.052 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.052 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:35.052 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.620359+0000", 2024-03-30T13:56:35.052 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.7800000000000001e-07 2024-03-30T13:56:35.052 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.052 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.052 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:35.052 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.620419+0000", 2024-03-30T13:56:35.052 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9920000000000002e-05 2024-03-30T13:56:35.052 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.052 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.052 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:35.053 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.620429+0000", 2024-03-30T13:56:35.053 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.8099999999999992e-06 2024-03-30T13:56:35.053 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.053 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.053 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:35.053 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.620436+0000", 2024-03-30T13:56:35.053 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.3819999999999998e-06 2024-03-30T13:56:35.053 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.053 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.053 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:35.053 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.620510+0000", 2024-03-30T13:56:35.053 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.3495e-05 2024-03-30T13:56:35.053 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.053 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.054 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:35.054 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.620644+0000", 2024-03-30T13:56:35.054 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013414199999999999 2024-03-30T13:56:35.054 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.054 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.054 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 2,4", 2024-03-30T13:56:35.054 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.620778+0000", 2024-03-30T13:56:35.054 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013415700000000001 2024-03-30T13:56:35.054 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.054 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.054 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:35.054 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.621756+0000", 2024-03-30T13:56:35.054 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00097763800000000007 2024-03-30T13:56:35.054 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.054 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.055 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.055 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.622314+0000", 2024-03-30T13:56:35.055 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00055802399999999995 2024-03-30T13:56:35.055 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.055 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.055 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.055 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.779972+0000", 2024-03-30T13:56:35.056 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15765843800000001 2024-03-30T13:56:35.056 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.056 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.056 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:35.056 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.780086+0000", 2024-03-30T13:56:35.056 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011418599999999999 2024-03-30T13:56:35.056 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.057 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.057 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:35.057 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.780128+0000", 2024-03-30T13:56:35.057 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1471000000000003e-05 2024-03-30T13:56:35.057 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.057 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.057 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.057 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.057 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.057 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:31317 8.4 8:244a887a:::benchmark_data_smithi043_119832_object31316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-30T13:56:35.057 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:49:27.624141+0000", 2024-03-30T13:56:35.057 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 427.40732230499998, 2024-03-30T13:56:35.057 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15630417999999999, 2024-03-30T13:56:35.057 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:35.058 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:35.058 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:35.058 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:35.058 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:35.058 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:35.058 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 31317 2024-03-30T13:56:35.058 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.058 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:35.058 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.058 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:35.058 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.624141+0000", 2024-03-30T13:56:35.058 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.058 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.058 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.059 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:35.059 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.624141+0000", 2024-03-30T13:56:35.059 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.059 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.059 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.059 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:35.059 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.624142+0000", 2024-03-30T13:56:35.059 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.4700000000000001e-07 2024-03-30T13:56:35.059 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.059 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.059 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:35.059 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.624205+0000", 2024-03-30T13:56:35.059 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.2424999999999999e-05 2024-03-30T13:56:35.060 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.060 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.060 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:35.060 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.624215+0000", 2024-03-30T13:56:35.060 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.9629999999999993e-06 2024-03-30T13:56:35.060 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.060 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.060 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:35.060 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.624231+0000", 2024-03-30T13:56:35.060 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.6421000000000001e-05 2024-03-30T13:56:35.060 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.060 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.060 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:35.060 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.624274+0000", 2024-03-30T13:56:35.060 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.2549000000000002e-05 2024-03-30T13:56:35.061 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.061 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.061 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:35.061 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.624360+0000", 2024-03-30T13:56:35.061 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6681000000000006e-05 2024-03-30T13:56:35.061 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.061 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.061 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 2,4", 2024-03-30T13:56:35.061 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.624446+0000", 2024-03-30T13:56:35.061 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6037000000000003e-05 2024-03-30T13:56:35.061 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.061 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.061 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:35.061 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.625196+0000", 2024-03-30T13:56:35.062 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00074996299999999998 2024-03-30T13:56:35.062 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.062 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.062 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.062 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.625529+0000", 2024-03-30T13:56:35.062 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00033245200000000001 2024-03-30T13:56:35.062 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.062 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.062 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.062 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.780307+0000", 2024-03-30T13:56:35.062 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.154778215 2024-03-30T13:56:35.062 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.062 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.062 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:35.063 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.780407+0000", 2024-03-30T13:56:35.063 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000100058 2024-03-30T13:56:35.063 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.063 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.063 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:35.063 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.780446+0000", 2024-03-30T13:56:35.063 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.8668999999999999e-05 2024-03-30T13:56:35.063 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.063 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.063 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.063 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.063 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.063 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:249827 8.8 8:1adcec11:::benchmark_data_smithi043_119832_object119473:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:56:35.063 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:19.321904+0000", 2024-03-30T13:56:35.064 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 255.70955928500001, 2024-03-30T13:56:35.064 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.127091228, 2024-03-30T13:56:35.064 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:35.064 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:35.064 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:35.064 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:35.064 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:35.064 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:35.064 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 249827 2024-03-30T13:56:35.064 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.064 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:35.064 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.064 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:35.064 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.321904+0000", 2024-03-30T13:56:35.065 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.065 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.065 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.065 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:35.065 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.321904+0000", 2024-03-30T13:56:35.065 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.065 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.065 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.065 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:35.065 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.321905+0000", 2024-03-30T13:56:35.065 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0109999999999999e-06 2024-03-30T13:56:35.065 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.065 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.065 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:35.066 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.321911+0000", 2024-03-30T13:56:35.066 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8810000000000001e-06 2024-03-30T13:56:35.066 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.066 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.066 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:35.066 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.321922+0000", 2024-03-30T13:56:35.066 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0659000000000001e-05 2024-03-30T13:56:35.066 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.066 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.066 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:35.066 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.321929+0000", 2024-03-30T13:56:35.066 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.08e-06 2024-03-30T13:56:35.066 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.066 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.067 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:35.067 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.322098+0000", 2024-03-30T13:56:35.067 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00016901000000000001 2024-03-30T13:56:35.067 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.067 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.067 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:35.067 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.322244+0000", 2024-03-30T13:56:35.067 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00014547799999999999 2024-03-30T13:56:35.067 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.067 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.067 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:56:35.067 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.322367+0000", 2024-03-30T13:56:35.067 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012317299999999999 2024-03-30T13:56:35.067 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.068 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.068 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.068 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.328056+0000", 2024-03-30T13:56:35.068 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0056895740000000002 2024-03-30T13:56:35.068 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.068 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.068 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:35.068 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.328217+0000", 2024-03-30T13:56:35.068 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000161101 2024-03-30T13:56:35.068 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.068 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.068 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.068 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.448923+0000", 2024-03-30T13:56:35.068 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.12070563199999999 2024-03-30T13:56:35.068 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.069 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.069 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:35.069 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.448958+0000", 2024-03-30T13:56:35.069 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.5289e-05 2024-03-30T13:56:35.069 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.069 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.069 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:35.069 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.448996+0000", 2024-03-30T13:56:35.069 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.7339999999999998e-05 2024-03-30T13:56:35.069 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.069 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.069 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.069 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.069 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.070 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:249832 8.8 8:176df1f4:::benchmark_data_smithi043_119832_object119478:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:56:35.070 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:19.322778+0000", 2024-03-30T13:56:35.070 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 255.70868596, 2024-03-30T13:56:35.070 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.12633915500000001, 2024-03-30T13:56:35.070 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:35.070 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:35.070 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:35.070 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:35.070 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:35.070 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:35.070 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 249832 2024-03-30T13:56:35.070 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.070 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:35.070 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.071 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:35.071 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.322778+0000", 2024-03-30T13:56:35.071 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.071 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.071 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.071 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:35.071 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.322778+0000", 2024-03-30T13:56:35.071 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.071 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.071 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.071 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:35.071 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.322779+0000", 2024-03-30T13:56:35.071 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.1200000000000002e-07 2024-03-30T13:56:35.071 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.072 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.072 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:35.072 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.322783+0000", 2024-03-30T13:56:35.072 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6020000000000002e-06 2024-03-30T13:56:35.072 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.072 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.072 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:35.072 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.322795+0000", 2024-03-30T13:56:35.072 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1806e-05 2024-03-30T13:56:35.072 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.072 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.072 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:35.072 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.322802+0000", 2024-03-30T13:56:35.072 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.4240000000000004e-06 2024-03-30T13:56:35.073 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.073 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.073 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:35.073 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.325035+0000", 2024-03-30T13:56:35.073 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0022325980000000001 2024-03-30T13:56:35.073 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.073 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.073 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:35.073 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.325133+0000", 2024-03-30T13:56:35.073 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.8256000000000003e-05 2024-03-30T13:56:35.073 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.073 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.073 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:56:35.074 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.325242+0000", 2024-03-30T13:56:35.074 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00010881 2024-03-30T13:56:35.074 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.074 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.074 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:35.074 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.328269+0000", 2024-03-30T13:56:35.074 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.003026782 2024-03-30T13:56:35.074 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.074 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.074 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.074 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.328961+0000", 2024-03-30T13:56:35.074 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00069232300000000001 2024-03-30T13:56:35.074 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.075 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.075 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.075 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.449044+0000", 2024-03-30T13:56:35.075 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.12008284 2024-03-30T13:56:35.075 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.075 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.075 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:35.075 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.449090+0000", 2024-03-30T13:56:35.075 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6076999999999999e-05 2024-03-30T13:56:35.075 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.075 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.075 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:35.075 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.449117+0000", 2024-03-30T13:56:35.076 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.6825000000000002e-05 2024-03-30T13:56:35.076 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.076 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.076 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.076 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.076 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.076 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:250368 8.8 8:1d8b7580:::benchmark_data_smithi043_119832_object120014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:56:35.076 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:19.958219+0000", 2024-03-30T13:56:35.076 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 255.073245155, 2024-03-30T13:56:35.076 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.108421089, 2024-03-30T13:56:35.076 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:35.076 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:35.076 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:35.076 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:35.077 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:35.077 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:35.077 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 250368 2024-03-30T13:56:35.077 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.077 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:35.077 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.077 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:35.077 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.958219+0000", 2024-03-30T13:56:35.077 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.077 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.077 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.077 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:35.077 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.958219+0000", 2024-03-30T13:56:35.078 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.078 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.078 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.078 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:35.078 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.958220+0000", 2024-03-30T13:56:35.078 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2950000000000001e-06 2024-03-30T13:56:35.078 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.078 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.078 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:35.078 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.958232+0000", 2024-03-30T13:56:35.078 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1756000000000001e-05 2024-03-30T13:56:35.078 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.078 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.079 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:35.079 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.958245+0000", 2024-03-30T13:56:35.079 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3329000000000001e-05 2024-03-30T13:56:35.079 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.079 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.079 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:35.079 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.958254+0000", 2024-03-30T13:56:35.079 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.9290000000000004e-06 2024-03-30T13:56:35.079 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.079 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.079 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:35.079 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.958313+0000", 2024-03-30T13:56:35.079 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8714000000000001e-05 2024-03-30T13:56:35.079 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.080 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.080 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:35.080 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.958460+0000", 2024-03-30T13:56:35.080 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00014767299999999999 2024-03-30T13:56:35.080 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.080 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.080 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:56:35.080 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.958580+0000", 2024-03-30T13:56:35.080 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000119349 2024-03-30T13:56:35.080 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.080 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.080 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:35.080 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.964242+0000", 2024-03-30T13:56:35.080 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0056619929999999997 2024-03-30T13:56:35.081 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.081 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.081 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.081 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.967163+0000", 2024-03-30T13:56:35.081 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0029217069999999999 2024-03-30T13:56:35.081 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.081 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.081 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.081 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:20.066562+0000", 2024-03-30T13:56:35.081 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.099398891000000003 2024-03-30T13:56:35.081 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.081 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.081 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:35.081 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:20.066603+0000", 2024-03-30T13:56:35.082 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.0794000000000003e-05 2024-03-30T13:56:35.082 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.082 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.082 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:35.082 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:20.066640+0000", 2024-03-30T13:56:35.082 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.6659000000000003e-05 2024-03-30T13:56:35.082 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.082 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.082 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.082 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.082 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.082 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:23290 9.f 9:fbbaa984:::benchmark_data_smithi043_145506_object23289:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-30T13:56:35.082 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:59.494863+0000", 2024-03-30T13:56:35.083 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 215.53660075100001, 2024-03-30T13:56:35.083 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.095788047000000001, 2024-03-30T13:56:35.083 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:35.083 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:35.083 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:35.083 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:35.083 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T13:56:35.083 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:56:35.083 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 23290 2024-03-30T13:56:35.083 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.083 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:35.083 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.083 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:35.084 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:59.494863+0000", 2024-03-30T13:56:35.084 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.084 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.084 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.084 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:35.084 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:59.494863+0000", 2024-03-30T13:56:35.084 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.084 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.084 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.084 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:35.084 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:59.494863+0000", 2024-03-30T13:56:35.084 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.08e-07 2024-03-30T13:56:35.084 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.084 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.085 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:35.085 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:59.494893+0000", 2024-03-30T13:56:35.085 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.0049e-05 2024-03-30T13:56:35.085 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.085 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.085 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:35.085 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:59.494900+0000", 2024-03-30T13:56:35.085 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.4919999999999998e-06 2024-03-30T13:56:35.085 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.085 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.085 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:35.085 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:59.494905+0000", 2024-03-30T13:56:35.085 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0479999999999998e-06 2024-03-30T13:56:35.085 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.085 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.086 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:35.086 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:59.494958+0000", 2024-03-30T13:56:35.086 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3368000000000001e-05 2024-03-30T13:56:35.086 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.086 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.086 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:35.086 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:59.495092+0000", 2024-03-30T13:56:35.086 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013348799999999999 2024-03-30T13:56:35.086 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.087 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:34 smithi113 ceph-mon[34341]: pgmap v593: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 60 MiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s; 1669/3456 objects degraded (48.293%); 2225/3456 objects misplaced (64.381%); 0 B/s, 82 objects/s recovering 2024-03-30T13:56:35.087 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.087 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:56:35.087 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:59.495244+0000", 2024-03-30T13:56:35.087 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00015194500000000001 2024-03-30T13:56:35.087 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.087 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.087 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.087 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:59.496909+0000", 2024-03-30T13:56:35.087 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0016648450000000001 2024-03-30T13:56:35.088 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.088 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.088 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:35.088 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:59.496934+0000", 2024-03-30T13:56:35.088 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.4990999999999998e-05 2024-03-30T13:56:35.088 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.088 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.088 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.088 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:59.590555+0000", 2024-03-30T13:56:35.088 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.093621287999999997 2024-03-30T13:56:35.088 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.088 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.088 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:35.088 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:59.590607+0000", 2024-03-30T13:56:35.088 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1653000000000002e-05 2024-03-30T13:56:35.089 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.089 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.089 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:35.089 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:59.590651+0000", 2024-03-30T13:56:35.089 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.4471999999999997e-05 2024-03-30T13:56:35.089 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.089 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.089 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.089 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.089 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.089 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:27882 9.a 9:5572eb80:::benchmark_data_smithi043_145506_object27881:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-30T13:56:35.089 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:01.598280+0000", 2024-03-30T13:56:35.089 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 213.43318345599999, 2024-03-30T13:56:35.089 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.097180534999999998, 2024-03-30T13:56:35.090 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:35.090 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:35.090 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:35.090 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:35.090 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T13:56:35.090 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:56:35.090 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 27882 2024-03-30T13:56:35.090 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.090 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:35.090 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.090 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:35.090 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.598280+0000", 2024-03-30T13:56:35.090 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.091 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.091 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.091 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:35.091 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.598280+0000", 2024-03-30T13:56:35.091 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.091 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.091 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.091 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:35.091 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.598281+0000", 2024-03-30T13:56:35.091 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.116e-06 2024-03-30T13:56:35.091 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.091 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.091 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:35.092 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.598327+0000", 2024-03-30T13:56:35.092 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.5595e-05 2024-03-30T13:56:35.092 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.092 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.092 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:35.092 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.598338+0000", 2024-03-30T13:56:35.092 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1007e-05 2024-03-30T13:56:35.092 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.092 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.092 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:35.092 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.598346+0000", 2024-03-30T13:56:35.092 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.1319999999999994e-06 2024-03-30T13:56:35.092 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.093 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.093 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:35.093 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.599654+0000", 2024-03-30T13:56:35.093 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0013073570000000001 2024-03-30T13:56:35.093 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.093 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.093 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:35.093 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.599782+0000", 2024-03-30T13:56:35.093 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012827100000000001 2024-03-30T13:56:35.093 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.093 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.093 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,7", 2024-03-30T13:56:35.093 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.599979+0000", 2024-03-30T13:56:35.094 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000197147 2024-03-30T13:56:35.094 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.094 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.094 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:35.094 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.602709+0000", 2024-03-30T13:56:35.094 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0027303449999999999 2024-03-30T13:56:35.094 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.094 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.094 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.094 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.604600+0000", 2024-03-30T13:56:35.094 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0018910070000000001 2024-03-30T13:56:35.094 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.095 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.095 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.095 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.695375+0000", 2024-03-30T13:56:35.095 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.090774438999999998 2024-03-30T13:56:35.095 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.095 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.095 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:35.095 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.695423+0000", 2024-03-30T13:56:35.095 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8430000000000002e-05 2024-03-30T13:56:35.095 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.095 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.095 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:35.095 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.695461+0000", 2024-03-30T13:56:35.096 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.7688999999999998e-05 2024-03-30T13:56:35.096 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.096 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.096 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.096 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.096 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.096 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:27886 9.a 9:58701930:::benchmark_data_smithi043_145506_object27885:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-30T13:56:35.096 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:01.599849+0000", 2024-03-30T13:56:35.096 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 213.431614518, 2024-03-30T13:56:35.096 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.095764848999999999, 2024-03-30T13:56:35.096 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:35.096 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:35.096 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:35.097 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:35.097 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T13:56:35.097 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:56:35.097 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 27886 2024-03-30T13:56:35.097 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.097 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:35.097 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.097 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:35.097 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.599849+0000", 2024-03-30T13:56:35.097 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.097 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.097 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.097 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:35.097 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.599849+0000", 2024-03-30T13:56:35.097 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.098 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.098 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.098 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:35.098 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.599850+0000", 2024-03-30T13:56:35.098 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5000000000000003e-07 2024-03-30T13:56:35.098 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.098 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.098 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:35.098 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.599881+0000", 2024-03-30T13:56:35.098 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.1161000000000002e-05 2024-03-30T13:56:35.098 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.098 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.098 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:35.098 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.599888+0000", 2024-03-30T13:56:35.099 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.4769999999999997e-06 2024-03-30T13:56:35.099 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.099 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.099 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:35.099 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.599895+0000", 2024-03-30T13:56:35.099 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.516e-06 2024-03-30T13:56:35.099 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.099 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.099 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:35.099 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.601337+0000", 2024-03-30T13:56:35.099 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0014419890000000001 2024-03-30T13:56:35.099 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.099 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.099 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:35.099 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.601463+0000", 2024-03-30T13:56:35.100 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012646099999999999 2024-03-30T13:56:35.100 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.100 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.100 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,7", 2024-03-30T13:56:35.100 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.601601+0000", 2024-03-30T13:56:35.100 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000137656 2024-03-30T13:56:35.100 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.100 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.100 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:35.100 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.602988+0000", 2024-03-30T13:56:35.100 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0013873679999999999 2024-03-30T13:56:35.100 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.100 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.101 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.101 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.695324+0000", 2024-03-30T13:56:35.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:35.102 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:35.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:35.103 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.092335807000000006 2024-03-30T13:56:35.103 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.103 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.103 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.103 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.695539+0000", 2024-03-30T13:56:35.103 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000214296 2024-03-30T13:56:35.103 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.103 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.103 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:35.103 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.695582+0000", 2024-03-30T13:56:35.103 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3738e-05 2024-03-30T13:56:35.104 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.104 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.104 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:35.104 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.695614+0000", 2024-03-30T13:56:35.104 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.1829999999999998e-05 2024-03-30T13:56:35.104 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.104 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.104 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.104 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.104 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.104 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:27899 9.f 9:f2d08f9d:::benchmark_data_smithi043_145506_object27898:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-30T13:56:35.104 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:01.606630+0000", 2024-03-30T13:56:35.104 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 213.42483338700001, 2024-03-30T13:56:35.104 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.095713935, 2024-03-30T13:56:35.105 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:35.105 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:35.105 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:35.105 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:35.105 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T13:56:35.105 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:56:35.105 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 27899 2024-03-30T13:56:35.105 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.105 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:35.105 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.105 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:35.105 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.606630+0000", 2024-03-30T13:56:35.105 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.105 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.106 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.106 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:35.106 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.606630+0000", 2024-03-30T13:56:35.106 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.106 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.106 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.106 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:35.106 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.606632+0000", 2024-03-30T13:56:35.106 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5149999999999999e-06 2024-03-30T13:56:35.106 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.106 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.106 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:35.106 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.606695+0000", 2024-03-30T13:56:35.106 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.2654e-05 2024-03-30T13:56:35.107 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.107 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.107 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:35.107 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.694575+0000", 2024-03-30T13:56:35.107 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.087880357000000006 2024-03-30T13:56:35.107 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.107 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.107 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:35.107 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.694588+0000", 2024-03-30T13:56:35.107 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3165e-05 2024-03-30T13:56:35.107 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.107 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.107 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:35.107 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.694722+0000", 2024-03-30T13:56:35.108 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000133769 2024-03-30T13:56:35.108 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.108 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.108 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:35.108 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.694904+0000", 2024-03-30T13:56:35.108 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000182175 2024-03-30T13:56:35.108 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.108 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.108 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:56:35.108 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.695339+0000", 2024-03-30T13:56:35.108 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00043473399999999999 2024-03-30T13:56:35.108 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.108 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.108 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:35.109 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.698245+0000", 2024-03-30T13:56:35.109 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0029059099999999998 2024-03-30T13:56:35.109 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.109 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.109 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.109 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.698948+0000", 2024-03-30T13:56:35.109 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00070378699999999999 2024-03-30T13:56:35.109 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.109 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.109 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.109 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.702002+0000", 2024-03-30T13:56:35.109 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.003053886 2024-03-30T13:56:35.109 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.110 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.110 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:35.110 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.702304+0000", 2024-03-30T13:56:35.110 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000301992 2024-03-30T13:56:35.110 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.110 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.110 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:35.110 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:01.702344+0000", 2024-03-30T13:56:35.110 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.9990999999999997e-05 2024-03-30T13:56:35.110 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.110 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.110 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.110 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.110 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.111 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:29677 9.7 9:eded116d:::benchmark_data_smithi043_145506_object29676:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-30T13:56:35.111 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:02.623619+0000", 2024-03-30T13:56:35.111 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 212.407845192, 2024-03-30T13:56:35.111 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.14263978499999999, 2024-03-30T13:56:35.111 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:35.111 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:35.111 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:35.111 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:35.111 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T13:56:35.111 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:56:35.111 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 29677 2024-03-30T13:56:35.111 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.111 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:35.111 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.112 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:35.112 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.623619+0000", 2024-03-30T13:56:35.112 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.112 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.112 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.112 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:35.112 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.623619+0000", 2024-03-30T13:56:35.112 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.112 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.112 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.112 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:35.112 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.623619+0000", 2024-03-30T13:56:35.112 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.2099999999999996e-07 2024-03-30T13:56:35.112 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.113 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.113 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:35.113 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.623677+0000", 2024-03-30T13:56:35.113 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7639000000000002e-05 2024-03-30T13:56:35.113 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.113 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.113 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:35.113 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.623685+0000", 2024-03-30T13:56:35.113 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.49e-06 2024-03-30T13:56:35.113 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.113 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.113 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:35.113 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.623692+0000", 2024-03-30T13:56:35.114 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5899999999999996e-06 2024-03-30T13:56:35.114 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.114 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.114 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:35.114 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.623732+0000", 2024-03-30T13:56:35.114 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.9975000000000003e-05 2024-03-30T13:56:35.114 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.114 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.114 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:35.114 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.623851+0000", 2024-03-30T13:56:35.114 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011944599999999999 2024-03-30T13:56:35.114 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.114 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.114 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,7", 2024-03-30T13:56:35.115 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.623967+0000", 2024-03-30T13:56:35.115 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000115523 2024-03-30T13:56:35.115 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.115 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.115 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:35.115 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.625862+0000", 2024-03-30T13:56:35.115 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0018955140000000001 2024-03-30T13:56:35.115 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.115 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.115 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.115 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.625953+0000", 2024-03-30T13:56:35.115 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.0957000000000003e-05 2024-03-30T13:56:35.115 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.116 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.116 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.116 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.766156+0000", 2024-03-30T13:56:35.116 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.14020278 2024-03-30T13:56:35.116 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.116 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.116 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:35.116 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.766206+0000", 2024-03-30T13:56:35.116 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9410000000000003e-05 2024-03-30T13:56:35.116 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.116 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.116 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:35.117 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.766258+0000", 2024-03-30T13:56:35.117 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.274e-05 2024-03-30T13:56:35.117 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.117 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.117 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.117 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.117 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.117 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:29678 9.f 9:f579b2d4:::benchmark_data_smithi043_145506_object29677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-30T13:56:35.117 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:02.624029+0000", 2024-03-30T13:56:35.117 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 212.407434491, 2024-03-30T13:56:35.117 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.14234263999999999, 2024-03-30T13:56:35.117 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:35.117 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:35.118 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:35.118 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:35.118 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T13:56:35.118 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:56:35.118 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 29678 2024-03-30T13:56:35.118 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.118 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:35.118 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.118 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:35.118 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.624029+0000", 2024-03-30T13:56:35.118 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.118 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.118 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.119 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:35.119 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.624029+0000", 2024-03-30T13:56:35.119 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.119 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.119 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.119 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:35.119 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.624030+0000", 2024-03-30T13:56:35.119 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8199999999999999e-07 2024-03-30T13:56:35.119 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.119 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.119 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:35.119 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.624058+0000", 2024-03-30T13:56:35.119 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.7841e-05 2024-03-30T13:56:35.119 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.119 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.120 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:35.120 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.624066+0000", 2024-03-30T13:56:35.120 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.2039999999999997e-06 2024-03-30T13:56:35.120 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.120 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.120 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:35.120 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.624073+0000", 2024-03-30T13:56:35.120 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.4760000000000003e-06 2024-03-30T13:56:35.120 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.120 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.120 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:35.120 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.624156+0000", 2024-03-30T13:56:35.121 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.3628999999999995e-05 2024-03-30T13:56:35.121 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.121 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.121 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:35.121 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.624278+0000", 2024-03-30T13:56:35.121 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000121853 2024-03-30T13:56:35.121 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.121 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.121 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3", 2024-03-30T13:56:35.121 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.624430+0000", 2024-03-30T13:56:35.121 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00015242700000000001 2024-03-30T13:56:35.121 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.121 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.122 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:35.122 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.625868+0000", 2024-03-30T13:56:35.122 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0014377560000000001 2024-03-30T13:56:35.122 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.122 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.122 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.122 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.627020+0000", 2024-03-30T13:56:35.122 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001151539 2024-03-30T13:56:35.122 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.122 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.122 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.122 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.766298+0000", 2024-03-30T13:56:35.123 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.13927857599999999 2024-03-30T13:56:35.123 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.123 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.123 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:35.123 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.766341+0000", 2024-03-30T13:56:35.123 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.2809000000000003e-05 2024-03-30T13:56:35.123 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.123 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.123 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:35.123 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.766372+0000", 2024-03-30T13:56:35.123 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.0747999999999998e-05 2024-03-30T13:56:35.123 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.123 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.123 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.123 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.124 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.124 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:29683 9.6 9:685e1853:::benchmark_data_smithi043_145506_object29682:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-30T13:56:35.124 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:02.626896+0000", 2024-03-30T13:56:35.124 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 212.40456736199999, 2024-03-30T13:56:35.124 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.14255252400000001, 2024-03-30T13:56:35.124 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:35.124 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:35.124 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:35.124 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:35.124 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T13:56:35.124 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:56:35.124 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 29683 2024-03-30T13:56:35.124 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.125 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:35.125 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.125 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:35.125 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.626896+0000", 2024-03-30T13:56:35.125 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.125 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.125 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.125 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:35.125 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.626896+0000", 2024-03-30T13:56:35.125 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.125 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.125 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.126 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:35.126 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.626898+0000", 2024-03-30T13:56:35.126 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2619999999999999e-06 2024-03-30T13:56:35.126 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.126 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.126 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:35.126 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.626956+0000", 2024-03-30T13:56:35.126 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8374e-05 2024-03-30T13:56:35.126 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.126 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.126 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:35.126 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.626967+0000", 2024-03-30T13:56:35.126 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1389000000000001e-05 2024-03-30T13:56:35.126 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.127 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.127 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:35.127 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.626975+0000", 2024-03-30T13:56:35.127 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.644e-06 2024-03-30T13:56:35.127 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.127 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.127 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:35.127 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.627029+0000", 2024-03-30T13:56:35.127 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4360000000000001e-05 2024-03-30T13:56:35.127 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.127 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.127 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:35.127 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.627160+0000", 2024-03-30T13:56:35.127 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013063199999999999 2024-03-30T13:56:35.128 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.128 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.128 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1", 2024-03-30T13:56:35.128 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.627405+0000", 2024-03-30T13:56:35.128 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00024529899999999998 2024-03-30T13:56:35.128 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.128 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.128 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.128 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.766658+0000", 2024-03-30T13:56:35.128 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.139252665 2024-03-30T13:56:35.128 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.128 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.128 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:35.129 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.769299+0000", 2024-03-30T13:56:35.129 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.002640795 2024-03-30T13:56:35.129 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.129 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.129 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:35.129 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.769352+0000", 2024-03-30T13:56:35.129 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3146000000000002e-05 2024-03-30T13:56:35.129 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.129 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.129 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:35.129 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.769449+0000", 2024-03-30T13:56:35.129 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.6958000000000002e-05 2024-03-30T13:56:35.129 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.130 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.130 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.130 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.130 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.130 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop(client.15354.0:29664 9.9 e219/211 9:99d08681:::benchmark_data_smithi043_145506_object29663:head v 219'1904, mlcod=219'1901)", 2024-03-30T13:56:35.130 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:02.627101+0000", 2024-03-30T13:56:35.130 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 212.404362354, 2024-03-30T13:56:35.130 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.13849988799999999, 2024-03-30T13:56:35.130 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:35.130 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:35.130 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:35.130 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:35.130 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.130 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:35.131 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.627101+0000", 2024-03-30T13:56:35.131 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.131 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.131 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.131 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:35.131 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.627101+0000", 2024-03-30T13:56:35.131 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.131 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.131 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.131 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:35.131 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.627102+0000", 2024-03-30T13:56:35.131 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2900000000000004e-07 2024-03-30T13:56:35.131 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.132 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.132 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:35.132 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.627179+0000", 2024-03-30T13:56:35.132 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.6824999999999997e-05 2024-03-30T13:56:35.132 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.132 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.132 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:35.132 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.627185+0000", 2024-03-30T13:56:35.132 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.2519999999999996e-06 2024-03-30T13:56:35.132 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.132 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.132 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:35.132 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.627190+0000", 2024-03-30T13:56:35.132 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7160000000000002e-06 2024-03-30T13:56:35.132 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.133 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.133 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:35.133 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.627248+0000", 2024-03-30T13:56:35.133 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8236000000000003e-05 2024-03-30T13:56:35.133 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.133 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.133 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:35.133 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.627267+0000", 2024-03-30T13:56:35.133 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.9286000000000001e-05 2024-03-30T13:56:35.133 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.133 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.133 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:35.133 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.765560+0000", 2024-03-30T13:56:35.133 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.13829275299999999 2024-03-30T13:56:35.133 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.134 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.134 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:35.134 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.765601+0000", 2024-03-30T13:56:35.134 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1291000000000002e-05 2024-03-30T13:56:35.134 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.134 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.134 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.134 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.134 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.134 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop(client.15354.0:29682 9.d e219/211 9:b8ecca61:::benchmark_data_smithi043_145506_object29681:head v 219'1874, mlcod=219'1872)", 2024-03-30T13:56:35.134 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:02.627377+0000", 2024-03-30T13:56:35.134 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 212.404086455, 2024-03-30T13:56:35.134 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.14022393899999999, 2024-03-30T13:56:35.135 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:35.135 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:35.135 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:35.135 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:35.135 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.135 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:35.135 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.627377+0000", 2024-03-30T13:56:35.135 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.135 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.135 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.135 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:35.135 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.627377+0000", 2024-03-30T13:56:35.135 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.135 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.135 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.136 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:35.136 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.627378+0000", 2024-03-30T13:56:35.136 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8400000000000004e-07 2024-03-30T13:56:35.136 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.136 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.136 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:35.136 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.627402+0000", 2024-03-30T13:56:35.136 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.387e-05 2024-03-30T13:56:35.136 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.136 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.136 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:35.136 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.627412+0000", 2024-03-30T13:56:35.136 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.8090000000000004e-06 2024-03-30T13:56:35.136 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.136 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.136 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:35.137 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.627418+0000", 2024-03-30T13:56:35.137 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3729999999999998e-06 2024-03-30T13:56:35.137 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.137 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.137 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:35.137 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.765411+0000", 2024-03-30T13:56:35.137 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.13799249599999999 2024-03-30T13:56:35.137 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.137 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.137 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:35.137 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.765433+0000", 2024-03-30T13:56:35.137 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.2929999999999999e-05 2024-03-30T13:56:35.137 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.137 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.137 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:35.138 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.767531+0000", 2024-03-30T13:56:35.138 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0020973960000000001 2024-03-30T13:56:35.138 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.138 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.138 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:35.138 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.767601+0000", 2024-03-30T13:56:35.138 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.0381000000000003e-05 2024-03-30T13:56:35.138 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.138 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.138 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.138 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.138 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.138 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:29687 9.6 9:6cf976f5:::benchmark_data_smithi043_145506_object29686:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-30T13:56:35.138 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:02.628638+0000", 2024-03-30T13:56:35.139 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 212.40282555100001, 2024-03-30T13:56:35.139 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.140944493, 2024-03-30T13:56:35.139 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:35.139 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:35.139 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:35.139 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:35.139 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T13:56:35.139 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:56:35.139 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 29687 2024-03-30T13:56:35.139 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.139 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:35.139 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.139 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:35.139 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.628638+0000", 2024-03-30T13:56:35.139 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.140 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.140 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.140 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:35.140 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.628638+0000", 2024-03-30T13:56:35.140 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.140 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.140 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.140 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:35.140 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.628639+0000", 2024-03-30T13:56:35.140 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0729999999999999e-06 2024-03-30T13:56:35.140 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.140 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.140 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:35.140 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.628716+0000", 2024-03-30T13:56:35.141 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.6429999999999998e-05 2024-03-30T13:56:35.141 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.141 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.141 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:35.141 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.765626+0000", 2024-03-30T13:56:35.141 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.13691035300000001 2024-03-30T13:56:35.141 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.141 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.141 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:35.141 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.765636+0000", 2024-03-30T13:56:35.141 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0360000000000001e-05 2024-03-30T13:56:35.141 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.141 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.141 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:35.141 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.766869+0000", 2024-03-30T13:56:35.142 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001232807 2024-03-30T13:56:35.142 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.142 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.142 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:35.142 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.766979+0000", 2024-03-30T13:56:35.142 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000109955 2024-03-30T13:56:35.142 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.142 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.142 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1", 2024-03-30T13:56:35.142 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.767127+0000", 2024-03-30T13:56:35.142 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00014822899999999999 2024-03-30T13:56:35.142 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.142 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.143 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:35.143 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.769212+0000", 2024-03-30T13:56:35.143 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.002084252 2024-03-30T13:56:35.143 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.143 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.143 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:35.143 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.769459+0000", 2024-03-30T13:56:35.143 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00024761 2024-03-30T13:56:35.143 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.143 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.143 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:35.143 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.769507+0000", 2024-03-30T13:56:35.143 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7373999999999997e-05 2024-03-30T13:56:35.143 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.144 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.144 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:35.144 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.769583+0000", 2024-03-30T13:56:35.144 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.6050000000000005e-05 2024-03-30T13:56:35.144 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.144 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.144 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.144 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.144 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.144 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop(client.15354.0:29691 9.d e219/211 9:bf9e9255:::benchmark_data_smithi043_145506_object29690:head v 219'1875, mlcod=219'1872)", 2024-03-30T13:56:35.144 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:02.630240+0000", 2024-03-30T13:56:35.144 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 212.401224136, 2024-03-30T13:56:35.145 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.13895160300000001, 2024-03-30T13:56:35.145 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:35.145 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:35.145 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:35.145 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:35.145 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.145 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:35.145 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.630240+0000", 2024-03-30T13:56:35.145 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.145 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.145 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.145 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:35.145 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.630240+0000", 2024-03-30T13:56:35.145 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.146 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.146 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.146 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:35.146 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.630240+0000", 2024-03-30T13:56:35.146 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6899999999999996e-07 2024-03-30T13:56:35.146 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.146 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.146 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:35.146 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.630269+0000", 2024-03-30T13:56:35.146 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.8912999999999999e-05 2024-03-30T13:56:35.146 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.146 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.146 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:35.146 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.765211+0000", 2024-03-30T13:56:35.147 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.13494110100000001 2024-03-30T13:56:35.147 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.147 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.147 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:35.147 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.765234+0000", 2024-03-30T13:56:35.147 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.3992e-05 2024-03-30T13:56:35.147 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.147 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.147 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:35.147 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.766568+0000", 2024-03-30T13:56:35.147 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001333973 2024-03-30T13:56:35.147 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.147 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.148 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:35.148 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.766597+0000", 2024-03-30T13:56:35.148 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.8631000000000001e-05 2024-03-30T13:56:35.148 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.148 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.148 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:35.148 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.769158+0000", 2024-03-30T13:56:35.148 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.002561144 2024-03-30T13:56:35.148 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.148 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.148 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:35.149 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.769191+0000", 2024-03-30T13:56:35.149 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2979999999999999e-05 2024-03-30T13:56:35.149 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.149 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.149 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.149 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.149 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.149 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop(client.15354.0:29688 9.b e219/211 9:d61db7eb:::benchmark_data_smithi043_145506_object29687:head v 219'1943, mlcod=219'1941)", 2024-03-30T13:56:35.149 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:02.630386+0000", 2024-03-30T13:56:35.149 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 212.40107749500001, 2024-03-30T13:56:35.149 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.13728396700000001, 2024-03-30T13:56:35.150 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:35.150 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:35.150 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:35.150 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:35.150 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.150 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:35.150 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.630386+0000", 2024-03-30T13:56:35.150 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.150 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.150 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.150 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:35.150 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.630386+0000", 2024-03-30T13:56:35.150 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:35.150 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.151 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.151 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:35.151 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.630387+0000", 2024-03-30T13:56:35.151 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.0800000000000004e-07 2024-03-30T13:56:35.151 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.151 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.151 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:35.151 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.630453+0000", 2024-03-30T13:56:35.151 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6278000000000005e-05 2024-03-30T13:56:35.151 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.151 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.151 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:35.151 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.765241+0000", 2024-03-30T13:56:35.151 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.134787619 2024-03-30T13:56:35.152 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.152 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.152 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:35.152 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.765252+0000", 2024-03-30T13:56:35.152 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1466e-05 2024-03-30T13:56:35.152 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.152 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.152 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:35.152 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.765324+0000", 2024-03-30T13:56:35.152 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1863e-05 2024-03-30T13:56:35.152 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.152 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.152 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:35.152 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.765348+0000", 2024-03-30T13:56:35.153 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.3873999999999999e-05 2024-03-30T13:56:35.153 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.153 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.153 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:35.153 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.767633+0000", 2024-03-30T13:56:35.153 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.002284763 2024-03-30T13:56:35.153 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:35.153 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:35.153 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:35.153 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:02.767670+0000", 2024-03-30T13:56:35.153 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.7395999999999999e-05 2024-03-30T13:56:35.153 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.153 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.154 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.154 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:35.154 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:35.154 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:56:35.154 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T13:56:35.155 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:34 smithi043 ceph-mon[29963]: pgmap v593: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 60 MiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s; 1669/3456 objects degraded (48.293%); 2225/3456 objects misplaced (64.381%); 0 B/s, 82 objects/s recovering 2024-03-30T13:56:35.155 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:34 smithi043 ceph-mon[42099]: pgmap v593: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 60 MiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s; 1669/3456 objects degraded (48.293%); 2225/3456 objects misplaced (64.381%); 0 B/s, 82 objects/s recovering 2024-03-30T13:56:35.204 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:35.232 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:35.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:35.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:35.334 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:35.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:35.359 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:35.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:35.460 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:35.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:35.484 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:35.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:35.586 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:35.614 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:35.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:35.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:35.716 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:35.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:35.757 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:56:35.758 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:56:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:56:35.850 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:35.875 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:35.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:35.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:35.943 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-30T13:56:35.977 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:36.004 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:36.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:36.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:36.106 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:36.130 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:36.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:36.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:36.180 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:36.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:36.261 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:36.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:36.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:36.363 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:36.386 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:56:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:56:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:56:36.389 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:36.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:36.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:36.492 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:36.516 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:36.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:36.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:36.617 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:36.644 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:36.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:36.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:36.745 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:36.769 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:36.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:36.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:36.871 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:36.896 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:36.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:36.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:36.997 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:37.021 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:37.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:37.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:37.123 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:37.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:36 smithi043 ceph-mon[29963]: pgmap v594: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 43 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 975 B/s rd, 0 B/s wr, 991 op/s; 1669/2628 objects degraded (63.508%); 2139/2628 objects misplaced (81.393%); 0 B/s, 74 objects/s recovering 2024-03-30T13:56:37.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:36 smithi043 ceph-mon[29963]: osdmap e279: 8 total, 7 up, 6 in 2024-03-30T13:56:37.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:36 smithi043 ceph-mon[42099]: pgmap v594: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 43 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 975 B/s rd, 0 B/s wr, 991 op/s; 1669/2628 objects degraded (63.508%); 2139/2628 objects misplaced (81.393%); 0 B/s, 74 objects/s recovering 2024-03-30T13:56:37.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:36 smithi043 ceph-mon[42099]: osdmap e279: 8 total, 7 up, 6 in 2024-03-30T13:56:37.146 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:37.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:37.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:37.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:36 smithi113 ceph-mon[34341]: pgmap v594: 158 pgs: 89 active+clean, 12 active+undersized, 2 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded, 46 active+clean+remapped; 43 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 975 B/s rd, 0 B/s wr, 991 op/s; 1669/2628 objects degraded (63.508%); 2139/2628 objects misplaced (81.393%); 0 B/s, 74 objects/s recovering 2024-03-30T13:56:37.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:36 smithi113 ceph-mon[34341]: osdmap e279: 8 total, 7 up, 6 in 2024-03-30T13:56:37.248 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:37.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:37.272 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:37.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:37.373 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:37.379 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-03-30T13:56:37.379 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-03-30T13:56:37.398 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:37.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:37.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:37.477 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6] out_osds: [5, 7] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-30T13:56:37.477 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:56:37.478 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-03-30T13:56:37.478 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-30T13:56:37.478 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-30T13:56:37.478 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-03-30T13:56:37.500 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:37.524 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:37.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:37.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:37.626 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:37.651 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:37.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:37.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:37.753 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:37.759 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:56:37.775 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:37.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:37.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:37.877 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:37.902 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:37.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:37.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:37.935 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:56:37.936 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [], 2024-03-30T13:56:37.936 INFO:teuthology.orchestra.run.smithi113.stdout: "num_ops": 0 2024-03-30T13:56:37.936 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:56:38.004 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:38.028 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:38.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:38.130 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:38.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:37 smithi043 ceph-mon[29963]: osdmap e280: 8 total, 7 up, 6 in 2024-03-30T13:56:38.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:37 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 1669/2628 objects degraded (63.508%), 11 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:38.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:37 smithi043 ceph-mon[29963]: osdmap e281: 8 total, 7 up, 6 in 2024-03-30T13:56:38.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:37 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:38.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:37 smithi043 ceph-mon[42099]: osdmap e280: 8 total, 7 up, 6 in 2024-03-30T13:56:38.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:37 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 1669/2628 objects degraded (63.508%), 11 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:38.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:37 smithi043 ceph-mon[42099]: osdmap e281: 8 total, 7 up, 6 in 2024-03-30T13:56:38.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:37 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:38.154 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:38.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:38.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:38.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:37 smithi113 ceph-mon[34341]: osdmap e280: 8 total, 7 up, 6 in 2024-03-30T13:56:38.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:37 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 1669/2628 objects degraded (63.508%), 11 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:38.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:37 smithi113 ceph-mon[34341]: osdmap e281: 8 total, 7 up, 6 in 2024-03-30T13:56:38.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:37 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:38.256 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:38.279 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:38.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:38.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:38.381 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:38.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:38.406 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:38.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:38.507 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:38.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:38.533 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:38.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:38.634 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:38.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:38.659 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:38.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:38.701 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-03-30T13:56:38.760 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:38.783 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:38.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:38.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:38.885 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:38.909 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:56:38.909 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:38.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:38.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:38.937 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:38.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:38 smithi113 ceph-mon[34341]: pgmap v598: 158 pgs: 89 active+clean, 1 active+recovering+undersized+remapped, 14 active+undersized, 1 active+undersized+remapped, 7 active+undersized+degraded, 46 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 192 op/s; 47/597 objects degraded (7.873%); 1024/597 objects misplaced (171.524%); 0 B/s, 406 objects/s recovering 2024-03-30T13:56:38.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:38 smithi113 ceph-mon[34341]: osdmap e282: 8 total, 7 up, 6 in 2024-03-30T13:56:38.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:38 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1791733901' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:56:39.011 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:39.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:39.036 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:39.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:39.137 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:39.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:38 smithi043 ceph-mon[42099]: pgmap v598: 158 pgs: 89 active+clean, 1 active+recovering+undersized+remapped, 14 active+undersized, 1 active+undersized+remapped, 7 active+undersized+degraded, 46 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 192 op/s; 47/597 objects degraded (7.873%); 1024/597 objects misplaced (171.524%); 0 B/s, 406 objects/s recovering 2024-03-30T13:56:39.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:38 smithi043 ceph-mon[42099]: osdmap e282: 8 total, 7 up, 6 in 2024-03-30T13:56:39.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:38 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1791733901' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:56:39.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:38 smithi043 ceph-mon[29963]: pgmap v598: 158 pgs: 89 active+clean, 1 active+recovering+undersized+remapped, 14 active+undersized, 1 active+undersized+remapped, 7 active+undersized+degraded, 46 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 192 op/s; 47/597 objects degraded (7.873%); 1024/597 objects misplaced (171.524%); 0 B/s, 406 objects/s recovering 2024-03-30T13:56:39.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:38 smithi043 ceph-mon[29963]: osdmap e282: 8 total, 7 up, 6 in 2024-03-30T13:56:39.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:38 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1791733901' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:56:39.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:39.161 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:39.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:39.262 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:39.286 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:39.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:39.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:39.357 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:56:39 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:56:39.389 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:39.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:39.413 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:39.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:39.514 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:39.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:39.539 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:39.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:39.565 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-03-30T13:56:39.641 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:39.665 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:39.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:39.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:39.767 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:39.791 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:39.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:39.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:39.892 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:39.916 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:39.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:39.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:40.018 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:40.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:40.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:40.126 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:39 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:40.127 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:39 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:40.127 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:39 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:56:40.127 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:39 smithi113 ceph-mon[34341]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:56:40.127 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:39 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1791733901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:56:40.127 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:39 smithi113 ceph-mon[34341]: osdmap e283: 8 total, 7 up, 6 in 2024-03-30T13:56:40.146 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:40.170 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:40.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:40.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:40.272 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:40.300 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:40.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:40.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:40.306 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:39 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:40.306 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:39 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:40.306 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:39 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:56:40.306 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:39 smithi043 ceph-mon[29963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:56:40.306 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:39 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1791733901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:56:40.306 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:39 smithi043 ceph-mon[29963]: osdmap e283: 8 total, 7 up, 6 in 2024-03-30T13:56:40.306 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:39 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:40.306 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:39 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:40.307 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:39 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:56:40.307 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:39 smithi043 ceph-mon[42099]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:56:40.307 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:39 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1791733901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:56:40.307 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:39 smithi043 ceph-mon[42099]: osdmap e283: 8 total, 7 up, 6 in 2024-03-30T13:56:40.402 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:40.426 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:40.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:40.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:40.527 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:40.551 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:40.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:40.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:40.653 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:40.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:40.677 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:40.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:40.764 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:56:40.765 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [], 2024-03-30T13:56:40.765 INFO:teuthology.orchestra.run.smithi113.stdout: "complaint_time": 30, 2024-03-30T13:56:40.765 INFO:teuthology.orchestra.run.smithi113.stdout: "num_blocked_ops": 0 2024-03-30T13:56:40.765 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:56:40.778 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:40.802 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:40.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:40.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:40.904 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:40.928 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:40.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:40.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:41.030 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:41.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:41.054 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:41.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:41.092 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:40 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:56:41.093 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:40 smithi113 ceph-mon[34341]: pgmap v601: 158 pgs: 89 active+clean, 1 active+recovering+undersized+remapped, 14 active+undersized, 1 active+undersized+remapped, 7 active+undersized+degraded, 46 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 91 op/s; 47/597 objects degraded (7.873%); 1024/597 objects misplaced (171.524%) 2024-03-30T13:56:41.155 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:41.179 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:41.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:41.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:41.281 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:41.310 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:41.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:41.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:41.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:40 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:56:41.429 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:40 smithi043 ceph-mon[29963]: pgmap v601: 158 pgs: 89 active+clean, 1 active+recovering+undersized+remapped, 14 active+undersized, 1 active+undersized+remapped, 7 active+undersized+degraded, 46 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 91 op/s; 47/597 objects degraded (7.873%); 1024/597 objects misplaced (171.524%) 2024-03-30T13:56:41.429 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:40 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:56:41.430 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:40 smithi043 ceph-mon[42099]: pgmap v601: 158 pgs: 89 active+clean, 1 active+recovering+undersized+remapped, 14 active+undersized, 1 active+undersized+remapped, 7 active+undersized+degraded, 46 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 91 op/s; 47/597 objects degraded (7.873%); 1024/597 objects misplaced (171.524%) 2024-03-30T13:56:41.430 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:41.431 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-03-30T13:56:41.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:41.496 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:41.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:41.597 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:41.621 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:41.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:41.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:41.712 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:41.723 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:41.752 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:41.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:41.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:41.854 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:41.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:41.879 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:41.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:41.980 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:42.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:42.004 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:42.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:42.105 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:42.130 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:42.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:42.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:42.232 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:42.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:42.256 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:42.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:42.357 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:42.381 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:42.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:42.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:42.483 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:42.507 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:42.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:42.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:42.609 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:42.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:42.634 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:42.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:42.665 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:56:42.735 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:42.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:42.761 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:42.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:42.861 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:42.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:42.887 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:42.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:42.988 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:43.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:43.012 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:43.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:43.034 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:42 smithi043 ceph-mon[29963]: pgmap v602: 158 pgs: 90 active+clean, 14 active+undersized, 7 active+undersized+degraded, 47 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%); 0 B/s, 211 objects/s recovering 2024-03-30T13:56:43.035 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:42 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 47/597 objects degraded (7.873%), 7 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:43.035 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:42 smithi043 ceph-mon[29963]: osdmap e284: 8 total, 7 up, 6 in 2024-03-30T13:56:43.035 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:42 smithi043 ceph-mon[42099]: pgmap v602: 158 pgs: 90 active+clean, 14 active+undersized, 7 active+undersized+degraded, 47 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%); 0 B/s, 211 objects/s recovering 2024-03-30T13:56:43.035 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:42 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 47/597 objects degraded (7.873%), 7 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:43.035 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:42 smithi043 ceph-mon[42099]: osdmap e284: 8 total, 7 up, 6 in 2024-03-30T13:56:43.087 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:56:43.113 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:43.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:43.139 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:43.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:43.165 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:42 smithi113 ceph-mon[34341]: pgmap v602: 158 pgs: 90 active+clean, 14 active+undersized, 7 active+undersized+degraded, 47 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%); 0 B/s, 211 objects/s recovering 2024-03-30T13:56:43.166 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:42 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 47/597 objects degraded (7.873%), 7 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:43.166 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:42 smithi113 ceph-mon[34341]: osdmap e284: 8 total, 7 up, 6 in 2024-03-30T13:56:43.241 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:43.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:43.265 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:43.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:43.366 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:43.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:43.390 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:43.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:43.397 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:56:43.397 INFO:teuthology.orchestra.run.smithi113.stdout: "size": 20, 2024-03-30T13:56:43.397 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 600, 2024-03-30T13:56:43.397 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [ 2024-03-30T13:56:43.397 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.397 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14817.0:145811 7.d 7:b6b9c34e:::benchmark_data_smithi043_86989_object23473:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-30T13:56:43.397 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:46:55.503454+0000", 2024-03-30T13:56:43.398 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 587.89529413599996, 2024-03-30T13:56:43.398 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.162165796, 2024-03-30T13:56:43.398 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:43.398 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:43.398 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:43.398 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:43.398 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14817", 2024-03-30T13:56:43.398 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:56:43.398 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 145811 2024-03-30T13:56:43.398 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.398 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:43.398 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.399 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:43.399 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.503454+0000", 2024-03-30T13:56:43.399 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.399 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.399 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.399 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:43.399 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.503454+0000", 2024-03-30T13:56:43.399 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.399 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.399 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.399 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:43.399 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.503455+0000", 2024-03-30T13:56:43.399 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.305e-06 2024-03-30T13:56:43.399 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.399 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.400 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:43.400 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.503460+0000", 2024-03-30T13:56:43.400 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.079e-06 2024-03-30T13:56:43.400 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.400 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.400 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:43.400 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.503473+0000", 2024-03-30T13:56:43.400 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2904999999999999e-05 2024-03-30T13:56:43.400 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.400 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.400 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:43.400 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.503481+0000", 2024-03-30T13:56:43.400 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.3989999999999998e-06 2024-03-30T13:56:43.400 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.400 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.401 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.401 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.503601+0000", 2024-03-30T13:56:43.401 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012061900000000001 2024-03-30T13:56:43.401 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.401 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.401 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:43.401 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.503720+0000", 2024-03-30T13:56:43.401 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000118393 2024-03-30T13:56:43.401 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.401 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.401 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1", 2024-03-30T13:56:43.402 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.506427+0000", 2024-03-30T13:56:43.402 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0027076890000000001 2024-03-30T13:56:43.402 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.402 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.402 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:43.402 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.511180+0000", 2024-03-30T13:56:43.402 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0047527400000000001 2024-03-30T13:56:43.402 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.402 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.402 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.402 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.512587+0000", 2024-03-30T13:56:43.402 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001406504 2024-03-30T13:56:43.403 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.403 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.403 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.403 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.665514+0000", 2024-03-30T13:56:43.403 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.15292704800000001 2024-03-30T13:56:43.403 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.403 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.403 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:43.403 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.665573+0000", 2024-03-30T13:56:43.403 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9477e-05 2024-03-30T13:56:43.403 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.403 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.403 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:43.403 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:46:55.665620+0000", 2024-03-30T13:56:43.403 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6638000000000003e-05 2024-03-30T13:56:43.404 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.404 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.404 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.404 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.404 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.404 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:31274 8.5 8:a5782853:::benchmark_data_smithi043_119832_object31273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-30T13:56:43.404 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:49:27.602094+0000", 2024-03-30T13:56:43.404 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 435.79665372599999, 2024-03-30T13:56:43.404 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.177822909, 2024-03-30T13:56:43.404 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:43.404 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:43.404 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:43.404 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:43.404 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:43.404 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 31274 2024-03-30T13:56:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.602094+0000", 2024-03-30T13:56:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.602094+0000", 2024-03-30T13:56:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:43.406 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.602095+0000", 2024-03-30T13:56:43.406 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.8700000000000004e-07 2024-03-30T13:56:43.406 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.406 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.406 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:43.406 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.602128+0000", 2024-03-30T13:56:43.406 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2987000000000001e-05 2024-03-30T13:56:43.406 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.406 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.406 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:43.406 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.602140+0000", 2024-03-30T13:56:43.407 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1474999999999999e-05 2024-03-30T13:56:43.407 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.407 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.407 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:43.407 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.602148+0000", 2024-03-30T13:56:43.407 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8380000000000002e-06 2024-03-30T13:56:43.407 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.407 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.407 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.408 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.602196+0000", 2024-03-30T13:56:43.408 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8050999999999998e-05 2024-03-30T13:56:43.408 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.408 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.408 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:43.408 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.602314+0000", 2024-03-30T13:56:43.408 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000118026 2024-03-30T13:56:43.408 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.409 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.409 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 4,7", 2024-03-30T13:56:43.409 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.602437+0000", 2024-03-30T13:56:43.409 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012336899999999999 2024-03-30T13:56:43.409 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.409 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.409 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:43.409 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.603685+0000", 2024-03-30T13:56:43.409 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0012479329999999999 2024-03-30T13:56:43.409 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.409 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.409 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.409 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.604229+0000", 2024-03-30T13:56:43.410 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00054412000000000004 2024-03-30T13:56:43.410 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.410 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.410 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.410 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.779807+0000", 2024-03-30T13:56:43.410 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.17557811200000001 2024-03-30T13:56:43.410 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.410 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.410 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:43.410 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.779871+0000", 2024-03-30T13:56:43.410 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3381999999999997e-05 2024-03-30T13:56:43.411 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.411 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.411 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:43.411 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.779917+0000", 2024-03-30T13:56:43.411 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6728999999999999e-05 2024-03-30T13:56:43.411 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.411 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.411 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.411 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.412 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.412 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:31292 8.5 8:afab0343:::benchmark_data_smithi043_119832_object31291:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-30T13:56:43.412 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:49:27.608619+0000", 2024-03-30T13:56:43.412 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 435.79012869600001, 2024-03-30T13:56:43.412 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.171446764, 2024-03-30T13:56:43.412 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:43.412 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:43.412 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:43.412 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:43.412 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:43.412 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:43.413 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 31292 2024-03-30T13:56:43.413 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.413 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:43.413 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.413 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:43.413 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.608619+0000", 2024-03-30T13:56:43.413 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.413 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.413 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.413 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:43.413 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.608619+0000", 2024-03-30T13:56:43.413 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.413 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.414 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.414 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:43.414 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.608621+0000", 2024-03-30T13:56:43.414 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.544e-06 2024-03-30T13:56:43.414 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.414 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.414 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:43.414 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.608675+0000", 2024-03-30T13:56:43.414 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4211e-05 2024-03-30T13:56:43.414 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.414 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.414 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:43.414 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.608694+0000", 2024-03-30T13:56:43.414 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.8989999999999999e-05 2024-03-30T13:56:43.415 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.415 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.415 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:43.415 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.608703+0000", 2024-03-30T13:56:43.415 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.2050000000000008e-06 2024-03-30T13:56:43.415 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.415 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.415 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.415 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.608772+0000", 2024-03-30T13:56:43.415 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8232000000000007e-05 2024-03-30T13:56:43.415 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.415 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.415 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:43.415 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.608890+0000", 2024-03-30T13:56:43.416 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000118066 2024-03-30T13:56:43.416 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.416 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.416 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 4,7", 2024-03-30T13:56:43.416 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.609032+0000", 2024-03-30T13:56:43.416 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000142263 2024-03-30T13:56:43.416 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.416 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.416 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.416 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.610546+0000", 2024-03-30T13:56:43.416 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0015144640000000001 2024-03-30T13:56:43.416 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.416 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.416 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:43.417 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.610564+0000", 2024-03-30T13:56:43.417 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.7280999999999999e-05 2024-03-30T13:56:43.417 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.417 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.417 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.417 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.779971+0000", 2024-03-30T13:56:43.417 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.169407591 2024-03-30T13:56:43.417 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.417 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.417 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:43.417 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.780024+0000", 2024-03-30T13:56:43.418 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2621999999999999e-05 2024-03-30T13:56:43.418 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.418 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.418 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:43.418 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.780066+0000", 2024-03-30T13:56:43.418 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.2295000000000001e-05 2024-03-30T13:56:43.418 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.418 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.418 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.419 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.419 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.419 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:103274 8.5 8:aece27e6:::benchmark_data_smithi043_119832_object103273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-03-30T13:56:43.419 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:20.661642+0000", 2024-03-30T13:56:43.419 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 382.73710639500001, 2024-03-30T13:56:43.419 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.166393232, 2024-03-30T13:56:43.419 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:43.419 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:43.419 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:43.420 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:43.420 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:43.420 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:43.420 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 103274 2024-03-30T13:56:43.420 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.420 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:43.420 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.420 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:43.421 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.661642+0000", 2024-03-30T13:56:43.421 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.421 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.421 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.421 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:43.421 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.661642+0000", 2024-03-30T13:56:43.421 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.421 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.422 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.422 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:43.422 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.661643+0000", 2024-03-30T13:56:43.422 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5710000000000001e-06 2024-03-30T13:56:43.422 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.422 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.422 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:43.422 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.661691+0000", 2024-03-30T13:56:43.423 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7673000000000002e-05 2024-03-30T13:56:43.423 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.423 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.423 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:43.423 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.661726+0000", 2024-03-30T13:56:43.423 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.4582999999999997e-05 2024-03-30T13:56:43.423 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.423 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.423 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:43.424 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.661747+0000", 2024-03-30T13:56:43.424 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.158e-05 2024-03-30T13:56:43.424 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.424 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.424 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.424 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.693111+0000", 2024-03-30T13:56:43.424 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.031364013000000003 2024-03-30T13:56:43.424 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.425 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.425 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:43.425 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.693255+0000", 2024-03-30T13:56:43.425 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000144151 2024-03-30T13:56:43.425 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.425 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.425 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 4,7", 2024-03-30T13:56:43.425 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.695786+0000", 2024-03-30T13:56:43.425 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0025310889999999998 2024-03-30T13:56:43.426 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.426 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.426 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:43.426 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.706470+0000", 2024-03-30T13:56:43.426 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.010683727000000001 2024-03-30T13:56:43.426 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.426 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.426 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.427 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.720609+0000", 2024-03-30T13:56:43.427 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.014139073 2024-03-30T13:56:43.427 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.427 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.427 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.427 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.827914+0000", 2024-03-30T13:56:43.427 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.10730523 2024-03-30T13:56:43.427 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.427 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.428 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:43.428 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.827973+0000", 2024-03-30T13:56:43.428 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8773000000000001e-05 2024-03-30T13:56:43.428 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.428 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.428 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:43.428 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.828035+0000", 2024-03-30T13:56:43.428 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1768999999999998e-05 2024-03-30T13:56:43.429 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.429 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.429 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.429 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.429 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.429 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:103554 8.5 8:a257ada5:::benchmark_data_smithi043_119832_object103553:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-03-30T13:56:43.429 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:20.970350+0000", 2024-03-30T13:56:43.429 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 382.42839790300002, 2024-03-30T13:56:43.430 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.162027909, 2024-03-30T13:56:43.430 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:43.430 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:43.430 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:43.430 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:43.430 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:43.430 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:43.430 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 103554 2024-03-30T13:56:43.431 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.431 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:43.431 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.431 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:43.431 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.970350+0000", 2024-03-30T13:56:43.431 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.431 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.431 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.432 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:43.432 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.970350+0000", 2024-03-30T13:56:43.432 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.432 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.432 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.432 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:43.434 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.970351+0000", 2024-03-30T13:56:43.434 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2309999999999999e-06 2024-03-30T13:56:43.434 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.434 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.434 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:43.434 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.970380+0000", 2024-03-30T13:56:43.434 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.8237000000000001e-05 2024-03-30T13:56:43.434 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.434 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.434 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:43.434 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.970408+0000", 2024-03-30T13:56:43.435 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.8656000000000001e-05 2024-03-30T13:56:43.435 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.435 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.435 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:43.435 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.970420+0000", 2024-03-30T13:56:43.435 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1620000000000001e-05 2024-03-30T13:56:43.435 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.435 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.435 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.435 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.000822+0000", 2024-03-30T13:56:43.435 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.030402080000000001 2024-03-30T13:56:43.435 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.435 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.436 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:43.436 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.000961+0000", 2024-03-30T13:56:43.436 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013883499999999999 2024-03-30T13:56:43.436 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.436 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.436 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 4,7", 2024-03-30T13:56:43.436 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.001124+0000", 2024-03-30T13:56:43.436 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00016319899999999999 2024-03-30T13:56:43.436 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.436 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.436 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.436 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.032639+0000", 2024-03-30T13:56:43.437 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.031515133000000001 2024-03-30T13:56:43.437 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.437 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.437 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:43.437 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.032783+0000", 2024-03-30T13:56:43.437 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00014418800000000001 2024-03-30T13:56:43.437 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.437 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.437 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.437 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.132239+0000", 2024-03-30T13:56:43.437 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.099456000000000003 2024-03-30T13:56:43.437 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.437 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.437 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:43.437 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.132308+0000", 2024-03-30T13:56:43.438 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8670999999999994e-05 2024-03-30T13:56:43.438 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.438 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.438 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:43.438 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.132378+0000", 2024-03-30T13:56:43.438 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.0059000000000001e-05 2024-03-30T13:56:43.438 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.438 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.438 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.438 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.438 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.438 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:103636 8.5 8:a71576a6:::benchmark_data_smithi043_119832_object103635:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-03-30T13:56:43.439 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:21.071814+0000", 2024-03-30T13:56:43.439 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 382.32693459299998, 2024-03-30T13:56:43.439 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.19432798900000001, 2024-03-30T13:56:43.439 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:43.439 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:43.439 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:43.439 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:43.439 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:43.439 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:43.439 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 103636 2024-03-30T13:56:43.439 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.439 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:43.439 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.440 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:43.440 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.071814+0000", 2024-03-30T13:56:43.440 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.440 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.440 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.440 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:43.440 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.071814+0000", 2024-03-30T13:56:43.440 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.440 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.440 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.440 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:43.440 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.071815+0000", 2024-03-30T13:56:43.440 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3209999999999999e-06 2024-03-30T13:56:43.440 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.440 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.441 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:43.441 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.071849+0000", 2024-03-30T13:56:43.441 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.4428999999999999e-05 2024-03-30T13:56:43.441 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.441 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.441 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:43.441 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.071860+0000", 2024-03-30T13:56:43.441 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.049e-05 2024-03-30T13:56:43.441 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.441 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.441 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:43.441 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.071867+0000", 2024-03-30T13:56:43.441 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8140000000000003e-06 2024-03-30T13:56:43.442 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.442 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.442 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.442 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.145045+0000", 2024-03-30T13:56:43.442 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.073178178999999996 2024-03-30T13:56:43.442 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.442 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.442 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:43.442 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.145182+0000", 2024-03-30T13:56:43.442 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013687500000000001 2024-03-30T13:56:43.442 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.442 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.443 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 4,7", 2024-03-30T13:56:43.443 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.145356+0000", 2024-03-30T13:56:43.443 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000174587 2024-03-30T13:56:43.443 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.443 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.443 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:43.443 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.157621+0000", 2024-03-30T13:56:43.443 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.012264287 2024-03-30T13:56:43.443 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.443 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.443 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.443 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.192220+0000", 2024-03-30T13:56:43.443 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.034599824000000001 2024-03-30T13:56:43.443 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.444 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.444 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.444 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.266058+0000", 2024-03-30T13:56:43.444 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.073837557999999998 2024-03-30T13:56:43.444 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.444 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.444 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:43.444 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.266096+0000", 2024-03-30T13:56:43.444 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.7836999999999997e-05 2024-03-30T13:56:43.444 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.444 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.444 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:43.444 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.266142+0000", 2024-03-30T13:56:43.444 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.5788000000000003e-05 2024-03-30T13:56:43.445 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.445 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.445 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.445 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.445 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.445 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:106297 8.5 8:a6e04b40:::benchmark_data_smithi043_119832_object106296:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-30T13:56:43.445 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:23.807369+0000", 2024-03-30T13:56:43.445 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 379.59137901000003, 2024-03-30T13:56:43.445 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.20054035200000001, 2024-03-30T13:56:43.445 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:43.445 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:43.445 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:43.445 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:43.445 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:43.445 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:43.446 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 106297 2024-03-30T13:56:43.446 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.446 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:43.446 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.446 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:43.446 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.807369+0000", 2024-03-30T13:56:43.446 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.446 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.446 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.446 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:43.446 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.807369+0000", 2024-03-30T13:56:43.446 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.446 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.447 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.447 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:43.447 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.807370+0000", 2024-03-30T13:56:43.447 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.282e-06 2024-03-30T13:56:43.447 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.447 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.447 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:43.447 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.807404+0000", 2024-03-30T13:56:43.447 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.3379e-05 2024-03-30T13:56:43.447 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.447 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.447 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:43.447 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.807418+0000", 2024-03-30T13:56:43.447 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4031e-05 2024-03-30T13:56:43.447 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.448 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.448 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:43.448 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.807426+0000", 2024-03-30T13:56:43.448 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.977e-06 2024-03-30T13:56:43.448 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.448 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.448 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.448 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.849483+0000", 2024-03-30T13:56:43.448 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.042057630999999998 2024-03-30T13:56:43.448 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.448 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.448 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:43.448 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.849625+0000", 2024-03-30T13:56:43.448 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000141302 2024-03-30T13:56:43.449 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.449 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.449 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 4,7", 2024-03-30T13:56:43.450 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.849787+0000", 2024-03-30T13:56:43.450 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000162292 2024-03-30T13:56:43.450 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.450 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.450 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:43.450 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.855696+0000", 2024-03-30T13:56:43.450 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0059085830000000002 2024-03-30T13:56:43.450 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.450 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.450 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.450 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.864383+0000", 2024-03-30T13:56:43.450 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0086872870000000001 2024-03-30T13:56:43.450 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.450 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.451 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.451 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.007775+0000", 2024-03-30T13:56:43.451 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.14339182 2024-03-30T13:56:43.451 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.451 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.451 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:43.451 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.007843+0000", 2024-03-30T13:56:43.451 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8184999999999999e-05 2024-03-30T13:56:43.451 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.451 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.451 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:43.451 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.007909+0000", 2024-03-30T13:56:43.451 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6582999999999996e-05 2024-03-30T13:56:43.451 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.452 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.452 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.452 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.452 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.452 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:106311 8.5 8:a5146b1c:::benchmark_data_smithi043_119832_object106310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-30T13:56:43.452 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:23.820912+0000", 2024-03-30T13:56:43.452 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 379.577836451, 2024-03-30T13:56:43.452 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.18714597299999999, 2024-03-30T13:56:43.452 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:43.452 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:43.453 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:43.453 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:43.453 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:43.453 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:43.453 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 106311 2024-03-30T13:56:43.453 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.453 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:43.453 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.453 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:43.453 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.820912+0000", 2024-03-30T13:56:43.453 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.453 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.453 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.454 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:43.454 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.820912+0000", 2024-03-30T13:56:43.454 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.454 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.454 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.454 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:43.454 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.820913+0000", 2024-03-30T13:56:43.454 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.731e-06 2024-03-30T13:56:43.454 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.454 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.454 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:43.454 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.820942+0000", 2024-03-30T13:56:43.455 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.8503000000000001e-05 2024-03-30T13:56:43.455 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.455 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.455 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:43.455 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.820951+0000", 2024-03-30T13:56:43.455 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.9590000000000001e-06 2024-03-30T13:56:43.455 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.455 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.455 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:43.455 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.820958+0000", 2024-03-30T13:56:43.455 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.2429999999999999e-06 2024-03-30T13:56:43.455 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.455 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.455 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.455 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.852772+0000", 2024-03-30T13:56:43.456 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.031813461000000001 2024-03-30T13:56:43.456 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.456 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.456 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:43.456 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.852904+0000", 2024-03-30T13:56:43.456 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013242899999999999 2024-03-30T13:56:43.456 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.456 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.456 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 4,7", 2024-03-30T13:56:43.456 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.853073+0000", 2024-03-30T13:56:43.456 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00016875399999999999 2024-03-30T13:56:43.456 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.456 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.456 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:43.457 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.856285+0000", 2024-03-30T13:56:43.457 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.003212258 2024-03-30T13:56:43.457 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.457 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.457 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.457 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.864431+0000", 2024-03-30T13:56:43.457 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0081460860000000003 2024-03-30T13:56:43.457 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.457 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.457 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.457 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.007953+0000", 2024-03-30T13:56:43.457 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.14352188699999999 2024-03-30T13:56:43.458 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.458 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.458 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:43.458 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.008008+0000", 2024-03-30T13:56:43.458 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5322000000000003e-05 2024-03-30T13:56:43.458 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.458 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.458 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:43.458 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.008058+0000", 2024-03-30T13:56:43.458 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9339999999999999e-05 2024-03-30T13:56:43.458 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.458 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.459 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.459 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.459 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.459 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:106325 8.5 8:a3263736:::benchmark_data_smithi043_119832_object106324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-30T13:56:43.459 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:23.832037+0000", 2024-03-30T13:56:43.459 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 379.56671105999999, 2024-03-30T13:56:43.460 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.176300234, 2024-03-30T13:56:43.460 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:43.460 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:43.460 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:43.460 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:43.460 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:43.460 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:43.461 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 106325 2024-03-30T13:56:43.461 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.461 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:43.461 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.461 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:43.461 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.832037+0000", 2024-03-30T13:56:43.461 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.461 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.461 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.461 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:43.461 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.832037+0000", 2024-03-30T13:56:43.461 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.461 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.462 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.462 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:43.462 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.832039+0000", 2024-03-30T13:56:43.462 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4640000000000001e-06 2024-03-30T13:56:43.462 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.462 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.462 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:43.462 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.832092+0000", 2024-03-30T13:56:43.462 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3749999999999999e-05 2024-03-30T13:56:43.462 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.462 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.462 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:43.462 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.832119+0000", 2024-03-30T13:56:43.463 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.6639999999999999e-05 2024-03-30T13:56:43.463 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.463 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.463 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:43.463 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.832128+0000", 2024-03-30T13:56:43.463 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.7369999999999995e-06 2024-03-30T13:56:43.463 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.463 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.463 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.463 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.856537+0000", 2024-03-30T13:56:43.463 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.024409836000000001 2024-03-30T13:56:43.463 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.463 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.463 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:43.463 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.856675+0000", 2024-03-30T13:56:43.464 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000137286 2024-03-30T13:56:43.464 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.464 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.464 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 4,7", 2024-03-30T13:56:43.464 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.856831+0000", 2024-03-30T13:56:43.464 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000155847 2024-03-30T13:56:43.464 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.464 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.464 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:43.465 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.864443+0000", 2024-03-30T13:56:43.465 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0076122230000000004 2024-03-30T13:56:43.465 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.465 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.465 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.465 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.865999+0000", 2024-03-30T13:56:43.465 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0015556719999999999 2024-03-30T13:56:43.465 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.465 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.465 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.465 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.008234+0000", 2024-03-30T13:56:43.465 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.14223533599999999 2024-03-30T13:56:43.465 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.465 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.465 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:43.466 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.008287+0000", 2024-03-30T13:56:43.466 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3236000000000003e-05 2024-03-30T13:56:43.466 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.466 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.466 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:43.466 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.008337+0000", 2024-03-30T13:56:43.466 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0207000000000002e-05 2024-03-30T13:56:43.466 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.466 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.466 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.466 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.466 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.466 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:106331 8.5 8:a0900ba3:::benchmark_data_smithi043_119832_object106330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-30T13:56:43.467 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:23.834241+0000", 2024-03-30T13:56:43.467 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 379.56450688500001, 2024-03-30T13:56:43.467 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.17422878999999999, 2024-03-30T13:56:43.467 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:43.467 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:43.467 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:43.467 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:43.468 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:43.468 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:43.468 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 106331 2024-03-30T13:56:43.468 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.468 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:43.468 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.468 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:43.468 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.834241+0000", 2024-03-30T13:56:43.468 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.468 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.468 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.468 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:43.468 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.834241+0000", 2024-03-30T13:56:43.469 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.469 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.469 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.469 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:43.469 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.834243+0000", 2024-03-30T13:56:43.469 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.418e-06 2024-03-30T13:56:43.469 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.469 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.469 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:43.469 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.834300+0000", 2024-03-30T13:56:43.469 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7487e-05 2024-03-30T13:56:43.469 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.469 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.469 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:43.470 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.834312+0000", 2024-03-30T13:56:43.470 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1722999999999999e-05 2024-03-30T13:56:43.470 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.470 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.470 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:43.470 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.834319+0000", 2024-03-30T13:56:43.470 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.4279999999999999e-06 2024-03-30T13:56:43.470 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.470 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.470 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.471 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.860430+0000", 2024-03-30T13:56:43.471 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.026110209999999998 2024-03-30T13:56:43.471 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.471 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.471 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:43.471 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.860545+0000", 2024-03-30T13:56:43.471 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011513799999999999 2024-03-30T13:56:43.471 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.471 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.471 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 4,7", 2024-03-30T13:56:43.471 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.860694+0000", 2024-03-30T13:56:43.471 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00014947300000000001 2024-03-30T13:56:43.471 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.471 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.472 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:43.472 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.866011+0000", 2024-03-30T13:56:43.472 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.005317291 2024-03-30T13:56:43.472 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.472 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.472 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.472 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.879751+0000", 2024-03-30T13:56:43.472 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.013739711999999999 2024-03-30T13:56:43.472 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.472 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.472 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.472 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.008377+0000", 2024-03-30T13:56:43.472 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.12862538700000001 2024-03-30T13:56:43.472 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.472 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.473 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:43.473 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.008427+0000", 2024-03-30T13:56:43.473 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0152999999999999e-05 2024-03-30T13:56:43.473 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.473 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.473 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:43.473 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.008470+0000", 2024-03-30T13:56:43.473 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.337e-05 2024-03-30T13:56:43.473 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.473 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.473 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.473 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.474 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.474 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:250162 8.9 8:9755cc2d:::benchmark_data_smithi043_119832_object119808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:56:43.474 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:19.661245+0000", 2024-03-30T13:56:43.474 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 263.73750327800002, 2024-03-30T13:56:43.474 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.18734693499999999, 2024-03-30T13:56:43.474 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:43.474 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:43.474 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:43.475 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:43.475 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:43.475 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:43.475 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 250162 2024-03-30T13:56:43.475 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.475 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:43.475 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.475 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:43.475 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.661245+0000", 2024-03-30T13:56:43.475 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.475 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.475 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.475 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:43.475 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.661245+0000", 2024-03-30T13:56:43.476 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.476 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.476 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.476 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:43.476 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.661245+0000", 2024-03-30T13:56:43.476 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5599999999999995e-07 2024-03-30T13:56:43.476 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.476 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.476 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:43.476 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.661249+0000", 2024-03-30T13:56:43.476 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.382e-06 2024-03-30T13:56:43.476 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.476 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.476 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:43.477 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.661259+0000", 2024-03-30T13:56:43.477 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0178e-05 2024-03-30T13:56:43.477 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.477 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.477 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:43.477 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.661265+0000", 2024-03-30T13:56:43.477 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0190000000000003e-06 2024-03-30T13:56:43.477 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.477 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.477 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.477 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.668796+0000", 2024-03-30T13:56:43.477 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0075311880000000003 2024-03-30T13:56:43.477 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.477 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.477 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:43.478 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.669080+0000", 2024-03-30T13:56:43.478 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00028389600000000001 2024-03-30T13:56:43.478 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.478 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.478 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,5", 2024-03-30T13:56:43.478 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.669467+0000", 2024-03-30T13:56:43.478 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00038705100000000002 2024-03-30T13:56:43.478 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.478 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.478 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:43.478 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.676682+0000", 2024-03-30T13:56:43.478 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0072146010000000002 2024-03-30T13:56:43.478 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.479 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.479 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.479 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.683310+0000", 2024-03-30T13:56:43.479 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0066280150000000001 2024-03-30T13:56:43.479 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.479 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.479 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.479 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.848485+0000", 2024-03-30T13:56:43.479 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.16517526699999999 2024-03-30T13:56:43.479 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.479 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.479 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:43.479 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.848544+0000", 2024-03-30T13:56:43.480 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9209000000000002e-05 2024-03-30T13:56:43.480 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.480 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.480 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:43.480 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.848592+0000", 2024-03-30T13:56:43.480 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7573e-05 2024-03-30T13:56:43.480 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.480 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.481 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.481 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.481 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.481 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:250165 8.9 8:97c487c5:::benchmark_data_smithi043_119832_object119811:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:56:43.481 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:19.665737+0000", 2024-03-30T13:56:43.481 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 263.73301157899999, 2024-03-30T13:56:43.482 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.186351143, 2024-03-30T13:56:43.482 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:43.482 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:43.482 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:43.482 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:43.482 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:43.482 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:43.482 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 250165 2024-03-30T13:56:43.482 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.482 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:43.483 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.483 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:43.483 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.665737+0000", 2024-03-30T13:56:43.483 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.483 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.483 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.483 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:43.483 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.665737+0000", 2024-03-30T13:56:43.483 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.483 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.483 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.483 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:43.483 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.665738+0000", 2024-03-30T13:56:43.483 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.153e-06 2024-03-30T13:56:43.484 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.484 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.484 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:43.484 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.665742+0000", 2024-03-30T13:56:43.484 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.4710000000000001e-06 2024-03-30T13:56:43.484 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.484 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.484 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:43.484 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.665755+0000", 2024-03-30T13:56:43.484 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.306e-05 2024-03-30T13:56:43.484 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.484 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.484 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:43.484 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.665762+0000", 2024-03-30T13:56:43.485 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.2110000000000001e-06 2024-03-30T13:56:43.485 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.485 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.485 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.485 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.673253+0000", 2024-03-30T13:56:43.485 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0074901059999999998 2024-03-30T13:56:43.485 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.485 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.485 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:43.486 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.673389+0000", 2024-03-30T13:56:43.486 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013632 2024-03-30T13:56:43.486 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.486 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.486 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,5", 2024-03-30T13:56:43.486 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.673545+0000", 2024-03-30T13:56:43.486 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000156229 2024-03-30T13:56:43.486 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.486 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.486 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:43.486 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.677858+0000", 2024-03-30T13:56:43.486 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0043133470000000004 2024-03-30T13:56:43.487 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.487 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.487 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.487 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.683392+0000", 2024-03-30T13:56:43.487 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0055340340000000002 2024-03-30T13:56:43.487 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.487 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.487 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.487 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.851943+0000", 2024-03-30T13:56:43.487 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.16855024900000001 2024-03-30T13:56:43.487 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.487 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.487 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:43.487 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.852026+0000", 2024-03-30T13:56:43.488 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.3486000000000007e-05 2024-03-30T13:56:43.488 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.488 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.488 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:43.488 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.852088+0000", 2024-03-30T13:56:43.488 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1476999999999994e-05 2024-03-30T13:56:43.488 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.488 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.488 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.488 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.488 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.489 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:250174 8.9 8:93ced427:::benchmark_data_smithi043_119832_object119820:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:56:43.489 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:19.672736+0000", 2024-03-30T13:56:43.489 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 263.72601227400003, 2024-03-30T13:56:43.489 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.17950729900000001, 2024-03-30T13:56:43.489 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:43.489 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:43.489 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:43.489 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:43.489 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:43.489 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:43.489 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 250174 2024-03-30T13:56:43.489 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.489 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:43.489 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.489 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:43.490 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.672736+0000", 2024-03-30T13:56:43.490 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.490 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.490 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.490 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:43.490 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.672736+0000", 2024-03-30T13:56:43.490 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.490 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.490 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.491 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:43.491 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.672737+0000", 2024-03-30T13:56:43.491 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.469e-06 2024-03-30T13:56:43.491 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.491 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.491 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:43.492 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.672742+0000", 2024-03-30T13:56:43.492 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.848e-06 2024-03-30T13:56:43.492 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.492 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.492 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:43.492 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.672757+0000", 2024-03-30T13:56:43.493 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5129e-05 2024-03-30T13:56:43.493 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.493 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.493 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:43.493 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.672765+0000", 2024-03-30T13:56:43.493 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8909999999999995e-06 2024-03-30T13:56:43.494 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.494 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.494 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.494 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.677945+0000", 2024-03-30T13:56:43.494 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0051795360000000002 2024-03-30T13:56:43.494 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.495 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.495 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:43.495 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.678414+0000", 2024-03-30T13:56:43.495 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00046906500000000001 2024-03-30T13:56:43.495 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.495 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.496 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,5", 2024-03-30T13:56:43.496 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.678982+0000", 2024-03-30T13:56:43.496 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00056849199999999998 2024-03-30T13:56:43.496 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.496 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.496 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:43.497 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.688439+0000", 2024-03-30T13:56:43.497 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0094570650000000006 2024-03-30T13:56:43.497 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.497 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.497 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.497 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.689525+0000", 2024-03-30T13:56:43.498 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001086072 2024-03-30T13:56:43.498 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.498 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.498 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.498 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.852137+0000", 2024-03-30T13:56:43.498 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.16261124199999999 2024-03-30T13:56:43.499 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.499 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.499 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:43.499 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.852194+0000", 2024-03-30T13:56:43.499 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.728e-05 2024-03-30T13:56:43.500 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.500 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.500 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:43.500 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.852243+0000", 2024-03-30T13:56:43.500 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9209999999999998e-05 2024-03-30T13:56:43.500 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.500 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.500 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.500 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.501 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.501 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:250180 8.9 8:9d13047d:::benchmark_data_smithi043_119832_object119826:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:56:43.501 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:19.680512+0000", 2024-03-30T13:56:43.501 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 263.71823654899998, 2024-03-30T13:56:43.501 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.172133181, 2024-03-30T13:56:43.502 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:43.502 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:43.502 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:56:43.502 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:43.502 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:56:43.502 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:56:43.503 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 250180 2024-03-30T13:56:43.503 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.503 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:43.503 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.503 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:43.503 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.680512+0000", 2024-03-30T13:56:43.504 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.504 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.504 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.504 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:43.504 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.680512+0000", 2024-03-30T13:56:43.504 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.505 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.505 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.505 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:43.505 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.680513+0000", 2024-03-30T13:56:43.505 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.209e-06 2024-03-30T13:56:43.505 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.506 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.506 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:43.506 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.680517+0000", 2024-03-30T13:56:43.506 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1069999999999998e-06 2024-03-30T13:56:43.506 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.506 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.507 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:43.507 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.680531+0000", 2024-03-30T13:56:43.507 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4219999999999999e-05 2024-03-30T13:56:43.507 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.507 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.507 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:43.508 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.680539+0000", 2024-03-30T13:56:43.508 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.9300000000000003e-06 2024-03-30T13:56:43.508 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.508 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.508 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.508 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.683421+0000", 2024-03-30T13:56:43.509 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0028816990000000002 2024-03-30T13:56:43.509 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.509 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.509 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:43.509 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.683549+0000", 2024-03-30T13:56:43.510 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000128169 2024-03-30T13:56:43.510 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.510 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.510 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,5", 2024-03-30T13:56:43.510 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.683669+0000", 2024-03-30T13:56:43.510 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012008800000000001 2024-03-30T13:56:43.511 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.511 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.511 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:56:43.511 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.688764+0000", 2024-03-30T13:56:43.511 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0050948030000000002 2024-03-30T13:56:43.512 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.512 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.512 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.512 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.689664+0000", 2024-03-30T13:56:43.512 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00090005100000000002 2024-03-30T13:56:43.512 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.513 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.513 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:56:43.513 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.852537+0000", 2024-03-30T13:56:43.513 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.16287290800000001 2024-03-30T13:56:43.513 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.513 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.514 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:56:43.514 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.852595+0000", 2024-03-30T13:56:43.514 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7784000000000001e-05 2024-03-30T13:56:43.514 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.514 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.515 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:43.515 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.852645+0000", 2024-03-30T13:56:43.515 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0213000000000002e-05 2024-03-30T13:56:43.515 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.515 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.515 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.516 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.516 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.516 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14451.0:2642 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94764809509632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-03-30T13:56:43.516 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:35.493028+0000", 2024-03-30T13:56:43.516 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 187.905719979, 2024-03-30T13:56:43.517 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.361405609, 2024-03-30T13:56:43.517 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:43.517 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:43.517 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:56:43.517 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:43.518 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14451", 2024-03-30T13:56:43.518 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T13:56:43.518 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 2642 2024-03-30T13:56:43.518 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.518 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:43.518 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.519 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:43.519 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493028+0000", 2024-03-30T13:56:43.519 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.519 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.519 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.519 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:43.520 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493028+0000", 2024-03-30T13:56:43.520 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.520 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.520 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.520 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:43.520 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493029+0000", 2024-03-30T13:56:43.521 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1600000000000001e-07 2024-03-30T13:56:43.521 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.521 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.521 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:43.521 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493033+0000", 2024-03-30T13:56:43.521 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3080000000000001e-06 2024-03-30T13:56:43.522 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.522 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.522 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:43.522 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493040+0000", 2024-03-30T13:56:43.522 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.3139999999999998e-06 2024-03-30T13:56:43.522 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.522 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.522 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:43.523 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493047+0000", 2024-03-30T13:56:43.523 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8889999999999996e-06 2024-03-30T13:56:43.523 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.523 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.523 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.523 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493230+0000", 2024-03-30T13:56:43.523 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00018283500000000001 2024-03-30T13:56:43.523 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.524 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.524 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T13:56:43.524 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493238+0000", 2024-03-30T13:56:43.524 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8120000000000008e-06 2024-03-30T13:56:43.524 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.524 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.524 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.524 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.846292+0000", 2024-03-30T13:56:43.524 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.35305440700000001 2024-03-30T13:56:43.525 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.525 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.525 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T13:56:43.525 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.846304+0000", 2024-03-30T13:56:43.525 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2001000000000001e-05 2024-03-30T13:56:43.525 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.526 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.526 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.526 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.854172+0000", 2024-03-30T13:56:43.526 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.00786771 2024-03-30T13:56:43.526 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.526 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.526 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:43.526 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.854348+0000", 2024-03-30T13:56:43.526 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00017568299999999999 2024-03-30T13:56:43.526 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.526 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.527 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:43.527 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.854434+0000", 2024-03-30T13:56:43.527 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5933999999999994e-05 2024-03-30T13:56:43.527 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.527 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.527 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.527 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.527 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.527 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14451.0:2643 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch ping cookie 94764813905280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-03-30T13:56:43.527 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:35.493069+0000", 2024-03-30T13:56:43.527 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 187.90567937099999, 2024-03-30T13:56:43.527 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.37558174100000002, 2024-03-30T13:56:43.527 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:43.528 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:43.528 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:56:43.528 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:43.528 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14451", 2024-03-30T13:56:43.528 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T13:56:43.528 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 2643 2024-03-30T13:56:43.528 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.528 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:43.528 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.528 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:43.528 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493069+0000", 2024-03-30T13:56:43.528 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.528 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.528 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.529 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:43.529 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493069+0000", 2024-03-30T13:56:43.529 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.529 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.529 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.529 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:43.529 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493069+0000", 2024-03-30T13:56:43.529 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.8799999999999998e-07 2024-03-30T13:56:43.529 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.529 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.529 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:43.529 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493072+0000", 2024-03-30T13:56:43.529 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.418e-06 2024-03-30T13:56:43.529 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.530 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.530 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:43.530 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493081+0000", 2024-03-30T13:56:43.530 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4010000000000003e-06 2024-03-30T13:56:43.530 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.530 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.530 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:43.530 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493086+0000", 2024-03-30T13:56:43.530 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1379999999999998e-06 2024-03-30T13:56:43.530 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.530 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.530 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.531 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493171+0000", 2024-03-30T13:56:43.531 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.4764000000000004e-05 2024-03-30T13:56:43.531 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.531 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.531 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T13:56:43.531 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493185+0000", 2024-03-30T13:56:43.531 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3919e-05 2024-03-30T13:56:43.531 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.531 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.531 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.531 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.865616+0000", 2024-03-30T13:56:43.531 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.37243163200000001 2024-03-30T13:56:43.531 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.531 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.532 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for degraded object", 2024-03-30T13:56:43.532 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.866216+0000", 2024-03-30T13:56:43.532 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00060001100000000001 2024-03-30T13:56:43.532 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.532 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.532 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.532 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.868509+0000", 2024-03-30T13:56:43.532 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0022922469999999999 2024-03-30T13:56:43.532 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.532 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.532 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:43.532 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.868564+0000", 2024-03-30T13:56:43.533 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4864999999999997e-05 2024-03-30T13:56:43.533 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.533 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.533 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:43.533 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.868650+0000", 2024-03-30T13:56:43.533 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6958000000000003e-05 2024-03-30T13:56:43.534 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.534 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.534 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.534 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.534 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.534 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14451.0:2647 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94764809509632 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:56:43.534 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:35.493415+0000", 2024-03-30T13:56:43.535 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 187.90533316200001, 2024-03-30T13:56:43.535 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.363426126, 2024-03-30T13:56:43.535 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:43.535 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:43.535 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:56:43.535 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:43.535 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14451", 2024-03-30T13:56:43.535 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T13:56:43.535 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 2647 2024-03-30T13:56:43.535 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.535 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:43.535 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.535 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:43.535 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493415+0000", 2024-03-30T13:56:43.536 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.536 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.536 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.536 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:43.536 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493415+0000", 2024-03-30T13:56:43.536 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.536 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.536 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.536 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:43.536 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493416+0000", 2024-03-30T13:56:43.536 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1399999999999996e-07 2024-03-30T13:56:43.537 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.537 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.537 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:43.537 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493420+0000", 2024-03-30T13:56:43.537 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.9389999999999999e-06 2024-03-30T13:56:43.537 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.537 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.537 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:43.537 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493434+0000", 2024-03-30T13:56:43.537 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4347e-05 2024-03-30T13:56:43.537 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.537 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.537 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:43.537 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493441+0000", 2024-03-30T13:56:43.537 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6120000000000004e-06 2024-03-30T13:56:43.538 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.538 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.538 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.538 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493506+0000", 2024-03-30T13:56:43.538 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5545999999999999e-05 2024-03-30T13:56:43.538 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.538 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.538 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T13:56:43.538 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493516+0000", 2024-03-30T13:56:43.538 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.9320000000000008e-06 2024-03-30T13:56:43.538 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.538 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.538 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.538 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.846331+0000", 2024-03-30T13:56:43.539 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.35281492800000003 2024-03-30T13:56:43.539 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.539 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.539 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T13:56:43.539 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.846339+0000", 2024-03-30T13:56:43.539 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.2160000000000006e-06 2024-03-30T13:56:43.539 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.539 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.539 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.539 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.855384+0000", 2024-03-30T13:56:43.539 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0090448649999999 2024-03-30T13:56:43.539 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.539 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.539 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:43.540 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.855471+0000", 2024-03-30T13:56:43.540 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.7027999999999994e-05 2024-03-30T13:56:43.540 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.540 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.540 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:43.540 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.856841+0000", 2024-03-30T13:56:43.540 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001369999 2024-03-30T13:56:43.540 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.540 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.540 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.540 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.540 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.540 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14451.0:2648 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94764813905280 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:56:43.541 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:35.493457+0000", 2024-03-30T13:56:43.541 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 187.905291533, 2024-03-30T13:56:43.541 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.37619107499999999, 2024-03-30T13:56:43.541 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:43.541 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:43.541 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:56:43.541 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:43.541 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14451", 2024-03-30T13:56:43.541 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T13:56:43.541 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 2648 2024-03-30T13:56:43.541 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.541 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:43.541 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.542 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:43.542 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493457+0000", 2024-03-30T13:56:43.542 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.542 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.542 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.542 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:43.542 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493457+0000", 2024-03-30T13:56:43.542 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.542 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.542 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.542 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:43.542 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493457+0000", 2024-03-30T13:56:43.542 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.7399999999999999e-07 2024-03-30T13:56:43.543 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.543 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.543 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:43.543 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493459+0000", 2024-03-30T13:56:43.543 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.1490000000000001e-06 2024-03-30T13:56:43.543 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.543 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.543 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:43.543 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493517+0000", 2024-03-30T13:56:43.543 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7615999999999999e-05 2024-03-30T13:56:43.543 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.543 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.544 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:43.544 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493524+0000", 2024-03-30T13:56:43.544 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.464e-06 2024-03-30T13:56:43.544 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.544 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.544 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.544 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493564+0000", 2024-03-30T13:56:43.545 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.9317999999999999e-05 2024-03-30T13:56:43.545 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.545 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.545 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T13:56:43.545 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493575+0000", 2024-03-30T13:56:43.545 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1892999999999999e-05 2024-03-30T13:56:43.545 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.545 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.545 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.545 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.866266+0000", 2024-03-30T13:56:43.545 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.37269074600000002 2024-03-30T13:56:43.546 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.546 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.546 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for degraded object", 2024-03-30T13:56:43.546 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.866320+0000", 2024-03-30T13:56:43.546 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3650999999999998e-05 2024-03-30T13:56:43.546 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.546 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.546 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.546 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.868673+0000", 2024-03-30T13:56:43.546 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0023534929999999999 2024-03-30T13:56:43.546 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.546 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.547 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:43.547 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.868717+0000", 2024-03-30T13:56:43.547 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.4190000000000002e-05 2024-03-30T13:56:43.547 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.547 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.547 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:43.547 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.869648+0000", 2024-03-30T13:56:43.547 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00093018099999999998 2024-03-30T13:56:43.547 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.547 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.547 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.547 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.547 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.548 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14451.0:2650 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94764809515392 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:56:43.548 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:35.493539+0000", 2024-03-30T13:56:43.548 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 187.90520889499999, 2024-03-30T13:56:43.548 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.39050606900000001, 2024-03-30T13:56:43.548 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:43.548 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:43.548 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:56:43.548 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:43.548 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14451", 2024-03-30T13:56:43.548 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T13:56:43.549 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 2650 2024-03-30T13:56:43.549 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.549 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:43.549 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.549 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:43.549 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493539+0000", 2024-03-30T13:56:43.549 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.549 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.549 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.549 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:43.549 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493539+0000", 2024-03-30T13:56:43.549 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.549 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.549 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.550 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:43.550 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493540+0000", 2024-03-30T13:56:43.550 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.6000000000000003e-07 2024-03-30T13:56:43.550 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.550 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.550 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:43.550 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493544+0000", 2024-03-30T13:56:43.550 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1439999999999996e-06 2024-03-30T13:56:43.550 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.550 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.550 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:43.550 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493552+0000", 2024-03-30T13:56:43.550 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8359999999999992e-06 2024-03-30T13:56:43.551 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.551 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.551 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:43.551 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493557+0000", 2024-03-30T13:56:43.551 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3299999999999998e-06 2024-03-30T13:56:43.551 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.551 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.551 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.551 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493605+0000", 2024-03-30T13:56:43.551 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7611999999999997e-05 2024-03-30T13:56:43.551 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.552 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.552 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T13:56:43.552 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493613+0000", 2024-03-30T13:56:43.552 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.6229999999999997e-06 2024-03-30T13:56:43.552 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.552 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.552 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.552 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.862721+0000", 2024-03-30T13:56:43.552 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.369108504 2024-03-30T13:56:43.552 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.552 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.553 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for degraded object", 2024-03-30T13:56:43.553 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.863669+0000", 2024-03-30T13:56:43.553 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00094758500000000003 2024-03-30T13:56:43.553 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.553 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.553 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.553 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.876266+0000", 2024-03-30T13:56:43.553 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.012597486 2024-03-30T13:56:43.553 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.553 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.553 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:43.553 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.876360+0000", 2024-03-30T13:56:43.553 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4313000000000003e-05 2024-03-30T13:56:43.554 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.554 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.554 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:43.554 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.884045+0000", 2024-03-30T13:56:43.554 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0076848760000000002 2024-03-30T13:56:43.554 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.554 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.554 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.554 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.554 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.554 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14451.0:2652 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94764809509632 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-03-30T13:56:43.554 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:35.852719+0000", 2024-03-30T13:56:43.554 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 187.54602915300001, 2024-03-30T13:56:43.554 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.004857439, 2024-03-30T13:56:43.555 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:56:43.555 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:56:43.555 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:56:43.555 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:56:43.555 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14451", 2024-03-30T13:56:43.555 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T13:56:43.555 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 2652 2024-03-30T13:56:43.555 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.555 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:56:43.555 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.555 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:56:43.555 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.852719+0000", 2024-03-30T13:56:43.555 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.555 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.555 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.556 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:56:43.556 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.852719+0000", 2024-03-30T13:56:43.556 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:56:43.556 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.556 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.556 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:56:43.556 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.852720+0000", 2024-03-30T13:56:43.556 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2160000000000001e-06 2024-03-30T13:56:43.556 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.556 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.556 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:56:43.556 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.852724+0000", 2024-03-30T13:56:43.556 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.563e-06 2024-03-30T13:56:43.556 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.557 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.557 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:56:43.557 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.852733+0000", 2024-03-30T13:56:43.557 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.0909999999999999e-06 2024-03-30T13:56:43.557 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.557 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.557 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:56:43.557 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.852743+0000", 2024-03-30T13:56:43.557 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0006999999999999e-05 2024-03-30T13:56:43.557 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.557 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.557 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.557 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.853116+0000", 2024-03-30T13:56:43.557 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00037315500000000001 2024-03-30T13:56:43.558 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.558 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.558 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T13:56:43.558 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.853128+0000", 2024-03-30T13:56:43.558 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2007999999999999e-05 2024-03-30T13:56:43.558 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.558 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.558 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:56:43.558 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.855865+0000", 2024-03-30T13:56:43.558 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0027372569999999 2024-03-30T13:56:43.558 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.558 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.558 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:56:43.558 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.855917+0000", 2024-03-30T13:56:43.559 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1595000000000003e-05 2024-03-30T13:56:43.559 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:56:43.559 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:56:43.559 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:56:43.559 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.857576+0000", 2024-03-30T13:56:43.559 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0016595469999999999 2024-03-30T13:56:43.559 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.559 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.559 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.559 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:56:43.559 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:56:43.559 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:56:43.560 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:43.584 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:43.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:43.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:43.686 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:43.709 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:43.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:43.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:43.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:43.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:43.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:43.937 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:43.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:43.961 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:43.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:44.062 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:44.085 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:44.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:44.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:44.187 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:44.211 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:44.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:44.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:44.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:43 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:56:44.251 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:43 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:56:44.251 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:43 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:56:44.268 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-30T13:56:44.313 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:44.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:44.337 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:44.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:44.439 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:44.463 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:44.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:44.508 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:44.565 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:44.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:44.590 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:44.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:44.691 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:44.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:44.716 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:44.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:44.817 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:44.840 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:44.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:44.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:44.942 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:44.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:44.967 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:44.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:45.063 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T13:56:45.068 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:45.089 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:44 smithi043 ceph-mon[29963]: pgmap v604: 158 pgs: 90 active+clean, 14 active+undersized, 7 active+undersized+degraded, 47 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%); 0 B/s, 165 objects/s recovering 2024-03-30T13:56:45.090 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:44 smithi043 ceph-mon[42099]: pgmap v604: 158 pgs: 90 active+clean, 14 active+undersized, 7 active+undersized+degraded, 47 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%); 0 B/s, 165 objects/s recovering 2024-03-30T13:56:45.114 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:45.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:45.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:45.216 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:45.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:44 smithi113 ceph-mon[34341]: pgmap v604: 158 pgs: 90 active+clean, 14 active+undersized, 7 active+undersized+degraded, 47 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%); 0 B/s, 165 objects/s recovering 2024-03-30T13:56:45.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:45.241 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:45.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:45.342 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:45.366 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:45.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:45.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:45.468 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:45.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:45.492 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:45.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:45.593 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:45.617 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:45.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:45.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:45.719 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:45.743 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:45.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:45.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:45.845 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:45.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:45.870 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:45.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:45.971 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:45.996 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:45.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:45.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:46.098 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:46.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:46.123 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:46.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:46.213 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:56:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:56:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:56:46.224 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:46.247 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:46.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:46.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:46.347 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:56:46.347 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "" 2024-03-30T13:56:46.347 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:56:46.349 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:46.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:46.373 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:46.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:46.379 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:56:46.379 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [], 2024-03-30T13:56:46.379 INFO:teuthology.orchestra.run.smithi113.stdout: "num_ops": 0 2024-03-30T13:56:46.380 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:56:46.474 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:46.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:46.500 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:46.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:46.601 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:46.624 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:46.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:46.726 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:46.751 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:46.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:46.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:46.853 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:46.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:46.878 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:46.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:46.951 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'unique_pool_2' removed 2024-03-30T13:56:46.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:46 smithi043 ceph-mon[29963]: pgmap v605: 158 pgs: 90 active+clean, 14 active+undersized, 7 active+undersized+degraded, 47 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 725 B/s rd, 0 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%); 0 B/s, 141 objects/s recovering 2024-03-30T13:56:46.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:46 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1067628777' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T13:56:46.951 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:46 smithi043 ceph-mon[42099]: pgmap v605: 158 pgs: 90 active+clean, 14 active+undersized, 7 active+undersized+degraded, 47 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 725 B/s rd, 0 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%); 0 B/s, 141 objects/s recovering 2024-03-30T13:56:46.951 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:46 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1067628777' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T13:56:46.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:46 smithi113 ceph-mon[34341]: pgmap v605: 158 pgs: 90 active+clean, 14 active+undersized, 7 active+undersized+degraded, 47 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 725 B/s rd, 0 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%); 0 B/s, 141 objects/s recovering 2024-03-30T13:56:46.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:46 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1067628777' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T13:56:46.978 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:47.003 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:47.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:47.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:47.034 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-03-30T13:56:47.082 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:56:47.106 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:47.109 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:56:47.129 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:56:47.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:47.142 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:47.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:47.149 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:56:47.169 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:56:47.188 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:56:47.208 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:56:47.208 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:56:47.209 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:56:47.209 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:56:47.209 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:56:47.209 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:56:47.209 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:56:47.209 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:56:47.209 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:56:47.243 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:47.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:47.267 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:47.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:47.315 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:47.368 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:47.391 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:47.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:47.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:47.493 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:47.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:47.517 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:47.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:47.618 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:47.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:47.642 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:47.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:47.743 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:47.767 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:47.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:47.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:47.868 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:47.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:47.893 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:47.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:47.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:47 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1067628777' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T13:56:47.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:47 smithi113 ceph-mon[34341]: osdmap e285: 8 total, 7 up, 6 in 2024-03-30T13:56:47.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:47 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3307937410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:56:47.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:47 smithi113 ceph-mon[34341]: osdmap e286: 8 total, 7 up, 6 in 2024-03-30T13:56:47.994 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:48.018 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:48.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:48.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:48.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:47 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1067628777' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T13:56:48.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:47 smithi043 ceph-mon[29963]: osdmap e285: 8 total, 7 up, 6 in 2024-03-30T13:56:48.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:47 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3307937410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:56:48.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:47 smithi043 ceph-mon[29963]: osdmap e286: 8 total, 7 up, 6 in 2024-03-30T13:56:48.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:47 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1067628777' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T13:56:48.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:47 smithi043 ceph-mon[42099]: osdmap e285: 8 total, 7 up, 6 in 2024-03-30T13:56:48.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:47 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3307937410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:56:48.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:47 smithi043 ceph-mon[42099]: osdmap e286: 8 total, 7 up, 6 in 2024-03-30T13:56:48.069 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:56:48.069 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:56:48.121 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:48.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:48.146 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:48.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:48.247 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:48.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:48.273 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:48.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:48.374 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:48.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:48.399 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:48.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:48.500 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:48.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:48.524 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:48.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:48.625 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:48.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:48.650 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:48.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:48.746 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-30T13:56:48.747 INFO:tasks.radosbench:Beginning radosbench... 2024-03-30T13:56:48.747 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-03-30T13:56:48.747 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool create unique_pool_3 16 2024-03-30T13:56:48.751 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:48.764 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:56:48.774 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:48.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:48.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:48.876 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:48.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:48.902 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:48.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:49.003 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:49.013 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:48 smithi043 ceph-mon[29963]: pgmap v608: 142 pgs: 80 active+clean, 12 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%) 2024-03-30T13:56:49.014 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:48 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 47/597 objects degraded (7.873%), 7 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:49.014 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:48 smithi043 ceph-mon[42099]: pgmap v608: 142 pgs: 80 active+clean, 12 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%) 2024-03-30T13:56:49.014 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:48 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 47/597 objects degraded (7.873%), 7 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:49.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:49.032 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:49.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:49.133 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:49.152 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:56:49.153 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:56:49.153 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:56:49 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:56:49.153 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:48 smithi113 ceph-mon[34341]: pgmap v608: 142 pgs: 80 active+clean, 12 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%) 2024-03-30T13:56:49.153 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:48 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 47/597 objects degraded (7.873%), 7 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:49.157 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:49.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:49.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:49.259 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:49.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:49.284 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:49.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:49.385 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:49.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:49.410 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:49.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:49.511 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:49.520 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:56:49.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:49.537 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:49.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:49.638 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:49.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:49.664 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:49.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:49.765 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:49.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:49.790 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:49.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:49.841 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-03-30T13:56:49.891 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:49.916 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:49.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:50.017 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:50.042 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:50.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:50.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:50.071 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:56:50.082 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:50.144 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:50.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:50.169 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:50.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:50.270 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:50.294 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:50.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:50.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:50.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:50.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:50.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:50.521 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:50.546 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:50.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:50.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:50.648 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:50.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:50.674 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:50.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:50.775 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:50.800 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:50.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:50.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:50.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:50.928 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:50.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:50.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:51.030 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:51.054 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:51.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:51.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:51.156 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:51.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:51.181 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:51.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:51.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:50 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:56:51.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:50 smithi113 ceph-mon[34341]: pgmap v609: 142 pgs: 80 active+clean, 12 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 579 B/s rd, 0 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%) 2024-03-30T13:56:51.282 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:51.305 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:51.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:51.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:51.381 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:50 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:56:51.381 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:50 smithi043 ceph-mon[29963]: pgmap v609: 142 pgs: 80 active+clean, 12 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 579 B/s rd, 0 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%) 2024-03-30T13:56:51.382 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:50 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:56:51.382 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:50 smithi043 ceph-mon[42099]: pgmap v609: 142 pgs: 80 active+clean, 12 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 579 B/s rd, 0 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%) 2024-03-30T13:56:51.407 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:51.430 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:51.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:51.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:51.532 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:51.555 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:51.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:51.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:51.657 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:51.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:51.681 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:51.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:51.782 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:51.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:51.807 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:51.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:51.820 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:56:51.860 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:56:51.861 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:56:51.915 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:51.930 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T13:56:51.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:51.942 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:51.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:52.043 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:52.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:52.068 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:52.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:52.069 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:51 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3476201289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:56:52.070 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:51 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3476201289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:56:52.169 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:52.193 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:52.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:52.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:52.211 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:51 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3476201289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:56:52.295 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:52.321 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:52.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:52.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:52.423 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:52.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:52.448 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:52.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:52.467 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6] out_osds: [5, 7] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-30T13:56:52.467 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:56:52.467 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-03-30T13:56:52.468 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 reset_purged_snaps_last 2024-03-30T13:56:52.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-30T13:56:52.549 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:52.573 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:52.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:52.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:52.675 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:52.699 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:52.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:52.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:52.801 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:52.830 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:52.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:52.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:52.932 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:52.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:52.958 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:52.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:53.059 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:53.082 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:53.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:53.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:53.184 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:53.209 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:53.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:53.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:53.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:52 smithi113 ceph-mon[34341]: pgmap v610: 142 pgs: 80 active+clean, 12 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%) 2024-03-30T13:56:53.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:52 smithi113 ceph-mon[34341]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:56:53.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:52 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3476201289' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:56:53.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:52 smithi113 ceph-mon[34341]: osdmap e287: 8 total, 7 up, 6 in 2024-03-30T13:56:53.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:52 smithi113 ceph-mon[34341]: osdmap e288: 8 total, 7 up, 6 in 2024-03-30T13:56:53.311 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:53.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:53.335 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:53.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:53.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:52 smithi043 ceph-mon[42099]: pgmap v610: 142 pgs: 80 active+clean, 12 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%) 2024-03-30T13:56:53.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:52 smithi043 ceph-mon[42099]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:56:53.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:52 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3476201289' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:56:53.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:52 smithi043 ceph-mon[42099]: osdmap e287: 8 total, 7 up, 6 in 2024-03-30T13:56:53.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:52 smithi043 ceph-mon[42099]: osdmap e288: 8 total, 7 up, 6 in 2024-03-30T13:56:53.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:52 smithi043 ceph-mon[29963]: pgmap v610: 142 pgs: 80 active+clean, 12 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%) 2024-03-30T13:56:53.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:52 smithi043 ceph-mon[29963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:56:53.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:52 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3476201289' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:56:53.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:52 smithi043 ceph-mon[29963]: osdmap e287: 8 total, 7 up, 6 in 2024-03-30T13:56:53.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:52 smithi043 ceph-mon[29963]: osdmap e288: 8 total, 7 up, 6 in 2024-03-30T13:56:53.437 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:53.461 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:53.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:53.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:53.563 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:53.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:53.588 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:53.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:53.689 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:53.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:53.715 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:53.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:53.816 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:53.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:53.841 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:53.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:53.943 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:53.967 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:53.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:53.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:54.068 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:54.093 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:54.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:54.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:54.103 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:56:54.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:53 smithi043 ceph-mon[29963]: 1.2 scrub starts 2024-03-30T13:56:54.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:53 smithi043 ceph-mon[29963]: 1.2 scrub ok 2024-03-30T13:56:54.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:53 smithi043 ceph-mon[42099]: 1.2 scrub starts 2024-03-30T13:56:54.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:53 smithi043 ceph-mon[42099]: 1.2 scrub ok 2024-03-30T13:56:54.148 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:56:54.194 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:54.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:53 smithi113 ceph-mon[34341]: 1.2 scrub starts 2024-03-30T13:56:54.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:53 smithi113 ceph-mon[34341]: 1.2 scrub ok 2024-03-30T13:56:54.219 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:54.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:54.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:54.320 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:54.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:54.345 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:54.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:54.446 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:54.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:54.471 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:54.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:54.572 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:54.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:54.596 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:54.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:54.697 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:54.721 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:54.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:54.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:54.823 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:54.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:54.847 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:54.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:54.944 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'unique_pool_3' created 2024-03-30T13:56:54.948 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:54.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:54.974 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:54.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:55.075 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:55.098 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:55.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:55.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:55.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:54 smithi043 ceph-mon[42099]: pgmap v613: 142 pgs: 80 active+clean, 12 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 843 B/s rd, 0 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%) 2024-03-30T13:56:55.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:54 smithi043 ceph-mon[42099]: 1.0 scrub starts 2024-03-30T13:56:55.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:54 smithi043 ceph-mon[42099]: 1.0 scrub ok 2024-03-30T13:56:55.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:54 smithi043 ceph-mon[42099]: 6.1c scrub starts 2024-03-30T13:56:55.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:54 smithi043 ceph-mon[42099]: 6.1c scrub ok 2024-03-30T13:56:55.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:54 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3979090787' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-30T13:56:55.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:54 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-30T13:56:55.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:54 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2991976763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:56:55.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:54 smithi043 ceph-mon[29963]: pgmap v613: 142 pgs: 80 active+clean, 12 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 843 B/s rd, 0 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%) 2024-03-30T13:56:55.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:54 smithi043 ceph-mon[29963]: 1.0 scrub starts 2024-03-30T13:56:55.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:54 smithi043 ceph-mon[29963]: 1.0 scrub ok 2024-03-30T13:56:55.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:54 smithi043 ceph-mon[29963]: 6.1c scrub starts 2024-03-30T13:56:55.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:54 smithi043 ceph-mon[29963]: 6.1c scrub ok 2024-03-30T13:56:55.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:54 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3979090787' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-30T13:56:55.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:54 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-30T13:56:55.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:54 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2991976763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:56:55.201 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:55.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:54 smithi113 ceph-mon[34341]: pgmap v613: 142 pgs: 80 active+clean, 12 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 588 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 843 B/s rd, 0 op/s; 47/597 objects degraded (7.873%); 28/597 objects misplaced (4.690%) 2024-03-30T13:56:55.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:54 smithi113 ceph-mon[34341]: 1.0 scrub starts 2024-03-30T13:56:55.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:54 smithi113 ceph-mon[34341]: 1.0 scrub ok 2024-03-30T13:56:55.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:54 smithi113 ceph-mon[34341]: 6.1c scrub starts 2024-03-30T13:56:55.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:54 smithi113 ceph-mon[34341]: 6.1c scrub ok 2024-03-30T13:56:55.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:54 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3979090787' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-30T13:56:55.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:54 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-30T13:56:55.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:54 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2991976763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:56:55.225 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:55.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:55.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:55.327 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:55.350 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:55.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:55.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:55.389 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:56:55.389 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:56:55.390 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:56:55.390 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:56:55.390 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:56:55.390 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:56:55.390 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:56:55.390 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:56:55.390 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:56:55.390 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:56:55.390 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:56:55.390 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:56:55.390 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:56:55.391 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:56:55.391 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:56:55.452 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:55.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:55.477 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:55.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:55.578 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:55.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:55.603 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:55.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:55.704 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:55.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:55.728 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:55.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:55.830 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:55.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:55.854 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:55.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:55.892 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T13:56:55.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:55.979 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:55.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:55.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:56.081 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:56.104 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:56.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:56.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:56.206 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:56.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:55 smithi113 ceph-mon[34341]: 1.1 scrub starts 2024-03-30T13:56:56.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:55 smithi113 ceph-mon[34341]: 1.1 scrub ok 2024-03-30T13:56:56.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:55 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-30T13:56:56.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:55 smithi113 ceph-mon[34341]: 1.8 scrub starts 2024-03-30T13:56:56.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:55 smithi113 ceph-mon[34341]: osdmap e289: 8 total, 7 up, 6 in 2024-03-30T13:56:56.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:55 smithi113 ceph-mon[34341]: 1.8 scrub ok 2024-03-30T13:56:56.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:55 smithi113 ceph-mon[34341]: 3.4 scrub starts 2024-03-30T13:56:56.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:55 smithi113 ceph-mon[34341]: 3.4 scrub ok 2024-03-30T13:56:56.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:55 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:56:56.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:55 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:56:56.230 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:56.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:56.331 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:56.356 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:56.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:56.394 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:56:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:56:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:56:56.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:55 smithi043 ceph-mon[42099]: 1.1 scrub starts 2024-03-30T13:56:56.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:55 smithi043 ceph-mon[42099]: 1.1 scrub ok 2024-03-30T13:56:56.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:55 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-30T13:56:56.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:55 smithi043 ceph-mon[42099]: 1.8 scrub starts 2024-03-30T13:56:56.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:55 smithi043 ceph-mon[42099]: osdmap e289: 8 total, 7 up, 6 in 2024-03-30T13:56:56.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:55 smithi043 ceph-mon[42099]: 1.8 scrub ok 2024-03-30T13:56:56.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:55 smithi043 ceph-mon[42099]: 3.4 scrub starts 2024-03-30T13:56:56.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:55 smithi043 ceph-mon[42099]: 3.4 scrub ok 2024-03-30T13:56:56.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:55 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:56:56.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:55 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:56:56.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:55 smithi043 ceph-mon[29963]: 1.1 scrub starts 2024-03-30T13:56:56.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:55 smithi043 ceph-mon[29963]: 1.1 scrub ok 2024-03-30T13:56:56.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:55 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-30T13:56:56.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:55 smithi043 ceph-mon[29963]: 1.8 scrub starts 2024-03-30T13:56:56.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:55 smithi043 ceph-mon[29963]: osdmap e289: 8 total, 7 up, 6 in 2024-03-30T13:56:56.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:55 smithi043 ceph-mon[29963]: 1.8 scrub ok 2024-03-30T13:56:56.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:55 smithi043 ceph-mon[29963]: 3.4 scrub starts 2024-03-30T13:56:56.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:55 smithi043 ceph-mon[29963]: 3.4 scrub ok 2024-03-30T13:56:56.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:55 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:56:56.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:55 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:56:56.457 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:56.481 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:56.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:56.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:56.583 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:56.596 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-03-30T13:56:56.607 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:56.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:56.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:56.709 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:56.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:56.734 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:56.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:56.835 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:56.859 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:56.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:56.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:56.961 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:56.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:56.986 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:56.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:57.087 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:57.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:57.111 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:57.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:57.213 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:57.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:56 smithi113 ceph-mon[34341]: pgmap v615: 158 pgs: 4 creating+peering, 12 unknown, 80 active+clean, 12 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 431 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 47/591 objects degraded (7.953%); 26/591 objects misplaced (4.399%) 2024-03-30T13:56:57.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:56 smithi113 ceph-mon[34341]: 1.9 scrub starts 2024-03-30T13:56:57.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:56 smithi113 ceph-mon[34341]: 1.9 scrub ok 2024-03-30T13:56:57.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:56 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 47/591 objects degraded (7.953%), 7 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:57.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:56 smithi113 ceph-mon[34341]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:56:57.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:56 smithi113 ceph-mon[34341]: osdmap e290: 8 total, 7 up, 6 in 2024-03-30T13:56:57.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:56 smithi113 ceph-mon[34341]: 5.7 scrub starts 2024-03-30T13:56:57.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:56 smithi113 ceph-mon[34341]: 5.7 scrub ok 2024-03-30T13:56:57.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:57.239 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:57.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:57.341 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:57.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:57.367 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:57.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:57.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:56 smithi043 ceph-mon[42099]: pgmap v615: 158 pgs: 4 creating+peering, 12 unknown, 80 active+clean, 12 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 431 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 47/591 objects degraded (7.953%); 26/591 objects misplaced (4.399%) 2024-03-30T13:56:57.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:56 smithi043 ceph-mon[42099]: 1.9 scrub starts 2024-03-30T13:56:57.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:56 smithi043 ceph-mon[42099]: 1.9 scrub ok 2024-03-30T13:56:57.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:56 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 47/591 objects degraded (7.953%), 7 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:57.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:56 smithi043 ceph-mon[42099]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:56:57.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:56 smithi043 ceph-mon[42099]: osdmap e290: 8 total, 7 up, 6 in 2024-03-30T13:56:57.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:56 smithi043 ceph-mon[42099]: 5.7 scrub starts 2024-03-30T13:56:57.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:56 smithi043 ceph-mon[42099]: 5.7 scrub ok 2024-03-30T13:56:57.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:56 smithi043 ceph-mon[29963]: pgmap v615: 158 pgs: 4 creating+peering, 12 unknown, 80 active+clean, 12 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 431 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 47/591 objects degraded (7.953%); 26/591 objects misplaced (4.399%) 2024-03-30T13:56:57.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:56 smithi043 ceph-mon[29963]: 1.9 scrub starts 2024-03-30T13:56:57.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:56 smithi043 ceph-mon[29963]: 1.9 scrub ok 2024-03-30T13:56:57.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:56 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 47/591 objects degraded (7.953%), 7 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:56:57.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:56 smithi043 ceph-mon[29963]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:56:57.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:56 smithi043 ceph-mon[29963]: osdmap e290: 8 total, 7 up, 6 in 2024-03-30T13:56:57.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:56 smithi043 ceph-mon[29963]: 5.7 scrub starts 2024-03-30T13:56:57.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:56 smithi043 ceph-mon[29963]: 5.7 scrub ok 2024-03-30T13:56:57.468 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:57.492 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:57.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:57.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:57.594 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:57.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:57.620 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:57.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:57.721 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:57.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:57.746 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:57.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:57.847 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:56:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:57.871 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:57.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:56:57.971 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:57.995 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:57.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:57.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:58.097 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:58.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:58.121 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:58.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:58.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:57 smithi113 ceph-mon[34341]: osdmap e291: 8 total, 7 up, 6 in 2024-03-30T13:56:58.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:57 smithi113 ceph-mon[34341]: 6.19 scrub starts 2024-03-30T13:56:58.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:57 smithi113 ceph-mon[34341]: 6.19 scrub ok 2024-03-30T13:56:58.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:57 smithi113 ceph-mon[34341]: osdmap e292: 8 total, 7 up, 6 in 2024-03-30T13:56:58.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:57 smithi113 ceph-mon[34341]: 3.9 scrub starts 2024-03-30T13:56:58.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:57 smithi113 ceph-mon[34341]: 3.9 scrub ok 2024-03-30T13:56:58.222 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:58.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:58.247 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:58.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:58.256 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:57 smithi043 ceph-mon[29963]: osdmap e291: 8 total, 7 up, 6 in 2024-03-30T13:56:58.256 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:57 smithi043 ceph-mon[29963]: 6.19 scrub starts 2024-03-30T13:56:58.256 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:57 smithi043 ceph-mon[29963]: 6.19 scrub ok 2024-03-30T13:56:58.256 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:57 smithi043 ceph-mon[29963]: osdmap e292: 8 total, 7 up, 6 in 2024-03-30T13:56:58.256 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:57 smithi043 ceph-mon[29963]: 3.9 scrub starts 2024-03-30T13:56:58.256 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:57 smithi043 ceph-mon[29963]: 3.9 scrub ok 2024-03-30T13:56:58.256 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:57 smithi043 ceph-mon[42099]: osdmap e291: 8 total, 7 up, 6 in 2024-03-30T13:56:58.257 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:57 smithi043 ceph-mon[42099]: 6.19 scrub starts 2024-03-30T13:56:58.257 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:57 smithi043 ceph-mon[42099]: 6.19 scrub ok 2024-03-30T13:56:58.257 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:57 smithi043 ceph-mon[42099]: osdmap e292: 8 total, 7 up, 6 in 2024-03-30T13:56:58.257 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:57 smithi043 ceph-mon[42099]: 3.9 scrub starts 2024-03-30T13:56:58.257 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:57 smithi043 ceph-mon[42099]: 3.9 scrub ok 2024-03-30T13:56:58.348 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:58.374 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:58.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:58.475 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:58.500 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:58.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:58.539 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:56:58.539 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T13:56:58.539 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 0 2024-03-30T13:56:58.539 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:56:58.600 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:58.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:58.625 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:58.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:58.726 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:58.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:58.751 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:58.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:58.852 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:58.875 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:58.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:58.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:58.977 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:59.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:59.002 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:59.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:59.091 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:58 smithi043 ceph-mon[42099]: 5.11 scrub starts 2024-03-30T13:56:59.092 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:58 smithi043 ceph-mon[42099]: 5.11 scrub ok 2024-03-30T13:56:59.092 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:58 smithi043 ceph-mon[42099]: pgmap v619: 158 pgs: 5 creating+peering, 10 unknown, 81 active+clean, 12 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 635 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 47/594 objects degraded (7.912%); 27/594 objects misplaced (4.545%) 2024-03-30T13:56:59.092 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:58 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:56:59.092 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:58 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:56:59.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:58 smithi043 ceph-mon[29963]: 5.11 scrub starts 2024-03-30T13:56:59.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:58 smithi043 ceph-mon[29963]: 5.11 scrub ok 2024-03-30T13:56:59.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:58 smithi043 ceph-mon[29963]: pgmap v619: 158 pgs: 5 creating+peering, 10 unknown, 81 active+clean, 12 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 635 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 47/594 objects degraded (7.912%); 27/594 objects misplaced (4.545%) 2024-03-30T13:56:59.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:58 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:56:59.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:58 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:56:59.103 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:59.131 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:59.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:59.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:59.217 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:56:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:56:59.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:58 smithi113 ceph-mon[34341]: 5.11 scrub starts 2024-03-30T13:56:59.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:58 smithi113 ceph-mon[34341]: 5.11 scrub ok 2024-03-30T13:56:59.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:58 smithi113 ceph-mon[34341]: pgmap v619: 158 pgs: 5 creating+peering, 10 unknown, 81 active+clean, 12 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 635 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 47/594 objects degraded (7.912%); 27/594 objects misplaced (4.545%) 2024-03-30T13:56:59.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:58 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:56:59.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:58 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:56:59.233 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:56:59.260 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:59.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:59.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:56:59.362 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:56:59.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:59.389 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:59.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:56:59.490 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:56:59.491 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:56:59.495 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:56:59.495 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:56:59.518 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:59.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:59.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:56:59.619 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:56:59.643 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:59.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:59.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:56:59.745 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:56:59.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:59.769 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:56:59.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:56:59.871 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:56:59.899 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:56:59.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:56:59.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:56:59.937 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-03-30T13:57:00.001 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:00.024 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:00.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:00.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:00.126 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:00.155 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:00.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:00.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:00.161 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:00.199 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:59 smithi043 ceph-mon[29963]: 3.1f scrub starts 2024-03-30T13:57:00.199 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:59 smithi043 ceph-mon[29963]: 3.1f scrub ok 2024-03-30T13:57:00.199 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:59 smithi043 ceph-mon[29963]: 4.14 scrub starts 2024-03-30T13:57:00.199 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:59 smithi043 ceph-mon[29963]: 4.14 scrub ok 2024-03-30T13:57:00.200 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:59 smithi043 ceph-mon[29963]: 6.4 scrub starts 2024-03-30T13:57:00.200 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:56:59 smithi043 ceph-mon[29963]: 6.4 scrub ok 2024-03-30T13:57:00.200 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:59 smithi043 ceph-mon[42099]: 3.1f scrub starts 2024-03-30T13:57:00.200 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:59 smithi043 ceph-mon[42099]: 3.1f scrub ok 2024-03-30T13:57:00.200 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:59 smithi043 ceph-mon[42099]: 4.14 scrub starts 2024-03-30T13:57:00.200 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:59 smithi043 ceph-mon[42099]: 4.14 scrub ok 2024-03-30T13:57:00.200 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:59 smithi043 ceph-mon[42099]: 6.4 scrub starts 2024-03-30T13:57:00.200 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:56:59 smithi043 ceph-mon[42099]: 6.4 scrub ok 2024-03-30T13:57:00.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:59 smithi113 ceph-mon[34341]: 3.1f scrub starts 2024-03-30T13:57:00.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:59 smithi113 ceph-mon[34341]: 3.1f scrub ok 2024-03-30T13:57:00.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:59 smithi113 ceph-mon[34341]: 4.14 scrub starts 2024-03-30T13:57:00.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:59 smithi113 ceph-mon[34341]: 4.14 scrub ok 2024-03-30T13:57:00.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:59 smithi113 ceph-mon[34341]: 6.4 scrub starts 2024-03-30T13:57:00.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:56:59 smithi113 ceph-mon[34341]: 6.4 scrub ok 2024-03-30T13:57:00.257 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:00.283 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:00.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:00.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:00.385 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:00.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:00.412 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:00.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:00.513 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:00.540 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:00.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:00.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:00.641 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:00.667 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:00.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:00.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:00.769 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:00.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:00.794 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:00.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:00.896 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:00.924 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:00.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:00.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:01.026 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:01.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:01.052 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:01.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:01.153 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:01.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:01.179 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:01.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:01.211 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:00 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:01.212 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:00 smithi043 ceph-mon[29963]: pgmap v620: 158 pgs: 5 creating+peering, 86 active+clean, 17 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 635 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 47/594 objects degraded (7.912%); 27/594 objects misplaced (4.545%) 2024-03-30T13:57:01.212 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:00 smithi043 ceph-mon[29963]: 6.9 scrub starts 2024-03-30T13:57:01.212 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:00 smithi043 ceph-mon[29963]: 6.9 scrub ok 2024-03-30T13:57:01.212 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:00 smithi043 ceph-mon[29963]: 3.5 scrub starts 2024-03-30T13:57:01.212 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:00 smithi043 ceph-mon[29963]: 3.5 scrub ok 2024-03-30T13:57:01.212 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:00 smithi043 ceph-mon[29963]: 6.1d scrub starts 2024-03-30T13:57:01.212 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:00 smithi043 ceph-mon[29963]: 6.1d scrub ok 2024-03-30T13:57:01.212 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:00 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:01.212 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:00 smithi043 ceph-mon[42099]: pgmap v620: 158 pgs: 5 creating+peering, 86 active+clean, 17 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 635 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 47/594 objects degraded (7.912%); 27/594 objects misplaced (4.545%) 2024-03-30T13:57:01.212 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:00 smithi043 ceph-mon[42099]: 6.9 scrub starts 2024-03-30T13:57:01.213 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:00 smithi043 ceph-mon[42099]: 6.9 scrub ok 2024-03-30T13:57:01.213 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:00 smithi043 ceph-mon[42099]: 3.5 scrub starts 2024-03-30T13:57:01.213 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:00 smithi043 ceph-mon[42099]: 3.5 scrub ok 2024-03-30T13:57:01.213 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:00 smithi043 ceph-mon[42099]: 6.1d scrub starts 2024-03-30T13:57:01.213 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:00 smithi043 ceph-mon[42099]: 6.1d scrub ok 2024-03-30T13:57:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:00 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:00 smithi113 ceph-mon[34341]: pgmap v620: 158 pgs: 5 creating+peering, 86 active+clean, 17 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 635 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 47/594 objects degraded (7.912%); 27/594 objects misplaced (4.545%) 2024-03-30T13:57:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:00 smithi113 ceph-mon[34341]: 6.9 scrub starts 2024-03-30T13:57:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:00 smithi113 ceph-mon[34341]: 6.9 scrub ok 2024-03-30T13:57:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:00 smithi113 ceph-mon[34341]: 3.5 scrub starts 2024-03-30T13:57:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:00 smithi113 ceph-mon[34341]: 3.5 scrub ok 2024-03-30T13:57:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:00 smithi113 ceph-mon[34341]: 6.1d scrub starts 2024-03-30T13:57:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:00 smithi113 ceph-mon[34341]: 6.1d scrub ok 2024-03-30T13:57:01.280 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:01.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:01.304 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:01.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:01.406 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:01.430 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:01.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:01.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:01.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:57:01.531 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:01.563 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:01.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:01.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:01.665 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:01.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:01.691 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:01.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:01.792 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:01.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:01.816 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:01.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:01.917 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:01.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:01.945 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:01.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:02.027 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 reset_purged_snaps_last 2024-03-30T13:57:02.046 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:02.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:02.095 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:02.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:02.197 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:02.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:01 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:57:02.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:01 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:57:02.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:01 smithi113 ceph-mon[34341]: osdmap e293: 8 total, 7 up, 6 in 2024-03-30T13:57:02.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:01 smithi113 ceph-mon[34341]: 5.c scrub starts 2024-03-30T13:57:02.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:01 smithi113 ceph-mon[34341]: 5.c scrub ok 2024-03-30T13:57:02.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:01 smithi113 ceph-mon[34341]: 6.14 scrub starts 2024-03-30T13:57:02.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:01 smithi113 ceph-mon[34341]: 6.14 scrub ok 2024-03-30T13:57:02.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:02.223 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:02.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:02.324 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:02.333 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:01 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:57:02.334 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:01 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:57:02.334 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:01 smithi043 ceph-mon[29963]: osdmap e293: 8 total, 7 up, 6 in 2024-03-30T13:57:02.334 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:01 smithi043 ceph-mon[29963]: 5.c scrub starts 2024-03-30T13:57:02.334 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:01 smithi043 ceph-mon[29963]: 5.c scrub ok 2024-03-30T13:57:02.334 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:01 smithi043 ceph-mon[29963]: 6.14 scrub starts 2024-03-30T13:57:02.334 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:01 smithi043 ceph-mon[29963]: 6.14 scrub ok 2024-03-30T13:57:02.334 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:01 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:57:02.334 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:01 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:57:02.334 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:01 smithi043 ceph-mon[42099]: osdmap e293: 8 total, 7 up, 6 in 2024-03-30T13:57:02.334 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:01 smithi043 ceph-mon[42099]: 5.c scrub starts 2024-03-30T13:57:02.334 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:01 smithi043 ceph-mon[42099]: 5.c scrub ok 2024-03-30T13:57:02.335 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:01 smithi043 ceph-mon[42099]: 6.14 scrub starts 2024-03-30T13:57:02.335 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:01 smithi043 ceph-mon[42099]: 6.14 scrub ok 2024-03-30T13:57:02.347 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:02.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:02.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:02.450 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:02.474 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:02.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:02.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:02.575 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:02.599 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:02.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:02.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:02.701 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:02.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:02.725 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:02.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:02.826 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:02.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:02.851 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:02.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:02.952 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:02.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:02.977 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:02.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:03.078 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:03.079 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:02 smithi043 ceph-mon[29963]: pgmap v622: 158 pgs: 88 active+clean, 20 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 47/594 objects degraded (7.912%); 27/594 objects misplaced (4.545%) 2024-03-30T13:57:03.079 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:02 smithi043 ceph-mon[29963]: 5.14 scrub starts 2024-03-30T13:57:03.079 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:02 smithi043 ceph-mon[29963]: 5.14 scrub ok 2024-03-30T13:57:03.079 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:02 smithi043 ceph-mon[29963]: 6.d scrub starts 2024-03-30T13:57:03.080 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:02 smithi043 ceph-mon[29963]: 6.d scrub ok 2024-03-30T13:57:03.080 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:02 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 47/594 objects degraded (7.912%), 7 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:03.080 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:02 smithi043 ceph-mon[29963]: osdmap e294: 8 total, 7 up, 6 in 2024-03-30T13:57:03.080 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:02 smithi043 ceph-mon[42099]: pgmap v622: 158 pgs: 88 active+clean, 20 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 47/594 objects degraded (7.912%); 27/594 objects misplaced (4.545%) 2024-03-30T13:57:03.080 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:02 smithi043 ceph-mon[42099]: 5.14 scrub starts 2024-03-30T13:57:03.080 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:02 smithi043 ceph-mon[42099]: 5.14 scrub ok 2024-03-30T13:57:03.080 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:02 smithi043 ceph-mon[42099]: 6.d scrub starts 2024-03-30T13:57:03.080 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:02 smithi043 ceph-mon[42099]: 6.d scrub ok 2024-03-30T13:57:03.081 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:02 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 47/594 objects degraded (7.912%), 7 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:03.081 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:02 smithi043 ceph-mon[42099]: osdmap e294: 8 total, 7 up, 6 in 2024-03-30T13:57:03.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:03.105 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:03.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:03.206 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:03.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:02 smithi113 ceph-mon[34341]: pgmap v622: 158 pgs: 88 active+clean, 20 active+undersized, 7 active+undersized+degraded, 43 active+clean+remapped; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 47/594 objects degraded (7.912%); 27/594 objects misplaced (4.545%) 2024-03-30T13:57:03.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:02 smithi113 ceph-mon[34341]: 5.14 scrub starts 2024-03-30T13:57:03.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:02 smithi113 ceph-mon[34341]: 5.14 scrub ok 2024-03-30T13:57:03.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:02 smithi113 ceph-mon[34341]: 6.d scrub starts 2024-03-30T13:57:03.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:02 smithi113 ceph-mon[34341]: 6.d scrub ok 2024-03-30T13:57:03.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:02 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 47/594 objects degraded (7.912%), 7 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:03.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:02 smithi113 ceph-mon[34341]: osdmap e294: 8 total, 7 up, 6 in 2024-03-30T13:57:03.229 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:03.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:03.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:03.331 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:03.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:03.356 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:03.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:03.458 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:03.482 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:03.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:03.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:03.585 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:03.609 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:03.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:03.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:03.653 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:03.674 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:03.677 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:03.711 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:03.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:03.736 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:03.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:03.837 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:03.860 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:03.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:03.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:03.962 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:03.988 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:03.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:03.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:04.090 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:04.113 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:03 smithi043 ceph-mon[29963]: 4.5 scrub starts 2024-03-30T13:57:04.114 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:03 smithi043 ceph-mon[29963]: 4.5 scrub ok 2024-03-30T13:57:04.114 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:03 smithi043 ceph-mon[29963]: 6.12 scrub starts 2024-03-30T13:57:04.114 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:03 smithi043 ceph-mon[29963]: 6.12 scrub ok 2024-03-30T13:57:04.114 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:03 smithi043 ceph-mon[42099]: 4.5 scrub starts 2024-03-30T13:57:04.114 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:03 smithi043 ceph-mon[42099]: 4.5 scrub ok 2024-03-30T13:57:04.114 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:03 smithi043 ceph-mon[42099]: 6.12 scrub starts 2024-03-30T13:57:04.114 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:03 smithi043 ceph-mon[42099]: 6.12 scrub ok 2024-03-30T13:57:04.115 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:04.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:04.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:04.217 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:04.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:03 smithi113 ceph-mon[34341]: 4.5 scrub starts 2024-03-30T13:57:04.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:03 smithi113 ceph-mon[34341]: 4.5 scrub ok 2024-03-30T13:57:04.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:03 smithi113 ceph-mon[34341]: 6.12 scrub starts 2024-03-30T13:57:04.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:03 smithi113 ceph-mon[34341]: 6.12 scrub ok 2024-03-30T13:57:04.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:04.243 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:04.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:04.344 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:04.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:04.369 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:04.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:04.470 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:04.494 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:04.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:04.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:04.596 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:04.619 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:04.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:04.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:04.721 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:04.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:04.745 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:04.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:04.846 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:04.870 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:04.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:04.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:04.972 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:04.976 INFO:teuthology.orchestra.run.smithi043.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-03-30T13:57:04.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:04.997 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:04.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:05.098 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:05.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:05.123 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:05.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:05.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:04 smithi043 ceph-mon[42099]: pgmap v624: 158 pgs: 88 active+clean, 1 active+clean+remapped+scrubbing, 20 active+undersized, 7 active+undersized+degraded, 42 active+clean+remapped; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 47/594 objects degraded (7.912%); 27/594 objects misplaced (4.545%) 2024-03-30T13:57:05.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:04 smithi043 ceph-mon[42099]: 6.8 scrub starts 2024-03-30T13:57:05.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:04 smithi043 ceph-mon[42099]: 6.8 scrub ok 2024-03-30T13:57:05.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:04 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2937761790' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:57:05.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:04 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:57:05.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:04 smithi043 ceph-mon[42099]: 3.13 scrub starts 2024-03-30T13:57:05.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:04 smithi043 ceph-mon[42099]: 3.13 scrub ok 2024-03-30T13:57:05.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:04 smithi043 ceph-mon[29963]: pgmap v624: 158 pgs: 88 active+clean, 1 active+clean+remapped+scrubbing, 20 active+undersized, 7 active+undersized+degraded, 42 active+clean+remapped; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 47/594 objects degraded (7.912%); 27/594 objects misplaced (4.545%) 2024-03-30T13:57:05.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:04 smithi043 ceph-mon[29963]: 6.8 scrub starts 2024-03-30T13:57:05.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:04 smithi043 ceph-mon[29963]: 6.8 scrub ok 2024-03-30T13:57:05.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:04 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2937761790' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:57:05.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:04 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:57:05.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:04 smithi043 ceph-mon[29963]: 3.13 scrub starts 2024-03-30T13:57:05.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:04 smithi043 ceph-mon[29963]: 3.13 scrub ok 2024-03-30T13:57:05.224 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:05.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:05.249 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:05.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:05.350 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:05.374 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:05.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:05.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:05.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:04 smithi113 ceph-mon[34341]: pgmap v624: 158 pgs: 88 active+clean, 1 active+clean+remapped+scrubbing, 20 active+undersized, 7 active+undersized+degraded, 42 active+clean+remapped; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 47/594 objects degraded (7.912%); 27/594 objects misplaced (4.545%) 2024-03-30T13:57:05.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:04 smithi113 ceph-mon[34341]: 6.8 scrub starts 2024-03-30T13:57:05.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:04 smithi113 ceph-mon[34341]: 6.8 scrub ok 2024-03-30T13:57:05.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:04 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2937761790' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:57:05.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:04 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:57:05.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:04 smithi113 ceph-mon[34341]: 3.13 scrub starts 2024-03-30T13:57:05.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:04 smithi113 ceph-mon[34341]: 3.13 scrub ok 2024-03-30T13:57:05.476 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:05.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:05.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:05.602 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:05.626 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:05.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:05.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:05.729 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:05.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:05.753 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:05.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:05.854 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:05.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:05.879 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:05.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:05.980 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:06.004 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:06.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:06.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:06.106 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:06.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:06.131 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:06.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:06.144 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:57:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:57:06.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:05 smithi043 ceph-mon[29963]: 5.1e scrub starts 2024-03-30T13:57:06.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:05 smithi043 ceph-mon[29963]: 5.1e scrub ok 2024-03-30T13:57:06.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:05 smithi043 ceph-mon[29963]: 3.c scrub starts 2024-03-30T13:57:06.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:05 smithi043 ceph-mon[29963]: 3.c scrub ok 2024-03-30T13:57:06.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:05 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-30T13:57:06.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:05 smithi043 ceph-mon[29963]: osdmap e295: 8 total, 7 up, 6 in 2024-03-30T13:57:06.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:05 smithi043 ceph-mon[29963]: 6.15 scrub starts 2024-03-30T13:57:06.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:05 smithi043 ceph-mon[29963]: 5.6 scrub starts 2024-03-30T13:57:06.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:05 smithi043 ceph-mon[29963]: 5.6 scrub ok 2024-03-30T13:57:06.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:05 smithi043 ceph-mon[42099]: 5.1e scrub starts 2024-03-30T13:57:06.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:05 smithi043 ceph-mon[42099]: 5.1e scrub ok 2024-03-30T13:57:06.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:05 smithi043 ceph-mon[42099]: 3.c scrub starts 2024-03-30T13:57:06.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:05 smithi043 ceph-mon[42099]: 3.c scrub ok 2024-03-30T13:57:06.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:05 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-30T13:57:06.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:05 smithi043 ceph-mon[42099]: osdmap e295: 8 total, 7 up, 6 in 2024-03-30T13:57:06.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:05 smithi043 ceph-mon[42099]: 6.15 scrub starts 2024-03-30T13:57:06.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:05 smithi043 ceph-mon[42099]: 5.6 scrub starts 2024-03-30T13:57:06.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:05 smithi043 ceph-mon[42099]: 5.6 scrub ok 2024-03-30T13:57:06.232 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:06.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:06.257 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:06.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:06.358 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:06.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:06.383 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:06.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:06.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:05 smithi113 ceph-mon[34341]: 5.1e scrub starts 2024-03-30T13:57:06.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:05 smithi113 ceph-mon[34341]: 5.1e scrub ok 2024-03-30T13:57:06.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:05 smithi113 ceph-mon[34341]: 3.c scrub starts 2024-03-30T13:57:06.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:05 smithi113 ceph-mon[34341]: 3.c scrub ok 2024-03-30T13:57:06.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:05 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-30T13:57:06.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:05 smithi113 ceph-mon[34341]: osdmap e295: 8 total, 7 up, 6 in 2024-03-30T13:57:06.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:05 smithi113 ceph-mon[34341]: 6.15 scrub starts 2024-03-30T13:57:06.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:05 smithi113 ceph-mon[34341]: 5.6 scrub starts 2024-03-30T13:57:06.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:05 smithi113 ceph-mon[34341]: 5.6 scrub ok 2024-03-30T13:57:06.484 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:06.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:06.509 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:06.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:06.610 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:06.635 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:06.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:06.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:06.737 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:06.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:06.761 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:06.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:06.862 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:06.886 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:06.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:06.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:06.987 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-03-30T13:57:06.987 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:07.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:07.012 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:07.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:07.112 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:07.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:07.136 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:07.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:07.237 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:07.262 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:07.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:07.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:07.364 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:07.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:07.388 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:07.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:07.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-mon[29963]: 6.15 scrub ok 2024-03-30T13:57:07.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-mon[29963]: pgmap v626: 158 pgs: 88 active+clean, 1 active+clean+remapped+scrubbing, 20 active+undersized, 7 active+undersized+degraded, 42 active+clean+remapped; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 47/594 objects degraded (7.912%); 27/594 objects misplaced (4.545%) 2024-03-30T13:57:07.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-mon[29963]: 5.4 scrub starts 2024-03-30T13:57:07.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-mon[29963]: 5.4 scrub ok 2024-03-30T13:57:07.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-mon[29963]: 3.b scrub starts 2024-03-30T13:57:07.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-mon[29963]: 3.b scrub ok 2024-03-30T13:57:07.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-mon[29963]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T13:57:07.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1442778063' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:57:07.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-mon[29963]: 6.17 scrub starts 2024-03-30T13:57:07.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-mon[29963]: 6.17 scrub ok 2024-03-30T13:57:07.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-mon[42099]: 6.15 scrub ok 2024-03-30T13:57:07.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-mon[42099]: pgmap v626: 158 pgs: 88 active+clean, 1 active+clean+remapped+scrubbing, 20 active+undersized, 7 active+undersized+degraded, 42 active+clean+remapped; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 47/594 objects degraded (7.912%); 27/594 objects misplaced (4.545%) 2024-03-30T13:57:07.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-mon[42099]: 5.4 scrub starts 2024-03-30T13:57:07.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-mon[42099]: 5.4 scrub ok 2024-03-30T13:57:07.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-mon[42099]: 3.b scrub starts 2024-03-30T13:57:07.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-mon[42099]: 3.b scrub ok 2024-03-30T13:57:07.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-mon[42099]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T13:57:07.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1442778063' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:57:07.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-mon[42099]: 6.17 scrub starts 2024-03-30T13:57:07.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:06 smithi043 ceph-mon[42099]: 6.17 scrub ok 2024-03-30T13:57:07.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:06 smithi113 ceph-mon[34341]: 6.15 scrub ok 2024-03-30T13:57:07.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:06 smithi113 ceph-mon[34341]: pgmap v626: 158 pgs: 88 active+clean, 1 active+clean+remapped+scrubbing, 20 active+undersized, 7 active+undersized+degraded, 42 active+clean+remapped; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 47/594 objects degraded (7.912%); 27/594 objects misplaced (4.545%) 2024-03-30T13:57:07.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:06 smithi113 ceph-mon[34341]: 5.4 scrub starts 2024-03-30T13:57:07.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:06 smithi113 ceph-mon[34341]: 5.4 scrub ok 2024-03-30T13:57:07.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:06 smithi113 ceph-mon[34341]: 3.b scrub starts 2024-03-30T13:57:07.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:06 smithi113 ceph-mon[34341]: 3.b scrub ok 2024-03-30T13:57:07.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:06 smithi113 ceph-mon[34341]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T13:57:07.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:06 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1442778063' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:57:07.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:06 smithi113 ceph-mon[34341]: 6.17 scrub starts 2024-03-30T13:57:07.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:06 smithi113 ceph-mon[34341]: 6.17 scrub ok 2024-03-30T13:57:07.489 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:07.513 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:07.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:07.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:07.615 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:07.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:07.639 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:07.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:07.740 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:07.767 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:07.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:07.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:07.869 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:07.895 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:07.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:07.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:07.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:08.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:08.022 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:08.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:08.100 DEBUG:teuthology.orchestra.run.smithi043:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-03-30T13:57:08.102 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-30T13:57:08.103 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-30T13:57:08.123 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:08.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:57:08.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:57:08.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[29963]: 5.b scrub starts 2024-03-30T13:57:08.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[29963]: 5.b scrub ok 2024-03-30T13:57:08.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[29963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:57:08.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1442778063' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:57:08.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[29963]: osdmap e296: 8 total, 7 up, 6 in 2024-03-30T13:57:08.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[29963]: 3.16 scrub starts 2024-03-30T13:57:08.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[29963]: 3.16 scrub ok 2024-03-30T13:57:08.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[29963]: 5.1f scrub starts 2024-03-30T13:57:08.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[29963]: 5.1f scrub ok 2024-03-30T13:57:08.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:57:08.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:57:08.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[42099]: 5.b scrub starts 2024-03-30T13:57:08.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[42099]: 5.b scrub ok 2024-03-30T13:57:08.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[42099]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:57:08.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1442778063' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:57:08.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[42099]: osdmap e296: 8 total, 7 up, 6 in 2024-03-30T13:57:08.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[42099]: 3.16 scrub starts 2024-03-30T13:57:08.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[42099]: 3.16 scrub ok 2024-03-30T13:57:08.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[42099]: 5.1f scrub starts 2024-03-30T13:57:08.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:07 smithi043 ceph-mon[42099]: 5.1f scrub ok 2024-03-30T13:57:08.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:08.149 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:08.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:08.250 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:08.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:08.276 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:08.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:08.364 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:hints = 1 2024-03-30T13:57:08.364 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-30T13:57:08.364 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Object prefix: benchmark_data_smithi043_169264 2024-03-30T13:57:08.364 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T13:57:08.364 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 0 0 0 0 0 0 - 0 2024-03-30T13:57:08.377 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:08.422 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:08.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:08.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:08.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:07 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:57:08.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:07 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:57:08.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:07 smithi113 ceph-mon[34341]: 5.b scrub starts 2024-03-30T13:57:08.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:07 smithi113 ceph-mon[34341]: 5.b scrub ok 2024-03-30T13:57:08.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:07 smithi113 ceph-mon[34341]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:57:08.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:07 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1442778063' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:57:08.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:07 smithi113 ceph-mon[34341]: osdmap e296: 8 total, 7 up, 6 in 2024-03-30T13:57:08.469 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:07 smithi113 ceph-mon[34341]: 3.16 scrub starts 2024-03-30T13:57:08.469 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:07 smithi113 ceph-mon[34341]: 3.16 scrub ok 2024-03-30T13:57:08.469 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:07 smithi113 ceph-mon[34341]: 5.1f scrub starts 2024-03-30T13:57:08.469 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:07 smithi113 ceph-mon[34341]: 5.1f scrub ok 2024-03-30T13:57:08.524 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:08.559 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:08.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:08.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:08.661 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:08.696 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:08.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:08.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:08.798 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:08.832 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:08.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:08.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:08.934 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:08.976 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:08.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:08.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:09.078 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:09.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:09.117 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:09.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:09.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:08 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:57:09.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:08 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:57:09.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:08 smithi043 ceph-mon[29963]: pgmap v628: 158 pgs: 88 active+clean, 1 active+clean+remapped+scrubbing, 20 active+undersized, 7 active+undersized+degraded, 42 active+clean+remapped; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 47/594 objects degraded (7.912%); 27/594 objects misplaced (4.545%) 2024-03-30T13:57:09.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:08 smithi043 ceph-mon[29963]: 4.10 scrub starts 2024-03-30T13:57:09.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:08 smithi043 ceph-mon[29963]: 4.10 scrub ok 2024-03-30T13:57:09.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:08 smithi043 ceph-mon[29963]: osdmap e297: 8 total, 7 up, 6 in 2024-03-30T13:57:09.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:08 smithi043 ceph-mon[29963]: 5.1d scrub starts 2024-03-30T13:57:09.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:08 smithi043 ceph-mon[29963]: 5.1d scrub ok 2024-03-30T13:57:09.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:08 smithi043 ceph-mon[29963]: 6.1a scrub starts 2024-03-30T13:57:09.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:08 smithi043 ceph-mon[29963]: 6.1a scrub ok 2024-03-30T13:57:09.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:08 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:57:09.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:08 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:57:09.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:08 smithi043 ceph-mon[42099]: pgmap v628: 158 pgs: 88 active+clean, 1 active+clean+remapped+scrubbing, 20 active+undersized, 7 active+undersized+degraded, 42 active+clean+remapped; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 47/594 objects degraded (7.912%); 27/594 objects misplaced (4.545%) 2024-03-30T13:57:09.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:08 smithi043 ceph-mon[42099]: 4.10 scrub starts 2024-03-30T13:57:09.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:08 smithi043 ceph-mon[42099]: 4.10 scrub ok 2024-03-30T13:57:09.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:08 smithi043 ceph-mon[42099]: osdmap e297: 8 total, 7 up, 6 in 2024-03-30T13:57:09.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:08 smithi043 ceph-mon[42099]: 5.1d scrub starts 2024-03-30T13:57:09.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:08 smithi043 ceph-mon[42099]: 5.1d scrub ok 2024-03-30T13:57:09.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:08 smithi043 ceph-mon[42099]: 6.1a scrub starts 2024-03-30T13:57:09.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:08 smithi043 ceph-mon[42099]: 6.1a scrub ok 2024-03-30T13:57:09.219 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:09.259 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:09.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:09.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:09.361 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:09.364 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 1 16 4127 4111 256.921 256.938 0.00449915 0.00387338 2024-03-30T13:57:09.400 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:09.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:09.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:09.467 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:57:09 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:57:09.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:08 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:57:09.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:08 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:57:09.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:08 smithi113 ceph-mon[34341]: pgmap v628: 158 pgs: 88 active+clean, 1 active+clean+remapped+scrubbing, 20 active+undersized, 7 active+undersized+degraded, 42 active+clean+remapped; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 47/594 objects degraded (7.912%); 27/594 objects misplaced (4.545%) 2024-03-30T13:57:09.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:08 smithi113 ceph-mon[34341]: 4.10 scrub starts 2024-03-30T13:57:09.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:08 smithi113 ceph-mon[34341]: 4.10 scrub ok 2024-03-30T13:57:09.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:08 smithi113 ceph-mon[34341]: osdmap e297: 8 total, 7 up, 6 in 2024-03-30T13:57:09.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:08 smithi113 ceph-mon[34341]: 5.1d scrub starts 2024-03-30T13:57:09.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:08 smithi113 ceph-mon[34341]: 5.1d scrub ok 2024-03-30T13:57:09.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:08 smithi113 ceph-mon[34341]: 6.1a scrub starts 2024-03-30T13:57:09.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:08 smithi113 ceph-mon[34341]: 6.1a scrub ok 2024-03-30T13:57:09.502 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:09.538 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:09.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:09.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:09.640 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:09.675 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:09.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:09.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:09.777 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:09.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:09.818 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:09.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:09.919 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:09.953 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:09.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:09.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:10.000 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:09 smithi043 ceph-mon[29963]: 5.d scrub starts 2024-03-30T13:57:10.055 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:10.094 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:10.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:10.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:10.200 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:10.239 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:10.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:10.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:10.313 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:09 smithi043 ceph-mon[29963]: 5.d scrub ok 2024-03-30T13:57:10.313 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:09 smithi043 ceph-mon[42099]: 5.d scrub starts 2024-03-30T13:57:10.313 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:09 smithi043 ceph-mon[42099]: 5.d scrub ok 2024-03-30T13:57:10.341 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:10.349 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:57:10.364 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 2 16 7806 7790 243.418 229.938 0.00528999 0.00409891 2024-03-30T13:57:10.375 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:10.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:10.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:10.400 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:57:10.444 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:57:10.445 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:57:10.445 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:57:10.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:09 smithi113 ceph-mon[34341]: 5.d scrub starts 2024-03-30T13:57:10.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:09 smithi113 ceph-mon[34341]: 5.d scrub ok 2024-03-30T13:57:10.476 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:10.490 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:57:10.511 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:10.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:10.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:10.527 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:57:10.569 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:57:10.612 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:57:10.612 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:57:10.613 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:57:10.613 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:57:10.613 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:57:10.613 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:57:10.613 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:57:10.613 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:57:10.613 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:57:10.613 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:10.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:10.652 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:10.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:10.754 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:10.795 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:10.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:10.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:10.897 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:10.934 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:10.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:10.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:11.036 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:11.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:11.078 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:11.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:11.179 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:11.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:11.214 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:11.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:11.315 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:11.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:11 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:11.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:11 smithi043 ceph-mon[29963]: pgmap v630: 158 pgs: 88 active+clean, 15 active+undersized, 12 active+undersized+degraded, 43 active+clean+remapped; 45 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 7.3 MiB/s wr, 118 op/s; 408/2709 objects degraded (15.061%); 27/2709 objects misplaced (0.997%) 2024-03-30T13:57:11.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:11 smithi043 ceph-mon[29963]: 5.16 deep-scrub starts 2024-03-30T13:57:11.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:11 smithi043 ceph-mon[29963]: 5.16 deep-scrub ok 2024-03-30T13:57:11.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:11 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 408/2709 objects degraded (15.061%), 12 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:11.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:11 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1926119676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:57:11.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:11 smithi043 ceph-mon[29963]: 5.1 scrub starts 2024-03-30T13:57:11.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:11 smithi043 ceph-mon[29963]: 5.1 scrub ok 2024-03-30T13:57:11.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:11 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:11.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:11 smithi043 ceph-mon[42099]: pgmap v630: 158 pgs: 88 active+clean, 15 active+undersized, 12 active+undersized+degraded, 43 active+clean+remapped; 45 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 7.3 MiB/s wr, 118 op/s; 408/2709 objects degraded (15.061%); 27/2709 objects misplaced (0.997%) 2024-03-30T13:57:11.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:11 smithi043 ceph-mon[42099]: 5.16 deep-scrub starts 2024-03-30T13:57:11.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:11 smithi043 ceph-mon[42099]: 5.16 deep-scrub ok 2024-03-30T13:57:11.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:11 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 408/2709 objects degraded (15.061%), 12 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:11.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:11 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1926119676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:57:11.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:11 smithi043 ceph-mon[42099]: 5.1 scrub starts 2024-03-30T13:57:11.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:11 smithi043 ceph-mon[42099]: 5.1 scrub ok 2024-03-30T13:57:11.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:11.357 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:11.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:11.364 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 3 15 11285 11270 234.771 217.5 0.00450643 0.00425209 2024-03-30T13:57:11.458 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:11.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:11 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:11.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:11 smithi113 ceph-mon[34341]: pgmap v630: 158 pgs: 88 active+clean, 15 active+undersized, 12 active+undersized+degraded, 43 active+clean+remapped; 45 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 7.3 MiB/s wr, 118 op/s; 408/2709 objects degraded (15.061%); 27/2709 objects misplaced (0.997%) 2024-03-30T13:57:11.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:11 smithi113 ceph-mon[34341]: 5.16 deep-scrub starts 2024-03-30T13:57:11.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:11 smithi113 ceph-mon[34341]: 5.16 deep-scrub ok 2024-03-30T13:57:11.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:11 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 408/2709 objects degraded (15.061%), 12 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:11.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:11 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1926119676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:57:11.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:11 smithi113 ceph-mon[34341]: 5.1 scrub starts 2024-03-30T13:57:11.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:11 smithi113 ceph-mon[34341]: 5.1 scrub ok 2024-03-30T13:57:11.494 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:11.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:11.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:11.506 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T13:57:11.595 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:11.627 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:11.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:11.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:11.729 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:11.762 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:11.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:11.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:11.864 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:11.899 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:11.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:11.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:12.001 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:12.036 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:12.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:12.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:12.138 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:12.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:12 smithi043 ceph-mon[29963]: 4.1f scrub starts 2024-03-30T13:57:12.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:12 smithi043 ceph-mon[29963]: 4.1f scrub ok 2024-03-30T13:57:12.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:12 smithi043 ceph-mon[42099]: 4.1f scrub starts 2024-03-30T13:57:12.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:12 smithi043 ceph-mon[42099]: 4.1f scrub ok 2024-03-30T13:57:12.186 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:12.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:12.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:12.288 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:12.323 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:12.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:12.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:12.364 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 4 16 14640 14624 228.478 209.625 0.0033414 0.00436964 2024-03-30T13:57:12.425 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:12.458 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:12.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:12.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:12.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:12 smithi113 ceph-mon[34341]: 4.1f scrub starts 2024-03-30T13:57:12.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:12 smithi113 ceph-mon[34341]: 4.1f scrub ok 2024-03-30T13:57:12.560 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:12.589 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:12.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:12.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:12.691 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:12.726 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:12.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:12.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:12.828 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:12.852 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:12.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:12.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:12.954 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:12.991 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:12.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:12.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:13.093 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:13.133 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:13.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:13.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:13.235 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:13.275 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:13.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:13.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:13.306 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:13 smithi043 ceph-mon[29963]: pgmap v631: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 243 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.5 KiB/s rd, 36 MiB/s wr, 584 op/s; 2020/12225 objects degraded (16.524%); 27/12225 objects misplaced (0.221%) 2024-03-30T13:57:13.306 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:13 smithi043 ceph-mon[29963]: 4.15 scrub starts 2024-03-30T13:57:13.307 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:13 smithi043 ceph-mon[29963]: 4.15 scrub ok 2024-03-30T13:57:13.307 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:13 smithi043 ceph-mon[29963]: 3.a deep-scrub starts 2024-03-30T13:57:13.307 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:13 smithi043 ceph-mon[29963]: 3.a deep-scrub ok 2024-03-30T13:57:13.307 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:13 smithi043 ceph-mon[29963]: osdmap e298: 8 total, 7 up, 6 in 2024-03-30T13:57:13.307 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:13 smithi043 ceph-mon[29963]: 5.18 scrub starts 2024-03-30T13:57:13.307 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:13 smithi043 ceph-mon[29963]: 5.18 scrub ok 2024-03-30T13:57:13.307 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:13 smithi043 ceph-mon[42099]: pgmap v631: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 243 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.5 KiB/s rd, 36 MiB/s wr, 584 op/s; 2020/12225 objects degraded (16.524%); 27/12225 objects misplaced (0.221%) 2024-03-30T13:57:13.307 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:13 smithi043 ceph-mon[42099]: 4.15 scrub starts 2024-03-30T13:57:13.307 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:13 smithi043 ceph-mon[42099]: 4.15 scrub ok 2024-03-30T13:57:13.307 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:13 smithi043 ceph-mon[42099]: 3.a deep-scrub starts 2024-03-30T13:57:13.307 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:13 smithi043 ceph-mon[42099]: 3.a deep-scrub ok 2024-03-30T13:57:13.308 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:13 smithi043 ceph-mon[42099]: osdmap e298: 8 total, 7 up, 6 in 2024-03-30T13:57:13.308 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:13 smithi043 ceph-mon[42099]: 5.18 scrub starts 2024-03-30T13:57:13.308 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:13 smithi043 ceph-mon[42099]: 5.18 scrub ok 2024-03-30T13:57:13.364 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 5 16 17550 17534 219.154 181.875 0.00446033 0.00455678 2024-03-30T13:57:13.378 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:13.413 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:13.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:13.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:13.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:13 smithi113 ceph-mon[34341]: pgmap v631: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 243 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.5 KiB/s rd, 36 MiB/s wr, 584 op/s; 2020/12225 objects degraded (16.524%); 27/12225 objects misplaced (0.221%) 2024-03-30T13:57:13.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:13 smithi113 ceph-mon[34341]: 4.15 scrub starts 2024-03-30T13:57:13.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:13 smithi113 ceph-mon[34341]: 4.15 scrub ok 2024-03-30T13:57:13.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:13 smithi113 ceph-mon[34341]: 3.a deep-scrub starts 2024-03-30T13:57:13.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:13 smithi113 ceph-mon[34341]: 3.a deep-scrub ok 2024-03-30T13:57:13.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:13 smithi113 ceph-mon[34341]: osdmap e298: 8 total, 7 up, 6 in 2024-03-30T13:57:13.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:13 smithi113 ceph-mon[34341]: 5.18 scrub starts 2024-03-30T13:57:13.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:13 smithi113 ceph-mon[34341]: 5.18 scrub ok 2024-03-30T13:57:13.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-03-30T13:57:13.516 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:13.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:13.578 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:13.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:13.679 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:13.725 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:13.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:13.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:13.827 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:13.860 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:13.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:13.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:13.963 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:13.997 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:13.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:13.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:14.099 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:14.140 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:14.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:14.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:14.242 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:14.302 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:14 smithi043 ceph-mon[42099]: 4.2 scrub starts 2024-03-30T13:57:14.302 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:14 smithi043 ceph-mon[42099]: 4.2 scrub ok 2024-03-30T13:57:14.302 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:14 smithi043 ceph-mon[42099]: 3.1c scrub starts 2024-03-30T13:57:14.302 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:14 smithi043 ceph-mon[42099]: 3.1c scrub ok 2024-03-30T13:57:14.302 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:14 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:57:14.303 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:14 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:14.303 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:14.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:14.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:14.304 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:14 smithi043 ceph-mon[29963]: 4.2 scrub starts 2024-03-30T13:57:14.304 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:14 smithi043 ceph-mon[29963]: 4.2 scrub ok 2024-03-30T13:57:14.304 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:14 smithi043 ceph-mon[29963]: 3.1c scrub starts 2024-03-30T13:57:14.305 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:14 smithi043 ceph-mon[29963]: 3.1c scrub ok 2024-03-30T13:57:14.305 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:14 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:57:14.305 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:14 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:14.306 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:57:14.306 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:57:14.364 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 6 16 20431 20415 212.636 180.062 0.00555512 0.00469698 2024-03-30T13:57:14.405 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:14.439 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:14.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:14.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:14.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:14 smithi113 ceph-mon[34341]: 4.2 scrub starts 2024-03-30T13:57:14.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:14 smithi113 ceph-mon[34341]: 4.2 scrub ok 2024-03-30T13:57:14.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:14 smithi113 ceph-mon[34341]: 3.1c scrub starts 2024-03-30T13:57:14.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:14 smithi113 ceph-mon[34341]: 3.1c scrub ok 2024-03-30T13:57:14.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:14 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:57:14.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:14 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:14.542 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:14.581 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:14.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:14.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:14.683 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:14.723 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:14.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:14.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:14.798 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 reset_purged_snaps_last 2024-03-30T13:57:14.825 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:14.852 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:14.853 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:14.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:14.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:14.905 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:14.956 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:14.986 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:14.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:14.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:15.088 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:15.139 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:15.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:15.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:15.241 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:15.255 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:15 smithi043 ceph-mon[29963]: pgmap v633: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 511 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 771 B/s rd, 77 MiB/s wr, 1.23k op/s; 4202/25116 objects degraded (16.730%); 27/25116 objects misplaced (0.108%) 2024-03-30T13:57:15.255 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:15 smithi043 ceph-mon[29963]: 6.10 scrub starts 2024-03-30T13:57:15.255 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:15 smithi043 ceph-mon[29963]: 6.10 scrub ok 2024-03-30T13:57:15.255 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:15 smithi043 ceph-mon[29963]: 1.6 deep-scrub starts 2024-03-30T13:57:15.255 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:15 smithi043 ceph-mon[29963]: 1.6 deep-scrub ok 2024-03-30T13:57:15.256 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:15 smithi043 ceph-mon[42099]: pgmap v633: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 511 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 771 B/s rd, 77 MiB/s wr, 1.23k op/s; 4202/25116 objects degraded (16.730%); 27/25116 objects misplaced (0.108%) 2024-03-30T13:57:15.256 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:15 smithi043 ceph-mon[42099]: 6.10 scrub starts 2024-03-30T13:57:15.256 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:15 smithi043 ceph-mon[42099]: 6.10 scrub ok 2024-03-30T13:57:15.256 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:15 smithi043 ceph-mon[42099]: 1.6 deep-scrub starts 2024-03-30T13:57:15.256 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:15 smithi043 ceph-mon[42099]: 1.6 deep-scrub ok 2024-03-30T13:57:15.286 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:15.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:15.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:15.364 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 7 16 23119 23103 206.257 168 0.00267028 0.00484215 2024-03-30T13:57:15.388 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:15.430 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:15.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:15.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:15.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:15 smithi113 ceph-mon[34341]: pgmap v633: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 511 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 771 B/s rd, 77 MiB/s wr, 1.23k op/s; 4202/25116 objects degraded (16.730%); 27/25116 objects misplaced (0.108%) 2024-03-30T13:57:15.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:15 smithi113 ceph-mon[34341]: 6.10 scrub starts 2024-03-30T13:57:15.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:15 smithi113 ceph-mon[34341]: 6.10 scrub ok 2024-03-30T13:57:15.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:15 smithi113 ceph-mon[34341]: 1.6 deep-scrub starts 2024-03-30T13:57:15.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:15 smithi113 ceph-mon[34341]: 1.6 deep-scrub ok 2024-03-30T13:57:15.532 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:15.566 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:15.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:15.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:15.668 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:15.710 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:15.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:15.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:15.812 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:15.846 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:15.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:15.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:15.948 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:15.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:15.983 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:15.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:16.084 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:16.121 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:16.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:16.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:16.223 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:16.263 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:16.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:16.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:16.307 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:57:16.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:16 smithi043 ceph-mon[29963]: 4.7 scrub starts 2024-03-30T13:57:16.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:16 smithi043 ceph-mon[29963]: 4.7 scrub ok 2024-03-30T13:57:16.347 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:16 smithi043 ceph-mon[42099]: 4.7 scrub starts 2024-03-30T13:57:16.347 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:16 smithi043 ceph-mon[42099]: 4.7 scrub ok 2024-03-30T13:57:16.347 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:57:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:57:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:57:16.364 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 8 16 25905 25889 202.238 174.125 0.00733078 0.00493781 2024-03-30T13:57:16.365 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:16.418 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:16.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:16.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:16.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:16 smithi113 ceph-mon[34341]: 4.7 scrub starts 2024-03-30T13:57:16.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:16 smithi113 ceph-mon[34341]: 4.7 scrub ok 2024-03-30T13:57:16.523 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:16.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:16.567 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:16.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:16.667 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:16.701 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:16.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:16.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:16.803 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:16.838 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:16.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:16.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:16.940 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:16.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:16.972 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:16.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:17.073 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:17.106 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:17.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:17.207 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:17.268 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:17.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:17.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:17.355 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:17 smithi043 ceph-mon[29963]: pgmap v634: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 1.1 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 145 MiB/s wr, 2.31k op/s; 9440/56109 objects degraded (16.824%); 27/56109 objects misplaced (0.048%) 2024-03-30T13:57:17.355 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:17 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 9440/56109 objects degraded (16.824%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:17.355 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:17 smithi043 ceph-mon[29963]: 5.2 scrub starts 2024-03-30T13:57:17.355 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:17 smithi043 ceph-mon[29963]: 5.2 scrub ok 2024-03-30T13:57:17.356 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:17 smithi043 ceph-mon[42099]: pgmap v634: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 1.1 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 145 MiB/s wr, 2.31k op/s; 9440/56109 objects degraded (16.824%); 27/56109 objects misplaced (0.048%) 2024-03-30T13:57:17.356 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:17 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 9440/56109 objects degraded (16.824%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:17.356 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:17 smithi043 ceph-mon[42099]: 5.2 scrub starts 2024-03-30T13:57:17.356 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:17 smithi043 ceph-mon[42099]: 5.2 scrub ok 2024-03-30T13:57:17.366 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 9 16 28440 28424 197.355 158.438 0.00413234 0.00506195 2024-03-30T13:57:17.370 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:17.413 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:17.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:17.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:17.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:17 smithi113 ceph-mon[34341]: pgmap v634: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 1.1 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 145 MiB/s wr, 2.31k op/s; 9440/56109 objects degraded (16.824%); 27/56109 objects misplaced (0.048%) 2024-03-30T13:57:17.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:17 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 9440/56109 objects degraded (16.824%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:17.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:17 smithi113 ceph-mon[34341]: 5.2 scrub starts 2024-03-30T13:57:17.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:17 smithi113 ceph-mon[34341]: 5.2 scrub ok 2024-03-30T13:57:17.514 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:17.548 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:17.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:17.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:17.650 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:17.684 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:17.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:17.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:17.786 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:17.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:17.827 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:17.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:17.928 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:17.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:17.963 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:17.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:18.064 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:18.104 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:18.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:18.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:18.206 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:18.230 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:18.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:18.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:18.332 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:18.366 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 10 16 30790 30774 192.304 146.875 0.0109628 0.00519448 2024-03-30T13:57:18.370 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:18.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:18.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:18 smithi043 ceph-mon[29963]: 6.11 scrub starts 2024-03-30T13:57:18.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:18 smithi043 ceph-mon[29963]: 6.11 scrub ok 2024-03-30T13:57:18.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:18 smithi043 ceph-mon[29963]: 3.10 scrub starts 2024-03-30T13:57:18.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:18 smithi043 ceph-mon[29963]: 3.10 scrub ok 2024-03-30T13:57:18.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:18 smithi043 ceph-mon[29963]: 2.0 scrub starts 2024-03-30T13:57:18.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:18 smithi043 ceph-mon[29963]: 2.0 scrub ok 2024-03-30T13:57:18.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:18 smithi043 ceph-mon[42099]: 6.11 scrub starts 2024-03-30T13:57:18.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:18 smithi043 ceph-mon[42099]: 6.11 scrub ok 2024-03-30T13:57:18.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:18 smithi043 ceph-mon[42099]: 3.10 scrub starts 2024-03-30T13:57:18.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:18 smithi043 ceph-mon[42099]: 3.10 scrub ok 2024-03-30T13:57:18.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:18 smithi043 ceph-mon[42099]: 2.0 scrub starts 2024-03-30T13:57:18.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:18 smithi043 ceph-mon[42099]: 2.0 scrub ok 2024-03-30T13:57:18.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:18 smithi113 ceph-mon[34341]: 6.11 scrub starts 2024-03-30T13:57:18.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:18 smithi113 ceph-mon[34341]: 6.11 scrub ok 2024-03-30T13:57:18.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:18 smithi113 ceph-mon[34341]: 3.10 scrub starts 2024-03-30T13:57:18.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:18 smithi113 ceph-mon[34341]: 3.10 scrub ok 2024-03-30T13:57:18.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:18 smithi113 ceph-mon[34341]: 2.0 scrub starts 2024-03-30T13:57:18.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:18 smithi113 ceph-mon[34341]: 2.0 scrub ok 2024-03-30T13:57:18.472 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:18.506 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:18.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:18.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:18.608 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:18.641 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:18.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:18.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:18.743 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:18.781 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:18.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:18.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:18.883 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:18.927 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:18.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:18.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:19.009 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:19.031 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:19.064 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:19.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:19.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:19.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:19 smithi043 ceph-mon[29963]: pgmap v635: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 1.2 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 1.0 KiB/s rd, 132 MiB/s wr, 2.11k op/s; 10354/61593 objects degraded (16.810%); 27/61593 objects misplaced (0.044%) 2024-03-30T13:57:19.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:19 smithi043 ceph-mon[29963]: 3.1 scrub starts 2024-03-30T13:57:19.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:19 smithi043 ceph-mon[29963]: 3.1 scrub ok 2024-03-30T13:57:19.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:19 smithi043 ceph-mon[29963]: 1.7 scrub starts 2024-03-30T13:57:19.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:19 smithi043 ceph-mon[29963]: 1.7 scrub ok 2024-03-30T13:57:19.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:19 smithi043 ceph-mon[42099]: pgmap v635: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 1.2 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 1.0 KiB/s rd, 132 MiB/s wr, 2.11k op/s; 10354/61593 objects degraded (16.810%); 27/61593 objects misplaced (0.044%) 2024-03-30T13:57:19.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:19 smithi043 ceph-mon[42099]: 3.1 scrub starts 2024-03-30T13:57:19.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:19 smithi043 ceph-mon[42099]: 3.1 scrub ok 2024-03-30T13:57:19.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:19 smithi043 ceph-mon[42099]: 1.7 scrub starts 2024-03-30T13:57:19.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:19 smithi043 ceph-mon[42099]: 1.7 scrub ok 2024-03-30T13:57:19.166 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:19.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:19.190 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:19.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:19.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:19.330 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:19.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:19.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:19.366 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 11 16 32925 32909 186.952 133.438 0.00811848 0.00534429 2024-03-30T13:57:19.368 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:57:19 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:57:19.368 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:19 smithi113 ceph-mon[34341]: pgmap v635: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 1.2 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 1.0 KiB/s rd, 132 MiB/s wr, 2.11k op/s; 10354/61593 objects degraded (16.810%); 27/61593 objects misplaced (0.044%) 2024-03-30T13:57:19.368 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:19 smithi113 ceph-mon[34341]: 3.1 scrub starts 2024-03-30T13:57:19.368 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:19 smithi113 ceph-mon[34341]: 3.1 scrub ok 2024-03-30T13:57:19.368 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:19 smithi113 ceph-mon[34341]: 1.7 scrub starts 2024-03-30T13:57:19.368 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:19 smithi113 ceph-mon[34341]: 1.7 scrub ok 2024-03-30T13:57:19.437 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:19.472 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:19.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:19.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:19.576 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:19.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:19.611 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:19.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:19.712 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:19.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:19.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:19.847 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:19.885 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:19.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:19.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:19.987 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:20.016 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:20.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:20.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:20.118 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:20.154 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:20.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:20.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:20 smithi113 ceph-mon[34341]: 3.15 scrub starts 2024-03-30T13:57:20.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:20 smithi113 ceph-mon[34341]: 3.15 scrub ok 2024-03-30T13:57:20.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:20 smithi113 ceph-mon[34341]: 4.1a scrub starts 2024-03-30T13:57:20.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:20 smithi113 ceph-mon[34341]: 4.1a scrub ok 2024-03-30T13:57:20.256 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:20.290 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:20.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:20.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:20.366 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 12 16 35020 35004 182.283 130.938 0.00375659 0.00548018 2024-03-30T13:57:20.393 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:20.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:20 smithi043 ceph-mon[29963]: 3.15 scrub starts 2024-03-30T13:57:20.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:20 smithi043 ceph-mon[29963]: 3.15 scrub ok 2024-03-30T13:57:20.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:20 smithi043 ceph-mon[29963]: 4.1a scrub starts 2024-03-30T13:57:20.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:20 smithi043 ceph-mon[29963]: 4.1a scrub ok 2024-03-30T13:57:20.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:20 smithi043 ceph-mon[42099]: 3.15 scrub starts 2024-03-30T13:57:20.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:20 smithi043 ceph-mon[42099]: 3.15 scrub ok 2024-03-30T13:57:20.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:20 smithi043 ceph-mon[42099]: 4.1a scrub starts 2024-03-30T13:57:20.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:20 smithi043 ceph-mon[42099]: 4.1a scrub ok 2024-03-30T13:57:20.432 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:20.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:20.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:20.534 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:20.572 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:20.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:20.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:20.674 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:20.713 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:20.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:20.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:20.815 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:20.850 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:20.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:20.952 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:20.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:20.988 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:20.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:21.089 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:21.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:21.121 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:21.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:21.222 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:21.262 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:21.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:21.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:21.364 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:21.365 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 13 16 37070 37054 178.117 128.125 0.00625838 0.00560916 2024-03-30T13:57:21.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:21 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:21.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:21 smithi043 ceph-mon[29963]: pgmap v636: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 1.7 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 174 MiB/s wr, 2.78k op/s; 14466/86211 objects degraded (16.780%); 27/86211 objects misplaced (0.031%) 2024-03-30T13:57:21.373 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:21 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:21.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:21 smithi043 ceph-mon[42099]: pgmap v636: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 1.7 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 174 MiB/s wr, 2.78k op/s; 14466/86211 objects degraded (16.780%); 27/86211 objects misplaced (0.031%) 2024-03-30T13:57:21.384 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:21.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:21.435 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:21.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:21.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:21 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:21.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:21 smithi113 ceph-mon[34341]: pgmap v636: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 1.7 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 174 MiB/s wr, 2.78k op/s; 14466/86211 objects degraded (16.780%); 27/86211 objects misplaced (0.031%) 2024-03-30T13:57:21.536 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:21.581 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:21.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:21.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:21.683 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:21.725 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:21.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:21.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:21.827 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:21.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:21.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:21.974 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:22.013 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:22.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:22.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:22.067 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-03-30T13:57:22.115 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:22.150 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:22.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:22.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:22.252 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:22.286 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:22.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:22.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:22.366 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 14 16 39034 39018 174.161 122.75 0.00463234 0.00573688 2024-03-30T13:57:22.388 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:22.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:22 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/911406078' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:57:22.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:22 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/911406078' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:57:22.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:22.419 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:22.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:22.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:22 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/911406078' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:57:22.520 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:22.560 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:22.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:22.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:22.662 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:22.701 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:22.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:22.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:22.803 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:22.842 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:22.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:22.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:22.945 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:22.984 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:22.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:22.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:23.086 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:23.113 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:23.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:23.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:23.215 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:23.252 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:23.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:23.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:23.354 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:23.366 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 15 16 41059 41043 170.988 126.562 0.00686685 0.00584349 2024-03-30T13:57:23.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:23 smithi043 ceph-mon[29963]: pgmap v637: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 2.0 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 181 MiB/s wr, 2.90k op/s; 16649/99246 objects degraded (16.775%); 27/99246 objects misplaced (0.027%) 2024-03-30T13:57:23.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:23 smithi043 ceph-mon[29963]: 5.0 scrub starts 2024-03-30T13:57:23.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:23 smithi043 ceph-mon[29963]: 5.0 scrub ok 2024-03-30T13:57:23.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:23 smithi043 ceph-mon[29963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:57:23.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:23 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 16649/99246 objects degraded (16.775%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:23.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:23 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/911406078' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:57:23.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:23 smithi043 ceph-mon[29963]: osdmap e299: 8 total, 7 up, 6 in 2024-03-30T13:57:23.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:23 smithi043 ceph-mon[29963]: osdmap e300: 8 total, 7 up, 6 in 2024-03-30T13:57:23.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:23 smithi043 ceph-mon[42099]: pgmap v637: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 2.0 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 181 MiB/s wr, 2.90k op/s; 16649/99246 objects degraded (16.775%); 27/99246 objects misplaced (0.027%) 2024-03-30T13:57:23.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:23 smithi043 ceph-mon[42099]: 5.0 scrub starts 2024-03-30T13:57:23.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:23 smithi043 ceph-mon[42099]: 5.0 scrub ok 2024-03-30T13:57:23.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:23 smithi043 ceph-mon[42099]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:57:23.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:23 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 16649/99246 objects degraded (16.775%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:23.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:23 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/911406078' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:57:23.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:23 smithi043 ceph-mon[42099]: osdmap e299: 8 total, 7 up, 6 in 2024-03-30T13:57:23.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:23 smithi043 ceph-mon[42099]: osdmap e300: 8 total, 7 up, 6 in 2024-03-30T13:57:23.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:23.396 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:23.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:23.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:23 smithi113 ceph-mon[34341]: pgmap v637: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 2.0 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 181 MiB/s wr, 2.90k op/s; 16649/99246 objects degraded (16.775%); 27/99246 objects misplaced (0.027%) 2024-03-30T13:57:23.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:23 smithi113 ceph-mon[34341]: 5.0 scrub starts 2024-03-30T13:57:23.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:23 smithi113 ceph-mon[34341]: 5.0 scrub ok 2024-03-30T13:57:23.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:23 smithi113 ceph-mon[34341]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:57:23.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:23 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 16649/99246 objects degraded (16.775%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:23.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:23 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/911406078' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:57:23.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:23 smithi113 ceph-mon[34341]: osdmap e299: 8 total, 7 up, 6 in 2024-03-30T13:57:23.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:23 smithi113 ceph-mon[34341]: osdmap e300: 8 total, 7 up, 6 in 2024-03-30T13:57:23.497 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:23.530 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:23.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:23.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:23.632 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:23.672 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:23.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:23.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:23.774 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:23.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:23.808 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:23.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:23.910 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:23.942 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:23.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:23.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:24.044 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:24.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:24.077 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:24.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:24.178 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:24.209 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:24.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:24.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:24.311 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:24.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:24.349 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:24.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:24.366 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 16 15 43315 43300 169.117 141.062 0.00655646 0.00590801 2024-03-30T13:57:24.449 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:24.489 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:24.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:24.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:24.591 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:24.624 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:24.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:24.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:24.726 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:24.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:24.758 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:24.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:24.859 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:24.893 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:24.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:24.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:24.995 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:25.028 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:25.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:25.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:25.131 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:25.160 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:25.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:25.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:25.262 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:25.304 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:25.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:25.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:25.366 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 17 16 45462 45446 167.058 134.125 0.00914845 0.00598138 2024-03-30T13:57:25.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:25 smithi043 ceph-mon[29963]: pgmap v640: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 2.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.11k op/s; 17921/106767 objects degraded (16.785%); 27/106767 objects misplaced (0.025%) 2024-03-30T13:57:25.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:25 smithi043 ceph-mon[42099]: pgmap v640: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 2.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.11k op/s; 17921/106767 objects degraded (16.785%); 27/106767 objects misplaced (0.025%) 2024-03-30T13:57:25.406 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:25.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:25.446 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:25.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:25.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:25 smithi113 ceph-mon[34341]: pgmap v640: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 2.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.11k op/s; 17921/106767 objects degraded (16.785%); 27/106767 objects misplaced (0.025%) 2024-03-30T13:57:25.548 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:25.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:25.588 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:25.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:25.689 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:25.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:25.739 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:25.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:25.840 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:25.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:25.874 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:25.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:25.906 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T13:57:25.962 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:57:25.962 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:57:25.975 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:26.002 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:26.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:26.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:26.104 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:26.137 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:26.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:26.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:26.238 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:26.271 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:26.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:26.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:26.331 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:57:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:57:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:57:26.332 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:26 smithi043 ceph-mon[42099]: pgmap v641: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 2.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 169 MiB/s wr, 2.70k op/s; 21235/126339 objects degraded (16.808%); 27/126339 objects misplaced (0.021%) 2024-03-30T13:57:26.332 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:26 smithi043 ceph-mon[29963]: pgmap v641: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 2.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 169 MiB/s wr, 2.70k op/s; 21235/126339 objects degraded (16.808%); 27/126339 objects misplaced (0.021%) 2024-03-30T13:57:26.366 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 18 16 47360 47344 164.366 118.625 0.00728698 0.00607981 2024-03-30T13:57:26.373 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:26.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:26.406 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:26.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:26.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:26 smithi113 ceph-mon[34341]: pgmap v641: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 2.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 169 MiB/s wr, 2.70k op/s; 21235/126339 objects degraded (16.808%); 27/126339 objects misplaced (0.021%) 2024-03-30T13:57:26.506 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:26.539 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:26.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:26.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:26.641 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:26.681 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:26.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:26.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:26.782 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:26.815 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:26.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:26.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:26.917 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:26.948 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:26.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:26.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:27.003 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-30T13:57:27.051 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:27.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:27.083 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:27.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:27.184 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:27.213 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:27.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:27.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:27.315 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:27.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:27.347 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:27.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:27.366 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 19 16 49306 49290 162.115 121.625 0.022379 0.00616101 2024-03-30T13:57:27.448 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:27.477 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:27.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:27.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:27.545 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:57:27.579 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:27.608 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:27.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:27.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:27.710 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:27.741 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:27.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:27.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:27.843 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:27.874 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:27.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:27.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:27.875 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:57:27.875 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:57:27.875 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:57:27.876 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:57:27.876 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:57:27.895 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:57:27.935 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:57:27.935 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:57:27.935 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:57:27.935 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:57:27.935 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:57:27.935 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:57:27.935 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:57:27.936 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:57:27.936 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:57:27.976 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:28.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:28.021 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:28.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:28.122 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:28.146 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:28.155 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:28.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:28.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:28.256 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:28.291 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:28.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:28.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:28.366 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-03-30T13:57:28.371478+0000 min lat: 0.00120663 max lat: 0.104947 avg lat: 0.00626286 2024-03-30T13:57:28.367 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T13:57:28.367 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 20 16 51074 51058 159.534 110.5 0.00513189 0.00626286 2024-03-30T13:57:28.392 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:28.425 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:28.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:28.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:28.527 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:28.556 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:28 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:57:28.556 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:28 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:57:28.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:28 smithi043 ceph-mon[29963]: osdmap e301: 8 total, 7 up, 6 in 2024-03-30T13:57:28.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:28 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 21235/126339 objects degraded (16.808%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:28.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:28 smithi043 ceph-mon[29963]: osdmap e302: 8 total, 7 up, 6 in 2024-03-30T13:57:28.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:28 smithi043 ceph-mon[29963]: pgmap v644: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 2.6 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 920 B/s rd, 117 MiB/s wr, 1.87k op/s; 21920/130500 objects degraded (16.797%); 27/130500 objects misplaced (0.021%) 2024-03-30T13:57:28.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:28 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2139594624' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:57:28.557 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:28 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:57:28.557 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:28 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:57:28.557 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:28 smithi043 ceph-mon[42099]: osdmap e301: 8 total, 7 up, 6 in 2024-03-30T13:57:28.557 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:28 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 21235/126339 objects degraded (16.808%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:28.557 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:28 smithi043 ceph-mon[42099]: osdmap e302: 8 total, 7 up, 6 in 2024-03-30T13:57:28.557 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:28 smithi043 ceph-mon[42099]: pgmap v644: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 2.6 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 920 B/s rd, 117 MiB/s wr, 1.87k op/s; 21920/130500 objects degraded (16.797%); 27/130500 objects misplaced (0.021%) 2024-03-30T13:57:28.557 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:28 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2139594624' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:57:28.561 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:28.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:28.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:28.607 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:57:28.607 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2024-03-30T13:57:28.607 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 reset_purged_snaps_last 2024-03-30T13:57:28.663 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:28.696 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:28.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:28.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:28.717 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:28 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:57:28.717 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:28 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:57:28.717 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:28 smithi113 ceph-mon[34341]: osdmap e301: 8 total, 7 up, 6 in 2024-03-30T13:57:28.717 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:28 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 21235/126339 objects degraded (16.808%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:28.717 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:28 smithi113 ceph-mon[34341]: osdmap e302: 8 total, 7 up, 6 in 2024-03-30T13:57:28.717 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:28 smithi113 ceph-mon[34341]: pgmap v644: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 2.6 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 920 B/s rd, 117 MiB/s wr, 1.87k op/s; 21920/130500 objects degraded (16.797%); 27/130500 objects misplaced (0.021%) 2024-03-30T13:57:28.717 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:28 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2139594624' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:57:28.799 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:28.833 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:28.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:28.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:28.935 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:28.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:28.968 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:28.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:29.069 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:29.100 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:29.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:29.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:29.202 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:29.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:29.237 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:29.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:29.338 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:29.366 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 21 16 52964 52948 157.562 118.125 0.00345747 0.00634163 2024-03-30T13:57:29.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:29.385 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:29.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:29.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:57:29 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:57:29.486 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:29.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:29.526 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:29.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:29.627 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:29.659 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:29.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:29.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:29.761 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:29.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:29.795 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:29.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:29.879 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:57:29.880 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:57:29.898 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:29.930 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:29.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:29.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:30.032 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:30.074 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:30.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:30.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:30.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:29 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:57:30.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:29 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:30.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:29 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:57:30.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:29 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:30.176 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:30.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:30.213 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:30.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:30.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:29 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:57:30.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:29 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:30.240 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:30.315 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:30.346 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:30.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:30.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:30.367 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 22 16 54827 54811 155.691 116.438 0.00725979 0.0064181 2024-03-30T13:57:30.448 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:30.492 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:30.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:30.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:30.545 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:30.594 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:30.626 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:30.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:30.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:30.727 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:30.765 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:30.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:30.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:30.867 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:30.901 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:30.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:30.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:31.003 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:31.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:31.037 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:31.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:31.094 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:30 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:31.095 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:30 smithi043 ceph-mon[29963]: pgmap v645: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 3.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 724 B/s rd, 147 MiB/s wr, 2.35k op/s; 24960/149163 objects degraded (16.733%); 27/149163 objects misplaced (0.018%) 2024-03-30T13:57:31.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:30 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:31.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:30 smithi043 ceph-mon[42099]: pgmap v645: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 3.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 724 B/s rd, 147 MiB/s wr, 2.35k op/s; 24960/149163 objects degraded (16.733%); 27/149163 objects misplaced (0.018%) 2024-03-30T13:57:31.137 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:31.190 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:31.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:31.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:31.292 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:31.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:31.335 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:31.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:31.336 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:30 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:31.337 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:30 smithi113 ceph-mon[34341]: pgmap v645: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 3.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 724 B/s rd, 147 MiB/s wr, 2.35k op/s; 24960/149163 objects degraded (16.733%); 27/149163 objects misplaced (0.018%) 2024-03-30T13:57:31.367 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 23 16 56573 56557 153.666 109.125 0.0168979 0.00650264 2024-03-30T13:57:31.436 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:31.469 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:31.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:31.570 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:31.608 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:31.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:31.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:31.710 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:31.742 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:31.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:31.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:31.844 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:31.881 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:57:31.883 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:31.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:31.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:31.985 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:31.994 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-03-30T13:57:32.012 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:32.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:32.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:32.113 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:32.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:31 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4003721541' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:57:32.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:31 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:57:32.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:31 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/4003721541' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:57:32.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:31 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:57:32.160 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:32.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:32.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:32.262 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:32.294 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:32.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:32.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:32.367 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 24 15 58259 58244 151.652 105.438 0.0104759 0.00658958 2024-03-30T13:57:32.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:32.437 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:32.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:32.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:32.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:31 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/4003721541' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:57:32.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:31 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:57:32.540 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:32.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:32.573 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:32.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:32.675 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:32.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:32.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:32.815 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:32.854 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:32.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:32.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:32.956 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:32.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:32.989 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:32.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:33.090 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:33.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:33.123 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:33.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:33.224 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:33.253 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:32 smithi043 ceph-mon[29963]: pgmap v646: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 3.2 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.20k op/s; 26722/159597 objects degraded (16.743%); 27/159597 objects misplaced (0.017%) 2024-03-30T13:57:33.253 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:32 smithi043 ceph-mon[29963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:57:33.253 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:32 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:57:33.253 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:32 smithi043 ceph-mon[29963]: osdmap e303: 8 total, 7 up, 6 in 2024-03-30T13:57:33.253 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:32 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 26722/159597 objects degraded (16.743%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:33.253 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:32 smithi043 ceph-mon[29963]: osdmap e304: 8 total, 7 up, 6 in 2024-03-30T13:57:33.253 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:32 smithi043 ceph-mon[42099]: pgmap v646: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 3.2 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.20k op/s; 26722/159597 objects degraded (16.743%); 27/159597 objects misplaced (0.017%) 2024-03-30T13:57:33.253 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:32 smithi043 ceph-mon[42099]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:57:33.254 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:32 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:57:33.254 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:32 smithi043 ceph-mon[42099]: osdmap e303: 8 total, 7 up, 6 in 2024-03-30T13:57:33.254 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:32 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 26722/159597 objects degraded (16.743%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:33.254 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:32 smithi043 ceph-mon[42099]: osdmap e304: 8 total, 7 up, 6 in 2024-03-30T13:57:33.256 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:33.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:33.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:33.358 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:33.368 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 25 16 60075 60059 150.123 113.438 0.00418084 0.00665591 2024-03-30T13:57:33.389 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:33.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:33.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:33.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:32 smithi113 ceph-mon[34341]: pgmap v646: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 3.2 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.20k op/s; 26722/159597 objects degraded (16.743%); 27/159597 objects misplaced (0.017%) 2024-03-30T13:57:33.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:32 smithi113 ceph-mon[34341]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:57:33.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:32 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:57:33.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:32 smithi113 ceph-mon[34341]: osdmap e303: 8 total, 7 up, 6 in 2024-03-30T13:57:33.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:32 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 26722/159597 objects degraded (16.743%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:33.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:32 smithi113 ceph-mon[34341]: osdmap e304: 8 total, 7 up, 6 in 2024-03-30T13:57:33.491 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:33.528 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:33.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:33.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:33.630 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:33.666 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:33.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:33.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:33.767 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:33.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:33.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:33.913 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:33.951 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:33.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:33.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:34.054 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:34.089 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:34.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:34.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:34.190 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:34.224 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:34.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:34.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:34.326 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:34.368 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 26 16 61846 61830 148.606 110.688 0.00418874 0.00672342 2024-03-30T13:57:34.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:34.374 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:34.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:34.475 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:34.509 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:34.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:34.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:34.610 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:34.640 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:34.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:34.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:34.658 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:57:34.658 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-03-30T13:57:34.658 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.659 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.15531.0:62295 10.1 e304/289 10:868f60fd:::benchmark_data_smithi043_169264_object62294:head v 304'3815, mlcod=304'3813)", 2024-03-30T13:57:34.659 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:57:34.657703+0000", 2024-03-30T13:57:34.659 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0037649010000000002, 2024-03-30T13:57:34.659 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0037991729999999999, 2024-03-30T13:57:34.659 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:57:34.659 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:57:34.659 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T13:57:34.659 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:57:34.659 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.659 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:57:34.659 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.657703+0000", 2024-03-30T13:57:34.659 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:57:34.659 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.660 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:57:34.660 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.657703+0000", 2024-03-30T13:57:34.660 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:57:34.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.660 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:57:34.660 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.657704+0000", 2024-03-30T13:57:34.660 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.243e-06 2024-03-30T13:57:34.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.660 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:57:34.660 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.657749+0000", 2024-03-30T13:57:34.660 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4771999999999998e-05 2024-03-30T13:57:34.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.661 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.661 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:57:34.661 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.657770+0000", 2024-03-30T13:57:34.661 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0868000000000001e-05 2024-03-30T13:57:34.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.661 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.661 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:57:34.661 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.657780+0000", 2024-03-30T13:57:34.661 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9909999999999997e-06 2024-03-30T13:57:34.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.661 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.662 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:57:34.662 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.657946+0000", 2024-03-30T13:57:34.662 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016648499999999999 2024-03-30T13:57:34.662 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.662 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.662 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:57:34.662 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.657970+0000", 2024-03-30T13:57:34.662 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.317e-05 2024-03-30T13:57:34.662 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:57:34.662 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:57:34.662 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:57:34.662 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.662 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.663 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.15531.0:62297 10.0 e304/289 10:0c743855:::benchmark_data_smithi043_169264_object62296:head v 304'3888, mlcod=304'3886)", 2024-03-30T13:57:34.663 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:57:34.657812+0000", 2024-03-30T13:57:34.663 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.003655728, 2024-03-30T13:57:34.663 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0037647449999999999, 2024-03-30T13:57:34.663 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:57:34.663 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:57:34.663 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T13:57:34.663 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:57:34.663 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.663 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:57:34.663 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.657812+0000", 2024-03-30T13:57:34.663 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:57:34.663 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.663 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.664 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:57:34.664 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.657812+0000", 2024-03-30T13:57:34.664 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:57:34.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.664 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.664 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:57:34.664 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.657813+0000", 2024-03-30T13:57:34.664 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5699999999999998e-07 2024-03-30T13:57:34.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.664 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.664 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:57:34.664 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.657850+0000", 2024-03-30T13:57:34.664 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7333000000000003e-05 2024-03-30T13:57:34.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.665 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.665 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:57:34.665 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.657859+0000", 2024-03-30T13:57:34.665 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.9090000000000006e-06 2024-03-30T13:57:34.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.665 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.665 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:57:34.665 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.657871+0000", 2024-03-30T13:57:34.665 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2187999999999999e-05 2024-03-30T13:57:34.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.665 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.665 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:57:34.665 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.659966+0000", 2024-03-30T13:57:34.666 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0020954160000000001 2024-03-30T13:57:34.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.666 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.666 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:57:34.666 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.659987+0000", 2024-03-30T13:57:34.666 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0355000000000001e-05 2024-03-30T13:57:34.666 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:57:34.666 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:57:34.666 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:57:34.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.666 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.666 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.15531.0:62298 10.6 e304/289 10:689c123c:::benchmark_data_smithi043_169264_object62297:head v 304'3907, mlcod=304'3905)", 2024-03-30T13:57:34.666 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:57:34.659356+0000", 2024-03-30T13:57:34.667 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0021116870000000001, 2024-03-30T13:57:34.667 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022892429999999998, 2024-03-30T13:57:34.667 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:57:34.667 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:57:34.667 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T13:57:34.667 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:57:34.667 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.667 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:57:34.667 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.659356+0000", 2024-03-30T13:57:34.667 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:57:34.667 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.667 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.667 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:57:34.667 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.659356+0000", 2024-03-30T13:57:34.667 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:57:34.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.668 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.668 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:57:34.668 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.659357+0000", 2024-03-30T13:57:34.668 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.9100000000000002e-07 2024-03-30T13:57:34.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.668 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.668 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:57:34.668 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.659386+0000", 2024-03-30T13:57:34.668 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.8421000000000001e-05 2024-03-30T13:57:34.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.668 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.668 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:57:34.669 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.659401+0000", 2024-03-30T13:57:34.669 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5238e-05 2024-03-30T13:57:34.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.669 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.669 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:57:34.669 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.659409+0000", 2024-03-30T13:57:34.669 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8120000000000008e-06 2024-03-30T13:57:34.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.669 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.669 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:57:34.669 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.660328+0000", 2024-03-30T13:57:34.669 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00091930899999999999 2024-03-30T13:57:34.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.669 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.670 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:57:34.670 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.660350+0000", 2024-03-30T13:57:34.670 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1801999999999999e-05 2024-03-30T13:57:34.670 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:57:34.670 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:57:34.670 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:57:34.670 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.670 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.670 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15531.0:62306 10.2 10:4e9273e4:::benchmark_data_smithi043_169264_object62305:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e304)", 2024-03-30T13:57:34.670 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:57:34.661461+0000", 2024-03-30T13:57:34.670 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 7.3610000000000003e-06, 2024-03-30T13:57:34.670 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00025527200000000002, 2024-03-30T13:57:34.670 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:57:34.670 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:57:34.671 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-03-30T13:57:34.671 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:57:34.671 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15531", 2024-03-30T13:57:34.671 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T13:57:34.671 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 62306 2024-03-30T13:57:34.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.671 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:57:34.671 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.671 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:57:34.671 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.661461+0000", 2024-03-30T13:57:34.671 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:57:34.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.672 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.672 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:57:34.672 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.661461+0000", 2024-03-30T13:57:34.672 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:57:34.672 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.672 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.672 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:57:34.672 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.661462+0000", 2024-03-30T13:57:34.672 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7600000000000001e-06 2024-03-30T13:57:34.672 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.672 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.672 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:57:34.672 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.661486+0000", 2024-03-30T13:57:34.673 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3249000000000001e-05 2024-03-30T13:57:34.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.673 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.673 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:57:34.673 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.661500+0000", 2024-03-30T13:57:34.673 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4272e-05 2024-03-30T13:57:34.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.673 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.673 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:57:34.673 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.661509+0000", 2024-03-30T13:57:34.673 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4779999999999997e-06 2024-03-30T13:57:34.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:57:34.673 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:57:34.674 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:57:34.674 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:57:34.661624+0000", 2024-03-30T13:57:34.674 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000114611 2024-03-30T13:57:34.674 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:57:34.674 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:57:34.674 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:57:34.674 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:57:34.674 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:57:34.674 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 4 2024-03-30T13:57:34.674 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:57:34.742 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:34.784 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:34.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:34.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:34.886 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:34.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:34.929 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:34.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:35.030 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:35.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:35.063 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:35.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:35.159 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T13:57:35.164 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:35.196 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:35.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:35.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:35.199 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:34 smithi043 ceph-mon[29963]: pgmap v649: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 3.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 844 B/s rd, 124 MiB/s wr, 1.98k op/s; 27925/166530 objects degraded (16.769%); 27/166530 objects misplaced (0.016%) 2024-03-30T13:57:35.199 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:34 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:57:35.199 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:34 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:57:35.199 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:34 smithi043 ceph-mon[42099]: pgmap v649: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 3.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 844 B/s rd, 124 MiB/s wr, 1.98k op/s; 27925/166530 objects degraded (16.769%); 27/166530 objects misplaced (0.016%) 2024-03-30T13:57:35.199 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:34 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:57:35.199 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:34 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:57:35.298 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:35.328 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:35.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:35.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:35.368 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 27 16 63423 63407 146.753 98.5625 0.00857863 0.0068087 2024-03-30T13:57:35.431 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:35.462 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:35.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:35.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:35.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:34 smithi113 ceph-mon[34341]: pgmap v649: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 3.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 844 B/s rd, 124 MiB/s wr, 1.98k op/s; 27925/166530 objects degraded (16.769%); 27/166530 objects misplaced (0.016%) 2024-03-30T13:57:35.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:34 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:57:35.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:34 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:57:35.564 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:35.594 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:35.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:35.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:35.696 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:35.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:35.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:35.837 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:35.868 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:35.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:35.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:35.970 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:36.002 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:36.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:36.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:36.104 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:36.139 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:36.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:36.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:36.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:35 smithi043 ceph-mon[29963]: osdmap e305: 8 total, 7 up, 6 in 2024-03-30T13:57:36.144 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:57:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:57:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:57:36.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:35 smithi043 ceph-mon[42099]: osdmap e305: 8 total, 7 up, 6 in 2024-03-30T13:57:36.241 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:36.272 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:36.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:36.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:36.368 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 28 16 65007 64991 145.047 99 0.00809031 0.00688996 2024-03-30T13:57:36.375 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:36.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:36.414 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:36.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:36.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:35 smithi113 ceph-mon[34341]: osdmap e305: 8 total, 7 up, 6 in 2024-03-30T13:57:36.515 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:36.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:36.546 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:36.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:36.647 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:36.677 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:36.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:36.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:36.779 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:36.809 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:36.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:36.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:36.911 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:36.918 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:36.929 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-03-30T13:57:36.940 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:36.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:36.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:37.030 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:37 smithi043 ceph-mon[29963]: pgmap v651: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 852 B/s rd, 116 MiB/s wr, 1.86k op/s; 30635/182724 objects degraded (16.766%); 27/182724 objects misplaced (0.015%) 2024-03-30T13:57:37.030 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:37 smithi043 ceph-mon[42099]: pgmap v651: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 852 B/s rd, 116 MiB/s wr, 1.86k op/s; 30635/182724 objects degraded (16.766%); 27/182724 objects misplaced (0.015%) 2024-03-30T13:57:37.042 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:37.084 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:37.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:37.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:37.186 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:37.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:37.216 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:37.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:37.317 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:37.351 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:37.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:37.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:37.368 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 29 16 66471 66455 143.2 91.5 0.00551426 0.00697746 2024-03-30T13:57:37.453 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:37.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:37 smithi113 ceph-mon[34341]: pgmap v651: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 852 B/s rd, 116 MiB/s wr, 1.86k op/s; 30635/182724 objects degraded (16.766%); 27/182724 objects misplaced (0.015%) 2024-03-30T13:57:37.503 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:37.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:37.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:37.605 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:37.633 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:37.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:37.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:37.735 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:37.765 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:37.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:37.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:37.868 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:37.899 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:37.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:37.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:38.001 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:38.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:38.033 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:38.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:38.122 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:38 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 30635/182724 objects degraded (16.766%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:38.122 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:38 smithi043 ceph-mon[42099]: osdmap e306: 8 total, 7 up, 6 in 2024-03-30T13:57:38.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:38 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 30635/182724 objects degraded (16.766%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:38.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:38 smithi043 ceph-mon[29963]: osdmap e306: 8 total, 7 up, 6 in 2024-03-30T13:57:38.133 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:38.175 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:38.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:38.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:38.277 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:38.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:38.311 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:38.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:38.350 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:38.370 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 30 16 68101 68085 141.815 101.875 0.00966892 0.00704601 2024-03-30T13:57:38.412 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:38.453 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:38.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:38.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:38.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:38 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 30635/182724 objects degraded (16.766%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:38.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:38 smithi113 ceph-mon[34341]: osdmap e306: 8 total, 7 up, 6 in 2024-03-30T13:57:38.555 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:38.590 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:38.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:38.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:38.695 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:38.727 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:38.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:38.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:38.829 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:38.863 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:38.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:38.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:38.965 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:39.005 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:39.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:39.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:39.050 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 reset_purged_snaps_last 2024-03-30T13:57:39.107 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:39.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:39.135 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:39.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:39.236 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:39.267 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:39.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:39.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:39.369 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:39.370 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 31 15 69753 69738 140.573 103.312 0.0168005 0.00710906 2024-03-30T13:57:39.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:39 smithi043 ceph-mon[42099]: pgmap v653: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 3.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 909 B/s rd, 98 MiB/s wr, 1.56k op/s; 31188/185970 objects degraded (16.770%); 27/185970 objects misplaced (0.015%) 2024-03-30T13:57:39.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:39 smithi043 ceph-mon[29963]: pgmap v653: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 3.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 909 B/s rd, 98 MiB/s wr, 1.56k op/s; 31188/185970 objects degraded (16.770%); 27/185970 objects misplaced (0.015%) 2024-03-30T13:57:39.416 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:39.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:39.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:39.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:39 smithi113 ceph-mon[34341]: pgmap v653: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 3.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 909 B/s rd, 98 MiB/s wr, 1.56k op/s; 31188/185970 objects degraded (16.770%); 27/185970 objects misplaced (0.015%) 2024-03-30T13:57:39.467 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:57:39 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:57:39.519 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:39.598 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:39.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:39.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:39.700 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:39.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:39.740 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:39.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:39.842 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:39.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:39.881 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:39.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:39.982 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:40.024 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:40 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:57:40.030 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:40.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:40.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:40.132 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:40.188 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:40.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:40.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:40.279 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:40 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:57:40.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:40.332 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:40 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:57:40.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:40.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:40.370 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 32 16 71293 71277 139.184 96.1875 0.00716006 0.00718022 2024-03-30T13:57:40.446 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:40.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:40.482 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:40.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:40.583 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:40.616 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:40.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:40.628 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:40.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:40.729 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:40.764 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:40.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:40.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:40.866 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:40.910 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:40.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:40.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:41.011 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:41.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:41.055 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:41.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:41.156 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:41.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:41.198 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:41.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:41.299 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:41.302 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:41 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:41.302 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:41 smithi043 ceph-mon[29963]: pgmap v654: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 4.1 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 724 B/s rd, 122 MiB/s wr, 1.95k op/s; 33662/200967 objects degraded (16.750%); 27/200967 objects misplaced (0.013%) 2024-03-30T13:57:41.302 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:41 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:41.302 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:41 smithi043 ceph-mon[42099]: pgmap v654: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 4.1 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 724 B/s rd, 122 MiB/s wr, 1.95k op/s; 33662/200967 objects degraded (16.750%); 27/200967 objects misplaced (0.013%) 2024-03-30T13:57:41.342 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:41.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:41.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:41.370 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 33 16 72899 72883 138.008 100.375 0.00383294 0.00724085 2024-03-30T13:57:41.445 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:41.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:41 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:41.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:41 smithi113 ceph-mon[34341]: pgmap v654: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 4.1 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 724 B/s rd, 122 MiB/s wr, 1.95k op/s; 33662/200967 objects degraded (16.750%); 27/200967 objects misplaced (0.013%) 2024-03-30T13:57:41.475 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:41.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:41.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:41.577 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:41.615 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:41.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:41.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:41.717 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:41.750 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:41.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:41.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:41.852 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:41.876 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:41.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:41.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:41.979 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:42.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:42.017 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:42.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:42.118 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:42.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:42.166 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:42.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:42.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:42 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2952855590' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:57:42.267 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:42.296 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:42.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:42.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:42.370 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 34 16 74344 74328 136.605 90.3125 0.00679103 0.0073158 2024-03-30T13:57:42.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:42 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2952855590' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:57:42.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:42 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2952855590' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:57:42.397 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:42.433 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:42.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:42.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:42.535 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:42.566 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:42.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:42.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:42.668 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:42.706 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:42.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:42.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:42.808 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:42.837 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:42.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:42.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:43.002 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:43.045 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:43.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:43.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:43.147 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:43.182 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:43.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:43.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:43.284 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:43.322 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:43.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:43.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:43.324 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:43 smithi113 ceph-mon[34341]: pgmap v655: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 4.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 35185/209949 objects degraded (16.759%); 27/209949 objects misplaced (0.013%) 2024-03-30T13:57:43.324 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:43 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 35185/209949 objects degraded (16.759%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:43.370 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 35 16 75920 75904 135.516 98.5 0.00748756 0.00737484 2024-03-30T13:57:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:43 smithi043 ceph-mon[42099]: pgmap v655: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 4.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 35185/209949 objects degraded (16.759%); 27/209949 objects misplaced (0.013%) 2024-03-30T13:57:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:43 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 35185/209949 objects degraded (16.759%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:43.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:43 smithi043 ceph-mon[29963]: pgmap v655: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 4.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 35185/209949 objects degraded (16.759%); 27/209949 objects misplaced (0.013%) 2024-03-30T13:57:43.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:43 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 35185/209949 objects degraded (16.759%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:43.424 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:43.452 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:43.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:43.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:43.553 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:43.589 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:57:43.589 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:57:43.589 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:57:43.589 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:57:43.589 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:57:43.589 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:57:43.589 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:57:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:57:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:57:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:57:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:57:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:57:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:57:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:57:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:57:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:43.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:43.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:43.696 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:43.728 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:43.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:43.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:43.830 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:43.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:43.863 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:43.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:43.964 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:43.993 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:43.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:43.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:44.094 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:44.130 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:44.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:44.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:44.231 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:44.266 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:44.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:44.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:44.368 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:44.370 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 36 16 77309 77293 134.163 86.8125 0.00856973 0.00744892 2024-03-30T13:57:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:44.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:44.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:44.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:44 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:44.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:44 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:44.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:44 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:44.494 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:44.534 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:44.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:44.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:44.638 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:44.668 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:44.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:44.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:44.770 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:44.800 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:44.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:44.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:44.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:44.940 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:44.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:44.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:45.042 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:45.081 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:45.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:45.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:45.183 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:45.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:45.213 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:45.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:45.314 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:45.349 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:45.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:45.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:45.371 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 37 16 78684 78668 132.86 85.9375 0.0140084 0.00751922 2024-03-30T13:57:45.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:45 smithi043 ceph-mon[42099]: pgmap v656: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.51k op/s; 36170/215748 objects degraded (16.765%); 27/215748 objects misplaced (0.013%) 2024-03-30T13:57:45.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:45 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:57:45.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:45 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:57:45.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:45 smithi043 ceph-mon[42099]: osdmap e307: 8 total, 7 up, 6 in 2024-03-30T13:57:45.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:45 smithi043 ceph-mon[29963]: pgmap v656: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.51k op/s; 36170/215748 objects degraded (16.765%); 27/215748 objects misplaced (0.013%) 2024-03-30T13:57:45.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:45 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:57:45.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:45 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:57:45.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:45 smithi043 ceph-mon[29963]: osdmap e307: 8 total, 7 up, 6 in 2024-03-30T13:57:45.406 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:45.450 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:45.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:45 smithi113 ceph-mon[34341]: pgmap v656: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.51k op/s; 36170/215748 objects degraded (16.765%); 27/215748 objects misplaced (0.013%) 2024-03-30T13:57:45.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:45 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:57:45.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:45 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:57:45.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:45 smithi113 ceph-mon[34341]: osdmap e307: 8 total, 7 up, 6 in 2024-03-30T13:57:45.481 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:45.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:45.583 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:45.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:45.607 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:45.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:45.708 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:45.740 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:45.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:45.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:45.843 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:45.883 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:45.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:45.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:45.985 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:46.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:46.022 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:46.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:46.123 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:46.164 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:46.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:46.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:46.266 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:46.298 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:46.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:46.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:46.371 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 38 16 80051 80035 131.612 85.4375 0.0071634 0.0075938 2024-03-30T13:57:46.393 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:57:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:57:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:57:46.400 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:46.435 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:46.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:46.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:46.537 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:46.568 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:46.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:46.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:46.670 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:46.699 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:46.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:46.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:46.713 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:57:46.713 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:57:46.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:46.842 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:46.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:46.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:46.944 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:46.986 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:46.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:47.088 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:47.126 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:47.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:47.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:47.227 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:47.262 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:47.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:47.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:47.364 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:47.370 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 39 16 81738 81722 130.94 105.438 0.00945359 0.00763157 2024-03-30T13:57:47.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:47 smithi043 ceph-mon[29963]: pgmap v658: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 4.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 634 B/s rd, 113 MiB/s wr, 1.81k op/s; 38571/229809 objects degraded (16.784%); 27/229809 objects misplaced (0.012%) 2024-03-30T13:57:47.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:47 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:57:47.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:47 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:57:47.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:47 smithi043 ceph-mon[42099]: pgmap v658: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 4.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 634 B/s rd, 113 MiB/s wr, 1.81k op/s; 38571/229809 objects degraded (16.784%); 27/229809 objects misplaced (0.012%) 2024-03-30T13:57:47.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:47 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:57:47.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:47 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:57:47.408 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:47.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:47.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:47.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:47 smithi113 ceph-mon[34341]: pgmap v658: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 4.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 634 B/s rd, 113 MiB/s wr, 1.81k op/s; 38571/229809 objects degraded (16.784%); 27/229809 objects misplaced (0.012%) 2024-03-30T13:57:47.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:47 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:57:47.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:47 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:57:47.510 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:47.549 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:47.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:47.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:47.651 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:47.694 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:47.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:47.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:47.796 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:47.831 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:47.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:47.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:47.934 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:47.977 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:47.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:47.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:48.079 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:48.125 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:48.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:48.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:48.226 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:48.260 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:48.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:48.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:48.362 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:48.371 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-03-30T13:57:48.376045+0000 min lat: 0.00120663 max lat: 0.10669 avg lat: 0.00766846 2024-03-30T13:57:48.371 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T13:57:48.371 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 40 16 83441 83425 130.328 106.438 0.00802164 0.00766846 2024-03-30T13:57:48.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:48 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 38571/229809 objects degraded (16.784%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:48.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:48 smithi043 ceph-mon[29963]: osdmap e308: 8 total, 7 up, 6 in 2024-03-30T13:57:48.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:48 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 38571/229809 objects degraded (16.784%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:48.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:48 smithi043 ceph-mon[42099]: osdmap e308: 8 total, 7 up, 6 in 2024-03-30T13:57:48.394 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:48.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:48.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:48.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:48 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 38571/229809 objects degraded (16.784%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:48.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:48 smithi113 ceph-mon[34341]: osdmap e308: 8 total, 7 up, 6 in 2024-03-30T13:57:48.496 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:48.527 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:48.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:48.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:48.629 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:48.664 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:48.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:48.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:48.715 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:57:48.765 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:48.840 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:48.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:48.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:48.942 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:48.973 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:48.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:48.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:49.075 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:49.109 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:49.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:49.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:49.210 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:49.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:49.244 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:49.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:49.345 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:49.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:49 smithi043 ceph-mon[29963]: pgmap v660: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 4.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 39022/232584 objects degraded (16.778%); 27/232584 objects misplaced (0.012%) 2024-03-30T13:57:49.352 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:49 smithi043 ceph-mon[42099]: pgmap v660: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 4.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 39022/232584 objects degraded (16.778%); 27/232584 objects misplaced (0.012%) 2024-03-30T13:57:49.368 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:49 smithi113 ceph-mon[34341]: pgmap v660: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 4.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 39022/232584 objects degraded (16.778%); 27/232584 objects misplaced (0.012%) 2024-03-30T13:57:49.368 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:57:49 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:57:49.371 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 41 15 85096 85081 129.672 103.5 0.0122225 0.00770738 2024-03-30T13:57:49.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:49.397 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:49.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:49.497 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:49.529 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:49.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:49.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:49.631 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:49.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:49.665 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:49.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:49.766 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:49.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:49.804 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:49.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:49.905 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:49.939 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:49.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:49.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:50.041 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:50.085 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:50.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:50.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:50.188 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:50.227 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:50.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:50.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:50.329 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:50.364 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:50.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:50.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:50.371 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 42 16 86759 86743 129.058 103.875 0.0080959 0.00774416 2024-03-30T13:57:50.466 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:50.502 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:50.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:50.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:50.604 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:50.639 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:50.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:50.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:50.741 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:50.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:50.782 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:50.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:50.883 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:50.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:50.930 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:50.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:51.031 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:51.070 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:51.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:51.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:51.172 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:51.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:51.213 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:51.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:51.267 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:57:51.267 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:57:51.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:51 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:51.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:51 smithi043 ceph-mon[29963]: pgmap v661: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 5.0 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 96 MiB/s wr, 1.54k op/s; 41409/246891 objects degraded (16.772%); 27/246891 objects misplaced (0.011%) 2024-03-30T13:57:51.268 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:51 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:51.268 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:51 smithi043 ceph-mon[42099]: pgmap v661: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 5.0 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 96 MiB/s wr, 1.54k op/s; 41409/246891 objects degraded (16.772%); 27/246891 objects misplaced (0.011%) 2024-03-30T13:57:51.314 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:51.352 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:51.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:51.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:51.366 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:51 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:51.366 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:51 smithi113 ceph-mon[34341]: pgmap v661: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 5.0 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 96 MiB/s wr, 1.54k op/s; 41409/246891 objects degraded (16.772%); 27/246891 objects misplaced (0.011%) 2024-03-30T13:57:51.371 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 43 16 88375 88359 128.405 101 0.00620056 0.00778312 2024-03-30T13:57:51.454 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:51.491 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:51.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:51.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:51.593 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:51.624 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:51.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:51.726 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:51.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:51.758 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:51.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:51.859 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:51.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:51.893 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:51.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:51.995 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:52.027 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:52.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:52.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:52.129 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:52.156 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:52.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:52.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:52.258 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:52.290 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:52.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:52.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:52.372 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 44 15 89972 89957 127.756 99.875 0.00809993 0.00782292 2024-03-30T13:57:52.393 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:52.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:52.425 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:52.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:52.527 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:52.559 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:52.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:52.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:52.661 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:52.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:52.708 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:52.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:52.748 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 reset_purged_snaps_last 2024-03-30T13:57:52.809 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:52.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:52.876 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:52.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:52.978 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:53.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:53.003 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:53.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:53.087 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T13:57:53.104 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:53.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:53.147 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:53.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:53.247 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:53.283 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:53.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:53.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:53.371 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 45 16 91568 91552 127.133 99.6875 0.00516072 0.00786133 2024-03-30T13:57:53.385 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:53.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:53 smithi043 ceph-mon[42099]: pgmap v662: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 42883/256056 objects degraded (16.748%); 27/256056 objects misplaced (0.011%) 2024-03-30T13:57:53.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:53 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3953453142' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:57:53.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:53 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:57:53.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:53 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 42883/256056 objects degraded (16.748%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:53.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:53 smithi043 ceph-mon[29963]: pgmap v662: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 42883/256056 objects degraded (16.748%); 27/256056 objects misplaced (0.011%) 2024-03-30T13:57:53.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:53 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3953453142' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:57:53.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:53 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:57:53.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:53 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 42883/256056 objects degraded (16.748%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:53.424 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:53.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:53.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:53.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:53 smithi113 ceph-mon[34341]: pgmap v662: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 42883/256056 objects degraded (16.748%); 27/256056 objects misplaced (0.011%) 2024-03-30T13:57:53.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:53 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3953453142' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:57:53.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:53 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:57:53.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:53 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 42883/256056 objects degraded (16.748%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:53.526 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:53.564 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:53.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:53.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:53.666 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:53.706 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:53.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:53.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:53.808 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:53.841 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:53.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:53.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:53.887 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-03-30T13:57:53.943 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:54.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:54.001 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:54.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:54.095 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:54.103 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:54.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:54 smithi043 ceph-mon[29963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:57:54.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:54 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:57:54.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:54 smithi043 ceph-mon[29963]: osdmap e309: 8 total, 7 up, 6 in 2024-03-30T13:57:54.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:54 smithi043 ceph-mon[42099]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:57:54.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:54 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:57:54.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:54 smithi043 ceph-mon[42099]: osdmap e309: 8 total, 7 up, 6 in 2024-03-30T13:57:54.179 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:54.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:54.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:54.281 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:54.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:54.326 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:54.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:54.372 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 46 16 93242 93226 126.643 104.625 0.00759928 0.00789192 2024-03-30T13:57:54.427 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:54.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:54 smithi113 ceph-mon[34341]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:57:54.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:54 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:57:54.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:54 smithi113 ceph-mon[34341]: osdmap e309: 8 total, 7 up, 6 in 2024-03-30T13:57:54.467 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:54.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:54.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:54.569 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:54.609 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:54.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:54.711 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:54.745 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:54.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:54.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:54.847 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:54.878 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:54.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:54.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:54.980 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:55.026 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:55.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:55.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:55.128 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:55.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:55 smithi043 ceph-mon[42099]: pgmap v664: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s; 42883/256056 objects degraded (16.748%); 27/256056 objects misplaced (0.011%) 2024-03-30T13:57:55.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:55 smithi043 ceph-mon[29963]: pgmap v664: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s; 42883/256056 objects degraded (16.748%); 27/256056 objects misplaced (0.011%) 2024-03-30T13:57:55.158 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:55.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:55.260 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:55.292 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:55.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:55.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:55.372 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 47 15 94907 94892 126.164 104.125 0.00972763 0.00792131 2024-03-30T13:57:55.394 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:55.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:55.436 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:55.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:55.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:55 smithi113 ceph-mon[34341]: pgmap v664: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s; 42883/256056 objects degraded (16.748%); 27/256056 objects misplaced (0.011%) 2024-03-30T13:57:55.537 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:55.569 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:55.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:55.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:55.671 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:55.708 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:55.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:55.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:55.810 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:55.861 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:55.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:55.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:55.963 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:55.995 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:55.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:55.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:56.097 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:56.130 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:56.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:56.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:56.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:56.272 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:56.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:56.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:56.297 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:56.372 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 48 16 96517 96501 125.629 100.562 0.0100365 0.00795564 2024-03-30T13:57:56.374 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:56.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:56 smithi043 ceph-mon[29963]: 5.9 scrub starts 2024-03-30T13:57:56.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:56 smithi043 ceph-mon[29963]: 5.9 scrub ok 2024-03-30T13:57:56.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:56 smithi043 ceph-mon[42099]: 5.9 scrub starts 2024-03-30T13:57:56.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:56 smithi043 ceph-mon[42099]: 5.9 scrub ok 2024-03-30T13:57:56.394 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:57:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:57:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:57:56.406 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:56.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:56.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:56.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:56 smithi113 ceph-mon[34341]: 5.9 scrub starts 2024-03-30T13:57:56.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:56 smithi113 ceph-mon[34341]: 5.9 scrub ok 2024-03-30T13:57:56.507 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:56.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:56.541 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:56.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:56.642 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:56.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:56.676 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:56.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:56.777 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:56.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:56.810 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:56.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:56.870 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:56.873 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T13:57:56.911 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:56.941 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:56.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:56.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:57.043 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:57.076 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:57.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:57.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:57.126 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:57 smithi043 ceph-mon[42099]: pgmap v665: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 634 B/s rd, 116 MiB/s wr, 1.85k op/s; 46380/277365 objects degraded (16.722%); 27/277365 objects misplaced (0.010%) 2024-03-30T13:57:57.126 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:57 smithi043 ceph-mon[42099]: 5.19 scrub starts 2024-03-30T13:57:57.126 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:57 smithi043 ceph-mon[42099]: 5.19 scrub ok 2024-03-30T13:57:57.126 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:57 smithi043 ceph-mon[42099]: 6.a scrub starts 2024-03-30T13:57:57.126 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:57 smithi043 ceph-mon[42099]: 6.a scrub ok 2024-03-30T13:57:57.126 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:57 smithi043 ceph-mon[29963]: pgmap v665: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 634 B/s rd, 116 MiB/s wr, 1.85k op/s; 46380/277365 objects degraded (16.722%); 27/277365 objects misplaced (0.010%) 2024-03-30T13:57:57.127 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:57 smithi043 ceph-mon[29963]: 5.19 scrub starts 2024-03-30T13:57:57.127 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:57 smithi043 ceph-mon[29963]: 5.19 scrub ok 2024-03-30T13:57:57.127 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:57 smithi043 ceph-mon[29963]: 6.a scrub starts 2024-03-30T13:57:57.127 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:57 smithi043 ceph-mon[29963]: 6.a scrub ok 2024-03-30T13:57:57.178 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:57.233 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:57.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:57.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:57.335 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:57.372 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 49 16 98128 98112 125.12 100.688 0.0135239 0.00798767 2024-03-30T13:57:57.375 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:57.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:57.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:57.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:57 smithi113 ceph-mon[34341]: pgmap v665: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 634 B/s rd, 116 MiB/s wr, 1.85k op/s; 46380/277365 objects degraded (16.722%); 27/277365 objects misplaced (0.010%) 2024-03-30T13:57:57.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:57 smithi113 ceph-mon[34341]: 5.19 scrub starts 2024-03-30T13:57:57.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:57 smithi113 ceph-mon[34341]: 5.19 scrub ok 2024-03-30T13:57:57.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:57 smithi113 ceph-mon[34341]: 6.a scrub starts 2024-03-30T13:57:57.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:57 smithi113 ceph-mon[34341]: 6.a scrub ok 2024-03-30T13:57:57.476 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:57.509 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:57.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:57.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:57.611 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:57.650 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:57.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:57.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:57.752 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:57.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:57.792 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:57.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:57.893 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:57.923 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:57.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:57.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:58.025 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:57:58.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:58.067 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:58.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:57:58.168 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:58.211 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:58.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:58.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:58.313 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:58.340 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:58 smithi043 ceph-mon[29963]: 3.1b scrub starts 2024-03-30T13:57:58.340 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:58 smithi043 ceph-mon[29963]: 3.1b scrub ok 2024-03-30T13:57:58.340 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:58 smithi043 ceph-mon[29963]: 5.15 scrub starts 2024-03-30T13:57:58.340 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:58 smithi043 ceph-mon[29963]: 5.15 scrub ok 2024-03-30T13:57:58.340 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:58 smithi043 ceph-mon[29963]: 6.6 scrub starts 2024-03-30T13:57:58.340 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:58 smithi043 ceph-mon[29963]: 6.6 scrub ok 2024-03-30T13:57:58.340 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:58 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 46380/277365 objects degraded (16.722%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:58.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:58 smithi043 ceph-mon[29963]: osdmap e310: 8 total, 7 up, 6 in 2024-03-30T13:57:58.341 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:58 smithi043 ceph-mon[42099]: 3.1b scrub starts 2024-03-30T13:57:58.341 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:58 smithi043 ceph-mon[42099]: 3.1b scrub ok 2024-03-30T13:57:58.341 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:58 smithi043 ceph-mon[42099]: 5.15 scrub starts 2024-03-30T13:57:58.341 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:58 smithi043 ceph-mon[42099]: 5.15 scrub ok 2024-03-30T13:57:58.341 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:58 smithi043 ceph-mon[42099]: 6.6 scrub starts 2024-03-30T13:57:58.341 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:58 smithi043 ceph-mon[42099]: 6.6 scrub ok 2024-03-30T13:57:58.342 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:58 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 46380/277365 objects degraded (16.722%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:58.342 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:58 smithi043 ceph-mon[42099]: osdmap e310: 8 total, 7 up, 6 in 2024-03-30T13:57:58.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:58.346 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:58.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:58.373 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 50 15 99731 99716 124.621 100.25 0.0119806 0.00802014 2024-03-30T13:57:58.447 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:57:58.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:58 smithi113 ceph-mon[34341]: 3.1b scrub starts 2024-03-30T13:57:58.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:58 smithi113 ceph-mon[34341]: 3.1b scrub ok 2024-03-30T13:57:58.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:58 smithi113 ceph-mon[34341]: 5.15 scrub starts 2024-03-30T13:57:58.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:58 smithi113 ceph-mon[34341]: 5.15 scrub ok 2024-03-30T13:57:58.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:58 smithi113 ceph-mon[34341]: 6.6 scrub starts 2024-03-30T13:57:58.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:58 smithi113 ceph-mon[34341]: 6.6 scrub ok 2024-03-30T13:57:58.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:58 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 46380/277365 objects degraded (16.722%), 15 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-03-30T13:57:58.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:58 smithi113 ceph-mon[34341]: osdmap e310: 8 total, 7 up, 6 in 2024-03-30T13:57:58.479 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:58.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:58.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:57:58.581 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:58.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:58.615 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:58.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:58.683 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:57:58.683 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:57:58.683 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:57:58.683 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:57:58.683 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:57:58.683 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:57:58.684 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:57:58.684 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:57:58.684 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:57:58.684 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:57:58.684 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:57:58.684 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:57:58.684 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:57:58.684 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:57:58.684 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:57:58.716 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:58.748 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:58.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:58.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:58.850 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:58.888 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:58.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:58.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:58.990 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:57:59.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:59.023 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:59.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:57:59.124 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:57:59.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:59.159 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:59.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:57:59.260 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:57:59.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:59.291 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:57:59.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:57:59.373 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 51 16 101459 101443 124.294 107.938 0.00703324 0.0080413 2024-03-30T13:57:59.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:59.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:59 smithi043 ceph-mon[29963]: pgmap v667: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.40k op/s; 46955/280575 objects degraded (16.735%); 27/280575 objects misplaced (0.010%) 2024-03-30T13:57:59.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:59 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1479414489' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:57:59.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:59 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:57:59.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:59 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:57:59.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:57:59 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:59.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:59 smithi043 ceph-mon[42099]: pgmap v667: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.40k op/s; 46955/280575 objects degraded (16.735%); 27/280575 objects misplaced (0.010%) 2024-03-30T13:57:59.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:59 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1479414489' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:57:59.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:59 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:57:59.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:59 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:57:59.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:57:59 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:59.430 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:59.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:59.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:59.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:57:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:57:59.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:59 smithi113 ceph-mon[34341]: pgmap v667: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.40k op/s; 46955/280575 objects degraded (16.735%); 27/280575 objects misplaced (0.010%) 2024-03-30T13:57:59.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:59 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1479414489' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:57:59.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:59 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:57:59.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:59 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:57:59.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:57:59 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:59.531 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:59.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:59.569 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:59.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:59.670 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:57:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:59.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:59.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:57:59.813 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:57:59.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:57:59.859 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:57:59.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:57:59.895 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:57:59.895 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:57:59.961 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:00.010 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:00.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:00.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:00.112 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:00.152 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:00.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:00.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:00.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:00.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:00.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:00.373 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 52 16 103166 103150 123.955 106.688 0.00685827 0.00806341 2024-03-30T13:58:00.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:00.470 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:00.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:00.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:00.572 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:00.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:00.621 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:00.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:00.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:00 smithi043 ceph-mon[29963]: 4.b scrub starts 2024-03-30T13:58:00.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:00 smithi043 ceph-mon[29963]: 4.b scrub ok 2024-03-30T13:58:00.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:00 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:00.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:00 smithi043 ceph-mon[29963]: osdmap e311: 8 total, 7 up, 6 in 2024-03-30T13:58:00.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:00 smithi043 ceph-mon[29963]: pgmap v669: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 779 B/s rd, 126 MiB/s wr, 2.01k op/s; 49428/295647 objects degraded (16.719%); 27/295647 objects misplaced (0.009%) 2024-03-30T13:58:00.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:00 smithi043 ceph-mon[42099]: 4.b scrub starts 2024-03-30T13:58:00.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:00 smithi043 ceph-mon[42099]: 4.b scrub ok 2024-03-30T13:58:00.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:00 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:00.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:00 smithi043 ceph-mon[42099]: osdmap e311: 8 total, 7 up, 6 in 2024-03-30T13:58:00.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:00 smithi043 ceph-mon[42099]: pgmap v669: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 779 B/s rd, 126 MiB/s wr, 2.01k op/s; 49428/295647 objects degraded (16.719%); 27/295647 objects misplaced (0.009%) 2024-03-30T13:58:00.722 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:00.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:00.765 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:00.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:00.859 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:58:00.867 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:00.897 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:00.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:00.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:00.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:00 smithi113 ceph-mon[34341]: 4.b scrub starts 2024-03-30T13:58:00.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:00 smithi113 ceph-mon[34341]: 4.b scrub ok 2024-03-30T13:58:00.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:00 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:00.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:00 smithi113 ceph-mon[34341]: osdmap e311: 8 total, 7 up, 6 in 2024-03-30T13:58:00.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:00 smithi113 ceph-mon[34341]: pgmap v669: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 779 B/s rd, 126 MiB/s wr, 2.01k op/s; 49428/295647 objects degraded (16.719%); 27/295647 objects misplaced (0.009%) 2024-03-30T13:58:00.999 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:01.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:01.047 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:01.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:01.148 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:01.190 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:01.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:01.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:01.292 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:01.324 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:01.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:01.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:01.373 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 53 16 104801 104785 123.544 102.188 0.00790447 0.00809013 2024-03-30T13:58:01.426 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:01.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:01.463 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:01.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:01.565 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:01.611 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:01.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:01.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:01 smithi043 ceph-mon[29963]: 4.3 scrub starts 2024-03-30T13:58:01.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:01 smithi043 ceph-mon[29963]: 4.3 scrub ok 2024-03-30T13:58:01.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:01 smithi043 ceph-mon[29963]: 3.19 scrub starts 2024-03-30T13:58:01.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:01 smithi043 ceph-mon[29963]: 3.19 scrub ok 2024-03-30T13:58:01.681 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:01 smithi043 ceph-mon[42099]: 4.3 scrub starts 2024-03-30T13:58:01.681 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:01 smithi043 ceph-mon[42099]: 4.3 scrub ok 2024-03-30T13:58:01.681 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:01 smithi043 ceph-mon[42099]: 3.19 scrub starts 2024-03-30T13:58:01.681 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:01 smithi043 ceph-mon[42099]: 3.19 scrub ok 2024-03-30T13:58:01.712 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:01.739 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:01.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:01.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:01.841 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:01.877 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:01.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:01.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:01.897 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:58:01.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:01 smithi113 ceph-mon[34341]: 4.3 scrub starts 2024-03-30T13:58:01.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:01 smithi113 ceph-mon[34341]: 4.3 scrub ok 2024-03-30T13:58:01.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:01 smithi113 ceph-mon[34341]: 3.19 scrub starts 2024-03-30T13:58:01.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:01 smithi113 ceph-mon[34341]: 3.19 scrub ok 2024-03-30T13:58:01.979 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:58:01.979 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-03-30T13:58:01.979 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-03-30T13:58:01.979 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-03-30T13:58:01.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.979 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:236062 7.b 7:d7a6a88c:::benchmark_data_smithi043_86989_object113724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-30T13:58:01.979 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:48:29.619701+0000", 2024-03-30T13:58:01.979 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 572.06305253899995, 2024-03-30T13:58:01.980 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.246167516, 2024-03-30T13:58:01.980 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:01.980 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:01.980 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:01.980 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:01.980 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:58:01.980 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:58:01.980 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 236062 2024-03-30T13:58:01.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.980 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:01.980 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.980 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:01.980 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.619701+0000", 2024-03-30T13:58:01.981 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:01.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.981 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:01.981 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.619701+0000", 2024-03-30T13:58:01.981 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:01.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.981 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:01.981 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.619703+0000", 2024-03-30T13:58:01.981 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4300000000000001e-06 2024-03-30T13:58:01.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.982 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:01.982 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.619708+0000", 2024-03-30T13:58:01.982 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4770000000000002e-06 2024-03-30T13:58:01.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.982 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:01.982 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.619720+0000", 2024-03-30T13:58:01.982 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2303000000000001e-05 2024-03-30T13:58:01.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.982 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:01.982 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.619729+0000", 2024-03-30T13:58:01.982 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8009999999999993e-06 2024-03-30T13:58:01.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.983 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:01.983 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.714050+0000", 2024-03-30T13:58:01.983 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.094320483999999996 2024-03-30T13:58:01.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.983 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:01.983 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.714208+0000", 2024-03-30T13:58:01.983 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015793400000000001 2024-03-30T13:58:01.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.983 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 5,6", 2024-03-30T13:58:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.714349+0000", 2024-03-30T13:58:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000141101 2024-03-30T13:58:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.718720+0000", 2024-03-30T13:58:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0043713199999999997 2024-03-30T13:58:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:01.985 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.722094+0000", 2024-03-30T13:58:01.985 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003373975 2024-03-30T13:58:01.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.985 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:01.985 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.865731+0000", 2024-03-30T13:58:01.985 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.143637242 2024-03-30T13:58:01.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.985 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.865802+0000", 2024-03-30T13:58:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1017000000000001e-05 2024-03-30T13:58:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.865869+0000", 2024-03-30T13:58:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6432000000000004e-05 2024-03-30T13:58:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:236064 7.b 7:d891dc63:::benchmark_data_smithi043_86989_object113726:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-30T13:58:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:48:29.622368+0000", 2024-03-30T13:58:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 572.06038619200001, 2024-03-30T13:58:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.24362557500000001, 2024-03-30T13:58:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:58:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:58:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 236064 2024-03-30T13:58:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.622368+0000", 2024-03-30T13:58:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.622368+0000", 2024-03-30T13:58:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:01.989 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.622369+0000", 2024-03-30T13:58:01.989 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0690000000000001e-06 2024-03-30T13:58:01.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.989 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:01.989 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.622373+0000", 2024-03-30T13:58:01.989 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.211e-06 2024-03-30T13:58:01.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.990 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:01.990 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.622382+0000", 2024-03-30T13:58:01.990 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2699999999999993e-06 2024-03-30T13:58:01.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.991 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:01.991 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.622389+0000", 2024-03-30T13:58:01.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4500000000000001e-06 2024-03-30T13:58:01.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.991 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:01.991 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.718748+0000", 2024-03-30T13:58:01.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.096359108999999998 2024-03-30T13:58:01.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.992 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:01.992 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.718886+0000", 2024-03-30T13:58:01.992 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013824400000000001 2024-03-30T13:58:01.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.992 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 5,6", 2024-03-30T13:58:01.992 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.719011+0000", 2024-03-30T13:58:01.992 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000124778 2024-03-30T13:58:01.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.992 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:01.992 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.722131+0000", 2024-03-30T13:58:01.992 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0031203770000000001 2024-03-30T13:58:01.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.993 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:01.993 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.725341+0000", 2024-03-30T13:58:01.993 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00321032 2024-03-30T13:58:01.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.993 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:01.993 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.865881+0000", 2024-03-30T13:58:01.993 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.140540006 2024-03-30T13:58:01.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.993 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:01.993 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.865938+0000", 2024-03-30T13:58:01.994 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7009999999999998e-05 2024-03-30T13:58:01.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.994 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:01.994 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.865993+0000", 2024-03-30T13:58:01.994 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4731000000000002e-05 2024-03-30T13:58:01.994 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:01.994 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:01.994 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:01.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.994 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14817.0:236083 7.b 7:d41e9be5:::benchmark_data_smithi043_86989_object113745:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-30T13:58:01.994 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:48:29.641804+0000", 2024-03-30T13:58:01.994 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 572.04095003600003, 2024-03-30T13:58:01.995 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.22430096899999999, 2024-03-30T13:58:01.996 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:01.996 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:01.996 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:01.997 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:01.997 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14817", 2024-03-30T13:58:01.997 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1026309455", 2024-03-30T13:58:01.997 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 236083 2024-03-30T13:58:01.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.998 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:01.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.998 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:01.998 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.641804+0000", 2024-03-30T13:58:01.998 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:01.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.998 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:01.998 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.641804+0000", 2024-03-30T13:58:01.998 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:01.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.999 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:01.999 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.641805+0000", 2024-03-30T13:58:01.999 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5090000000000001e-06 2024-03-30T13:58:01.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.999 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:01.999 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.641810+0000", 2024-03-30T13:58:01.999 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.515e-06 2024-03-30T13:58:01.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:01.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:01.999 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:01.999 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.641819+0000", 2024-03-30T13:58:01.999 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.6339999999999996e-06 2024-03-30T13:58:01.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.000 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:02.000 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.641826+0000", 2024-03-30T13:58:02.000 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5420000000000002e-06 2024-03-30T13:58:02.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.722158+0000", 2024-03-30T13:58:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.080332039999999993 2024-03-30T13:58:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.722287+0000", 2024-03-30T13:58:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012925499999999999 2024-03-30T13:58:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 5,6", 2024-03-30T13:58:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.722424+0000", 2024-03-30T13:58:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000137231 2024-03-30T13:58:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.731771+0000", 2024-03-30T13:58:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0093462770000000001 2024-03-30T13:58:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.731811+0000", 2024-03-30T13:58:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0679000000000001e-05 2024-03-30T13:58:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.866000+0000", 2024-03-30T13:58:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13418876299999999 2024-03-30T13:58:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.866056+0000", 2024-03-30T13:58:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6119999999999998e-05 2024-03-30T13:58:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:48:29.866105+0000", 2024-03-30T13:58:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8403999999999999e-05 2024-03-30T13:58:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61015 9.d 9:bd835e52:::benchmark_data_smithi043_145506_object61014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:58:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.849021+0000", 2024-03-30T13:58:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 266.83373264400001, 2024-03-30T13:58:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2177895259999998, 2024-03-30T13:58:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-03-30T13:58:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61015 2024-03-30T13:58:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849021+0000", 2024-03-30T13:58:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849021+0000", 2024-03-30T13:58:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849022+0000", 2024-03-30T13:58:02.006 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.254e-06 2024-03-30T13:58:02.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.006 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:02.006 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849056+0000", 2024-03-30T13:58:02.006 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4051000000000003e-05 2024-03-30T13:58:02.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.006 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:02.006 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849067+0000", 2024-03-30T13:58:02.006 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0173000000000001e-05 2024-03-30T13:58:02.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.007 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:02.007 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849230+0000", 2024-03-30T13:58:02.007 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000163086 2024-03-30T13:58:02.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.007 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.007 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854000+0000", 2024-03-30T13:58:02.007 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.004770552 2024-03-30T13:58:02.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.007 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.007 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854011+0000", 2024-03-30T13:58:02.007 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0679000000000001e-05 2024-03-30T13:58:02.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.008 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.008 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845267+0000", 2024-03-30T13:58:02.008 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99125657899999997 2024-03-30T13:58:02.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.008 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.008 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845280+0000", 2024-03-30T13:58:02.008 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2313999999999999e-05 2024-03-30T13:58:02.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.008 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.008 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.065485+0000", 2024-03-30T13:58:02.008 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.220205703 2024-03-30T13:58:02.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.009 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:02.009 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.066811+0000", 2024-03-30T13:58:02.009 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0013251350000000001 2024-03-30T13:58:02.009 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.009 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:02.009 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.009 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61042 9.d 9:bd1a2220:::benchmark_data_smithi043_145506_object61041:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:58:02.009 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.849245+0000", 2024-03-30T13:58:02.009 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 266.83350840700001, 2024-03-30T13:58:02.009 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4398415450000002, 2024-03-30T13:58:02.009 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:02.009 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:02.010 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:02.010 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:02.010 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:02.010 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:02.010 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61042 2024-03-30T13:58:02.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.010 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:02.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.010 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:02.010 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849245+0000", 2024-03-30T13:58:02.010 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.011 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:02.011 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849245+0000", 2024-03-30T13:58:02.011 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.011 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:02.011 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849246+0000", 2024-03-30T13:58:02.011 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9999999999999997e-07 2024-03-30T13:58:02.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.011 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:02.011 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849272+0000", 2024-03-30T13:58:02.011 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5956999999999999e-05 2024-03-30T13:58:02.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.012 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:02.012 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849281+0000", 2024-03-30T13:58:02.012 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2089999999999994e-06 2024-03-30T13:58:02.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.012 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:02.012 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849288+0000", 2024-03-30T13:58:02.012 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8920000000000003e-06 2024-03-30T13:58:02.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.012 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.012 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854033+0000", 2024-03-30T13:58:02.012 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0047449299999999996 2024-03-30T13:58:02.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.013 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.013 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854040+0000", 2024-03-30T13:58:02.013 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.737e-06 2024-03-30T13:58:02.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.013 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.013 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845318+0000", 2024-03-30T13:58:02.013 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.991278561 2024-03-30T13:58:02.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.013 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.013 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845327+0000", 2024-03-30T13:58:02.013 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.166e-06 2024-03-30T13:58:02.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.014 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.014 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.233130+0000", 2024-03-30T13:58:02.014 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.38780266 2024-03-30T13:58:02.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.014 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:02.014 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.233697+0000", 2024-03-30T13:58:02.014 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00056653400000000005 2024-03-30T13:58:02.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.014 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T13:58:02.014 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.233994+0000", 2024-03-30T13:58:02.015 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00029792300000000001 2024-03-30T13:58:02.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.015 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.015 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.257518+0000", 2024-03-30T13:58:02.015 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.023523143 2024-03-30T13:58:02.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.015 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.015 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.257567+0000", 2024-03-30T13:58:02.015 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8928999999999999e-05 2024-03-30T13:58:02.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.015 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:02.015 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.286372+0000", 2024-03-30T13:58:02.016 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.028805173 2024-03-30T13:58:02.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.016 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:02.016 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.286429+0000", 2024-03-30T13:58:02.016 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7568000000000002e-05 2024-03-30T13:58:02.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.016 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:02.016 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.289087+0000", 2024-03-30T13:58:02.016 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026575629999999999 2024-03-30T13:58:02.016 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.016 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:02.016 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.017 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61052 9.d 9:bc7be88d:::benchmark_data_smithi043_145506_object61051:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:58:02.017 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.849296+0000", 2024-03-30T13:58:02.017 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 266.83345803100002, 2024-03-30T13:58:02.017 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4426081989999999, 2024-03-30T13:58:02.017 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:02.017 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:02.017 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:02.017 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:02.017 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:02.017 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:02.017 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61052 2024-03-30T13:58:02.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.018 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:02.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.018 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:02.018 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849296+0000", 2024-03-30T13:58:02.018 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.018 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:02.018 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849296+0000", 2024-03-30T13:58:02.018 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.018 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:02.018 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849296+0000", 2024-03-30T13:58:02.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2e-07 2024-03-30T13:58:02.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.019 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:02.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849427+0000", 2024-03-30T13:58:02.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013042800000000001 2024-03-30T13:58:02.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.019 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:02.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849436+0000", 2024-03-30T13:58:02.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2010000000000005e-06 2024-03-30T13:58:02.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.019 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:02.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849443+0000", 2024-03-30T13:58:02.020 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1339999999999998e-06 2024-03-30T13:58:02.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.020 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854060+0000", 2024-03-30T13:58:02.020 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0046171880000000004 2024-03-30T13:58:02.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.020 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854067+0000", 2024-03-30T13:58:02.020 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.72e-06 2024-03-30T13:58:02.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.021 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845359+0000", 2024-03-30T13:58:02.021 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99129192799999999 2024-03-30T13:58:02.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.021 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.021 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845367+0000", 2024-03-30T13:58:02.021 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1249999999999993e-06 2024-03-30T13:58:02.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.021 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.021 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.250580+0000", 2024-03-30T13:58:02.021 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4052128429999999 2024-03-30T13:58:02.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.021 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:02.022 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.250754+0000", 2024-03-30T13:58:02.022 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017425599999999999 2024-03-30T13:58:02.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.022 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T13:58:02.022 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.250985+0000", 2024-03-30T13:58:02.022 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00023110900000000001 2024-03-30T13:58:02.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.022 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.022 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.257611+0000", 2024-03-30T13:58:02.022 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0066258840000000003 2024-03-30T13:58:02.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.023 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.023 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.257656+0000", 2024-03-30T13:58:02.023 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5240000000000001e-05 2024-03-30T13:58:02.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.023 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:02.023 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.289106+0000", 2024-03-30T13:58:02.023 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.031449612000000002 2024-03-30T13:58:02.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.023 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:02.023 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.289162+0000", 2024-03-30T13:58:02.023 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6039999999999999e-05 2024-03-30T13:58:02.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.291904+0000", 2024-03-30T13:58:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002742071 2024-03-30T13:58:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61062 9.d 9:be3b0b08:::benchmark_data_smithi043_145506_object61061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:58:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.849454+0000", 2024-03-30T13:58:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 266.83329976499999, 2024-03-30T13:58:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4481625989999998, 2024-03-30T13:58:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:02.024 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61062 2024-03-30T13:58:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849454+0000", 2024-03-30T13:58:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.025 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849454+0000", 2024-03-30T13:58:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849455+0000", 2024-03-30T13:58:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5499999999999998e-07 2024-03-30T13:58:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849502+0000", 2024-03-30T13:58:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7818999999999997e-05 2024-03-30T13:58:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849510+0000", 2024-03-30T13:58:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4189999999999996e-06 2024-03-30T13:58:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849516+0000", 2024-03-30T13:58:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9309999999999996e-06 2024-03-30T13:58:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854101+0000", 2024-03-30T13:58:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0045853860000000003 2024-03-30T13:58:02.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854108+0000", 2024-03-30T13:58:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4070000000000004e-06 2024-03-30T13:58:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845395+0000", 2024-03-30T13:58:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99128673099999998 2024-03-30T13:58:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845404+0000", 2024-03-30T13:58:02.028 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4540000000000007e-06 2024-03-30T13:58:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.257690+0000", 2024-03-30T13:58:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4122865120000001 2024-03-30T13:58:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.257880+0000", 2024-03-30T13:58:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000190309 2024-03-30T13:58:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T13:58:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.258100+0000", 2024-03-30T13:58:02.029 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00021951300000000001 2024-03-30T13:58:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.260161+0000", 2024-03-30T13:58:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0020608380000000002 2024-03-30T13:58:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.276325+0000", 2024-03-30T13:58:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.016164380999999999 2024-03-30T13:58:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:02.030 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.291984+0000", 2024-03-30T13:58:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.015658436000000001 2024-03-30T13:58:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.293663+0000", 2024-03-30T13:58:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0016796319999999999 2024-03-30T13:58:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.297617+0000", 2024-03-30T13:58:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0039532760000000004 2024-03-30T13:58:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:02.031 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61080 9.d 9:bc5be45f:::benchmark_data_smithi043_145506_object61079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:58:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.849523+0000", 2024-03-30T13:58:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 266.83323046300001, 2024-03-30T13:58:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4527278949999998, 2024-03-30T13:58:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61080 2024-03-30T13:58:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.032 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849523+0000", 2024-03-30T13:58:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849523+0000", 2024-03-30T13:58:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:02.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849524+0000", 2024-03-30T13:58:02.034 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4400000000000001e-07 2024-03-30T13:58:02.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.034 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:02.034 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849566+0000", 2024-03-30T13:58:02.034 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1857000000000003e-05 2024-03-30T13:58:02.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.034 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:02.034 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850156+0000", 2024-03-30T13:58:02.034 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00059000000000000003 2024-03-30T13:58:02.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.035 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:02.035 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850167+0000", 2024-03-30T13:58:02.035 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1904e-05 2024-03-30T13:58:02.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.035 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.035 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854131+0000", 2024-03-30T13:58:02.035 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0039639139999999998 2024-03-30T13:58:02.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.035 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.035 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854139+0000", 2024-03-30T13:58:02.035 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3780000000000003e-06 2024-03-30T13:58:02.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.036 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.036 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845432+0000", 2024-03-30T13:58:02.036 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99129344100000005 2024-03-30T13:58:02.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.036 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.036 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845441+0000", 2024-03-30T13:58:02.036 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2839999999999995e-06 2024-03-30T13:58:02.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.036 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.036 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.260197+0000", 2024-03-30T13:58:02.036 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.414755948 2024-03-30T13:58:02.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.037 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:02.037 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.260372+0000", 2024-03-30T13:58:02.037 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000174864 2024-03-30T13:58:02.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.037 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T13:58:02.037 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.260584+0000", 2024-03-30T13:58:02.037 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000212031 2024-03-30T13:58:02.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.037 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.037 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.276519+0000", 2024-03-30T13:58:02.037 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.015934308000000001 2024-03-30T13:58:02.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.038 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.038 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.276685+0000", 2024-03-30T13:58:02.038 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000166238 2024-03-30T13:58:02.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.038 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:02.038 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.297637+0000", 2024-03-30T13:58:02.038 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.020952028000000001 2024-03-30T13:58:02.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.038 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:02.039 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.297704+0000", 2024-03-30T13:58:02.039 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7503999999999996e-05 2024-03-30T13:58:02.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.039 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:02.039 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.302251+0000", 2024-03-30T13:58:02.039 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0045467520000000003 2024-03-30T13:58:02.039 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.039 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:02.039 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.039 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61090 9.d 9:b53a3867:::benchmark_data_smithi043_145506_object61089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:58:02.039 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850180+0000", 2024-03-30T13:58:02.039 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 266.83257359800001, 2024-03-30T13:58:02.040 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4542226359999999, 2024-03-30T13:58:02.040 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:02.040 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:02.040 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:02.040 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:02.040 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:02.040 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:02.040 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61090 2024-03-30T13:58:02.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.040 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:02.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.040 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:02.040 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850180+0000", 2024-03-30T13:58:02.040 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.041 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:02.041 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850180+0000", 2024-03-30T13:58:02.041 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.041 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:02.041 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850181+0000", 2024-03-30T13:58:02.041 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4099999999999998e-07 2024-03-30T13:58:02.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.041 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:02.041 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850236+0000", 2024-03-30T13:58:02.042 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5337999999999998e-05 2024-03-30T13:58:02.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.042 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:02.042 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850245+0000", 2024-03-30T13:58:02.042 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0699999999999996e-06 2024-03-30T13:58:02.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.042 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:02.042 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850251+0000", 2024-03-30T13:58:02.042 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8819999999999998e-06 2024-03-30T13:58:02.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.042 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.043 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854162+0000", 2024-03-30T13:58:02.043 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0039110409999999996 2024-03-30T13:58:02.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.043 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.043 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854169+0000", 2024-03-30T13:58:02.043 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2869999999999999e-06 2024-03-30T13:58:02.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.043 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.043 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845471+0000", 2024-03-30T13:58:02.043 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99130147599999996 2024-03-30T13:58:02.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.043 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.044 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845479+0000", 2024-03-30T13:58:02.044 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3119999999999994e-06 2024-03-30T13:58:02.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.044 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.044 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.044 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.266915+0000", 2024-03-30T13:58:02.044 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.421435687 2024-03-30T13:58:02.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.044 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.044 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:02.044 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.267087+0000", 2024-03-30T13:58:02.044 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017207799999999999 2024-03-30T13:58:02.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.044 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.045 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T13:58:02.045 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.267313+0000", 2024-03-30T13:58:02.045 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00022567400000000001 2024-03-30T13:58:02.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.045 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.045 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.045 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.276854+0000", 2024-03-30T13:58:02.045 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0095411190000000007 2024-03-30T13:58:02.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.045 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.045 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.045 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.277083+0000", 2024-03-30T13:58:02.045 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00022885599999999999 2024-03-30T13:58:02.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.045 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.046 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:02.046 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.302266+0000", 2024-03-30T13:58:02.046 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.025182942999999999 2024-03-30T13:58:02.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.046 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.046 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:02.046 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.302375+0000", 2024-03-30T13:58:02.046 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000109098 2024-03-30T13:58:02.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.046 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.046 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:02.046 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.304403+0000", 2024-03-30T13:58:02.046 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002028134 2024-03-30T13:58:02.046 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.047 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:02.047 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.047 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.047 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61125 9.d 9:bd41aaa2:::benchmark_data_smithi043_145506_object61124:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:58:02.047 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850259+0000", 2024-03-30T13:58:02.047 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 266.83249428699997, 2024-03-30T13:58:02.047 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4559017920000001, 2024-03-30T13:58:02.047 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:02.047 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:02.047 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:02.047 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:02.047 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:02.047 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:02.048 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61125 2024-03-30T13:58:02.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.048 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:02.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.048 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:02.048 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850259+0000", 2024-03-30T13:58:02.048 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.048 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:02.049 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850259+0000", 2024-03-30T13:58:02.049 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.049 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.049 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:02.049 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850260+0000", 2024-03-30T13:58:02.049 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.39e-07 2024-03-30T13:58:02.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.049 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.049 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:02.049 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850303+0000", 2024-03-30T13:58:02.049 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3279999999999999e-05 2024-03-30T13:58:02.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.049 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.049 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:02.050 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850310+0000", 2024-03-30T13:58:02.050 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0589999999999997e-06 2024-03-30T13:58:02.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.050 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:02.050 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850316+0000", 2024-03-30T13:58:02.050 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3859999999999998e-06 2024-03-30T13:58:02.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.050 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.050 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854192+0000", 2024-03-30T13:58:02.050 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003876759 2024-03-30T13:58:02.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.051 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.051 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854199+0000", 2024-03-30T13:58:02.051 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.109e-06 2024-03-30T13:58:02.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.051 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.051 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845507+0000", 2024-03-30T13:58:02.051 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99130747100000005 2024-03-30T13:58:02.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.051 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.051 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845515+0000", 2024-03-30T13:58:02.051 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9079999999999995e-06 2024-03-30T13:58:02.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.052 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.052 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.277120+0000", 2024-03-30T13:58:02.052 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4316047009999999 2024-03-30T13:58:02.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.052 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:02.052 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.277293+0000", 2024-03-30T13:58:02.052 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017354 2024-03-30T13:58:02.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.053 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T13:58:02.053 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.277471+0000", 2024-03-30T13:58:02.053 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017787799999999999 2024-03-30T13:58:02.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.053 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.053 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.291965+0000", 2024-03-30T13:58:02.053 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.014493852999999999 2024-03-30T13:58:02.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.053 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:02.053 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.304419+0000", 2024-03-30T13:58:02.053 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.012453697 2024-03-30T13:58:02.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.054 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.054 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.304796+0000", 2024-03-30T13:58:02.054 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00037768200000000001 2024-03-30T13:58:02.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.054 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:02.054 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.304833+0000", 2024-03-30T13:58:02.054 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7119000000000002e-05 2024-03-30T13:58:02.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.054 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:02.054 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.306161+0000", 2024-03-30T13:58:02.055 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001327911 2024-03-30T13:58:02.055 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.055 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:02.055 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.055 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.055 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61137 9.d 9:b639fb2d:::benchmark_data_smithi043_145506_object61136:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:58:02.055 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850323+0000", 2024-03-30T13:58:02.055 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 266.83243080300002, 2024-03-30T13:58:02.055 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4574525619999998, 2024-03-30T13:58:02.055 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:02.055 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:02.055 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:02.055 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:02.055 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:02.056 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:02.056 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61137 2024-03-30T13:58:02.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.056 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:02.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.056 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:02.056 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850323+0000", 2024-03-30T13:58:02.056 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.056 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:02.057 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850323+0000", 2024-03-30T13:58:02.057 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.057 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:02.057 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850323+0000", 2024-03-30T13:58:02.057 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.65e-07 2024-03-30T13:58:02.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.057 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:02.057 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850366+0000", 2024-03-30T13:58:02.057 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2586000000000002e-05 2024-03-30T13:58:02.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.058 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:02.058 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850373+0000", 2024-03-30T13:58:02.058 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9079999999999998e-06 2024-03-30T13:58:02.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.058 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:02.058 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850378+0000", 2024-03-30T13:58:02.058 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6860000000000003e-06 2024-03-30T13:58:02.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.058 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.058 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854221+0000", 2024-03-30T13:58:02.058 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0038425569999999999 2024-03-30T13:58:02.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.059 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.059 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854228+0000", 2024-03-30T13:58:02.059 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.861e-06 2024-03-30T13:58:02.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.059 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.059 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845543+0000", 2024-03-30T13:58:02.059 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99131543300000002 2024-03-30T13:58:02.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.059 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.059 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845552+0000", 2024-03-30T13:58:02.059 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3520000000000007e-06 2024-03-30T13:58:02.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.060 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.060 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.279430+0000", 2024-03-30T13:58:02.060 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4338782210000001 2024-03-30T13:58:02.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.060 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:02.060 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.279622+0000", 2024-03-30T13:58:02.060 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00019243299999999999 2024-03-30T13:58:02.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.060 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T13:58:02.060 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.280952+0000", 2024-03-30T13:58:02.061 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0013300860000000001 2024-03-30T13:58:02.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.061 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:02.061 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.304430+0000", 2024-03-30T13:58:02.061 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.023477554000000001 2024-03-30T13:58:02.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.061 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.061 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.304490+0000", 2024-03-30T13:58:02.061 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0494000000000001e-05 2024-03-30T13:58:02.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.062 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.062 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.306238+0000", 2024-03-30T13:58:02.062 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001747342 2024-03-30T13:58:02.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.062 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:02.062 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.306342+0000", 2024-03-30T13:58:02.062 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010447500000000001 2024-03-30T13:58:02.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.062 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:02.062 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.307775+0000", 2024-03-30T13:58:02.062 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001433209 2024-03-30T13:58:02.063 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.063 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:02.063 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.063 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61140 9.d 9:b67c2d0c:::benchmark_data_smithi043_145506_object61139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:58:02.063 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850385+0000", 2024-03-30T13:58:02.063 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 266.83236829200001, 2024-03-30T13:58:02.063 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4594105769999999, 2024-03-30T13:58:02.063 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:02.063 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:02.063 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:02.063 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:02.063 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:02.063 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:02.064 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61140 2024-03-30T13:58:02.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.064 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:02.064 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.064 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:02.064 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850385+0000", 2024-03-30T13:58:02.064 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.064 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.064 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:02.065 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850385+0000", 2024-03-30T13:58:02.065 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.065 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.065 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:02.065 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850386+0000", 2024-03-30T13:58:02.065 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4999999999999998e-07 2024-03-30T13:58:02.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.065 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.065 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:02.065 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850432+0000", 2024-03-30T13:58:02.065 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6236999999999998e-05 2024-03-30T13:58:02.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.065 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.065 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:02.066 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850439+0000", 2024-03-30T13:58:02.066 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7530000000000004e-06 2024-03-30T13:58:02.066 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.066 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.066 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:02.066 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850444+0000", 2024-03-30T13:58:02.066 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.169e-06 2024-03-30T13:58:02.066 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.066 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.066 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.066 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854251+0000", 2024-03-30T13:58:02.066 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003806781 2024-03-30T13:58:02.066 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.067 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.067 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854258+0000", 2024-03-30T13:58:02.067 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4000000000000003e-06 2024-03-30T13:58:02.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.067 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.067 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845580+0000", 2024-03-30T13:58:02.067 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99132166899999996 2024-03-30T13:58:02.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.067 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.067 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845588+0000", 2024-03-30T13:58:02.067 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3669999999999997e-06 2024-03-30T13:58:02.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.068 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.068 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.283446+0000", 2024-03-30T13:58:02.068 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4378582849999999 2024-03-30T13:58:02.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.068 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:02.068 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.283622+0000", 2024-03-30T13:58:02.068 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017582700000000001 2024-03-30T13:58:02.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.068 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T13:58:02.068 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.283835+0000", 2024-03-30T13:58:02.069 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00021268799999999999 2024-03-30T13:58:02.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.069 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:02.069 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.304605+0000", 2024-03-30T13:58:02.069 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.020769823999999999 2024-03-30T13:58:02.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.069 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.069 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.304753+0000", 2024-03-30T13:58:02.069 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014781700000000001 2024-03-30T13:58:02.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.069 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.070 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.307918+0000", 2024-03-30T13:58:02.070 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0031653660000000002 2024-03-30T13:58:02.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.070 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:02.070 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.308027+0000", 2024-03-30T13:58:02.070 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000108789 2024-03-30T13:58:02.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.070 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:02.070 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.309796+0000", 2024-03-30T13:58:02.070 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001769255 2024-03-30T13:58:02.070 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.070 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:02.071 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.071 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61145 9.d 9:ba09f9de:::benchmark_data_smithi043_145506_object61144:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:58:02.071 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850451+0000", 2024-03-30T13:58:02.071 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 266.832303002, 2024-03-30T13:58:02.071 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4658662370000002, 2024-03-30T13:58:02.071 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:02.071 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:02.071 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:02.071 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:02.071 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:02.071 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:02.071 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61145 2024-03-30T13:58:02.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.072 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:02.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.072 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:02.072 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850451+0000", 2024-03-30T13:58:02.072 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.072 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:02.072 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850451+0000", 2024-03-30T13:58:02.072 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.072 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:02.072 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850451+0000", 2024-03-30T13:58:02.073 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7e-07 2024-03-30T13:58:02.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.073 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:02.073 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850494+0000", 2024-03-30T13:58:02.073 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2991000000000002e-05 2024-03-30T13:58:02.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.073 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:02.073 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850501+0000", 2024-03-30T13:58:02.073 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8310000000000003e-06 2024-03-30T13:58:02.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.073 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:02.074 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850506+0000", 2024-03-30T13:58:02.074 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.485e-06 2024-03-30T13:58:02.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.074 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.074 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854282+0000", 2024-03-30T13:58:02.074 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0037756550000000002 2024-03-30T13:58:02.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.074 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.074 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854290+0000", 2024-03-30T13:58:02.074 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5490000000000001e-06 2024-03-30T13:58:02.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.075 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.075 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845616+0000", 2024-03-30T13:58:02.075 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.991326343 2024-03-30T13:58:02.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.075 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.075 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845624+0000", 2024-03-30T13:58:02.075 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9410000000000007e-06 2024-03-30T13:58:02.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.075 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.075 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.309839+0000", 2024-03-30T13:58:02.075 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4642154810000001 2024-03-30T13:58:02.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.076 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:02.076 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.310036+0000", 2024-03-30T13:58:02.076 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000196687 2024-03-30T13:58:02.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.076 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T13:58:02.076 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.310262+0000", 2024-03-30T13:58:02.076 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00022566700000000001 2024-03-30T13:58:02.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.076 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.076 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.314606+0000", 2024-03-30T13:58:02.076 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0043444149999999999 2024-03-30T13:58:02.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.077 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.077 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.314671+0000", 2024-03-30T13:58:02.077 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4560000000000005e-05 2024-03-30T13:58:02.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.077 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:02.077 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.314689+0000", 2024-03-30T13:58:02.077 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8709999999999999e-05 2024-03-30T13:58:02.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.077 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:02.077 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.314804+0000", 2024-03-30T13:58:02.077 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000114811 2024-03-30T13:58:02.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.078 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:02.078 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.316317+0000", 2024-03-30T13:58:02.078 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001512741 2024-03-30T13:58:02.078 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.078 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:02.078 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.078 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61147 9.d 9:b6510d4c:::benchmark_data_smithi043_145506_object61146:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:58:02.078 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850513+0000", 2024-03-30T13:58:02.078 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 266.83224064199999, 2024-03-30T13:58:02.078 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4759084520000001, 2024-03-30T13:58:02.079 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:02.079 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:02.079 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:02.079 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:02.079 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:02.079 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:02.079 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61147 2024-03-30T13:58:02.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.079 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:02.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.079 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:02.079 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850513+0000", 2024-03-30T13:58:02.079 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.080 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:02.080 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850513+0000", 2024-03-30T13:58:02.080 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.080 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:02.080 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850513+0000", 2024-03-30T13:58:02.080 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4499999999999998e-07 2024-03-30T13:58:02.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.080 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:02.080 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850554+0000", 2024-03-30T13:58:02.080 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0902999999999998e-05 2024-03-30T13:58:02.081 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.081 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.081 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:02.081 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850561+0000", 2024-03-30T13:58:02.081 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3280000000000002e-06 2024-03-30T13:58:02.081 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.081 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.081 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:02.081 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850566+0000", 2024-03-30T13:58:02.081 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9219999999999996e-06 2024-03-30T13:58:02.081 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.081 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.081 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.081 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854312+0000", 2024-03-30T13:58:02.082 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0037467680000000001 2024-03-30T13:58:02.082 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.082 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.082 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.082 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854320+0000", 2024-03-30T13:58:02.082 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3150000000000003e-06 2024-03-30T13:58:02.082 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.082 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.082 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.082 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845655+0000", 2024-03-30T13:58:02.082 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99133570199999999 2024-03-30T13:58:02.082 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.082 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.082 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.083 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845664+0000", 2024-03-30T13:58:02.083 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.9020000000000005e-06 2024-03-30T13:58:02.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.083 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.083 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.083 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.316361+0000", 2024-03-30T13:58:02.083 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4706968279999999 2024-03-30T13:58:02.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.083 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.083 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:02.083 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.316547+0000", 2024-03-30T13:58:02.083 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000186059 2024-03-30T13:58:02.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.083 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.084 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T13:58:02.084 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.316770+0000", 2024-03-30T13:58:02.084 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000223119 2024-03-30T13:58:02.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.084 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.084 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.084 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.321387+0000", 2024-03-30T13:58:02.084 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0046164140000000001 2024-03-30T13:58:02.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.084 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.084 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:02.084 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.321405+0000", 2024-03-30T13:58:02.084 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8094999999999999e-05 2024-03-30T13:58:02.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.084 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.085 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.085 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.324536+0000", 2024-03-30T13:58:02.085 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0031307269999999998 2024-03-30T13:58:02.085 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.085 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.085 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:02.085 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.324576+0000", 2024-03-30T13:58:02.085 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0374999999999999e-05 2024-03-30T13:58:02.085 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.085 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.085 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:02.085 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.326422+0000", 2024-03-30T13:58:02.085 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0018456499999999999 2024-03-30T13:58:02.085 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.086 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:02.086 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.086 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.086 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.086 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61150 9.d 9:baf726bb:::benchmark_data_smithi043_145506_object61149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:58:02.086 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850572+0000", 2024-03-30T13:58:02.086 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 266.83218140600002, 2024-03-30T13:58:02.086 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4862311400000001, 2024-03-30T13:58:02.086 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:02.086 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:02.086 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:02.086 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:02.086 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:02.086 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:02.086 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61150 2024-03-30T13:58:02.087 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.087 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:02.087 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.087 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:02.087 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850572+0000", 2024-03-30T13:58:02.087 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.087 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.087 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.087 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:02.087 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850572+0000", 2024-03-30T13:58:02.087 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.087 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.087 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.087 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:02.088 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850573+0000", 2024-03-30T13:58:02.088 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4900000000000001e-07 2024-03-30T13:58:02.088 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.088 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.088 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:02.088 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850613+0000", 2024-03-30T13:58:02.088 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0605000000000002e-05 2024-03-30T13:58:02.088 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.088 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.088 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:02.088 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850620+0000", 2024-03-30T13:58:02.088 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4749999999999998e-06 2024-03-30T13:58:02.088 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.088 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.088 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:02.089 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850625+0000", 2024-03-30T13:58:02.089 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0259999999999998e-06 2024-03-30T13:58:02.089 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.089 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.089 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.089 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854345+0000", 2024-03-30T13:58:02.089 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00372048 2024-03-30T13:58:02.089 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.089 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.089 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.089 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854353+0000", 2024-03-30T13:58:02.090 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3549999999999999e-06 2024-03-30T13:58:02.090 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.090 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.090 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.090 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845694+0000", 2024-03-30T13:58:02.090 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99134108600000004 2024-03-30T13:58:02.090 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.090 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.090 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.090 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845702+0000", 2024-03-30T13:58:02.090 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0770000000000008e-06 2024-03-30T13:58:02.090 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.090 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.090 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.091 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.318701+0000", 2024-03-30T13:58:02.091 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4729991899999999 2024-03-30T13:58:02.091 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.091 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.091 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:02.091 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.318816+0000", 2024-03-30T13:58:02.091 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011537899999999999 2024-03-30T13:58:02.091 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.091 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.091 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T13:58:02.092 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.318948+0000", 2024-03-30T13:58:02.092 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000132037 2024-03-30T13:58:02.092 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.092 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.092 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:02.092 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.321413+0000", 2024-03-30T13:58:02.092 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024647940000000002 2024-03-30T13:58:02.092 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.092 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.092 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.092 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.324424+0000", 2024-03-30T13:58:02.092 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003011208 2024-03-30T13:58:02.092 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.092 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.092 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.093 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.334395+0000", 2024-03-30T13:58:02.093 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0099709849999999999 2024-03-30T13:58:02.093 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.093 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.093 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:02.093 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.334467+0000", 2024-03-30T13:58:02.093 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2028999999999998e-05 2024-03-30T13:58:02.093 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.093 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.093 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:02.093 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.336803+0000", 2024-03-30T13:58:02.093 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002336065 2024-03-30T13:58:02.093 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.093 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:02.094 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.094 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.094 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.094 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61169 9.d 9:b3588fe6:::benchmark_data_smithi043_145506_object61168:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:58:02.094 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850631+0000", 2024-03-30T13:58:02.094 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 266.83212256100001, 2024-03-30T13:58:02.094 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4878158689999998, 2024-03-30T13:58:02.094 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:02.094 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:02.094 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:02.094 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:02.094 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:02.094 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:02.094 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61169 2024-03-30T13:58:02.095 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.095 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:02.095 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.095 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:02.095 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850631+0000", 2024-03-30T13:58:02.095 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.095 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.095 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.095 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:02.095 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850631+0000", 2024-03-30T13:58:02.095 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.095 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.095 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.095 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:02.096 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850632+0000", 2024-03-30T13:58:02.096 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4900000000000001e-07 2024-03-30T13:58:02.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.096 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.096 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:02.096 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850678+0000", 2024-03-30T13:58:02.096 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6363999999999999e-05 2024-03-30T13:58:02.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.096 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.096 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:02.096 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850684+0000", 2024-03-30T13:58:02.096 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5289999999999997e-06 2024-03-30T13:58:02.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.096 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.096 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:02.097 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850689+0000", 2024-03-30T13:58:02.097 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8060000000000002e-06 2024-03-30T13:58:02.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.097 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.097 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.097 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854375+0000", 2024-03-30T13:58:02.097 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0036859369999999998 2024-03-30T13:58:02.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.097 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.097 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.098 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854383+0000", 2024-03-30T13:58:02.098 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.43e-06 2024-03-30T13:58:02.098 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.098 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.098 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.098 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845730+0000", 2024-03-30T13:58:02.098 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99134741999999998 2024-03-30T13:58:02.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.099 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.099 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.099 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845738+0000", 2024-03-30T13:58:02.099 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1599999999999998e-06 2024-03-30T13:58:02.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.099 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.099 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.099 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.319984+0000", 2024-03-30T13:58:02.099 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4742460390000001 2024-03-30T13:58:02.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.099 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.099 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:02.099 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.320135+0000", 2024-03-30T13:58:02.100 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015038000000000001 2024-03-30T13:58:02.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.100 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.100 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T13:58:02.100 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.320335+0000", 2024-03-30T13:58:02.100 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00020085799999999999 2024-03-30T13:58:02.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.100 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.100 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:02.100 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.324379+0000", 2024-03-30T13:58:02.100 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0040431310000000002 2024-03-30T13:58:02.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.100 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.100 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.324479+0000", 2024-03-30T13:58:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000100487 2024-03-30T13:58:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.336884+0000", 2024-03-30T13:58:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.012405143 2024-03-30T13:58:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.336961+0000", 2024-03-30T13:58:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7127999999999997e-05 2024-03-30T13:58:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.101 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.338447+0000", 2024-03-30T13:58:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0014857080000000001 2024-03-30T13:58:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61174 9.d 9:b691a112:::benchmark_data_smithi043_145506_object61173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:58:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850696+0000", 2024-03-30T13:58:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 266.832057744, 2024-03-30T13:58:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4896642509999998, 2024-03-30T13:58:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:02.102 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61174 2024-03-30T13:58:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850696+0000", 2024-03-30T13:58:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:02.103 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850696+0000", 2024-03-30T13:58:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850696+0000", 2024-03-30T13:58:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3599999999999999e-07 2024-03-30T13:58:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850738+0000", 2024-03-30T13:58:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1473000000000001e-05 2024-03-30T13:58:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:02.104 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850744+0000", 2024-03-30T13:58:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6379999999999998e-06 2024-03-30T13:58:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850749+0000", 2024-03-30T13:58:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8690000000000003e-06 2024-03-30T13:58:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854408+0000", 2024-03-30T13:58:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0036582339999999998 2024-03-30T13:58:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.105 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854415+0000", 2024-03-30T13:58:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.537e-06 2024-03-30T13:58:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845768+0000", 2024-03-30T13:58:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99135278699999996 2024-03-30T13:58:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845776+0000", 2024-03-30T13:58:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3100000000000001e-06 2024-03-30T13:58:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.106 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.321437+0000", 2024-03-30T13:58:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.475661302 2024-03-30T13:58:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.321525+0000", 2024-03-30T13:58:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7298000000000003e-05 2024-03-30T13:58:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T13:58:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.321632+0000", 2024-03-30T13:58:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000107734 2024-03-30T13:58:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.107 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.108 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:02.108 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.324496+0000", 2024-03-30T13:58:02.108 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0028631220000000001 2024-03-30T13:58:02.108 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.108 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.108 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.108 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.326464+0000", 2024-03-30T13:58:02.108 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0019687020000000001 2024-03-30T13:58:02.108 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.108 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.108 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.108 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.338506+0000", 2024-03-30T13:58:02.108 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.012041530999999999 2024-03-30T13:58:02.108 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.109 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.109 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:02.109 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.338570+0000", 2024-03-30T13:58:02.109 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4144000000000001e-05 2024-03-30T13:58:02.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.109 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.109 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:02.109 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.340360+0000", 2024-03-30T13:58:02.109 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0017902339999999999 2024-03-30T13:58:02.109 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.109 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:02.109 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.109 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.110 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61190 9.d 9:ba843ea7:::benchmark_data_smithi043_145506_object61189:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:58:02.110 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850756+0000", 2024-03-30T13:58:02.110 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 266.83199798800001, 2024-03-30T13:58:02.110 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4909397279999999, 2024-03-30T13:58:02.110 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:02.110 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:02.110 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:02.110 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:02.110 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:02.110 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:02.110 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61190 2024-03-30T13:58:02.110 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.110 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:02.110 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.111 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:02.111 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850756+0000", 2024-03-30T13:58:02.111 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.111 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.111 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:02.111 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850756+0000", 2024-03-30T13:58:02.111 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.111 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.111 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:02.111 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850756+0000", 2024-03-30T13:58:02.111 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4499999999999998e-07 2024-03-30T13:58:02.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.112 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:02.112 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850801+0000", 2024-03-30T13:58:02.112 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4878e-05 2024-03-30T13:58:02.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.112 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:02.112 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850808+0000", 2024-03-30T13:58:02.112 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.72e-06 2024-03-30T13:58:02.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.112 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:02.112 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850813+0000", 2024-03-30T13:58:02.112 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9389999999999996e-06 2024-03-30T13:58:02.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.113 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.113 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854440+0000", 2024-03-30T13:58:02.113 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003627052 2024-03-30T13:58:02.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.113 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.113 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854447+0000", 2024-03-30T13:58:02.113 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7049999999999999e-06 2024-03-30T13:58:02.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.113 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.113 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845805+0000", 2024-03-30T13:58:02.113 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99135801000000001 2024-03-30T13:58:02.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.114 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.114 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845814+0000", 2024-03-30T13:58:02.114 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4840000000000004e-06 2024-03-30T13:58:02.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.114 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.114 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.323021+0000", 2024-03-30T13:58:02.114 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.477207148 2024-03-30T13:58:02.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.114 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:02.114 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.323113+0000", 2024-03-30T13:58:02.115 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1707999999999997e-05 2024-03-30T13:58:02.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.115 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T13:58:02.115 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.323248+0000", 2024-03-30T13:58:02.115 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013484 2024-03-30T13:58:02.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.115 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:02.115 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.326474+0000", 2024-03-30T13:58:02.115 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0032268739999999998 2024-03-30T13:58:02.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.115 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.115 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.326510+0000", 2024-03-30T13:58:02.116 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5444000000000001e-05 2024-03-30T13:58:02.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.116 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.116 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.340411+0000", 2024-03-30T13:58:02.116 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.013900859999999999 2024-03-30T13:58:02.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.116 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:02.116 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.340451+0000", 2024-03-30T13:58:02.116 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.032e-05 2024-03-30T13:58:02.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.116 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:02.117 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.341695+0000", 2024-03-30T13:58:02.117 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001244401 2024-03-30T13:58:02.117 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.117 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:02.117 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.117 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.117 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61214 9.d 9:bfa3a72f:::benchmark_data_smithi043_145506_object61213:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:58:02.117 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850819+0000", 2024-03-30T13:58:02.117 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 266.83193433000002, 2024-03-30T13:58:02.117 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4922764219999998, 2024-03-30T13:58:02.117 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:02.117 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:02.117 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:02.117 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:02.118 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:02.118 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:02.118 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61214 2024-03-30T13:58:02.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.118 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:02.118 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.118 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:02.118 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850819+0000", 2024-03-30T13:58:02.118 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.118 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.118 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:02.118 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850819+0000", 2024-03-30T13:58:02.118 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.119 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.119 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:02.119 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850820+0000", 2024-03-30T13:58:02.119 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2300000000000002e-07 2024-03-30T13:58:02.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.119 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.119 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:02.119 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850877+0000", 2024-03-30T13:58:02.119 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6759000000000003e-05 2024-03-30T13:58:02.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.119 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.119 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:02.119 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850884+0000", 2024-03-30T13:58:02.119 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5619999999999998e-06 2024-03-30T13:58:02.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.120 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.120 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:02.120 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850890+0000", 2024-03-30T13:58:02.120 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4750000000000001e-06 2024-03-30T13:58:02.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.120 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.120 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.120 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854471+0000", 2024-03-30T13:58:02.120 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0035816419999999999 2024-03-30T13:58:02.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.120 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.120 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.120 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854479+0000", 2024-03-30T13:58:02.121 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4660000000000002e-06 2024-03-30T13:58:02.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.121 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.121 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.121 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.847038+0000", 2024-03-30T13:58:02.121 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99255926400000005 2024-03-30T13:58:02.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.121 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.121 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.121 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.847052+0000", 2024-03-30T13:58:02.121 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.447e-05 2024-03-30T13:58:02.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.121 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.121 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.122 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.326536+0000", 2024-03-30T13:58:02.122 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4794833030000001 2024-03-30T13:58:02.122 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.122 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.122 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:02.122 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.326641+0000", 2024-03-30T13:58:02.122 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000105073 2024-03-30T13:58:02.122 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.122 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.122 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T13:58:02.122 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.326758+0000", 2024-03-30T13:58:02.122 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000116749 2024-03-30T13:58:02.122 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.122 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.123 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:02.123 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.328416+0000", 2024-03-30T13:58:02.123 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001658358 2024-03-30T13:58:02.123 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.123 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.123 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.123 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.334339+0000", 2024-03-30T13:58:02.123 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0059233610000000003 2024-03-30T13:58:02.123 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.123 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.123 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:02.123 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.341763+0000", 2024-03-30T13:58:02.123 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0074236479999999997 2024-03-30T13:58:02.123 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.123 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.124 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:02.124 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.341826+0000", 2024-03-30T13:58:02.124 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3196999999999998e-05 2024-03-30T13:58:02.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.124 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.124 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:02.124 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.343096+0000", 2024-03-30T13:58:02.124 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0012696719999999999 2024-03-30T13:58:02.124 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.124 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:02.124 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.124 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.124 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14535.0:637 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:58:02.125 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:35.634326+0000", 2024-03-30T13:58:02.125 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 266.04842794699999, 2024-03-30T13:58:02.125 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.25576208900000003, 2024-03-30T13:58:02.125 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:02.125 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:02.125 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T13:58:02.125 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:02.125 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14535", 2024-03-30T13:58:02.125 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.113:0/4128665673", 2024-03-30T13:58:02.125 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 637 2024-03-30T13:58:02.125 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.125 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:02.125 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.125 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:02.125 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.634326+0000", 2024-03-30T13:58:02.126 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.126 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.126 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.126 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:02.126 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.634326+0000", 2024-03-30T13:58:02.126 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:02.126 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.126 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.126 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:02.126 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.634327+0000", 2024-03-30T13:58:02.126 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.218e-06 2024-03-30T13:58:02.126 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.126 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.126 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:02.127 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.634332+0000", 2024-03-30T13:58:02.127 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6410000000000001e-06 2024-03-30T13:58:02.127 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.127 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.127 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:02.127 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.634339+0000", 2024-03-30T13:58:02.127 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1389999999999998e-06 2024-03-30T13:58:02.127 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.127 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.127 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:02.127 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.634345+0000", 2024-03-30T13:58:02.127 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5749999999999997e-06 2024-03-30T13:58:02.127 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.127 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.127 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.128 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.634393+0000", 2024-03-30T13:58:02.128 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8028999999999997e-05 2024-03-30T13:58:02.128 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.128 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.128 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:02.128 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.634401+0000", 2024-03-30T13:58:02.128 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4970000000000004e-06 2024-03-30T13:58:02.128 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.128 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.128 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:02.128 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.889778+0000", 2024-03-30T13:58:02.128 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.25537702600000001 2024-03-30T13:58:02.128 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.128 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.129 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:02.129 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.889935+0000", 2024-03-30T13:58:02.129 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015727199999999999 2024-03-30T13:58:02.129 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:02.129 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:02.129 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:02.129 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.890088+0000", 2024-03-30T13:58:02.129 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015269200000000001 2024-03-30T13:58:02.129 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.129 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:02.129 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.129 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:02.129 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:02.129 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:58:02.130 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:02.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:02.177 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:02.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:02.278 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:02.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:02.313 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:02.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:02.374 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 54 16 106198 106182 122.872 87.3125 0.00457281 0.00813361 2024-03-30T13:58:02.414 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:02.447 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:02.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:02.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:02.549 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:02.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:02.599 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:02.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:02.700 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:02.739 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:02.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:02.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:02.842 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:02.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:02.883 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:02.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:02.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:02 smithi043 ceph-mon[29963]: 1.3 scrub starts 2024-03-30T13:58:02.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:02 smithi043 ceph-mon[29963]: 1.3 scrub ok 2024-03-30T13:58:02.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:02 smithi043 ceph-mon[29963]: pgmap v670: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 6.2 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.04k op/s; 50968/304992 objects degraded (16.711%); 27/304992 objects misplaced (0.009%) 2024-03-30T13:58:02.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:02 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 49428/295647 objects degraded (16.719%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:02.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:02 smithi043 ceph-mon[29963]: osdmap e312: 8 total, 7 up, 6 in 2024-03-30T13:58:02.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:02 smithi043 ceph-mon[42099]: 1.3 scrub starts 2024-03-30T13:58:02.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:02 smithi043 ceph-mon[42099]: 1.3 scrub ok 2024-03-30T13:58:02.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:02 smithi043 ceph-mon[42099]: pgmap v670: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 6.2 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.04k op/s; 50968/304992 objects degraded (16.711%); 27/304992 objects misplaced (0.009%) 2024-03-30T13:58:02.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:02 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 49428/295647 objects degraded (16.719%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:02.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:02 smithi043 ceph-mon[42099]: osdmap e312: 8 total, 7 up, 6 in 2024-03-30T13:58:02.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:02 smithi113 ceph-mon[34341]: 1.3 scrub starts 2024-03-30T13:58:02.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:02 smithi113 ceph-mon[34341]: 1.3 scrub ok 2024-03-30T13:58:02.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:02 smithi113 ceph-mon[34341]: pgmap v670: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 6.2 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.04k op/s; 50968/304992 objects degraded (16.711%); 27/304992 objects misplaced (0.009%) 2024-03-30T13:58:02.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:02 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 49428/295647 objects degraded (16.719%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:02.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:02 smithi113 ceph-mon[34341]: osdmap e312: 8 total, 7 up, 6 in 2024-03-30T13:58:02.984 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:03.016 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:03.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:03.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:03.119 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:03.152 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:03.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:03.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:03.254 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:03.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:03.289 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:03.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:03.326 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-30T13:58:03.373 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 55 16 107857 107841 122.523 103.688 0.0066318 0.0081571 2024-03-30T13:58:03.390 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:03.438 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:03.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:03.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:03.540 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:03.573 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:03.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:03.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:03.675 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:03.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:03.708 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:03.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:03.809 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:03.839 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:03.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:03.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:03.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:03 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:58:03.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:03 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:58:03.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:03 smithi043 ceph-mon[29963]: 5.17 scrub starts 2024-03-30T13:58:03.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:03 smithi043 ceph-mon[29963]: 5.17 scrub ok 2024-03-30T13:58:03.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:03 smithi043 ceph-mon[29963]: 6.3 scrub starts 2024-03-30T13:58:03.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:03 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:58:03.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:03 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:58:03.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:03 smithi043 ceph-mon[42099]: 5.17 scrub starts 2024-03-30T13:58:03.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:03 smithi043 ceph-mon[42099]: 5.17 scrub ok 2024-03-30T13:58:03.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:03 smithi043 ceph-mon[42099]: 6.3 scrub starts 2024-03-30T13:58:03.941 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:03.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:03 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:58:03.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:03 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:58:03.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:03 smithi113 ceph-mon[34341]: 5.17 scrub starts 2024-03-30T13:58:03.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:03 smithi113 ceph-mon[34341]: 5.17 scrub ok 2024-03-30T13:58:03.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:03 smithi113 ceph-mon[34341]: 6.3 scrub starts 2024-03-30T13:58:03.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:03.987 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:03.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:04.088 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:04.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:04.120 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:04.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:04.221 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:04.267 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:04.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:04.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:04.369 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:04.374 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 56 16 109334 109318 121.983 92.3125 0.0114903 0.00819357 2024-03-30T13:58:04.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:04.411 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:04.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:04.512 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:04.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:04.558 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:04.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:04.659 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:04.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:04.689 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:04.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:04.721 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:04 smithi043 ceph-mon[29963]: 5.12 scrub starts 2024-03-30T13:58:04.722 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:04 smithi043 ceph-mon[29963]: 5.12 scrub ok 2024-03-30T13:58:04.722 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:04 smithi043 ceph-mon[29963]: 6.3 scrub ok 2024-03-30T13:58:04.722 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:04 smithi043 ceph-mon[29963]: pgmap v672: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 844 B/s rd, 105 MiB/s wr, 1.68k op/s; 52016/311202 objects degraded (16.715%); 27/311202 objects misplaced (0.009%) 2024-03-30T13:58:04.722 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:04 smithi043 ceph-mon[29963]: 6.5 scrub starts 2024-03-30T13:58:04.722 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:04 smithi043 ceph-mon[29963]: 6.5 scrub ok 2024-03-30T13:58:04.723 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:04 smithi043 ceph-mon[42099]: 5.12 scrub starts 2024-03-30T13:58:04.723 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:04 smithi043 ceph-mon[42099]: 5.12 scrub ok 2024-03-30T13:58:04.723 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:04 smithi043 ceph-mon[42099]: 6.3 scrub ok 2024-03-30T13:58:04.723 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:04 smithi043 ceph-mon[42099]: pgmap v672: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 844 B/s rd, 105 MiB/s wr, 1.68k op/s; 52016/311202 objects degraded (16.715%); 27/311202 objects misplaced (0.009%) 2024-03-30T13:58:04.723 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:04 smithi043 ceph-mon[42099]: 6.5 scrub starts 2024-03-30T13:58:04.724 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:04 smithi043 ceph-mon[42099]: 6.5 scrub ok 2024-03-30T13:58:04.790 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:04.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:04.824 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:04.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:04.925 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:04.955 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:04.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:04.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:04.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:04 smithi113 ceph-mon[34341]: 5.12 scrub starts 2024-03-30T13:58:04.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:04 smithi113 ceph-mon[34341]: 5.12 scrub ok 2024-03-30T13:58:04.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:04 smithi113 ceph-mon[34341]: 6.3 scrub ok 2024-03-30T13:58:04.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:04 smithi113 ceph-mon[34341]: pgmap v672: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 844 B/s rd, 105 MiB/s wr, 1.68k op/s; 52016/311202 objects degraded (16.715%); 27/311202 objects misplaced (0.009%) 2024-03-30T13:58:04.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:04 smithi113 ceph-mon[34341]: 6.5 scrub starts 2024-03-30T13:58:04.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:04 smithi113 ceph-mon[34341]: 6.5 scrub ok 2024-03-30T13:58:05.057 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:05.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:05.112 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:05.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:05.213 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:05.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:05.255 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:05.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:05.356 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:05.374 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 57 16 110638 110622 121.273 81.5 0.0149278 0.00824154 2024-03-30T13:58:05.388 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:05.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:05.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:05.463 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:58:05.490 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:05.521 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:05.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:05.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:05.623 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:05.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:05.653 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:05.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:05.755 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:05.795 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:05.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:05.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:05.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:05 smithi043 ceph-mon[29963]: 3.12 scrub starts 2024-03-30T13:58:05.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:05 smithi043 ceph-mon[29963]: 3.12 scrub ok 2024-03-30T13:58:05.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:05 smithi043 ceph-mon[29963]: 2.2 scrub starts 2024-03-30T13:58:05.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:05 smithi043 ceph-mon[29963]: 2.2 scrub ok 2024-03-30T13:58:05.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:05 smithi043 ceph-mon[29963]: 3.3 scrub starts 2024-03-30T13:58:05.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:05 smithi043 ceph-mon[29963]: 3.3 scrub ok 2024-03-30T13:58:05.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:05 smithi043 ceph-mon[29963]: 1.a scrub starts 2024-03-30T13:58:05.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:05 smithi043 ceph-mon[29963]: 1.a scrub ok 2024-03-30T13:58:05.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:05 smithi043 ceph-mon[42099]: 3.12 scrub starts 2024-03-30T13:58:05.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:05 smithi043 ceph-mon[42099]: 3.12 scrub ok 2024-03-30T13:58:05.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:05 smithi043 ceph-mon[42099]: 2.2 scrub starts 2024-03-30T13:58:05.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:05 smithi043 ceph-mon[42099]: 2.2 scrub ok 2024-03-30T13:58:05.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:05 smithi043 ceph-mon[42099]: 3.3 scrub starts 2024-03-30T13:58:05.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:05 smithi043 ceph-mon[42099]: 3.3 scrub ok 2024-03-30T13:58:05.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:05 smithi043 ceph-mon[42099]: 1.a scrub starts 2024-03-30T13:58:05.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:05 smithi043 ceph-mon[42099]: 1.a scrub ok 2024-03-30T13:58:05.896 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:05.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:05.931 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:05.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:05.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:05 smithi113 ceph-mon[34341]: 3.12 scrub starts 2024-03-30T13:58:05.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:05 smithi113 ceph-mon[34341]: 3.12 scrub ok 2024-03-30T13:58:05.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:05 smithi113 ceph-mon[34341]: 2.2 scrub starts 2024-03-30T13:58:05.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:05 smithi113 ceph-mon[34341]: 2.2 scrub ok 2024-03-30T13:58:05.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:05 smithi113 ceph-mon[34341]: 3.3 scrub starts 2024-03-30T13:58:05.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:05 smithi113 ceph-mon[34341]: 3.3 scrub ok 2024-03-30T13:58:05.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:05 smithi113 ceph-mon[34341]: 1.a scrub starts 2024-03-30T13:58:05.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:05 smithi113 ceph-mon[34341]: 1.a scrub ok 2024-03-30T13:58:06.032 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:06.064 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:06.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:06.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:06.139 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:58:06.167 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:06.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:06.220 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:06.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:06.241 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:58:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:58:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:58:06.321 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:06.360 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:06.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:06.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:06.374 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 58 16 112177 112161 120.84 96.1875 0.0118352 0.00827096 2024-03-30T13:58:06.462 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:06.497 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:06.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:06.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:06.599 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:06.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:06.634 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:06.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:06.643 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-03-30T13:58:06.735 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:06.772 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:06.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:06.866 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:06 smithi043 ceph-mon[29963]: pgmap v673: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 6.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 639 B/s rd, 118 MiB/s wr, 1.89k op/s; 54480/325881 objects degraded (16.718%); 27/325881 objects misplaced (0.008%) 2024-03-30T13:58:06.866 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:06 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3821375906' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:58:06.866 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:06 smithi043 ceph-mon[42099]: pgmap v673: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 6.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 639 B/s rd, 118 MiB/s wr, 1.89k op/s; 54480/325881 objects degraded (16.718%); 27/325881 objects misplaced (0.008%) 2024-03-30T13:58:06.866 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:06 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3821375906' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:58:06.874 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:06.917 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:06.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:06.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:06.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:06 smithi113 ceph-mon[34341]: pgmap v673: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 6.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 639 B/s rd, 118 MiB/s wr, 1.89k op/s; 54480/325881 objects degraded (16.718%); 27/325881 objects misplaced (0.008%) 2024-03-30T13:58:06.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:06 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3821375906' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:58:07.019 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:07.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:07.055 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:07.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:07.156 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:07.210 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:07.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:07.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:07.312 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:07.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:07.352 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:07.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:07.374 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 59 16 113897 113881 120.614 107.5 0.00905072 0.00828663 2024-03-30T13:58:07.453 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:07.489 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:07.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:07.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:07.591 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:07.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:07.615 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:07.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:07.716 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:07.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:07.751 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:07.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:07.852 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:07.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:07.885 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:07.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:07.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:07 smithi043 ceph-mon[29963]: 5.5 scrub starts 2024-03-30T13:58:07.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:07 smithi043 ceph-mon[29963]: 5.5 scrub ok 2024-03-30T13:58:07.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:07 smithi043 ceph-mon[29963]: 5.e scrub starts 2024-03-30T13:58:07.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:07 smithi043 ceph-mon[29963]: 5.e scrub ok 2024-03-30T13:58:07.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:07 smithi043 ceph-mon[29963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:58:07.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:07 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3821375906' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:58:07.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:07 smithi043 ceph-mon[29963]: osdmap e313: 8 total, 7 up, 6 in 2024-03-30T13:58:07.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:07 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 54480/325881 objects degraded (16.718%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:07.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:07 smithi043 ceph-mon[29963]: osdmap e314: 8 total, 7 up, 6 in 2024-03-30T13:58:07.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:07 smithi043 ceph-mon[42099]: 5.5 scrub starts 2024-03-30T13:58:07.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:07 smithi043 ceph-mon[42099]: 5.5 scrub ok 2024-03-30T13:58:07.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:07 smithi043 ceph-mon[42099]: 5.e scrub starts 2024-03-30T13:58:07.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:07 smithi043 ceph-mon[42099]: 5.e scrub ok 2024-03-30T13:58:07.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:07 smithi043 ceph-mon[42099]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:58:07.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:07 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3821375906' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:58:07.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:07 smithi043 ceph-mon[42099]: osdmap e313: 8 total, 7 up, 6 in 2024-03-30T13:58:07.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:07 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 54480/325881 objects degraded (16.718%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:07.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:07 smithi043 ceph-mon[42099]: osdmap e314: 8 total, 7 up, 6 in 2024-03-30T13:58:07.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:07 smithi113 ceph-mon[34341]: 5.5 scrub starts 2024-03-30T13:58:07.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:07 smithi113 ceph-mon[34341]: 5.5 scrub ok 2024-03-30T13:58:07.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:07 smithi113 ceph-mon[34341]: 5.e scrub starts 2024-03-30T13:58:07.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:07 smithi113 ceph-mon[34341]: 5.e scrub ok 2024-03-30T13:58:07.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:07 smithi113 ceph-mon[34341]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:58:07.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:07 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3821375906' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:58:07.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:07 smithi113 ceph-mon[34341]: osdmap e313: 8 total, 7 up, 6 in 2024-03-30T13:58:07.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:07 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 54480/325881 objects degraded (16.718%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:07.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:07 smithi113 ceph-mon[34341]: osdmap e314: 8 total, 7 up, 6 in 2024-03-30T13:58:07.987 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:08.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:08.012 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:08.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:08.114 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:08.167 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:08.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:08.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:08.268 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:08.309 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:08.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:08.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:08.374 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-03-30T13:58:08.379822+0000 min lat: 0.00120663 max lat: 0.143119 avg lat: 0.00831599 2024-03-30T13:58:08.375 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T13:58:08.375 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 60 16 115414 115398 120.184 94.8125 0.00833342 0.00831599 2024-03-30T13:58:08.411 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:08.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:08.446 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:08.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:08.548 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:08.582 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:08.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:08.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:08.684 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:08.727 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:08.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:08.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:08.829 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:08.864 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:08.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:08.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:08.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:08 smithi043 ceph-mon[29963]: pgmap v676: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 6.7 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 852 B/s rd, 82 MiB/s wr, 1.32k op/s; 54933/328731 objects degraded (16.711%); 27/328731 objects misplaced (0.008%) 2024-03-30T13:58:08.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:08 smithi043 ceph-mon[29963]: 4.8 scrub starts 2024-03-30T13:58:08.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:08 smithi043 ceph-mon[29963]: 4.8 scrub ok 2024-03-30T13:58:08.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:08 smithi043 ceph-mon[29963]: 6.1b scrub starts 2024-03-30T13:58:08.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:08 smithi043 ceph-mon[42099]: pgmap v676: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 6.7 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 852 B/s rd, 82 MiB/s wr, 1.32k op/s; 54933/328731 objects degraded (16.711%); 27/328731 objects misplaced (0.008%) 2024-03-30T13:58:08.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:08 smithi043 ceph-mon[42099]: 4.8 scrub starts 2024-03-30T13:58:08.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:08 smithi043 ceph-mon[42099]: 4.8 scrub ok 2024-03-30T13:58:08.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:08 smithi043 ceph-mon[42099]: 6.1b scrub starts 2024-03-30T13:58:08.966 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:08.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:08 smithi113 ceph-mon[34341]: pgmap v676: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 6.7 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 852 B/s rd, 82 MiB/s wr, 1.32k op/s; 54933/328731 objects degraded (16.711%); 27/328731 objects misplaced (0.008%) 2024-03-30T13:58:08.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:08 smithi113 ceph-mon[34341]: 4.8 scrub starts 2024-03-30T13:58:08.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:08 smithi113 ceph-mon[34341]: 4.8 scrub ok 2024-03-30T13:58:08.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:08 smithi113 ceph-mon[34341]: 6.1b scrub starts 2024-03-30T13:58:09.000 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:09.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:09.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:09.102 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:09.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:09.143 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:09.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:09.244 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:09.286 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:09.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:09.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:09.374 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 61 16 117016 117000 119.855 100.125 0.0673288 0.00833838 2024-03-30T13:58:09.388 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:09.428 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:09.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:09.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:09.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:58:09 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:58:09.530 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:09.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:09.572 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:09.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:09.673 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:09.713 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:09.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:09.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:09.816 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:09.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:09.853 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:09.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:09.907 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:09 smithi043 ceph-mon[29963]: 5.1b scrub starts 2024-03-30T13:58:09.908 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:09 smithi043 ceph-mon[29963]: 5.1b scrub ok 2024-03-30T13:58:09.908 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:09 smithi043 ceph-mon[29963]: 6.1b scrub ok 2024-03-30T13:58:09.908 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:09 smithi043 ceph-mon[29963]: 6.c scrub starts 2024-03-30T13:58:09.908 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:09 smithi043 ceph-mon[29963]: 6.c scrub ok 2024-03-30T13:58:09.908 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:09 smithi043 ceph-mon[42099]: 5.1b scrub starts 2024-03-30T13:58:09.908 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:09 smithi043 ceph-mon[42099]: 5.1b scrub ok 2024-03-30T13:58:09.908 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:09 smithi043 ceph-mon[42099]: 6.1b scrub ok 2024-03-30T13:58:09.908 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:09 smithi043 ceph-mon[42099]: 6.c scrub starts 2024-03-30T13:58:09.908 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:09 smithi043 ceph-mon[42099]: 6.c scrub ok 2024-03-30T13:58:09.954 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:09.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:09 smithi113 ceph-mon[34341]: 5.1b scrub starts 2024-03-30T13:58:09.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:09 smithi113 ceph-mon[34341]: 5.1b scrub ok 2024-03-30T13:58:09.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:09 smithi113 ceph-mon[34341]: 6.1b scrub ok 2024-03-30T13:58:09.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:09 smithi113 ceph-mon[34341]: 6.c scrub starts 2024-03-30T13:58:09.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:09 smithi113 ceph-mon[34341]: 6.c scrub ok 2024-03-30T13:58:09.985 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:09.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:09.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:10.087 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:10.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:10.116 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:10.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:10.217 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:10.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:10.255 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:10.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:10.356 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:10.374 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 62 15 118485 118470 119.404 91.875 0.0049862 0.00837082 2024-03-30T13:58:10.388 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:10.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:10.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:10.490 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:10.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:10.515 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:10.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:10.616 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:10.655 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:10.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:10.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:10.757 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:10.784 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:10.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:10.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:10.886 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:10.913 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:10 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:10.914 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:10 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:58:10.914 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:10 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:58:10.914 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:10 smithi043 ceph-mon[29963]: osdmap e315: 8 total, 7 up, 6 in 2024-03-30T13:58:10.914 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:10 smithi043 ceph-mon[29963]: pgmap v678: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 7.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 852 B/s rd, 110 MiB/s wr, 1.75k op/s; 57290/342780 objects degraded (16.713%); 27/342780 objects misplaced (0.008%) 2024-03-30T13:58:10.914 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:10 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:10.914 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:10 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:58:10.914 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:10 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:58:10.914 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:10 smithi043 ceph-mon[42099]: osdmap e315: 8 total, 7 up, 6 in 2024-03-30T13:58:10.915 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:10 smithi043 ceph-mon[42099]: pgmap v678: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 7.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 852 B/s rd, 110 MiB/s wr, 1.75k op/s; 57290/342780 objects degraded (16.713%); 27/342780 objects misplaced (0.008%) 2024-03-30T13:58:10.920 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:10.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:10.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:10.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:10 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:10.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:10 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:58:10.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:10 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:58:10.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:10 smithi113 ceph-mon[34341]: osdmap e315: 8 total, 7 up, 6 in 2024-03-30T13:58:10.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:10 smithi113 ceph-mon[34341]: pgmap v678: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 7.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 852 B/s rd, 110 MiB/s wr, 1.75k op/s; 57290/342780 objects degraded (16.713%); 27/342780 objects misplaced (0.008%) 2024-03-30T13:58:11.022 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:11.059 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:11.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:11.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:11.118 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6] out_osds: [5, 7] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-30T13:58:11.118 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:58:11.119 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_3 2024-03-30T13:58:11.119 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_3 by 10 min 8 2024-03-30T13:58:11.119 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T13:58:11.161 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:11.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:11.191 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:11.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:11.292 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:11.338 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:11.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:11.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:11.375 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 63 16 119768 119752 118.779 80.125 0.00634945 0.00841497 2024-03-30T13:58:11.439 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:11.471 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:11.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:11.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:11.573 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:11.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:11.599 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:11.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:11.700 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:11.734 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:11.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:11.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:11.836 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:11.880 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:11.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:11.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:11.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:11 smithi113 ceph-mon[34341]: 4.f scrub starts 2024-03-30T13:58:11.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:11 smithi113 ceph-mon[34341]: 4.f scrub ok 2024-03-30T13:58:11.982 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:12.006 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:12.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:12.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:12.108 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:12.137 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:11 smithi043 ceph-mon[29963]: 4.f scrub starts 2024-03-30T13:58:12.137 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:11 smithi043 ceph-mon[29963]: 4.f scrub ok 2024-03-30T13:58:12.138 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:11 smithi043 ceph-mon[42099]: 4.f scrub starts 2024-03-30T13:58:12.138 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:11 smithi043 ceph-mon[42099]: 4.f scrub ok 2024-03-30T13:58:12.142 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:12.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:12.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:12.244 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:12.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:12.282 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:12.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:12.375 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 64 16 121288 121272 118.407 95 0.0112809 0.00844123 2024-03-30T13:58:12.383 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:12.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:12.423 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:12.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:12.524 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:12.569 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:12.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:12.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:12.671 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:12.703 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:12.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:12.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:12.805 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:12.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:12.842 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:12.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:12.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:12 smithi043 ceph-mon[29963]: 6.18 scrub starts 2024-03-30T13:58:12.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:12 smithi043 ceph-mon[29963]: 6.18 scrub ok 2024-03-30T13:58:12.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:12 smithi043 ceph-mon[29963]: pgmap v679: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 7.1 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.7 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 58811/351558 objects degraded (16.729%); 27/351558 objects misplaced (0.008%) 2024-03-30T13:58:12.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:12 smithi043 ceph-mon[29963]: 6.b scrub starts 2024-03-30T13:58:12.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:12 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 58811/351558 objects degraded (16.729%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:12.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:12 smithi043 ceph-mon[29963]: osdmap e316: 8 total, 7 up, 6 in 2024-03-30T13:58:12.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:12 smithi043 ceph-mon[42099]: 6.18 scrub starts 2024-03-30T13:58:12.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:12 smithi043 ceph-mon[42099]: 6.18 scrub ok 2024-03-30T13:58:12.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:12 smithi043 ceph-mon[42099]: pgmap v679: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 7.1 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.7 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 58811/351558 objects degraded (16.729%); 27/351558 objects misplaced (0.008%) 2024-03-30T13:58:12.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:12 smithi043 ceph-mon[42099]: 6.b scrub starts 2024-03-30T13:58:12.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:12 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 58811/351558 objects degraded (16.729%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:12.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:12 smithi043 ceph-mon[42099]: osdmap e316: 8 total, 7 up, 6 in 2024-03-30T13:58:12.944 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:12.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:12 smithi113 ceph-mon[34341]: 6.18 scrub starts 2024-03-30T13:58:12.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:12 smithi113 ceph-mon[34341]: 6.18 scrub ok 2024-03-30T13:58:12.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:12 smithi113 ceph-mon[34341]: pgmap v679: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 7.1 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.7 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 58811/351558 objects degraded (16.729%); 27/351558 objects misplaced (0.008%) 2024-03-30T13:58:12.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:12 smithi113 ceph-mon[34341]: 6.b scrub starts 2024-03-30T13:58:12.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:12 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 58811/351558 objects degraded (16.729%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:12.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:12 smithi113 ceph-mon[34341]: osdmap e316: 8 total, 7 up, 6 in 2024-03-30T13:58:12.981 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:12.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:12.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:13.083 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:13.121 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:13.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:13.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:13.223 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:13.258 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:13.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:13.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:13.360 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:13.376 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 65 16 122986 122970 118.218 106.125 0.00520631 0.00845481 2024-03-30T13:58:13.391 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:13.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:13.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:13.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:13.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:13.633 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:13.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:13.734 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:13.769 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:13.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:13.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:13.871 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:13.919 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:13 smithi043 ceph-mon[29963]: 3.4 scrub starts 2024-03-30T13:58:13.919 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:13 smithi043 ceph-mon[29963]: 3.4 scrub ok 2024-03-30T13:58:13.919 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:13 smithi043 ceph-mon[29963]: 6.b scrub ok 2024-03-30T13:58:13.919 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:13 smithi043 ceph-mon[29963]: 4.17 deep-scrub starts 2024-03-30T13:58:13.920 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:13 smithi043 ceph-mon[29963]: 4.17 deep-scrub ok 2024-03-30T13:58:13.921 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:13 smithi043 ceph-mon[42099]: 3.4 scrub starts 2024-03-30T13:58:13.922 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:13 smithi043 ceph-mon[42099]: 3.4 scrub ok 2024-03-30T13:58:13.922 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:13 smithi043 ceph-mon[42099]: 6.b scrub ok 2024-03-30T13:58:13.922 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:13 smithi043 ceph-mon[42099]: 4.17 deep-scrub starts 2024-03-30T13:58:13.922 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:13 smithi043 ceph-mon[42099]: 4.17 deep-scrub ok 2024-03-30T13:58:13.928 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:13.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:13.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:13.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:13 smithi113 ceph-mon[34341]: 3.4 scrub starts 2024-03-30T13:58:13.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:13 smithi113 ceph-mon[34341]: 3.4 scrub ok 2024-03-30T13:58:13.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:13 smithi113 ceph-mon[34341]: 6.b scrub ok 2024-03-30T13:58:13.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:13 smithi113 ceph-mon[34341]: 4.17 deep-scrub starts 2024-03-30T13:58:13.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:13 smithi113 ceph-mon[34341]: 4.17 deep-scrub ok 2024-03-30T13:58:14.030 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:14.077 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:14.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:14.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:14.179 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:14.210 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:14.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:14.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:14.240 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:58:14.286 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:58:14.312 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:14.318 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:58:14.327 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:58:14.340 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:14.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:14.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:14.376 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 66 15 124531 124516 117.89 96.625 0.0161927 0.00847784 2024-03-30T13:58:14.380 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:58:14.397 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:58:14.397 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:58:14.422 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:58:14.442 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:14.466 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:58:14.474 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:14.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:14.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:14.506 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:58:14.507 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:58:14.507 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:58:14.507 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:58:14.507 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:58:14.507 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:58:14.507 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:58:14.507 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:58:14.507 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:58:14.576 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:14.606 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:14.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:14.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:14.708 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:14.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:14.743 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:14.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:14.844 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:14.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:14.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:14.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:14 smithi113 ceph-mon[34341]: 3.6 scrub starts 2024-03-30T13:58:14.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:14 smithi113 ceph-mon[34341]: 3.6 scrub ok 2024-03-30T13:58:14.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:14 smithi113 ceph-mon[34341]: 6.1d scrub starts 2024-03-30T13:58:14.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:14 smithi113 ceph-mon[34341]: 6.1d scrub ok 2024-03-30T13:58:14.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:14 smithi113 ceph-mon[34341]: 3.e scrub starts 2024-03-30T13:58:14.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:14 smithi113 ceph-mon[34341]: 3.e scrub ok 2024-03-30T13:58:14.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:14 smithi113 ceph-mon[34341]: pgmap v681: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 7.3 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 842 B/s rd, 98 MiB/s wr, 1.56k op/s; 59760/357216 objects degraded (16.729%); 27/357216 objects misplaced (0.008%) 2024-03-30T13:58:14.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:14 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:58:14.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:14 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:14.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:14 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1875524886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:58:14.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:14 smithi113 ceph-mon[34341]: 3.18 scrub starts 2024-03-30T13:58:14.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:14 smithi113 ceph-mon[34341]: 3.18 scrub ok 2024-03-30T13:58:14.993 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:15.013 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[29963]: 3.6 scrub starts 2024-03-30T13:58:15.014 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[29963]: 3.6 scrub ok 2024-03-30T13:58:15.014 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[29963]: 6.1d scrub starts 2024-03-30T13:58:15.014 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[29963]: 6.1d scrub ok 2024-03-30T13:58:15.014 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[29963]: 3.e scrub starts 2024-03-30T13:58:15.014 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[29963]: 3.e scrub ok 2024-03-30T13:58:15.014 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[29963]: pgmap v681: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 7.3 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 842 B/s rd, 98 MiB/s wr, 1.56k op/s; 59760/357216 objects degraded (16.729%); 27/357216 objects misplaced (0.008%) 2024-03-30T13:58:15.014 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:58:15.014 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:15.014 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1875524886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:58:15.014 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[29963]: 3.18 scrub starts 2024-03-30T13:58:15.014 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[29963]: 3.18 scrub ok 2024-03-30T13:58:15.015 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[42099]: 3.6 scrub starts 2024-03-30T13:58:15.015 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[42099]: 3.6 scrub ok 2024-03-30T13:58:15.015 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[42099]: 6.1d scrub starts 2024-03-30T13:58:15.015 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[42099]: 6.1d scrub ok 2024-03-30T13:58:15.015 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[42099]: 3.e scrub starts 2024-03-30T13:58:15.015 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[42099]: 3.e scrub ok 2024-03-30T13:58:15.015 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[42099]: pgmap v681: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 7.3 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 842 B/s rd, 98 MiB/s wr, 1.56k op/s; 59760/357216 objects degraded (16.729%); 27/357216 objects misplaced (0.008%) 2024-03-30T13:58:15.015 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:58:15.015 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:15.015 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1875524886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:58:15.015 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[42099]: 3.18 scrub starts 2024-03-30T13:58:15.015 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:14 smithi043 ceph-mon[42099]: 3.18 scrub ok 2024-03-30T13:58:15.016 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T13:58:15.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:15.028 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:15.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:15.129 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:15.164 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:15.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:15.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:15.266 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:15.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:15.314 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:15.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:15.377 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 67 16 126152 126136 117.64 101.25 0.00772003 0.00849573 2024-03-30T13:58:15.415 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:15.443 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:15.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:15.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:15.457 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:58:15.457 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:58:15.546 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:15.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:15.582 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:15.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:15.683 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:15.725 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:15.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:15.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:15.827 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:15.858 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:15.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:15.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:15.960 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:15.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:15 smithi113 ceph-mon[34341]: 5.3 scrub starts 2024-03-30T13:58:15.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:15 smithi113 ceph-mon[34341]: 5.3 scrub ok 2024-03-30T13:58:15.996 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:15.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:15.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:16.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:15 smithi043 ceph-mon[29963]: 5.3 scrub starts 2024-03-30T13:58:16.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:15 smithi043 ceph-mon[29963]: 5.3 scrub ok 2024-03-30T13:58:16.092 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:15 smithi043 ceph-mon[42099]: 5.3 scrub starts 2024-03-30T13:58:16.092 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:15 smithi043 ceph-mon[42099]: 5.3 scrub ok 2024-03-30T13:58:16.098 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:16.138 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:16.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:16.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:16.240 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:16.271 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:16.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:16.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:16.373 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:16.377 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 68 16 127853 127837 117.473 106.312 0.0114917 0.00850837 2024-03-30T13:58:16.393 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:58:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:58:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:58:16.408 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:16.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:16.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:16.510 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:16.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:16.545 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:16.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:16.647 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:16.679 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:16.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:16.781 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:16.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:16.815 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:16.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:16.898 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-03-30T13:58:16.916 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:16.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:16.951 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:16.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:16.963 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:16 smithi043 ceph-mon[29963]: 5.f scrub starts 2024-03-30T13:58:16.963 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:16 smithi043 ceph-mon[29963]: 5.f scrub ok 2024-03-30T13:58:16.963 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:16 smithi043 ceph-mon[29963]: pgmap v682: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 7.5 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 111 MiB/s wr, 1.78k op/s; 62145/371517 objects degraded (16.727%); 27/371517 objects misplaced (0.007%) 2024-03-30T13:58:16.964 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:16 smithi043 ceph-mon[29963]: 3.d scrub starts 2024-03-30T13:58:16.964 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:16 smithi043 ceph-mon[29963]: 3.d scrub ok 2024-03-30T13:58:16.964 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:16 smithi043 ceph-mon[29963]: 3.2 scrub starts 2024-03-30T13:58:16.964 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:16 smithi043 ceph-mon[29963]: 3.2 scrub ok 2024-03-30T13:58:16.964 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:16 smithi043 ceph-mon[42099]: 5.f scrub starts 2024-03-30T13:58:16.964 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:16 smithi043 ceph-mon[42099]: 5.f scrub ok 2024-03-30T13:58:16.964 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:16 smithi043 ceph-mon[42099]: pgmap v682: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 7.5 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 111 MiB/s wr, 1.78k op/s; 62145/371517 objects degraded (16.727%); 27/371517 objects misplaced (0.007%) 2024-03-30T13:58:16.964 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:16 smithi043 ceph-mon[42099]: 3.d scrub starts 2024-03-30T13:58:16.964 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:16 smithi043 ceph-mon[42099]: 3.d scrub ok 2024-03-30T13:58:16.965 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:16 smithi043 ceph-mon[42099]: 3.2 scrub starts 2024-03-30T13:58:16.965 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:16 smithi043 ceph-mon[42099]: 3.2 scrub ok 2024-03-30T13:58:16.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:16 smithi113 ceph-mon[34341]: 5.f scrub starts 2024-03-30T13:58:16.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:16 smithi113 ceph-mon[34341]: 5.f scrub ok 2024-03-30T13:58:16.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:16 smithi113 ceph-mon[34341]: pgmap v682: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 7.5 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 111 MiB/s wr, 1.78k op/s; 62145/371517 objects degraded (16.727%); 27/371517 objects misplaced (0.007%) 2024-03-30T13:58:16.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:16 smithi113 ceph-mon[34341]: 3.d scrub starts 2024-03-30T13:58:16.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:16 smithi113 ceph-mon[34341]: 3.d scrub ok 2024-03-30T13:58:16.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:16 smithi113 ceph-mon[34341]: 3.2 scrub starts 2024-03-30T13:58:16.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:16 smithi113 ceph-mon[34341]: 3.2 scrub ok 2024-03-30T13:58:17.053 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:17.100 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:17.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:17.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:17.202 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:17.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:17.237 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:17.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:17.338 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:17.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:17.367 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:17.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:17.377 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 69 16 129481 129465 117.245 101.75 0.0110586 0.00852528 2024-03-30T13:58:17.458 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:58:17.468 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:17.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:17.502 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:17.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:17.603 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:17.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:17.653 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:17.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:17.755 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:17.786 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:17.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:17.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:17.888 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:17.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:17 smithi043 ceph-mon[29963]: 3.11 scrub starts 2024-03-30T13:58:17.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:17 smithi043 ceph-mon[29963]: 3.11 scrub ok 2024-03-30T13:58:17.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:17 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 62145/371517 objects degraded (16.727%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:17.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:17 smithi043 ceph-mon[42099]: 3.11 scrub starts 2024-03-30T13:58:17.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:17 smithi043 ceph-mon[42099]: 3.11 scrub ok 2024-03-30T13:58:17.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:17 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 62145/371517 objects degraded (16.727%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:17.919 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:17.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:17.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:17.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:17 smithi113 ceph-mon[34341]: 3.11 scrub starts 2024-03-30T13:58:17.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:17 smithi113 ceph-mon[34341]: 3.11 scrub ok 2024-03-30T13:58:17.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:17 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 62145/371517 objects degraded (16.727%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:18.022 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:18.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:18.060 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:18.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:18.161 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:18.197 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:18.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:18.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:18.300 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:18.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:18.336 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:18.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:18.378 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 70 15 131031 131016 116.955 96.9375 0.0100845 0.00854625 2024-03-30T13:58:18.437 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:18.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:18.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:18.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:18.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:18.624 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:18.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:18.726 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:18.764 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:18.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:18.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:18.866 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:18.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:18 smithi043 ceph-mon[29963]: 3.17 scrub starts 2024-03-30T13:58:18.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:18 smithi043 ceph-mon[29963]: 3.17 scrub ok 2024-03-30T13:58:18.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:18 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:58:18.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:18 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:58:18.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:18 smithi043 ceph-mon[29963]: pgmap v683: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 7.6 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 62632/374511 objects degraded (16.724%); 27/374511 objects misplaced (0.007%) 2024-03-30T13:58:18.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:18 smithi043 ceph-mon[29963]: osdmap e317: 8 total, 7 up, 6 in 2024-03-30T13:58:18.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:18 smithi043 ceph-mon[42099]: 3.17 scrub starts 2024-03-30T13:58:18.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:18 smithi043 ceph-mon[42099]: 3.17 scrub ok 2024-03-30T13:58:18.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:18 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:58:18.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:18 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:58:18.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:18 smithi043 ceph-mon[42099]: pgmap v683: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 7.6 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 62632/374511 objects degraded (16.724%); 27/374511 objects misplaced (0.007%) 2024-03-30T13:58:18.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:18 smithi043 ceph-mon[42099]: osdmap e317: 8 total, 7 up, 6 in 2024-03-30T13:58:18.897 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:18.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:18.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:18.999 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:19.030 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:19.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:19.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:19.132 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:19.165 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:19.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:19.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:19.183 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:18 smithi113 ceph-mon[34341]: 3.17 scrub starts 2024-03-30T13:58:19.183 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:18 smithi113 ceph-mon[34341]: 3.17 scrub ok 2024-03-30T13:58:19.183 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:18 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:58:19.183 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:18 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:58:19.183 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:18 smithi113 ceph-mon[34341]: pgmap v683: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 7.6 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 62632/374511 objects degraded (16.724%); 27/374511 objects misplaced (0.007%) 2024-03-30T13:58:19.184 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:18 smithi113 ceph-mon[34341]: osdmap e317: 8 total, 7 up, 6 in 2024-03-30T13:58:19.267 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:19.306 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:19.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:19.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:19.378 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 71 15 132679 132664 116.758 103 0.016413 0.00856063 2024-03-30T13:58:19.409 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:19.439 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:19.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:19.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:19.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:58:19 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:58:19.517 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:58:19.532 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:58:19.542 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:19.563 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:58:19.651 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:19.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:19.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:19.753 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:19.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:19.786 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:19.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:19.887 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:19.923 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:19.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:19.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:20.006 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:19 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:58:20.007 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:19 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:58:20.007 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:19 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:58:20.007 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:19 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:58:20.025 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:20.058 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:20.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:20.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:20.150 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:19 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:58:20.150 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:19 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:58:20.160 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:20.193 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:20.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:20.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:20.295 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:20.334 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:20.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:20.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:20.378 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 72 16 134083 134067 116.354 87.6875 0.00630247 0.00859041 2024-03-30T13:58:20.435 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:20.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:20.474 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:20.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:20.575 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:20.615 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:20.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:20.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:20.717 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:20.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:20.753 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:20.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:20.854 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:20.888 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:20.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:20.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:20.990 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:21.027 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:21.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:21.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:21.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:20 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:21.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:20 smithi043 ceph-mon[29963]: pgmap v685: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 7.9 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.59k op/s; 65214/389715 objects degraded (16.734%); 27/389715 objects misplaced (0.007%) 2024-03-30T13:58:21.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:20 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:58:21.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:20 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:58:21.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:20 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:58:21.058 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:20 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:21.058 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:20 smithi043 ceph-mon[42099]: pgmap v685: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 7.9 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.59k op/s; 65214/389715 objects degraded (16.734%); 27/389715 objects misplaced (0.007%) 2024-03-30T13:58:21.058 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:20 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:58:21.058 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:20 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:58:21.058 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:20 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:58:21.129 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:21.167 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:21.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:21.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:21.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:20 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:21.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:20 smithi113 ceph-mon[34341]: pgmap v685: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 7.9 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.59k op/s; 65214/389715 objects degraded (16.734%); 27/389715 objects misplaced (0.007%) 2024-03-30T13:58:21.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:20 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:58:21.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:20 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:58:21.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:20 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:58:21.269 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:21.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:21.307 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:21.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:21.378 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 73 16 135785 135769 116.218 106.375 0.011511 0.0086006 2024-03-30T13:58:21.408 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:21.448 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:21.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:21.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:21.550 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:21.595 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:21.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:21.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:21.697 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:21.739 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:21.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:21.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:21.841 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:21.881 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:21.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:21.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:21.983 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:22.028 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:22.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:22.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:22.130 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:22.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:21 smithi043 ceph-mon[29963]: 6.1e scrub starts 2024-03-30T13:58:22.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:21 smithi043 ceph-mon[29963]: 6.1e scrub ok 2024-03-30T13:58:22.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:21 smithi043 ceph-mon[42099]: 6.1e scrub starts 2024-03-30T13:58:22.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:21 smithi043 ceph-mon[42099]: 6.1e scrub ok 2024-03-30T13:58:22.166 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:22.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:22.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:22.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:21 smithi113 ceph-mon[34341]: 6.1e scrub starts 2024-03-30T13:58:22.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:21 smithi113 ceph-mon[34341]: 6.1e scrub ok 2024-03-30T13:58:22.269 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:22.303 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:22.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:22.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:22.378 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 74 16 137469 137453 116.069 105.25 0.0101259 0.00861128 2024-03-30T13:58:22.404 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:22.453 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:22.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:22.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:22.555 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:22.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:22.592 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:22.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:22.693 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:22.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:22.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:22.833 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:22.874 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:22.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:22.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:22.976 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:23.027 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:23.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:23.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:23.090 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:22 smithi043 ceph-mon[29963]: pgmap v686: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 8.1 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 66701/398751 objects degraded (16.727%); 27/398751 objects misplaced (0.007%) 2024-03-30T13:58:23.090 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:22 smithi043 ceph-mon[29963]: 4.a scrub starts 2024-03-30T13:58:23.090 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:22 smithi043 ceph-mon[29963]: 4.a scrub ok 2024-03-30T13:58:23.090 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:22 smithi043 ceph-mon[29963]: 4.c scrub starts 2024-03-30T13:58:23.090 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:22 smithi043 ceph-mon[29963]: 4.c scrub ok 2024-03-30T13:58:23.091 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:22 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 66701/398751 objects degraded (16.727%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:23.091 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:22 smithi043 ceph-mon[29963]: osdmap e318: 8 total, 7 up, 6 in 2024-03-30T13:58:23.091 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:22 smithi043 ceph-mon[42099]: pgmap v686: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 8.1 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 66701/398751 objects degraded (16.727%); 27/398751 objects misplaced (0.007%) 2024-03-30T13:58:23.091 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:22 smithi043 ceph-mon[42099]: 4.a scrub starts 2024-03-30T13:58:23.091 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:22 smithi043 ceph-mon[42099]: 4.a scrub ok 2024-03-30T13:58:23.091 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:22 smithi043 ceph-mon[42099]: 4.c scrub starts 2024-03-30T13:58:23.091 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:22 smithi043 ceph-mon[42099]: 4.c scrub ok 2024-03-30T13:58:23.091 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:22 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 66701/398751 objects degraded (16.727%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:23.091 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:22 smithi043 ceph-mon[42099]: osdmap e318: 8 total, 7 up, 6 in 2024-03-30T13:58:23.129 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:23.159 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:23.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:23.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:23.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:22 smithi113 ceph-mon[34341]: pgmap v686: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 8.1 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 66701/398751 objects degraded (16.727%); 27/398751 objects misplaced (0.007%) 2024-03-30T13:58:23.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:22 smithi113 ceph-mon[34341]: 4.a scrub starts 2024-03-30T13:58:23.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:22 smithi113 ceph-mon[34341]: 4.a scrub ok 2024-03-30T13:58:23.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:22 smithi113 ceph-mon[34341]: 4.c scrub starts 2024-03-30T13:58:23.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:22 smithi113 ceph-mon[34341]: 4.c scrub ok 2024-03-30T13:58:23.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:22 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 66701/398751 objects degraded (16.727%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:23.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:22 smithi113 ceph-mon[34341]: osdmap e318: 8 total, 7 up, 6 in 2024-03-30T13:58:23.261 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:23.295 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:23.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:23.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:23.378 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 75 16 138983 138967 115.783 94.625 0.00873495 0.00863268 2024-03-30T13:58:23.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:23.436 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:23.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:23.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:23.472 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:58:23.483 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-30T13:58:23.538 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:23.589 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:23.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:23.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:23.692 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:23.725 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:23.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:23.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:23.827 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:23.879 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:23.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:23.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:23.960 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-03-30T13:58:23.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:23 smithi043 ceph-mon[29963]: 6.d scrub starts 2024-03-30T13:58:23.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:23 smithi043 ceph-mon[29963]: 6.d scrub ok 2024-03-30T13:58:23.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:23 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1159813288' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:58:23.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:23 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:58:23.981 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:23 smithi043 ceph-mon[42099]: 6.d scrub starts 2024-03-30T13:58:23.981 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:23 smithi043 ceph-mon[42099]: 6.d scrub ok 2024-03-30T13:58:23.981 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:23 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1159813288' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:58:23.981 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:23 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:58:23.981 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:24.013 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:24.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:24.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:24.115 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:24.146 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:24.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:24.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:24.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:23 smithi113 ceph-mon[34341]: 6.d scrub starts 2024-03-30T13:58:24.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:23 smithi113 ceph-mon[34341]: 6.d scrub ok 2024-03-30T13:58:24.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:23 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1159813288' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:58:24.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:23 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:58:24.248 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:24.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:24.284 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:24.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:24.378 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 76 16 140505 140489 115.511 95.125 0.00681825 0.00865313 2024-03-30T13:58:24.385 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:24.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:24.424 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:24.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:24.525 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:24.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:24.568 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:24.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:24.670 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:24.703 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:24.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:24.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:24.805 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:24.851 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:24.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:24.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:24.953 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:24.986 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:24.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:24.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:25.089 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:25.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:24 smithi043 ceph-mon[29963]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:58:25.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:24 smithi043 ceph-mon[29963]: pgmap v688: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 8.1 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s; 66701/398751 objects degraded (16.727%); 27/398751 objects misplaced (0.007%) 2024-03-30T13:58:25.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:24 smithi043 ceph-mon[29963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:58:25.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:24 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:58:25.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:24 smithi043 ceph-mon[29963]: osdmap e319: 8 total, 7 up, 6 in 2024-03-30T13:58:25.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:24 smithi043 ceph-mon[42099]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:58:25.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:24 smithi043 ceph-mon[42099]: pgmap v688: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 8.1 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s; 66701/398751 objects degraded (16.727%); 27/398751 objects misplaced (0.007%) 2024-03-30T13:58:25.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:24 smithi043 ceph-mon[42099]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:58:25.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:24 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:58:25.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:24 smithi043 ceph-mon[42099]: osdmap e319: 8 total, 7 up, 6 in 2024-03-30T13:58:25.123 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:25.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:25.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:25.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:24 smithi113 ceph-mon[34341]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:58:25.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:24 smithi113 ceph-mon[34341]: pgmap v688: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 8.1 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s; 66701/398751 objects degraded (16.727%); 27/398751 objects misplaced (0.007%) 2024-03-30T13:58:25.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:24 smithi113 ceph-mon[34341]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:58:25.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:24 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:58:25.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:24 smithi113 ceph-mon[34341]: osdmap e319: 8 total, 7 up, 6 in 2024-03-30T13:58:25.225 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:25.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:25.263 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:25.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:25.364 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:25.378 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 77 16 142253 142237 115.43 109.25 0.0127644 0.00865928 2024-03-30T13:58:25.405 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:25.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:25.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:25.507 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:25.545 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:25.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:25.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:25.647 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:25.680 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:25.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:25.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:25.782 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:25.813 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:25.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:25.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:25.915 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:25.948 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T13:58:25.949 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:58:25.956 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:25.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:25.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:26.058 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:26.089 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:26.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:26.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:26.191 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:26.232 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:26.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:26.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:26.334 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:26.375 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:26.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:26.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:26.378 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 78 16 143897 143881 115.267 102.75 0.00640889 0.00867134 2024-03-30T13:58:26.393 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:58:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:58:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:58:26.477 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:26.511 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:26.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:26.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:26.613 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:26.641 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:26.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:26.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:26.744 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:26.776 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:26.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:26.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:26.878 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:26.913 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:26.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:26.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:27.015 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:27.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:27.052 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:27.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:27.153 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:27.160 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-03-30T13:58:27.186 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:27.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:27.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:27.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:26 smithi113 ceph-mon[34341]: pgmap v690: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 8.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 645 B/s rd, 118 MiB/s wr, 1.89k op/s; 70223/419499 objects degraded (16.740%); 27/419499 objects misplaced (0.006%) 2024-03-30T13:58:27.239 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:26 smithi043 ceph-mon[42099]: pgmap v690: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 8.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 645 B/s rd, 118 MiB/s wr, 1.89k op/s; 70223/419499 objects degraded (16.740%); 27/419499 objects misplaced (0.006%) 2024-03-30T13:58:27.239 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:26 smithi043 ceph-mon[29963]: pgmap v690: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 8.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 645 B/s rd, 118 MiB/s wr, 1.89k op/s; 70223/419499 objects degraded (16.740%); 27/419499 objects misplaced (0.006%) 2024-03-30T13:58:27.289 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:27.321 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:27.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:27.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:27.378 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 79 16 145638 145622 115.185 108.812 0.0125063 0.00867744 2024-03-30T13:58:27.423 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:27.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:27.457 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:27.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:27.555 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":687,"stamp":"2024-03-30T13:58:22.591676+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8698468702,"num_objects":132917,"num_object_clones":0,"num_object_copies":398751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":66701,"num_objects_misplaced":27,"num_objects_unfound":0,"num_objects_dirty":132917,"num_whiteouts":0,"num_read":1428,"num_read_kb":1180,"num_write":133441,"num_write_kb":8494061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":659760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63467,"ondisk_log_size":63467,"up":404,"acting":447,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":449,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":26705276,"kb_used_data":23226564,"kb_used_omap":9,"kb_used_meta":3478646,"kb_avail":535765636,"statfs":{"total":575970213888,"available":548624011264,"internally_reserved":0,"allocated":23784001536,"data_stored":23777044917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":3562134177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":907345920,"num_objects":13845,"num_object_clones":0,"num_object_copies":41535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13845,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":13845,"num_write_kb":886080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.933735"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":560,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.729523+0000","last_change":"2024-03-30T13:57:08.625466+0000","last_active":"2024-03-30T13:58:17.729523+0000","last_peered":"2024-03-30T13:58:17.729523+0000","last_clean":"2024-03-30T13:58:17.729523+0000","last_became_active":"2024-03-30T13:53:35.861853+0000","last_became_peered":"2024-03-30T13:53:35.861853+0000","last_unstale":"2024-03-30T13:58:17.729523+0000","last_undegraded":"2024-03-30T13:58:17.729523+0000","last_fullsized":"2024-03-30T13:58:17.729523+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:08.625338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:08.625338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:31.305689+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":577,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.735327+0000","last_change":"2024-03-30T13:57:56.016167+0000","last_active":"2024-03-30T13:58:17.735327+0000","last_peered":"2024-03-30T13:58:17.735327+0000","last_clean":"2024-03-30T13:58:17.735327+0000","last_became_active":"2024-03-30T13:53:36.860312+0000","last_became_peered":"2024-03-30T13:53:36.860312+0000","last_unstale":"2024-03-30T13:58:17.735327+0000","last_undegraded":"2024-03-30T13:58:17.735327+0000","last_fullsized":"2024-03-30T13:58:17.735327+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:56.015905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:17.548567+0000","last_clean_scrub_stamp":"2024-03-30T13:57:56.015905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:10.331736+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6,7],"avail_no_missing":["1","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"65'1","reported_seq":569,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.744366+0000","last_change":"2024-03-30T13:56:58.649057+0000","last_active":"2024-03-30T13:58:17.744366+0000","last_peered":"2024-03-30T13:58:17.744366+0000","last_clean":"2024-03-30T13:58:17.744366+0000","last_became_active":"2024-03-30T13:53:14.657693+0000","last_became_peered":"2024-03-30T13:53:14.657693+0000","last_unstale":"2024-03-30T13:58:17.744366+0000","last_undegraded":"2024-03-30T13:58:17.744366+0000","last_fullsized":"2024-03-30T13:58:17.744366+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:56:58.648903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:56:58.648903+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,5],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"216'15","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.730368+0000","last_change":"2024-03-30T13:52:56.028286+0000","last_active":"2024-03-30T13:58:17.730368+0000","last_peered":"2024-03-30T13:58:17.730368+0000","last_clean":"2024-03-30T13:58:17.730368+0000","last_became_active":"2024-03-30T13:42:47.319638+0000","last_became_peered":"2024-03-30T13:42:47.319638+0000","last_unstale":"2024-03-30T13:58:17.730368+0000","last_undegraded":"2024-03-30T13:58:17.730368+0000","last_fullsized":"2024-03-30T13:58:17.730368+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'15","last_scrub_stamp":"2024-03-30T13:52:56.028141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:56.028141+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":554,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.734424+0000","last_change":"2024-03-30T13:58:08.388962+0000","last_active":"2024-03-30T13:58:17.734424+0000","last_peered":"2024-03-30T13:58:17.734424+0000","last_clean":"2024-03-30T13:58:17.734424+0000","last_became_active":"2024-03-30T13:53:35.862085+0000","last_became_peered":"2024-03-30T13:53:35.862085+0000","last_unstale":"2024-03-30T13:58:17.734424+0000","last_undegraded":"2024-03-30T13:58:17.734424+0000","last_fullsized":"2024-03-30T13:58:17.734424+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:08.388764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:08.388764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:30.819838+0000","scrub_duration":168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":552,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.730211+0000","last_change":"2024-03-30T13:57:12.673777+0000","last_active":"2024-03-30T13:58:17.730211+0000","last_peered":"2024-03-30T13:58:17.730211+0000","last_clean":"2024-03-30T13:58:17.730211+0000","last_became_active":"2024-03-30T13:42:49.327124+0000","last_became_peered":"2024-03-30T13:42:49.327124+0000","last_unstale":"2024-03-30T13:58:17.730211+0000","last_undegraded":"2024-03-30T13:58:17.730211+0000","last_fullsized":"2024-03-30T13:58:17.730211+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:12.673593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:12.673593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:26.107025+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":548,"reported_epoch":317,"state":"active+undersized","last_fresh":"2024-03-30T13:58:17.734227+0000","last_change":"2024-03-30T13:50:08.512316+0000","last_active":"2024-03-30T13:58:17.734227+0000","last_peered":"2024-03-30T13:58:17.734227+0000","last_clean":"2024-03-30T13:50:05.504688+0000","last_became_active":"2024-03-30T13:50:08.512316+0000","last_became_peered":"2024-03-30T13:50:08.512316+0000","last_unstale":"2024-03-30T13:58:17.734227+0000","last_undegraded":"2024-03-30T13:58:17.734227+0000","last_fullsized":"2024-03-30T13:50:08.501439+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:20.499768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:20.499768+0000","last_clean_scrub_stamp":"2024-03-30T13:49:20.499768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:19.213065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"217'34","reported_seq":585,"reported_epoch":317,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:58:17.734351+0000","last_change":"2024-03-30T13:50:08.511099+0000","last_active":"2024-03-30T13:58:17.734351+0000","last_peered":"2024-03-30T13:58:17.734351+0000","last_clean":"2024-03-30T13:50:05.499201+0000","last_became_active":"2024-03-30T13:50:08.511099+0000","last_became_peered":"2024-03-30T13:50:08.511099+0000","last_unstale":"2024-03-30T13:58:17.734351+0000","last_undegraded":"2024-03-30T13:50:08.502733+0000","last_fullsized":"2024-03-30T13:50:08.502606+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'22","last_scrub_stamp":"2024-03-30T13:46:49.339954+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:29.151791+0000","last_clean_scrub_stamp":"2024-03-30T13:46:49.339954+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:19.213025+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":559,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.741956+0000","last_change":"2024-03-30T13:58:10.911460+0000","last_active":"2024-03-30T13:58:17.741956+0000","last_peered":"2024-03-30T13:58:17.741956+0000","last_clean":"2024-03-30T13:58:17.741956+0000","last_became_active":"2024-03-30T13:53:36.855710+0000","last_became_peered":"2024-03-30T13:53:36.855710+0000","last_unstale":"2024-03-30T13:58:17.741956+0000","last_undegraded":"2024-03-30T13:58:17.741956+0000","last_fullsized":"2024-03-30T13:58:17.741956+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:10.911245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:27.061281+0000","last_clean_scrub_stamp":"2024-03-30T13:58:10.911245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:33.994963+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":218,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.741924+0000","last_change":"2024-03-30T13:58:07.825025+0000","last_active":"2024-03-30T13:58:17.741924+0000","last_peered":"2024-03-30T13:58:17.741924+0000","last_clean":"2024-03-30T13:58:17.741924+0000","last_became_active":"2024-03-30T13:53:35.856684+0000","last_became_peered":"2024-03-30T13:53:35.856684+0000","last_unstale":"2024-03-30T13:58:17.741924+0000","last_undegraded":"2024-03-30T13:58:17.741924+0000","last_fullsized":"2024-03-30T13:58:17.741924+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:07.824904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:20.069864+0000","last_clean_scrub_stamp":"2024-03-30T13:58:07.824904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:25.617218+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":203,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.731184+0000","last_change":"2024-03-30T13:53:13.645288+0000","last_active":"2024-03-30T13:58:17.731184+0000","last_peered":"2024-03-30T13:58:17.731184+0000","last_clean":"2024-03-30T13:58:17.731184+0000","last_became_active":"2024-03-30T13:53:13.644821+0000","last_became_peered":"2024-03-30T13:53:13.644821+0000","last_unstale":"2024-03-30T13:58:17.731184+0000","last_undegraded":"2024-03-30T13:58:17.731184+0000","last_fullsized":"2024-03-30T13:58:17.731184+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:07.129142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:37.513596+0000","last_clean_scrub_stamp":"2024-03-30T13:52:07.129142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"216'15","reported_seq":589,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.730252+0000","last_change":"2024-03-30T13:57:19.638146+0000","last_active":"2024-03-30T13:58:17.730252+0000","last_peered":"2024-03-30T13:58:17.730252+0000","last_clean":"2024-03-30T13:58:17.730252+0000","last_became_active":"2024-03-30T13:42:47.310289+0000","last_became_peered":"2024-03-30T13:42:47.310289+0000","last_unstale":"2024-03-30T13:58:17.730252+0000","last_undegraded":"2024-03-30T13:58:17.730252+0000","last_fullsized":"2024-03-30T13:58:17.730252+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'15","last_scrub_stamp":"2024-03-30T13:57:19.638024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:19.638024+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:39.206527+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":211,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.739067+0000","last_change":"2024-03-30T13:56:57.194703+0000","last_active":"2024-03-30T13:58:17.739067+0000","last_peered":"2024-03-30T13:58:17.739067+0000","last_clean":"2024-03-30T13:58:17.739067+0000","last_became_active":"2024-03-30T13:53:14.658167+0000","last_became_peered":"2024-03-30T13:53:14.658167+0000","last_unstale":"2024-03-30T13:58:17.739067+0000","last_undegraded":"2024-03-30T13:58:17.739067+0000","last_fullsized":"2024-03-30T13:58:17.739067+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:57.194555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:56:57.194555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3,5],"avail_no_missing":["1","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":188,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.731056+0000","last_change":"2024-03-30T13:53:35.862161+0000","last_active":"2024-03-30T13:58:17.731056+0000","last_peered":"2024-03-30T13:58:17.731056+0000","last_clean":"2024-03-30T13:58:17.731056+0000","last_became_active":"2024-03-30T13:53:35.861405+0000","last_became_peered":"2024-03-30T13:53:35.861405+0000","last_unstale":"2024-03-30T13:58:17.731056+0000","last_undegraded":"2024-03-30T13:58:17.731056+0000","last_fullsized":"2024-03-30T13:58:17.731056+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:50.953533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:41.633284+0000","last_clean_scrub_stamp":"2024-03-30T13:52:50.953533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"186'2","reported_seq":510,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.730065+0000","last_change":"2024-03-30T13:57:13.664902+0000","last_active":"2024-03-30T13:58:17.730065+0000","last_peered":"2024-03-30T13:58:17.730065+0000","last_clean":"2024-03-30T13:58:17.730065+0000","last_became_active":"2024-03-30T13:53:13.646659+0000","last_became_peered":"2024-03-30T13:53:13.646659+0000","last_unstale":"2024-03-30T13:58:17.730065+0000","last_undegraded":"2024-03-30T13:58:17.730065+0000","last_fullsized":"2024-03-30T13:58:17.730065+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-30T13:57:13.664762+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-30T13:51:15.932076+0000","last_clean_scrub_stamp":"2024-03-30T13:57:13.664762+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:29.059258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"216'9","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.729500+0000","last_change":"2024-03-30T13:52:19.931416+0000","last_active":"2024-03-30T13:58:17.729500+0000","last_peered":"2024-03-30T13:58:17.729500+0000","last_clean":"2024-03-30T13:58:17.729500+0000","last_became_active":"2024-03-30T13:42:47.317352+0000","last_became_peered":"2024-03-30T13:42:47.317352+0000","last_unstale":"2024-03-30T13:58:17.729500+0000","last_undegraded":"2024-03-30T13:58:17.729500+0000","last_fullsized":"2024-03-30T13:58:17.729500+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-30T13:52:19.931197+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-30T13:43:50.960980+0000","last_clean_scrub_stamp":"2024-03-30T13:52:19.931197+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":555,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.730432+0000","last_change":"2024-03-30T13:53:13.655592+0000","last_active":"2024-03-30T13:58:17.730432+0000","last_peered":"2024-03-30T13:58:17.730432+0000","last_clean":"2024-03-30T13:58:17.730432+0000","last_became_active":"2024-03-30T13:53:13.654803+0000","last_became_peered":"2024-03-30T13:53:13.654803+0000","last_unstale":"2024-03-30T13:58:17.730432+0000","last_undegraded":"2024-03-30T13:58:17.730432+0000","last_fullsized":"2024-03-30T13:58:17.730432+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:07.996724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:07.996724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":567,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.735145+0000","last_change":"2024-03-30T13:57:08.097180+0000","last_active":"2024-03-30T13:58:17.735145+0000","last_peered":"2024-03-30T13:58:17.735145+0000","last_clean":"2024-03-30T13:58:17.735145+0000","last_became_active":"2024-03-30T13:42:49.322665+0000","last_became_peered":"2024-03-30T13:42:49.322665+0000","last_unstale":"2024-03-30T13:58:17.735145+0000","last_undegraded":"2024-03-30T13:58:17.735145+0000","last_fullsized":"2024-03-30T13:58:17.735145+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:08.097101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:30.465974+0000","last_clean_scrub_stamp":"2024-03-30T13:57:08.097101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:19.206320+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":572,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.742392+0000","last_change":"2024-03-30T13:57:56.945998+0000","last_active":"2024-03-30T13:58:17.742392+0000","last_peered":"2024-03-30T13:58:17.742392+0000","last_clean":"2024-03-30T13:58:17.742392+0000","last_became_active":"2024-03-30T13:53:35.861043+0000","last_became_peered":"2024-03-30T13:53:35.861043+0000","last_unstale":"2024-03-30T13:58:17.742392+0000","last_undegraded":"2024-03-30T13:58:17.742392+0000","last_fullsized":"2024-03-30T13:58:17.742392+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:56.945861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:43.088964+0000","last_clean_scrub_stamp":"2024-03-30T13:57:56.945861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:22.686748+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"216'27","reported_seq":385,"reported_epoch":317,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:58:17.734495+0000","last_change":"2024-03-30T13:50:08.508437+0000","last_active":"2024-03-30T13:58:17.734495+0000","last_peered":"2024-03-30T13:58:17.734495+0000","last_clean":"2024-03-30T13:50:05.490228+0000","last_became_active":"2024-03-30T13:50:08.508437+0000","last_became_peered":"2024-03-30T13:50:08.508437+0000","last_unstale":"2024-03-30T13:58:17.734495+0000","last_undegraded":"2024-03-30T13:50:08.506233+0000","last_fullsized":"2024-03-30T13:50:08.506116+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-30T13:49:08.415681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:49:08.415681+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":547,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.733409+0000","last_change":"2024-03-30T13:53:13.665423+0000","last_active":"2024-03-30T13:58:17.733409+0000","last_peered":"2024-03-30T13:58:17.733409+0000","last_clean":"2024-03-30T13:58:17.733409+0000","last_became_active":"2024-03-30T13:53:13.664873+0000","last_became_peered":"2024-03-30T13:53:13.664873+0000","last_unstale":"2024-03-30T13:58:17.733409+0000","last_undegraded":"2024-03-30T13:58:17.733409+0000","last_fullsized":"2024-03-30T13:58:17.733409+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:52.054302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:52.054302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":539,"reported_epoch":317,"state":"active+undersized","last_fresh":"2024-03-30T13:58:17.730679+0000","last_change":"2024-03-30T13:50:08.506226+0000","last_active":"2024-03-30T13:58:17.730679+0000","last_peered":"2024-03-30T13:58:17.730679+0000","last_clean":"2024-03-30T13:50:05.490914+0000","last_became_active":"2024-03-30T13:50:08.506226+0000","last_became_peered":"2024-03-30T13:50:08.506226+0000","last_unstale":"2024-03-30T13:58:17.730679+0000","last_undegraded":"2024-03-30T13:58:17.730679+0000","last_fullsized":"2024-03-30T13:50:08.502611+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:25.104610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:48:25.104610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:18.505160+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":547,"reported_epoch":317,"state":"active+undersized","last_fresh":"2024-03-30T13:58:17.730543+0000","last_change":"2024-03-30T13:50:08.508556+0000","last_active":"2024-03-30T13:58:17.730543+0000","last_peered":"2024-03-30T13:58:17.730543+0000","last_clean":"2024-03-30T13:50:07.261781+0000","last_became_active":"2024-03-30T13:50:08.508556+0000","last_became_peered":"2024-03-30T13:50:08.508556+0000","last_unstale":"2024-03-30T13:58:17.730543+0000","last_undegraded":"2024-03-30T13:58:17.730543+0000","last_fullsized":"2024-03-30T13:50:08.502793+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:07.261595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:50:07.261595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:18.505189+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"217'23","reported_seq":593,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.732701+0000","last_change":"2024-03-30T13:52:50.945958+0000","last_active":"2024-03-30T13:58:17.732701+0000","last_peered":"2024-03-30T13:58:17.732701+0000","last_clean":"2024-03-30T13:58:17.732701+0000","last_became_active":"2024-03-30T13:42:47.317881+0000","last_became_peered":"2024-03-30T13:42:47.317881+0000","last_unstale":"2024-03-30T13:58:17.732701+0000","last_undegraded":"2024-03-30T13:58:17.732701+0000","last_fullsized":"2024-03-30T13:58:17.732701+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'17","last_scrub_stamp":"2024-03-30T13:52:50.945844+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-03-30T13:43:52.044427+0000","last_clean_scrub_stamp":"2024-03-30T13:52:50.945844+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":539,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.733442+0000","last_change":"2024-03-30T13:56:53.919136+0000","last_active":"2024-03-30T13:58:17.733442+0000","last_peered":"2024-03-30T13:58:17.733442+0000","last_clean":"2024-03-30T13:58:17.733442+0000","last_became_active":"2024-03-30T13:53:37.109227+0000","last_became_peered":"2024-03-30T13:53:37.109227+0000","last_unstale":"2024-03-30T13:58:17.733442+0000","last_undegraded":"2024-03-30T13:58:17.733442+0000","last_fullsized":"2024-03-30T13:58:17.733442+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T13:56:53.918975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:56:53.918975+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,7],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"230'10","reported_seq":770,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.734359+0000","last_change":"2024-03-30T13:57:07.246065+0000","last_active":"2024-03-30T13:58:17.734359+0000","last_peered":"2024-03-30T13:58:17.734359+0000","last_clean":"2024-03-30T13:58:17.734359+0000","last_became_active":"2024-03-30T13:53:36.852879+0000","last_became_peered":"2024-03-30T13:53:36.852879+0000","last_unstale":"2024-03-30T13:58:17.734359+0000","last_undegraded":"2024-03-30T13:58:17.734359+0000","last_fullsized":"2024-03-30T13:58:17.734359+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'10","last_scrub_stamp":"2024-03-30T13:57:07.245825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:07.245825+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,7],"avail_no_missing":["6","4","7"],"object_location_counts":[{"shards":"4,6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"65'1","reported_seq":576,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.735168+0000","last_change":"2024-03-30T13:58:00.032332+0000","last_active":"2024-03-30T13:58:17.735168+0000","last_peered":"2024-03-30T13:58:17.735168+0000","last_clean":"2024-03-30T13:58:17.735168+0000","last_became_active":"2024-03-30T13:42:45.307832+0000","last_became_peered":"2024-03-30T13:42:45.307832+0000","last_unstale":"2024-03-30T13:58:17.735168+0000","last_undegraded":"2024-03-30T13:58:17.735168+0000","last_fullsized":"2024-03-30T13:58:17.735168+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:58:00.031186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:00.031186+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:25.305529+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"217'25","reported_seq":616,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.746377+0000","last_change":"2024-03-30T13:53:35.974113+0000","last_active":"2024-03-30T13:58:17.746377+0000","last_peered":"2024-03-30T13:58:17.746377+0000","last_clean":"2024-03-30T13:58:17.746377+0000","last_became_active":"2024-03-30T13:53:35.858156+0000","last_became_peered":"2024-03-30T13:53:35.858156+0000","last_unstale":"2024-03-30T13:58:17.746377+0000","last_undegraded":"2024-03-30T13:58:17.746377+0000","last_fullsized":"2024-03-30T13:58:17.746377+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'25","last_scrub_stamp":"2024-03-30T13:52:54.925758+0000","last_deep_scrub":"153'15","last_deep_scrub_stamp":"2024-03-30T13:51:22.252831+0000","last_clean_scrub_stamp":"2024-03-30T13:52:54.925758+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":568,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.734405+0000","last_change":"2024-03-30T13:58:13.029728+0000","last_active":"2024-03-30T13:58:17.734405+0000","last_peered":"2024-03-30T13:58:17.734405+0000","last_clean":"2024-03-30T13:58:17.734405+0000","last_became_active":"2024-03-30T13:53:14.659650+0000","last_became_peered":"2024-03-30T13:53:14.659650+0000","last_unstale":"2024-03-30T13:58:17.734405+0000","last_undegraded":"2024-03-30T13:58:17.734405+0000","last_fullsized":"2024-03-30T13:58:17.734405+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:13.029471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:13.029471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:16.554109+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,5],"avail_no_missing":["1","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":559,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.746740+0000","last_change":"2024-03-30T13:57:04.801994+0000","last_active":"2024-03-30T13:58:17.746740+0000","last_peered":"2024-03-30T13:58:17.746740+0000","last_clean":"2024-03-30T13:58:17.746740+0000","last_became_active":"2024-03-30T13:53:36.857751+0000","last_became_peered":"2024-03-30T13:53:36.857751+0000","last_unstale":"2024-03-30T13:58:17.746740+0000","last_undegraded":"2024-03-30T13:58:17.746740+0000","last_fullsized":"2024-03-30T13:58:17.746740+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:04.801834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:04.801834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":560,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.733049+0000","last_change":"2024-03-30T13:58:14.227217+0000","last_active":"2024-03-30T13:58:17.733049+0000","last_peered":"2024-03-30T13:58:17.733049+0000","last_clean":"2024-03-30T13:58:17.733049+0000","last_became_active":"2024-03-30T13:42:45.310165+0000","last_became_peered":"2024-03-30T13:42:45.310165+0000","last_unstale":"2024-03-30T13:58:17.733049+0000","last_undegraded":"2024-03-30T13:58:17.733049+0000","last_fullsized":"2024-03-30T13:58:17.733049+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:14.224328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:14.224328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:35.989219+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"217'19","reported_seq":619,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.734560+0000","last_change":"2024-03-30T13:57:11.200267+0000","last_active":"2024-03-30T13:58:17.734560+0000","last_peered":"2024-03-30T13:58:17.734560+0000","last_clean":"2024-03-30T13:58:17.734560+0000","last_became_active":"2024-03-30T13:53:13.650706+0000","last_became_peered":"2024-03-30T13:53:13.650706+0000","last_unstale":"2024-03-30T13:58:17.734560+0000","last_undegraded":"2024-03-30T13:58:17.734560+0000","last_fullsized":"2024-03-30T13:58:17.734560+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'19","last_scrub_stamp":"2024-03-30T13:57:11.200126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:11.200126+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:36.452224+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":422,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.729159+0000","last_change":"2024-03-30T13:57:03.611202+0000","last_active":"2024-03-30T13:58:17.729159+0000","last_peered":"2024-03-30T13:58:17.729159+0000","last_clean":"2024-03-30T13:58:17.729159+0000","last_became_active":"2024-03-30T13:53:36.863938+0000","last_became_peered":"2024-03-30T13:53:36.863938+0000","last_unstale":"2024-03-30T13:58:17.729159+0000","last_undegraded":"2024-03-30T13:58:17.729159+0000","last_fullsized":"2024-03-30T13:58:17.729159+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:03.611042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:03.611042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3,7],"avail_no_missing":["4","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.732922+0000","last_change":"2024-03-30T13:56:57.393571+0000","last_active":"2024-03-30T13:58:17.732922+0000","last_peered":"2024-03-30T13:58:17.732922+0000","last_clean":"2024-03-30T13:58:17.732922+0000","last_became_active":"2024-03-30T13:53:35.862467+0000","last_became_peered":"2024-03-30T13:53:35.862467+0000","last_unstale":"2024-03-30T13:58:17.732922+0000","last_undegraded":"2024-03-30T13:58:17.732922+0000","last_fullsized":"2024-03-30T13:58:17.732922+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:57.393495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:54.893653+0000","last_clean_scrub_stamp":"2024-03-30T13:56:57.393495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":580,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.744688+0000","last_change":"2024-03-30T13:58:16.924928+0000","last_active":"2024-03-30T13:58:17.744688+0000","last_peered":"2024-03-30T13:58:17.744688+0000","last_clean":"2024-03-30T13:58:17.744688+0000","last_became_active":"2024-03-30T13:53:13.661128+0000","last_became_peered":"2024-03-30T13:53:13.661128+0000","last_unstale":"2024-03-30T13:58:17.744688+0000","last_undegraded":"2024-03-30T13:58:17.744688+0000","last_fullsized":"2024-03-30T13:58:17.744688+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:16.924797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:16.924797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:29.938729+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"216'8","reported_seq":568,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.733148+0000","last_change":"2024-03-30T13:57:07.952284+0000","last_active":"2024-03-30T13:58:17.733148+0000","last_peered":"2024-03-30T13:58:17.733148+0000","last_clean":"2024-03-30T13:58:17.733148+0000","last_became_active":"2024-03-30T13:42:47.321428+0000","last_became_peered":"2024-03-30T13:42:47.321428+0000","last_unstale":"2024-03-30T13:58:17.733148+0000","last_undegraded":"2024-03-30T13:58:17.733148+0000","last_fullsized":"2024-03-30T13:58:17.733148+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'8","last_scrub_stamp":"2024-03-30T13:57:07.952145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:07.952145+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:21.304369+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":541,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.734399+0000","last_change":"2024-03-30T13:52:26.149264+0000","last_active":"2024-03-30T13:58:17.734399+0000","last_peered":"2024-03-30T13:58:17.734399+0000","last_clean":"2024-03-30T13:58:17.734399+0000","last_became_active":"2024-03-30T13:42:51.374304+0000","last_became_peered":"2024-03-30T13:42:51.374304+0000","last_unstale":"2024-03-30T13:58:17.734399+0000","last_undegraded":"2024-03-30T13:58:17.734399+0000","last_fullsized":"2024-03-30T13:58:17.734399+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:26.149132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:26.149132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":189,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.729138+0000","last_change":"2024-03-30T13:53:35.854347+0000","last_active":"2024-03-30T13:58:17.729138+0000","last_peered":"2024-03-30T13:58:17.729138+0000","last_clean":"2024-03-30T13:58:17.729138+0000","last_became_active":"2024-03-30T13:53:35.854002+0000","last_became_peered":"2024-03-30T13:53:35.854002+0000","last_unstale":"2024-03-30T13:58:17.729138+0000","last_undegraded":"2024-03-30T13:58:17.729138+0000","last_fullsized":"2024-03-30T13:58:17.729138+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:56.878297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:42.616155+0000","last_clean_scrub_stamp":"2024-03-30T13:52:56.878297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":198,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.734569+0000","last_change":"2024-03-30T13:57:07.098975+0000","last_active":"2024-03-30T13:58:17.734569+0000","last_peered":"2024-03-30T13:58:17.734569+0000","last_clean":"2024-03-30T13:58:17.734569+0000","last_became_active":"2024-03-30T13:53:36.869434+0000","last_became_peered":"2024-03-30T13:53:36.869434+0000","last_unstale":"2024-03-30T13:58:17.734569+0000","last_undegraded":"2024-03-30T13:58:17.734569+0000","last_fullsized":"2024-03-30T13:58:17.734569+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:07.098782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:25.494903+0000","last_clean_scrub_stamp":"2024-03-30T13:57:07.098782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3,7],"avail_no_missing":["1","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"217'19","reported_seq":605,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.734237+0000","last_change":"2024-03-30T13:53:36.012242+0000","last_active":"2024-03-30T13:58:17.734237+0000","last_peered":"2024-03-30T13:58:17.734237+0000","last_clean":"2024-03-30T13:58:17.734237+0000","last_became_active":"2024-03-30T13:53:35.863880+0000","last_became_peered":"2024-03-30T13:53:35.863880+0000","last_unstale":"2024-03-30T13:58:17.734237+0000","last_undegraded":"2024-03-30T13:58:17.734237+0000","last_fullsized":"2024-03-30T13:58:17.734237+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'19","last_scrub_stamp":"2024-03-30T13:52:57.970377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:57.970377+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":560,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.741592+0000","last_change":"2024-03-30T13:57:13.749488+0000","last_active":"2024-03-30T13:58:17.741592+0000","last_peered":"2024-03-30T13:58:17.741592+0000","last_clean":"2024-03-30T13:58:17.741592+0000","last_became_active":"2024-03-30T13:53:13.650411+0000","last_became_peered":"2024-03-30T13:53:13.650411+0000","last_unstale":"2024-03-30T13:58:17.741592+0000","last_undegraded":"2024-03-30T13:58:17.741592+0000","last_fullsized":"2024-03-30T13:58:17.741592+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:13.749366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:28.314391+0000","last_clean_scrub_stamp":"2024-03-30T13:57:13.749366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:32.857101+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":544,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.732808+0000","last_change":"2024-03-30T13:52:16.009739+0000","last_active":"2024-03-30T13:58:17.732808+0000","last_peered":"2024-03-30T13:58:17.732808+0000","last_clean":"2024-03-30T13:58:17.732808+0000","last_became_active":"2024-03-30T13:42:49.325219+0000","last_became_peered":"2024-03-30T13:42:49.325219+0000","last_unstale":"2024-03-30T13:58:17.732808+0000","last_undegraded":"2024-03-30T13:58:17.732808+0000","last_fullsized":"2024-03-30T13:58:17.732808+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:16.009499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:16.009499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":197,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.732533+0000","last_change":"2024-03-30T13:57:19.071948+0000","last_active":"2024-03-30T13:58:17.732533+0000","last_peered":"2024-03-30T13:58:17.732533+0000","last_clean":"2024-03-30T13:58:17.732533+0000","last_became_active":"2024-03-30T13:53:35.859253+0000","last_became_peered":"2024-03-30T13:53:35.859253+0000","last_unstale":"2024-03-30T13:58:17.732533+0000","last_undegraded":"2024-03-30T13:58:17.732533+0000","last_fullsized":"2024-03-30T13:58:17.732533+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:19.071814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:11.701294+0000","last_clean_scrub_stamp":"2024-03-30T13:57:19.071814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:40.463550+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"216'15","reported_seq":573,"reported_epoch":317,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:58:17.738557+0000","last_change":"2024-03-30T13:50:08.513156+0000","last_active":"2024-03-30T13:58:17.738557+0000","last_peered":"2024-03-30T13:58:17.738557+0000","last_clean":"2024-03-30T13:50:05.496678+0000","last_became_active":"2024-03-30T13:50:08.513156+0000","last_became_peered":"2024-03-30T13:50:08.513156+0000","last_unstale":"2024-03-30T13:58:17.738557+0000","last_undegraded":"2024-03-30T13:50:08.507726+0000","last_fullsized":"2024-03-30T13:50:08.507616+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:49:26.480540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:49:26.480540+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":553,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.733446+0000","last_change":"2024-03-30T13:57:17.058336+0000","last_active":"2024-03-30T13:58:17.733446+0000","last_peered":"2024-03-30T13:58:17.733446+0000","last_clean":"2024-03-30T13:58:17.733446+0000","last_became_active":"2024-03-30T13:53:14.822437+0000","last_became_peered":"2024-03-30T13:53:14.822437+0000","last_unstale":"2024-03-30T13:58:17.733446+0000","last_undegraded":"2024-03-30T13:58:17.733446+0000","last_fullsized":"2024-03-30T13:58:17.733446+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:17.058094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:24.598976+0000","last_clean_scrub_stamp":"2024-03-30T13:57:17.058094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:26.763149+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0,5],"avail_no_missing":["3","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":577,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.738369+0000","last_change":"2024-03-30T13:58:02.985628+0000","last_active":"2024-03-30T13:58:17.738369+0000","last_peered":"2024-03-30T13:58:17.738369+0000","last_clean":"2024-03-30T13:58:17.738369+0000","last_became_active":"2024-03-30T13:53:35.870924+0000","last_became_peered":"2024-03-30T13:53:35.870924+0000","last_unstale":"2024-03-30T13:58:17.738369+0000","last_undegraded":"2024-03-30T13:58:17.738369+0000","last_fullsized":"2024-03-30T13:58:17.738369+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:02.985176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:34.815783+0000","last_clean_scrub_stamp":"2024-03-30T13:58:02.985176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:03.839994+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":565,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.730268+0000","last_change":"2024-03-30T13:53:36.849109+0000","last_active":"2024-03-30T13:58:17.730268+0000","last_peered":"2024-03-30T13:58:17.730268+0000","last_clean":"2024-03-30T13:58:17.730268+0000","last_became_active":"2024-03-30T13:53:36.848656+0000","last_became_peered":"2024-03-30T13:53:36.848656+0000","last_unstale":"2024-03-30T13:58:17.730268+0000","last_undegraded":"2024-03-30T13:58:17.730268+0000","last_fullsized":"2024-03-30T13:58:17.730268+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:14.015356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:14.015356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,7],"avail_no_missing":["4","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"216'19","reported_seq":592,"reported_epoch":317,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:58:17.729594+0000","last_change":"2024-03-30T13:50:08.513382+0000","last_active":"2024-03-30T13:58:17.729594+0000","last_peered":"2024-03-30T13:58:17.729594+0000","last_clean":"2024-03-30T13:50:05.491029+0000","last_became_active":"2024-03-30T13:50:08.513382+0000","last_became_peered":"2024-03-30T13:50:08.513382+0000","last_unstale":"2024-03-30T13:58:17.729594+0000","last_undegraded":"2024-03-30T13:50:08.503759+0000","last_fullsized":"2024-03-30T13:50:08.503643+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:48:22.068206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:22.068206+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:18.505152+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":560,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.745449+0000","last_change":"2024-03-30T13:53:35.856379+0000","last_active":"2024-03-30T13:58:17.745449+0000","last_peered":"2024-03-30T13:58:17.745449+0000","last_clean":"2024-03-30T13:58:17.745449+0000","last_became_active":"2024-03-30T13:53:35.855996+0000","last_became_peered":"2024-03-30T13:53:35.855996+0000","last_unstale":"2024-03-30T13:58:17.745449+0000","last_undegraded":"2024-03-30T13:58:17.745449+0000","last_fullsized":"2024-03-30T13:58:17.745449+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:56.942611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:56.942611+0000","last_clean_scrub_stamp":"2024-03-30T13:52:56.942611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"222'9","reported_seq":303,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.735091+0000","last_change":"2024-03-30T13:57:57.056334+0000","last_active":"2024-03-30T13:58:17.735091+0000","last_peered":"2024-03-30T13:58:17.735091+0000","last_clean":"2024-03-30T13:58:17.735091+0000","last_became_active":"2024-03-30T13:53:13.651996+0000","last_became_peered":"2024-03-30T13:53:13.651996+0000","last_unstale":"2024-03-30T13:58:17.735091+0000","last_undegraded":"2024-03-30T13:58:17.735091+0000","last_fullsized":"2024-03-30T13:58:17.735091+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'9","last_scrub_stamp":"2024-03-30T13:57:57.056166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:57.056166+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:13.651057+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":434,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.729739+0000","last_change":"2024-03-30T13:57:04.646873+0000","last_active":"2024-03-30T13:58:17.729739+0000","last_peered":"2024-03-30T13:58:17.729739+0000","last_clean":"2024-03-30T13:58:17.729739+0000","last_became_active":"2024-03-30T13:53:36.849971+0000","last_became_peered":"2024-03-30T13:53:36.849971+0000","last_unstale":"2024-03-30T13:58:17.729739+0000","last_undegraded":"2024-03-30T13:58:17.729739+0000","last_fullsized":"2024-03-30T13:58:17.729739+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:04.646741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:57:04.646741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:25.974096+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,7],"avail_no_missing":["4","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"217'20","reported_seq":608,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.735115+0000","last_change":"2024-03-30T13:56:58.941975+0000","last_active":"2024-03-30T13:58:17.735115+0000","last_peered":"2024-03-30T13:58:17.735115+0000","last_clean":"2024-03-30T13:58:17.735115+0000","last_became_active":"2024-03-30T13:53:37.110194+0000","last_became_peered":"2024-03-30T13:53:37.110194+0000","last_unstale":"2024-03-30T13:58:17.735115+0000","last_undegraded":"2024-03-30T13:58:17.735115+0000","last_fullsized":"2024-03-30T13:58:17.735115+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'20","last_scrub_stamp":"2024-03-30T13:56:58.941755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:56:58.941755+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":548,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.729912+0000","last_change":"2024-03-30T13:57:06.727910+0000","last_active":"2024-03-30T13:58:17.729912+0000","last_peered":"2024-03-30T13:58:17.729912+0000","last_clean":"2024-03-30T13:58:17.729912+0000","last_became_active":"2024-03-30T13:53:14.658940+0000","last_became_peered":"2024-03-30T13:53:14.658940+0000","last_unstale":"2024-03-30T13:58:17.729912+0000","last_undegraded":"2024-03-30T13:58:17.729912+0000","last_fullsized":"2024-03-30T13:58:17.729912+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:06.727774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:06.727774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,5],"avail_no_missing":["4","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"230'11","reported_seq":754,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.735428+0000","last_change":"2024-03-30T13:57:01.991621+0000","last_active":"2024-03-30T13:58:17.735428+0000","last_peered":"2024-03-30T13:58:17.735428+0000","last_clean":"2024-03-30T13:58:17.735428+0000","last_became_active":"2024-03-30T13:53:37.127826+0000","last_became_peered":"2024-03-30T13:53:37.127826+0000","last_unstale":"2024-03-30T13:58:17.735428+0000","last_undegraded":"2024-03-30T13:58:17.735428+0000","last_fullsized":"2024-03-30T13:58:17.735428+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'11","last_scrub_stamp":"2024-03-30T13:57:01.991476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:01.991476+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":572,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.743374+0000","last_change":"2024-03-30T13:58:03.861834+0000","last_active":"2024-03-30T13:58:17.743374+0000","last_peered":"2024-03-30T13:58:17.743374+0000","last_clean":"2024-03-30T13:58:17.743374+0000","last_became_active":"2024-03-30T13:53:35.857456+0000","last_became_peered":"2024-03-30T13:53:35.857456+0000","last_unstale":"2024-03-30T13:58:17.743374+0000","last_undegraded":"2024-03-30T13:58:17.743374+0000","last_fullsized":"2024-03-30T13:58:17.743374+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:03.861718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:28.734080+0000","last_clean_scrub_stamp":"2024-03-30T13:58:03.861718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:15.297097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"217'15","reported_seq":583,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.732806+0000","last_change":"2024-03-30T13:57:12.040139+0000","last_active":"2024-03-30T13:58:17.732806+0000","last_peered":"2024-03-30T13:58:17.732806+0000","last_clean":"2024-03-30T13:58:17.732806+0000","last_became_active":"2024-03-30T13:53:35.865891+0000","last_became_peered":"2024-03-30T13:53:35.865891+0000","last_unstale":"2024-03-30T13:58:17.732806+0000","last_undegraded":"2024-03-30T13:58:17.732806+0000","last_fullsized":"2024-03-30T13:58:17.732806+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'15","last_scrub_stamp":"2024-03-30T13:57:12.039992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:12.039992+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:21.020394+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":336,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.731743+0000","last_change":"2024-03-30T13:57:01.689045+0000","last_active":"2024-03-30T13:58:17.731743+0000","last_peered":"2024-03-30T13:58:17.731743+0000","last_clean":"2024-03-30T13:58:17.731743+0000","last_became_active":"2024-03-30T13:53:36.857026+0000","last_became_peered":"2024-03-30T13:53:36.857026+0000","last_unstale":"2024-03-30T13:58:17.731743+0000","last_undegraded":"2024-03-30T13:58:17.731743+0000","last_fullsized":"2024-03-30T13:58:17.731743+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:01.688893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:01.688893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:30.571747+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,7],"avail_no_missing":["4","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":558,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.733912+0000","last_change":"2024-03-30T13:58:03.191278+0000","last_active":"2024-03-30T13:58:17.733912+0000","last_peered":"2024-03-30T13:58:17.733912+0000","last_clean":"2024-03-30T13:58:17.733912+0000","last_became_active":"2024-03-30T13:53:35.875783+0000","last_became_peered":"2024-03-30T13:53:35.875783+0000","last_unstale":"2024-03-30T13:58:17.733912+0000","last_undegraded":"2024-03-30T13:58:17.733912+0000","last_fullsized":"2024-03-30T13:58:17.733912+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:03.191052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:58:03.191052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:21.859148+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":197,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.732004+0000","last_change":"2024-03-30T13:58:16.531285+0000","last_active":"2024-03-30T13:58:17.732004+0000","last_peered":"2024-03-30T13:58:17.732004+0000","last_clean":"2024-03-30T13:58:17.732004+0000","last_became_active":"2024-03-30T13:53:35.858261+0000","last_became_peered":"2024-03-30T13:53:35.858261+0000","last_unstale":"2024-03-30T13:58:17.732004+0000","last_undegraded":"2024-03-30T13:58:17.732004+0000","last_fullsized":"2024-03-30T13:58:17.732004+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:16.530762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:16.530762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:27.572482+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"218'16","reported_seq":593,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.746470+0000","last_change":"2024-03-30T13:53:36.861902+0000","last_active":"2024-03-30T13:58:17.746470+0000","last_peered":"2024-03-30T13:58:17.746470+0000","last_clean":"2024-03-30T13:58:17.746470+0000","last_became_active":"2024-03-30T13:53:36.861455+0000","last_became_peered":"2024-03-30T13:53:36.861455+0000","last_unstale":"2024-03-30T13:58:17.746470+0000","last_undegraded":"2024-03-30T13:58:17.746470+0000","last_fullsized":"2024-03-30T13:58:17.746470+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'16","last_scrub_stamp":"2024-03-30T13:52:59.883027+0000","last_deep_scrub":"218'16","last_deep_scrub_stamp":"2024-03-30T13:52:59.883027+0000","last_clean_scrub_stamp":"2024-03-30T13:52:59.883027+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3,7],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"72'1","reported_seq":212,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.734240+0000","last_change":"2024-03-30T13:57:05.320998+0000","last_active":"2024-03-30T13:58:17.734240+0000","last_peered":"2024-03-30T13:58:17.734240+0000","last_clean":"2024-03-30T13:58:17.734240+0000","last_became_active":"2024-03-30T13:53:35.864200+0000","last_became_peered":"2024-03-30T13:53:35.864200+0000","last_unstale":"2024-03-30T13:58:17.734240+0000","last_undegraded":"2024-03-30T13:58:17.734240+0000","last_fullsized":"2024-03-30T13:58:17.734240+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T13:57:05.320870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:05.320870+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:28.158735+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":212,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.734348+0000","last_change":"2024-03-30T13:57:09.967641+0000","last_active":"2024-03-30T13:58:17.734348+0000","last_peered":"2024-03-30T13:58:17.734348+0000","last_clean":"2024-03-30T13:58:17.734348+0000","last_became_active":"2024-03-30T13:53:13.661052+0000","last_became_peered":"2024-03-30T13:53:13.661052+0000","last_unstale":"2024-03-30T13:58:17.734348+0000","last_undegraded":"2024-03-30T13:58:17.734348+0000","last_fullsized":"2024-03-30T13:58:17.734348+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:09.967517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:57:09.967517+0000","last_clean_scrub_stamp":"2024-03-30T13:57:09.967517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.733007+0000","last_change":"2024-03-30T13:57:17.132124+0000","last_active":"2024-03-30T13:58:17.733007+0000","last_peered":"2024-03-30T13:58:17.733007+0000","last_clean":"2024-03-30T13:58:17.733007+0000","last_became_active":"2024-03-30T13:53:13.657810+0000","last_became_peered":"2024-03-30T13:53:13.657810+0000","last_unstale":"2024-03-30T13:58:17.733007+0000","last_undegraded":"2024-03-30T13:58:17.733007+0000","last_fullsized":"2024-03-30T13:58:17.733007+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:17.132009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:51:20.175438+0000","last_clean_scrub_stamp":"2024-03-30T13:57:17.132009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:43.306526+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"316'15","reported_seq":582,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.733806+0000","last_change":"2024-03-30T13:58:13.210664+0000","last_active":"2024-03-30T13:58:17.733806+0000","last_peered":"2024-03-30T13:58:17.733806+0000","last_clean":"2024-03-30T13:58:17.733806+0000","last_became_active":"2024-03-30T13:53:37.098834+0000","last_became_peered":"2024-03-30T13:53:37.098834+0000","last_unstale":"2024-03-30T13:58:17.733806+0000","last_undegraded":"2024-03-30T13:58:17.733806+0000","last_fullsized":"2024-03-30T13:58:17.733806+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'15","last_scrub_stamp":"2024-03-30T13:58:13.210392+0000","last_deep_scrub":"316'15","last_deep_scrub_stamp":"2024-03-30T13:58:13.210392+0000","last_clean_scrub_stamp":"2024-03-30T13:58:13.210392+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:34.819233+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"317'8435","reported_seq":8500,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:20.565214+0000","last_change":"2024-03-30T13:56:55.955502+0000","last_active":"2024-03-30T13:58:20.565214+0000","last_peered":"2024-03-30T13:58:20.565214+0000","last_clean":"2024-03-30T13:58:20.565214+0000","last_became_active":"2024-03-30T13:56:55.955212+0000","last_became_peered":"2024-03-30T13:56:55.955212+0000","last_unstale":"2024-03-30T13:58:20.565214+0000","last_undegraded":"2024-03-30T13:58:20.565214+0000","last_fullsized":"2024-03-30T13:58:20.565214+0000","mapping_epoch":289,"log_start":"306'4700","ondisk_log_start":"306'4700","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":552796160,"num_objects":8435,"num_object_clones":0,"num_object_copies":25305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8435,"num_write_kb":539840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":219,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.733045+0000","last_change":"2024-03-30T13:57:56.190301+0000","last_active":"2024-03-30T13:58:17.733045+0000","last_peered":"2024-03-30T13:58:17.733045+0000","last_clean":"2024-03-30T13:58:17.733045+0000","last_became_active":"2024-03-30T13:53:14.655517+0000","last_became_peered":"2024-03-30T13:53:14.655517+0000","last_unstale":"2024-03-30T13:58:17.733045+0000","last_undegraded":"2024-03-30T13:58:17.733045+0000","last_fullsized":"2024-03-30T13:58:17.733045+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:56.190076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:56.190076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:12.664415+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0,5],"avail_no_missing":["6","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":204,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.729527+0000","last_change":"2024-03-30T13:53:36.015472+0000","last_active":"2024-03-30T13:58:17.729527+0000","last_peered":"2024-03-30T13:58:17.729527+0000","last_clean":"2024-03-30T13:58:17.729527+0000","last_became_active":"2024-03-30T13:53:35.874521+0000","last_became_peered":"2024-03-30T13:53:35.874521+0000","last_unstale":"2024-03-30T13:58:17.729527+0000","last_undegraded":"2024-03-30T13:58:17.729527+0000","last_fullsized":"2024-03-30T13:58:17.729527+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-30T13:52:09.680567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:09.680567+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"216'27","reported_seq":613,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.733040+0000","last_change":"2024-03-30T13:58:08.152191+0000","last_active":"2024-03-30T13:58:17.733040+0000","last_peered":"2024-03-30T13:58:17.733040+0000","last_clean":"2024-03-30T13:58:17.733040+0000","last_became_active":"2024-03-30T13:53:35.887732+0000","last_became_peered":"2024-03-30T13:53:35.887732+0000","last_unstale":"2024-03-30T13:58:17.733040+0000","last_undegraded":"2024-03-30T13:58:17.733040+0000","last_fullsized":"2024-03-30T13:58:17.733040+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'27","last_scrub_stamp":"2024-03-30T13:58:08.152048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:58:08.152048+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:36.857346+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"229'9","reported_seq":291,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.732800+0000","last_change":"2024-03-30T13:57:55.147011+0000","last_active":"2024-03-30T13:58:17.732800+0000","last_peered":"2024-03-30T13:58:17.732800+0000","last_clean":"2024-03-30T13:58:17.732800+0000","last_became_active":"2024-03-30T13:53:35.862126+0000","last_became_peered":"2024-03-30T13:53:35.862126+0000","last_unstale":"2024-03-30T13:58:17.732800+0000","last_undegraded":"2024-03-30T13:58:17.732800+0000","last_fullsized":"2024-03-30T13:58:17.732800+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'9","last_scrub_stamp":"2024-03-30T13:57:55.146893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:55.146893+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:17.257278+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"317'8400","reported_seq":8463,"reported_epoch":317,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:58:18.715813+0000","last_change":"2024-03-30T13:57:08.380068+0000","last_active":"2024-03-30T13:58:18.715813+0000","last_peered":"2024-03-30T13:58:18.715813+0000","last_clean":"2024-03-30T13:56:54.930191+0000","last_became_active":"2024-03-30T13:56:55.955577+0000","last_became_peered":"2024-03-30T13:56:55.955577+0000","last_unstale":"2024-03-30T13:58:18.715813+0000","last_undegraded":"2024-03-30T13:57:08.376613+0000","last_fullsized":"2024-03-30T13:56:55.947822+0000","mapping_epoch":289,"log_start":"306'4300","ondisk_log_start":"306'4300","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4100,"log_dups_size":0,"ondisk_log_size":4100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":550502400,"num_objects":8400,"num_object_clones":0,"num_object_copies":25200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8400,"num_write_kb":537600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":8400}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":554,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.734281+0000","last_change":"2024-03-30T13:58:12.361908+0000","last_active":"2024-03-30T13:58:17.734281+0000","last_peered":"2024-03-30T13:58:17.734281+0000","last_clean":"2024-03-30T13:58:17.734281+0000","last_became_active":"2024-03-30T13:53:35.862056+0000","last_became_peered":"2024-03-30T13:53:35.862056+0000","last_unstale":"2024-03-30T13:58:17.734281+0000","last_undegraded":"2024-03-30T13:58:17.734281+0000","last_fullsized":"2024-03-30T13:58:17.734281+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:12.361782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:12.361782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:14.785915+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":197,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.729841+0000","last_change":"2024-03-30T13:58:13.511206+0000","last_active":"2024-03-30T13:58:17.729841+0000","last_peered":"2024-03-30T13:58:17.729841+0000","last_clean":"2024-03-30T13:58:17.729841+0000","last_became_active":"2024-03-30T13:53:35.861370+0000","last_became_peered":"2024-03-30T13:53:35.861370+0000","last_unstale":"2024-03-30T13:58:17.729841+0000","last_undegraded":"2024-03-30T13:58:17.729841+0000","last_fullsized":"2024-03-30T13:58:17.729841+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:13.511094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:13.511094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:24.132961+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"216'18","reported_seq":589,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.732057+0000","last_change":"2024-03-30T13:52:11.994083+0000","last_active":"2024-03-30T13:58:17.732057+0000","last_peered":"2024-03-30T13:58:17.732057+0000","last_clean":"2024-03-30T13:58:17.732057+0000","last_became_active":"2024-03-30T13:42:47.316187+0000","last_became_peered":"2024-03-30T13:42:47.316187+0000","last_unstale":"2024-03-30T13:58:17.732057+0000","last_undegraded":"2024-03-30T13:58:17.732057+0000","last_fullsized":"2024-03-30T13:58:17.732057+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-30T13:52:11.993946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:11.993946+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":324,"reported_epoch":317,"state":"active+undersized","last_fresh":"2024-03-30T13:58:17.742136+0000","last_change":"2024-03-30T13:50:08.509701+0000","last_active":"2024-03-30T13:58:17.742136+0000","last_peered":"2024-03-30T13:58:17.742136+0000","last_clean":"2024-03-30T13:50:05.499237+0000","last_became_active":"2024-03-30T13:50:08.509701+0000","last_became_peered":"2024-03-30T13:50:08.509701+0000","last_unstale":"2024-03-30T13:58:17.742136+0000","last_undegraded":"2024-03-30T13:58:17.742136+0000","last_fullsized":"2024-03-30T13:50:08.501671+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:22.401087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:22.401087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:21.914250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"317'8262","reported_seq":8328,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:20.526470+0000","last_change":"2024-03-30T13:56:55.952448+0000","last_active":"2024-03-30T13:58:20.526470+0000","last_peered":"2024-03-30T13:58:20.526470+0000","last_clean":"2024-03-30T13:58:20.526470+0000","last_became_active":"2024-03-30T13:56:55.951841+0000","last_became_peered":"2024-03-30T13:56:55.951841+0000","last_unstale":"2024-03-30T13:58:20.526470+0000","last_undegraded":"2024-03-30T13:58:20.526470+0000","last_fullsized":"2024-03-30T13:58:20.526470+0000","mapping_epoch":289,"log_start":"306'4600","ondisk_log_start":"306'4600","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":3662,"log_dups_size":0,"ondisk_log_size":3662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":541458432,"num_objects":8262,"num_object_clones":0,"num_object_copies":24786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8262,"num_write_kb":528768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":431,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.732841+0000","last_change":"2024-03-30T13:57:03.960613+0000","last_active":"2024-03-30T13:58:17.732841+0000","last_peered":"2024-03-30T13:58:17.732841+0000","last_clean":"2024-03-30T13:58:17.732841+0000","last_became_active":"2024-03-30T13:53:36.862437+0000","last_became_peered":"2024-03-30T13:53:36.862437+0000","last_unstale":"2024-03-30T13:58:17.732841+0000","last_undegraded":"2024-03-30T13:58:17.732841+0000","last_fullsized":"2024-03-30T13:58:17.732841+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:03.960362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:03.960362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,7],"avail_no_missing":["3","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":342,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.742020+0000","last_change":"2024-03-30T13:57:06.715296+0000","last_active":"2024-03-30T13:58:17.742020+0000","last_peered":"2024-03-30T13:58:17.742020+0000","last_clean":"2024-03-30T13:58:17.742020+0000","last_became_active":"2024-03-30T13:53:36.850279+0000","last_became_peered":"2024-03-30T13:53:36.850279+0000","last_unstale":"2024-03-30T13:58:17.742020+0000","last_undegraded":"2024-03-30T13:58:17.742020+0000","last_fullsized":"2024-03-30T13:58:17.742020+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:06.715115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:06.715115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:18.975417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":560,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.734070+0000","last_change":"2024-03-30T13:58:16.087808+0000","last_active":"2024-03-30T13:58:17.734070+0000","last_peered":"2024-03-30T13:58:17.734070+0000","last_clean":"2024-03-30T13:58:17.734070+0000","last_became_active":"2024-03-30T13:53:35.852959+0000","last_became_peered":"2024-03-30T13:53:35.852959+0000","last_unstale":"2024-03-30T13:58:17.734070+0000","last_undegraded":"2024-03-30T13:58:17.734070+0000","last_fullsized":"2024-03-30T13:58:17.734070+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:16.087693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:17.366740+0000","last_clean_scrub_stamp":"2024-03-30T13:58:16.087693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:37.823766+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"217'44","reported_seq":660,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.733637+0000","last_change":"2024-03-30T13:56:24.249597+0000","last_active":"2024-03-30T13:58:17.733637+0000","last_peered":"2024-03-30T13:58:17.733637+0000","last_clean":"2024-03-30T13:58:17.733637+0000","last_became_active":"2024-03-30T13:53:35.862052+0000","last_became_peered":"2024-03-30T13:53:35.862052+0000","last_unstale":"2024-03-30T13:58:17.733637+0000","last_undegraded":"2024-03-30T13:58:17.733637+0000","last_fullsized":"2024-03-30T13:58:17.733637+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'44","last_scrub_stamp":"2024-03-30T13:56:24.249485+0000","last_deep_scrub":"108'28","last_deep_scrub_stamp":"2024-03-30T13:45:49.140418+0000","last_clean_scrub_stamp":"2024-03-30T13:56:24.249485+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"317'8339","reported_seq":8402,"reported_epoch":317,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:58:20.560217+0000","last_change":"2024-03-30T13:57:08.381005+0000","last_active":"2024-03-30T13:58:20.560217+0000","last_peered":"2024-03-30T13:58:20.560217+0000","last_clean":"2024-03-30T13:56:54.930191+0000","last_became_active":"2024-03-30T13:56:55.950831+0000","last_became_peered":"2024-03-30T13:56:55.950831+0000","last_unstale":"2024-03-30T13:58:20.560217+0000","last_undegraded":"2024-03-30T13:57:08.371957+0000","last_fullsized":"2024-03-30T13:56:55.946732+0000","mapping_epoch":289,"log_start":"306'4600","ondisk_log_start":"306'4600","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":3739,"log_dups_size":0,"ondisk_log_size":3739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":546504704,"num_objects":8339,"num_object_clones":0,"num_object_copies":25017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8339,"num_write_kb":533696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":8339}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":547,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.745006+0000","last_change":"2024-03-30T13:56:59.724692+0000","last_active":"2024-03-30T13:58:17.745006+0000","last_peered":"2024-03-30T13:58:17.745006+0000","last_clean":"2024-03-30T13:58:17.745006+0000","last_became_active":"2024-03-30T13:53:37.359258+0000","last_became_peered":"2024-03-30T13:53:37.359258+0000","last_unstale":"2024-03-30T13:58:17.745006+0000","last_undegraded":"2024-03-30T13:58:17.745006+0000","last_fullsized":"2024-03-30T13:58:17.745006+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:59.724551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:44.095338+0000","last_clean_scrub_stamp":"2024-03-30T13:56:59.724551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:24.306217+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3,7],"avail_no_missing":["0","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":323,"reported_epoch":317,"state":"active+undersized","last_fresh":"2024-03-30T13:58:17.731191+0000","last_change":"2024-03-30T13:50:08.507875+0000","last_active":"2024-03-30T13:58:17.731191+0000","last_peered":"2024-03-30T13:58:17.731191+0000","last_clean":"2024-03-30T13:50:05.490245+0000","last_became_active":"2024-03-30T13:50:08.507875+0000","last_became_peered":"2024-03-30T13:50:08.507875+0000","last_unstale":"2024-03-30T13:58:17.731191+0000","last_undegraded":"2024-03-30T13:58:17.731191+0000","last_fullsized":"2024-03-30T13:50:08.502700+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:13.399683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:28.225742+0000","last_clean_scrub_stamp":"2024-03-30T13:49:13.399683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:18.505182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":212,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.734959+0000","last_change":"2024-03-30T13:57:04.919188+0000","last_active":"2024-03-30T13:58:17.734959+0000","last_peered":"2024-03-30T13:58:17.734959+0000","last_clean":"2024-03-30T13:58:17.734959+0000","last_became_active":"2024-03-30T13:53:13.666227+0000","last_became_peered":"2024-03-30T13:53:13.666227+0000","last_unstale":"2024-03-30T13:58:17.734959+0000","last_undegraded":"2024-03-30T13:58:17.734959+0000","last_fullsized":"2024-03-30T13:58:17.734959+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:04.919039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:57:04.919039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:30.945262+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"216'15","reported_seq":627,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.742249+0000","last_change":"2024-03-30T13:57:58.907928+0000","last_active":"2024-03-30T13:58:17.742249+0000","last_peered":"2024-03-30T13:58:17.742249+0000","last_clean":"2024-03-30T13:58:17.742249+0000","last_became_active":"2024-03-30T13:53:13.656465+0000","last_became_peered":"2024-03-30T13:53:13.656465+0000","last_unstale":"2024-03-30T13:58:17.742249+0000","last_undegraded":"2024-03-30T13:58:17.742249+0000","last_fullsized":"2024-03-30T13:58:17.742249+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'15","last_scrub_stamp":"2024-03-30T13:57:58.907779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:58.907779+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:19.149130+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"317'8278","reported_seq":8341,"reported_epoch":317,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:58:18.715803+0000","last_change":"2024-03-30T13:57:08.405740+0000","last_active":"2024-03-30T13:58:18.715803+0000","last_peered":"2024-03-30T13:58:18.715803+0000","last_clean":"2024-03-30T13:56:54.930191+0000","last_became_active":"2024-03-30T13:56:55.952697+0000","last_became_peered":"2024-03-30T13:56:55.952697+0000","last_unstale":"2024-03-30T13:58:18.715803+0000","last_undegraded":"2024-03-30T13:57:08.401357+0000","last_fullsized":"2024-03-30T13:56:55.946905+0000","mapping_epoch":289,"log_start":"305'4200","ondisk_log_start":"305'4200","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":542507008,"num_objects":8278,"num_object_clones":0,"num_object_copies":24834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8278,"num_write_kb":529792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":8278}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":518,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.744568+0000","last_change":"2024-03-30T13:56:54.694994+0000","last_active":"2024-03-30T13:58:17.744568+0000","last_peered":"2024-03-30T13:58:17.744568+0000","last_clean":"2024-03-30T13:58:17.744568+0000","last_became_active":"2024-03-30T13:53:16.690961+0000","last_became_peered":"2024-03-30T13:53:16.690961+0000","last_unstale":"2024-03-30T13:58:17.744568+0000","last_undegraded":"2024-03-30T13:58:17.744568+0000","last_fullsized":"2024-03-30T13:58:17.744568+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:56:54.694786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.694786+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4,5],"avail_no_missing":["0","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":555,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.734517+0000","last_change":"2024-03-30T13:57:57.278091+0000","last_active":"2024-03-30T13:58:17.734517+0000","last_peered":"2024-03-30T13:58:17.734517+0000","last_clean":"2024-03-30T13:58:17.734517+0000","last_became_active":"2024-03-30T13:53:35.855186+0000","last_became_peered":"2024-03-30T13:53:35.855186+0000","last_unstale":"2024-03-30T13:58:17.734517+0000","last_undegraded":"2024-03-30T13:58:17.734517+0000","last_fullsized":"2024-03-30T13:58:17.734517+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:57.277970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:57.277970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:12.590589+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":238,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.735307+0000","last_change":"2024-03-30T13:58:03.986390+0000","last_active":"2024-03-30T13:58:17.735307+0000","last_peered":"2024-03-30T13:58:17.735307+0000","last_clean":"2024-03-30T13:58:17.735307+0000","last_became_active":"2024-03-30T13:53:13.655879+0000","last_became_peered":"2024-03-30T13:53:13.655879+0000","last_unstale":"2024-03-30T13:58:17.735307+0000","last_undegraded":"2024-03-30T13:58:17.735307+0000","last_fullsized":"2024-03-30T13:58:17.735307+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-30T13:58:03.985919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:58:03.985919+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:32.282760+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":561,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.729335+0000","last_change":"2024-03-30T13:58:04.478097+0000","last_active":"2024-03-30T13:58:17.729335+0000","last_peered":"2024-03-30T13:58:17.729335+0000","last_clean":"2024-03-30T13:58:17.729335+0000","last_became_active":"2024-03-30T13:42:45.310209+0000","last_became_peered":"2024-03-30T13:42:45.310209+0000","last_unstale":"2024-03-30T13:58:17.729335+0000","last_undegraded":"2024-03-30T13:58:17.729335+0000","last_fullsized":"2024-03-30T13:58:17.729335+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:04.478028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:04.478028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:21.405419+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"217'41","reported_seq":798,"reported_epoch":317,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:58:17.734926+0000","last_change":"2024-03-30T13:50:08.508727+0000","last_active":"2024-03-30T13:58:17.734926+0000","last_peered":"2024-03-30T13:58:17.734926+0000","last_clean":"2024-03-30T13:50:05.490389+0000","last_became_active":"2024-03-30T13:50:08.508727+0000","last_became_peered":"2024-03-30T13:50:08.508727+0000","last_unstale":"2024-03-30T13:58:17.734926+0000","last_undegraded":"2024-03-30T13:50:08.506113+0000","last_fullsized":"2024-03-30T13:50:08.505993+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-03-30T13:48:27.377242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:27.377242+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":567,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.744705+0000","last_change":"2024-03-30T13:58:05.858433+0000","last_active":"2024-03-30T13:58:17.744705+0000","last_peered":"2024-03-30T13:58:17.744705+0000","last_clean":"2024-03-30T13:58:17.744705+0000","last_became_active":"2024-03-30T13:42:49.322154+0000","last_became_peered":"2024-03-30T13:42:49.322154+0000","last_unstale":"2024-03-30T13:58:17.744705+0000","last_undegraded":"2024-03-30T13:58:17.744705+0000","last_fullsized":"2024-03-30T13:58:17.744705+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:05.858319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:18.862784+0000","last_clean_scrub_stamp":"2024-03-30T13:58:05.858319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:08.445018+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"317'8347","reported_seq":8410,"reported_epoch":317,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:58:18.654803+0000","last_change":"2024-03-30T13:57:08.373694+0000","last_active":"2024-03-30T13:58:18.654803+0000","last_peered":"2024-03-30T13:58:18.654803+0000","last_clean":"2024-03-30T13:56:54.930191+0000","last_became_active":"2024-03-30T13:56:55.946947+0000","last_became_peered":"2024-03-30T13:56:55.946947+0000","last_unstale":"2024-03-30T13:58:18.654803+0000","last_undegraded":"2024-03-30T13:57:08.373558+0000","last_fullsized":"2024-03-30T13:56:55.941745+0000","mapping_epoch":289,"log_start":"306'4800","ondisk_log_start":"306'4800","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":3547,"log_dups_size":0,"ondisk_log_size":3547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:19.345214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":547028992,"num_objects":8347,"num_object_clones":0,"num_object_copies":25041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8347,"num_write_kb":534208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":8347}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":493,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.745032+0000","last_change":"2024-03-30T13:56:52.636459+0000","last_active":"2024-03-30T13:58:17.745032+0000","last_peered":"2024-03-30T13:58:17.745032+0000","last_clean":"2024-03-30T13:58:17.745032+0000","last_became_active":"2024-03-30T13:53:36.851158+0000","last_became_peered":"2024-03-30T13:53:36.851158+0000","last_unstale":"2024-03-30T13:58:17.745032+0000","last_undegraded":"2024-03-30T13:58:17.745032+0000","last_fullsized":"2024-03-30T13:58:17.745032+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:56:52.636303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:56:52.636303+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4,7],"avail_no_missing":["0","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":204,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.734168+0000","last_change":"2024-03-30T13:58:04.252009+0000","last_active":"2024-03-30T13:58:17.734168+0000","last_peered":"2024-03-30T13:58:17.734168+0000","last_clean":"2024-03-30T13:58:17.734168+0000","last_became_active":"2024-03-30T13:53:36.860597+0000","last_became_peered":"2024-03-30T13:53:36.860597+0000","last_unstale":"2024-03-30T13:58:17.734168+0000","last_undegraded":"2024-03-30T13:58:17.734168+0000","last_fullsized":"2024-03-30T13:58:17.734168+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:04.251869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:04.251869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:08.363084+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3,7],"avail_no_missing":["6","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":555,"reported_epoch":317,"state":"active+undersized","last_fresh":"2024-03-30T13:58:17.739536+0000","last_change":"2024-03-30T13:50:08.512561+0000","last_active":"2024-03-30T13:58:17.739536+0000","last_peered":"2024-03-30T13:58:17.739536+0000","last_clean":"2024-03-30T13:50:05.489457+0000","last_became_active":"2024-03-30T13:50:08.512561+0000","last_became_peered":"2024-03-30T13:50:08.512561+0000","last_unstale":"2024-03-30T13:58:17.739536+0000","last_undegraded":"2024-03-30T13:58:17.739536+0000","last_fullsized":"2024-03-30T13:50:08.506886+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:16.464234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:59.810189+0000","last_clean_scrub_stamp":"2024-03-30T13:49:16.464234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":325,"reported_epoch":317,"state":"active+undersized","last_fresh":"2024-03-30T13:58:17.735668+0000","last_change":"2024-03-30T13:50:08.511337+0000","last_active":"2024-03-30T13:58:17.735668+0000","last_peered":"2024-03-30T13:58:17.735668+0000","last_clean":"2024-03-30T13:50:05.498892+0000","last_became_active":"2024-03-30T13:50:08.511337+0000","last_became_peered":"2024-03-30T13:50:08.511337+0000","last_unstale":"2024-03-30T13:58:17.735668+0000","last_undegraded":"2024-03-30T13:58:17.735668+0000","last_fullsized":"2024-03-30T13:50:08.501456+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:25.398232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:51.919203+0000","last_clean_scrub_stamp":"2024-03-30T13:49:25.398232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"216'17","reported_seq":641,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.736779+0000","last_change":"2024-03-30T13:57:15.236817+0000","last_active":"2024-03-30T13:58:17.736779+0000","last_peered":"2024-03-30T13:58:17.736779+0000","last_clean":"2024-03-30T13:58:17.736779+0000","last_became_active":"2024-03-30T13:53:35.867159+0000","last_became_peered":"2024-03-30T13:53:35.867159+0000","last_unstale":"2024-03-30T13:58:17.736779+0000","last_undegraded":"2024-03-30T13:58:17.736779+0000","last_fullsized":"2024-03-30T13:58:17.736779+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'17","last_scrub_stamp":"2024-03-30T13:57:15.236671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:15.236671+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:28.405018+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":216,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.729591+0000","last_change":"2024-03-30T13:57:05.690641+0000","last_active":"2024-03-30T13:58:17.729591+0000","last_peered":"2024-03-30T13:58:17.729591+0000","last_clean":"2024-03-30T13:58:17.729591+0000","last_became_active":"2024-03-30T13:53:36.864365+0000","last_became_peered":"2024-03-30T13:53:36.864365+0000","last_unstale":"2024-03-30T13:58:17.729591+0000","last_undegraded":"2024-03-30T13:58:17.729591+0000","last_fullsized":"2024-03-30T13:58:17.729591+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:05.690511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:28.373277+0000","last_clean_scrub_stamp":"2024-03-30T13:57:05.690511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:22.643597+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,7],"avail_no_missing":["4","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.b","version":"317'8128","reported_seq":8194,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:18.706300+0000","last_change":"2024-03-30T13:56:55.953007+0000","last_active":"2024-03-30T13:58:18.706300+0000","last_peered":"2024-03-30T13:58:18.706300+0000","last_clean":"2024-03-30T13:58:18.706300+0000","last_became_active":"2024-03-30T13:56:55.952660+0000","last_became_peered":"2024-03-30T13:56:55.952660+0000","last_unstale":"2024-03-30T13:58:18.706300+0000","last_undegraded":"2024-03-30T13:58:18.706300+0000","last_fullsized":"2024-03-30T13:58:18.706300+0000","mapping_epoch":289,"log_start":"305'4000","ondisk_log_start":"305'4000","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4128,"log_dups_size":0,"ondisk_log_size":4128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":532676608,"num_objects":8128,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8128,"num_write_kb":520192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":205,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.734023+0000","last_change":"2024-03-30T13:53:13.660514+0000","last_active":"2024-03-30T13:58:17.734023+0000","last_peered":"2024-03-30T13:58:17.734023+0000","last_clean":"2024-03-30T13:58:17.734023+0000","last_became_active":"2024-03-30T13:53:13.660154+0000","last_became_peered":"2024-03-30T13:53:13.660154+0000","last_unstale":"2024-03-30T13:58:17.734023+0000","last_undegraded":"2024-03-30T13:58:17.734023+0000","last_fullsized":"2024-03-30T13:58:17.734023+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:28.079029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:12.433166+0000","last_clean_scrub_stamp":"2024-03-30T13:52:28.079029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":497,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.741904+0000","last_change":"2024-03-30T13:56:53.653367+0000","last_active":"2024-03-30T13:58:17.741904+0000","last_peered":"2024-03-30T13:58:17.741904+0000","last_clean":"2024-03-30T13:58:17.741904+0000","last_became_active":"2024-03-30T13:53:38.545543+0000","last_became_peered":"2024-03-30T13:53:38.545543+0000","last_unstale":"2024-03-30T13:58:17.741904+0000","last_undegraded":"2024-03-30T13:58:17.741904+0000","last_fullsized":"2024-03-30T13:58:17.741904+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:56:53.653242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:56:53.653242+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":567,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.733699+0000","last_change":"2024-03-30T13:58:16.238325+0000","last_active":"2024-03-30T13:58:17.733699+0000","last_peered":"2024-03-30T13:58:17.733699+0000","last_clean":"2024-03-30T13:58:17.733699+0000","last_became_active":"2024-03-30T13:53:13.662164+0000","last_became_peered":"2024-03-30T13:53:13.662164+0000","last_unstale":"2024-03-30T13:58:17.733699+0000","last_undegraded":"2024-03-30T13:58:17.733699+0000","last_fullsized":"2024-03-30T13:58:17.733699+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:16.238202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:12.975288+0000","last_clean_scrub_stamp":"2024-03-30T13:58:16.238202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:16.243750+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"229'29","reported_seq":821,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.734114+0000","last_change":"2024-03-30T13:57:02.380198+0000","last_active":"2024-03-30T13:58:17.734114+0000","last_peered":"2024-03-30T13:58:17.734114+0000","last_clean":"2024-03-30T13:58:17.734114+0000","last_became_active":"2024-03-30T13:53:35.864416+0000","last_became_peered":"2024-03-30T13:53:35.864416+0000","last_unstale":"2024-03-30T13:58:17.734114+0000","last_undegraded":"2024-03-30T13:58:17.734114+0000","last_fullsized":"2024-03-30T13:58:17.734114+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'29","last_scrub_stamp":"2024-03-30T13:57:02.380102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:02.380102+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":547,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.746619+0000","last_change":"2024-03-30T13:57:05.760106+0000","last_active":"2024-03-30T13:58:17.746619+0000","last_peered":"2024-03-30T13:58:17.746619+0000","last_clean":"2024-03-30T13:58:17.746619+0000","last_became_active":"2024-03-30T13:53:36.853892+0000","last_became_peered":"2024-03-30T13:53:36.853892+0000","last_unstale":"2024-03-30T13:58:17.746619+0000","last_undegraded":"2024-03-30T13:58:17.746619+0000","last_fullsized":"2024-03-30T13:58:17.746619+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:05.759975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:05.759975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:31.802854+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.8","version":"317'8434","reported_seq":8497,"reported_epoch":317,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:58:20.535222+0000","last_change":"2024-03-30T13:57:08.372715+0000","last_active":"2024-03-30T13:58:20.535222+0000","last_peered":"2024-03-30T13:58:20.535222+0000","last_clean":"2024-03-30T13:56:54.930191+0000","last_became_active":"2024-03-30T13:56:55.945757+0000","last_became_peered":"2024-03-30T13:56:55.945757+0000","last_unstale":"2024-03-30T13:58:20.535222+0000","last_undegraded":"2024-03-30T13:57:08.372522+0000","last_fullsized":"2024-03-30T13:56:55.944337+0000","mapping_epoch":289,"log_start":"306'4700","ondisk_log_start":"306'4700","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":3734,"log_dups_size":0,"ondisk_log_size":3734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:21.074035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":552730624,"num_objects":8434,"num_object_clones":0,"num_object_copies":25302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8434,"num_write_kb":539776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":8434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":547,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.737342+0000","last_change":"2024-03-30T13:58:01.061819+0000","last_active":"2024-03-30T13:58:17.737342+0000","last_peered":"2024-03-30T13:58:17.737342+0000","last_clean":"2024-03-30T13:58:17.737342+0000","last_became_active":"2024-03-30T13:52:36.489806+0000","last_became_peered":"2024-03-30T13:52:36.489806+0000","last_unstale":"2024-03-30T13:58:17.737342+0000","last_undegraded":"2024-03-30T13:58:17.737342+0000","last_fullsized":"2024-03-30T13:58:17.737342+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:58:01.061633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:58:01.061633+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:06.886356+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":562,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.734212+0000","last_change":"2024-03-30T13:56:59.155399+0000","last_active":"2024-03-30T13:58:17.734212+0000","last_peered":"2024-03-30T13:58:17.734212+0000","last_clean":"2024-03-30T13:58:17.734212+0000","last_became_active":"2024-03-30T13:53:13.664207+0000","last_became_peered":"2024-03-30T13:53:13.664207+0000","last_unstale":"2024-03-30T13:58:17.734212+0000","last_undegraded":"2024-03-30T13:58:17.734212+0000","last_fullsized":"2024-03-30T13:58:17.734212+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:59.155318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:56:59.155318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":567,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.744800+0000","last_change":"2024-03-30T13:57:17.744960+0000","last_active":"2024-03-30T13:58:17.744800+0000","last_peered":"2024-03-30T13:58:17.744800+0000","last_clean":"2024-03-30T13:58:17.744800+0000","last_became_active":"2024-03-30T13:42:45.307905+0000","last_became_peered":"2024-03-30T13:42:45.307905+0000","last_unstale":"2024-03-30T13:58:17.744800+0000","last_undegraded":"2024-03-30T13:58:17.744800+0000","last_fullsized":"2024-03-30T13:58:17.744800+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:17.744839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:57:17.744839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:21.974721+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":530,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:19.676764+0000","last_change":"2024-03-30T13:57:17.250283+0000","last_active":"2024-03-30T13:58:19.676764+0000","last_peered":"2024-03-30T13:58:19.676764+0000","last_clean":"2024-03-30T13:58:19.676764+0000","last_became_active":"2024-03-30T13:53:35.889071+0000","last_became_peered":"2024-03-30T13:53:35.889071+0000","last_unstale":"2024-03-30T13:58:19.676764+0000","last_undegraded":"2024-03-30T13:58:19.676764+0000","last_fullsized":"2024-03-30T13:58:19.676764+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-30T13:57:17.250150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:57:17.250150+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:24.323932+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.00126391,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":290,"num_read_kb":285,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"216'30","reported_seq":592,"reported_epoch":317,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:58:17.746079+0000","last_change":"2024-03-30T13:50:08.510256+0000","last_active":"2024-03-30T13:58:17.746079+0000","last_peered":"2024-03-30T13:58:17.746079+0000","last_clean":"2024-03-30T13:50:05.503358+0000","last_became_active":"2024-03-30T13:50:08.510256+0000","last_became_peered":"2024-03-30T13:50:08.510256+0000","last_unstale":"2024-03-30T13:58:17.746079+0000","last_undegraded":"2024-03-30T13:50:08.504536+0000","last_fullsized":"2024-03-30T13:50:08.504398+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'18","last_scrub_stamp":"2024-03-30T13:49:13.774982+0000","last_deep_scrub":"93'18","last_deep_scrub_stamp":"2024-03-30T13:47:28.872529+0000","last_clean_scrub_stamp":"2024-03-30T13:49:13.774982+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:21.914267+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":211,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.737014+0000","last_change":"2024-03-30T13:56:56.114039+0000","last_active":"2024-03-30T13:58:17.737014+0000","last_peered":"2024-03-30T13:58:17.737014+0000","last_clean":"2024-03-30T13:58:17.737014+0000","last_became_active":"2024-03-30T13:53:13.649288+0000","last_became_peered":"2024-03-30T13:53:13.649288+0000","last_unstale":"2024-03-30T13:58:17.737014+0000","last_undegraded":"2024-03-30T13:58:17.737014+0000","last_fullsized":"2024-03-30T13:58:17.737014+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:56.113961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:56:56.113961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"317'8154","reported_seq":8218,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:18.731606+0000","last_change":"2024-03-30T13:56:55.953429+0000","last_active":"2024-03-30T13:58:18.731606+0000","last_peered":"2024-03-30T13:58:18.731606+0000","last_clean":"2024-03-30T13:58:18.731606+0000","last_became_active":"2024-03-30T13:56:55.953088+0000","last_became_peered":"2024-03-30T13:56:55.953088+0000","last_unstale":"2024-03-30T13:58:18.731606+0000","last_undegraded":"2024-03-30T13:58:18.731606+0000","last_fullsized":"2024-03-30T13:58:18.731606+0000","mapping_epoch":289,"log_start":"305'4000","ondisk_log_start":"305'4000","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4154,"log_dups_size":0,"ondisk_log_size":4154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534380544,"num_objects":8154,"num_object_clones":0,"num_object_copies":24462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8154,"num_write_kb":521856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":552,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.734074+0000","last_change":"2024-03-30T13:57:14.131392+0000","last_active":"2024-03-30T13:58:17.734074+0000","last_peered":"2024-03-30T13:58:17.734074+0000","last_clean":"2024-03-30T13:58:17.734074+0000","last_became_active":"2024-03-30T13:52:36.483298+0000","last_became_peered":"2024-03-30T13:52:36.483298+0000","last_unstale":"2024-03-30T13:58:17.734074+0000","last_undegraded":"2024-03-30T13:58:17.734074+0000","last_fullsized":"2024-03-30T13:58:17.734074+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:57:14.131256+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T13:57:14.131256+0000","last_clean_scrub_stamp":"2024-03-30T13:57:14.131256+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:21.260624+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":543,"reported_epoch":317,"state":"active+undersized","last_fresh":"2024-03-30T13:58:17.735506+0000","last_change":"2024-03-30T13:50:08.512648+0000","last_active":"2024-03-30T13:58:17.735506+0000","last_peered":"2024-03-30T13:58:17.735506+0000","last_clean":"2024-03-30T13:50:05.489180+0000","last_became_active":"2024-03-30T13:50:08.512648+0000","last_became_peered":"2024-03-30T13:50:08.512648+0000","last_unstale":"2024-03-30T13:58:17.735506+0000","last_undegraded":"2024-03-30T13:58:17.735506+0000","last_fullsized":"2024-03-30T13:50:08.505931+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:21.527712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:21.527712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":576,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.734719+0000","last_change":"2024-03-30T13:58:11.910087+0000","last_active":"2024-03-30T13:58:17.734719+0000","last_peered":"2024-03-30T13:58:17.734719+0000","last_clean":"2024-03-30T13:58:17.734719+0000","last_became_active":"2024-03-30T13:53:14.658165+0000","last_became_peered":"2024-03-30T13:53:14.658165+0000","last_unstale":"2024-03-30T13:58:17.734719+0000","last_undegraded":"2024-03-30T13:58:17.734719+0000","last_fullsized":"2024-03-30T13:58:17.734719+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:11.909857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:45.975163+0000","last_clean_scrub_stamp":"2024-03-30T13:58:11.909857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:15.612889+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0,5],"avail_no_missing":["1","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"216'29","reported_seq":643,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.742489+0000","last_change":"2024-03-30T13:57:59.935648+0000","last_active":"2024-03-30T13:58:17.742489+0000","last_peered":"2024-03-30T13:58:17.742489+0000","last_clean":"2024-03-30T13:58:17.742489+0000","last_became_active":"2024-03-30T13:53:36.857573+0000","last_became_peered":"2024-03-30T13:53:36.857573+0000","last_unstale":"2024-03-30T13:58:17.742489+0000","last_undegraded":"2024-03-30T13:58:17.742489+0000","last_fullsized":"2024-03-30T13:58:17.742489+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'29","last_scrub_stamp":"2024-03-30T13:57:59.935365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:59.935365+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:07.329886+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[{"shards":"0,1,7","objects":8}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.733742+0000","last_change":"2024-03-30T13:57:16.165427+0000","last_active":"2024-03-30T13:58:17.733742+0000","last_peered":"2024-03-30T13:58:17.733742+0000","last_clean":"2024-03-30T13:58:17.733742+0000","last_became_active":"2024-03-30T13:53:14.654346+0000","last_became_peered":"2024-03-30T13:53:14.654346+0000","last_unstale":"2024-03-30T13:58:17.733742+0000","last_undegraded":"2024-03-30T13:58:17.733742+0000","last_fullsized":"2024-03-30T13:58:17.733742+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:16.165134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:52.887014+0000","last_clean_scrub_stamp":"2024-03-30T13:57:16.165134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:30.941683+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0,5],"avail_no_missing":["6","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"317'8194","reported_seq":8260,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:18.629647+0000","last_change":"2024-03-30T13:56:55.948911+0000","last_active":"2024-03-30T13:58:18.629647+0000","last_peered":"2024-03-30T13:58:18.629647+0000","last_clean":"2024-03-30T13:58:18.629647+0000","last_became_active":"2024-03-30T13:56:55.948591+0000","last_became_peered":"2024-03-30T13:56:55.948591+0000","last_unstale":"2024-03-30T13:58:18.629647+0000","last_undegraded":"2024-03-30T13:58:18.629647+0000","last_fullsized":"2024-03-30T13:58:18.629647+0000","mapping_epoch":289,"log_start":"306'4600","ondisk_log_start":"306'4600","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":3594,"log_dups_size":0,"ondisk_log_size":3594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537001984,"num_objects":8194,"num_object_clones":0,"num_object_copies":24582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8194,"num_write_kb":524416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":557,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.734468+0000","last_change":"2024-03-30T13:57:18.133598+0000","last_active":"2024-03-30T13:58:17.734468+0000","last_peered":"2024-03-30T13:58:17.734468+0000","last_clean":"2024-03-30T13:58:17.734468+0000","last_became_active":"2024-03-30T13:53:37.107659+0000","last_became_peered":"2024-03-30T13:53:37.107659+0000","last_unstale":"2024-03-30T13:58:17.734468+0000","last_undegraded":"2024-03-30T13:58:17.734468+0000","last_fullsized":"2024-03-30T13:58:17.734468+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:57:18.133345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:57:18.133345+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:38.756415+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3,7],"avail_no_missing":["6","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":542,"reported_epoch":317,"state":"active+undersized","last_fresh":"2024-03-30T13:58:17.741703+0000","last_change":"2024-03-30T13:50:08.510025+0000","last_active":"2024-03-30T13:58:17.741703+0000","last_peered":"2024-03-30T13:58:17.741703+0000","last_clean":"2024-03-30T13:50:05.498930+0000","last_became_active":"2024-03-30T13:50:08.510025+0000","last_became_peered":"2024-03-30T13:50:08.510025+0000","last_unstale":"2024-03-30T13:58:17.741703+0000","last_undegraded":"2024-03-30T13:58:17.741703+0000","last_fullsized":"2024-03-30T13:50:08.502478+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:28.946303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:41.142941+0000","last_clean_scrub_stamp":"2024-03-30T13:49:28.946303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:21.914281+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":458,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.732326+0000","last_change":"2024-03-30T13:56:59.986364+0000","last_active":"2024-03-30T13:58:17.732326+0000","last_peered":"2024-03-30T13:58:17.732326+0000","last_clean":"2024-03-30T13:58:17.732326+0000","last_became_active":"2024-03-30T13:53:14.650772+0000","last_became_peered":"2024-03-30T13:53:14.650772+0000","last_unstale":"2024-03-30T13:58:17.732326+0000","last_undegraded":"2024-03-30T13:58:17.732326+0000","last_fullsized":"2024-03-30T13:58:17.732326+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:59.986230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:55.458779+0000","last_clean_scrub_stamp":"2024-03-30T13:56:59.986230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,5],"avail_no_missing":["3","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"217'25","reported_seq":623,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.738461+0000","last_change":"2024-03-30T13:57:13.195976+0000","last_active":"2024-03-30T13:58:17.738461+0000","last_peered":"2024-03-30T13:58:17.738461+0000","last_clean":"2024-03-30T13:58:17.738461+0000","last_became_active":"2024-03-30T13:53:13.657632+0000","last_became_peered":"2024-03-30T13:53:13.657632+0000","last_unstale":"2024-03-30T13:58:17.738461+0000","last_undegraded":"2024-03-30T13:58:17.738461+0000","last_fullsized":"2024-03-30T13:58:17.738461+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'25","last_scrub_stamp":"2024-03-30T13:57:13.195836+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-03-30T13:45:50.945837+0000","last_clean_scrub_stamp":"2024-03-30T13:57:13.195836+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:32.425668+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"222'10","reported_seq":775,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.742750+0000","last_change":"2024-03-30T13:58:13.880934+0000","last_active":"2024-03-30T13:58:17.742750+0000","last_peered":"2024-03-30T13:58:17.742750+0000","last_clean":"2024-03-30T13:58:17.742750+0000","last_became_active":"2024-03-30T13:53:13.658557+0000","last_became_peered":"2024-03-30T13:53:13.658557+0000","last_unstale":"2024-03-30T13:58:17.742750+0000","last_undegraded":"2024-03-30T13:58:17.742750+0000","last_fullsized":"2024-03-30T13:58:17.742750+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'10","last_scrub_stamp":"2024-03-30T13:58:13.880792+0000","last_deep_scrub":"218'9","last_deep_scrub_stamp":"2024-03-30T13:52:57.873554+0000","last_clean_scrub_stamp":"2024-03-30T13:58:13.880792+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:26.029765+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"317'8089","reported_seq":8152,"reported_epoch":317,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:58:18.730110+0000","last_change":"2024-03-30T13:57:08.373767+0000","last_active":"2024-03-30T13:58:18.730110+0000","last_peered":"2024-03-30T13:58:18.730110+0000","last_clean":"2024-03-30T13:56:54.930191+0000","last_became_active":"2024-03-30T13:56:55.952893+0000","last_became_peered":"2024-03-30T13:56:55.952893+0000","last_unstale":"2024-03-30T13:58:18.730110+0000","last_undegraded":"2024-03-30T13:57:08.373532+0000","last_fullsized":"2024-03-30T13:56:55.947530+0000","mapping_epoch":289,"log_start":"305'4000","ondisk_log_start":"305'4000","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4089,"log_dups_size":0,"ondisk_log_size":4089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":530120704,"num_objects":8089,"num_object_clones":0,"num_object_copies":24267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8089,"num_write_kb":517696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":8089}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":566,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.733757+0000","last_change":"2024-03-30T13:53:38.553854+0000","last_active":"2024-03-30T13:58:17.733757+0000","last_peered":"2024-03-30T13:58:17.733757+0000","last_clean":"2024-03-30T13:58:17.733757+0000","last_became_active":"2024-03-30T13:53:38.553065+0000","last_became_peered":"2024-03-30T13:53:38.553065+0000","last_unstale":"2024-03-30T13:58:17.733757+0000","last_undegraded":"2024-03-30T13:58:17.733757+0000","last_fullsized":"2024-03-30T13:58:17.733757+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:48.866100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:48.866100+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":430,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.733458+0000","last_change":"2024-03-30T13:58:03.303778+0000","last_active":"2024-03-30T13:58:17.733458+0000","last_peered":"2024-03-30T13:58:17.733458+0000","last_clean":"2024-03-30T13:58:17.733458+0000","last_became_active":"2024-03-30T13:53:36.852004+0000","last_became_peered":"2024-03-30T13:53:36.852004+0000","last_unstale":"2024-03-30T13:58:17.733458+0000","last_undegraded":"2024-03-30T13:58:17.733458+0000","last_fullsized":"2024-03-30T13:58:17.733458+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:03.303563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:31.900908+0000","last_clean_scrub_stamp":"2024-03-30T13:58:03.303563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:18.791865+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1,7],"avail_no_missing":["6","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":576,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.746284+0000","last_change":"2024-03-30T13:58:12.869168+0000","last_active":"2024-03-30T13:58:17.746284+0000","last_peered":"2024-03-30T13:58:17.746284+0000","last_clean":"2024-03-30T13:58:17.746284+0000","last_became_active":"2024-03-30T13:42:45.309886+0000","last_became_peered":"2024-03-30T13:42:45.309886+0000","last_unstale":"2024-03-30T13:58:17.746284+0000","last_undegraded":"2024-03-30T13:58:17.746284+0000","last_fullsized":"2024-03-30T13:58:17.746284+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:58:12.869021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:12.869021+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:17.722136+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"217'35","reported_seq":636,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.731858+0000","last_change":"2024-03-30T13:53:13.687444+0000","last_active":"2024-03-30T13:58:17.731858+0000","last_peered":"2024-03-30T13:58:17.731858+0000","last_clean":"2024-03-30T13:58:17.731858+0000","last_became_active":"2024-03-30T13:53:13.652175+0000","last_became_peered":"2024-03-30T13:53:13.652175+0000","last_unstale":"2024-03-30T13:58:17.731858+0000","last_undegraded":"2024-03-30T13:58:17.731858+0000","last_fullsized":"2024-03-30T13:58:17.731858+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'21","last_scrub_stamp":"2024-03-30T13:52:44.925926+0000","last_deep_scrub":"91'21","last_deep_scrub_stamp":"2024-03-30T13:44:05.917064+0000","last_clean_scrub_stamp":"2024-03-30T13:52:44.925926+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":749,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.733773+0000","last_change":"2024-03-30T13:57:22.046830+0000","last_active":"2024-03-30T13:58:17.733773+0000","last_peered":"2024-03-30T13:58:17.733773+0000","last_clean":"2024-03-30T13:58:17.733773+0000","last_became_active":"2024-03-30T13:42:49.321423+0000","last_became_peered":"2024-03-30T13:42:49.321423+0000","last_unstale":"2024-03-30T13:58:17.733773+0000","last_undegraded":"2024-03-30T13:58:17.733773+0000","last_fullsized":"2024-03-30T13:58:17.733773+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:57:22.046689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:22.046689+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:25.809668+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"317'8214","reported_seq":8277,"reported_epoch":317,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:58:18.642666+0000","last_change":"2024-03-30T13:57:08.372029+0000","last_active":"2024-03-30T13:58:18.642666+0000","last_peered":"2024-03-30T13:58:18.642666+0000","last_clean":"2024-03-30T13:56:54.930191+0000","last_became_active":"2024-03-30T13:56:55.949377+0000","last_became_peered":"2024-03-30T13:56:55.949377+0000","last_unstale":"2024-03-30T13:58:18.642666+0000","last_undegraded":"2024-03-30T13:57:08.371728+0000","last_fullsized":"2024-03-30T13:56:55.941808+0000","mapping_epoch":289,"log_start":"306'4600","ondisk_log_start":"306'4600","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":3614,"log_dups_size":0,"ondisk_log_size":3614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538312704,"num_objects":8214,"num_object_clones":0,"num_object_copies":24642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8214,"num_write_kb":525696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":8214}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":539,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.730718+0000","last_change":"2024-03-30T13:52:47.990769+0000","last_active":"2024-03-30T13:58:17.730718+0000","last_peered":"2024-03-30T13:58:17.730718+0000","last_clean":"2024-03-30T13:58:17.730718+0000","last_became_active":"2024-03-30T13:52:36.490909+0000","last_became_peered":"2024-03-30T13:52:36.490909+0000","last_unstale":"2024-03-30T13:58:17.730718+0000","last_undegraded":"2024-03-30T13:58:17.730718+0000","last_fullsized":"2024-03-30T13:58:17.730718+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:47.990635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:47.990635+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,0],"avail_no_missing":["4","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":536,"reported_epoch":317,"state":"active+undersized","last_fresh":"2024-03-30T13:58:17.729988+0000","last_change":"2024-03-30T13:50:08.504411+0000","last_active":"2024-03-30T13:58:17.729988+0000","last_peered":"2024-03-30T13:58:17.729988+0000","last_clean":"2024-03-30T13:50:05.489944+0000","last_became_active":"2024-03-30T13:50:08.504411+0000","last_became_peered":"2024-03-30T13:50:08.504411+0000","last_unstale":"2024-03-30T13:58:17.729988+0000","last_undegraded":"2024-03-30T13:58:17.729988+0000","last_fullsized":"2024-03-30T13:50:08.501823+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:10.216387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:10.216387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:18.505141+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":567,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.733641+0000","last_change":"2024-03-30T13:53:35.851353+0000","last_active":"2024-03-30T13:58:17.733641+0000","last_peered":"2024-03-30T13:58:17.733641+0000","last_clean":"2024-03-30T13:58:17.733641+0000","last_became_active":"2024-03-30T13:53:35.851006+0000","last_became_peered":"2024-03-30T13:53:35.851006+0000","last_unstale":"2024-03-30T13:58:17.733641+0000","last_undegraded":"2024-03-30T13:58:17.733641+0000","last_fullsized":"2024-03-30T13:58:17.733641+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:53:01.031216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:53:01.031216+0000","last_clean_scrub_stamp":"2024-03-30T13:53:01.031216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"216'35","reported_seq":636,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.733620+0000","last_change":"2024-03-30T13:53:36.081652+0000","last_active":"2024-03-30T13:58:17.733620+0000","last_peered":"2024-03-30T13:58:17.733620+0000","last_clean":"2024-03-30T13:58:17.733620+0000","last_became_active":"2024-03-30T13:53:35.888652+0000","last_became_peered":"2024-03-30T13:53:35.888652+0000","last_unstale":"2024-03-30T13:58:17.733620+0000","last_undegraded":"2024-03-30T13:58:17.733620+0000","last_fullsized":"2024-03-30T13:58:17.733620+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'29","last_scrub_stamp":"2024-03-30T13:52:52.973602+0000","last_deep_scrub":"216'29","last_deep_scrub_stamp":"2024-03-30T13:52:52.973602+0000","last_clean_scrub_stamp":"2024-03-30T13:52:52.973602+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":559,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.731704+0000","last_change":"2024-03-30T13:57:10.660140+0000","last_active":"2024-03-30T13:58:17.731704+0000","last_peered":"2024-03-30T13:58:17.731704+0000","last_clean":"2024-03-30T13:58:17.731704+0000","last_became_active":"2024-03-30T13:53:35.863124+0000","last_became_peered":"2024-03-30T13:53:35.863124+0000","last_unstale":"2024-03-30T13:58:17.731704+0000","last_undegraded":"2024-03-30T13:58:17.731704+0000","last_fullsized":"2024-03-30T13:58:17.731704+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:10.660011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:10.660011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:31.396346+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"317'8085","reported_seq":8151,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:18.651614+0000","last_change":"2024-03-30T13:56:55.950105+0000","last_active":"2024-03-30T13:58:18.651614+0000","last_peered":"2024-03-30T13:58:18.651614+0000","last_clean":"2024-03-30T13:58:18.651614+0000","last_became_active":"2024-03-30T13:56:55.949813+0000","last_became_peered":"2024-03-30T13:56:55.949813+0000","last_unstale":"2024-03-30T13:58:18.651614+0000","last_undegraded":"2024-03-30T13:58:18.651614+0000","last_fullsized":"2024-03-30T13:58:18.651614+0000","mapping_epoch":289,"log_start":"306'4500","ondisk_log_start":"306'4500","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":3585,"log_dups_size":0,"ondisk_log_size":3585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":529858560,"num_objects":8085,"num_object_clones":0,"num_object_copies":24255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8085,"num_write_kb":517440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":517,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.732992+0000","last_change":"2024-03-30T13:58:05.194623+0000","last_active":"2024-03-30T13:58:17.732992+0000","last_peered":"2024-03-30T13:58:17.732992+0000","last_clean":"2024-03-30T13:58:17.732992+0000","last_became_active":"2024-03-30T13:53:16.672717+0000","last_became_peered":"2024-03-30T13:53:16.672717+0000","last_unstale":"2024-03-30T13:58:17.732992+0000","last_undegraded":"2024-03-30T13:58:17.732992+0000","last_fullsized":"2024-03-30T13:58:17.732992+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:58:05.194482+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T13:56:08.385059+0000","last_clean_scrub_stamp":"2024-03-30T13:58:05.194482+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:17.788428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,5],"avail_no_missing":["6","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":211,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.734196+0000","last_change":"2024-03-30T13:57:02.253896+0000","last_active":"2024-03-30T13:58:17.734196+0000","last_peered":"2024-03-30T13:58:17.734196+0000","last_clean":"2024-03-30T13:58:17.734196+0000","last_became_active":"2024-03-30T13:53:14.660483+0000","last_became_peered":"2024-03-30T13:53:14.660483+0000","last_unstale":"2024-03-30T13:58:17.734196+0000","last_undegraded":"2024-03-30T13:58:17.734196+0000","last_fullsized":"2024-03-30T13:58:17.734196+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:02.253737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:02.253737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0,5],"avail_no_missing":["1","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"223'11","reported_seq":763,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.732033+0000","last_change":"2024-03-30T13:58:06.437549+0000","last_active":"2024-03-30T13:58:17.732033+0000","last_peered":"2024-03-30T13:58:17.732033+0000","last_clean":"2024-03-30T13:58:17.732033+0000","last_became_active":"2024-03-30T13:53:14.650289+0000","last_became_peered":"2024-03-30T13:53:14.650289+0000","last_unstale":"2024-03-30T13:58:17.732033+0000","last_undegraded":"2024-03-30T13:58:17.732033+0000","last_fullsized":"2024-03-30T13:58:17.732033+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'11","last_scrub_stamp":"2024-03-30T13:58:06.437298+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-30T13:51:15.146255+0000","last_clean_scrub_stamp":"2024-03-30T13:58:06.437298+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:23.061064+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,5],"avail_no_missing":["4","0","5"],"object_location_counts":[{"shards":"0,4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":559,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.733250+0000","last_change":"2024-03-30T13:53:35.887442+0000","last_active":"2024-03-30T13:58:17.733250+0000","last_peered":"2024-03-30T13:58:17.733250+0000","last_clean":"2024-03-30T13:58:17.733250+0000","last_became_active":"2024-03-30T13:53:35.886750+0000","last_became_peered":"2024-03-30T13:53:35.886750+0000","last_unstale":"2024-03-30T13:58:17.733250+0000","last_undegraded":"2024-03-30T13:58:17.733250+0000","last_fullsized":"2024-03-30T13:58:17.733250+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:27.058807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:27.058807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"217'34","reported_seq":621,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.736567+0000","last_change":"2024-03-30T13:58:09.947780+0000","last_active":"2024-03-30T13:58:17.736567+0000","last_peered":"2024-03-30T13:58:17.736567+0000","last_clean":"2024-03-30T13:58:17.736567+0000","last_became_active":"2024-03-30T13:42:47.324350+0000","last_became_peered":"2024-03-30T13:42:47.324350+0000","last_unstale":"2024-03-30T13:58:17.736567+0000","last_undegraded":"2024-03-30T13:58:17.736567+0000","last_fullsized":"2024-03-30T13:58:17.736567+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'34","last_scrub_stamp":"2024-03-30T13:58:09.947626+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:28.831527+0000","last_clean_scrub_stamp":"2024-03-30T13:58:09.947626+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:15.653767+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"317'8267","reported_seq":8333,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:18.720865+0000","last_change":"2024-03-30T13:56:55.955019+0000","last_active":"2024-03-30T13:58:18.720865+0000","last_peered":"2024-03-30T13:58:18.720865+0000","last_clean":"2024-03-30T13:58:18.720865+0000","last_became_active":"2024-03-30T13:56:55.954690+0000","last_became_peered":"2024-03-30T13:56:55.954690+0000","last_unstale":"2024-03-30T13:58:18.720865+0000","last_undegraded":"2024-03-30T13:58:18.720865+0000","last_fullsized":"2024-03-30T13:58:18.720865+0000","mapping_epoch":289,"log_start":"306'4200","ondisk_log_start":"306'4200","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4067,"log_dups_size":0,"ondisk_log_size":4067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":541786112,"num_objects":8267,"num_object_clones":0,"num_object_copies":24801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8267,"num_write_kb":529088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":560,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.734860+0000","last_change":"2024-03-30T13:58:09.195823+0000","last_active":"2024-03-30T13:58:17.734860+0000","last_peered":"2024-03-30T13:58:17.734860+0000","last_clean":"2024-03-30T13:58:17.734860+0000","last_became_active":"2024-03-30T13:53:35.875118+0000","last_became_peered":"2024-03-30T13:53:35.875118+0000","last_unstale":"2024-03-30T13:58:17.734860+0000","last_undegraded":"2024-03-30T13:58:17.734860+0000","last_fullsized":"2024-03-30T13:58:17.734860+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:09.195697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:21.081938+0000","last_clean_scrub_stamp":"2024-03-30T13:58:09.195697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:38.573534+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":210,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.729731+0000","last_change":"2024-03-30T13:58:15.581171+0000","last_active":"2024-03-30T13:58:17.729731+0000","last_peered":"2024-03-30T13:58:17.729731+0000","last_clean":"2024-03-30T13:58:17.729731+0000","last_became_active":"2024-03-30T13:53:14.655908+0000","last_became_peered":"2024-03-30T13:53:14.655908+0000","last_unstale":"2024-03-30T13:58:17.729731+0000","last_undegraded":"2024-03-30T13:58:17.729731+0000","last_fullsized":"2024-03-30T13:58:17.729731+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:15.580974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:52.603772+0000","last_clean_scrub_stamp":"2024-03-30T13:58:15.580974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:36.742712+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,5],"avail_no_missing":["4","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":561,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.731747+0000","last_change":"2024-03-30T13:56:57.664929+0000","last_active":"2024-03-30T13:58:17.731747+0000","last_peered":"2024-03-30T13:58:17.731747+0000","last_clean":"2024-03-30T13:58:17.731747+0000","last_became_active":"2024-03-30T13:53:36.852500+0000","last_became_peered":"2024-03-30T13:53:36.852500+0000","last_unstale":"2024-03-30T13:58:17.731747+0000","last_undegraded":"2024-03-30T13:58:17.731747+0000","last_fullsized":"2024-03-30T13:58:17.731747+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:57.664795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:56:57.664795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:24.223493+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,7],"avail_no_missing":["4","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"217'19","reported_seq":595,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.730336+0000","last_change":"2024-03-30T13:52:17.057583+0000","last_active":"2024-03-30T13:58:17.730336+0000","last_peered":"2024-03-30T13:58:17.730336+0000","last_clean":"2024-03-30T13:58:17.730336+0000","last_became_active":"2024-03-30T13:42:47.315667+0000","last_became_peered":"2024-03-30T13:42:47.315667+0000","last_unstale":"2024-03-30T13:58:17.730336+0000","last_undegraded":"2024-03-30T13:58:17.730336+0000","last_fullsized":"2024-03-30T13:58:17.730336+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:52:17.057433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:17.057433+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"317'8553","reported_seq":8616,"reported_epoch":317,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:58:20.874473+0000","last_change":"2024-03-30T13:57:08.383441+0000","last_active":"2024-03-30T13:58:20.874473+0000","last_peered":"2024-03-30T13:58:20.874473+0000","last_clean":"2024-03-30T13:56:54.930191+0000","last_became_active":"2024-03-30T13:56:55.945479+0000","last_became_peered":"2024-03-30T13:56:55.945479+0000","last_unstale":"2024-03-30T13:58:20.874473+0000","last_undegraded":"2024-03-30T13:57:08.380450+0000","last_fullsized":"2024-03-30T13:56:55.942431+0000","mapping_epoch":289,"log_start":"306'4800","ondisk_log_start":"306'4800","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":560529408,"num_objects":8553,"num_object_clones":0,"num_object_copies":25659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":547392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":8553}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":517,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.732534+0000","last_change":"2024-03-30T13:56:54.941079+0000","last_active":"2024-03-30T13:58:17.732534+0000","last_peered":"2024-03-30T13:58:17.732534+0000","last_clean":"2024-03-30T13:58:17.732534+0000","last_became_active":"2024-03-30T13:53:16.671377+0000","last_became_peered":"2024-03-30T13:53:16.671377+0000","last_unstale":"2024-03-30T13:58:17.732534+0000","last_undegraded":"2024-03-30T13:58:17.732534+0000","last_fullsized":"2024-03-30T13:58:17.732534+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:56:54.940801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.940801+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:24.401844+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,5],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":324,"reported_epoch":317,"state":"active+undersized","last_fresh":"2024-03-30T13:58:17.733211+0000","last_change":"2024-03-30T13:50:08.511676+0000","last_active":"2024-03-30T13:58:17.733211+0000","last_peered":"2024-03-30T13:58:17.733211+0000","last_clean":"2024-03-30T13:50:05.504989+0000","last_became_active":"2024-03-30T13:50:08.511676+0000","last_became_peered":"2024-03-30T13:50:08.511676+0000","last_unstale":"2024-03-30T13:58:17.733211+0000","last_undegraded":"2024-03-30T13:58:17.733211+0000","last_fullsized":"2024-03-30T13:50:08.504157+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:09.357378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:09.357378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:19.213048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":567,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.734915+0000","last_change":"2024-03-30T13:57:01.170627+0000","last_active":"2024-03-30T13:58:17.734915+0000","last_peered":"2024-03-30T13:58:17.734915+0000","last_clean":"2024-03-30T13:58:17.734915+0000","last_became_active":"2024-03-30T13:42:49.321360+0000","last_became_peered":"2024-03-30T13:42:49.321360+0000","last_unstale":"2024-03-30T13:58:17.734915+0000","last_undegraded":"2024-03-30T13:58:17.734915+0000","last_fullsized":"2024-03-30T13:58:17.734915+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:01.170540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:01.170540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:18.933224+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":582,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.733252+0000","last_change":"2024-03-30T13:57:12.187448+0000","last_active":"2024-03-30T13:58:17.733252+0000","last_peered":"2024-03-30T13:58:17.733252+0000","last_clean":"2024-03-30T13:58:17.733252+0000","last_became_active":"2024-03-30T13:42:45.315251+0000","last_became_peered":"2024-03-30T13:42:45.315251+0000","last_unstale":"2024-03-30T13:58:17.733252+0000","last_undegraded":"2024-03-30T13:58:17.733252+0000","last_fullsized":"2024-03-30T13:58:17.733252+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:12.187291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:57:12.187291+0000","last_clean_scrub_stamp":"2024-03-30T13:57:12.187291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:39.267948+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"217'31","reported_seq":592,"reported_epoch":317,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:58:17.731217+0000","last_change":"2024-03-30T13:50:08.508030+0000","last_active":"2024-03-30T13:58:17.731217+0000","last_peered":"2024-03-30T13:58:17.731217+0000","last_clean":"2024-03-30T13:50:05.491159+0000","last_became_active":"2024-03-30T13:50:08.508030+0000","last_became_peered":"2024-03-30T13:50:08.508030+0000","last_unstale":"2024-03-30T13:58:17.731217+0000","last_undegraded":"2024-03-30T13:50:08.502958+0000","last_fullsized":"2024-03-30T13:50:08.502815+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-03-30T13:48:27.096261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:27.096261+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:18.505168+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"317'8540","reported_seq":8606,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:20.997994+0000","last_change":"2024-03-30T13:56:55.954799+0000","last_active":"2024-03-30T13:58:20.997994+0000","last_peered":"2024-03-30T13:58:20.997994+0000","last_clean":"2024-03-30T13:58:20.997994+0000","last_became_active":"2024-03-30T13:56:55.953793+0000","last_became_peered":"2024-03-30T13:56:55.953793+0000","last_unstale":"2024-03-30T13:58:20.997994+0000","last_undegraded":"2024-03-30T13:58:20.997994+0000","last_fullsized":"2024-03-30T13:58:20.997994+0000","mapping_epoch":289,"log_start":"307'4900","ondisk_log_start":"307'4900","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":3640,"log_dups_size":0,"ondisk_log_size":3640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":559677440,"num_objects":8540,"num_object_clones":0,"num_object_copies":25620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8540,"num_write_kb":546560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":792,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.733216+0000","last_change":"2024-03-30T13:56:55.928547+0000","last_active":"2024-03-30T13:58:17.733216+0000","last_peered":"2024-03-30T13:58:17.733216+0000","last_clean":"2024-03-30T13:58:17.733216+0000","last_became_active":"2024-03-30T13:53:38.543500+0000","last_became_peered":"2024-03-30T13:53:38.543500+0000","last_unstale":"2024-03-30T13:58:17.733216+0000","last_undegraded":"2024-03-30T13:58:17.733216+0000","last_fullsized":"2024-03-30T13:58:17.733216+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:56:55.928341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:56:55.928341+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:24.749657+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,7],"avail_no_missing":["3","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":535,"reported_epoch":317,"state":"active+undersized","last_fresh":"2024-03-30T13:58:17.730539+0000","last_change":"2024-03-30T13:50:08.511010+0000","last_active":"2024-03-30T13:58:17.730539+0000","last_peered":"2024-03-30T13:58:17.730539+0000","last_clean":"2024-03-30T13:50:05.490535+0000","last_became_active":"2024-03-30T13:50:08.511010+0000","last_became_peered":"2024-03-30T13:50:08.511010+0000","last_unstale":"2024-03-30T13:58:17.730539+0000","last_undegraded":"2024-03-30T13:58:17.730539+0000","last_fullsized":"2024-03-30T13:50:08.503118+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:06.154805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:06.154805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:18.505175+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"231'13","reported_seq":456,"reported_epoch":317,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:58:17.744366+0000","last_change":"2024-03-30T13:57:08.674470+0000","last_active":"2024-03-30T13:58:17.744366+0000","last_peered":"2024-03-30T13:58:17.744366+0000","last_clean":"2024-03-30T13:58:17.744366+0000","last_became_active":"2024-03-30T13:53:37.518434+0000","last_became_peered":"2024-03-30T13:53:37.518434+0000","last_unstale":"2024-03-30T13:58:17.744366+0000","last_undegraded":"2024-03-30T13:58:17.744366+0000","last_fullsized":"2024-03-30T13:58:17.744366+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'13","last_scrub_stamp":"2024-03-30T13:57:08.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:08.673207+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:58:34.047875+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3,7],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":560,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.732713+0000","last_change":"2024-03-30T13:57:05.955711+0000","last_active":"2024-03-30T13:58:17.732713+0000","last_peered":"2024-03-30T13:58:17.732713+0000","last_clean":"2024-03-30T13:58:17.732713+0000","last_became_active":"2024-03-30T13:42:45.309265+0000","last_became_peered":"2024-03-30T13:42:45.309265+0000","last_unstale":"2024-03-30T13:58:17.732713+0000","last_undegraded":"2024-03-30T13:58:17.732713+0000","last_fullsized":"2024-03-30T13:58:17.732713+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:05.955630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:57:05.955630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"216'20","reported_seq":586,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-03-30T13:58:17.730585+0000","last_change":"2024-03-30T13:51:59.022459+0000","last_active":"2024-03-30T13:58:17.730585+0000","last_peered":"2024-03-30T13:58:17.730585+0000","last_clean":"2024-03-30T13:58:17.730585+0000","last_became_active":"2024-03-30T13:42:47.313026+0000","last_became_peered":"2024-03-30T13:42:47.313026+0000","last_unstale":"2024-03-30T13:58:17.730585+0000","last_undegraded":"2024-03-30T13:58:17.730585+0000","last_fullsized":"2024-03-30T13:58:17.730585+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-30T13:51:59.022304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:51:59.022304+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":8697872384,"num_objects":132719,"num_object_clones":0,"num_object_copies":398157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":66654,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132719,"num_write_kb":8494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21820407808,"data_stored":21820407808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61219,"ondisk_log_size":61219,"up":40,"acting":40,"num_store_stats":5},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":290,"num_read_kb":285,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":26,"acting":33,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":5246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":86,"acting":93,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47,"num_objects_misplaced":20,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":2244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":12206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":743,"ondisk_log_size":743,"up":85,"acting":89,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":82,"acting":93,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":77,"acting":91,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":249108103360,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397700,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4283756,"kb_used_data":3652836,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":89461396,"statfs":{"total":95995035648,"available":91608469504,"internally_reserved":0,"allocated":3740504064,"data_stored":3739225183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":646052294},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575070,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3404852,"kb_used_data":2701740,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":90340300,"statfs":{"total":95995035648,"available":92508467200,"internally_reserved":0,"allocated":2766581760,"data_stored":2765229926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738595,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5981076,"kb_used_data":5314056,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":87764076,"statfs":{"total":95995035648,"available":89870413824,"internally_reserved":0,"allocated":5441593344,"data_stored":5440277954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411446,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2213288,"kb_used_data":2015520,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":91531864,"statfs":{"total":95995035648,"available":93728628736,"internally_reserved":0,"allocated":2063892480,"data_stored":2063509110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182612,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4477592,"kb_used_data":3756432,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":89267560,"statfs":{"total":95995035648,"available":91409981440,"internally_reserved":0,"allocated":3846586368,"data_stored":3845263125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":738458055},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953678,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6344712,"kb_used_data":5785980,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":87400440,"statfs":{"total":95995035648,"available":89498050560,"internally_reserved":0,"allocated":5924843520,"data_stored":5923539619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":171798692041,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":2778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5483462656,"data_stored":5483462656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2763653120,"data_stored":2763653120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3873046528,"data_stored":3873046528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5929107456,"data_stored":5929107456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3771138048,"data_stored":3771138048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:58:27.564 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-03-30T13:58:27.564 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:27.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:27.598 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:27.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:27.620 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T13:58:27.699 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:27.730 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:27.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:27.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:27.832 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:27.845 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:58:27.890 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:27.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:27.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:27.904 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:58:27.945 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:58:27.988 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:58:27.991 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:28.012 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:27 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 70223/419499 objects degraded (16.740%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:28.013 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:27 smithi043 ceph-mon[42099]: osdmap e320: 8 total, 7 up, 6 in 2024-03-30T13:58:28.013 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:27 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/984312635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:58:28.013 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:27 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 70223/419499 objects degraded (16.740%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:28.013 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:27 smithi043 ceph-mon[29963]: osdmap e320: 8 total, 7 up, 6 in 2024-03-30T13:58:28.013 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:27 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/984312635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:58:28.025 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:28.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:28.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:28.032 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:58:28.070 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:58:28.111 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:58:28.126 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:28.149 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:58:28.149 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:58:28.149 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:58:28.149 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:58:28.149 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:58:28.149 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:58:28.149 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:58:28.149 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:58:28.150 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:58:28.167 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:28.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:28.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:28.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:27 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 70223/419499 objects degraded (16.740%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:28.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:27 smithi113 ceph-mon[34341]: osdmap e320: 8 total, 7 up, 6 in 2024-03-30T13:58:28.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:27 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/984312635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:58:28.269 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:28.302 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:28.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:28.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:28.378 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-03-30T13:58:28.383753+0000 min lat: 0.00120663 max lat: 0.148211 avg lat: 0.00869903 2024-03-30T13:58:28.379 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T13:58:28.379 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 80 16 147105 147089 114.892 91.6875 0.02808 0.00869903 2024-03-30T13:58:28.404 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:28.439 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:28.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:28.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:28.541 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:28.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:28.584 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:28.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:28.685 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:28.717 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:28.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:28.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:28.819 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:28.856 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:28.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:28.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:28.958 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:28.996 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:28.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:29.096 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:29.131 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:29.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:29.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:29.234 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:28 smithi043 ceph-mon[29963]: pgmap v692: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 8.6 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 852 B/s rd, 83 MiB/s wr, 1.32k op/s; 70742/422571 objects degraded (16.741%); 27/422571 objects misplaced (0.006%) 2024-03-30T13:58:29.234 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:28 smithi043 ceph-mon[42099]: pgmap v692: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 8.6 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 852 B/s rd, 83 MiB/s wr, 1.32k op/s; 70742/422571 objects degraded (16.741%); 27/422571 objects misplaced (0.006%) 2024-03-30T13:58:29.235 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:29.285 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:29.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:29.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:29.380 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 81 16 148805 148789 114.783 106.25 0.00668079 0.0087081 2024-03-30T13:58:29.387 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:29.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:29.434 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:29.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:29.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:28 smithi113 ceph-mon[34341]: pgmap v692: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 8.6 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 852 B/s rd, 83 MiB/s wr, 1.32k op/s; 70742/422571 objects degraded (16.741%); 27/422571 objects misplaced (0.006%) 2024-03-30T13:58:29.467 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:58:29 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:58:29.535 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:29.575 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:29.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:29.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:29.678 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:29.708 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:29.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:29.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:29.750 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:58:29.750 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:58:29.772 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:58:29.810 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:29.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:29.849 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:29.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:29.950 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:29.982 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:29.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:29.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:30.084 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:30.115 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:30.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:30.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:30.171 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:58:30.172 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:30.172 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:58:30.172 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:30.217 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:30.257 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:30.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:30.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:30.359 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:30.381 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 82 16 150472 150456 114.652 104.188 0.0112319 0.00871777 2024-03-30T13:58:30.404 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:30.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:30.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:30.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:58:30.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:30.506 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:30.536 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:30.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:30.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:30.638 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:30.672 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:30.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:30.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:30.774 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:30.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:30.812 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:30.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:30.913 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:30.944 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:30.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:30.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:31.046 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:31.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:31.079 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:31.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:31.180 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:31.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:31.214 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:31.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:31.315 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:31.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:31.360 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:31.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:31.381 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 83 16 152140 152124 114.526 104.25 0.00548653 0.00872756 2024-03-30T13:58:31.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:31 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:31.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:31 smithi043 ceph-mon[29963]: pgmap v693: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 8.9 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 723 B/s rd, 117 MiB/s wr, 1.88k op/s; 73510/438609 objects degraded (16.760%); 27/438609 objects misplaced (0.006%) 2024-03-30T13:58:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:31 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:31.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:31 smithi043 ceph-mon[42099]: pgmap v693: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 8.9 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 723 B/s rd, 117 MiB/s wr, 1.88k op/s; 73510/438609 objects degraded (16.760%); 27/438609 objects misplaced (0.006%) 2024-03-30T13:58:31.461 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:31.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:31 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:31.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:31 smithi113 ceph-mon[34341]: pgmap v693: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 8.9 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 723 B/s rd, 117 MiB/s wr, 1.88k op/s; 73510/438609 objects degraded (16.760%); 27/438609 objects misplaced (0.006%) 2024-03-30T13:58:31.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:31.494 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:31.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:31.595 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:31.630 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:31.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:31.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:31.732 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:31.752 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:58:31.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:31.766 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:31.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:31.867 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:31.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:31.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:31.993 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:32.025 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:32.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:32.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:32.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:32.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:32.152 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:32.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:32.253 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:32.285 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:32.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:32.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:32.345 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:58:32.345 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-03-30T13:58:32.345 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-03-30T13:58:32.345 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-03-30T13:58:32.345 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.345 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14451.0:2640 5.14 5:2b04a3e9:::notify.6:head [watch ping cookie 94764809503872 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-03-30T13:58:32.345 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:35.493031+0000", 2024-03-30T13:58:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 295.17399256599998, 2024-03-30T13:58:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.683516824, 2024-03-30T13:58:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T13:58:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14451", 2024-03-30T13:58:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T13:58:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 2640 2024-03-30T13:58:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.493031+0000", 2024-03-30T13:58:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.493031+0000", 2024-03-30T13:58:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.493031+0000", 2024-03-30T13:58:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6599999999999996e-07 2024-03-30T13:58:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.493034+0000", 2024-03-30T13:58:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.959e-06 2024-03-30T13:58:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.493052+0000", 2024-03-30T13:58:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8020999999999999e-05 2024-03-30T13:58:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.493060+0000", 2024-03-30T13:58:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3679999999999996e-06 2024-03-30T13:58:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.493122+0000", 2024-03-30T13:58:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2626000000000007e-05 2024-03-30T13:58:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.493131+0000", 2024-03-30T13:58:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3589999999999997e-06 2024-03-30T13:58:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.850640+0000", 2024-03-30T13:58:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.35750842399999999 2024-03-30T13:58:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.850650+0000", 2024-03-30T13:58:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9990000000000003e-06 2024-03-30T13:58:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.162772+0000", 2024-03-30T13:58:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.312121995 2024-03-30T13:58:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for degraded object", 2024-03-30T13:58:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.163105+0000", 2024-03-30T13:58:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00033263699999999999 2024-03-30T13:58:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.176307+0000", 2024-03-30T13:58:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.013202551 2024-03-30T13:58:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.176404+0000", 2024-03-30T13:58:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.6759e-05 2024-03-30T13:58:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.176547+0000", 2024-03-30T13:58:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014355999999999999 2024-03-30T13:58:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14451.0:2646 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94764809503872 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:58:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:35.493442+0000", 2024-03-30T13:58:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 295.17358121199999, 2024-03-30T13:58:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7172575370000001, 2024-03-30T13:58:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T13:58:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14451", 2024-03-30T13:58:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T13:58:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 2646 2024-03-30T13:58:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.493442+0000", 2024-03-30T13:58:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.493442+0000", 2024-03-30T13:58:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.493443+0000", 2024-03-30T13:58:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4e-07 2024-03-30T13:58:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.493448+0000", 2024-03-30T13:58:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6920000000000001e-06 2024-03-30T13:58:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.493458+0000", 2024-03-30T13:58:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0514e-05 2024-03-30T13:58:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.493464+0000", 2024-03-30T13:58:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.212e-06 2024-03-30T13:58:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.493502+0000", 2024-03-30T13:58:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7982999999999999e-05 2024-03-30T13:58:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:32.356 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.493507+0000", 2024-03-30T13:58:32.356 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9599999999999999e-06 2024-03-30T13:58:32.356 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.356 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.356 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.356 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.851304+0000", 2024-03-30T13:58:32.356 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.35779657100000001 2024-03-30T13:58:32.356 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.356 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.356 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:32.356 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.851312+0000", 2024-03-30T13:58:32.356 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4300000000000006e-06 2024-03-30T13:58:32.356 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.356 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.356 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.357 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.163139+0000", 2024-03-30T13:58:32.357 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.311826317 2024-03-30T13:58:32.357 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.357 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.357 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for degraded object", 2024-03-30T13:58:32.357 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.163193+0000", 2024-03-30T13:58:32.357 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4171e-05 2024-03-30T13:58:32.357 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.357 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.357 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.357 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.176581+0000", 2024-03-30T13:58:32.358 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.013388027 2024-03-30T13:58:32.358 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.359 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.359 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:32.359 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.176661+0000", 2024-03-30T13:58:32.359 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0603000000000001e-05 2024-03-30T13:58:32.359 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.359 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.359 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:32.359 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.210699+0000", 2024-03-30T13:58:32.359 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.034038117 2024-03-30T13:58:32.359 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.359 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.359 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.360 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.360 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.360 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14451.0:2651 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94764809503872 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-03-30T13:58:32.360 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:35.850859+0000", 2024-03-30T13:58:32.360 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 294.81616450299998, 2024-03-30T13:58:32.360 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.360098982, 2024-03-30T13:58:32.360 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:32.360 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:32.360 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T13:58:32.360 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:32.360 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14451", 2024-03-30T13:58:32.360 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T13:58:32.360 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 2651 2024-03-30T13:58:32.361 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.361 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:32.361 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.361 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:32.361 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.850859+0000", 2024-03-30T13:58:32.361 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.361 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.361 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.361 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:32.361 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.850859+0000", 2024-03-30T13:58:32.361 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.361 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.361 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.361 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:32.362 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.850860+0000", 2024-03-30T13:58:32.362 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.575e-06 2024-03-30T13:58:32.362 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.362 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.362 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:32.362 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.850871+0000", 2024-03-30T13:58:32.362 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1270000000000001e-05 2024-03-30T13:58:32.362 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.362 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.362 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:32.362 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.850881+0000", 2024-03-30T13:58:32.362 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.8719999999999997e-06 2024-03-30T13:58:32.362 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.363 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.363 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:32.363 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.850891+0000", 2024-03-30T13:58:32.363 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9669999999999996e-06 2024-03-30T13:58:32.363 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.363 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.363 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.363 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.851335+0000", 2024-03-30T13:58:32.363 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000443456 2024-03-30T13:58:32.363 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.363 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.363 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T13:58:32.363 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.851355+0000", 2024-03-30T13:58:32.363 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0401999999999999e-05 2024-03-30T13:58:32.364 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.364 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.364 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.364 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.171401+0000", 2024-03-30T13:58:32.364 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3200462319999999 2024-03-30T13:58:32.364 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.364 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.364 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for degraded object", 2024-03-30T13:58:32.364 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.171473+0000", 2024-03-30T13:58:32.364 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1082999999999997e-05 2024-03-30T13:58:32.364 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.364 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.364 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.364 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.177146+0000", 2024-03-30T13:58:32.365 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0056739829999999996 2024-03-30T13:58:32.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.365 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.365 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:32.365 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.177218+0000", 2024-03-30T13:58:32.365 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1138999999999998e-05 2024-03-30T13:58:32.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.365 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.365 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:32.365 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.210958+0000", 2024-03-30T13:58:32.365 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.033740002999999998 2024-03-30T13:58:32.365 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.365 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.365 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.366 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.366 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "MOSDPGPushReply(9.8 229/228 [PushReplyOp(9:14d55118:::benchmark_data_smithi043_145506_object1012:head)])", 2024-03-30T13:58:32.366 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:36.853555+0000", 2024-03-30T13:58:32.366 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 293.81346850699998, 2024-03-30T13:58:32.366 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.21337526700000001, 2024-03-30T13:58:32.366 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:32.366 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:32.366 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-03-30T13:58:32.366 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:32.366 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.366 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:32.366 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:36.853555+0000", 2024-03-30T13:58:32.366 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.367 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:32.367 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:36.853555+0000", 2024-03-30T13:58:32.367 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.367 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:32.367 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:36.853555+0000", 2024-03-30T13:58:32.367 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.96e-07 2024-03-30T13:58:32.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.367 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:32.367 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:36.853570+0000", 2024-03-30T13:58:32.367 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5407999999999999e-05 2024-03-30T13:58:32.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.368 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:32.368 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:36.853579+0000", 2024-03-30T13:58:32.368 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.602e-06 2024-03-30T13:58:32.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.368 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:32.368 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:36.853596+0000", 2024-03-30T13:58:32.368 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7544999999999999e-05 2024-03-30T13:58:32.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.368 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.368 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:36.876610+0000", 2024-03-30T13:58:32.369 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.023013853000000001 2024-03-30T13:58:32.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.369 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:32.369 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.066930+0000", 2024-03-30T13:58:32.369 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.190319563 2024-03-30T13:58:32.369 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.369 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.369 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.369 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61746 9.8 9:1490bcb0:::benchmark_data_smithi043_145506_object61745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-03-30T13:58:32.369 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:38.352933+0000", 2024-03-30T13:58:32.369 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 292.31408981200002, 2024-03-30T13:58:32.370 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.20359448699999999, 2024-03-30T13:58:32.370 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:32.370 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:32.370 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:32.370 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:32.370 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:32.370 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:32.370 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61746 2024-03-30T13:58:32.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.370 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:32.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.370 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:32.370 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.352933+0000", 2024-03-30T13:58:32.370 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.371 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:32.371 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.352933+0000", 2024-03-30T13:58:32.371 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.371 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:32.371 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.352934+0000", 2024-03-30T13:58:32.371 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.68e-07 2024-03-30T13:58:32.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.371 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:32.371 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.352955+0000", 2024-03-30T13:58:32.371 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0899000000000001e-05 2024-03-30T13:58:32.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.372 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:32.372 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.352965+0000", 2024-03-30T13:58:32.372 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0052e-05 2024-03-30T13:58:32.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.372 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:32.372 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.352972+0000", 2024-03-30T13:58:32.372 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5860000000000001e-06 2024-03-30T13:58:32.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.372 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.372 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.365024+0000", 2024-03-30T13:58:32.373 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.012052942000000001 2024-03-30T13:58:32.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.373 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:32.373 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.365197+0000", 2024-03-30T13:58:32.373 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017210899999999999 2024-03-30T13:58:32.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.373 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-03-30T13:58:32.373 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.365442+0000", 2024-03-30T13:58:32.373 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00024565999999999998 2024-03-30T13:58:32.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.373 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:32.373 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.383419+0000", 2024-03-30T13:58:32.374 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.017977018000000001 2024-03-30T13:58:32.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.374 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.374 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.383478+0000", 2024-03-30T13:58:32.374 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8467000000000001e-05 2024-03-30T13:58:32.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.374 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.374 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.550548+0000", 2024-03-30T13:58:32.374 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.167069889 2024-03-30T13:58:32.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.375 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:32.375 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.550657+0000", 2024-03-30T13:58:32.375 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000109549 2024-03-30T13:58:32.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.375 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:32.375 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.556528+0000", 2024-03-30T13:58:32.375 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.005870648 2024-03-30T13:58:32.375 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.375 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.375 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.376 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.376 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61758 9.8 9:19dbd1b6:::benchmark_data_smithi043_145506_object61757:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-03-30T13:58:32.376 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:38.373207+0000", 2024-03-30T13:58:32.376 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 292.29381636199997, 2024-03-30T13:58:32.376 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.18663471300000001, 2024-03-30T13:58:32.376 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:32.376 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:32.376 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:32.376 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:32.376 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:32.376 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:32.376 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61758 2024-03-30T13:58:32.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.376 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:32.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.377 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:32.377 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.373207+0000", 2024-03-30T13:58:32.377 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.377 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:32.377 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.373207+0000", 2024-03-30T13:58:32.377 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.377 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:32.377 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.373209+0000", 2024-03-30T13:58:32.377 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8870000000000001e-06 2024-03-30T13:58:32.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.378 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:32.378 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.373238+0000", 2024-03-30T13:58:32.378 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9213999999999998e-05 2024-03-30T13:58:32.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.378 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:32.378 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.373252+0000", 2024-03-30T13:58:32.378 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.362e-05 2024-03-30T13:58:32.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.378 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:32.378 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.373259+0000", 2024-03-30T13:58:32.379 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8480000000000001e-06 2024-03-30T13:58:32.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.379 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.379 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.383515+0000", 2024-03-30T13:58:32.379 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010256124 2024-03-30T13:58:32.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.379 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:32.379 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.383607+0000", 2024-03-30T13:58:32.379 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1583999999999996e-05 2024-03-30T13:58:32.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.379 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-03-30T13:58:32.380 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.383713+0000", 2024-03-30T13:58:32.380 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000105461 2024-03-30T13:58:32.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.380 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.380 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.380 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.527387+0000", 2024-03-30T13:58:32.380 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.143674787 2024-03-30T13:58:32.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.380 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.380 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:32.380 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.550497+0000", 2024-03-30T13:58:32.380 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.023109667 2024-03-30T13:58:32.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.380 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.381 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.381 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.556582+0000", 2024-03-30T13:58:32.381 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0060855090000000002 2024-03-30T13:58:32.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.381 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:32.381 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.556680+0000", 2024-03-30T13:58:32.381 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7157999999999994e-05 2024-03-30T13:58:32.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.381 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:32.381 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:38.559842+0000", 2024-03-30T13:58:32.381 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003161854 2024-03-30T13:58:32.381 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.382 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.382 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.382 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.382 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:62321 9.8 9:1d32f2ca:::benchmark_data_smithi043_145506_object62320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-30T13:58:32.382 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:39.450650+0000", 2024-03-30T13:58:32.382 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 291.21637297799998, 2024-03-30T13:58:32.382 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.224521, 2024-03-30T13:58:32.382 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:32.382 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:32.382 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:32.382 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:32.382 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:32.382 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:32.383 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 62321 2024-03-30T13:58:32.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.383 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:32.383 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.383 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:32.383 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.450650+0000", 2024-03-30T13:58:32.383 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.383 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.383 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:32.383 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.450650+0000", 2024-03-30T13:58:32.383 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.383 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.384 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:32.384 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.450652+0000", 2024-03-30T13:58:32.384 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.66e-06 2024-03-30T13:58:32.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.384 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.384 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:32.384 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.450677+0000", 2024-03-30T13:58:32.384 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4941999999999999e-05 2024-03-30T13:58:32.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.384 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.384 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:32.384 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.450689+0000", 2024-03-30T13:58:32.384 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1922999999999999e-05 2024-03-30T13:58:32.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.385 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.385 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:32.385 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.450696+0000", 2024-03-30T13:58:32.385 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7279999999999995e-06 2024-03-30T13:58:32.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.385 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.385 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.385 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.454781+0000", 2024-03-30T13:58:32.385 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0040849060000000001 2024-03-30T13:58:32.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.385 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.385 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:32.385 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.454954+0000", 2024-03-30T13:58:32.385 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017218600000000001 2024-03-30T13:58:32.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.386 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.386 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-03-30T13:58:32.386 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.455148+0000", 2024-03-30T13:58:32.386 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000194711 2024-03-30T13:58:32.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.386 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.386 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.386 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.473949+0000", 2024-03-30T13:58:32.386 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.018801198000000002 2024-03-30T13:58:32.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.386 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.386 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.386 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.473988+0000", 2024-03-30T13:58:32.387 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8939000000000001e-05 2024-03-30T13:58:32.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.387 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:32.387 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.654843+0000", 2024-03-30T13:58:32.387 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.18085440899999999 2024-03-30T13:58:32.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.387 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:32.387 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.654915+0000", 2024-03-30T13:58:32.387 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2203e-05 2024-03-30T13:58:32.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.387 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:32.388 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.675171+0000", 2024-03-30T13:58:32.388 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.020256195000000001 2024-03-30T13:58:32.388 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.388 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.388 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.388 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.388 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:62350 9.8 9:1bbf8487:::benchmark_data_smithi043_145506_object62349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-30T13:58:32.388 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:39.489894+0000", 2024-03-30T13:58:32.388 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 291.17712909199997, 2024-03-30T13:58:32.388 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.20651594600000001, 2024-03-30T13:58:32.388 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:32.388 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:32.388 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 62350 2024-03-30T13:58:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.489894+0000", 2024-03-30T13:58:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.489894+0000", 2024-03-30T13:58:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.489896+0000", 2024-03-30T13:58:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.485e-06 2024-03-30T13:58:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.489923+0000", 2024-03-30T13:58:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7521e-05 2024-03-30T13:58:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.490125+0000", 2024-03-30T13:58:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00020163599999999999 2024-03-30T13:58:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.490136+0000", 2024-03-30T13:58:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1525999999999999e-05 2024-03-30T13:58:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.492613+0000", 2024-03-30T13:58:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002476677 2024-03-30T13:58:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.492768+0000", 2024-03-30T13:58:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015541400000000001 2024-03-30T13:58:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-03-30T13:58:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.492949+0000", 2024-03-30T13:58:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018092800000000001 2024-03-30T13:58:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.497943+0000", 2024-03-30T13:58:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0049938459999999997 2024-03-30T13:58:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.675227+0000", 2024-03-30T13:58:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17728360900000001 2024-03-30T13:58:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.689255+0000", 2024-03-30T13:58:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.014028489999999999 2024-03-30T13:58:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.689326+0000", 2024-03-30T13:58:32.394 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1013999999999995e-05 2024-03-30T13:58:32.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.394 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.394 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:32.394 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.696410+0000", 2024-03-30T13:58:32.394 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0070838000000000003 2024-03-30T13:58:32.394 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.394 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.394 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.394 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.394 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:62355 9.8 9:1ebd8874:::benchmark_data_smithi043_145506_object62354:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-30T13:58:32.394 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:39.496069+0000", 2024-03-30T13:58:32.394 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 291.17095381000001, 2024-03-30T13:58:32.395 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.219885634, 2024-03-30T13:58:32.395 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:32.395 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:32.395 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:32.395 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:32.395 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:32.395 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:32.395 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 62355 2024-03-30T13:58:32.395 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.395 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:32.395 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.395 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:32.395 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.496069+0000", 2024-03-30T13:58:32.396 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.396 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.396 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:32.396 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.496069+0000", 2024-03-30T13:58:32.396 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.396 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.396 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:32.396 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.496071+0000", 2024-03-30T13:58:32.396 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3790000000000001e-06 2024-03-30T13:58:32.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.396 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.396 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:32.396 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.496097+0000", 2024-03-30T13:58:32.397 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.6687999999999999e-05 2024-03-30T13:58:32.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.397 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:32.397 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.496110+0000", 2024-03-30T13:58:32.397 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2412000000000001e-05 2024-03-30T13:58:32.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.397 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:32.397 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.496119+0000", 2024-03-30T13:58:32.397 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7900000000000005e-06 2024-03-30T13:58:32.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.397 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.397 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.497980+0000", 2024-03-30T13:58:32.398 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0018611120000000001 2024-03-30T13:58:32.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.398 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:32.398 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.498166+0000", 2024-03-30T13:58:32.398 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018670899999999999 2024-03-30T13:58:32.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.398 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-03-30T13:58:32.398 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.498376+0000", 2024-03-30T13:58:32.398 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000209583 2024-03-30T13:58:32.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.398 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.399 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.654042+0000", 2024-03-30T13:58:32.399 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.15566637 2024-03-30T13:58:32.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.399 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.399 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.675280+0000", 2024-03-30T13:58:32.399 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.021237767000000001 2024-03-30T13:58:32.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.399 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:32.399 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.712800+0000", 2024-03-30T13:58:32.399 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.037520254000000003 2024-03-30T13:58:32.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.400 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:32.400 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.712852+0000", 2024-03-30T13:58:32.400 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2065999999999999e-05 2024-03-30T13:58:32.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.400 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:32.400 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.715955+0000", 2024-03-30T13:58:32.400 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0031025039999999999 2024-03-30T13:58:32.400 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.400 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.400 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.400 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:62506 9.8 9:1b8c72f3:::benchmark_data_smithi043_145506_object62505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-30T13:58:32.401 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:39.872735+0000", 2024-03-30T13:58:32.401 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 290.794288333, 2024-03-30T13:58:32.401 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.18540820099999999, 2024-03-30T13:58:32.401 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:32.401 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:32.401 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:32.401 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:32.401 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:32.401 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:32.401 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 62506 2024-03-30T13:58:32.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.401 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:32.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.402 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:32.402 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.872735+0000", 2024-03-30T13:58:32.402 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.402 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.402 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:32.402 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.872735+0000", 2024-03-30T13:58:32.402 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.402 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.402 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:32.402 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.872737+0000", 2024-03-30T13:58:32.402 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.716e-06 2024-03-30T13:58:32.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.403 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.403 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:32.403 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.872760+0000", 2024-03-30T13:58:32.403 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2969e-05 2024-03-30T13:58:32.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.403 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.403 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:32.404 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.872773+0000", 2024-03-30T13:58:32.404 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3026e-05 2024-03-30T13:58:32.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.404 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.404 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:32.404 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.872782+0000", 2024-03-30T13:58:32.404 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3640000000000005e-06 2024-03-30T13:58:32.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.404 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.404 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.404 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.882782+0000", 2024-03-30T13:58:32.404 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010000223000000001 2024-03-30T13:58:32.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.405 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:32.405 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.883026+0000", 2024-03-30T13:58:32.405 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00024386699999999999 2024-03-30T13:58:32.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.405 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-03-30T13:58:32.405 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.883730+0000", 2024-03-30T13:58:32.405 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00070403100000000004 2024-03-30T13:58:32.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.405 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.406 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.906675+0000", 2024-03-30T13:58:32.406 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.022944836 2024-03-30T13:58:32.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.406 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.406 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:32.406 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.906690+0000", 2024-03-30T13:58:32.406 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5089000000000001e-05 2024-03-30T13:58:32.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.406 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.406 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.406 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.052690+0000", 2024-03-30T13:58:32.406 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.146000246 2024-03-30T13:58:32.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.407 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:32.407 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.052737+0000", 2024-03-30T13:58:32.407 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7108999999999999e-05 2024-03-30T13:58:32.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.407 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:32.407 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.058143+0000", 2024-03-30T13:58:32.407 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0054057250000000001 2024-03-30T13:58:32.407 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.407 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.407 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.408 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:62941 9.8 9:1ebc15b0:::benchmark_data_smithi043_145506_object62940:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-30T13:58:32.408 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:40.841362+0000", 2024-03-30T13:58:32.408 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 289.82566110800002, 2024-03-30T13:58:32.408 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.22600500200000001, 2024-03-30T13:58:32.408 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:32.408 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:32.408 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:32.408 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:32.408 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:32.408 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:32.408 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 62941 2024-03-30T13:58:32.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.408 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:32.408 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.409 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:32.409 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.841362+0000", 2024-03-30T13:58:32.409 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.409 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:32.409 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.841362+0000", 2024-03-30T13:58:32.409 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.409 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:32.409 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.841364+0000", 2024-03-30T13:58:32.409 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.638e-06 2024-03-30T13:58:32.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.410 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:32.410 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.841391+0000", 2024-03-30T13:58:32.410 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7016999999999999e-05 2024-03-30T13:58:32.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.410 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:32.410 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.841405+0000", 2024-03-30T13:58:32.410 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4063000000000001e-05 2024-03-30T13:58:32.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.410 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:32.410 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.841417+0000", 2024-03-30T13:58:32.410 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2300000000000001e-05 2024-03-30T13:58:32.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.411 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.411 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.843309+0000", 2024-03-30T13:58:32.411 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001892 2024-03-30T13:58:32.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.411 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:32.411 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.843473+0000", 2024-03-30T13:58:32.411 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016429500000000001 2024-03-30T13:58:32.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.411 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-03-30T13:58:32.411 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.843660+0000", 2024-03-30T13:58:32.411 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000186904 2024-03-30T13:58:32.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.412 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.412 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.854327+0000", 2024-03-30T13:58:32.412 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010666271999999999 2024-03-30T13:58:32.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.412 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:32.412 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.854352+0000", 2024-03-30T13:58:32.412 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5944000000000001e-05 2024-03-30T13:58:32.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.412 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.412 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:41.064213+0000", 2024-03-30T13:58:32.413 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.20986065600000001 2024-03-30T13:58:32.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.413 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:32.413 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:41.064276+0000", 2024-03-30T13:58:32.413 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2563000000000004e-05 2024-03-30T13:58:32.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.413 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:32.413 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:41.067367+0000", 2024-03-30T13:58:32.413 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0030913500000000001 2024-03-30T13:58:32.413 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.413 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.413 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.414 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:62964 9.8 9:11e0008e:::benchmark_data_smithi043_145506_object62963:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-30T13:58:32.414 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:40.876642+0000", 2024-03-30T13:58:32.414 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 289.79038136600002, 2024-03-30T13:58:32.414 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.19417015500000001, 2024-03-30T13:58:32.414 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:32.414 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:32.414 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:32.414 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:32.414 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:32.414 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:32.414 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 62964 2024-03-30T13:58:32.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.415 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:32.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.415 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:32.415 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.876642+0000", 2024-03-30T13:58:32.415 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.415 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:32.415 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.876642+0000", 2024-03-30T13:58:32.415 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.415 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:32.415 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.876643+0000", 2024-03-30T13:58:32.416 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5009999999999999e-06 2024-03-30T13:58:32.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.416 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:32.416 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.876671+0000", 2024-03-30T13:58:32.416 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.8170999999999998e-05 2024-03-30T13:58:32.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.416 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:32.416 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.876688+0000", 2024-03-30T13:58:32.416 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6368999999999999e-05 2024-03-30T13:58:32.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.416 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:32.417 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.876696+0000", 2024-03-30T13:58:32.417 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6850000000000001e-06 2024-03-30T13:58:32.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.417 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.417 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.880100+0000", 2024-03-30T13:58:32.417 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0034048749999999999 2024-03-30T13:58:32.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.417 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:32.417 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.880294+0000", 2024-03-30T13:58:32.417 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00019400799999999999 2024-03-30T13:58:32.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.417 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-03-30T13:58:32.418 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.880500+0000", 2024-03-30T13:58:32.418 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00020511100000000001 2024-03-30T13:58:32.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.418 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.418 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.891487+0000", 2024-03-30T13:58:32.418 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010987133 2024-03-30T13:58:32.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.418 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:32.418 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.891504+0000", 2024-03-30T13:58:32.418 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7768999999999999e-05 2024-03-30T13:58:32.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.419 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.419 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:41.067427+0000", 2024-03-30T13:58:32.419 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17592274499999999 2024-03-30T13:58:32.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.419 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:32.419 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:41.067481+0000", 2024-03-30T13:58:32.419 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3542000000000003e-05 2024-03-30T13:58:32.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.419 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:32.419 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:41.070812+0000", 2024-03-30T13:58:32.419 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0033312459999999999 2024-03-30T13:58:32.420 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.420 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.420 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.420 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:62965 9.8 9:1d8a07dd:::benchmark_data_smithi043_145506_object62964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-30T13:58:32.420 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:40.879066+0000", 2024-03-30T13:58:32.420 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 289.78795672899997, 2024-03-30T13:58:32.420 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.19756965400000001, 2024-03-30T13:58:32.420 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:32.420 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:32.420 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:32.420 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:32.420 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:32.420 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:32.421 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 62965 2024-03-30T13:58:32.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.421 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:32.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.421 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:32.421 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.879066+0000", 2024-03-30T13:58:32.421 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.421 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:32.421 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.879066+0000", 2024-03-30T13:58:32.421 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.422 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:32.422 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.879068+0000", 2024-03-30T13:58:32.422 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4780000000000001e-06 2024-03-30T13:58:32.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.422 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:32.422 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.879094+0000", 2024-03-30T13:58:32.422 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.6086999999999999e-05 2024-03-30T13:58:32.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.422 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:32.422 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.879107+0000", 2024-03-30T13:58:32.422 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2704999999999999e-05 2024-03-30T13:58:32.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.423 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:32.423 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.879115+0000", 2024-03-30T13:58:32.423 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7590000000000003e-06 2024-03-30T13:58:32.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.423 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.423 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.882577+0000", 2024-03-30T13:58:32.423 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003461801 2024-03-30T13:58:32.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.423 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:32.423 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.882775+0000", 2024-03-30T13:58:32.423 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00019734 2024-03-30T13:58:32.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.424 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-03-30T13:58:32.424 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.882988+0000", 2024-03-30T13:58:32.424 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000213465 2024-03-30T13:58:32.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.424 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:32.424 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.891515+0000", 2024-03-30T13:58:32.424 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0085271949999999996 2024-03-30T13:58:32.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.424 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.424 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.891555+0000", 2024-03-30T13:58:32.425 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9462000000000003e-05 2024-03-30T13:58:32.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.425 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.425 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:41.070895+0000", 2024-03-30T13:58:32.425 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17934053 2024-03-30T13:58:32.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.425 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:32.425 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:41.070960+0000", 2024-03-30T13:58:32.425 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4585999999999995e-05 2024-03-30T13:58:32.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.425 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:32.426 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:41.076636+0000", 2024-03-30T13:58:32.426 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0056762460000000002 2024-03-30T13:58:32.426 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.426 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.426 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.426 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.426 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:91801 9.7 9:e39f0226:::benchmark_data_smithi043_145506_object4985:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-03-30T13:58:32.426 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:54:31.761508+0000", 2024-03-30T13:58:32.426 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 238.90551564899999, 2024-03-30T13:58:32.426 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.21157531500000001, 2024-03-30T13:58:32.426 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:32.426 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:32.426 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:32.427 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:32.427 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:32.427 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:32.427 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 91801 2024-03-30T13:58:32.427 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.427 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:32.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.427 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:32.427 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.761508+0000", 2024-03-30T13:58:32.427 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.427 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.427 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:32.427 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.761508+0000", 2024-03-30T13:58:32.427 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.428 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.428 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.428 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:32.428 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.761509+0000", 2024-03-30T13:58:32.428 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.15e-06 2024-03-30T13:58:32.428 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.428 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.428 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:32.428 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.761515+0000", 2024-03-30T13:58:32.428 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.879e-06 2024-03-30T13:58:32.428 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.428 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.428 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:32.429 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.761525+0000", 2024-03-30T13:58:32.429 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0543999999999999e-05 2024-03-30T13:58:32.429 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.429 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.429 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:32.429 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.761532+0000", 2024-03-30T13:58:32.429 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9550000000000003e-06 2024-03-30T13:58:32.429 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.429 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.429 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.429 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.763274+0000", 2024-03-30T13:58:32.429 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001741682 2024-03-30T13:58:32.429 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.429 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.430 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:32.430 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.763463+0000", 2024-03-30T13:58:32.430 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018943399999999999 2024-03-30T13:58:32.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.430 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.430 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,7", 2024-03-30T13:58:32.430 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.763627+0000", 2024-03-30T13:58:32.430 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016405900000000001 2024-03-30T13:58:32.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.430 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.430 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.430 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.766073+0000", 2024-03-30T13:58:32.430 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024459379999999999 2024-03-30T13:58:32.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.431 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.431 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:32.431 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.768554+0000", 2024-03-30T13:58:32.431 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024812039999999999 2024-03-30T13:58:32.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.431 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.431 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.431 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.969649+0000", 2024-03-30T13:58:32.431 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.20109442899999999 2024-03-30T13:58:32.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.431 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.431 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:32.431 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.969702+0000", 2024-03-30T13:58:32.431 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3668999999999997e-05 2024-03-30T13:58:32.432 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.432 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.432 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:32.432 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.973083+0000", 2024-03-30T13:58:32.432 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003380372 2024-03-30T13:58:32.432 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.432 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.432 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.432 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.432 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.432 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:91805 9.2 9:43712f2a:::benchmark_data_smithi043_145506_object4989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-03-30T13:58:32.432 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:54:31.768607+0000", 2024-03-30T13:58:32.432 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 238.898416072, 2024-03-30T13:58:32.432 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.20916811799999999, 2024-03-30T13:58:32.433 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:32.433 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:32.433 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:32.433 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:32.433 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:32.433 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:32.433 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 91805 2024-03-30T13:58:32.433 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.433 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:32.433 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.433 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:32.434 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.768607+0000", 2024-03-30T13:58:32.434 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.434 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:32.434 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.768607+0000", 2024-03-30T13:58:32.434 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.434 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:32.434 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.768609+0000", 2024-03-30T13:58:32.434 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.505e-06 2024-03-30T13:58:32.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.435 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:32.435 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.768614+0000", 2024-03-30T13:58:32.435 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7019999999999998e-06 2024-03-30T13:58:32.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.435 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:32.435 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.768625+0000", 2024-03-30T13:58:32.435 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0559e-05 2024-03-30T13:58:32.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.435 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:32.435 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.768634+0000", 2024-03-30T13:58:32.435 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8789999999999992e-06 2024-03-30T13:58:32.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.436 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.436 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.768721+0000", 2024-03-30T13:58:32.436 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7144000000000005e-05 2024-03-30T13:58:32.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.436 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:32.436 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.768859+0000", 2024-03-30T13:58:32.436 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013771000000000001 2024-03-30T13:58:32.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.436 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1", 2024-03-30T13:58:32.436 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.768995+0000", 2024-03-30T13:58:32.436 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000136613 2024-03-30T13:58:32.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.437 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:32.437 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.771435+0000", 2024-03-30T13:58:32.437 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00243947 2024-03-30T13:58:32.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.437 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.437 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.977626+0000", 2024-03-30T13:58:32.437 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.20619151399999999 2024-03-30T13:58:32.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.437 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:32.437 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.977686+0000", 2024-03-30T13:58:32.438 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9629000000000001e-05 2024-03-30T13:58:32.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.438 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:32.438 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.977775+0000", 2024-03-30T13:58:32.438 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.9393000000000003e-05 2024-03-30T13:58:32.438 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.438 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.438 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.438 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:94741 9.c 9:3eb2be37:::benchmark_data_smithi043_145506_object7925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-03-30T13:58:32.438 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:54:36.936006+0000", 2024-03-30T13:58:32.438 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 233.73101669100001, 2024-03-30T13:58:32.439 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.19792824000000001, 2024-03-30T13:58:32.439 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:32.439 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:32.439 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:32.439 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:32.439 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:32.439 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:32.439 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 94741 2024-03-30T13:58:32.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.439 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:32.439 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.439 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:32.439 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.936006+0000", 2024-03-30T13:58:32.439 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.440 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:32.440 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.936006+0000", 2024-03-30T13:58:32.440 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.440 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:32.440 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.936007+0000", 2024-03-30T13:58:32.440 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9700000000000002e-07 2024-03-30T13:58:32.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.440 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:32.440 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.936009+0000", 2024-03-30T13:58:32.440 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.446e-06 2024-03-30T13:58:32.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.441 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:32.441 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.936015+0000", 2024-03-30T13:58:32.441 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.57e-06 2024-03-30T13:58:32.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.441 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:32.441 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.936020+0000", 2024-03-30T13:58:32.441 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.95e-06 2024-03-30T13:58:32.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.441 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.441 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.938340+0000", 2024-03-30T13:58:32.442 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002320506 2024-03-30T13:58:32.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.442 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.442 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:32.442 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.938494+0000", 2024-03-30T13:58:32.442 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015381499999999999 2024-03-30T13:58:32.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.442 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.442 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,4", 2024-03-30T13:58:32.442 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.938645+0000", 2024-03-30T13:58:32.442 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015067600000000001 2024-03-30T13:58:32.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.442 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.443 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.443 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.944611+0000", 2024-03-30T13:58:32.443 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0059656789999999998 2024-03-30T13:58:32.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.443 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:32.443 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.949697+0000", 2024-03-30T13:58:32.443 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0050869740000000002 2024-03-30T13:58:32.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.443 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.443 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.984020+0000", 2024-03-30T13:58:32.443 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.034322566999999998 2024-03-30T13:58:32.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.444 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:32.444 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.984079+0000", 2024-03-30T13:58:32.444 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9194000000000003e-05 2024-03-30T13:58:32.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.444 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:32.444 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.133935+0000", 2024-03-30T13:58:32.444 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14985546599999999 2024-03-30T13:58:32.444 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.444 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.444 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.445 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:94749 9.c 9:31d281fe:::benchmark_data_smithi043_145506_object7933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-03-30T13:58:32.445 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:54:36.944743+0000", 2024-03-30T13:58:32.445 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 233.722280396, 2024-03-30T13:58:32.445 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.19413675499999999, 2024-03-30T13:58:32.445 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:32.445 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:32.445 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:32.445 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:32.445 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:32.445 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:32.445 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 94749 2024-03-30T13:58:32.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.445 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:32.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.446 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:32.446 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.944743+0000", 2024-03-30T13:58:32.446 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.446 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.446 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:32.446 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.944743+0000", 2024-03-30T13:58:32.446 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.446 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.446 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:32.446 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.944744+0000", 2024-03-30T13:58:32.446 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.37e-06 2024-03-30T13:58:32.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.447 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:32.447 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.944749+0000", 2024-03-30T13:58:32.447 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4000000000000002e-06 2024-03-30T13:58:32.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.447 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:32.447 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.944757+0000", 2024-03-30T13:58:32.447 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7490000000000004e-06 2024-03-30T13:58:32.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.447 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:32.447 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.944764+0000", 2024-03-30T13:58:32.448 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4559999999999997e-06 2024-03-30T13:58:32.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.448 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.448 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.946847+0000", 2024-03-30T13:58:32.448 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0020835039999999999 2024-03-30T13:58:32.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.448 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:32.448 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.947003+0000", 2024-03-30T13:58:32.448 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015556599999999999 2024-03-30T13:58:32.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.448 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,4", 2024-03-30T13:58:32.448 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.947140+0000", 2024-03-30T13:58:32.449 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013759300000000001 2024-03-30T13:58:32.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.449 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.449 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.949741+0000", 2024-03-30T13:58:32.449 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026005609999999999 2024-03-30T13:58:32.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.449 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:32.449 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.956211+0000", 2024-03-30T13:58:32.449 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0064701730000000001 2024-03-30T13:58:32.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.449 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.450 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.135803+0000", 2024-03-30T13:58:32.450 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17959153999999999 2024-03-30T13:58:32.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.450 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.450 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:32.450 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.135894+0000", 2024-03-30T13:58:32.450 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1390000000000004e-05 2024-03-30T13:58:32.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.450 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.450 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:32.450 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.138880+0000", 2024-03-30T13:58:32.450 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0029854529999999999 2024-03-30T13:58:32.450 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.450 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.451 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.451 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:94770 9.8 9:19e3ddc0:::benchmark_data_smithi043_145506_object7954:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-03-30T13:58:32.451 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:54:36.974156+0000", 2024-03-30T13:58:32.451 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 233.692867237, 2024-03-30T13:58:32.451 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.19024108100000001, 2024-03-30T13:58:32.451 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:32.451 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:32.451 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:32.451 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:32.451 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:32.451 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:32.451 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 94770 2024-03-30T13:58:32.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.452 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:32.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.452 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:32.452 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.974156+0000", 2024-03-30T13:58:32.452 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.452 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:32.452 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.974156+0000", 2024-03-30T13:58:32.452 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.452 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:32.452 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.974157+0000", 2024-03-30T13:58:32.453 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3600000000000002e-07 2024-03-30T13:58:32.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.453 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:32.453 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.974162+0000", 2024-03-30T13:58:32.453 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7040000000000002e-06 2024-03-30T13:58:32.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.453 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:32.453 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.974172+0000", 2024-03-30T13:58:32.453 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0927000000000001e-05 2024-03-30T13:58:32.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.453 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:32.454 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.974180+0000", 2024-03-30T13:58:32.454 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5490000000000001e-06 2024-03-30T13:58:32.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.454 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.454 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.454 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.982428+0000", 2024-03-30T13:58:32.454 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0082478259999999998 2024-03-30T13:58:32.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.454 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.454 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:32.454 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.982567+0000", 2024-03-30T13:58:32.454 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000138794 2024-03-30T13:58:32.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.454 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.455 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-03-30T13:58:32.455 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.982710+0000", 2024-03-30T13:58:32.455 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014296 2024-03-30T13:58:32.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.455 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.455 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.141576+0000", 2024-03-30T13:58:32.455 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.15886668200000001 2024-03-30T13:58:32.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.455 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:32.455 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.141601+0000", 2024-03-30T13:58:32.455 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4324e-05 2024-03-30T13:58:32.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.456 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.456 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.162253+0000", 2024-03-30T13:58:32.456 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.02065283 2024-03-30T13:58:32.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.456 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.456 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:32.456 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.162314+0000", 2024-03-30T13:58:32.456 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0062000000000002e-05 2024-03-30T13:58:32.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.456 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.456 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:32.456 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.164397+0000", 2024-03-30T13:58:32.456 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0020834870000000002 2024-03-30T13:58:32.456 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.457 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.457 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.457 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.457 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:94771 9.8 9:125ee1c4:::benchmark_data_smithi043_145506_object7955:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-03-30T13:58:32.457 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:54:36.975996+0000", 2024-03-30T13:58:32.457 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 233.691026857, 2024-03-30T13:58:32.457 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.19043966800000001, 2024-03-30T13:58:32.457 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:32.457 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:32.457 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:32.457 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:32.457 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:32.458 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:32.458 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 94771 2024-03-30T13:58:32.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.458 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:32.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.458 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:32.458 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.975996+0000", 2024-03-30T13:58:32.458 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.458 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:32.458 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.975996+0000", 2024-03-30T13:58:32.458 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.459 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:32.459 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.975998+0000", 2024-03-30T13:58:32.459 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.888e-06 2024-03-30T13:58:32.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.459 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:32.459 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.976003+0000", 2024-03-30T13:58:32.459 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2009999999999998e-06 2024-03-30T13:58:32.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.459 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:32.459 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.976015+0000", 2024-03-30T13:58:32.459 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1664e-05 2024-03-30T13:58:32.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.460 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:32.460 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.976023+0000", 2024-03-30T13:58:32.460 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2400000000000007e-06 2024-03-30T13:58:32.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.460 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.460 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.139089+0000", 2024-03-30T13:58:32.460 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.16306605499999999 2024-03-30T13:58:32.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.460 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:32.460 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.139220+0000", 2024-03-30T13:58:32.460 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000130813 2024-03-30T13:58:32.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.461 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-03-30T13:58:32.461 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.139364+0000", 2024-03-30T13:58:32.461 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000143757 2024-03-30T13:58:32.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.461 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.461 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.156274+0000", 2024-03-30T13:58:32.461 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.016910090999999999 2024-03-30T13:58:32.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.461 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:32.461 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.164418+0000", 2024-03-30T13:58:32.462 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0081439189999999995 2024-03-30T13:58:32.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.462 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.462 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.164471+0000", 2024-03-30T13:58:32.462 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2994000000000001e-05 2024-03-30T13:58:32.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.462 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:32.462 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.164523+0000", 2024-03-30T13:58:32.462 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1999000000000001e-05 2024-03-30T13:58:32.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.462 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:32.463 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.166436+0000", 2024-03-30T13:58:32.463 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001913047 2024-03-30T13:58:32.463 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.463 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.463 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.463 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:94775 9.8 9:1007198a:::benchmark_data_smithi043_145506_object7959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-03-30T13:58:32.463 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:54:36.982284+0000", 2024-03-30T13:58:32.463 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 233.684739638, 2024-03-30T13:58:32.463 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.19173500299999999, 2024-03-30T13:58:32.463 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:58:32.463 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:58:32.463 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:32.464 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:58:32.464 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T13:58:32.464 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T13:58:32.464 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 94775 2024-03-30T13:58:32.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.464 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:58:32.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.464 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:58:32.464 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.982284+0000", 2024-03-30T13:58:32.464 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.464 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:58:32.464 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.982284+0000", 2024-03-30T13:58:32.464 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:58:32.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.465 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:58:32.465 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.982284+0000", 2024-03-30T13:58:32.465 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7700000000000005e-07 2024-03-30T13:58:32.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.465 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:58:32.465 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.982287+0000", 2024-03-30T13:58:32.465 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9040000000000001e-06 2024-03-30T13:58:32.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.465 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:58:32.465 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.982294+0000", 2024-03-30T13:58:32.466 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6449999999999999e-06 2024-03-30T13:58:32.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.466 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:58:32.466 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.982299+0000", 2024-03-30T13:58:32.466 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.074e-06 2024-03-30T13:58:32.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.466 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:58:32.466 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.143661+0000", 2024-03-30T13:58:32.466 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.16136198099999999 2024-03-30T13:58:32.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.466 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:58:32.467 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.143814+0000", 2024-03-30T13:58:32.467 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000153733 2024-03-30T13:58:32.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.467 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-03-30T13:58:32.467 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.143982+0000", 2024-03-30T13:58:32.467 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016750399999999999 2024-03-30T13:58:32.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.467 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.467 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.156329+0000", 2024-03-30T13:58:32.467 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.012347387 2024-03-30T13:58:32.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.467 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T13:58:32.468 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.164425+0000", 2024-03-30T13:58:32.468 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0080956159999999999 2024-03-30T13:58:32.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.468 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:32.468 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.166495+0000", 2024-03-30T13:58:32.468 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0020697110000000001 2024-03-30T13:58:32.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.468 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T13:58:32.468 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.166545+0000", 2024-03-30T13:58:32.468 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0739000000000003e-05 2024-03-30T13:58:32.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:32.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:32.469 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T13:58:32.469 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.174019+0000", 2024-03-30T13:58:32.469 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0074732319999999998 2024-03-30T13:58:32.469 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.469 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.469 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.469 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:32.469 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:32.469 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:58:32.469 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:32.470 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 84 15 153582 153567 114.237 90.1875 0.00878204 0.00874986 2024-03-30T13:58:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:32.494 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:32.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:32.565 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6] out_osds: [5, 7] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-30T13:58:32.565 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:58:32.565 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-03-30T13:58:32.565 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd in 5 2024-03-30T13:58:32.595 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:32.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:32.659 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:32.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:32.760 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:32.784 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:32.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:32.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:32.887 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:32.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:32.932 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:32.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:33.033 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:33.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:33.066 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:33.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:33.168 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:33.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:33.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:33.307 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:33.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:33.331 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:33.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:33.381 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 85 16 154709 154693 113.72 70.375 0.00284975 0.00878186 2024-03-30T13:58:33.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:33 smithi043 ceph-mon[29963]: pgmap v694: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.1 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.99k op/s; 74544/446550 objects degraded (16.693%); 27/446550 objects misplaced (0.006%) 2024-03-30T13:58:33.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:33 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 74544/446550 objects degraded (16.693%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:33.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:33 smithi043 ceph-mon[42099]: pgmap v694: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.1 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.99k op/s; 74544/446550 objects degraded (16.693%); 27/446550 objects misplaced (0.006%) 2024-03-30T13:58:33.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:33 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 74544/446550 objects degraded (16.693%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:33.432 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:33.463 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:33.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:33.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:33.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:33 smithi113 ceph-mon[34341]: pgmap v694: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.1 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.99k op/s; 74544/446550 objects degraded (16.693%); 27/446550 objects misplaced (0.006%) 2024-03-30T13:58:33.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:33 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 74544/446550 objects degraded (16.693%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:33.566 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:33.594 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:33.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:33.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:33.696 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:33.732 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:33.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:33.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:33.834 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:33.875 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:33.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:33.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:33.977 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:34.015 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:34.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:34.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:34.117 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:34.152 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:34.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:34.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:34.254 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:34.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:34.290 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:34.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:34.382 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 86 16 156132 156116 113.432 88.9375 0.00485474 0.00880656 2024-03-30T13:58:34.391 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:34.422 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:34.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:34.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:34.525 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:34.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:34.567 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:34.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:34.671 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:34.701 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:34.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:34.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:34.803 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:34.835 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:34.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:34.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:34.936 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:34.974 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:34.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:34.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:35.080 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:35.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:35.123 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:35.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:35.224 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:35.261 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:35.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:35.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:35.362 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:35 smithi043 ceph-mon[29963]: pgmap v695: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.1 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 75015/447963 objects degraded (16.746%); 27/447963 objects misplaced (0.006%) 2024-03-30T13:58:35.362 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:35.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:35 smithi043 ceph-mon[42099]: pgmap v695: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.1 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 75015/447963 objects degraded (16.746%); 27/447963 objects misplaced (0.006%) 2024-03-30T13:58:35.386 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 87 16 157690 157674 113.245 97.375 0.00675329 0.00882622 2024-03-30T13:58:35.388 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:35.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:35.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:35.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:35 smithi113 ceph-mon[34341]: pgmap v695: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.1 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 75015/447963 objects degraded (16.746%); 27/447963 objects misplaced (0.006%) 2024-03-30T13:58:35.490 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:35.531 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:35.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:35.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:35.633 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:35.667 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:35.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:35.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:35.770 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:35.793 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:35.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:35.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:35.895 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:35.928 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:35.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:35.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:36.030 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:36.065 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:36.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:36.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:36.144 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:58:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:58:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:58:36.167 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:36.191 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:36.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:36.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:36.293 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:36.326 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:36.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:36.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:36.383 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 88 16 159113 159097 112.969 88.9375 0.00321266 0.00884801 2024-03-30T13:58:36.428 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:36.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:36.462 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:36.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:36.563 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:36.594 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:36.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:36.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:36.696 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:36.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:36.726 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:36.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:36.827 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:36.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:36.867 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:36.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:36.968 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:36.998 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:36.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:36.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:37.100 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:37.142 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:37.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:37.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:37.244 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:37.275 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:37.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:37.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:37.377 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:37.384 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 89 16 160637 160621 112.77 95.25 0.00859011 0.00886354 2024-03-30T13:58:37.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:37 smithi043 ceph-mon[29963]: pgmap v696: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.5 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.60k op/s; 78258/467421 objects degraded (16.743%); 27/467421 objects misplaced (0.006%) 2024-03-30T13:58:37.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:37 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:58:37.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:37 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:58:37.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:37 smithi043 ceph-mon[42099]: pgmap v696: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.5 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.60k op/s; 78258/467421 objects degraded (16.743%); 27/467421 objects misplaced (0.006%) 2024-03-30T13:58:37.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:37 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:58:37.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:37 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:58:37.412 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:37.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:37.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:37 smithi113 ceph-mon[34341]: pgmap v696: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.5 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.60k op/s; 78258/467421 objects degraded (16.743%); 27/467421 objects misplaced (0.006%) 2024-03-30T13:58:37.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:37 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:58:37.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:37 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:58:37.514 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:37.547 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:37.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:37.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:37.649 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:37.689 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:37.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:37.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:37.791 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:37.825 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:37.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:37.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:37.927 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:37.958 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:37.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:37.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:37.967 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-30T13:58:38.061 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:38.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:38.103 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:38.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:38.171 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:38 smithi043 ceph-mon[29963]: osdmap e321: 8 total, 7 up, 6 in 2024-03-30T13:58:38.171 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:38 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 78258/467421 objects degraded (16.743%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:38.171 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:38 smithi043 ceph-mon[29963]: osdmap e322: 8 total, 7 up, 6 in 2024-03-30T13:58:38.171 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:38 smithi043 ceph-mon[29963]: pgmap v699: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.5 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.26k op/s; 78258/468747 objects degraded (16.695%); 27/468747 objects misplaced (0.006%) 2024-03-30T13:58:38.172 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:38 smithi043 ceph-mon[42099]: osdmap e321: 8 total, 7 up, 6 in 2024-03-30T13:58:38.172 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:38 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 78258/467421 objects degraded (16.743%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:38.172 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:38 smithi043 ceph-mon[42099]: osdmap e322: 8 total, 7 up, 6 in 2024-03-30T13:58:38.172 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:38 smithi043 ceph-mon[42099]: pgmap v699: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.5 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.26k op/s; 78258/468747 objects degraded (16.695%); 27/468747 objects misplaced (0.006%) 2024-03-30T13:58:38.204 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:38.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:38.244 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:38.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:38.345 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:38.367 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:58:38.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:38.381 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:38.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:38.383 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Total time run: 90.0061 2024-03-30T13:58:38.384 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Total writes made: 162224 2024-03-30T13:58:38.384 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Write size: 65536 2024-03-30T13:58:38.384 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Object size: 65536 2024-03-30T13:58:38.384 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Bandwidth (MB/sec): 112.648 2024-03-30T13:58:38.384 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev Bandwidth: 32.7911 2024-03-30T13:58:38.384 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max bandwidth (MB/sec): 256.938 2024-03-30T13:58:38.384 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min bandwidth (MB/sec): 70.375 2024-03-30T13:58:38.384 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Average IOPS: 1802 2024-03-30T13:58:38.384 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev IOPS: 524.658 2024-03-30T13:58:38.384 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max IOPS: 4111 2024-03-30T13:58:38.384 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min IOPS: 1126 2024-03-30T13:58:38.384 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Average Latency(s): 0.00887343 2024-03-30T13:58:38.384 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev Latency(s): 0.00664904 2024-03-30T13:58:38.384 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max latency(s): 0.148211 2024-03-30T13:58:38.385 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min latency(s): 0.00120663 2024-03-30T13:58:38.389 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Cleaning up (deleting benchmark objects) 2024-03-30T13:58:38.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:38 smithi113 ceph-mon[34341]: osdmap e321: 8 total, 7 up, 6 in 2024-03-30T13:58:38.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:38 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 78258/467421 objects degraded (16.743%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:38.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:38 smithi113 ceph-mon[34341]: osdmap e322: 8 total, 7 up, 6 in 2024-03-30T13:58:38.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:38 smithi113 ceph-mon[34341]: pgmap v699: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.5 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.26k op/s; 78258/468747 objects degraded (16.695%); 27/468747 objects misplaced (0.006%) 2024-03-30T13:58:38.482 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:38.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:38.524 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:38.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:38.618 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:58:38.619 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:58:38.625 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:38.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:38.662 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:38.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:38.763 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:38.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:38.797 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:38.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:38.899 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:38.929 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:38.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:38.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:39.031 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:39.061 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:39.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:39.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:39.163 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:39.207 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:39.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:39.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:39.297 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:39 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:58:39.297 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:39 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:58:39.297 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:39 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:58:39.297 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:39 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:58:39.309 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:39.338 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:39.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:39.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:39.441 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:39.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:58:39 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:58:39.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:39 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:58:39.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:39 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:58:39.474 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:39.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:39.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:39.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:39.615 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:39.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:39.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:39.717 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:39.756 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:39.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:39.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:39.858 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:39.896 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:39.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:39.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:39.998 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:40.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:40.036 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:40.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:40.118 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-03-30T13:58:40.136 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:40.173 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:40.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:40.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:40.275 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:40.310 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:40.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:40.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:40.388 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:40 smithi113 ceph-mon[34341]: osdmap e323: 8 total, 7 up, 6 in 2024-03-30T13:58:40.388 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:40 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/4218652096' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:58:40.388 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:40 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:58:40.388 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:40 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:40.389 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:40 smithi113 ceph-mon[34341]: pgmap v701: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.8 GiB data, 30 GiB used, 507 GiB / 536 GiB avail; 1023 B/s rd, 123 MiB/s wr, 2.02k op/s; 80712/482352 objects degraded (16.733%); 27/482352 objects misplaced (0.006%) 2024-03-30T13:58:40.389 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:58:40.389 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [ 2024-03-30T13:58:40.389 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.389 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:165748 10.c 10:335cb088:::benchmark_data_smithi043_169264_object3521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-03-30T13:58:40.390 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:58:40.389257+0000", 2024-03-30T13:58:40.390 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.0032046560000000002, 2024-03-30T13:58:40.390 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.003236788, 2024-03-30T13:58:40.390 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:58:40.390 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:58:40.390 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "waiting for sub ops", 2024-03-30T13:58:40.390 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:58:40.390 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T13:58:40.390 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T13:58:40.390 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 165748 2024-03-30T13:58:40.390 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.390 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:58:40.390 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.391 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:58:40.391 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.389257+0000", 2024-03-30T13:58:40.391 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:58:40.391 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.391 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.391 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:58:40.391 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.389257+0000", 2024-03-30T13:58:40.391 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:58:40.391 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.391 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.391 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:58:40.391 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.389258+0000", 2024-03-30T13:58:40.391 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.9899999999999999e-07 2024-03-30T13:58:40.392 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.392 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.392 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:58:40.392 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.389263+0000", 2024-03-30T13:58:40.392 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9919999999999998e-06 2024-03-30T13:58:40.392 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.392 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.392 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:58:40.392 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.389273+0000", 2024-03-30T13:58:40.392 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.9830000000000008e-06 2024-03-30T13:58:40.392 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.392 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.392 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:58:40.392 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.389280+0000", 2024-03-30T13:58:40.393 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.4730000000000003e-06 2024-03-30T13:58:40.393 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.393 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.393 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:58:40.393 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.389352+0000", 2024-03-30T13:58:40.393 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1668999999999994e-05 2024-03-30T13:58:40.393 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.393 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.393 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:58:40.393 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.389468+0000", 2024-03-30T13:58:40.393 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000116039 2024-03-30T13:58:40.393 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.393 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.393 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,3", 2024-03-30T13:58:40.394 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.389582+0000", 2024-03-30T13:58:40.394 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000114079 2024-03-30T13:58:40.394 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.394 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.394 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:58:40.394 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.391815+0000", 2024-03-30T13:58:40.394 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0022328489999999999 2024-03-30T13:58:40.394 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.394 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.394 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:40.394 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392149+0000", 2024-03-30T13:58:40.394 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00033477799999999999 2024-03-30T13:58:40.395 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.395 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.395 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:40.395 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392459+0000", 2024-03-30T13:58:40.395 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00031003600000000002 2024-03-30T13:58:40.395 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:58:40.395 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:58:40.395 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:58:40.395 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.395 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.395 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:165752 10.9 10:96ee4f2a:::benchmark_data_smithi043_169264_object3525:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-03-30T13:58:40.395 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:58:40.389715+0000", 2024-03-30T13:58:40.395 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.0027460940000000001, 2024-03-30T13:58:40.396 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0028884219999999999, 2024-03-30T13:58:40.396 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:58:40.396 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:58:40.396 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "waiting for sub ops", 2024-03-30T13:58:40.396 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:58:40.396 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T13:58:40.396 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T13:58:40.396 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 165752 2024-03-30T13:58:40.396 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.396 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:58:40.396 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.396 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:58:40.396 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.389715+0000", 2024-03-30T13:58:40.396 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:58:40.396 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.397 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.397 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:58:40.397 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.389715+0000", 2024-03-30T13:58:40.397 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:58:40.397 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.397 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.397 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:58:40.397 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.389716+0000", 2024-03-30T13:58:40.397 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.9699999999999994e-07 2024-03-30T13:58:40.397 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.397 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.397 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:58:40.397 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.389722+0000", 2024-03-30T13:58:40.397 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.592e-06 2024-03-30T13:58:40.398 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.398 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.398 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:58:40.398 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.389736+0000", 2024-03-30T13:58:40.398 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4374e-05 2024-03-30T13:58:40.398 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.398 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.398 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:58:40.398 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.389745+0000", 2024-03-30T13:58:40.398 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6729999999999998e-06 2024-03-30T13:58:40.398 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.398 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.398 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:58:40.399 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.389807+0000", 2024-03-30T13:58:40.399 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1618000000000005e-05 2024-03-30T13:58:40.399 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.399 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.399 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:58:40.399 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.389935+0000", 2024-03-30T13:58:40.399 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012817899999999999 2024-03-30T13:58:40.399 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.399 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.399 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3", 2024-03-30T13:58:40.399 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.390190+0000", 2024-03-30T13:58:40.399 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00025503100000000002 2024-03-30T13:58:40.399 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:58:40.399 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:58:40.400 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:58:40.400 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.400 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.400 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop_reply(client.15531.0:165745 10.9 e324/289)", 2024-03-30T13:58:40.400 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:58:40.390022+0000", 2024-03-30T13:58:40.400 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.0024391230000000001, 2024-03-30T13:58:40.400 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0026659990000000001, 2024-03-30T13:58:40.400 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:58:40.400 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:58:40.400 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "reached pg", 2024-03-30T13:58:40.400 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:58:40.400 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.400 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:58:40.400 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.390022+0000", 2024-03-30T13:58:40.401 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:58:40.401 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.401 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.401 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:58:40.401 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.390022+0000", 2024-03-30T13:58:40.401 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:58:40.401 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.401 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.401 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:58:40.401 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.390023+0000", 2024-03-30T13:58:40.401 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1200000000000002e-07 2024-03-30T13:58:40.401 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.401 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.401 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:58:40.401 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.390027+0000", 2024-03-30T13:58:40.402 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1989999999999999e-06 2024-03-30T13:58:40.402 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.402 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.402 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:58:40.402 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.390036+0000", 2024-03-30T13:58:40.402 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.3820000000000004e-06 2024-03-30T13:58:40.402 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.402 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.402 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:58:40.402 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.390043+0000", 2024-03-30T13:58:40.402 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.3069999999999997e-06 2024-03-30T13:58:40.402 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.402 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.402 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:58:40.403 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392678+0000", 2024-03-30T13:58:40.403 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0026348410000000002 2024-03-30T13:58:40.403 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:58:40.403 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:58:40.403 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:58:40.403 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.403 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.403 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop(client.15531.0:165746 10.3 e324/289 10:c959a544:::benchmark_data_smithi043_169264_object3519:head v 324'10503, mlcod=324'10498)", 2024-03-30T13:58:40.403 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:58:40.392239+0000", 2024-03-30T13:58:40.403 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.00022225700000000001, 2024-03-30T13:58:40.403 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00052753999999999995, 2024-03-30T13:58:40.403 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:58:40.403 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:58:40.403 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:58:40.404 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:58:40.404 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.404 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:58:40.404 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392239+0000", 2024-03-30T13:58:40.404 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:58:40.404 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.404 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.404 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:58:40.404 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392239+0000", 2024-03-30T13:58:40.404 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:58:40.404 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.404 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.404 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:58:40.404 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392240+0000", 2024-03-30T13:58:40.404 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4099999999999999e-07 2024-03-30T13:58:40.405 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.405 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.405 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:58:40.405 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392245+0000", 2024-03-30T13:58:40.405 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7450000000000001e-06 2024-03-30T13:58:40.405 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.405 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.405 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:58:40.405 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392253+0000", 2024-03-30T13:58:40.405 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.4150000000000002e-06 2024-03-30T13:58:40.405 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.405 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.405 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:58:40.405 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392259+0000", 2024-03-30T13:58:40.406 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.2389999999999999e-06 2024-03-30T13:58:40.406 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.406 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.406 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:58:40.406 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392316+0000", 2024-03-30T13:58:40.406 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6538e-05 2024-03-30T13:58:40.406 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.406 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.406 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:58:40.406 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392336+0000", 2024-03-30T13:58:40.406 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.0305999999999998e-05 2024-03-30T13:58:40.406 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:58:40.406 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:58:40.406 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:58:40.407 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.407 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.407 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop(client.15531.0:165753 10.d e324/289 10:b21f7cd4:::benchmark_data_smithi043_169264_object3526:head v 324'10331, mlcod=324'10329)", 2024-03-30T13:58:40.407 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:58:40.392714+0000", 2024-03-30T13:58:40.407 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 4294967295.9997473, 2024-03-30T13:58:40.407 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00010481599999999999, 2024-03-30T13:58:40.407 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:58:40.407 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:58:40.407 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:58:40.407 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:58:40.407 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.407 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:58:40.407 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392714+0000", 2024-03-30T13:58:40.407 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:58:40.408 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.408 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.408 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:58:40.408 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392714+0000", 2024-03-30T13:58:40.408 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:58:40.408 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.408 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.408 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:58:40.408 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392715+0000", 2024-03-30T13:58:40.408 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3300000000000002e-07 2024-03-30T13:58:40.408 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.408 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.408 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:58:40.408 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392721+0000", 2024-03-30T13:58:40.408 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8289999999999996e-06 2024-03-30T13:58:40.409 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.409 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.409 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:58:40.409 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392734+0000", 2024-03-30T13:58:40.409 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3368e-05 2024-03-30T13:58:40.409 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.409 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.409 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:58:40.409 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392741+0000", 2024-03-30T13:58:40.409 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1430000000000001e-06 2024-03-30T13:58:40.409 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.409 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.409 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:58:40.409 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392790+0000", 2024-03-30T13:58:40.410 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8739999999999998e-05 2024-03-30T13:58:40.410 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.410 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.410 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:58:40.410 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392809+0000", 2024-03-30T13:58:40.410 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.9125000000000001e-05 2024-03-30T13:58:40.410 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:58:40.410 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:58:40.410 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:58:40.410 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.410 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.410 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:165745 10.9 10:96fdb318:::benchmark_data_smithi043_169264_object3518:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-03-30T13:58:40.410 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:58:40.387074+0000", 2024-03-30T13:58:40.410 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.0053873200000000001, 2024-03-30T13:58:40.411 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0057990799999999999, 2024-03-30T13:58:40.411 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:58:40.411 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:58:40.411 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:58:40.411 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:58:40.411 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T13:58:40.411 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T13:58:40.411 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 165745 2024-03-30T13:58:40.411 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.411 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:58:40.411 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.411 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:58:40.411 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.387074+0000", 2024-03-30T13:58:40.412 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:58:40.412 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.412 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.412 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:58:40.412 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.387074+0000", 2024-03-30T13:58:40.412 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:58:40.412 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.412 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.412 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:58:40.412 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.387075+0000", 2024-03-30T13:58:40.412 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.92e-07 2024-03-30T13:58:40.412 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.412 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.412 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:58:40.413 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.387078+0000", 2024-03-30T13:58:40.413 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.596e-06 2024-03-30T13:58:40.413 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.413 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.413 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:58:40.413 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.387086+0000", 2024-03-30T13:58:40.413 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8420000000000005e-06 2024-03-30T13:58:40.413 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.413 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.413 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:58:40.413 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.387093+0000", 2024-03-30T13:58:40.413 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.9349999999999997e-06 2024-03-30T13:58:40.413 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.413 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.414 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:58:40.414 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.387154+0000", 2024-03-30T13:58:40.414 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1152e-05 2024-03-30T13:58:40.414 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.414 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.414 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:58:40.414 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.387281+0000", 2024-03-30T13:58:40.414 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000127038 2024-03-30T13:58:40.414 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.414 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.414 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3", 2024-03-30T13:58:40.414 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.387398+0000", 2024-03-30T13:58:40.414 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011705999999999999 2024-03-30T13:58:40.414 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.415 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.415 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:58:40.415 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.389439+0000", 2024-03-30T13:58:40.415 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0020405250000000001 2024-03-30T13:58:40.415 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.415 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.415 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:58:40.415 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392770+0000", 2024-03-30T13:58:40.415 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0033309030000000001 2024-03-30T13:58:40.415 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:40.415 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:40.415 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:58:40.415 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:40.392821+0000", 2024-03-30T13:58:40.415 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0868000000000002e-05 2024-03-30T13:58:40.416 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:58:40.416 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:58:40.416 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:58:40.416 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:58:40.416 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-03-30T13:58:40.416 INFO:teuthology.orchestra.run.smithi113.stdout: "num_ops": 6 2024-03-30T13:58:40.416 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:58:40.416 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:40.417 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:40 smithi043 ceph-mon[29963]: osdmap e323: 8 total, 7 up, 6 in 2024-03-30T13:58:40.417 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:40 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4218652096' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:58:40.417 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:40 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:58:40.417 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:40 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:40.417 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:40 smithi043 ceph-mon[29963]: pgmap v701: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.8 GiB data, 30 GiB used, 507 GiB / 536 GiB avail; 1023 B/s rd, 123 MiB/s wr, 2.02k op/s; 80712/482352 objects degraded (16.733%); 27/482352 objects misplaced (0.006%) 2024-03-30T13:58:40.418 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:40 smithi043 ceph-mon[42099]: osdmap e323: 8 total, 7 up, 6 in 2024-03-30T13:58:40.418 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:40 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/4218652096' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:58:40.418 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:40 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:58:40.418 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:40 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:40.418 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:40 smithi043 ceph-mon[42099]: pgmap v701: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.8 GiB data, 30 GiB used, 507 GiB / 536 GiB avail; 1023 B/s rd, 123 MiB/s wr, 2.02k op/s; 80712/482352 objects degraded (16.733%); 27/482352 objects misplaced (0.006%) 2024-03-30T13:58:40.452 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:40.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:40.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:40.553 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:40.592 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:40.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:40.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:40.693 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:40.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:40.720 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:40.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:40.821 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:40.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:40.856 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:40.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:40.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:40.997 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:40.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:40.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:41.100 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:41.134 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:41.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:41.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:41.236 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:41.268 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:41.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:41.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:41.370 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:41.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:41 smithi043 ceph-mon[29963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:58:41.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:41 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:58:41.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:41 smithi043 ceph-mon[29963]: osdmap e324: 8 total, 7 up, 6 in 2024-03-30T13:58:41.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:41 smithi043 ceph-mon[42099]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:58:41.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:41 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:58:41.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:41 smithi043 ceph-mon[42099]: osdmap e324: 8 total, 7 up, 6 in 2024-03-30T13:58:41.405 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:41.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:41.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:41.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:41 smithi113 ceph-mon[34341]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:58:41.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:41 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:58:41.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:41 smithi113 ceph-mon[34341]: osdmap e324: 8 total, 7 up, 6 in 2024-03-30T13:58:41.507 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:41.542 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:41.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:41.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:41.644 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:41.684 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:41.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:41.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:41.786 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:41.819 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:41.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:41.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:41.921 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:41.948 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:41.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:42.049 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:42.086 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:42.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:42.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:42.189 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:42.224 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:42.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:42.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:42.326 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:42.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:42.368 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:42.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:42.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:42 smithi043 ceph-mon[29963]: pgmap v703: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.8 GiB data, 30 GiB used, 507 GiB / 536 GiB avail; 1.3 KiB/s rd, 84 MiB/s wr, 1.69k op/s; 80757/482529 objects degraded (16.736%); 27/482529 objects misplaced (0.006%) 2024-03-30T13:58:42.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:42 smithi043 ceph-mon[42099]: pgmap v703: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.8 GiB data, 30 GiB used, 507 GiB / 536 GiB avail; 1.3 KiB/s rd, 84 MiB/s wr, 1.69k op/s; 80757/482529 objects degraded (16.736%); 27/482529 objects misplaced (0.006%) 2024-03-30T13:58:42.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:42 smithi113 ceph-mon[34341]: pgmap v703: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.8 GiB data, 30 GiB used, 507 GiB / 536 GiB avail; 1.3 KiB/s rd, 84 MiB/s wr, 1.69k op/s; 80757/482529 objects degraded (16.736%); 27/482529 objects misplaced (0.006%) 2024-03-30T13:58:42.468 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:42.499 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:42.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:42.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:42.601 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:42.635 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:42.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:42.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:42.737 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:42.768 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:42.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:42.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:42.870 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:42.903 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:42.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:42.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:43.005 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:43.038 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:43.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:43.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:43.140 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:43.168 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:43.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:43.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:43.270 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:43.303 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:43.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:43.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:43.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:43 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 80757/482529 objects degraded (16.736%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:43 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 80757/482529 objects degraded (16.736%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:43.405 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:43.446 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:43.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:43.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:43.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:43 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 80757/482529 objects degraded (16.736%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:43.548 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:43.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:43.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:43.687 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:43.715 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-03-30T13:58:43.725 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:43.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:43.827 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:43.863 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:43.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:43.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:43.964 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:44.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:44.005 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:44.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:44.096 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:58:44.106 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:44.148 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:44.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:44.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:44.250 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:44.283 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:44.305 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:44 smithi043 ceph-mon[42099]: pgmap v704: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.8 GiB data, 30 GiB used, 507 GiB / 536 GiB avail; 1011 B/s rd, 64 MiB/s wr, 1.36k op/s; 80237/480969 objects degraded (16.682%); 27/480969 objects misplaced (0.006%) 2024-03-30T13:58:44.305 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:44 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:44.305 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:44 smithi043 ceph-mon[29963]: pgmap v704: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.8 GiB data, 30 GiB used, 507 GiB / 536 GiB avail; 1011 B/s rd, 64 MiB/s wr, 1.36k op/s; 80237/480969 objects degraded (16.682%); 27/480969 objects misplaced (0.006%) 2024-03-30T13:58:44.305 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:44 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:44.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:44.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:44.400 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:44 smithi113 ceph-mon[34341]: pgmap v704: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.8 GiB data, 30 GiB used, 507 GiB / 536 GiB avail; 1011 B/s rd, 64 MiB/s wr, 1.36k op/s; 80237/480969 objects degraded (16.682%); 27/480969 objects misplaced (0.006%) 2024-03-30T13:58:44.400 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:44 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:44.409 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:44.445 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:44.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:44.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:44.533 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T13:58:44.547 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:44.621 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:44.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:44.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:44.704 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:58:44.724 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:44.742 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:58:44.766 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:44.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:44.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:44.781 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:58:44.822 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:58:44.864 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:58:44.868 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:44.897 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:44.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:44.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:44.910 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:58:44.950 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:58:44.950 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:58:44.950 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:58:44.950 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:58:44.950 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:58:44.951 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:58:44.951 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:58:44.951 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:58:44.951 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:58:44.999 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:45.040 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:45.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:45.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:45.141 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:45.174 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:45.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:45.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:45.276 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:45.313 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:45.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:45.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:45.372 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:45 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/622095369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:58:45.409 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:45 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/622095369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:58:45.409 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:45 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/622095369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:58:45.415 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:45.448 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:45.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:45.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:45.549 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:45.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:45.580 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:45.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:45.681 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:45.720 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:45.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:45.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:45.822 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:45.857 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:45.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:45.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:45.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:45 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a[29940]: 2024-03-30T13:58:45.596+0000 7ff6065d3700 -1 mon.a@0(leader).osd e324 definitely_dead 0 2024-03-30T13:58:45.959 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:45.996 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:45.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:45.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:46.098 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:46.112 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:58:46.112 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:58:46.132 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:46.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:46.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:46.167 INFO:teuthology.orchestra.run.smithi043.stderr:marked in osd.5. 2024-03-30T13:58:46.227 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:58:46.228 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:58:46.234 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:46.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:46.291 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:46.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:46.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:46.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:46 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2564823135' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-30T13:58:46.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:46 smithi043 ceph-mon[29963]: pgmap v705: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.3 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 767 B/s rd, 48 MiB/s wr, 1.97k op/s; 76527/458574 objects degraded (16.688%); 27/458574 objects misplaced (0.006%) 2024-03-30T13:58:46.394 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:58:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:58:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:58:46.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:46 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2564823135' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-30T13:58:46.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:46 smithi043 ceph-mon[42099]: pgmap v705: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.3 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 767 B/s rd, 48 MiB/s wr, 1.97k op/s; 76527/458574 objects degraded (16.688%); 27/458574 objects misplaced (0.006%) 2024-03-30T13:58:46.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:46.419 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:46.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:46.456 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:46 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2564823135' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-30T13:58:46.456 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:46 smithi113 ceph-mon[34341]: pgmap v705: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.3 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 767 B/s rd, 48 MiB/s wr, 1.97k op/s; 76527/458574 objects degraded (16.688%); 27/458574 objects misplaced (0.006%) 2024-03-30T13:58:46.521 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:46.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:46.545 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:46.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:46.646 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:46.671 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:46.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:46.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:46.772 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:46.774 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-03-30T13:58:46.816 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:58:46.836 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:46.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:46.842 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-03-30T13:58:46.938 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:46.964 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:46.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:46.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:47.050 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:58:47.066 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:47.091 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:47.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:47.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:47.172 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:47 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2564823135' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-30T13:58:47.172 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:47 smithi113 ceph-mon[34341]: osdmap e325: 8 total, 7 up, 7 in 2024-03-30T13:58:47.193 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:47.224 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:47.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:47.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:47.326 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:47.329 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:47 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2564823135' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-30T13:58:47.329 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:47 smithi043 ceph-mon[29963]: osdmap e325: 8 total, 7 up, 7 in 2024-03-30T13:58:47.329 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:47 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2564823135' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-30T13:58:47.329 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:47 smithi043 ceph-mon[42099]: osdmap e325: 8 total, 7 up, 7 in 2024-03-30T13:58:47.355 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:47.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:47.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:47.457 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:47.481 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:47.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:47.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:47.583 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:47.608 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:47.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:47.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:47.710 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:47.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:47.737 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:47.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:47.838 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:47.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:47.863 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:47.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:47.964 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:47.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:47.988 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:47.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:48.090 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:48.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:48.115 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:48.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:48.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:48 smithi113 ceph-mon[34341]: osdmap e326: 8 total, 7 up, 7 in 2024-03-30T13:58:48.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:48 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 76527/458574 objects degraded (16.688%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:48.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:48 smithi113 ceph-mon[34341]: pgmap v708: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.3 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.3 KiB/s rd, 11 MiB/s wr, 1.46k op/s; 76527/456981 objects degraded (16.746%); 27/456981 objects misplaced (0.006%) 2024-03-30T13:58:48.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:48 smithi113 ceph-mon[34341]: osdmap e327: 8 total, 7 up, 7 in 2024-03-30T13:58:48.216 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:48.228 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:58:48.257 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:48.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:48.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:48.361 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:48.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:48 smithi043 ceph-mon[29963]: osdmap e326: 8 total, 7 up, 7 in 2024-03-30T13:58:48.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:48 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 76527/458574 objects degraded (16.688%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:48.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:48 smithi043 ceph-mon[29963]: pgmap v708: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.3 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.3 KiB/s rd, 11 MiB/s wr, 1.46k op/s; 76527/456981 objects degraded (16.746%); 27/456981 objects misplaced (0.006%) 2024-03-30T13:58:48.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:48 smithi043 ceph-mon[29963]: osdmap e327: 8 total, 7 up, 7 in 2024-03-30T13:58:48.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:48 smithi043 ceph-mon[42099]: osdmap e326: 8 total, 7 up, 7 in 2024-03-30T13:58:48.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:48 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 76527/458574 objects degraded (16.688%), 15 pgs degraded, 27 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:48.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:48 smithi043 ceph-mon[42099]: pgmap v708: 158 pgs: 88 active+clean, 12 active+undersized, 15 active+undersized+degraded, 43 active+clean+remapped; 9.3 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.3 KiB/s rd, 11 MiB/s wr, 1.46k op/s; 76527/456981 objects degraded (16.746%); 27/456981 objects misplaced (0.006%) 2024-03-30T13:58:48.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:48 smithi043 ceph-mon[42099]: osdmap e327: 8 total, 7 up, 7 in 2024-03-30T13:58:48.403 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:48.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:48.509 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:48.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:48.545 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:48.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:48.646 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:48.691 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:48.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:48.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:48.793 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:48.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:48.832 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:48.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:58:48.933 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:48.969 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:48.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:48.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:49.062 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:58:49.062 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:58:49.071 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:49.110 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:49.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:49.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:49.212 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:49.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:49.253 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:49.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:49.354 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:49.392 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:49.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:49 smithi043 ceph-mon[29963]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 324'10998 2024-03-30T13:58:49.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:49 smithi043 ceph-mon[29963]: 10.9 starting backfill to osd.5 from (0'0,0'0] MAX to 324'11014 2024-03-30T13:58:49.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:49 smithi043 ceph-mon[29963]: 10.3 starting backfill to osd.5 from (0'0,0'0] MAX to 324'11109 2024-03-30T13:58:49.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:49 smithi043 ceph-mon[29963]: 10.7 starting backfill to osd.5 from (0'0,0'0] MAX to 324'11168 2024-03-30T13:58:49.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:49 smithi043 ceph-mon[29963]: 10.a starting backfill to osd.5 from (0'0,0'0] MAX to 324'11085 2024-03-30T13:58:49.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:49 smithi043 ceph-mon[29963]: 10.6 starting backfill to osd.5 from (0'0,0'0] MAX to 324'11044 2024-03-30T13:58:49.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:49 smithi043 ceph-mon[42099]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 324'10998 2024-03-30T13:58:49.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:49 smithi043 ceph-mon[42099]: 10.9 starting backfill to osd.5 from (0'0,0'0] MAX to 324'11014 2024-03-30T13:58:49.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:49 smithi043 ceph-mon[42099]: 10.3 starting backfill to osd.5 from (0'0,0'0] MAX to 324'11109 2024-03-30T13:58:49.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:49 smithi043 ceph-mon[42099]: 10.7 starting backfill to osd.5 from (0'0,0'0] MAX to 324'11168 2024-03-30T13:58:49.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:49 smithi043 ceph-mon[42099]: 10.a starting backfill to osd.5 from (0'0,0'0] MAX to 324'11085 2024-03-30T13:58:49.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:49 smithi043 ceph-mon[42099]: 10.6 starting backfill to osd.5 from (0'0,0'0] MAX to 324'11044 2024-03-30T13:58:49.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:49.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:49.409 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:49 smithi113 ceph-mon[34341]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 324'10998 2024-03-30T13:58:49.409 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:49 smithi113 ceph-mon[34341]: 10.9 starting backfill to osd.5 from (0'0,0'0] MAX to 324'11014 2024-03-30T13:58:49.410 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:49 smithi113 ceph-mon[34341]: 10.3 starting backfill to osd.5 from (0'0,0'0] MAX to 324'11109 2024-03-30T13:58:49.410 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:49 smithi113 ceph-mon[34341]: 10.7 starting backfill to osd.5 from (0'0,0'0] MAX to 324'11168 2024-03-30T13:58:49.410 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:49 smithi113 ceph-mon[34341]: 10.a starting backfill to osd.5 from (0'0,0'0] MAX to 324'11085 2024-03-30T13:58:49.410 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:49 smithi113 ceph-mon[34341]: 10.6 starting backfill to osd.5 from (0'0,0'0] MAX to 324'11044 2024-03-30T13:58:49.410 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:58:49 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:58:49.497 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:49.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:49.536 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:49.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:49.637 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:49.673 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:49.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:49.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:49.775 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:49.782 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-30T13:58:49.812 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:49.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:49.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:49.913 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:49.951 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:49.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:49.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:50.053 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:50.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:50.107 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:50.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:50.186 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T13:58:50.190 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:58:50.208 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:50.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:50.250 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:50.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:50.354 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:50.395 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:50.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:50.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:50.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:50 smithi113 ceph-mon[34341]: osdmap e328: 8 total, 7 up, 7 in 2024-03-30T13:58:50.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:50 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:50.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:50 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1358590875' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:58:50.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:50 smithi113 ceph-mon[34341]: pgmap v711: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 99 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 3 active+undersized+remapped, 11 active+undersized+degraded, 29 active+clean+remapped; 9.1 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 950 op/s; 74631/447177 objects degraded (16.689%); 18842/447177 objects misplaced (4.214%); 1.3 MiB/s, 27 objects/s recovering 2024-03-30T13:58:50.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:50 smithi113 ceph-mon[34341]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:58:50.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:50 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1358590875' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:58:50.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:50 smithi113 ceph-mon[34341]: osdmap e329: 8 total, 7 up, 7 in 2024-03-30T13:58:50.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:50.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:50.541 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:50.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:50.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:50 smithi043 ceph-mon[29963]: osdmap e328: 8 total, 7 up, 7 in 2024-03-30T13:58:50.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:50 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:50.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:50 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1358590875' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:58:50.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:50 smithi043 ceph-mon[29963]: pgmap v711: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 99 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 3 active+undersized+remapped, 11 active+undersized+degraded, 29 active+clean+remapped; 9.1 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 950 op/s; 74631/447177 objects degraded (16.689%); 18842/447177 objects misplaced (4.214%); 1.3 MiB/s, 27 objects/s recovering 2024-03-30T13:58:50.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:50 smithi043 ceph-mon[29963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:58:50.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:50 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1358590875' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:58:50.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:50 smithi043 ceph-mon[29963]: osdmap e329: 8 total, 7 up, 7 in 2024-03-30T13:58:50.611 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:50 smithi043 ceph-mon[42099]: osdmap e328: 8 total, 7 up, 7 in 2024-03-30T13:58:50.611 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:50 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:50.611 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:50 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1358590875' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:58:50.611 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:50 smithi043 ceph-mon[42099]: pgmap v711: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 99 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 3 active+undersized+remapped, 11 active+undersized+degraded, 29 active+clean+remapped; 9.1 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 950 op/s; 74631/447177 objects degraded (16.689%); 18842/447177 objects misplaced (4.214%); 1.3 MiB/s, 27 objects/s recovering 2024-03-30T13:58:50.611 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:50 smithi043 ceph-mon[42099]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:58:50.611 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:50 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1358590875' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:58:50.611 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:50 smithi043 ceph-mon[42099]: osdmap e329: 8 total, 7 up, 7 in 2024-03-30T13:58:50.643 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:50.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:50.686 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:50.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:50.787 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:50.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:50.832 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:50.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:50.934 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:50.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:50.977 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:50.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:51.078 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:51.115 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:51.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:51.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:51.217 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:51.252 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:51.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:51.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:51.354 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:51.396 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:51.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:51.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:51.498 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:51.536 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:51.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:51.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:51.638 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:51.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:51.680 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:51.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:51.701 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:58:51.740 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:58:51.779 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:58:51.781 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:51.820 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:51 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3456634286' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:58:51.821 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:51.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:51.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:51.830 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:58:51.842 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 5] out_osds: [7] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-30T13:58:51.842 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:58:51.842 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-30T13:58:51.843 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-03-30T13:58:51.870 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:58:51.913 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:58:51.922 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:51.952 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:58:51.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:51 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3456634286' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:58:51.953 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:58:51.953 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:58:51.953 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:58:51.953 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:58:51.953 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:58:51.953 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:58:51.953 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:58:51.953 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:58:51.954 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:51 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3456634286' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:58:51.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:51.962 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:51.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:52.063 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:52.101 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:52.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:52.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:52.202 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:52.247 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:52.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:52.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:52.349 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:52.390 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:52.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:52.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:52.492 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:52.531 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:52.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:52.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:52.633 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:52.651 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:58:52.652 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:58:52.670 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:52.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:52.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:52.719 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:58:52.772 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:52.807 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:52.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:52.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:52.819 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:58:52.819 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [ 2024-03-30T13:58:52.819 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:52.819 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop(client.15531.0:181794 10.3 e330/326)", 2024-03-30T13:58:52.819 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:58:52.821482+0000", 2024-03-30T13:58:52.819 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.001914792, 2024-03-30T13:58:52.820 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001940195, 2024-03-30T13:58:52.820 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:58:52.820 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:58:52.820 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "queued for pg", 2024-03-30T13:58:52.820 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:58:52.820 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:52.820 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:58:52.820 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:52.821482+0000", 2024-03-30T13:58:52.820 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:58:52.820 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:52.820 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:52.820 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:58:52.820 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:52.821482+0000", 2024-03-30T13:58:52.820 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:58:52.821 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:52.821 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:52.821 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:58:52.821 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:52.821490+0000", 2024-03-30T13:58:52.821 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.7489999999999998e-06 2024-03-30T13:58:52.821 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:52.821 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:52.821 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:58:52.821 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:52.821497+0000", 2024-03-30T13:58:52.821 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6669999999999999e-06 2024-03-30T13:58:52.821 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:52.821 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:52.821 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:58:52.821 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:52.821524+0000", 2024-03-30T13:58:52.822 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.7257999999999999e-05 2024-03-30T13:58:52.822 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:52.822 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:52.822 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:58:52.822 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:52.821531+0000", 2024-03-30T13:58:52.822 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.63e-06 2024-03-30T13:58:52.822 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:58:52.822 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:58:52.822 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:58:52.822 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:52.822 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:52.822 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_repop(client.15531.0:181791 10.8 e330/326 10:1a75be97:::benchmark_data_smithi043_169264_object19564:head v 330'11441, mlcod=330'11438)", 2024-03-30T13:58:52.822 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:58:52.822325+0000", 2024-03-30T13:58:52.823 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 0.001072527, 2024-03-30T13:58:52.823 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0011542589999999999, 2024-03-30T13:58:52.823 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:58:52.823 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:58:52.823 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:58:52.823 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:58:52.823 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:52.825 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:58:52.825 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:52.822325+0000", 2024-03-30T13:58:52.825 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:58:52.825 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:52.825 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:52.825 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:58:52.825 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:52.822325+0000", 2024-03-30T13:58:52.825 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:58:52.826 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:52.826 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:52.826 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:58:52.826 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:52.822325+0000", 2024-03-30T13:58:52.826 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5899999999999996e-07 2024-03-30T13:58:52.826 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:52.826 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:52.826 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:58:52.826 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:52.822332+0000", 2024-03-30T13:58:52.826 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3110000000000002e-06 2024-03-30T13:58:52.826 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:52.826 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:52.826 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:58:52.827 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:52.822345+0000", 2024-03-30T13:58:52.827 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3006999999999999e-05 2024-03-30T13:58:52.827 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:52.827 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:52.827 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:58:52.827 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:52.822351+0000", 2024-03-30T13:58:52.827 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6719999999999998e-06 2024-03-30T13:58:52.827 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:52.827 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:52.827 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:58:52.827 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:52.822414+0000", 2024-03-30T13:58:52.827 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.2470000000000003e-05 2024-03-30T13:58:52.827 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:58:52.827 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:58:52.827 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:58:52.828 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:52.822433+0000", 2024-03-30T13:58:52.828 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.9355999999999999e-05 2024-03-30T13:58:52.828 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:58:52.828 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:58:52.828 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:58:52.828 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:58:52.828 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-03-30T13:58:52.828 INFO:teuthology.orchestra.run.smithi113.stdout: "num_ops": 2 2024-03-30T13:58:52.828 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:58:52.881 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:52 smithi043 ceph-mon[29963]: pgmap v713: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 3 active+undersized+remapped, 10 active+undersized+degraded, 27 active+clean+remapped; 9.0 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.03k op/s; 73749/443067 objects degraded (16.645%); 37069/443067 objects misplaced (8.366%); 5.6 MiB/s, 100 objects/s recovering 2024-03-30T13:58:52.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:52 smithi043 ceph-mon[29963]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T13:58:52.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:52 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 73749/443067 objects degraded (16.645%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:52.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:52 smithi043 ceph-mon[29963]: osdmap e330: 8 total, 7 up, 7 in 2024-03-30T13:58:52.882 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:52 smithi043 ceph-mon[42099]: pgmap v713: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 3 active+undersized+remapped, 10 active+undersized+degraded, 27 active+clean+remapped; 9.0 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.03k op/s; 73749/443067 objects degraded (16.645%); 37069/443067 objects misplaced (8.366%); 5.6 MiB/s, 100 objects/s recovering 2024-03-30T13:58:52.882 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:52 smithi043 ceph-mon[42099]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T13:58:52.882 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:52 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 73749/443067 objects degraded (16.645%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:52.882 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:52 smithi043 ceph-mon[42099]: osdmap e330: 8 total, 7 up, 7 in 2024-03-30T13:58:52.909 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:52.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:52.948 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:52.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:52.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:52 smithi113 ceph-mon[34341]: pgmap v713: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 3 active+undersized+remapped, 10 active+undersized+degraded, 27 active+clean+remapped; 9.0 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.03k op/s; 73749/443067 objects degraded (16.645%); 37069/443067 objects misplaced (8.366%); 5.6 MiB/s, 100 objects/s recovering 2024-03-30T13:58:52.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:52 smithi113 ceph-mon[34341]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T13:58:52.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:52 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 73749/443067 objects degraded (16.645%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:52.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:52 smithi113 ceph-mon[34341]: osdmap e330: 8 total, 7 up, 7 in 2024-03-30T13:58:53.049 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:53.081 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:53.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:53.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:53.183 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:53.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:53.232 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:53.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:53.333 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:53.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:53.377 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:53.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:53.409 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T13:58:53.478 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:53.525 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:53.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:53.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:53.624 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-03-30T13:58:53.628 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:53.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:53.676 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:53.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:53.777 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:53.821 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:53.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:53.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:53.924 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:53.964 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:53.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:53.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:53.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:53 smithi113 ceph-mon[34341]: 3.14 scrub starts 2024-03-30T13:58:53.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:53 smithi113 ceph-mon[34341]: 3.14 scrub ok 2024-03-30T13:58:54.026 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:58:54.066 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:54.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:54.105 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:54.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:54.131 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:53 smithi043 ceph-mon[29963]: 3.14 scrub starts 2024-03-30T13:58:54.132 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:53 smithi043 ceph-mon[29963]: 3.14 scrub ok 2024-03-30T13:58:54.132 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:53 smithi043 ceph-mon[42099]: 3.14 scrub starts 2024-03-30T13:58:54.132 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:53 smithi043 ceph-mon[42099]: 3.14 scrub ok 2024-03-30T13:58:54.207 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:54.277 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:54.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:54.296 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:58:54.379 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:54.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:54.418 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:54.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:54.519 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:54.565 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:54.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:54.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:54.652 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:58:54.667 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:54.755 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:54.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:54.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:54.857 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:54.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:54.906 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:54.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:54.939 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:54 smithi043 ceph-mon[29963]: 10.2 scrub starts 2024-03-30T13:58:54.939 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:54 smithi043 ceph-mon[29963]: 4.e scrub starts 2024-03-30T13:58:54.939 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:54 smithi043 ceph-mon[29963]: 4.e scrub ok 2024-03-30T13:58:54.939 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:54 smithi043 ceph-mon[29963]: pgmap v715: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 101 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 1 active+undersized+remapped, 10 active+undersized+degraded, 28 active+clean+remapped; 9.0 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 929 B/s rd, 0 B/s wr, 994 op/s; 72137/440568 objects degraded (16.374%); 36236/440568 objects misplaced (8.225%); 14 MiB/s, 226 objects/s recovering 2024-03-30T13:58:54.939 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:54 smithi043 ceph-mon[42099]: 10.2 scrub starts 2024-03-30T13:58:54.939 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:54 smithi043 ceph-mon[42099]: 4.e scrub starts 2024-03-30T13:58:54.939 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:54 smithi043 ceph-mon[42099]: 4.e scrub ok 2024-03-30T13:58:54.940 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:54 smithi043 ceph-mon[42099]: pgmap v715: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 101 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 1 active+undersized+remapped, 10 active+undersized+degraded, 28 active+clean+remapped; 9.0 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 929 B/s rd, 0 B/s wr, 994 op/s; 72137/440568 objects degraded (16.374%); 36236/440568 objects misplaced (8.225%); 14 MiB/s, 226 objects/s recovering 2024-03-30T13:58:54.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:54 smithi113 ceph-mon[34341]: 10.2 scrub starts 2024-03-30T13:58:54.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:54 smithi113 ceph-mon[34341]: 4.e scrub starts 2024-03-30T13:58:54.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:54 smithi113 ceph-mon[34341]: 4.e scrub ok 2024-03-30T13:58:54.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:54 smithi113 ceph-mon[34341]: pgmap v715: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 101 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 1 active+undersized+remapped, 10 active+undersized+degraded, 28 active+clean+remapped; 9.0 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 929 B/s rd, 0 B/s wr, 994 op/s; 72137/440568 objects degraded (16.374%); 36236/440568 objects misplaced (8.225%); 14 MiB/s, 226 objects/s recovering 2024-03-30T13:58:55.008 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:58:55.049 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:55.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:55.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:58:55.150 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:55.183 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:55.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:55.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:55.285 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:55.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:55.319 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:55.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:55.420 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:55.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:55.473 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:55.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:55.568 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:58:55.569 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:58:55.569 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 330, 2024-03-30T13:58:55.569 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "7ee932fa-ee9a-11ee-b647-cb9ed24678a4", 2024-03-30T13:58:55.569 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-03-30T13:37:07.346020+0000", 2024-03-30T13:58:55.569 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-03-30T13:58:52.599607+0000", 2024-03-30T13:58:55.569 INFO:teuthology.orchestra.run.smithi043.stdout: "last_up_change": "2024-03-30T13:50:07.489747+0000", 2024-03-30T13:58:55.569 INFO:teuthology.orchestra.run.smithi043.stdout: "last_in_change": "2024-03-30T13:58:46.155265+0000", 2024-03-30T13:58:55.570 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-30T13:58:55.570 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_num": 5804032, 2024-03-30T13:58:55.570 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_set": [ 2024-03-30T13:58:55.570 INFO:teuthology.orchestra.run.smithi043.stdout: "nodeep-scrub", 2024-03-30T13:58:55.570 INFO:teuthology.orchestra.run.smithi043.stdout: "pglog_hardlimit", 2024-03-30T13:58:55.570 INFO:teuthology.orchestra.run.smithi043.stdout: "purged_snapdirs", 2024-03-30T13:58:55.571 INFO:teuthology.orchestra.run.smithi043.stdout: "recovery_deletes", 2024-03-30T13:58:55.571 INFO:teuthology.orchestra.run.smithi043.stdout: "sortbitwise" 2024-03-30T13:58:55.571 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:58:55.571 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_version": 18, 2024-03-30T13:58:55.571 INFO:teuthology.orchestra.run.smithi043.stdout: "full_ratio": 0.94999998807907104, 2024-03-30T13:58:55.571 INFO:teuthology.orchestra.run.smithi043.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-30T13:58:55.572 INFO:teuthology.orchestra.run.smithi043.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-30T13:58:55.572 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_snapshot": "", 2024-03-30T13:58:55.572 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_max": 10, 2024-03-30T13:58:55.572 INFO:teuthology.orchestra.run.smithi043.stdout: "max_osd": 8, 2024-03-30T13:58:55.572 INFO:teuthology.orchestra.run.smithi043.stdout: "require_min_compat_client": "luminous", 2024-03-30T13:58:55.573 INFO:teuthology.orchestra.run.smithi043.stdout: "min_compat_client": "luminous", 2024-03-30T13:58:55.573 INFO:teuthology.orchestra.run.smithi043.stdout: "require_osd_release": "squid", 2024-03-30T13:58:55.573 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_crimson": false, 2024-03-30T13:58:55.574 INFO:teuthology.orchestra.run.smithi043.stdout: "pools": [ 2024-03-30T13:58:55.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.574 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 1, 2024-03-30T13:58:55.574 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".mgr", 2024-03-30T13:58:55.574 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:40:27.658979+0000", 2024-03-30T13:58:55.575 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T13:58:55.575 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T13:58:55.575 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:58:55.575 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:58:55.575 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:58:55.575 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:58:55.575 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:58:55.576 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:58:55.576 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:58:55.576 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:58:55.576 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:58:55.576 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:58:55.576 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 11, 2024-03-30T13:58:55.576 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 11, 2024-03-30T13:58:55.576 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 11, 2024-03-30T13:58:55.576 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 11, 2024-03-30T13:58:55.577 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 11, 2024-03-30T13:58:55.577 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:58:55.577 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:58:55.577 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:58:55.577 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:58:55.577 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:58:55.577 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:58:55.577 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:58:55.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.577 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "248", 2024-03-30T13:58:55.578 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:58:55.578 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:58:55.578 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "202", 2024-03-30T13:58:55.578 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:58:55.579 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:58:55.579 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T13:58:55.579 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T13:58:55.580 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:58:55.580 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:58:55.580 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:58:55.580 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:58:55.580 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:58:55.580 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:58:55.581 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:58:55.581 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:58:55.581 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:58:55.581 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:58:55.581 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:58:55.581 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:58:55.581 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:58:55.581 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:58:55.581 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:58:55.582 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:58:55.582 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:58:55.582 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:58:55.582 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:58:55.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.582 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:58:55.583 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:58:55.583 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:58:55.583 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:58:55.583 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:58:55.583 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:58:55.583 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:58:55.583 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:58:55.583 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:58:55.584 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:58:55.584 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:58:55.584 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-03-30T13:58:55.584 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_max": 32, 2024-03-30T13:58:55.584 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_min": 1 2024-03-30T13:58:55.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.584 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:58:55.584 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": {} 2024-03-30T13:58:55.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.584 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:58:55.585 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:58:55.585 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.9099999666213989, 2024-03-30T13:58:55.585 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.9099999666213989, 2024-03-30T13:58:55.585 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.86000001430511475, 2024-03-30T13:58:55.585 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.6399999856948853, 2024-03-30T13:58:55.585 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.6399999856948853, 2024-03-30T13:58:55.585 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:58:55.585 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:58:55.585 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:58:55.585 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.586 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 2, 2024-03-30T13:58:55.586 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "datapool", 2024-03-30T13:58:55.586 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:42.989975+0000", 2024-03-30T13:58:55.586 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 8193, 2024-03-30T13:58:55.586 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T13:58:55.586 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:58:55.586 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:58:55.587 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:58:55.587 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:58:55.587 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:58:55.587 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:58:55.587 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:58:55.587 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:58:55.587 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:58:55.587 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:58:55.588 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 3, 2024-03-30T13:58:55.588 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 3, 2024-03-30T13:58:55.588 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 3, 2024-03-30T13:58:55.588 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 3, 2024-03-30T13:58:55.588 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 3, 2024-03-30T13:58:55.588 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:58:55.588 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:58:55.588 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:58:55.588 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:58:55.588 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:58:55.588 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:58:55.588 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:58:55.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.589 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "68", 2024-03-30T13:58:55.589 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:58:55.589 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:58:55.589 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T13:58:55.589 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:58:55.589 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:58:55.589 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 2, 2024-03-30T13:58:55.589 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 68, 2024-03-30T13:58:55.590 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:58:55.590 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:58:55.590 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:58:55.590 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:58:55.590 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:58:55.590 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:58:55.590 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:58:55.590 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:58:55.590 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:58:55.590 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:58:55.590 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:58:55.590 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:58:55.591 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:58:55.591 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:58:55.591 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:58:55.591 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:58:55.591 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:58:55.591 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:58:55.591 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:58:55.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.591 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:58:55.591 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:58:55.591 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:58:55.591 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:58:55.591 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:58:55.592 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:58:55.592 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:58:55.592 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:58:55.592 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:58:55.592 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:58:55.592 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:58:55.592 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T13:58:55.592 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:58:55.592 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd": {} 2024-03-30T13:58:55.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.593 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:58:55.593 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:58:55.593 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.3499999046325684, 2024-03-30T13:58:55.593 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.3499999046325684, 2024-03-30T13:58:55.593 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.70999997854232788, 2024-03-30T13:58:55.593 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.6699999570846558, 2024-03-30T13:58:55.593 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.6699999570846558, 2024-03-30T13:58:55.593 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:58:55.593 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:58:55.593 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:58:55.593 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.594 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 3, 2024-03-30T13:58:55.594 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".rgw.root", 2024-03-30T13:58:55.594 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:43.910306+0000", 2024-03-30T13:58:55.595 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T13:58:55.595 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T13:58:55.595 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:58:55.595 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:58:55.595 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:58:55.595 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:58:55.595 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:58:55.595 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:58:55.595 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:58:55.595 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:58:55.595 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:58:55.595 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:58:55.595 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T13:58:55.595 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T13:58:55.596 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T13:58:55.596 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T13:58:55.596 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T13:58:55.596 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:58:55.596 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:58:55.596 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:58:55.596 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:58:55.596 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:58:55.596 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:55.598 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:58:55.598 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:58:55.598 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.598 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "66", 2024-03-30T13:58:55.598 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:58:55.598 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:58:55.599 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T13:58:55.599 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:58:55.599 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:58:55.599 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T13:58:55.599 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T13:58:55.599 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:58:55.599 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:58:55.599 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:58:55.599 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:58:55.599 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:58:55.599 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:58:55.599 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:58:55.599 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:58:55.599 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:58:55.600 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:58:55.600 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:58:55.600 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:58:55.600 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:58:55.600 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:58:55.600 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:58:55.600 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:58:55.600 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:58:55.600 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:58:55.600 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:58:55.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.600 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:58:55.600 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:58:55.600 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:58:55.601 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:58:55.601 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:58:55.601 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:58:55.601 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:58:55.601 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:58:55.601 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:58:55.601 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:58:55.601 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:58:55.601 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T13:58:55.601 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:58:55.601 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T13:58:55.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.601 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:58:55.601 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:58:55.602 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.7400000095367432, 2024-03-30T13:58:55.602 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.7400000095367432, 2024-03-30T13:58:55.602 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.86000001430511475, 2024-03-30T13:58:55.602 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.5, 2024-03-30T13:58:55.602 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.5, 2024-03-30T13:58:55.602 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:58:55.602 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:58:55.602 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:58:55.602 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.602 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.602 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 4, 2024-03-30T13:58:55.602 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.log", 2024-03-30T13:58:55.602 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:45.633132+0000", 2024-03-30T13:58:55.603 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T13:58:55.603 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T13:58:55.603 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:58:55.603 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:58:55.603 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:58:55.603 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:58:55.603 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:58:55.603 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:58:55.603 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:58:55.603 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:58:55.603 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:58:55.603 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:58:55.603 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T13:58:55.604 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T13:58:55.604 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T13:58:55.604 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T13:58:55.604 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T13:58:55.604 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:58:55.604 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:58:55.604 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:58:55.604 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:58:55.604 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:58:55.604 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:58:55.604 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:58:55.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.604 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "68", 2024-03-30T13:58:55.604 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:58:55.605 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:58:55.605 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T13:58:55.605 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:58:55.605 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:58:55.605 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T13:58:55.605 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T13:58:55.605 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:58:55.605 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:58:55.605 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:58:55.605 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:58:55.605 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:58:55.605 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:58:55.605 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:58:55.605 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:58:55.606 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:58:55.606 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:58:55.606 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:58:55.606 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:58:55.606 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:58:55.606 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:58:55.606 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:58:55.606 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:58:55.606 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:58:55.606 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:58:55.606 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:58:55.606 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.606 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:58:55.606 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:58:55.607 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:58:55.607 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:58:55.607 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:58:55.607 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:58:55.607 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:58:55.607 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:58:55.607 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:58:55.607 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:58:55.607 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:58:55.607 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T13:58:55.607 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:58:55.607 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T13:58:55.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.607 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:58:55.607 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:58:55.608 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.9700000286102295, 2024-03-30T13:58:55.608 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.9700000286102295, 2024-03-30T13:58:55.608 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.86000001430511475, 2024-03-30T13:58:55.608 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.690000057220459, 2024-03-30T13:58:55.608 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.690000057220459, 2024-03-30T13:58:55.608 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:58:55.608 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:58:55.608 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:58:55.608 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.608 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.608 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 5, 2024-03-30T13:58:55.608 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.control", 2024-03-30T13:58:55.609 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:47.570084+0000", 2024-03-30T13:58:55.609 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T13:58:55.609 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T13:58:55.609 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:58:55.609 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:58:55.609 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:58:55.609 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:58:55.609 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:58:55.609 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:58:55.610 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:58:55.610 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:58:55.610 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:58:55.610 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:58:55.610 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T13:58:55.610 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T13:58:55.610 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T13:58:55.610 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T13:58:55.610 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T13:58:55.610 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:58:55.610 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:58:55.610 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:58:55.610 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:58:55.611 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:58:55.611 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:58:55.611 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:58:55.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.611 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "70", 2024-03-30T13:58:55.611 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:58:55.611 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:58:55.611 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T13:58:55.611 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:58:55.611 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:58:55.612 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T13:58:55.612 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T13:58:55.612 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:58:55.612 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:58:55.612 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:58:55.612 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:58:55.612 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:58:55.612 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:58:55.612 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:58:55.612 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:58:55.612 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:58:55.612 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:58:55.612 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:58:55.612 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:58:55.613 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:58:55.613 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:58:55.613 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:58:55.613 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:58:55.613 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:58:55.613 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:58:55.613 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:58:55.613 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.613 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:58:55.613 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:58:55.613 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:58:55.613 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:58:55.613 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:58:55.613 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:58:55.614 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:58:55.614 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:58:55.614 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:58:55.614 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:58:55.614 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:58:55.614 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T13:58:55.614 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:58:55.614 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T13:58:55.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.614 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:58:55.614 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:58:55.614 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.7400000095367432, 2024-03-30T13:58:55.614 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.7400000095367432, 2024-03-30T13:58:55.614 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.86000001430511475, 2024-03-30T13:58:55.615 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.5, 2024-03-30T13:58:55.615 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.5, 2024-03-30T13:58:55.615 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:58:55.615 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:58:55.615 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:58:55.615 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.615 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.615 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 6, 2024-03-30T13:58:55.615 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.meta", 2024-03-30T13:58:55.615 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:49.637189+0000", 2024-03-30T13:58:55.615 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T13:58:55.615 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T13:58:55.616 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:58:55.616 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:58:55.616 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:58:55.616 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:58:55.616 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:58:55.616 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:58:55.616 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:58:55.616 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:58:55.616 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:58:55.616 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:58:55.616 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T13:58:55.616 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T13:58:55.616 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T13:58:55.616 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T13:58:55.617 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T13:58:55.617 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:58:55.617 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:58:55.617 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:58:55.617 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:58:55.617 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:58:55.617 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:58:55.617 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:58:55.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.617 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "72", 2024-03-30T13:58:55.617 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:58:55.617 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:58:55.617 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T13:58:55.617 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:58:55.618 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:58:55.618 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T13:58:55.618 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T13:58:55.618 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:58:55.618 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:58:55.618 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:58:55.618 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:58:55.618 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:58:55.618 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:58:55.618 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:58:55.618 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:58:55.618 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:58:55.618 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:58:55.618 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:58:55.619 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:58:55.619 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:58:55.619 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:58:55.619 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:58:55.619 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:58:55.619 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:58:55.619 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:58:55.619 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:58:55.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.620 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:58:55.620 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:58:55.620 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:58:55.620 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:58:55.620 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:58:55.620 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:58:55.620 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:58:55.620 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:58:55.620 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:58:55.620 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:58:55.620 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:58:55.620 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-03-30T13:58:55.620 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_bias": 4 2024-03-30T13:58:55.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.621 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:58:55.621 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T13:58:55.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.621 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:58:55.621 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:58:55.621 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.9700000286102295, 2024-03-30T13:58:55.621 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.9700000286102295, 2024-03-30T13:58:55.621 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.86000001430511475, 2024-03-30T13:58:55.621 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.690000057220459, 2024-03-30T13:58:55.621 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.690000057220459, 2024-03-30T13:58:55.621 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:58:55.621 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:58:55.621 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:58:55.622 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.622 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 10, 2024-03-30T13:58:55.622 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "unique_pool_3", 2024-03-30T13:58:55.622 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:56:54.614701+0000", 2024-03-30T13:58:55.622 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T13:58:55.622 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T13:58:55.622 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:58:55.622 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:58:55.622 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:58:55.622 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:58:55.622 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:58:55.623 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:58:55.623 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:58:55.623 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:58:55.623 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:58:55.623 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:58:55.623 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 16, 2024-03-30T13:58:55.623 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 16, 2024-03-30T13:58:55.623 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 16, 2024-03-30T13:58:55.623 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 16, 2024-03-30T13:58:55.623 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 16, 2024-03-30T13:58:55.623 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:58:55.623 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:58:55.623 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:58:55.623 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:58:55.624 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:58:55.624 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:58:55.624 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:58:55.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.624 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "295", 2024-03-30T13:58:55.624 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:58:55.624 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:58:55.624 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T13:58:55.624 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:58:55.624 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:58:55.624 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T13:58:55.624 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T13:58:55.624 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:58:55.624 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:58:55.625 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:58:55.625 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:58:55.625 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:58:55.625 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:58:55.625 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:58:55.625 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:58:55.625 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:58:55.625 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:58:55.625 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:58:55.625 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:58:55.625 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:58:55.625 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:58:55.625 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:58:55.625 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:58:55.625 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:58:55.626 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:58:55.626 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:58:55.626 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.626 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:58:55.626 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:58:55.626 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:58:55.626 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:58:55.626 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:58:55.626 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:58:55.626 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:58:55.626 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:58:55.626 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:58:55.626 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:58:55.627 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:58:55.627 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T13:58:55.627 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:58:55.627 INFO:teuthology.orchestra.run.smithi043.stdout: "rados": {} 2024-03-30T13:58:55.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.627 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:58:55.627 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:58:55.627 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.619999885559082, 2024-03-30T13:58:55.627 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.619999885559082, 2024-03-30T13:58:55.627 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.86000001430511475, 2024-03-30T13:58:55.627 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2.25, 2024-03-30T13:58:55.627 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2.25, 2024-03-30T13:58:55.627 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:58:55.627 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:58:55.628 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:58:55.628 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.628 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.628 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:58:55.628 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.628 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.628 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-03-30T13:58:55.628 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "94ea083e-6a27-41db-83af-2f7d3cae1956", 2024-03-30T13:58:55.628 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:58:55.628 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:58:55.628 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:58:55.628 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:58:55.628 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:58:55.628 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:58:55.628 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 8, 2024-03-30T13:58:55.628 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 326, 2024-03-30T13:58:55.629 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:58:55.629 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:58:55.629 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:58:55.629 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.629 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.629 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.629 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6802", 2024-03-30T13:58:55.629 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:58:55.629 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.629 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.629 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.629 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6803", 2024-03-30T13:58:55.629 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:58:55.629 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.630 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.630 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.630 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:58:55.630 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.630 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.630 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.630 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6804", 2024-03-30T13:58:55.630 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:58:55.630 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.630 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.630 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.630 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6805", 2024-03-30T13:58:55.630 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:58:55.630 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.631 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.631 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:58:55.631 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.631 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.631 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.631 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6808", 2024-03-30T13:58:55.631 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:58:55.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.631 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.631 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.631 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6809", 2024-03-30T13:58:55.631 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:58:55.632 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.632 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.632 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.632 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:58:55.632 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.632 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.632 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.632 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6806", 2024-03-30T13:58:55.632 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:58:55.632 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.632 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.632 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.632 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6807", 2024-03-30T13:58:55.632 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:58:55.633 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.633 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.633 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6803/1144592572", 2024-03-30T13:58:55.633 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6805/1144592572", 2024-03-30T13:58:55.633 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6809/1144592572", 2024-03-30T13:58:55.633 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6807/1144592572", 2024-03-30T13:58:55.633 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:58:55.633 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:58:55.633 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:58:55.633 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.633 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.633 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-03-30T13:58:55.634 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a", 2024-03-30T13:58:55.634 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:58:55.634 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:58:55.634 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:58:55.634 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:58:55.634 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:58:55.634 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:58:55.634 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 13, 2024-03-30T13:58:55.634 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 326, 2024-03-30T13:58:55.634 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:58:55.634 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:58:55.634 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:58:55.634 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.634 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.635 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.635 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6810", 2024-03-30T13:58:55.635 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:58:55.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.635 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.635 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.635 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6811", 2024-03-30T13:58:55.635 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:58:55.635 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.635 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.635 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:58:55.635 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.636 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.636 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.636 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6812", 2024-03-30T13:58:55.636 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:58:55.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.636 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.636 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.636 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6813", 2024-03-30T13:58:55.636 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:58:55.636 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.636 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.636 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:58:55.636 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.637 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.637 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.637 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6816", 2024-03-30T13:58:55.637 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:58:55.637 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.637 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.637 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.637 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6817", 2024-03-30T13:58:55.637 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:58:55.637 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.637 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.637 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.637 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:58:55.637 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.638 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.638 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.638 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6814", 2024-03-30T13:58:55.638 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:58:55.638 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.638 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.638 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.638 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6815", 2024-03-30T13:58:55.638 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:58:55.638 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.638 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.638 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.638 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6811/3901165928", 2024-03-30T13:58:55.639 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6813/3901165928", 2024-03-30T13:58:55.639 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6817/3901165928", 2024-03-30T13:58:55.639 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6815/3901165928", 2024-03-30T13:58:55.639 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:58:55.639 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:58:55.639 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:58:55.639 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.639 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.639 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-03-30T13:58:55.639 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "6db6782e-5889-443e-bc6c-df47a1f93bc8", 2024-03-30T13:58:55.639 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 0, 2024-03-30T13:58:55.639 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:58:55.640 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:58:55.640 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 0, 2024-03-30T13:58:55.640 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:58:55.640 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:58:55.640 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 18, 2024-03-30T13:58:55.640 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 98, 2024-03-30T13:58:55.640 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 167, 2024-03-30T13:58:55.640 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:58:55.640 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:58:55.640 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.640 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.640 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.640 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6818", 2024-03-30T13:58:55.640 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:58:55.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.641 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.641 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.641 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6819", 2024-03-30T13:58:55.641 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:58:55.641 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.641 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.641 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:58:55.641 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.641 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.641 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.641 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6820", 2024-03-30T13:58:55.641 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:58:55.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6821", 2024-03-30T13:58:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:58:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:58:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6824", 2024-03-30T13:58:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:58:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6825", 2024-03-30T13:58:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:58:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:58:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6822", 2024-03-30T13:58:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:58:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6823", 2024-03-30T13:58:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:58:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6819/3415037523", 2024-03-30T13:58:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6821/3415037523", 2024-03-30T13:58:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6825/3415037523", 2024-03-30T13:58:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6823/3415037523", 2024-03-30T13:58:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:58:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: "exists" 2024-03-30T13:58:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-03-30T13:58:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "7048ff46-2cba-4d18-9835-95ecbddb2bb6", 2024-03-30T13:58:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:58:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:58:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:58:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:58:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:58:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:58:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 25, 2024-03-30T13:58:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 328, 2024-03-30T13:58:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:58:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:58:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:58:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6826", 2024-03-30T13:58:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:58:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6827", 2024-03-30T13:58:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:58:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:58:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6828", 2024-03-30T13:58:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:58:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6829", 2024-03-30T13:58:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:58:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.648 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.648 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:58:55.648 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.648 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.648 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.648 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6832", 2024-03-30T13:58:55.648 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:58:55.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.648 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.648 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.648 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6833", 2024-03-30T13:58:55.648 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:58:55.649 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.649 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.649 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.649 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:58:55.649 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.649 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.649 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.649 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6830", 2024-03-30T13:58:55.649 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:58:55.649 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.649 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.649 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.649 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6831", 2024-03-30T13:58:55.649 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:58:55.649 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.650 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.650 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.650 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6827/2933589376", 2024-03-30T13:58:55.650 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6829/2933589376", 2024-03-30T13:58:55.650 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6833/2933589376", 2024-03-30T13:58:55.651 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6831/2933589376", 2024-03-30T13:58:55.651 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:58:55.651 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:58:55.651 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:58:55.651 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.651 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.651 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.651 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-03-30T13:58:55.651 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "42da30e6-9eec-4ed5-8c73-9ce78aa8ff63", 2024-03-30T13:58:55.652 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:58:55.652 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:58:55.652 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:58:55.652 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:58:55.652 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:58:55.652 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:58:55.652 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 32, 2024-03-30T13:58:55.652 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 326, 2024-03-30T13:58:55.652 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:58:55.652 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:58:55.652 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:58:55.652 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.652 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.652 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.652 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6800", 2024-03-30T13:58:55.653 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:58:55.653 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.653 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.653 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.653 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6801", 2024-03-30T13:58:55.653 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:58:55.653 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.653 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.653 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.653 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:58:55.653 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.653 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.653 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.653 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6802", 2024-03-30T13:58:55.654 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:58:55.654 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.654 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.654 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.654 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6803", 2024-03-30T13:58:55.654 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:58:55.654 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.654 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.654 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.654 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:58:55.654 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.654 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.654 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.655 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6806", 2024-03-30T13:58:55.655 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:58:55.655 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.655 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.655 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.655 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6807", 2024-03-30T13:58:55.655 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:58:55.655 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.655 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.655 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.655 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:58:55.655 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.655 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.655 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.656 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6804", 2024-03-30T13:58:55.656 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:58:55.656 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.656 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.656 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.656 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6805", 2024-03-30T13:58:55.656 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:58:55.656 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.656 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.656 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.656 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6801/1945833081", 2024-03-30T13:58:55.656 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6803/1945833081", 2024-03-30T13:58:55.656 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6807/1945833081", 2024-03-30T13:58:55.657 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6805/1945833081", 2024-03-30T13:58:55.657 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:58:55.657 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:58:55.657 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:58:55.657 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.657 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.657 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.657 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-03-30T13:58:55.657 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "99a11b81-0df7-4b1d-946a-4297ba62e263", 2024-03-30T13:58:55.657 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:58:55.657 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:58:55.657 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:58:55.657 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:58:55.657 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:58:55.657 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:58:55.658 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 40, 2024-03-30T13:58:55.658 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 326, 2024-03-30T13:58:55.658 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:58:55.658 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:58:55.658 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:58:55.658 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.658 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.658 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.658 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6808", 2024-03-30T13:58:55.658 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:58:55.658 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.658 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.659 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.659 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6809", 2024-03-30T13:58:55.659 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:58:55.659 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.659 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.659 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.659 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:58:55.659 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.659 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.659 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.659 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6810", 2024-03-30T13:58:55.659 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:58:55.659 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.659 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6811", 2024-03-30T13:58:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:58:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:58:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6814", 2024-03-30T13:58:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:58:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6815", 2024-03-30T13:58:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:58:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:58:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6812", 2024-03-30T13:58:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:58:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6813", 2024-03-30T13:58:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:58:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6809/3426360325", 2024-03-30T13:58:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6811/3426360325", 2024-03-30T13:58:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6815/3426360325", 2024-03-30T13:58:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6813/3426360325", 2024-03-30T13:58:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:58:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:58:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:58:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-03-30T13:58:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "cafd0e26-78e9-4164-a43a-96ca8382fefe", 2024-03-30T13:58:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:58:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:58:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:58:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:58:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:58:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:58:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 49, 2024-03-30T13:58:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 328, 2024-03-30T13:58:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:58:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:58:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:58:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6816", 2024-03-30T13:58:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:58:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6817", 2024-03-30T13:58:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:58:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:58:55.665 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.665 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.665 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.665 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6818", 2024-03-30T13:58:55.665 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:58:55.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.665 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.665 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.665 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6819", 2024-03-30T13:58:55.665 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:58:55.665 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.665 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.665 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:58:55.666 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.666 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.666 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.666 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6822", 2024-03-30T13:58:55.666 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:58:55.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.666 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.666 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.666 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6823", 2024-03-30T13:58:55.666 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:58:55.666 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.666 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.666 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:58:55.667 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.667 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.667 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.667 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6820", 2024-03-30T13:58:55.667 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:58:55.667 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.667 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.667 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.667 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6821", 2024-03-30T13:58:55.667 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:58:55.667 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.667 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.667 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.667 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6817/1527017698", 2024-03-30T13:58:55.668 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6819/1527017698", 2024-03-30T13:58:55.668 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6823/1527017698", 2024-03-30T13:58:55.668 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6821/1527017698", 2024-03-30T13:58:55.668 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:58:55.668 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:58:55.668 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:58:55.668 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.668 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.668 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-03-30T13:58:55.668 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "e9d59662-9132-48f4-9e95-842a3cfbd79f", 2024-03-30T13:58:55.668 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:58:55.668 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 0, 2024-03-30T13:58:55.668 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-03-30T13:58:55.669 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:58:55.669 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:58:55.669 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:58:55.669 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 58, 2024-03-30T13:58:55.669 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 228, 2024-03-30T13:58:55.669 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:58:55.669 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:58:55.669 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:58:55.669 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.669 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.669 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.669 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6824", 2024-03-30T13:58:55.669 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:58:55.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.670 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.670 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.670 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6825", 2024-03-30T13:58:55.670 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:58:55.670 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.670 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.670 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.670 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:58:55.670 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.670 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.670 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.670 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6826", 2024-03-30T13:58:55.670 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:58:55.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.671 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.671 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.671 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6827", 2024-03-30T13:58:55.671 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:58:55.671 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.671 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.671 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:58:55.671 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.671 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.671 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.671 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6830", 2024-03-30T13:58:55.671 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:58:55.672 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.672 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.672 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.672 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6831", 2024-03-30T13:58:55.672 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:58:55.672 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.672 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.672 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.672 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:58:55.672 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:58:55.672 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.672 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:58:55.672 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6828", 2024-03-30T13:58:55.672 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:58:55.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.673 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.673 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:58:55.673 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6829", 2024-03-30T13:58:55.673 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:58:55.673 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.673 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.673 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6825/3182476015", 2024-03-30T13:58:55.673 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6827/3182476015", 2024-03-30T13:58:55.673 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6831/3182476015", 2024-03-30T13:58:55.673 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6829/3182476015", 2024-03-30T13:58:55.673 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:58:55.673 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:58:55.674 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:58:55.674 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.674 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.674 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:58:55.674 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_xinfo": [ 2024-03-30T13:58:55.674 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.674 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-03-30T13:58:55.674 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:58:55.674 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:58:55.674 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:58:55.674 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:58:55.674 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:58:55.674 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:58:09.417388+0000", 2024-03-30T13:58:55.675 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:58:55.675 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.675 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.675 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-03-30T13:58:55.675 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:58:55.675 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:58:55.675 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:58:55.675 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:58:55.675 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:58:55.675 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:58:36.388795+0000", 2024-03-30T13:58:55.675 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:58:55.675 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.675 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.675 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-03-30T13:58:55.676 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:58:55.676 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:58:55.676 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:58:55.676 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:58:55.676 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:58:55.676 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:49:03.444490+0000", 2024-03-30T13:58:55.676 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 166 2024-03-30T13:58:55.676 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.676 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.676 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-03-30T13:58:55.676 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:58:55.676 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:58:55.676 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:58:55.677 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:58:55.677 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:58:55.677 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:58:38.119388+0000", 2024-03-30T13:58:55.677 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:58:55.677 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.677 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.677 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-03-30T13:58:55.677 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:58:55.677 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:58:55.677 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:58:55.677 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:58:55.677 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:58:55.677 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:57:58.847762+0000", 2024-03-30T13:58:55.677 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:58:55.678 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.678 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.678 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-03-30T13:58:55.678 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:58:55.678 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:58:55.678 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:58:55.678 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:58:55.678 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:58:55.678 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:57:44.008850+0000", 2024-03-30T13:58:55.678 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:58:55.678 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.678 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.678 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-03-30T13:58:55.679 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:58:55.679 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:58:55.679 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:58:55.679 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:58:55.679 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:58:55.679 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:58:17.340072+0000", 2024-03-30T13:58:55.679 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:58:55.679 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.679 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.679 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-03-30T13:58:55.679 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:58:55.679 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:58:55.679 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:58:55.679 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:58:55.680 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 65536, 2024-03-30T13:58:55.680 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:58:02.540701+0000", 2024-03-30T13:58:55.680 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:58:55.680 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.680 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:58:55.680 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap": [], 2024-03-30T13:58:55.680 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_items": [ 2024-03-30T13:58:55.680 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.680 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.13", 2024-03-30T13:58:55.680 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-03-30T13:58:55.680 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.680 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-03-30T13:58:55.680 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-03-30T13:58:55.680 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.680 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.681 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.681 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.681 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.14", 2024-03-30T13:58:55.681 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-03-30T13:58:55.681 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.681 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-03-30T13:58:55.681 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-03-30T13:58:55.681 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.681 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.681 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.681 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:58:55.681 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_primaries": [], 2024-03-30T13:58:55.681 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_temp": [ 2024-03-30T13:58:55.681 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.681 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "1.1", 2024-03-30T13:58:55.682 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.682 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:58:55.682 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:58:55.682 INFO:teuthology.orchestra.run.smithi043.stdout: 4 2024-03-30T13:58:55.682 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.682 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.682 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.682 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "1.2", 2024-03-30T13:58:55.682 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.682 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:58:55.686 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:58:55.686 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:58:55.686 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.686 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.686 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.686 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "1.5", 2024-03-30T13:58:55.687 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.687 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:58:55.687 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T13:58:55.687 INFO:teuthology.orchestra.run.smithi043.stdout: 0 2024-03-30T13:58:55.687 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.687 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.687 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.687 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "1.8", 2024-03-30T13:58:55.687 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.687 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:58:55.687 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:58:55.687 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-03-30T13:58:55.687 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.687 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.688 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.688 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "1.9", 2024-03-30T13:58:55.688 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.688 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:58:55.688 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:58:55.688 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-03-30T13:58:55.688 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.688 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.688 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.688 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "1.a", 2024-03-30T13:58:55.688 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.688 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:58:55.688 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T13:58:55.688 INFO:teuthology.orchestra.run.smithi043.stdout: 4 2024-03-30T13:58:55.689 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.689 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.689 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.689 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "3.4", 2024-03-30T13:58:55.689 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.689 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T13:58:55.689 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:58:55.689 INFO:teuthology.orchestra.run.smithi043.stdout: 0 2024-03-30T13:58:55.689 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.689 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.689 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.689 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "3.5", 2024-03-30T13:58:55.689 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.689 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:58:55.690 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:58:55.690 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-03-30T13:58:55.690 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.690 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.690 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.690 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "3.9", 2024-03-30T13:58:55.690 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.690 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:58:55.690 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T13:58:55.690 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:58:55.690 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.690 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.690 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.691 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "3.13", 2024-03-30T13:58:55.691 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.691 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:58:55.691 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T13:58:55.691 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:58:55.691 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.691 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.691 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.691 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "3.14", 2024-03-30T13:58:55.691 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.691 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:58:55.691 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T13:58:55.691 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:58:55.691 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.692 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.692 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.692 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "3.1f", 2024-03-30T13:58:55.692 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.692 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:58:55.692 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:58:55.692 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-03-30T13:58:55.692 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.692 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.692 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.692 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.14", 2024-03-30T13:58:55.692 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.692 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:58:55.692 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T13:58:55.692 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:58:55.693 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.693 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.693 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.693 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.16", 2024-03-30T13:58:55.693 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.693 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:58:55.693 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:58:55.693 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:58:55.693 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.693 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.693 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.693 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.17", 2024-03-30T13:58:55.693 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.693 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:58:55.693 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T13:58:55.694 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:58:55.694 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.694 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.694 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.694 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.4", 2024-03-30T13:58:55.694 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.694 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:58:55.694 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:58:55.694 INFO:teuthology.orchestra.run.smithi043.stdout: 1 2024-03-30T13:58:55.694 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.694 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.694 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.694 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.6", 2024-03-30T13:58:55.694 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.695 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:58:55.695 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:58:55.695 INFO:teuthology.orchestra.run.smithi043.stdout: 1 2024-03-30T13:58:55.695 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.695 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.695 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.695 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.b", 2024-03-30T13:58:55.695 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.695 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:58:55.695 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:58:55.695 INFO:teuthology.orchestra.run.smithi043.stdout: 1 2024-03-30T13:58:55.695 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.695 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.695 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.695 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.d", 2024-03-30T13:58:55.696 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.696 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:58:55.696 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:58:55.696 INFO:teuthology.orchestra.run.smithi043.stdout: 3 2024-03-30T13:58:55.696 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.696 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.696 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.696 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.14", 2024-03-30T13:58:55.696 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.696 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:58:55.696 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T13:58:55.696 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:58:55.696 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.696 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.696 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.697 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.19", 2024-03-30T13:58:55.697 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.697 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T13:58:55.697 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:58:55.697 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-03-30T13:58:55.697 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.697 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.697 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.697 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.1e", 2024-03-30T13:58:55.697 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.697 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:58:55.697 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T13:58:55.697 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:58:55.697 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.698 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.698 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.1f", 2024-03-30T13:58:55.698 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.698 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T13:58:55.698 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:58:55.698 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:58:55.698 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.698 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.698 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.3", 2024-03-30T13:58:55.698 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.698 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T13:58:55.699 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T13:58:55.699 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:58:55.699 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.699 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.699 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.5", 2024-03-30T13:58:55.699 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.699 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T13:58:55.699 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:58:55.699 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:58:55.699 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.699 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.699 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.8", 2024-03-30T13:58:55.699 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.700 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:58:55.700 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T13:58:55.700 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:58:55.700 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.700 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.700 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.9", 2024-03-30T13:58:55.700 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.700 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:58:55.700 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:58:55.700 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:58:55.700 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.700 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.701 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.12", 2024-03-30T13:58:55.701 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.701 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:58:55.701 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:58:55.701 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:58:55.701 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.701 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.701 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.14", 2024-03-30T13:58:55.701 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.701 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:58:55.701 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:58:55.701 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:58:55.701 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.702 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.702 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.17", 2024-03-30T13:58:55.702 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.702 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:58:55.702 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:58:55.702 INFO:teuthology.orchestra.run.smithi043.stdout: 0 2024-03-30T13:58:55.702 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.702 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.702 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.18", 2024-03-30T13:58:55.702 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.702 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:58:55.702 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T13:58:55.703 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:58:55.703 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.703 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.703 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.703 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.1c", 2024-03-30T13:58:55.703 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.703 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:58:55.703 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:58:55.703 INFO:teuthology.orchestra.run.smithi043.stdout: 4 2024-03-30T13:58:55.703 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.703 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.703 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.703 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "10.3", 2024-03-30T13:58:55.703 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.703 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:58:55.704 INFO:teuthology.orchestra.run.smithi043.stdout: 4 2024-03-30T13:58:55.704 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.704 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.704 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.704 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "10.6", 2024-03-30T13:58:55.704 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.704 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:58:55.704 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:58:55.704 INFO:teuthology.orchestra.run.smithi043.stdout: 1 2024-03-30T13:58:55.704 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.704 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.704 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.704 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "10.7", 2024-03-30T13:58:55.704 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.705 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T13:58:55.705 INFO:teuthology.orchestra.run.smithi043.stdout: 3 2024-03-30T13:58:55.705 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.705 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.705 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "10.8", 2024-03-30T13:58:55.705 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.705 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:58:55.705 INFO:teuthology.orchestra.run.smithi043.stdout: 4 2024-03-30T13:58:55.705 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.705 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.705 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "10.9", 2024-03-30T13:58:55.705 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.706 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:58:55.706 INFO:teuthology.orchestra.run.smithi043.stdout: 4 2024-03-30T13:58:55.706 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.706 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:58:55.706 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "10.a", 2024-03-30T13:58:55.706 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:58:55.706 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T13:58:55.706 INFO:teuthology.orchestra.run.smithi043.stdout: 0 2024-03-30T13:58:55.706 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:58:55.706 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.706 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:58:55.706 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_temp": [], 2024-03-30T13:58:55.706 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklist": { 2024-03-30T13:58:55.707 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/538304234": "2024-03-31T13:43:27.253365+0000", 2024-03-30T13:58:55.707 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6801/3411723337": "2024-03-31T13:37:41.733515+0000", 2024-03-30T13:58:55.707 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3656823679": "2024-03-31T13:37:41.733515+0000", 2024-03-30T13:58:55.707 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2618817647": "2024-03-31T13:38:09.321740+0000", 2024-03-30T13:58:55.707 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/3411723337": "2024-03-31T13:37:41.733515+0000", 2024-03-30T13:58:55.707 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1206218784": "2024-03-31T13:37:41.733515+0000", 2024-03-30T13:58:55.707 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/886631486": "2024-03-31T13:37:41.733515+0000", 2024-03-30T13:58:55.707 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/1339921781": "2024-03-31T13:38:09.321740+0000", 2024-03-30T13:58:55.707 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6801/1339921781": "2024-03-31T13:38:09.321740+0000", 2024-03-30T13:58:55.707 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2794494528": "2024-03-31T13:38:09.321740+0000", 2024-03-30T13:58:55.707 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/564868778": "2024-03-31T13:38:09.321740+0000", 2024-03-30T13:58:55.707 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/3033707884": "2024-03-31T13:43:27.253365+0000", 2024-03-30T13:58:55.707 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3539398453": "2024-03-31T13:43:27.253365+0000", 2024-03-30T13:58:55.707 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2414364740": "2024-03-31T13:43:27.253365+0000", 2024-03-30T13:58:55.707 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6801/3033707884": "2024-03-31T13:43:27.253365+0000", 2024-03-30T13:58:55.708 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3923597429": "2024-03-31T13:43:27.253365+0000", 2024-03-30T13:58:55.708 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2524269526": "2024-03-31T13:43:27.253365+0000" 2024-03-30T13:58:55.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.708 INFO:teuthology.orchestra.run.smithi043.stdout: "range_blocklist": {}, 2024-03-30T13:58:55.708 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profiles": { 2024-03-30T13:58:55.708 INFO:teuthology.orchestra.run.smithi043.stdout: "default": { 2024-03-30T13:58:55.708 INFO:teuthology.orchestra.run.smithi043.stdout: "crush-failure-domain": "osd", 2024-03-30T13:58:55.708 INFO:teuthology.orchestra.run.smithi043.stdout: "k": "2", 2024-03-30T13:58:55.708 INFO:teuthology.orchestra.run.smithi043.stdout: "m": "1", 2024-03-30T13:58:55.708 INFO:teuthology.orchestra.run.smithi043.stdout: "plugin": "jerasure", 2024-03-30T13:58:55.708 INFO:teuthology.orchestra.run.smithi043.stdout: "technique": "reed_sol_van" 2024-03-30T13:58:55.708 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:58:55.708 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps_queue": [], 2024-03-30T13:58:55.709 INFO:teuthology.orchestra.run.smithi043.stdout: "new_removed_snaps": [], 2024-03-30T13:58:55.709 INFO:teuthology.orchestra.run.smithi043.stdout: "new_purged_snaps": [], 2024-03-30T13:58:55.709 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_node_flags": {}, 2024-03-30T13:58:55.709 INFO:teuthology.orchestra.run.smithi043.stdout: "device_class_flags": {}, 2024-03-30T13:58:55.709 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": { 2024-03-30T13:58:55.709 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_enabled": false, 2024-03-30T13:58:55.709 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_bucket_count": 0, 2024-03-30T13:58:55.709 INFO:teuthology.orchestra.run.smithi043.stdout: "degraded_stretch_mode": 0, 2024-03-30T13:58:55.709 INFO:teuthology.orchestra.run.smithi043.stdout: "recovering_stretch_mode": 0, 2024-03-30T13:58:55.709 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_bucket": 0 2024-03-30T13:58:55.709 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:58:55.709 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:58:55.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:55.711 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:55.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:55.813 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:55.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:55.858 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:55.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:55.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:55 smithi043 ceph-mon[29963]: 4.9 scrub starts 2024-03-30T13:58:55.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:55 smithi043 ceph-mon[29963]: 4.9 scrub ok 2024-03-30T13:58:55.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:55 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3861963685' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T13:58:55.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:55 smithi043 ceph-mon[42099]: 4.9 scrub starts 2024-03-30T13:58:55.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:55 smithi043 ceph-mon[42099]: 4.9 scrub ok 2024-03-30T13:58:55.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:55 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3861963685' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T13:58:55.959 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:55.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:55 smithi113 ceph-mon[34341]: 4.9 scrub starts 2024-03-30T13:58:55.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:55 smithi113 ceph-mon[34341]: 4.9 scrub ok 2024-03-30T13:58:55.967 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:55 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3861963685' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T13:58:55.995 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:55.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:55.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:56.011 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:58:56.011 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [], 2024-03-30T13:58:56.011 INFO:teuthology.orchestra.run.smithi113.stdout: "complaint_time": 30, 2024-03-30T13:58:56.011 INFO:teuthology.orchestra.run.smithi113.stdout: "num_blocked_ops": 0 2024-03-30T13:58:56.011 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:58:56.098 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:56.134 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:56.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:56.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:56.235 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:56.272 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:56.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:56.373 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:56.393 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:58:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:58:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:58:56.410 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:56.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:56.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:56.512 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:56.537 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:56.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:56.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:56.639 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:56.676 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:56.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:56.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:56.778 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:56.815 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:56.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:56.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:56.918 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:56.954 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:56.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:56.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:57.056 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:57.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:57.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:57.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:56 smithi043 ceph-mon[29963]: pgmap v716: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 8.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 788 B/s rd, 0 B/s wr, 931 op/s; 69642/429036 objects degraded (16.232%); 36236/429036 objects misplaced (8.446%); 19 MiB/s, 302 objects/s recovering 2024-03-30T13:58:57.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:56 smithi043 ceph-mon[42099]: pgmap v716: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 8.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 788 B/s rd, 0 B/s wr, 931 op/s; 69642/429036 objects degraded (16.232%); 36236/429036 objects misplaced (8.446%); 19 MiB/s, 302 objects/s recovering 2024-03-30T13:58:57.196 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:57.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:56 smithi113 ceph-mon[34341]: pgmap v716: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 8.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 788 B/s rd, 0 B/s wr, 931 op/s; 69642/429036 objects degraded (16.232%); 36236/429036 objects misplaced (8.446%); 19 MiB/s, 302 objects/s recovering 2024-03-30T13:58:57.233 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:57.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:57.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:57.334 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:57.371 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:57.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:57.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:57.473 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:57.513 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:57.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:57.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:57.615 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:57.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:57.648 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:57.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:57.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:57.770 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:57 smithi043 ceph-mon[29963]: 10.2 scrub ok 2024-03-30T13:58:57.770 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:57 smithi043 ceph-mon[29963]: 6.4 scrub starts 2024-03-30T13:58:57.770 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:57 smithi043 ceph-mon[29963]: 6.4 scrub ok 2024-03-30T13:58:57.770 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:57 smithi043 ceph-mon[29963]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T13:58:57.771 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:57 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 69642/429036 objects degraded (16.232%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:57.771 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:57 smithi043 ceph-mon[42099]: 10.2 scrub ok 2024-03-30T13:58:57.771 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:57 smithi043 ceph-mon[42099]: 6.4 scrub starts 2024-03-30T13:58:57.771 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:57 smithi043 ceph-mon[42099]: 6.4 scrub ok 2024-03-30T13:58:57.771 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:57 smithi043 ceph-mon[42099]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T13:58:57.771 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:57 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 69642/429036 objects degraded (16.232%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:57.794 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:57.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:57.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:57.816 INFO:tasks.thrashosds.thrasher:j is {'epoch': 330, 'fsid': '7ee932fa-ee9a-11ee-b647-cb9ed24678a4', 'created': '2024-03-30T13:37:07.346020+0000', 'modified': '2024-03-30T13:58:52.599607+0000', 'last_up_change': '2024-03-30T13:50:07.489747+0000', 'last_in_change': '2024-03-30T13:58:46.155265+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T13:40:27.658979+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '202', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.6399999856948853, 'raw_score_stable': 1.6399999856948853, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-30T13:42:42.989975+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3499999046325684, 'score_stable': 2.3499999046325684, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 1.6699999570846558, 'raw_score_stable': 1.6699999570846558, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-30T13:42:43.910306+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-30T13:42:45.633132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-30T13:42:47.570084+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-30T13:42:49.637189+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-03-30T13:56:54.614701+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '295', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.619999885559082, 'score_stable': 2.619999885559082, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '94ea083e-6a27-41db-83af-2f7d3cae1956', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 326, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6802', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6803', 'nonce': 1144592572}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6804', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6805', 'nonce': 1144592572}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6808', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6809', 'nonce': 1144592572}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6806', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6807', 'nonce': 1144592572}]}, 'public_addr': '172.21.15.43:6803/1144592572', 'cluster_addr': '172.21.15.43:6805/1144592572', 'heartbeat_back_addr': '172.21.15.43:6809/1144592572', 'heartbeat_front_addr': '172.21.15.43:6807/1144592572', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 326, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6810', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6811', 'nonce': 3901165928}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6812', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6813', 'nonce': 3901165928}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6816', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6817', 'nonce': 3901165928}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6814', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6815', 'nonce': 3901165928}]}, 'public_addr': '172.21.15.43:6811/3901165928', 'cluster_addr': '172.21.15.43:6813/3901165928', 'heartbeat_back_addr': '172.21.15.43:6817/3901165928', 'heartbeat_front_addr': '172.21.15.43:6815/3901165928', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6db6782e-5889-443e-bc6c-df47a1f93bc8', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 98, 'down_at': 167, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6818', 'nonce': 3415037523}, {'type': 'v1', 'addr': '172.21.15.43:6819', 'nonce': 3415037523}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6820', 'nonce': 3415037523}, {'type': 'v1', 'addr': '172.21.15.43:6821', 'nonce': 3415037523}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6824', 'nonce': 3415037523}, {'type': 'v1', 'addr': '172.21.15.43:6825', 'nonce': 3415037523}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6822', 'nonce': 3415037523}, {'type': 'v1', 'addr': '172.21.15.43:6823', 'nonce': 3415037523}]}, 'public_addr': '172.21.15.43:6819/3415037523', 'cluster_addr': '172.21.15.43:6821/3415037523', 'heartbeat_back_addr': '172.21.15.43:6825/3415037523', 'heartbeat_front_addr': '172.21.15.43:6823/3415037523', 'state': ['exists']}, {'osd': 3, 'uuid': '7048ff46-2cba-4d18-9835-95ecbddb2bb6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 328, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6826', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6827', 'nonce': 2933589376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6828', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6829', 'nonce': 2933589376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6832', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6833', 'nonce': 2933589376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6830', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6831', 'nonce': 2933589376}]}, 'public_addr': '172.21.15.43:6827/2933589376', 'cluster_addr': '172.21.15.43:6829/2933589376', 'heartbeat_back_addr': '172.21.15.43:6833/2933589376', 'heartbeat_front_addr': '172.21.15.43:6831/2933589376', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '42da30e6-9eec-4ed5-8c73-9ce78aa8ff63', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 326, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6800', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6801', 'nonce': 1945833081}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6802', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6803', 'nonce': 1945833081}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6806', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6807', 'nonce': 1945833081}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6804', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6805', 'nonce': 1945833081}]}, 'public_addr': '172.21.15.113:6801/1945833081', 'cluster_addr': '172.21.15.113:6803/1945833081', 'heartbeat_back_addr': '172.21.15.113:6807/1945833081', 'heartbeat_front_addr': '172.21.15.113:6805/1945833081', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '99a11b81-0df7-4b1d-946a-4297ba62e263', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 326, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6808', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6809', 'nonce': 3426360325}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6810', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6811', 'nonce': 3426360325}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6814', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6815', 'nonce': 3426360325}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6812', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6813', 'nonce': 3426360325}]}, 'public_addr': '172.21.15.113:6809/3426360325', 'cluster_addr': '172.21.15.113:6811/3426360325', 'heartbeat_back_addr': '172.21.15.113:6815/3426360325', 'heartbeat_front_addr': '172.21.15.113:6813/3426360325', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'cafd0e26-78e9-4164-a43a-96ca8382fefe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 328, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6816', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6817', 'nonce': 1527017698}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6818', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6819', 'nonce': 1527017698}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6822', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6823', 'nonce': 1527017698}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6820', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6821', 'nonce': 1527017698}]}, 'public_addr': '172.21.15.113:6817/1527017698', 'cluster_addr': '172.21.15.113:6819/1527017698', 'heartbeat_back_addr': '172.21.15.113:6823/1527017698', 'heartbeat_front_addr': '172.21.15.113:6821/1527017698', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e9d59662-9132-48f4-9e95-842a3cfbd79f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 228, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6824', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6825', 'nonce': 3182476015}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6826', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6827', 'nonce': 3182476015}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6830', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6831', 'nonce': 3182476015}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6828', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6829', 'nonce': 3182476015}]}, 'public_addr': '172.21.15.113:6825/3182476015', 'cluster_addr': '172.21.15.113:6827/3182476015', 'heartbeat_back_addr': '172.21.15.113:6831/3182476015', 'heartbeat_front_addr': '172.21.15.113:6829/3182476015', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:58:09.417388+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:58:36.388795+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:49:03.444490+0000', 'dead_epoch': 166}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:58:38.119388+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:57:58.847762+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:57:44.008850+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:58:17.340072+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-30T13:58:02.540701+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.1', 'osds': [5, 0, 4]}, {'pgid': '1.2', 'osds': [0, 4, 7]}, {'pgid': '1.5', 'osds': [4, 6, 0]}, {'pgid': '1.8', 'osds': [5, 3, 6]}, {'pgid': '1.9', 'osds': [3, 5, 6]}, {'pgid': '1.a', 'osds': [5, 6, 4]}, {'pgid': '3.4', 'osds': [1, 5, 0]}, {'pgid': '3.5', 'osds': [5, 3, 6]}, {'pgid': '3.9', 'osds': [4, 1, 7]}, {'pgid': '3.13', 'osds': [4, 6, 7]}, {'pgid': '3.14', 'osds': [4, 6, 7]}, {'pgid': '3.1f', 'osds': [0, 5, 6]}, {'pgid': '4.14', 'osds': [3, 1, 7]}, {'pgid': '4.16', 'osds': [0, 3, 7]}, {'pgid': '4.17', 'osds': [3, 1, 7]}, {'pgid': '5.4', 'osds': [5, 0, 1]}, {'pgid': '5.6', 'osds': [5, 4, 1]}, {'pgid': '5.b', 'osds': [0, 5, 1]}, {'pgid': '5.d', 'osds': [5, 0, 3]}, {'pgid': '5.14', 'osds': [3, 1, 7]}, {'pgid': '5.19', 'osds': [1, 5, 6]}, {'pgid': '5.1e', 'osds': [0, 1, 7]}, {'pgid': '5.1f', 'osds': [6, 4, 7]}, {'pgid': '6.3', 'osds': [6, 1, 7]}, {'pgid': '6.5', 'osds': [6, 3, 7]}, {'pgid': '6.8', 'osds': [3, 6, 7]}, {'pgid': '6.9', 'osds': [0, 3, 7]}, {'pgid': '6.12', 'osds': [4, 3, 7]}, {'pgid': '6.14', 'osds': [4, 0, 7]}, {'pgid': '6.17', 'osds': [4, 5, 0]}, {'pgid': '6.18', 'osds': [0, 1, 7]}, {'pgid': '6.1c', 'osds': [5, 3, 4]}, {'pgid': '10.3', 'osds': [3, 4]}, {'pgid': '10.6', 'osds': [0, 4, 1]}, {'pgid': '10.7', 'osds': [6, 3]}, {'pgid': '10.8', 'osds': [0, 4]}, {'pgid': '10.9', 'osds': [3, 4]}, {'pgid': '10.a', 'osds': [6, 0]}], 'primary_temp': [], 'blocklist': {'172.21.15.43:0/538304234': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:6801/3411723337': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/3656823679': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/2618817647': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:6800/3411723337': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/1206218784': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/886631486': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:6800/1339921781': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:6801/1339921781': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:0/2794494528': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:0/564868778': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:6800/3033707884': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/3539398453': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/2414364740': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:6801/3033707884': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/3923597429': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/2524269526': '2024-03-31T13:43:27.253365+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-30T13:58:57.816 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T13:58:57.852 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:58:57.895 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:57.935 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:57.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:57.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:58.037 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:58.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:58.079 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:58.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:58.139 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:57 smithi113 ceph-mon[34341]: 10.2 scrub ok 2024-03-30T13:58:58.140 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:57 smithi113 ceph-mon[34341]: 6.4 scrub starts 2024-03-30T13:58:58.140 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:57 smithi113 ceph-mon[34341]: 6.4 scrub ok 2024-03-30T13:58:58.140 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:57 smithi113 ceph-mon[34341]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T13:58:58.140 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:57 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 69642/429036 objects degraded (16.232%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:58:58.180 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:58.217 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:58.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:58.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:58.319 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:58.358 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:58.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:58.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:58.459 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:58.496 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:58.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:58.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:58.598 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:58.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:58.634 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:58.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:58.735 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:58:58.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:58.768 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:58.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:58:58.870 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:58.902 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:58.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:58.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:59.004 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:58:59.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:59.039 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:59.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:58:59.140 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:59.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:58 smithi043 ceph-mon[29963]: pgmap v717: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 8.6 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 68114/423006 objects degraded (16.102%); 35415/423006 objects misplaced (8.372%); 21 MiB/s, 334 objects/s recovering 2024-03-30T13:58:59.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:58 smithi043 ceph-mon[42099]: pgmap v717: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 8.6 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 68114/423006 objects degraded (16.102%); 35415/423006 objects misplaced (8.372%); 21 MiB/s, 334 objects/s recovering 2024-03-30T13:58:59.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:59.182 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:59.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:59.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:58 smithi113 ceph-mon[34341]: pgmap v717: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 8.6 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 68114/423006 objects degraded (16.102%); 35415/423006 objects misplaced (8.372%); 21 MiB/s, 334 objects/s recovering 2024-03-30T13:58:59.284 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:59.318 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:59.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:59.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:58:59.420 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:58:59.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:59.457 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:59.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:58:59.558 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:58:59.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:59.582 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:59.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:58:59.683 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:59.716 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:58:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:58:59.719 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:59.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:59.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:59.768 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-03-30T13:58:59.821 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:58:59.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:58:59.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:58:59.949 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:58:59.984 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:58:59.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:59.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:00.086 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:00.119 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:00.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:00.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:00.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:59 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3615593150' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:59:00.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:59 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:59:00.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:59 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:00.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:58:59 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:00.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:59 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3615593150' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:59:00.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:59 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:59:00.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:59 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:00.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:58:59 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:00.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:59 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3615593150' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:59:00.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:59 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:59:00.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:59 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:00.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:58:59 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:00.221 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:00.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:00.251 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:00.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:00.352 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:00.388 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:00.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:00.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:00.490 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:00.526 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:00.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:00.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:00.628 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:00.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:00.664 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:00.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:00.765 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:00.801 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:00.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:00.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:00.836 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:00 smithi043 ceph-mon[29963]: 4.7 scrub starts 2024-03-30T13:59:00.836 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:00 smithi043 ceph-mon[29963]: 4.7 scrub ok 2024-03-30T13:59:00.836 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:00 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:00.836 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:00 smithi043 ceph-mon[29963]: pgmap v718: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 8.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s; 63907/409698 objects degraded (15.599%); 34584/409698 objects misplaced (8.441%); 29 MiB/s, 473 objects/s recovering 2024-03-30T13:59:00.836 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:00 smithi043 ceph-mon[29963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:59:00.836 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:00 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:59:00.836 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:00 smithi043 ceph-mon[29963]: osdmap e331: 8 total, 7 up, 7 in 2024-03-30T13:59:00.837 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:00 smithi043 ceph-mon[42099]: 4.7 scrub starts 2024-03-30T13:59:00.837 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:00 smithi043 ceph-mon[42099]: 4.7 scrub ok 2024-03-30T13:59:00.837 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:00 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:00.837 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:00 smithi043 ceph-mon[42099]: pgmap v718: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 8.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s; 63907/409698 objects degraded (15.599%); 34584/409698 objects misplaced (8.441%); 29 MiB/s, 473 objects/s recovering 2024-03-30T13:59:00.837 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:00 smithi043 ceph-mon[42099]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:59:00.837 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:00 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:59:00.837 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:00 smithi043 ceph-mon[42099]: osdmap e331: 8 total, 7 up, 7 in 2024-03-30T13:59:00.904 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:00.945 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:00.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:00.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:01.004 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:00 smithi113 ceph-mon[34341]: 4.7 scrub starts 2024-03-30T13:59:01.004 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:00 smithi113 ceph-mon[34341]: 4.7 scrub ok 2024-03-30T13:59:01.004 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:00 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:01.004 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:00 smithi113 ceph-mon[34341]: pgmap v718: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 8.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s; 63907/409698 objects degraded (15.599%); 34584/409698 objects misplaced (8.441%); 29 MiB/s, 473 objects/s recovering 2024-03-30T13:59:01.004 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:00 smithi113 ceph-mon[34341]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T13:59:01.004 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:00 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T13:59:01.005 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:00 smithi113 ceph-mon[34341]: osdmap e331: 8 total, 7 up, 7 in 2024-03-30T13:59:01.006 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-03-30T13:59:01.046 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:01.088 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:01.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:01.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:01.190 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:01.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:01.231 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:01.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:01.234 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:59:01.264 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:59:01.306 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:59:01.332 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:01.345 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:59:01.387 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:59:01.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:01.388 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:01.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:01.417 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:59:01.424 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:59:01.460 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:59:01.490 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:01.501 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:59:01.501 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:59:01.501 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:59:01.501 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:59:01.502 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:59:01.502 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:59:01.502 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:59:01.502 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:59:01.502 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:59:01.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:01.538 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:01.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:01.639 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:01.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:01.681 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:01.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:01.782 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:01.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:01.823 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:01.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:01.898 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:01 smithi113 ceph-mon[34341]: 4.2 scrub starts 2024-03-30T13:59:01.898 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:01 smithi113 ceph-mon[34341]: 4.2 scrub ok 2024-03-30T13:59:01.898 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:01 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/203780553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:59:01.924 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:01.966 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:01.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:02.068 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:02.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:02.103 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:02.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:02.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:01 smithi043 ceph-mon[29963]: 4.2 scrub starts 2024-03-30T13:59:02.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:01 smithi043 ceph-mon[29963]: 4.2 scrub ok 2024-03-30T13:59:02.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:01 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/203780553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:59:02.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:01 smithi043 ceph-mon[42099]: 4.2 scrub starts 2024-03-30T13:59:02.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:01 smithi043 ceph-mon[42099]: 4.2 scrub ok 2024-03-30T13:59:02.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:01 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/203780553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:59:02.204 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:02.242 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:02.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:02.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:02.346 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:02.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:02.383 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:02.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:02.484 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:02.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:02.519 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:02.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:02.580 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T13:59:02.621 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:02.650 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:02.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:02.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:02.752 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:02.785 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:02.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:02.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:02.853 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:02 smithi113 ceph-mon[34341]: pgmap v720: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 8.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.44k op/s; 62262/403758 objects degraded (15.421%); 33771/403758 objects misplaced (8.364%); 35 MiB/s, 556 objects/s recovering 2024-03-30T13:59:02.853 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:02 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:59:02.853 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:02 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:59:02.853 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:02 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 62262/403758 objects degraded (15.421%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:02.853 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:02 smithi113 ceph-mon[34341]: osdmap e332: 8 total, 7 up, 7 in 2024-03-30T13:59:02.887 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:02.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:02 smithi043 ceph-mon[29963]: pgmap v720: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 8.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.44k op/s; 62262/403758 objects degraded (15.421%); 33771/403758 objects misplaced (8.364%); 35 MiB/s, 556 objects/s recovering 2024-03-30T13:59:02.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:02 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:59:02.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:02 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:59:02.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:02 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 62262/403758 objects degraded (15.421%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:02.895 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:02 smithi043 ceph-mon[29963]: osdmap e332: 8 total, 7 up, 7 in 2024-03-30T13:59:02.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:02 smithi043 ceph-mon[42099]: pgmap v720: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 8.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.44k op/s; 62262/403758 objects degraded (15.421%); 33771/403758 objects misplaced (8.364%); 35 MiB/s, 556 objects/s recovering 2024-03-30T13:59:02.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:02 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:59:02.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:02 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:59:02.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:02 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 62262/403758 objects degraded (15.421%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:02.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:02 smithi043 ceph-mon[42099]: osdmap e332: 8 total, 7 up, 7 in 2024-03-30T13:59:02.930 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:02.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:02.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:03.032 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:03.071 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:03.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:03.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:03.173 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:03.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:03.228 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:03.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:03.329 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:03.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:03.369 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:03.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:03.470 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:03.504 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:03.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:03.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:03.582 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:59:03.582 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:59:03.605 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:03.642 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:03.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:03.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:03.744 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:03.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:03.784 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:03.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:03.885 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:03.911 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:03.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:03.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:04.013 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:04.048 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:04.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:04.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:04.151 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:04.187 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:04.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:04.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:04.288 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:04.310 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-30T13:59:04.332 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:04.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:04.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:04.434 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:04.482 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:04.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:04.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:04.513 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:59:04.514 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:59:04.584 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:04.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:04.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:04.719 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:04.751 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:59:04.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:04.771 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:04.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:04.871 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:04.897 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:04.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:04.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:04.999 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:05.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:05.026 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:05.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:05.049 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:04 smithi113 ceph-mon[34341]: pgmap v722: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 8.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 60607/401334 objects degraded (15.101%); 32963/401334 objects misplaced (8.213%); 34 MiB/s, 546 objects/s recovering 2024-03-30T13:59:05.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:05.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:04 smithi043 ceph-mon[29963]: pgmap v722: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 8.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 60607/401334 objects degraded (15.101%); 32963/401334 objects misplaced (8.213%); 34 MiB/s, 546 objects/s recovering 2024-03-30T13:59:05.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:04 smithi043 ceph-mon[42099]: pgmap v722: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 8.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 60607/401334 objects degraded (15.101%); 32963/401334 objects misplaced (8.213%); 34 MiB/s, 546 objects/s recovering 2024-03-30T13:59:05.170 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:05.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:05.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:05.272 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:05.306 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:05.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:05.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:05.408 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:05.436 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:05.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:05.538 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:05.577 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:05.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:05.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:05.679 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:05.716 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:05.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:05.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:05.818 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:05.860 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:05.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:05.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:05.962 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:05.999 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:06.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:06.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:06.101 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:06.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:06.132 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:06.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:06.233 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:06.260 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:06.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:06.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:06.362 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:06.393 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:59:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:59:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:59:06.400 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:06.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:06.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:06.502 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:06.515 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:59:06.527 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:06.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:06.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:06.629 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:06.685 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:06.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:06.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:06.789 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:06.823 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:06.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:06.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:06.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:06 smithi043 ceph-mon[29963]: pgmap v723: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.35k op/s; 58331/390528 objects degraded (14.936%); 32963/390528 objects misplaced (8.441%); 35 MiB/s, 560 objects/s recovering 2024-03-30T13:59:06.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:06 smithi043 ceph-mon[42099]: pgmap v723: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.35k op/s; 58331/390528 objects degraded (14.936%); 32963/390528 objects misplaced (8.441%); 35 MiB/s, 560 objects/s recovering 2024-03-30T13:59:06.925 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:06.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:06.962 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:06.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:07.063 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:07.091 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:07.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:07.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:07.193 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:07.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:06 smithi113 ceph-mon[34341]: pgmap v723: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.35k op/s; 58331/390528 objects degraded (14.936%); 32963/390528 objects misplaced (8.441%); 35 MiB/s, 560 objects/s recovering 2024-03-30T13:59:07.217 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:07.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:07.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:07.319 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:07.360 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:07.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:07.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:07.462 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:07.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:07.503 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:07.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:07.604 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:07.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:07.639 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:07.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:07.740 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:07.775 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:07.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:07.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:07.877 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:07.912 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:07.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:07.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:08.014 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:08.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:08.051 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:08.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:08.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:07 smithi043 ceph-mon[29963]: 5.c deep-scrub starts 2024-03-30T13:59:08.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:07 smithi043 ceph-mon[29963]: 5.c deep-scrub ok 2024-03-30T13:59:08.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:07 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 58331/390528 objects degraded (14.936%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:08.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:07 smithi043 ceph-mon[42099]: 5.c deep-scrub starts 2024-03-30T13:59:08.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:07 smithi043 ceph-mon[42099]: 5.c deep-scrub ok 2024-03-30T13:59:08.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:07 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 58331/390528 objects degraded (14.936%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:08.153 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:08.186 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:08.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:08.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:08.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:07 smithi113 ceph-mon[34341]: 5.c deep-scrub starts 2024-03-30T13:59:08.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:07 smithi113 ceph-mon[34341]: 5.c deep-scrub ok 2024-03-30T13:59:08.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:07 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 58331/390528 objects degraded (14.936%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:08.288 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:08.319 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:08.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:08.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:08.421 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:08.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:08.458 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:08.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:08.559 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:08.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:08.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:08.691 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:08.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:08.725 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:08.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:08.826 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:08.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:08.852 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:08.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:08.953 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:08.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:08.992 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:08.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:09.093 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:09.116 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:09.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:09.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:09.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:08 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:59:09.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:08 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:59:09.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:08 smithi043 ceph-mon[29963]: pgmap v724: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 56931/384672 objects degraded (14.800%); 32177/384672 objects misplaced (8.365%); 26 MiB/s, 423 objects/s recovering 2024-03-30T13:59:09.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:08 smithi043 ceph-mon[29963]: osdmap e333: 8 total, 7 up, 7 in 2024-03-30T13:59:09.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:08 smithi043 ceph-mon[29963]: 5.1d scrub starts 2024-03-30T13:59:09.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:08 smithi043 ceph-mon[29963]: 5.1d scrub ok 2024-03-30T13:59:09.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:08 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:59:09.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:08 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:59:09.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:08 smithi043 ceph-mon[42099]: pgmap v724: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 56931/384672 objects degraded (14.800%); 32177/384672 objects misplaced (8.365%); 26 MiB/s, 423 objects/s recovering 2024-03-30T13:59:09.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:08 smithi043 ceph-mon[42099]: osdmap e333: 8 total, 7 up, 7 in 2024-03-30T13:59:09.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:08 smithi043 ceph-mon[42099]: 5.1d scrub starts 2024-03-30T13:59:09.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:08 smithi043 ceph-mon[42099]: 5.1d scrub ok 2024-03-30T13:59:09.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:08 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:59:09.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:08 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:59:09.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:08 smithi113 ceph-mon[34341]: pgmap v724: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 56931/384672 objects degraded (14.800%); 32177/384672 objects misplaced (8.365%); 26 MiB/s, 423 objects/s recovering 2024-03-30T13:59:09.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:08 smithi113 ceph-mon[34341]: osdmap e333: 8 total, 7 up, 7 in 2024-03-30T13:59:09.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:08 smithi113 ceph-mon[34341]: 5.1d scrub starts 2024-03-30T13:59:09.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:08 smithi113 ceph-mon[34341]: 5.1d scrub ok 2024-03-30T13:59:09.218 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:09.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:09.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:09.359 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:09.394 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:09.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:09.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:09.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:09.542 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:09.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:09.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:09.644 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:09.669 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:59:09.696 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:09.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:09.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:09.716 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:59:09 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:59:09.797 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:09.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:09.832 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:09.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:09.933 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:09.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:09.974 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:09.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:10.075 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:10.114 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:10.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:10.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:10.216 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:10.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:10.246 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:10.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:10.347 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:10.382 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:10.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:10.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:10.484 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:10.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:10.508 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:10.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:10.609 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:10.648 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:10.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:10.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:10.750 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:10.777 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:10.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:10.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:10.879 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:10.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:10.915 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:10.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:11.016 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:11.046 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:11.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:11.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:11.123 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:10 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:11.123 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:10 smithi113 ceph-mon[34341]: pgmap v726: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.27k op/s; 53116/373143 objects degraded (14.235%); 31484/373143 objects misplaced (8.438%); 34 MiB/s, 551 objects/s recovering 2024-03-30T13:59:11.123 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:10 smithi113 ceph-mon[34341]: 2.0 scrub starts 2024-03-30T13:59:11.123 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:10 smithi113 ceph-mon[34341]: 2.0 scrub ok 2024-03-30T13:59:11.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:10 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:11.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:10 smithi043 ceph-mon[29963]: pgmap v726: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.27k op/s; 53116/373143 objects degraded (14.235%); 31484/373143 objects misplaced (8.438%); 34 MiB/s, 551 objects/s recovering 2024-03-30T13:59:11.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:10 smithi043 ceph-mon[29963]: 2.0 scrub starts 2024-03-30T13:59:11.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:10 smithi043 ceph-mon[29963]: 2.0 scrub ok 2024-03-30T13:59:11.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:10 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:11.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:10 smithi043 ceph-mon[42099]: pgmap v726: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.27k op/s; 53116/373143 objects degraded (14.235%); 31484/373143 objects misplaced (8.438%); 34 MiB/s, 551 objects/s recovering 2024-03-30T13:59:11.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:10 smithi043 ceph-mon[42099]: 2.0 scrub starts 2024-03-30T13:59:11.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:10 smithi043 ceph-mon[42099]: 2.0 scrub ok 2024-03-30T13:59:11.148 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:11.188 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:11.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:11.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:11.290 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:11.331 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:11.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:11.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:11.433 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:11.468 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:11.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:11.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:11.570 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:11.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:11.605 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:11.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:11.706 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:11.725 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:59:11.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:11.759 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:11.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:11.860 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:11.899 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:11.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:11.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:12.002 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:12.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:12.037 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:12.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:12.138 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:12.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:12.163 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:12.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:12.264 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:12.300 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:12.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:12.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:12.402 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:12.427 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:12.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:12.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:12.529 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:12.566 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:12.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:12.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:12.667 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:12.708 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:12.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:12.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:12.810 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:12.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:12.850 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:12.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:12.951 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:12.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:12 smithi113 ceph-mon[34341]: 3.f scrub starts 2024-03-30T13:59:12.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:12 smithi113 ceph-mon[34341]: 3.f scrub ok 2024-03-30T13:59:12.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:12 smithi113 ceph-mon[34341]: pgmap v727: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s; 51811/367773 objects degraded (14.088%); 30776/367773 objects misplaced (8.368%); 35 MiB/s, 565 objects/s recovering 2024-03-30T13:59:12.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:12 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 51811/367773 objects degraded (14.088%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:12.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:12 smithi113 ceph-mon[34341]: osdmap e334: 8 total, 7 up, 7 in 2024-03-30T13:59:12.986 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:12.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:12.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:13.088 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:13.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:13.124 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:13.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:12 smithi043 ceph-mon[29963]: 3.f scrub starts 2024-03-30T13:59:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:12 smithi043 ceph-mon[29963]: 3.f scrub ok 2024-03-30T13:59:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:12 smithi043 ceph-mon[29963]: pgmap v727: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s; 51811/367773 objects degraded (14.088%); 30776/367773 objects misplaced (8.368%); 35 MiB/s, 565 objects/s recovering 2024-03-30T13:59:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:12 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 51811/367773 objects degraded (14.088%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:12 smithi043 ceph-mon[29963]: osdmap e334: 8 total, 7 up, 7 in 2024-03-30T13:59:13.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:12 smithi043 ceph-mon[42099]: 3.f scrub starts 2024-03-30T13:59:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:12 smithi043 ceph-mon[42099]: 3.f scrub ok 2024-03-30T13:59:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:12 smithi043 ceph-mon[42099]: pgmap v727: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s; 51811/367773 objects degraded (14.088%); 30776/367773 objects misplaced (8.368%); 35 MiB/s, 565 objects/s recovering 2024-03-30T13:59:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:12 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 51811/367773 objects degraded (14.088%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:12 smithi043 ceph-mon[42099]: osdmap e334: 8 total, 7 up, 7 in 2024-03-30T13:59:13.226 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:13.265 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:13.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:13.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:13.368 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:13.405 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:13.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:13.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:13.507 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:13.524 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:59:13.528 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-30T13:59:13.545 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:13.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:13.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:13.647 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:13.681 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:13.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:13.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:13.783 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:13.816 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:13.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:13.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:13.918 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:13.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:13.954 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:13.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:14.055 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:14.091 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:14.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:14.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:14.193 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:14.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:14.231 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:14.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:14.322 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:59:14.322 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:59:14.333 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:14.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:14.361 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:14.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:14.462 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:14.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:14.502 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:14.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:14.603 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:14.639 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:14.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:14.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:14.741 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:14.776 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:14.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:14.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:14.878 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:14.928 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:14.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:14.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:15.001 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:14 smithi043 ceph-mon[29963]: 4.1b scrub starts 2024-03-30T13:59:15.001 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:14 smithi043 ceph-mon[29963]: 4.1b scrub ok 2024-03-30T13:59:15.001 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:14 smithi043 ceph-mon[29963]: from='client.25523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:59:15.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:14 smithi043 ceph-mon[29963]: pgmap v729: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 50379/365493 objects degraded (13.784%); 30016/365493 objects misplaced (8.212%); 33 MiB/s, 524 objects/s recovering 2024-03-30T13:59:15.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:14 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:15.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:14 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:15.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:14 smithi043 ceph-mon[29963]: 5.19 scrub starts 2024-03-30T13:59:15.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:14 smithi043 ceph-mon[29963]: 5.19 scrub ok 2024-03-30T13:59:15.002 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:14 smithi043 ceph-mon[42099]: 4.1b scrub starts 2024-03-30T13:59:15.002 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:14 smithi043 ceph-mon[42099]: 4.1b scrub ok 2024-03-30T13:59:15.002 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:14 smithi043 ceph-mon[42099]: from='client.25523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:59:15.002 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:14 smithi043 ceph-mon[42099]: pgmap v729: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 50379/365493 objects degraded (13.784%); 30016/365493 objects misplaced (8.212%); 33 MiB/s, 524 objects/s recovering 2024-03-30T13:59:15.003 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:14 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:15.003 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:14 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:15.003 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:14 smithi043 ceph-mon[42099]: 5.19 scrub starts 2024-03-30T13:59:15.003 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:14 smithi043 ceph-mon[42099]: 5.19 scrub ok 2024-03-30T13:59:15.029 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:15.037 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:14 smithi113 ceph-mon[34341]: 4.1b scrub starts 2024-03-30T13:59:15.037 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:14 smithi113 ceph-mon[34341]: 4.1b scrub ok 2024-03-30T13:59:15.037 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:14 smithi113 ceph-mon[34341]: from='client.25523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:59:15.038 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:14 smithi113 ceph-mon[34341]: pgmap v729: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 50379/365493 objects degraded (13.784%); 30016/365493 objects misplaced (8.212%); 33 MiB/s, 524 objects/s recovering 2024-03-30T13:59:15.038 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:14 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:15.038 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:14 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:15.038 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:14 smithi113 ceph-mon[34341]: 5.19 scrub starts 2024-03-30T13:59:15.038 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:14 smithi113 ceph-mon[34341]: 5.19 scrub ok 2024-03-30T13:59:15.080 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-03-30T13:59:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:15.081 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:15.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:15.183 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:15.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:15.221 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:15.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:15.322 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:15.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:15.346 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:15.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:15.447 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:15.472 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:59:15.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:15.492 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:15.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:15.593 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:15.631 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:15.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:15.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:15.733 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:15.771 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:15.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:15.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:15.873 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:15.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:15.914 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:15.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:15.951 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:15 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:59:15.951 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:15 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:59:15.951 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:15 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:59:15.951 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:15 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:59:15.951 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:15 smithi113 ceph-mon[34341]: osdmap e335: 8 total, 7 up, 7 in 2024-03-30T13:59:15.951 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:15 smithi113 ceph-mon[34341]: 1.3 scrub starts 2024-03-30T13:59:15.951 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:15 smithi113 ceph-mon[34341]: 1.3 scrub ok 2024-03-30T13:59:16.016 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:16.062 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:16.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:16.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:16.072 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:15 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:59:16.072 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:15 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:59:16.072 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:15 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:59:16.072 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:15 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:59:16.072 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:15 smithi043 ceph-mon[29963]: osdmap e335: 8 total, 7 up, 7 in 2024-03-30T13:59:16.072 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:15 smithi043 ceph-mon[29963]: 1.3 scrub starts 2024-03-30T13:59:16.073 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:15 smithi043 ceph-mon[29963]: 1.3 scrub ok 2024-03-30T13:59:16.073 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:15 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:59:16.073 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:15 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:59:16.073 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:15 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:59:16.073 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:15 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:59:16.073 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:15 smithi043 ceph-mon[42099]: osdmap e335: 8 total, 7 up, 7 in 2024-03-30T13:59:16.073 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:15 smithi043 ceph-mon[42099]: 1.3 scrub starts 2024-03-30T13:59:16.073 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:15 smithi043 ceph-mon[42099]: 1.3 scrub ok 2024-03-30T13:59:16.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:16.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:16.204 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:16.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:16.305 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:16.341 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:16.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:16.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:16.393 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:59:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:59:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:59:16.442 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:16.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:16.478 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:16.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:16.579 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:16.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:16.622 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:16.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:16.723 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:16.757 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:16.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:16.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:16.858 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:16.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:16.914 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:16.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:17.015 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:17.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:17.058 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:17.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:17.089 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:16 smithi043 ceph-mon[29963]: pgmap v731: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 648 B/s rd, 0 B/s wr, 1.22k op/s; 48163/355833 objects degraded (13.535%); 30016/355833 objects misplaced (8.435%); 35 MiB/s, 565 objects/s recovering 2024-03-30T13:59:17.089 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:16 smithi043 ceph-mon[42099]: pgmap v731: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 648 B/s rd, 0 B/s wr, 1.22k op/s; 48163/355833 objects degraded (13.535%); 30016/355833 objects misplaced (8.435%); 35 MiB/s, 565 objects/s recovering 2024-03-30T13:59:17.091 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":728,"stamp":"2024-03-30T13:59:12.610213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8021678454,"num_objects":122591,"num_object_clones":0,"num_object_copies":367773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":51811,"num_objects_misplaced":30776,"num_objects_unfound":0,"num_objects_dirty":122591,"num_whiteouts":0,"num_read":1479,"num_read_kb":1231,"num_write":202779,"num_write_kb":10382382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10756,"num_bytes_recovered":694752909,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69610,"ondisk_log_size":69610,"up":417,"acting":447,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":480,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":26504004,"kb_used_data":22306624,"kb_used_omap":10,"kb_used_meta":4197045,"kb_avail":629712060,"statfs":{"total":671965249536,"available":644825149440,"internally_reserved":0,"allocated":22841982976,"data_stored":22833121280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11162,"internal_metadata":4297774182},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-733151232,"num_objects":-11187,"num_object_clones":0,"num_object_copies":-33561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-8796,"num_objects_misplaced":-2187,"num_objects_unfound":0,"num_objects_dirty":-11187,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11187,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4188,"num_bytes_recovered":274464768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.926473"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":598,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.792422+0000","last_change":"2024-03-30T13:58:47.182999+0000","last_active":"2024-03-30T13:59:07.792422+0000","last_peered":"2024-03-30T13:59:07.792422+0000","last_clean":"2024-03-30T13:59:07.792422+0000","last_became_active":"2024-03-30T13:58:47.182681+0000","last_became_peered":"2024-03-30T13:58:47.182681+0000","last_unstale":"2024-03-30T13:59:07.792422+0000","last_undegraded":"2024-03-30T13:59:07.792422+0000","last_fullsized":"2024-03-30T13:59:07.792422+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:08.625338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:08.625338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":614,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.795901+0000","last_change":"2024-03-30T13:58:48.180029+0000","last_active":"2024-03-30T13:59:07.795901+0000","last_peered":"2024-03-30T13:59:07.795901+0000","last_clean":"2024-03-30T13:59:07.795901+0000","last_became_active":"2024-03-30T13:58:48.179562+0000","last_became_peered":"2024-03-30T13:58:48.179562+0000","last_unstale":"2024-03-30T13:59:07.795901+0000","last_undegraded":"2024-03-30T13:59:07.795901+0000","last_fullsized":"2024-03-30T13:59:07.795901+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:56.015905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:17.548567+0000","last_clean_scrub_stamp":"2024-03-30T13:57:56.015905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5,6],"avail_no_missing":["1","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"65'1","reported_seq":606,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.802771+0000","last_change":"2024-03-30T13:58:48.188322+0000","last_active":"2024-03-30T13:59:07.802771+0000","last_peered":"2024-03-30T13:59:07.802771+0000","last_clean":"2024-03-30T13:59:07.802771+0000","last_became_active":"2024-03-30T13:58:48.185898+0000","last_became_peered":"2024-03-30T13:58:48.185898+0000","last_unstale":"2024-03-30T13:59:07.802771+0000","last_undegraded":"2024-03-30T13:59:07.802771+0000","last_fullsized":"2024-03-30T13:59:07.802771+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:56:58.648903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:56:58.648903+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,6],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"216'15","reported_seq":613,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.794771+0000","last_change":"2024-03-30T13:52:56.028286+0000","last_active":"2024-03-30T13:59:07.794771+0000","last_peered":"2024-03-30T13:59:07.794771+0000","last_clean":"2024-03-30T13:59:07.794771+0000","last_became_active":"2024-03-30T13:42:47.319638+0000","last_became_peered":"2024-03-30T13:42:47.319638+0000","last_unstale":"2024-03-30T13:59:07.794771+0000","last_undegraded":"2024-03-30T13:59:07.794771+0000","last_fullsized":"2024-03-30T13:59:07.794771+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'15","last_scrub_stamp":"2024-03-30T13:52:56.028141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:56.028141+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":586,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.800979+0000","last_change":"2024-03-30T13:58:08.388962+0000","last_active":"2024-03-30T13:59:07.800979+0000","last_peered":"2024-03-30T13:59:07.800979+0000","last_clean":"2024-03-30T13:59:07.800979+0000","last_became_active":"2024-03-30T13:53:35.862085+0000","last_became_peered":"2024-03-30T13:53:35.862085+0000","last_unstale":"2024-03-30T13:59:07.800979+0000","last_undegraded":"2024-03-30T13:59:07.800979+0000","last_fullsized":"2024-03-30T13:59:07.800979+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:08.388764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:08.388764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:30.819838+0000","scrub_duration":168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":584,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.794577+0000","last_change":"2024-03-30T13:57:12.673777+0000","last_active":"2024-03-30T13:59:07.794577+0000","last_peered":"2024-03-30T13:59:07.794577+0000","last_clean":"2024-03-30T13:59:07.794577+0000","last_became_active":"2024-03-30T13:42:49.327124+0000","last_became_peered":"2024-03-30T13:42:49.327124+0000","last_unstale":"2024-03-30T13:59:07.794577+0000","last_undegraded":"2024-03-30T13:59:07.794577+0000","last_fullsized":"2024-03-30T13:59:07.794577+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:12.673593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:12.673593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":581,"reported_epoch":333,"state":"active+undersized","last_fresh":"2024-03-30T13:59:07.801304+0000","last_change":"2024-03-30T13:50:08.512316+0000","last_active":"2024-03-30T13:59:07.801304+0000","last_peered":"2024-03-30T13:59:07.801304+0000","last_clean":"2024-03-30T13:50:05.504688+0000","last_became_active":"2024-03-30T13:50:08.512316+0000","last_became_peered":"2024-03-30T13:50:08.512316+0000","last_unstale":"2024-03-30T13:59:07.801304+0000","last_undegraded":"2024-03-30T13:59:07.801304+0000","last_fullsized":"2024-03-30T13:50:08.501439+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:20.499768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:20.499768+0000","last_clean_scrub_stamp":"2024-03-30T13:49:20.499768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"217'34","reported_seq":617,"reported_epoch":333,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:59:07.803338+0000","last_change":"2024-03-30T13:50:08.511099+0000","last_active":"2024-03-30T13:59:07.803338+0000","last_peered":"2024-03-30T13:59:07.803338+0000","last_clean":"2024-03-30T13:50:05.499201+0000","last_became_active":"2024-03-30T13:50:08.511099+0000","last_became_peered":"2024-03-30T13:50:08.511099+0000","last_unstale":"2024-03-30T13:59:07.803338+0000","last_undegraded":"2024-03-30T13:50:08.502733+0000","last_fullsized":"2024-03-30T13:50:08.502606+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'22","last_scrub_stamp":"2024-03-30T13:46:49.339954+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:29.151791+0000","last_clean_scrub_stamp":"2024-03-30T13:46:49.339954+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":591,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.801896+0000","last_change":"2024-03-30T13:58:10.911460+0000","last_active":"2024-03-30T13:59:07.801896+0000","last_peered":"2024-03-30T13:59:07.801896+0000","last_clean":"2024-03-30T13:59:07.801896+0000","last_became_active":"2024-03-30T13:53:36.855710+0000","last_became_peered":"2024-03-30T13:53:36.855710+0000","last_unstale":"2024-03-30T13:59:07.801896+0000","last_undegraded":"2024-03-30T13:59:07.801896+0000","last_fullsized":"2024-03-30T13:59:07.801896+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:10.911245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:27.061281+0000","last_clean_scrub_stamp":"2024-03-30T13:58:10.911245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:33.994963+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.802643+0000","last_change":"2024-03-30T13:58:47.192572+0000","last_active":"2024-03-30T13:59:07.802643+0000","last_peered":"2024-03-30T13:59:07.802643+0000","last_clean":"2024-03-30T13:59:07.802643+0000","last_became_active":"2024-03-30T13:58:47.192207+0000","last_became_peered":"2024-03-30T13:58:47.192207+0000","last_unstale":"2024-03-30T13:59:07.802643+0000","last_undegraded":"2024-03-30T13:59:07.802643+0000","last_fullsized":"2024-03-30T13:59:07.802643+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:07.824904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:20.069864+0000","last_clean_scrub_stamp":"2024-03-30T13:58:07.824904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:17.308272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.803595+0000","last_change":"2024-03-30T13:58:47.199973+0000","last_active":"2024-03-30T13:59:07.803595+0000","last_peered":"2024-03-30T13:59:07.803595+0000","last_clean":"2024-03-30T13:59:07.803595+0000","last_became_active":"2024-03-30T13:58:47.199130+0000","last_became_peered":"2024-03-30T13:58:47.199130+0000","last_unstale":"2024-03-30T13:59:07.803595+0000","last_undegraded":"2024-03-30T13:59:07.803595+0000","last_fullsized":"2024-03-30T13:59:07.803595+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:07.129142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:37.513596+0000","last_clean_scrub_stamp":"2024-03-30T13:52:07.129142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"216'15","reported_seq":621,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.792577+0000","last_change":"2024-03-30T13:57:19.638146+0000","last_active":"2024-03-30T13:59:07.792577+0000","last_peered":"2024-03-30T13:59:07.792577+0000","last_clean":"2024-03-30T13:59:07.792577+0000","last_became_active":"2024-03-30T13:42:47.310289+0000","last_became_peered":"2024-03-30T13:42:47.310289+0000","last_unstale":"2024-03-30T13:59:07.792577+0000","last_undegraded":"2024-03-30T13:59:07.792577+0000","last_fullsized":"2024-03-30T13:59:07.792577+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'15","last_scrub_stamp":"2024-03-30T13:57:19.638024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:19.638024+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":374,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.804229+0000","last_change":"2024-03-30T13:58:48.202829+0000","last_active":"2024-03-30T13:59:07.804229+0000","last_peered":"2024-03-30T13:59:07.804229+0000","last_clean":"2024-03-30T13:59:07.804229+0000","last_became_active":"2024-03-30T13:58:48.202430+0000","last_became_peered":"2024-03-30T13:58:48.202430+0000","last_unstale":"2024-03-30T13:59:07.804229+0000","last_undegraded":"2024-03-30T13:59:07.804229+0000","last_fullsized":"2024-03-30T13:59:07.804229+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:57.194555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:56:57.194555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":221,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.793635+0000","last_change":"2024-03-30T13:53:35.862161+0000","last_active":"2024-03-30T13:59:07.793635+0000","last_peered":"2024-03-30T13:59:07.793635+0000","last_clean":"2024-03-30T13:59:07.793635+0000","last_became_active":"2024-03-30T13:53:35.861405+0000","last_became_peered":"2024-03-30T13:53:35.861405+0000","last_unstale":"2024-03-30T13:59:07.793635+0000","last_undegraded":"2024-03-30T13:59:07.793635+0000","last_fullsized":"2024-03-30T13:59:07.793635+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:50.953533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:41.633284+0000","last_clean_scrub_stamp":"2024-03-30T13:52:50.953533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"186'2","reported_seq":29,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.803044+0000","last_change":"2024-03-30T13:58:47.201377+0000","last_active":"2024-03-30T13:59:07.803044+0000","last_peered":"2024-03-30T13:59:07.803044+0000","last_clean":"2024-03-30T13:59:07.803044+0000","last_became_active":"2024-03-30T13:58:47.193793+0000","last_became_peered":"2024-03-30T13:58:47.193793+0000","last_unstale":"2024-03-30T13:59:07.803044+0000","last_undegraded":"2024-03-30T13:59:07.803044+0000","last_fullsized":"2024-03-30T13:59:07.803044+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-30T13:57:13.664762+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-30T13:51:15.932076+0000","last_clean_scrub_stamp":"2024-03-30T13:57:13.664762+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"216'9","reported_seq":611,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.791728+0000","last_change":"2024-03-30T13:52:19.931416+0000","last_active":"2024-03-30T13:59:07.791728+0000","last_peered":"2024-03-30T13:59:07.791728+0000","last_clean":"2024-03-30T13:59:07.791728+0000","last_became_active":"2024-03-30T13:42:47.317352+0000","last_became_peered":"2024-03-30T13:42:47.317352+0000","last_unstale":"2024-03-30T13:59:07.791728+0000","last_undegraded":"2024-03-30T13:59:07.791728+0000","last_fullsized":"2024-03-30T13:59:07.791728+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'7","last_scrub_stamp":"2024-03-30T13:52:19.931197+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-30T13:43:50.960980+0000","last_clean_scrub_stamp":"2024-03-30T13:52:19.931197+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":602,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.794759+0000","last_change":"2024-03-30T13:58:47.199570+0000","last_active":"2024-03-30T13:59:07.794759+0000","last_peered":"2024-03-30T13:59:07.794759+0000","last_clean":"2024-03-30T13:59:07.794759+0000","last_became_active":"2024-03-30T13:58:47.199001+0000","last_became_peered":"2024-03-30T13:58:47.199001+0000","last_unstale":"2024-03-30T13:59:07.794759+0000","last_undegraded":"2024-03-30T13:59:07.794759+0000","last_fullsized":"2024-03-30T13:59:07.794759+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:21.602164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:21.602164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:27.907037+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":607,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:08.299676+0000","last_change":"2024-03-30T13:59:08.299710+0000","last_active":"2024-03-30T13:59:08.299676+0000","last_peered":"2024-03-30T13:59:08.299676+0000","last_clean":"2024-03-30T13:59:08.299676+0000","last_became_active":"2024-03-30T13:42:49.322665+0000","last_became_peered":"2024-03-30T13:42:49.322665+0000","last_unstale":"2024-03-30T13:59:08.299676+0000","last_undegraded":"2024-03-30T13:59:08.299676+0000","last_fullsized":"2024-03-30T13:59:08.299676+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:59:08.299596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:30.465974+0000","last_clean_scrub_stamp":"2024-03-30T13:59:08.299596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:15.688749+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":604,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.803667+0000","last_change":"2024-03-30T13:57:56.945998+0000","last_active":"2024-03-30T13:59:07.803667+0000","last_peered":"2024-03-30T13:59:07.803667+0000","last_clean":"2024-03-30T13:59:07.803667+0000","last_became_active":"2024-03-30T13:53:35.861043+0000","last_became_peered":"2024-03-30T13:53:35.861043+0000","last_unstale":"2024-03-30T13:59:07.803667+0000","last_undegraded":"2024-03-30T13:59:07.803667+0000","last_fullsized":"2024-03-30T13:59:07.803667+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:56.945861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:43.088964+0000","last_clean_scrub_stamp":"2024-03-30T13:57:56.945861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:22.686748+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"216'27","reported_seq":417,"reported_epoch":333,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:59:07.796273+0000","last_change":"2024-03-30T13:50:08.508437+0000","last_active":"2024-03-30T13:59:07.796273+0000","last_peered":"2024-03-30T13:59:07.796273+0000","last_clean":"2024-03-30T13:50:05.490228+0000","last_became_active":"2024-03-30T13:50:08.508437+0000","last_became_peered":"2024-03-30T13:50:08.508437+0000","last_unstale":"2024-03-30T13:59:07.796273+0000","last_undegraded":"2024-03-30T13:50:08.506233+0000","last_fullsized":"2024-03-30T13:50:08.506116+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-30T13:49:08.415681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:49:08.415681+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:11.264089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":585,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.802573+0000","last_change":"2024-03-30T13:58:47.195100+0000","last_active":"2024-03-30T13:59:07.802573+0000","last_peered":"2024-03-30T13:59:07.802573+0000","last_clean":"2024-03-30T13:59:07.802573+0000","last_became_active":"2024-03-30T13:58:47.194643+0000","last_became_peered":"2024-03-30T13:58:47.194643+0000","last_unstale":"2024-03-30T13:59:07.802573+0000","last_undegraded":"2024-03-30T13:59:07.802573+0000","last_fullsized":"2024-03-30T13:59:07.802573+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:52.054302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:52.054302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":571,"reported_epoch":333,"state":"active+undersized","last_fresh":"2024-03-30T13:59:07.795211+0000","last_change":"2024-03-30T13:50:08.506226+0000","last_active":"2024-03-30T13:59:07.795211+0000","last_peered":"2024-03-30T13:59:07.795211+0000","last_clean":"2024-03-30T13:50:05.490914+0000","last_became_active":"2024-03-30T13:50:08.506226+0000","last_became_peered":"2024-03-30T13:50:08.506226+0000","last_unstale":"2024-03-30T13:59:07.795211+0000","last_undegraded":"2024-03-30T13:59:07.795211+0000","last_fullsized":"2024-03-30T13:50:08.502611+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:25.104610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:48:25.104610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":579,"reported_epoch":333,"state":"active+undersized","last_fresh":"2024-03-30T13:59:07.792737+0000","last_change":"2024-03-30T13:50:08.508556+0000","last_active":"2024-03-30T13:59:07.792737+0000","last_peered":"2024-03-30T13:59:07.792737+0000","last_clean":"2024-03-30T13:50:07.261781+0000","last_became_active":"2024-03-30T13:50:08.508556+0000","last_became_peered":"2024-03-30T13:50:08.508556+0000","last_unstale":"2024-03-30T13:59:07.792737+0000","last_undegraded":"2024-03-30T13:59:07.792737+0000","last_fullsized":"2024-03-30T13:50:08.502793+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:07.261595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:50:07.261595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"217'23","reported_seq":625,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.802246+0000","last_change":"2024-03-30T13:52:50.945958+0000","last_active":"2024-03-30T13:59:07.802246+0000","last_peered":"2024-03-30T13:59:07.802246+0000","last_clean":"2024-03-30T13:59:07.802246+0000","last_became_active":"2024-03-30T13:42:47.317881+0000","last_became_peered":"2024-03-30T13:42:47.317881+0000","last_unstale":"2024-03-30T13:59:07.802246+0000","last_undegraded":"2024-03-30T13:59:07.802246+0000","last_fullsized":"2024-03-30T13:59:07.802246+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'17","last_scrub_stamp":"2024-03-30T13:52:50.945844+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-03-30T13:43:52.044427+0000","last_clean_scrub_stamp":"2024-03-30T13:52:50.945844+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":27,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.802951+0000","last_change":"2024-03-30T13:58:48.204829+0000","last_active":"2024-03-30T13:59:07.802951+0000","last_peered":"2024-03-30T13:59:07.802951+0000","last_clean":"2024-03-30T13:59:07.802951+0000","last_became_active":"2024-03-30T13:58:48.191902+0000","last_became_peered":"2024-03-30T13:58:48.191902+0000","last_unstale":"2024-03-30T13:59:07.802951+0000","last_undegraded":"2024-03-30T13:59:07.802951+0000","last_fullsized":"2024-03-30T13:59:07.802951+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T13:56:53.918975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:56:53.918975+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"230'10","reported_seq":812,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.801894+0000","last_change":"2024-03-30T13:57:07.246065+0000","last_active":"2024-03-30T13:59:07.801894+0000","last_peered":"2024-03-30T13:59:07.801894+0000","last_clean":"2024-03-30T13:59:07.801894+0000","last_became_active":"2024-03-30T13:53:36.852879+0000","last_became_peered":"2024-03-30T13:53:36.852879+0000","last_unstale":"2024-03-30T13:59:07.801894+0000","last_undegraded":"2024-03-30T13:59:07.801894+0000","last_fullsized":"2024-03-30T13:59:07.801894+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'10","last_scrub_stamp":"2024-03-30T13:57:07.245825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:07.245825+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,7],"avail_no_missing":["6","4","7"],"object_location_counts":[{"shards":"4,6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"65'1","reported_seq":608,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.795700+0000","last_change":"2024-03-30T13:58:00.032332+0000","last_active":"2024-03-30T13:59:07.795700+0000","last_peered":"2024-03-30T13:59:07.795700+0000","last_clean":"2024-03-30T13:59:07.795700+0000","last_became_active":"2024-03-30T13:42:45.307832+0000","last_became_peered":"2024-03-30T13:42:45.307832+0000","last_unstale":"2024-03-30T13:59:07.795700+0000","last_undegraded":"2024-03-30T13:59:07.795700+0000","last_fullsized":"2024-03-30T13:59:07.795700+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:58:00.031186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:00.031186+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:25.305529+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"217'25","reported_seq":649,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.804534+0000","last_change":"2024-03-30T13:53:35.974113+0000","last_active":"2024-03-30T13:59:07.804534+0000","last_peered":"2024-03-30T13:59:07.804534+0000","last_clean":"2024-03-30T13:59:07.804534+0000","last_became_active":"2024-03-30T13:53:35.858156+0000","last_became_peered":"2024-03-30T13:53:35.858156+0000","last_unstale":"2024-03-30T13:59:07.804534+0000","last_undegraded":"2024-03-30T13:59:07.804534+0000","last_fullsized":"2024-03-30T13:59:07.804534+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'25","last_scrub_stamp":"2024-03-30T13:52:54.925758+0000","last_deep_scrub":"153'15","last_deep_scrub_stamp":"2024-03-30T13:51:22.252831+0000","last_clean_scrub_stamp":"2024-03-30T13:52:54.925758+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":605,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.797605+0000","last_change":"2024-03-30T13:58:48.181184+0000","last_active":"2024-03-30T13:59:07.797605+0000","last_peered":"2024-03-30T13:59:07.797605+0000","last_clean":"2024-03-30T13:59:07.797605+0000","last_became_active":"2024-03-30T13:58:48.180840+0000","last_became_peered":"2024-03-30T13:58:48.180840+0000","last_unstale":"2024-03-30T13:59:07.797605+0000","last_undegraded":"2024-03-30T13:59:07.797605+0000","last_fullsized":"2024-03-30T13:59:07.797605+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:13.029471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:13.029471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:35.500156+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":592,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.805278+0000","last_change":"2024-03-30T13:57:04.801994+0000","last_active":"2024-03-30T13:59:07.805278+0000","last_peered":"2024-03-30T13:59:07.805278+0000","last_clean":"2024-03-30T13:59:07.805278+0000","last_became_active":"2024-03-30T13:53:36.857751+0000","last_became_peered":"2024-03-30T13:53:36.857751+0000","last_unstale":"2024-03-30T13:59:07.805278+0000","last_undegraded":"2024-03-30T13:59:07.805278+0000","last_fullsized":"2024-03-30T13:59:07.805278+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:04.801834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:04.801834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":592,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.797684+0000","last_change":"2024-03-30T13:58:14.227217+0000","last_active":"2024-03-30T13:59:07.797684+0000","last_peered":"2024-03-30T13:59:07.797684+0000","last_clean":"2024-03-30T13:59:07.797684+0000","last_became_active":"2024-03-30T13:42:45.310165+0000","last_became_peered":"2024-03-30T13:42:45.310165+0000","last_unstale":"2024-03-30T13:59:07.797684+0000","last_undegraded":"2024-03-30T13:59:07.797684+0000","last_fullsized":"2024-03-30T13:59:07.797684+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:14.224328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:14.224328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:35.989219+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"217'19","reported_seq":669,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.802078+0000","last_change":"2024-03-30T13:58:47.224883+0000","last_active":"2024-03-30T13:59:07.802078+0000","last_peered":"2024-03-30T13:59:07.802078+0000","last_clean":"2024-03-30T13:59:07.802078+0000","last_became_active":"2024-03-30T13:58:47.204696+0000","last_became_peered":"2024-03-30T13:58:47.204696+0000","last_unstale":"2024-03-30T13:59:07.802078+0000","last_undegraded":"2024-03-30T13:59:07.802078+0000","last_fullsized":"2024-03-30T13:59:07.802078+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'19","last_scrub_stamp":"2024-03-30T13:57:11.200126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:11.200126+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":454,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.791479+0000","last_change":"2024-03-30T13:57:03.611202+0000","last_active":"2024-03-30T13:59:07.791479+0000","last_peered":"2024-03-30T13:59:07.791479+0000","last_clean":"2024-03-30T13:59:07.791479+0000","last_became_active":"2024-03-30T13:53:36.863938+0000","last_became_peered":"2024-03-30T13:53:36.863938+0000","last_unstale":"2024-03-30T13:59:07.791479+0000","last_undegraded":"2024-03-30T13:59:07.791479+0000","last_fullsized":"2024-03-30T13:59:07.791479+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:03.611042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:03.611042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3,7],"avail_no_missing":["4","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":612,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.799440+0000","last_change":"2024-03-30T13:56:57.393571+0000","last_active":"2024-03-30T13:59:07.799440+0000","last_peered":"2024-03-30T13:59:07.799440+0000","last_clean":"2024-03-30T13:59:07.799440+0000","last_became_active":"2024-03-30T13:53:35.862467+0000","last_became_peered":"2024-03-30T13:53:35.862467+0000","last_unstale":"2024-03-30T13:59:07.799440+0000","last_undegraded":"2024-03-30T13:59:07.799440+0000","last_fullsized":"2024-03-30T13:59:07.799440+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:57.393495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:54.893653+0000","last_clean_scrub_stamp":"2024-03-30T13:56:57.393495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":618,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.803322+0000","last_change":"2024-03-30T13:58:47.200252+0000","last_active":"2024-03-30T13:59:07.803322+0000","last_peered":"2024-03-30T13:59:07.803322+0000","last_clean":"2024-03-30T13:59:07.803322+0000","last_became_active":"2024-03-30T13:58:47.199798+0000","last_became_peered":"2024-03-30T13:58:47.199798+0000","last_unstale":"2024-03-30T13:59:07.803322+0000","last_undegraded":"2024-03-30T13:59:07.803322+0000","last_fullsized":"2024-03-30T13:59:07.803322+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:16.924797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:16.924797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:19.423352+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"216'8","reported_seq":601,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.797818+0000","last_change":"2024-03-30T13:57:07.952284+0000","last_active":"2024-03-30T13:59:07.797818+0000","last_peered":"2024-03-30T13:59:07.797818+0000","last_clean":"2024-03-30T13:59:07.797818+0000","last_became_active":"2024-03-30T13:42:47.321428+0000","last_became_peered":"2024-03-30T13:42:47.321428+0000","last_unstale":"2024-03-30T13:59:07.797818+0000","last_undegraded":"2024-03-30T13:59:07.797818+0000","last_fullsized":"2024-03-30T13:59:07.797818+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'8","last_scrub_stamp":"2024-03-30T13:57:07.952145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:07.952145+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":573,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.801173+0000","last_change":"2024-03-30T13:52:26.149264+0000","last_active":"2024-03-30T13:59:07.801173+0000","last_peered":"2024-03-30T13:59:07.801173+0000","last_clean":"2024-03-30T13:59:07.801173+0000","last_became_active":"2024-03-30T13:42:51.374304+0000","last_became_peered":"2024-03-30T13:42:51.374304+0000","last_unstale":"2024-03-30T13:59:07.801173+0000","last_undegraded":"2024-03-30T13:59:07.801173+0000","last_fullsized":"2024-03-30T13:59:07.801173+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:26.149132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:52:26.149132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":221,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.794428+0000","last_change":"2024-03-30T13:53:35.854347+0000","last_active":"2024-03-30T13:59:07.794428+0000","last_peered":"2024-03-30T13:59:07.794428+0000","last_clean":"2024-03-30T13:59:07.794428+0000","last_became_active":"2024-03-30T13:53:35.854002+0000","last_became_peered":"2024-03-30T13:53:35.854002+0000","last_unstale":"2024-03-30T13:59:07.794428+0000","last_undegraded":"2024-03-30T13:59:07.794428+0000","last_fullsized":"2024-03-30T13:59:07.794428+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:56.878297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:42.616155+0000","last_clean_scrub_stamp":"2024-03-30T13:52:56.878297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":23,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.802707+0000","last_change":"2024-03-30T13:58:48.187642+0000","last_active":"2024-03-30T13:59:07.802707+0000","last_peered":"2024-03-30T13:59:07.802707+0000","last_clean":"2024-03-30T13:59:07.802707+0000","last_became_active":"2024-03-30T13:58:48.187064+0000","last_became_peered":"2024-03-30T13:58:48.187064+0000","last_unstale":"2024-03-30T13:59:07.802707+0000","last_undegraded":"2024-03-30T13:59:07.802707+0000","last_fullsized":"2024-03-30T13:59:07.802707+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:07.098782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:25.494903+0000","last_clean_scrub_stamp":"2024-03-30T13:57:07.098782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"217'19","reported_seq":656,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.803281+0000","last_change":"2024-03-30T13:58:47.213397+0000","last_active":"2024-03-30T13:59:07.803281+0000","last_peered":"2024-03-30T13:59:07.803281+0000","last_clean":"2024-03-30T13:59:07.803281+0000","last_became_active":"2024-03-30T13:58:47.193590+0000","last_became_peered":"2024-03-30T13:58:47.193590+0000","last_unstale":"2024-03-30T13:59:07.803281+0000","last_undegraded":"2024-03-30T13:59:07.803281+0000","last_fullsized":"2024-03-30T13:59:07.803281+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'19","last_scrub_stamp":"2024-03-30T13:52:57.970377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:52:57.970377+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":598,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.800809+0000","last_change":"2024-03-30T13:58:47.198718+0000","last_active":"2024-03-30T13:59:07.800809+0000","last_peered":"2024-03-30T13:59:07.800809+0000","last_clean":"2024-03-30T13:59:07.800809+0000","last_became_active":"2024-03-30T13:58:47.198442+0000","last_became_peered":"2024-03-30T13:58:47.198442+0000","last_unstale":"2024-03-30T13:59:07.800809+0000","last_undegraded":"2024-03-30T13:59:07.800809+0000","last_fullsized":"2024-03-30T13:59:07.800809+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:13.749366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:28.314391+0000","last_clean_scrub_stamp":"2024-03-30T13:57:13.749366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":576,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.800883+0000","last_change":"2024-03-30T13:52:16.009739+0000","last_active":"2024-03-30T13:59:07.800883+0000","last_peered":"2024-03-30T13:59:07.800883+0000","last_clean":"2024-03-30T13:59:07.800883+0000","last_became_active":"2024-03-30T13:42:49.325219+0000","last_became_peered":"2024-03-30T13:42:49.325219+0000","last_unstale":"2024-03-30T13:59:07.800883+0000","last_undegraded":"2024-03-30T13:59:07.800883+0000","last_fullsized":"2024-03-30T13:59:07.800883+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:16.009499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:52:16.009499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":230,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.802069+0000","last_change":"2024-03-30T13:57:19.071948+0000","last_active":"2024-03-30T13:59:07.802069+0000","last_peered":"2024-03-30T13:59:07.802069+0000","last_clean":"2024-03-30T13:59:07.802069+0000","last_became_active":"2024-03-30T13:53:35.859253+0000","last_became_peered":"2024-03-30T13:53:35.859253+0000","last_unstale":"2024-03-30T13:59:07.802069+0000","last_undegraded":"2024-03-30T13:59:07.802069+0000","last_fullsized":"2024-03-30T13:59:07.802069+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:19.071814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:11.701294+0000","last_clean_scrub_stamp":"2024-03-30T13:57:19.071814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"216'15","reported_seq":605,"reported_epoch":333,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:59:07.799144+0000","last_change":"2024-03-30T13:50:08.513156+0000","last_active":"2024-03-30T13:59:07.799144+0000","last_peered":"2024-03-30T13:59:07.799144+0000","last_clean":"2024-03-30T13:50:05.496678+0000","last_became_active":"2024-03-30T13:50:08.513156+0000","last_became_peered":"2024-03-30T13:50:08.513156+0000","last_unstale":"2024-03-30T13:59:07.799144+0000","last_undegraded":"2024-03-30T13:50:08.507726+0000","last_fullsized":"2024-03-30T13:50:08.507616+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:49:26.480540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:49:26.480540+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:11.264100+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":591,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.801837+0000","last_change":"2024-03-30T13:58:47.194818+0000","last_active":"2024-03-30T13:59:07.801837+0000","last_peered":"2024-03-30T13:59:07.801837+0000","last_clean":"2024-03-30T13:59:07.801837+0000","last_became_active":"2024-03-30T13:58:47.194434+0000","last_became_peered":"2024-03-30T13:58:47.194434+0000","last_unstale":"2024-03-30T13:59:07.801837+0000","last_undegraded":"2024-03-30T13:59:07.801837+0000","last_fullsized":"2024-03-30T13:59:07.801837+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:17.058094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:24.598976+0000","last_clean_scrub_stamp":"2024-03-30T13:57:17.058094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":615,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.798958+0000","last_change":"2024-03-30T13:58:47.186077+0000","last_active":"2024-03-30T13:59:07.798958+0000","last_peered":"2024-03-30T13:59:07.798958+0000","last_clean":"2024-03-30T13:59:07.798958+0000","last_became_active":"2024-03-30T13:58:47.185594+0000","last_became_peered":"2024-03-30T13:58:47.185594+0000","last_unstale":"2024-03-30T13:59:07.798958+0000","last_undegraded":"2024-03-30T13:59:07.798958+0000","last_fullsized":"2024-03-30T13:59:07.798958+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:02.985176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:34.815783+0000","last_clean_scrub_stamp":"2024-03-30T13:58:02.985176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:25.589455+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":606,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.795104+0000","last_change":"2024-03-30T13:58:52.482066+0000","last_active":"2024-03-30T13:59:07.795104+0000","last_peered":"2024-03-30T13:59:07.795104+0000","last_clean":"2024-03-30T13:59:07.795104+0000","last_became_active":"2024-03-30T13:53:36.848656+0000","last_became_peered":"2024-03-30T13:53:36.848656+0000","last_unstale":"2024-03-30T13:59:07.795104+0000","last_undegraded":"2024-03-30T13:59:07.795104+0000","last_fullsized":"2024-03-30T13:59:07.795104+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:52.481860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:52.481860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:08.491959+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,7],"avail_no_missing":["4","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"216'19","reported_seq":625,"reported_epoch":333,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:59:07.792432+0000","last_change":"2024-03-30T13:50:08.513382+0000","last_active":"2024-03-30T13:59:07.792432+0000","last_peered":"2024-03-30T13:59:07.792432+0000","last_clean":"2024-03-30T13:50:05.491029+0000","last_became_active":"2024-03-30T13:50:08.513382+0000","last_became_peered":"2024-03-30T13:50:08.513382+0000","last_unstale":"2024-03-30T13:59:07.792432+0000","last_undegraded":"2024-03-30T13:50:08.503759+0000","last_fullsized":"2024-03-30T13:50:08.503643+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-30T13:48:22.068206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:22.068206+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":592,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.803619+0000","last_change":"2024-03-30T13:53:35.856379+0000","last_active":"2024-03-30T13:59:07.803619+0000","last_peered":"2024-03-30T13:59:07.803619+0000","last_clean":"2024-03-30T13:59:07.803619+0000","last_became_active":"2024-03-30T13:53:35.855996+0000","last_became_peered":"2024-03-30T13:53:35.855996+0000","last_unstale":"2024-03-30T13:59:07.803619+0000","last_undegraded":"2024-03-30T13:59:07.803619+0000","last_fullsized":"2024-03-30T13:59:07.803619+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:56.942611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:56.942611+0000","last_clean_scrub_stamp":"2024-03-30T13:52:56.942611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"326'10","reported_seq":34,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.803678+0000","last_change":"2024-03-30T13:58:47.205173+0000","last_active":"2024-03-30T13:59:07.803678+0000","last_peered":"2024-03-30T13:59:07.803678+0000","last_clean":"2024-03-30T13:59:07.803678+0000","last_became_active":"2024-03-30T13:58:47.194143+0000","last_became_peered":"2024-03-30T13:58:47.194143+0000","last_unstale":"2024-03-30T13:59:07.803678+0000","last_undegraded":"2024-03-30T13:59:07.803678+0000","last_fullsized":"2024-03-30T13:59:07.803678+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'9","last_scrub_stamp":"2024-03-30T13:57:57.056166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:57.056166+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":466,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.791831+0000","last_change":"2024-03-30T13:57:04.646873+0000","last_active":"2024-03-30T13:59:07.791831+0000","last_peered":"2024-03-30T13:59:07.791831+0000","last_clean":"2024-03-30T13:59:07.791831+0000","last_became_active":"2024-03-30T13:53:36.849971+0000","last_became_peered":"2024-03-30T13:53:36.849971+0000","last_unstale":"2024-03-30T13:59:07.791831+0000","last_undegraded":"2024-03-30T13:59:07.791831+0000","last_fullsized":"2024-03-30T13:59:07.791831+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:04.646741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:57:04.646741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,7],"avail_no_missing":["4","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"217'20","reported_seq":640,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.805677+0000","last_change":"2024-03-30T13:56:58.941975+0000","last_active":"2024-03-30T13:59:07.805677+0000","last_peered":"2024-03-30T13:59:07.805677+0000","last_clean":"2024-03-30T13:59:07.805677+0000","last_became_active":"2024-03-30T13:53:37.110194+0000","last_became_peered":"2024-03-30T13:53:37.110194+0000","last_unstale":"2024-03-30T13:59:07.805677+0000","last_undegraded":"2024-03-30T13:59:07.805677+0000","last_fullsized":"2024-03-30T13:59:07.805677+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'20","last_scrub_stamp":"2024-03-30T13:56:58.941755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:56:58.941755+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":585,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.791845+0000","last_change":"2024-03-30T13:58:48.187038+0000","last_active":"2024-03-30T13:59:07.791845+0000","last_peered":"2024-03-30T13:59:07.791845+0000","last_clean":"2024-03-30T13:59:07.791845+0000","last_became_active":"2024-03-30T13:58:48.186613+0000","last_became_peered":"2024-03-30T13:58:48.186613+0000","last_unstale":"2024-03-30T13:59:07.791845+0000","last_undegraded":"2024-03-30T13:59:07.791845+0000","last_fullsized":"2024-03-30T13:59:07.791845+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:06.727774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:06.727774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,0],"avail_no_missing":["4","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"230'11","reported_seq":795,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.805950+0000","last_change":"2024-03-30T13:57:01.991621+0000","last_active":"2024-03-30T13:59:07.805950+0000","last_peered":"2024-03-30T13:59:07.805950+0000","last_clean":"2024-03-30T13:59:07.805950+0000","last_became_active":"2024-03-30T13:53:37.127826+0000","last_became_peered":"2024-03-30T13:53:37.127826+0000","last_unstale":"2024-03-30T13:59:07.805950+0000","last_undegraded":"2024-03-30T13:59:07.805950+0000","last_fullsized":"2024-03-30T13:59:07.805950+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'11","last_scrub_stamp":"2024-03-30T13:57:01.991476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:01.991476+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":605,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.802572+0000","last_change":"2024-03-30T13:58:03.861834+0000","last_active":"2024-03-30T13:59:07.802572+0000","last_peered":"2024-03-30T13:59:07.802572+0000","last_clean":"2024-03-30T13:59:07.802572+0000","last_became_active":"2024-03-30T13:53:35.857456+0000","last_became_peered":"2024-03-30T13:53:35.857456+0000","last_unstale":"2024-03-30T13:59:07.802572+0000","last_undegraded":"2024-03-30T13:59:07.802572+0000","last_fullsized":"2024-03-30T13:59:07.802572+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:03.861718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:28.734080+0000","last_clean_scrub_stamp":"2024-03-30T13:58:03.861718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:15.297097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"217'15","reported_seq":35,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.803432+0000","last_change":"2024-03-30T13:58:47.214215+0000","last_active":"2024-03-30T13:59:07.803432+0000","last_peered":"2024-03-30T13:59:07.803432+0000","last_clean":"2024-03-30T13:59:07.803432+0000","last_became_active":"2024-03-30T13:58:47.198411+0000","last_became_peered":"2024-03-30T13:58:47.198411+0000","last_unstale":"2024-03-30T13:59:07.803432+0000","last_undegraded":"2024-03-30T13:59:07.803432+0000","last_fullsized":"2024-03-30T13:59:07.803432+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'15","last_scrub_stamp":"2024-03-30T13:57:12.039992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:12.039992+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":368,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.795234+0000","last_change":"2024-03-30T13:57:01.689045+0000","last_active":"2024-03-30T13:59:07.795234+0000","last_peered":"2024-03-30T13:59:07.795234+0000","last_clean":"2024-03-30T13:59:07.795234+0000","last_became_active":"2024-03-30T13:53:36.857026+0000","last_became_peered":"2024-03-30T13:53:36.857026+0000","last_unstale":"2024-03-30T13:59:07.795234+0000","last_undegraded":"2024-03-30T13:59:07.795234+0000","last_fullsized":"2024-03-30T13:59:07.795234+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:01.688893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:01.688893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,7],"avail_no_missing":["4","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":590,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.802935+0000","last_change":"2024-03-30T13:58:03.191278+0000","last_active":"2024-03-30T13:59:07.802935+0000","last_peered":"2024-03-30T13:59:07.802935+0000","last_clean":"2024-03-30T13:59:07.802935+0000","last_became_active":"2024-03-30T13:53:35.875783+0000","last_became_peered":"2024-03-30T13:53:35.875783+0000","last_unstale":"2024-03-30T13:59:07.802935+0000","last_undegraded":"2024-03-30T13:59:07.802935+0000","last_fullsized":"2024-03-30T13:59:07.802935+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:03.191052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:58:03.191052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:21.859148+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":229,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.793543+0000","last_change":"2024-03-30T13:58:16.531285+0000","last_active":"2024-03-30T13:59:07.793543+0000","last_peered":"2024-03-30T13:59:07.793543+0000","last_clean":"2024-03-30T13:59:07.793543+0000","last_became_active":"2024-03-30T13:53:35.858261+0000","last_became_peered":"2024-03-30T13:53:35.858261+0000","last_unstale":"2024-03-30T13:59:07.793543+0000","last_undegraded":"2024-03-30T13:59:07.793543+0000","last_fullsized":"2024-03-30T13:59:07.793543+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:16.530762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:16.530762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:27.572482+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"218'16","reported_seq":625,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.804654+0000","last_change":"2024-03-30T13:53:36.861902+0000","last_active":"2024-03-30T13:59:07.804654+0000","last_peered":"2024-03-30T13:59:07.804654+0000","last_clean":"2024-03-30T13:59:07.804654+0000","last_became_active":"2024-03-30T13:53:36.861455+0000","last_became_peered":"2024-03-30T13:53:36.861455+0000","last_unstale":"2024-03-30T13:59:07.804654+0000","last_undegraded":"2024-03-30T13:59:07.804654+0000","last_fullsized":"2024-03-30T13:59:07.804654+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'16","last_scrub_stamp":"2024-03-30T13:52:59.883027+0000","last_deep_scrub":"218'16","last_deep_scrub_stamp":"2024-03-30T13:52:59.883027+0000","last_clean_scrub_stamp":"2024-03-30T13:52:59.883027+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3,7],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"72'1","reported_seq":255,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.800849+0000","last_change":"2024-03-30T13:58:47.219345+0000","last_active":"2024-03-30T13:59:07.800849+0000","last_peered":"2024-03-30T13:59:07.800849+0000","last_clean":"2024-03-30T13:59:07.800849+0000","last_became_active":"2024-03-30T13:58:47.198798+0000","last_became_peered":"2024-03-30T13:58:47.198798+0000","last_unstale":"2024-03-30T13:59:07.800849+0000","last_undegraded":"2024-03-30T13:59:07.800849+0000","last_fullsized":"2024-03-30T13:59:07.800849+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T13:57:05.320870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:05.320870+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.804465+0000","last_change":"2024-03-30T13:58:47.193833+0000","last_active":"2024-03-30T13:59:07.804465+0000","last_peered":"2024-03-30T13:59:07.804465+0000","last_clean":"2024-03-30T13:59:07.804465+0000","last_became_active":"2024-03-30T13:58:47.193472+0000","last_became_peered":"2024-03-30T13:58:47.193472+0000","last_unstale":"2024-03-30T13:59:07.804465+0000","last_undegraded":"2024-03-30T13:59:07.804465+0000","last_fullsized":"2024-03-30T13:59:07.804465+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:09.967517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:57:09.967517+0000","last_clean_scrub_stamp":"2024-03-30T13:57:09.967517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":626,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.798695+0000","last_change":"2024-03-30T13:58:47.205520+0000","last_active":"2024-03-30T13:59:07.798695+0000","last_peered":"2024-03-30T13:59:07.798695+0000","last_clean":"2024-03-30T13:59:07.798695+0000","last_became_active":"2024-03-30T13:58:47.203903+0000","last_became_peered":"2024-03-30T13:58:47.203903+0000","last_unstale":"2024-03-30T13:59:07.798695+0000","last_undegraded":"2024-03-30T13:59:07.798695+0000","last_fullsized":"2024-03-30T13:59:07.798695+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:17.132009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:51:20.175438+0000","last_clean_scrub_stamp":"2024-03-30T13:57:17.132009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"316'15","reported_seq":614,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.802906+0000","last_change":"2024-03-30T13:58:13.210664+0000","last_active":"2024-03-30T13:59:07.802906+0000","last_peered":"2024-03-30T13:59:07.802906+0000","last_clean":"2024-03-30T13:59:07.802906+0000","last_became_active":"2024-03-30T13:53:37.098834+0000","last_became_peered":"2024-03-30T13:53:37.098834+0000","last_unstale":"2024-03-30T13:59:07.802906+0000","last_undegraded":"2024-03-30T13:59:07.802906+0000","last_fullsized":"2024-03-30T13:59:07.802906+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'15","last_scrub_stamp":"2024-03-30T13:58:13.210392+0000","last_deep_scrub":"316'15","last_deep_scrub_stamp":"2024-03-30T13:58:13.210392+0000","last_clean_scrub_stamp":"2024-03-30T13:58:13.210392+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:34.819233+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"333'12812","reported_seq":12923,"reported_epoch":333,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T13:59:10.675023+0000","last_change":"2024-03-30T13:58:48.330795+0000","last_active":"2024-03-30T13:59:10.675023+0000","last_peered":"2024-03-30T13:59:10.675023+0000","last_clean":"2024-03-30T13:58:46.191255+0000","last_became_active":"2024-03-30T13:58:48.292518+0000","last_became_peered":"2024-03-30T13:58:48.292518+0000","last_unstale":"2024-03-30T13:59:10.675023+0000","last_undegraded":"2024-03-30T13:59:10.675023+0000","last_fullsized":"2024-03-30T13:59:10.675023+0000","mapping_epoch":326,"log_start":"318'8700","ondisk_log_start":"318'8700","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4112,"log_dups_size":0,"ondisk_log_size":4112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":496631808,"num_objects":7578,"num_object_clones":0,"num_object_copies":22734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7578,"num_objects_unfound":0,"num_objects_dirty":7578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12812,"num_write_kb":652480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[0,4,1],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":7578}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":366,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.803140+0000","last_change":"2024-03-30T13:58:48.234962+0000","last_active":"2024-03-30T13:59:07.803140+0000","last_peered":"2024-03-30T13:59:07.803140+0000","last_clean":"2024-03-30T13:59:07.803140+0000","last_became_active":"2024-03-30T13:58:48.234649+0000","last_became_peered":"2024-03-30T13:58:48.234649+0000","last_unstale":"2024-03-30T13:59:07.803140+0000","last_undegraded":"2024-03-30T13:59:07.803140+0000","last_fullsized":"2024-03-30T13:59:07.803140+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:56.190076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:56.190076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":236,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.791521+0000","last_change":"2024-03-30T13:53:36.015472+0000","last_active":"2024-03-30T13:59:07.791521+0000","last_peered":"2024-03-30T13:59:07.791521+0000","last_clean":"2024-03-30T13:59:07.791521+0000","last_became_active":"2024-03-30T13:53:35.874521+0000","last_became_peered":"2024-03-30T13:53:35.874521+0000","last_unstale":"2024-03-30T13:59:07.791521+0000","last_undegraded":"2024-03-30T13:59:07.791521+0000","last_fullsized":"2024-03-30T13:59:07.791521+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-30T13:52:09.680567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:09.680567+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"216'27","reported_seq":41,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.803230+0000","last_change":"2024-03-30T13:58:47.222352+0000","last_active":"2024-03-30T13:59:07.803230+0000","last_peered":"2024-03-30T13:59:07.803230+0000","last_clean":"2024-03-30T13:59:07.803230+0000","last_became_active":"2024-03-30T13:58:47.197778+0000","last_became_peered":"2024-03-30T13:58:47.197778+0000","last_unstale":"2024-03-30T13:59:07.803230+0000","last_undegraded":"2024-03-30T13:59:07.803230+0000","last_fullsized":"2024-03-30T13:59:07.803230+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'27","last_scrub_stamp":"2024-03-30T13:58:08.152048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:58:08.152048+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:36.136996+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"229'9","reported_seq":333,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.799246+0000","last_change":"2024-03-30T13:57:55.147011+0000","last_active":"2024-03-30T13:59:07.799246+0000","last_peered":"2024-03-30T13:59:07.799246+0000","last_clean":"2024-03-30T13:59:07.799246+0000","last_became_active":"2024-03-30T13:53:35.862126+0000","last_became_peered":"2024-03-30T13:53:35.862126+0000","last_unstale":"2024-03-30T13:59:07.799246+0000","last_undegraded":"2024-03-30T13:59:07.799246+0000","last_fullsized":"2024-03-30T13:59:07.799246+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'9","last_scrub_stamp":"2024-03-30T13:57:55.146893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:55.146893+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:17.257278+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"333'12856","reported_seq":12957,"reported_epoch":333,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-03-30T13:59:08.802712+0000","last_change":"2024-03-30T13:58:48.335986+0000","last_active":"2024-03-30T13:59:08.802712+0000","last_peered":"2024-03-30T13:59:08.802712+0000","last_clean":"2024-03-30T13:56:54.930191+0000","last_became_active":"2024-03-30T13:58:48.296956+0000","last_became_peered":"2024-03-30T13:58:48.296956+0000","last_unstale":"2024-03-30T13:59:08.802712+0000","last_undegraded":"2024-03-30T13:58:48.199227+0000","last_fullsized":"2024-03-30T13:58:48.178788+0000","mapping_epoch":326,"log_start":"317'8300","ondisk_log_start":"317'8300","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":507379712,"num_objects":7742,"num_object_clones":0,"num_object_copies":23226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12856,"num_write_kb":659136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":7742}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":587,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.801075+0000","last_change":"2024-03-30T13:58:12.361908+0000","last_active":"2024-03-30T13:59:07.801075+0000","last_peered":"2024-03-30T13:59:07.801075+0000","last_clean":"2024-03-30T13:59:07.801075+0000","last_became_active":"2024-03-30T13:53:35.862056+0000","last_became_peered":"2024-03-30T13:53:35.862056+0000","last_unstale":"2024-03-30T13:59:07.801075+0000","last_undegraded":"2024-03-30T13:59:07.801075+0000","last_fullsized":"2024-03-30T13:59:07.801075+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:12.361782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:12.361782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:14.785915+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":229,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.794670+0000","last_change":"2024-03-30T13:58:13.511206+0000","last_active":"2024-03-30T13:59:07.794670+0000","last_peered":"2024-03-30T13:59:07.794670+0000","last_clean":"2024-03-30T13:59:07.794670+0000","last_became_active":"2024-03-30T13:53:35.861370+0000","last_became_peered":"2024-03-30T13:53:35.861370+0000","last_unstale":"2024-03-30T13:59:07.794670+0000","last_undegraded":"2024-03-30T13:59:07.794670+0000","last_fullsized":"2024-03-30T13:59:07.794670+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:13.511094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:13.511094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:24.132961+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"216'18","reported_seq":629,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.793640+0000","last_change":"2024-03-30T13:58:54.451235+0000","last_active":"2024-03-30T13:59:07.793640+0000","last_peered":"2024-03-30T13:59:07.793640+0000","last_clean":"2024-03-30T13:59:07.793640+0000","last_became_active":"2024-03-30T13:42:47.316187+0000","last_became_peered":"2024-03-30T13:42:47.316187+0000","last_unstale":"2024-03-30T13:59:07.793640+0000","last_undegraded":"2024-03-30T13:59:07.793640+0000","last_fullsized":"2024-03-30T13:59:07.793640+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'18","last_scrub_stamp":"2024-03-30T13:58:54.450973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:58:54.450973+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:21.061429+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":357,"reported_epoch":333,"state":"active+undersized","last_fresh":"2024-03-30T13:59:07.801276+0000","last_change":"2024-03-30T13:50:08.509701+0000","last_active":"2024-03-30T13:59:07.801276+0000","last_peered":"2024-03-30T13:59:07.801276+0000","last_clean":"2024-03-30T13:50:05.499237+0000","last_became_active":"2024-03-30T13:50:08.509701+0000","last_became_peered":"2024-03-30T13:50:08.509701+0000","last_unstale":"2024-03-30T13:59:07.801276+0000","last_undegraded":"2024-03-30T13:59:07.801276+0000","last_fullsized":"2024-03-30T13:50:08.501671+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:22.401087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:22.401087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"333'12568","reported_seq":12667,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:10.688902+0000","last_change":"2024-03-30T13:56:55.952448+0000","last_active":"2024-03-30T13:59:10.688902+0000","last_peered":"2024-03-30T13:59:10.688902+0000","last_clean":"2024-03-30T13:59:10.688902+0000","last_became_active":"2024-03-30T13:56:55.951841+0000","last_became_peered":"2024-03-30T13:56:55.951841+0000","last_unstale":"2024-03-30T13:59:10.688902+0000","last_undegraded":"2024-03-30T13:59:10.688902+0000","last_fullsized":"2024-03-30T13:59:10.688902+0000","mapping_epoch":289,"log_start":"318'8500","ondisk_log_start":"318'8500","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484048896,"num_objects":7386,"num_object_clones":0,"num_object_copies":22158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12568,"num_write_kb":638528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":463,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.797488+0000","last_change":"2024-03-30T13:57:03.960613+0000","last_active":"2024-03-30T13:59:07.797488+0000","last_peered":"2024-03-30T13:59:07.797488+0000","last_clean":"2024-03-30T13:59:07.797488+0000","last_became_active":"2024-03-30T13:53:36.862437+0000","last_became_peered":"2024-03-30T13:53:36.862437+0000","last_unstale":"2024-03-30T13:59:07.797488+0000","last_undegraded":"2024-03-30T13:59:07.797488+0000","last_fullsized":"2024-03-30T13:59:07.797488+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:03.960362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:03.960362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,7],"avail_no_missing":["3","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":380,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.803222+0000","last_change":"2024-03-30T13:58:48.185511+0000","last_active":"2024-03-30T13:59:07.803222+0000","last_peered":"2024-03-30T13:59:07.803222+0000","last_clean":"2024-03-30T13:59:07.803222+0000","last_became_active":"2024-03-30T13:58:48.184571+0000","last_became_peered":"2024-03-30T13:58:48.184571+0000","last_unstale":"2024-03-30T13:59:07.803222+0000","last_undegraded":"2024-03-30T13:59:07.803222+0000","last_fullsized":"2024-03-30T13:59:07.803222+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:06.715115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:06.715115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,1],"avail_no_missing":["0","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.803518+0000","last_change":"2024-03-30T13:58:47.200041+0000","last_active":"2024-03-30T13:59:07.803518+0000","last_peered":"2024-03-30T13:59:07.803518+0000","last_clean":"2024-03-30T13:59:07.803518+0000","last_became_active":"2024-03-30T13:58:47.199395+0000","last_became_peered":"2024-03-30T13:58:47.199395+0000","last_unstale":"2024-03-30T13:59:07.803518+0000","last_undegraded":"2024-03-30T13:59:07.803518+0000","last_fullsized":"2024-03-30T13:59:07.803518+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:16.087693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:17.366740+0000","last_clean_scrub_stamp":"2024-03-30T13:58:16.087693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:27.796567+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"217'44","reported_seq":726,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.799748+0000","last_change":"2024-03-30T13:58:47.227571+0000","last_active":"2024-03-30T13:59:07.799748+0000","last_peered":"2024-03-30T13:59:07.799748+0000","last_clean":"2024-03-30T13:59:07.799748+0000","last_became_active":"2024-03-30T13:58:47.204577+0000","last_became_peered":"2024-03-30T13:58:47.204577+0000","last_unstale":"2024-03-30T13:59:07.799748+0000","last_undegraded":"2024-03-30T13:59:07.799748+0000","last_fullsized":"2024-03-30T13:59:07.799748+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'44","last_scrub_stamp":"2024-03-30T13:58:21.999653+0000","last_deep_scrub":"108'28","last_deep_scrub_stamp":"2024-03-30T13:45:49.140418+0000","last_clean_scrub_stamp":"2024-03-30T13:58:21.999653+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:42.534294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"333'12682","reported_seq":12778,"reported_epoch":333,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:59:10.704771+0000","last_change":"2024-03-30T13:57:08.381005+0000","last_active":"2024-03-30T13:59:10.704771+0000","last_peered":"2024-03-30T13:59:10.704771+0000","last_clean":"2024-03-30T13:56:54.930191+0000","last_became_active":"2024-03-30T13:56:55.950831+0000","last_became_peered":"2024-03-30T13:56:55.950831+0000","last_unstale":"2024-03-30T13:59:10.704771+0000","last_undegraded":"2024-03-30T13:57:08.371957+0000","last_fullsized":"2024-03-30T13:56:55.946732+0000","mapping_epoch":289,"log_start":"318'8600","ondisk_log_start":"318'8600","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4082,"log_dups_size":0,"ondisk_log_size":4082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":494010368,"num_objects":7538,"num_object_clones":0,"num_object_copies":22614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12682,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":7538}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":579,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.802826+0000","last_change":"2024-03-30T13:56:59.724692+0000","last_active":"2024-03-30T13:59:07.802826+0000","last_peered":"2024-03-30T13:59:07.802826+0000","last_clean":"2024-03-30T13:59:07.802826+0000","last_became_active":"2024-03-30T13:53:37.359258+0000","last_became_peered":"2024-03-30T13:53:37.359258+0000","last_unstale":"2024-03-30T13:59:07.802826+0000","last_undegraded":"2024-03-30T13:59:07.802826+0000","last_fullsized":"2024-03-30T13:59:07.802826+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:59.724551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:44.095338+0000","last_clean_scrub_stamp":"2024-03-30T13:56:59.724551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3,7],"avail_no_missing":["0","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":356,"reported_epoch":333,"state":"active+undersized","last_fresh":"2024-03-30T13:59:07.792932+0000","last_change":"2024-03-30T13:50:08.507875+0000","last_active":"2024-03-30T13:59:07.792932+0000","last_peered":"2024-03-30T13:59:07.792932+0000","last_clean":"2024-03-30T13:50:05.490245+0000","last_became_active":"2024-03-30T13:50:08.507875+0000","last_became_peered":"2024-03-30T13:50:08.507875+0000","last_unstale":"2024-03-30T13:59:07.792932+0000","last_undegraded":"2024-03-30T13:59:07.792932+0000","last_fullsized":"2024-03-30T13:50:08.502700+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:13.399683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:28.225742+0000","last_clean_scrub_stamp":"2024-03-30T13:49:13.399683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.803125+0000","last_change":"2024-03-30T13:58:47.195053+0000","last_active":"2024-03-30T13:59:07.803125+0000","last_peered":"2024-03-30T13:59:07.803125+0000","last_clean":"2024-03-30T13:59:07.803125+0000","last_became_active":"2024-03-30T13:58:47.194288+0000","last_became_peered":"2024-03-30T13:58:47.194288+0000","last_unstale":"2024-03-30T13:59:07.803125+0000","last_undegraded":"2024-03-30T13:59:07.803125+0000","last_fullsized":"2024-03-30T13:59:07.803125+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:04.919039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:57:04.919039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"216'15","reported_seq":660,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.803503+0000","last_change":"2024-03-30T13:57:58.907928+0000","last_active":"2024-03-30T13:59:07.803503+0000","last_peered":"2024-03-30T13:59:07.803503+0000","last_clean":"2024-03-30T13:59:07.803503+0000","last_became_active":"2024-03-30T13:53:13.656465+0000","last_became_peered":"2024-03-30T13:53:13.656465+0000","last_unstale":"2024-03-30T13:59:07.803503+0000","last_undegraded":"2024-03-30T13:59:07.803503+0000","last_fullsized":"2024-03-30T13:59:07.803503+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'15","last_scrub_stamp":"2024-03-30T13:57:58.907779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:58.907779+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:19.149130+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"333'12710","reported_seq":20483,"reported_epoch":333,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-03-30T13:59:08.797354+0000","last_change":"2024-03-30T13:58:48.331802+0000","last_active":"2024-03-30T13:59:08.797354+0000","last_peered":"2024-03-30T13:59:08.797354+0000","last_clean":"2024-03-30T13:56:54.930191+0000","last_became_active":"2024-03-30T13:58:48.301633+0000","last_became_peered":"2024-03-30T13:58:48.301633+0000","last_unstale":"2024-03-30T13:59:08.797354+0000","last_undegraded":"2024-03-30T13:58:48.321853+0000","last_fullsized":"2024-03-30T13:58:48.178230+0000","mapping_epoch":326,"log_start":"316'8100","ondisk_log_start":"316'8100","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4610,"log_dups_size":0,"ondisk_log_size":4610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":507510784,"num_objects":7744,"num_object_clones":0,"num_object_copies":23232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12710,"num_write_kb":654528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3836,"num_bytes_recovered":251330560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":7744}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":367,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.804074+0000","last_change":"2024-03-30T13:58:48.190429+0000","last_active":"2024-03-30T13:59:07.804074+0000","last_peered":"2024-03-30T13:59:07.804074+0000","last_clean":"2024-03-30T13:59:07.804074+0000","last_became_active":"2024-03-30T13:58:48.189456+0000","last_became_peered":"2024-03-30T13:58:48.189456+0000","last_unstale":"2024-03-30T13:59:07.804074+0000","last_undegraded":"2024-03-30T13:59:07.804074+0000","last_fullsized":"2024-03-30T13:59:07.804074+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:56:54.694786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.694786+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0,4],"avail_no_missing":["5","0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":587,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.801677+0000","last_change":"2024-03-30T13:57:57.278091+0000","last_active":"2024-03-30T13:59:07.801677+0000","last_peered":"2024-03-30T13:59:07.801677+0000","last_clean":"2024-03-30T13:59:07.801677+0000","last_became_active":"2024-03-30T13:53:35.855186+0000","last_became_peered":"2024-03-30T13:53:35.855186+0000","last_unstale":"2024-03-30T13:59:07.801677+0000","last_undegraded":"2024-03-30T13:59:07.801677+0000","last_fullsized":"2024-03-30T13:59:07.801677+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:57.277970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:57.277970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:12.590589+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":29,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.802691+0000","last_change":"2024-03-30T13:58:47.215143+0000","last_active":"2024-03-30T13:59:07.802691+0000","last_peered":"2024-03-30T13:59:07.802691+0000","last_clean":"2024-03-30T13:59:07.802691+0000","last_became_active":"2024-03-30T13:58:47.199657+0000","last_became_peered":"2024-03-30T13:58:47.199657+0000","last_unstale":"2024-03-30T13:59:07.802691+0000","last_undegraded":"2024-03-30T13:59:07.802691+0000","last_fullsized":"2024-03-30T13:59:07.802691+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-30T13:58:03.985919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:58:03.985919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:22.983627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":593,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.791447+0000","last_change":"2024-03-30T13:58:04.478097+0000","last_active":"2024-03-30T13:59:07.791447+0000","last_peered":"2024-03-30T13:59:07.791447+0000","last_clean":"2024-03-30T13:59:07.791447+0000","last_became_active":"2024-03-30T13:42:45.310209+0000","last_became_peered":"2024-03-30T13:42:45.310209+0000","last_unstale":"2024-03-30T13:59:07.791447+0000","last_undegraded":"2024-03-30T13:59:07.791447+0000","last_fullsized":"2024-03-30T13:59:07.791447+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:04.478028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:04.478028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:21.405419+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"217'41","reported_seq":840,"reported_epoch":333,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:59:07.798326+0000","last_change":"2024-03-30T13:50:08.508727+0000","last_active":"2024-03-30T13:59:07.798326+0000","last_peered":"2024-03-30T13:59:07.798326+0000","last_clean":"2024-03-30T13:50:05.490389+0000","last_became_active":"2024-03-30T13:50:08.508727+0000","last_became_peered":"2024-03-30T13:50:08.508727+0000","last_unstale":"2024-03-30T13:59:07.798326+0000","last_undegraded":"2024-03-30T13:50:08.506113+0000","last_fullsized":"2024-03-30T13:50:08.505993+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-03-30T13:48:27.377242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:27.377242+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:11.264064+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":599,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.804716+0000","last_change":"2024-03-30T13:58:05.858433+0000","last_active":"2024-03-30T13:59:07.804716+0000","last_peered":"2024-03-30T13:59:07.804716+0000","last_clean":"2024-03-30T13:59:07.804716+0000","last_became_active":"2024-03-30T13:42:49.322154+0000","last_became_peered":"2024-03-30T13:42:49.322154+0000","last_unstale":"2024-03-30T13:59:07.804716+0000","last_undegraded":"2024-03-30T13:59:07.804716+0000","last_fullsized":"2024-03-30T13:59:07.804716+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:05.858319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:18.862784+0000","last_clean_scrub_stamp":"2024-03-30T13:58:05.858319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:08.445018+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"333'12645","reported_seq":19478,"reported_epoch":333,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-03-30T13:59:08.331093+0000","last_change":"2024-03-30T13:58:48.288427+0000","last_active":"2024-03-30T13:59:08.331093+0000","last_peered":"2024-03-30T13:59:08.331093+0000","last_clean":"2024-03-30T13:56:54.930191+0000","last_became_active":"2024-03-30T13:58:48.277724+0000","last_became_peered":"2024-03-30T13:58:48.277724+0000","last_unstale":"2024-03-30T13:59:08.331093+0000","last_undegraded":"2024-03-30T13:58:48.194555+0000","last_fullsized":"2024-03-30T13:58:48.174057+0000","mapping_epoch":326,"log_start":"317'8600","ondisk_log_start":"317'8600","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4045,"log_dups_size":0,"ondisk_log_size":4045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":509280256,"num_objects":7771,"num_object_clones":0,"num_object_copies":23313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4702,"num_objects_misplaced":7771,"num_objects_unfound":0,"num_objects_dirty":7771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12645,"num_write_kb":653312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3367,"num_bytes_recovered":220659712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":7771}],"blocked_by":[],"up_primary":5,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":525,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.803144+0000","last_change":"2024-03-30T13:56:52.636459+0000","last_active":"2024-03-30T13:59:07.803144+0000","last_peered":"2024-03-30T13:59:07.803144+0000","last_clean":"2024-03-30T13:59:07.803144+0000","last_became_active":"2024-03-30T13:53:36.851158+0000","last_became_peered":"2024-03-30T13:53:36.851158+0000","last_unstale":"2024-03-30T13:59:07.803144+0000","last_undegraded":"2024-03-30T13:59:07.803144+0000","last_fullsized":"2024-03-30T13:59:07.803144+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:56:52.636303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:56:52.636303+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4,7],"avail_no_missing":["0","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":236,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.800656+0000","last_change":"2024-03-30T13:58:04.252009+0000","last_active":"2024-03-30T13:59:07.800656+0000","last_peered":"2024-03-30T13:59:07.800656+0000","last_clean":"2024-03-30T13:59:07.800656+0000","last_became_active":"2024-03-30T13:53:36.860597+0000","last_became_peered":"2024-03-30T13:53:36.860597+0000","last_unstale":"2024-03-30T13:59:07.800656+0000","last_undegraded":"2024-03-30T13:59:07.800656+0000","last_fullsized":"2024-03-30T13:59:07.800656+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:04.251869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:04.251869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:08.363084+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3,7],"avail_no_missing":["6","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":588,"reported_epoch":333,"state":"active+undersized","last_fresh":"2024-03-30T13:59:07.796172+0000","last_change":"2024-03-30T13:50:08.512561+0000","last_active":"2024-03-30T13:59:07.796172+0000","last_peered":"2024-03-30T13:59:07.796172+0000","last_clean":"2024-03-30T13:50:05.489457+0000","last_became_active":"2024-03-30T13:50:08.512561+0000","last_became_peered":"2024-03-30T13:50:08.512561+0000","last_unstale":"2024-03-30T13:59:07.796172+0000","last_undegraded":"2024-03-30T13:59:07.796172+0000","last_fullsized":"2024-03-30T13:50:08.506886+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:16.464234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:59.810189+0000","last_clean_scrub_stamp":"2024-03-30T13:49:16.464234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:11.264081+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":357,"reported_epoch":333,"state":"active+undersized","last_fresh":"2024-03-30T13:59:07.802190+0000","last_change":"2024-03-30T13:50:08.511337+0000","last_active":"2024-03-30T13:59:07.802190+0000","last_peered":"2024-03-30T13:59:07.802190+0000","last_clean":"2024-03-30T13:50:05.498892+0000","last_became_active":"2024-03-30T13:50:08.511337+0000","last_became_peered":"2024-03-30T13:50:08.511337+0000","last_unstale":"2024-03-30T13:59:07.802190+0000","last_undegraded":"2024-03-30T13:59:07.802190+0000","last_fullsized":"2024-03-30T13:50:08.501456+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:25.398232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:51.919203+0000","last_clean_scrub_stamp":"2024-03-30T13:49:25.398232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"216'17","reported_seq":699,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.797245+0000","last_change":"2024-03-30T13:58:59.170542+0000","last_active":"2024-03-30T13:59:07.797245+0000","last_peered":"2024-03-30T13:59:07.797245+0000","last_clean":"2024-03-30T13:59:07.797245+0000","last_became_active":"2024-03-30T13:58:47.192640+0000","last_became_peered":"2024-03-30T13:58:47.192640+0000","last_unstale":"2024-03-30T13:59:07.797245+0000","last_undegraded":"2024-03-30T13:59:07.797245+0000","last_fullsized":"2024-03-30T13:59:07.797245+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'17","last_scrub_stamp":"2024-03-30T13:58:59.170397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:58:59.170397+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:26.474947+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":23,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.804288+0000","last_change":"2024-03-30T13:58:48.178732+0000","last_active":"2024-03-30T13:59:07.804288+0000","last_peered":"2024-03-30T13:59:07.804288+0000","last_clean":"2024-03-30T13:59:07.804288+0000","last_became_active":"2024-03-30T13:58:48.178266+0000","last_became_peered":"2024-03-30T13:58:48.178266+0000","last_unstale":"2024-03-30T13:59:07.804288+0000","last_undegraded":"2024-03-30T13:59:07.804288+0000","last_fullsized":"2024-03-30T13:59:07.804288+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:05.690511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:28.373277+0000","last_clean_scrub_stamp":"2024-03-30T13:57:05.690511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"333'12457","reported_seq":12555,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:08.785298+0000","last_change":"2024-03-30T13:56:55.953007+0000","last_active":"2024-03-30T13:59:08.785298+0000","last_peered":"2024-03-30T13:59:08.785298+0000","last_clean":"2024-03-30T13:59:08.785298+0000","last_became_active":"2024-03-30T13:56:55.952660+0000","last_became_peered":"2024-03-30T13:56:55.952660+0000","last_unstale":"2024-03-30T13:59:08.785298+0000","last_undegraded":"2024-03-30T13:59:08.785298+0000","last_fullsized":"2024-03-30T13:59:08.785298+0000","mapping_epoch":289,"log_start":"316'7900","ondisk_log_start":"316'7900","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4557,"log_dups_size":0,"ondisk_log_size":4557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498794496,"num_objects":7611,"num_object_clones":0,"num_object_copies":22833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12457,"num_write_kb":642176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":25,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.804752+0000","last_change":"2024-03-30T13:58:47.195043+0000","last_active":"2024-03-30T13:59:07.804752+0000","last_peered":"2024-03-30T13:59:07.804752+0000","last_clean":"2024-03-30T13:59:07.804752+0000","last_became_active":"2024-03-30T13:58:47.194374+0000","last_became_peered":"2024-03-30T13:58:47.194374+0000","last_unstale":"2024-03-30T13:59:07.804752+0000","last_undegraded":"2024-03-30T13:59:07.804752+0000","last_fullsized":"2024-03-30T13:59:07.804752+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:28.079029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:12.433166+0000","last_clean_scrub_stamp":"2024-03-30T13:52:28.079029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":529,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.801121+0000","last_change":"2024-03-30T13:56:53.653367+0000","last_active":"2024-03-30T13:59:07.801121+0000","last_peered":"2024-03-30T13:59:07.801121+0000","last_clean":"2024-03-30T13:59:07.801121+0000","last_became_active":"2024-03-30T13:53:38.545543+0000","last_became_peered":"2024-03-30T13:53:38.545543+0000","last_unstale":"2024-03-30T13:59:07.801121+0000","last_undegraded":"2024-03-30T13:59:07.801121+0000","last_fullsized":"2024-03-30T13:59:07.801121+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:56:53.653242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:56:53.653242+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":605,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.800588+0000","last_change":"2024-03-30T13:58:47.193901+0000","last_active":"2024-03-30T13:59:07.800588+0000","last_peered":"2024-03-30T13:59:07.800588+0000","last_clean":"2024-03-30T13:59:07.800588+0000","last_became_active":"2024-03-30T13:58:47.193538+0000","last_became_peered":"2024-03-30T13:58:47.193538+0000","last_unstale":"2024-03-30T13:59:07.800588+0000","last_undegraded":"2024-03-30T13:59:07.800588+0000","last_fullsized":"2024-03-30T13:59:07.800588+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:16.238202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:12.975288+0000","last_clean_scrub_stamp":"2024-03-30T13:58:16.238202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:40.530260+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"229'29","reported_seq":863,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.800552+0000","last_change":"2024-03-30T13:57:02.380198+0000","last_active":"2024-03-30T13:59:07.800552+0000","last_peered":"2024-03-30T13:59:07.800552+0000","last_clean":"2024-03-30T13:59:07.800552+0000","last_became_active":"2024-03-30T13:53:35.864416+0000","last_became_peered":"2024-03-30T13:53:35.864416+0000","last_unstale":"2024-03-30T13:59:07.800552+0000","last_undegraded":"2024-03-30T13:59:07.800552+0000","last_fullsized":"2024-03-30T13:59:07.800552+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'29","last_scrub_stamp":"2024-03-30T13:57:02.380102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:02.380102+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":23,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.802858+0000","last_change":"2024-03-30T13:58:48.181574+0000","last_active":"2024-03-30T13:59:07.802858+0000","last_peered":"2024-03-30T13:59:07.802858+0000","last_clean":"2024-03-30T13:59:07.802858+0000","last_became_active":"2024-03-30T13:58:48.179382+0000","last_became_peered":"2024-03-30T13:58:48.179382+0000","last_unstale":"2024-03-30T13:59:07.802858+0000","last_undegraded":"2024-03-30T13:59:07.802858+0000","last_fullsized":"2024-03-30T13:59:07.802858+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:05.759975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:05.759975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.8","version":"333'12819","reported_seq":19680,"reported_epoch":333,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-03-30T13:59:10.705847+0000","last_change":"2024-03-30T13:58:48.286044+0000","last_active":"2024-03-30T13:59:10.705847+0000","last_peered":"2024-03-30T13:59:10.705847+0000","last_clean":"2024-03-30T13:56:54.930191+0000","last_became_active":"2024-03-30T13:58:48.272190+0000","last_became_peered":"2024-03-30T13:58:48.272190+0000","last_unstale":"2024-03-30T13:59:10.705847+0000","last_undegraded":"2024-03-30T13:58:48.188574+0000","last_fullsized":"2024-03-30T13:58:48.171020+0000","mapping_epoch":326,"log_start":"318'8700","ondisk_log_start":"318'8700","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4119,"log_dups_size":0,"ondisk_log_size":4119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":493420544,"num_objects":7529,"num_object_clones":0,"num_object_copies":22587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4457,"num_objects_misplaced":7529,"num_objects_unfound":0,"num_objects_dirty":7529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12819,"num_write_kb":651136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3380,"num_bytes_recovered":221511680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":7529}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":580,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.804392+0000","last_change":"2024-03-30T13:58:01.061819+0000","last_active":"2024-03-30T13:59:07.804392+0000","last_peered":"2024-03-30T13:59:07.804392+0000","last_clean":"2024-03-30T13:59:07.804392+0000","last_became_active":"2024-03-30T13:52:36.489806+0000","last_became_peered":"2024-03-30T13:52:36.489806+0000","last_unstale":"2024-03-30T13:59:07.804392+0000","last_undegraded":"2024-03-30T13:59:07.804392+0000","last_fullsized":"2024-03-30T13:59:07.804392+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:58:01.061633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:58:01.061633+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":608,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.796113+0000","last_change":"2024-03-30T13:58:56.161545+0000","last_active":"2024-03-30T13:59:07.796113+0000","last_peered":"2024-03-30T13:59:07.796113+0000","last_clean":"2024-03-30T13:59:07.796113+0000","last_became_active":"2024-03-30T13:58:47.189419+0000","last_became_peered":"2024-03-30T13:58:47.189419+0000","last_unstale":"2024-03-30T13:59:07.796113+0000","last_undegraded":"2024-03-30T13:59:07.796113+0000","last_fullsized":"2024-03-30T13:59:07.796113+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:56.161347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:56.161347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:16.416287+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":599,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.802494+0000","last_change":"2024-03-30T13:57:17.744960+0000","last_active":"2024-03-30T13:59:07.802494+0000","last_peered":"2024-03-30T13:59:07.802494+0000","last_clean":"2024-03-30T13:59:07.802494+0000","last_became_active":"2024-03-30T13:42:45.307905+0000","last_became_peered":"2024-03-30T13:42:45.307905+0000","last_unstale":"2024-03-30T13:59:07.802494+0000","last_undegraded":"2024-03-30T13:59:07.802494+0000","last_fullsized":"2024-03-30T13:59:07.802494+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:17.744839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:57:17.744839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":620,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:10.350921+0000","last_change":"2024-03-30T13:59:10.350921+0000","last_active":"2024-03-30T13:59:10.350921+0000","last_peered":"2024-03-30T13:59:10.350921+0000","last_clean":"2024-03-30T13:59:10.350921+0000","last_became_active":"2024-03-30T13:53:35.889071+0000","last_became_peered":"2024-03-30T13:53:35.889071+0000","last_unstale":"2024-03-30T13:59:10.350921+0000","last_undegraded":"2024-03-30T13:59:10.350921+0000","last_fullsized":"2024-03-30T13:59:10.350921+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-30T13:59:10.350780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:59:10.350780+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:28.127398+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0.00126391,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":340,"num_read_kb":335,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"216'30","reported_seq":624,"reported_epoch":333,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:59:07.803716+0000","last_change":"2024-03-30T13:50:08.510256+0000","last_active":"2024-03-30T13:59:07.803716+0000","last_peered":"2024-03-30T13:59:07.803716+0000","last_clean":"2024-03-30T13:50:05.503358+0000","last_became_active":"2024-03-30T13:50:08.510256+0000","last_became_peered":"2024-03-30T13:50:08.510256+0000","last_unstale":"2024-03-30T13:59:07.803716+0000","last_undegraded":"2024-03-30T13:50:08.504536+0000","last_fullsized":"2024-03-30T13:50:08.504398+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'18","last_scrub_stamp":"2024-03-30T13:49:13.774982+0000","last_deep_scrub":"93'18","last_deep_scrub_stamp":"2024-03-30T13:47:28.872529+0000","last_clean_scrub_stamp":"2024-03-30T13:49:13.774982+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.804764+0000","last_change":"2024-03-30T13:58:47.196443+0000","last_active":"2024-03-30T13:59:07.804764+0000","last_peered":"2024-03-30T13:59:07.804764+0000","last_clean":"2024-03-30T13:59:07.804764+0000","last_became_active":"2024-03-30T13:58:47.195511+0000","last_became_peered":"2024-03-30T13:58:47.195511+0000","last_unstale":"2024-03-30T13:59:07.804764+0000","last_undegraded":"2024-03-30T13:59:07.804764+0000","last_fullsized":"2024-03-30T13:59:07.804764+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:56.113961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:56:56.113961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"333'12524","reported_seq":12623,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:08.796711+0000","last_change":"2024-03-30T13:56:55.953429+0000","last_active":"2024-03-30T13:59:08.796711+0000","last_peered":"2024-03-30T13:59:08.796711+0000","last_clean":"2024-03-30T13:59:08.796711+0000","last_became_active":"2024-03-30T13:56:55.953088+0000","last_became_peered":"2024-03-30T13:56:55.953088+0000","last_unstale":"2024-03-30T13:59:08.796711+0000","last_undegraded":"2024-03-30T13:59:08.796711+0000","last_fullsized":"2024-03-30T13:59:08.796711+0000","mapping_epoch":289,"log_start":"316'7900","ondisk_log_start":"316'7900","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4624,"log_dups_size":0,"ondisk_log_size":4624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502726680,"num_objects":7672,"num_object_clones":0,"num_object_copies":23016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7672,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12524,"num_write_kb":646209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":585,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.800065+0000","last_change":"2024-03-30T13:57:14.131392+0000","last_active":"2024-03-30T13:59:07.800065+0000","last_peered":"2024-03-30T13:59:07.800065+0000","last_clean":"2024-03-30T13:59:07.800065+0000","last_became_active":"2024-03-30T13:52:36.483298+0000","last_became_peered":"2024-03-30T13:52:36.483298+0000","last_unstale":"2024-03-30T13:59:07.800065+0000","last_undegraded":"2024-03-30T13:59:07.800065+0000","last_fullsized":"2024-03-30T13:59:07.800065+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:57:14.131256+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T13:57:14.131256+0000","last_clean_scrub_stamp":"2024-03-30T13:57:14.131256+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":575,"reported_epoch":333,"state":"active+undersized","last_fresh":"2024-03-30T13:59:07.796083+0000","last_change":"2024-03-30T13:50:08.512648+0000","last_active":"2024-03-30T13:59:07.796083+0000","last_peered":"2024-03-30T13:59:07.796083+0000","last_clean":"2024-03-30T13:50:05.489180+0000","last_became_active":"2024-03-30T13:50:08.512648+0000","last_became_peered":"2024-03-30T13:50:08.512648+0000","last_unstale":"2024-03-30T13:59:07.796083+0000","last_undegraded":"2024-03-30T13:59:07.796083+0000","last_fullsized":"2024-03-30T13:50:08.505931+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:21.527712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:21.527712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:11.264074+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":613,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.796927+0000","last_change":"2024-03-30T13:58:48.178329+0000","last_active":"2024-03-30T13:59:07.796927+0000","last_peered":"2024-03-30T13:59:07.796927+0000","last_clean":"2024-03-30T13:59:07.796927+0000","last_became_active":"2024-03-30T13:58:48.177702+0000","last_became_peered":"2024-03-30T13:58:48.177702+0000","last_unstale":"2024-03-30T13:59:07.796927+0000","last_undegraded":"2024-03-30T13:59:07.796927+0000","last_fullsized":"2024-03-30T13:59:07.796927+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:11.909857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:45.975163+0000","last_clean_scrub_stamp":"2024-03-30T13:58:11.909857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:25.299990+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5,0],"avail_no_missing":["1","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"216'29","reported_seq":698,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.804066+0000","last_change":"2024-03-30T13:58:48.219313+0000","last_active":"2024-03-30T13:59:07.804066+0000","last_peered":"2024-03-30T13:59:07.804066+0000","last_clean":"2024-03-30T13:59:07.804066+0000","last_became_active":"2024-03-30T13:58:48.183735+0000","last_became_peered":"2024-03-30T13:58:48.183735+0000","last_unstale":"2024-03-30T13:59:07.804066+0000","last_undegraded":"2024-03-30T13:59:07.804066+0000","last_fullsized":"2024-03-30T13:59:07.804066+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'29","last_scrub_stamp":"2024-03-30T13:57:59.935365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:59.935365+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:18.842287+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":617,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.799852+0000","last_change":"2024-03-30T13:58:47.205660+0000","last_active":"2024-03-30T13:59:07.799852+0000","last_peered":"2024-03-30T13:59:07.799852+0000","last_clean":"2024-03-30T13:59:07.799852+0000","last_became_active":"2024-03-30T13:58:47.204999+0000","last_became_peered":"2024-03-30T13:58:47.204999+0000","last_unstale":"2024-03-30T13:59:07.799852+0000","last_undegraded":"2024-03-30T13:59:07.799852+0000","last_fullsized":"2024-03-30T13:59:07.799852+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:16.165134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:52.887014+0000","last_clean_scrub_stamp":"2024-03-30T13:57:16.165134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"333'12578","reported_seq":12676,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:08.719780+0000","last_change":"2024-03-30T13:56:55.948911+0000","last_active":"2024-03-30T13:59:08.719780+0000","last_peered":"2024-03-30T13:59:08.719780+0000","last_clean":"2024-03-30T13:59:08.719780+0000","last_became_active":"2024-03-30T13:56:55.948591+0000","last_became_peered":"2024-03-30T13:56:55.948591+0000","last_unstale":"2024-03-30T13:59:08.719780+0000","last_undegraded":"2024-03-30T13:59:08.719780+0000","last_fullsized":"2024-03-30T13:59:08.719780+0000","mapping_epoch":289,"log_start":"319'8800","ondisk_log_start":"319'8800","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":3778,"log_dups_size":0,"ondisk_log_size":3778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505151488,"num_objects":7708,"num_object_clones":0,"num_object_copies":23124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12578,"num_write_kb":649152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":599,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.801978+0000","last_change":"2024-03-30T13:58:50.216157+0000","last_active":"2024-03-30T13:59:07.801978+0000","last_peered":"2024-03-30T13:59:07.801978+0000","last_clean":"2024-03-30T13:59:07.801978+0000","last_became_active":"2024-03-30T13:58:50.215325+0000","last_became_peered":"2024-03-30T13:58:50.215325+0000","last_unstale":"2024-03-30T13:59:07.801978+0000","last_undegraded":"2024-03-30T13:59:07.801978+0000","last_fullsized":"2024-03-30T13:59:07.801978+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:57:18.133345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:57:18.133345+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":575,"reported_epoch":333,"state":"active+undersized","last_fresh":"2024-03-30T13:59:07.800926+0000","last_change":"2024-03-30T13:50:08.510025+0000","last_active":"2024-03-30T13:59:07.800926+0000","last_peered":"2024-03-30T13:59:07.800926+0000","last_clean":"2024-03-30T13:50:05.498930+0000","last_became_active":"2024-03-30T13:50:08.510025+0000","last_became_peered":"2024-03-30T13:50:08.510025+0000","last_unstale":"2024-03-30T13:59:07.800926+0000","last_undegraded":"2024-03-30T13:59:07.800926+0000","last_fullsized":"2024-03-30T13:50:08.502478+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:28.946303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:41.142941+0000","last_clean_scrub_stamp":"2024-03-30T13:49:28.946303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":381,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.803914+0000","last_change":"2024-03-30T13:58:48.192282+0000","last_active":"2024-03-30T13:59:07.803914+0000","last_peered":"2024-03-30T13:59:07.803914+0000","last_clean":"2024-03-30T13:59:07.803914+0000","last_became_active":"2024-03-30T13:58:48.191799+0000","last_became_peered":"2024-03-30T13:58:48.191799+0000","last_unstale":"2024-03-30T13:59:07.803914+0000","last_undegraded":"2024-03-30T13:59:07.803914+0000","last_fullsized":"2024-03-30T13:59:07.803914+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:59.986230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:55.458779+0000","last_clean_scrub_stamp":"2024-03-30T13:56:59.986230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3,6],"avail_no_missing":["5","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"217'25","reported_seq":681,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.799044+0000","last_change":"2024-03-30T13:59:00.243888+0000","last_active":"2024-03-30T13:59:07.799044+0000","last_peered":"2024-03-30T13:59:07.799044+0000","last_clean":"2024-03-30T13:59:07.799044+0000","last_became_active":"2024-03-30T13:58:47.186574+0000","last_became_peered":"2024-03-30T13:58:47.186574+0000","last_unstale":"2024-03-30T13:59:07.799044+0000","last_undegraded":"2024-03-30T13:59:07.799044+0000","last_fullsized":"2024-03-30T13:59:07.799044+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'25","last_scrub_stamp":"2024-03-30T13:59:00.243726+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-03-30T13:45:50.945837+0000","last_clean_scrub_stamp":"2024-03-30T13:59:00.243726+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:06.504421+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"326'11","reported_seq":828,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.804221+0000","last_change":"2024-03-30T13:58:47.220214+0000","last_active":"2024-03-30T13:59:07.804221+0000","last_peered":"2024-03-30T13:59:07.804221+0000","last_clean":"2024-03-30T13:59:07.804221+0000","last_became_active":"2024-03-30T13:58:47.199445+0000","last_became_peered":"2024-03-30T13:58:47.199445+0000","last_unstale":"2024-03-30T13:59:07.804221+0000","last_undegraded":"2024-03-30T13:59:07.804221+0000","last_fullsized":"2024-03-30T13:59:07.804221+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'10","last_scrub_stamp":"2024-03-30T13:58:13.880792+0000","last_deep_scrub":"218'9","last_deep_scrub_stamp":"2024-03-30T13:52:57.873554+0000","last_clean_scrub_stamp":"2024-03-30T13:58:13.880792+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:21.281896+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"333'12435","reported_seq":12530,"reported_epoch":333,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:59:08.750961+0000","last_change":"2024-03-30T13:57:08.373767+0000","last_active":"2024-03-30T13:59:08.750961+0000","last_peered":"2024-03-30T13:59:08.750961+0000","last_clean":"2024-03-30T13:56:54.930191+0000","last_became_active":"2024-03-30T13:56:55.952893+0000","last_became_peered":"2024-03-30T13:56:55.952893+0000","last_unstale":"2024-03-30T13:59:08.750961+0000","last_undegraded":"2024-03-30T13:57:08.373532+0000","last_fullsized":"2024-03-30T13:56:55.947530+0000","mapping_epoch":289,"log_start":"316'7800","ondisk_log_start":"316'7800","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4635,"log_dups_size":0,"ondisk_log_size":4635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":493944832,"num_objects":7537,"num_object_clones":0,"num_object_copies":22611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12435,"num_write_kb":639104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":7537}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":612,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.805233+0000","last_change":"2024-03-30T13:58:50.203822+0000","last_active":"2024-03-30T13:59:07.805233+0000","last_peered":"2024-03-30T13:59:07.805233+0000","last_clean":"2024-03-30T13:59:07.805233+0000","last_became_active":"2024-03-30T13:58:50.203408+0000","last_became_peered":"2024-03-30T13:58:50.203408+0000","last_unstale":"2024-03-30T13:59:07.805233+0000","last_undegraded":"2024-03-30T13:59:07.805233+0000","last_fullsized":"2024-03-30T13:59:07.805233+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:48.866100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:48.866100+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":462,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.799764+0000","last_change":"2024-03-30T13:58:03.303778+0000","last_active":"2024-03-30T13:59:07.799764+0000","last_peered":"2024-03-30T13:59:07.799764+0000","last_clean":"2024-03-30T13:59:07.799764+0000","last_became_active":"2024-03-30T13:53:36.852004+0000","last_became_peered":"2024-03-30T13:53:36.852004+0000","last_unstale":"2024-03-30T13:59:07.799764+0000","last_undegraded":"2024-03-30T13:59:07.799764+0000","last_fullsized":"2024-03-30T13:59:07.799764+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:03.303563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:31.900908+0000","last_clean_scrub_stamp":"2024-03-30T13:58:03.303563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:18.791865+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1,7],"avail_no_missing":["6","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":608,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.804277+0000","last_change":"2024-03-30T13:58:12.869168+0000","last_active":"2024-03-30T13:59:07.804277+0000","last_peered":"2024-03-30T13:59:07.804277+0000","last_clean":"2024-03-30T13:59:07.804277+0000","last_became_active":"2024-03-30T13:42:45.309886+0000","last_became_peered":"2024-03-30T13:42:45.309886+0000","last_unstale":"2024-03-30T13:59:07.804277+0000","last_undegraded":"2024-03-30T13:59:07.804277+0000","last_fullsized":"2024-03-30T13:59:07.804277+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:58:12.869021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:12.869021+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:17.722136+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"217'35","reported_seq":690,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.793442+0000","last_change":"2024-03-30T13:58:47.208948+0000","last_active":"2024-03-30T13:59:07.793442+0000","last_peered":"2024-03-30T13:59:07.793442+0000","last_clean":"2024-03-30T13:59:07.793442+0000","last_became_active":"2024-03-30T13:58:47.192909+0000","last_became_peered":"2024-03-30T13:58:47.192909+0000","last_unstale":"2024-03-30T13:59:07.793442+0000","last_undegraded":"2024-03-30T13:59:07.793442+0000","last_fullsized":"2024-03-30T13:59:07.793442+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'21","last_scrub_stamp":"2024-03-30T13:52:44.925926+0000","last_deep_scrub":"91'21","last_deep_scrub_stamp":"2024-03-30T13:44:05.917064+0000","last_clean_scrub_stamp":"2024-03-30T13:52:44.925926+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":791,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.798198+0000","last_change":"2024-03-30T13:57:22.046830+0000","last_active":"2024-03-30T13:59:07.798198+0000","last_peered":"2024-03-30T13:59:07.798198+0000","last_clean":"2024-03-30T13:59:07.798198+0000","last_became_active":"2024-03-30T13:42:49.321423+0000","last_became_peered":"2024-03-30T13:42:49.321423+0000","last_unstale":"2024-03-30T13:59:07.798198+0000","last_undegraded":"2024-03-30T13:59:07.798198+0000","last_fullsized":"2024-03-30T13:59:07.798198+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:57:22.046689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:22.046689+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"333'12609","reported_seq":12704,"reported_epoch":333,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:59:08.727974+0000","last_change":"2024-03-30T13:57:08.372029+0000","last_active":"2024-03-30T13:59:08.727974+0000","last_peered":"2024-03-30T13:59:08.727974+0000","last_clean":"2024-03-30T13:56:54.930191+0000","last_became_active":"2024-03-30T13:56:55.949377+0000","last_became_peered":"2024-03-30T13:56:55.949377+0000","last_unstale":"2024-03-30T13:59:08.727974+0000","last_undegraded":"2024-03-30T13:57:08.371728+0000","last_fullsized":"2024-03-30T13:56:55.941808+0000","mapping_epoch":289,"log_start":"319'8800","ondisk_log_start":"319'8800","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":3809,"log_dups_size":0,"ondisk_log_size":3809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502333440,"num_objects":7665,"num_object_clones":0,"num_object_copies":22995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12609,"num_write_kb":648768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":7665}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":572,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.791498+0000","last_change":"2024-03-30T13:52:47.990769+0000","last_active":"2024-03-30T13:59:07.791498+0000","last_peered":"2024-03-30T13:59:07.791498+0000","last_clean":"2024-03-30T13:59:07.791498+0000","last_became_active":"2024-03-30T13:52:36.490909+0000","last_became_peered":"2024-03-30T13:52:36.490909+0000","last_unstale":"2024-03-30T13:59:07.791498+0000","last_undegraded":"2024-03-30T13:59:07.791498+0000","last_fullsized":"2024-03-30T13:59:07.791498+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:52:47.990635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:52:47.990635+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,0],"avail_no_missing":["4","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":568,"reported_epoch":333,"state":"active+undersized","last_fresh":"2024-03-30T13:59:07.792448+0000","last_change":"2024-03-30T13:50:08.504411+0000","last_active":"2024-03-30T13:59:07.792448+0000","last_peered":"2024-03-30T13:59:07.792448+0000","last_clean":"2024-03-30T13:50:05.489944+0000","last_became_active":"2024-03-30T13:50:08.504411+0000","last_became_peered":"2024-03-30T13:50:08.504411+0000","last_unstale":"2024-03-30T13:59:07.792448+0000","last_undegraded":"2024-03-30T13:59:07.792448+0000","last_fullsized":"2024-03-30T13:50:08.501823+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:10.216387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:10.216387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":605,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.802727+0000","last_change":"2024-03-30T13:58:47.194618+0000","last_active":"2024-03-30T13:59:07.802727+0000","last_peered":"2024-03-30T13:59:07.802727+0000","last_clean":"2024-03-30T13:59:07.802727+0000","last_became_active":"2024-03-30T13:58:47.194093+0000","last_became_peered":"2024-03-30T13:58:47.194093+0000","last_unstale":"2024-03-30T13:59:07.802727+0000","last_undegraded":"2024-03-30T13:59:07.802727+0000","last_fullsized":"2024-03-30T13:59:07.802727+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:53:01.031216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:53:01.031216+0000","last_clean_scrub_stamp":"2024-03-30T13:53:01.031216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"216'35","reported_seq":669,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.798051+0000","last_change":"2024-03-30T13:53:36.081652+0000","last_active":"2024-03-30T13:59:07.798051+0000","last_peered":"2024-03-30T13:59:07.798051+0000","last_clean":"2024-03-30T13:59:07.798051+0000","last_became_active":"2024-03-30T13:53:35.888652+0000","last_became_peered":"2024-03-30T13:53:35.888652+0000","last_unstale":"2024-03-30T13:59:07.798051+0000","last_undegraded":"2024-03-30T13:59:07.798051+0000","last_fullsized":"2024-03-30T13:59:07.798051+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'29","last_scrub_stamp":"2024-03-30T13:52:52.973602+0000","last_deep_scrub":"216'29","last_deep_scrub_stamp":"2024-03-30T13:52:52.973602+0000","last_clean_scrub_stamp":"2024-03-30T13:52:52.973602+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":591,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.793287+0000","last_change":"2024-03-30T13:57:10.660140+0000","last_active":"2024-03-30T13:59:07.793287+0000","last_peered":"2024-03-30T13:59:07.793287+0000","last_clean":"2024-03-30T13:59:07.793287+0000","last_became_active":"2024-03-30T13:53:35.863124+0000","last_became_peered":"2024-03-30T13:53:35.863124+0000","last_unstale":"2024-03-30T13:59:07.793287+0000","last_undegraded":"2024-03-30T13:59:07.793287+0000","last_fullsized":"2024-03-30T13:59:07.793287+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:10.660011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:10.660011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"333'12345","reported_seq":12444,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:08.707275+0000","last_change":"2024-03-30T13:56:55.950105+0000","last_active":"2024-03-30T13:59:08.707275+0000","last_peered":"2024-03-30T13:59:08.707275+0000","last_clean":"2024-03-30T13:59:08.707275+0000","last_became_active":"2024-03-30T13:56:55.949813+0000","last_became_peered":"2024-03-30T13:56:55.949813+0000","last_unstale":"2024-03-30T13:59:08.707275+0000","last_undegraded":"2024-03-30T13:59:08.707275+0000","last_fullsized":"2024-03-30T13:59:08.707275+0000","mapping_epoch":289,"log_start":"318'8500","ondisk_log_start":"318'8500","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498925568,"num_objects":7613,"num_object_clones":0,"num_object_copies":22839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12345,"num_write_kb":638656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":367,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.802851+0000","last_change":"2024-03-30T13:58:48.185795+0000","last_active":"2024-03-30T13:59:07.802851+0000","last_peered":"2024-03-30T13:59:07.802851+0000","last_clean":"2024-03-30T13:59:07.802851+0000","last_became_active":"2024-03-30T13:58:48.184258+0000","last_became_peered":"2024-03-30T13:58:48.184258+0000","last_unstale":"2024-03-30T13:59:07.802851+0000","last_undegraded":"2024-03-30T13:59:07.802851+0000","last_fullsized":"2024-03-30T13:59:07.802851+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:58:05.194482+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T13:56:08.385059+0000","last_clean_scrub_stamp":"2024-03-30T13:58:05.194482+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:24.494181+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6,4],"avail_no_missing":["5","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":374,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.803826+0000","last_change":"2024-03-30T13:58:48.184750+0000","last_active":"2024-03-30T13:59:07.803826+0000","last_peered":"2024-03-30T13:59:07.803826+0000","last_clean":"2024-03-30T13:59:07.803826+0000","last_became_active":"2024-03-30T13:58:48.183934+0000","last_became_peered":"2024-03-30T13:58:48.183934+0000","last_unstale":"2024-03-30T13:59:07.803826+0000","last_undegraded":"2024-03-30T13:59:07.803826+0000","last_fullsized":"2024-03-30T13:59:07.803826+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:23.126245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:23.126245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:25.399071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"327'13","reported_seq":813,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.794866+0000","last_change":"2024-03-30T13:58:48.205934+0000","last_active":"2024-03-30T13:59:07.794866+0000","last_peered":"2024-03-30T13:59:07.794866+0000","last_clean":"2024-03-30T13:59:07.794866+0000","last_became_active":"2024-03-30T13:58:48.204824+0000","last_became_peered":"2024-03-30T13:58:48.204824+0000","last_unstale":"2024-03-30T13:59:07.794866+0000","last_undegraded":"2024-03-30T13:59:07.794866+0000","last_fullsized":"2024-03-30T13:59:07.794866+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'11","last_scrub_stamp":"2024-03-30T13:58:06.437298+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-30T13:51:15.146255+0000","last_clean_scrub_stamp":"2024-03-30T13:58:06.437298+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:31.371009+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":591,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.797876+0000","last_change":"2024-03-30T13:53:35.887442+0000","last_active":"2024-03-30T13:59:07.797876+0000","last_peered":"2024-03-30T13:59:07.797876+0000","last_clean":"2024-03-30T13:59:07.797876+0000","last_became_active":"2024-03-30T13:53:35.886750+0000","last_became_peered":"2024-03-30T13:53:35.886750+0000","last_unstale":"2024-03-30T13:59:07.797876+0000","last_undegraded":"2024-03-30T13:59:07.797876+0000","last_fullsized":"2024-03-30T13:59:07.797876+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:52:27.058807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:52:27.058807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"217'34","reported_seq":654,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.797090+0000","last_change":"2024-03-30T13:58:09.947780+0000","last_active":"2024-03-30T13:59:07.797090+0000","last_peered":"2024-03-30T13:59:07.797090+0000","last_clean":"2024-03-30T13:59:07.797090+0000","last_became_active":"2024-03-30T13:42:47.324350+0000","last_became_peered":"2024-03-30T13:42:47.324350+0000","last_unstale":"2024-03-30T13:59:07.797090+0000","last_undegraded":"2024-03-30T13:59:07.797090+0000","last_fullsized":"2024-03-30T13:59:07.797090+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'34","last_scrub_stamp":"2024-03-30T13:58:09.947626+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:28.831527+0000","last_clean_scrub_stamp":"2024-03-30T13:58:09.947626+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:15.653767+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"333'12582","reported_seq":12681,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:08.738970+0000","last_change":"2024-03-30T13:56:55.955019+0000","last_active":"2024-03-30T13:59:08.738970+0000","last_peered":"2024-03-30T13:59:08.738970+0000","last_clean":"2024-03-30T13:59:08.738970+0000","last_became_active":"2024-03-30T13:56:55.954690+0000","last_became_peered":"2024-03-30T13:56:55.954690+0000","last_unstale":"2024-03-30T13:59:08.738970+0000","last_undegraded":"2024-03-30T13:59:08.738970+0000","last_fullsized":"2024-03-30T13:59:08.738970+0000","mapping_epoch":289,"log_start":"316'8000","ondisk_log_start":"316'8000","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4582,"log_dups_size":0,"ondisk_log_size":4582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":507117568,"num_objects":7738,"num_object_clones":0,"num_object_copies":23214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12582,"num_write_kb":650240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":599,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.805509+0000","last_change":"2024-03-30T13:58:47.194433+0000","last_active":"2024-03-30T13:59:07.805509+0000","last_peered":"2024-03-30T13:59:07.805509+0000","last_clean":"2024-03-30T13:59:07.805509+0000","last_became_active":"2024-03-30T13:58:47.193821+0000","last_became_peered":"2024-03-30T13:58:47.193821+0000","last_unstale":"2024-03-30T13:59:07.805509+0000","last_undegraded":"2024-03-30T13:59:07.805509+0000","last_fullsized":"2024-03-30T13:59:07.805509+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:09.195697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:21.081938+0000","last_clean_scrub_stamp":"2024-03-30T13:58:09.195697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:36.467245+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":378,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.804917+0000","last_change":"2024-03-30T13:58:48.216887+0000","last_active":"2024-03-30T13:59:07.804917+0000","last_peered":"2024-03-30T13:59:07.804917+0000","last_clean":"2024-03-30T13:59:07.804917+0000","last_became_active":"2024-03-30T13:58:48.216542+0000","last_became_peered":"2024-03-30T13:58:48.216542+0000","last_unstale":"2024-03-30T13:59:07.804917+0000","last_undegraded":"2024-03-30T13:59:07.804917+0000","last_fullsized":"2024-03-30T13:59:07.804917+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:15.580974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:52.603772+0000","last_clean_scrub_stamp":"2024-03-30T13:58:15.580974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:36.539750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":593,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.793295+0000","last_change":"2024-03-30T13:56:57.664929+0000","last_active":"2024-03-30T13:59:07.793295+0000","last_peered":"2024-03-30T13:59:07.793295+0000","last_clean":"2024-03-30T13:59:07.793295+0000","last_became_active":"2024-03-30T13:53:36.852500+0000","last_became_peered":"2024-03-30T13:53:36.852500+0000","last_unstale":"2024-03-30T13:59:07.793295+0000","last_undegraded":"2024-03-30T13:59:07.793295+0000","last_fullsized":"2024-03-30T13:59:07.793295+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:57.664795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:56:57.664795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,7],"avail_no_missing":["4","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"217'19","reported_seq":636,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.794756+0000","last_change":"2024-03-30T13:58:53.440615+0000","last_active":"2024-03-30T13:59:07.794756+0000","last_peered":"2024-03-30T13:59:07.794756+0000","last_clean":"2024-03-30T13:59:07.794756+0000","last_became_active":"2024-03-30T13:42:47.315667+0000","last_became_peered":"2024-03-30T13:42:47.315667+0000","last_unstale":"2024-03-30T13:59:07.794756+0000","last_undegraded":"2024-03-30T13:59:07.794756+0000","last_fullsized":"2024-03-30T13:59:07.794756+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'19","last_scrub_stamp":"2024-03-30T13:58:53.440486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:58:53.440486+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:01.331209+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"333'12646","reported_seq":12758,"reported_epoch":333,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-03-30T13:59:08.250939+0000","last_change":"2024-03-30T13:58:48.331067+0000","last_active":"2024-03-30T13:59:08.250939+0000","last_peered":"2024-03-30T13:59:08.250939+0000","last_clean":"2024-03-30T13:56:54.930191+0000","last_became_active":"2024-03-30T13:58:48.283279+0000","last_became_peered":"2024-03-30T13:58:48.283279+0000","last_unstale":"2024-03-30T13:59:08.250939+0000","last_undegraded":"2024-03-30T13:58:48.199934+0000","last_fullsized":"2024-03-30T13:58:48.174829+0000","mapping_epoch":326,"log_start":"317'8600","ondisk_log_start":"317'8600","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4046,"log_dups_size":0,"ondisk_log_size":4046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":516554752,"num_objects":7882,"num_object_clones":0,"num_object_copies":23646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7879,"num_objects_misplaced":7882,"num_objects_unfound":0,"num_objects_dirty":7882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12646,"num_write_kb":656896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":262144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":7882}],"blocked_by":[],"up_primary":5,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":367,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.803410+0000","last_change":"2024-03-30T13:58:48.191817+0000","last_active":"2024-03-30T13:59:07.803410+0000","last_peered":"2024-03-30T13:59:07.803410+0000","last_clean":"2024-03-30T13:59:07.803410+0000","last_became_active":"2024-03-30T13:58:48.191314+0000","last_became_peered":"2024-03-30T13:58:48.191314+0000","last_unstale":"2024-03-30T13:59:07.803410+0000","last_undegraded":"2024-03-30T13:59:07.803410+0000","last_fullsized":"2024-03-30T13:59:07.803410+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:56:54.940801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.940801+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3,6],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":357,"reported_epoch":333,"state":"active+undersized","last_fresh":"2024-03-30T13:59:07.802479+0000","last_change":"2024-03-30T13:50:08.511676+0000","last_active":"2024-03-30T13:59:07.802479+0000","last_peered":"2024-03-30T13:59:07.802479+0000","last_clean":"2024-03-30T13:50:05.504989+0000","last_became_active":"2024-03-30T13:50:08.511676+0000","last_became_peered":"2024-03-30T13:50:08.511676+0000","last_unstale":"2024-03-30T13:59:07.802479+0000","last_undegraded":"2024-03-30T13:59:07.802479+0000","last_fullsized":"2024-03-30T13:50:08.504157+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:09.357378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:09.357378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":608,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.798529+0000","last_change":"2024-03-30T13:59:06.350933+0000","last_active":"2024-03-30T13:59:07.798529+0000","last_peered":"2024-03-30T13:59:07.798529+0000","last_clean":"2024-03-30T13:59:07.798529+0000","last_became_active":"2024-03-30T13:42:49.321360+0000","last_became_peered":"2024-03-30T13:42:49.321360+0000","last_unstale":"2024-03-30T13:59:07.798529+0000","last_undegraded":"2024-03-30T13:59:07.798529+0000","last_fullsized":"2024-03-30T13:59:07.798529+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:59:06.350813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:59:06.350813+0000","last_clean_scrub_stamp":"2024-03-30T13:59:06.350813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:07.237850+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":614,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.799566+0000","last_change":"2024-03-30T13:57:12.187448+0000","last_active":"2024-03-30T13:59:07.799566+0000","last_peered":"2024-03-30T13:59:07.799566+0000","last_clean":"2024-03-30T13:59:07.799566+0000","last_became_active":"2024-03-30T13:42:45.315251+0000","last_became_peered":"2024-03-30T13:42:45.315251+0000","last_unstale":"2024-03-30T13:59:07.799566+0000","last_undegraded":"2024-03-30T13:59:07.799566+0000","last_fullsized":"2024-03-30T13:59:07.799566+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:12.187291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:57:12.187291+0000","last_clean_scrub_stamp":"2024-03-30T13:57:12.187291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"217'31","reported_seq":625,"reported_epoch":333,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:59:07.792778+0000","last_change":"2024-03-30T13:50:08.508030+0000","last_active":"2024-03-30T13:59:07.792778+0000","last_peered":"2024-03-30T13:59:07.792778+0000","last_clean":"2024-03-30T13:50:05.491159+0000","last_became_active":"2024-03-30T13:50:08.508030+0000","last_became_peered":"2024-03-30T13:50:08.508030+0000","last_unstale":"2024-03-30T13:59:07.792778+0000","last_undegraded":"2024-03-30T13:50:08.502958+0000","last_fullsized":"2024-03-30T13:50:08.502815+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-03-30T13:48:27.096261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:27.096261+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"333'12789","reported_seq":12987,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:11.034078+0000","last_change":"2024-03-30T13:58:55.948850+0000","last_active":"2024-03-30T13:59:11.034078+0000","last_peered":"2024-03-30T13:59:11.034078+0000","last_clean":"2024-03-30T13:59:11.034078+0000","last_became_active":"2024-03-30T13:56:55.953793+0000","last_became_peered":"2024-03-30T13:56:55.953793+0000","last_unstale":"2024-03-30T13:59:11.034078+0000","last_undegraded":"2024-03-30T13:59:11.034078+0000","last_fullsized":"2024-03-30T13:59:11.034078+0000","mapping_epoch":289,"log_start":"319'8900","ondisk_log_start":"319'8900","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'11631","last_scrub_stamp":"2024-03-30T13:58:55.948602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:58:55.948602+0000","objects_scrubbed":8948,"log_size":3889,"log_dups_size":0,"ondisk_log_size":3889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:19.475818+0000","scrub_duration":2801,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503250944,"num_objects":7679,"num_object_clones":0,"num_object_copies":23037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12789,"num_write_kb":654976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":833,"reported_epoch":333,"state":"active+clean+remapped","last_fresh":"2024-03-30T13:59:07.801630+0000","last_change":"2024-03-30T13:58:50.203750+0000","last_active":"2024-03-30T13:59:07.801630+0000","last_peered":"2024-03-30T13:59:07.801630+0000","last_clean":"2024-03-30T13:59:07.801630+0000","last_became_active":"2024-03-30T13:58:50.203221+0000","last_became_peered":"2024-03-30T13:58:50.203221+0000","last_unstale":"2024-03-30T13:59:07.801630+0000","last_undegraded":"2024-03-30T13:59:07.801630+0000","last_fullsized":"2024-03-30T13:59:07.801630+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:56:55.928341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:56:55.928341+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5,6],"avail_no_missing":["3","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":568,"reported_epoch":333,"state":"active+undersized","last_fresh":"2024-03-30T13:59:07.794965+0000","last_change":"2024-03-30T13:50:08.511010+0000","last_active":"2024-03-30T13:59:07.794965+0000","last_peered":"2024-03-30T13:59:07.794965+0000","last_clean":"2024-03-30T13:50:05.490535+0000","last_became_active":"2024-03-30T13:50:08.511010+0000","last_became_peered":"2024-03-30T13:50:08.511010+0000","last_unstale":"2024-03-30T13:59:07.794965+0000","last_undegraded":"2024-03-30T13:59:07.794965+0000","last_fullsized":"2024-03-30T13:50:08.503118+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:06.154805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:06.154805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"327'14","reported_seq":32,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.803341+0000","last_change":"2024-03-30T13:58:48.236949+0000","last_active":"2024-03-30T13:59:07.803341+0000","last_peered":"2024-03-30T13:59:07.803341+0000","last_clean":"2024-03-30T13:59:07.803341+0000","last_became_active":"2024-03-30T13:58:48.191842+0000","last_became_peered":"2024-03-30T13:58:48.191842+0000","last_unstale":"2024-03-30T13:59:07.803341+0000","last_undegraded":"2024-03-30T13:59:07.803341+0000","last_fullsized":"2024-03-30T13:59:07.803341+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'13","last_scrub_stamp":"2024-03-30T13:57:08.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:08.673207+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":593,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.800778+0000","last_change":"2024-03-30T13:57:05.955711+0000","last_active":"2024-03-30T13:59:07.800778+0000","last_peered":"2024-03-30T13:59:07.800778+0000","last_clean":"2024-03-30T13:59:07.800778+0000","last_became_active":"2024-03-30T13:42:45.309265+0000","last_became_peered":"2024-03-30T13:42:45.309265+0000","last_unstale":"2024-03-30T13:59:07.800778+0000","last_undegraded":"2024-03-30T13:59:07.800778+0000","last_fullsized":"2024-03-30T13:59:07.800778+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:05.955630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:57:05.955630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"216'20","reported_seq":627,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-03-30T13:59:07.795118+0000","last_change":"2024-03-30T13:58:22.508345+0000","last_active":"2024-03-30T13:59:07.795118+0000","last_peered":"2024-03-30T13:59:07.795118+0000","last_clean":"2024-03-30T13:59:07.795118+0000","last_became_active":"2024-03-30T13:42:47.313026+0000","last_became_peered":"2024-03-30T13:42:47.313026+0000","last_unstale":"2024-03-30T13:59:07.795118+0000","last_undegraded":"2024-03-30T13:59:07.795118+0000","last_fullsized":"2024-03-30T13:59:07.795118+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'20","last_scrub_stamp":"2024-03-30T13:58:22.508219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:58:22.508219+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:59:40.120341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":8021082136,"num_objects":122393,"num_object_clones":0,"num_object_copies":367179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":51764,"num_objects_misplaced":30760,"num_objects_unfound":0,"num_objects_dirty":122393,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":202057,"num_write_kb":10382337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10588,"num_bytes_recovered":693764096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20651061248,"data_stored":20651049032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67357,"ondisk_log_size":67357,"up":42,"acting":40,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":340,"num_read_kb":335,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":27,"acting":33,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":5246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":87,"acting":93,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":3234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":743,"ondisk_log_size":743,"up":86,"acting":89,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":85,"acting":93,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":82,"acting":91,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":249108103370,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397710,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4138224,"kb_used_data":3471204,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":89606928,"statfs":{"total":95995035648,"available":91757494272,"internally_reserved":0,"allocated":3554512896,"data_stored":3553124142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":683014598},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575080,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3156148,"kb_used_data":2434988,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":90589004,"statfs":{"total":95995035648,"available":92763140096,"internally_reserved":0,"allocated":2493427712,"data_stored":2491966379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738605,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5560188,"kb_used_data":4838900,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":88184964,"statfs":{"total":95995035648,"available":90301403136,"internally_reserved":0,"allocated":4955033600,"data_stored":4953628743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411446,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2213288,"kb_used_data":2015520,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":91531864,"statfs":{"total":95995035648,"available":93728628736,"internally_reserved":0,"allocated":2063892480,"data_stored":2063509110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182622,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4315448,"kb_used_data":3494256,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":89429704,"statfs":{"total":95995035648,"available":91576016896,"internally_reserved":0,"allocated":3578118144,"data_stored":3576708370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":840825287},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953688,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6091852,"kb_used_data":5451136,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":87653300,"statfs":{"total":95995035648,"available":89756979200,"internally_reserved":0,"allocated":5581963264,"data_stored":5580545792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692051,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1028856,"kb_used_data":600620,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":92716296,"statfs":{"total":95995035648,"available":94941487104,"internally_reserved":0,"allocated":615034880,"data_stored":613638744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":2778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4942008320,"data_stored":4942004248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2471100416,"data_stored":2471100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3538288640,"data_stored":3538288640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5558374400,"data_stored":5558370328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":622526464,"data_stored":622526464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3518763008,"data_stored":3518758936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:59:17.101 INFO:tasks.thrashosds.thrasher:Setting 4.1a to [7, 3, 2] 2024-03-30T13:59:17.101 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.1a', '7', '3', '2'] 2024-03-30T13:59:17.101 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pg-upmap 4.1a 7 3 2 2024-03-30T13:59:17.123 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:16 smithi113 ceph-mon[34341]: pgmap v731: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 648 B/s rd, 0 B/s wr, 1.22k op/s; 48163/355833 objects degraded (13.535%); 30016/355833 objects misplaced (8.435%); 35 MiB/s, 565 objects/s recovering 2024-03-30T13:59:17.159 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:17.193 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:17.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:17.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:17.295 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:17.335 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:17.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:17.436 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:17.472 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:17.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:17.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:17.507 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:59:17.508 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:59:17.574 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:17.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:17.629 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:17.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:17.730 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:17.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:17.774 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:17.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:17.875 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:17.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:17.913 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:17.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:17.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:17 smithi043 ceph-mon[29963]: 5.1a scrub starts 2024-03-30T13:59:17.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:17 smithi043 ceph-mon[29963]: 5.1a scrub ok 2024-03-30T13:59:17.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:17 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 48163/355833 objects degraded (13.535%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:17.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:17 smithi043 ceph-mon[29963]: osdmap e336: 8 total, 7 up, 7 in 2024-03-30T13:59:17.975 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:17 smithi043 ceph-mon[42099]: 5.1a scrub starts 2024-03-30T13:59:17.975 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:17 smithi043 ceph-mon[42099]: 5.1a scrub ok 2024-03-30T13:59:17.975 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:17 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 48163/355833 objects degraded (13.535%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:17.975 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:17 smithi043 ceph-mon[42099]: osdmap e336: 8 total, 7 up, 7 in 2024-03-30T13:59:18.014 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:18.053 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:18.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:18.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:18.155 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:18.167 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:17 smithi113 ceph-mon[34341]: 5.1a scrub starts 2024-03-30T13:59:18.167 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:17 smithi113 ceph-mon[34341]: 5.1a scrub ok 2024-03-30T13:59:18.167 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:17 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 48163/355833 objects degraded (13.535%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:18.167 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:17 smithi113 ceph-mon[34341]: osdmap e336: 8 total, 7 up, 7 in 2024-03-30T13:59:18.199 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:18.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:18.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:18.211 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-03-30T13:59:18.301 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:18.334 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:18.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:18.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:18.437 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:18.485 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:18.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:18.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:18.587 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:18.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:18.645 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:18.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:18.653 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:59:18.746 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:18.801 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:18.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:18.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:18.903 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:18.940 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:18.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:18.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:19.042 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:19.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:18 smithi043 ceph-mon[29963]: 4.18 scrub starts 2024-03-30T13:59:19.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:18 smithi043 ceph-mon[29963]: 4.18 scrub ok 2024-03-30T13:59:19.044 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:18 smithi043 ceph-mon[29963]: pgmap v733: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 980 op/s; 46850/350127 objects degraded (13.381%); 29276/350127 objects misplaced (8.362%); 27 MiB/s, 427 objects/s recovering 2024-03-30T13:59:19.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:18 smithi043 ceph-mon[42099]: 4.18 scrub starts 2024-03-30T13:59:19.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:18 smithi043 ceph-mon[42099]: 4.18 scrub ok 2024-03-30T13:59:19.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:18 smithi043 ceph-mon[42099]: pgmap v733: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 980 op/s; 46850/350127 objects degraded (13.381%); 29276/350127 objects misplaced (8.362%); 27 MiB/s, 427 objects/s recovering 2024-03-30T13:59:19.049 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:18 smithi113 ceph-mon[34341]: 4.18 scrub starts 2024-03-30T13:59:19.049 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:18 smithi113 ceph-mon[34341]: 4.18 scrub ok 2024-03-30T13:59:19.050 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:18 smithi113 ceph-mon[34341]: pgmap v733: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 7.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 980 op/s; 46850/350127 objects degraded (13.381%); 29276/350127 objects misplaced (8.362%); 27 MiB/s, 427 objects/s recovering 2024-03-30T13:59:19.086 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:19.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:19.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:19.188 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:19.233 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:19.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:19.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:19.332 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:59:19.335 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:19.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:19.398 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:19.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:19.429 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:59:19 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:59:19.442 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:59:19.483 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:59:19.500 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:19.528 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:59:19.533 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:19.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:19.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:19.568 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:59:19.612 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:59:19.635 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:19.655 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:59:19.682 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:19.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:19.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:19.705 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:59:19.705 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:59:19.705 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:59:19.706 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:59:19.706 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:59:19.706 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:59:19.706 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:59:19.706 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:59:19.706 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:59:19.785 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:19.820 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:19.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:19.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:19.847 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-03-30T13:59:19.921 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:19.954 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:19.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:19.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:20.056 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:20.092 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:20.093 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:19 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3742007248' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:59:20.093 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:19 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:59:20.093 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:19 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1261902439' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:59:20.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:20.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:20.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:19 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3742007248' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:59:20.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:19 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:59:20.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:19 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1261902439' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:59:20.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:19 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3742007248' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:59:20.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:19 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T13:59:20.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:19 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1261902439' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:59:20.195 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:20.231 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:20.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:20.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:20.333 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:20.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:20.376 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:20.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:20.477 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:20.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:20.530 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:20.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:20.631 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:20.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:20.671 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:20.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:20.772 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:20.814 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:20.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:20.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:20.853 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:59:20.853 INFO:teuthology.orchestra.run.smithi113.stdout: "size": 20, 2024-03-30T13:59:20.853 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 600, 2024-03-30T13:59:20.853 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [ 2024-03-30T13:59:20.854 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.854 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:31274 8.5 8:a5782853:::benchmark_data_smithi043_119832_object31273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-30T13:59:20.854 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:49:27.602094+0000", 2024-03-30T13:59:20.854 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 593.25116711199996, 2024-03-30T13:59:20.854 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.177822909, 2024-03-30T13:59:20.854 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:59:20.855 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:59:20.855 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:59:20.855 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:59:20.855 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:59:20.855 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:59:20.855 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 31274 2024-03-30T13:59:20.856 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.856 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:59:20.856 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.856 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:59:20.856 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.602094+0000", 2024-03-30T13:59:20.856 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.857 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.857 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.857 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:59:20.857 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.602094+0000", 2024-03-30T13:59:20.857 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.857 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.858 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.858 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:59:20.858 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.602095+0000", 2024-03-30T13:59:20.858 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.8700000000000004e-07 2024-03-30T13:59:20.858 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.859 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.859 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:59:20.859 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.602128+0000", 2024-03-30T13:59:20.859 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2987000000000001e-05 2024-03-30T13:59:20.859 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.859 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.860 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:59:20.860 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.602140+0000", 2024-03-30T13:59:20.860 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1474999999999999e-05 2024-03-30T13:59:20.860 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.860 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.860 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:59:20.861 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.602148+0000", 2024-03-30T13:59:20.861 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8380000000000002e-06 2024-03-30T13:59:20.861 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.861 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.861 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.861 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.602196+0000", 2024-03-30T13:59:20.862 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8050999999999998e-05 2024-03-30T13:59:20.862 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.862 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.862 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:59:20.862 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.602314+0000", 2024-03-30T13:59:20.863 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000118026 2024-03-30T13:59:20.863 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.863 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.863 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 4,7", 2024-03-30T13:59:20.863 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.602437+0000", 2024-03-30T13:59:20.864 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012336899999999999 2024-03-30T13:59:20.864 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.864 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.864 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:59:20.864 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.603685+0000", 2024-03-30T13:59:20.864 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0012479329999999999 2024-03-30T13:59:20.864 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.865 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.865 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.865 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.604229+0000", 2024-03-30T13:59:20.865 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00054412000000000004 2024-03-30T13:59:20.865 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.866 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.866 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.866 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.779807+0000", 2024-03-30T13:59:20.866 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.17557811200000001 2024-03-30T13:59:20.866 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.866 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.867 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:59:20.867 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.779871+0000", 2024-03-30T13:59:20.867 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3381999999999997e-05 2024-03-30T13:59:20.867 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.867 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.867 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:59:20.867 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.779917+0000", 2024-03-30T13:59:20.867 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6728999999999999e-05 2024-03-30T13:59:20.868 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.868 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:20.868 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.868 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.868 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.868 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:31292 8.5 8:afab0343:::benchmark_data_smithi043_119832_object31291:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-03-30T13:59:20.868 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:49:27.608619+0000", 2024-03-30T13:59:20.868 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 593.24464208200004, 2024-03-30T13:59:20.869 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.171446764, 2024-03-30T13:59:20.869 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:59:20.869 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:59:20.869 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:59:20.869 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:59:20.869 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:59:20.869 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:59:20.870 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 31292 2024-03-30T13:59:20.870 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.870 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:59:20.870 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.870 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:59:20.870 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.608619+0000", 2024-03-30T13:59:20.870 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.870 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.871 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.871 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:59:20.871 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.608619+0000", 2024-03-30T13:59:20.871 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.871 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.871 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.871 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:59:20.872 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.608621+0000", 2024-03-30T13:59:20.872 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.544e-06 2024-03-30T13:59:20.872 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.872 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.872 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:59:20.872 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.608675+0000", 2024-03-30T13:59:20.872 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4211e-05 2024-03-30T13:59:20.872 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.872 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.872 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:59:20.873 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.608694+0000", 2024-03-30T13:59:20.873 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.8989999999999999e-05 2024-03-30T13:59:20.873 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.873 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.873 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:59:20.873 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.608703+0000", 2024-03-30T13:59:20.873 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.2050000000000008e-06 2024-03-30T13:59:20.873 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.873 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.874 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.874 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.608772+0000", 2024-03-30T13:59:20.874 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8232000000000007e-05 2024-03-30T13:59:20.874 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.874 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.874 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:59:20.874 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.608890+0000", 2024-03-30T13:59:20.874 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000118066 2024-03-30T13:59:20.874 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.874 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.874 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 4,7", 2024-03-30T13:59:20.874 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.609032+0000", 2024-03-30T13:59:20.874 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000142263 2024-03-30T13:59:20.875 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.875 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.875 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.875 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.610546+0000", 2024-03-30T13:59:20.875 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0015144640000000001 2024-03-30T13:59:20.875 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.875 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.875 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:59:20.875 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.610564+0000", 2024-03-30T13:59:20.875 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.7280999999999999e-05 2024-03-30T13:59:20.875 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.875 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.875 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.875 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.779971+0000", 2024-03-30T13:59:20.876 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.169407591 2024-03-30T13:59:20.876 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.876 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.876 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:59:20.876 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.780024+0000", 2024-03-30T13:59:20.876 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2621999999999999e-05 2024-03-30T13:59:20.876 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.876 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.876 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:59:20.876 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:49:27.780066+0000", 2024-03-30T13:59:20.876 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.2295000000000001e-05 2024-03-30T13:59:20.876 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.876 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:20.876 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.877 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.877 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.877 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:103274 8.5 8:aece27e6:::benchmark_data_smithi043_119832_object103273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-03-30T13:59:20.877 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:20.661642+0000", 2024-03-30T13:59:20.877 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 540.19161978099999, 2024-03-30T13:59:20.877 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.166393232, 2024-03-30T13:59:20.877 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:59:20.877 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:59:20.877 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:59:20.878 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:59:20.878 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:59:20.879 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:59:20.879 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 103274 2024-03-30T13:59:20.879 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.879 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:59:20.879 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.879 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:59:20.879 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.661642+0000", 2024-03-30T13:59:20.879 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.879 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.879 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.879 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:59:20.879 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.661642+0000", 2024-03-30T13:59:20.879 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.879 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.880 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.880 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:59:20.880 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.661643+0000", 2024-03-30T13:59:20.880 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5710000000000001e-06 2024-03-30T13:59:20.880 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.880 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.880 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:59:20.880 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.661691+0000", 2024-03-30T13:59:20.880 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7673000000000002e-05 2024-03-30T13:59:20.880 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.880 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.880 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:59:20.880 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.661726+0000", 2024-03-30T13:59:20.881 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.4582999999999997e-05 2024-03-30T13:59:20.881 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.881 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.881 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:59:20.881 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.661747+0000", 2024-03-30T13:59:20.881 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.158e-05 2024-03-30T13:59:20.881 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.881 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.881 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.881 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.693111+0000", 2024-03-30T13:59:20.881 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.031364013000000003 2024-03-30T13:59:20.881 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.881 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.881 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:59:20.881 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.693255+0000", 2024-03-30T13:59:20.882 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000144151 2024-03-30T13:59:20.882 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.882 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.882 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 4,7", 2024-03-30T13:59:20.882 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.695786+0000", 2024-03-30T13:59:20.882 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0025310889999999998 2024-03-30T13:59:20.882 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.882 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.882 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:59:20.883 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.706470+0000", 2024-03-30T13:59:20.883 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.010683727000000001 2024-03-30T13:59:20.883 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.883 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.883 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.883 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.720609+0000", 2024-03-30T13:59:20.883 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.014139073 2024-03-30T13:59:20.884 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.884 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.884 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.884 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.827914+0000", 2024-03-30T13:59:20.884 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.10730523 2024-03-30T13:59:20.884 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.884 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.884 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:59:20.884 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.827973+0000", 2024-03-30T13:59:20.885 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8773000000000001e-05 2024-03-30T13:59:20.885 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.885 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.885 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:59:20.885 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.828035+0000", 2024-03-30T13:59:20.885 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1768999999999998e-05 2024-03-30T13:59:20.885 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.885 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:20.885 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.885 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.885 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.885 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:103554 8.5 8:a257ada5:::benchmark_data_smithi043_119832_object103553:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-03-30T13:59:20.885 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:20.970350+0000", 2024-03-30T13:59:20.885 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 539.88291128900005, 2024-03-30T13:59:20.886 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.162027909, 2024-03-30T13:59:20.886 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:59:20.886 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:59:20.886 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:59:20.886 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:59:20.886 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:59:20.886 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:59:20.886 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 103554 2024-03-30T13:59:20.886 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.886 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:59:20.886 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.886 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:59:20.886 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.970350+0000", 2024-03-30T13:59:20.887 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.887 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.887 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.887 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:59:20.887 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.970350+0000", 2024-03-30T13:59:20.887 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.887 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.887 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.887 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:59:20.887 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.970351+0000", 2024-03-30T13:59:20.887 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2309999999999999e-06 2024-03-30T13:59:20.887 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.887 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.887 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:59:20.888 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.970380+0000", 2024-03-30T13:59:20.888 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.8237000000000001e-05 2024-03-30T13:59:20.888 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.888 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.888 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:59:20.888 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.970408+0000", 2024-03-30T13:59:20.888 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.8656000000000001e-05 2024-03-30T13:59:20.888 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.888 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.888 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:59:20.888 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:20.970420+0000", 2024-03-30T13:59:20.888 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1620000000000001e-05 2024-03-30T13:59:20.888 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.888 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.889 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.889 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.000822+0000", 2024-03-30T13:59:20.889 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.030402080000000001 2024-03-30T13:59:20.889 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.889 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.889 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:59:20.889 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.000961+0000", 2024-03-30T13:59:20.889 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013883499999999999 2024-03-30T13:59:20.889 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.889 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.889 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 4,7", 2024-03-30T13:59:20.889 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.001124+0000", 2024-03-30T13:59:20.889 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00016319899999999999 2024-03-30T13:59:20.889 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.889 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.890 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.890 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.032639+0000", 2024-03-30T13:59:20.890 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.031515133000000001 2024-03-30T13:59:20.890 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.890 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.890 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:59:20.890 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.032783+0000", 2024-03-30T13:59:20.890 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00014418800000000001 2024-03-30T13:59:20.891 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.891 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.891 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.891 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.132239+0000", 2024-03-30T13:59:20.891 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.099456000000000003 2024-03-30T13:59:20.891 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.892 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.892 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:59:20.892 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.132308+0000", 2024-03-30T13:59:20.892 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8670999999999994e-05 2024-03-30T13:59:20.892 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.892 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.893 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:59:20.893 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.132378+0000", 2024-03-30T13:59:20.893 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.0059000000000001e-05 2024-03-30T13:59:20.893 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.893 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:20.893 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.894 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.894 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.894 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:103636 8.5 8:a71576a6:::benchmark_data_smithi043_119832_object103635:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-03-30T13:59:20.894 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:21.071814+0000", 2024-03-30T13:59:20.894 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 539.78144797899995, 2024-03-30T13:59:20.895 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.19432798900000001, 2024-03-30T13:59:20.895 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:59:20.895 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:59:20.895 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:59:20.895 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:59:20.895 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:59:20.896 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:59:20.896 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 103636 2024-03-30T13:59:20.896 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.896 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:59:20.896 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.896 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:59:20.897 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.071814+0000", 2024-03-30T13:59:20.897 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.897 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.897 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.897 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:59:20.898 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.071814+0000", 2024-03-30T13:59:20.898 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.898 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.898 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.898 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:59:20.898 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.071815+0000", 2024-03-30T13:59:20.899 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3209999999999999e-06 2024-03-30T13:59:20.899 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.899 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.899 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:59:20.899 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.071849+0000", 2024-03-30T13:59:20.899 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.4428999999999999e-05 2024-03-30T13:59:20.900 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.900 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.900 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:59:20.900 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.071860+0000", 2024-03-30T13:59:20.900 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.049e-05 2024-03-30T13:59:20.900 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.901 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.901 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:59:20.901 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.071867+0000", 2024-03-30T13:59:20.901 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8140000000000003e-06 2024-03-30T13:59:20.901 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.901 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.902 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.902 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.145045+0000", 2024-03-30T13:59:20.902 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.073178178999999996 2024-03-30T13:59:20.902 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.902 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.903 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:59:20.903 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.145182+0000", 2024-03-30T13:59:20.903 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013687500000000001 2024-03-30T13:59:20.903 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.903 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.903 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 4,7", 2024-03-30T13:59:20.904 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.145356+0000", 2024-03-30T13:59:20.904 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000174587 2024-03-30T13:59:20.904 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.904 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.904 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:59:20.904 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.157621+0000", 2024-03-30T13:59:20.905 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.012264287 2024-03-30T13:59:20.905 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.905 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.905 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.905 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.192220+0000", 2024-03-30T13:59:20.905 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.034599824000000001 2024-03-30T13:59:20.905 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.905 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.906 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.906 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.266058+0000", 2024-03-30T13:59:20.906 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.073837557999999998 2024-03-30T13:59:20.906 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.906 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.907 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:59:20.907 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.266096+0000", 2024-03-30T13:59:20.907 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.7836999999999997e-05 2024-03-30T13:59:20.907 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.907 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.907 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:59:20.907 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:21.266142+0000", 2024-03-30T13:59:20.908 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.5788000000000003e-05 2024-03-30T13:59:20.908 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.908 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:20.908 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.908 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.908 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.908 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:106297 8.5 8:a6e04b40:::benchmark_data_smithi043_119832_object106296:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-30T13:59:20.909 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:23.807369+0000", 2024-03-30T13:59:20.909 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 537.045892396, 2024-03-30T13:59:20.909 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.20054035200000001, 2024-03-30T13:59:20.909 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:59:20.909 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:59:20.909 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:59:20.909 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:59:20.909 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:59:20.910 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:59:20.910 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 106297 2024-03-30T13:59:20.910 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.910 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:59:20.910 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.910 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:59:20.910 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.807369+0000", 2024-03-30T13:59:20.910 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.911 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.911 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.911 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:59:20.911 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.807369+0000", 2024-03-30T13:59:20.911 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.911 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.911 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.912 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:59:20.912 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.807370+0000", 2024-03-30T13:59:20.912 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.282e-06 2024-03-30T13:59:20.912 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.912 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.912 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:59:20.912 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.807404+0000", 2024-03-30T13:59:20.912 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.3379e-05 2024-03-30T13:59:20.912 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.912 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.913 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:59:20.913 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.807418+0000", 2024-03-30T13:59:20.913 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4031e-05 2024-03-30T13:59:20.913 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.913 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.913 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:59:20.914 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.807426+0000", 2024-03-30T13:59:20.914 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.977e-06 2024-03-30T13:59:20.914 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.914 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.914 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.914 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.849483+0000", 2024-03-30T13:59:20.914 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.042057630999999998 2024-03-30T13:59:20.914 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.915 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.915 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:59:20.915 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.849625+0000", 2024-03-30T13:59:20.915 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000141302 2024-03-30T13:59:20.915 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.915 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.915 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 4,7", 2024-03-30T13:59:20.915 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.849787+0000", 2024-03-30T13:59:20.915 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000162292 2024-03-30T13:59:20.915 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.915 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.915 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:59:20.915 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.855696+0000", 2024-03-30T13:59:20.916 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0059085830000000002 2024-03-30T13:59:20.916 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.916 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.916 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.916 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.864383+0000", 2024-03-30T13:59:20.916 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0086872870000000001 2024-03-30T13:59:20.916 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.916 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.916 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.916 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.007775+0000", 2024-03-30T13:59:20.916 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.14339182 2024-03-30T13:59:20.916 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.916 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.917 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:59:20.917 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.007843+0000", 2024-03-30T13:59:20.917 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8184999999999999e-05 2024-03-30T13:59:20.917 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.917 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.917 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:59:20.917 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.007909+0000", 2024-03-30T13:59:20.917 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6582999999999996e-05 2024-03-30T13:59:20.917 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.917 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:20.917 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.917 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.918 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.918 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:106311 8.5 8:a5146b1c:::benchmark_data_smithi043_119832_object106310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-30T13:59:20.918 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:23.820912+0000", 2024-03-30T13:59:20.918 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 537.03234983699997, 2024-03-30T13:59:20.918 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.18714597299999999, 2024-03-30T13:59:20.918 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:59:20.918 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:59:20.918 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:59:20.918 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:59:20.918 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:59:20.918 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:59:20.918 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 106311 2024-03-30T13:59:20.918 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.919 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:59:20.919 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.919 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:59:20.919 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.820912+0000", 2024-03-30T13:59:20.919 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.919 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.919 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.919 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:59:20.919 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.820912+0000", 2024-03-30T13:59:20.919 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.919 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.919 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.919 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:59:20.919 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.820913+0000", 2024-03-30T13:59:20.920 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.731e-06 2024-03-30T13:59:20.920 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.920 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.920 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:59:20.920 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.820942+0000", 2024-03-30T13:59:20.920 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.8503000000000001e-05 2024-03-30T13:59:20.920 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.920 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.920 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:59:20.920 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.820951+0000", 2024-03-30T13:59:20.920 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.9590000000000001e-06 2024-03-30T13:59:20.920 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.921 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.921 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:59:20.921 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.820958+0000", 2024-03-30T13:59:20.921 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.2429999999999999e-06 2024-03-30T13:59:20.921 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.921 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.921 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.921 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.852772+0000", 2024-03-30T13:59:20.921 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.031813461000000001 2024-03-30T13:59:20.921 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.921 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.921 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:59:20.921 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.852904+0000", 2024-03-30T13:59:20.921 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013242899999999999 2024-03-30T13:59:20.922 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.922 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.922 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 4,7", 2024-03-30T13:59:20.922 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.853073+0000", 2024-03-30T13:59:20.922 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00016875399999999999 2024-03-30T13:59:20.922 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.922 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.922 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:59:20.922 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.856285+0000", 2024-03-30T13:59:20.922 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.003212258 2024-03-30T13:59:20.923 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.923 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.923 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.923 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.864431+0000", 2024-03-30T13:59:20.923 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0081460860000000003 2024-03-30T13:59:20.923 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.923 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.923 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.923 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.007953+0000", 2024-03-30T13:59:20.923 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.14352188699999999 2024-03-30T13:59:20.923 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.923 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.923 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:59:20.924 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.008008+0000", 2024-03-30T13:59:20.924 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5322000000000003e-05 2024-03-30T13:59:20.924 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.924 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.924 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:59:20.924 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.008058+0000", 2024-03-30T13:59:20.924 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9339999999999999e-05 2024-03-30T13:59:20.924 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.924 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:20.924 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.924 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.924 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.924 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:106325 8.5 8:a3263736:::benchmark_data_smithi043_119832_object106324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-30T13:59:20.925 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:23.832037+0000", 2024-03-30T13:59:20.925 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 537.02122444600002, 2024-03-30T13:59:20.925 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.176300234, 2024-03-30T13:59:20.925 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:59:20.925 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:59:20.925 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:59:20.925 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:59:20.926 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:59:20.926 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:59:20.926 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 106325 2024-03-30T13:59:20.926 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.926 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:59:20.926 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.926 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:59:20.927 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.832037+0000", 2024-03-30T13:59:20.927 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.927 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.927 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.927 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:59:20.927 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.832037+0000", 2024-03-30T13:59:20.927 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.927 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.927 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.927 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:59:20.927 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.832039+0000", 2024-03-30T13:59:20.927 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4640000000000001e-06 2024-03-30T13:59:20.928 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.928 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.928 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:59:20.928 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.832092+0000", 2024-03-30T13:59:20.928 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3749999999999999e-05 2024-03-30T13:59:20.928 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.928 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.928 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:59:20.928 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.832119+0000", 2024-03-30T13:59:20.928 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.6639999999999999e-05 2024-03-30T13:59:20.929 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.929 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.929 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:59:20.929 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.832128+0000", 2024-03-30T13:59:20.929 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.7369999999999995e-06 2024-03-30T13:59:20.929 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.930 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.930 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.930 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.856537+0000", 2024-03-30T13:59:20.930 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.024409836000000001 2024-03-30T13:59:20.930 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.930 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.930 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:59:20.930 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.856675+0000", 2024-03-30T13:59:20.930 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000137286 2024-03-30T13:59:20.930 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.930 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.930 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 4,7", 2024-03-30T13:59:20.930 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.856831+0000", 2024-03-30T13:59:20.931 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000155847 2024-03-30T13:59:20.931 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.931 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.931 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:59:20.931 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.864443+0000", 2024-03-30T13:59:20.931 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0076122230000000004 2024-03-30T13:59:20.931 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.931 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.931 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.931 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.865999+0000", 2024-03-30T13:59:20.931 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0015556719999999999 2024-03-30T13:59:20.931 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.931 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.931 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.932 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.008234+0000", 2024-03-30T13:59:20.932 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.14223533599999999 2024-03-30T13:59:20.932 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.932 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.932 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:59:20.932 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.008287+0000", 2024-03-30T13:59:20.932 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3236000000000003e-05 2024-03-30T13:59:20.932 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.932 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.932 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:59:20.932 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.008337+0000", 2024-03-30T13:59:20.932 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0207000000000002e-05 2024-03-30T13:59:20.932 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.932 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:20.933 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.933 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.933 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.933 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:106331 8.5 8:a0900ba3:::benchmark_data_smithi043_119832_object106330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-30T13:59:20.933 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:50:23.834241+0000", 2024-03-30T13:59:20.933 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 537.01902027100004, 2024-03-30T13:59:20.933 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.17422878999999999, 2024-03-30T13:59:20.933 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:59:20.933 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:59:20.933 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:59:20.933 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:59:20.933 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:59:20.933 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:59:20.934 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 106331 2024-03-30T13:59:20.934 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.934 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:59:20.934 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.934 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:59:20.934 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.834241+0000", 2024-03-30T13:59:20.934 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.934 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.934 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.934 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:59:20.934 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.834241+0000", 2024-03-30T13:59:20.934 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.934 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.935 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.935 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:59:20.935 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.834243+0000", 2024-03-30T13:59:20.935 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.418e-06 2024-03-30T13:59:20.935 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.935 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.935 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:59:20.935 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.834300+0000", 2024-03-30T13:59:20.935 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7487e-05 2024-03-30T13:59:20.935 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.935 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.935 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:59:20.935 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.834312+0000", 2024-03-30T13:59:20.935 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1722999999999999e-05 2024-03-30T13:59:20.936 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.936 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.936 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:59:20.936 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.834319+0000", 2024-03-30T13:59:20.936 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.4279999999999999e-06 2024-03-30T13:59:20.936 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.936 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.936 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.936 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.860430+0000", 2024-03-30T13:59:20.936 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.026110209999999998 2024-03-30T13:59:20.936 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.936 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.936 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:59:20.936 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.860545+0000", 2024-03-30T13:59:20.937 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011513799999999999 2024-03-30T13:59:20.937 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.937 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.937 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 4,7", 2024-03-30T13:59:20.937 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.860694+0000", 2024-03-30T13:59:20.937 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00014947300000000001 2024-03-30T13:59:20.937 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.937 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.937 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:59:20.937 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.866011+0000", 2024-03-30T13:59:20.937 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.005317291 2024-03-30T13:59:20.937 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.937 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.938 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.938 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:23.879751+0000", 2024-03-30T13:59:20.938 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.013739711999999999 2024-03-30T13:59:20.938 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.938 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.938 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.938 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.008377+0000", 2024-03-30T13:59:20.938 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.12862538700000001 2024-03-30T13:59:20.938 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.938 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.938 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:59:20.938 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.008427+0000", 2024-03-30T13:59:20.938 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0152999999999999e-05 2024-03-30T13:59:20.939 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.939 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.939 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:59:20.939 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:50:24.008470+0000", 2024-03-30T13:59:20.939 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.337e-05 2024-03-30T13:59:20.939 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.939 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:20.940 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.940 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.940 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.940 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:250162 8.9 8:9755cc2d:::benchmark_data_smithi043_119832_object119808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:59:20.940 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:19.661245+0000", 2024-03-30T13:59:20.940 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 421.19201666399999, 2024-03-30T13:59:20.940 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.18734693499999999, 2024-03-30T13:59:20.940 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:59:20.940 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:59:20.941 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:59:20.941 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:59:20.941 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:59:20.941 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:59:20.941 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 250162 2024-03-30T13:59:20.941 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.941 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:59:20.941 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.942 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:59:20.942 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.661245+0000", 2024-03-30T13:59:20.942 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.942 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.942 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.942 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:59:20.942 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.661245+0000", 2024-03-30T13:59:20.942 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.943 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.943 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.943 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:59:20.943 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.661245+0000", 2024-03-30T13:59:20.943 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5599999999999995e-07 2024-03-30T13:59:20.943 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.944 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.944 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:59:20.944 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.661249+0000", 2024-03-30T13:59:20.944 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.382e-06 2024-03-30T13:59:20.944 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.944 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.945 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:59:20.945 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.661259+0000", 2024-03-30T13:59:20.945 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0178e-05 2024-03-30T13:59:20.945 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.945 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.945 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:59:20.945 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.661265+0000", 2024-03-30T13:59:20.946 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0190000000000003e-06 2024-03-30T13:59:20.946 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.946 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.946 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.946 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.668796+0000", 2024-03-30T13:59:20.946 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0075311880000000003 2024-03-30T13:59:20.946 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.947 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.947 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:59:20.947 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.669080+0000", 2024-03-30T13:59:20.947 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00028389600000000001 2024-03-30T13:59:20.947 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.947 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.947 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,5", 2024-03-30T13:59:20.947 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.669467+0000", 2024-03-30T13:59:20.948 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00038705100000000002 2024-03-30T13:59:20.948 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.948 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.948 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:59:20.948 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.676682+0000", 2024-03-30T13:59:20.948 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0072146010000000002 2024-03-30T13:59:20.948 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.948 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.949 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.949 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.683310+0000", 2024-03-30T13:59:20.949 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0066280150000000001 2024-03-30T13:59:20.949 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.949 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.949 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.949 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.848485+0000", 2024-03-30T13:59:20.950 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.16517526699999999 2024-03-30T13:59:20.950 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.950 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.950 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:59:20.950 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.848544+0000", 2024-03-30T13:59:20.950 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9209000000000002e-05 2024-03-30T13:59:20.950 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.950 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.950 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:59:20.950 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.848592+0000", 2024-03-30T13:59:20.950 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7573e-05 2024-03-30T13:59:20.950 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.950 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:20.951 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.951 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.951 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.951 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:250165 8.9 8:97c487c5:::benchmark_data_smithi043_119832_object119811:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:59:20.951 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:19.665737+0000", 2024-03-30T13:59:20.951 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 421.18752496500002, 2024-03-30T13:59:20.951 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.186351143, 2024-03-30T13:59:20.951 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:59:20.951 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:59:20.951 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:59:20.951 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:59:20.951 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:59:20.951 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:59:20.951 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 250165 2024-03-30T13:59:20.952 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.952 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:59:20.952 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.952 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:59:20.952 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.665737+0000", 2024-03-30T13:59:20.952 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.952 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.952 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.952 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:59:20.952 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.665737+0000", 2024-03-30T13:59:20.952 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.952 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.952 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.952 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:59:20.952 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.665738+0000", 2024-03-30T13:59:20.953 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.153e-06 2024-03-30T13:59:20.953 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.953 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.953 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:59:20.953 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.665742+0000", 2024-03-30T13:59:20.953 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.4710000000000001e-06 2024-03-30T13:59:20.953 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.953 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.953 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:59:20.953 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.665755+0000", 2024-03-30T13:59:20.953 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.306e-05 2024-03-30T13:59:20.953 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.953 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.954 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:59:20.954 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.665762+0000", 2024-03-30T13:59:20.954 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.2110000000000001e-06 2024-03-30T13:59:20.954 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.954 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.954 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.954 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.673253+0000", 2024-03-30T13:59:20.954 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0074901059999999998 2024-03-30T13:59:20.954 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.954 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.954 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:59:20.954 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.673389+0000", 2024-03-30T13:59:20.954 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013632 2024-03-30T13:59:20.954 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.955 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.955 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,5", 2024-03-30T13:59:20.955 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.673545+0000", 2024-03-30T13:59:20.955 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000156229 2024-03-30T13:59:20.955 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.955 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.955 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:59:20.955 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.677858+0000", 2024-03-30T13:59:20.955 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0043133470000000004 2024-03-30T13:59:20.955 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.955 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.955 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.955 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.683392+0000", 2024-03-30T13:59:20.955 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0055340340000000002 2024-03-30T13:59:20.956 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.956 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.956 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:20.957 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:59:20.958 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.958 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.851943+0000", 2024-03-30T13:59:20.958 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.16855024900000001 2024-03-30T13:59:20.958 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.958 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.958 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:59:20.958 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.852026+0000", 2024-03-30T13:59:20.959 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.3486000000000007e-05 2024-03-30T13:59:20.959 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.959 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.959 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:59:20.959 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.852088+0000", 2024-03-30T13:59:20.959 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1476999999999994e-05 2024-03-30T13:59:20.959 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.959 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:20.959 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.959 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.959 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.959 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:250174 8.9 8:93ced427:::benchmark_data_smithi043_119832_object119820:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:59:20.959 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:19.672736+0000", 2024-03-30T13:59:20.959 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 421.18052566, 2024-03-30T13:59:20.960 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.17950729900000001, 2024-03-30T13:59:20.960 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:59:20.960 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:59:20.960 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:59:20.960 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:59:20.960 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:59:20.960 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:59:20.960 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 250174 2024-03-30T13:59:20.960 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.960 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:59:20.960 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.960 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:59:20.960 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.672736+0000", 2024-03-30T13:59:20.961 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.961 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.961 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.961 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:59:20.961 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.672736+0000", 2024-03-30T13:59:20.961 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.961 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.961 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.961 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:59:20.961 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.672737+0000", 2024-03-30T13:59:20.961 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.469e-06 2024-03-30T13:59:20.961 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.961 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.961 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:59:20.962 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.672742+0000", 2024-03-30T13:59:20.962 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.848e-06 2024-03-30T13:59:20.962 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.962 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.962 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:59:20.962 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.672757+0000", 2024-03-30T13:59:20.962 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5129e-05 2024-03-30T13:59:20.962 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.962 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.962 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:59:20.962 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.672765+0000", 2024-03-30T13:59:20.962 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8909999999999995e-06 2024-03-30T13:59:20.962 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.962 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.963 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.963 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.677945+0000", 2024-03-30T13:59:20.963 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0051795360000000002 2024-03-30T13:59:20.963 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.963 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.963 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:59:20.963 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.678414+0000", 2024-03-30T13:59:20.963 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00046906500000000001 2024-03-30T13:59:20.963 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.963 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.963 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,5", 2024-03-30T13:59:20.963 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.678982+0000", 2024-03-30T13:59:20.963 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00056849199999999998 2024-03-30T13:59:20.963 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.964 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.964 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:59:20.964 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.688439+0000", 2024-03-30T13:59:20.964 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0094570650000000006 2024-03-30T13:59:20.964 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.964 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.964 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.964 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.689525+0000", 2024-03-30T13:59:20.964 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001086072 2024-03-30T13:59:20.964 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.964 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.964 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.964 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.852137+0000", 2024-03-30T13:59:20.964 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.16261124199999999 2024-03-30T13:59:20.965 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.965 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.965 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:59:20.965 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.852194+0000", 2024-03-30T13:59:20.965 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.728e-05 2024-03-30T13:59:20.965 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.965 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.965 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:59:20.965 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.852243+0000", 2024-03-30T13:59:20.965 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9209999999999998e-05 2024-03-30T13:59:20.965 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.965 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:20.965 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.965 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.966 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.966 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.24977.0:250180 8.9 8:9d13047d:::benchmark_data_smithi043_119832_object119826:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-30T13:59:20.966 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:52:19.680512+0000", 2024-03-30T13:59:20.966 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 421.17274993500001, 2024-03-30T13:59:20.966 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.172133181, 2024-03-30T13:59:20.966 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:59:20.966 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:59:20.966 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:59:20.966 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:59:20.966 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.24977", 2024-03-30T13:59:20.966 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2502350755", 2024-03-30T13:59:20.966 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 250180 2024-03-30T13:59:20.966 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.967 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:59:20.967 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.967 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:59:20.967 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.680512+0000", 2024-03-30T13:59:20.967 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.967 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.967 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.967 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:59:20.967 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.680512+0000", 2024-03-30T13:59:20.967 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.967 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.967 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.967 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:59:20.967 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.680513+0000", 2024-03-30T13:59:20.967 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.209e-06 2024-03-30T13:59:20.968 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.968 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.968 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:59:20.968 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.680517+0000", 2024-03-30T13:59:20.968 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1069999999999998e-06 2024-03-30T13:59:20.968 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.968 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.968 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:59:20.968 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.680531+0000", 2024-03-30T13:59:20.968 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4219999999999999e-05 2024-03-30T13:59:20.968 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.968 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.968 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:59:20.968 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.680539+0000", 2024-03-30T13:59:20.969 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.9300000000000003e-06 2024-03-30T13:59:20.969 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.969 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.969 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.969 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.683421+0000", 2024-03-30T13:59:20.969 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0028816990000000002 2024-03-30T13:59:20.969 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.969 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.969 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:59:20.969 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.683549+0000", 2024-03-30T13:59:20.969 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000128169 2024-03-30T13:59:20.969 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.969 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.969 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,5", 2024-03-30T13:59:20.970 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.683669+0000", 2024-03-30T13:59:20.970 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012008800000000001 2024-03-30T13:59:20.970 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.970 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.970 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:59:20.970 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.688764+0000", 2024-03-30T13:59:20.970 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0050948030000000002 2024-03-30T13:59:20.970 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.970 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.970 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.970 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.689664+0000", 2024-03-30T13:59:20.970 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00090005100000000002 2024-03-30T13:59:20.970 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.971 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.971 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:20.971 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.852537+0000", 2024-03-30T13:59:20.971 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.16287290800000001 2024-03-30T13:59:20.971 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.971 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.971 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:59:20.971 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.852595+0000", 2024-03-30T13:59:20.971 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7784000000000001e-05 2024-03-30T13:59:20.971 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.971 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.971 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:59:20.971 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:52:19.852645+0000", 2024-03-30T13:59:20.971 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0213000000000002e-05 2024-03-30T13:59:20.972 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.972 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:20.972 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.972 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.972 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.972 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14451.0:2642 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94764809509632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-03-30T13:59:20.972 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:35.493028+0000", 2024-03-30T13:59:20.972 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 345.360233365, 2024-03-30T13:59:20.972 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.361405609, 2024-03-30T13:59:20.972 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:59:20.972 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:59:20.972 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:59:20.972 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:59:20.973 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14451", 2024-03-30T13:59:20.973 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T13:59:20.973 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 2642 2024-03-30T13:59:20.973 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.973 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:59:20.973 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.973 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:59:20.973 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493028+0000", 2024-03-30T13:59:20.973 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.973 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.973 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.973 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:59:20.973 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493028+0000", 2024-03-30T13:59:20.973 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.974 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.974 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.974 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:59:20.974 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493029+0000", 2024-03-30T13:59:20.974 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1600000000000001e-07 2024-03-30T13:59:20.974 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.974 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.974 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:59:20.974 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493033+0000", 2024-03-30T13:59:20.974 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3080000000000001e-06 2024-03-30T13:59:20.974 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.974 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.974 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:59:20.975 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493040+0000", 2024-03-30T13:59:20.975 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.3139999999999998e-06 2024-03-30T13:59:20.975 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.975 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.975 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:59:20.975 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493047+0000", 2024-03-30T13:59:20.975 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8889999999999996e-06 2024-03-30T13:59:20.975 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.975 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.975 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.975 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493230+0000", 2024-03-30T13:59:20.975 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00018283500000000001 2024-03-30T13:59:20.975 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.976 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.976 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T13:59:20.976 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493238+0000", 2024-03-30T13:59:20.976 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8120000000000008e-06 2024-03-30T13:59:20.976 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.976 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.976 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.976 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.846292+0000", 2024-03-30T13:59:20.976 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.35305440700000001 2024-03-30T13:59:20.976 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.976 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.976 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T13:59:20.976 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.846304+0000", 2024-03-30T13:59:20.976 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2001000000000001e-05 2024-03-30T13:59:20.977 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.977 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.977 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.977 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.854172+0000", 2024-03-30T13:59:20.977 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.00786771 2024-03-30T13:59:20.977 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.977 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.977 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:59:20.977 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.854348+0000", 2024-03-30T13:59:20.977 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00017568299999999999 2024-03-30T13:59:20.977 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.977 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.977 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:59:20.977 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.854434+0000", 2024-03-30T13:59:20.977 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5933999999999994e-05 2024-03-30T13:59:20.978 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.978 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:20.978 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.978 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.978 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.978 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14451.0:2643 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch ping cookie 94764813905280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-03-30T13:59:20.978 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:35.493069+0000", 2024-03-30T13:59:20.978 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 345.36019275699999, 2024-03-30T13:59:20.978 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.37558174100000002, 2024-03-30T13:59:20.978 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:59:20.978 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:59:20.978 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:59:20.978 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:59:20.979 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14451", 2024-03-30T13:59:20.979 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T13:59:20.979 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 2643 2024-03-30T13:59:20.979 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.979 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:59:20.979 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.979 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:59:20.979 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493069+0000", 2024-03-30T13:59:20.979 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.979 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.979 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.979 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:59:20.979 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493069+0000", 2024-03-30T13:59:20.979 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.980 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.980 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.980 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:59:20.980 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493069+0000", 2024-03-30T13:59:20.980 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.8799999999999998e-07 2024-03-30T13:59:20.980 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.980 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.980 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:59:20.980 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493072+0000", 2024-03-30T13:59:20.980 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.418e-06 2024-03-30T13:59:20.980 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.980 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.980 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:59:20.980 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493081+0000", 2024-03-30T13:59:20.981 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4010000000000003e-06 2024-03-30T13:59:20.981 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.981 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.981 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:59:20.981 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493086+0000", 2024-03-30T13:59:20.981 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1379999999999998e-06 2024-03-30T13:59:20.981 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.981 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.981 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.981 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493171+0000", 2024-03-30T13:59:20.981 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.4764000000000004e-05 2024-03-30T13:59:20.981 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.981 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.981 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T13:59:20.982 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493185+0000", 2024-03-30T13:59:20.982 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3919e-05 2024-03-30T13:59:20.982 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.982 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.982 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.982 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.865616+0000", 2024-03-30T13:59:20.982 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.37243163200000001 2024-03-30T13:59:20.982 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.982 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.982 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for degraded object", 2024-03-30T13:59:20.982 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.866216+0000", 2024-03-30T13:59:20.982 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00060001100000000001 2024-03-30T13:59:20.982 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.983 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.983 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.983 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.868509+0000", 2024-03-30T13:59:20.983 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0022922469999999999 2024-03-30T13:59:20.983 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.983 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.983 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:59:20.983 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.868564+0000", 2024-03-30T13:59:20.983 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4864999999999997e-05 2024-03-30T13:59:20.983 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.983 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.983 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:59:20.983 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.868650+0000", 2024-03-30T13:59:20.983 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6958000000000003e-05 2024-03-30T13:59:20.984 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.984 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:20.984 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.984 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.984 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.984 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14451.0:2647 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94764809509632 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:59:20.984 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:35.493415+0000", 2024-03-30T13:59:20.984 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 345.35984654800001, 2024-03-30T13:59:20.984 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.363426126, 2024-03-30T13:59:20.984 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:59:20.984 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:59:20.984 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:59:20.984 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:59:20.984 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14451", 2024-03-30T13:59:20.985 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T13:59:20.985 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 2647 2024-03-30T13:59:20.985 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.985 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:59:20.985 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.985 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:59:20.985 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493415+0000", 2024-03-30T13:59:20.985 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.985 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.985 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.985 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:59:20.985 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493415+0000", 2024-03-30T13:59:20.985 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.986 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.986 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.986 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:59:20.986 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493416+0000", 2024-03-30T13:59:20.986 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1399999999999996e-07 2024-03-30T13:59:20.986 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.986 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.986 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:59:20.986 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493420+0000", 2024-03-30T13:59:20.986 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.9389999999999999e-06 2024-03-30T13:59:20.986 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.986 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.987 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:59:20.987 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493434+0000", 2024-03-30T13:59:20.987 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4347e-05 2024-03-30T13:59:20.987 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.987 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.987 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:59:20.987 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493441+0000", 2024-03-30T13:59:20.987 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6120000000000004e-06 2024-03-30T13:59:20.987 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.987 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.987 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.987 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493506+0000", 2024-03-30T13:59:20.987 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5545999999999999e-05 2024-03-30T13:59:20.987 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.988 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.988 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T13:59:20.988 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493516+0000", 2024-03-30T13:59:20.988 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.9320000000000008e-06 2024-03-30T13:59:20.988 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.988 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.988 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.988 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.846331+0000", 2024-03-30T13:59:20.988 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.35281492800000003 2024-03-30T13:59:20.988 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.988 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.988 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T13:59:20.988 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.846339+0000", 2024-03-30T13:59:20.988 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.2160000000000006e-06 2024-03-30T13:59:20.989 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.989 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.989 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.989 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.855384+0000", 2024-03-30T13:59:20.989 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0090448649999999 2024-03-30T13:59:20.989 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.989 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.989 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:59:20.989 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.855471+0000", 2024-03-30T13:59:20.989 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.7027999999999994e-05 2024-03-30T13:59:20.989 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.989 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.989 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:59:20.990 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.856841+0000", 2024-03-30T13:59:20.990 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001369999 2024-03-30T13:59:20.990 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.990 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:20.990 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.990 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.990 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.990 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14451.0:2648 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94764813905280 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:59:20.990 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:35.493457+0000", 2024-03-30T13:59:20.990 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 345.359804919, 2024-03-30T13:59:20.990 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.37619107499999999, 2024-03-30T13:59:20.990 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:59:20.991 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:59:20.991 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:59:20.991 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:59:20.991 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14451", 2024-03-30T13:59:20.991 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T13:59:20.991 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 2648 2024-03-30T13:59:20.991 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.991 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:59:20.991 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.991 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:59:20.991 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493457+0000", 2024-03-30T13:59:20.991 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.991 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.991 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.992 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:59:20.992 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493457+0000", 2024-03-30T13:59:20.992 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.992 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.992 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.992 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:59:20.992 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493457+0000", 2024-03-30T13:59:20.992 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.7399999999999999e-07 2024-03-30T13:59:20.992 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.992 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.992 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:59:20.992 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493459+0000", 2024-03-30T13:59:20.992 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.1490000000000001e-06 2024-03-30T13:59:20.992 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.993 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.993 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:59:20.993 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493517+0000", 2024-03-30T13:59:20.993 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7615999999999999e-05 2024-03-30T13:59:20.993 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.993 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.993 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:59:20.993 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493524+0000", 2024-03-30T13:59:20.993 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.464e-06 2024-03-30T13:59:20.993 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.993 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.993 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.993 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493564+0000", 2024-03-30T13:59:20.993 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.9317999999999999e-05 2024-03-30T13:59:20.994 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.994 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.994 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T13:59:20.994 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493575+0000", 2024-03-30T13:59:20.994 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1892999999999999e-05 2024-03-30T13:59:20.994 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.994 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.994 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.994 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.866266+0000", 2024-03-30T13:59:20.994 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.37269074600000002 2024-03-30T13:59:20.994 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.994 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.994 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for degraded object", 2024-03-30T13:59:20.995 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.866320+0000", 2024-03-30T13:59:20.995 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3650999999999998e-05 2024-03-30T13:59:20.995 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.995 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.995 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.995 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.868673+0000", 2024-03-30T13:59:20.995 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0023534929999999999 2024-03-30T13:59:20.995 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.995 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.995 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:59:20.995 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.868717+0000", 2024-03-30T13:59:20.995 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.4190000000000002e-05 2024-03-30T13:59:20.995 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.995 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.996 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:59:20.996 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.869648+0000", 2024-03-30T13:59:20.996 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00093018099999999998 2024-03-30T13:59:20.996 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.996 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:20.996 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:20.996 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.996 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.996 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14451.0:2650 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94764809515392 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T13:59:20.996 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:35.493539+0000", 2024-03-30T13:59:20.996 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 345.35972228100002, 2024-03-30T13:59:20.996 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.39050606900000001, 2024-03-30T13:59:20.996 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:59:20.996 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:59:20.997 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:59:20.997 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:59:20.997 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14451", 2024-03-30T13:59:20.997 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T13:59:20.997 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 2650 2024-03-30T13:59:20.997 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.997 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:59:20.997 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.997 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:59:20.997 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493539+0000", 2024-03-30T13:59:20.997 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.997 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.997 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.997 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:59:20.998 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493539+0000", 2024-03-30T13:59:20.998 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:20.998 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.998 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.998 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:59:20.998 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493540+0000", 2024-03-30T13:59:20.998 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.6000000000000003e-07 2024-03-30T13:59:20.998 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.998 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.998 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:59:20.998 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493544+0000", 2024-03-30T13:59:20.998 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1439999999999996e-06 2024-03-30T13:59:20.998 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.998 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.999 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:59:20.999 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493552+0000", 2024-03-30T13:59:20.999 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8359999999999992e-06 2024-03-30T13:59:20.999 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.999 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.999 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:59:20.999 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493557+0000", 2024-03-30T13:59:20.999 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3299999999999998e-06 2024-03-30T13:59:20.999 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:20.999 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:20.999 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:20.999 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493605+0000", 2024-03-30T13:59:20.999 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7611999999999997e-05 2024-03-30T13:59:20.999 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.000 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.000 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T13:59:21.000 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493613+0000", 2024-03-30T13:59:21.000 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.6229999999999997e-06 2024-03-30T13:59:21.000 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.000 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.000 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:21.000 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.862721+0000", 2024-03-30T13:59:21.000 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.369108504 2024-03-30T13:59:21.000 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.000 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.000 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for degraded object", 2024-03-30T13:59:21.000 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.863669+0000", 2024-03-30T13:59:21.001 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00094758500000000003 2024-03-30T13:59:21.001 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.001 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.001 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:21.001 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.876266+0000", 2024-03-30T13:59:21.001 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.012597486 2024-03-30T13:59:21.001 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.001 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.001 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:59:21.001 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.876360+0000", 2024-03-30T13:59:21.001 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4313000000000003e-05 2024-03-30T13:59:21.001 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.001 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.001 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:59:21.002 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.884045+0000", 2024-03-30T13:59:21.002 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0076848760000000002 2024-03-30T13:59:21.002 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:21.002 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:21.002 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:21.002 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.002 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.002 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14451.0:2652 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94764809509632 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-03-30T13:59:21.002 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:35.852719+0000", 2024-03-30T13:59:21.002 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 345.00054253899998, 2024-03-30T13:59:21.002 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.004857439, 2024-03-30T13:59:21.002 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:59:21.002 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:59:21.003 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T13:59:21.003 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:59:21.003 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14451", 2024-03-30T13:59:21.003 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T13:59:21.003 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 2652 2024-03-30T13:59:21.003 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.003 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:59:21.003 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.003 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:59:21.003 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.852719+0000", 2024-03-30T13:59:21.003 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:21.003 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.003 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.004 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:59:21.004 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.852719+0000", 2024-03-30T13:59:21.004 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:21.004 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.004 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.004 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:59:21.004 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.852720+0000", 2024-03-30T13:59:21.004 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2160000000000001e-06 2024-03-30T13:59:21.004 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.004 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.004 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:59:21.004 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.852724+0000", 2024-03-30T13:59:21.004 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.563e-06 2024-03-30T13:59:21.004 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.005 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.005 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:59:21.005 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.852733+0000", 2024-03-30T13:59:21.005 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.0909999999999999e-06 2024-03-30T13:59:21.005 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.005 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.005 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:59:21.005 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.852743+0000", 2024-03-30T13:59:21.005 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0006999999999999e-05 2024-03-30T13:59:21.005 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.005 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.005 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:21.005 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.853116+0000", 2024-03-30T13:59:21.005 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00037315500000000001 2024-03-30T13:59:21.006 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.006 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.006 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T13:59:21.006 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.853128+0000", 2024-03-30T13:59:21.006 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2007999999999999e-05 2024-03-30T13:59:21.006 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.006 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.006 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:21.006 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.855865+0000", 2024-03-30T13:59:21.006 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0027372569999999 2024-03-30T13:59:21.006 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.006 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.006 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:59:21.007 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.855917+0000", 2024-03-30T13:59:21.007 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1595000000000003e-05 2024-03-30T13:59:21.007 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.007 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.007 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:59:21.007 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.857576+0000", 2024-03-30T13:59:21.007 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0016595469999999999 2024-03-30T13:59:21.007 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:21.007 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:21.007 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:21.007 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.007 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.007 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:118624 10.a 10:58dfbf2a:::benchmark_data_smithi043_169264_object118623:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-30T13:59:21.007 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:58:10.452890+0000", 2024-03-30T13:59:21.008 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 70.400371114999999, 2024-03-30T13:59:21.008 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.14768361399999999, 2024-03-30T13:59:21.008 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T13:59:21.008 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T13:59:21.008 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T13:59:21.008 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T13:59:21.008 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T13:59:21.008 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T13:59:21.008 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 118624 2024-03-30T13:59:21.008 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.008 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T13:59:21.008 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.008 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T13:59:21.009 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.452890+0000", 2024-03-30T13:59:21.009 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:21.009 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.009 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.009 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T13:59:21.009 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.452890+0000", 2024-03-30T13:59:21.009 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T13:59:21.009 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.009 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.009 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T13:59:21.009 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.452891+0000", 2024-03-30T13:59:21.009 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.44e-07 2024-03-30T13:59:21.009 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.010 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.010 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T13:59:21.010 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.452918+0000", 2024-03-30T13:59:21.010 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.6781e-05 2024-03-30T13:59:21.010 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.010 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.010 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T13:59:21.010 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.452926+0000", 2024-03-30T13:59:21.010 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8220000000000007e-06 2024-03-30T13:59:21.010 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.010 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.010 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T13:59:21.010 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.452931+0000", 2024-03-30T13:59:21.010 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.609e-06 2024-03-30T13:59:21.011 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.011 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.011 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T13:59:21.011 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.578360+0000", 2024-03-30T13:59:21.011 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.125428753 2024-03-30T13:59:21.011 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.011 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.011 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T13:59:21.011 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.578515+0000", 2024-03-30T13:59:21.011 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000155291 2024-03-30T13:59:21.011 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.011 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.011 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0", 2024-03-30T13:59:21.011 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.578675+0000", 2024-03-30T13:59:21.012 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00015995699999999999 2024-03-30T13:59:21.012 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.012 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.012 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T13:59:21.012 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.581572+0000", 2024-03-30T13:59:21.012 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0028971449999999998 2024-03-30T13:59:21.012 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.012 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.012 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T13:59:21.012 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.600202+0000", 2024-03-30T13:59:21.012 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.018629521999999999 2024-03-30T13:59:21.012 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.012 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.012 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T13:59:21.013 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.600487+0000", 2024-03-30T13:59:21.013 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00028477500000000002 2024-03-30T13:59:21.013 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T13:59:21.013 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T13:59:21.013 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T13:59:21.013 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.600574+0000", 2024-03-30T13:59:21.013 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.7315e-05 2024-03-30T13:59:21.013 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:21.013 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:21.013 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:21.013 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T13:59:21.013 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T13:59:21.013 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:59:21.014 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:59:21.033 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:20 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:21.033 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:20 smithi113 ceph-mon[34341]: pgmap v734: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 723 B/s rd, 0 B/s wr, 1.50k op/s; 43113/335823 objects degraded (12.838%); 28413/335823 objects misplaced (8.461%); 38 MiB/s, 601 objects/s recovering 2024-03-30T13:59:21.033 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:20 smithi113 ceph-mon[34341]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:59:21.033 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:20 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:59:21.033 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:20 smithi113 ceph-mon[34341]: osdmap e337: 8 total, 7 up, 7 in 2024-03-30T13:59:21.033 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:20 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:59:21.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:21.057 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:21.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:21.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:20 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:21.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:20 smithi043 ceph-mon[29963]: pgmap v734: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 723 B/s rd, 0 B/s wr, 1.50k op/s; 43113/335823 objects degraded (12.838%); 28413/335823 objects misplaced (8.461%); 38 MiB/s, 601 objects/s recovering 2024-03-30T13:59:21.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:20 smithi043 ceph-mon[29963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:59:21.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:20 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:59:21.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:20 smithi043 ceph-mon[29963]: osdmap e337: 8 total, 7 up, 7 in 2024-03-30T13:59:21.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:20 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:59:21.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:20 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:21.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:20 smithi043 ceph-mon[42099]: pgmap v734: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 723 B/s rd, 0 B/s wr, 1.50k op/s; 43113/335823 objects degraded (12.838%); 28413/335823 objects misplaced (8.461%); 38 MiB/s, 601 objects/s recovering 2024-03-30T13:59:21.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:20 smithi043 ceph-mon[42099]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:59:21.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:20 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T13:59:21.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:20 smithi043 ceph-mon[42099]: osdmap e337: 8 total, 7 up, 7 in 2024-03-30T13:59:21.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:20 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:59:21.158 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:21.201 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:21.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:21.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:21.302 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:21.337 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:21.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:21.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:21.438 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:21.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:21.478 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:21.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:21.579 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:21.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:21.613 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:21.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:21.715 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:21.756 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:21.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:21.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:21.829 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 13:59:21 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=sqlstore.transactions t=2024-03-30T13:59:21.505546702Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-30T13:59:21.858 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:21.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:21.912 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:21.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:21.957 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-30T13:59:22.013 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:22.048 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:22.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:22.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:22.150 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:22.185 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:22.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:22.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:22.287 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:22.322 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:22.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:22.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:22.394 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:59:22.424 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:22.475 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:22.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:22.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:22.577 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:22.625 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:22.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:22.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:22.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:22 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:22.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:22 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:22.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:22 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:22.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:22 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:22.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:22 smithi043 ceph-mon[29963]: pgmap v736: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.28k op/s; 41786/329457 objects degraded (12.683%); 27541/329457 objects misplaced (8.360%); 29 MiB/s, 456 objects/s recovering 2024-03-30T13:59:22.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:22 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:22.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:22 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:22.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:22 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:22.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:22 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:22.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:22 smithi043 ceph-mon[42099]: pgmap v736: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.28k op/s; 41786/329457 objects degraded (12.683%); 27541/329457 objects misplaced (8.360%); 29 MiB/s, 456 objects/s recovering 2024-03-30T13:59:22.690 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:22 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:22.691 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:22 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:22.691 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:22 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:22.691 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:22 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:22.691 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:22 smithi113 ceph-mon[34341]: pgmap v736: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.28k op/s; 41786/329457 objects degraded (12.683%); 27541/329457 objects misplaced (8.360%); 29 MiB/s, 456 objects/s recovering 2024-03-30T13:59:22.727 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:22.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:22.766 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:22.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:22.867 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:22.913 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:22.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:22.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:23.015 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:59:23.016 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:23.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:23.055 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:23.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:23.157 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:23.191 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:23.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:23.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:23.293 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:23.345 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:23.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:23.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:23.447 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:23.468 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:59:23.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:23.515 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:23.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:23.537 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T13:59:23.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:23 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 41786/329457 objects degraded (12.683%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:23.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:23 smithi043 ceph-mon[29963]: osdmap e338: 8 total, 7 up, 7 in 2024-03-30T13:59:23.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:23 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:59:23.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:23 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:59:23.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:23 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:23.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:23 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 41786/329457 objects degraded (12.683%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:23.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:23 smithi043 ceph-mon[42099]: osdmap e338: 8 total, 7 up, 7 in 2024-03-30T13:59:23.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:23 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:59:23.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:23 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:59:23.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:23 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:23.616 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:23.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:23.653 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:23.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:23.692 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:23 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 41786/329457 objects degraded (12.683%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:23.693 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:23 smithi113 ceph-mon[34341]: osdmap e338: 8 total, 7 up, 7 in 2024-03-30T13:59:23.693 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:23 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:59:23.693 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:23 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:59:23.693 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:23 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:23.754 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:23.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:23.809 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:23.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:23.910 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:23.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:23.951 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:23.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:24.052 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:24.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:24.095 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:24.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:24.196 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:24.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:24.245 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:24.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:24.346 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:24.401 INFO:teuthology.orchestra.run.smithi043.stderr:set 4.1a pg_upmap mapping to [7,3,2] 2024-03-30T13:59:24.411 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T13:59:24.411 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [], 2024-03-30T13:59:24.411 INFO:teuthology.orchestra.run.smithi113.stdout: "num_ops": 0 2024-03-30T13:59:24.411 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T13:59:24.413 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:24.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:24.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:24.515 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:24.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:24.559 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:24.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:24.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:24 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2931045389' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.1a", "id": [7, 3, 2]}]: dispatch 2024-03-30T13:59:24.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:24 smithi043 ceph-mon[29963]: pgmap v738: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 843 B/s rd, 0 B/s wr, 1.27k op/s; 40515/326982 objects degraded (12.391%); 26716/326982 objects misplaced (8.170%); 32 MiB/s, 514 objects/s recovering 2024-03-30T13:59:24.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:24 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2931045389' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.1a", "id": [7, 3, 2]}]: dispatch 2024-03-30T13:59:24.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:24 smithi043 ceph-mon[42099]: pgmap v738: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 843 B/s rd, 0 B/s wr, 1.27k op/s; 40515/326982 objects degraded (12.391%); 26716/326982 objects misplaced (8.170%); 32 MiB/s, 514 objects/s recovering 2024-03-30T13:59:24.660 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:24.698 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:24.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:24.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:24.716 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:24 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2931045389' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.1a", "id": [7, 3, 2]}]: dispatch 2024-03-30T13:59:24.716 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:24 smithi113 ceph-mon[34341]: pgmap v738: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 843 B/s rd, 0 B/s wr, 1.27k op/s; 40515/326982 objects degraded (12.391%); 26716/326982 objects misplaced (8.170%); 32 MiB/s, 514 objects/s recovering 2024-03-30T13:59:24.799 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:24.836 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:24.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:24.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:24.938 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:24.973 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:24.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:24.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:25.075 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:25.116 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:25.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:25.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:25.156 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:59:25.219 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:25.252 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:25.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:25.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:25.309 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-03-30T13:59:25.354 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:25.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:25.403 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:25.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:25.504 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:25.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:25.542 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:25.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:25.642 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:25.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:25 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2931045389' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.1a", "id": [7, 3, 2]}]': finished 2024-03-30T13:59:25.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:25 smithi043 ceph-mon[29963]: osdmap e339: 8 total, 7 up, 7 in 2024-03-30T13:59:25.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:25 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2931045389' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.1a", "id": [7, 3, 2]}]': finished 2024-03-30T13:59:25.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:25 smithi043 ceph-mon[42099]: osdmap e339: 8 total, 7 up, 7 in 2024-03-30T13:59:25.678 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:25.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:25.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:25.716 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:25 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2931045389' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.1a", "id": [7, 3, 2]}]': finished 2024-03-30T13:59:25.716 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:25 smithi113 ceph-mon[34341]: osdmap e339: 8 total, 7 up, 7 in 2024-03-30T13:59:25.718 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:59:25.780 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:25.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:25.819 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:25.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:25.920 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:25.961 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:25.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:25.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:26.063 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:26.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:26.106 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:26.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:26.207 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:26.245 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:59:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:59:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:59:26.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:26.247 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:26.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:26.348 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:26.359 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:59:26.383 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:26.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:26.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:26.398 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:59:26.446 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:59:26.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:26 smithi113 ceph-mon[34341]: pgmap v740: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s; 38464/316698 objects degraded (12.145%); 26716/316698 objects misplaced (8.436%); 23 MiB/s, 368 objects/s recovering 2024-03-30T13:59:26.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:26 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1362790785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:59:26.485 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:26.491 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:59:26.520 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:26.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:26.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:26.537 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:26 smithi043 ceph-mon[29963]: pgmap v740: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s; 38464/316698 objects degraded (12.145%); 26716/316698 objects misplaced (8.436%); 23 MiB/s, 368 objects/s recovering 2024-03-30T13:59:26.537 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:26 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1362790785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:59:26.538 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:59:26.538 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:26 smithi043 ceph-mon[42099]: pgmap v740: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s; 38464/316698 objects degraded (12.145%); 26716/316698 objects misplaced (8.436%); 23 MiB/s, 368 objects/s recovering 2024-03-30T13:59:26.538 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:26 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1362790785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:59:26.584 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:59:26.622 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:26.631 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:59:26.631 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:59:26.631 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:59:26.631 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:59:26.632 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:59:26.632 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:59:26.632 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:59:26.632 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:59:26.632 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:59:26.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:26.667 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:26.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:26.768 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:26.806 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:26.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:26.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:26.907 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:26.944 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:26.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:26.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:27.046 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:27.100 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:27.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:27.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:27.202 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:27.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:27.247 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:27.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:27.348 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:27.394 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:27.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:27.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:27.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:27.538 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:27.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:27.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:27.640 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:27.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:27.679 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:27.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:27.751 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:59:27.751 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:59:27.781 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:27.817 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:27.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:27.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:27.919 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:27.956 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:27.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:27.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:28.059 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:28.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:28.113 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:28.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:28.214 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:28.244 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:28.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:28.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:28.346 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:28.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:28.384 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:28.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:28.485 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:28.520 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:28.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:28.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:28.623 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:28.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:28 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T13:59:28.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:28 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T13:59:28.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:28 smithi043 ceph-mon[29963]: osdmap e340: 8 total, 7 up, 7 in 2024-03-30T13:59:28.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:28 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 38464/316698 objects degraded (12.145%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:28.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:28 smithi043 ceph-mon[29963]: pgmap v742: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 37252/311001 objects degraded (11.978%); 25962/311001 objects misplaced (8.348%); 23 MiB/s, 367 objects/s recovering 2024-03-30T13:59:28.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:28 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T13:59:28.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:28 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T13:59:28.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:28 smithi043 ceph-mon[42099]: osdmap e340: 8 total, 7 up, 7 in 2024-03-30T13:59:28.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:28 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 38464/316698 objects degraded (12.145%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:28.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:28 smithi043 ceph-mon[42099]: pgmap v742: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 37252/311001 objects degraded (11.978%); 25962/311001 objects misplaced (8.348%); 23 MiB/s, 367 objects/s recovering 2024-03-30T13:59:28.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:28.664 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:28.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:28.716 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:59:28.716 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:59:28.765 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:28.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:28.803 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:28.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:28.823 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:28 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T13:59:28.823 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:28 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T13:59:28.823 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:28 smithi113 ceph-mon[34341]: osdmap e340: 8 total, 7 up, 7 in 2024-03-30T13:59:28.823 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:28 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 38464/316698 objects degraded (12.145%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:28.823 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:28 smithi113 ceph-mon[34341]: pgmap v742: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 37252/311001 objects degraded (11.978%); 25962/311001 objects misplaced (8.348%); 23 MiB/s, 367 objects/s recovering 2024-03-30T13:59:28.871 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-03-30T13:59:28.905 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:28.951 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:28.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:28.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:29.053 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:29.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:29.091 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:29.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:29.192 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:29.242 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:29.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:29.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:29.333 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:59:29.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:29.386 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:29.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:29.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:29.488 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:29.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:29.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:29.626 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:29.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:29.666 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:29.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:29.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:59:29 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:59:29.766 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:29.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:29.805 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:29.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:29.906 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:29.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:29.950 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:29.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:30.051 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:30.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:30.094 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:30.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:30.176 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-03-30T13:59:30.194 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:30.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:30.232 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:30.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:30.333 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:30.367 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:30.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:30.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:30.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:30.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:30.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:30 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:30.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:30 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1245030472' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:59:30.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:30 smithi113 ceph-mon[34341]: pgmap v743: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 725 B/s rd, 0 B/s wr, 1.56k op/s; 33715/296385 objects degraded (11.375%); 25074/296385 objects misplaced (8.460%); 34 MiB/s, 542 objects/s recovering 2024-03-30T13:59:30.469 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:30.504 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:30.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:30.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:30.606 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:30.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:30.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:30.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:30 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:30.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:30 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1245030472' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:59:30.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:30 smithi043 ceph-mon[29963]: pgmap v743: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 725 B/s rd, 0 B/s wr, 1.56k op/s; 33715/296385 objects degraded (11.375%); 25074/296385 objects misplaced (8.460%); 34 MiB/s, 542 objects/s recovering 2024-03-30T13:59:30.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:30.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:30.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:30 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:30.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:30 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1245030472' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T13:59:30.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:30 smithi043 ceph-mon[42099]: pgmap v743: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 725 B/s rd, 0 B/s wr, 1.56k op/s; 33715/296385 objects degraded (11.375%); 25074/296385 objects misplaced (8.460%); 34 MiB/s, 542 objects/s recovering 2024-03-30T13:59:30.648 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:30.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:30.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:30.718 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T13:59:30.737 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 5] out_osds: [7] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-30T13:59:30.737 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T13:59:30.738 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-30T13:59:30.738 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-03-30T13:59:30.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:30.838 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:30.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:30.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:30.940 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:30.995 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:30.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:30.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:31.097 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:31.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:31.145 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:31.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:31.246 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:31.254 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:59:31.270 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:59:31.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:31.279 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:31.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:31.380 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:31.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:31 smithi043 ceph-mon[29963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:59:31.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:31 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1245030472' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:59:31.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:31 smithi043 ceph-mon[29963]: osdmap e341: 8 total, 7 up, 7 in 2024-03-30T13:59:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:31 smithi043 ceph-mon[42099]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:59:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:31 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1245030472' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:59:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:31 smithi043 ceph-mon[42099]: osdmap e341: 8 total, 7 up, 7 in 2024-03-30T13:59:31.409 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:31.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:31.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:31.432 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:31 smithi113 ceph-mon[34341]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:59:31.432 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:31 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1245030472' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T13:59:31.433 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:31 smithi113 ceph-mon[34341]: osdmap e341: 8 total, 7 up, 7 in 2024-03-30T13:59:31.511 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:31.539 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:31.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:31.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:31.641 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:31.671 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:31.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:31.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:31.747 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T13:59:31.747 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T13:59:31.773 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:31.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:31.803 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:31.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:31.904 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:31.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:31.932 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:31.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:32.033 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:32.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:32.062 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:32.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:32.162 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:32.193 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:32.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:32.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:32.295 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:32.329 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:32.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:32.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:32.388 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:32 smithi043 ceph-mon[29963]: osdmap e342: 8 total, 7 up, 7 in 2024-03-30T13:59:32.389 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:32 smithi043 ceph-mon[29963]: pgmap v746: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.49k op/s; 32433/289899 objects degraded (11.188%); 24197/289899 objects misplaced (8.347%); 32 MiB/s, 510 objects/s recovering 2024-03-30T13:59:32.389 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:32 smithi043 ceph-mon[42099]: osdmap e342: 8 total, 7 up, 7 in 2024-03-30T13:59:32.389 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:32 smithi043 ceph-mon[42099]: pgmap v746: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.49k op/s; 32433/289899 objects degraded (11.188%); 24197/289899 objects misplaced (8.347%); 32 MiB/s, 510 objects/s recovering 2024-03-30T13:59:32.431 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:32.436 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:32 smithi113 ceph-mon[34341]: osdmap e342: 8 total, 7 up, 7 in 2024-03-30T13:59:32.436 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:32 smithi113 ceph-mon[34341]: pgmap v746: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 28 active+clean+remapped; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.49k op/s; 32433/289899 objects degraded (11.188%); 24197/289899 objects misplaced (8.347%); 32 MiB/s, 510 objects/s recovering 2024-03-30T13:59:32.462 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:32.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:32.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:32.475 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-30T13:59:32.564 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:32.591 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:32.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:32.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:32.693 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:32.727 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:32.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:32.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:32.829 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:32.857 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:32.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:32.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:32.959 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:32.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:32.989 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:32.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:33.090 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:33.118 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:33.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:33.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:33.134 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T13:59:33.220 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:33.248 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:33.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:33.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:33.350 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:33.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:33.380 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:33.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:33.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:33 smithi043 ceph-mon[29963]: osdmap e343: 8 total, 7 up, 7 in 2024-03-30T13:59:33.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:33 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 32433/289899 objects degraded (11.188%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:33.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:33 smithi043 ceph-mon[29963]: osdmap e344: 8 total, 7 up, 7 in 2024-03-30T13:59:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:33 smithi043 ceph-mon[42099]: osdmap e343: 8 total, 7 up, 7 in 2024-03-30T13:59:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:33 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 32433/289899 objects degraded (11.188%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:33 smithi043 ceph-mon[42099]: osdmap e344: 8 total, 7 up, 7 in 2024-03-30T13:59:33.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:33 smithi113 ceph-mon[34341]: osdmap e343: 8 total, 7 up, 7 in 2024-03-30T13:59:33.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:33 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 32433/289899 objects degraded (11.188%), 15 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:33.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:33 smithi113 ceph-mon[34341]: osdmap e344: 8 total, 7 up, 7 in 2024-03-30T13:59:33.481 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:33.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:33.508 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:33.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:33.609 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:33.639 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:33.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:33.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:33.741 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:33.785 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:33.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:33.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:33.887 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:33.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:33.929 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:33.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:34.030 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:34.069 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:34.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:34.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:34.171 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:34.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:34.195 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:34.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:34.296 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:34.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:34.334 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:34.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:34.435 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:34.470 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:34.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:34.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:34.573 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:34.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:34.615 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:34.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:34.716 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:34.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:34.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:34.873 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:34.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:34 smithi043 ceph-mon[42099]: osdmap e345: 8 total, 7 up, 7 in 2024-03-30T13:59:34.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:34 smithi043 ceph-mon[42099]: pgmap v750: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 29 active+clean+remapped; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 776 op/s; 30387/288186 objects degraded (10.544%); 23626/288186 objects misplaced (8.198%); 44 MiB/s, 700 objects/s recovering 2024-03-30T13:59:34.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:34 smithi043 ceph-mon[29963]: osdmap e345: 8 total, 7 up, 7 in 2024-03-30T13:59:34.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:34 smithi043 ceph-mon[29963]: pgmap v750: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 29 active+clean+remapped; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 776 op/s; 30387/288186 objects degraded (10.544%); 23626/288186 objects misplaced (8.198%); 44 MiB/s, 700 objects/s recovering 2024-03-30T13:59:34.911 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:34.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:34.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:34.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:34 smithi113 ceph-mon[34341]: osdmap e345: 8 total, 7 up, 7 in 2024-03-30T13:59:34.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:34 smithi113 ceph-mon[34341]: pgmap v750: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 102 active+clean, 1 active+undersized+degraded+remapped+backfill_wait, 12 active+undersized, 10 active+undersized+degraded, 29 active+clean+remapped; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 776 op/s; 30387/288186 objects degraded (10.544%); 23626/288186 objects misplaced (8.198%); 44 MiB/s, 700 objects/s recovering 2024-03-30T13:59:35.012 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:35.052 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:35.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:35.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:35.154 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:35.190 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:35.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:35.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:35.292 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:35.318 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:35.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:35.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:35.420 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:35.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:35.455 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:35.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:35.557 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:35.598 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:35.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:35.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:35.700 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:35.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:35.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:35.841 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:35.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:35.881 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:35.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:35.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:35 smithi043 ceph-mon[29963]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T13:59:35.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:35 smithi043 ceph-mon[42099]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T13:59:35.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:35 smithi113 ceph-mon[34341]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T13:59:35.982 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:36.008 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:36.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:36.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:36.110 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:36.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:36.155 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:36.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:36.256 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:36.282 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:36.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:36.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:36.384 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:36.392 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:59:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:59:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:59:36.422 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:36.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:36.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:36.523 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:36.559 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:36.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:36.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:36.661 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:36.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:36.695 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:36.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:36.796 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:36.831 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:36.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:36.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:36.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:36 smithi043 ceph-mon[29963]: pgmap v751: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 29 active+clean+remapped; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 617 op/s; 27942/281544 objects degraded (9.925%); 23626/281544 objects misplaced (8.392%); 48 MiB/s, 761 objects/s recovering 2024-03-30T13:59:36.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:36 smithi043 ceph-mon[42099]: pgmap v751: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 29 active+clean+remapped; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 617 op/s; 27942/281544 objects degraded (9.925%); 23626/281544 objects misplaced (8.392%); 48 MiB/s, 761 objects/s recovering 2024-03-30T13:59:36.933 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:36.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:36.959 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:36.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:36.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:36 smithi113 ceph-mon[34341]: pgmap v751: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 29 active+clean+remapped; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 617 op/s; 27942/281544 objects degraded (9.925%); 23626/281544 objects misplaced (8.392%); 48 MiB/s, 761 objects/s recovering 2024-03-30T13:59:37.060 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:37.100 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:37.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:37.201 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:37.235 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:37.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:37.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:37.337 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:37.371 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:37.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:37.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:37.473 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:37.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:37.499 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:37.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:37.600 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:37.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:37.637 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:37.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:37.738 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:37.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:37.776 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:37.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:37.877 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:37.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:37.911 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:37.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:37.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:37 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 27942/281544 objects degraded (9.925%), 13 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:37.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:37 smithi113 ceph-mon[34341]: osdmap e346: 8 total, 7 up, 7 in 2024-03-30T13:59:38.012 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:38.048 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:38.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:38.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:38.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:37 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 27942/281544 objects degraded (9.925%), 13 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:38.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:37 smithi043 ceph-mon[29963]: osdmap e346: 8 total, 7 up, 7 in 2024-03-30T13:59:38.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:37 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 27942/281544 objects degraded (9.925%), 13 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:38.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:37 smithi043 ceph-mon[42099]: osdmap e346: 8 total, 7 up, 7 in 2024-03-30T13:59:38.150 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:38.185 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:38.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:38.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:38.287 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:38.324 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:38.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:38.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:38.426 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:38.458 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:38.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:38.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:38.560 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:38.591 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:38.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:38.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:38.693 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:38.722 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:38.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:38.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:38.827 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:38.868 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:38.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:38.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:38.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:38 smithi043 ceph-mon[29963]: pgmap v753: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 929 B/s rd, 0 B/s wr, 730 op/s; 26796/277842 objects degraded (9.644%); 22166/277842 objects misplaced (7.978%); 61 MiB/s, 968 objects/s recovering 2024-03-30T13:59:38.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:38 smithi043 ceph-mon[42099]: pgmap v753: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 929 B/s rd, 0 B/s wr, 730 op/s; 26796/277842 objects degraded (9.644%); 22166/277842 objects misplaced (7.978%); 61 MiB/s, 968 objects/s recovering 2024-03-30T13:59:38.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:38 smithi113 ceph-mon[34341]: pgmap v753: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 929 B/s rd, 0 B/s wr, 730 op/s; 26796/277842 objects degraded (9.644%); 22166/277842 objects misplaced (7.978%); 61 MiB/s, 968 objects/s recovering 2024-03-30T13:59:38.970 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:38.995 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:38.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:38.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:39.097 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:39.129 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:39.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:39.234 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:39.260 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:39.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:39.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:39.362 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:39.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:39.399 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:39.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:39.478 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T13:59:39.478 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:59:39.478 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 346, 2024-03-30T13:59:39.478 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "7ee932fa-ee9a-11ee-b647-cb9ed24678a4", 2024-03-30T13:59:39.478 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-03-30T13:37:07.346020+0000", 2024-03-30T13:59:39.478 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-03-30T13:59:37.620100+0000", 2024-03-30T13:59:39.479 INFO:teuthology.orchestra.run.smithi043.stdout: "last_up_change": "2024-03-30T13:50:07.489747+0000", 2024-03-30T13:59:39.479 INFO:teuthology.orchestra.run.smithi043.stdout: "last_in_change": "2024-03-30T13:58:46.155265+0000", 2024-03-30T13:59:39.479 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-30T13:59:39.479 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_num": 5806080, 2024-03-30T13:59:39.479 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_set": [ 2024-03-30T13:59:39.479 INFO:teuthology.orchestra.run.smithi043.stdout: "nodeep-scrub", 2024-03-30T13:59:39.479 INFO:teuthology.orchestra.run.smithi043.stdout: "noscrub", 2024-03-30T13:59:39.479 INFO:teuthology.orchestra.run.smithi043.stdout: "pglog_hardlimit", 2024-03-30T13:59:39.479 INFO:teuthology.orchestra.run.smithi043.stdout: "purged_snapdirs", 2024-03-30T13:59:39.479 INFO:teuthology.orchestra.run.smithi043.stdout: "recovery_deletes", 2024-03-30T13:59:39.479 INFO:teuthology.orchestra.run.smithi043.stdout: "sortbitwise" 2024-03-30T13:59:39.480 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:59:39.480 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_version": 18, 2024-03-30T13:59:39.480 INFO:teuthology.orchestra.run.smithi043.stdout: "full_ratio": 0.94999998807907104, 2024-03-30T13:59:39.480 INFO:teuthology.orchestra.run.smithi043.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-30T13:59:39.480 INFO:teuthology.orchestra.run.smithi043.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-30T13:59:39.480 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_snapshot": "", 2024-03-30T13:59:39.480 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_max": 10, 2024-03-30T13:59:39.480 INFO:teuthology.orchestra.run.smithi043.stdout: "max_osd": 8, 2024-03-30T13:59:39.480 INFO:teuthology.orchestra.run.smithi043.stdout: "require_min_compat_client": "luminous", 2024-03-30T13:59:39.480 INFO:teuthology.orchestra.run.smithi043.stdout: "min_compat_client": "luminous", 2024-03-30T13:59:39.480 INFO:teuthology.orchestra.run.smithi043.stdout: "require_osd_release": "squid", 2024-03-30T13:59:39.480 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_crimson": false, 2024-03-30T13:59:39.481 INFO:teuthology.orchestra.run.smithi043.stdout: "pools": [ 2024-03-30T13:59:39.481 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.481 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 1, 2024-03-30T13:59:39.481 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".mgr", 2024-03-30T13:59:39.481 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:40:27.658979+0000", 2024-03-30T13:59:39.481 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T13:59:39.481 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T13:59:39.482 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:59:39.482 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:59:39.482 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:59:39.482 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:59:39.482 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:59:39.482 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:59:39.482 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:59:39.482 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:59:39.483 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:59:39.483 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:59:39.483 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 11, 2024-03-30T13:59:39.483 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 11, 2024-03-30T13:59:39.483 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 11, 2024-03-30T13:59:39.483 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 11, 2024-03-30T13:59:39.483 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 11, 2024-03-30T13:59:39.483 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:59:39.483 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:59:39.483 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:59:39.483 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:59:39.483 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:59:39.484 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:59:39.484 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:59:39.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.484 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "248", 2024-03-30T13:59:39.484 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:59:39.484 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:59:39.484 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "202", 2024-03-30T13:59:39.484 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:59:39.485 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:59:39.485 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T13:59:39.485 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T13:59:39.485 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:59:39.485 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:59:39.485 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:59:39.486 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:59:39.486 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:59:39.486 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:59:39.486 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:59:39.486 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:59:39.486 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:59:39.486 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:59:39.486 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:59:39.486 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:59:39.486 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:59:39.487 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:59:39.487 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:59:39.488 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:59:39.488 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:59:39.488 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:59:39.488 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:59:39.488 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.488 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:59:39.488 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:59:39.488 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:59:39.489 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:59:39.489 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:59:39.489 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:59:39.489 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:59:39.489 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:59:39.489 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:59:39.489 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:59:39.489 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:59:39.489 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-03-30T13:59:39.489 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_max": 32, 2024-03-30T13:59:39.489 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_min": 1 2024-03-30T13:59:39.489 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.490 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:59:39.490 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": {} 2024-03-30T13:59:39.490 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.490 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:59:39.490 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:59:39.490 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.9099999666213989, 2024-03-30T13:59:39.490 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.9099999666213989, 2024-03-30T13:59:39.490 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.86000001430511475, 2024-03-30T13:59:39.490 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.6399999856948853, 2024-03-30T13:59:39.490 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.6399999856948853, 2024-03-30T13:59:39.490 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:59:39.491 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:59:39.491 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:59:39.491 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.491 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.491 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 2, 2024-03-30T13:59:39.491 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "datapool", 2024-03-30T13:59:39.491 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:42.989975+0000", 2024-03-30T13:59:39.491 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 8193, 2024-03-30T13:59:39.491 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T13:59:39.491 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:59:39.491 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:59:39.491 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:59:39.492 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:59:39.492 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:59:39.492 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:59:39.492 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:59:39.492 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:59:39.492 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:59:39.492 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:59:39.492 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 3, 2024-03-30T13:59:39.492 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 3, 2024-03-30T13:59:39.492 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 3, 2024-03-30T13:59:39.492 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 3, 2024-03-30T13:59:39.492 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 3, 2024-03-30T13:59:39.492 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:59:39.493 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:59:39.493 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:59:39.493 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:59:39.493 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:59:39.493 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:59:39.493 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:59:39.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.493 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "68", 2024-03-30T13:59:39.493 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:59:39.493 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:59:39.493 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T13:59:39.493 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:59:39.493 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:59:39.494 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 2, 2024-03-30T13:59:39.494 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 68, 2024-03-30T13:59:39.494 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:59:39.494 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:59:39.494 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:59:39.494 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:59:39.494 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:59:39.494 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:59:39.494 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:59:39.494 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:59:39.494 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:59:39.494 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:59:39.494 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:59:39.495 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:59:39.495 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:59:39.495 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:59:39.495 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:59:39.495 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:59:39.495 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:59:39.495 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:59:39.495 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:59:39.495 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.495 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:59:39.495 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:59:39.495 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:59:39.495 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:59:39.495 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:59:39.496 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:59:39.496 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:59:39.496 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:59:39.496 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:59:39.496 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:59:39.496 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:59:39.496 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T13:59:39.496 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:59:39.496 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd": {} 2024-03-30T13:59:39.496 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.496 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:59:39.496 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:59:39.496 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.3499999046325684, 2024-03-30T13:59:39.497 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.3499999046325684, 2024-03-30T13:59:39.497 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.70999997854232788, 2024-03-30T13:59:39.497 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.6699999570846558, 2024-03-30T13:59:39.497 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.6699999570846558, 2024-03-30T13:59:39.497 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:59:39.497 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:59:39.497 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:59:39.497 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.497 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.497 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 3, 2024-03-30T13:59:39.497 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".rgw.root", 2024-03-30T13:59:39.497 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:43.910306+0000", 2024-03-30T13:59:39.498 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T13:59:39.498 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T13:59:39.498 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:59:39.498 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:59:39.498 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:59:39.498 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:59:39.498 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:59:39.498 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:59:39.498 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:59:39.498 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:59:39.498 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:59:39.498 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:59:39.498 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T13:59:39.499 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T13:59:39.499 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T13:59:39.499 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T13:59:39.499 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T13:59:39.499 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:59:39.499 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:59:39.499 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:59:39.499 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:59:39.499 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:59:39.499 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:59:39.499 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:59:39.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.499 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "66", 2024-03-30T13:59:39.499 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:59:39.500 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:59:39.500 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T13:59:39.500 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:59:39.500 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:59:39.500 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T13:59:39.500 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T13:59:39.500 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:59:39.500 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:59:39.500 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:59:39.500 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:59:39.500 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:59:39.500 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:59:39.500 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:59:39.500 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:59:39.501 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:59:39.501 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:59:39.501 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:59:39.501 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:59:39.501 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:59:39.501 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:59:39.501 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:59:39.501 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:59:39.501 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:59:39.501 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:59:39.501 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:59:39.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.501 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:59:39.501 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:59:39.502 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:59:39.502 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:59:39.502 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:59:39.502 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:59:39.502 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:59:39.502 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:59:39.502 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:59:39.502 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:59:39.502 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:59:39.502 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T13:59:39.502 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:59:39.502 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T13:59:39.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.503 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:59:39.503 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:59:39.503 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.7400000095367432, 2024-03-30T13:59:39.503 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.7400000095367432, 2024-03-30T13:59:39.503 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.86000001430511475, 2024-03-30T13:59:39.503 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.5, 2024-03-30T13:59:39.503 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.5, 2024-03-30T13:59:39.503 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:59:39.503 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:59:39.503 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:59:39.503 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.503 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 4, 2024-03-30T13:59:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.log", 2024-03-30T13:59:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:45.633132+0000", 2024-03-30T13:59:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T13:59:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T13:59:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:59:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:59:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:59:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:59:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:59:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:59:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:59:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:59:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:59:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:59:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T13:59:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T13:59:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T13:59:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T13:59:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T13:59:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:59:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:59:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:59:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:59:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:59:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:59:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:59:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "68", 2024-03-30T13:59:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:59:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:59:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T13:59:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:59:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:59:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T13:59:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T13:59:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:59:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:59:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:59:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:59:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:59:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:59:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:59:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:59:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:59:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:59:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:59:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:59:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:59:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:59:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:59:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:59:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:59:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:59:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:59:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:59:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:59:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:59:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:59:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:59:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:59:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:59:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:59:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:59:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:59:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:59:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T13:59:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:59:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T13:59:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:59:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:59:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.9700000286102295, 2024-03-30T13:59:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.9700000286102295, 2024-03-30T13:59:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.86000001430511475, 2024-03-30T13:59:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.690000057220459, 2024-03-30T13:59:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.690000057220459, 2024-03-30T13:59:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:59:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:59:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:59:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 5, 2024-03-30T13:59:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.control", 2024-03-30T13:59:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:47.570084+0000", 2024-03-30T13:59:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T13:59:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T13:59:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:59:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:59:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:59:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:59:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:59:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:59:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:59:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:59:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:59:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:59:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T13:59:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T13:59:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T13:59:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T13:59:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T13:59:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:59:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:59:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:59:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:59:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:59:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:59:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:59:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "70", 2024-03-30T13:59:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:59:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:59:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T13:59:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:59:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:59:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T13:59:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T13:59:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:59:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:59:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:59:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:59:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:59:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:59:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:59:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:59:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:59:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:59:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:59:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:59:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:59:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:59:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:59:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:59:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:59:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:59:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:59:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:59:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:59:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:59:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:59:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:59:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:59:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:59:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:59:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:59:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:59:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:59:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T13:59:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:59:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T13:59:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:59:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:59:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.7400000095367432, 2024-03-30T13:59:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.7400000095367432, 2024-03-30T13:59:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.86000001430511475, 2024-03-30T13:59:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.5, 2024-03-30T13:59:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.5, 2024-03-30T13:59:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:59:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:59:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:59:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 6, 2024-03-30T13:59:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.meta", 2024-03-30T13:59:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:49.637189+0000", 2024-03-30T13:59:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T13:59:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T13:59:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:59:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:59:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:59:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:59:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:59:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:59:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:59:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:59:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:59:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:59:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T13:59:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T13:59:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T13:59:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T13:59:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T13:59:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:59:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:59:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:59:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:59:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:59:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:59:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:59:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "72", 2024-03-30T13:59:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:59:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:59:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T13:59:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:59:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:59:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T13:59:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T13:59:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:59:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:59:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:59:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:59:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:59:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:59:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:59:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:59:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:59:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:59:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:59:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:59:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:59:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:59:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:59:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:59:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:59:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:59:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:59:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:59:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:59:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:59:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:59:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:59:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:59:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:59:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:59:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:59:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:59:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:59:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-03-30T13:59:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_bias": 4 2024-03-30T13:59:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:59:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T13:59:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:59:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:59:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.9700000286102295, 2024-03-30T13:59:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.9700000286102295, 2024-03-30T13:59:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.86000001430511475, 2024-03-30T13:59:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.690000057220459, 2024-03-30T13:59:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.690000057220459, 2024-03-30T13:59:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:59:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:59:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:59:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 10, 2024-03-30T13:59:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "unique_pool_3", 2024-03-30T13:59:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:56:54.614701+0000", 2024-03-30T13:59:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T13:59:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T13:59:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T13:59:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T13:59:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T13:59:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T13:59:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T13:59:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T13:59:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T13:59:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T13:59:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T13:59:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T13:59:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 16, 2024-03-30T13:59:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 16, 2024-03-30T13:59:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 16, 2024-03-30T13:59:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 16, 2024-03-30T13:59:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 16, 2024-03-30T13:59:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T13:59:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T13:59:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T13:59:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T13:59:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T13:59:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T13:59:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T13:59:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "295", 2024-03-30T13:59:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T13:59:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T13:59:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T13:59:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T13:59:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T13:59:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T13:59:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T13:59:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T13:59:39.526 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T13:59:39.526 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T13:59:39.526 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T13:59:39.526 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T13:59:39.526 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T13:59:39.526 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T13:59:39.526 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T13:59:39.526 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T13:59:39.526 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T13:59:39.526 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T13:59:39.526 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T13:59:39.526 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T13:59:39.527 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T13:59:39.527 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T13:59:39.527 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T13:59:39.527 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T13:59:39.527 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T13:59:39.527 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T13:59:39.527 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.527 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T13:59:39.527 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T13:59:39.527 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T13:59:39.527 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T13:59:39.527 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T13:59:39.528 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T13:59:39.528 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T13:59:39.528 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T13:59:39.528 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T13:59:39.528 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T13:59:39.528 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T13:59:39.528 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T13:59:39.528 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T13:59:39.528 INFO:teuthology.orchestra.run.smithi043.stdout: "rados": {} 2024-03-30T13:59:39.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.528 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T13:59:39.528 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T13:59:39.528 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.619999885559082, 2024-03-30T13:59:39.528 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.619999885559082, 2024-03-30T13:59:39.529 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.86000001430511475, 2024-03-30T13:59:39.529 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2.25, 2024-03-30T13:59:39.529 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2.25, 2024-03-30T13:59:39.529 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T13:59:39.529 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T13:59:39.529 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T13:59:39.529 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.529 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.529 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:59:39.529 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.529 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.529 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-03-30T13:59:39.529 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "94ea083e-6a27-41db-83af-2f7d3cae1956", 2024-03-30T13:59:39.529 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:59:39.530 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:59:39.530 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:59:39.530 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:59:39.530 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:59:39.530 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:59:39.530 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 8, 2024-03-30T13:59:39.530 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 344, 2024-03-30T13:59:39.530 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:59:39.530 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:59:39.530 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:59:39.530 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.530 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.530 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.531 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6802", 2024-03-30T13:59:39.531 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:59:39.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.531 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.531 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.531 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6803", 2024-03-30T13:59:39.531 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:59:39.531 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.531 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.531 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:59:39.531 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.531 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.531 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.532 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6804", 2024-03-30T13:59:39.532 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:59:39.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.532 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.532 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.532 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6805", 2024-03-30T13:59:39.532 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:59:39.532 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.532 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.532 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:59:39.532 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.532 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.533 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.533 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6808", 2024-03-30T13:59:39.533 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:59:39.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.533 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.533 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6809", 2024-03-30T13:59:39.533 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:59:39.533 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.533 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.533 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:59:39.533 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.534 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.534 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6806", 2024-03-30T13:59:39.534 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:59:39.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.534 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.534 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.534 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6807", 2024-03-30T13:59:39.534 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T13:59:39.534 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.534 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.534 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6803/1144592572", 2024-03-30T13:59:39.534 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6805/1144592572", 2024-03-30T13:59:39.535 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6809/1144592572", 2024-03-30T13:59:39.535 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6807/1144592572", 2024-03-30T13:59:39.535 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:59:39.535 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:59:39.535 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:59:39.535 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.535 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-03-30T13:59:39.535 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a", 2024-03-30T13:59:39.535 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:59:39.535 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:59:39.535 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:59:39.535 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:59:39.536 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:59:39.536 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:59:39.536 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 13, 2024-03-30T13:59:39.536 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 326, 2024-03-30T13:59:39.536 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:59:39.536 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:59:39.536 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:59:39.536 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.536 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.536 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.536 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6810", 2024-03-30T13:59:39.536 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:59:39.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.536 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.537 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.537 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6811", 2024-03-30T13:59:39.537 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:59:39.537 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.537 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.537 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:59:39.537 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.537 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.537 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.537 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6812", 2024-03-30T13:59:39.537 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:59:39.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.537 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.538 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.538 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6813", 2024-03-30T13:59:39.538 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:59:39.538 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.538 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.538 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.538 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:59:39.538 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.538 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.538 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.538 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6816", 2024-03-30T13:59:39.538 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:59:39.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.539 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.539 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6817", 2024-03-30T13:59:39.539 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:59:39.539 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.539 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.539 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:59:39.539 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.539 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.539 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6814", 2024-03-30T13:59:39.540 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:59:39.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.540 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.540 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.540 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6815", 2024-03-30T13:59:39.540 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T13:59:39.540 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.540 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.540 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6811/3901165928", 2024-03-30T13:59:39.540 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6813/3901165928", 2024-03-30T13:59:39.540 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6817/3901165928", 2024-03-30T13:59:39.540 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6815/3901165928", 2024-03-30T13:59:39.540 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:59:39.541 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:59:39.541 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:59:39.541 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.541 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.541 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.541 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-03-30T13:59:39.541 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "6db6782e-5889-443e-bc6c-df47a1f93bc8", 2024-03-30T13:59:39.541 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 0, 2024-03-30T13:59:39.541 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:59:39.541 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:59:39.541 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 0, 2024-03-30T13:59:39.541 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:59:39.542 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:59:39.542 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 18, 2024-03-30T13:59:39.542 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 98, 2024-03-30T13:59:39.542 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 167, 2024-03-30T13:59:39.542 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:59:39.542 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:59:39.542 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.542 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.542 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6818", 2024-03-30T13:59:39.542 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:59:39.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.542 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.543 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6819", 2024-03-30T13:59:39.543 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:59:39.543 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.543 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.543 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:59:39.543 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.543 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.543 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.543 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6820", 2024-03-30T13:59:39.543 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:59:39.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.543 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.543 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.544 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6821", 2024-03-30T13:59:39.544 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:59:39.544 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.544 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.544 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:59:39.544 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.544 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.544 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.544 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6824", 2024-03-30T13:59:39.544 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:59:39.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.544 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.544 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.545 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6825", 2024-03-30T13:59:39.545 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:59:39.545 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.545 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.545 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:59:39.545 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.545 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.545 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6822", 2024-03-30T13:59:39.545 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:59:39.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.545 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.546 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6823", 2024-03-30T13:59:39.546 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T13:59:39.546 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.546 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.546 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6819/3415037523", 2024-03-30T13:59:39.546 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6821/3415037523", 2024-03-30T13:59:39.546 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6825/3415037523", 2024-03-30T13:59:39.546 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6823/3415037523", 2024-03-30T13:59:39.546 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:59:39.546 INFO:teuthology.orchestra.run.smithi043.stdout: "exists" 2024-03-30T13:59:39.547 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.547 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.547 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-03-30T13:59:39.547 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "7048ff46-2cba-4d18-9835-95ecbddb2bb6", 2024-03-30T13:59:39.547 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:59:39.547 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:59:39.547 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:59:39.547 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:59:39.547 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:59:39.547 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:59:39.547 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 25, 2024-03-30T13:59:39.547 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 342, 2024-03-30T13:59:39.548 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:59:39.548 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:59:39.548 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:59:39.548 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.548 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.548 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6826", 2024-03-30T13:59:39.548 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:59:39.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.548 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.548 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6827", 2024-03-30T13:59:39.548 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:59:39.548 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.548 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.549 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:59:39.549 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.549 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.549 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.549 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6828", 2024-03-30T13:59:39.549 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:59:39.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.549 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.549 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.549 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6829", 2024-03-30T13:59:39.549 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:59:39.549 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.549 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.550 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:59:39.550 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.550 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.550 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6832", 2024-03-30T13:59:39.550 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:59:39.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.550 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.550 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6833", 2024-03-30T13:59:39.550 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:59:39.550 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.551 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.551 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.551 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:59:39.554 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.554 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.554 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.554 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6830", 2024-03-30T13:59:39.554 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:59:39.554 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.554 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.554 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.554 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6831", 2024-03-30T13:59:39.554 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T13:59:39.554 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.554 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.555 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6827/2933589376", 2024-03-30T13:59:39.555 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6829/2933589376", 2024-03-30T13:59:39.555 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6833/2933589376", 2024-03-30T13:59:39.555 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6831/2933589376", 2024-03-30T13:59:39.555 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:59:39.555 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:59:39.555 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:59:39.555 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.555 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.555 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-03-30T13:59:39.556 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "42da30e6-9eec-4ed5-8c73-9ce78aa8ff63", 2024-03-30T13:59:39.556 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:59:39.556 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:59:39.556 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:59:39.556 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:59:39.556 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:59:39.557 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:59:39.557 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 32, 2024-03-30T13:59:39.557 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 326, 2024-03-30T13:59:39.557 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:59:39.557 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:59:39.557 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:59:39.557 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.557 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.557 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.558 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6800", 2024-03-30T13:59:39.558 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:59:39.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.558 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.558 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6801", 2024-03-30T13:59:39.558 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:59:39.558 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.558 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.558 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:59:39.558 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.559 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.559 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6802", 2024-03-30T13:59:39.559 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:59:39.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.559 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.559 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6803", 2024-03-30T13:59:39.559 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:59:39.559 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.559 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.559 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:59:39.559 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.560 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.560 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6806", 2024-03-30T13:59:39.560 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:59:39.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.560 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.560 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6807", 2024-03-30T13:59:39.560 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:59:39.560 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.560 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.560 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:59:39.560 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.561 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.561 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.561 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6804", 2024-03-30T13:59:39.561 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:59:39.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.561 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.561 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.561 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6805", 2024-03-30T13:59:39.561 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T13:59:39.561 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.561 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.561 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6801/1945833081", 2024-03-30T13:59:39.561 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6803/1945833081", 2024-03-30T13:59:39.562 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6807/1945833081", 2024-03-30T13:59:39.562 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6805/1945833081", 2024-03-30T13:59:39.562 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:59:39.562 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:59:39.562 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:59:39.562 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.562 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.562 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-03-30T13:59:39.562 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "99a11b81-0df7-4b1d-946a-4297ba62e263", 2024-03-30T13:59:39.562 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:59:39.562 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:59:39.562 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:59:39.563 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:59:39.563 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:59:39.563 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:59:39.563 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 40, 2024-03-30T13:59:39.563 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 343, 2024-03-30T13:59:39.563 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:59:39.563 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:59:39.563 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:59:39.563 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.563 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.563 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.563 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6808", 2024-03-30T13:59:39.564 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:59:39.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.564 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.564 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6809", 2024-03-30T13:59:39.564 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:59:39.564 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.564 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.564 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:59:39.564 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.564 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.565 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6810", 2024-03-30T13:59:39.565 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:59:39.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.565 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.565 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.565 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6811", 2024-03-30T13:59:39.565 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:59:39.565 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.565 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.565 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:59:39.565 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.565 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.566 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.566 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6814", 2024-03-30T13:59:39.566 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:59:39.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.566 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.566 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.566 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6815", 2024-03-30T13:59:39.566 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:59:39.566 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.566 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.566 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:59:39.566 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.567 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.567 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.567 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6812", 2024-03-30T13:59:39.567 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:59:39.567 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.567 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.567 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.567 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6813", 2024-03-30T13:59:39.567 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T13:59:39.567 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.567 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.567 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.567 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6809/3426360325", 2024-03-30T13:59:39.567 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6811/3426360325", 2024-03-30T13:59:39.568 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6815/3426360325", 2024-03-30T13:59:39.568 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6813/3426360325", 2024-03-30T13:59:39.568 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:59:39.568 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:59:39.568 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:59:39.568 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.568 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-03-30T13:59:39.568 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "cafd0e26-78e9-4164-a43a-96ca8382fefe", 2024-03-30T13:59:39.568 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:59:39.568 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T13:59:39.568 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T13:59:39.568 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:59:39.569 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:59:39.569 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:59:39.569 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 49, 2024-03-30T13:59:39.569 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 342, 2024-03-30T13:59:39.569 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:59:39.569 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:59:39.569 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:59:39.569 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.569 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.569 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6816", 2024-03-30T13:59:39.569 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:59:39.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.570 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.570 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6817", 2024-03-30T13:59:39.570 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:59:39.570 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.570 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.570 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:59:39.570 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.570 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.570 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6818", 2024-03-30T13:59:39.570 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:59:39.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.571 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.571 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6819", 2024-03-30T13:59:39.571 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:59:39.571 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.571 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.571 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:59:39.571 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.571 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.571 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6822", 2024-03-30T13:59:39.571 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:59:39.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.572 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.572 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.572 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6823", 2024-03-30T13:59:39.572 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:59:39.572 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.572 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.572 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:59:39.572 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.572 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.572 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.572 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6820", 2024-03-30T13:59:39.572 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:59:39.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.573 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.573 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.573 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6821", 2024-03-30T13:59:39.573 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T13:59:39.573 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.573 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.573 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6817/1527017698", 2024-03-30T13:59:39.573 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6819/1527017698", 2024-03-30T13:59:39.573 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6823/1527017698", 2024-03-30T13:59:39.573 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6821/1527017698", 2024-03-30T13:59:39.573 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:59:39.573 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:59:39.573 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:59:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-03-30T13:59:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "e9d59662-9132-48f4-9e95-842a3cfbd79f", 2024-03-30T13:59:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T13:59:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 0, 2024-03-30T13:59:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-03-30T13:59:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T13:59:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T13:59:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T13:59:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 58, 2024-03-30T13:59:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 228, 2024-03-30T13:59:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T13:59:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T13:59:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T13:59:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6824", 2024-03-30T13:59:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:59:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6825", 2024-03-30T13:59:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:59:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T13:59:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6826", 2024-03-30T13:59:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:59:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6827", 2024-03-30T13:59:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:59:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T13:59:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6830", 2024-03-30T13:59:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:59:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6831", 2024-03-30T13:59:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:59:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T13:59:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T13:59:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T13:59:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6828", 2024-03-30T13:59:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:59:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T13:59:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6829", 2024-03-30T13:59:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T13:59:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6825/3182476015", 2024-03-30T13:59:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6827/3182476015", 2024-03-30T13:59:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6831/3182476015", 2024-03-30T13:59:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6829/3182476015", 2024-03-30T13:59:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T13:59:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T13:59:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T13:59:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:59:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_xinfo": [ 2024-03-30T13:59:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-03-30T13:59:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:59:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:59:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:59:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:59:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:59:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:59:14.128544+0000", 2024-03-30T13:59:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:59:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-03-30T13:59:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:59:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:59:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:59:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:59:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:59:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:58:36.388795+0000", 2024-03-30T13:59:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:59:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-03-30T13:59:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:59:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:59:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:59:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:59:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:59:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:49:03.444490+0000", 2024-03-30T13:59:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 166 2024-03-30T13:59:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-03-30T13:59:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:59:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:59:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:59:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:59:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:59:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:58:38.119388+0000", 2024-03-30T13:59:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:59:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-03-30T13:59:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:59:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:59:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:59:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:59:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:59:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:59:01.956659+0000", 2024-03-30T13:59:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:59:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-03-30T13:59:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:59:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:59:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:59:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:59:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:59:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:59:14.665162+0000", 2024-03-30T13:59:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:59:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-03-30T13:59:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:59:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:59:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:59:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:59:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T13:59:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:59:27.485636+0000", 2024-03-30T13:59:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:59:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-03-30T13:59:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T13:59:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T13:59:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T13:59:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T13:59:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 65536, 2024-03-30T13:59:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:59:07.293614+0000", 2024-03-30T13:59:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T13:59:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:59:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap": [ 2024-03-30T13:59:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.1a", 2024-03-30T13:59:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-03-30T13:59:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:59:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-03-30T13:59:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:59:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_items": [ 2024-03-30T13:59:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.13", 2024-03-30T13:59:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-03-30T13:59:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-03-30T13:59:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-03-30T13:59:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.14", 2024-03-30T13:59:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-03-30T13:59:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-03-30T13:59:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-03-30T13:59:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:59:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_primaries": [], 2024-03-30T13:59:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_temp": [ 2024-03-30T13:59:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "1.1", 2024-03-30T13:59:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:59:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:59:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: 4 2024-03-30T13:59:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "1.2", 2024-03-30T13:59:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:59:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:59:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:59:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "1.5", 2024-03-30T13:59:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:59:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T13:59:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: 0 2024-03-30T13:59:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "1.8", 2024-03-30T13:59:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:59:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:59:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-03-30T13:59:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "1.9", 2024-03-30T13:59:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:59:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:59:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-03-30T13:59:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "1.a", 2024-03-30T13:59:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:59:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T13:59:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: 4 2024-03-30T13:59:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "3.4", 2024-03-30T13:59:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T13:59:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:59:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: 0 2024-03-30T13:59:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "3.5", 2024-03-30T13:59:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:59:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:59:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-03-30T13:59:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "3.9", 2024-03-30T13:59:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:59:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T13:59:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:59:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "3.13", 2024-03-30T13:59:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:59:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T13:59:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:59:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "3.14", 2024-03-30T13:59:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:59:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T13:59:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:59:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "3.1f", 2024-03-30T13:59:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:59:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:59:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-03-30T13:59:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.14", 2024-03-30T13:59:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:59:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T13:59:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:59:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.16", 2024-03-30T13:59:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:59:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:59:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:59:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.17", 2024-03-30T13:59:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:59:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T13:59:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:59:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.4", 2024-03-30T13:59:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:59:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:59:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: 1 2024-03-30T13:59:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.6", 2024-03-30T13:59:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:59:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:59:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: 1 2024-03-30T13:59:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.b", 2024-03-30T13:59:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:59:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:59:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: 1 2024-03-30T13:59:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.d", 2024-03-30T13:59:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:59:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:59:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: 3 2024-03-30T13:59:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.14", 2024-03-30T13:59:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:59:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T13:59:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:59:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.19", 2024-03-30T13:59:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T13:59:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:59:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-03-30T13:59:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.1e", 2024-03-30T13:59:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:59:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T13:59:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:59:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.1f", 2024-03-30T13:59:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T13:59:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:59:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:59:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.3", 2024-03-30T13:59:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T13:59:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T13:59:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:59:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.5", 2024-03-30T13:59:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T13:59:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:59:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:59:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.8", 2024-03-30T13:59:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:59:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T13:59:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:59:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.9", 2024-03-30T13:59:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:59:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:59:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:59:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.12", 2024-03-30T13:59:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:59:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:59:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:59:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.14", 2024-03-30T13:59:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:59:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:59:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:59:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.17", 2024-03-30T13:59:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:59:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:59:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: 0 2024-03-30T13:59:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.18", 2024-03-30T13:59:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:59:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T13:59:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T13:59:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.1c", 2024-03-30T13:59:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:59:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:59:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: 4 2024-03-30T13:59:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "10.3", 2024-03-30T13:59:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T13:59:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: 4 2024-03-30T13:59:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "10.6", 2024-03-30T13:59:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:59:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:59:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: 1 2024-03-30T13:59:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "10.7", 2024-03-30T13:59:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T13:59:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: 3 2024-03-30T13:59:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "10.8", 2024-03-30T13:59:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T13:59:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:59:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: 4 2024-03-30T13:59:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "10.9", 2024-03-30T13:59:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T13:59:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T13:59:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T13:59:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: 3 2024-03-30T13:59:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:59:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_temp": [], 2024-03-30T13:59:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklist": { 2024-03-30T13:59:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/538304234": "2024-03-31T13:43:27.253365+0000", 2024-03-30T13:59:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6801/3411723337": "2024-03-31T13:37:41.733515+0000", 2024-03-30T13:59:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3656823679": "2024-03-31T13:37:41.733515+0000", 2024-03-30T13:59:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2618817647": "2024-03-31T13:38:09.321740+0000", 2024-03-30T13:59:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/3411723337": "2024-03-31T13:37:41.733515+0000", 2024-03-30T13:59:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1206218784": "2024-03-31T13:37:41.733515+0000", 2024-03-30T13:59:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/886631486": "2024-03-31T13:37:41.733515+0000", 2024-03-30T13:59:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/1339921781": "2024-03-31T13:38:09.321740+0000", 2024-03-30T13:59:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6801/1339921781": "2024-03-31T13:38:09.321740+0000", 2024-03-30T13:59:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2794494528": "2024-03-31T13:38:09.321740+0000", 2024-03-30T13:59:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/564868778": "2024-03-31T13:38:09.321740+0000", 2024-03-30T13:59:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/3033707884": "2024-03-31T13:43:27.253365+0000", 2024-03-30T13:59:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3539398453": "2024-03-31T13:43:27.253365+0000", 2024-03-30T13:59:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2414364740": "2024-03-31T13:43:27.253365+0000", 2024-03-30T13:59:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6801/3033707884": "2024-03-31T13:43:27.253365+0000", 2024-03-30T13:59:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3923597429": "2024-03-31T13:43:27.253365+0000", 2024-03-30T13:59:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2524269526": "2024-03-31T13:43:27.253365+0000" 2024-03-30T13:59:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "range_blocklist": {}, 2024-03-30T13:59:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profiles": { 2024-03-30T13:59:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "default": { 2024-03-30T13:59:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "crush-failure-domain": "osd", 2024-03-30T13:59:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "k": "2", 2024-03-30T13:59:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "m": "1", 2024-03-30T13:59:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "plugin": "jerasure", 2024-03-30T13:59:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "technique": "reed_sol_van" 2024-03-30T13:59:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps_queue": [], 2024-03-30T13:59:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "new_removed_snaps": [], 2024-03-30T13:59:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "new_purged_snaps": [], 2024-03-30T13:59:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_node_flags": {}, 2024-03-30T13:59:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "device_class_flags": {}, 2024-03-30T13:59:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": { 2024-03-30T13:59:39.613 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_enabled": false, 2024-03-30T13:59:39.613 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_bucket_count": 0, 2024-03-30T13:59:39.613 INFO:teuthology.orchestra.run.smithi043.stdout: "degraded_stretch_mode": 0, 2024-03-30T13:59:39.613 INFO:teuthology.orchestra.run.smithi043.stdout: "recovering_stretch_mode": 0, 2024-03-30T13:59:39.613 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_bucket": 0 2024-03-30T13:59:39.613 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:39.613 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:59:39.613 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:39.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:39.652 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:39.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:39.673 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:59:39 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:59:39.753 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:39.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:39 smithi043 ceph-mon[29963]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-30T13:59:39.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:39 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1993188985' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T13:59:39.768 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:39 smithi043 ceph-mon[42099]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-30T13:59:39.768 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:39 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1993188985' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T13:59:39.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:39.789 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:39.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:39.890 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:39.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:39.915 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:39.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:39.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:39 smithi113 ceph-mon[34341]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-30T13:59:39.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:39 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1993188985' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T13:59:40.016 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:40.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:40.051 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:40.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:40.152 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:40.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:40.191 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:40.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:40.292 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:40.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:40.332 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:40.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:40.433 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:40.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:40.473 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:40.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:40.574 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:40.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:40.663 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:40.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:40.766 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:40.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:40.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:40.909 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:40.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:40.946 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:40.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:40.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:40 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:40.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:40 smithi113 ceph-mon[34341]: pgmap v754: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 723 B/s rd, 0 B/s wr, 1.19k op/s; 23562/264690 objects degraded (8.902%); 21370/264690 objects misplaced (8.074%); 64 MiB/s, 1.02k objects/s recovering 2024-03-30T13:59:41.047 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:41.083 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:41.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:41.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:41.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:40 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:41.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:40 smithi043 ceph-mon[29963]: pgmap v754: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 723 B/s rd, 0 B/s wr, 1.19k op/s; 23562/264690 objects degraded (8.902%); 21370/264690 objects misplaced (8.074%); 64 MiB/s, 1.02k objects/s recovering 2024-03-30T13:59:41.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:40 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:41.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:40 smithi043 ceph-mon[42099]: pgmap v754: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 723 B/s rd, 0 B/s wr, 1.19k op/s; 23562/264690 objects degraded (8.902%); 21370/264690 objects misplaced (8.074%); 64 MiB/s, 1.02k objects/s recovering 2024-03-30T13:59:41.185 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:41.222 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:41.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:41.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:41.324 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:41.361 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:41.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:41.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:41.463 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:41.502 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:41.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:41.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:41.604 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:41.627 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:41.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:41.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:41.729 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:41.768 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:41.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:41.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:41.870 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:41.899 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:41.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:41.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:42.001 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:42.037 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:42.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:42.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:42.139 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:42.172 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:42.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:42.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:42.274 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:42.316 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:42.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:42.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:42.418 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:42.462 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:42.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:42.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:42.564 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:42.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:42.592 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:42.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:42.694 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:42.730 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:42.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:42.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:42.832 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:42.868 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:42.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:42.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:42.971 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:43.008 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:43.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:43.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:43.110 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:43.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:42 smithi043 ceph-mon[29963]: pgmap v755: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 23149/259110 objects degraded (8.934%); 20128/259110 objects misplaced (7.768%); 54 MiB/s, 868 objects/s recovering 2024-03-30T13:59:43.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:42 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 23149/259110 objects degraded (8.934%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:43.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:42 smithi043 ceph-mon[42099]: pgmap v755: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 23149/259110 objects degraded (8.934%); 20128/259110 objects misplaced (7.768%); 54 MiB/s, 868 objects/s recovering 2024-03-30T13:59:43.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:42 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 23149/259110 objects degraded (8.934%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:43.146 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:43.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:43.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:43.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:42 smithi113 ceph-mon[34341]: pgmap v755: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 23149/259110 objects degraded (8.934%); 20128/259110 objects misplaced (7.768%); 54 MiB/s, 868 objects/s recovering 2024-03-30T13:59:43.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:42 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 23149/259110 objects degraded (8.934%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:43.247 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:43.282 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:43.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:43.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:43.384 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:43.420 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:43.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:43.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:43.522 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:43.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:43.560 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:43.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:43.661 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:43.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:43.701 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:43.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:43.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:43.839 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:43.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:43.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:43.941 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:43.975 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:43.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:43.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:44.077 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:44.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:44.113 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:44.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:44.214 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:44.254 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:44.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:44.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:44.356 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:44.395 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:44.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:44.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:44.497 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:44.531 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:44.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:44.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:44.633 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:44.672 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:44.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:44.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:44.773 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:44.810 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:44.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:44.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:44.914 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:44.945 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:44.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:44.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:45.047 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:45.082 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:45.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:45.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:45.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:44 smithi043 ceph-mon[29963]: pgmap v756: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 22147/256692 objects degraded (8.628%); 19322/256692 objects misplaced (7.527%); 43 MiB/s, 685 objects/s recovering 2024-03-30T13:59:45.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:44 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:45.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:44 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:45.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:44 smithi043 ceph-mon[42099]: pgmap v756: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 22147/256692 objects degraded (8.628%); 19322/256692 objects misplaced (7.527%); 43 MiB/s, 685 objects/s recovering 2024-03-30T13:59:45.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:44 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:45.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:44 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:45.184 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:45.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:44 smithi113 ceph-mon[34341]: pgmap v756: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 22147/256692 objects degraded (8.628%); 19322/256692 objects misplaced (7.527%); 43 MiB/s, 685 objects/s recovering 2024-03-30T13:59:45.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:44 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T13:59:45.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:44 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:45.217 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:45.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:45.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:45.319 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:45.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:45.357 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:45.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:45.458 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:45.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:45.496 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:45.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:45.597 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:45.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:45.627 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:45.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:45.728 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:45.765 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:45.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:45.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:45.867 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:45.905 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:45.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:45.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:46.007 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:46.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:46.038 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:46.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:46.139 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:46.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:46.181 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:46.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:46.282 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:46.319 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:46.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:46.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:46.393 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:59:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:59:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:59:46.421 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:46.459 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:46.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:46.561 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:46.597 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:46.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:46.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:46.699 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:46.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:46.737 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:46.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:46.839 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:46.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:46.876 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:46.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:46.977 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:47.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:47.014 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:47.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:47.115 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:47.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:46 smithi043 ceph-mon[29963]: pgmap v757: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 20166/245970 objects degraded (8.199%); 19322/245970 objects misplaced (7.855%); 38 MiB/s, 612 objects/s recovering 2024-03-30T13:59:47.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:46 smithi043 ceph-mon[42099]: pgmap v757: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 20166/245970 objects degraded (8.199%); 19322/245970 objects misplaced (7.855%); 38 MiB/s, 612 objects/s recovering 2024-03-30T13:59:47.151 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:47.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:47.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:47.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:46 smithi113 ceph-mon[34341]: pgmap v757: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 20166/245970 objects degraded (8.199%); 19322/245970 objects misplaced (7.855%); 38 MiB/s, 612 objects/s recovering 2024-03-30T13:59:47.253 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:47.289 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:47.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:47.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:47.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:47.431 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:47.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:47.532 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:47.569 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:47.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:47.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:47.671 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:47.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:47.702 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:47.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:47.803 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:47.840 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:47.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:47.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:47.942 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:47.955 INFO:tasks.thrashosds.thrasher:j is {'epoch': 346, 'fsid': '7ee932fa-ee9a-11ee-b647-cb9ed24678a4', 'created': '2024-03-30T13:37:07.346020+0000', 'modified': '2024-03-30T13:59:37.620100+0000', 'last_up_change': '2024-03-30T13:50:07.489747+0000', 'last_in_change': '2024-03-30T13:58:46.155265+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T13:40:27.658979+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '202', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.6399999856948853, 'raw_score_stable': 1.6399999856948853, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-30T13:42:42.989975+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3499999046325684, 'score_stable': 2.3499999046325684, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 1.6699999570846558, 'raw_score_stable': 1.6699999570846558, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-30T13:42:43.910306+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-30T13:42:45.633132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-30T13:42:47.570084+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-30T13:42:49.637189+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-03-30T13:56:54.614701+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '295', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.619999885559082, 'score_stable': 2.619999885559082, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '94ea083e-6a27-41db-83af-2f7d3cae1956', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 344, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6802', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6803', 'nonce': 1144592572}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6804', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6805', 'nonce': 1144592572}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6808', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6809', 'nonce': 1144592572}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6806', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6807', 'nonce': 1144592572}]}, 'public_addr': '172.21.15.43:6803/1144592572', 'cluster_addr': '172.21.15.43:6805/1144592572', 'heartbeat_back_addr': '172.21.15.43:6809/1144592572', 'heartbeat_front_addr': '172.21.15.43:6807/1144592572', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 326, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6810', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6811', 'nonce': 3901165928}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6812', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6813', 'nonce': 3901165928}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6816', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6817', 'nonce': 3901165928}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6814', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6815', 'nonce': 3901165928}]}, 'public_addr': '172.21.15.43:6811/3901165928', 'cluster_addr': '172.21.15.43:6813/3901165928', 'heartbeat_back_addr': '172.21.15.43:6817/3901165928', 'heartbeat_front_addr': '172.21.15.43:6815/3901165928', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6db6782e-5889-443e-bc6c-df47a1f93bc8', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 98, 'down_at': 167, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6818', 'nonce': 3415037523}, {'type': 'v1', 'addr': '172.21.15.43:6819', 'nonce': 3415037523}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6820', 'nonce': 3415037523}, {'type': 'v1', 'addr': '172.21.15.43:6821', 'nonce': 3415037523}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6824', 'nonce': 3415037523}, {'type': 'v1', 'addr': '172.21.15.43:6825', 'nonce': 3415037523}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6822', 'nonce': 3415037523}, {'type': 'v1', 'addr': '172.21.15.43:6823', 'nonce': 3415037523}]}, 'public_addr': '172.21.15.43:6819/3415037523', 'cluster_addr': '172.21.15.43:6821/3415037523', 'heartbeat_back_addr': '172.21.15.43:6825/3415037523', 'heartbeat_front_addr': '172.21.15.43:6823/3415037523', 'state': ['exists']}, {'osd': 3, 'uuid': '7048ff46-2cba-4d18-9835-95ecbddb2bb6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 342, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6826', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6827', 'nonce': 2933589376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6828', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6829', 'nonce': 2933589376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6832', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6833', 'nonce': 2933589376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6830', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6831', 'nonce': 2933589376}]}, 'public_addr': '172.21.15.43:6827/2933589376', 'cluster_addr': '172.21.15.43:6829/2933589376', 'heartbeat_back_addr': '172.21.15.43:6833/2933589376', 'heartbeat_front_addr': '172.21.15.43:6831/2933589376', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '42da30e6-9eec-4ed5-8c73-9ce78aa8ff63', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 326, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6800', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6801', 'nonce': 1945833081}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6802', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6803', 'nonce': 1945833081}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6806', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6807', 'nonce': 1945833081}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6804', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6805', 'nonce': 1945833081}]}, 'public_addr': '172.21.15.113:6801/1945833081', 'cluster_addr': '172.21.15.113:6803/1945833081', 'heartbeat_back_addr': '172.21.15.113:6807/1945833081', 'heartbeat_front_addr': '172.21.15.113:6805/1945833081', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '99a11b81-0df7-4b1d-946a-4297ba62e263', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 343, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6808', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6809', 'nonce': 3426360325}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6810', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6811', 'nonce': 3426360325}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6814', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6815', 'nonce': 3426360325}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6812', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6813', 'nonce': 3426360325}]}, 'public_addr': '172.21.15.113:6809/3426360325', 'cluster_addr': '172.21.15.113:6811/3426360325', 'heartbeat_back_addr': '172.21.15.113:6815/3426360325', 'heartbeat_front_addr': '172.21.15.113:6813/3426360325', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'cafd0e26-78e9-4164-a43a-96ca8382fefe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 342, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6816', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6817', 'nonce': 1527017698}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6818', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6819', 'nonce': 1527017698}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6822', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6823', 'nonce': 1527017698}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6820', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6821', 'nonce': 1527017698}]}, 'public_addr': '172.21.15.113:6817/1527017698', 'cluster_addr': '172.21.15.113:6819/1527017698', 'heartbeat_back_addr': '172.21.15.113:6823/1527017698', 'heartbeat_front_addr': '172.21.15.113:6821/1527017698', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e9d59662-9132-48f4-9e95-842a3cfbd79f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 228, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6824', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6825', 'nonce': 3182476015}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6826', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6827', 'nonce': 3182476015}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6830', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6831', 'nonce': 3182476015}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6828', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6829', 'nonce': 3182476015}]}, 'public_addr': '172.21.15.113:6825/3182476015', 'cluster_addr': '172.21.15.113:6827/3182476015', 'heartbeat_back_addr': '172.21.15.113:6831/3182476015', 'heartbeat_front_addr': '172.21.15.113:6829/3182476015', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:59:14.128544+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:58:36.388795+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:49:03.444490+0000', 'dead_epoch': 166}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:58:38.119388+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:59:01.956659+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:59:14.665162+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:59:27.485636+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-30T13:59:07.293614+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.1a', 'osds': [7, 3, 2]}], 'pg_upmap_items': [{'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.1', 'osds': [5, 0, 4]}, {'pgid': '1.2', 'osds': [0, 4, 7]}, {'pgid': '1.5', 'osds': [4, 6, 0]}, {'pgid': '1.8', 'osds': [5, 3, 6]}, {'pgid': '1.9', 'osds': [3, 5, 6]}, {'pgid': '1.a', 'osds': [5, 6, 4]}, {'pgid': '3.4', 'osds': [1, 5, 0]}, {'pgid': '3.5', 'osds': [5, 3, 6]}, {'pgid': '3.9', 'osds': [4, 1, 7]}, {'pgid': '3.13', 'osds': [4, 6, 7]}, {'pgid': '3.14', 'osds': [4, 6, 7]}, {'pgid': '3.1f', 'osds': [0, 5, 6]}, {'pgid': '4.14', 'osds': [3, 1, 7]}, {'pgid': '4.16', 'osds': [0, 3, 7]}, {'pgid': '4.17', 'osds': [3, 1, 7]}, {'pgid': '5.4', 'osds': [5, 0, 1]}, {'pgid': '5.6', 'osds': [5, 4, 1]}, {'pgid': '5.b', 'osds': [0, 5, 1]}, {'pgid': '5.d', 'osds': [5, 0, 3]}, {'pgid': '5.14', 'osds': [3, 1, 7]}, {'pgid': '5.19', 'osds': [1, 5, 6]}, {'pgid': '5.1e', 'osds': [0, 1, 7]}, {'pgid': '5.1f', 'osds': [6, 4, 7]}, {'pgid': '6.3', 'osds': [6, 1, 7]}, {'pgid': '6.5', 'osds': [6, 3, 7]}, {'pgid': '6.8', 'osds': [3, 6, 7]}, {'pgid': '6.9', 'osds': [0, 3, 7]}, {'pgid': '6.12', 'osds': [4, 3, 7]}, {'pgid': '6.14', 'osds': [4, 0, 7]}, {'pgid': '6.17', 'osds': [4, 5, 0]}, {'pgid': '6.18', 'osds': [0, 1, 7]}, {'pgid': '6.1c', 'osds': [5, 3, 4]}, {'pgid': '10.3', 'osds': [3, 4]}, {'pgid': '10.6', 'osds': [0, 4, 1]}, {'pgid': '10.7', 'osds': [6, 3]}, {'pgid': '10.8', 'osds': [0, 5, 4]}, {'pgid': '10.9', 'osds': [5, 4, 3]}], 'primary_temp': [], 'blocklist': {'172.21.15.43:0/538304234': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:6801/3411723337': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/3656823679': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/2618817647': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:6800/3411723337': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/1206218784': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/886631486': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:6800/1339921781': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:6801/1339921781': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:0/2794494528': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:0/564868778': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:6800/3033707884': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/3539398453': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/2414364740': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:6801/3033707884': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/3923597429': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/2524269526': '2024-03-31T13:43:27.253365+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-30T13:59:47.955 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.13 2024-03-30T13:59:47.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd rm-pg-upmap-items 4.13 2024-03-30T13:59:47.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:47.981 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:47.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:48.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:47 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 20166/245970 objects degraded (8.199%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:48.057 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:47 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 20166/245970 objects degraded (8.199%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:48.082 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:48.117 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:48.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:48.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:48.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:47 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 20166/245970 objects degraded (8.199%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:48.219 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:48.256 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:48.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:48.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:48.358 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:48.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:48.395 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:48.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:48.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:48.548 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:59:48.552 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:59:48.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:48.553 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:48.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:48.654 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:48.662 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T13:59:48.690 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:48.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:48.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:48.698 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T13:59:48.741 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T13:59:48.783 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T13:59:48.792 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:48.827 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T13:59:48.830 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:48.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:48.870 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T13:59:48.910 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T13:59:48.910 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T13:59:48.910 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T13:59:48.910 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T13:59:48.910 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T13:59:48.911 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T13:59:48.911 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T13:59:48.911 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T13:59:48.911 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T13:59:48.932 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:48.966 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:48 smithi043 ceph-mon[29963]: pgmap v758: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1015 B/s rd, 0 B/s wr, 1.26k op/s; 19776/239670 objects degraded (8.251%); 17988/239670 objects misplaced (7.505%); 30 MiB/s, 484 objects/s recovering 2024-03-30T13:59:48.966 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:48 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/464661306' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:59:48.966 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:48 smithi043 ceph-mon[42099]: pgmap v758: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1015 B/s rd, 0 B/s wr, 1.26k op/s; 19776/239670 objects degraded (8.251%); 17988/239670 objects misplaced (7.505%); 30 MiB/s, 484 objects/s recovering 2024-03-30T13:59:48.966 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:48 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/464661306' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:59:48.967 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:48.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:48.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:49.069 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:49.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:49.106 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:49.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:49.121 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:48 smithi113 ceph-mon[34341]: pgmap v758: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1015 B/s rd, 0 B/s wr, 1.26k op/s; 19776/239670 objects degraded (8.251%); 17988/239670 objects misplaced (7.505%); 30 MiB/s, 484 objects/s recovering 2024-03-30T13:59:49.121 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:48 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/464661306' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T13:59:49.207 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:49.252 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:49.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:49.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:49.354 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:49.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:49.390 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:49.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:49.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:59:49 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:59:49.491 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:49.529 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:49.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:49.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:49.631 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:49.672 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:49.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:49.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:49.773 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:49.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:49.815 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:49.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:49.916 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:49.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:49.954 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:49.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:50.055 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:50.097 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:50.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:50.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:50.166 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:59:50.198 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:50.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:50.240 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:50.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:50.341 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:50.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:50.378 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:50.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:50.479 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:50.521 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:50.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:50.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:50.623 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:50.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:50.657 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:50.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:50.758 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:50.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:50.795 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:50.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:50.896 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:50.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:50.932 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:50.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:51.033 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:51.073 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:51.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:51.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:51.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:50 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:51.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:50 smithi043 ceph-mon[29963]: pgmap v759: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.45k op/s; 16867/225612 objects degraded (7.476%); 17149/225612 objects misplaced (7.601%); 35 MiB/s, 553 objects/s recovering 2024-03-30T13:59:51.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:50 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:51.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:50 smithi043 ceph-mon[42099]: pgmap v759: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.45k op/s; 16867/225612 objects degraded (7.476%); 17149/225612 objects misplaced (7.601%); 35 MiB/s, 553 objects/s recovering 2024-03-30T13:59:51.175 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:51.211 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:51.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:51.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:51.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:50 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:51.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:50 smithi113 ceph-mon[34341]: pgmap v759: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.45k op/s; 16867/225612 objects degraded (7.476%); 17149/225612 objects misplaced (7.601%); 35 MiB/s, 553 objects/s recovering 2024-03-30T13:59:51.228 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T13:59:51.229 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T13:59:51.314 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:51.352 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:51.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:51.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:51.454 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:51.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:51.494 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:51.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:51.595 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:51.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:51.635 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:51.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:51.736 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:51.773 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:51.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:51.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:51.875 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:51.913 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:51.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:51.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:52.015 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:52.049 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:52.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:52.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:52.151 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:52.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:52.189 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:52.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:52.290 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:52.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:52.328 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:52.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:52.429 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:52.465 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:52.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:52.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:52.566 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:52.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:52.606 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:52.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:52.707 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:52.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:52.743 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:52.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:52.844 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:52.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:52.883 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:52.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:52.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:52 smithi113 ceph-mon[34341]: pgmap v760: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 16427/219126 objects degraded (7.497%); 15849/219126 objects misplaced (7.233%); 28 MiB/s, 452 objects/s recovering 2024-03-30T13:59:52.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:52 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 16427/219126 objects degraded (7.497%), 12 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:52.985 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:53.022 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:53.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:53.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:53.124 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:53.133 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:52 smithi043 ceph-mon[29963]: pgmap v760: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 16427/219126 objects degraded (7.497%); 15849/219126 objects misplaced (7.233%); 28 MiB/s, 452 objects/s recovering 2024-03-30T13:59:53.133 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:52 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 16427/219126 objects degraded (7.497%), 12 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:53.133 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:52 smithi043 ceph-mon[42099]: pgmap v760: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 16427/219126 objects degraded (7.497%); 15849/219126 objects misplaced (7.233%); 28 MiB/s, 452 objects/s recovering 2024-03-30T13:59:53.133 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:52 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 16427/219126 objects degraded (7.497%), 12 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:53.159 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:53.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:53.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:53.230 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T13:59:53.260 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:53.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:53.302 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:53.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:53.403 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:53.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:53.442 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:53.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:53.543 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:53.558 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T13:59:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-03-30T13:59:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15531.0:257304 10.6 10:66b4eeff:::benchmark_data_smithi043_169264_object95077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-30T13:59:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:59:53.560714+0000", 2024-03-30T13:59:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0020512989999999999, 2024-03-30T13:59:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0020767060000000002, 2024-03-30T13:59:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:59:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:59:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-03-30T13:59:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:59:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15531", 2024-03-30T13:59:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T13:59:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 257304 2024-03-30T13:59:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:59:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:59:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:59:53.560714+0000", 2024-03-30T13:59:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:59:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:59:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:59:53.560714+0000", 2024-03-30T13:59:53.560 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:59:53.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:53.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:53.560 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:59:53.560 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:59:53.560715+0000", 2024-03-30T13:59:53.560 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.46e-06 2024-03-30T13:59:53.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:53.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:59:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:59:53.560722+0000", 2024-03-30T13:59:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8909999999999998e-06 2024-03-30T13:59:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:59:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:59:53.560736+0000", 2024-03-30T13:59:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4062e-05 2024-03-30T13:59:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:59:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:59:53.560745+0000", 2024-03-30T13:59:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0529999999999996e-06 2024-03-30T13:59:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T13:59:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:59:53.561202+0000", 2024-03-30T13:59:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00045644799999999998 2024-03-30T13:59:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T13:59:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:59:53.561350+0000", 2024-03-30T13:59:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000148732 2024-03-30T13:59:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4,5", 2024-03-30T13:59:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:59:53.561500+0000", 2024-03-30T13:59:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000149693 2024-03-30T13:59:53.563 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:53.563 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:53.563 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:53.563 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:53.563 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:53.563 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15531.0:257306 10.6 10.fc480786 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-30T13:59:53.563 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:59:53.560804+0000", 2024-03-30T13:59:53.563 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0019607069999999999, 2024-03-30T13:59:53.563 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0020375969999999999, 2024-03-30T13:59:53.563 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T13:59:53.563 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T13:59:53.563 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "queued for pg", 2024-03-30T13:59:53.563 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T13:59:53.563 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15531", 2024-03-30T13:59:53.564 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T13:59:53.564 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 257306 2024-03-30T13:59:53.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:53.564 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T13:59:53.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:53.564 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T13:59:53.564 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:59:53.560804+0000", 2024-03-30T13:59:53.564 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:59:53.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:53.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:53.564 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T13:59:53.564 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:59:53.560804+0000", 2024-03-30T13:59:53.564 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T13:59:53.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:53.565 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:53.565 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T13:59:53.565 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:59:53.560805+0000", 2024-03-30T13:59:53.565 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6799999999999999e-07 2024-03-30T13:59:53.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:53.565 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:53.565 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T13:59:53.565 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:59:53.560809+0000", 2024-03-30T13:59:53.565 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3059999999999999e-06 2024-03-30T13:59:53.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:53.565 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:53.565 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T13:59:53.565 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:59:53.560819+0000", 2024-03-30T13:59:53.565 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.8020000000000003e-06 2024-03-30T13:59:53.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T13:59:53.566 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T13:59:53.566 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T13:59:53.566 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:59:53.560827+0000", 2024-03-30T13:59:53.566 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.841e-06 2024-03-30T13:59:53.566 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:53.566 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T13:59:53.566 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:53.566 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T13:59:53.566 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T13:59:53.566 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 2 2024-03-30T13:59:53.566 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T13:59:53.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:53.582 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:53.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:53.683 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:53.717 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:53.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:53.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:53.819 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:53.860 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:53.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:53.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:53.962 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:53.995 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:53.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:53.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:54.097 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:54.138 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:54.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:54.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:54.240 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:54.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:54.279 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:54.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:54.380 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:54.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:54.415 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:54.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:54.516 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:54.550 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:54.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:54.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:54.652 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:54.689 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:54.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:54.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:54.792 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:54.803 INFO:teuthology.orchestra.run.smithi043.stderr:clear 4.13 pg_upmap_items mapping 2024-03-30T13:59:54.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:54.830 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:54.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:54.931 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:54.944 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:54 smithi043 ceph-mon[29963]: pgmap v761: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.19k op/s; 15547/216378 objects degraded (7.185%); 14933/216378 objects misplaced (6.901%); 29 MiB/s, 462 objects/s recovering 2024-03-30T13:59:54.944 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:54 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2611702866' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.13"}]: dispatch 2024-03-30T13:59:54.945 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:54 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.13"}]: dispatch 2024-03-30T13:59:54.945 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:54 smithi043 ceph-mon[42099]: pgmap v761: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.19k op/s; 15547/216378 objects degraded (7.185%); 14933/216378 objects misplaced (6.901%); 29 MiB/s, 462 objects/s recovering 2024-03-30T13:59:54.945 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:54 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2611702866' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.13"}]: dispatch 2024-03-30T13:59:54.945 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:54 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.13"}]: dispatch 2024-03-30T13:59:54.946 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-03-30T13:59:54.965 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:54.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:54.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:55.066 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:55.099 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:55.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:55.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:55.201 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:55.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:54 smithi113 ceph-mon[34341]: pgmap v761: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.19k op/s; 15547/216378 objects degraded (7.185%); 14933/216378 objects misplaced (6.901%); 29 MiB/s, 462 objects/s recovering 2024-03-30T13:59:55.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:54 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2611702866' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.13"}]: dispatch 2024-03-30T13:59:55.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:54 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.13"}]: dispatch 2024-03-30T13:59:55.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:55.240 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:55.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:55.341 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:55.376 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:55.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:55.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:55.405 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:59:55.478 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:55.523 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:55.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:55.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:55.625 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:55.669 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:55.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:55.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:55.771 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:55.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:55.808 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:55.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:55.908 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:55.942 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:55.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:55.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:56.044 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:56.071 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:56.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:56.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:56.090 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:55 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.13"}]': finished 2024-03-30T13:59:56.090 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:55 smithi043 ceph-mon[29963]: osdmap e347: 8 total, 7 up, 7 in 2024-03-30T13:59:56.090 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:55 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.13"}]': finished 2024-03-30T13:59:56.091 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:55 smithi043 ceph-mon[42099]: osdmap e347: 8 total, 7 up, 7 in 2024-03-30T13:59:56.173 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:56.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:56.204 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:56.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:56.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:55 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.13"}]': finished 2024-03-30T13:59:56.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:55 smithi113 ceph-mon[34341]: osdmap e347: 8 total, 7 up, 7 in 2024-03-30T13:59:56.305 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:56.330 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:56.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:56.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:56.392 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 13:59:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:13:59:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:59:56.432 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:56.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:56.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:56.561 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:56.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:56.587 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:56.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:56.660 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:59:56.688 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:56.712 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:56.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:56.813 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:56.825 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T13:59:56.842 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:56.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:56.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:56.944 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:56.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:56.979 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:56.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:57.080 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T13:59:57.104 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:56 smithi043 ceph-mon[29963]: pgmap v763: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s; 14026/204798 objects degraded (6.849%); 14933/204798 objects misplaced (7.292%); 28 MiB/s, 447 objects/s recovering 2024-03-30T13:59:57.104 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:56 smithi043 ceph-mon[29963]: osdmap e348: 8 total, 7 up, 7 in 2024-03-30T13:59:57.104 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:56 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2887945064' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:59:57.104 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:56 smithi043 ceph-mon[42099]: pgmap v763: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s; 14026/204798 objects degraded (6.849%); 14933/204798 objects misplaced (7.292%); 28 MiB/s, 447 objects/s recovering 2024-03-30T13:59:57.104 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:56 smithi043 ceph-mon[42099]: osdmap e348: 8 total, 7 up, 7 in 2024-03-30T13:59:57.105 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:56 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2887945064' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:59:57.105 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:57.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:57.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T13:59:57.207 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:57.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:56 smithi113 ceph-mon[34341]: pgmap v763: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s; 14026/204798 objects degraded (6.849%); 14933/204798 objects misplaced (7.292%); 28 MiB/s, 447 objects/s recovering 2024-03-30T13:59:57.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:56 smithi113 ceph-mon[34341]: osdmap e348: 8 total, 7 up, 7 in 2024-03-30T13:59:57.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:56 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2887945064' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T13:59:57.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:57.232 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:57.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:57.333 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:57.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:57.357 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:57.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:57.458 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:57.484 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:57.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:57.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:57.586 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:57.612 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:57.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:57.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:57.714 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:57.751 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:57.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:57.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:57.853 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:57.891 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:57.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:57.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:57.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:57 smithi043 ceph-mon[29963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:59:57.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:57 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2887945064' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:59:57.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:57 smithi043 ceph-mon[29963]: osdmap e349: 8 total, 7 up, 7 in 2024-03-30T13:59:57.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:57 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 14026/204798 objects degraded (6.849%), 12 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:57.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:57 smithi043 ceph-mon[29963]: osdmap e350: 8 total, 7 up, 7 in 2024-03-30T13:59:57.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:57 smithi043 ceph-mon[42099]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:59:57.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:57 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2887945064' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:59:57.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:57 smithi043 ceph-mon[42099]: osdmap e349: 8 total, 7 up, 7 in 2024-03-30T13:59:57.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:57 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 14026/204798 objects degraded (6.849%), 12 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:57.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:57 smithi043 ceph-mon[42099]: osdmap e350: 8 total, 7 up, 7 in 2024-03-30T13:59:57.992 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:58.032 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:58.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:58.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:58.134 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:58.170 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:58.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:58.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:58.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:57 smithi113 ceph-mon[34341]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T13:59:58.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:57 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2887945064' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T13:59:58.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:57 smithi113 ceph-mon[34341]: osdmap e349: 8 total, 7 up, 7 in 2024-03-30T13:59:58.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:57 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 14026/204798 objects degraded (6.849%), 12 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-03-30T13:59:58.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:57 smithi113 ceph-mon[34341]: osdmap e350: 8 total, 7 up, 7 in 2024-03-30T13:59:58.273 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:58.306 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:58.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:58.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:58.409 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:58.443 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:58.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:58.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:58.544 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:58.579 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:58.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:58.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:58.681 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T13:59:58.711 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:58.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:58.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T13:59:58.813 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T13:59:58.840 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:58.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:58.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T13:59:58.941 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:58.967 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:58.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:58.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:59.069 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:59.093 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:59.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:59.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:59.112 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:58 smithi043 ceph-mon[29963]: pgmap v767: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 13596/198369 objects degraded (6.854%); 13655/198369 objects misplaced (6.884%); 21 MiB/s, 333 objects/s recovering 2024-03-30T13:59:59.112 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:58 smithi043 ceph-mon[29963]: osdmap e351: 8 total, 7 up, 7 in 2024-03-30T13:59:59.112 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:58 smithi043 ceph-mon[42099]: pgmap v767: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 13596/198369 objects degraded (6.854%); 13655/198369 objects misplaced (6.884%); 21 MiB/s, 333 objects/s recovering 2024-03-30T13:59:59.112 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:58 smithi043 ceph-mon[42099]: osdmap e351: 8 total, 7 up, 7 in 2024-03-30T13:59:59.194 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:59.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:58 smithi113 ceph-mon[34341]: pgmap v767: 158 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 103 active+clean, 12 active+undersized, 10 active+undersized+degraded, 30 active+clean+remapped; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 13596/198369 objects degraded (6.854%); 13655/198369 objects misplaced (6.884%); 21 MiB/s, 333 objects/s recovering 2024-03-30T13:59:59.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:58 smithi113 ceph-mon[34341]: osdmap e351: 8 total, 7 up, 7 in 2024-03-30T13:59:59.218 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:59.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:59.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:59.321 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T13:59:59.345 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:59.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:59.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T13:59:59.447 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T13:59:59.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:59.472 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:59.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T13:59:59.573 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:59.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:59.597 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:59.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:59.698 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T13:59:59.716 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 13:59:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T13:59:59.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:59.739 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T13:59:59.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T13:59:59.841 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T13:59:59.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:59:59.872 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T13:59:59.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T13:59:59.973 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:00.004 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:59 smithi043 ceph-mon[29963]: 1.5 scrub starts 2024-03-30T14:00:00.004 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:59 smithi043 ceph-mon[29963]: 1.5 scrub ok 2024-03-30T14:00:00.005 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:59 smithi043 ceph-mon[29963]: 10.7 scrub starts 2024-03-30T14:00:00.005 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:59 smithi043 ceph-mon[29963]: 6.16 scrub starts 2024-03-30T14:00:00.005 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:59 smithi043 ceph-mon[29963]: 6.16 scrub ok 2024-03-30T14:00:00.005 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:59 smithi043 ceph-mon[29963]: 3.1d scrub starts 2024-03-30T14:00:00.005 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:59 smithi043 ceph-mon[29963]: 3.1d scrub ok 2024-03-30T14:00:00.005 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:59 smithi043 ceph-mon[29963]: 10.7 scrub ok 2024-03-30T14:00:00.005 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:59 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:00.005 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 13:59:59 smithi043 ceph-mon[29963]: osdmap e352: 8 total, 7 up, 7 in 2024-03-30T14:00:00.005 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:59 smithi043 ceph-mon[42099]: 1.5 scrub starts 2024-03-30T14:00:00.006 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:59 smithi043 ceph-mon[42099]: 1.5 scrub ok 2024-03-30T14:00:00.006 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:59 smithi043 ceph-mon[42099]: 10.7 scrub starts 2024-03-30T14:00:00.006 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:59 smithi043 ceph-mon[42099]: 6.16 scrub starts 2024-03-30T14:00:00.006 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:59 smithi043 ceph-mon[42099]: 6.16 scrub ok 2024-03-30T14:00:00.006 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:59 smithi043 ceph-mon[42099]: 3.1d scrub starts 2024-03-30T14:00:00.006 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:59 smithi043 ceph-mon[42099]: 3.1d scrub ok 2024-03-30T14:00:00.006 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:59 smithi043 ceph-mon[42099]: 10.7 scrub ok 2024-03-30T14:00:00.006 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:59 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:00.006 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 13:59:59 smithi043 ceph-mon[42099]: osdmap e352: 8 total, 7 up, 7 in 2024-03-30T14:00:00.006 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T14:00:00.007 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T14:00:00.007 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-03-30T14:00:00.007 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-03-30T14:00:00.007 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T14:00:00.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:00.008 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:00.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:00.109 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:00.148 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:00.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:00.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:00.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:59 smithi113 ceph-mon[34341]: 1.5 scrub starts 2024-03-30T14:00:00.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:59 smithi113 ceph-mon[34341]: 1.5 scrub ok 2024-03-30T14:00:00.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:59 smithi113 ceph-mon[34341]: 10.7 scrub starts 2024-03-30T14:00:00.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:59 smithi113 ceph-mon[34341]: 6.16 scrub starts 2024-03-30T14:00:00.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:59 smithi113 ceph-mon[34341]: 6.16 scrub ok 2024-03-30T14:00:00.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:59 smithi113 ceph-mon[34341]: 3.1d scrub starts 2024-03-30T14:00:00.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:59 smithi113 ceph-mon[34341]: 3.1d scrub ok 2024-03-30T14:00:00.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:59 smithi113 ceph-mon[34341]: 10.7 scrub ok 2024-03-30T14:00:00.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:59 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:00.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 13:59:59 smithi113 ceph-mon[34341]: osdmap e352: 8 total, 7 up, 7 in 2024-03-30T14:00:00.250 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:00.258 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T14:00:00.290 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:00.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:00.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:00.293 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T14:00:00.295 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T14:00:00.334 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T14:00:00.371 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T14:00:00.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:00.413 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T14:00:00.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:00.437 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:00.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:00.454 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T14:00:00.483 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T14:00:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:00:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:00:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T14:00:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:00:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:00:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:00:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:00:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:00:00.538 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:00.571 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:00.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:00.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:00.673 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:00.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:00.712 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:00.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:00.813 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:00.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:00.848 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:00.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:00.895 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 5] out_osds: [7] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-30T14:00:00.896 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T14:00:00.896 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-30T14:00:00.896 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-03-30T14:00:00.948 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:00.983 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:00.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:00.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:01.085 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:01.135 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:01.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:01.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:01.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:01.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: 4.1e scrub starts 2024-03-30T14:00:01.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: 4.1e scrub ok 2024-03-30T14:00:01.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pgmap v770: 158 pgs: 1 active+clean+scrubbing, 1 peering, 104 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.43k op/s; 11793/188115 objects degraded (6.269%); 11805/188115 objects misplaced (6.275%); 49 MiB/s, 787 objects/s recovering 2024-03-30T14:00:01.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 13596/198369 objects degraded (6.854%), 12 pgs degraded, 24 pgs undersized 2024-03-30T14:00:01.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-30T14:00:01.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: [WRN] OSD_DOWN: 1 osds down 2024-03-30T14:00:01.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: osd.2 (root=default,host=smithi043) is down 2024-03-30T14:00:01.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: [WRN] PG_DEGRADED: Degraded data redundancy: 13596/198369 objects degraded (6.854%), 12 pgs degraded, 24 pgs undersized 2024-03-30T14:00:01.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 2.1 is stuck undersized for 9m, current state active+undersized, last acting [3,0] 2024-03-30T14:00:01.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 3.0 is stuck undersized for 9m, current state active+undersized, last acting [1,6] 2024-03-30T14:00:01.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 3.1a is stuck undersized for 9m, current state active+undersized, last acting [4,1] 2024-03-30T14:00:01.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 3.1e is stuck undersized for 9m, current state active+undersized, last acting [3,6] 2024-03-30T14:00:01.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 4.4 is stuck undersized for 9m, current state active+undersized+degraded, last acting [1,3] 2024-03-30T14:00:01.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 4.6 is stuck undersized for 9m, current state active+undersized+degraded, last acting [0,1] 2024-03-30T14:00:01.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 4.d is stuck undersized for 9m, current state active+undersized+degraded, last acting [4,1] 2024-03-30T14:00:01.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 4.12 is stuck undersized for 9m, current state active+undersized+degraded, last acting [1,6] 2024-03-30T14:00:01.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 4.13 is stuck undersized for 9m, current state active+undersized+degraded, last acting [4,6] 2024-03-30T14:00:01.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 4.19 is stuck undersized for 9m, current state active+undersized+degraded, last acting [3,0] 2024-03-30T14:00:01.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 4.1c is stuck undersized for 9m, current state active+undersized+degraded, last acting [1,3] 2024-03-30T14:00:01.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 5.8 is stuck undersized for 9m, current state active+undersized, last acting [0,1] 2024-03-30T14:00:01.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 5.a is stuck undersized for 9m, current state active+undersized, last acting [4,3] 2024-03-30T14:00:01.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 5.1c is stuck undersized for 9m, current state active+undersized, last acting [4,3] 2024-03-30T14:00:01.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 6.0 is stuck undersized for 9m, current state active+undersized, last acting [0,3] 2024-03-30T14:00:01.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 6.1 is stuck undersized for 9m, current state active+undersized, last acting [1,6] 2024-03-30T14:00:01.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 6.2 is stuck undersized for 9m, current state active+undersized, last acting [4,3] 2024-03-30T14:00:01.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 6.e is stuck undersized for 9m, current state active+undersized, last acting [4,1] 2024-03-30T14:00:01.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 6.f is stuck undersized for 9m, current state active+undersized, last acting [3,4] 2024-03-30T14:00:01.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 10.3 is stuck undersized for 69s, current state active+undersized+degraded+remapped+backfilling, last acting [3,4] 2024-03-30T14:00:01.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:01.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: 4.1e scrub starts 2024-03-30T14:00:01.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: 4.1e scrub ok 2024-03-30T14:00:01.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pgmap v770: 158 pgs: 1 active+clean+scrubbing, 1 peering, 104 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.43k op/s; 11793/188115 objects degraded (6.269%); 11805/188115 objects misplaced (6.275%); 49 MiB/s, 787 objects/s recovering 2024-03-30T14:00:01.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 13596/198369 objects degraded (6.854%), 12 pgs degraded, 24 pgs undersized 2024-03-30T14:00:01.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-30T14:00:01.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: [WRN] OSD_DOWN: 1 osds down 2024-03-30T14:00:01.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: osd.2 (root=default,host=smithi043) is down 2024-03-30T14:00:01.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: [WRN] PG_DEGRADED: Degraded data redundancy: 13596/198369 objects degraded (6.854%), 12 pgs degraded, 24 pgs undersized 2024-03-30T14:00:01.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 2.1 is stuck undersized for 9m, current state active+undersized, last acting [3,0] 2024-03-30T14:00:01.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 3.0 is stuck undersized for 9m, current state active+undersized, last acting [1,6] 2024-03-30T14:00:01.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 3.1a is stuck undersized for 9m, current state active+undersized, last acting [4,1] 2024-03-30T14:00:01.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 3.1e is stuck undersized for 9m, current state active+undersized, last acting [3,6] 2024-03-30T14:00:01.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 4.4 is stuck undersized for 9m, current state active+undersized+degraded, last acting [1,3] 2024-03-30T14:00:01.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 4.6 is stuck undersized for 9m, current state active+undersized+degraded, last acting [0,1] 2024-03-30T14:00:01.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 4.d is stuck undersized for 9m, current state active+undersized+degraded, last acting [4,1] 2024-03-30T14:00:01.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 4.12 is stuck undersized for 9m, current state active+undersized+degraded, last acting [1,6] 2024-03-30T14:00:01.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 4.13 is stuck undersized for 9m, current state active+undersized+degraded, last acting [4,6] 2024-03-30T14:00:01.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 4.19 is stuck undersized for 9m, current state active+undersized+degraded, last acting [3,0] 2024-03-30T14:00:01.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 4.1c is stuck undersized for 9m, current state active+undersized+degraded, last acting [1,3] 2024-03-30T14:00:01.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 5.8 is stuck undersized for 9m, current state active+undersized, last acting [0,1] 2024-03-30T14:00:01.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 5.a is stuck undersized for 9m, current state active+undersized, last acting [4,3] 2024-03-30T14:00:01.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 5.1c is stuck undersized for 9m, current state active+undersized, last acting [4,3] 2024-03-30T14:00:01.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 6.0 is stuck undersized for 9m, current state active+undersized, last acting [0,3] 2024-03-30T14:00:01.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 6.1 is stuck undersized for 9m, current state active+undersized, last acting [1,6] 2024-03-30T14:00:01.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 6.2 is stuck undersized for 9m, current state active+undersized, last acting [4,3] 2024-03-30T14:00:01.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 6.e is stuck undersized for 9m, current state active+undersized, last acting [4,1] 2024-03-30T14:00:01.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 6.f is stuck undersized for 9m, current state active+undersized, last acting [3,4] 2024-03-30T14:00:01.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 10.3 is stuck undersized for 69s, current state active+undersized+degraded+remapped+backfilling, last acting [3,4] 2024-03-30T14:00:01.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 10.5 is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,3] 2024-03-30T14:00:01.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 10.7 is stuck undersized for 69s, current state active+undersized+degraded+remapped+backfilling, last acting [6,3] 2024-03-30T14:00:01.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 10.e is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-03-30T14:00:01.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: pg 10.f is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,3] 2024-03-30T14:00:01.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/4115234847' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:00:01.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 10.5 is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,3] 2024-03-30T14:00:01.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 10.7 is stuck undersized for 69s, current state active+undersized+degraded+remapped+backfilling, last acting [6,3] 2024-03-30T14:00:01.148 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 10.e is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-03-30T14:00:01.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: pg 10.f is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,3] 2024-03-30T14:00:01.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:00 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4115234847' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:00:01.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:01.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: 4.1e scrub starts 2024-03-30T14:00:01.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: 4.1e scrub ok 2024-03-30T14:00:01.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pgmap v770: 158 pgs: 1 active+clean+scrubbing, 1 peering, 104 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.43k op/s; 11793/188115 objects degraded (6.269%); 11805/188115 objects misplaced (6.275%); 49 MiB/s, 787 objects/s recovering 2024-03-30T14:00:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 13596/198369 objects degraded (6.854%), 12 pgs degraded, 24 pgs undersized 2024-03-30T14:00:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-30T14:00:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: [WRN] OSD_DOWN: 1 osds down 2024-03-30T14:00:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: osd.2 (root=default,host=smithi043) is down 2024-03-30T14:00:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: [WRN] PG_DEGRADED: Degraded data redundancy: 13596/198369 objects degraded (6.854%), 12 pgs degraded, 24 pgs undersized 2024-03-30T14:00:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 2.1 is stuck undersized for 9m, current state active+undersized, last acting [3,0] 2024-03-30T14:00:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 3.0 is stuck undersized for 9m, current state active+undersized, last acting [1,6] 2024-03-30T14:00:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 3.1a is stuck undersized for 9m, current state active+undersized, last acting [4,1] 2024-03-30T14:00:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 3.1e is stuck undersized for 9m, current state active+undersized, last acting [3,6] 2024-03-30T14:00:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 4.4 is stuck undersized for 9m, current state active+undersized+degraded, last acting [1,3] 2024-03-30T14:00:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 4.6 is stuck undersized for 9m, current state active+undersized+degraded, last acting [0,1] 2024-03-30T14:00:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 4.d is stuck undersized for 9m, current state active+undersized+degraded, last acting [4,1] 2024-03-30T14:00:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 4.12 is stuck undersized for 9m, current state active+undersized+degraded, last acting [1,6] 2024-03-30T14:00:01.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 4.13 is stuck undersized for 9m, current state active+undersized+degraded, last acting [4,6] 2024-03-30T14:00:01.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 4.19 is stuck undersized for 9m, current state active+undersized+degraded, last acting [3,0] 2024-03-30T14:00:01.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 4.1c is stuck undersized for 9m, current state active+undersized+degraded, last acting [1,3] 2024-03-30T14:00:01.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 5.8 is stuck undersized for 9m, current state active+undersized, last acting [0,1] 2024-03-30T14:00:01.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 5.a is stuck undersized for 9m, current state active+undersized, last acting [4,3] 2024-03-30T14:00:01.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 5.1c is stuck undersized for 9m, current state active+undersized, last acting [4,3] 2024-03-30T14:00:01.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 6.0 is stuck undersized for 9m, current state active+undersized, last acting [0,3] 2024-03-30T14:00:01.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 6.1 is stuck undersized for 9m, current state active+undersized, last acting [1,6] 2024-03-30T14:00:01.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 6.2 is stuck undersized for 9m, current state active+undersized, last acting [4,3] 2024-03-30T14:00:01.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 6.e is stuck undersized for 9m, current state active+undersized, last acting [4,1] 2024-03-30T14:00:01.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 6.f is stuck undersized for 9m, current state active+undersized, last acting [3,4] 2024-03-30T14:00:01.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 10.3 is stuck undersized for 69s, current state active+undersized+degraded+remapped+backfilling, last acting [3,4] 2024-03-30T14:00:01.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 10.5 is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,3] 2024-03-30T14:00:01.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 10.7 is stuck undersized for 69s, current state active+undersized+degraded+remapped+backfilling, last acting [6,3] 2024-03-30T14:00:01.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 10.e is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-03-30T14:00:01.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: pg 10.f is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,3] 2024-03-30T14:00:01.219 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:00 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/4115234847' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:00:01.237 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:01.242 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-03-30T14:00:01.276 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:01.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:01.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:01.378 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:01.413 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:01.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:01.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:01.515 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:01.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:01.559 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:01.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:01.660 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:01.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:01.698 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:01.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:01.799 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:01.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:01.835 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:01.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:01.936 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:01.973 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T14:00:01.973 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T14:00:01.993 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:01.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:01.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:02.074 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:00:02.078 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:00:02.095 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:02.140 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:00:02.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:01 smithi043 ceph-mon[29963]: 4.16 scrub starts 2024-03-30T14:00:02.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:01 smithi043 ceph-mon[29963]: 4.16 scrub ok 2024-03-30T14:00:02.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:01 smithi043 ceph-mon[29963]: 5.13 scrub starts 2024-03-30T14:00:02.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:01 smithi043 ceph-mon[29963]: 5.13 scrub ok 2024-03-30T14:00:02.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:01 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:00:02.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:01 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:00:02.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:01 smithi043 ceph-mon[42099]: 4.16 scrub starts 2024-03-30T14:00:02.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:01 smithi043 ceph-mon[42099]: 4.16 scrub ok 2024-03-30T14:00:02.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:01 smithi043 ceph-mon[42099]: 5.13 scrub starts 2024-03-30T14:00:02.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:01 smithi043 ceph-mon[42099]: 5.13 scrub ok 2024-03-30T14:00:02.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:01 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:00:02.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:01 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:00:02.169 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:02.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:02.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:02.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:01 smithi113 ceph-mon[34341]: 4.16 scrub starts 2024-03-30T14:00:02.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:01 smithi113 ceph-mon[34341]: 4.16 scrub ok 2024-03-30T14:00:02.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:01 smithi113 ceph-mon[34341]: 5.13 scrub starts 2024-03-30T14:00:02.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:01 smithi113 ceph-mon[34341]: 5.13 scrub ok 2024-03-30T14:00:02.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:01 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:00:02.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:01 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:00:02.271 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:02.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:02.314 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:02.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:02.415 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:02.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:02.445 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:02.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:02.546 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:02.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:02.586 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:02.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:02.688 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:02.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:02.729 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:02.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:02.830 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:02.874 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:02.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:02.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:02.976 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:03.010 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:03.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:03.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:03.111 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:03.139 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:02 smithi113 ceph-mon[34341]: 10.4 scrub starts 2024-03-30T14:00:03.139 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:02 smithi113 ceph-mon[34341]: pgmap v771: 158 pgs: 1 active+clean+scrubbing, 1 peering, 104 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 895 op/s; 11554/185250 objects degraded (6.237%); 11571/185250 objects misplaced (6.246%); 30 MiB/s, 485 objects/s recovering 2024-03-30T14:00:03.139 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:02 smithi113 ceph-mon[34341]: osdmap e353: 8 total, 7 up, 7 in 2024-03-30T14:00:03.140 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:02 smithi113 ceph-mon[34341]: 4.f scrub starts 2024-03-30T14:00:03.140 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:02 smithi113 ceph-mon[34341]: 4.f scrub ok 2024-03-30T14:00:03.140 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:02 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 11554/185250 objects degraded (6.237%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:03.140 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:02 smithi113 ceph-mon[34341]: osdmap e354: 8 total, 7 up, 7 in 2024-03-30T14:00:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:02 smithi043 ceph-mon[29963]: 10.4 scrub starts 2024-03-30T14:00:03.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:02 smithi043 ceph-mon[29963]: pgmap v771: 158 pgs: 1 active+clean+scrubbing, 1 peering, 104 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 895 op/s; 11554/185250 objects degraded (6.237%); 11571/185250 objects misplaced (6.246%); 30 MiB/s, 485 objects/s recovering 2024-03-30T14:00:03.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:02 smithi043 ceph-mon[29963]: osdmap e353: 8 total, 7 up, 7 in 2024-03-30T14:00:03.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:02 smithi043 ceph-mon[29963]: 4.f scrub starts 2024-03-30T14:00:03.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:02 smithi043 ceph-mon[29963]: 4.f scrub ok 2024-03-30T14:00:03.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:02 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 11554/185250 objects degraded (6.237%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:03.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:02 smithi043 ceph-mon[29963]: osdmap e354: 8 total, 7 up, 7 in 2024-03-30T14:00:03.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:02 smithi043 ceph-mon[42099]: 10.4 scrub starts 2024-03-30T14:00:03.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:02 smithi043 ceph-mon[42099]: pgmap v771: 158 pgs: 1 active+clean+scrubbing, 1 peering, 104 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 895 op/s; 11554/185250 objects degraded (6.237%); 11571/185250 objects misplaced (6.246%); 30 MiB/s, 485 objects/s recovering 2024-03-30T14:00:03.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:02 smithi043 ceph-mon[42099]: osdmap e353: 8 total, 7 up, 7 in 2024-03-30T14:00:03.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:02 smithi043 ceph-mon[42099]: 4.f scrub starts 2024-03-30T14:00:03.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:02 smithi043 ceph-mon[42099]: 4.f scrub ok 2024-03-30T14:00:03.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:02 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 11554/185250 objects degraded (6.237%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:03.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:02 smithi043 ceph-mon[42099]: osdmap e354: 8 total, 7 up, 7 in 2024-03-30T14:00:03.146 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:03.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:03.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:03.248 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:03.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:03.290 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:03.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:03.391 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:03.425 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:03.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:03.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:03.527 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:03.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:03.570 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:03.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:03.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:03.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:03.710 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:03.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:03.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:03.838 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:03.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:03.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:03.940 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:03.975 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T14:00:03.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:03.977 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:03.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:04.079 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:04.103 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:04.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:04.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:04.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:03 smithi043 ceph-mon[29963]: 10.0 scrub starts 2024-03-30T14:00:04.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:03 smithi043 ceph-mon[29963]: 3.1f scrub starts 2024-03-30T14:00:04.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:03 smithi043 ceph-mon[29963]: 3.1f scrub ok 2024-03-30T14:00:04.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:03 smithi043 ceph-mon[29963]: 10.4 scrub ok 2024-03-30T14:00:04.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:03 smithi043 ceph-mon[42099]: 10.0 scrub starts 2024-03-30T14:00:04.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:03 smithi043 ceph-mon[42099]: 3.1f scrub starts 2024-03-30T14:00:04.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:03 smithi043 ceph-mon[42099]: 3.1f scrub ok 2024-03-30T14:00:04.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:03 smithi043 ceph-mon[42099]: 10.4 scrub ok 2024-03-30T14:00:04.205 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:04.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:03 smithi113 ceph-mon[34341]: 10.0 scrub starts 2024-03-30T14:00:04.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:03 smithi113 ceph-mon[34341]: 3.1f scrub starts 2024-03-30T14:00:04.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:03 smithi113 ceph-mon[34341]: 3.1f scrub ok 2024-03-30T14:00:04.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:03 smithi113 ceph-mon[34341]: 10.4 scrub ok 2024-03-30T14:00:04.241 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:04.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:04.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:04.343 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:04.379 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:04.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:04.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:04.481 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:04.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:04.515 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:04.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:04.616 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:04.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:04.651 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:04.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:04.752 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:04.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:04.784 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:04.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:04.885 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:04.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:04.919 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:04.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:05.020 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:05.056 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:05.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:05.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:05.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:04 smithi043 ceph-mon[29963]: 4.5 scrub starts 2024-03-30T14:00:05.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:04 smithi043 ceph-mon[29963]: 4.5 scrub ok 2024-03-30T14:00:05.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:04 smithi043 ceph-mon[29963]: 10.0 scrub ok 2024-03-30T14:00:05.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:04 smithi043 ceph-mon[29963]: pgmap v774: 158 pgs: 1 active+clean+scrubbing, 105 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1010 B/s rd, 0 B/s wr, 1.08k op/s; 11554/181959 objects degraded (6.350%); 10840/181959 objects misplaced (5.957%); 29 MiB/s, 468 objects/s recovering 2024-03-30T14:00:05.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:04 smithi043 ceph-mon[29963]: 3.4 scrub starts 2024-03-30T14:00:05.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:04 smithi043 ceph-mon[29963]: 3.4 scrub ok 2024-03-30T14:00:05.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:04 smithi043 ceph-mon[42099]: 4.5 scrub starts 2024-03-30T14:00:05.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:04 smithi043 ceph-mon[42099]: 4.5 scrub ok 2024-03-30T14:00:05.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:04 smithi043 ceph-mon[42099]: 10.0 scrub ok 2024-03-30T14:00:05.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:04 smithi043 ceph-mon[42099]: pgmap v774: 158 pgs: 1 active+clean+scrubbing, 105 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1010 B/s rd, 0 B/s wr, 1.08k op/s; 11554/181959 objects degraded (6.350%); 10840/181959 objects misplaced (5.957%); 29 MiB/s, 468 objects/s recovering 2024-03-30T14:00:05.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:04 smithi043 ceph-mon[42099]: 3.4 scrub starts 2024-03-30T14:00:05.148 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:04 smithi043 ceph-mon[42099]: 3.4 scrub ok 2024-03-30T14:00:05.158 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:05.192 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:05.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:05.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:05.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:04 smithi113 ceph-mon[34341]: 4.5 scrub starts 2024-03-30T14:00:05.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:04 smithi113 ceph-mon[34341]: 4.5 scrub ok 2024-03-30T14:00:05.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:04 smithi113 ceph-mon[34341]: 10.0 scrub ok 2024-03-30T14:00:05.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:04 smithi113 ceph-mon[34341]: pgmap v774: 158 pgs: 1 active+clean+scrubbing, 105 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1010 B/s rd, 0 B/s wr, 1.08k op/s; 11554/181959 objects degraded (6.350%); 10840/181959 objects misplaced (5.957%); 29 MiB/s, 468 objects/s recovering 2024-03-30T14:00:05.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:04 smithi113 ceph-mon[34341]: 3.4 scrub starts 2024-03-30T14:00:05.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:04 smithi113 ceph-mon[34341]: 3.4 scrub ok 2024-03-30T14:00:05.293 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:05.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:05.330 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:05.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:05.431 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:05.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:05.466 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:05.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:05.567 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:05.606 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:05.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:05.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:05.708 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:05.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:05.747 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:05.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:05.848 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:05.884 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:05.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:05.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:05.987 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:06.023 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:06.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:06.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:06.089 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:05 smithi043 ceph-mon[42099]: 4.13 scrub starts 2024-03-30T14:00:06.090 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:05 smithi043 ceph-mon[42099]: 4.13 scrub ok 2024-03-30T14:00:06.090 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:05 smithi043 ceph-mon[42099]: 3.19 scrub starts 2024-03-30T14:00:06.090 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:05 smithi043 ceph-mon[42099]: 3.19 scrub ok 2024-03-30T14:00:06.090 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:05 smithi043 ceph-mon[42099]: 6.13 scrub starts 2024-03-30T14:00:06.090 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:05 smithi043 ceph-mon[29963]: 4.13 scrub starts 2024-03-30T14:00:06.090 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:05 smithi043 ceph-mon[29963]: 4.13 scrub ok 2024-03-30T14:00:06.090 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:05 smithi043 ceph-mon[29963]: 3.19 scrub starts 2024-03-30T14:00:06.090 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:05 smithi043 ceph-mon[29963]: 3.19 scrub ok 2024-03-30T14:00:06.090 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:05 smithi043 ceph-mon[29963]: 6.13 scrub starts 2024-03-30T14:00:06.126 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:06.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:06.165 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:06.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:06.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:05 smithi113 ceph-mon[34341]: 4.13 scrub starts 2024-03-30T14:00:06.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:05 smithi113 ceph-mon[34341]: 4.13 scrub ok 2024-03-30T14:00:06.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:05 smithi113 ceph-mon[34341]: 3.19 scrub starts 2024-03-30T14:00:06.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:05 smithi113 ceph-mon[34341]: 3.19 scrub ok 2024-03-30T14:00:06.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:05 smithi113 ceph-mon[34341]: 6.13 scrub starts 2024-03-30T14:00:06.267 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:06.301 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:06.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:06.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:06.393 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:00:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:00:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:00:06.403 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:06.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:06.442 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:06.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:06.543 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:06.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:06.583 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:06.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:06.684 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:06.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:06.721 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:06.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:06.822 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:06.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:06.859 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:06.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:06.960 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:06.997 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:06.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:06.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:07.099 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:07.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:06 smithi043 ceph-mon[29963]: 6.13 scrub ok 2024-03-30T14:00:07.125 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:06 smithi043 ceph-mon[29963]: 10.b scrub starts 2024-03-30T14:00:07.125 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:06 smithi043 ceph-mon[29963]: pgmap v775: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 3.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 843 B/s rd, 0 B/s wr, 877 op/s; 10798/172146 objects degraded (6.273%); 10840/172146 objects misplaced (6.297%) 2024-03-30T14:00:07.125 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:06 smithi043 ceph-mon[29963]: 6.7 scrub starts 2024-03-30T14:00:07.125 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:06 smithi043 ceph-mon[29963]: 6.7 scrub ok 2024-03-30T14:00:07.125 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:06 smithi043 ceph-mon[42099]: 6.13 scrub ok 2024-03-30T14:00:07.125 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:06 smithi043 ceph-mon[42099]: 10.b scrub starts 2024-03-30T14:00:07.125 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:06 smithi043 ceph-mon[42099]: pgmap v775: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 3.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 843 B/s rd, 0 B/s wr, 877 op/s; 10798/172146 objects degraded (6.273%); 10840/172146 objects misplaced (6.297%) 2024-03-30T14:00:07.126 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:06 smithi043 ceph-mon[42099]: 6.7 scrub starts 2024-03-30T14:00:07.126 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:06 smithi043 ceph-mon[42099]: 6.7 scrub ok 2024-03-30T14:00:07.136 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:07.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:07.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:07.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:06 smithi113 ceph-mon[34341]: 6.13 scrub ok 2024-03-30T14:00:07.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:06 smithi113 ceph-mon[34341]: 10.b scrub starts 2024-03-30T14:00:07.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:06 smithi113 ceph-mon[34341]: pgmap v775: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 3.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 843 B/s rd, 0 B/s wr, 877 op/s; 10798/172146 objects degraded (6.273%); 10840/172146 objects misplaced (6.297%) 2024-03-30T14:00:07.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:06 smithi113 ceph-mon[34341]: 6.7 scrub starts 2024-03-30T14:00:07.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:06 smithi113 ceph-mon[34341]: 6.7 scrub ok 2024-03-30T14:00:07.238 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:07.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:07.272 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:07.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:07.373 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:07.419 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:07.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:07.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:07.521 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:07.560 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:07.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:07.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:07.661 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:07.696 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:07.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:07.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:07.798 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:07.835 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:07.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:07.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:07.937 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:07.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:07.984 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:07.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:08.085 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:08.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:08.123 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:08.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:08.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:07 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:00:08.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:07 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:00:08.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:07 smithi043 ceph-mon[29963]: 10.b scrub ok 2024-03-30T14:00:08.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:07 smithi043 ceph-mon[29963]: 1.0 scrub starts 2024-03-30T14:00:08.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:07 smithi043 ceph-mon[29963]: 1.0 scrub ok 2024-03-30T14:00:08.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:07 smithi043 ceph-mon[29963]: 6.1c scrub starts 2024-03-30T14:00:08.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:07 smithi043 ceph-mon[29963]: 6.1c scrub ok 2024-03-30T14:00:08.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:07 smithi043 ceph-mon[29963]: osdmap e355: 8 total, 7 up, 7 in 2024-03-30T14:00:08.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:07 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 10798/172146 objects degraded (6.273%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:08.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:07 smithi043 ceph-mon[29963]: osdmap e356: 8 total, 7 up, 7 in 2024-03-30T14:00:08.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:07 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:00:08.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:07 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:00:08.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:07 smithi043 ceph-mon[42099]: 10.b scrub ok 2024-03-30T14:00:08.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:07 smithi043 ceph-mon[42099]: 1.0 scrub starts 2024-03-30T14:00:08.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:07 smithi043 ceph-mon[42099]: 1.0 scrub ok 2024-03-30T14:00:08.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:07 smithi043 ceph-mon[42099]: 6.1c scrub starts 2024-03-30T14:00:08.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:07 smithi043 ceph-mon[42099]: 6.1c scrub ok 2024-03-30T14:00:08.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:07 smithi043 ceph-mon[42099]: osdmap e355: 8 total, 7 up, 7 in 2024-03-30T14:00:08.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:07 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 10798/172146 objects degraded (6.273%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:08.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:07 smithi043 ceph-mon[42099]: osdmap e356: 8 total, 7 up, 7 in 2024-03-30T14:00:08.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:07 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:00:08.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:07 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:00:08.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:07 smithi113 ceph-mon[34341]: 10.b scrub ok 2024-03-30T14:00:08.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:07 smithi113 ceph-mon[34341]: 1.0 scrub starts 2024-03-30T14:00:08.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:07 smithi113 ceph-mon[34341]: 1.0 scrub ok 2024-03-30T14:00:08.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:07 smithi113 ceph-mon[34341]: 6.1c scrub starts 2024-03-30T14:00:08.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:07 smithi113 ceph-mon[34341]: 6.1c scrub ok 2024-03-30T14:00:08.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:07 smithi113 ceph-mon[34341]: osdmap e355: 8 total, 7 up, 7 in 2024-03-30T14:00:08.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:07 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 10798/172146 objects degraded (6.273%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:08.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:07 smithi113 ceph-mon[34341]: osdmap e356: 8 total, 7 up, 7 in 2024-03-30T14:00:08.225 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:08.263 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:08.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:08.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:08.365 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:08.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:08.400 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:08.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:08.501 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:08.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:08.539 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:08.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:08.640 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:08.679 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:08.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:08.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:08.781 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:08.819 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:08.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:08.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:08.921 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:08.955 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:08.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:08.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:08.999 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:00:09.056 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:09.110 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:09.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:09.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:09.137 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:08 smithi043 ceph-mon[29963]: pgmap v778: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 874 B/s rd, 0 B/s wr, 1.04k op/s; 10348/167010 objects degraded (6.196%); 10410/167010 objects misplaced (6.233%) 2024-03-30T14:00:09.138 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:08 smithi043 ceph-mon[42099]: pgmap v778: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 874 B/s rd, 0 B/s wr, 1.04k op/s; 10348/167010 objects degraded (6.196%); 10410/167010 objects misplaced (6.233%) 2024-03-30T14:00:09.213 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:09.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:08 smithi113 ceph-mon[34341]: pgmap v778: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 874 B/s rd, 0 B/s wr, 1.04k op/s; 10348/167010 objects degraded (6.196%); 10410/167010 objects misplaced (6.233%) 2024-03-30T14:00:09.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:09.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:09.352 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:09.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:09.390 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:09.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:09.491 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:09.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:09.528 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:09.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:09.629 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:09.669 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:09.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:09.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:09.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:00:09 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:00:09.771 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:09.811 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:09.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:09.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:09.913 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:09.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:09.949 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:09.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:10.050 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:10.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:10.090 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:10.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:10.191 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:10.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:09 smithi113 ceph-mon[34341]: 5.11 scrub starts 2024-03-30T14:00:10.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:09 smithi113 ceph-mon[34341]: 5.11 scrub ok 2024-03-30T14:00:10.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:09 smithi113 ceph-mon[34341]: 10.8 scrub starts 2024-03-30T14:00:10.226 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:10.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:10.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:10.328 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:10.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:10.368 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:10.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:10.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:09 smithi043 ceph-mon[29963]: 5.11 scrub starts 2024-03-30T14:00:10.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:09 smithi043 ceph-mon[29963]: 5.11 scrub ok 2024-03-30T14:00:10.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:09 smithi043 ceph-mon[29963]: 10.8 scrub starts 2024-03-30T14:00:10.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:09 smithi043 ceph-mon[42099]: 5.11 scrub starts 2024-03-30T14:00:10.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:09 smithi043 ceph-mon[42099]: 5.11 scrub ok 2024-03-30T14:00:10.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:09 smithi043 ceph-mon[42099]: 10.8 scrub starts 2024-03-30T14:00:10.469 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:10.504 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:10.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:10.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:10.606 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:10.639 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:10.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:10.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:10.741 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:10.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:10.780 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:10.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:10.881 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:10.920 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:10.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:10.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:11.022 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:11.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:11.056 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:11.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:11.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:10 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:11.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:10 smithi043 ceph-mon[29963]: 6.1f scrub starts 2024-03-30T14:00:11.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:10 smithi043 ceph-mon[29963]: 10.d scrub starts 2024-03-30T14:00:11.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:10 smithi043 ceph-mon[29963]: 5.1e scrub starts 2024-03-30T14:00:11.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:10 smithi043 ceph-mon[29963]: 5.1e scrub ok 2024-03-30T14:00:11.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:10 smithi043 ceph-mon[29963]: 6.1f scrub ok 2024-03-30T14:00:11.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:10 smithi043 ceph-mon[29963]: 10.8 scrub ok 2024-03-30T14:00:11.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:10 smithi043 ceph-mon[29963]: pgmap v779: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 3.1 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 722 B/s rd, 0 B/s wr, 1.62k op/s; 9455/150792 objects degraded (6.270%); 9508/150792 objects misplaced (6.305%) 2024-03-30T14:00:11.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:10 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:00:11.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:10 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:00:11.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:10 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:11.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:10 smithi043 ceph-mon[42099]: 6.1f scrub starts 2024-03-30T14:00:11.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:10 smithi043 ceph-mon[42099]: 10.d scrub starts 2024-03-30T14:00:11.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:10 smithi043 ceph-mon[42099]: 5.1e scrub starts 2024-03-30T14:00:11.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:10 smithi043 ceph-mon[42099]: 5.1e scrub ok 2024-03-30T14:00:11.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:10 smithi043 ceph-mon[42099]: 6.1f scrub ok 2024-03-30T14:00:11.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:10 smithi043 ceph-mon[42099]: 10.8 scrub ok 2024-03-30T14:00:11.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:10 smithi043 ceph-mon[42099]: pgmap v779: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 3.1 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 722 B/s rd, 0 B/s wr, 1.62k op/s; 9455/150792 objects degraded (6.270%); 9508/150792 objects misplaced (6.305%) 2024-03-30T14:00:11.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:10 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:00:11.147 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:10 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:00:11.157 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:11.194 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:11.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:11.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:11.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:10 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:11.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:10 smithi113 ceph-mon[34341]: 6.1f scrub starts 2024-03-30T14:00:11.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:10 smithi113 ceph-mon[34341]: 10.d scrub starts 2024-03-30T14:00:11.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:10 smithi113 ceph-mon[34341]: 5.1e scrub starts 2024-03-30T14:00:11.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:10 smithi113 ceph-mon[34341]: 5.1e scrub ok 2024-03-30T14:00:11.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:10 smithi113 ceph-mon[34341]: 6.1f scrub ok 2024-03-30T14:00:11.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:10 smithi113 ceph-mon[34341]: 10.8 scrub ok 2024-03-30T14:00:11.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:10 smithi113 ceph-mon[34341]: pgmap v779: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 3.1 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 722 B/s rd, 0 B/s wr, 1.62k op/s; 9455/150792 objects degraded (6.270%); 9508/150792 objects misplaced (6.305%) 2024-03-30T14:00:11.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:10 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:00:11.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:10 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:00:11.296 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:11.326 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:11.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:11.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:11.428 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:11.452 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:11.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:11.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:11.554 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:11.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:11.592 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:11.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:11.693 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:11.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:11.728 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:11.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:11.829 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:11.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:11.865 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:11.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:11.966 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:12.002 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:12.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:12.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:12.104 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:12.140 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:12.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:12.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:12.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:11 smithi113 ceph-mon[34341]: 10.d scrub ok 2024-03-30T14:00:12.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:11 smithi113 ceph-mon[34341]: osdmap e357: 8 total, 7 up, 7 in 2024-03-30T14:00:12.241 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:12.277 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:12.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:12.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:12.379 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:12.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:11 smithi043 ceph-mon[29963]: 10.d scrub ok 2024-03-30T14:00:12.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:11 smithi043 ceph-mon[29963]: osdmap e357: 8 total, 7 up, 7 in 2024-03-30T14:00:12.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:11 smithi043 ceph-mon[42099]: 10.d scrub ok 2024-03-30T14:00:12.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:11 smithi043 ceph-mon[42099]: osdmap e357: 8 total, 7 up, 7 in 2024-03-30T14:00:12.414 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:12.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:12.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:12.515 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:12.551 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:12.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:12.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:12.653 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:12.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:12.693 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:12.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:12.794 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:12.831 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:12.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:12.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:12.934 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:12.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:12.959 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:12.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:13.060 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:13.084 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:13.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:13.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[29963]: 5.10 scrub starts 2024-03-30T14:00:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[29963]: 5.10 scrub ok 2024-03-30T14:00:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[29963]: 3.8 scrub starts 2024-03-30T14:00:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[29963]: 3.8 scrub ok 2024-03-30T14:00:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[29963]: pgmap v781: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.50k op/s; 9003/145107 objects degraded (6.204%); 9009/145107 objects misplaced (6.209%) 2024-03-30T14:00:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:00:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:00:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[29963]: osdmap e358: 8 total, 7 up, 7 in 2024-03-30T14:00:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 9003/145107 objects degraded (6.204%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3077388345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:00:13.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:00:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[42099]: 5.10 scrub starts 2024-03-30T14:00:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[42099]: 5.10 scrub ok 2024-03-30T14:00:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[42099]: 3.8 scrub starts 2024-03-30T14:00:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[42099]: 3.8 scrub ok 2024-03-30T14:00:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[42099]: pgmap v781: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.50k op/s; 9003/145107 objects degraded (6.204%); 9009/145107 objects misplaced (6.209%) 2024-03-30T14:00:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:00:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:00:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[42099]: osdmap e358: 8 total, 7 up, 7 in 2024-03-30T14:00:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 9003/145107 objects degraded (6.204%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3077388345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:00:13.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:12 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:00:13.186 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:13.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:12 smithi113 ceph-mon[34341]: 5.10 scrub starts 2024-03-30T14:00:13.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:12 smithi113 ceph-mon[34341]: 5.10 scrub ok 2024-03-30T14:00:13.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:12 smithi113 ceph-mon[34341]: 3.8 scrub starts 2024-03-30T14:00:13.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:12 smithi113 ceph-mon[34341]: 3.8 scrub ok 2024-03-30T14:00:13.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:12 smithi113 ceph-mon[34341]: pgmap v781: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.50k op/s; 9003/145107 objects degraded (6.204%); 9009/145107 objects misplaced (6.209%) 2024-03-30T14:00:13.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:12 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:00:13.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:12 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:00:13.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:12 smithi113 ceph-mon[34341]: osdmap e358: 8 total, 7 up, 7 in 2024-03-30T14:00:13.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:12 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 9003/145107 objects degraded (6.204%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:13.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:12 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3077388345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:00:13.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:12 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:00:13.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:13.221 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:13.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:13.322 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:13.361 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:13.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:13.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:13.463 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:13.497 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:13.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:13.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:13.599 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:13.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:13.639 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:13.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:13.640 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-03-30T14:00:13.740 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:13.775 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:13.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:13.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:13.877 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:13.900 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:13.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:13.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:14.002 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:14.026 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:14.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:14.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:14.128 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:14.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:14.168 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:14.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:14.195 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T14:00:14.196 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:00:14.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:13 smithi043 ceph-mon[29963]: 4.1 scrub starts 2024-03-30T14:00:14.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:13 smithi043 ceph-mon[29963]: 4.1 scrub ok 2024-03-30T14:00:14.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:13 smithi043 ceph-mon[29963]: 5.1f scrub starts 2024-03-30T14:00:14.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:13 smithi043 ceph-mon[29963]: 5.1f scrub ok 2024-03-30T14:00:14.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:13 smithi043 ceph-mon[29963]: 5.12 scrub starts 2024-03-30T14:00:14.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:13 smithi043 ceph-mon[29963]: 5.12 scrub ok 2024-03-30T14:00:14.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:13 smithi043 ceph-mon[29963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:00:14.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:13 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:00:14.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:13 smithi043 ceph-mon[29963]: osdmap e359: 8 total, 7 up, 7 in 2024-03-30T14:00:14.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:13 smithi043 ceph-mon[42099]: 4.1 scrub starts 2024-03-30T14:00:14.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:13 smithi043 ceph-mon[42099]: 4.1 scrub ok 2024-03-30T14:00:14.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:13 smithi043 ceph-mon[42099]: 5.1f scrub starts 2024-03-30T14:00:14.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:13 smithi043 ceph-mon[42099]: 5.1f scrub ok 2024-03-30T14:00:14.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:13 smithi043 ceph-mon[42099]: 5.12 scrub starts 2024-03-30T14:00:14.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:13 smithi043 ceph-mon[42099]: 5.12 scrub ok 2024-03-30T14:00:14.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:13 smithi043 ceph-mon[42099]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:00:14.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:13 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:00:14.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:13 smithi043 ceph-mon[42099]: osdmap e359: 8 total, 7 up, 7 in 2024-03-30T14:00:14.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:13 smithi113 ceph-mon[34341]: 4.1 scrub starts 2024-03-30T14:00:14.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:13 smithi113 ceph-mon[34341]: 4.1 scrub ok 2024-03-30T14:00:14.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:13 smithi113 ceph-mon[34341]: 5.1f scrub starts 2024-03-30T14:00:14.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:13 smithi113 ceph-mon[34341]: 5.1f scrub ok 2024-03-30T14:00:14.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:13 smithi113 ceph-mon[34341]: 5.12 scrub starts 2024-03-30T14:00:14.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:13 smithi113 ceph-mon[34341]: 5.12 scrub ok 2024-03-30T14:00:14.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:13 smithi113 ceph-mon[34341]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:00:14.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:13 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:00:14.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:13 smithi113 ceph-mon[34341]: osdmap e359: 8 total, 7 up, 7 in 2024-03-30T14:00:14.269 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:14.309 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:14.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:14.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:14.411 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:14.447 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:14.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:14.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:14.548 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:14.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:14.584 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:14.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:14.685 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:14.719 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:14.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:14.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:14.821 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:14.858 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:14.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:14.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:14.959 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:14.994 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:14.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:14.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:15.096 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:15.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:15.136 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:15.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:15.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:14 smithi113 ceph-mon[34341]: 10.a scrub starts 2024-03-30T14:00:15.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:14 smithi113 ceph-mon[34341]: pgmap v784: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.22k op/s; 9003/145107 objects degraded (6.204%); 9009/145107 objects misplaced (6.209%) 2024-03-30T14:00:15.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:14 smithi113 ceph-mon[34341]: 10.6 scrub starts 2024-03-30T14:00:15.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:14 smithi113 ceph-mon[34341]: 1.2 scrub starts 2024-03-30T14:00:15.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:14 smithi113 ceph-mon[34341]: 1.2 scrub ok 2024-03-30T14:00:15.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:14 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:00:15.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:14 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:15.238 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:15.268 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:15.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:15.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:15.373 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:14 smithi043 ceph-mon[29963]: 10.a scrub starts 2024-03-30T14:00:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:14 smithi043 ceph-mon[29963]: pgmap v784: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.22k op/s; 9003/145107 objects degraded (6.204%); 9009/145107 objects misplaced (6.209%) 2024-03-30T14:00:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:14 smithi043 ceph-mon[29963]: 10.6 scrub starts 2024-03-30T14:00:15.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:14 smithi043 ceph-mon[29963]: 1.2 scrub starts 2024-03-30T14:00:15.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:14 smithi043 ceph-mon[29963]: 1.2 scrub ok 2024-03-30T14:00:15.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:14 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:00:15.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:14 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:15.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:14 smithi043 ceph-mon[42099]: 10.a scrub starts 2024-03-30T14:00:15.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:14 smithi043 ceph-mon[42099]: pgmap v784: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.22k op/s; 9003/145107 objects degraded (6.204%); 9009/145107 objects misplaced (6.209%) 2024-03-30T14:00:15.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:14 smithi043 ceph-mon[42099]: 10.6 scrub starts 2024-03-30T14:00:15.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:14 smithi043 ceph-mon[42099]: 1.2 scrub starts 2024-03-30T14:00:15.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:14 smithi043 ceph-mon[42099]: 1.2 scrub ok 2024-03-30T14:00:15.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:14 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:00:15.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:14 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:15.407 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:15.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:15.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:15.509 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:15.547 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:15.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:15.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:15.649 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:15.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:15.679 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:15.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:15.780 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:15.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:15.826 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:15.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:15.927 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:15.964 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:15.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:15.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:16.066 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:16.090 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:15 smithi043 ceph-mon[29963]: 6.12 scrub starts 2024-03-30T14:00:16.090 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:15 smithi043 ceph-mon[29963]: 6.12 scrub ok 2024-03-30T14:00:16.090 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:15 smithi043 ceph-mon[29963]: 10.a scrub ok 2024-03-30T14:00:16.090 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:15 smithi043 ceph-mon[29963]: 10.6 scrub ok 2024-03-30T14:00:16.091 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:15 smithi043 ceph-mon[42099]: 6.12 scrub starts 2024-03-30T14:00:16.091 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:15 smithi043 ceph-mon[42099]: 6.12 scrub ok 2024-03-30T14:00:16.091 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:15 smithi043 ceph-mon[42099]: 10.a scrub ok 2024-03-30T14:00:16.091 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:15 smithi043 ceph-mon[42099]: 10.6 scrub ok 2024-03-30T14:00:16.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:16.104 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:16.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:16.205 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:16.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:15 smithi113 ceph-mon[34341]: 6.12 scrub starts 2024-03-30T14:00:16.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:15 smithi113 ceph-mon[34341]: 6.12 scrub ok 2024-03-30T14:00:16.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:15 smithi113 ceph-mon[34341]: 10.a scrub ok 2024-03-30T14:00:16.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:15 smithi113 ceph-mon[34341]: 10.6 scrub ok 2024-03-30T14:00:16.235 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:16.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:16.337 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:16.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:16.376 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:16.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:16.394 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:00:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:00:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:00:16.477 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:16.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:16.514 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:16.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:16.615 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:16.651 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:16.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:16.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:16.753 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:16.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:16.788 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:16.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:16.889 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:16.923 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:16.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:16.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:17.025 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:17.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:17.064 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:17.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:17.165 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:17.189 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:16 smithi043 ceph-mon[29963]: 10.c scrub starts 2024-03-30T14:00:17.189 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:16 smithi043 ceph-mon[29963]: pgmap v785: 158 pgs: 2 active+clean+scrubbing, 104 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.18k op/s; 8168/129492 objects degraded (6.308%); 8129/129492 objects misplaced (6.278%) 2024-03-30T14:00:17.189 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:16 smithi043 ceph-mon[29963]: 1.8 scrub starts 2024-03-30T14:00:17.189 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:16 smithi043 ceph-mon[29963]: 1.8 scrub ok 2024-03-30T14:00:17.189 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:16 smithi043 ceph-mon[29963]: 10.c scrub ok 2024-03-30T14:00:17.190 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:16 smithi043 ceph-mon[42099]: 10.c scrub starts 2024-03-30T14:00:17.190 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:16 smithi043 ceph-mon[42099]: pgmap v785: 158 pgs: 2 active+clean+scrubbing, 104 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.18k op/s; 8168/129492 objects degraded (6.308%); 8129/129492 objects misplaced (6.278%) 2024-03-30T14:00:17.190 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:16 smithi043 ceph-mon[42099]: 1.8 scrub starts 2024-03-30T14:00:17.190 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:16 smithi043 ceph-mon[42099]: 1.8 scrub ok 2024-03-30T14:00:17.190 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:16 smithi043 ceph-mon[42099]: 10.c scrub ok 2024-03-30T14:00:17.202 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:17.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:17.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:17.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:16 smithi113 ceph-mon[34341]: 10.c scrub starts 2024-03-30T14:00:17.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:16 smithi113 ceph-mon[34341]: pgmap v785: 158 pgs: 2 active+clean+scrubbing, 104 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.18k op/s; 8168/129492 objects degraded (6.308%); 8129/129492 objects misplaced (6.278%) 2024-03-30T14:00:17.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:16 smithi113 ceph-mon[34341]: 1.8 scrub starts 2024-03-30T14:00:17.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:16 smithi113 ceph-mon[34341]: 1.8 scrub ok 2024-03-30T14:00:17.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:16 smithi113 ceph-mon[34341]: 10.c scrub ok 2024-03-30T14:00:17.236 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-30T14:00:17.304 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:17.348 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:17.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:17.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:17.450 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:17.484 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:17.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:17.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:17.490 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T14:00:17.490 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T14:00:17.490 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 359, 2024-03-30T14:00:17.490 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "7ee932fa-ee9a-11ee-b647-cb9ed24678a4", 2024-03-30T14:00:17.490 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-03-30T13:37:07.346020+0000", 2024-03-30T14:00:17.490 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-03-30T14:00:13.634096+0000", 2024-03-30T14:00:17.491 INFO:teuthology.orchestra.run.smithi043.stdout: "last_up_change": "2024-03-30T13:50:07.489747+0000", 2024-03-30T14:00:17.491 INFO:teuthology.orchestra.run.smithi043.stdout: "last_in_change": "2024-03-30T13:58:46.155265+0000", 2024-03-30T14:00:17.491 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-30T14:00:17.491 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_num": 5799936, 2024-03-30T14:00:17.491 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_set": [ 2024-03-30T14:00:17.491 INFO:teuthology.orchestra.run.smithi043.stdout: "pglog_hardlimit", 2024-03-30T14:00:17.491 INFO:teuthology.orchestra.run.smithi043.stdout: "purged_snapdirs", 2024-03-30T14:00:17.491 INFO:teuthology.orchestra.run.smithi043.stdout: "recovery_deletes", 2024-03-30T14:00:17.491 INFO:teuthology.orchestra.run.smithi043.stdout: "sortbitwise" 2024-03-30T14:00:17.491 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T14:00:17.491 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_version": 18, 2024-03-30T14:00:17.491 INFO:teuthology.orchestra.run.smithi043.stdout: "full_ratio": 0.94999998807907104, 2024-03-30T14:00:17.492 INFO:teuthology.orchestra.run.smithi043.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-30T14:00:17.492 INFO:teuthology.orchestra.run.smithi043.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-30T14:00:17.492 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_snapshot": "", 2024-03-30T14:00:17.492 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_max": 10, 2024-03-30T14:00:17.492 INFO:teuthology.orchestra.run.smithi043.stdout: "max_osd": 8, 2024-03-30T14:00:17.492 INFO:teuthology.orchestra.run.smithi043.stdout: "require_min_compat_client": "luminous", 2024-03-30T14:00:17.492 INFO:teuthology.orchestra.run.smithi043.stdout: "min_compat_client": "luminous", 2024-03-30T14:00:17.492 INFO:teuthology.orchestra.run.smithi043.stdout: "require_osd_release": "squid", 2024-03-30T14:00:17.492 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_crimson": false, 2024-03-30T14:00:17.492 INFO:teuthology.orchestra.run.smithi043.stdout: "pools": [ 2024-03-30T14:00:17.492 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.492 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 1, 2024-03-30T14:00:17.492 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".mgr", 2024-03-30T14:00:17.492 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:40:27.658979+0000", 2024-03-30T14:00:17.493 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T14:00:17.493 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T14:00:17.493 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:00:17.493 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:00:17.493 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:00:17.493 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:00:17.493 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:00:17.493 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:00:17.493 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:00:17.493 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:00:17.493 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:00:17.493 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:00:17.493 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 11, 2024-03-30T14:00:17.494 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 11, 2024-03-30T14:00:17.494 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 11, 2024-03-30T14:00:17.494 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 11, 2024-03-30T14:00:17.494 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 11, 2024-03-30T14:00:17.494 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:00:17.494 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:00:17.494 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:00:17.494 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:00:17.494 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:00:17.494 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:00:17.494 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:00:17.494 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.494 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "248", 2024-03-30T14:00:17.495 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:00:17.495 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:00:17.495 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "202", 2024-03-30T14:00:17.495 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:00:17.495 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:00:17.495 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T14:00:17.495 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T14:00:17.495 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:00:17.495 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:00:17.495 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:00:17.495 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:00:17.495 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:00:17.495 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:00:17.496 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:00:17.496 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:00:17.496 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:00:17.496 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:00:17.496 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:00:17.496 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:00:17.496 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:00:17.496 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:00:17.496 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:00:17.496 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:00:17.496 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:00:17.496 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:00:17.496 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:00:17.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.497 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:00:17.497 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:00:17.497 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:00:17.497 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:00:17.497 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:00:17.497 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:00:17.497 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:00:17.497 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:00:17.497 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:00:17.497 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:00:17.497 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:00:17.497 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-03-30T14:00:17.498 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_max": 32, 2024-03-30T14:00:17.498 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_min": 1 2024-03-30T14:00:17.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.498 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:00:17.498 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": {} 2024-03-30T14:00:17.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.498 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:00:17.498 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:00:17.498 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.9099999666213989, 2024-03-30T14:00:17.498 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.9099999666213989, 2024-03-30T14:00:17.498 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.86000001430511475, 2024-03-30T14:00:17.498 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.6399999856948853, 2024-03-30T14:00:17.498 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.6399999856948853, 2024-03-30T14:00:17.499 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T14:00:17.499 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:00:17.499 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T14:00:17.499 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.499 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.499 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 2, 2024-03-30T14:00:17.499 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "datapool", 2024-03-30T14:00:17.499 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:42.989975+0000", 2024-03-30T14:00:17.499 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 8193, 2024-03-30T14:00:17.499 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T14:00:17.499 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:00:17.500 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:00:17.500 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:00:17.500 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:00:17.500 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:00:17.500 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:00:17.500 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:00:17.500 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:00:17.500 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:00:17.500 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:00:17.500 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 3, 2024-03-30T14:00:17.500 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 3, 2024-03-30T14:00:17.500 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 3, 2024-03-30T14:00:17.500 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 3, 2024-03-30T14:00:17.500 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 3, 2024-03-30T14:00:17.501 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:00:17.501 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:00:17.501 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:00:17.501 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:00:17.501 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:00:17.501 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:00:17.501 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:00:17.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.501 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "68", 2024-03-30T14:00:17.501 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:00:17.501 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:00:17.501 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T14:00:17.501 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:00:17.501 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:00:17.501 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 2, 2024-03-30T14:00:17.502 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 68, 2024-03-30T14:00:17.502 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:00:17.502 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:00:17.502 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:00:17.502 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:00:17.502 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:00:17.502 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:00:17.502 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:00:17.502 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:00:17.502 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:00:17.502 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:00:17.502 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:00:17.503 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:00:17.503 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:00:17.503 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:00:17.503 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:00:17.503 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:00:17.503 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:00:17.503 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:00:17.503 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:00:17.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.503 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:00:17.503 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:00:17.503 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:00:17.504 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:00:17.504 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:00:17.504 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:00:17.504 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:00:17.504 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:00:17.504 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:00:17.504 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:00:17.504 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:00:17.504 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T14:00:17.504 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:00:17.504 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd": {} 2024-03-30T14:00:17.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.504 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:00:17.504 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:00:17.504 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.3499999046325684, 2024-03-30T14:00:17.505 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.3499999046325684, 2024-03-30T14:00:17.505 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.70999997854232788, 2024-03-30T14:00:17.505 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.6699999570846558, 2024-03-30T14:00:17.505 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.6699999570846558, 2024-03-30T14:00:17.505 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T14:00:17.505 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:00:17.505 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T14:00:17.505 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.505 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.505 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 3, 2024-03-30T14:00:17.505 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".rgw.root", 2024-03-30T14:00:17.505 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:43.910306+0000", 2024-03-30T14:00:17.505 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T14:00:17.506 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T14:00:17.506 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:00:17.506 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:00:17.506 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:00:17.506 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:00:17.506 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:00:17.506 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:00:17.506 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:00:17.506 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:00:17.506 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:00:17.506 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:00:17.506 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T14:00:17.506 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T14:00:17.506 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T14:00:17.507 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T14:00:17.507 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T14:00:17.507 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:00:17.507 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:00:17.507 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:00:17.507 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:00:17.507 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:00:17.507 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:00:17.507 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:00:17.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.507 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "66", 2024-03-30T14:00:17.507 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:00:17.507 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:00:17.507 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T14:00:17.508 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:00:17.508 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:00:17.508 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T14:00:17.508 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T14:00:17.508 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:00:17.508 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:00:17.508 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:00:17.508 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:00:17.508 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:00:17.508 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:00:17.508 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:00:17.508 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:00:17.508 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:00:17.508 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:00:17.509 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:00:17.509 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:00:17.509 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:00:17.509 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:00:17.509 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:00:17.509 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:00:17.509 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:00:17.509 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:00:17.509 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:00:17.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.509 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:00:17.509 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:00:17.510 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:00:17.510 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:00:17.510 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:00:17.510 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:00:17.510 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:00:17.510 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:00:17.510 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:00:17.510 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:00:17.510 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:00:17.510 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T14:00:17.510 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:00:17.510 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T14:00:17.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.510 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:00:17.511 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:00:17.511 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.7400000095367432, 2024-03-30T14:00:17.511 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.7400000095367432, 2024-03-30T14:00:17.511 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.86000001430511475, 2024-03-30T14:00:17.511 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.5, 2024-03-30T14:00:17.511 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.5, 2024-03-30T14:00:17.511 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T14:00:17.511 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:00:17.511 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T14:00:17.511 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.512 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 4, 2024-03-30T14:00:17.512 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.log", 2024-03-30T14:00:17.512 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:45.633132+0000", 2024-03-30T14:00:17.512 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T14:00:17.512 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T14:00:17.512 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:00:17.512 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:00:17.512 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:00:17.512 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:00:17.513 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:00:17.513 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:00:17.513 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:00:17.513 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:00:17.513 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:00:17.513 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:00:17.513 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T14:00:17.513 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T14:00:17.513 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T14:00:17.513 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T14:00:17.513 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T14:00:17.513 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:00:17.513 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:00:17.513 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:00:17.514 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:00:17.514 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:00:17.514 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:00:17.514 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:00:17.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.514 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "68", 2024-03-30T14:00:17.514 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:00:17.514 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:00:17.514 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T14:00:17.514 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:00:17.514 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:00:17.514 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T14:00:17.514 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T14:00:17.514 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:00:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:00:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:00:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:00:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:00:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:00:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:00:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:00:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:00:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:00:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:00:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:00:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:00:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:00:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:00:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:00:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:00:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:00:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:00:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:00:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:00:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:00:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:00:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:00:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:00:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:00:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:00:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:00:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:00:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:00:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T14:00:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:00:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T14:00:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:00:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:00:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.9700000286102295, 2024-03-30T14:00:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.9700000286102295, 2024-03-30T14:00:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.86000001430511475, 2024-03-30T14:00:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.690000057220459, 2024-03-30T14:00:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.690000057220459, 2024-03-30T14:00:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T14:00:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:00:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T14:00:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 5, 2024-03-30T14:00:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.control", 2024-03-30T14:00:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:47.570084+0000", 2024-03-30T14:00:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T14:00:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T14:00:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:00:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:00:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:00:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:00:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:00:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:00:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:00:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:00:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:00:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:00:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T14:00:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T14:00:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T14:00:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T14:00:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T14:00:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:00:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:00:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:00:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:00:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:00:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:00:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:00:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "70", 2024-03-30T14:00:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:00:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:00:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T14:00:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:00:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:00:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T14:00:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T14:00:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:00:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:00:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:00:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:00:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:00:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:00:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:00:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:00:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:00:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:00:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:00:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:00:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:00:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:00:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:00:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:00:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:00:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:00:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:00:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:00:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:00:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:00:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:00:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:00:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:00:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:00:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:00:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:00:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:00:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:00:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T14:00:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:00:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T14:00:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:00:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:00:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.7400000095367432, 2024-03-30T14:00:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.7400000095367432, 2024-03-30T14:00:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.86000001430511475, 2024-03-30T14:00:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.5, 2024-03-30T14:00:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.5, 2024-03-30T14:00:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T14:00:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:00:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T14:00:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 6, 2024-03-30T14:00:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.meta", 2024-03-30T14:00:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:49.637189+0000", 2024-03-30T14:00:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T14:00:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T14:00:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:00:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:00:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:00:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:00:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:00:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:00:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:00:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:00:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:00:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:00:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T14:00:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T14:00:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T14:00:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T14:00:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T14:00:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:00:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:00:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:00:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:00:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:00:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:00:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:00:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "72", 2024-03-30T14:00:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:00:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:00:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T14:00:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:00:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:00:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T14:00:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T14:00:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:00:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:00:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:00:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:00:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:00:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:00:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:00:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:00:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:00:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:00:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:00:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:00:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:00:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:00:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:00:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:00:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:00:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:00:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:00:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:00:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:00:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:00:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:00:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:00:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:00:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:00:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:00:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:00:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:00:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:00:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-03-30T14:00:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_bias": 4 2024-03-30T14:00:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:00:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T14:00:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:00:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:00:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.9700000286102295, 2024-03-30T14:00:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.9700000286102295, 2024-03-30T14:00:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.86000001430511475, 2024-03-30T14:00:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.690000057220459, 2024-03-30T14:00:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.690000057220459, 2024-03-30T14:00:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T14:00:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:00:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T14:00:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 10, 2024-03-30T14:00:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "unique_pool_3", 2024-03-30T14:00:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:56:54.614701+0000", 2024-03-30T14:00:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T14:00:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T14:00:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:00:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:00:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:00:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:00:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:00:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:00:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:00:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:00:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:00:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:00:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 16, 2024-03-30T14:00:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 16, 2024-03-30T14:00:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 16, 2024-03-30T14:00:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 16, 2024-03-30T14:00:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 16, 2024-03-30T14:00:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:00:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:00:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:00:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:00:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:00:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:00:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:00:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "295", 2024-03-30T14:00:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:00:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:00:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T14:00:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:00:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:00:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T14:00:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T14:00:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:00:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:00:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:00:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:00:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:00:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:00:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:00:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:00:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:00:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:00:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:00:17.536 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:00:17.536 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:00:17.536 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:00:17.536 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:00:17.536 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:00:17.536 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:00:17.536 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:00:17.536 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:00:17.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.536 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:00:17.536 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:00:17.536 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:00:17.536 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:00:17.536 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:00:17.537 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:00:17.537 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:00:17.537 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:00:17.537 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:00:17.537 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:00:17.537 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:00:17.537 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T14:00:17.537 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:00:17.537 INFO:teuthology.orchestra.run.smithi043.stdout: "rados": {} 2024-03-30T14:00:17.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.537 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:00:17.537 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:00:17.537 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.619999885559082, 2024-03-30T14:00:17.537 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.619999885559082, 2024-03-30T14:00:17.537 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.86000001430511475, 2024-03-30T14:00:17.538 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2.25, 2024-03-30T14:00:17.538 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2.25, 2024-03-30T14:00:17.538 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.875, 2024-03-30T14:00:17.538 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:00:17.538 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-30T14:00:17.538 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.538 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.538 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T14:00:17.538 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.538 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.538 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-03-30T14:00:17.538 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "94ea083e-6a27-41db-83af-2f7d3cae1956", 2024-03-30T14:00:17.538 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:00:17.539 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T14:00:17.539 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T14:00:17.539 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T14:00:17.539 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:00:17.539 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:00:17.539 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 8, 2024-03-30T14:00:17.539 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 350, 2024-03-30T14:00:17.539 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:00:17.539 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:00:17.539 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:00:17.539 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.539 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.539 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6802", 2024-03-30T14:00:17.539 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:00:17.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.540 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.540 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.540 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6803", 2024-03-30T14:00:17.540 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:00:17.540 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.540 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.540 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:00:17.540 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.540 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.540 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.540 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6804", 2024-03-30T14:00:17.540 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:00:17.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.541 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.541 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.541 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6805", 2024-03-30T14:00:17.541 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:00:17.541 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.541 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.541 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.541 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:00:17.541 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.541 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.541 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.541 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6808", 2024-03-30T14:00:17.541 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:00:17.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.542 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.542 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6809", 2024-03-30T14:00:17.542 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:00:17.542 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.542 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.542 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:00:17.542 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.542 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.542 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6806", 2024-03-30T14:00:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:00:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6807", 2024-03-30T14:00:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:00:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6803/1144592572", 2024-03-30T14:00:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6805/1144592572", 2024-03-30T14:00:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6809/1144592572", 2024-03-30T14:00:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6807/1144592572", 2024-03-30T14:00:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:00:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:00:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:00:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-03-30T14:00:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a", 2024-03-30T14:00:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:00:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T14:00:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T14:00:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T14:00:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:00:17.545 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:00:17.545 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 13, 2024-03-30T14:00:17.545 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 326, 2024-03-30T14:00:17.545 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:00:17.545 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:00:17.545 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:00:17.545 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.545 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.545 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6810", 2024-03-30T14:00:17.545 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:00:17.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.546 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.546 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6811", 2024-03-30T14:00:17.546 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:00:17.546 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.546 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.546 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:00:17.546 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.546 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.546 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6812", 2024-03-30T14:00:17.546 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:00:17.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.547 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.547 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.547 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6813", 2024-03-30T14:00:17.547 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:00:17.547 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.547 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.547 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:00:17.547 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.547 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.547 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.547 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6816", 2024-03-30T14:00:17.547 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:00:17.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.548 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.548 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6817", 2024-03-30T14:00:17.548 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:00:17.548 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.548 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.548 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:00:17.548 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.548 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.548 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6814", 2024-03-30T14:00:17.548 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:00:17.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.549 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.549 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.549 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6815", 2024-03-30T14:00:17.549 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:00:17.549 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.549 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.549 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6811/3901165928", 2024-03-30T14:00:17.549 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6813/3901165928", 2024-03-30T14:00:17.549 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6817/3901165928", 2024-03-30T14:00:17.549 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6815/3901165928", 2024-03-30T14:00:17.549 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:00:17.550 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:00:17.550 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:00:17.550 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.550 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-03-30T14:00:17.550 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "6db6782e-5889-443e-bc6c-df47a1f93bc8", 2024-03-30T14:00:17.550 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 0, 2024-03-30T14:00:17.550 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T14:00:17.550 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T14:00:17.550 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 0, 2024-03-30T14:00:17.550 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:00:17.551 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:00:17.551 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 18, 2024-03-30T14:00:17.551 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 98, 2024-03-30T14:00:17.551 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 167, 2024-03-30T14:00:17.551 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:00:17.551 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:00:17.551 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.551 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.551 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.551 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6818", 2024-03-30T14:00:17.551 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T14:00:17.551 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.551 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.551 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.552 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6819", 2024-03-30T14:00:17.552 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T14:00:17.552 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.552 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.552 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:00:17.552 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.552 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.552 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.552 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6820", 2024-03-30T14:00:17.552 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T14:00:17.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.552 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.552 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.553 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6821", 2024-03-30T14:00:17.553 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T14:00:17.553 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.553 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.553 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.553 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:00:17.553 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.553 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.553 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.553 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6824", 2024-03-30T14:00:17.553 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T14:00:17.553 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.553 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.553 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.554 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6825", 2024-03-30T14:00:17.554 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T14:00:17.554 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.554 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.554 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.554 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:00:17.554 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.554 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.554 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.554 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6822", 2024-03-30T14:00:17.554 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T14:00:17.554 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.554 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.555 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.555 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6823", 2024-03-30T14:00:17.555 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3415037523 2024-03-30T14:00:17.555 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.555 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.555 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6819/3415037523", 2024-03-30T14:00:17.555 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6821/3415037523", 2024-03-30T14:00:17.555 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6825/3415037523", 2024-03-30T14:00:17.555 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6823/3415037523", 2024-03-30T14:00:17.555 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:00:17.555 INFO:teuthology.orchestra.run.smithi043.stdout: "exists" 2024-03-30T14:00:17.555 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-03-30T14:00:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "7048ff46-2cba-4d18-9835-95ecbddb2bb6", 2024-03-30T14:00:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:00:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T14:00:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T14:00:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T14:00:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:00:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:00:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 25, 2024-03-30T14:00:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 348, 2024-03-30T14:00:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:00:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:00:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:00:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6826", 2024-03-30T14:00:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:00:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6827", 2024-03-30T14:00:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:00:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:00:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6828", 2024-03-30T14:00:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:00:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6829", 2024-03-30T14:00:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:00:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:00:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6832", 2024-03-30T14:00:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:00:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6833", 2024-03-30T14:00:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:00:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:00:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6830", 2024-03-30T14:00:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:00:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6831", 2024-03-30T14:00:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:00:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.561 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.561 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6827/2933589376", 2024-03-30T14:00:17.561 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6829/2933589376", 2024-03-30T14:00:17.561 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6833/2933589376", 2024-03-30T14:00:17.561 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6831/2933589376", 2024-03-30T14:00:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:00:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:00:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:00:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-03-30T14:00:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "42da30e6-9eec-4ed5-8c73-9ce78aa8ff63", 2024-03-30T14:00:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:00:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T14:00:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T14:00:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T14:00:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:00:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:00:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 32, 2024-03-30T14:00:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 347, 2024-03-30T14:00:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:00:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:00:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:00:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6800", 2024-03-30T14:00:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:00:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6801", 2024-03-30T14:00:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:00:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:00:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6802", 2024-03-30T14:00:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:00:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6803", 2024-03-30T14:00:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:00:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:00:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6806", 2024-03-30T14:00:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:00:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6807", 2024-03-30T14:00:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:00:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:00:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6804", 2024-03-30T14:00:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:00:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.567 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6805", 2024-03-30T14:00:17.567 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:00:17.567 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.567 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.567 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.567 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6801/1945833081", 2024-03-30T14:00:17.567 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6803/1945833081", 2024-03-30T14:00:17.567 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6807/1945833081", 2024-03-30T14:00:17.567 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6805/1945833081", 2024-03-30T14:00:17.567 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:00:17.567 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:00:17.567 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:00:17.567 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.567 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.568 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-03-30T14:00:17.568 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "99a11b81-0df7-4b1d-946a-4297ba62e263", 2024-03-30T14:00:17.568 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:00:17.568 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T14:00:17.568 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T14:00:17.568 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T14:00:17.568 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:00:17.568 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:00:17.568 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 40, 2024-03-30T14:00:17.568 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 351, 2024-03-30T14:00:17.568 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:00:17.568 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:00:17.568 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:00:17.569 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.569 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.569 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6808", 2024-03-30T14:00:17.569 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:00:17.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.569 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.569 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6809", 2024-03-30T14:00:17.569 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:00:17.569 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.569 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.569 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:00:17.569 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.570 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.570 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6810", 2024-03-30T14:00:17.570 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:00:17.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.570 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.570 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6811", 2024-03-30T14:00:17.570 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:00:17.570 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.570 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.570 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:00:17.570 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.571 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.571 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6814", 2024-03-30T14:00:17.571 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:00:17.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.571 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.571 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6815", 2024-03-30T14:00:17.571 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:00:17.571 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.571 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.571 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:00:17.571 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.572 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.572 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.572 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6812", 2024-03-30T14:00:17.572 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:00:17.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.572 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.572 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.572 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6813", 2024-03-30T14:00:17.572 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:00:17.572 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.572 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.572 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6809/3426360325", 2024-03-30T14:00:17.572 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6811/3426360325", 2024-03-30T14:00:17.573 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6815/3426360325", 2024-03-30T14:00:17.573 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6813/3426360325", 2024-03-30T14:00:17.573 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:00:17.573 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:00:17.573 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:00:17.573 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.573 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.573 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-03-30T14:00:17.573 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "cafd0e26-78e9-4164-a43a-96ca8382fefe", 2024-03-30T14:00:17.573 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:00:17.573 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T14:00:17.573 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T14:00:17.573 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T14:00:17.574 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:00:17.574 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:00:17.574 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 49, 2024-03-30T14:00:17.574 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 348, 2024-03-30T14:00:17.574 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:00:17.574 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:00:17.574 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:00:17.574 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.574 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.574 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6816", 2024-03-30T14:00:17.574 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:00:17.574 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.575 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.575 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6817", 2024-03-30T14:00:17.575 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:00:17.575 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.575 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.575 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:00:17.575 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.575 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.575 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.575 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6818", 2024-03-30T14:00:17.575 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:00:17.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.575 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.576 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.576 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6819", 2024-03-30T14:00:17.576 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:00:17.576 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.576 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.576 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:00:17.576 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.576 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.576 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.576 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6822", 2024-03-30T14:00:17.576 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:00:17.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.576 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.577 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.577 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6823", 2024-03-30T14:00:17.577 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:00:17.577 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.577 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.577 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:00:17.577 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.577 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.577 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.577 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6820", 2024-03-30T14:00:17.577 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:00:17.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.578 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.578 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.578 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6821", 2024-03-30T14:00:17.578 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:00:17.578 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.578 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.578 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6817/1527017698", 2024-03-30T14:00:17.578 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6819/1527017698", 2024-03-30T14:00:17.578 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6823/1527017698", 2024-03-30T14:00:17.578 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6821/1527017698", 2024-03-30T14:00:17.578 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:00:17.578 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:00:17.578 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:00:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-03-30T14:00:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "e9d59662-9132-48f4-9e95-842a3cfbd79f", 2024-03-30T14:00:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:00:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 0, 2024-03-30T14:00:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-03-30T14:00:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T14:00:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:00:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:00:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 58, 2024-03-30T14:00:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 228, 2024-03-30T14:00:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:00:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:00:17.580 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:00:17.580 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.580 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.580 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6824", 2024-03-30T14:00:17.580 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:00:17.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.580 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.580 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6825", 2024-03-30T14:00:17.580 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:00:17.580 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.580 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.581 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:00:17.581 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.581 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.581 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6826", 2024-03-30T14:00:17.581 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:00:17.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.581 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.581 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6827", 2024-03-30T14:00:17.581 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:00:17.581 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.581 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.582 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:00:17.582 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.582 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.582 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6830", 2024-03-30T14:00:17.582 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:00:17.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.582 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.582 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6831", 2024-03-30T14:00:17.582 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:00:17.582 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.582 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.583 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:00:17.583 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:00:17.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.583 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:00:17.583 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6828", 2024-03-30T14:00:17.583 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:00:17.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.583 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:00:17.583 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6829", 2024-03-30T14:00:17.584 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:00:17.584 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.584 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.584 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6825/3182476015", 2024-03-30T14:00:17.584 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6827/3182476015", 2024-03-30T14:00:17.584 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6831/3182476015", 2024-03-30T14:00:17.584 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6829/3182476015", 2024-03-30T14:00:17.584 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:00:17.584 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:00:17.584 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:00:17.584 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.584 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.585 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T14:00:17.585 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_xinfo": [ 2024-03-30T14:00:17.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.585 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-03-30T14:00:17.585 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:00:17.585 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:00:17.585 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:00:17.585 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:00:17.585 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T14:00:17.585 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:59:14.128544+0000", 2024-03-30T14:00:17.585 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:00:17.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.586 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-03-30T14:00:17.586 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:00:17.586 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:00:17.586 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:00:17.586 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:00:17.586 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T14:00:17.586 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T14:00:01.546624+0000", 2024-03-30T14:00:17.586 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:00:17.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.586 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-03-30T14:00:17.586 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:00:17.587 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:00:17.587 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:00:17.587 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:00:17.587 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T14:00:17.587 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:49:03.444490+0000", 2024-03-30T14:00:17.587 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 166 2024-03-30T14:00:17.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.587 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.587 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-03-30T14:00:17.587 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:00:17.587 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:00:17.587 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:00:17.587 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:00:17.587 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T14:00:17.588 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T14:00:06.394162+0000", 2024-03-30T14:00:17.588 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:00:17.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.588 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-03-30T14:00:17.588 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:00:17.588 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:00:17.588 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:00:17.588 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:00:17.588 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T14:00:17.588 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T14:00:11.785775+0000", 2024-03-30T14:00:17.588 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:00:17.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.589 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-03-30T14:00:17.589 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:00:17.589 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:00:17.589 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:00:17.589 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:00:17.589 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T14:00:17.589 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:59:14.665162+0000", 2024-03-30T14:00:17.589 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:00:17.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.589 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.589 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-03-30T14:00:17.589 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:00:17.589 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:00:17.589 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:00:17.589 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:00:17.590 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T14:00:17.590 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T13:59:27.485636+0000", 2024-03-30T14:00:17.590 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:00:17.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.590 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.590 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-03-30T14:00:17.590 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:00:17.590 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:00:17.590 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:00:17.590 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:00:17.590 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 65536, 2024-03-30T14:00:17.590 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T14:00:10.093197+0000", 2024-03-30T14:00:17.590 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:00:17.591 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.591 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T14:00:17.591 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap": [ 2024-03-30T14:00:17.591 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.591 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.1a", 2024-03-30T14:00:17.591 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.591 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-03-30T14:00:17.591 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T14:00:17.591 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-03-30T14:00:17.591 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.591 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.592 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T14:00:17.592 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_items": [ 2024-03-30T14:00:17.592 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.592 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.14", 2024-03-30T14:00:17.592 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-03-30T14:00:17.592 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.592 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-03-30T14:00:17.592 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-03-30T14:00:17.592 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.592 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.592 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.592 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T14:00:17.592 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_primaries": [], 2024-03-30T14:00:17.593 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_temp": [ 2024-03-30T14:00:17.593 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.593 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "1.1", 2024-03-30T14:00:17.593 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.593 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T14:00:17.593 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T14:00:17.593 INFO:teuthology.orchestra.run.smithi043.stdout: 4 2024-03-30T14:00:17.593 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.593 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.593 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "1.2", 2024-03-30T14:00:17.593 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.593 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T14:00:17.593 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T14:00:17.593 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T14:00:17.593 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.594 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "1.5", 2024-03-30T14:00:17.594 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.594 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T14:00:17.594 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T14:00:17.594 INFO:teuthology.orchestra.run.smithi043.stdout: 0 2024-03-30T14:00:17.594 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.594 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "1.8", 2024-03-30T14:00:17.594 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.594 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T14:00:17.594 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T14:00:17.594 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-03-30T14:00:17.595 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.595 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.595 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "1.9", 2024-03-30T14:00:17.595 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.595 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T14:00:17.595 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T14:00:17.595 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-03-30T14:00:17.595 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.595 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.595 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "1.a", 2024-03-30T14:00:17.595 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.596 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T14:00:17.596 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T14:00:17.596 INFO:teuthology.orchestra.run.smithi043.stdout: 4 2024-03-30T14:00:17.596 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.596 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.596 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "3.4", 2024-03-30T14:00:17.596 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.596 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T14:00:17.596 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T14:00:17.596 INFO:teuthology.orchestra.run.smithi043.stdout: 0 2024-03-30T14:00:17.596 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.596 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.597 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "3.5", 2024-03-30T14:00:17.597 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.597 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T14:00:17.597 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T14:00:17.597 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-03-30T14:00:17.597 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.597 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.597 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "3.9", 2024-03-30T14:00:17.597 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.597 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T14:00:17.597 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T14:00:17.597 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T14:00:17.597 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.598 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.598 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.598 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "3.13", 2024-03-30T14:00:17.598 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.598 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T14:00:17.598 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T14:00:17.598 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T14:00:17.598 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.598 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.598 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.598 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "3.14", 2024-03-30T14:00:17.598 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.598 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T14:00:17.598 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T14:00:17.599 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T14:00:17.599 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.599 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.599 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "3.1f", 2024-03-30T14:00:17.599 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.599 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T14:00:17.599 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T14:00:17.599 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-03-30T14:00:17.599 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.599 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.599 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.14", 2024-03-30T14:00:17.600 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.600 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T14:00:17.600 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T14:00:17.600 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T14:00:17.600 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.600 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.600 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.16", 2024-03-30T14:00:17.600 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.600 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T14:00:17.600 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T14:00:17.600 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T14:00:17.600 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.601 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.601 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.17", 2024-03-30T14:00:17.601 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.601 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T14:00:17.601 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T14:00:17.601 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T14:00:17.601 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.601 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.601 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.4", 2024-03-30T14:00:17.601 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.601 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T14:00:17.601 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T14:00:17.601 INFO:teuthology.orchestra.run.smithi043.stdout: 1 2024-03-30T14:00:17.602 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.602 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.602 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.6", 2024-03-30T14:00:17.602 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.602 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T14:00:17.602 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T14:00:17.602 INFO:teuthology.orchestra.run.smithi043.stdout: 1 2024-03-30T14:00:17.602 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.602 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.602 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.b", 2024-03-30T14:00:17.603 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.603 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T14:00:17.603 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T14:00:17.603 INFO:teuthology.orchestra.run.smithi043.stdout: 1 2024-03-30T14:00:17.603 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.603 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.603 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.603 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.d", 2024-03-30T14:00:17.603 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.603 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T14:00:17.603 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T14:00:17.603 INFO:teuthology.orchestra.run.smithi043.stdout: 3 2024-03-30T14:00:17.603 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.603 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.604 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.604 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.14", 2024-03-30T14:00:17.604 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.604 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T14:00:17.604 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T14:00:17.604 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T14:00:17.604 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.604 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.604 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.19", 2024-03-30T14:00:17.604 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.604 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T14:00:17.604 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T14:00:17.605 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-03-30T14:00:17.605 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.605 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.605 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.605 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.1e", 2024-03-30T14:00:17.605 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:17.606 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.606 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T14:00:17.607 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T14:00:17.607 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T14:00:17.607 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.607 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.607 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.1f", 2024-03-30T14:00:17.607 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.607 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T14:00:17.607 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T14:00:17.607 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T14:00:17.607 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.607 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.607 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.3", 2024-03-30T14:00:17.608 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.608 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T14:00:17.608 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T14:00:17.608 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T14:00:17.608 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.608 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.608 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.5", 2024-03-30T14:00:17.608 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.608 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T14:00:17.608 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T14:00:17.608 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T14:00:17.608 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.609 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.8", 2024-03-30T14:00:17.609 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.609 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T14:00:17.609 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T14:00:17.609 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T14:00:17.609 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.609 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.9", 2024-03-30T14:00:17.609 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.609 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T14:00:17.610 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T14:00:17.610 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T14:00:17.610 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.610 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.610 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.12", 2024-03-30T14:00:17.610 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.610 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T14:00:17.610 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T14:00:17.610 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T14:00:17.610 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.610 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.611 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.14", 2024-03-30T14:00:17.611 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.611 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T14:00:17.611 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T14:00:17.611 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T14:00:17.611 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.611 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.611 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.17", 2024-03-30T14:00:17.611 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.611 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T14:00:17.611 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T14:00:17.611 INFO:teuthology.orchestra.run.smithi043.stdout: 0 2024-03-30T14:00:17.612 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.612 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.612 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.18", 2024-03-30T14:00:17.612 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.612 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T14:00:17.612 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T14:00:17.612 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-03-30T14:00:17.612 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.613 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.613 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.1c", 2024-03-30T14:00:17.613 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.613 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T14:00:17.613 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T14:00:17.613 INFO:teuthology.orchestra.run.smithi043.stdout: 4 2024-03-30T14:00:17.613 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.613 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.613 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.613 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "10.3", 2024-03-30T14:00:17.613 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.613 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T14:00:17.613 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T14:00:17.614 INFO:teuthology.orchestra.run.smithi043.stdout: 4 2024-03-30T14:00:17.614 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.614 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "10.8", 2024-03-30T14:00:17.614 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.614 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T14:00:17.614 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T14:00:17.614 INFO:teuthology.orchestra.run.smithi043.stdout: 4 2024-03-30T14:00:17.614 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:17.614 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "10.9", 2024-03-30T14:00:17.615 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:00:17.615 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T14:00:17.615 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-03-30T14:00:17.615 INFO:teuthology.orchestra.run.smithi043.stdout: 3 2024-03-30T14:00:17.615 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:17.615 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.615 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T14:00:17.615 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_temp": [], 2024-03-30T14:00:17.615 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklist": { 2024-03-30T14:00:17.615 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/538304234": "2024-03-31T13:43:27.253365+0000", 2024-03-30T14:00:17.615 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6801/3411723337": "2024-03-31T13:37:41.733515+0000", 2024-03-30T14:00:17.615 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3656823679": "2024-03-31T13:37:41.733515+0000", 2024-03-30T14:00:17.615 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2618817647": "2024-03-31T13:38:09.321740+0000", 2024-03-30T14:00:17.615 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/3411723337": "2024-03-31T13:37:41.733515+0000", 2024-03-30T14:00:17.616 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1206218784": "2024-03-31T13:37:41.733515+0000", 2024-03-30T14:00:17.616 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/886631486": "2024-03-31T13:37:41.733515+0000", 2024-03-30T14:00:17.616 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/1339921781": "2024-03-31T13:38:09.321740+0000", 2024-03-30T14:00:17.616 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6801/1339921781": "2024-03-31T13:38:09.321740+0000", 2024-03-30T14:00:17.616 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2794494528": "2024-03-31T13:38:09.321740+0000", 2024-03-30T14:00:17.616 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/564868778": "2024-03-31T13:38:09.321740+0000", 2024-03-30T14:00:17.616 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/3033707884": "2024-03-31T13:43:27.253365+0000", 2024-03-30T14:00:17.616 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3539398453": "2024-03-31T13:43:27.253365+0000", 2024-03-30T14:00:17.616 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2414364740": "2024-03-31T13:43:27.253365+0000", 2024-03-30T14:00:17.616 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6801/3033707884": "2024-03-31T13:43:27.253365+0000", 2024-03-30T14:00:17.616 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3923597429": "2024-03-31T13:43:27.253365+0000", 2024-03-30T14:00:17.616 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2524269526": "2024-03-31T13:43:27.253365+0000" 2024-03-30T14:00:17.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.616 INFO:teuthology.orchestra.run.smithi043.stdout: "range_blocklist": {}, 2024-03-30T14:00:17.616 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profiles": { 2024-03-30T14:00:17.617 INFO:teuthology.orchestra.run.smithi043.stdout: "default": { 2024-03-30T14:00:17.617 INFO:teuthology.orchestra.run.smithi043.stdout: "crush-failure-domain": "osd", 2024-03-30T14:00:17.617 INFO:teuthology.orchestra.run.smithi043.stdout: "k": "2", 2024-03-30T14:00:17.617 INFO:teuthology.orchestra.run.smithi043.stdout: "m": "1", 2024-03-30T14:00:17.617 INFO:teuthology.orchestra.run.smithi043.stdout: "plugin": "jerasure", 2024-03-30T14:00:17.617 INFO:teuthology.orchestra.run.smithi043.stdout: "technique": "reed_sol_van" 2024-03-30T14:00:17.617 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:17.617 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps_queue": [], 2024-03-30T14:00:17.617 INFO:teuthology.orchestra.run.smithi043.stdout: "new_removed_snaps": [], 2024-03-30T14:00:17.617 INFO:teuthology.orchestra.run.smithi043.stdout: "new_purged_snaps": [], 2024-03-30T14:00:17.617 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_node_flags": {}, 2024-03-30T14:00:17.617 INFO:teuthology.orchestra.run.smithi043.stdout: "device_class_flags": {}, 2024-03-30T14:00:17.618 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": { 2024-03-30T14:00:17.618 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_enabled": false, 2024-03-30T14:00:17.618 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_bucket_count": 0, 2024-03-30T14:00:17.618 INFO:teuthology.orchestra.run.smithi043.stdout: "degraded_stretch_mode": 0, 2024-03-30T14:00:17.618 INFO:teuthology.orchestra.run.smithi043.stdout: "recovering_stretch_mode": 0, 2024-03-30T14:00:17.618 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_bucket": 0 2024-03-30T14:00:17.618 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:17.618 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T14:00:17.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:17.638 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:17.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:17.739 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:17.773 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:17.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:17.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:17.820 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T14:00:17.875 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:17.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:17.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:18.023 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:18.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:18.063 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:18.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:18.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:17 smithi043 ceph-mon[29963]: 6.1d scrub starts 2024-03-30T14:00:18.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:17 smithi043 ceph-mon[29963]: 6.1d scrub ok 2024-03-30T14:00:18.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:17 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/703482396' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T14:00:18.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:17 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 8168/129492 objects degraded (6.308%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:18.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:17 smithi043 ceph-mon[29963]: osdmap e360: 8 total, 7 up, 7 in 2024-03-30T14:00:18.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:17 smithi043 ceph-mon[42099]: 6.1d scrub starts 2024-03-30T14:00:18.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:17 smithi043 ceph-mon[42099]: 6.1d scrub ok 2024-03-30T14:00:18.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:17 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/703482396' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T14:00:18.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:17 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 8168/129492 objects degraded (6.308%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:18.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:17 smithi043 ceph-mon[42099]: osdmap e360: 8 total, 7 up, 7 in 2024-03-30T14:00:18.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:18.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:18.202 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:18.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:18.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:17 smithi113 ceph-mon[34341]: 6.1d scrub starts 2024-03-30T14:00:18.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:17 smithi113 ceph-mon[34341]: 6.1d scrub ok 2024-03-30T14:00:18.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:17 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/703482396' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T14:00:18.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:17 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 8168/129492 objects degraded (6.308%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:18.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:17 smithi113 ceph-mon[34341]: osdmap e360: 8 total, 7 up, 7 in 2024-03-30T14:00:18.303 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:18.335 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:18.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:18.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:18.437 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:18.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:18.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:18.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:18.616 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:18.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:18.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:18.718 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:18.754 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:18.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:18.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:18.856 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:18.893 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:18.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:18.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:18.995 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:19.028 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:19.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:19.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:19.130 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:19.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:18 smithi043 ceph-mon[29963]: 10.1 scrub starts 2024-03-30T14:00:19.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:18 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:00:19.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:18 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:00:19.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:18 smithi043 ceph-mon[29963]: pgmap v787: 158 pgs: 2 active+clean+scrubbing, 104 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.15k op/s; 7718/124329 objects degraded (6.208%); 7687/124329 objects misplaced (6.183%) 2024-03-30T14:00:19.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:18 smithi043 ceph-mon[29963]: 10.9 scrub starts 2024-03-30T14:00:19.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:18 smithi043 ceph-mon[29963]: 10.1 scrub ok 2024-03-30T14:00:19.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:18 smithi043 ceph-mon[29963]: 3.7 scrub starts 2024-03-30T14:00:19.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:18 smithi043 ceph-mon[29963]: 3.7 scrub ok 2024-03-30T14:00:19.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:18 smithi043 ceph-mon[29963]: 10.9 scrub ok 2024-03-30T14:00:19.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:18 smithi043 ceph-mon[42099]: 10.1 scrub starts 2024-03-30T14:00:19.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:18 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:00:19.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:18 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:00:19.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:18 smithi043 ceph-mon[42099]: pgmap v787: 158 pgs: 2 active+clean+scrubbing, 104 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.15k op/s; 7718/124329 objects degraded (6.208%); 7687/124329 objects misplaced (6.183%) 2024-03-30T14:00:19.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:18 smithi043 ceph-mon[42099]: 10.9 scrub starts 2024-03-30T14:00:19.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:18 smithi043 ceph-mon[42099]: 10.1 scrub ok 2024-03-30T14:00:19.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:18 smithi043 ceph-mon[42099]: 3.7 scrub starts 2024-03-30T14:00:19.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:18 smithi043 ceph-mon[42099]: 3.7 scrub ok 2024-03-30T14:00:19.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:18 smithi043 ceph-mon[42099]: 10.9 scrub ok 2024-03-30T14:00:19.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:19.170 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:19.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:19.206 INFO:tasks.thrashosds.thrasher:j is {'epoch': 359, 'fsid': '7ee932fa-ee9a-11ee-b647-cb9ed24678a4', 'created': '2024-03-30T13:37:07.346020+0000', 'modified': '2024-03-30T14:00:13.634096+0000', 'last_up_change': '2024-03-30T13:50:07.489747+0000', 'last_in_change': '2024-03-30T13:58:46.155265+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T13:40:27.658979+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '202', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.6399999856948853, 'raw_score_stable': 1.6399999856948853, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-30T13:42:42.989975+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3499999046325684, 'score_stable': 2.3499999046325684, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 1.6699999570846558, 'raw_score_stable': 1.6699999570846558, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-30T13:42:43.910306+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-30T13:42:45.633132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-30T13:42:47.570084+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-30T13:42:49.637189+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-03-30T13:56:54.614701+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '295', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.619999885559082, 'score_stable': 2.619999885559082, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '94ea083e-6a27-41db-83af-2f7d3cae1956', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 350, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6802', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6803', 'nonce': 1144592572}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6804', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6805', 'nonce': 1144592572}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6808', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6809', 'nonce': 1144592572}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6806', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6807', 'nonce': 1144592572}]}, 'public_addr': '172.21.15.43:6803/1144592572', 'cluster_addr': '172.21.15.43:6805/1144592572', 'heartbeat_back_addr': '172.21.15.43:6809/1144592572', 'heartbeat_front_addr': '172.21.15.43:6807/1144592572', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 326, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6810', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6811', 'nonce': 3901165928}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6812', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6813', 'nonce': 3901165928}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6816', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6817', 'nonce': 3901165928}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6814', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6815', 'nonce': 3901165928}]}, 'public_addr': '172.21.15.43:6811/3901165928', 'cluster_addr': '172.21.15.43:6813/3901165928', 'heartbeat_back_addr': '172.21.15.43:6817/3901165928', 'heartbeat_front_addr': '172.21.15.43:6815/3901165928', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6db6782e-5889-443e-bc6c-df47a1f93bc8', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 98, 'down_at': 167, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6818', 'nonce': 3415037523}, {'type': 'v1', 'addr': '172.21.15.43:6819', 'nonce': 3415037523}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6820', 'nonce': 3415037523}, {'type': 'v1', 'addr': '172.21.15.43:6821', 'nonce': 3415037523}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6824', 'nonce': 3415037523}, {'type': 'v1', 'addr': '172.21.15.43:6825', 'nonce': 3415037523}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6822', 'nonce': 3415037523}, {'type': 'v1', 'addr': '172.21.15.43:6823', 'nonce': 3415037523}]}, 'public_addr': '172.21.15.43:6819/3415037523', 'cluster_addr': '172.21.15.43:6821/3415037523', 'heartbeat_back_addr': '172.21.15.43:6825/3415037523', 'heartbeat_front_addr': '172.21.15.43:6823/3415037523', 'state': ['exists']}, {'osd': 3, 'uuid': '7048ff46-2cba-4d18-9835-95ecbddb2bb6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 348, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6826', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6827', 'nonce': 2933589376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6828', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6829', 'nonce': 2933589376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6832', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6833', 'nonce': 2933589376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6830', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6831', 'nonce': 2933589376}]}, 'public_addr': '172.21.15.43:6827/2933589376', 'cluster_addr': '172.21.15.43:6829/2933589376', 'heartbeat_back_addr': '172.21.15.43:6833/2933589376', 'heartbeat_front_addr': '172.21.15.43:6831/2933589376', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '42da30e6-9eec-4ed5-8c73-9ce78aa8ff63', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 347, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6800', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6801', 'nonce': 1945833081}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6802', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6803', 'nonce': 1945833081}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6806', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6807', 'nonce': 1945833081}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6804', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6805', 'nonce': 1945833081}]}, 'public_addr': '172.21.15.113:6801/1945833081', 'cluster_addr': '172.21.15.113:6803/1945833081', 'heartbeat_back_addr': '172.21.15.113:6807/1945833081', 'heartbeat_front_addr': '172.21.15.113:6805/1945833081', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '99a11b81-0df7-4b1d-946a-4297ba62e263', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 351, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6808', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6809', 'nonce': 3426360325}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6810', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6811', 'nonce': 3426360325}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6814', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6815', 'nonce': 3426360325}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6812', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6813', 'nonce': 3426360325}]}, 'public_addr': '172.21.15.113:6809/3426360325', 'cluster_addr': '172.21.15.113:6811/3426360325', 'heartbeat_back_addr': '172.21.15.113:6815/3426360325', 'heartbeat_front_addr': '172.21.15.113:6813/3426360325', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'cafd0e26-78e9-4164-a43a-96ca8382fefe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 348, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6816', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6817', 'nonce': 1527017698}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6818', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6819', 'nonce': 1527017698}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6822', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6823', 'nonce': 1527017698}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6820', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6821', 'nonce': 1527017698}]}, 'public_addr': '172.21.15.113:6817/1527017698', 'cluster_addr': '172.21.15.113:6819/1527017698', 'heartbeat_back_addr': '172.21.15.113:6823/1527017698', 'heartbeat_front_addr': '172.21.15.113:6821/1527017698', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e9d59662-9132-48f4-9e95-842a3cfbd79f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 228, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6824', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6825', 'nonce': 3182476015}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6826', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6827', 'nonce': 3182476015}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6830', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6831', 'nonce': 3182476015}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6828', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6829', 'nonce': 3182476015}]}, 'public_addr': '172.21.15.113:6825/3182476015', 'cluster_addr': '172.21.15.113:6827/3182476015', 'heartbeat_back_addr': '172.21.15.113:6831/3182476015', 'heartbeat_front_addr': '172.21.15.113:6829/3182476015', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:59:14.128544+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T14:00:01.546624+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:49:03.444490+0000', 'dead_epoch': 166}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T14:00:06.394162+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T14:00:11.785775+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:59:14.665162+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T13:59:27.485636+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-30T14:00:10.093197+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.1a', 'osds': [7, 3, 2]}], 'pg_upmap_items': [{'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.1', 'osds': [5, 0, 4]}, {'pgid': '1.2', 'osds': [0, 4, 7]}, {'pgid': '1.5', 'osds': [4, 6, 0]}, {'pgid': '1.8', 'osds': [5, 3, 6]}, {'pgid': '1.9', 'osds': [3, 5, 6]}, {'pgid': '1.a', 'osds': [5, 6, 4]}, {'pgid': '3.4', 'osds': [1, 5, 0]}, {'pgid': '3.5', 'osds': [5, 3, 6]}, {'pgid': '3.9', 'osds': [4, 1, 7]}, {'pgid': '3.13', 'osds': [4, 6, 7]}, {'pgid': '3.14', 'osds': [4, 6, 7]}, {'pgid': '3.1f', 'osds': [0, 5, 6]}, {'pgid': '4.14', 'osds': [3, 1, 7]}, {'pgid': '4.16', 'osds': [0, 3, 7]}, {'pgid': '4.17', 'osds': [3, 1, 7]}, {'pgid': '5.4', 'osds': [5, 0, 1]}, {'pgid': '5.6', 'osds': [5, 4, 1]}, {'pgid': '5.b', 'osds': [0, 5, 1]}, {'pgid': '5.d', 'osds': [5, 0, 3]}, {'pgid': '5.14', 'osds': [3, 1, 7]}, {'pgid': '5.19', 'osds': [1, 5, 6]}, {'pgid': '5.1e', 'osds': [0, 1, 7]}, {'pgid': '5.1f', 'osds': [6, 4, 7]}, {'pgid': '6.3', 'osds': [6, 1, 7]}, {'pgid': '6.5', 'osds': [6, 3, 7]}, {'pgid': '6.8', 'osds': [3, 6, 7]}, {'pgid': '6.9', 'osds': [0, 3, 7]}, {'pgid': '6.12', 'osds': [4, 3, 7]}, {'pgid': '6.14', 'osds': [4, 0, 7]}, {'pgid': '6.17', 'osds': [4, 5, 0]}, {'pgid': '6.18', 'osds': [0, 1, 7]}, {'pgid': '6.1c', 'osds': [5, 3, 4]}, {'pgid': '10.3', 'osds': [5, 3, 4]}, {'pgid': '10.8', 'osds': [0, 5, 4]}, {'pgid': '10.9', 'osds': [5, 4, 3]}], 'primary_temp': [], 'blocklist': {'172.21.15.43:0/538304234': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:6801/3411723337': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/3656823679': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/2618817647': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:6800/3411723337': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/1206218784': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/886631486': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:6800/1339921781': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:6801/1339921781': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:0/2794494528': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:0/564868778': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:6800/3033707884': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/3539398453': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/2414364740': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:6801/3033707884': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/3923597429': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/2524269526': '2024-03-31T13:43:27.253365+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-30T14:00:19.206 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.1a 2024-03-30T14:00:19.206 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd rm-pg-upmap 4.1a 2024-03-30T14:00:19.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:18 smithi113 ceph-mon[34341]: 10.1 scrub starts 2024-03-30T14:00:19.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:18 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:00:19.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:18 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:00:19.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:18 smithi113 ceph-mon[34341]: pgmap v787: 158 pgs: 2 active+clean+scrubbing, 104 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.15k op/s; 7718/124329 objects degraded (6.208%); 7687/124329 objects misplaced (6.183%) 2024-03-30T14:00:19.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:18 smithi113 ceph-mon[34341]: 10.9 scrub starts 2024-03-30T14:00:19.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:18 smithi113 ceph-mon[34341]: 10.1 scrub ok 2024-03-30T14:00:19.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:18 smithi113 ceph-mon[34341]: 3.7 scrub starts 2024-03-30T14:00:19.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:18 smithi113 ceph-mon[34341]: 3.7 scrub ok 2024-03-30T14:00:19.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:18 smithi113 ceph-mon[34341]: 10.9 scrub ok 2024-03-30T14:00:19.273 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:19.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:19.323 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:19.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:19.424 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:19.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:19.467 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:19.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:19.568 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:19.607 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:19.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:19.709 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:19.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:00:19 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:00:19.744 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:19.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:19.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:19.848 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:19.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:19.883 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:19.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:19.984 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:20.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:20.019 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:20.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:20.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:20.155 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:20.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:20.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:19 smithi113 ceph-mon[34341]: 6.1a scrub starts 2024-03-30T14:00:20.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:19 smithi113 ceph-mon[34341]: 6.1a scrub ok 2024-03-30T14:00:20.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:19 smithi113 ceph-mon[34341]: 6.15 scrub starts 2024-03-30T14:00:20.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:19 smithi113 ceph-mon[34341]: 6.15 scrub ok 2024-03-30T14:00:20.255 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:20.288 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:20.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:20.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:20.367 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:00:20.376 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:19 smithi043 ceph-mon[29963]: 6.1a scrub starts 2024-03-30T14:00:20.376 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:19 smithi043 ceph-mon[29963]: 6.1a scrub ok 2024-03-30T14:00:20.376 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:19 smithi043 ceph-mon[29963]: 6.15 scrub starts 2024-03-30T14:00:20.376 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:19 smithi043 ceph-mon[29963]: 6.15 scrub ok 2024-03-30T14:00:20.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:19 smithi043 ceph-mon[42099]: 6.1a scrub starts 2024-03-30T14:00:20.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:19 smithi043 ceph-mon[42099]: 6.1a scrub ok 2024-03-30T14:00:20.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:19 smithi043 ceph-mon[42099]: 6.15 scrub starts 2024-03-30T14:00:20.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:19 smithi043 ceph-mon[42099]: 6.15 scrub ok 2024-03-30T14:00:20.390 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:20.425 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:20.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:20.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:20.526 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:20.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:20.563 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:20.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:20.664 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:20.697 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:20.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:20.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:20.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:20.851 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:20.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:20.912 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T14:00:20.912 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T14:00:20.940 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T14:00:20.953 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:20.978 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T14:00:20.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:20.991 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:20.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:21.018 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T14:00:21.058 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T14:00:21.092 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:21.099 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T14:00:21.099 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:00:21.100 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:00:21.100 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T14:00:21.100 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:00:21.100 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:00:21.100 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:00:21.100 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:00:21.100 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:00:21.126 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:21.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:21.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:20 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:21.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:20 smithi113 ceph-mon[34341]: pgmap v788: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 657 B/s rd, 0 B/s wr, 1.57k op/s; 6819/108381 objects degraded (6.292%); 6809/108381 objects misplaced (6.282%) 2024-03-30T14:00:21.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:20 smithi113 ceph-mon[34341]: 1.1 scrub starts 2024-03-30T14:00:21.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:20 smithi113 ceph-mon[34341]: 1.1 scrub ok 2024-03-30T14:00:21.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:20 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/297546142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:00:21.228 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:21.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:21.263 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:21.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:21.282 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:20 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:21.283 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:20 smithi043 ceph-mon[42099]: pgmap v788: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 657 B/s rd, 0 B/s wr, 1.57k op/s; 6819/108381 objects degraded (6.292%); 6809/108381 objects misplaced (6.282%) 2024-03-30T14:00:21.283 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:20 smithi043 ceph-mon[42099]: 1.1 scrub starts 2024-03-30T14:00:21.283 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:20 smithi043 ceph-mon[42099]: 1.1 scrub ok 2024-03-30T14:00:21.283 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:20 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/297546142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:00:21.283 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:20 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:21.283 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:20 smithi043 ceph-mon[29963]: pgmap v788: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 657 B/s rd, 0 B/s wr, 1.57k op/s; 6819/108381 objects degraded (6.292%); 6809/108381 objects misplaced (6.282%) 2024-03-30T14:00:21.283 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:20 smithi043 ceph-mon[29963]: 1.1 scrub starts 2024-03-30T14:00:21.283 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:20 smithi043 ceph-mon[29963]: 1.1 scrub ok 2024-03-30T14:00:21.283 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:20 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/297546142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:00:21.364 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:21.399 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:21.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:21.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:21.501 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:21.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:21.542 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:21.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:21.643 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:21.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:21.675 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:21.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:21.776 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:21.814 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:21.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:21.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:21.915 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:21.954 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:21.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:21.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:21.970 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:00:22.056 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:22.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:22.106 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:22.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:22.207 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:22.244 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:22.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:22.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:22.346 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:22.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:22.371 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:22.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:22.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:21 smithi043 ceph-mon[29963]: 5.b scrub starts 2024-03-30T14:00:22.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:21 smithi043 ceph-mon[29963]: 5.b scrub ok 2024-03-30T14:00:22.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:21 smithi043 ceph-mon[42099]: 5.b scrub starts 2024-03-30T14:00:22.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:21 smithi043 ceph-mon[42099]: 5.b scrub ok 2024-03-30T14:00:22.440 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T14:00:22.440 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T14:00:22.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:21 smithi113 ceph-mon[34341]: 5.b scrub starts 2024-03-30T14:00:22.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:21 smithi113 ceph-mon[34341]: 5.b scrub ok 2024-03-30T14:00:22.472 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:22.510 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:22.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:22.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:22.612 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:22.653 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:22.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:22.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:22.755 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:22.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:22.788 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:22.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:22.889 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:22.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:22.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:23.029 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:23.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:23.074 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:23.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:23.175 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:23.228 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:23.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:23.329 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:23.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:23.375 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:23.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:23.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:22 smithi043 ceph-mon[29963]: 3.9 deep-scrub starts 2024-03-30T14:00:23.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:22 smithi043 ceph-mon[29963]: 3.9 deep-scrub ok 2024-03-30T14:00:23.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:22 smithi043 ceph-mon[29963]: 3.5 scrub starts 2024-03-30T14:00:23.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:22 smithi043 ceph-mon[29963]: pgmap v789: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.74k op/s; 6371/102861 objects degraded (6.194%); 6345/102861 objects misplaced (6.169%) 2024-03-30T14:00:23.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:22 smithi043 ceph-mon[29963]: 3.5 scrub ok 2024-03-30T14:00:23.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:22 smithi043 ceph-mon[29963]: 4.1d scrub starts 2024-03-30T14:00:23.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:22 smithi043 ceph-mon[29963]: 4.1d scrub ok 2024-03-30T14:00:23.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:22 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 6371/102861 objects degraded (6.194%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:23.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:22 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:00:23.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:22 smithi043 ceph-mon[42099]: 3.9 deep-scrub starts 2024-03-30T14:00:23.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:22 smithi043 ceph-mon[42099]: 3.9 deep-scrub ok 2024-03-30T14:00:23.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:22 smithi043 ceph-mon[42099]: 3.5 scrub starts 2024-03-30T14:00:23.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:22 smithi043 ceph-mon[42099]: pgmap v789: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.74k op/s; 6371/102861 objects degraded (6.194%); 6345/102861 objects misplaced (6.169%) 2024-03-30T14:00:23.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:22 smithi043 ceph-mon[42099]: 3.5 scrub ok 2024-03-30T14:00:23.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:22 smithi043 ceph-mon[42099]: 4.1d scrub starts 2024-03-30T14:00:23.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:22 smithi043 ceph-mon[42099]: 4.1d scrub ok 2024-03-30T14:00:23.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:22 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 6371/102861 objects degraded (6.194%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:23.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:22 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:00:23.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:22 smithi113 ceph-mon[34341]: 3.9 deep-scrub starts 2024-03-30T14:00:23.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:22 smithi113 ceph-mon[34341]: 3.9 deep-scrub ok 2024-03-30T14:00:23.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:22 smithi113 ceph-mon[34341]: 3.5 scrub starts 2024-03-30T14:00:23.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:22 smithi113 ceph-mon[34341]: pgmap v789: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.74k op/s; 6371/102861 objects degraded (6.194%); 6345/102861 objects misplaced (6.169%) 2024-03-30T14:00:23.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:22 smithi113 ceph-mon[34341]: 3.5 scrub ok 2024-03-30T14:00:23.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:22 smithi113 ceph-mon[34341]: 4.1d scrub starts 2024-03-30T14:00:23.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:22 smithi113 ceph-mon[34341]: 4.1d scrub ok 2024-03-30T14:00:23.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:22 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 6371/102861 objects degraded (6.194%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:23.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:22 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:00:23.477 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:23.506 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:00:23.523 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:23.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:23.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:23.625 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:23.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:23.666 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:23.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:23.767 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:23.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:23.803 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:23.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:23.904 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:23.940 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:23.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:24.042 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:24.077 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:24.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:24.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:24.179 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:24.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:24.219 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:24.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:24.320 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:24.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:23 smithi043 ceph-mon[29963]: 1.6 scrub starts 2024-03-30T14:00:24.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[29963]: 1.6 scrub ok 2024-03-30T14:00:24.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[29963]: 5.7 scrub starts 2024-03-30T14:00:24.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[29963]: 5.7 scrub ok 2024-03-30T14:00:24.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[29963]: 4.2 deep-scrub starts 2024-03-30T14:00:24.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[29963]: 4.2 deep-scrub ok 2024-03-30T14:00:24.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[29963]: 1.4 scrub starts 2024-03-30T14:00:24.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[29963]: 1.4 scrub ok 2024-03-30T14:00:24.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:00:24.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:00:24.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:00:24.327 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[42099]: 1.6 scrub starts 2024-03-30T14:00:24.327 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[42099]: 1.6 scrub ok 2024-03-30T14:00:24.327 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[42099]: 5.7 scrub starts 2024-03-30T14:00:24.327 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[42099]: 5.7 scrub ok 2024-03-30T14:00:24.327 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[42099]: 4.2 deep-scrub starts 2024-03-30T14:00:24.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[42099]: 4.2 deep-scrub ok 2024-03-30T14:00:24.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[42099]: 1.4 scrub starts 2024-03-30T14:00:24.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[42099]: 1.4 scrub ok 2024-03-30T14:00:24.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:00:24.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:00:24.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:24 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:00:24.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:24.360 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:24.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:24.441 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T14:00:24.461 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:24.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:24 smithi113 ceph-mon[34341]: 1.6 scrub starts 2024-03-30T14:00:24.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:24 smithi113 ceph-mon[34341]: 1.6 scrub ok 2024-03-30T14:00:24.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:24 smithi113 ceph-mon[34341]: 5.7 scrub starts 2024-03-30T14:00:24.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:24 smithi113 ceph-mon[34341]: 5.7 scrub ok 2024-03-30T14:00:24.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:24 smithi113 ceph-mon[34341]: 4.2 deep-scrub starts 2024-03-30T14:00:24.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:24 smithi113 ceph-mon[34341]: 4.2 deep-scrub ok 2024-03-30T14:00:24.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:24 smithi113 ceph-mon[34341]: 1.4 scrub starts 2024-03-30T14:00:24.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:24 smithi113 ceph-mon[34341]: 1.4 scrub ok 2024-03-30T14:00:24.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:24 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:00:24.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:24 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:00:24.468 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:24 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:00:24.497 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:24.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:24.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:24.599 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:24.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:24.656 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:24.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:24.757 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:24.793 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:24.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:24.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:24.895 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:24.933 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:24.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:24.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:25.035 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:25.072 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:25.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:25.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:25.146 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T14:00:25.146 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-03-30T14:00:25.146 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.146 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.15531.0:296700 10.1 e360/289 10:8cd7c280:::benchmark_data_smithi043_169264_object134473:head v 360'18252, mlcod=360'18249)", 2024-03-30T14:00:25.146 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:00:23.837894+0000", 2024-03-30T14:00:25.147 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.002174311, 2024-03-30T14:00:25.147 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00220586, 2024-03-30T14:00:25.147 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:00:25.147 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:00:25.147 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T14:00:25.147 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:00:25.147 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.147 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:00:25.147 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.837894+0000", 2024-03-30T14:00:25.147 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:00:25.147 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.147 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.147 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:00:25.148 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.837894+0000", 2024-03-30T14:00:25.148 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:00:25.148 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.148 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.148 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:00:25.148 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.837894+0000", 2024-03-30T14:00:25.148 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1799999999999995e-07 2024-03-30T14:00:25.148 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.148 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.148 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:00:25.148 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.837899+0000", 2024-03-30T14:00:25.148 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7650000000000001e-06 2024-03-30T14:00:25.148 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.149 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.149 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:00:25.149 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.837914+0000", 2024-03-30T14:00:25.149 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4744e-05 2024-03-30T14:00:25.149 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.149 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.149 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:00:25.149 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.837922+0000", 2024-03-30T14:00:25.149 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6310000000000003e-06 2024-03-30T14:00:25.149 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.149 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.149 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:00:25.149 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.838065+0000", 2024-03-30T14:00:25.150 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014342599999999999 2024-03-30T14:00:25.150 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.150 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.150 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:00:25.150 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.838085+0000", 2024-03-30T14:00:25.150 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0420000000000001e-05 2024-03-30T14:00:25.150 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:25.150 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:25.150 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:25.150 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.150 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.150 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.15531.0:296702 10.b e360/289 10:d4fabb5a:::benchmark_data_smithi043_169264_object134475:head v 360'18326, mlcod=360'18324)", 2024-03-30T14:00:25.151 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:00:23.838000+0000", 2024-03-30T14:00:25.151 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0020677690000000001, 2024-03-30T14:00:25.151 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0021733059999999998, 2024-03-30T14:00:25.151 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:00:25.151 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:00:25.151 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T14:00:25.151 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:00:25.152 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.152 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:00:25.152 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.838000+0000", 2024-03-30T14:00:25.152 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:00:25.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.152 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.152 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:00:25.152 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.838000+0000", 2024-03-30T14:00:25.152 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:00:25.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.152 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.152 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:00:25.152 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.838001+0000", 2024-03-30T14:00:25.152 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0100000000000005e-07 2024-03-30T14:00:25.153 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.153 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.153 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:00:25.153 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.838009+0000", 2024-03-30T14:00:25.153 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8830000000000006e-06 2024-03-30T14:00:25.153 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.153 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.153 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:00:25.153 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.838019+0000", 2024-03-30T14:00:25.153 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0318e-05 2024-03-30T14:00:25.153 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.153 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.153 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:00:25.154 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.838024+0000", 2024-03-30T14:00:25.154 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3990000000000003e-06 2024-03-30T14:00:25.154 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.154 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.154 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:00:25.154 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.838080+0000", 2024-03-30T14:00:25.154 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5398000000000001e-05 2024-03-30T14:00:25.154 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.154 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.154 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:00:25.154 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.838101+0000", 2024-03-30T14:00:25.154 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0775e-05 2024-03-30T14:00:25.154 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:25.154 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:25.155 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:25.155 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.155 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.155 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15531.0:296703 10.2 10:429dc705:::benchmark_data_smithi043_169264_object134476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-30T14:00:25.155 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:00:23.839299+0000", 2024-03-30T14:00:25.155 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.00076920899999999997, 2024-03-30T14:00:25.155 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00094113599999999997, 2024-03-30T14:00:25.155 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:00:25.155 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:00:25.155 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-03-30T14:00:25.155 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:00:25.155 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15531", 2024-03-30T14:00:25.155 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:00:25.155 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 296703 2024-03-30T14:00:25.156 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.156 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:00:25.156 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.156 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:00:25.156 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.839299+0000", 2024-03-30T14:00:25.156 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:00:25.156 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.156 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.156 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:00:25.156 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.839299+0000", 2024-03-30T14:00:25.156 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:00:25.156 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.156 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.156 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:00:25.157 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.839300+0000", 2024-03-30T14:00:25.157 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3319999999999999e-06 2024-03-30T14:00:25.157 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.157 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.157 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:00:25.157 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.839307+0000", 2024-03-30T14:00:25.157 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2859999999999996e-06 2024-03-30T14:00:25.157 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.157 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.157 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:00:25.158 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.839365+0000", 2024-03-30T14:00:25.158 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8087000000000001e-05 2024-03-30T14:00:25.158 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.158 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.158 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:00:25.158 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.839374+0000", 2024-03-30T14:00:25.158 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5100000000000004e-06 2024-03-30T14:00:25.158 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.158 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.158 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:00:25.158 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.839441+0000", 2024-03-30T14:00:25.158 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7291000000000003e-05 2024-03-30T14:00:25.158 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.159 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.159 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:00:25.159 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.839564+0000", 2024-03-30T14:00:25.159 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012270799999999999 2024-03-30T14:00:25.159 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.159 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.159 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,4", 2024-03-30T14:00:25.159 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.839679+0000", 2024-03-30T14:00:25.159 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000114622 2024-03-30T14:00:25.159 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:25.159 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:25.159 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:25.159 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.159 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.160 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.15531.0:296701 10.1 e360/289)", 2024-03-30T14:00:25.160 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:00:23.839604+0000", 2024-03-30T14:00:25.160 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.00046456699999999999, 2024-03-30T14:00:25.160 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00072821500000000005, 2024-03-30T14:00:25.160 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:00:25.160 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:00:25.160 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "queued for pg", 2024-03-30T14:00:25.160 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:00:25.160 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.160 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:00:25.161 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.839604+0000", 2024-03-30T14:00:25.161 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:00:25.161 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.161 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.161 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:00:25.161 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.839604+0000", 2024-03-30T14:00:25.161 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:00:25.161 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.161 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.161 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:00:25.161 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.839604+0000", 2024-03-30T14:00:25.161 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3099999999999997e-07 2024-03-30T14:00:25.161 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.162 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.162 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:00:25.162 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.839610+0000", 2024-03-30T14:00:25.162 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.755e-06 2024-03-30T14:00:25.162 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.162 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.162 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:00:25.162 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.839620+0000", 2024-03-30T14:00:25.162 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.6160000000000008e-06 2024-03-30T14:00:25.162 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:00:25.162 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:00:25.162 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:00:25.162 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:00:23.839626+0000", 2024-03-30T14:00:25.163 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5429999999999999e-06 2024-03-30T14:00:25.163 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:25.163 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:00:25.163 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:25.163 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:00:25.163 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T14:00:25.163 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 4 2024-03-30T14:00:25.163 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T14:00:25.174 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:25.215 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:25.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:25.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:25.317 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:25.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:25.353 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:25.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:25.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[29963]: 3.13 scrub starts 2024-03-30T14:00:25.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[29963]: 3.13 scrub ok 2024-03-30T14:00:25.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[29963]: pgmap v790: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.1 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s; 6371/102861 objects degraded (6.194%); 6345/102861 objects misplaced (6.169%) 2024-03-30T14:00:25.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:00:25.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:00:25.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[29963]: osdmap e361: 8 total, 7 up, 7 in 2024-03-30T14:00:25.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[29963]: 5.c scrub starts 2024-03-30T14:00:25.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[29963]: 5.c scrub ok 2024-03-30T14:00:25.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[29963]: 5.2 scrub starts 2024-03-30T14:00:25.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[29963]: 5.2 scrub ok 2024-03-30T14:00:25.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[29963]: 4.11 scrub starts 2024-03-30T14:00:25.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[29963]: 4.11 scrub ok 2024-03-30T14:00:25.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[42099]: 3.13 scrub starts 2024-03-30T14:00:25.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[42099]: 3.13 scrub ok 2024-03-30T14:00:25.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[42099]: pgmap v790: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.1 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s; 6371/102861 objects degraded (6.194%); 6345/102861 objects misplaced (6.169%) 2024-03-30T14:00:25.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:00:25.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:00:25.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[42099]: osdmap e361: 8 total, 7 up, 7 in 2024-03-30T14:00:25.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[42099]: 5.c scrub starts 2024-03-30T14:00:25.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[42099]: 5.c scrub ok 2024-03-30T14:00:25.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[42099]: 5.2 scrub starts 2024-03-30T14:00:25.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[42099]: 5.2 scrub ok 2024-03-30T14:00:25.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[42099]: 4.11 scrub starts 2024-03-30T14:00:25.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:25 smithi043 ceph-mon[42099]: 4.11 scrub ok 2024-03-30T14:00:25.454 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:25.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:25 smithi113 ceph-mon[34341]: 3.13 scrub starts 2024-03-30T14:00:25.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:25 smithi113 ceph-mon[34341]: 3.13 scrub ok 2024-03-30T14:00:25.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:25 smithi113 ceph-mon[34341]: pgmap v790: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 2.1 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s; 6371/102861 objects degraded (6.194%); 6345/102861 objects misplaced (6.169%) 2024-03-30T14:00:25.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:25 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:00:25.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:25 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:00:25.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:25 smithi113 ceph-mon[34341]: osdmap e361: 8 total, 7 up, 7 in 2024-03-30T14:00:25.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:25 smithi113 ceph-mon[34341]: 5.c scrub starts 2024-03-30T14:00:25.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:25 smithi113 ceph-mon[34341]: 5.c scrub ok 2024-03-30T14:00:25.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:25 smithi113 ceph-mon[34341]: 5.2 scrub starts 2024-03-30T14:00:25.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:25 smithi113 ceph-mon[34341]: 5.2 scrub ok 2024-03-30T14:00:25.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:25 smithi113 ceph-mon[34341]: 4.11 scrub starts 2024-03-30T14:00:25.467 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:25 smithi113 ceph-mon[34341]: 4.11 scrub ok 2024-03-30T14:00:25.488 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:25.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:25.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:25.590 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:25.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:25.624 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:25.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:25.725 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:25.751 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:25.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:25.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:25.854 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:25.891 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:25.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:25.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:25.993 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:26.010 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-03-30T14:00:26.019 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:26.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:26.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:26.121 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:26.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:26.160 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:26.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:26.261 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:26.298 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:26.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:26.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:26.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:26 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3530191445' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:00:26.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:26 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:00:26.393 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:00:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:00:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:00:26.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:26 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3530191445' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:00:26.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:26 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:00:26.400 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:26.439 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:26.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:26.446 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-03-30T14:00:26.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:26 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3530191445' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:00:26.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:26 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:00:26.540 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:26.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:26.580 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:26.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:26.681 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:26.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:26.716 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:26.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:26.817 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:26.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:26.856 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:26.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:26.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:26.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:26.998 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:26.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:27.091 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:27 smithi043 ceph-mon[29963]: 3.1 deep-scrub starts 2024-03-30T14:00:27.091 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:27 smithi043 ceph-mon[29963]: 3.1 deep-scrub ok 2024-03-30T14:00:27.091 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:27 smithi043 ceph-mon[29963]: pgmap v792: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 1.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 632 B/s rd, 0 B/s wr, 1.55k op/s; 5481/86793 objects degraded (6.315%); 5430/86793 objects misplaced (6.256%) 2024-03-30T14:00:27.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:27 smithi043 ceph-mon[29963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:00:27.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:27 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:00:27.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:27 smithi043 ceph-mon[29963]: osdmap e362: 8 total, 7 up, 7 in 2024-03-30T14:00:27.092 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:27 smithi043 ceph-mon[42099]: 3.1 deep-scrub starts 2024-03-30T14:00:27.092 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:27 smithi043 ceph-mon[42099]: 3.1 deep-scrub ok 2024-03-30T14:00:27.092 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:27 smithi043 ceph-mon[42099]: pgmap v792: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 1.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 632 B/s rd, 0 B/s wr, 1.55k op/s; 5481/86793 objects degraded (6.315%); 5430/86793 objects misplaced (6.256%) 2024-03-30T14:00:27.092 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:27 smithi043 ceph-mon[42099]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:00:27.093 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:27 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:00:27.093 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:27 smithi043 ceph-mon[42099]: osdmap e362: 8 total, 7 up, 7 in 2024-03-30T14:00:27.099 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:27.133 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:27.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:27.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:27.235 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:27.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:27.270 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:27.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:27.371 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:27.408 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:27.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:27.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:27.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:27 smithi113 ceph-mon[34341]: 3.1 deep-scrub starts 2024-03-30T14:00:27.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:27 smithi113 ceph-mon[34341]: 3.1 deep-scrub ok 2024-03-30T14:00:27.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:27 smithi113 ceph-mon[34341]: pgmap v792: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 1.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 632 B/s rd, 0 B/s wr, 1.55k op/s; 5481/86793 objects degraded (6.315%); 5430/86793 objects misplaced (6.256%) 2024-03-30T14:00:27.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:27 smithi113 ceph-mon[34341]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:00:27.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:27 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:00:27.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:27 smithi113 ceph-mon[34341]: osdmap e362: 8 total, 7 up, 7 in 2024-03-30T14:00:27.509 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:27.549 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:27.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:27.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:27.651 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:27.684 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:27.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:27.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:27.786 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:27.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:27.824 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:27.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:27.925 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:27.962 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:27.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:27.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:28.064 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:28.066 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:00:28.066 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:00:28.115 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:28.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:28.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:28.172 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:28 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 5481/86793 objects degraded (6.315%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:28.173 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:28 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 5481/86793 objects degraded (6.315%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:28.218 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:28.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:28.252 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:28.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:28.353 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:28.387 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:28.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:28.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:28.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:28 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 5481/86793 objects degraded (6.315%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:28.489 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:28.525 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:28.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:28.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:28.627 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:28.663 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:28.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:28.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:28.765 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:28.805 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:28.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:28.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:28.907 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:28.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:28.947 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:28.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:29.048 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:29.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:29.087 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:29.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:29.089 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:29 smithi043 ceph-mon[29963]: pgmap v794: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 1.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 5047/81573 objects degraded (6.187%); 4984/81573 objects misplaced (6.110%) 2024-03-30T14:00:29.091 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:29 smithi043 ceph-mon[42099]: pgmap v794: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 1.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 5047/81573 objects degraded (6.187%); 4984/81573 objects misplaced (6.110%) 2024-03-30T14:00:29.141 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T14:00:29.187 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:29.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:29.228 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:29.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:29.329 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:29.332 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:29 smithi113 ceph-mon[34341]: pgmap v794: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 1.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 5047/81573 objects degraded (6.187%); 4984/81573 objects misplaced (6.110%) 2024-03-30T14:00:29.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:29.363 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:29.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:29.464 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:29.500 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:29.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:29.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:29.602 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:29.638 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:29.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:29.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:29.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:00:29 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:00:29.740 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:29.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:29.776 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:29.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:29.877 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:29.912 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:29.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:29.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:30.013 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:30.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:30.063 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:30.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:30.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:30.211 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:30.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:30.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:30.263 INFO:teuthology.orchestra.run.smithi043.stderr:clear 4.1a pg_upmap mapping 2024-03-30T14:00:30.313 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:30.343 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:30.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:30.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:30.445 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:30.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:30.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:30.582 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:30.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:30.619 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:30.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:00:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:30.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:30 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:30.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:30 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2059209807' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.1a"}]: dispatch 2024-03-30T14:00:30.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:30 smithi043 ceph-mon[29963]: pgmap v795: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 1.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.58k op/s; 4103/64827 objects degraded (6.329%); 4070/64827 objects misplaced (6.278%) 2024-03-30T14:00:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:00:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:30 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:30 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2059209807' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.1a"}]: dispatch 2024-03-30T14:00:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:30 smithi043 ceph-mon[42099]: pgmap v795: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 1.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.58k op/s; 4103/64827 objects degraded (6.329%); 4070/64827 objects misplaced (6.278%) 2024-03-30T14:00:30.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:00:30.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:30.716 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:30 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:30.716 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:30 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2059209807' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.1a"}]: dispatch 2024-03-30T14:00:30.716 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:30 smithi113 ceph-mon[34341]: pgmap v795: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 1.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.58k op/s; 4103/64827 objects degraded (6.329%); 4070/64827 objects misplaced (6.278%) 2024-03-30T14:00:30.721 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:30.765 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:30.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:30.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:30.867 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:30.901 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:30.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:30.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:31.002 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:31.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:31.040 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:31.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:31.141 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:31.181 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:31.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:31.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:31.283 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:31.319 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:31.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:31.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:31 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:00:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:31 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:00:31.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:31 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2059209807' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.1a"}]': finished 2024-03-30T14:00:31.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:31 smithi043 ceph-mon[29963]: osdmap e363: 8 total, 7 up, 7 in 2024-03-30T14:00:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:31 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:00:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:31 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:00:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:31 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2059209807' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.1a"}]': finished 2024-03-30T14:00:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:31 smithi043 ceph-mon[42099]: osdmap e363: 8 total, 7 up, 7 in 2024-03-30T14:00:31.421 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:31.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:31.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:31.557 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:31.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:31.594 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:31.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:31.695 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:31.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:31 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:00:31.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:31 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:00:31.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:31 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2059209807' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.1a"}]': finished 2024-03-30T14:00:31.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:31 smithi113 ceph-mon[34341]: osdmap e363: 8 total, 7 up, 7 in 2024-03-30T14:00:31.730 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:31.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:31.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:31.832 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:31.864 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:31.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:31.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:31.966 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:32.002 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:32.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:32.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:32.104 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:32.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:32.148 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:32.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:32.249 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:32.278 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:32.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:32.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:32.356 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:32 smithi043 ceph-mon[29963]: pgmap v797: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 1.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.18k op/s; 3650/59364 objects degraded (6.149%); 3625/59364 objects misplaced (6.106%) 2024-03-30T14:00:32.356 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:32 smithi043 ceph-mon[42099]: pgmap v797: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 1.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.18k op/s; 3650/59364 objects degraded (6.149%); 3625/59364 objects misplaced (6.106%) 2024-03-30T14:00:32.381 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:32.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:32.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:32.524 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:32.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:32.563 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:32.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:32.664 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:32.696 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:32.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:32.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:32.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:32 smithi113 ceph-mon[34341]: pgmap v797: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 1.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.18k op/s; 3650/59364 objects degraded (6.149%); 3625/59364 objects misplaced (6.106%) 2024-03-30T14:00:32.799 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:32.834 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:32.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:32.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:32.936 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:32.973 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:32.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:32.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:33.075 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:33.111 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:33.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:33.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:33.213 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:33.258 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:33.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:33.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:33.360 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:33.393 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:33.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:33.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:33.495 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:33.503 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:33 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 3650/59364 objects degraded (6.149%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:33.503 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:33 smithi043 ceph-mon[29963]: osdmap e364: 8 total, 7 up, 7 in 2024-03-30T14:00:33.503 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:33 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 3650/59364 objects degraded (6.149%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:33.503 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:33 smithi043 ceph-mon[42099]: osdmap e364: 8 total, 7 up, 7 in 2024-03-30T14:00:33.529 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:33.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:33.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:33.631 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:33.670 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:33.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:33.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:33.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:33 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 3650/59364 objects degraded (6.149%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:33.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:33 smithi113 ceph-mon[34341]: osdmap e364: 8 total, 7 up, 7 in 2024-03-30T14:00:33.772 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:33.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:33.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:33.853 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T14:00:33.891 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T14:00:33.913 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:33.930 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T14:00:33.948 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:33.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:33.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:33.970 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T14:00:34.024 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T14:00:34.050 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:34.086 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T14:00:34.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:34.102 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:34.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:34.130 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T14:00:34.130 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:00:34.130 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:00:34.130 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T14:00:34.130 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:00:34.130 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:00:34.130 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:00:34.131 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:00:34.131 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:00:34.170 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T14:00:34.170 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:00:34.203 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:34.235 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:34.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:34.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:34.337 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:34.374 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:34.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:34.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:34.476 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:34.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:34.512 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:34.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:34.613 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:34.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:34 smithi043 ceph-mon[29963]: pgmap v799: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 1.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.02k op/s; 3650/59364 objects degraded (6.149%); 3625/59364 objects misplaced (6.106%) 2024-03-30T14:00:34.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:34 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4144950906' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:00:34.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:34 smithi043 ceph-mon[42099]: pgmap v799: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 1.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.02k op/s; 3650/59364 objects degraded (6.149%); 3625/59364 objects misplaced (6.106%) 2024-03-30T14:00:34.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:34 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/4144950906' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:00:34.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:34.649 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:34.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:34.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:34 smithi113 ceph-mon[34341]: pgmap v799: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 1.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.02k op/s; 3650/59364 objects degraded (6.149%); 3625/59364 objects misplaced (6.106%) 2024-03-30T14:00:34.716 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:34 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/4144950906' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:00:34.750 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:34.785 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:34.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:34.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:34.887 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:34.920 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:34.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:34.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:35.022 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:35.045 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T14:00:35.046 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T14:00:35.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:35.063 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:35.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:35.164 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:35.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:35.199 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:35.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:35.300 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:35.334 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:35.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:35.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:35.436 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:35.476 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:35.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:35.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:35.577 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:35.598 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:00:35.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:35.628 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:35.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:35.641 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-03-30T14:00:35.729 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:35.765 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:35.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:35.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:35.868 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:35.902 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:35.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:35.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:36.004 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:36.037 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:36.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:36.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:36.139 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:36.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:36.179 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:36.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:36.280 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:36.319 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:36.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:36.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:36.393 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:00:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:00:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:00:36.421 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:36.454 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:36.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:36.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:36.556 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:36.594 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:36.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:36.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:36.696 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:36.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:36.734 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:36.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:36.835 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:36.871 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:36.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:36.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:36.886 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:36 smithi043 ceph-mon[42099]: pgmap v800: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 887 MiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.60k op/s; 2772/43122 objects degraded (6.428%); 2731/43122 objects misplaced (6.333%) 2024-03-30T14:00:36.886 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:36 smithi043 ceph-mon[29963]: pgmap v800: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 887 MiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.60k op/s; 2772/43122 objects degraded (6.428%); 2731/43122 objects misplaced (6.333%) 2024-03-30T14:00:36.953 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:00:36.973 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:37.008 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:37.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:37.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:37.047 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T14:00:37.110 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:37.144 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:37.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:37.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:37.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:36 smithi113 ceph-mon[34341]: pgmap v800: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 887 MiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.60k op/s; 2772/43122 objects degraded (6.428%); 2731/43122 objects misplaced (6.333%) 2024-03-30T14:00:37.246 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:37.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:37.283 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:37.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:37.384 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:37.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:37.429 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:37.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:37.530 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:37.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:37.565 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:37.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:37.666 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:37.703 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:37.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:37.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:37.805 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:37.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:37.834 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:37.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:37.907 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 5] out_osds: [7] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-30T14:00:37.907 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T14:00:37.907 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-30T14:00:37.908 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T14:00:37.936 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:37.996 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:37.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:37.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:38.098 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:38.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:38.132 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:38.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:38.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:37 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 2772/43122 objects degraded (6.428%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:38.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:37 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 2772/43122 objects degraded (6.428%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:38.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:37 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 2772/43122 objects degraded (6.428%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:38.234 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:38.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:38.270 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:38.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:38.371 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:38.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:38.407 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:38.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:38.508 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:38.543 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:38.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:38.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:38.645 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:38.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:38.684 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:38.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:38.785 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:38.822 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:38.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:38.925 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:38.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:38.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:39.063 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:39.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:39.100 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:39.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:39.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:38 smithi043 ceph-mon[29963]: pgmap v801: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 770 MiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 2321/37551 objects degraded (6.181%); 2269/37551 objects misplaced (6.042%) 2024-03-30T14:00:39.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:38 smithi043 ceph-mon[42099]: pgmap v801: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 770 MiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 2321/37551 objects degraded (6.181%); 2269/37551 objects misplaced (6.042%) 2024-03-30T14:00:39.200 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:39.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:38 smithi113 ceph-mon[34341]: pgmap v801: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 770 MiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 2321/37551 objects degraded (6.181%); 2269/37551 objects misplaced (6.042%) 2024-03-30T14:00:39.231 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:39.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:39.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:39.333 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:39.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:39.371 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:39.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:39.472 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:39.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:39.511 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:39.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:39.612 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:39.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:39.643 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:39.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:39.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:00:39 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:00:39.745 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:39.778 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:39.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:39.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:39.880 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:39.916 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:39.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:39.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:40.018 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:40.055 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:40.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:40.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:40.157 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:40.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:40.194 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:40.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:40.227 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:00:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:00:40.294 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:40.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:40.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:40.441 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:40.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:40.468 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:40.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:40.569 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:40.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:40.604 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:40.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:40.705 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:40.742 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:40.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:40.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-03-30T14:00:40.843 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:40.890 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:40.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:40.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:40.992 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:41.030 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:41.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:41.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:41.132 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:41.135 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:40 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:41.135 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:40 smithi043 ceph-mon[29963]: pgmap v802: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 426 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.54k op/s; 1415/21021 objects degraded (6.731%); 1404/21021 objects misplaced (6.679%) 2024-03-30T14:00:41.136 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:40 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/738723466' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:00:41.136 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:40 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:41.136 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:40 smithi043 ceph-mon[42099]: pgmap v802: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 426 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.54k op/s; 1415/21021 objects degraded (6.731%); 1404/21021 objects misplaced (6.679%) 2024-03-30T14:00:41.136 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:40 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/738723466' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:00:41.171 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:41.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:41.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:41.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:40 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:41.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:40 smithi113 ceph-mon[34341]: pgmap v802: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 426 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.54k op/s; 1415/21021 objects degraded (6.731%); 1404/21021 objects misplaced (6.679%) 2024-03-30T14:00:41.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:40 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/738723466' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:00:41.273 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:41.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:41.309 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:41.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:41.410 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:41.446 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:41.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:41.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:41.548 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:41.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:41.586 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:41.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:41.688 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:41.724 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:41.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:41.826 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:41.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:41.862 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:41.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:41.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:41 smithi043 ceph-mon[29963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:00:41.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:41 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/738723466' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:00:41.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:41 smithi043 ceph-mon[29963]: osdmap e365: 8 total, 7 up, 7 in 2024-03-30T14:00:41.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:41 smithi043 ceph-mon[42099]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:00:41.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:41 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/738723466' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:00:41.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:41 smithi043 ceph-mon[42099]: osdmap e365: 8 total, 7 up, 7 in 2024-03-30T14:00:41.963 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:41.998 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:41.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:41.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:42.100 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:42.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:42.138 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:42.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:42.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:41 smithi113 ceph-mon[34341]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:00:42.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:41 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/738723466' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:00:42.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:41 smithi113 ceph-mon[34341]: osdmap e365: 8 total, 7 up, 7 in 2024-03-30T14:00:42.238 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:42.275 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:42.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:42.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:42.377 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:42.410 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:42.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:42.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:42.512 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:42.548 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:42.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:42.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:42.650 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:42.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:42.682 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:42.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:42.704 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Removed 162224 objects 2024-03-30T14:00:42.704 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Clean up completed and total clean up time :124.316 2024-03-30T14:00:42.742 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T14:00:42.742 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:00:42.783 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:42.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:42.808 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:42.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:42.909 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:42.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:42.935 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:42.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:43.036 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:43.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:43.061 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:43.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:43.100 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:42 smithi043 ceph-mon[29963]: pgmap v804: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 313 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.61k op/s; 941/15576 objects degraded (6.041%); 947/15576 objects misplaced (6.080%) 2024-03-30T14:00:43.100 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:42 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 941/15576 objects degraded (6.041%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:43.101 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:42 smithi043 ceph-mon[29963]: osdmap e366: 8 total, 7 up, 7 in 2024-03-30T14:00:43.101 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:42 smithi043 ceph-mon[42099]: pgmap v804: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 313 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.61k op/s; 941/15576 objects degraded (6.041%); 947/15576 objects misplaced (6.080%) 2024-03-30T14:00:43.101 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:42 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 941/15576 objects degraded (6.041%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:43.101 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:42 smithi043 ceph-mon[42099]: osdmap e366: 8 total, 7 up, 7 in 2024-03-30T14:00:43.162 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:43.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:43.187 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:43.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:43.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:42 smithi113 ceph-mon[34341]: pgmap v804: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 313 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.61k op/s; 941/15576 objects degraded (6.041%); 947/15576 objects misplaced (6.080%) 2024-03-30T14:00:43.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:42 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 941/15576 objects degraded (6.041%), 9 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:43.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:42 smithi113 ceph-mon[34341]: osdmap e366: 8 total, 7 up, 7 in 2024-03-30T14:00:43.288 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:43.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:43.312 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:43.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:43.413 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:43.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:43.439 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:43.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:43.540 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:43.544 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-30T14:00:43.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:43.564 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:43.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:43.665 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:43.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:43.689 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:43.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:43.790 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:43.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:43.815 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:43.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:43.916 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:43.940 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:43.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:43.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:44.042 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:44.067 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:44.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:44.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:44.169 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:44.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:44.194 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:44.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:44.295 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:44.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:44.321 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:44.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:44.422 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:44.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:44.447 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:44.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:44.548 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:44.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:44.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:44.673 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:44.701 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:44.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:44.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:44.802 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:44 smithi043 ceph-mon[29963]: pgmap v806: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 313 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 941/15576 objects degraded (6.041%); 947/15576 objects misplaced (6.080%) 2024-03-30T14:00:44.803 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:44 smithi043 ceph-mon[42099]: pgmap v806: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 313 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 941/15576 objects degraded (6.041%); 947/15576 objects misplaced (6.080%) 2024-03-30T14:00:44.804 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:44.831 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:44.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:44.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:44.933 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:44.959 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:44.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:44.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:44.989 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T14:00:44.989 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T14:00:44.989 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T14:00:44.989 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T14:00:44.989 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T14:00:44.989 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T14:00:44.989 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T14:00:44.989 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:00:44.989 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:00:44.989 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T14:00:44.990 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:00:44.990 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:00:44.990 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:00:44.990 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:00:44.990 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:00:45.061 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:45.085 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:45.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:45.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:45.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:44 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:45.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:44 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:45.187 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:45.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:44 smithi113 ceph-mon[34341]: pgmap v806: 158 pgs: 106 active+clean, 12 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 313 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 941/15576 objects degraded (6.041%); 947/15576 objects misplaced (6.080%) 2024-03-30T14:00:45.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:44 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:45.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:45.216 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:45.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:45.317 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:45.345 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:45.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:45.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:45.447 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:45.475 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:45.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:45.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:45.577 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:45.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:45.603 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:45.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:45.644 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T14:00:45.647 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-30T14:00:45.704 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:45.732 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:45.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:45.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:45.834 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:45.861 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:45.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:45.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:45.863 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T14:00:45.863 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T14:00:45.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:45 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3203971374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:00:45.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:45 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3203971374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:00:45.963 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:45.990 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:45.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:45.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:46.092 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:46.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:46.117 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:46.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:46.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:45 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3203971374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:00:46.218 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:46.245 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:46.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:46.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:46.347 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:46.372 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:46.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:46.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:46.392 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:00:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:00:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:00:46.474 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:46.499 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:46.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:46.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:46.512 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T14:00:46.527 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":806,"stamp":"2024-03-30T14:00:43.732491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":327817590,"num_objects":5192,"num_object_clones":0,"num_object_copies":15576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":941,"num_objects_misplaced":947,"num_objects_unfound":0,"num_objects_dirty":5192,"num_whiteouts":0,"num_read":1569,"num_read_kb":1321,"num_write":320178,"num_write_kb":10382382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37393,"num_bytes_recovered":2440107771,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71805,"ondisk_log_size":71805,"up":418,"acting":453,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":481,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":7837524,"kb_used_data":3046164,"kb_used_omap":10,"kb_used_meta":4791221,"kb_avail":648378540,"statfs":{"total":671965249536,"available":663939624960,"internally_reserved":0,"allocated":3119271936,"data_stored":3109067404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11162,"internal_metadata":4906210406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-601751552,"num_objects":-9182,"num_object_clones":0,"num_object_copies":-27546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1831,"num_objects_misplaced":-1784,"num_objects_unfound":0,"num_objects_dirty":-9182,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9182,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003910"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":669,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.645188+0000","last_change":"2024-03-30T14:00:18.152378+0000","last_active":"2024-03-30T14:00:32.645188+0000","last_peered":"2024-03-30T14:00:32.645188+0000","last_clean":"2024-03-30T14:00:32.645188+0000","last_became_active":"2024-03-30T13:58:47.182681+0000","last_became_peered":"2024-03-30T13:58:47.182681+0000","last_unstale":"2024-03-30T14:00:32.645188+0000","last_undegraded":"2024-03-30T14:00:32.645188+0000","last_fullsized":"2024-03-30T14:00:32.645188+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:18.152265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:00:18.152265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:29.293470+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":687,"reported_epoch":365,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:40.814262+0000","last_change":"2024-03-30T13:59:14.336451+0000","last_active":"2024-03-30T14:00:40.814262+0000","last_peered":"2024-03-30T14:00:40.814262+0000","last_clean":"2024-03-30T14:00:40.814262+0000","last_became_active":"2024-03-30T13:58:48.179562+0000","last_became_peered":"2024-03-30T13:58:48.179562+0000","last_unstale":"2024-03-30T14:00:40.814262+0000","last_undegraded":"2024-03-30T14:00:40.814262+0000","last_fullsized":"2024-03-30T14:00:40.814262+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:59:14.336241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:17.548567+0000","last_clean_scrub_stamp":"2024-03-30T13:59:14.336241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:44.085693+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5,6],"avail_no_missing":["1","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"65'1","reported_seq":679,"reported_epoch":365,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:40.808395+0000","last_change":"2024-03-30T14:00:02.574522+0000","last_active":"2024-03-30T14:00:40.808395+0000","last_peered":"2024-03-30T14:00:40.808395+0000","last_clean":"2024-03-30T14:00:40.808395+0000","last_became_active":"2024-03-30T13:58:48.185898+0000","last_became_peered":"2024-03-30T13:58:48.185898+0000","last_unstale":"2024-03-30T14:00:40.808395+0000","last_undegraded":"2024-03-30T14:00:40.808395+0000","last_fullsized":"2024-03-30T14:00:40.808395+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T14:00:02.574261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:00:02.574261+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:08.359816+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,6],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"216'15","reported_seq":684,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.647981+0000","last_change":"2024-03-30T13:59:17.249544+0000","last_active":"2024-03-30T14:00:32.647981+0000","last_peered":"2024-03-30T14:00:32.647981+0000","last_clean":"2024-03-30T14:00:32.647981+0000","last_became_active":"2024-03-30T13:42:47.319638+0000","last_became_peered":"2024-03-30T13:42:47.319638+0000","last_unstale":"2024-03-30T14:00:32.647981+0000","last_undegraded":"2024-03-30T14:00:32.647981+0000","last_fullsized":"2024-03-30T14:00:32.647981+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'15","last_scrub_stamp":"2024-03-30T13:59:17.249429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:59:17.249429+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:42.173812+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":653,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.649493+0000","last_change":"2024-03-30T13:58:08.388962+0000","last_active":"2024-03-30T14:00:42.649493+0000","last_peered":"2024-03-30T14:00:42.649493+0000","last_clean":"2024-03-30T14:00:42.649493+0000","last_became_active":"2024-03-30T13:53:35.862085+0000","last_became_peered":"2024-03-30T13:53:35.862085+0000","last_unstale":"2024-03-30T14:00:42.649493+0000","last_undegraded":"2024-03-30T14:00:42.649493+0000","last_fullsized":"2024-03-30T14:00:42.649493+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:08.388764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:08.388764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337661+0000","scrub_duration":168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":647,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.647799+0000","last_change":"2024-03-30T13:57:12.673777+0000","last_active":"2024-03-30T14:00:32.647799+0000","last_peered":"2024-03-30T14:00:32.647799+0000","last_clean":"2024-03-30T14:00:32.647799+0000","last_became_active":"2024-03-30T13:42:49.327124+0000","last_became_peered":"2024-03-30T13:42:49.327124+0000","last_unstale":"2024-03-30T14:00:32.647799+0000","last_undegraded":"2024-03-30T14:00:32.647799+0000","last_fullsized":"2024-03-30T14:00:32.647799+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:12.673593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:12.673593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:33.023637+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":647,"reported_epoch":366,"state":"active+undersized","last_fresh":"2024-03-30T14:00:42.652272+0000","last_change":"2024-03-30T13:50:08.512316+0000","last_active":"2024-03-30T14:00:42.652272+0000","last_peered":"2024-03-30T14:00:42.652272+0000","last_clean":"2024-03-30T13:50:05.504688+0000","last_became_active":"2024-03-30T13:50:08.512316+0000","last_became_peered":"2024-03-30T13:50:08.512316+0000","last_unstale":"2024-03-30T14:00:42.652272+0000","last_undegraded":"2024-03-30T14:00:42.652272+0000","last_fullsized":"2024-03-30T13:50:08.501439+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:20.499768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:20.499768+0000","last_clean_scrub_stamp":"2024-03-30T13:49:20.499768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337239+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"217'34","reported_seq":684,"reported_epoch":366,"state":"active+undersized+degraded","last_fresh":"2024-03-30T14:00:42.651127+0000","last_change":"2024-03-30T13:50:08.511099+0000","last_active":"2024-03-30T14:00:42.651127+0000","last_peered":"2024-03-30T14:00:42.651127+0000","last_clean":"2024-03-30T13:50:05.499201+0000","last_became_active":"2024-03-30T13:50:08.511099+0000","last_became_peered":"2024-03-30T13:50:08.511099+0000","last_unstale":"2024-03-30T14:00:42.651127+0000","last_undegraded":"2024-03-30T13:50:08.502733+0000","last_fullsized":"2024-03-30T13:50:08.502606+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'22","last_scrub_stamp":"2024-03-30T13:46:49.339954+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:29.151791+0000","last_clean_scrub_stamp":"2024-03-30T13:46:49.339954+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337210+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":656,"reported_epoch":365,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:40.808388+0000","last_change":"2024-03-30T13:58:10.911460+0000","last_active":"2024-03-30T14:00:40.808388+0000","last_peered":"2024-03-30T14:00:40.808388+0000","last_clean":"2024-03-30T14:00:40.808388+0000","last_became_active":"2024-03-30T13:53:36.855710+0000","last_became_peered":"2024-03-30T13:53:36.855710+0000","last_unstale":"2024-03-30T14:00:40.808388+0000","last_undegraded":"2024-03-30T14:00:40.808388+0000","last_fullsized":"2024-03-30T14:00:40.808388+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:10.911245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:27.061281+0000","last_clean_scrub_stamp":"2024-03-30T13:58:10.911245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":87,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.650022+0000","last_change":"2024-03-30T13:58:47.192572+0000","last_active":"2024-03-30T14:00:32.650022+0000","last_peered":"2024-03-30T14:00:32.650022+0000","last_clean":"2024-03-30T14:00:32.650022+0000","last_became_active":"2024-03-30T13:58:47.192207+0000","last_became_peered":"2024-03-30T13:58:47.192207+0000","last_unstale":"2024-03-30T14:00:32.650022+0000","last_undegraded":"2024-03-30T14:00:32.650022+0000","last_fullsized":"2024-03-30T14:00:32.650022+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:07.824904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:20.069864+0000","last_clean_scrub_stamp":"2024-03-30T13:58:07.824904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":95,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.649018+0000","last_change":"2024-03-30T13:59:58.707437+0000","last_active":"2024-03-30T14:00:32.649018+0000","last_peered":"2024-03-30T14:00:32.649018+0000","last_clean":"2024-03-30T14:00:32.649018+0000","last_became_active":"2024-03-30T13:58:47.199130+0000","last_became_peered":"2024-03-30T13:58:47.199130+0000","last_unstale":"2024-03-30T14:00:32.649018+0000","last_undegraded":"2024-03-30T14:00:32.649018+0000","last_fullsized":"2024-03-30T14:00:32.649018+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:59:58.707314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:37.513596+0000","last_clean_scrub_stamp":"2024-03-30T13:59:58.707314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:28.318535+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"216'15","reported_seq":684,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.645641+0000","last_change":"2024-03-30T13:57:19.638146+0000","last_active":"2024-03-30T14:00:32.645641+0000","last_peered":"2024-03-30T14:00:32.645641+0000","last_clean":"2024-03-30T14:00:32.645641+0000","last_became_active":"2024-03-30T13:42:47.310289+0000","last_became_peered":"2024-03-30T13:42:47.310289+0000","last_unstale":"2024-03-30T14:00:32.645641+0000","last_undegraded":"2024-03-30T14:00:32.645641+0000","last_fullsized":"2024-03-30T14:00:32.645641+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'15","last_scrub_stamp":"2024-03-30T13:57:19.638024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:19.638024+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:33.023749+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":436,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.649205+0000","last_change":"2024-03-30T13:58:48.202829+0000","last_active":"2024-03-30T14:00:32.649205+0000","last_peered":"2024-03-30T14:00:32.649205+0000","last_clean":"2024-03-30T14:00:32.649205+0000","last_became_active":"2024-03-30T13:58:48.202430+0000","last_became_peered":"2024-03-30T13:58:48.202430+0000","last_unstale":"2024-03-30T14:00:32.649205+0000","last_undegraded":"2024-03-30T14:00:32.649205+0000","last_fullsized":"2024-03-30T14:00:32.649205+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:57.194555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:56:57.194555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":291,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.646946+0000","last_change":"2024-03-30T13:59:16.269252+0000","last_active":"2024-03-30T14:00:32.646946+0000","last_peered":"2024-03-30T14:00:32.646946+0000","last_clean":"2024-03-30T14:00:32.646946+0000","last_became_active":"2024-03-30T13:53:35.861405+0000","last_became_peered":"2024-03-30T13:53:35.861405+0000","last_unstale":"2024-03-30T14:00:32.646946+0000","last_undegraded":"2024-03-30T14:00:32.646946+0000","last_fullsized":"2024-03-30T14:00:32.646946+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:59:16.269136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:41.633284+0000","last_clean_scrub_stamp":"2024-03-30T13:59:16.269136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:33.618246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"186'2","reported_seq":91,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.649560+0000","last_change":"2024-03-30T13:58:47.201377+0000","last_active":"2024-03-30T14:00:32.649560+0000","last_peered":"2024-03-30T14:00:32.649560+0000","last_clean":"2024-03-30T14:00:32.649560+0000","last_became_active":"2024-03-30T13:58:47.193793+0000","last_became_peered":"2024-03-30T13:58:47.193793+0000","last_unstale":"2024-03-30T14:00:32.649560+0000","last_undegraded":"2024-03-30T14:00:32.649560+0000","last_fullsized":"2024-03-30T14:00:32.649560+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-30T13:57:13.664762+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-30T13:51:15.932076+0000","last_clean_scrub_stamp":"2024-03-30T13:57:13.664762+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"216'9","reported_seq":682,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.645609+0000","last_change":"2024-03-30T13:59:13.285172+0000","last_active":"2024-03-30T14:00:32.645609+0000","last_peered":"2024-03-30T14:00:32.645609+0000","last_clean":"2024-03-30T14:00:32.645609+0000","last_became_active":"2024-03-30T13:42:47.317352+0000","last_became_peered":"2024-03-30T13:42:47.317352+0000","last_unstale":"2024-03-30T14:00:32.645609+0000","last_undegraded":"2024-03-30T14:00:32.645609+0000","last_fullsized":"2024-03-30T14:00:32.645609+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'9","last_scrub_stamp":"2024-03-30T13:59:13.285052+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-30T13:43:50.960980+0000","last_clean_scrub_stamp":"2024-03-30T13:59:13.285052+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:33.024000+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":664,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.646324+0000","last_change":"2024-03-30T13:58:47.199570+0000","last_active":"2024-03-30T14:00:32.646324+0000","last_peered":"2024-03-30T14:00:32.646324+0000","last_clean":"2024-03-30T14:00:32.646324+0000","last_became_active":"2024-03-30T13:58:47.199001+0000","last_became_peered":"2024-03-30T13:58:47.199001+0000","last_unstale":"2024-03-30T14:00:32.646324+0000","last_undegraded":"2024-03-30T14:00:32.646324+0000","last_fullsized":"2024-03-30T14:00:32.646324+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:21.602164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:21.602164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:33.023854+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":672,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.816676+0000","last_change":"2024-03-30T13:59:08.299710+0000","last_active":"2024-03-30T14:00:40.816676+0000","last_peered":"2024-03-30T14:00:40.816676+0000","last_clean":"2024-03-30T14:00:40.816676+0000","last_became_active":"2024-03-30T13:42:49.322665+0000","last_became_peered":"2024-03-30T13:42:49.322665+0000","last_unstale":"2024-03-30T14:00:40.816676+0000","last_undegraded":"2024-03-30T14:00:40.816676+0000","last_fullsized":"2024-03-30T14:00:40.816676+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:59:08.299596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:30.465974+0000","last_clean_scrub_stamp":"2024-03-30T13:59:08.299596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:44.085175+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":669,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.808690+0000","last_change":"2024-03-30T13:57:56.945998+0000","last_active":"2024-03-30T14:00:40.808690+0000","last_peered":"2024-03-30T14:00:40.808690+0000","last_clean":"2024-03-30T14:00:40.808690+0000","last_became_active":"2024-03-30T13:53:35.861043+0000","last_became_peered":"2024-03-30T13:53:35.861043+0000","last_unstale":"2024-03-30T14:00:40.808690+0000","last_undegraded":"2024-03-30T14:00:40.808690+0000","last_fullsized":"2024-03-30T14:00:40.808690+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:56.945861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:43.088964+0000","last_clean_scrub_stamp":"2024-03-30T13:57:56.945861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"216'27","reported_seq":482,"reported_epoch":365,"state":"active+undersized+degraded","last_fresh":"2024-03-30T14:00:40.815235+0000","last_change":"2024-03-30T13:50:08.508437+0000","last_active":"2024-03-30T14:00:40.815235+0000","last_peered":"2024-03-30T14:00:40.815235+0000","last_clean":"2024-03-30T13:50:05.490228+0000","last_became_active":"2024-03-30T13:50:08.508437+0000","last_became_peered":"2024-03-30T13:50:08.508437+0000","last_unstale":"2024-03-30T14:00:40.815235+0000","last_undegraded":"2024-03-30T13:50:08.506233+0000","last_fullsized":"2024-03-30T13:50:08.506116+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-30T13:49:08.415681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:49:08.415681+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:44.085131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":660,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.653883+0000","last_change":"2024-03-30T14:00:09.659366+0000","last_active":"2024-03-30T14:00:42.653883+0000","last_peered":"2024-03-30T14:00:42.653883+0000","last_clean":"2024-03-30T14:00:42.653883+0000","last_became_active":"2024-03-30T13:58:47.194643+0000","last_became_peered":"2024-03-30T13:58:47.194643+0000","last_unstale":"2024-03-30T14:00:42.653883+0000","last_undegraded":"2024-03-30T14:00:42.653883+0000","last_fullsized":"2024-03-30T14:00:42.653883+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:09.659259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:00:09.659259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:13.284156+0000","scrub_duration":220,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":634,"reported_epoch":364,"state":"active+undersized","last_fresh":"2024-03-30T14:00:32.646701+0000","last_change":"2024-03-30T13:50:08.506226+0000","last_active":"2024-03-30T14:00:32.646701+0000","last_peered":"2024-03-30T14:00:32.646701+0000","last_clean":"2024-03-30T13:50:05.490914+0000","last_became_active":"2024-03-30T13:50:08.506226+0000","last_became_peered":"2024-03-30T13:50:08.506226+0000","last_unstale":"2024-03-30T14:00:32.646701+0000","last_undegraded":"2024-03-30T14:00:32.646701+0000","last_fullsized":"2024-03-30T13:50:08.502611+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:25.104610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:48:25.104610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:37.015639+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":642,"reported_epoch":364,"state":"active+undersized","last_fresh":"2024-03-30T14:00:32.645800+0000","last_change":"2024-03-30T13:50:08.508556+0000","last_active":"2024-03-30T14:00:32.645800+0000","last_peered":"2024-03-30T14:00:32.645800+0000","last_clean":"2024-03-30T13:50:07.261781+0000","last_became_active":"2024-03-30T13:50:08.508556+0000","last_became_peered":"2024-03-30T13:50:08.508556+0000","last_unstale":"2024-03-30T14:00:32.645800+0000","last_undegraded":"2024-03-30T14:00:32.645800+0000","last_fullsized":"2024-03-30T13:50:08.502793+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:07.261595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:50:07.261595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:37.015696+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"217'23","reported_seq":700,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.652425+0000","last_change":"2024-03-30T14:00:22.339054+0000","last_active":"2024-03-30T14:00:42.652425+0000","last_peered":"2024-03-30T14:00:42.652425+0000","last_clean":"2024-03-30T14:00:42.652425+0000","last_became_active":"2024-03-30T13:42:47.317881+0000","last_became_peered":"2024-03-30T13:42:47.317881+0000","last_unstale":"2024-03-30T14:00:42.652425+0000","last_undegraded":"2024-03-30T14:00:42.652425+0000","last_fullsized":"2024-03-30T14:00:42.652425+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'23","last_scrub_stamp":"2024-03-30T14:00:22.338785+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-03-30T13:43:52.044427+0000","last_clean_scrub_stamp":"2024-03-30T14:00:22.338785+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:41.237247+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":97,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.649463+0000","last_change":"2024-03-30T14:00:06.704173+0000","last_active":"2024-03-30T14:00:32.649463+0000","last_peered":"2024-03-30T14:00:32.649463+0000","last_clean":"2024-03-30T14:00:32.649463+0000","last_became_active":"2024-03-30T13:58:48.191902+0000","last_became_peered":"2024-03-30T13:58:48.191902+0000","last_unstale":"2024-03-30T14:00:32.649463+0000","last_undegraded":"2024-03-30T14:00:32.649463+0000","last_fullsized":"2024-03-30T14:00:32.649463+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T14:00:06.703980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:00:06.703980+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:29.963207+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"230'10","reported_seq":901,"reported_epoch":364,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:32.658922+0000","last_change":"2024-03-30T14:00:12.475945+0000","last_active":"2024-03-30T14:00:32.658922+0000","last_peered":"2024-03-30T14:00:32.658922+0000","last_clean":"2024-03-30T14:00:32.658922+0000","last_became_active":"2024-03-30T13:53:36.852879+0000","last_became_peered":"2024-03-30T13:53:36.852879+0000","last_unstale":"2024-03-30T14:00:32.658922+0000","last_undegraded":"2024-03-30T14:00:32.658922+0000","last_fullsized":"2024-03-30T14:00:32.658922+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'10","last_scrub_stamp":"2024-03-30T14:00:12.474921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:00:12.474921+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:29.936204+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,7],"avail_no_missing":["6","4","7"],"object_location_counts":[{"shards":"4,6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"65'1","reported_seq":681,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.814088+0000","last_change":"2024-03-30T14:00:05.061932+0000","last_active":"2024-03-30T14:00:40.814088+0000","last_peered":"2024-03-30T14:00:40.814088+0000","last_clean":"2024-03-30T14:00:40.814088+0000","last_became_active":"2024-03-30T13:42:45.307832+0000","last_became_peered":"2024-03-30T13:42:45.307832+0000","last_unstale":"2024-03-30T14:00:40.814088+0000","last_undegraded":"2024-03-30T14:00:40.814088+0000","last_fullsized":"2024-03-30T14:00:40.814088+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T14:00:05.061791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:00:05.061791+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:13.090694+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"217'25","reported_seq":721,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.809331+0000","last_change":"2024-03-30T13:59:59.559708+0000","last_active":"2024-03-30T14:00:40.809331+0000","last_peered":"2024-03-30T14:00:40.809331+0000","last_clean":"2024-03-30T14:00:40.809331+0000","last_became_active":"2024-03-30T13:53:35.858156+0000","last_became_peered":"2024-03-30T13:53:35.858156+0000","last_unstale":"2024-03-30T14:00:40.809331+0000","last_undegraded":"2024-03-30T14:00:40.809331+0000","last_fullsized":"2024-03-30T14:00:40.809331+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'25","last_scrub_stamp":"2024-03-30T13:59:59.559555+0000","last_deep_scrub":"153'15","last_deep_scrub_stamp":"2024-03-30T13:51:22.252831+0000","last_clean_scrub_stamp":"2024-03-30T13:59:59.559555+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:06.679539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":678,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.815061+0000","last_change":"2024-03-30T14:00:17.296945+0000","last_active":"2024-03-30T14:00:40.815061+0000","last_peered":"2024-03-30T14:00:40.815061+0000","last_clean":"2024-03-30T14:00:40.815061+0000","last_became_active":"2024-03-30T13:58:48.180840+0000","last_became_peered":"2024-03-30T13:58:48.180840+0000","last_unstale":"2024-03-30T14:00:40.815061+0000","last_undegraded":"2024-03-30T14:00:40.815061+0000","last_fullsized":"2024-03-30T14:00:40.815061+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:17.296689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:00:17.296689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:31.513030+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":664,"reported_epoch":365,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:40.809487+0000","last_change":"2024-03-30T14:00:09.655829+0000","last_active":"2024-03-30T14:00:40.809487+0000","last_peered":"2024-03-30T14:00:40.809487+0000","last_clean":"2024-03-30T14:00:40.809487+0000","last_became_active":"2024-03-30T13:53:36.857751+0000","last_became_peered":"2024-03-30T13:53:36.857751+0000","last_unstale":"2024-03-30T14:00:40.809487+0000","last_undegraded":"2024-03-30T14:00:40.809487+0000","last_fullsized":"2024-03-30T14:00:40.809487+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:09.655585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:00:09.655585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:26.153726+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":659,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.649172+0000","last_change":"2024-03-30T13:58:14.227217+0000","last_active":"2024-03-30T14:00:42.649172+0000","last_peered":"2024-03-30T14:00:42.649172+0000","last_clean":"2024-03-30T14:00:42.649172+0000","last_became_active":"2024-03-30T13:42:45.310165+0000","last_became_peered":"2024-03-30T13:42:45.310165+0000","last_unstale":"2024-03-30T14:00:42.649172+0000","last_undegraded":"2024-03-30T14:00:42.649172+0000","last_fullsized":"2024-03-30T14:00:42.649172+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:14.224328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:14.224328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337720+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"217'19","reported_seq":732,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.658960+0000","last_change":"2024-03-30T13:58:47.224883+0000","last_active":"2024-03-30T14:00:32.658960+0000","last_peered":"2024-03-30T14:00:32.658960+0000","last_clean":"2024-03-30T14:00:32.658960+0000","last_became_active":"2024-03-30T13:58:47.204696+0000","last_became_peered":"2024-03-30T13:58:47.204696+0000","last_unstale":"2024-03-30T14:00:32.658960+0000","last_undegraded":"2024-03-30T14:00:32.658960+0000","last_fullsized":"2024-03-30T14:00:32.658960+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'19","last_scrub_stamp":"2024-03-30T13:57:11.200126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:11.200126+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:34.214861+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":525,"reported_epoch":364,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:32.645334+0000","last_change":"2024-03-30T14:00:14.193873+0000","last_active":"2024-03-30T14:00:32.645334+0000","last_peered":"2024-03-30T14:00:32.645334+0000","last_clean":"2024-03-30T14:00:32.645334+0000","last_became_active":"2024-03-30T13:53:36.863938+0000","last_became_peered":"2024-03-30T13:53:36.863938+0000","last_unstale":"2024-03-30T14:00:32.645334+0000","last_undegraded":"2024-03-30T14:00:32.645334+0000","last_fullsized":"2024-03-30T14:00:32.645334+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:14.193663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:00:14.193663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:32.380384+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3,7],"avail_no_missing":["4","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":682,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.655936+0000","last_change":"2024-03-30T14:00:08.488553+0000","last_active":"2024-03-30T14:00:32.655936+0000","last_peered":"2024-03-30T14:00:32.655936+0000","last_clean":"2024-03-30T14:00:32.655936+0000","last_became_active":"2024-03-30T13:53:35.862467+0000","last_became_peered":"2024-03-30T13:53:35.862467+0000","last_unstale":"2024-03-30T14:00:32.655936+0000","last_undegraded":"2024-03-30T14:00:32.655936+0000","last_fullsized":"2024-03-30T14:00:32.655936+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:08.488436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:54.893653+0000","last_clean_scrub_stamp":"2024-03-30T14:00:08.488436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:35.324627+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":683,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.808751+0000","last_change":"2024-03-30T13:58:47.200252+0000","last_active":"2024-03-30T14:00:40.808751+0000","last_peered":"2024-03-30T14:00:40.808751+0000","last_clean":"2024-03-30T14:00:40.808751+0000","last_became_active":"2024-03-30T13:58:47.199798+0000","last_became_peered":"2024-03-30T13:58:47.199798+0000","last_unstale":"2024-03-30T14:00:40.808751+0000","last_undegraded":"2024-03-30T14:00:40.808751+0000","last_fullsized":"2024-03-30T14:00:40.808751+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:16.924797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:16.924797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"216'8","reported_seq":667,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.649272+0000","last_change":"2024-03-30T13:57:07.952284+0000","last_active":"2024-03-30T14:00:42.649272+0000","last_peered":"2024-03-30T14:00:42.649272+0000","last_clean":"2024-03-30T14:00:42.649272+0000","last_became_active":"2024-03-30T13:42:47.321428+0000","last_became_peered":"2024-03-30T13:42:47.321428+0000","last_unstale":"2024-03-30T14:00:42.649272+0000","last_undegraded":"2024-03-30T14:00:42.649272+0000","last_fullsized":"2024-03-30T14:00:42.649272+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'8","last_scrub_stamp":"2024-03-30T13:57:07.952145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:07.952145+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337473+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":648,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.650760+0000","last_change":"2024-03-30T14:00:05.461525+0000","last_active":"2024-03-30T14:00:42.650760+0000","last_peered":"2024-03-30T14:00:42.650760+0000","last_clean":"2024-03-30T14:00:42.650760+0000","last_became_active":"2024-03-30T13:42:51.374304+0000","last_became_peered":"2024-03-30T13:42:51.374304+0000","last_unstale":"2024-03-30T14:00:42.650760+0000","last_undegraded":"2024-03-30T14:00:42.650760+0000","last_fullsized":"2024-03-30T14:00:42.650760+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:05.461415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:00:05.461415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:28.489975+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":292,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.647585+0000","last_change":"2024-03-30T14:00:11.156066+0000","last_active":"2024-03-30T14:00:32.647585+0000","last_peered":"2024-03-30T14:00:32.647585+0000","last_clean":"2024-03-30T14:00:32.647585+0000","last_became_active":"2024-03-30T13:53:35.854002+0000","last_became_peered":"2024-03-30T13:53:35.854002+0000","last_unstale":"2024-03-30T14:00:32.647585+0000","last_undegraded":"2024-03-30T14:00:32.647585+0000","last_fullsized":"2024-03-30T14:00:32.647585+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:11.155966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:42.616155+0000","last_clean_scrub_stamp":"2024-03-30T14:00:11.155966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:20.424329+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":85,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.649301+0000","last_change":"2024-03-30T13:58:48.187642+0000","last_active":"2024-03-30T14:00:32.649301+0000","last_peered":"2024-03-30T14:00:32.649301+0000","last_clean":"2024-03-30T14:00:32.649301+0000","last_became_active":"2024-03-30T13:58:48.187064+0000","last_became_peered":"2024-03-30T13:58:48.187064+0000","last_unstale":"2024-03-30T14:00:32.649301+0000","last_undegraded":"2024-03-30T14:00:32.649301+0000","last_fullsized":"2024-03-30T14:00:32.649301+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:07.098782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:25.494903+0000","last_clean_scrub_stamp":"2024-03-30T13:57:07.098782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"217'19","reported_seq":730,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.651107+0000","last_change":"2024-03-30T14:00:24.357354+0000","last_active":"2024-03-30T14:00:42.651107+0000","last_peered":"2024-03-30T14:00:42.651107+0000","last_clean":"2024-03-30T14:00:42.651107+0000","last_became_active":"2024-03-30T13:58:47.193590+0000","last_became_peered":"2024-03-30T13:58:47.193590+0000","last_unstale":"2024-03-30T14:00:42.651107+0000","last_undegraded":"2024-03-30T14:00:42.651107+0000","last_fullsized":"2024-03-30T14:00:42.651107+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'19","last_scrub_stamp":"2024-03-30T14:00:24.357069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:00:24.357069+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:30.729730+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":663,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.807859+0000","last_change":"2024-03-30T13:58:47.198718+0000","last_active":"2024-03-30T14:00:40.807859+0000","last_peered":"2024-03-30T14:00:40.807859+0000","last_clean":"2024-03-30T14:00:40.807859+0000","last_became_active":"2024-03-30T13:58:47.198442+0000","last_became_peered":"2024-03-30T13:58:47.198442+0000","last_unstale":"2024-03-30T14:00:40.807859+0000","last_undegraded":"2024-03-30T14:00:40.807859+0000","last_fullsized":"2024-03-30T14:00:40.807859+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:13.749366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:28.314391+0000","last_clean_scrub_stamp":"2024-03-30T13:57:13.749366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":651,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.649410+0000","last_change":"2024-03-30T14:00:01.326908+0000","last_active":"2024-03-30T14:00:42.649410+0000","last_peered":"2024-03-30T14:00:42.649410+0000","last_clean":"2024-03-30T14:00:42.649410+0000","last_became_active":"2024-03-30T13:42:49.325219+0000","last_became_peered":"2024-03-30T13:42:49.325219+0000","last_unstale":"2024-03-30T14:00:42.649410+0000","last_undegraded":"2024-03-30T14:00:42.649410+0000","last_fullsized":"2024-03-30T14:00:42.649410+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:01.326777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:00:01.326777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:29.159738+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":296,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.652265+0000","last_change":"2024-03-30T13:57:19.071948+0000","last_active":"2024-03-30T14:00:42.652265+0000","last_peered":"2024-03-30T14:00:42.652265+0000","last_clean":"2024-03-30T14:00:42.652265+0000","last_became_active":"2024-03-30T13:53:35.859253+0000","last_became_peered":"2024-03-30T13:53:35.859253+0000","last_unstale":"2024-03-30T14:00:42.652265+0000","last_undegraded":"2024-03-30T14:00:42.652265+0000","last_fullsized":"2024-03-30T14:00:42.652265+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:19.071814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:11.701294+0000","last_clean_scrub_stamp":"2024-03-30T13:57:19.071814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337557+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"216'15","reported_seq":670,"reported_epoch":365,"state":"active+undersized+degraded","last_fresh":"2024-03-30T14:00:40.818887+0000","last_change":"2024-03-30T13:50:08.513156+0000","last_active":"2024-03-30T14:00:40.818887+0000","last_peered":"2024-03-30T14:00:40.818887+0000","last_clean":"2024-03-30T13:50:05.496678+0000","last_became_active":"2024-03-30T13:50:08.513156+0000","last_became_peered":"2024-03-30T13:50:08.513156+0000","last_unstale":"2024-03-30T14:00:40.818887+0000","last_undegraded":"2024-03-30T13:50:08.507726+0000","last_fullsized":"2024-03-30T13:50:08.507616+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:49:26.480540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:49:26.480540+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:44.085145+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":658,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.650581+0000","last_change":"2024-03-30T13:58:47.194818+0000","last_active":"2024-03-30T14:00:42.650581+0000","last_peered":"2024-03-30T14:00:42.650581+0000","last_clean":"2024-03-30T14:00:42.650581+0000","last_became_active":"2024-03-30T13:58:47.194434+0000","last_became_peered":"2024-03-30T13:58:47.194434+0000","last_unstale":"2024-03-30T14:00:42.650581+0000","last_undegraded":"2024-03-30T14:00:42.650581+0000","last_fullsized":"2024-03-30T14:00:42.650581+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:17.058094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:24.598976+0000","last_clean_scrub_stamp":"2024-03-30T13:57:17.058094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337500+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":688,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.818698+0000","last_change":"2024-03-30T14:00:13.220615+0000","last_active":"2024-03-30T14:00:40.818698+0000","last_peered":"2024-03-30T14:00:40.818698+0000","last_clean":"2024-03-30T14:00:40.818698+0000","last_became_active":"2024-03-30T13:58:47.185594+0000","last_became_peered":"2024-03-30T13:58:47.185594+0000","last_unstale":"2024-03-30T14:00:40.818698+0000","last_undegraded":"2024-03-30T14:00:40.818698+0000","last_fullsized":"2024-03-30T14:00:40.818698+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:13.220494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:34.815783+0000","last_clean_scrub_stamp":"2024-03-30T14:00:13.220494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:28.946458+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":668,"reported_epoch":364,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:32.646333+0000","last_change":"2024-03-30T13:58:52.482066+0000","last_active":"2024-03-30T14:00:32.646333+0000","last_peered":"2024-03-30T14:00:32.646333+0000","last_clean":"2024-03-30T14:00:32.646333+0000","last_became_active":"2024-03-30T13:53:36.848656+0000","last_became_peered":"2024-03-30T13:53:36.848656+0000","last_unstale":"2024-03-30T14:00:32.646333+0000","last_undegraded":"2024-03-30T14:00:32.646333+0000","last_fullsized":"2024-03-30T14:00:32.646333+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:52.481860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:52.481860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:33.023969+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,7],"avail_no_missing":["4","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"216'19","reported_seq":713,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.645766+0000","last_change":"2024-03-30T14:00:04.206440+0000","last_active":"2024-03-30T14:00:32.645766+0000","last_peered":"2024-03-30T14:00:32.645766+0000","last_clean":"2024-03-30T14:00:32.645766+0000","last_became_active":"2024-03-30T13:59:55.845201+0000","last_became_peered":"2024-03-30T13:59:55.845201+0000","last_unstale":"2024-03-30T14:00:32.645766+0000","last_undegraded":"2024-03-30T14:00:32.645766+0000","last_fullsized":"2024-03-30T14:00:32.645766+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'19","last_scrub_stamp":"2024-03-30T14:00:04.206318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:00:04.206318+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:33.930228+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":665,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.809123+0000","last_change":"2024-03-30T13:59:58.656833+0000","last_active":"2024-03-30T14:00:40.809123+0000","last_peered":"2024-03-30T14:00:40.809123+0000","last_clean":"2024-03-30T14:00:40.809123+0000","last_became_active":"2024-03-30T13:53:35.855996+0000","last_became_peered":"2024-03-30T13:53:35.855996+0000","last_unstale":"2024-03-30T14:00:40.809123+0000","last_undegraded":"2024-03-30T14:00:40.809123+0000","last_fullsized":"2024-03-30T14:00:40.809123+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:59:58.656716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:56.942611+0000","last_clean_scrub_stamp":"2024-03-30T13:59:58.656716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:00.059178+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"326'10","reported_seq":114,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.649124+0000","last_change":"2024-03-30T13:58:47.205173+0000","last_active":"2024-03-30T14:00:32.649124+0000","last_peered":"2024-03-30T14:00:32.649124+0000","last_clean":"2024-03-30T14:00:32.649124+0000","last_became_active":"2024-03-30T13:58:47.194143+0000","last_became_peered":"2024-03-30T13:58:47.194143+0000","last_unstale":"2024-03-30T14:00:32.649124+0000","last_undegraded":"2024-03-30T14:00:32.649124+0000","last_fullsized":"2024-03-30T14:00:32.649124+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'9","last_scrub_stamp":"2024-03-30T13:57:57.056166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:57.056166+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":537,"reported_epoch":364,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:32.645801+0000","last_change":"2024-03-30T14:00:23.113883+0000","last_active":"2024-03-30T14:00:32.645801+0000","last_peered":"2024-03-30T14:00:32.645801+0000","last_clean":"2024-03-30T14:00:32.645801+0000","last_became_active":"2024-03-30T13:53:36.849971+0000","last_became_peered":"2024-03-30T13:53:36.849971+0000","last_unstale":"2024-03-30T14:00:32.645801+0000","last_undegraded":"2024-03-30T14:00:32.645801+0000","last_fullsized":"2024-03-30T14:00:32.645801+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:23.112463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:00:23.112463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:37.931934+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,7],"avail_no_missing":["4","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"217'20","reported_seq":707,"reported_epoch":366,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:42.650162+0000","last_change":"2024-03-30T13:56:58.941975+0000","last_active":"2024-03-30T14:00:42.650162+0000","last_peered":"2024-03-30T14:00:42.650162+0000","last_clean":"2024-03-30T14:00:42.650162+0000","last_became_active":"2024-03-30T13:53:37.110194+0000","last_became_peered":"2024-03-30T13:53:37.110194+0000","last_unstale":"2024-03-30T14:00:42.650162+0000","last_undegraded":"2024-03-30T14:00:42.650162+0000","last_fullsized":"2024-03-30T14:00:42.650162+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'20","last_scrub_stamp":"2024-03-30T13:56:58.941755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:56:58.941755+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":648,"reported_epoch":364,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:32.646411+0000","last_change":"2024-03-30T13:58:48.187038+0000","last_active":"2024-03-30T14:00:32.646411+0000","last_peered":"2024-03-30T14:00:32.646411+0000","last_clean":"2024-03-30T14:00:32.646411+0000","last_became_active":"2024-03-30T13:58:48.186613+0000","last_became_peered":"2024-03-30T13:58:48.186613+0000","last_unstale":"2024-03-30T14:00:32.646411+0000","last_undegraded":"2024-03-30T14:00:32.646411+0000","last_fullsized":"2024-03-30T14:00:32.646411+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:06.727774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:06.727774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:33.023720+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,0],"avail_no_missing":["4","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"230'11","reported_seq":881,"reported_epoch":366,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:42.650405+0000","last_change":"2024-03-30T13:57:01.991621+0000","last_active":"2024-03-30T14:00:42.650405+0000","last_peered":"2024-03-30T14:00:42.650405+0000","last_clean":"2024-03-30T14:00:42.650405+0000","last_became_active":"2024-03-30T13:53:37.127826+0000","last_became_peered":"2024-03-30T13:53:37.127826+0000","last_unstale":"2024-03-30T14:00:42.650405+0000","last_undegraded":"2024-03-30T14:00:42.650405+0000","last_fullsized":"2024-03-30T14:00:42.650405+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'11","last_scrub_stamp":"2024-03-30T13:57:01.991476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:01.991476+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":669,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.811729+0000","last_change":"2024-03-30T13:58:03.861834+0000","last_active":"2024-03-30T14:00:40.811729+0000","last_peered":"2024-03-30T14:00:40.811729+0000","last_clean":"2024-03-30T14:00:40.811729+0000","last_became_active":"2024-03-30T13:53:35.857456+0000","last_became_peered":"2024-03-30T13:53:35.857456+0000","last_unstale":"2024-03-30T14:00:40.811729+0000","last_undegraded":"2024-03-30T14:00:40.811729+0000","last_fullsized":"2024-03-30T14:00:40.811729+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:03.861718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:28.734080+0000","last_clean_scrub_stamp":"2024-03-30T13:58:03.861718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"217'15","reported_seq":97,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.648929+0000","last_change":"2024-03-30T13:58:47.214215+0000","last_active":"2024-03-30T14:00:32.648929+0000","last_peered":"2024-03-30T14:00:32.648929+0000","last_clean":"2024-03-30T14:00:32.648929+0000","last_became_active":"2024-03-30T13:58:47.198411+0000","last_became_peered":"2024-03-30T13:58:47.198411+0000","last_unstale":"2024-03-30T14:00:32.648929+0000","last_undegraded":"2024-03-30T14:00:32.648929+0000","last_fullsized":"2024-03-30T14:00:32.648929+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'15","last_scrub_stamp":"2024-03-30T13:57:12.039992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:12.039992+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":431,"reported_epoch":364,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:32.646779+0000","last_change":"2024-03-30T13:57:01.689045+0000","last_active":"2024-03-30T14:00:32.646779+0000","last_peered":"2024-03-30T14:00:32.646779+0000","last_clean":"2024-03-30T14:00:32.646779+0000","last_became_active":"2024-03-30T13:53:36.857026+0000","last_became_peered":"2024-03-30T13:53:36.857026+0000","last_unstale":"2024-03-30T14:00:32.646779+0000","last_undegraded":"2024-03-30T14:00:32.646779+0000","last_fullsized":"2024-03-30T14:00:32.646779+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:01.688893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:01.688893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:33.023668+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,7],"avail_no_missing":["4","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":657,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.654614+0000","last_change":"2024-03-30T13:58:03.191278+0000","last_active":"2024-03-30T14:00:42.654614+0000","last_peered":"2024-03-30T14:00:42.654614+0000","last_clean":"2024-03-30T14:00:42.654614+0000","last_became_active":"2024-03-30T13:53:35.875783+0000","last_became_peered":"2024-03-30T13:53:35.875783+0000","last_unstale":"2024-03-30T14:00:42.654614+0000","last_undegraded":"2024-03-30T14:00:42.654614+0000","last_fullsized":"2024-03-30T14:00:42.654614+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:03.191052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:58:03.191052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337635+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":292,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.645937+0000","last_change":"2024-03-30T13:58:16.531285+0000","last_active":"2024-03-30T14:00:32.645937+0000","last_peered":"2024-03-30T14:00:32.645937+0000","last_clean":"2024-03-30T14:00:32.645937+0000","last_became_active":"2024-03-30T13:53:35.858261+0000","last_became_peered":"2024-03-30T13:53:35.858261+0000","last_unstale":"2024-03-30T14:00:32.645937+0000","last_undegraded":"2024-03-30T14:00:32.645937+0000","last_fullsized":"2024-03-30T14:00:32.645937+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:16.530762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:16.530762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"218'16","reported_seq":698,"reported_epoch":365,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:40.809405+0000","last_change":"2024-03-30T14:00:00.547929+0000","last_active":"2024-03-30T14:00:40.809405+0000","last_peered":"2024-03-30T14:00:40.809405+0000","last_clean":"2024-03-30T14:00:40.809405+0000","last_became_active":"2024-03-30T13:53:36.861455+0000","last_became_peered":"2024-03-30T13:53:36.861455+0000","last_unstale":"2024-03-30T14:00:40.809405+0000","last_undegraded":"2024-03-30T14:00:40.809405+0000","last_fullsized":"2024-03-30T14:00:40.809405+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'16","last_scrub_stamp":"2024-03-30T14:00:00.547687+0000","last_deep_scrub":"218'16","last_deep_scrub_stamp":"2024-03-30T13:52:59.883027+0000","last_clean_scrub_stamp":"2024-03-30T14:00:00.547687+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:11.950912+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3,7],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"72'1","reported_seq":325,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.657188+0000","last_change":"2024-03-30T14:00:18.497292+0000","last_active":"2024-03-30T14:00:32.657188+0000","last_peered":"2024-03-30T14:00:32.657188+0000","last_clean":"2024-03-30T14:00:32.657188+0000","last_became_active":"2024-03-30T13:58:47.198798+0000","last_became_peered":"2024-03-30T13:58:47.198798+0000","last_unstale":"2024-03-30T14:00:32.657188+0000","last_undegraded":"2024-03-30T14:00:32.657188+0000","last_fullsized":"2024-03-30T14:00:32.657188+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T14:00:18.497174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:00:18.497174+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:25.649257+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":87,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.649683+0000","last_change":"2024-03-30T13:58:47.193833+0000","last_active":"2024-03-30T14:00:32.649683+0000","last_peered":"2024-03-30T14:00:32.649683+0000","last_clean":"2024-03-30T14:00:32.649683+0000","last_became_active":"2024-03-30T13:58:47.193472+0000","last_became_peered":"2024-03-30T13:58:47.193472+0000","last_unstale":"2024-03-30T14:00:32.649683+0000","last_undegraded":"2024-03-30T14:00:32.649683+0000","last_fullsized":"2024-03-30T14:00:32.649683+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:09.967517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:57:09.967517+0000","last_clean_scrub_stamp":"2024-03-30T13:57:09.967517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":688,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.657843+0000","last_change":"2024-03-30T13:58:47.205520+0000","last_active":"2024-03-30T14:00:32.657843+0000","last_peered":"2024-03-30T14:00:32.657843+0000","last_clean":"2024-03-30T14:00:32.657843+0000","last_became_active":"2024-03-30T13:58:47.203903+0000","last_became_peered":"2024-03-30T13:58:47.203903+0000","last_unstale":"2024-03-30T14:00:32.657843+0000","last_undegraded":"2024-03-30T14:00:32.657843+0000","last_fullsized":"2024-03-30T14:00:32.657843+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:17.132009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:51:20.175438+0000","last_clean_scrub_stamp":"2024-03-30T13:57:17.132009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:34.214827+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"316'15","reported_seq":681,"reported_epoch":366,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:42.654307+0000","last_change":"2024-03-30T13:58:13.210664+0000","last_active":"2024-03-30T14:00:42.654307+0000","last_peered":"2024-03-30T14:00:42.654307+0000","last_clean":"2024-03-30T14:00:42.654307+0000","last_became_active":"2024-03-30T13:53:37.098834+0000","last_became_peered":"2024-03-30T13:53:37.098834+0000","last_unstale":"2024-03-30T14:00:42.654307+0000","last_undegraded":"2024-03-30T14:00:42.654307+0000","last_fullsized":"2024-03-30T14:00:42.654307+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'15","last_scrub_stamp":"2024-03-30T13:58:13.210392+0000","last_deep_scrub":"316'15","last_deep_scrub_stamp":"2024-03-30T13:58:13.210392+0000","last_clean_scrub_stamp":"2024-03-30T13:58:13.210392+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337693+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"364'20010","reported_seq":29910,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:38.717872+0000","last_change":"2024-03-30T14:00:14.724261+0000","last_active":"2024-03-30T14:00:38.717872+0000","last_peered":"2024-03-30T14:00:38.717872+0000","last_clean":"2024-03-30T14:00:38.717872+0000","last_became_active":"2024-03-30T13:59:59.651012+0000","last_became_peered":"2024-03-30T13:59:59.651012+0000","last_unstale":"2024-03-30T14:00:38.717872+0000","last_undegraded":"2024-03-30T14:00:38.717872+0000","last_fullsized":"2024-03-30T14:00:38.717872+0000","mapping_epoch":351,"log_start":"346'15200","ondisk_log_start":"346'15200","created":289,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"359'17772","last_scrub_stamp":"2024-03-30T14:00:14.724015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T14:00:14.724015+0000","objects_scrubbed":2645,"log_size":4810,"log_dups_size":0,"ondisk_log_size":4810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:28.400645+0000","scrub_duration":1005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24903680,"num_objects":380,"num_object_clones":0,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20010,"num_write_kb":652480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4841,"num_bytes_recovered":317194240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":428,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.651955+0000","last_change":"2024-03-30T13:58:48.234962+0000","last_active":"2024-03-30T14:00:32.651955+0000","last_peered":"2024-03-30T14:00:32.651955+0000","last_clean":"2024-03-30T14:00:32.651955+0000","last_became_active":"2024-03-30T13:58:48.234649+0000","last_became_peered":"2024-03-30T13:58:48.234649+0000","last_unstale":"2024-03-30T14:00:32.651955+0000","last_undegraded":"2024-03-30T14:00:32.651955+0000","last_fullsized":"2024-03-30T14:00:32.651955+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:56.190076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:56.190076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":307,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.646142+0000","last_change":"2024-03-30T13:59:11.278099+0000","last_active":"2024-03-30T14:00:32.646142+0000","last_peered":"2024-03-30T14:00:32.646142+0000","last_clean":"2024-03-30T14:00:32.646142+0000","last_became_active":"2024-03-30T13:53:35.874521+0000","last_became_peered":"2024-03-30T13:53:35.874521+0000","last_unstale":"2024-03-30T14:00:32.646142+0000","last_undegraded":"2024-03-30T14:00:32.646142+0000","last_fullsized":"2024-03-30T14:00:32.646142+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-30T13:59:11.277957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:59:11.277957+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:33.024059+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"216'27","reported_seq":103,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.647982+0000","last_change":"2024-03-30T13:58:47.222352+0000","last_active":"2024-03-30T14:00:32.647982+0000","last_peered":"2024-03-30T14:00:32.647982+0000","last_clean":"2024-03-30T14:00:32.647982+0000","last_became_active":"2024-03-30T13:58:47.197778+0000","last_became_peered":"2024-03-30T13:58:47.197778+0000","last_unstale":"2024-03-30T14:00:32.647982+0000","last_undegraded":"2024-03-30T14:00:32.647982+0000","last_fullsized":"2024-03-30T14:00:32.647982+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'27","last_scrub_stamp":"2024-03-30T13:58:08.152048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:58:08.152048+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"229'9","reported_seq":414,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.655690+0000","last_change":"2024-03-30T13:57:55.147011+0000","last_active":"2024-03-30T14:00:32.655690+0000","last_peered":"2024-03-30T14:00:32.655690+0000","last_clean":"2024-03-30T14:00:32.655690+0000","last_became_active":"2024-03-30T13:53:35.862126+0000","last_became_peered":"2024-03-30T13:53:35.862126+0000","last_unstale":"2024-03-30T14:00:32.655690+0000","last_undegraded":"2024-03-30T14:00:32.655690+0000","last_fullsized":"2024-03-30T14:00:32.655690+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'9","last_scrub_stamp":"2024-03-30T13:57:55.146893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:55.146893+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:39.257785+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"364'20268","reported_seq":30946,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:38.938019+0000","last_change":"2024-03-30T13:59:59.104389+0000","last_active":"2024-03-30T14:00:38.938019+0000","last_peered":"2024-03-30T14:00:38.938019+0000","last_clean":"2024-03-30T14:00:38.938019+0000","last_became_active":"2024-03-30T13:59:56.832879+0000","last_became_peered":"2024-03-30T13:59:56.832879+0000","last_unstale":"2024-03-30T14:00:38.938019+0000","last_undegraded":"2024-03-30T14:00:38.938019+0000","last_fullsized":"2024-03-30T14:00:38.938019+0000","mapping_epoch":348,"log_start":"346'15700","ondisk_log_start":"346'15700","created":289,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'16686","last_scrub_stamp":"2024-03-30T13:59:59.104197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:59:59.104197+0000","objects_scrubbed":3916,"log_size":4568,"log_dups_size":0,"ondisk_log_size":4568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:10.499931+0000","scrub_duration":627,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21626880,"num_objects":330,"num_object_clones":0,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20268,"num_write_kb":659136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5228,"num_bytes_recovered":342622208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":653,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.649587+0000","last_change":"2024-03-30T13:58:12.361908+0000","last_active":"2024-03-30T14:00:42.649587+0000","last_peered":"2024-03-30T14:00:42.649587+0000","last_clean":"2024-03-30T14:00:42.649587+0000","last_became_active":"2024-03-30T13:53:35.862056+0000","last_became_peered":"2024-03-30T13:53:35.862056+0000","last_unstale":"2024-03-30T14:00:42.649587+0000","last_undegraded":"2024-03-30T14:00:42.649587+0000","last_fullsized":"2024-03-30T14:00:42.649587+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:12.361782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:12.361782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337609+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":292,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.646237+0000","last_change":"2024-03-30T13:58:13.511206+0000","last_active":"2024-03-30T14:00:32.646237+0000","last_peered":"2024-03-30T14:00:32.646237+0000","last_clean":"2024-03-30T14:00:32.646237+0000","last_became_active":"2024-03-30T13:53:35.861370+0000","last_became_peered":"2024-03-30T13:53:35.861370+0000","last_unstale":"2024-03-30T14:00:32.646237+0000","last_undegraded":"2024-03-30T14:00:32.646237+0000","last_fullsized":"2024-03-30T14:00:32.646237+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:13.511094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:13.511094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:33.023803+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"216'18","reported_seq":692,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.646053+0000","last_change":"2024-03-30T13:58:54.451235+0000","last_active":"2024-03-30T14:00:32.646053+0000","last_peered":"2024-03-30T14:00:32.646053+0000","last_clean":"2024-03-30T14:00:32.646053+0000","last_became_active":"2024-03-30T13:42:47.316187+0000","last_became_peered":"2024-03-30T13:42:47.316187+0000","last_unstale":"2024-03-30T14:00:32.646053+0000","last_undegraded":"2024-03-30T14:00:32.646053+0000","last_fullsized":"2024-03-30T14:00:32.646053+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'18","last_scrub_stamp":"2024-03-30T13:58:54.450973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:58:54.450973+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:33.024030+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":421,"reported_epoch":365,"state":"active+undersized","last_fresh":"2024-03-30T14:00:40.808759+0000","last_change":"2024-03-30T13:50:08.509701+0000","last_active":"2024-03-30T14:00:40.808759+0000","last_peered":"2024-03-30T14:00:40.808759+0000","last_clean":"2024-03-30T13:50:05.499237+0000","last_became_active":"2024-03-30T13:50:08.509701+0000","last_became_peered":"2024-03-30T13:50:08.509701+0000","last_unstale":"2024-03-30T14:00:40.808759+0000","last_undegraded":"2024-03-30T14:00:40.808759+0000","last_fullsized":"2024-03-30T13:50:08.501671+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:22.401087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:22.401087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"365'19775","reported_seq":19981,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.938741+0000","last_change":"2024-03-30T14:00:02.950272+0000","last_active":"2024-03-30T14:00:40.938741+0000","last_peered":"2024-03-30T14:00:40.938741+0000","last_clean":"2024-03-30T14:00:40.938741+0000","last_became_active":"2024-03-30T13:56:55.951841+0000","last_became_peered":"2024-03-30T13:56:55.951841+0000","last_unstale":"2024-03-30T14:00:40.938741+0000","last_undegraded":"2024-03-30T14:00:40.938741+0000","last_fullsized":"2024-03-30T14:00:40.938741+0000","mapping_epoch":289,"log_start":"346'15700","ondisk_log_start":"346'15700","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'16454","last_scrub_stamp":"2024-03-30T14:00:02.950005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T14:00:02.950005+0000","objects_scrubbed":3543,"log_size":4075,"log_dups_size":0,"ondisk_log_size":4075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:28.203427+0000","scrub_duration":1391,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11730944,"num_objects":179,"num_object_clones":0,"num_object_copies":537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19775,"num_write_kb":638528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":530,"reported_epoch":366,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:42.650529+0000","last_change":"2024-03-30T13:57:03.960613+0000","last_active":"2024-03-30T14:00:42.650529+0000","last_peered":"2024-03-30T14:00:42.650529+0000","last_clean":"2024-03-30T14:00:42.650529+0000","last_became_active":"2024-03-30T13:53:36.862437+0000","last_became_peered":"2024-03-30T13:53:36.862437+0000","last_unstale":"2024-03-30T14:00:42.650529+0000","last_undegraded":"2024-03-30T14:00:42.650529+0000","last_fullsized":"2024-03-30T14:00:42.650529+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:03.960362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:03.960362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,7],"avail_no_missing":["3","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":452,"reported_epoch":365,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:40.808225+0000","last_change":"2024-03-30T14:00:20.674508+0000","last_active":"2024-03-30T14:00:40.808225+0000","last_peered":"2024-03-30T14:00:40.808225+0000","last_clean":"2024-03-30T14:00:40.808225+0000","last_became_active":"2024-03-30T13:58:48.184571+0000","last_became_peered":"2024-03-30T13:58:48.184571+0000","last_unstale":"2024-03-30T14:00:40.808225+0000","last_undegraded":"2024-03-30T14:00:40.808225+0000","last_fullsized":"2024-03-30T14:00:40.808225+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:20.674306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:00:20.674306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:23.998678+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,1],"avail_no_missing":["0","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":87,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.648981+0000","last_change":"2024-03-30T13:58:47.200041+0000","last_active":"2024-03-30T14:00:32.648981+0000","last_peered":"2024-03-30T14:00:32.648981+0000","last_clean":"2024-03-30T14:00:32.648981+0000","last_became_active":"2024-03-30T13:58:47.199395+0000","last_became_peered":"2024-03-30T13:58:47.199395+0000","last_unstale":"2024-03-30T14:00:32.648981+0000","last_undegraded":"2024-03-30T14:00:32.648981+0000","last_fullsized":"2024-03-30T14:00:32.648981+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:16.087693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:17.366740+0000","last_clean_scrub_stamp":"2024-03-30T13:58:16.087693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"217'44","reported_seq":789,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.656364+0000","last_change":"2024-03-30T13:58:47.227571+0000","last_active":"2024-03-30T14:00:32.656364+0000","last_peered":"2024-03-30T14:00:32.656364+0000","last_clean":"2024-03-30T14:00:32.656364+0000","last_became_active":"2024-03-30T13:58:47.204577+0000","last_became_peered":"2024-03-30T13:58:47.204577+0000","last_unstale":"2024-03-30T14:00:32.656364+0000","last_undegraded":"2024-03-30T14:00:32.656364+0000","last_fullsized":"2024-03-30T14:00:32.656364+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'44","last_scrub_stamp":"2024-03-30T13:58:21.999653+0000","last_deep_scrub":"108'28","last_deep_scrub_stamp":"2024-03-30T13:45:49.140418+0000","last_clean_scrub_stamp":"2024-03-30T13:58:21.999653+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:34.215039+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"365'20029","reported_seq":20189,"reported_epoch":365,"state":"active+undersized+degraded","last_fresh":"2024-03-30T14:00:40.922333+0000","last_change":"2024-03-30T13:57:08.381005+0000","last_active":"2024-03-30T14:00:40.922333+0000","last_peered":"2024-03-30T14:00:40.922333+0000","last_clean":"2024-03-30T13:56:54.930191+0000","last_became_active":"2024-03-30T13:56:55.950831+0000","last_became_peered":"2024-03-30T13:56:55.950831+0000","last_unstale":"2024-03-30T14:00:40.922333+0000","last_undegraded":"2024-03-30T13:57:08.371957+0000","last_fullsized":"2024-03-30T13:56:55.946732+0000","mapping_epoch":289,"log_start":"346'15900","ondisk_log_start":"346'15900","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4129,"log_dups_size":0,"ondisk_log_size":4129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12517376,"num_objects":191,"num_object_clones":0,"num_object_copies":573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20029,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":191}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":644,"reported_epoch":365,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:40.813640+0000","last_change":"2024-03-30T13:56:59.724692+0000","last_active":"2024-03-30T14:00:40.813640+0000","last_peered":"2024-03-30T14:00:40.813640+0000","last_clean":"2024-03-30T14:00:40.813640+0000","last_became_active":"2024-03-30T13:53:37.359258+0000","last_became_peered":"2024-03-30T13:53:37.359258+0000","last_unstale":"2024-03-30T14:00:40.813640+0000","last_undegraded":"2024-03-30T14:00:40.813640+0000","last_fullsized":"2024-03-30T14:00:40.813640+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:59.724551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:44.095338+0000","last_clean_scrub_stamp":"2024-03-30T13:56:59.724551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3,7],"avail_no_missing":["0","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":418,"reported_epoch":364,"state":"active+undersized","last_fresh":"2024-03-30T14:00:32.646059+0000","last_change":"2024-03-30T13:50:08.507875+0000","last_active":"2024-03-30T14:00:32.646059+0000","last_peered":"2024-03-30T14:00:32.646059+0000","last_clean":"2024-03-30T13:50:05.490245+0000","last_became_active":"2024-03-30T13:50:08.507875+0000","last_became_peered":"2024-03-30T13:50:08.507875+0000","last_unstale":"2024-03-30T14:00:32.646059+0000","last_undegraded":"2024-03-30T14:00:32.646059+0000","last_fullsized":"2024-03-30T13:50:08.502700+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:13.399683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:28.225742+0000","last_clean_scrub_stamp":"2024-03-30T13:49:13.399683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:37.015683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":87,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.649647+0000","last_change":"2024-03-30T13:58:47.195053+0000","last_active":"2024-03-30T14:00:32.649647+0000","last_peered":"2024-03-30T14:00:32.649647+0000","last_clean":"2024-03-30T14:00:32.649647+0000","last_became_active":"2024-03-30T13:58:47.194288+0000","last_became_peered":"2024-03-30T13:58:47.194288+0000","last_unstale":"2024-03-30T14:00:32.649647+0000","last_undegraded":"2024-03-30T14:00:32.649647+0000","last_fullsized":"2024-03-30T14:00:32.649647+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:04.919039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:57:04.919039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"216'15","reported_seq":724,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.808505+0000","last_change":"2024-03-30T13:57:58.907928+0000","last_active":"2024-03-30T14:00:40.808505+0000","last_peered":"2024-03-30T14:00:40.808505+0000","last_clean":"2024-03-30T14:00:40.808505+0000","last_became_active":"2024-03-30T13:53:13.656465+0000","last_became_peered":"2024-03-30T13:53:13.656465+0000","last_unstale":"2024-03-30T14:00:40.808505+0000","last_undegraded":"2024-03-30T14:00:40.808505+0000","last_fullsized":"2024-03-30T14:00:40.808505+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'15","last_scrub_stamp":"2024-03-30T13:57:58.907779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:58.907779+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"364'20113","reported_seq":35501,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:38.940446+0000","last_change":"2024-03-30T14:00:14.408265+0000","last_active":"2024-03-30T14:00:38.940446+0000","last_peered":"2024-03-30T14:00:38.940446+0000","last_clean":"2024-03-30T14:00:38.940446+0000","last_became_active":"2024-03-30T13:59:32.202366+0000","last_became_peered":"2024-03-30T13:59:32.202366+0000","last_unstale":"2024-03-30T14:00:38.940446+0000","last_undegraded":"2024-03-30T14:00:38.940446+0000","last_fullsized":"2024-03-30T14:00:38.940446+0000","mapping_epoch":342,"log_start":"346'15500","ondisk_log_start":"346'15500","created":289,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"359'17809","last_scrub_stamp":"2024-03-30T14:00:14.407894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T14:00:14.407894+0000","objects_scrubbed":2697,"log_size":4613,"log_dups_size":0,"ondisk_log_size":4613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:37.549870+0000","scrub_duration":1003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22347776,"num_objects":341,"num_object_clones":0,"num_object_copies":1023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20113,"num_write_kb":654528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7590,"num_bytes_recovered":497352704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":437,"reported_epoch":364,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:32.649007+0000","last_change":"2024-03-30T14:00:19.739388+0000","last_active":"2024-03-30T14:00:32.649007+0000","last_peered":"2024-03-30T14:00:32.649007+0000","last_clean":"2024-03-30T14:00:32.649007+0000","last_became_active":"2024-03-30T13:58:48.189456+0000","last_became_peered":"2024-03-30T13:58:48.189456+0000","last_unstale":"2024-03-30T14:00:32.649007+0000","last_undegraded":"2024-03-30T14:00:32.649007+0000","last_fullsized":"2024-03-30T14:00:32.649007+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:00:19.739169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:00:19.739169+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:29.527332+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0,4],"avail_no_missing":["5","0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":654,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.652724+0000","last_change":"2024-03-30T13:57:57.278091+0000","last_active":"2024-03-30T14:00:42.652724+0000","last_peered":"2024-03-30T14:00:42.652724+0000","last_clean":"2024-03-30T14:00:42.652724+0000","last_became_active":"2024-03-30T13:53:35.855186+0000","last_became_peered":"2024-03-30T13:53:35.855186+0000","last_unstale":"2024-03-30T14:00:42.652724+0000","last_undegraded":"2024-03-30T14:00:42.652724+0000","last_fullsized":"2024-03-30T14:00:42.652724+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:57.277970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:57.277970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337583+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":91,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.651466+0000","last_change":"2024-03-30T13:58:47.215143+0000","last_active":"2024-03-30T14:00:32.651466+0000","last_peered":"2024-03-30T14:00:32.651466+0000","last_clean":"2024-03-30T14:00:32.651466+0000","last_became_active":"2024-03-30T13:58:47.199657+0000","last_became_peered":"2024-03-30T13:58:47.199657+0000","last_unstale":"2024-03-30T14:00:32.651466+0000","last_undegraded":"2024-03-30T14:00:32.651466+0000","last_fullsized":"2024-03-30T14:00:32.651466+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-30T13:58:03.985919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:58:03.985919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":656,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.645502+0000","last_change":"2024-03-30T13:58:04.478097+0000","last_active":"2024-03-30T14:00:32.645502+0000","last_peered":"2024-03-30T14:00:32.645502+0000","last_clean":"2024-03-30T14:00:32.645502+0000","last_became_active":"2024-03-30T13:42:45.310209+0000","last_became_peered":"2024-03-30T13:42:45.310209+0000","last_unstale":"2024-03-30T14:00:32.645502+0000","last_undegraded":"2024-03-30T14:00:32.645502+0000","last_fullsized":"2024-03-30T14:00:32.645502+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:04.478028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:04.478028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:33.023777+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"217'41","reported_seq":923,"reported_epoch":365,"state":"active+undersized+degraded","last_fresh":"2024-03-30T14:00:40.815674+0000","last_change":"2024-03-30T13:50:08.508727+0000","last_active":"2024-03-30T14:00:40.815674+0000","last_peered":"2024-03-30T14:00:40.815674+0000","last_clean":"2024-03-30T13:50:05.490389+0000","last_became_active":"2024-03-30T13:50:08.508727+0000","last_became_peered":"2024-03-30T13:50:08.508727+0000","last_unstale":"2024-03-30T14:00:40.815674+0000","last_undegraded":"2024-03-30T13:50:08.506113+0000","last_fullsized":"2024-03-30T13:50:08.505993+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'28","last_scrub_stamp":"2024-03-30T13:48:27.377242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:27.377242+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:44.085089+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":664,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.816942+0000","last_change":"2024-03-30T13:58:05.858433+0000","last_active":"2024-03-30T14:00:40.816942+0000","last_peered":"2024-03-30T14:00:40.816942+0000","last_clean":"2024-03-30T14:00:40.816942+0000","last_became_active":"2024-03-30T13:42:49.322154+0000","last_became_peered":"2024-03-30T13:42:49.322154+0000","last_unstale":"2024-03-30T14:00:40.816942+0000","last_undegraded":"2024-03-30T14:00:40.816942+0000","last_fullsized":"2024-03-30T14:00:40.816942+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:05.858319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:18.862784+0000","last_clean_scrub_stamp":"2024-03-30T13:58:05.858319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"364'20049","reported_seq":35087,"reported_epoch":364,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:38.722651+0000","last_change":"2024-03-30T14:00:18.559196+0000","last_active":"2024-03-30T14:00:38.722651+0000","last_peered":"2024-03-30T14:00:38.722651+0000","last_clean":"2024-03-30T14:00:38.722651+0000","last_became_active":"2024-03-30T13:59:32.629713+0000","last_became_peered":"2024-03-30T13:59:32.629713+0000","last_unstale":"2024-03-30T14:00:38.722651+0000","last_undegraded":"2024-03-30T14:00:38.722651+0000","last_fullsized":"2024-03-30T14:00:38.722651+0000","mapping_epoch":343,"log_start":"346'15200","ondisk_log_start":"346'15200","created":289,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"360'18254","last_scrub_stamp":"2024-03-30T14:00:18.558727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T14:00:18.558727+0000","objects_scrubbed":2202,"log_size":4849,"log_dups_size":0,"ondisk_log_size":4849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:41.695865+0000","scrub_duration":813,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24051712,"num_objects":367,"num_object_clones":0,"num_object_copies":1101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":367,"num_objects_unfound":0,"num_objects_dirty":367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20049,"num_write_kb":653312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7421,"num_bytes_recovered":486342656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,3],"avail_no_missing":["5","3","4"],"object_location_counts":[{"shards":"3,4,5","objects":367}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":598,"reported_epoch":365,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:40.811834+0000","last_change":"2024-03-30T14:00:13.751692+0000","last_active":"2024-03-30T14:00:40.811834+0000","last_peered":"2024-03-30T14:00:40.811834+0000","last_clean":"2024-03-30T14:00:40.811834+0000","last_became_active":"2024-03-30T13:53:36.851158+0000","last_became_peered":"2024-03-30T13:53:36.851158+0000","last_unstale":"2024-03-30T14:00:40.811834+0000","last_undegraded":"2024-03-30T14:00:40.811834+0000","last_fullsized":"2024-03-30T14:00:40.811834+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:00:13.751292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:00:13.751292+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:18.885219+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4,7],"avail_no_missing":["0","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":299,"reported_epoch":364,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:32.656631+0000","last_change":"2024-03-30T13:58:04.252009+0000","last_active":"2024-03-30T14:00:32.656631+0000","last_peered":"2024-03-30T14:00:32.656631+0000","last_clean":"2024-03-30T14:00:32.656631+0000","last_became_active":"2024-03-30T13:53:36.860597+0000","last_became_peered":"2024-03-30T13:53:36.860597+0000","last_unstale":"2024-03-30T14:00:32.656631+0000","last_undegraded":"2024-03-30T14:00:32.656631+0000","last_fullsized":"2024-03-30T14:00:32.656631+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:04.251869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:04.251869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:34.214947+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3,7],"avail_no_missing":["6","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":652,"reported_epoch":365,"state":"active+undersized","last_fresh":"2024-03-30T14:00:40.814571+0000","last_change":"2024-03-30T13:50:08.512561+0000","last_active":"2024-03-30T14:00:40.814571+0000","last_peered":"2024-03-30T14:00:40.814571+0000","last_clean":"2024-03-30T13:50:05.489457+0000","last_became_active":"2024-03-30T13:50:08.512561+0000","last_became_peered":"2024-03-30T13:50:08.512561+0000","last_unstale":"2024-03-30T14:00:40.814571+0000","last_undegraded":"2024-03-30T14:00:40.814571+0000","last_fullsized":"2024-03-30T13:50:08.506886+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:16.464234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:59.810189+0000","last_clean_scrub_stamp":"2024-03-30T13:49:16.464234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:44.085117+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":424,"reported_epoch":366,"state":"active+undersized","last_fresh":"2024-03-30T14:00:42.650925+0000","last_change":"2024-03-30T13:50:08.511337+0000","last_active":"2024-03-30T14:00:42.650925+0000","last_peered":"2024-03-30T14:00:42.650925+0000","last_clean":"2024-03-30T13:50:05.498892+0000","last_became_active":"2024-03-30T13:50:08.511337+0000","last_became_peered":"2024-03-30T13:50:08.511337+0000","last_unstale":"2024-03-30T14:00:42.650925+0000","last_undegraded":"2024-03-30T14:00:42.650925+0000","last_fullsized":"2024-03-30T13:50:08.501456+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:25.398232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:51.919203+0000","last_clean_scrub_stamp":"2024-03-30T13:49:25.398232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"216'17","reported_seq":764,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.816159+0000","last_change":"2024-03-30T13:58:59.170542+0000","last_active":"2024-03-30T14:00:40.816159+0000","last_peered":"2024-03-30T14:00:40.816159+0000","last_clean":"2024-03-30T14:00:40.816159+0000","last_became_active":"2024-03-30T13:58:47.192640+0000","last_became_peered":"2024-03-30T13:58:47.192640+0000","last_unstale":"2024-03-30T14:00:40.816159+0000","last_undegraded":"2024-03-30T14:00:40.816159+0000","last_fullsized":"2024-03-30T14:00:40.816159+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'17","last_scrub_stamp":"2024-03-30T13:58:59.170397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:58:59.170397+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:44.085626+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":85,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.649470+0000","last_change":"2024-03-30T13:58:48.178732+0000","last_active":"2024-03-30T14:00:32.649470+0000","last_peered":"2024-03-30T14:00:32.649470+0000","last_clean":"2024-03-30T14:00:32.649470+0000","last_became_active":"2024-03-30T13:58:48.178266+0000","last_became_peered":"2024-03-30T13:58:48.178266+0000","last_unstale":"2024-03-30T14:00:32.649470+0000","last_undegraded":"2024-03-30T14:00:32.649470+0000","last_fullsized":"2024-03-30T14:00:32.649470+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:05.690511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:28.373277+0000","last_clean_scrub_stamp":"2024-03-30T13:57:05.690511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"364'19703","reported_seq":19905,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:38.933609+0000","last_change":"2024-03-30T14:00:06.430449+0000","last_active":"2024-03-30T14:00:38.933609+0000","last_peered":"2024-03-30T14:00:38.933609+0000","last_clean":"2024-03-30T14:00:38.933609+0000","last_became_active":"2024-03-30T13:56:55.952660+0000","last_became_peered":"2024-03-30T13:56:55.952660+0000","last_unstale":"2024-03-30T14:00:38.933609+0000","last_undegraded":"2024-03-30T14:00:38.933609+0000","last_fullsized":"2024-03-30T14:00:38.933609+0000","mapping_epoch":289,"log_start":"346'15100","ondisk_log_start":"346'15100","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'16846","last_scrub_stamp":"2024-03-30T14:00:06.430130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T14:00:06.430130+0000","objects_scrubbed":3267,"log_size":4603,"log_dups_size":0,"ondisk_log_size":4603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:20.460637+0000","scrub_duration":956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23920640,"num_objects":365,"num_object_clones":0,"num_object_copies":1095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19703,"num_write_kb":642176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":95,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.649642+0000","last_change":"2024-03-30T14:00:05.877131+0000","last_active":"2024-03-30T14:00:32.649642+0000","last_peered":"2024-03-30T14:00:32.649642+0000","last_clean":"2024-03-30T14:00:32.649642+0000","last_became_active":"2024-03-30T13:58:47.194374+0000","last_became_peered":"2024-03-30T13:58:47.194374+0000","last_unstale":"2024-03-30T14:00:32.649642+0000","last_undegraded":"2024-03-30T14:00:32.649642+0000","last_fullsized":"2024-03-30T14:00:32.649642+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:05.877011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:12.433166+0000","last_clean_scrub_stamp":"2024-03-30T14:00:05.877011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:24.361283+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":602,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.808278+0000","last_change":"2024-03-30T14:00:06.612216+0000","last_active":"2024-03-30T14:00:40.808278+0000","last_peered":"2024-03-30T14:00:40.808278+0000","last_clean":"2024-03-30T14:00:40.808278+0000","last_became_active":"2024-03-30T13:53:38.545543+0000","last_became_peered":"2024-03-30T13:53:38.545543+0000","last_unstale":"2024-03-30T14:00:40.808278+0000","last_undegraded":"2024-03-30T14:00:40.808278+0000","last_fullsized":"2024-03-30T14:00:40.808278+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:00:06.611924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:00:06.611924+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:31.304401+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":672,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.649257+0000","last_change":"2024-03-30T13:58:47.193901+0000","last_active":"2024-03-30T14:00:42.649257+0000","last_peered":"2024-03-30T14:00:42.649257+0000","last_clean":"2024-03-30T14:00:42.649257+0000","last_became_active":"2024-03-30T13:58:47.193538+0000","last_became_peered":"2024-03-30T13:58:47.193538+0000","last_unstale":"2024-03-30T14:00:42.649257+0000","last_undegraded":"2024-03-30T14:00:42.649257+0000","last_fullsized":"2024-03-30T14:00:42.649257+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:16.238202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:12.975288+0000","last_clean_scrub_stamp":"2024-03-30T13:58:16.238202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337773+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"229'29","reported_seq":952,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.656574+0000","last_change":"2024-03-30T14:00:03.512484+0000","last_active":"2024-03-30T14:00:32.656574+0000","last_peered":"2024-03-30T14:00:32.656574+0000","last_clean":"2024-03-30T14:00:32.656574+0000","last_became_active":"2024-03-30T13:53:35.864416+0000","last_became_peered":"2024-03-30T13:53:35.864416+0000","last_unstale":"2024-03-30T14:00:32.656574+0000","last_undegraded":"2024-03-30T14:00:32.656574+0000","last_fullsized":"2024-03-30T14:00:32.656574+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'29","last_scrub_stamp":"2024-03-30T14:00:03.512142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:00:03.512142+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:29.918338+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":85,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.649372+0000","last_change":"2024-03-30T13:58:48.181574+0000","last_active":"2024-03-30T14:00:32.649372+0000","last_peered":"2024-03-30T14:00:32.649372+0000","last_clean":"2024-03-30T14:00:32.649372+0000","last_became_active":"2024-03-30T13:58:48.179382+0000","last_became_peered":"2024-03-30T13:58:48.179382+0000","last_unstale":"2024-03-30T14:00:32.649372+0000","last_undegraded":"2024-03-30T14:00:32.649372+0000","last_fullsized":"2024-03-30T14:00:32.649372+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:05.759975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:05.759975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.8","version":"365'20169","reported_seq":34751,"reported_epoch":365,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:40.908898+0000","last_change":"2024-03-30T14:00:09.668633+0000","last_active":"2024-03-30T14:00:40.908898+0000","last_peered":"2024-03-30T14:00:40.908898+0000","last_clean":"2024-03-30T14:00:40.908898+0000","last_became_active":"2024-03-30T13:59:33.636270+0000","last_became_peered":"2024-03-30T13:59:33.636270+0000","last_unstale":"2024-03-30T14:00:40.908898+0000","last_undegraded":"2024-03-30T14:00:40.908898+0000","last_fullsized":"2024-03-30T14:00:40.908898+0000","mapping_epoch":344,"log_start":"346'16100","ondisk_log_start":"346'16100","created":289,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'17334","last_scrub_stamp":"2024-03-30T14:00:09.668011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T14:00:09.668011+0000","objects_scrubbed":3057,"log_size":4069,"log_dups_size":0,"ondisk_log_size":4069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:13.059317+0000","scrub_duration":1022,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11730944,"num_objects":179,"num_object_clones":0,"num_object_copies":537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":179,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20169,"num_write_kb":651136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7185,"num_bytes_recovered":470876160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,4],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":179}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":652,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.816268+0000","last_change":"2024-03-30T13:59:15.374370+0000","last_active":"2024-03-30T14:00:40.816268+0000","last_peered":"2024-03-30T14:00:40.816268+0000","last_clean":"2024-03-30T14:00:40.816268+0000","last_became_active":"2024-03-30T13:52:36.489806+0000","last_became_peered":"2024-03-30T13:52:36.489806+0000","last_unstale":"2024-03-30T14:00:40.816268+0000","last_undegraded":"2024-03-30T14:00:40.816268+0000","last_fullsized":"2024-03-30T14:00:40.816268+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:59:15.374217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:59:15.374217+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:41.136496+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":673,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.814413+0000","last_change":"2024-03-30T13:58:56.161545+0000","last_active":"2024-03-30T14:00:40.814413+0000","last_peered":"2024-03-30T14:00:40.814413+0000","last_clean":"2024-03-30T14:00:40.814413+0000","last_became_active":"2024-03-30T13:58:47.189419+0000","last_became_peered":"2024-03-30T13:58:47.189419+0000","last_unstale":"2024-03-30T14:00:40.814413+0000","last_undegraded":"2024-03-30T14:00:40.814413+0000","last_fullsized":"2024-03-30T14:00:40.814413+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:56.161347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:56.161347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:44.085201+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":672,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.812580+0000","last_change":"2024-03-30T14:00:25.597969+0000","last_active":"2024-03-30T14:00:40.812580+0000","last_peered":"2024-03-30T14:00:40.812580+0000","last_clean":"2024-03-30T14:00:40.812580+0000","last_became_active":"2024-03-30T13:42:45.307905+0000","last_became_peered":"2024-03-30T13:42:45.307905+0000","last_unstale":"2024-03-30T14:00:40.812580+0000","last_undegraded":"2024-03-30T14:00:40.812580+0000","last_fullsized":"2024-03-30T14:00:40.812580+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:25.597831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:00:25.597831+0000","last_clean_scrub_stamp":"2024-03-30T14:00:25.597831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:48.523988+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":775,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.814400+0000","last_change":"2024-03-30T13:59:10.350921+0000","last_active":"2024-03-30T14:00:40.814400+0000","last_peered":"2024-03-30T14:00:40.814400+0000","last_clean":"2024-03-30T14:00:40.814400+0000","last_became_active":"2024-03-30T13:53:35.889071+0000","last_became_peered":"2024-03-30T13:53:35.889071+0000","last_unstale":"2024-03-30T14:00:40.814400+0000","last_undegraded":"2024-03-30T14:00:40.814400+0000","last_fullsized":"2024-03-30T14:00:40.814400+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-30T13:59:10.350780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:59:10.350780+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:44.085661+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0.00126391,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":430,"num_read_kb":425,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"216'30","reported_seq":689,"reported_epoch":365,"state":"active+undersized+degraded","last_fresh":"2024-03-30T14:00:40.809189+0000","last_change":"2024-03-30T13:50:08.510256+0000","last_active":"2024-03-30T14:00:40.809189+0000","last_peered":"2024-03-30T14:00:40.809189+0000","last_clean":"2024-03-30T13:50:05.503358+0000","last_became_active":"2024-03-30T13:50:08.510256+0000","last_became_peered":"2024-03-30T13:50:08.510256+0000","last_unstale":"2024-03-30T14:00:40.809189+0000","last_undegraded":"2024-03-30T13:50:08.504536+0000","last_fullsized":"2024-03-30T13:50:08.504398+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'18","last_scrub_stamp":"2024-03-30T13:49:13.774982+0000","last_deep_scrub":"93'18","last_deep_scrub_stamp":"2024-03-30T13:47:28.872529+0000","last_clean_scrub_stamp":"2024-03-30T13:49:13.774982+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":95,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.649741+0000","last_change":"2024-03-30T14:00:22.722109+0000","last_active":"2024-03-30T14:00:32.649741+0000","last_peered":"2024-03-30T14:00:32.649741+0000","last_clean":"2024-03-30T14:00:32.649741+0000","last_became_active":"2024-03-30T13:58:47.195511+0000","last_became_peered":"2024-03-30T13:58:47.195511+0000","last_unstale":"2024-03-30T14:00:32.649741+0000","last_undegraded":"2024-03-30T14:00:32.649741+0000","last_fullsized":"2024-03-30T14:00:32.649741+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:22.722033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:00:22.722033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:35.443776+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"364'19859","reported_seq":20058,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:38.940445+0000","last_change":"2024-03-30T14:00:10.397438+0000","last_active":"2024-03-30T14:00:38.940445+0000","last_peered":"2024-03-30T14:00:38.940445+0000","last_clean":"2024-03-30T14:00:38.940445+0000","last_became_active":"2024-03-30T13:56:55.953088+0000","last_became_peered":"2024-03-30T13:56:55.953088+0000","last_unstale":"2024-03-30T14:00:38.940445+0000","last_undegraded":"2024-03-30T14:00:38.940445+0000","last_fullsized":"2024-03-30T14:00:38.940445+0000","mapping_epoch":289,"log_start":"346'15300","ondisk_log_start":"346'15300","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'17271","last_scrub_stamp":"2024-03-30T14:00:10.397147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T14:00:10.397147+0000","objects_scrubbed":2958,"log_size":4559,"log_dups_size":0,"ondisk_log_size":4559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:19.121732+0000","scrub_duration":902,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22020120,"num_objects":337,"num_object_clones":0,"num_object_copies":1011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":19859,"num_write_kb":646209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":655,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.656075+0000","last_change":"2024-03-30T14:00:22.478488+0000","last_active":"2024-03-30T14:00:32.656075+0000","last_peered":"2024-03-30T14:00:32.656075+0000","last_clean":"2024-03-30T14:00:32.656075+0000","last_became_active":"2024-03-30T13:52:36.483298+0000","last_became_peered":"2024-03-30T13:52:36.483298+0000","last_unstale":"2024-03-30T14:00:32.656075+0000","last_undegraded":"2024-03-30T14:00:32.656075+0000","last_fullsized":"2024-03-30T14:00:32.656075+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:00:22.478281+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T13:57:14.131256+0000","last_clean_scrub_stamp":"2024-03-30T14:00:22.478281+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:39.743018+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":640,"reported_epoch":365,"state":"active+undersized","last_fresh":"2024-03-30T14:00:40.814349+0000","last_change":"2024-03-30T13:50:08.512648+0000","last_active":"2024-03-30T14:00:40.814349+0000","last_peered":"2024-03-30T14:00:40.814349+0000","last_clean":"2024-03-30T13:50:05.489180+0000","last_became_active":"2024-03-30T13:50:08.512648+0000","last_became_peered":"2024-03-30T13:50:08.512648+0000","last_unstale":"2024-03-30T14:00:40.814349+0000","last_undegraded":"2024-03-30T14:00:40.814349+0000","last_fullsized":"2024-03-30T13:50:08.505931+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:21.527712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:21.527712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:44.085104+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":686,"reported_epoch":365,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:40.815438+0000","last_change":"2024-03-30T14:00:04.161911+0000","last_active":"2024-03-30T14:00:40.815438+0000","last_peered":"2024-03-30T14:00:40.815438+0000","last_clean":"2024-03-30T14:00:40.815438+0000","last_became_active":"2024-03-30T13:58:48.177702+0000","last_became_peered":"2024-03-30T13:58:48.177702+0000","last_unstale":"2024-03-30T14:00:40.815438+0000","last_undegraded":"2024-03-30T14:00:40.815438+0000","last_fullsized":"2024-03-30T14:00:40.815438+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:04.161648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:45.975163+0000","last_clean_scrub_stamp":"2024-03-30T14:00:04.161648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:31.222566+0000","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5,0],"avail_no_missing":["1","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"216'29","reported_seq":763,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.808989+0000","last_change":"2024-03-30T13:58:48.219313+0000","last_active":"2024-03-30T14:00:40.808989+0000","last_peered":"2024-03-30T14:00:40.808989+0000","last_clean":"2024-03-30T14:00:40.808989+0000","last_became_active":"2024-03-30T13:58:48.183735+0000","last_became_peered":"2024-03-30T13:58:48.183735+0000","last_unstale":"2024-03-30T14:00:40.808989+0000","last_undegraded":"2024-03-30T14:00:40.808989+0000","last_fullsized":"2024-03-30T14:00:40.808989+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'29","last_scrub_stamp":"2024-03-30T13:57:59.935365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:59.935365+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":688,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.656475+0000","last_change":"2024-03-30T14:00:24.340982+0000","last_active":"2024-03-30T14:00:32.656475+0000","last_peered":"2024-03-30T14:00:32.656475+0000","last_clean":"2024-03-30T14:00:32.656475+0000","last_became_active":"2024-03-30T13:58:47.204999+0000","last_became_peered":"2024-03-30T13:58:47.204999+0000","last_unstale":"2024-03-30T14:00:32.656475+0000","last_undegraded":"2024-03-30T14:00:32.656475+0000","last_fullsized":"2024-03-30T14:00:32.656475+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:24.340873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:52.887014+0000","last_clean_scrub_stamp":"2024-03-30T14:00:24.340873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:49.796664+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"364'19897","reported_seq":20090,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:38.831353+0000","last_change":"2024-03-30T14:00:15.906502+0000","last_active":"2024-03-30T14:00:38.831353+0000","last_peered":"2024-03-30T14:00:38.831353+0000","last_clean":"2024-03-30T14:00:38.831353+0000","last_became_active":"2024-03-30T13:56:55.948591+0000","last_became_peered":"2024-03-30T13:56:55.948591+0000","last_unstale":"2024-03-30T14:00:38.831353+0000","last_undegraded":"2024-03-30T14:00:38.831353+0000","last_fullsized":"2024-03-30T14:00:38.831353+0000","mapping_epoch":289,"log_start":"346'16100","ondisk_log_start":"346'16100","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"359'17842","last_scrub_stamp":"2024-03-30T14:00:15.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T14:00:15.906327+0000","objects_scrubbed":2476,"log_size":3797,"log_dups_size":0,"ondisk_log_size":3797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:33.009790+0000","scrub_duration":753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25493504,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19897,"num_write_kb":649152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":661,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.658871+0000","last_change":"2024-03-30T13:58:50.216157+0000","last_active":"2024-03-30T14:00:32.658871+0000","last_peered":"2024-03-30T14:00:32.658871+0000","last_clean":"2024-03-30T14:00:32.658871+0000","last_became_active":"2024-03-30T13:58:50.215325+0000","last_became_peered":"2024-03-30T13:58:50.215325+0000","last_unstale":"2024-03-30T14:00:32.658871+0000","last_undegraded":"2024-03-30T14:00:32.658871+0000","last_fullsized":"2024-03-30T14:00:32.658871+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:57:18.133345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:57:18.133345+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:34.214918+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":639,"reported_epoch":365,"state":"active+undersized","last_fresh":"2024-03-30T14:00:40.808004+0000","last_change":"2024-03-30T13:50:08.510025+0000","last_active":"2024-03-30T14:00:40.808004+0000","last_peered":"2024-03-30T14:00:40.808004+0000","last_clean":"2024-03-30T13:50:05.498930+0000","last_became_active":"2024-03-30T13:50:08.510025+0000","last_became_peered":"2024-03-30T13:50:08.510025+0000","last_unstale":"2024-03-30T14:00:40.808004+0000","last_undegraded":"2024-03-30T14:00:40.808004+0000","last_fullsized":"2024-03-30T13:50:08.502478+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:28.946303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:41.142941+0000","last_clean_scrub_stamp":"2024-03-30T13:49:28.946303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":451,"reported_epoch":364,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:32.649183+0000","last_change":"2024-03-30T14:00:21.725005+0000","last_active":"2024-03-30T14:00:32.649183+0000","last_peered":"2024-03-30T14:00:32.649183+0000","last_clean":"2024-03-30T14:00:32.649183+0000","last_became_active":"2024-03-30T13:58:48.191799+0000","last_became_peered":"2024-03-30T13:58:48.191799+0000","last_unstale":"2024-03-30T14:00:32.649183+0000","last_undegraded":"2024-03-30T14:00:32.649183+0000","last_fullsized":"2024-03-30T14:00:32.649183+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:21.724810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:55.458779+0000","last_clean_scrub_stamp":"2024-03-30T14:00:21.724810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:25.317439+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3,6],"avail_no_missing":["5","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"217'25","reported_seq":754,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.818787+0000","last_change":"2024-03-30T14:00:23.161109+0000","last_active":"2024-03-30T14:00:40.818787+0000","last_peered":"2024-03-30T14:00:40.818787+0000","last_clean":"2024-03-30T14:00:40.818787+0000","last_became_active":"2024-03-30T13:58:47.186574+0000","last_became_peered":"2024-03-30T13:58:47.186574+0000","last_unstale":"2024-03-30T14:00:40.818787+0000","last_undegraded":"2024-03-30T14:00:40.818787+0000","last_fullsized":"2024-03-30T14:00:40.818787+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'25","last_scrub_stamp":"2024-03-30T14:00:23.160941+0000","last_deep_scrub":"217'25","last_deep_scrub_stamp":"2024-03-30T14:00:23.160941+0000","last_clean_scrub_stamp":"2024-03-30T14:00:23.160941+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:40.203499+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"326'11","reported_seq":911,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.809013+0000","last_change":"2024-03-30T13:58:47.220214+0000","last_active":"2024-03-30T14:00:40.809013+0000","last_peered":"2024-03-30T14:00:40.809013+0000","last_clean":"2024-03-30T14:00:40.809013+0000","last_became_active":"2024-03-30T13:58:47.199445+0000","last_became_peered":"2024-03-30T13:58:47.199445+0000","last_unstale":"2024-03-30T14:00:40.809013+0000","last_undegraded":"2024-03-30T14:00:40.809013+0000","last_fullsized":"2024-03-30T14:00:40.809013+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'10","last_scrub_stamp":"2024-03-30T13:58:13.880792+0000","last_deep_scrub":"218'9","last_deep_scrub_stamp":"2024-03-30T13:52:57.873554+0000","last_clean_scrub_stamp":"2024-03-30T13:58:13.880792+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"364'19627","reported_seq":19785,"reported_epoch":364,"state":"active+undersized+degraded","last_fresh":"2024-03-30T14:00:38.932145+0000","last_change":"2024-03-30T13:57:08.373767+0000","last_active":"2024-03-30T14:00:38.932145+0000","last_peered":"2024-03-30T14:00:38.932145+0000","last_clean":"2024-03-30T13:56:54.930191+0000","last_became_active":"2024-03-30T13:56:55.952893+0000","last_became_peered":"2024-03-30T13:56:55.952893+0000","last_unstale":"2024-03-30T14:00:38.932145+0000","last_undegraded":"2024-03-30T13:57:08.373532+0000","last_fullsized":"2024-03-30T13:56:55.947530+0000","mapping_epoch":289,"log_start":"346'15000","ondisk_log_start":"346'15000","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:41.289852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22609920,"num_objects":345,"num_object_clones":0,"num_object_copies":1035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19627,"num_write_kb":639104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":345}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":686,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.649849+0000","last_change":"2024-03-30T14:00:23.341383+0000","last_active":"2024-03-30T14:00:42.649849+0000","last_peered":"2024-03-30T14:00:42.649849+0000","last_clean":"2024-03-30T14:00:42.649849+0000","last_became_active":"2024-03-30T13:58:50.203408+0000","last_became_peered":"2024-03-30T13:58:50.203408+0000","last_unstale":"2024-03-30T14:00:42.649849+0000","last_undegraded":"2024-03-30T14:00:42.649849+0000","last_fullsized":"2024-03-30T14:00:42.649849+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:00:23.341247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:00:23.341247+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:33.927922+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":525,"reported_epoch":364,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:32.656355+0000","last_change":"2024-03-30T13:58:03.303778+0000","last_active":"2024-03-30T14:00:32.656355+0000","last_peered":"2024-03-30T14:00:32.656355+0000","last_clean":"2024-03-30T14:00:32.656355+0000","last_became_active":"2024-03-30T13:53:36.852004+0000","last_became_peered":"2024-03-30T13:53:36.852004+0000","last_unstale":"2024-03-30T14:00:32.656355+0000","last_undegraded":"2024-03-30T14:00:32.656355+0000","last_fullsized":"2024-03-30T14:00:32.656355+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:03.303563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:31.900908+0000","last_clean_scrub_stamp":"2024-03-30T13:58:03.303563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:34.215007+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1,7],"avail_no_missing":["6","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":673,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.809224+0000","last_change":"2024-03-30T13:58:12.869168+0000","last_active":"2024-03-30T14:00:40.809224+0000","last_peered":"2024-03-30T14:00:40.809224+0000","last_clean":"2024-03-30T14:00:40.809224+0000","last_became_active":"2024-03-30T13:42:45.309886+0000","last_became_peered":"2024-03-30T13:42:45.309886+0000","last_unstale":"2024-03-30T14:00:40.809224+0000","last_undegraded":"2024-03-30T14:00:40.809224+0000","last_fullsized":"2024-03-30T14:00:40.809224+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T13:58:12.869021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:12.869021+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"217'35","reported_seq":761,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.645704+0000","last_change":"2024-03-30T14:00:12.151979+0000","last_active":"2024-03-30T14:00:32.645704+0000","last_peered":"2024-03-30T14:00:32.645704+0000","last_clean":"2024-03-30T14:00:32.645704+0000","last_became_active":"2024-03-30T13:58:47.192909+0000","last_became_peered":"2024-03-30T13:58:47.192909+0000","last_unstale":"2024-03-30T14:00:32.645704+0000","last_undegraded":"2024-03-30T14:00:32.645704+0000","last_fullsized":"2024-03-30T14:00:32.645704+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'35","last_scrub_stamp":"2024-03-30T14:00:12.151846+0000","last_deep_scrub":"91'21","last_deep_scrub_stamp":"2024-03-30T13:44:05.917064+0000","last_clean_scrub_stamp":"2024-03-30T14:00:12.151846+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:14.356638+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":876,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.649523+0000","last_change":"2024-03-30T13:57:22.046830+0000","last_active":"2024-03-30T14:00:42.649523+0000","last_peered":"2024-03-30T14:00:42.649523+0000","last_clean":"2024-03-30T14:00:42.649523+0000","last_became_active":"2024-03-30T13:42:49.321423+0000","last_became_peered":"2024-03-30T13:42:49.321423+0000","last_unstale":"2024-03-30T14:00:42.649523+0000","last_undegraded":"2024-03-30T14:00:42.649523+0000","last_fullsized":"2024-03-30T14:00:42.649523+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:57:22.046689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:22.046689+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337530+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"364'19911","reported_seq":20069,"reported_epoch":364,"state":"active+undersized+degraded","last_fresh":"2024-03-30T14:00:38.854870+0000","last_change":"2024-03-30T13:57:08.372029+0000","last_active":"2024-03-30T14:00:38.854870+0000","last_peered":"2024-03-30T14:00:38.854870+0000","last_clean":"2024-03-30T13:56:54.930191+0000","last_became_active":"2024-03-30T13:56:55.949377+0000","last_became_peered":"2024-03-30T13:56:55.949377+0000","last_unstale":"2024-03-30T14:00:38.854870+0000","last_undegraded":"2024-03-30T13:57:08.371728+0000","last_fullsized":"2024-03-30T13:56:55.941808+0000","mapping_epoch":289,"log_start":"346'16100","ondisk_log_start":"346'16100","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":3811,"log_dups_size":0,"ondisk_log_size":3811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23789568,"num_objects":363,"num_object_clones":0,"num_object_copies":1089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19911,"num_write_kb":648768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":363}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":642,"reported_epoch":364,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:32.646738+0000","last_change":"2024-03-30T13:59:58.326706+0000","last_active":"2024-03-30T14:00:32.646738+0000","last_peered":"2024-03-30T14:00:32.646738+0000","last_clean":"2024-03-30T14:00:32.646738+0000","last_became_active":"2024-03-30T13:52:36.490909+0000","last_became_peered":"2024-03-30T13:52:36.490909+0000","last_unstale":"2024-03-30T14:00:32.646738+0000","last_undegraded":"2024-03-30T14:00:32.646738+0000","last_fullsized":"2024-03-30T14:00:32.646738+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:59:58.326371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:59:58.326371+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:21.262800+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,0],"avail_no_missing":["4","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":631,"reported_epoch":364,"state":"active+undersized","last_fresh":"2024-03-30T14:00:32.645297+0000","last_change":"2024-03-30T13:50:08.504411+0000","last_active":"2024-03-30T14:00:32.645297+0000","last_peered":"2024-03-30T14:00:32.645297+0000","last_clean":"2024-03-30T13:50:05.489944+0000","last_became_active":"2024-03-30T13:50:08.504411+0000","last_became_peered":"2024-03-30T13:50:08.504411+0000","last_unstale":"2024-03-30T14:00:32.645297+0000","last_undegraded":"2024-03-30T14:00:32.645297+0000","last_fullsized":"2024-03-30T13:50:08.501823+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:10.216387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:10.216387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:37.015623+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":680,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.654048+0000","last_change":"2024-03-30T14:00:18.364366+0000","last_active":"2024-03-30T14:00:42.654048+0000","last_peered":"2024-03-30T14:00:42.654048+0000","last_clean":"2024-03-30T14:00:42.654048+0000","last_became_active":"2024-03-30T13:58:47.194093+0000","last_became_peered":"2024-03-30T13:58:47.194093+0000","last_unstale":"2024-03-30T14:00:42.654048+0000","last_undegraded":"2024-03-30T14:00:42.654048+0000","last_fullsized":"2024-03-30T14:00:42.654048+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:18.364248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:53:01.031216+0000","last_clean_scrub_stamp":"2024-03-30T14:00:18.364248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:36.622196+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"216'35","reported_seq":735,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.649436+0000","last_change":"2024-03-30T13:53:36.081652+0000","last_active":"2024-03-30T14:00:42.649436+0000","last_peered":"2024-03-30T14:00:42.649436+0000","last_clean":"2024-03-30T14:00:42.649436+0000","last_became_active":"2024-03-30T13:53:35.888652+0000","last_became_peered":"2024-03-30T13:53:35.888652+0000","last_unstale":"2024-03-30T14:00:42.649436+0000","last_undegraded":"2024-03-30T14:00:42.649436+0000","last_fullsized":"2024-03-30T14:00:42.649436+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'29","last_scrub_stamp":"2024-03-30T13:52:52.973602+0000","last_deep_scrub":"216'29","last_deep_scrub_stamp":"2024-03-30T13:52:52.973602+0000","last_clean_scrub_stamp":"2024-03-30T13:52:52.973602+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337288+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":654,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.645288+0000","last_change":"2024-03-30T13:57:10.660140+0000","last_active":"2024-03-30T14:00:32.645288+0000","last_peered":"2024-03-30T14:00:32.645288+0000","last_clean":"2024-03-30T14:00:32.645288+0000","last_became_active":"2024-03-30T13:53:35.863124+0000","last_became_peered":"2024-03-30T13:53:35.863124+0000","last_unstale":"2024-03-30T14:00:32.645288+0000","last_undegraded":"2024-03-30T14:00:32.645288+0000","last_fullsized":"2024-03-30T14:00:32.645288+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:10.660011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:10.660011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:33.023693+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"364'19601","reported_seq":19792,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:38.838245+0000","last_change":"2024-03-30T14:00:17.879960+0000","last_active":"2024-03-30T14:00:38.838245+0000","last_peered":"2024-03-30T14:00:38.838245+0000","last_clean":"2024-03-30T14:00:38.838245+0000","last_became_active":"2024-03-30T13:56:55.949813+0000","last_became_peered":"2024-03-30T13:56:55.949813+0000","last_unstale":"2024-03-30T14:00:38.838245+0000","last_undegraded":"2024-03-30T14:00:38.838245+0000","last_fullsized":"2024-03-30T14:00:38.838245+0000","mapping_epoch":289,"log_start":"346'15800","ondisk_log_start":"346'15800","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"360'17699","last_scrub_stamp":"2024-03-30T14:00:17.879595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T14:00:17.879595+0000","objects_scrubbed":2294,"log_size":3801,"log_dups_size":0,"ondisk_log_size":3801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:24.689975+0000","scrub_duration":712,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23396352,"num_objects":357,"num_object_clones":0,"num_object_copies":1071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19601,"num_write_kb":638656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":429,"reported_epoch":364,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:32.651698+0000","last_change":"2024-03-30T13:58:48.185795+0000","last_active":"2024-03-30T14:00:32.651698+0000","last_peered":"2024-03-30T14:00:32.651698+0000","last_clean":"2024-03-30T14:00:32.651698+0000","last_became_active":"2024-03-30T13:58:48.184258+0000","last_became_peered":"2024-03-30T13:58:48.184258+0000","last_unstale":"2024-03-30T14:00:32.651698+0000","last_undegraded":"2024-03-30T14:00:32.651698+0000","last_fullsized":"2024-03-30T14:00:32.651698+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:58:05.194482+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T13:56:08.385059+0000","last_clean_scrub_stamp":"2024-03-30T13:58:05.194482+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6,4],"avail_no_missing":["5","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":436,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.649155+0000","last_change":"2024-03-30T13:58:48.184750+0000","last_active":"2024-03-30T14:00:32.649155+0000","last_peered":"2024-03-30T14:00:32.649155+0000","last_clean":"2024-03-30T14:00:32.649155+0000","last_became_active":"2024-03-30T13:58:48.183934+0000","last_became_peered":"2024-03-30T13:58:48.183934+0000","last_unstale":"2024-03-30T14:00:32.649155+0000","last_undegraded":"2024-03-30T14:00:32.649155+0000","last_fullsized":"2024-03-30T14:00:32.649155+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:23.126245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:23.126245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"327'13","reported_seq":893,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.646562+0000","last_change":"2024-03-30T13:58:48.205934+0000","last_active":"2024-03-30T14:00:32.646562+0000","last_peered":"2024-03-30T14:00:32.646562+0000","last_clean":"2024-03-30T14:00:32.646562+0000","last_became_active":"2024-03-30T13:58:48.204824+0000","last_became_peered":"2024-03-30T13:58:48.204824+0000","last_unstale":"2024-03-30T14:00:32.646562+0000","last_undegraded":"2024-03-30T14:00:32.646562+0000","last_fullsized":"2024-03-30T14:00:32.646562+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'11","last_scrub_stamp":"2024-03-30T13:58:06.437298+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-30T13:51:15.146255+0000","last_clean_scrub_stamp":"2024-03-30T13:58:06.437298+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":666,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.649356+0000","last_change":"2024-03-30T14:00:11.446913+0000","last_active":"2024-03-30T14:00:42.649356+0000","last_peered":"2024-03-30T14:00:42.649356+0000","last_clean":"2024-03-30T14:00:42.649356+0000","last_became_active":"2024-03-30T13:53:35.886750+0000","last_became_peered":"2024-03-30T13:53:35.886750+0000","last_unstale":"2024-03-30T14:00:42.649356+0000","last_undegraded":"2024-03-30T14:00:42.649356+0000","last_fullsized":"2024-03-30T14:00:42.649356+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:11.446804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:00:11.446804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:16.331270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"217'34","reported_seq":726,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.816016+0000","last_change":"2024-03-30T14:00:02.158918+0000","last_active":"2024-03-30T14:00:40.816016+0000","last_peered":"2024-03-30T14:00:40.816016+0000","last_clean":"2024-03-30T14:00:40.816016+0000","last_became_active":"2024-03-30T13:42:47.324350+0000","last_became_peered":"2024-03-30T13:42:47.324350+0000","last_unstale":"2024-03-30T14:00:40.816016+0000","last_undegraded":"2024-03-30T14:00:40.816016+0000","last_fullsized":"2024-03-30T14:00:40.816016+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'34","last_scrub_stamp":"2024-03-30T14:00:02.158736+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:28.831527+0000","last_clean_scrub_stamp":"2024-03-30T14:00:02.158736+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:03.039588+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"364'19978","reported_seq":20183,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:38.943161+0000","last_change":"2024-03-30T14:00:03.544262+0000","last_active":"2024-03-30T14:00:38.943161+0000","last_peered":"2024-03-30T14:00:38.943161+0000","last_clean":"2024-03-30T14:00:38.943161+0000","last_became_active":"2024-03-30T13:56:55.954690+0000","last_became_peered":"2024-03-30T13:56:55.954690+0000","last_unstale":"2024-03-30T14:00:38.943161+0000","last_undegraded":"2024-03-30T14:00:38.943161+0000","last_fullsized":"2024-03-30T14:00:38.943161+0000","mapping_epoch":289,"log_start":"346'15400","ondisk_log_start":"346'15400","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'16769","last_scrub_stamp":"2024-03-30T14:00:03.543777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T14:00:03.543777+0000","objects_scrubbed":3584,"log_size":4578,"log_dups_size":0,"ondisk_log_size":4578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:08.460175+0000","scrub_duration":1017,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22413312,"num_objects":342,"num_object_clones":0,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19978,"num_write_kb":650240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":665,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.650036+0000","last_change":"2024-03-30T13:58:47.194433+0000","last_active":"2024-03-30T14:00:42.650036+0000","last_peered":"2024-03-30T14:00:42.650036+0000","last_clean":"2024-03-30T14:00:42.650036+0000","last_became_active":"2024-03-30T13:58:47.193821+0000","last_became_peered":"2024-03-30T13:58:47.193821+0000","last_unstale":"2024-03-30T14:00:42.650036+0000","last_undegraded":"2024-03-30T14:00:42.650036+0000","last_fullsized":"2024-03-30T14:00:42.650036+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:09.195697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:21.081938+0000","last_clean_scrub_stamp":"2024-03-30T13:58:09.195697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337747+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":440,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.650025+0000","last_change":"2024-03-30T13:58:48.216887+0000","last_active":"2024-03-30T14:00:32.650025+0000","last_peered":"2024-03-30T14:00:32.650025+0000","last_clean":"2024-03-30T14:00:32.650025+0000","last_became_active":"2024-03-30T13:58:48.216542+0000","last_became_peered":"2024-03-30T13:58:48.216542+0000","last_unstale":"2024-03-30T14:00:32.650025+0000","last_undegraded":"2024-03-30T14:00:32.650025+0000","last_fullsized":"2024-03-30T14:00:32.650025+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:15.580974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:52.603772+0000","last_clean_scrub_stamp":"2024-03-30T13:58:15.580974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":664,"reported_epoch":364,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:32.645383+0000","last_change":"2024-03-30T14:00:21.110580+0000","last_active":"2024-03-30T14:00:32.645383+0000","last_peered":"2024-03-30T14:00:32.645383+0000","last_clean":"2024-03-30T14:00:32.645383+0000","last_became_active":"2024-03-30T13:53:36.852500+0000","last_became_peered":"2024-03-30T13:53:36.852500+0000","last_unstale":"2024-03-30T14:00:32.645383+0000","last_undegraded":"2024-03-30T14:00:32.645383+0000","last_fullsized":"2024-03-30T14:00:32.645383+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:21.110376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:00:21.110376+0000","last_clean_scrub_stamp":"2024-03-30T14:00:21.110376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:29.837075+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,7],"avail_no_missing":["4","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"217'19","reported_seq":698,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.646434+0000","last_change":"2024-03-30T13:58:53.440615+0000","last_active":"2024-03-30T14:00:32.646434+0000","last_peered":"2024-03-30T14:00:32.646434+0000","last_clean":"2024-03-30T14:00:32.646434+0000","last_became_active":"2024-03-30T13:42:47.315667+0000","last_became_peered":"2024-03-30T13:42:47.315667+0000","last_unstale":"2024-03-30T14:00:32.646434+0000","last_undegraded":"2024-03-30T14:00:32.646434+0000","last_fullsized":"2024-03-30T14:00:32.646434+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'19","last_scrub_stamp":"2024-03-30T13:58:53.440486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:58:53.440486+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:33.023941+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"364'20143","reported_seq":30229,"reported_epoch":364,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:38.741302+0000","last_change":"2024-03-30T13:59:57.664849+0000","last_active":"2024-03-30T14:00:38.741302+0000","last_peered":"2024-03-30T14:00:38.741302+0000","last_clean":"2024-03-30T14:00:38.741302+0000","last_became_active":"2024-03-30T13:59:57.632761+0000","last_became_peered":"2024-03-30T13:59:57.632761+0000","last_unstale":"2024-03-30T14:00:38.741302+0000","last_undegraded":"2024-03-30T14:00:38.741302+0000","last_fullsized":"2024-03-30T14:00:38.741302+0000","mapping_epoch":349,"log_start":"346'15300","ondisk_log_start":"346'15300","created":289,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:56:54.930191+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:41.730828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25231360,"num_objects":385,"num_object_clones":0,"num_object_copies":1155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":385,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20143,"num_write_kb":656896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4955,"num_bytes_recovered":324730880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3,4],"avail_no_missing":["5","3","4"],"object_location_counts":[{"shards":"3,4,5","objects":385}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":437,"reported_epoch":364,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:32.648129+0000","last_change":"2024-03-30T14:00:15.789762+0000","last_active":"2024-03-30T14:00:32.648129+0000","last_peered":"2024-03-30T14:00:32.648129+0000","last_clean":"2024-03-30T14:00:32.648129+0000","last_became_active":"2024-03-30T13:58:48.191314+0000","last_became_peered":"2024-03-30T13:58:48.191314+0000","last_unstale":"2024-03-30T14:00:32.648129+0000","last_undegraded":"2024-03-30T14:00:32.648129+0000","last_fullsized":"2024-03-30T14:00:32.648129+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:00:15.789401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:00:15.789401+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:21.678198+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3,6],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":423,"reported_epoch":366,"state":"active+undersized","last_fresh":"2024-03-30T14:00:42.653854+0000","last_change":"2024-03-30T13:50:08.511676+0000","last_active":"2024-03-30T14:00:42.653854+0000","last_peered":"2024-03-30T14:00:42.653854+0000","last_clean":"2024-03-30T13:50:05.504989+0000","last_became_active":"2024-03-30T13:50:08.511676+0000","last_became_peered":"2024-03-30T13:50:08.511676+0000","last_unstale":"2024-03-30T14:00:42.653854+0000","last_undegraded":"2024-03-30T14:00:42.653854+0000","last_fullsized":"2024-03-30T13:50:08.504157+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:09.357378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:09.357378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":680,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:40.815874+0000","last_change":"2024-03-30T14:00:24.179462+0000","last_active":"2024-03-30T14:00:40.815874+0000","last_peered":"2024-03-30T14:00:40.815874+0000","last_clean":"2024-03-30T14:00:40.815874+0000","last_became_active":"2024-03-30T13:42:49.321360+0000","last_became_peered":"2024-03-30T13:42:49.321360+0000","last_unstale":"2024-03-30T14:00:40.815874+0000","last_undegraded":"2024-03-30T14:00:40.815874+0000","last_fullsized":"2024-03-30T14:00:40.815874+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:24.179354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:59:06.350813+0000","last_clean_scrub_stamp":"2024-03-30T14:00:24.179354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:51.395112+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":677,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.655742+0000","last_change":"2024-03-30T13:57:12.187448+0000","last_active":"2024-03-30T14:00:32.655742+0000","last_peered":"2024-03-30T14:00:32.655742+0000","last_clean":"2024-03-30T14:00:32.655742+0000","last_became_active":"2024-03-30T13:42:45.315251+0000","last_became_peered":"2024-03-30T13:42:45.315251+0000","last_unstale":"2024-03-30T14:00:32.655742+0000","last_undegraded":"2024-03-30T14:00:32.655742+0000","last_fullsized":"2024-03-30T14:00:32.655742+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:12.187291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:57:12.187291+0000","last_clean_scrub_stamp":"2024-03-30T13:57:12.187291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:34.214888+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"217'31","reported_seq":687,"reported_epoch":364,"state":"active+undersized+degraded","last_fresh":"2024-03-30T14:00:32.647017+0000","last_change":"2024-03-30T13:50:08.508030+0000","last_active":"2024-03-30T14:00:32.647017+0000","last_peered":"2024-03-30T14:00:32.647017+0000","last_clean":"2024-03-30T13:50:05.491159+0000","last_became_active":"2024-03-30T13:50:08.508030+0000","last_became_peered":"2024-03-30T13:50:08.508030+0000","last_unstale":"2024-03-30T14:00:32.647017+0000","last_undegraded":"2024-03-30T13:50:08.502958+0000","last_fullsized":"2024-03-30T13:50:08.502815+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-03-30T13:48:27.096261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:27.096261+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:37.015655+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"365'20324","reported_seq":20586,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-30T14:00:41.056485+0000","last_change":"2024-03-30T13:58:55.948850+0000","last_active":"2024-03-30T14:00:41.056485+0000","last_peered":"2024-03-30T14:00:41.056485+0000","last_clean":"2024-03-30T14:00:41.056485+0000","last_became_active":"2024-03-30T13:56:55.953793+0000","last_became_peered":"2024-03-30T13:56:55.953793+0000","last_unstale":"2024-03-30T14:00:41.056485+0000","last_undegraded":"2024-03-30T14:00:41.056485+0000","last_fullsized":"2024-03-30T14:00:41.056485+0000","mapping_epoch":289,"log_start":"350'16504","ondisk_log_start":"350'16504","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'11631","last_scrub_stamp":"2024-03-30T13:58:55.948602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:54.930191+0000","last_clean_scrub_stamp":"2024-03-30T13:58:55.948602+0000","objects_scrubbed":8948,"log_size":3820,"log_dups_size":0,"ondisk_log_size":3820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:44.085587+0000","scrub_duration":2801,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9437184,"num_objects":144,"num_object_clones":0,"num_object_copies":432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20324,"num_write_kb":654976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":900,"reported_epoch":366,"state":"active+clean+remapped","last_fresh":"2024-03-30T14:00:42.650257+0000","last_change":"2024-03-30T13:58:50.203750+0000","last_active":"2024-03-30T14:00:42.650257+0000","last_peered":"2024-03-30T14:00:42.650257+0000","last_clean":"2024-03-30T14:00:42.650257+0000","last_became_active":"2024-03-30T13:58:50.203221+0000","last_became_peered":"2024-03-30T13:58:50.203221+0000","last_unstale":"2024-03-30T14:00:42.650257+0000","last_undegraded":"2024-03-30T14:00:42.650257+0000","last_fullsized":"2024-03-30T14:00:42.650257+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T13:56:55.928341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T13:56:55.928341+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5,6],"avail_no_missing":["3","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":630,"reported_epoch":364,"state":"active+undersized","last_fresh":"2024-03-30T14:00:32.646542+0000","last_change":"2024-03-30T13:50:08.511010+0000","last_active":"2024-03-30T14:00:32.646542+0000","last_peered":"2024-03-30T14:00:32.646542+0000","last_clean":"2024-03-30T13:50:05.490535+0000","last_became_active":"2024-03-30T13:50:08.511010+0000","last_became_peered":"2024-03-30T13:50:08.511010+0000","last_unstale":"2024-03-30T14:00:32.646542+0000","last_undegraded":"2024-03-30T14:00:32.646542+0000","last_fullsized":"2024-03-30T13:50:08.503118+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:06.154805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:06.154805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:37.015669+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"327'14","reported_seq":112,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.648874+0000","last_change":"2024-03-30T13:58:48.236949+0000","last_active":"2024-03-30T14:00:32.648874+0000","last_peered":"2024-03-30T14:00:32.648874+0000","last_clean":"2024-03-30T14:00:32.648874+0000","last_became_active":"2024-03-30T13:58:48.191842+0000","last_became_peered":"2024-03-30T13:58:48.191842+0000","last_unstale":"2024-03-30T14:00:32.648874+0000","last_undegraded":"2024-03-30T14:00:32.648874+0000","last_fullsized":"2024-03-30T14:00:32.648874+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'13","last_scrub_stamp":"2024-03-30T13:57:08.673207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:08.673207+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":660,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-30T14:00:42.649358+0000","last_change":"2024-03-30T13:57:05.955711+0000","last_active":"2024-03-30T14:00:42.649358+0000","last_peered":"2024-03-30T14:00:42.649358+0000","last_clean":"2024-03-30T14:00:42.649358+0000","last_became_active":"2024-03-30T13:42:45.309265+0000","last_became_peered":"2024-03-30T13:42:45.309265+0000","last_unstale":"2024-03-30T14:00:42.649358+0000","last_undegraded":"2024-03-30T14:00:42.649358+0000","last_fullsized":"2024-03-30T14:00:42.649358+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:05.955630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:57:05.955630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:45.337382+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"216'20","reported_seq":689,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-03-30T14:00:32.646674+0000","last_change":"2024-03-30T13:58:22.508345+0000","last_active":"2024-03-30T14:00:32.646674+0000","last_peered":"2024-03-30T14:00:32.646674+0000","last_clean":"2024-03-30T14:00:32.646674+0000","last_became_active":"2024-03-30T13:42:47.313026+0000","last_became_peered":"2024-03-30T13:42:47.313026+0000","last_unstale":"2024-03-30T14:00:32.646674+0000","last_undegraded":"2024-03-30T14:00:32.646674+0000","last_fullsized":"2024-03-30T14:00:32.646674+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'20","last_scrub_stamp":"2024-03-30T13:58:22.508219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:58:22.508219+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:00:33.023912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":327221272,"num_objects":4994,"num_object_clones":0,"num_object_copies":14982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":899,"num_objects_misplaced":931,"num_objects_unfound":0,"num_objects_dirty":4994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":319456,"num_write_kb":10382337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37220,"num_bytes_recovered":2439118848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":898183168,"data_stored":898170952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69552,"ondisk_log_size":69552,"up":42,"acting":45,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":430,"num_read_kb":425,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":27,"acting":33,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":5246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":87,"acting":93,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":3344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":11216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":743,"ondisk_log_size":743,"up":87,"acting":90,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":85,"acting":93,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":82,"acting":91,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":249108103389,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397728,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":910236,"kb_used_data":189076,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":92834916,"statfs":{"total":95995035648,"available":95062953984,"internally_reserved":0,"allocated":193613824,"data_stored":192005493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":738458054},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575098,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":993768,"kb_used_data":240672,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":92751384,"statfs":{"total":95995035648,"available":94977417216,"internally_reserved":0,"allocated":246448128,"data_stored":244763232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":771160517},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738623,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":962988,"kb_used_data":133540,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":92782164,"statfs":{"total":95995035648,"available":95008935936,"internally_reserved":0,"allocated":136744960,"data_stored":135120526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":849344965},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411446,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2213288,"kb_used_data":2015520,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":91531864,"statfs":{"total":95995035648,"available":93728628736,"internally_reserved":0,"allocated":2063892480,"data_stored":2063509110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182641,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888224,"kb_used_data":4624,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":92856928,"statfs":{"total":95995035648,"available":95085494272,"internally_reserved":0,"allocated":4734976,"data_stored":3084237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":904788423},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953706,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1044728,"kb_used_data":315248,"kb_used_omap":1,"kb_used_meta":729470,"kb_avail":92700424,"statfs":{"total":95995035648,"available":94925234176,"internally_reserved":0,"allocated":322813952,"data_stored":321176903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":746977733},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692069,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":824292,"kb_used_data":147484,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":92920860,"statfs":{"total":95995035648,"available":95150960640,"internally_reserved":0,"allocated":151023616,"data_stored":149407903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":2888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":116396032,"data_stored":116391960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":218431488,"data_stored":218431488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":260378624,"data_stored":260374552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":145227776,"data_stored":145227776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":157749248,"data_stored":157745176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T14:00:46.535 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-30T14:00:46.535 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool set .mgr pg_num 21 2024-03-30T14:00:46.555 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:00:46.601 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:46.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:46.631 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:46.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:46.732 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:46.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:46.759 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:46.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:46.860 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:46.888 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:46.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:46.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:46.990 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:47.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:47.019 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:47.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:47.101 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:46 smithi043 ceph-mon[29963]: from='client.25616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:00:47.101 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:46 smithi043 ceph-mon[29963]: pgmap v807: 158 pgs: 106 active+clean, 13 active+undersized, 8 active+undersized+degraded, 31 active+clean+remapped; 44 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.45k op/s; 233/2673 objects degraded (8.717%); 195/2673 objects misplaced (7.295%) 2024-03-30T14:00:47.101 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:46 smithi043 ceph-mon[42099]: from='client.25616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:00:47.102 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:46 smithi043 ceph-mon[42099]: pgmap v807: 158 pgs: 106 active+clean, 13 active+undersized, 8 active+undersized+degraded, 31 active+clean+remapped; 44 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.45k op/s; 233/2673 objects degraded (8.717%); 195/2673 objects misplaced (7.295%) 2024-03-30T14:00:47.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:47.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:47.145 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:47.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:47.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:46 smithi113 ceph-mon[34341]: from='client.25616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:00:47.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:46 smithi113 ceph-mon[34341]: pgmap v807: 158 pgs: 106 active+clean, 13 active+undersized, 8 active+undersized+degraded, 31 active+clean+remapped; 44 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.45k op/s; 233/2673 objects degraded (8.717%); 195/2673 objects misplaced (7.295%) 2024-03-30T14:00:47.246 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:47.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:47.275 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:47.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:47.329 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:00:47.333 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:00:47.376 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:47.403 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:47.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:47.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:47.505 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:47.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:47.531 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:47.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:47.632 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:47.660 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:47.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:47.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:47.762 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:47.790 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:47.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:47.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:47.864 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T14:00:47.892 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:47.919 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:47.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:47.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:48.020 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:48.046 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:48.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:48.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:47 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 233/2673 objects degraded (8.717%), 8 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:48.057 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:47 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 233/2673 objects degraded (8.717%), 8 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:48.148 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:48.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:48.173 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:48.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:48.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:47 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 233/2673 objects degraded (8.717%), 8 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:48.274 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:48.299 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:48.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:48.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:48.401 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:48.426 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:48.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:48.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:48.528 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:48.555 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:48.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:48.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:48.656 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:48.680 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:48.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:48.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:48.782 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:48.806 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:48.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:48.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:48.908 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:48.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:48.937 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:48.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:49.039 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:49.067 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:49.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:49.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:49.073 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:48 smithi043 ceph-mon[29963]: pgmap v808: 158 pgs: 106 active+clean, 13 active+undersized, 8 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 852 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:00:49.074 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:48 smithi043 ceph-mon[42099]: pgmap v808: 158 pgs: 106 active+clean, 13 active+undersized, 8 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 852 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:00:49.077 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T14:00:49.077 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T14:00:49.077 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 0 2024-03-30T14:00:49.077 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T14:00:49.117 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:48 smithi113 ceph-mon[34341]: pgmap v808: 158 pgs: 106 active+clean, 13 active+undersized, 8 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 852 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:00:49.169 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:49.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:49.194 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:49.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:49.295 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:49.318 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:49.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:49.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:49.420 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:49.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:49.447 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:49.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:49.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:00:49 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:00:49.548 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:49.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:49.572 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:49.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:49.673 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:49.701 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:49.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:49.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:49.803 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:49.831 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:49.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:49.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:49.933 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:49.958 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:49.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:49.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:50.060 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:50.088 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:50.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:50.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:50.190 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:50.216 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:50.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:50.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:50.318 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:50.346 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:50.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:50.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:50.448 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:50.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:50.476 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:50.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:50.545 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-03-30T14:00:50.550 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:00:50.577 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:50.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:50.607 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:50.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:50.708 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:50.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:50.737 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:50.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:50.838 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:50.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:50.863 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:50.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:50.964 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:50.990 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:50.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:50.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:51.092 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:51.118 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:50 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:51.119 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:50 smithi043 ceph-mon[29963]: pgmap v809: 158 pgs: 106 active+clean, 13 active+undersized, 8 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 763 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:00:51.119 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:50 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:51.119 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:50 smithi043 ceph-mon[42099]: pgmap v809: 158 pgs: 106 active+clean, 13 active+undersized, 8 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 763 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:00:51.120 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:51.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:51.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:51.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:50 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:51.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:50 smithi113 ceph-mon[34341]: pgmap v809: 158 pgs: 106 active+clean, 13 active+undersized, 8 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 763 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:00:51.222 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:51.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:51.250 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:51.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:51.352 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:51.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:51.380 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:51.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:51.433 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:00:51.481 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:51.510 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:51.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:51.612 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:51.640 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:51.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:51.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:51.742 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:51.768 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:51.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:51.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:51.870 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:51.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:51.897 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:51.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:51.998 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:52.027 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:52.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:52.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:52.129 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:52.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:52.153 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:52.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:52.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:52.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:52.279 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:52.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:52.380 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:52.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:52.405 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:52.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:52.506 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:52.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:52.530 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:52.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:52.631 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:52.659 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:52.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:52.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:52.761 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:52.788 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:52.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:52.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:52.889 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:52.915 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:52.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:52.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:53.017 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:53.044 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:53.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:53.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:53.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:52 smithi043 ceph-mon[29963]: pgmap v810: 158 pgs: 106 active+clean, 13 active+undersized, 8 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 500 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:00:53.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:52 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 42/594 objects degraded (7.071%), 8 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:53.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:52 smithi043 ceph-mon[42099]: pgmap v810: 158 pgs: 106 active+clean, 13 active+undersized, 8 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 500 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:00:53.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:52 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 42/594 objects degraded (7.071%), 8 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:53.146 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:53.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:53.174 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:53.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:53.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:52 smithi113 ceph-mon[34341]: pgmap v810: 158 pgs: 106 active+clean, 13 active+undersized, 8 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 500 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:00:53.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:52 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 42/594 objects degraded (7.071%), 8 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:53.275 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:53.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:53.304 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:53.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:53.406 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:53.433 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:53.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:53.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:53.535 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:53.559 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:53.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:53.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:53.661 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:53.689 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:53.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:53.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:53.791 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:53.818 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:53.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:53.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:53.827 INFO:teuthology.orchestra.run.smithi043.stderr:set pool 1 pg_num to 21 2024-03-30T14:00:53.827 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T14:00:53.920 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:53.948 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:53.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:53.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:54.050 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:54.078 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:54.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:54.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:54.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:53 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3904037489' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-30T14:00:54.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:53 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3009137105' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:00:54.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:53 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3904037489' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-30T14:00:54.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:53 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3009137105' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:00:54.179 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:54.204 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:54.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:54.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:54.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:53 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3904037489' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-30T14:00:54.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:53 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3009137105' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:00:54.306 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:54.334 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:54.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:54.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:54.436 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:54.463 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:54.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:54.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:54.565 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:54.593 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:54.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:54.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:54.694 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:54.720 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:54.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:54.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:54.822 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:54.850 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:54.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:54.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:54.952 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:54.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:54.978 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:54.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:55.079 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:55.103 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:55.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:55.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:55.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:54 smithi043 ceph-mon[42099]: pgmap v811: 158 pgs: 106 active+clean, 13 active+undersized, 8 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 923 B/s rd, 0 B/s wr, 451 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:00:55.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:54 smithi043 ceph-mon[42099]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:00:55.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:54 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3904037489' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-30T14:00:55.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:54 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3009137105' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:00:55.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:54 smithi043 ceph-mon[42099]: osdmap e367: 8 total, 7 up, 7 in 2024-03-30T14:00:55.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:54 smithi043 ceph-mon[29963]: pgmap v811: 158 pgs: 106 active+clean, 13 active+undersized, 8 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 923 B/s rd, 0 B/s wr, 451 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:00:55.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:54 smithi043 ceph-mon[29963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:00:55.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:54 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3904037489' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-30T14:00:55.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:54 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3009137105' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:00:55.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:54 smithi043 ceph-mon[29963]: osdmap e367: 8 total, 7 up, 7 in 2024-03-30T14:00:55.205 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:55.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:54 smithi113 ceph-mon[34341]: pgmap v811: 158 pgs: 106 active+clean, 13 active+undersized, 8 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 923 B/s rd, 0 B/s wr, 451 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:00:55.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:54 smithi113 ceph-mon[34341]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:00:55.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:54 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3904037489' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-30T14:00:55.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:54 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3009137105' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:00:55.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:54 smithi113 ceph-mon[34341]: osdmap e367: 8 total, 7 up, 7 in 2024-03-30T14:00:55.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:55.230 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:55.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:55.334 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:55.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:55.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:55.460 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:55.488 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:55.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:55.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:55.590 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:55.617 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:55.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:55.719 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:55.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:55.744 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:55.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:55.845 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:55.870 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:55.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:55.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:55.972 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:55.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:55.998 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:55.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:56.006 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:55 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-30T14:00:56.006 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:55 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-30T14:00:56.033 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-03-30T14:00:56.033 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-03-30T14:00:56.098 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:56.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:56.124 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:56.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:56.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:55 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-30T14:00:56.225 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:56.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:56.250 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:56.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:56.265 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:00:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:00:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:00:56.289 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T14:00:56.290 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T14:00:56.290 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-03-30T14:00:56.290 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-03-30T14:00:56.290 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T14:00:56.350 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:56.374 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:56.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:56.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:56.476 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:56.500 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:56.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:56.602 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:56.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:56.626 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:56.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:56.727 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:56.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:56.752 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:56.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:56.853 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:56.878 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:56.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:56.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:56.980 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:57.004 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:57.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:57.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:57.106 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:57.107 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:56 smithi043 ceph-mon[29963]: 6.9 scrub starts 2024-03-30T14:00:57.107 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:56 smithi043 ceph-mon[29963]: 6.9 scrub ok 2024-03-30T14:00:57.107 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:56 smithi043 ceph-mon[29963]: 1.a scrub starts 2024-03-30T14:00:57.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:56 smithi043 ceph-mon[29963]: 1.a scrub ok 2024-03-30T14:00:57.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:56 smithi043 ceph-mon[29963]: pgmap v813: 158 pgs: 106 active+clean, 14 active+undersized, 7 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 70 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:00:57.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:56 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-30T14:00:57.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:56 smithi043 ceph-mon[29963]: osdmap e368: 8 total, 7 up, 7 in 2024-03-30T14:00:57.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:56 smithi043 ceph-mon[29963]: 3.10 scrub starts 2024-03-30T14:00:57.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:56 smithi043 ceph-mon[29963]: 3.10 scrub ok 2024-03-30T14:00:57.108 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:56 smithi043 ceph-mon[42099]: 6.9 scrub starts 2024-03-30T14:00:57.108 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:56 smithi043 ceph-mon[42099]: 6.9 scrub ok 2024-03-30T14:00:57.108 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:56 smithi043 ceph-mon[42099]: 1.a scrub starts 2024-03-30T14:00:57.108 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:56 smithi043 ceph-mon[42099]: 1.a scrub ok 2024-03-30T14:00:57.108 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:56 smithi043 ceph-mon[42099]: pgmap v813: 158 pgs: 106 active+clean, 14 active+undersized, 7 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 70 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:00:57.108 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:56 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-30T14:00:57.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:56 smithi043 ceph-mon[42099]: osdmap e368: 8 total, 7 up, 7 in 2024-03-30T14:00:57.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:56 smithi043 ceph-mon[42099]: 3.10 scrub starts 2024-03-30T14:00:57.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:56 smithi043 ceph-mon[42099]: 3.10 scrub ok 2024-03-30T14:00:57.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:57.131 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:57.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:57.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:56 smithi113 ceph-mon[34341]: 6.9 scrub starts 2024-03-30T14:00:57.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:56 smithi113 ceph-mon[34341]: 6.9 scrub ok 2024-03-30T14:00:57.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:56 smithi113 ceph-mon[34341]: 1.a scrub starts 2024-03-30T14:00:57.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:56 smithi113 ceph-mon[34341]: 1.a scrub ok 2024-03-30T14:00:57.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:56 smithi113 ceph-mon[34341]: pgmap v813: 158 pgs: 106 active+clean, 14 active+undersized, 7 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 70 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:00:57.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:56 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-30T14:00:57.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:56 smithi113 ceph-mon[34341]: osdmap e368: 8 total, 7 up, 7 in 2024-03-30T14:00:57.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:56 smithi113 ceph-mon[34341]: 3.10 scrub starts 2024-03-30T14:00:57.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:56 smithi113 ceph-mon[34341]: 3.10 scrub ok 2024-03-30T14:00:57.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:57.256 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:57.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:57.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:57.358 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:57.383 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:57.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:57.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:57.480 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-03-30T14:00:57.485 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:57.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:57.510 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:57.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:57.611 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:57.635 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:57.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:57.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:57.737 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:57.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:57.761 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:57.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:57.862 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:00:57.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:57.887 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:57.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:00:57.933 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:00:57.975 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:00:57.988 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:57.989 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T14:00:57.989 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[42099]: 3.1c scrub starts 2024-03-30T14:00:57.990 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[42099]: 3.1c scrub ok 2024-03-30T14:00:57.990 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[42099]: osdmap e369: 8 total, 7 up, 7 in 2024-03-30T14:00:57.990 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[42099]: 1.5 scrub starts 2024-03-30T14:00:57.990 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[42099]: 1.5 scrub ok 2024-03-30T14:00:57.990 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[42099]: 1.3 scrub starts 2024-03-30T14:00:57.990 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[42099]: 1.3 scrub ok 2024-03-30T14:00:57.990 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[42099]: 1.4 scrub starts 2024-03-30T14:00:57.990 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[42099]: 1.4 scrub ok 2024-03-30T14:00:57.990 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[42099]: 1.7 scrub starts 2024-03-30T14:00:57.990 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[42099]: 1.7 scrub ok 2024-03-30T14:00:57.990 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 42/594 objects degraded (7.071%), 7 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:57.990 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[42099]: osdmap e370: 8 total, 7 up, 7 in 2024-03-30T14:00:57.990 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-30T14:00:57.991 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[29963]: 3.1c scrub starts 2024-03-30T14:00:57.993 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[29963]: 3.1c scrub ok 2024-03-30T14:00:57.993 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[29963]: osdmap e369: 8 total, 7 up, 7 in 2024-03-30T14:00:57.993 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[29963]: 1.5 scrub starts 2024-03-30T14:00:57.993 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[29963]: 1.5 scrub ok 2024-03-30T14:00:57.994 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[29963]: 1.3 scrub starts 2024-03-30T14:00:57.994 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[29963]: 1.3 scrub ok 2024-03-30T14:00:57.994 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[29963]: 1.4 scrub starts 2024-03-30T14:00:57.994 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[29963]: 1.4 scrub ok 2024-03-30T14:00:57.994 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[29963]: 1.7 scrub starts 2024-03-30T14:00:57.994 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[29963]: 1.7 scrub ok 2024-03-30T14:00:57.994 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 42/594 objects degraded (7.071%), 7 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:57.994 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[29963]: osdmap e370: 8 total, 7 up, 7 in 2024-03-30T14:00:57.994 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:57 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-30T14:00:58.012 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:58.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:58.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:58.013 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T14:00:58.033 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T14:00:58.053 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T14:00:58.073 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T14:00:58.092 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T14:00:58.113 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:58.114 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T14:00:58.114 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:00:58.114 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:00:58.114 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T14:00:58.115 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:00:58.115 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:00:58.115 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:00:58.115 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:00:58.115 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:00:58.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:58.140 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:58.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:58.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:57 smithi113 ceph-mon[34341]: 3.1c scrub starts 2024-03-30T14:00:58.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:57 smithi113 ceph-mon[34341]: 3.1c scrub ok 2024-03-30T14:00:58.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:57 smithi113 ceph-mon[34341]: osdmap e369: 8 total, 7 up, 7 in 2024-03-30T14:00:58.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:57 smithi113 ceph-mon[34341]: 1.5 scrub starts 2024-03-30T14:00:58.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:57 smithi113 ceph-mon[34341]: 1.5 scrub ok 2024-03-30T14:00:58.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:57 smithi113 ceph-mon[34341]: 1.3 scrub starts 2024-03-30T14:00:58.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:57 smithi113 ceph-mon[34341]: 1.3 scrub ok 2024-03-30T14:00:58.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:57 smithi113 ceph-mon[34341]: 1.4 scrub starts 2024-03-30T14:00:58.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:57 smithi113 ceph-mon[34341]: 1.4 scrub ok 2024-03-30T14:00:58.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:57 smithi113 ceph-mon[34341]: 1.7 scrub starts 2024-03-30T14:00:58.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:57 smithi113 ceph-mon[34341]: 1.7 scrub ok 2024-03-30T14:00:58.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:57 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 42/594 objects degraded (7.071%), 7 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:00:58.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:57 smithi113 ceph-mon[34341]: osdmap e370: 8 total, 7 up, 7 in 2024-03-30T14:00:58.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:57 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-30T14:00:58.241 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:58.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:58.266 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:58.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:58.367 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:58.391 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:58.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:58.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:58.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:00:58.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:58.519 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:58.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:00:58.620 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:58.645 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:58.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:58.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:58.706 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T14:00:58.747 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:58.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:58.772 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:58.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:58.874 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:00:58.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:58.899 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:58.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:00:59.000 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:59.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:59.025 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:59.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:59.126 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:59.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[42099]: 1.0 scrub starts 2024-03-30T14:00:59.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[42099]: 1.0 scrub ok 2024-03-30T14:00:59.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[42099]: 1.1 scrub starts 2024-03-30T14:00:59.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[42099]: 1.1 scrub ok 2024-03-30T14:00:59.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[42099]: pgmap v817: 168 pgs: 1 remapped+peering, 2 peering, 10 unknown, 104 active+clean, 15 active+undersized, 6 active+undersized+degraded, 30 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:00:59.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1892957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:00:59.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[42099]: 1.b scrub starts 2024-03-30T14:00:59.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[42099]: 1.b scrub ok 2024-03-30T14:00:59.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[42099]: 1.14 scrub starts 2024-03-30T14:00:59.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[42099]: 1.14 scrub ok 2024-03-30T14:00:59.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[42099]: 1.f scrub starts 2024-03-30T14:00:59.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[42099]: 1.f scrub ok 2024-03-30T14:00:59.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-30T14:00:59.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[42099]: osdmap e371: 8 total, 7 up, 7 in 2024-03-30T14:00:59.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[29963]: 1.0 scrub starts 2024-03-30T14:00:59.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[29963]: 1.0 scrub ok 2024-03-30T14:00:59.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[29963]: 1.1 scrub starts 2024-03-30T14:00:59.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[29963]: 1.1 scrub ok 2024-03-30T14:00:59.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[29963]: pgmap v817: 168 pgs: 1 remapped+peering, 2 peering, 10 unknown, 104 active+clean, 15 active+undersized, 6 active+undersized+degraded, 30 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:00:59.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1892957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:00:59.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[29963]: 1.b scrub starts 2024-03-30T14:00:59.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[29963]: 1.b scrub ok 2024-03-30T14:00:59.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[29963]: 1.14 scrub starts 2024-03-30T14:00:59.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[29963]: 1.14 scrub ok 2024-03-30T14:00:59.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[29963]: 1.f scrub starts 2024-03-30T14:00:59.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[29963]: 1.f scrub ok 2024-03-30T14:00:59.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-30T14:00:59.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:58 smithi043 ceph-mon[29963]: osdmap e371: 8 total, 7 up, 7 in 2024-03-30T14:00:59.149 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:59.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:59.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:59.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:58 smithi113 ceph-mon[34341]: 1.0 scrub starts 2024-03-30T14:00:59.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:58 smithi113 ceph-mon[34341]: 1.0 scrub ok 2024-03-30T14:00:59.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:58 smithi113 ceph-mon[34341]: 1.1 scrub starts 2024-03-30T14:00:59.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:58 smithi113 ceph-mon[34341]: 1.1 scrub ok 2024-03-30T14:00:59.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:58 smithi113 ceph-mon[34341]: pgmap v817: 168 pgs: 1 remapped+peering, 2 peering, 10 unknown, 104 active+clean, 15 active+undersized, 6 active+undersized+degraded, 30 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:00:59.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:58 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1892957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:00:59.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:58 smithi113 ceph-mon[34341]: 1.b scrub starts 2024-03-30T14:00:59.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:58 smithi113 ceph-mon[34341]: 1.b scrub ok 2024-03-30T14:00:59.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:58 smithi113 ceph-mon[34341]: 1.14 scrub starts 2024-03-30T14:00:59.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:58 smithi113 ceph-mon[34341]: 1.14 scrub ok 2024-03-30T14:00:59.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:58 smithi113 ceph-mon[34341]: 1.f scrub starts 2024-03-30T14:00:59.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:58 smithi113 ceph-mon[34341]: 1.f scrub ok 2024-03-30T14:00:59.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:58 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-30T14:00:59.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:58 smithi113 ceph-mon[34341]: osdmap e371: 8 total, 7 up, 7 in 2024-03-30T14:00:59.251 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:00:59.279 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:59.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:59.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:00:59.381 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:59.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:59.406 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:59.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:59.507 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:59.531 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:59.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:59.632 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:00:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:00:59.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:59.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:00:59.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:00:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:00:59.769 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:00:59.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:59.793 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:59.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:00:59.894 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:00:59.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:00:59.919 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:00:59.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:00:59.973 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:59 smithi043 ceph-mon[29963]: 1.10 scrub starts 2024-03-30T14:00:59.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:59 smithi043 ceph-mon[29963]: 1.10 scrub ok 2024-03-30T14:00:59.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:59 smithi043 ceph-mon[29963]: 5.1d scrub starts 2024-03-30T14:00:59.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:59 smithi043 ceph-mon[29963]: 5.1d scrub ok 2024-03-30T14:00:59.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:59 smithi043 ceph-mon[29963]: 1.6 scrub starts 2024-03-30T14:00:59.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:59 smithi043 ceph-mon[29963]: 1.6 scrub ok 2024-03-30T14:00:59.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:59 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:59.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:00:59 smithi043 ceph-mon[29963]: osdmap e372: 8 total, 7 up, 7 in 2024-03-30T14:00:59.974 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:59 smithi043 ceph-mon[42099]: 1.10 scrub starts 2024-03-30T14:00:59.974 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:59 smithi043 ceph-mon[42099]: 1.10 scrub ok 2024-03-30T14:00:59.974 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:59 smithi043 ceph-mon[42099]: 5.1d scrub starts 2024-03-30T14:00:59.975 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:59 smithi043 ceph-mon[42099]: 5.1d scrub ok 2024-03-30T14:00:59.975 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:59 smithi043 ceph-mon[42099]: 1.6 scrub starts 2024-03-30T14:00:59.975 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:59 smithi043 ceph-mon[42099]: 1.6 scrub ok 2024-03-30T14:00:59.975 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:59 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:59.975 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:00:59 smithi043 ceph-mon[42099]: osdmap e372: 8 total, 7 up, 7 in 2024-03-30T14:00:59.975 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T14:00:59.976 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T14:01:00.019 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:00.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:00.043 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:00.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:00.144 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:00.170 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:00.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:00.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:00.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:59 smithi113 ceph-mon[34341]: 1.10 scrub starts 2024-03-30T14:01:00.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:59 smithi113 ceph-mon[34341]: 1.10 scrub ok 2024-03-30T14:01:00.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:59 smithi113 ceph-mon[34341]: 5.1d scrub starts 2024-03-30T14:01:00.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:59 smithi113 ceph-mon[34341]: 5.1d scrub ok 2024-03-30T14:01:00.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:59 smithi113 ceph-mon[34341]: 1.6 scrub starts 2024-03-30T14:01:00.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:59 smithi113 ceph-mon[34341]: 1.6 scrub ok 2024-03-30T14:01:00.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:59 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:00.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:00:59 smithi113 ceph-mon[34341]: osdmap e372: 8 total, 7 up, 7 in 2024-03-30T14:01:00.272 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:00.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:00.298 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:00.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:00.399 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:00.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:00.423 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:00.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:00.524 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:00.548 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:00.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:00.649 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:00.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:00.682 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:00.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:00.783 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:00.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:00.810 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:00.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:00.912 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:00.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:00.936 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:00.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:01.034 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 5] out_osds: [7] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-30T14:01:01.034 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T14:01:01.034 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-30T14:01:01.034 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.000000 2024-03-30T14:01:01.034 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 0 0 2024-03-30T14:01:01.037 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:01.051 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:01.052 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[29963]: 1.2 scrub starts 2024-03-30T14:01:01.052 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[29963]: 1.2 scrub ok 2024-03-30T14:01:01.052 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[29963]: 10.3 scrub starts 2024-03-30T14:01:01.052 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[29963]: 10.3 scrub ok 2024-03-30T14:01:01.052 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[29963]: pgmap v820: 168 pgs: 2 remapped, 1 remapped+peering, 2 peering, 7 unknown, 105 active+clean, 15 active+undersized, 6 active+undersized+degraded, 30 active+clean+remapped; 369 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:01.052 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[29963]: 6.4 scrub starts 2024-03-30T14:01:01.052 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[29963]: 6.4 scrub ok 2024-03-30T14:01:01.052 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[29963]: 4.1f scrub starts 2024-03-30T14:01:01.052 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[29963]: 4.1f scrub ok 2024-03-30T14:01:01.052 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[29963]: osdmap e373: 8 total, 7 up, 7 in 2024-03-30T14:01:01.053 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:01.053 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[42099]: 1.2 scrub starts 2024-03-30T14:01:01.053 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[42099]: 1.2 scrub ok 2024-03-30T14:01:01.053 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[42099]: 10.3 scrub starts 2024-03-30T14:01:01.053 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[42099]: 10.3 scrub ok 2024-03-30T14:01:01.053 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[42099]: pgmap v820: 168 pgs: 2 remapped, 1 remapped+peering, 2 peering, 7 unknown, 105 active+clean, 15 active+undersized, 6 active+undersized+degraded, 30 active+clean+remapped; 369 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:01.053 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[42099]: 6.4 scrub starts 2024-03-30T14:01:01.053 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[42099]: 6.4 scrub ok 2024-03-30T14:01:01.053 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[42099]: 4.1f scrub starts 2024-03-30T14:01:01.053 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[42099]: 4.1f scrub ok 2024-03-30T14:01:01.053 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:00 smithi043 ceph-mon[42099]: osdmap e373: 8 total, 7 up, 7 in 2024-03-30T14:01:01.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:01.063 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:01.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:01.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:01.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:01.189 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:01.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:01.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:00 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:01.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:00 smithi113 ceph-mon[34341]: 1.2 scrub starts 2024-03-30T14:01:01.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:00 smithi113 ceph-mon[34341]: 1.2 scrub ok 2024-03-30T14:01:01.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:00 smithi113 ceph-mon[34341]: 10.3 scrub starts 2024-03-30T14:01:01.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:00 smithi113 ceph-mon[34341]: 10.3 scrub ok 2024-03-30T14:01:01.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:00 smithi113 ceph-mon[34341]: pgmap v820: 168 pgs: 2 remapped, 1 remapped+peering, 2 peering, 7 unknown, 105 active+clean, 15 active+undersized, 6 active+undersized+degraded, 30 active+clean+remapped; 369 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:01.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:00 smithi113 ceph-mon[34341]: 6.4 scrub starts 2024-03-30T14:01:01.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:00 smithi113 ceph-mon[34341]: 6.4 scrub ok 2024-03-30T14:01:01.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:00 smithi113 ceph-mon[34341]: 4.1f scrub starts 2024-03-30T14:01:01.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:00 smithi113 ceph-mon[34341]: 4.1f scrub ok 2024-03-30T14:01:01.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:00 smithi113 ceph-mon[34341]: osdmap e373: 8 total, 7 up, 7 in 2024-03-30T14:01:01.290 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:01.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:01.314 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:01.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:01.415 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:01.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:01.439 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:01.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:01.541 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:01.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:01.565 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:01.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:01.666 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:01.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:01.691 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:01.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:01.792 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:01.815 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:01.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:01.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:01.917 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:01.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:01.941 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:01.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:01.976 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T14:01:02.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:01 smithi043 ceph-mon[29963]: 5.6 scrub starts 2024-03-30T14:01:02.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:01 smithi043 ceph-mon[29963]: 5.6 scrub ok 2024-03-30T14:01:02.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:01 smithi043 ceph-mon[29963]: 3.a scrub starts 2024-03-30T14:01:02.003 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:01 smithi043 ceph-mon[29963]: 3.a scrub ok 2024-03-30T14:01:02.003 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:01 smithi043 ceph-mon[29963]: osdmap e374: 8 total, 7 up, 7 in 2024-03-30T14:01:02.003 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:01 smithi043 ceph-mon[42099]: 5.6 scrub starts 2024-03-30T14:01:02.003 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:01 smithi043 ceph-mon[42099]: 5.6 scrub ok 2024-03-30T14:01:02.003 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:01 smithi043 ceph-mon[42099]: 3.a scrub starts 2024-03-30T14:01:02.003 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:01 smithi043 ceph-mon[42099]: 3.a scrub ok 2024-03-30T14:01:02.003 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:01 smithi043 ceph-mon[42099]: osdmap e374: 8 total, 7 up, 7 in 2024-03-30T14:01:02.043 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:02.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:02.067 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:02.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:02.169 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:02.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:02.194 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:02.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:02.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:01 smithi113 ceph-mon[34341]: 5.6 scrub starts 2024-03-30T14:01:02.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:01 smithi113 ceph-mon[34341]: 5.6 scrub ok 2024-03-30T14:01:02.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:01 smithi113 ceph-mon[34341]: 3.a scrub starts 2024-03-30T14:01:02.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:01 smithi113 ceph-mon[34341]: 3.a scrub ok 2024-03-30T14:01:02.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:01 smithi113 ceph-mon[34341]: osdmap e374: 8 total, 7 up, 7 in 2024-03-30T14:01:02.295 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:02.319 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:02.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:02.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:02.336 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:01:02.384 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:01:02.421 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:02.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:02.445 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:02.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:02.546 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:02.587 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:02.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:02.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:02.663 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'unique_pool_3' removed 2024-03-30T14:01:02.690 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:02.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:02.756 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:02.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:02.857 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:02.883 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:02.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:02.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:02.985 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:03.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:03.009 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:03.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:03.110 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:03.122 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:02 smithi043 ceph-mon[29963]: pgmap v823: 168 pgs: 1 remapped, 2 unknown, 107 active+clean, 15 active+undersized, 6 active+undersized+remapped, 6 active+undersized+degraded, 31 active+clean+remapped; 395 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 5 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:03.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:02 smithi043 ceph-mon[29963]: 10.2 scrub starts 2024-03-30T14:01:03.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:02 smithi043 ceph-mon[29963]: 10.2 scrub ok 2024-03-30T14:01:03.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:02 smithi043 ceph-mon[29963]: 6.5 scrub starts 2024-03-30T14:01:03.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:02 smithi043 ceph-mon[29963]: 6.5 scrub ok 2024-03-30T14:01:03.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:02 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4023577135' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:01:03.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:02 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 42/594 objects degraded (7.071%), 6 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:01:03.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:02 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4023577135' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T14:01:03.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:02 smithi043 ceph-mon[29963]: osdmap e375: 8 total, 7 up, 7 in 2024-03-30T14:01:03.123 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:02 smithi043 ceph-mon[42099]: pgmap v823: 168 pgs: 1 remapped, 2 unknown, 107 active+clean, 15 active+undersized, 6 active+undersized+remapped, 6 active+undersized+degraded, 31 active+clean+remapped; 395 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 5 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:03.124 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:02 smithi043 ceph-mon[42099]: 10.2 scrub starts 2024-03-30T14:01:03.124 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:02 smithi043 ceph-mon[42099]: 10.2 scrub ok 2024-03-30T14:01:03.124 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:02 smithi043 ceph-mon[42099]: 6.5 scrub starts 2024-03-30T14:01:03.124 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:02 smithi043 ceph-mon[42099]: 6.5 scrub ok 2024-03-30T14:01:03.124 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:02 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/4023577135' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:01:03.124 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:02 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 42/594 objects degraded (7.071%), 6 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:01:03.124 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:02 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/4023577135' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T14:01:03.124 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:02 smithi043 ceph-mon[42099]: osdmap e375: 8 total, 7 up, 7 in 2024-03-30T14:01:03.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:03.135 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:03.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:03.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:02 smithi113 ceph-mon[34341]: pgmap v823: 168 pgs: 1 remapped, 2 unknown, 107 active+clean, 15 active+undersized, 6 active+undersized+remapped, 6 active+undersized+degraded, 31 active+clean+remapped; 395 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 5 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:03.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:02 smithi113 ceph-mon[34341]: 10.2 scrub starts 2024-03-30T14:01:03.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:02 smithi113 ceph-mon[34341]: 10.2 scrub ok 2024-03-30T14:01:03.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:02 smithi113 ceph-mon[34341]: 6.5 scrub starts 2024-03-30T14:01:03.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:02 smithi113 ceph-mon[34341]: 6.5 scrub ok 2024-03-30T14:01:03.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:02 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/4023577135' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:01:03.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:02 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 42/594 objects degraded (7.071%), 6 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-03-30T14:01:03.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:02 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/4023577135' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T14:01:03.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:02 smithi113 ceph-mon[34341]: osdmap e375: 8 total, 7 up, 7 in 2024-03-30T14:01:03.236 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:03.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:03.261 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:03.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:03.362 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:03.387 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:03.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:03.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:03.488 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:03.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:03.513 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:03.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:03.614 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:03.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:03.639 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:03.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:03.740 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:03.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:03.764 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:03.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:03.865 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:03.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:03.890 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:03.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:03.991 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:04.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:04.015 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:04.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:04.095 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:03 smithi043 ceph-mon[29963]: 6.10 scrub starts 2024-03-30T14:01:04.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:03 smithi043 ceph-mon[29963]: 6.10 scrub ok 2024-03-30T14:01:04.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:03 smithi043 ceph-mon[29963]: 6.19 scrub starts 2024-03-30T14:01:04.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:03 smithi043 ceph-mon[29963]: 6.19 scrub ok 2024-03-30T14:01:04.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:03 smithi043 ceph-mon[29963]: 1.d scrub starts 2024-03-30T14:01:04.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:03 smithi043 ceph-mon[29963]: 1.d scrub ok 2024-03-30T14:01:04.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:03 smithi043 ceph-mon[29963]: 1.13 scrub starts 2024-03-30T14:01:04.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:03 smithi043 ceph-mon[29963]: 1.13 scrub ok 2024-03-30T14:01:04.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:03 smithi043 ceph-mon[29963]: 1.c scrub starts 2024-03-30T14:01:04.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:03 smithi043 ceph-mon[29963]: 1.c scrub ok 2024-03-30T14:01:04.096 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:03 smithi043 ceph-mon[42099]: 6.10 scrub starts 2024-03-30T14:01:04.096 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:03 smithi043 ceph-mon[42099]: 6.10 scrub ok 2024-03-30T14:01:04.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:03 smithi043 ceph-mon[42099]: 6.19 scrub starts 2024-03-30T14:01:04.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:03 smithi043 ceph-mon[42099]: 6.19 scrub ok 2024-03-30T14:01:04.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:03 smithi043 ceph-mon[42099]: 1.d scrub starts 2024-03-30T14:01:04.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:03 smithi043 ceph-mon[42099]: 1.d scrub ok 2024-03-30T14:01:04.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:03 smithi043 ceph-mon[42099]: 1.13 scrub starts 2024-03-30T14:01:04.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:03 smithi043 ceph-mon[42099]: 1.13 scrub ok 2024-03-30T14:01:04.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:03 smithi043 ceph-mon[42099]: 1.c scrub starts 2024-03-30T14:01:04.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:03 smithi043 ceph-mon[42099]: 1.c scrub ok 2024-03-30T14:01:04.117 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:04.140 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:04.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:04.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:04.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:03 smithi113 ceph-mon[34341]: 6.10 scrub starts 2024-03-30T14:01:04.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:03 smithi113 ceph-mon[34341]: 6.10 scrub ok 2024-03-30T14:01:04.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:03 smithi113 ceph-mon[34341]: 6.19 scrub starts 2024-03-30T14:01:04.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:03 smithi113 ceph-mon[34341]: 6.19 scrub ok 2024-03-30T14:01:04.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:03 smithi113 ceph-mon[34341]: 1.d scrub starts 2024-03-30T14:01:04.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:03 smithi113 ceph-mon[34341]: 1.d scrub ok 2024-03-30T14:01:04.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:03 smithi113 ceph-mon[34341]: 1.13 scrub starts 2024-03-30T14:01:04.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:03 smithi113 ceph-mon[34341]: 1.13 scrub ok 2024-03-30T14:01:04.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:03 smithi113 ceph-mon[34341]: 1.c scrub starts 2024-03-30T14:01:04.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:03 smithi113 ceph-mon[34341]: 1.c scrub ok 2024-03-30T14:01:04.242 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:04.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:04.267 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:04.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:04.368 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:04.392 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:04.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:04.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:04.493 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:04.514 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:01:04.517 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:04.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:04.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:04.619 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:04.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:04.643 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:04.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:04.745 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:04.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:04.770 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:04.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:04.871 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:04.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:04.895 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:04.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:04.996 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:05.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:05.021 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:05.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:05.086 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:04 smithi043 ceph-mon[29963]: 1.11 scrub starts 2024-03-30T14:01:05.086 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:04 smithi043 ceph-mon[29963]: 1.11 scrub ok 2024-03-30T14:01:05.086 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:04 smithi043 ceph-mon[29963]: 1.12 scrub starts 2024-03-30T14:01:05.087 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:04 smithi043 ceph-mon[29963]: 1.12 scrub ok 2024-03-30T14:01:05.087 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:04 smithi043 ceph-mon[29963]: pgmap v825: 152 pgs: 1 remapped, 2 unknown, 97 active+clean, 12 active+undersized, 6 active+undersized+remapped, 6 active+undersized+degraded, 28 active+clean+remapped; 609 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 4.4 KiB/s rd, 3 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:05.087 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:04 smithi043 ceph-mon[29963]: 1.e scrub starts 2024-03-30T14:01:05.087 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:04 smithi043 ceph-mon[29963]: 1.e scrub ok 2024-03-30T14:01:05.087 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:04 smithi043 ceph-mon[42099]: 1.11 scrub starts 2024-03-30T14:01:05.087 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:04 smithi043 ceph-mon[42099]: 1.11 scrub ok 2024-03-30T14:01:05.087 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:04 smithi043 ceph-mon[42099]: 1.12 scrub starts 2024-03-30T14:01:05.087 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:04 smithi043 ceph-mon[42099]: 1.12 scrub ok 2024-03-30T14:01:05.087 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:04 smithi043 ceph-mon[42099]: pgmap v825: 152 pgs: 1 remapped, 2 unknown, 97 active+clean, 12 active+undersized, 6 active+undersized+remapped, 6 active+undersized+degraded, 28 active+clean+remapped; 609 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 4.4 KiB/s rd, 3 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:05.087 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:04 smithi043 ceph-mon[42099]: 1.e scrub starts 2024-03-30T14:01:05.087 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:04 smithi043 ceph-mon[42099]: 1.e scrub ok 2024-03-30T14:01:05.122 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:05.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:05.146 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:05.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:05.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:04 smithi113 ceph-mon[34341]: 1.11 scrub starts 2024-03-30T14:01:05.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:04 smithi113 ceph-mon[34341]: 1.11 scrub ok 2024-03-30T14:01:05.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:04 smithi113 ceph-mon[34341]: 1.12 scrub starts 2024-03-30T14:01:05.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:04 smithi113 ceph-mon[34341]: 1.12 scrub ok 2024-03-30T14:01:05.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:04 smithi113 ceph-mon[34341]: pgmap v825: 152 pgs: 1 remapped, 2 unknown, 97 active+clean, 12 active+undersized, 6 active+undersized+remapped, 6 active+undersized+degraded, 28 active+clean+remapped; 609 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 4.4 KiB/s rd, 3 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:05.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:04 smithi113 ceph-mon[34341]: 1.e scrub starts 2024-03-30T14:01:05.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:04 smithi113 ceph-mon[34341]: 1.e scrub ok 2024-03-30T14:01:05.247 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:05.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:05.273 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:05.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:05.374 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:05.398 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:05.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:05.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:05.490 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-30T14:01:05.491 INFO:tasks.radosbench:Beginning radosbench... 2024-03-30T14:01:05.492 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-03-30T14:01:05.492 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool create unique_pool_4 16 2024-03-30T14:01:05.501 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:05.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:05.526 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:05.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:05.627 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:05.652 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:05.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:05.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:05.753 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:05.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:05.778 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:05.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:05.879 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:05.903 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:05.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:05.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:06.005 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:06.031 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:06.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:06.132 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:06.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:05 smithi043 ceph-mon[29963]: 5.18 scrub starts 2024-03-30T14:01:06.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:05 smithi043 ceph-mon[29963]: 5.18 scrub ok 2024-03-30T14:01:06.142 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:01:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:01:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:01:06.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:05 smithi043 ceph-mon[42099]: 5.18 scrub starts 2024-03-30T14:01:06.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:05 smithi043 ceph-mon[42099]: 5.18 scrub ok 2024-03-30T14:01:06.156 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:06.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:06.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:06.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:05 smithi113 ceph-mon[34341]: 5.18 scrub starts 2024-03-30T14:01:06.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:05 smithi113 ceph-mon[34341]: 5.18 scrub ok 2024-03-30T14:01:06.258 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:06.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:06.283 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:06.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:06.376 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T14:01:06.377 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:01:06.384 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:06.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:06.409 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:06.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:06.510 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:06.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:06.536 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:06.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:06.637 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:06.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:06.662 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:06.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:06.763 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:06.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:06.791 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:06.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:06.892 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:06.917 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:06.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:06.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:07.014 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:06 smithi043 ceph-mon[29963]: pgmap v826: 152 pgs: 100 active+clean, 12 active+undersized, 3 active+undersized+remapped, 6 active+undersized+degraded, 31 active+clean+remapped; 609 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 3.0 KiB/s rd, 2 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:07.015 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:06 smithi043 ceph-mon[42099]: pgmap v826: 152 pgs: 100 active+clean, 12 active+undersized, 3 active+undersized+remapped, 6 active+undersized+degraded, 31 active+clean+remapped; 609 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 3.0 KiB/s rd, 2 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:07.019 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:07.045 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:07.047 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-30T14:01:07.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:07.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:07.150 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:07.175 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:07.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:07.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:07.218 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:06 smithi113 ceph-mon[34341]: pgmap v826: 152 pgs: 100 active+clean, 12 active+undersized, 3 active+undersized+remapped, 6 active+undersized+degraded, 31 active+clean+remapped; 609 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 3.0 KiB/s rd, 2 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:07.276 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:07.301 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:07.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:07.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:07.307 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:01:07.403 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:07.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:07.428 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:07.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:07.529 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:07.555 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:07.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:07.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:07.654 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-03-30T14:01:07.657 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:07.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:07.682 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:07.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:07.783 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:07.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:07.809 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:07.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:07.911 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:07.936 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:07.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:07.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:07.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:07 smithi113 ceph-mon[34341]: 4.0 scrub starts 2024-03-30T14:01:07.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:07 smithi113 ceph-mon[34341]: 4.0 scrub ok 2024-03-30T14:01:07.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:07 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1521859384' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:01:07.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:07 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:01:07.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:07 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 42/594 objects degraded (7.071%), 6 pgs degraded, 18 pgs undersized (PG_DEGRADED) 2024-03-30T14:01:07.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:07 smithi113 ceph-mon[34341]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:01:07.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:07 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:01:07.966 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:07 smithi113 ceph-mon[34341]: osdmap e376: 8 total, 7 up, 7 in 2024-03-30T14:01:08.038 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:08.064 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:08.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:08.138 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:07 smithi043 ceph-mon[42099]: 4.0 scrub starts 2024-03-30T14:01:08.138 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:07 smithi043 ceph-mon[42099]: 4.0 scrub ok 2024-03-30T14:01:08.138 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:07 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1521859384' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:01:08.139 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:07 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:01:08.139 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:07 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 42/594 objects degraded (7.071%), 6 pgs degraded, 18 pgs undersized (PG_DEGRADED) 2024-03-30T14:01:08.139 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:07 smithi043 ceph-mon[42099]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:01:08.139 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:07 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:01:08.139 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:07 smithi043 ceph-mon[42099]: osdmap e376: 8 total, 7 up, 7 in 2024-03-30T14:01:08.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:07 smithi043 ceph-mon[29963]: 4.0 scrub starts 2024-03-30T14:01:08.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:07 smithi043 ceph-mon[29963]: 4.0 scrub ok 2024-03-30T14:01:08.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:07 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1521859384' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:01:08.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:07 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:01:08.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:07 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 42/594 objects degraded (7.071%), 6 pgs degraded, 18 pgs undersized (PG_DEGRADED) 2024-03-30T14:01:08.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:07 smithi043 ceph-mon[29963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:01:08.140 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:07 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:01:08.140 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:07 smithi043 ceph-mon[29963]: osdmap e376: 8 total, 7 up, 7 in 2024-03-30T14:01:08.166 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:08.191 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:08.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:08.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:08.292 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:08.317 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:08.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:08.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:08.420 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:08.446 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:08.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:08.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:08.550 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:08.615 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:08.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:08.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:08.717 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:08.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:08.742 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:08.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:08.843 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:08.858 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:08 smithi043 ceph-mon[29963]: 5.5 scrub starts 2024-03-30T14:01:08.858 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:08 smithi043 ceph-mon[29963]: 5.5 scrub ok 2024-03-30T14:01:08.859 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:08 smithi043 ceph-mon[29963]: pgmap v828: 152 pgs: 103 active+clean, 12 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 609 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 673 B/s rd, 0 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:08.859 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:08 smithi043 ceph-mon[29963]: 6.14 scrub starts 2024-03-30T14:01:08.859 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:08 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1889050193' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-03-30T14:01:08.859 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:08 smithi043 ceph-mon[29963]: 5.9 deep-scrub starts 2024-03-30T14:01:08.859 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:08 smithi043 ceph-mon[29963]: 5.9 deep-scrub ok 2024-03-30T14:01:08.865 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.0 primary-affinity to 0 (802) 2024-03-30T14:01:08.876 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:08.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:08.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:08.978 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:09.002 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:09.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:09.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:09.105 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:09.129 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:09.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:09.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:09.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:08 smithi043 ceph-mon[42099]: 5.5 scrub starts 2024-03-30T14:01:09.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:08 smithi043 ceph-mon[42099]: 5.5 scrub ok 2024-03-30T14:01:09.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:08 smithi043 ceph-mon[42099]: pgmap v828: 152 pgs: 103 active+clean, 12 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 609 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 673 B/s rd, 0 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:09.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:08 smithi043 ceph-mon[42099]: 6.14 scrub starts 2024-03-30T14:01:09.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:08 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1889050193' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-03-30T14:01:09.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:08 smithi043 ceph-mon[42099]: 5.9 deep-scrub starts 2024-03-30T14:01:09.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:08 smithi043 ceph-mon[42099]: 5.9 deep-scrub ok 2024-03-30T14:01:09.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:08 smithi113 ceph-mon[34341]: 5.5 scrub starts 2024-03-30T14:01:09.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:08 smithi113 ceph-mon[34341]: 5.5 scrub ok 2024-03-30T14:01:09.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:08 smithi113 ceph-mon[34341]: pgmap v828: 152 pgs: 103 active+clean, 12 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 609 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 673 B/s rd, 0 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:09.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:08 smithi113 ceph-mon[34341]: 6.14 scrub starts 2024-03-30T14:01:09.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:08 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1889050193' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-03-30T14:01:09.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:08 smithi113 ceph-mon[34341]: 5.9 deep-scrub starts 2024-03-30T14:01:09.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:08 smithi113 ceph-mon[34341]: 5.9 deep-scrub ok 2024-03-30T14:01:09.231 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:09.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:09.256 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:09.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:09.357 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:09.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:09.382 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:09.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:09.483 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:09.507 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:09.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:09.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:09.543 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T14:01:09.544 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:01:09.610 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:09.630 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:01:09.635 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:09.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:09.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:09.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:01:09 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:01:09.737 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:09.762 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:09.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:09.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:09.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:09.891 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:09.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:09.992 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:10.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:10.018 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:10.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:10.117 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:09 smithi113 ceph-mon[34341]: 6.14 scrub ok 2024-03-30T14:01:10.117 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:09 smithi113 ceph-mon[34341]: 4.14 scrub starts 2024-03-30T14:01:10.117 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:09 smithi113 ceph-mon[34341]: 4.14 scrub ok 2024-03-30T14:01:10.118 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:09 smithi113 ceph-mon[34341]: 3.16 scrub starts 2024-03-30T14:01:10.118 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:09 smithi113 ceph-mon[34341]: 3.16 scrub ok 2024-03-30T14:01:10.118 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:09 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1889050193' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-03-30T14:01:10.118 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:09 smithi113 ceph-mon[34341]: osdmap e377: 8 total, 7 up, 7 in 2024-03-30T14:01:10.118 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:09 smithi113 ceph-mon[34341]: osdmap e378: 8 total, 7 up, 7 in 2024-03-30T14:01:10.120 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:10.125 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:09 smithi043 ceph-mon[29963]: 6.14 scrub ok 2024-03-30T14:01:10.125 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:09 smithi043 ceph-mon[29963]: 4.14 scrub starts 2024-03-30T14:01:10.125 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:09 smithi043 ceph-mon[29963]: 4.14 scrub ok 2024-03-30T14:01:10.125 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:09 smithi043 ceph-mon[29963]: 3.16 scrub starts 2024-03-30T14:01:10.126 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:09 smithi043 ceph-mon[29963]: 3.16 scrub ok 2024-03-30T14:01:10.126 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:09 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1889050193' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-03-30T14:01:10.126 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:09 smithi043 ceph-mon[29963]: osdmap e377: 8 total, 7 up, 7 in 2024-03-30T14:01:10.126 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:09 smithi043 ceph-mon[29963]: osdmap e378: 8 total, 7 up, 7 in 2024-03-30T14:01:10.126 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:09 smithi043 ceph-mon[42099]: 6.14 scrub ok 2024-03-30T14:01:10.126 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:09 smithi043 ceph-mon[42099]: 4.14 scrub starts 2024-03-30T14:01:10.126 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:09 smithi043 ceph-mon[42099]: 4.14 scrub ok 2024-03-30T14:01:10.126 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:09 smithi043 ceph-mon[42099]: 3.16 scrub starts 2024-03-30T14:01:10.126 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:09 smithi043 ceph-mon[42099]: 3.16 scrub ok 2024-03-30T14:01:10.126 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:09 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1889050193' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-03-30T14:01:10.126 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:09 smithi043 ceph-mon[42099]: osdmap e377: 8 total, 7 up, 7 in 2024-03-30T14:01:10.126 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:09 smithi043 ceph-mon[42099]: osdmap e378: 8 total, 7 up, 7 in 2024-03-30T14:01:10.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:10.145 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:10.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:10.147 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-03-30T14:01:10.247 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:10.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:10.275 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:10.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:10.376 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:10.400 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:10.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:10.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:10.424 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:01:10.502 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:10.526 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:10.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:10.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:10.552 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T14:01:10.628 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:10.652 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:10.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:10.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:10.753 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:10.778 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:10.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:10.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:10.880 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:10.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:10 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:10.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:10 smithi043 ceph-mon[29963]: pgmap v830: 152 pgs: 4 peering, 99 active+clean, 12 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 609 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 577 B/s rd, 0 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:10.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:10 smithi043 ceph-mon[29963]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-30T14:01:10.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:10 smithi043 ceph-mon[29963]: 1.0 deep-scrub starts 2024-03-30T14:01:10.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:10 smithi043 ceph-mon[29963]: 1.0 deep-scrub ok 2024-03-30T14:01:10.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:10 smithi043 ceph-mon[29963]: osdmap e379: 8 total, 7 up, 7 in 2024-03-30T14:01:10.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:10 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:10.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:10 smithi043 ceph-mon[42099]: pgmap v830: 152 pgs: 4 peering, 99 active+clean, 12 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 609 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 577 B/s rd, 0 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:10.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:10 smithi043 ceph-mon[42099]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-30T14:01:10.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:10 smithi043 ceph-mon[42099]: 1.0 deep-scrub starts 2024-03-30T14:01:10.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:10 smithi043 ceph-mon[42099]: 1.0 deep-scrub ok 2024-03-30T14:01:10.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:10 smithi043 ceph-mon[42099]: osdmap e379: 8 total, 7 up, 7 in 2024-03-30T14:01:10.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:10.906 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:10.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:10.946 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T14:01:10.966 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T14:01:10.985 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T14:01:11.006 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T14:01:11.007 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:11.026 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T14:01:11.031 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:11.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:11.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:11.046 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T14:01:11.066 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T14:01:11.066 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:01:11.067 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:01:11.067 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T14:01:11.067 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:01:11.067 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:01:11.067 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:01:11.067 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:01:11.067 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:01:11.133 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:11.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:11.157 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:11.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:11.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:10 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:11.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:10 smithi113 ceph-mon[34341]: pgmap v830: 152 pgs: 4 peering, 99 active+clean, 12 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 609 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 577 B/s rd, 0 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:11.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:10 smithi113 ceph-mon[34341]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-30T14:01:11.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:10 smithi113 ceph-mon[34341]: 1.0 deep-scrub starts 2024-03-30T14:01:11.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:10 smithi113 ceph-mon[34341]: 1.0 deep-scrub ok 2024-03-30T14:01:11.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:10 smithi113 ceph-mon[34341]: osdmap e379: 8 total, 7 up, 7 in 2024-03-30T14:01:11.258 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:11.282 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:11.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:11.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:11.384 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:11.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:11.408 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:11.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:11.509 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:11.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:11.535 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:11.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:11.636 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:11.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:11.661 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:11.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:11.762 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:11.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:11.788 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:11.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:11.889 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:11.913 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:11.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:11.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:11.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:11 smithi043 ceph-mon[29963]: 1.11 scrub starts 2024-03-30T14:01:11.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:11 smithi043 ceph-mon[29963]: 1.11 scrub ok 2024-03-30T14:01:11.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:11 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2042579717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:01:11.975 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:11 smithi043 ceph-mon[42099]: 1.11 scrub starts 2024-03-30T14:01:11.975 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:11 smithi043 ceph-mon[42099]: 1.11 scrub ok 2024-03-30T14:01:11.975 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:11 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2042579717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:01:12.015 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:12.039 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:12.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:12.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:12.141 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:12.165 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:12.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:12.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:12.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:11 smithi113 ceph-mon[34341]: 1.11 scrub starts 2024-03-30T14:01:12.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:11 smithi113 ceph-mon[34341]: 1.11 scrub ok 2024-03-30T14:01:12.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:11 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2042579717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:01:12.267 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:12.290 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:12.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:12.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:12.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:01:12.418 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:12.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:12.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:12.520 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:12.544 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:12.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:12.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:12.646 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:12.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:12.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:12.714 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T14:01:12.714 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [], 2024-03-30T14:01:12.714 INFO:teuthology.orchestra.run.smithi113.stdout: "complaint_time": 30, 2024-03-30T14:01:12.714 INFO:teuthology.orchestra.run.smithi113.stdout: "num_blocked_ops": 0 2024-03-30T14:01:12.714 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T14:01:12.772 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:12.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:12.798 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:12.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:12.815 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T14:01:12.815 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T14:01:12.899 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:12.904 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'unique_pool_4' created 2024-03-30T14:01:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:12.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:12.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:13.029 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:13.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:13.055 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:13.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:13.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:12 smithi043 ceph-mon[29963]: 1.2 deep-scrub starts 2024-03-30T14:01:13.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:12 smithi043 ceph-mon[29963]: 1.2 deep-scrub ok 2024-03-30T14:01:13.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:12 smithi043 ceph-mon[29963]: pgmap v833: 152 pgs: 4 peering, 99 active+clean, 12 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 609 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 751 B/s rd, 0 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:13.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:12 smithi043 ceph-mon[29963]: 6.3 scrub starts 2024-03-30T14:01:13.097 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:12 smithi043 ceph-mon[29963]: 6.3 scrub ok 2024-03-30T14:01:13.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:12 smithi043 ceph-mon[42099]: 1.2 deep-scrub starts 2024-03-30T14:01:13.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:12 smithi043 ceph-mon[42099]: 1.2 deep-scrub ok 2024-03-30T14:01:13.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:12 smithi043 ceph-mon[42099]: pgmap v833: 152 pgs: 4 peering, 99 active+clean, 12 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 609 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 751 B/s rd, 0 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:13.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:12 smithi043 ceph-mon[42099]: 6.3 scrub starts 2024-03-30T14:01:13.098 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:12 smithi043 ceph-mon[42099]: 6.3 scrub ok 2024-03-30T14:01:13.155 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:13.180 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:13.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:13.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:13.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:12 smithi113 ceph-mon[34341]: 1.2 deep-scrub starts 2024-03-30T14:01:13.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:12 smithi113 ceph-mon[34341]: 1.2 deep-scrub ok 2024-03-30T14:01:13.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:12 smithi113 ceph-mon[34341]: pgmap v833: 152 pgs: 4 peering, 99 active+clean, 12 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 609 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 751 B/s rd, 0 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:13.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:12 smithi113 ceph-mon[34341]: 6.3 scrub starts 2024-03-30T14:01:13.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:12 smithi113 ceph-mon[34341]: 6.3 scrub ok 2024-03-30T14:01:13.282 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:13.306 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:13.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:13.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:13.335 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-03-30T14:01:13.407 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:13.442 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:13.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:13.543 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:13.568 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:13.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:13.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:13.617 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:01:13.669 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:13.693 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:13.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:13.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:13.795 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:13.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:13.821 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:13.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:13.922 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:13.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:13.946 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:13.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:14.047 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:14.060 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/911198374' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-30T14:01:14.061 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-30T14:01:14.061 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 42/594 objects degraded (7.071%), 6 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-03-30T14:01:14.061 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-30T14:01:14.061 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[42099]: osdmap e380: 8 total, 7 up, 7 in 2024-03-30T14:01:14.061 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[42099]: 5.3 scrub starts 2024-03-30T14:01:14.061 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[42099]: 5.3 scrub ok 2024-03-30T14:01:14.061 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[42099]: 4.b scrub starts 2024-03-30T14:01:14.061 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[42099]: 4.b scrub ok 2024-03-30T14:01:14.061 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:01:14.061 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:01:14.061 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[42099]: osdmap e381: 8 total, 7 up, 7 in 2024-03-30T14:01:14.062 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/911198374' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-30T14:01:14.062 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-30T14:01:14.062 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 42/594 objects degraded (7.071%), 6 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-03-30T14:01:14.062 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-30T14:01:14.062 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[29963]: osdmap e380: 8 total, 7 up, 7 in 2024-03-30T14:01:14.062 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[29963]: 5.3 scrub starts 2024-03-30T14:01:14.062 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[29963]: 5.3 scrub ok 2024-03-30T14:01:14.062 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[29963]: 4.b scrub starts 2024-03-30T14:01:14.062 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[29963]: 4.b scrub ok 2024-03-30T14:01:14.062 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:01:14.062 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:01:14.062 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:13 smithi043 ceph-mon[29963]: osdmap e381: 8 total, 7 up, 7 in 2024-03-30T14:01:14.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:14.073 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:14.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:14.087 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-03-30T14:01:14.129 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:13 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/911198374' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-30T14:01:14.130 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:13 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-30T14:01:14.130 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:13 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 42/594 objects degraded (7.071%), 6 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-03-30T14:01:14.130 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:13 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-30T14:01:14.130 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:13 smithi113 ceph-mon[34341]: osdmap e380: 8 total, 7 up, 7 in 2024-03-30T14:01:14.130 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:13 smithi113 ceph-mon[34341]: 5.3 scrub starts 2024-03-30T14:01:14.130 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:13 smithi113 ceph-mon[34341]: 5.3 scrub ok 2024-03-30T14:01:14.130 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:13 smithi113 ceph-mon[34341]: 4.b scrub starts 2024-03-30T14:01:14.130 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:13 smithi113 ceph-mon[34341]: 4.b scrub ok 2024-03-30T14:01:14.130 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:13 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:01:14.130 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:13 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:01:14.130 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:13 smithi113 ceph-mon[34341]: osdmap e381: 8 total, 7 up, 7 in 2024-03-30T14:01:14.175 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:14.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:14.200 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:14.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:14.301 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:14.326 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:14.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:14.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:14.428 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:14.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:14.452 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:14.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:14.553 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:14.566 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 5] out_osds: [7] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-30T14:01:14.566 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T14:01:14.566 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-03-30T14:01:14.567 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2024-03-30T14:01:14.567 DEBUG:teuthology.orchestra.run.smithi043:osd.2> sudo journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.2.service 2024-03-30T14:01:14.569 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:01:14.571 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl start ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.2 2024-03-30T14:01:14.579 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:14.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:14.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:14.642 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:14 smithi043 systemd[1]: Starting Ceph osd.2 for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:01:14.682 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:14.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:14.707 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:14.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:14.808 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:14.816 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T14:01:14.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:14.833 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:14.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:14.934 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:14.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:14.960 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:14.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:15.061 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:15.088 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:15.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:15.170 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:14 smithi113 ceph-mon[34341]: pgmap v835: 168 pgs: 5 creating+peering, 11 unknown, 4 peering, 99 active+clean, 12 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 609 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 419 B/s rd, 0 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:15.170 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:14 smithi113 ceph-mon[34341]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:15.170 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:14 smithi113 ceph-mon[34341]: 4.a scrub starts 2024-03-30T14:01:15.170 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:14 smithi113 ceph-mon[34341]: 4.a scrub ok 2024-03-30T14:01:15.170 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:14 smithi113 ceph-mon[34341]: 1.9 scrub starts 2024-03-30T14:01:15.170 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:14 smithi113 ceph-mon[34341]: 1.9 scrub ok 2024-03-30T14:01:15.170 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:14 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:01:15.170 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:14 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:15.170 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:14 smithi113 ceph-mon[34341]: osdmap e382: 8 total, 7 up, 7 in 2024-03-30T14:01:15.189 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:15.217 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:15.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:15.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:15.318 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:15.340 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:14 smithi043 ceph-mon[42099]: pgmap v835: 168 pgs: 5 creating+peering, 11 unknown, 4 peering, 99 active+clean, 12 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 609 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 419 B/s rd, 0 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:15.340 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:14 smithi043 ceph-mon[42099]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:15.340 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:14 smithi043 ceph-mon[42099]: 4.a scrub starts 2024-03-30T14:01:15.340 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:14 smithi043 ceph-mon[42099]: 4.a scrub ok 2024-03-30T14:01:15.340 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:14 smithi043 ceph-mon[42099]: 1.9 scrub starts 2024-03-30T14:01:15.340 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:14 smithi043 ceph-mon[42099]: 1.9 scrub ok 2024-03-30T14:01:15.340 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:14 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:01:15.341 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:14 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:15.341 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:14 smithi043 ceph-mon[42099]: osdmap e382: 8 total, 7 up, 7 in 2024-03-30T14:01:15.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:14 smithi043 ceph-mon[29963]: pgmap v835: 168 pgs: 5 creating+peering, 11 unknown, 4 peering, 99 active+clean, 12 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 609 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 419 B/s rd, 0 op/s; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:15.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:14 smithi043 ceph-mon[29963]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:15.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:14 smithi043 ceph-mon[29963]: 4.a scrub starts 2024-03-30T14:01:15.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:14 smithi043 ceph-mon[29963]: 4.a scrub ok 2024-03-30T14:01:15.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:14 smithi043 ceph-mon[29963]: 1.9 scrub starts 2024-03-30T14:01:15.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:14 smithi043 ceph-mon[29963]: 1.9 scrub ok 2024-03-30T14:01:15.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:14 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:01:15.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:14 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:15.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:14 smithi043 ceph-mon[29963]: osdmap e382: 8 total, 7 up, 7 in 2024-03-30T14:01:15.343 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:15.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:15.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:15.406 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T14:01:15.406 INFO:teuthology.orchestra.run.smithi113.stdout: "size": 20, 2024-03-30T14:01:15.406 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 600, 2024-03-30T14:01:15.406 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [ 2024-03-30T14:01:15.406 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.406 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:61215 9.e 9:75beb107:::benchmark_data_smithi043_145506_object61214:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e230)", 2024-03-30T14:01:15.406 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:37.070928+0000", 2024-03-30T14:01:15.406 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 458.33825199099999, 2024-03-30T14:01:15.406 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.38644029000000002, 2024-03-30T14:01:15.407 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:15.407 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:15.407 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:15.407 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:15.407 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T14:01:15.407 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:01:15.407 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 61215 2024-03-30T14:01:15.407 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.407 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:15.407 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.407 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:15.407 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:37.070928+0000", 2024-03-30T14:01:15.407 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.407 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.408 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.408 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:15.408 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:37.070928+0000", 2024-03-30T14:01:15.408 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.408 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.408 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.408 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:15.408 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:37.070930+0000", 2024-03-30T14:01:15.408 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1149999999999999e-06 2024-03-30T14:01:15.408 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.408 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.408 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:15.408 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:37.071099+0000", 2024-03-30T14:01:15.409 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00016937399999999999 2024-03-30T14:01:15.409 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.409 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.409 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:15.409 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:37.071108+0000", 2024-03-30T14:01:15.409 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.3209999999999994e-06 2024-03-30T14:01:15.409 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.409 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.409 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:15.409 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:37.071118+0000", 2024-03-30T14:01:15.409 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.3000000000000007e-06 2024-03-30T14:01:15.409 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.409 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.409 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.410 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:37.071169+0000", 2024-03-30T14:01:15.410 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1598999999999998e-05 2024-03-30T14:01:15.410 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.410 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.410 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:15.410 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:37.071176+0000", 2024-03-30T14:01:15.410 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.2819999999999999e-06 2024-03-30T14:01:15.410 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.410 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.410 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.413 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:37.435350+0000", 2024-03-30T14:01:15.413 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.36417401100000002 2024-03-30T14:01:15.414 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.414 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.414 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:15.414 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:37.436267+0000", 2024-03-30T14:01:15.414 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00091653500000000001 2024-03-30T14:01:15.414 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.414 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.415 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1", 2024-03-30T14:01:15.415 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:37.436389+0000", 2024-03-30T14:01:15.415 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000121791 2024-03-30T14:01:15.415 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.415 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.415 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:15.415 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:37.441470+0000", 2024-03-30T14:01:15.415 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0050813660000000004 2024-03-30T14:01:15.415 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.415 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.415 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.415 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:37.449103+0000", 2024-03-30T14:01:15.415 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0076323889999999998 2024-03-30T14:01:15.416 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.416 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.416 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.416 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:37.456004+0000", 2024-03-30T14:01:15.416 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0069016119999999997 2024-03-30T14:01:15.416 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.416 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.416 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:15.416 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:37.456061+0000", 2024-03-30T14:01:15.416 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7185000000000002e-05 2024-03-30T14:01:15.416 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.416 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.416 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:15.417 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:37.457369+0000", 2024-03-30T14:01:15.417 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00130741 2024-03-30T14:01:15.417 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.417 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.417 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.417 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.417 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.417 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:88646 9.e 9:7f352dd0:::benchmark_data_smithi043_145506_object1830:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-30T14:01:15.417 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:54:25.713380+0000", 2024-03-30T14:01:15.417 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 409.69580045499998, 2024-03-30T14:01:15.417 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0218103080000001, 2024-03-30T14:01:15.417 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:15.417 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:15.417 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:15.418 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:15.418 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T14:01:15.418 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:01:15.418 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 88646 2024-03-30T14:01:15.418 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.418 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:15.418 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.418 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:15.418 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.713380+0000", 2024-03-30T14:01:15.418 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.418 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.418 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.418 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:15.418 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.713380+0000", 2024-03-30T14:01:15.418 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.419 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.419 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.419 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:15.419 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.713383+0000", 2024-03-30T14:01:15.419 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.8499999999999998e-06 2024-03-30T14:01:15.419 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.419 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.419 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:15.419 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.713396+0000", 2024-03-30T14:01:15.419 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3457e-05 2024-03-30T14:01:15.419 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.419 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.419 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:15.419 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.713437+0000", 2024-03-30T14:01:15.420 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.0899000000000003e-05 2024-03-30T14:01:15.420 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.420 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.420 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:15.420 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.713469+0000", 2024-03-30T14:01:15.420 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2069000000000001e-05 2024-03-30T14:01:15.420 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.420 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.420 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.420 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.713600+0000", 2024-03-30T14:01:15.420 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013088600000000001 2024-03-30T14:01:15.420 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.420 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.420 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:15.421 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.713609+0000", 2024-03-30T14:01:15.421 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.9800000000000004e-06 2024-03-30T14:01:15.421 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.421 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.421 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.421 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.709324+0000", 2024-03-30T14:01:15.421 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.99571526600000004 2024-03-30T14:01:15.421 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.421 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.421 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:15.421 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.709455+0000", 2024-03-30T14:01:15.421 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013091599999999999 2024-03-30T14:01:15.421 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.421 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.422 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1", 2024-03-30T14:01:15.422 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.709591+0000", 2024-03-30T14:01:15.422 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000135885 2024-03-30T14:01:15.422 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.422 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.422 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:15.422 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.718980+0000", 2024-03-30T14:01:15.422 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0093891809999999999 2024-03-30T14:01:15.422 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.422 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.422 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.422 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.719962+0000", 2024-03-30T14:01:15.422 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00098184899999999992 2024-03-30T14:01:15.422 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.423 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.423 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.423 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.735117+0000", 2024-03-30T14:01:15.423 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.015154993 2024-03-30T14:01:15.423 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.423 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.423 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:15.423 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.735157+0000", 2024-03-30T14:01:15.423 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.0234999999999997e-05 2024-03-30T14:01:15.423 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.423 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.423 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:15.423 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.735190+0000", 2024-03-30T14:01:15.424 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2842000000000002e-05 2024-03-30T14:01:15.424 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.424 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.424 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.424 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.424 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.424 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:88652 9.e 9:7923d925:::benchmark_data_smithi043_145506_object1836:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-30T14:01:15.424 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:54:25.722178+0000", 2024-03-30T14:01:15.424 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 409.68700275999998, 2024-03-30T14:01:15.424 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0131422109999999, 2024-03-30T14:01:15.424 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:15.425 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:15.425 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:15.425 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:15.425 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T14:01:15.425 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:01:15.425 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 88652 2024-03-30T14:01:15.425 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.426 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:15.426 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.426 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:15.426 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.722178+0000", 2024-03-30T14:01:15.426 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.426 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.426 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.426 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:15.426 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.722178+0000", 2024-03-30T14:01:15.426 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.426 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.426 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.426 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:15.426 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.722180+0000", 2024-03-30T14:01:15.427 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.5600000000000001e-06 2024-03-30T14:01:15.427 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.427 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.427 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:15.427 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.722192+0000", 2024-03-30T14:01:15.427 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1677e-05 2024-03-30T14:01:15.427 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.427 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.427 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:15.427 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.722255+0000", 2024-03-30T14:01:15.427 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3292000000000003e-05 2024-03-30T14:01:15.427 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.428 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.428 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:15.428 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.722265+0000", 2024-03-30T14:01:15.428 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4280000000000002e-06 2024-03-30T14:01:15.428 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.428 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.428 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.428 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.722322+0000", 2024-03-30T14:01:15.428 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7160000000000002e-05 2024-03-30T14:01:15.428 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.428 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.428 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:15.428 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.722330+0000", 2024-03-30T14:01:15.428 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.0209999999999999e-06 2024-03-30T14:01:15.429 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.429 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.429 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.429 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.711378+0000", 2024-03-30T14:01:15.429 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.98904863399999998 2024-03-30T14:01:15.429 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.429 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.429 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:15.429 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.711492+0000", 2024-03-30T14:01:15.429 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011352400000000001 2024-03-30T14:01:15.429 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.429 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.429 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1", 2024-03-30T14:01:15.430 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.711611+0000", 2024-03-30T14:01:15.430 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000118793 2024-03-30T14:01:15.430 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.430 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.430 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:15.430 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.720045+0000", 2024-03-30T14:01:15.430 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0084338999999999994 2024-03-30T14:01:15.430 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.430 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.430 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.430 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.729727+0000", 2024-03-30T14:01:15.430 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0096822320000000007 2024-03-30T14:01:15.430 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.430 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.431 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.431 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.735248+0000", 2024-03-30T14:01:15.431 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0055207880000000004 2024-03-30T14:01:15.431 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.431 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.431 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:15.431 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.735294+0000", 2024-03-30T14:01:15.431 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6244e-05 2024-03-30T14:01:15.431 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.431 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.431 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:15.431 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.735320+0000", 2024-03-30T14:01:15.431 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.5958000000000001e-05 2024-03-30T14:01:15.431 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.432 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.432 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.432 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.432 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.432 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:88666 9.e 9:7d6303d0:::benchmark_data_smithi043_145506_object1850:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-30T14:01:15.432 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:54:25.747532+0000", 2024-03-30T14:01:15.432 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 409.66164870400002, 2024-03-30T14:01:15.432 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.98789200899999996, 2024-03-30T14:01:15.432 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:15.432 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:15.432 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:15.432 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:15.432 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T14:01:15.432 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:01:15.432 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 88666 2024-03-30T14:01:15.433 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.433 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:15.433 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.433 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:15.433 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.747532+0000", 2024-03-30T14:01:15.433 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.433 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.433 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.433 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:15.433 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.747532+0000", 2024-03-30T14:01:15.433 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.433 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.433 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.433 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:15.433 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.747533+0000", 2024-03-30T14:01:15.434 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3599999999999999e-06 2024-03-30T14:01:15.434 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.434 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.434 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:15.434 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.747539+0000", 2024-03-30T14:01:15.434 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6890000000000001e-06 2024-03-30T14:01:15.434 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.434 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.434 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:15.434 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.747550+0000", 2024-03-30T14:01:15.434 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1644999999999999e-05 2024-03-30T14:01:15.434 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.434 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.435 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:15.435 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.747557+0000", 2024-03-30T14:01:15.435 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.404e-06 2024-03-30T14:01:15.435 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.435 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.435 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.435 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.747608+0000", 2024-03-30T14:01:15.435 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0816000000000003e-05 2024-03-30T14:01:15.436 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.436 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.436 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:15.436 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.747616+0000", 2024-03-30T14:01:15.436 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.4600000000000003e-06 2024-03-30T14:01:15.436 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.436 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.436 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.436 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.712517+0000", 2024-03-30T14:01:15.436 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.96490127299999995 2024-03-30T14:01:15.436 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.436 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.436 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:15.436 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.712621+0000", 2024-03-30T14:01:15.437 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000103485 2024-03-30T14:01:15.437 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.437 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.437 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1", 2024-03-30T14:01:15.437 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.712733+0000", 2024-03-30T14:01:15.437 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011229499999999999 2024-03-30T14:01:15.437 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.437 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.437 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:15.437 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.720052+0000", 2024-03-30T14:01:15.437 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0073183700000000003 2024-03-30T14:01:15.437 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.437 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.437 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.438 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.729768+0000", 2024-03-30T14:01:15.438 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0097160500000000004 2024-03-30T14:01:15.438 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.438 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.438 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.438 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.735349+0000", 2024-03-30T14:01:15.438 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0055810579999999999 2024-03-30T14:01:15.438 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.439 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.439 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:15.440 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.735378+0000", 2024-03-30T14:01:15.440 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.9836e-05 2024-03-30T14:01:15.440 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.440 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.440 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:15.440 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.735424+0000", 2024-03-30T14:01:15.440 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.5268000000000001e-05 2024-03-30T14:01:15.440 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.440 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.440 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.440 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.440 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.440 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:88676 9.e 9:75c69ca6:::benchmark_data_smithi043_145506_object1860:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-30T14:01:15.441 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:54:25.758029+0000", 2024-03-30T14:01:15.441 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 409.65115161799997, 2024-03-30T14:01:15.441 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.98168357900000003, 2024-03-30T14:01:15.441 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:15.441 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:15.441 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:15.441 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:15.441 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T14:01:15.441 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:01:15.441 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 88676 2024-03-30T14:01:15.441 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.441 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:15.441 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.441 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:15.442 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.758029+0000", 2024-03-30T14:01:15.442 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.442 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.442 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.442 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:15.442 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.758029+0000", 2024-03-30T14:01:15.442 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.442 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.442 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.442 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:15.442 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.758030+0000", 2024-03-30T14:01:15.442 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.6080000000000001e-06 2024-03-30T14:01:15.443 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.443 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.443 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:15.443 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.758037+0000", 2024-03-30T14:01:15.443 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5030000000000002e-06 2024-03-30T14:01:15.443 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.443 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.443 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:15.443 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.758050+0000", 2024-03-30T14:01:15.443 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3033e-05 2024-03-30T14:01:15.443 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.443 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.443 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:15.443 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.758058+0000", 2024-03-30T14:01:15.444 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.0539999999999995e-06 2024-03-30T14:01:15.444 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.444 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.444 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.444 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.758136+0000", 2024-03-30T14:01:15.444 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8448000000000005e-05 2024-03-30T14:01:15.444 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.444 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.444 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:15.444 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.758148+0000", 2024-03-30T14:01:15.444 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1671999999999999e-05 2024-03-30T14:01:15.444 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.444 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.445 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.445 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.713563+0000", 2024-03-30T14:01:15.445 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.95541482099999997 2024-03-30T14:01:15.445 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.445 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.445 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:15.445 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.713666+0000", 2024-03-30T14:01:15.445 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00010283 2024-03-30T14:01:15.445 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.445 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.445 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1", 2024-03-30T14:01:15.445 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.713781+0000", 2024-03-30T14:01:15.445 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000115525 2024-03-30T14:01:15.445 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.446 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.446 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:15.446 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.720057+0000", 2024-03-30T14:01:15.446 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0062757849999999999 2024-03-30T14:01:15.446 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.446 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.446 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.446 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.729819+0000", 2024-03-30T14:01:15.446 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0097619409999999997 2024-03-30T14:01:15.446 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.446 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.446 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.446 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.739608+0000", 2024-03-30T14:01:15.447 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0097886919999999999 2024-03-30T14:01:15.447 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.447 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.447 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:15.447 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.739663+0000", 2024-03-30T14:01:15.447 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4824000000000002e-05 2024-03-30T14:01:15.447 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.447 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.447 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:15.447 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.739712+0000", 2024-03-30T14:01:15.447 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9842999999999997e-05 2024-03-30T14:01:15.447 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.447 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.447 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.448 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.448 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.448 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:88680 9.e 9:7ce7ae95:::benchmark_data_smithi043_145506_object1864:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-30T14:01:15.448 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:54:25.764810+0000", 2024-03-30T14:01:15.448 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 409.644370638, 2024-03-30T14:01:15.448 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.97502488099999995, 2024-03-30T14:01:15.448 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:15.448 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:15.448 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:15.448 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:15.448 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T14:01:15.448 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:01:15.448 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 88680 2024-03-30T14:01:15.449 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.449 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:15.449 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.449 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:15.449 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.764810+0000", 2024-03-30T14:01:15.449 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.449 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.449 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.449 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:15.449 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.764810+0000", 2024-03-30T14:01:15.449 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.449 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.449 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.450 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:15.450 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.764812+0000", 2024-03-30T14:01:15.450 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.4559999999999999e-06 2024-03-30T14:01:15.450 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.450 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.450 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:15.450 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.764824+0000", 2024-03-30T14:01:15.450 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.134e-05 2024-03-30T14:01:15.450 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.450 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.450 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:15.450 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.764855+0000", 2024-03-30T14:01:15.450 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.1510000000000002e-05 2024-03-30T14:01:15.450 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.451 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.451 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:15.451 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.764880+0000", 2024-03-30T14:01:15.451 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.5082000000000001e-05 2024-03-30T14:01:15.451 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.451 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.451 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.451 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.764938+0000", 2024-03-30T14:01:15.451 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7574999999999997e-05 2024-03-30T14:01:15.451 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.451 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.451 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:15.451 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.764947+0000", 2024-03-30T14:01:15.451 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.2599999999999994e-06 2024-03-30T14:01:15.451 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.452 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.452 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.452 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.714591+0000", 2024-03-30T14:01:15.452 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.94964393300000005 2024-03-30T14:01:15.452 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.452 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.452 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:15.452 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.714697+0000", 2024-03-30T14:01:15.452 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00010566899999999999 2024-03-30T14:01:15.452 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.452 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.452 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1", 2024-03-30T14:01:15.452 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.714810+0000", 2024-03-30T14:01:15.452 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011309799999999999 2024-03-30T14:01:15.452 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.453 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.453 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:15.453 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.720064+0000", 2024-03-30T14:01:15.453 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.005254285 2024-03-30T14:01:15.453 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.453 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.453 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.453 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.729856+0000", 2024-03-30T14:01:15.453 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0097919470000000005 2024-03-30T14:01:15.453 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.453 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.454 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.454 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.739750+0000", 2024-03-30T14:01:15.454 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0098936149999999997 2024-03-30T14:01:15.454 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.454 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.454 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:15.454 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.739794+0000", 2024-03-30T14:01:15.454 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.4894000000000001e-05 2024-03-30T14:01:15.454 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.454 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.454 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:15.454 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.739835+0000", 2024-03-30T14:01:15.454 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.0216999999999998e-05 2024-03-30T14:01:15.455 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.455 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.455 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.455 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.455 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.455 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:88712 9.e 9:708649af:::benchmark_data_smithi043_145506_object1896:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-30T14:01:15.455 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:54:25.823086+0000", 2024-03-30T14:01:15.455 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 409.58609475899999, 2024-03-30T14:01:15.455 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.91686288599999999, 2024-03-30T14:01:15.455 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:15.455 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:15.455 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:15.455 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:15.456 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T14:01:15.456 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:01:15.456 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 88712 2024-03-30T14:01:15.456 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.456 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:15.456 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.456 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:15.456 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.823086+0000", 2024-03-30T14:01:15.456 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.456 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.456 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.456 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:15.456 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.823086+0000", 2024-03-30T14:01:15.456 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.457 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.457 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.457 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:15.457 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.823087+0000", 2024-03-30T14:01:15.457 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.311e-06 2024-03-30T14:01:15.457 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.457 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.457 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:15.457 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.823092+0000", 2024-03-30T14:01:15.457 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3480000000000003e-06 2024-03-30T14:01:15.457 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.457 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.457 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:15.458 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.823104+0000", 2024-03-30T14:01:15.458 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2043e-05 2024-03-30T14:01:15.458 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.458 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.458 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:15.458 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.823111+0000", 2024-03-30T14:01:15.458 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5100000000000004e-06 2024-03-30T14:01:15.458 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.458 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.458 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.458 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.823168+0000", 2024-03-30T14:01:15.458 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7093999999999999e-05 2024-03-30T14:01:15.458 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.459 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.459 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:15.459 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.823176+0000", 2024-03-30T14:01:15.459 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.371e-06 2024-03-30T14:01:15.459 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.459 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.459 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.460 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.715620+0000", 2024-03-30T14:01:15.460 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.89244399799999996 2024-03-30T14:01:15.460 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.460 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.460 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:15.460 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.715723+0000", 2024-03-30T14:01:15.460 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000103074 2024-03-30T14:01:15.460 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.461 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.461 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1", 2024-03-30T14:01:15.461 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.715837+0000", 2024-03-30T14:01:15.461 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000113139 2024-03-30T14:01:15.461 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.461 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.461 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:15.461 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.720070+0000", 2024-03-30T14:01:15.461 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0042329339999999998 2024-03-30T14:01:15.461 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.461 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.461 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.462 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.732165+0000", 2024-03-30T14:01:15.462 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.012095257 2024-03-30T14:01:15.462 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.462 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.462 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.462 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.739868+0000", 2024-03-30T14:01:15.462 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0077030429999999997 2024-03-30T14:01:15.462 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.462 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.462 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:15.462 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.739913+0000", 2024-03-30T14:01:15.462 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.4815999999999999e-05 2024-03-30T14:01:15.462 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.462 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.463 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:15.463 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.739949+0000", 2024-03-30T14:01:15.463 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.5948000000000002e-05 2024-03-30T14:01:15.463 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.463 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.463 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.463 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.463 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.463 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:88736 9.e 9:70fc24a2:::benchmark_data_smithi043_145506_object1920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-30T14:01:15.463 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:54:25.866474+0000", 2024-03-30T14:01:15.463 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 409.54270674000003, 2024-03-30T14:01:15.463 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.87358530899999998, 2024-03-30T14:01:15.463 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:15.463 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:15.464 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:15.464 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:15.464 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T14:01:15.464 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:01:15.464 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 88736 2024-03-30T14:01:15.464 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.464 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:15.464 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.464 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:15.464 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.866474+0000", 2024-03-30T14:01:15.464 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.464 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.464 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.464 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:15.464 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.866474+0000", 2024-03-30T14:01:15.465 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.465 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.465 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.465 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:15.465 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.866475+0000", 2024-03-30T14:01:15.465 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5969999999999999e-06 2024-03-30T14:01:15.465 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.465 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.465 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:15.465 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.866482+0000", 2024-03-30T14:01:15.465 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.9330000000000004e-06 2024-03-30T14:01:15.465 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.465 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.465 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:15.466 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.866498+0000", 2024-03-30T14:01:15.466 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5372000000000001e-05 2024-03-30T14:01:15.466 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.466 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.466 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:15.466 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.866507+0000", 2024-03-30T14:01:15.466 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.0729999999999994e-06 2024-03-30T14:01:15.466 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.466 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.466 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.466 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.866562+0000", 2024-03-30T14:01:15.466 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5565000000000001e-05 2024-03-30T14:01:15.466 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.466 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.467 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:15.467 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.866571+0000", 2024-03-30T14:01:15.467 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.8750000000000006e-06 2024-03-30T14:01:15.467 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.467 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.467 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.467 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.716664+0000", 2024-03-30T14:01:15.467 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.85009269099999996 2024-03-30T14:01:15.467 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.467 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.467 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:15.467 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.716765+0000", 2024-03-30T14:01:15.467 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000100902 2024-03-30T14:01:15.467 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.467 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.468 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1", 2024-03-30T14:01:15.468 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.716878+0000", 2024-03-30T14:01:15.468 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000112929 2024-03-30T14:01:15.468 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.468 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.468 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:15.468 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.720076+0000", 2024-03-30T14:01:15.468 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0031987130000000002 2024-03-30T14:01:15.468 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.468 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.468 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.468 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.732208+0000", 2024-03-30T14:01:15.468 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.012131179000000001 2024-03-30T14:01:15.468 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.469 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.469 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.469 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.739980+0000", 2024-03-30T14:01:15.469 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0077722360000000001 2024-03-30T14:01:15.469 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.469 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.469 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:15.469 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.740026+0000", 2024-03-30T14:01:15.469 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6560999999999997e-05 2024-03-30T14:01:15.469 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.469 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.469 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:15.469 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.740059+0000", 2024-03-30T14:01:15.469 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2682999999999999e-05 2024-03-30T14:01:15.470 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.470 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.470 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.470 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.470 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.470 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:88744 9.e 9:70df637d:::benchmark_data_smithi043_145506_object1928:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-30T14:01:15.470 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:54:25.880074+0000", 2024-03-30T14:01:15.470 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 409.52910648699998, 2024-03-30T14:01:15.470 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.86008798799999997, 2024-03-30T14:01:15.470 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:15.470 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:15.470 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:15.470 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:15.470 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T14:01:15.471 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:01:15.471 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 88744 2024-03-30T14:01:15.471 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.471 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:15.471 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.471 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:15.471 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.880074+0000", 2024-03-30T14:01:15.471 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.471 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.471 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.471 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:15.471 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.880074+0000", 2024-03-30T14:01:15.471 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.472 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.472 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.472 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:15.472 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.880075+0000", 2024-03-30T14:01:15.472 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.463e-06 2024-03-30T14:01:15.472 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.472 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.472 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:15.472 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.880081+0000", 2024-03-30T14:01:15.472 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0290000000000002e-06 2024-03-30T14:01:15.472 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.472 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.472 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:15.473 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.880095+0000", 2024-03-30T14:01:15.473 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4034e-05 2024-03-30T14:01:15.473 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.473 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.473 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:15.473 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.880104+0000", 2024-03-30T14:01:15.473 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.0150000000000003e-06 2024-03-30T14:01:15.473 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.473 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.473 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.473 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.880166+0000", 2024-03-30T14:01:15.473 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.2321000000000002e-05 2024-03-30T14:01:15.473 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.473 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.473 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:15.474 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.880175+0000", 2024-03-30T14:01:15.474 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6759999999999996e-06 2024-03-30T14:01:15.474 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.474 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.474 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.474 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.717763+0000", 2024-03-30T14:01:15.474 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.83758886499999996 2024-03-30T14:01:15.474 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.474 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.474 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:15.474 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.717848+0000", 2024-03-30T14:01:15.474 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.4862000000000002e-05 2024-03-30T14:01:15.474 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.475 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.475 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1", 2024-03-30T14:01:15.475 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.717967+0000", 2024-03-30T14:01:15.475 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000118775 2024-03-30T14:01:15.475 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.475 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.475 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:15.475 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.722272+0000", 2024-03-30T14:01:15.475 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.004305143 2024-03-30T14:01:15.475 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.475 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.475 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.475 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.735586+0000", 2024-03-30T14:01:15.476 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.013314286999999999 2024-03-30T14:01:15.476 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.476 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.476 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.476 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.740089+0000", 2024-03-30T14:01:15.476 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0045029609999999998 2024-03-30T14:01:15.476 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.476 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.476 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:15.476 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.740130+0000", 2024-03-30T14:01:15.476 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.0669e-05 2024-03-30T14:01:15.476 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.476 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.476 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:15.477 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.740162+0000", 2024-03-30T14:01:15.477 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.1887999999999997e-05 2024-03-30T14:01:15.477 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.477 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.477 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.477 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.477 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.477 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:88745 9.e 9:73c5959d:::benchmark_data_smithi043_145506_object1929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-30T14:01:15.477 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:54:25.882317+0000", 2024-03-30T14:01:15.477 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 409.52686299099997, 2024-03-30T14:01:15.477 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.85797838999999998, 2024-03-30T14:01:15.477 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:15.477 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:15.477 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:15.478 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:15.478 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T14:01:15.478 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:01:15.478 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 88745 2024-03-30T14:01:15.478 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.478 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:15.478 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.478 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:15.478 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.882317+0000", 2024-03-30T14:01:15.478 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.478 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.478 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.478 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:15.479 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.882317+0000", 2024-03-30T14:01:15.479 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.479 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.479 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.479 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:15.479 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.882319+0000", 2024-03-30T14:01:15.479 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.8330000000000001e-06 2024-03-30T14:01:15.479 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.479 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.479 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:15.479 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.882329+0000", 2024-03-30T14:01:15.479 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4830000000000005e-06 2024-03-30T14:01:15.479 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.479 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.480 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:15.480 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.882357+0000", 2024-03-30T14:01:15.480 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.7843000000000001e-05 2024-03-30T14:01:15.480 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.480 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.480 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:15.480 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.882372+0000", 2024-03-30T14:01:15.480 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4912999999999999e-05 2024-03-30T14:01:15.480 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.480 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.480 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.480 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.882536+0000", 2024-03-30T14:01:15.480 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00016425399999999999 2024-03-30T14:01:15.480 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.480 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.481 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:15.481 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.882553+0000", 2024-03-30T14:01:15.481 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.6956000000000002e-05 2024-03-30T14:01:15.481 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.481 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.481 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.481 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.719032+0000", 2024-03-30T14:01:15.481 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.83647957100000003 2024-03-30T14:01:15.481 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.481 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.481 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:15.481 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.719106+0000", 2024-03-30T14:01:15.481 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.3267000000000002e-05 2024-03-30T14:01:15.481 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.482 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.482 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1", 2024-03-30T14:01:15.482 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.719170+0000", 2024-03-30T14:01:15.482 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.4370999999999997e-05 2024-03-30T14:01:15.482 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.482 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.482 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:15.482 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.722290+0000", 2024-03-30T14:01:15.482 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0031196689999999998 2024-03-30T14:01:15.482 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.482 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.482 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.482 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.739019+0000", 2024-03-30T14:01:15.482 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.016729732000000001 2024-03-30T14:01:15.483 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.483 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.483 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.483 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.740212+0000", 2024-03-30T14:01:15.483 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0011922250000000001 2024-03-30T14:01:15.483 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.483 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.483 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:15.483 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.740262+0000", 2024-03-30T14:01:15.483 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0071999999999998e-05 2024-03-30T14:01:15.483 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.483 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.483 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:15.483 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.740296+0000", 2024-03-30T14:01:15.484 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.4201e-05 2024-03-30T14:01:15.484 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.484 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.484 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.484 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.484 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.484 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:88788 9.e 9:712f49b3:::benchmark_data_smithi043_145506_object1972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-30T14:01:15.484 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:54:25.963070+0000", 2024-03-30T14:01:15.484 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 409.44611054699999, 2024-03-30T14:01:15.484 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.786353576, 2024-03-30T14:01:15.484 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:15.484 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:15.484 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:15.484 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:15.485 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T14:01:15.485 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:01:15.485 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 88788 2024-03-30T14:01:15.485 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.485 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:15.485 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.485 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:15.485 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.963070+0000", 2024-03-30T14:01:15.485 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.485 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.485 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.485 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:15.485 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.963070+0000", 2024-03-30T14:01:15.485 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.486 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.486 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.486 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:15.486 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.963071+0000", 2024-03-30T14:01:15.486 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2810000000000001e-06 2024-03-30T14:01:15.486 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.486 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.486 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:15.486 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.963078+0000", 2024-03-30T14:01:15.486 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3829999999999997e-06 2024-03-30T14:01:15.486 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.486 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.486 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:15.486 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.963092+0000", 2024-03-30T14:01:15.487 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3946e-05 2024-03-30T14:01:15.487 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.487 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.487 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:15.487 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.963099+0000", 2024-03-30T14:01:15.487 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.4909999999999999e-06 2024-03-30T14:01:15.487 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.487 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.487 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.487 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.963156+0000", 2024-03-30T14:01:15.487 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6619999999999997e-05 2024-03-30T14:01:15.487 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.487 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.487 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:15.488 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:25.963165+0000", 2024-03-30T14:01:15.488 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4720000000000001e-06 2024-03-30T14:01:15.488 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.488 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.488 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.488 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.720092+0000", 2024-03-30T14:01:15.488 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.75692646699999999 2024-03-30T14:01:15.488 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.488 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.488 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:15.488 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.720152+0000", 2024-03-30T14:01:15.488 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0522000000000001e-05 2024-03-30T14:01:15.488 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.488 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.489 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1", 2024-03-30T14:01:15.489 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.720216+0000", 2024-03-30T14:01:15.489 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3459000000000003e-05 2024-03-30T14:01:15.489 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.489 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.489 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:15.489 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.724967+0000", 2024-03-30T14:01:15.489 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.004751145 2024-03-30T14:01:15.489 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.489 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.489 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.489 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.740329+0000", 2024-03-30T14:01:15.489 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.01536189 2024-03-30T14:01:15.489 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.490 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.490 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.490 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.749292+0000", 2024-03-30T14:01:15.490 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0089637220000000004 2024-03-30T14:01:15.490 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.490 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.490 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:15.490 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.749369+0000", 2024-03-30T14:01:15.490 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.6618000000000004e-05 2024-03-30T14:01:15.490 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.490 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.490 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:15.490 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.749423+0000", 2024-03-30T14:01:15.491 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4559999999999999e-05 2024-03-30T14:01:15.491 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.491 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.491 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.491 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.491 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.491 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:88829 9.e 9:77e96717:::benchmark_data_smithi043_145506_object2013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-30T14:01:15.491 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:54:26.032176+0000", 2024-03-30T14:01:15.491 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 409.37700438500002, 2024-03-30T14:01:15.491 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.71741295999999999, 2024-03-30T14:01:15.491 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:15.491 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:15.491 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:15.491 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:15.491 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T14:01:15.492 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:01:15.492 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 88829 2024-03-30T14:01:15.492 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.492 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:15.492 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.492 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:15.492 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.032176+0000", 2024-03-30T14:01:15.492 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.492 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.492 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.492 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:15.492 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.032176+0000", 2024-03-30T14:01:15.492 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.493 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.493 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.493 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:15.493 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.032177+0000", 2024-03-30T14:01:15.493 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2720000000000001e-06 2024-03-30T14:01:15.493 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.493 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.493 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:15.493 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.032192+0000", 2024-03-30T14:01:15.493 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4523e-05 2024-03-30T14:01:15.493 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.493 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.493 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:15.493 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.032203+0000", 2024-03-30T14:01:15.493 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1456e-05 2024-03-30T14:01:15.494 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.494 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.494 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:15.494 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.032209+0000", 2024-03-30T14:01:15.494 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0079999999999999e-06 2024-03-30T14:01:15.494 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.494 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.494 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.494 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.032270+0000", 2024-03-30T14:01:15.494 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0996999999999999e-05 2024-03-30T14:01:15.494 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.494 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.494 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:15.494 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.032279+0000", 2024-03-30T14:01:15.495 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.1419999999999999e-06 2024-03-30T14:01:15.495 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.495 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.495 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.495 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.720978+0000", 2024-03-30T14:01:15.495 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.68869877800000001 2024-03-30T14:01:15.495 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.495 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.495 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:15.495 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.721088+0000", 2024-03-30T14:01:15.495 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000110003 2024-03-30T14:01:15.495 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.495 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.495 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1", 2024-03-30T14:01:15.495 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.721206+0000", 2024-03-30T14:01:15.496 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000117563 2024-03-30T14:01:15.496 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.496 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.496 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:15.496 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.728071+0000", 2024-03-30T14:01:15.496 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0068647550000000002 2024-03-30T14:01:15.496 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.496 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.496 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.496 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.740364+0000", 2024-03-30T14:01:15.496 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.012293326 2024-03-30T14:01:15.496 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.496 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.496 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.497 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.749489+0000", 2024-03-30T14:01:15.497 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0091250700000000008 2024-03-30T14:01:15.497 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.497 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.497 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:15.497 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.749545+0000", 2024-03-30T14:01:15.497 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5628999999999999e-05 2024-03-30T14:01:15.497 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.497 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.497 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:15.497 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.749589+0000", 2024-03-30T14:01:15.497 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.4437999999999997e-05 2024-03-30T14:01:15.497 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.497 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.497 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.498 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.498 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.498 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:88872 9.e 9:721d1732:::benchmark_data_smithi043_145506_object2056:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-30T14:01:15.498 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:54:26.130690+0000", 2024-03-30T14:01:15.498 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 409.27849002900001, 2024-03-30T14:01:15.498 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.61903624199999996, 2024-03-30T14:01:15.498 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:15.498 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:15.498 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:15.498 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:15.498 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T14:01:15.498 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:01:15.498 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 88872 2024-03-30T14:01:15.498 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.499 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:15.499 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.499 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:15.499 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.130690+0000", 2024-03-30T14:01:15.499 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.499 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.499 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.499 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:15.499 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.130690+0000", 2024-03-30T14:01:15.499 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.499 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.499 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.499 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:15.499 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.130692+0000", 2024-03-30T14:01:15.500 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.7570000000000001e-06 2024-03-30T14:01:15.500 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.500 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.500 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:15.500 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.130699+0000", 2024-03-30T14:01:15.500 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.4130000000000003e-06 2024-03-30T14:01:15.500 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.500 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.500 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:15.500 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.130713+0000", 2024-03-30T14:01:15.500 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4055e-05 2024-03-30T14:01:15.500 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.500 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.500 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:15.501 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.130721+0000", 2024-03-30T14:01:15.501 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5059999999999995e-06 2024-03-30T14:01:15.501 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.501 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.501 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.501 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.130785+0000", 2024-03-30T14:01:15.501 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3602999999999994e-05 2024-03-30T14:01:15.501 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.501 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.501 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:15.501 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.130793+0000", 2024-03-30T14:01:15.501 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.7110000000000001e-06 2024-03-30T14:01:15.501 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.501 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.501 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.502 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.722326+0000", 2024-03-30T14:01:15.502 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.59153234899999996 2024-03-30T14:01:15.502 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.502 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.502 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:15.502 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.722439+0000", 2024-03-30T14:01:15.502 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000113498 2024-03-30T14:01:15.502 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.502 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.502 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1", 2024-03-30T14:01:15.502 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.722560+0000", 2024-03-30T14:01:15.502 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000120382 2024-03-30T14:01:15.502 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.502 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.503 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:15.503 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.728076+0000", 2024-03-30T14:01:15.503 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0055161660000000003 2024-03-30T14:01:15.503 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.503 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.503 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.503 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.741767+0000", 2024-03-30T14:01:15.503 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.013691258 2024-03-30T14:01:15.503 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.503 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.503 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.503 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.749635+0000", 2024-03-30T14:01:15.503 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0078674060000000004 2024-03-30T14:01:15.503 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.504 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.504 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:15.504 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.749689+0000", 2024-03-30T14:01:15.504 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4147999999999997e-05 2024-03-30T14:01:15.504 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.504 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.504 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:15.504 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.749727+0000", 2024-03-30T14:01:15.504 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.799e-05 2024-03-30T14:01:15.504 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.504 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.504 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.504 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.504 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.504 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:88892 9.e 9:7863f8a7:::benchmark_data_smithi043_145506_object2076:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-30T14:01:15.505 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:54:26.187656+0000", 2024-03-30T14:01:15.505 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 409.22152408199997, 2024-03-30T14:01:15.505 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.56218118399999994, 2024-03-30T14:01:15.505 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:15.505 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:15.505 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:15.505 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:15.505 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T14:01:15.505 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:01:15.505 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 88892 2024-03-30T14:01:15.505 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.505 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:15.505 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.505 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:15.505 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.187656+0000", 2024-03-30T14:01:15.506 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.506 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.506 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.506 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:15.506 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.187656+0000", 2024-03-30T14:01:15.506 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.506 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.506 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.506 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:15.506 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.187658+0000", 2024-03-30T14:01:15.506 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.206e-06 2024-03-30T14:01:15.506 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.506 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.506 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:15.507 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.187663+0000", 2024-03-30T14:01:15.507 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1479999999999997e-06 2024-03-30T14:01:15.507 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.507 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.507 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:15.507 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.187674+0000", 2024-03-30T14:01:15.507 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1219999999999999e-05 2024-03-30T14:01:15.507 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.507 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.507 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:15.507 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.187680+0000", 2024-03-30T14:01:15.507 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3260000000000001e-06 2024-03-30T14:01:15.507 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.507 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.508 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.508 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.187732+0000", 2024-03-30T14:01:15.508 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.189e-05 2024-03-30T14:01:15.508 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.508 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.508 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:15.508 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.187741+0000", 2024-03-30T14:01:15.508 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.4980000000000006e-06 2024-03-30T14:01:15.508 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.508 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.508 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.508 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.723747+0000", 2024-03-30T14:01:15.508 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.53600663400000004 2024-03-30T14:01:15.508 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.508 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.509 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:15.509 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.723862+0000", 2024-03-30T14:01:15.509 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000115103 2024-03-30T14:01:15.509 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.509 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.509 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1", 2024-03-30T14:01:15.509 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.723998+0000", 2024-03-30T14:01:15.509 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013554800000000001 2024-03-30T14:01:15.509 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.509 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.509 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:15.509 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.728084+0000", 2024-03-30T14:01:15.509 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0040865429999999998 2024-03-30T14:01:15.509 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.510 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.510 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.510 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.743824+0000", 2024-03-30T14:01:15.510 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.015739592 2024-03-30T14:01:15.510 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.510 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.510 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.510 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.749767+0000", 2024-03-30T14:01:15.510 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0059428759999999997 2024-03-30T14:01:15.510 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.510 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.510 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:15.510 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.749796+0000", 2024-03-30T14:01:15.510 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.9513999999999999e-05 2024-03-30T14:01:15.511 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.511 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.511 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:15.511 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.749838+0000", 2024-03-30T14:01:15.511 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1085999999999999e-05 2024-03-30T14:01:15.511 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.511 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.511 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.511 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.511 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.511 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:88903 9.e 9:79661009:::benchmark_data_smithi043_145506_object2087:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-30T14:01:15.511 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:54:26.210391+0000", 2024-03-30T14:01:15.511 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 409.198789347, 2024-03-30T14:01:15.511 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.53960190799999996, 2024-03-30T14:01:15.512 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:15.512 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:15.512 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:15.512 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:15.512 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T14:01:15.512 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:01:15.512 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 88903 2024-03-30T14:01:15.512 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.512 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:15.512 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.512 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:15.512 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.210391+0000", 2024-03-30T14:01:15.512 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.513 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.513 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.513 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:15.513 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.210391+0000", 2024-03-30T14:01:15.513 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.513 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.513 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.513 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:15.513 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.210393+0000", 2024-03-30T14:01:15.513 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5969999999999999e-06 2024-03-30T14:01:15.513 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.513 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.513 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:15.513 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.210399+0000", 2024-03-30T14:01:15.513 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.2720000000000003e-06 2024-03-30T14:01:15.514 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.514 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.514 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:15.514 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.210413+0000", 2024-03-30T14:01:15.514 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4344e-05 2024-03-30T14:01:15.514 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.514 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.514 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:15.514 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.210420+0000", 2024-03-30T14:01:15.514 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.888e-06 2024-03-30T14:01:15.514 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.514 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.514 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.515 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.210488+0000", 2024-03-30T14:01:15.515 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.7828000000000002e-05 2024-03-30T14:01:15.515 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.515 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.515 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:15.515 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.210500+0000", 2024-03-30T14:01:15.515 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1620000000000001e-05 2024-03-30T14:01:15.515 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.515 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.515 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.515 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.725013+0000", 2024-03-30T14:01:15.515 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.51451306799999996 2024-03-30T14:01:15.515 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.515 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.516 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:15.516 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.725121+0000", 2024-03-30T14:01:15.516 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000108356 2024-03-30T14:01:15.516 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.516 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.516 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1", 2024-03-30T14:01:15.516 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.725255+0000", 2024-03-30T14:01:15.516 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013375099999999999 2024-03-30T14:01:15.516 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.516 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.516 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:15.516 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.728091+0000", 2024-03-30T14:01:15.516 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0028364459999999998 2024-03-30T14:01:15.516 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.517 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.517 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.517 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.743859+0000", 2024-03-30T14:01:15.517 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.015767618000000001 2024-03-30T14:01:15.517 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.517 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.517 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.517 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.749898+0000", 2024-03-30T14:01:15.517 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0060387230000000002 2024-03-30T14:01:15.517 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.517 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.517 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:15.517 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.749952+0000", 2024-03-30T14:01:15.517 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4086999999999999e-05 2024-03-30T14:01:15.518 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.518 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.518 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:15.518 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.749993+0000", 2024-03-30T14:01:15.518 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1310000000000003e-05 2024-03-30T14:01:15.518 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.518 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.518 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.518 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.518 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.518 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:88926 9.e 9:7715f5bc:::benchmark_data_smithi043_145506_object2110:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-30T14:01:15.518 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:54:26.308210+0000", 2024-03-30T14:01:15.519 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 409.10097064500002, 2024-03-30T14:01:15.519 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.443989193, 2024-03-30T14:01:15.519 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:15.519 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:15.519 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:15.519 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:15.519 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T14:01:15.519 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:01:15.519 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 88926 2024-03-30T14:01:15.519 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.519 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:15.519 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.519 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:15.519 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.308210+0000", 2024-03-30T14:01:15.520 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.520 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.520 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.520 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:15.520 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.308210+0000", 2024-03-30T14:01:15.520 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.520 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.520 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.520 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:15.520 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.308211+0000", 2024-03-30T14:01:15.520 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.547e-06 2024-03-30T14:01:15.520 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.520 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.520 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:15.521 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.308217+0000", 2024-03-30T14:01:15.521 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.519e-06 2024-03-30T14:01:15.521 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.521 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.521 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:15.521 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.308241+0000", 2024-03-30T14:01:15.521 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.4029999999999999e-05 2024-03-30T14:01:15.521 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.521 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.521 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:15.521 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.308249+0000", 2024-03-30T14:01:15.521 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.4840000000000004e-06 2024-03-30T14:01:15.521 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.521 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.521 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.522 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.308300+0000", 2024-03-30T14:01:15.522 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0763000000000002e-05 2024-03-30T14:01:15.522 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.522 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.522 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:15.522 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.308309+0000", 2024-03-30T14:01:15.522 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.9700000000000005e-06 2024-03-30T14:01:15.522 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.522 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.522 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.522 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.726529+0000", 2024-03-30T14:01:15.522 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.41821949000000003 2024-03-30T14:01:15.522 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.522 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.523 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:15.523 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.726633+0000", 2024-03-30T14:01:15.523 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000104559 2024-03-30T14:01:15.523 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.523 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:15.524 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.524 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1", 2024-03-30T14:01:15.525 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.726748+0000", 2024-03-30T14:01:15.525 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000114333 2024-03-30T14:01:15.525 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.525 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.525 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:15.525 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.729333+0000", 2024-03-30T14:01:15.525 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0025854189999999998 2024-03-30T14:01:15.525 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.525 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.525 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.525 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.750052+0000", 2024-03-30T14:01:15.525 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.020718581 2024-03-30T14:01:15.525 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.526 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.526 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.526 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.752048+0000", 2024-03-30T14:01:15.526 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001996734 2024-03-30T14:01:15.526 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.526 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.526 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:15.526 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.752148+0000", 2024-03-30T14:01:15.526 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00010011899999999999 2024-03-30T14:01:15.526 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.526 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.526 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:15.526 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.752199+0000", 2024-03-30T14:01:15.527 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0645e-05 2024-03-30T14:01:15.527 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.527 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.527 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.527 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.527 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.527 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15354.0:88938 9.e 9:797820a5:::benchmark_data_smithi043_145506_object2122:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-30T14:01:15.527 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:54:26.444794+0000", 2024-03-30T14:01:15.527 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 408.964386741, 2024-03-30T14:01:15.527 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.30875860799999999, 2024-03-30T14:01:15.527 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:15.527 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:15.527 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:15.528 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:15.528 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15354", 2024-03-30T14:01:15.528 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:01:15.528 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 88938 2024-03-30T14:01:15.528 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.528 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:15.528 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.528 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:15.528 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.444794+0000", 2024-03-30T14:01:15.528 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.528 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.528 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.528 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:15.528 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.444794+0000", 2024-03-30T14:01:15.529 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.529 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.529 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.529 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:15.529 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.444795+0000", 2024-03-30T14:01:15.529 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3769999999999999e-06 2024-03-30T14:01:15.529 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.529 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.529 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:15.529 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.444800+0000", 2024-03-30T14:01:15.529 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2650000000000004e-06 2024-03-30T14:01:15.529 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.529 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.529 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:15.530 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.444813+0000", 2024-03-30T14:01:15.530 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.241e-05 2024-03-30T14:01:15.530 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.530 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.530 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:15.530 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.444819+0000", 2024-03-30T14:01:15.530 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.4239999999999998e-06 2024-03-30T14:01:15.530 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.530 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.530 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.530 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.444871+0000", 2024-03-30T14:01:15.530 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1870000000000003e-05 2024-03-30T14:01:15.530 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.530 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.531 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:15.531 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.444880+0000", 2024-03-30T14:01:15.531 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.8210000000000008e-06 2024-03-30T14:01:15.531 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.531 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.531 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.531 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.728125+0000", 2024-03-30T14:01:15.531 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.28324558700000002 2024-03-30T14:01:15.531 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.531 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.531 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:15.531 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.728235+0000", 2024-03-30T14:01:15.531 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000109209 2024-03-30T14:01:15.531 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.531 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.532 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1", 2024-03-30T14:01:15.532 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.728317+0000", 2024-03-30T14:01:15.532 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.2756000000000005e-05 2024-03-30T14:01:15.532 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.532 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.532 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:15.532 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.729461+0000", 2024-03-30T14:01:15.532 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001143256 2024-03-30T14:01:15.532 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.532 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.532 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.532 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.750096+0000", 2024-03-30T14:01:15.532 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.020635581 2024-03-30T14:01:15.532 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.533 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.533 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.533 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.753485+0000", 2024-03-30T14:01:15.533 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0033885629999999998 2024-03-30T14:01:15.533 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.533 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.533 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:15.533 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.753517+0000", 2024-03-30T14:01:15.533 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2267999999999997e-05 2024-03-30T14:01:15.533 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.533 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.533 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:15.533 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:54:26.753552+0000", 2024-03-30T14:01:15.533 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.5221e-05 2024-03-30T14:01:15.533 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.533 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.534 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.534 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.534 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.534 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:118621 10.c 10:3a544232:::benchmark_data_smithi043_169264_object118620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-30T14:01:15.534 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:58:10.452538+0000", 2024-03-30T14:01:15.534 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 184.95664196000001, 2024-03-30T14:01:15.534 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.13393395699999999, 2024-03-30T14:01:15.534 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:15.534 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:15.534 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:15.534 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:15.534 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:01:15.534 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:01:15.535 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 118621 2024-03-30T14:01:15.535 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.535 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:15.535 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.535 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:15.535 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.452538+0000", 2024-03-30T14:01:15.535 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.535 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.535 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.535 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:15.535 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.452538+0000", 2024-03-30T14:01:15.535 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.535 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.535 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.535 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:15.536 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.452539+0000", 2024-03-30T14:01:15.536 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0900000000000002e-07 2024-03-30T14:01:15.536 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.536 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.536 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:15.536 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.452566+0000", 2024-03-30T14:01:15.536 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.7123000000000001e-05 2024-03-30T14:01:15.536 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.536 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.536 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:15.536 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.452573+0000", 2024-03-30T14:01:15.536 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1620000000000002e-06 2024-03-30T14:01:15.536 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.536 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.537 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:15.537 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.452579+0000", 2024-03-30T14:01:15.537 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4960000000000004e-06 2024-03-30T14:01:15.537 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.537 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.537 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.537 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.452622+0000", 2024-03-30T14:01:15.537 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3231e-05 2024-03-30T14:01:15.537 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.537 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.537 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:15.537 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.452748+0000", 2024-03-30T14:01:15.537 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012574 2024-03-30T14:01:15.537 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.537 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.537 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,3", 2024-03-30T14:01:15.538 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.578487+0000", 2024-03-30T14:01:15.538 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.125739091 2024-03-30T14:01:15.538 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.538 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.538 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:15.538 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.582997+0000", 2024-03-30T14:01:15.538 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0045099659999999998 2024-03-30T14:01:15.538 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.538 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.538 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.538 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.584741+0000", 2024-03-30T14:01:15.538 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0017445830000000001 2024-03-30T14:01:15.538 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.539 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.539 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.539 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.586375+0000", 2024-03-30T14:01:15.539 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001633818 2024-03-30T14:01:15.539 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.539 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.539 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:15.539 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.586431+0000", 2024-03-30T14:01:15.539 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6029999999999997e-05 2024-03-30T14:01:15.539 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.539 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.539 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:15.539 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:10.586472+0000", 2024-03-30T14:01:15.539 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1208000000000003e-05 2024-03-30T14:01:15.540 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.540 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.540 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.540 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.540 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.540 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:153866 10.1 10:8ec74b9a:::benchmark_data_smithi043_169264_object153865:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-03-30T14:01:15.540 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:58:32.687364+0000", 2024-03-30T14:01:15.540 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 162.72181602399999, 2024-03-30T14:01:15.540 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.13246817599999999, 2024-03-30T14:01:15.540 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:15.540 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:15.540 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:15.540 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:15.540 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:01:15.540 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:01:15.541 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 153866 2024-03-30T14:01:15.541 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.541 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:15.541 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.541 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:15.541 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.687364+0000", 2024-03-30T14:01:15.541 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.541 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.541 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.541 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:15.541 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.687364+0000", 2024-03-30T14:01:15.541 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.541 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.541 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.541 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:15.541 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.687365+0000", 2024-03-30T14:01:15.542 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1399999999999997e-07 2024-03-30T14:01:15.542 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.542 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.542 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:15.542 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.687389+0000", 2024-03-30T14:01:15.542 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.4527000000000001e-05 2024-03-30T14:01:15.542 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.542 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.542 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:15.542 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.687395+0000", 2024-03-30T14:01:15.542 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9540000000000001e-06 2024-03-30T14:01:15.542 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.542 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.543 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:15.543 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.687401+0000", 2024-03-30T14:01:15.543 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8690000000000001e-06 2024-03-30T14:01:15.543 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.543 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.543 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.543 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.687453+0000", 2024-03-30T14:01:15.543 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1366999999999998e-05 2024-03-30T14:01:15.543 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.543 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.543 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:15.543 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.687586+0000", 2024-03-30T14:01:15.543 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000133662 2024-03-30T14:01:15.543 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.544 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.544 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1", 2024-03-30T14:01:15.544 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.687741+0000", 2024-03-30T14:01:15.544 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00015501299999999999 2024-03-30T14:01:15.544 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.544 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.544 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:15.544 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.690360+0000", 2024-03-30T14:01:15.544 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.002618811 2024-03-30T14:01:15.544 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.544 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.545 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.545 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.690969+0000", 2024-03-30T14:01:15.545 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00060870599999999998 2024-03-30T14:01:15.545 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.545 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.545 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.545 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.819650+0000", 2024-03-30T14:01:15.545 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.128681451 2024-03-30T14:01:15.545 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.545 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.545 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:15.545 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.819752+0000", 2024-03-30T14:01:15.545 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000101252 2024-03-30T14:01:15.546 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.546 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.546 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:15.546 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.819833+0000", 2024-03-30T14:01:15.546 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.1050000000000005e-05 2024-03-30T14:01:15.546 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.546 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.546 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.546 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.546 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.546 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:153873 10.c 10:3bbf1f2a:::benchmark_data_smithi043_169264_object153872:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-03-30T14:01:15.546 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:58:32.690246+0000", 2024-03-30T14:01:15.546 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 162.71893397599999, 2024-03-30T14:01:15.547 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.13866584200000001, 2024-03-30T14:01:15.547 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:15.547 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:15.547 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:15.547 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:15.547 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:01:15.547 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:01:15.547 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 153873 2024-03-30T14:01:15.547 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.547 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:15.547 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.547 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:15.547 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.690246+0000", 2024-03-30T14:01:15.547 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.548 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.548 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.548 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:15.548 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.690246+0000", 2024-03-30T14:01:15.548 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:15.548 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.548 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.548 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:15.548 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.690248+0000", 2024-03-30T14:01:15.548 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2529999999999999e-06 2024-03-30T14:01:15.548 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.548 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.548 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:15.548 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.690273+0000", 2024-03-30T14:01:15.548 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.5698e-05 2024-03-30T14:01:15.549 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.549 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.549 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:15.549 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.690285+0000", 2024-03-30T14:01:15.549 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1707e-05 2024-03-30T14:01:15.549 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.549 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.549 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:15.549 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.690293+0000", 2024-03-30T14:01:15.549 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.525e-06 2024-03-30T14:01:15.549 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.549 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.549 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:15.549 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.690353+0000", 2024-03-30T14:01:15.550 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0594999999999998e-05 2024-03-30T14:01:15.550 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.550 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.550 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:15.550 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.690478+0000", 2024-03-30T14:01:15.550 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000125091 2024-03-30T14:01:15.550 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.550 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.550 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,3", 2024-03-30T14:01:15.550 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.690627+0000", 2024-03-30T14:01:15.550 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00014827199999999999 2024-03-30T14:01:15.550 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.550 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.551 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:15.551 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.692751+0000", 2024-03-30T14:01:15.551 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0021245740000000002 2024-03-30T14:01:15.551 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.551 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.551 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.551 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.693355+0000", 2024-03-30T14:01:15.551 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00060350699999999998 2024-03-30T14:01:15.551 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.551 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.551 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:15.551 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.828779+0000", 2024-03-30T14:01:15.551 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.13542438400000001 2024-03-30T14:01:15.551 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.551 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.552 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:15.552 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.828851+0000", 2024-03-30T14:01:15.552 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1458000000000006e-05 2024-03-30T14:01:15.552 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:15.552 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:15.552 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:15.552 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:58:32.828912+0000", 2024-03-30T14:01:15.552 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1778000000000004e-05 2024-03-30T14:01:15.552 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.552 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.552 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.552 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:15.552 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:15.552 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T14:01:15.576 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:15.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:15.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:15.642 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:15 smithi043 podman[196733]: 2024-03-30 14:01:15.399945006 +0000 UTC m=+0.021412893 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:01:15.678 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:15.701 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:15.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:15.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:15.803 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:15.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:15.830 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:15.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:15.921 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-03-30T14:01:15.930 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:15.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:15.958 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:15.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:16.059 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:16.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:16.083 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:16.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:16.184 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:16.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:16.208 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:16.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:16.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:15 smithi113 ceph-mon[34341]: 5.1 scrub starts 2024-03-30T14:01:16.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:15 smithi113 ceph-mon[34341]: 5.1 scrub ok 2024-03-30T14:01:16.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:15 smithi113 ceph-mon[34341]: 3.6 scrub starts 2024-03-30T14:01:16.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:15 smithi113 ceph-mon[34341]: 3.6 scrub ok 2024-03-30T14:01:16.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:15 smithi113 ceph-mon[34341]: 3.b scrub starts 2024-03-30T14:01:16.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:15 smithi113 ceph-mon[34341]: 3.b scrub ok 2024-03-30T14:01:16.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:15 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1349724156' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:01:16.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:15 smithi113 ceph-mon[34341]: 5.d scrub starts 2024-03-30T14:01:16.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:15 smithi113 ceph-mon[34341]: 5.d scrub ok 2024-03-30T14:01:16.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:15 smithi043 ceph-mon[29963]: 5.1 scrub starts 2024-03-30T14:01:16.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:15 smithi043 ceph-mon[29963]: 5.1 scrub ok 2024-03-30T14:01:16.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:15 smithi043 ceph-mon[29963]: 3.6 scrub starts 2024-03-30T14:01:16.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:15 smithi043 ceph-mon[29963]: 3.6 scrub ok 2024-03-30T14:01:16.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:15 smithi043 ceph-mon[29963]: 3.b scrub starts 2024-03-30T14:01:16.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:15 smithi043 ceph-mon[29963]: 3.b scrub ok 2024-03-30T14:01:16.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:15 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1349724156' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:01:16.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:15 smithi043 ceph-mon[29963]: 5.d scrub starts 2024-03-30T14:01:16.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:15 smithi043 ceph-mon[29963]: 5.d scrub ok 2024-03-30T14:01:16.265 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:01:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:01:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:01:16.265 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:15 smithi043 ceph-mon[42099]: 5.1 scrub starts 2024-03-30T14:01:16.265 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:15 smithi043 ceph-mon[42099]: 5.1 scrub ok 2024-03-30T14:01:16.265 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:15 smithi043 ceph-mon[42099]: 3.6 scrub starts 2024-03-30T14:01:16.265 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:15 smithi043 ceph-mon[42099]: 3.6 scrub ok 2024-03-30T14:01:16.266 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:15 smithi043 ceph-mon[42099]: 3.b scrub starts 2024-03-30T14:01:16.266 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:15 smithi043 ceph-mon[42099]: 3.b scrub ok 2024-03-30T14:01:16.266 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:15 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1349724156' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:01:16.266 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:15 smithi043 ceph-mon[42099]: 5.d scrub starts 2024-03-30T14:01:16.266 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:15 smithi043 ceph-mon[42099]: 5.d scrub ok 2024-03-30T14:01:16.266 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:15 smithi043 podman[196733]: 2024-03-30 14:01:15.953844947 +0000 UTC m=+0.575312801 container create b2ee1ac7ae1692138c2ff617feedfa0b4cc252ab6e55855a7a5a9cbaab0b311b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:01:16.310 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:16.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:16.333 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:16.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:16.404 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-30T14:01:16.435 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:16.470 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:16.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:16.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:16.572 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:16.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:16.597 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:16.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:16.645 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:01:16.699 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:16.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:16.723 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:16.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:16.824 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:16.848 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:16.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:16.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:16.950 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:16.973 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:16.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:16.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:17.047 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:16 smithi043 ceph-mon[29963]: pgmap v838: 168 pgs: 9 creating+peering, 4 unknown, 105 active+clean, 13 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:17.048 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:16 smithi043 ceph-mon[29963]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-03-30T14:01:17.048 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:16 smithi043 ceph-mon[29963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:01:17.048 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:16 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1349724156' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:01:17.048 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:16 smithi043 ceph-mon[29963]: osdmap e383: 8 total, 7 up, 7 in 2024-03-30T14:01:17.048 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:16 smithi043 ceph-mon[42099]: pgmap v838: 168 pgs: 9 creating+peering, 4 unknown, 105 active+clean, 13 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:17.048 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:16 smithi043 ceph-mon[42099]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-03-30T14:01:17.048 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:16 smithi043 ceph-mon[42099]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:01:17.048 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:16 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1349724156' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:01:17.049 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:16 smithi043 ceph-mon[42099]: osdmap e383: 8 total, 7 up, 7 in 2024-03-30T14:01:17.065 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:16 smithi113 ceph-mon[34341]: pgmap v838: 168 pgs: 9 creating+peering, 4 unknown, 105 active+clean, 13 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:17.065 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:16 smithi113 ceph-mon[34341]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-03-30T14:01:17.065 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:16 smithi113 ceph-mon[34341]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:01:17.066 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:16 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1349724156' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:01:17.066 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:16 smithi113 ceph-mon[34341]: osdmap e383: 8 total, 7 up, 7 in 2024-03-30T14:01:17.075 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:17.100 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:17.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:17.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:17.202 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:17.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:17.226 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:17.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:17.328 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:17.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:17.354 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:17.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:17.455 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:17.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:17.481 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:17.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:17.582 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:17.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:17.607 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:17.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:17.634 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:17 smithi043 podman[196733]: 2024-03-30 14:01:17.414749196 +0000 UTC m=+2.036217063 container init b2ee1ac7ae1692138c2ff617feedfa0b4cc252ab6e55855a7a5a9cbaab0b311b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-activate, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-30T14:01:17.635 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:17 smithi043 podman[196733]: 2024-03-30 14:01:17.421214041 +0000 UTC m=+2.042681895 container start b2ee1ac7ae1692138c2ff617feedfa0b4cc252ab6e55855a7a5a9cbaab0b311b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-activate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326) 2024-03-30T14:01:17.635 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:17 smithi043 podman[196733]: 2024-03-30 14:01:17.640205132 +0000 UTC m=+2.261672988 container attach b2ee1ac7ae1692138c2ff617feedfa0b4cc252ab6e55855a7a5a9cbaab0b311b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, ceph=True) 2024-03-30T14:01:17.708 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:17.732 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:17.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:17.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:17.782 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:01:17.833 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:17.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:17.857 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:17.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:17.904 INFO:teuthology.orchestra.run.smithi043.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-03-30T14:01:17.959 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:17.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:17.984 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:17.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:18.086 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:18.109 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:18.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:18.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:18.186 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:17 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2893124851' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:18.187 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:17 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:18.187 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:17 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 42/594 objects degraded (7.071%), 6 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-03-30T14:01:18.187 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:17 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:18.187 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:17 smithi113 ceph-mon[34341]: osdmap e384: 8 total, 7 up, 7 in 2024-03-30T14:01:18.211 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:18.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:18.238 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:18.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:18.339 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:18.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:17 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2893124851' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:18.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:17 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:18.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:17 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 42/594 objects degraded (7.071%), 6 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-03-30T14:01:18.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:17 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:18.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:17 smithi043 ceph-mon[29963]: osdmap e384: 8 total, 7 up, 7 in 2024-03-30T14:01:18.341 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:17 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2893124851' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:18.341 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:17 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:18.341 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:17 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 42/594 objects degraded (7.071%), 6 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-03-30T14:01:18.342 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:17 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:18.342 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:17 smithi043 ceph-mon[42099]: osdmap e384: 8 total, 7 up, 7 in 2024-03-30T14:01:18.365 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:18.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:18.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:18.467 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:18.491 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:18.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:18.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:18.593 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:18.605 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T14:01:18.605 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [], 2024-03-30T14:01:18.605 INFO:teuthology.orchestra.run.smithi113.stdout: "num_ops": 0 2024-03-30T14:01:18.605 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T14:01:18.618 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:18.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:18.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:18.642 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:18 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-activate[196843]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-30T14:01:18.642 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:18 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-activate[196843]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-30T14:01:18.642 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:18 smithi043 bash[196733]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-30T14:01:18.642 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:18 smithi043 bash[196733]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-30T14:01:18.643 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:18 smithi043 bash[196733]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-30T14:01:18.643 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:18 smithi043 bash[196733]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-30T14:01:18.643 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:18 smithi043 bash[196733]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-30T14:01:18.643 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:18 smithi043 bash[196733]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-30T14:01:18.643 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:18 smithi043 bash[196733]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-30T14:01:18.643 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:18 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-activate[196843]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-30T14:01:18.643 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:18 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-activate[196843]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-30T14:01:18.643 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:18 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-activate[196843]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-30T14:01:18.643 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:18 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-activate[196843]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-30T14:01:18.643 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:18 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-activate[196843]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-30T14:01:18.643 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:18 smithi043 podman[196733]: 2024-03-30 14:01:18.372994832 +0000 UTC m=+2.994462698 container died b2ee1ac7ae1692138c2ff617feedfa0b4cc252ab6e55855a7a5a9cbaab0b311b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-30T14:01:18.720 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:18.744 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:18.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:18.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:18.846 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:18.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:18.872 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:18.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:18.973 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:18.998 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:18.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:18.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:19.077 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T14:01:19.100 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:19.147 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:19.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:19.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:19.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:18 smithi113 ceph-mon[34341]: pgmap v840: 168 pgs: 9 creating+peering, 106 active+clean, 16 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:19.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:18 smithi043 ceph-mon[29963]: pgmap v840: 168 pgs: 9 creating+peering, 106 active+clean, 16 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:19.246 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:18 smithi043 ceph-mon[42099]: pgmap v840: 168 pgs: 9 creating+peering, 106 active+clean, 16 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:19.249 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:19.272 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:19.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:19.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:19.374 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:19.400 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:19.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:19.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:19.406 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-03-30T14:01:19.470 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:01:19 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:01:19.502 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:19.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:19.528 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:19.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:19.629 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:19.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:19.655 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:19.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:19.684 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:01:19.755 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:19.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:19.780 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:19.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:19.881 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:19.892 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:19 smithi043 podman[197201]: 2024-03-30 14:01:19.610358869 +0000 UTC m=+1.229002529 container remove b2ee1ac7ae1692138c2ff617feedfa0b4cc252ab6e55855a7a5a9cbaab0b311b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-activate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, ceph=True) 2024-03-30T14:01:19.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:19.907 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:19.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:20.002 DEBUG:teuthology.orchestra.run.smithi043:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-03-30T14:01:20.005 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-30T14:01:20.005 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-30T14:01:20.008 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:20.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:20.032 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:20.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:20.053 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:hints = 1 2024-03-30T14:01:20.053 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-30T14:01:20.053 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Object prefix: benchmark_data_smithi043_197439 2024-03-30T14:01:20.053 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T14:01:20.053 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 0 0 0 0 0 0 - 0 2024-03-30T14:01:20.082 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:19 smithi113 ceph-mon[34341]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T14:01:20.133 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:20.180 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:20.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:20.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:20.282 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:20.302 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:19 smithi043 ceph-mon[29963]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T14:01:20.303 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:19 smithi043 ceph-mon[42099]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T14:01:20.303 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:20 smithi043 podman[197476]: 2024-03-30 14:01:20.130286271 +0000 UTC m=+0.035915780 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:01:20.319 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:20.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:20.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:20.421 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:20.462 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:20.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:20.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:20.564 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:20.597 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:20.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:20.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:20.699 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:20.744 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:20.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:20.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:20.846 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:20.886 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:20.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:20.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:20.988 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:21.023 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:21.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:21.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:21.053 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 1 16 3765 3749 234.299 234.312 0.00420043 0.00425322 2024-03-30T14:01:21.125 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:21.158 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:21.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:21.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:21.246 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:20 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:21.246 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:20 smithi113 ceph-mon[34341]: pgmap v842: 168 pgs: 113 active+clean, 18 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:21.260 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:21.300 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:21.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:21.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:20 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:21.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:20 smithi043 ceph-mon[29963]: pgmap v842: 168 pgs: 113 active+clean, 18 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:21.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:20 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:21.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:20 smithi043 ceph-mon[42099]: pgmap v842: 168 pgs: 113 active+clean, 18 active+undersized, 6 active+undersized+degraded, 31 active+clean+remapped; 582 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 42/594 objects degraded (7.071%); 16/594 objects misplaced (2.694%) 2024-03-30T14:01:21.393 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:21 smithi043 podman[197476]: 2024-03-30 14:01:21.008920151 +0000 UTC m=+0.914549637 container create 3f9ef916ccfdd3dd69f8f7b9f2233256f72308e9afe53adddf0b2a7911506877 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD) 2024-03-30T14:01:21.401 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:21.442 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:21.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:21.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:21.544 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:21.585 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:21.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:21.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:21.674 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-30T14:01:21.686 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:21.725 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:21 smithi043 podman[197476]: 2024-03-30 14:01:21.433361237 +0000 UTC m=+1.338990740 container init 3f9ef916ccfdd3dd69f8f7b9f2233256f72308e9afe53adddf0b2a7911506877 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-30T14:01:21.725 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:21 smithi043 podman[197476]: 2024-03-30 14:01:21.445034206 +0000 UTC m=+1.350663693 container start 3f9ef916ccfdd3dd69f8f7b9f2233256f72308e9afe53adddf0b2a7911506877 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:01:21.725 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:21 smithi043 ceph-osd[197647]: -- 172.21.15.43:0/3710576083 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5566f918bba0 con 0x5566f915d000 2024-03-30T14:01:21.725 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:21 smithi043 ceph-osd[197647]: -- 172.21.15.43:0/3710576083 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5566f9193680 con 0x5566f915d000 2024-03-30T14:01:21.725 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:21 smithi043 bash[197476]: 3f9ef916ccfdd3dd69f8f7b9f2233256f72308e9afe53adddf0b2a7911506877 2024-03-30T14:01:21.726 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:21 smithi043 systemd[1]: Started Ceph osd.2 for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T14:01:21.738 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:01:21.755 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T14:01:21.755 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [], 2024-03-30T14:01:21.755 INFO:teuthology.orchestra.run.smithi113.stdout: "complaint_time": 30, 2024-03-30T14:01:21.755 INFO:teuthology.orchestra.run.smithi113.stdout: "num_blocked_ops": 0 2024-03-30T14:01:21.755 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T14:01:21.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:21.774 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:21.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:21.875 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:21.924 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:21.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:21.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:22.026 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:22.029 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:21 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2[197592]: 2024-03-30T14:01:21.934+0000 7fe6e94307c0 -1 Falling back to public interface 2024-03-30T14:01:22.053 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 2 16 6983 6967 217.687 201.125 0.00699459 0.00457472 2024-03-30T14:01:22.078 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:22.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:22.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:22.180 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:22.220 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:22.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:22.322 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:22.365 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:22.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:22.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:22.467 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:22.500 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:22.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:22.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:22.582 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-03-30T14:01:22.602 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:22.641 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:22.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:22.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:22.657 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T14:01:22.701 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T14:01:22.743 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:22.748 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T14:01:22.779 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:22.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:22.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:22.789 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T14:01:22.828 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T14:01:22.867 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T14:01:22.881 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:22.909 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T14:01:22.909 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:01:22.909 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:01:22.909 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T14:01:22.909 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:01:22.909 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:01:22.909 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:01:22.910 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:01:22.910 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:01:22.919 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:22.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:22.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:22.980 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:01:23.061 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:23.062 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 3 16 9963 9947 207.206 186.25 0.00561239 0.00481763 2024-03-30T14:01:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:23.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:23.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:23.203 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:23.238 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:23.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:23.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:23.271 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:22 smithi043 ceph-mon[42099]: pgmap v843: 168 pgs: 113 active+clean, 15 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 62 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.5 KiB/s rd, 9.0 MiB/s wr, 145 op/s; 779/3558 objects degraded (21.894%); 16/3558 objects misplaced (0.450%) 2024-03-30T14:01:23.272 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:22 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:01:23.272 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:22 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:01:23.272 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:22 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/135925540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:01:23.272 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:22 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 779/3558 objects degraded (21.894%), 9 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-03-30T14:01:23.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:22 smithi043 ceph-mon[29963]: pgmap v843: 168 pgs: 113 active+clean, 15 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 62 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.5 KiB/s rd, 9.0 MiB/s wr, 145 op/s; 779/3558 objects degraded (21.894%); 16/3558 objects misplaced (0.450%) 2024-03-30T14:01:23.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:22 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:01:23.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:22 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:01:23.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:22 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/135925540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:01:23.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:22 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 779/3558 objects degraded (21.894%), 9 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-03-30T14:01:23.295 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:22 smithi113 ceph-mon[34341]: pgmap v843: 168 pgs: 113 active+clean, 15 active+undersized, 9 active+undersized+degraded, 31 active+clean+remapped; 62 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.5 KiB/s rd, 9.0 MiB/s wr, 145 op/s; 779/3558 objects degraded (21.894%); 16/3558 objects misplaced (0.450%) 2024-03-30T14:01:23.295 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:22 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:01:23.295 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:22 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:01:23.295 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:22 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/135925540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:01:23.295 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:22 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 779/3558 objects degraded (21.894%), 9 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-03-30T14:01:23.340 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:23.377 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:23.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:23.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:23.479 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:23.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:23.522 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:23.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:23.623 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:23.635 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T14:01:23.635 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T14:01:23.643 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:01:23.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:23.653 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:23.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:23.753 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:23.787 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:23.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:23.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:23.889 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:23.933 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:23.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:23.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:24.036 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:24.053 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 4 16 12986 12970 202.627 188.938 0.00490394 0.00492917 2024-03-30T14:01:24.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:24.078 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:24.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:24.179 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:24.212 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:24.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:24.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:24.313 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:24.348 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:24.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:24.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:24.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:24 smithi043 ceph-mon[29963]: osdmap e385: 8 total, 7 up, 7 in 2024-03-30T14:01:24.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:24 smithi043 ceph-mon[42099]: osdmap e385: 8 total, 7 up, 7 in 2024-03-30T14:01:24.425 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:24 smithi113 ceph-mon[34341]: osdmap e385: 8 total, 7 up, 7 in 2024-03-30T14:01:24.451 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:24.484 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:24.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:24.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:24.586 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:24.645 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:24.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:24.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:24.747 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:24.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:24.781 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:24.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:24.882 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:24.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:24.915 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:24.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:25.016 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:25.030 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:25 smithi113 ceph-mon[34341]: pgmap v845: 168 pgs: 113 active+clean, 14 active+undersized, 10 active+undersized+degraded, 31 active+clean+remapped; 281 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 1.3 KiB/s rd, 36 MiB/s wr, 575 op/s; 1416/14076 objects degraded (10.060%); 16/14076 objects misplaced (0.114%) 2024-03-30T14:01:25.030 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:25 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:01:25.031 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:25 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:01:25.031 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:25 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:01:25.031 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:25 smithi113 ceph-mon[34341]: osdmap e386: 8 total, 7 up, 7 in 2024-03-30T14:01:25.051 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:25.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:25.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:25.053 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 5 16 15216 15200 189.975 139.375 0.00432307 0.00525709 2024-03-30T14:01:25.127 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T14:01:25.127 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:01:25.153 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:25.193 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:25.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:25.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:25.294 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:25.325 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:25.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:25.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:25.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:25 smithi043 ceph-mon[29963]: pgmap v845: 168 pgs: 113 active+clean, 14 active+undersized, 10 active+undersized+degraded, 31 active+clean+remapped; 281 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 1.3 KiB/s rd, 36 MiB/s wr, 575 op/s; 1416/14076 objects degraded (10.060%); 16/14076 objects misplaced (0.114%) 2024-03-30T14:01:25.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:25 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:01:25.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:25 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:01:25.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:25 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:01:25.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:25 smithi043 ceph-mon[29963]: osdmap e386: 8 total, 7 up, 7 in 2024-03-30T14:01:25.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:25 smithi043 ceph-mon[42099]: pgmap v845: 168 pgs: 113 active+clean, 14 active+undersized, 10 active+undersized+degraded, 31 active+clean+remapped; 281 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 1.3 KiB/s rd, 36 MiB/s wr, 575 op/s; 1416/14076 objects degraded (10.060%); 16/14076 objects misplaced (0.114%) 2024-03-30T14:01:25.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:25 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:01:25.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:25 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:01:25.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:25 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:01:25.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:25 smithi043 ceph-mon[42099]: osdmap e386: 8 total, 7 up, 7 in 2024-03-30T14:01:25.427 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:25.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:25.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:25.562 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:25.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:25.604 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:25.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:25.636 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T14:01:25.706 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:25.777 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:25.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:25.878 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:25.892 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:25 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2[197592]: 2024-03-30T14:01:25.631+0000 7fe6e94307c0 -1 osd.2 166 log_to_monitors true 2024-03-30T14:01:25.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:25.925 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:25.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:26.026 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:26.054 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 6 16 17188 17172 178.852 123.25 0.00234221 0.00558175 2024-03-30T14:01:26.056 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:26.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:26.158 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:26.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:26.207 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:26.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:26.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:26 smithi113 ceph-mon[34341]: from='osd.2 [v2:172.21.15.43:6818/886354154,v1:172.21.15.43:6819/886354154]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T14:01:26.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:26 smithi113 ceph-mon[34341]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T14:01:26.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:26 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:01:26.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:26 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:01:26.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:26 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:01:26.308 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:26.338 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:26.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:26.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:26.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:26 smithi043 ceph-mon[29963]: from='osd.2 [v2:172.21.15.43:6818/886354154,v1:172.21.15.43:6819/886354154]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T14:01:26.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:26 smithi043 ceph-mon[29963]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T14:01:26.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:26 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:01:26.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:26 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:01:26.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:26 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:01:26.393 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:01:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:01:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:01:26.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:26 smithi043 ceph-mon[42099]: from='osd.2 [v2:172.21.15.43:6818/886354154,v1:172.21.15.43:6819/886354154]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T14:01:26.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:26 smithi043 ceph-mon[42099]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T14:01:26.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:26 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:01:26.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:26 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:01:26.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:26 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:01:26.410 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-30T14:01:26.440 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:26.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:26.488 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:26.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:26.589 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:26.621 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:26.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:26.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:26.722 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:26.743 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:01:26.752 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:26.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:26.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:26.853 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:01:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2[197592]: 2024-03-30T14:01:26.598+0000 7fe6d6880700 -1 osd.2 387 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:01:26.854 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:26.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:26.901 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:26.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:27.002 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:27.026 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:27.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:27.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:27.054 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 7 15 19008 18993 169.56 113.812 0.0728151 0.00587692 2024-03-30T14:01:27.056 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-03-30T14:01:27.128 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:27.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:27 smithi043 ceph-mon[29963]: pgmap v847: 168 pgs: 113 active+clean, 12 active+undersized, 12 active+undersized+degraded, 31 active+clean+remapped; 621 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 652 B/s rd, 79 MiB/s wr, 1.27k op/s; 3010/30381 objects degraded (9.908%); 16/30381 objects misplaced (0.053%) 2024-03-30T14:01:27.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:27 smithi043 ceph-mon[29963]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T14:01:27.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:27 smithi043 ceph-mon[29963]: from='osd.2 [v2:172.21.15.43:6818/886354154,v1:172.21.15.43:6819/886354154]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T14:01:27.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:27 smithi043 ceph-mon[29963]: osdmap e387: 8 total, 7 up, 7 in 2024-03-30T14:01:27.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:27 smithi043 ceph-mon[29963]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T14:01:27.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:27 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3717395045' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:01:27.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:27 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:01:27.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:27 smithi043 ceph-mon[42099]: pgmap v847: 168 pgs: 113 active+clean, 12 active+undersized, 12 active+undersized+degraded, 31 active+clean+remapped; 621 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 652 B/s rd, 79 MiB/s wr, 1.27k op/s; 3010/30381 objects degraded (9.908%); 16/30381 objects misplaced (0.053%) 2024-03-30T14:01:27.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:27 smithi043 ceph-mon[42099]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T14:01:27.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:27 smithi043 ceph-mon[42099]: from='osd.2 [v2:172.21.15.43:6818/886354154,v1:172.21.15.43:6819/886354154]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T14:01:27.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:27 smithi043 ceph-mon[42099]: osdmap e387: 8 total, 7 up, 7 in 2024-03-30T14:01:27.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:27 smithi043 ceph-mon[42099]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T14:01:27.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:27 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3717395045' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:01:27.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:27 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:01:27.151 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:27.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:27.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:27.165 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:27 smithi113 ceph-mon[34341]: pgmap v847: 168 pgs: 113 active+clean, 12 active+undersized, 12 active+undersized+degraded, 31 active+clean+remapped; 621 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 652 B/s rd, 79 MiB/s wr, 1.27k op/s; 3010/30381 objects degraded (9.908%); 16/30381 objects misplaced (0.053%) 2024-03-30T14:01:27.165 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:27 smithi113 ceph-mon[34341]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T14:01:27.166 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:27 smithi113 ceph-mon[34341]: from='osd.2 [v2:172.21.15.43:6818/886354154,v1:172.21.15.43:6819/886354154]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T14:01:27.166 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:27 smithi113 ceph-mon[34341]: osdmap e387: 8 total, 7 up, 7 in 2024-03-30T14:01:27.166 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:27 smithi113 ceph-mon[34341]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-30T14:01:27.166 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:27 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3717395045' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:01:27.166 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:27 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:01:27.252 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:27.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:27.278 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:27.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:27.379 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:27.403 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:27.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:27.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:27.504 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:27.530 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:27.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:27.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:27.632 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:27.658 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:27.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:27.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:27.759 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:27.785 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:27.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:27.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:27.887 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:27.921 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:27.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:27.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:28.023 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:28.047 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:28.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:28.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:28.054 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 8 16 19040 19024 148.607 1.9375 0.00281545 0.00590221 2024-03-30T14:01:28.148 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:28.173 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:28.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:28.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:28.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:28 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:01:28.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:28 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:01:28.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:28 smithi113 ceph-mon[34341]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:01:28.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:28 smithi113 ceph-mon[34341]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T14:01:28.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:28 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:01:28.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:28 smithi113 ceph-mon[34341]: osd.2 [v2:172.21.15.43:6818/886354154,v1:172.21.15.43:6819/886354154] boot 2024-03-30T14:01:28.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:28 smithi113 ceph-mon[34341]: osdmap e388: 8 total, 8 up, 7 in 2024-03-30T14:01:28.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:01:28.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:28 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 3010/30381 objects degraded (9.908%), 12 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-03-30T14:01:28.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:28 smithi113 ceph-mon[34341]: osdmap e389: 8 total, 8 up, 7 in 2024-03-30T14:01:28.275 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:28.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:28.300 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:28.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:28.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:28 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:01:28.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:28 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:01:28.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:28 smithi043 ceph-mon[29963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:01:28.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:28 smithi043 ceph-mon[29963]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T14:01:28.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:28 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:01:28.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:28 smithi043 ceph-mon[29963]: osd.2 [v2:172.21.15.43:6818/886354154,v1:172.21.15.43:6819/886354154] boot 2024-03-30T14:01:28.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:28 smithi043 ceph-mon[29963]: osdmap e388: 8 total, 8 up, 7 in 2024-03-30T14:01:28.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:01:28.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:28 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 3010/30381 objects degraded (9.908%), 12 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-03-30T14:01:28.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:28 smithi043 ceph-mon[29963]: osdmap e389: 8 total, 8 up, 7 in 2024-03-30T14:01:28.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:28 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:01:28.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:28 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:01:28.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:28 smithi043 ceph-mon[42099]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:01:28.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:28 smithi043 ceph-mon[42099]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T14:01:28.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:28 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:01:28.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:28 smithi043 ceph-mon[42099]: osd.2 [v2:172.21.15.43:6818/886354154,v1:172.21.15.43:6819/886354154] boot 2024-03-30T14:01:28.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:28 smithi043 ceph-mon[42099]: osdmap e388: 8 total, 8 up, 7 in 2024-03-30T14:01:28.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:01:28.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:28 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 3010/30381 objects degraded (9.908%), 12 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-03-30T14:01:28.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:28 smithi043 ceph-mon[42099]: osdmap e389: 8 total, 8 up, 7 in 2024-03-30T14:01:28.402 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:28.427 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:28.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:28.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:28.529 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:28.556 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:28.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:28.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:28.657 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:28.685 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:28.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:28.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:28.787 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:28.811 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:28.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:28.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:28.813 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T14:01:28.813 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T14:01:28.813 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 0 2024-03-30T14:01:28.813 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T14:01:28.913 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:28.953 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:28.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:28.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:29.054 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 9 16 19110 19094 132.583 4.375 0.00540174 0.00732076 2024-03-30T14:01:29.054 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:29.091 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:29.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:29.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:29.193 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:29.230 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:29.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:29.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:29.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:29 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:01:29.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:29 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:01:29.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:29 smithi043 ceph-mon[29963]: pgmap v850: 168 pgs: 113 active+clean, 12 active+undersized, 12 active+undersized+degraded, 31 active+clean+remapped; 825 MiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 161 MiB/s wr, 2.57k op/s; 5498/40173 objects degraded (13.686%); 16/40173 objects misplaced (0.040%) 2024-03-30T14:01:29.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:29 smithi043 ceph-mon[29963]: osdmap e390: 8 total, 8 up, 7 in 2024-03-30T14:01:29.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:29 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:01:29.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:29 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:01:29.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:29 smithi043 ceph-mon[42099]: pgmap v850: 168 pgs: 113 active+clean, 12 active+undersized, 12 active+undersized+degraded, 31 active+clean+remapped; 825 MiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 161 MiB/s wr, 2.57k op/s; 5498/40173 objects degraded (13.686%); 16/40173 objects misplaced (0.040%) 2024-03-30T14:01:29.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:29 smithi043 ceph-mon[42099]: osdmap e390: 8 total, 8 up, 7 in 2024-03-30T14:01:29.333 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:29.344 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:29 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:01:29.344 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:29 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:01:29.344 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:29 smithi113 ceph-mon[34341]: pgmap v850: 168 pgs: 113 active+clean, 12 active+undersized, 12 active+undersized+degraded, 31 active+clean+remapped; 825 MiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 161 MiB/s wr, 2.57k op/s; 5498/40173 objects degraded (13.686%); 16/40173 objects misplaced (0.040%) 2024-03-30T14:01:29.344 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:29 smithi113 ceph-mon[34341]: osdmap e390: 8 total, 8 up, 7 in 2024-03-30T14:01:29.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:29.371 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:29.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:29.402 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T14:01:29.402 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:01:29.472 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:29.513 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:29.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:29.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:29.614 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:29.647 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:29.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:29.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:29.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:01:29 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:01:29.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:29.797 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:29.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:29.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:29.900 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:29.944 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:29.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:29.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:30.047 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:30.054 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 10 16 20666 20650 129.049 97.25 0.00948756 0.00774101 2024-03-30T14:01:30.079 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:30.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:30.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:30.181 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:30.216 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:30.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:30.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:30.233 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-03-30T14:01:30.277 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:01:30.318 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:30.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:30.376 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:30.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:30.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:01:30.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:30.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:30 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:30.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:30 smithi113 ceph-mon[34341]: pgmap v853: 168 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+degraded, 19 peering, 11 remapped+peering, 122 active+clean, 3 active+undersized, 7 active+undersized+degraded, 4 active+clean+remapped; 1.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.16k op/s; 4437/56337 objects degraded (7.876%); 1/56337 objects misplaced (0.002%); 0 B/s, 3 objects/s recovering 2024-03-30T14:01:30.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:30 smithi113 ceph-mon[34341]: osdmap e391: 8 total, 8 up, 7 in 2024-03-30T14:01:30.477 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:30.511 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:30.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:30.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:30.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:01:30.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:30.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:30 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:30.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:30 smithi043 ceph-mon[29963]: pgmap v853: 168 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+degraded, 19 peering, 11 remapped+peering, 122 active+clean, 3 active+undersized, 7 active+undersized+degraded, 4 active+clean+remapped; 1.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.16k op/s; 4437/56337 objects degraded (7.876%); 1/56337 objects misplaced (0.002%); 0 B/s, 3 objects/s recovering 2024-03-30T14:01:30.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:30 smithi043 ceph-mon[29963]: osdmap e391: 8 total, 8 up, 7 in 2024-03-30T14:01:30.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:01:30.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:30.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:30 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:30.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:30 smithi043 ceph-mon[42099]: pgmap v853: 168 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+degraded, 19 peering, 11 remapped+peering, 122 active+clean, 3 active+undersized, 7 active+undersized+degraded, 4 active+clean+remapped; 1.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.16k op/s; 4437/56337 objects degraded (7.876%); 1/56337 objects misplaced (0.002%); 0 B/s, 3 objects/s recovering 2024-03-30T14:01:30.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:30 smithi043 ceph-mon[42099]: osdmap e391: 8 total, 8 up, 7 in 2024-03-30T14:01:30.612 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:30.614 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:01:30.653 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:30.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:30.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:30.755 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:30.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:30.787 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:30.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:30.888 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:30.917 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:30.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:30.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:31.018 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:31.054 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 11 16 22194 22178 125.998 95.5 0.00738077 0.00792833 2024-03-30T14:01:31.061 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:31.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:31.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:31.163 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:31.190 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:31.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:31.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:31.292 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:31.319 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:31.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:31.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:31.421 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:31.452 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:31.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:31.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:31.554 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:31.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:31.594 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:31.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:31.695 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:31.728 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:31.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:31.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:31.732 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T14:01:31.830 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:31.858 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:31.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:31.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:31.960 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:31.991 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:31.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:31.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:32.055 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 12 16 23650 23634 123.07 91 0.00618797 0.00811877 2024-03-30T14:01:32.093 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:32.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:32.124 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:32.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:32.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:31 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:01:32.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:31 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:01:32.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:31 smithi043 ceph-mon[29963]: osdmap e392: 8 total, 8 up, 7 in 2024-03-30T14:01:32.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:31 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:01:32.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:31 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:01:32.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:31 smithi043 ceph-mon[42099]: osdmap e392: 8 total, 8 up, 7 in 2024-03-30T14:01:32.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:31 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:01:32.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:31 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:01:32.217 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:31 smithi113 ceph-mon[34341]: osdmap e392: 8 total, 8 up, 7 in 2024-03-30T14:01:32.225 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:32.259 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:32.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:32.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:32.361 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:32.404 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:32.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:32.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:32.506 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:32.531 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:32.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:32.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:32.633 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:32.663 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:32.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:32.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:32.765 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:32.798 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:32.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:32.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:32.900 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:32.932 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:32.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:32.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:33.034 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:33.055 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 13 16 25012 24996 120.151 85.125 0.00644255 0.00827559 2024-03-30T14:01:33.058 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:33.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:33.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:33.160 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:33.200 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:33.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:33.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:33.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:32 smithi113 ceph-mon[34341]: pgmap v856: 168 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+degraded, 19 peering, 11 remapped+peering, 133 active+clean, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.2 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.66k op/s; 4543/60129 objects degraded (7.555%); 1/60129 objects misplaced (0.002%); 3.1 MiB/s, 57 objects/s recovering 2024-03-30T14:01:33.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:32 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 4543/60129 objects degraded (7.555%), 5 pgs degraded (PG_DEGRADED) 2024-03-30T14:01:33.302 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:33.328 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:33.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:33.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:32 smithi043 ceph-mon[29963]: pgmap v856: 168 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+degraded, 19 peering, 11 remapped+peering, 133 active+clean, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.2 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.66k op/s; 4543/60129 objects degraded (7.555%); 1/60129 objects misplaced (0.002%); 3.1 MiB/s, 57 objects/s recovering 2024-03-30T14:01:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:32 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 4543/60129 objects degraded (7.555%), 5 pgs degraded (PG_DEGRADED) 2024-03-30T14:01:33.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:32 smithi043 ceph-mon[42099]: pgmap v856: 168 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+degraded, 19 peering, 11 remapped+peering, 133 active+clean, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.2 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.66k op/s; 4543/60129 objects degraded (7.555%); 1/60129 objects misplaced (0.002%); 3.1 MiB/s, 57 objects/s recovering 2024-03-30T14:01:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:32 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 4543/60129 objects degraded (7.555%), 5 pgs degraded (PG_DEGRADED) 2024-03-30T14:01:33.430 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:33.472 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:33.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:33.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:33.574 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:33.597 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:33.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:33.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:33.699 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:33.730 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:33.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:33.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:33.832 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:33.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:33.877 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:33.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:33.978 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:34.009 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:34.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:34.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:34.055 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 14 16 26196 26180 116.854 74 0.0122654 0.0085509 2024-03-30T14:01:34.111 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:34.143 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:34.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:34.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:34.244 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:34.276 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:34.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:34.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:34.378 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:34.409 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:34.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:34.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:34.511 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:34.546 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:34.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:34.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:34.648 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:34.694 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:34.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:34.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:34.795 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:34.823 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:34.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:34.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:34.925 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:34.953 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:34.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:34.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:35.055 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:35.056 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 15 16 27561 27545 114.749 85.3125 0.0109539 0.00870305 2024-03-30T14:01:35.105 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:35.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:35.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:35.201 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:34 smithi043 ceph-mon[29963]: pgmap v857: 168 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 13 peering, 9 remapped+peering, 139 active+clean, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.3 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 875 B/s rd, 92 MiB/s wr, 1.47k op/s; 5677/65973 objects degraded (8.605%); 1/65973 objects misplaced (0.002%); 2.1 MiB/s, 41 objects/s recovering 2024-03-30T14:01:35.201 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:34 smithi043 ceph-mon[42099]: pgmap v857: 168 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 13 peering, 9 remapped+peering, 139 active+clean, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.3 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 875 B/s rd, 92 MiB/s wr, 1.47k op/s; 5677/65973 objects degraded (8.605%); 1/65973 objects misplaced (0.002%); 2.1 MiB/s, 41 objects/s recovering 2024-03-30T14:01:35.207 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:35.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:34 smithi113 ceph-mon[34341]: pgmap v857: 168 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 13 peering, 9 remapped+peering, 139 active+clean, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.3 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 875 B/s rd, 92 MiB/s wr, 1.47k op/s; 5677/65973 objects degraded (8.605%); 1/65973 objects misplaced (0.002%); 2.1 MiB/s, 41 objects/s recovering 2024-03-30T14:01:35.238 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:35.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:35.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:35.258 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-03-30T14:01:35.258 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T14:01:35.258 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-30T14:01:35.259 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-03-30T14:01:35.340 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:35.367 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:35.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:35.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:35.469 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:35.496 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:35.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:35.598 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:35.616 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:01:35.676 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:35.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:35.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:35.777 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:35.778 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T14:01:35.811 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T14:01:35.813 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:35.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:35.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:35.857 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T14:01:35.899 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T14:01:35.914 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:35.938 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T14:01:35.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:35.970 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:35.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:35.981 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T14:01:36.025 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T14:01:36.056 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 16 16 28625 28609 111.733 66.5 0.0101957 0.00894121 2024-03-30T14:01:36.071 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:36.076 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T14:01:36.077 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:35 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3347595196' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:01:36.078 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:35 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3347595196' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:01:36.083 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:01:36.083 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:01:36.084 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-03-30T14:01:36.084 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:01:36.084 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:01:36.084 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:01:36.084 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:01:36.084 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:01:36.116 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:36.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:36.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:36.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:35 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3347595196' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:01:36.218 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:36.250 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:36.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:36.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:36.352 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:36.381 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:36.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:36.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:36.392 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:01:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:01:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:01:36.483 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:36.519 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:36.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:36.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:36.621 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:36.664 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:36.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:36.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:36.754 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:01:36.766 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:36.791 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T14:01:36.791 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:01:36.793 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:36.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:36.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:36.895 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:36.921 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:36.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:36.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:37.023 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:37.054 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:37.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:37.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:37.056 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 17 16 29917 29901 109.91 80.75 0.00365161 0.00909194 2024-03-30T14:01:37.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:36 smithi043 ceph-mon[42099]: pgmap v858: 168 pgs: 1 active+recovery_wait+degraded, 161 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.5 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 747 B/s rd, 68 MiB/s wr, 1.09k op/s; 5775/75192 objects degraded (7.680%); 12 MiB/s, 195 objects/s recovering 2024-03-30T14:01:37.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:36 smithi043 ceph-mon[29963]: pgmap v858: 168 pgs: 1 active+recovery_wait+degraded, 161 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.5 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 747 B/s rd, 68 MiB/s wr, 1.09k op/s; 5775/75192 objects degraded (7.680%); 12 MiB/s, 195 objects/s recovering 2024-03-30T14:01:37.156 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:37.184 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:37.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:37.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:37.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:36 smithi113 ceph-mon[34341]: pgmap v858: 168 pgs: 1 active+recovery_wait+degraded, 161 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.5 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 747 B/s rd, 68 MiB/s wr, 1.09k op/s; 5775/75192 objects degraded (7.680%); 12 MiB/s, 195 objects/s recovering 2024-03-30T14:01:37.285 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:37.332 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:37.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:37.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:37.435 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:37.463 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:37.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:37.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:37.564 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:37.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:37.593 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:37.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:37.694 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:37.734 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:37.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:37.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:37.836 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:37.876 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:37.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:37.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:37.978 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:38.006 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:38.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:38.056 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 18 16 31198 31182 108.252 80.0625 0.00291724 0.00921217 2024-03-30T14:01:38.108 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:38.131 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:38.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:38.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:38.193 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:37 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 5775/75192 objects degraded (7.680%), 7 pgs degraded (PG_DEGRADED) 2024-03-30T14:01:38.193 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:37 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 5775/75192 objects degraded (7.680%), 7 pgs degraded (PG_DEGRADED) 2024-03-30T14:01:38.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:37 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 5775/75192 objects degraded (7.680%), 7 pgs degraded (PG_DEGRADED) 2024-03-30T14:01:38.233 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:38.261 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:38.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:38.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:38.363 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:38.404 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:38.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:38.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:38.506 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:38.550 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:38.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:38.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:38.652 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:38.695 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:38.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:38.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:38.796 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:38.828 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:38.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:38.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:38.929 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:38.957 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:38.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:38.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:39.056 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 19 16 32279 32263 106.111 67.5625 0.00366762 0.00940466 2024-03-30T14:01:39.060 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:39.087 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:39.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:39.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:39.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:38 smithi043 ceph-mon[29963]: pgmap v859: 168 pgs: 1 active+recovery_wait+degraded, 161 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.6 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 985 op/s; 5358/79968 objects degraded (6.700%); 14 MiB/s, 219 objects/s recovering 2024-03-30T14:01:39.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:38 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:01:39.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:38 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:01:39.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:38 smithi043 ceph-mon[29963]: osdmap e393: 8 total, 8 up, 7 in 2024-03-30T14:01:39.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:38 smithi043 ceph-mon[42099]: pgmap v859: 168 pgs: 1 active+recovery_wait+degraded, 161 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.6 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 985 op/s; 5358/79968 objects degraded (6.700%); 14 MiB/s, 219 objects/s recovering 2024-03-30T14:01:39.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:38 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:01:39.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:38 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:01:39.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:38 smithi043 ceph-mon[42099]: osdmap e393: 8 total, 8 up, 7 in 2024-03-30T14:01:39.188 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:39.214 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:39.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:39.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:39.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:38 smithi113 ceph-mon[34341]: pgmap v859: 168 pgs: 1 active+recovery_wait+degraded, 161 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.6 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 985 op/s; 5358/79968 objects degraded (6.700%); 14 MiB/s, 219 objects/s recovering 2024-03-30T14:01:39.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:38 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:01:39.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:38 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:01:39.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:38 smithi113 ceph-mon[34341]: osdmap e393: 8 total, 8 up, 7 in 2024-03-30T14:01:39.264 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-03-30T14:01:39.315 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:39.348 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:39.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:39.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:39.370 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T14:01:39.450 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:39.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:39.480 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:39.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:39.512 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:01:39.581 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:39.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:39.611 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:39.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:39.635 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:01:39 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:01:39.712 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:39.736 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:39.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:39.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:39.838 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:39.874 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:39.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:39.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:39.909 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:01:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T14:01:39.977 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:40.008 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:40.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:40.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:40.056 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-03-30T14:01:40.062932+0000 min lat: 0.00121029 max lat: 2.12887 avg lat: 0.00982635 2024-03-30T14:01:40.056 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T14:01:40.056 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 20 16 32510 32494 101.527 14.4375 0.00262303 0.00982635 2024-03-30T14:01:40.110 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:40.139 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:40.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:40.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:40.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:39 smithi043 ceph-mon[29963]: osdmap e394: 8 total, 8 up, 7 in 2024-03-30T14:01:40.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:39 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3926112814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:01:40.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:39 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:01:40.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:39 smithi043 ceph-mon[42099]: osdmap e394: 8 total, 8 up, 7 in 2024-03-30T14:01:40.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:39 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3926112814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:01:40.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:39 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:01:40.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:39 smithi113 ceph-mon[34341]: osdmap e394: 8 total, 8 up, 7 in 2024-03-30T14:01:40.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:39 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3926112814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:01:40.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:39 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:01:40.240 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:40.269 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:40.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:40.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:40.371 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:40.401 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:40.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:40.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:40.503 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:40.533 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:40.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:40.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:40.635 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:40.659 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:40.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:40.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:40.761 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:40.786 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:40.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:40.887 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:40.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:40.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:40.970 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T14:01:40.970 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T14:01:40.971 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 396, 2024-03-30T14:01:40.971 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "7ee932fa-ee9a-11ee-b647-cb9ed24678a4", 2024-03-30T14:01:40.971 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-03-30T13:37:07.346020+0000", 2024-03-30T14:01:40.971 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-03-30T14:01:40.963485+0000", 2024-03-30T14:01:40.971 INFO:teuthology.orchestra.run.smithi043.stdout: "last_up_change": "2024-03-30T14:01:27.027194+0000", 2024-03-30T14:01:40.971 INFO:teuthology.orchestra.run.smithi043.stdout: "last_in_change": "2024-03-30T13:58:46.155265+0000", 2024-03-30T14:01:40.971 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-30T14:01:40.971 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_num": 5804032, 2024-03-30T14:01:40.971 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_set": [ 2024-03-30T14:01:40.971 INFO:teuthology.orchestra.run.smithi043.stdout: "nodeep-scrub", 2024-03-30T14:01:40.971 INFO:teuthology.orchestra.run.smithi043.stdout: "pglog_hardlimit", 2024-03-30T14:01:40.971 INFO:teuthology.orchestra.run.smithi043.stdout: "purged_snapdirs", 2024-03-30T14:01:40.971 INFO:teuthology.orchestra.run.smithi043.stdout: "recovery_deletes", 2024-03-30T14:01:40.971 INFO:teuthology.orchestra.run.smithi043.stdout: "sortbitwise" 2024-03-30T14:01:40.972 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T14:01:40.972 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_version": 19, 2024-03-30T14:01:40.972 INFO:teuthology.orchestra.run.smithi043.stdout: "full_ratio": 0.94999998807907104, 2024-03-30T14:01:40.972 INFO:teuthology.orchestra.run.smithi043.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-30T14:01:40.972 INFO:teuthology.orchestra.run.smithi043.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-30T14:01:40.972 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_snapshot": "", 2024-03-30T14:01:40.972 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_max": 11, 2024-03-30T14:01:40.972 INFO:teuthology.orchestra.run.smithi043.stdout: "max_osd": 8, 2024-03-30T14:01:40.972 INFO:teuthology.orchestra.run.smithi043.stdout: "require_min_compat_client": "luminous", 2024-03-30T14:01:40.972 INFO:teuthology.orchestra.run.smithi043.stdout: "min_compat_client": "luminous", 2024-03-30T14:01:40.972 INFO:teuthology.orchestra.run.smithi043.stdout: "require_osd_release": "squid", 2024-03-30T14:01:40.972 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_crimson": false, 2024-03-30T14:01:40.972 INFO:teuthology.orchestra.run.smithi043.stdout: "pools": [ 2024-03-30T14:01:40.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:40.972 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 1, 2024-03-30T14:01:40.973 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".mgr", 2024-03-30T14:01:40.973 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:40:27.658979+0000", 2024-03-30T14:01:40.973 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T14:01:40.973 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T14:01:40.973 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:01:40.973 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:01:40.973 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:01:40.973 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:01:40.973 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:01:40.973 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:01:40.973 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:01:40.973 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:01:40.973 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:01:40.973 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:01:40.974 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 21, 2024-03-30T14:01:40.974 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 21, 2024-03-30T14:01:40.974 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 21, 2024-03-30T14:01:40.974 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 21, 2024-03-30T14:01:40.974 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 21, 2024-03-30T14:01:40.974 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:01:40.974 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:01:40.974 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:01:40.974 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:01:40.974 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:01:40.974 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:01:40.974 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:01:40.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:40.974 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "371", 2024-03-30T14:01:40.974 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:01:40.975 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:01:40.975 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "368", 2024-03-30T14:01:40.975 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:01:40.975 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:01:40.975 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T14:01:40.975 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T14:01:40.975 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:01:40.975 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:01:40.975 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:01:40.975 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:01:40.975 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:01:40.975 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:01:40.975 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:01:40.975 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:01:40.975 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:01:40.976 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:01:40.976 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:01:40.976 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:01:40.976 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:01:40.976 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:01:40.976 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:01:40.976 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:01:40.976 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:01:40.976 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:01:40.976 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:01:40.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:40.976 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:01:40.976 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:01:40.976 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:01:40.977 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:01:40.977 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:01:40.977 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:01:40.977 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:01:40.977 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:01:40.977 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:01:40.977 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:01:40.977 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:01:40.977 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-03-30T14:01:40.977 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_max": 32, 2024-03-30T14:01:40.977 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_min": 1 2024-03-30T14:01:40.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:40.977 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:01:40.977 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": {} 2024-03-30T14:01:40.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:40.978 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:01:40.978 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:01:40.978 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.7100000381469727, 2024-03-30T14:01:40.978 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.7100000381469727, 2024-03-30T14:01:40.978 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1.1699999570846558, 2024-03-30T14:01:40.978 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2, 2024-03-30T14:01:40.978 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2, 2024-03-30T14:01:40.978 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.75, 2024-03-30T14:01:40.978 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:01:40.978 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.75 2024-03-30T14:01:40.978 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:40.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:40.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:40.978 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 2, 2024-03-30T14:01:40.979 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "datapool", 2024-03-30T14:01:40.979 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:42.989975+0000", 2024-03-30T14:01:40.979 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 8193, 2024-03-30T14:01:40.979 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T14:01:40.979 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:01:40.979 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:01:40.979 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:01:40.979 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:01:40.979 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:01:40.979 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:01:40.979 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:01:40.979 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:01:40.979 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:01:40.979 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:01:40.980 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 3, 2024-03-30T14:01:40.980 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 3, 2024-03-30T14:01:40.980 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 3, 2024-03-30T14:01:40.980 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 3, 2024-03-30T14:01:40.980 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 3, 2024-03-30T14:01:40.980 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:01:40.980 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:01:40.980 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:01:40.980 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:01:40.980 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:01:40.980 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:01:40.980 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:01:40.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:40.980 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "68", 2024-03-30T14:01:40.980 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:01:40.981 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:01:40.981 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T14:01:40.981 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:01:40.981 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:01:40.981 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 2, 2024-03-30T14:01:40.981 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 68, 2024-03-30T14:01:40.981 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:01:40.981 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:01:40.981 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:01:40.981 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:01:40.981 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:01:40.981 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:01:40.981 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:01:40.981 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:01:40.981 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:01:40.982 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:01:40.982 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:01:40.982 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:01:40.982 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:01:40.982 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:01:40.982 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:01:40.982 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:01:40.982 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:01:40.982 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:01:40.982 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:01:40.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:40.982 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:01:40.982 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:01:40.982 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:01:40.982 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:01:40.982 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:01:40.983 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:01:40.983 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:01:40.983 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:01:40.983 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:01:40.983 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:01:40.983 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:01:40.983 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T14:01:40.983 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:01:40.983 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd": {} 2024-03-30T14:01:40.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:40.983 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:01:40.983 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:01:40.983 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2, 2024-03-30T14:01:40.983 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2, 2024-03-30T14:01:40.983 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1, 2024-03-30T14:01:40.984 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2, 2024-03-30T14:01:40.984 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2, 2024-03-30T14:01:40.984 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.75, 2024-03-30T14:01:40.984 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:01:40.984 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.75 2024-03-30T14:01:40.984 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:40.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:40.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:40.984 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 3, 2024-03-30T14:01:40.984 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".rgw.root", 2024-03-30T14:01:40.984 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:43.910306+0000", 2024-03-30T14:01:40.984 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T14:01:40.984 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T14:01:40.984 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:01:40.984 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:01:40.984 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:01:40.985 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:01:40.985 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:01:40.985 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:01:40.985 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:01:40.985 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:01:40.985 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:01:40.985 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:01:40.985 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T14:01:40.985 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T14:01:40.985 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T14:01:40.985 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T14:01:40.985 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T14:01:40.985 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:01:40.985 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:01:40.985 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:01:40.986 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:01:40.986 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:01:40.986 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:01:40.986 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:01:40.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:40.986 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "66", 2024-03-30T14:01:40.986 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:01:40.987 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:01:40.987 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T14:01:40.987 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:01:40.987 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:01:40.987 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T14:01:40.987 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T14:01:40.987 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:01:40.987 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:01:40.987 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:01:40.987 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:01:40.987 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:01:40.987 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:01:40.987 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:01:40.987 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:01:40.987 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:01:40.988 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:01:40.988 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:01:40.988 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:01:40.988 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:01:40.988 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:01:40.988 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:01:40.988 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:01:40.988 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:01:40.988 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:01:40.988 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:01:40.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:40.988 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:01:40.988 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:01:40.988 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:01:40.988 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:01:40.988 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:01:40.989 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:01:40.989 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:01:40.989 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:01:40.989 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:01:40.989 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:01:40.989 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:01:40.989 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T14:01:40.989 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:01:40.989 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T14:01:40.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:40.989 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:01:40.989 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:01:40.989 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.8700000047683716, 2024-03-30T14:01:40.989 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.8700000047683716, 2024-03-30T14:01:40.989 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1.1699999570846558, 2024-03-30T14:01:40.990 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2.190000057220459, 2024-03-30T14:01:40.990 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2.190000057220459, 2024-03-30T14:01:40.990 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.75, 2024-03-30T14:01:40.990 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:01:40.990 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.75 2024-03-30T14:01:40.990 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:40.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:40.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:40.990 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 4, 2024-03-30T14:01:40.990 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.log", 2024-03-30T14:01:40.990 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:45.633132+0000", 2024-03-30T14:01:40.990 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T14:01:40.990 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T14:01:40.990 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:01:40.991 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:01:40.991 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:01:40.991 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:01:40.991 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:01:40.991 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:01:40.991 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:01:40.991 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:01:40.991 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:01:40.991 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:01:40.991 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T14:01:40.991 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T14:01:40.991 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T14:01:40.991 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T14:01:40.991 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T14:01:40.991 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:01:40.992 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:01:40.992 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:01:40.992 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:01:40.992 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:01:40.992 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:01:40.992 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:01:40.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:40.992 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "68", 2024-03-30T14:01:40.992 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:01:40.992 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:01:40.992 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T14:01:40.992 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:01:40.992 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:01:40.992 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T14:01:40.992 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T14:01:40.993 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:01:40.993 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:01:40.993 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:01:40.993 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:01:40.993 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:01:40.993 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:01:40.993 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:01:40.993 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:01:40.993 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:01:40.993 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:01:40.993 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:01:40.993 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:01:40.993 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:01:40.993 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:01:40.993 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:01:40.993 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:01:40.994 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:01:40.994 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:01:40.994 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:01:40.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:40.994 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:01:40.994 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:01:40.994 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:01:40.994 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:01:40.994 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:01:40.994 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:01:40.994 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:01:40.994 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:01:40.994 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:01:40.994 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:01:40.995 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:01:40.995 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T14:01:40.995 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:01:40.995 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T14:01:40.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:40.995 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:01:40.995 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:01:40.995 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.6799999475479126, 2024-03-30T14:01:40.995 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.6799999475479126, 2024-03-30T14:01:40.995 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1.1699999570846558, 2024-03-30T14:01:40.995 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-30T14:01:40.995 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-30T14:01:40.995 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.75, 2024-03-30T14:01:40.995 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:01:40.995 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.75 2024-03-30T14:01:40.996 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:40.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:40.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:40.996 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 5, 2024-03-30T14:01:40.996 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.control", 2024-03-30T14:01:40.996 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:47.570084+0000", 2024-03-30T14:01:40.996 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T14:01:40.996 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T14:01:40.996 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:01:40.996 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:01:40.996 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:01:40.996 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:01:40.996 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:01:40.996 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:01:40.996 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:01:40.997 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:01:40.997 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:01:40.997 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:01:40.997 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T14:01:40.997 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T14:01:40.997 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T14:01:40.997 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T14:01:40.997 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T14:01:40.997 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:01:40.997 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:01:40.997 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:01:40.997 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:01:40.997 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:01:40.997 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:01:40.997 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:01:40.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:40.998 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "70", 2024-03-30T14:01:40.998 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:01:40.998 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:01:40.998 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T14:01:40.998 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:01:40.998 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:01:40.998 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T14:01:40.998 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T14:01:40.998 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:01:40.998 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:01:40.998 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:01:40.998 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:01:40.998 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:01:40.998 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:01:40.999 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:01:40.999 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:01:40.999 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:01:40.999 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:01:40.999 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:01:40.999 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:01:40.999 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:01:40.999 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:01:40.999 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:01:40.999 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:01:40.999 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:01:40.999 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:01:40.999 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:01:40.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:40.999 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:01:41.000 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:01:41.000 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:01:41.000 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:01:41.000 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:01:41.000 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:01:41.000 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:01:41.000 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:01:41.000 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:01:41.000 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:01:41.000 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:01:41.000 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T14:01:41.000 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:01:41.000 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T14:01:41.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.000 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:01:41.000 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:01:41.000 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.6799999475479126, 2024-03-30T14:01:41.001 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.6799999475479126, 2024-03-30T14:01:41.001 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1.1699999570846558, 2024-03-30T14:01:41.001 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-30T14:01:41.001 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-30T14:01:41.001 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.75, 2024-03-30T14:01:41.001 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:01:41.001 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.75 2024-03-30T14:01:41.001 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.001 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 6, 2024-03-30T14:01:41.001 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.meta", 2024-03-30T14:01:41.001 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:49.637189+0000", 2024-03-30T14:01:41.001 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T14:01:41.002 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T14:01:41.002 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:01:41.002 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:01:41.002 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:01:41.002 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:01:41.002 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:01:41.002 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:01:41.002 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:01:41.002 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:01:41.002 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:01:41.002 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:01:41.002 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T14:01:41.002 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T14:01:41.002 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T14:01:41.003 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T14:01:41.003 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T14:01:41.003 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:01:41.003 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:01:41.003 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:01:41.003 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:01:41.003 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:01:41.003 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:01:41.003 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:01:41.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.003 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "72", 2024-03-30T14:01:41.003 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:01:41.003 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:01:41.003 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T14:01:41.003 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:01:41.003 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:01:41.004 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T14:01:41.004 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T14:01:41.004 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:01:41.004 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:01:41.004 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:01:41.004 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:01:41.004 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:01:41.004 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:01:41.004 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:01:41.004 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:01:41.004 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:01:41.004 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:01:41.004 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:01:41.004 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:01:41.004 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:01:41.005 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:01:41.005 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:01:41.005 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:01:41.005 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:01:41.005 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:01:41.005 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:01:41.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.005 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:01:41.005 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:01:41.005 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:01:41.005 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:01:41.005 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:01:41.005 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:01:41.005 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:01:41.005 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:01:41.006 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:01:41.006 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:01:41.006 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:01:41.006 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-03-30T14:01:41.006 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_bias": 4 2024-03-30T14:01:41.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.006 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:01:41.006 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T14:01:41.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.006 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:01:41.006 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:01:41.006 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.25, 2024-03-30T14:01:41.006 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.25, 2024-03-30T14:01:41.006 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1.1699999570846558, 2024-03-30T14:01:41.007 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2.630000114440918, 2024-03-30T14:01:41.007 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2.630000114440918, 2024-03-30T14:01:41.007 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.75, 2024-03-30T14:01:41.007 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:01:41.007 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.75 2024-03-30T14:01:41.007 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.007 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 11, 2024-03-30T14:01:41.007 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "unique_pool_4", 2024-03-30T14:01:41.007 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T14:01:12.894591+0000", 2024-03-30T14:01:41.007 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T14:01:41.007 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T14:01:41.007 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:01:41.007 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:01:41.008 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:01:41.008 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:01:41.008 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:01:41.008 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:01:41.008 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:01:41.008 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:01:41.008 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:01:41.008 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:01:41.008 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 16, 2024-03-30T14:01:41.008 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 16, 2024-03-30T14:01:41.008 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 16, 2024-03-30T14:01:41.008 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 16, 2024-03-30T14:01:41.008 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 16, 2024-03-30T14:01:41.008 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:01:41.008 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:01:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:01:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:01:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:01:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:01:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:01:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "384", 2024-03-30T14:01:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:01:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:01:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T14:01:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:01:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:01:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T14:01:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T14:01:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:01:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:01:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:01:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:01:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:01:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:01:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:01:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:01:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:01:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:01:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:01:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:01:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:01:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:01:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:01:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:01:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:01:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:01:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:01:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:01:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:01:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:01:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:01:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:01:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:01:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:01:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:01:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:01:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:01:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:01:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T14:01:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:01:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "rados": {} 2024-03-30T14:01:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:01:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:01:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.8700000047683716, 2024-03-30T14:01:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.8700000047683716, 2024-03-30T14:01:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1.1699999570846558, 2024-03-30T14:01:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2.190000057220459, 2024-03-30T14:01:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2.190000057220459, 2024-03-30T14:01:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.75, 2024-03-30T14:01:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:01:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.75 2024-03-30T14:01:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T14:01:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:01:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-03-30T14:01:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "94ea083e-6a27-41db-83af-2f7d3cae1956", 2024-03-30T14:01:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:01:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T14:01:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T14:01:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 0, 2024-03-30T14:01:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:01:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:01:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 8, 2024-03-30T14:01:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 377, 2024-03-30T14:01:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:01:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:01:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:01:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6802", 2024-03-30T14:01:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:01:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6803", 2024-03-30T14:01:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:01:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:01:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6804", 2024-03-30T14:01:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:01:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6805", 2024-03-30T14:01:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:01:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:01:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6808", 2024-03-30T14:01:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:01:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6809", 2024-03-30T14:01:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:01:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:01:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6806", 2024-03-30T14:01:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:01:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6807", 2024-03-30T14:01:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:01:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6803/1144592572", 2024-03-30T14:01:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6805/1144592572", 2024-03-30T14:01:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6809/1144592572", 2024-03-30T14:01:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6807/1144592572", 2024-03-30T14:01:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:01:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:01:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:01:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-03-30T14:01:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a", 2024-03-30T14:01:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:01:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T14:01:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T14:01:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T14:01:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:01:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:01:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 13, 2024-03-30T14:01:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 395, 2024-03-30T14:01:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:01:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:01:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:01:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6810", 2024-03-30T14:01:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:01:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6811", 2024-03-30T14:01:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:01:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:01:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6812", 2024-03-30T14:01:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:01:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6813", 2024-03-30T14:01:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:01:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:01:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6816", 2024-03-30T14:01:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:01:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6817", 2024-03-30T14:01:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:01:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:01:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6814", 2024-03-30T14:01:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:01:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6815", 2024-03-30T14:01:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:01:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6811/3901165928", 2024-03-30T14:01:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6813/3901165928", 2024-03-30T14:01:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6817/3901165928", 2024-03-30T14:01:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6815/3901165928", 2024-03-30T14:01:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:01:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:01:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:01:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-03-30T14:01:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "6db6782e-5889-443e-bc6c-df47a1f93bc8", 2024-03-30T14:01:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:01:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T14:01:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T14:01:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 0, 2024-03-30T14:01:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 18, 2024-03-30T14:01:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 166, 2024-03-30T14:01:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 388, 2024-03-30T14:01:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 98, 2024-03-30T14:01:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 167, 2024-03-30T14:01:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:01:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:01:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6818", 2024-03-30T14:01:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 886354154 2024-03-30T14:01:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6819", 2024-03-30T14:01:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 886354154 2024-03-30T14:01:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:01:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6820", 2024-03-30T14:01:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 886354154 2024-03-30T14:01:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6821", 2024-03-30T14:01:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 886354154 2024-03-30T14:01:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:01:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6824", 2024-03-30T14:01:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 886354154 2024-03-30T14:01:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6825", 2024-03-30T14:01:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 886354154 2024-03-30T14:01:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:01:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6822", 2024-03-30T14:01:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 886354154 2024-03-30T14:01:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6823", 2024-03-30T14:01:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 886354154 2024-03-30T14:01:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6819/886354154", 2024-03-30T14:01:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6821/886354154", 2024-03-30T14:01:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6825/886354154", 2024-03-30T14:01:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6823/886354154", 2024-03-30T14:01:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:01:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:01:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:01:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-03-30T14:01:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "7048ff46-2cba-4d18-9835-95ecbddb2bb6", 2024-03-30T14:01:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:01:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T14:01:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T14:01:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T14:01:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:01:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:01:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 25, 2024-03-30T14:01:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 391, 2024-03-30T14:01:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:01:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:01:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:01:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6826", 2024-03-30T14:01:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:01:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6827", 2024-03-30T14:01:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:01:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:01:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6828", 2024-03-30T14:01:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:01:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6829", 2024-03-30T14:01:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:01:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:01:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6832", 2024-03-30T14:01:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:01:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6833", 2024-03-30T14:01:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:01:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:01:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6830", 2024-03-30T14:01:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:01:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6831", 2024-03-30T14:01:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:01:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6827/2933589376", 2024-03-30T14:01:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6829/2933589376", 2024-03-30T14:01:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6833/2933589376", 2024-03-30T14:01:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6831/2933589376", 2024-03-30T14:01:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:01:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:01:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:01:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.033 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:41.034 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-03-30T14:01:41.034 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "42da30e6-9eec-4ed5-8c73-9ce78aa8ff63", 2024-03-30T14:01:41.034 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:01:41.034 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T14:01:41.034 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T14:01:41.034 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T14:01:41.034 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:01:41.034 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:01:41.034 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 32, 2024-03-30T14:01:41.034 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 394, 2024-03-30T14:01:41.034 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:01:41.035 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:01:41.035 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:01:41.035 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.035 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.035 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6800", 2024-03-30T14:01:41.035 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:01:41.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.035 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.035 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6801", 2024-03-30T14:01:41.035 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:01:41.035 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.035 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.035 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:01:41.036 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.036 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.036 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6802", 2024-03-30T14:01:41.036 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:01:41.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.036 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.036 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6803", 2024-03-30T14:01:41.036 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:01:41.036 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.036 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.036 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:01:41.037 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.037 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.037 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6806", 2024-03-30T14:01:41.037 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:01:41.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.037 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.037 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6807", 2024-03-30T14:01:41.037 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:01:41.037 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.037 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.037 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:01:41.037 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.038 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.038 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6804", 2024-03-30T14:01:41.038 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:01:41.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.038 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.038 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6805", 2024-03-30T14:01:41.038 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:01:41.038 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.038 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.038 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6801/1945833081", 2024-03-30T14:01:41.038 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6803/1945833081", 2024-03-30T14:01:41.038 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6807/1945833081", 2024-03-30T14:01:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6805/1945833081", 2024-03-30T14:01:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:01:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:01:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:01:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-03-30T14:01:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "99a11b81-0df7-4b1d-946a-4297ba62e263", 2024-03-30T14:01:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:01:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T14:01:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T14:01:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T14:01:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:01:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:01:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 40, 2024-03-30T14:01:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 391, 2024-03-30T14:01:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:01:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:01:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:01:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6808", 2024-03-30T14:01:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:01:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6809", 2024-03-30T14:01:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:01:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.041 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:01:41.041 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.041 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.041 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6810", 2024-03-30T14:01:41.041 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:01:41.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.041 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.041 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6811", 2024-03-30T14:01:41.041 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:01:41.041 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.041 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.042 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:01:41.042 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.042 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.042 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6814", 2024-03-30T14:01:41.042 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:01:41.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.042 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.042 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6815", 2024-03-30T14:01:41.042 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:01:41.042 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.042 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.042 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:01:41.043 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.043 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.043 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6812", 2024-03-30T14:01:41.043 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:01:41.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.043 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.043 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6813", 2024-03-30T14:01:41.043 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:01:41.043 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.043 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.043 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6809/3426360325", 2024-03-30T14:01:41.043 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6811/3426360325", 2024-03-30T14:01:41.044 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6815/3426360325", 2024-03-30T14:01:41.044 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6813/3426360325", 2024-03-30T14:01:41.044 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:01:41.044 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:01:41.044 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:01:41.044 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.044 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.044 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-03-30T14:01:41.044 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "cafd0e26-78e9-4164-a43a-96ca8382fefe", 2024-03-30T14:01:41.044 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:01:41.044 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T14:01:41.044 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T14:01:41.044 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T14:01:41.044 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:01:41.045 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:01:41.045 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 49, 2024-03-30T14:01:41.045 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 395, 2024-03-30T14:01:41.045 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:01:41.045 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:01:41.045 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:01:41.045 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.045 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.045 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.045 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6816", 2024-03-30T14:01:41.045 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:01:41.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.045 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.045 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.045 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6817", 2024-03-30T14:01:41.045 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:01:41.046 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.046 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.046 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:01:41.046 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.046 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.046 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.046 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6818", 2024-03-30T14:01:41.046 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:01:41.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.046 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.046 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.046 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6819", 2024-03-30T14:01:41.046 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:01:41.046 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.047 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.047 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:01:41.047 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.047 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.047 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.047 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6822", 2024-03-30T14:01:41.047 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:01:41.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.047 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.047 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.047 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6823", 2024-03-30T14:01:41.047 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:01:41.047 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.047 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.048 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:01:41.048 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.048 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.048 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6820", 2024-03-30T14:01:41.048 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:01:41.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.048 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.048 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6821", 2024-03-30T14:01:41.048 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:01:41.048 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.048 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.048 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6817/1527017698", 2024-03-30T14:01:41.049 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6819/1527017698", 2024-03-30T14:01:41.049 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6823/1527017698", 2024-03-30T14:01:41.049 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6821/1527017698", 2024-03-30T14:01:41.049 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:01:41.049 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:01:41.049 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:01:41.049 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.049 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.049 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-03-30T14:01:41.049 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "e9d59662-9132-48f4-9e95-842a3cfbd79f", 2024-03-30T14:01:41.049 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:01:41.049 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 0, 2024-03-30T14:01:41.049 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-03-30T14:01:41.049 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T14:01:41.050 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:01:41.050 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:01:41.050 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 58, 2024-03-30T14:01:41.050 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 228, 2024-03-30T14:01:41.050 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:01:41.050 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:01:41.050 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:01:41.050 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.050 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.050 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6824", 2024-03-30T14:01:41.051 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:40 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:41.051 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:40 smithi113 ceph-mon[34341]: pgmap v862: 168 pgs: 162 active+clean, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.9 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 87 MiB/s wr, 1.40k op/s; 4235/93699 objects degraded (4.520%); 21 MiB/s, 332 objects/s recovering 2024-03-30T14:01:41.051 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:40 smithi113 ceph-mon[34341]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:01:41.051 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:40 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:01:41.051 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:40 smithi113 ceph-mon[34341]: osdmap e395: 8 total, 8 up, 7 in 2024-03-30T14:01:41.051 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:40 smithi113 ceph-mon[34341]: 6.15 scrub starts 2024-03-30T14:01:41.051 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:40 smithi113 ceph-mon[34341]: 6.15 scrub ok 2024-03-30T14:01:41.051 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:40 smithi113 ceph-mon[34341]: 5.4 scrub starts 2024-03-30T14:01:41.051 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:40 smithi113 ceph-mon[34341]: 5.4 scrub ok 2024-03-30T14:01:41.052 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:01:41.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.052 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.052 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6825", 2024-03-30T14:01:41.052 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:01:41.052 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.052 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.052 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:01:41.052 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.052 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.052 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6826", 2024-03-30T14:01:41.053 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:01:41.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.053 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.053 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6827", 2024-03-30T14:01:41.053 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:01:41.053 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.053 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.053 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:01:41.053 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.053 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.053 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6830", 2024-03-30T14:01:41.053 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:01:41.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.054 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.054 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6831", 2024-03-30T14:01:41.054 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:01:41.054 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.054 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.054 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:01:41.054 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:01:41.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.054 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:01:41.054 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6828", 2024-03-30T14:01:41.054 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:01:41.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.055 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.055 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:01:41.055 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6829", 2024-03-30T14:01:41.055 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:01:41.055 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.055 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.055 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6825/3182476015", 2024-03-30T14:01:41.055 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6827/3182476015", 2024-03-30T14:01:41.055 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6831/3182476015", 2024-03-30T14:01:41.055 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6829/3182476015", 2024-03-30T14:01:41.055 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:01:41.055 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:01:41.055 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:01:41.055 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.055 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.056 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T14:01:41.056 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_xinfo": [ 2024-03-30T14:01:41.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.056 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-03-30T14:01:41.056 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:01:41.056 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:01:41.056 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:01:41.056 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:01:41.056 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T14:01:41.056 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T14:01:30.245122+0000", 2024-03-30T14:01:41.056 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:01:41.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.056 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-03-30T14:01:41.056 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:01:41.056 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:01:41.057 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:01:41.057 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:01:41.057 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T14:01:41.057 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T14:01:13.621213+0000", 2024-03-30T14:01:41.057 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:01:41.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.057 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-03-30T14:01:41.057 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:01:41.057 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:01:41.057 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:01:41.057 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:01:41.057 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T14:01:41.057 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T14:01:26.644187+0000", 2024-03-30T14:01:41.057 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 166 2024-03-30T14:01:41.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.058 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-03-30T14:01:41.058 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:01:41.058 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:01:41.058 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:01:41.058 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:01:41.058 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T14:01:41.058 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T14:01:22.343676+0000", 2024-03-30T14:01:41.058 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:01:41.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.058 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-03-30T14:01:41.058 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:01:41.058 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:01:41.058 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:01:41.058 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:01:41.059 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T14:01:41.059 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T14:01:26.628194+0000", 2024-03-30T14:01:41.059 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:01:41.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.059 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-03-30T14:01:41.059 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:01:41.059 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:01:41.059 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:01:41.059 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:01:41.059 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T14:01:41.059 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T14:00:23.753024+0000", 2024-03-30T14:01:41.059 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:01:41.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.060 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-03-30T14:01:41.060 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:01:41.060 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:01:41.060 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:01:41.060 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:01:41.060 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T14:01:41.060 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T14:01:37.799242+0000", 2024-03-30T14:01:41.060 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:01:41.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.060 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-03-30T14:01:41.060 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:01:41.060 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:01:41.060 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:01:41.060 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:01:41.060 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 65536, 2024-03-30T14:01:41.061 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T14:01:23.807351+0000", 2024-03-30T14:01:41.061 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:01:41.061 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.061 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T14:01:41.061 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap": [], 2024-03-30T14:01:41.061 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_items": [ 2024-03-30T14:01:41.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.061 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.14", 2024-03-30T14:01:41.061 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-03-30T14:01:41.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.061 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-03-30T14:01:41.061 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-03-30T14:01:41.061 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.061 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.061 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.061 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T14:01:41.062 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_primaries": [], 2024-03-30T14:01:41.062 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_temp": [ 2024-03-30T14:01:41.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.062 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "11.2", 2024-03-30T14:01:41.062 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:01:41.062 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-03-30T14:01:41.062 INFO:teuthology.orchestra.run.smithi043.stdout: 0 2024-03-30T14:01:41.062 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.062 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "11.6", 2024-03-30T14:01:41.062 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:01:41.062 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T14:01:41.062 INFO:teuthology.orchestra.run.smithi043.stdout: 0 2024-03-30T14:01:41.062 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:01:41.063 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "11.f", 2024-03-30T14:01:41.063 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:01:41.063 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T14:01:41.063 INFO:teuthology.orchestra.run.smithi043.stdout: 0 2024-03-30T14:01:41.063 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:01:41.063 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.063 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T14:01:41.063 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_temp": [], 2024-03-30T14:01:41.063 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklist": { 2024-03-30T14:01:41.063 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/538304234": "2024-03-31T13:43:27.253365+0000", 2024-03-30T14:01:41.063 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6801/3411723337": "2024-03-31T13:37:41.733515+0000", 2024-03-30T14:01:41.063 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3656823679": "2024-03-31T13:37:41.733515+0000", 2024-03-30T14:01:41.063 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2618817647": "2024-03-31T13:38:09.321740+0000", 2024-03-30T14:01:41.063 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/3411723337": "2024-03-31T13:37:41.733515+0000", 2024-03-30T14:01:41.064 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1206218784": "2024-03-31T13:37:41.733515+0000", 2024-03-30T14:01:41.064 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/886631486": "2024-03-31T13:37:41.733515+0000", 2024-03-30T14:01:41.064 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/1339921781": "2024-03-31T13:38:09.321740+0000", 2024-03-30T14:01:41.064 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6801/1339921781": "2024-03-31T13:38:09.321740+0000", 2024-03-30T14:01:41.064 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2794494528": "2024-03-31T13:38:09.321740+0000", 2024-03-30T14:01:41.064 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/564868778": "2024-03-31T13:38:09.321740+0000", 2024-03-30T14:01:41.064 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/3033707884": "2024-03-31T13:43:27.253365+0000", 2024-03-30T14:01:41.064 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3539398453": "2024-03-31T13:43:27.253365+0000", 2024-03-30T14:01:41.064 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2414364740": "2024-03-31T13:43:27.253365+0000", 2024-03-30T14:01:41.064 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6801/3033707884": "2024-03-31T13:43:27.253365+0000", 2024-03-30T14:01:41.064 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3923597429": "2024-03-31T13:43:27.253365+0000", 2024-03-30T14:01:41.064 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2524269526": "2024-03-31T13:43:27.253365+0000" 2024-03-30T14:01:41.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.064 INFO:teuthology.orchestra.run.smithi043.stdout: "range_blocklist": {}, 2024-03-30T14:01:41.064 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profiles": { 2024-03-30T14:01:41.065 INFO:teuthology.orchestra.run.smithi043.stdout: "default": { 2024-03-30T14:01:41.065 INFO:teuthology.orchestra.run.smithi043.stdout: "crush-failure-domain": "osd", 2024-03-30T14:01:41.065 INFO:teuthology.orchestra.run.smithi043.stdout: "k": "2", 2024-03-30T14:01:41.065 INFO:teuthology.orchestra.run.smithi043.stdout: "m": "1", 2024-03-30T14:01:41.065 INFO:teuthology.orchestra.run.smithi043.stdout: "plugin": "jerasure", 2024-03-30T14:01:41.065 INFO:teuthology.orchestra.run.smithi043.stdout: "technique": "reed_sol_van" 2024-03-30T14:01:41.065 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:01:41.065 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps_queue": [], 2024-03-30T14:01:41.065 INFO:teuthology.orchestra.run.smithi043.stdout: "new_removed_snaps": [], 2024-03-30T14:01:41.065 INFO:teuthology.orchestra.run.smithi043.stdout: "new_purged_snaps": [], 2024-03-30T14:01:41.065 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_node_flags": {}, 2024-03-30T14:01:41.065 INFO:teuthology.orchestra.run.smithi043.stdout: "device_class_flags": {}, 2024-03-30T14:01:41.065 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": { 2024-03-30T14:01:41.065 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_enabled": false, 2024-03-30T14:01:41.066 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_bucket_count": 0, 2024-03-30T14:01:41.066 INFO:teuthology.orchestra.run.smithi043.stdout: "degraded_stretch_mode": 0, 2024-03-30T14:01:41.066 INFO:teuthology.orchestra.run.smithi043.stdout: "recovering_stretch_mode": 0, 2024-03-30T14:01:41.066 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_bucket": 0 2024-03-30T14:01:41.066 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:01:41.066 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T14:01:41.067 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 21 16 32611 32595 96.9935 6.3125 0.0194928 0.0103024 2024-03-30T14:01:41.079 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:41.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:41.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:40 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:40 smithi043 ceph-mon[29963]: pgmap v862: 168 pgs: 162 active+clean, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.9 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 87 MiB/s wr, 1.40k op/s; 4235/93699 objects degraded (4.520%); 21 MiB/s, 332 objects/s recovering 2024-03-30T14:01:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:40 smithi043 ceph-mon[29963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:01:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:40 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:01:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:40 smithi043 ceph-mon[29963]: osdmap e395: 8 total, 8 up, 7 in 2024-03-30T14:01:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:40 smithi043 ceph-mon[29963]: 6.15 scrub starts 2024-03-30T14:01:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:40 smithi043 ceph-mon[29963]: 6.15 scrub ok 2024-03-30T14:01:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:40 smithi043 ceph-mon[29963]: 5.4 scrub starts 2024-03-30T14:01:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:40 smithi043 ceph-mon[29963]: 5.4 scrub ok 2024-03-30T14:01:41.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:40 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:41.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:40 smithi043 ceph-mon[42099]: pgmap v862: 168 pgs: 162 active+clean, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.9 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 87 MiB/s wr, 1.40k op/s; 4235/93699 objects degraded (4.520%); 21 MiB/s, 332 objects/s recovering 2024-03-30T14:01:41.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:40 smithi043 ceph-mon[42099]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:01:41.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:40 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:01:41.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:40 smithi043 ceph-mon[42099]: osdmap e395: 8 total, 8 up, 7 in 2024-03-30T14:01:41.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:40 smithi043 ceph-mon[42099]: 6.15 scrub starts 2024-03-30T14:01:41.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:40 smithi043 ceph-mon[42099]: 6.15 scrub ok 2024-03-30T14:01:41.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:40 smithi043 ceph-mon[42099]: 5.4 scrub starts 2024-03-30T14:01:41.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:40 smithi043 ceph-mon[42099]: 5.4 scrub ok 2024-03-30T14:01:41.180 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:41.225 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:41.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:41.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:41.327 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:41.341 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T14:01:41.342 INFO:teuthology.orchestra.run.smithi113.stdout: "size": 20, 2024-03-30T14:01:41.342 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 600, 2024-03-30T14:01:41.342 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [ 2024-03-30T14:01:41.342 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.342 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14451.0:2642 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94764809509632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-03-30T14:01:41.342 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:35.493028+0000", 2024-03-30T14:01:41.342 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 485.85195595300002, 2024-03-30T14:01:41.342 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.361405609, 2024-03-30T14:01:41.342 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:41.342 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:41.342 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T14:01:41.342 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:41.342 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14451", 2024-03-30T14:01:41.343 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T14:01:41.343 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 2642 2024-03-30T14:01:41.343 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.343 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:41.343 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.343 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:41.343 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493028+0000", 2024-03-30T14:01:41.343 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.343 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.343 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.343 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:41.343 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493028+0000", 2024-03-30T14:01:41.343 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.344 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.344 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.344 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:41.344 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493029+0000", 2024-03-30T14:01:41.344 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1600000000000001e-07 2024-03-30T14:01:41.344 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.344 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.344 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:41.344 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493033+0000", 2024-03-30T14:01:41.344 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3080000000000001e-06 2024-03-30T14:01:41.344 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.344 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.344 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:41.344 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493040+0000", 2024-03-30T14:01:41.344 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.3139999999999998e-06 2024-03-30T14:01:41.345 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.345 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.345 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:41.345 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493047+0000", 2024-03-30T14:01:41.345 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8889999999999996e-06 2024-03-30T14:01:41.345 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.345 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.345 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.345 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493230+0000", 2024-03-30T14:01:41.345 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00018283500000000001 2024-03-30T14:01:41.345 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.345 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.345 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.346 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493238+0000", 2024-03-30T14:01:41.346 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8120000000000008e-06 2024-03-30T14:01:41.346 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.346 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.346 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.346 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.846292+0000", 2024-03-30T14:01:41.346 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.35305440700000001 2024-03-30T14:01:41.346 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.346 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.346 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.346 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.846304+0000", 2024-03-30T14:01:41.346 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2001000000000001e-05 2024-03-30T14:01:41.346 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.346 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.346 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.347 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.854172+0000", 2024-03-30T14:01:41.347 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.00786771 2024-03-30T14:01:41.347 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.347 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.347 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:41.347 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.854348+0000", 2024-03-30T14:01:41.347 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00017568299999999999 2024-03-30T14:01:41.347 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.347 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.347 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:41.347 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.854434+0000", 2024-03-30T14:01:41.347 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5933999999999994e-05 2024-03-30T14:01:41.347 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.347 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.348 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.348 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.348 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.348 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14451.0:2647 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94764809509632 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T14:01:41.348 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:35.493415+0000", 2024-03-30T14:01:41.348 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 485.85156913600002, 2024-03-30T14:01:41.348 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.363426126, 2024-03-30T14:01:41.348 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:41.348 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:41.348 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T14:01:41.348 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:41.348 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14451", 2024-03-30T14:01:41.348 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T14:01:41.348 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 2647 2024-03-30T14:01:41.348 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.349 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:41.349 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.349 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:41.349 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493415+0000", 2024-03-30T14:01:41.349 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.349 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.349 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.349 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:41.349 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493415+0000", 2024-03-30T14:01:41.349 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.349 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.349 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.349 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:41.349 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493416+0000", 2024-03-30T14:01:41.349 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1399999999999996e-07 2024-03-30T14:01:41.350 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.350 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.350 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:41.350 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493420+0000", 2024-03-30T14:01:41.350 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.9389999999999999e-06 2024-03-30T14:01:41.350 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.350 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.350 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:41.350 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493434+0000", 2024-03-30T14:01:41.350 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4347e-05 2024-03-30T14:01:41.350 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.350 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.350 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:41.350 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493441+0000", 2024-03-30T14:01:41.350 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6120000000000004e-06 2024-03-30T14:01:41.351 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.351 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.351 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.351 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493506+0000", 2024-03-30T14:01:41.351 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5545999999999999e-05 2024-03-30T14:01:41.351 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.351 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.351 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.351 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.493516+0000", 2024-03-30T14:01:41.351 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.9320000000000008e-06 2024-03-30T14:01:41.351 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.351 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.351 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.351 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.846331+0000", 2024-03-30T14:01:41.351 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.35281492800000003 2024-03-30T14:01:41.351 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.352 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.352 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.352 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.846339+0000", 2024-03-30T14:01:41.352 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.2160000000000006e-06 2024-03-30T14:01:41.352 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.352 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.352 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.352 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.855384+0000", 2024-03-30T14:01:41.352 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0090448649999999 2024-03-30T14:01:41.352 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.352 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.352 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:41.352 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.855471+0000", 2024-03-30T14:01:41.352 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.7027999999999994e-05 2024-03-30T14:01:41.352 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.352 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.353 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:41.353 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.856841+0000", 2024-03-30T14:01:41.353 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001369999 2024-03-30T14:01:41.353 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.353 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.353 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.353 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.353 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.353 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14451.0:2652 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94764809509632 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-03-30T14:01:41.353 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:53:35.852719+0000", 2024-03-30T14:01:41.353 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 485.492265127, 2024-03-30T14:01:41.353 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.004857439, 2024-03-30T14:01:41.353 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:41.353 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:41.353 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T14:01:41.353 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:41.354 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14451", 2024-03-30T14:01:41.354 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T14:01:41.354 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 2652 2024-03-30T14:01:41.354 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.354 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:41.354 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.354 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:41.354 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.852719+0000", 2024-03-30T14:01:41.354 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.354 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.354 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.354 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:41.354 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.852719+0000", 2024-03-30T14:01:41.354 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.355 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.355 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.355 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:41.355 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.852720+0000", 2024-03-30T14:01:41.355 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2160000000000001e-06 2024-03-30T14:01:41.356 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.356 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.356 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:41.356 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.852724+0000", 2024-03-30T14:01:41.356 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.563e-06 2024-03-30T14:01:41.356 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.356 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.356 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:41.356 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.852733+0000", 2024-03-30T14:01:41.356 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.0909999999999999e-06 2024-03-30T14:01:41.356 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.356 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.356 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:41.356 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.852743+0000", 2024-03-30T14:01:41.356 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0006999999999999e-05 2024-03-30T14:01:41.357 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.357 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.357 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.357 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.853116+0000", 2024-03-30T14:01:41.357 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00037315500000000001 2024-03-30T14:01:41.357 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.357 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.357 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.357 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:35.853128+0000", 2024-03-30T14:01:41.357 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2007999999999999e-05 2024-03-30T14:01:41.357 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.357 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.357 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.357 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.855865+0000", 2024-03-30T14:01:41.357 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0027372569999999 2024-03-30T14:01:41.358 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.358 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.358 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:41.358 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.855917+0000", 2024-03-30T14:01:41.358 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1595000000000003e-05 2024-03-30T14:01:41.358 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.358 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.358 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:41.358 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:53:36.857576+0000", 2024-03-30T14:01:41.358 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0016595469999999999 2024-03-30T14:01:41.358 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.358 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.358 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.358 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.358 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.359 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:260334 10.7 10:efe0db43:::benchmark_data_smithi043_169264_object98107:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-30T14:01:41.359 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:55.834827+0000", 2024-03-30T14:01:41.359 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 105.510156743, 2024-03-30T14:01:41.359 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0284955389999999, 2024-03-30T14:01:41.359 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:41.359 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:41.359 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:41.359 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:41.359 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:01:41.359 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:01:41.359 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 260334 2024-03-30T14:01:41.359 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.359 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:41.359 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.359 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:41.359 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.834827+0000", 2024-03-30T14:01:41.360 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.360 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.360 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.360 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:41.360 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.834827+0000", 2024-03-30T14:01:41.360 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.360 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.360 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.360 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:41.360 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.834828+0000", 2024-03-30T14:01:41.360 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7599999999999997e-07 2024-03-30T14:01:41.360 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.360 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.360 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:41.360 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.834831+0000", 2024-03-30T14:01:41.360 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.0510000000000001e-06 2024-03-30T14:01:41.361 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.361 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.361 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:41.361 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.834839+0000", 2024-03-30T14:01:41.361 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.5730000000000002e-06 2024-03-30T14:01:41.361 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.361 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.361 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:41.361 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.834846+0000", 2024-03-30T14:01:41.361 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.0859999999999996e-06 2024-03-30T14:01:41.361 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.361 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.361 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.361 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.838202+0000", 2024-03-30T14:01:41.361 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.003356496 2024-03-30T14:01:41.361 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.362 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.362 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.362 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.838210+0000", 2024-03-30T14:01:41.362 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.5320000000000001e-06 2024-03-30T14:01:41.362 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.362 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.362 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.362 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.857161+0000", 2024-03-30T14:01:41.362 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.018951312 2024-03-30T14:01:41.362 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.362 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.362 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:41.362 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.857275+0000", 2024-03-30T14:01:41.362 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011414899999999999 2024-03-30T14:01:41.362 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.363 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.363 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,5", 2024-03-30T14:01:41.363 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.857357+0000", 2024-03-30T14:01:41.363 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.2050000000000002e-05 2024-03-30T14:01:41.363 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.363 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.363 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:41.363 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.861588+0000", 2024-03-30T14:01:41.363 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00423121 2024-03-30T14:01:41.363 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.363 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.363 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.363 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.863214+0000", 2024-03-30T14:01:41.363 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001625821 2024-03-30T14:01:41.363 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.363 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.364 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.364 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.863258+0000", 2024-03-30T14:01:41.364 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3872999999999998e-05 2024-03-30T14:01:41.364 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.364 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.364 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:41.364 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.863292+0000", 2024-03-30T14:01:41.364 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.4314999999999999e-05 2024-03-30T14:01:41.364 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.364 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.364 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:41.364 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.863323+0000", 2024-03-30T14:01:41.364 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.0494999999999999e-05 2024-03-30T14:01:41.364 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.364 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.364 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.365 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.365 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.365 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:260340 10.7 10:eb7c54a9:::benchmark_data_smithi043_169264_object98113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-30T14:01:41.365 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:55.841923+0000", 2024-03-30T14:01:41.365 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 105.50306080199999, 2024-03-30T14:01:41.365 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0254439769999999, 2024-03-30T14:01:41.365 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:41.365 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:41.365 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:41.365 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:41.365 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:01:41.365 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:01:41.365 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 260340 2024-03-30T14:01:41.365 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.365 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:41.366 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.366 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:41.366 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.841923+0000", 2024-03-30T14:01:41.366 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.366 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.366 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.366 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:41.366 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.841923+0000", 2024-03-30T14:01:41.366 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.366 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.366 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.366 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:41.366 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.841924+0000", 2024-03-30T14:01:41.366 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5600000000000005e-07 2024-03-30T14:01:41.367 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.367 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.367 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:41.367 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.841928+0000", 2024-03-30T14:01:41.367 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.5580000000000001e-06 2024-03-30T14:01:41.367 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.367 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.367 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:41.367 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.841936+0000", 2024-03-30T14:01:41.367 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5520000000000005e-06 2024-03-30T14:01:41.367 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.367 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.367 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:41.367 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.841942+0000", 2024-03-30T14:01:41.367 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5979999999999996e-06 2024-03-30T14:01:41.367 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.368 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.368 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.368 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.842102+0000", 2024-03-30T14:01:41.368 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000160137 2024-03-30T14:01:41.368 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.368 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.368 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.368 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.842111+0000", 2024-03-30T14:01:41.368 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.9069999999999996e-06 2024-03-30T14:01:41.368 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.368 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.368 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.368 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.859768+0000", 2024-03-30T14:01:41.368 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0176572770000001 2024-03-30T14:01:41.369 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.369 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.369 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:41.369 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.859857+0000", 2024-03-30T14:01:41.369 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.8713000000000003e-05 2024-03-30T14:01:41.369 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.369 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.369 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,5", 2024-03-30T14:01:41.369 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.859924+0000", 2024-03-30T14:01:41.369 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.7358000000000001e-05 2024-03-30T14:01:41.369 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.369 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.369 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:41.369 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.863335+0000", 2024-03-30T14:01:41.369 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0034107360000000001 2024-03-30T14:01:41.370 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.370 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.370 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.370 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.863367+0000", 2024-03-30T14:01:41.370 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2129999999999999e-05 2024-03-30T14:01:41.370 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.370 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.370 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.370 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.867267+0000", 2024-03-30T14:01:41.370 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0039002849999999999 2024-03-30T14:01:41.370 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.370 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.370 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:41.370 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.867322+0000", 2024-03-30T14:01:41.371 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5158999999999998e-05 2024-03-30T14:01:41.371 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.371 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.371 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:41.371 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.867367+0000", 2024-03-30T14:01:41.371 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.4910999999999997e-05 2024-03-30T14:01:41.371 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.371 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.371 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.371 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.371 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.371 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:260342 10.7 10:ec9ba1bd:::benchmark_data_smithi043_169264_object98115:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-30T14:01:41.371 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:55.844063+0000", 2024-03-30T14:01:41.371 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 105.50092118000001, 2024-03-30T14:01:41.371 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0234843279999999, 2024-03-30T14:01:41.372 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:41.372 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:41.372 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:41.372 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:41.372 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:01:41.372 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:01:41.372 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 260342 2024-03-30T14:01:41.372 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.372 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:41.372 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.372 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:41.372 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.844063+0000", 2024-03-30T14:01:41.372 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.372 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.372 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.373 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:41.373 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.844063+0000", 2024-03-30T14:01:41.373 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.373 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.373 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.373 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:41.373 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.844064+0000", 2024-03-30T14:01:41.373 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5199999999999996e-07 2024-03-30T14:01:41.373 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.373 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.373 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:41.373 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.844068+0000", 2024-03-30T14:01:41.373 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.493e-06 2024-03-30T14:01:41.373 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.373 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.373 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:41.374 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.844080+0000", 2024-03-30T14:01:41.374 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2203e-05 2024-03-30T14:01:41.374 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.374 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.374 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:41.374 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.844087+0000", 2024-03-30T14:01:41.374 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.9530000000000002e-06 2024-03-30T14:01:41.374 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.374 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.374 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.374 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.844137+0000", 2024-03-30T14:01:41.374 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9953000000000001e-05 2024-03-30T14:01:41.374 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.375 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.375 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.375 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.844146+0000", 2024-03-30T14:01:41.375 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5050000000000007e-06 2024-03-30T14:01:41.375 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.375 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.375 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.375 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.861620+0000", 2024-03-30T14:01:41.375 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.017474392 2024-03-30T14:01:41.375 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.375 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.375 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:41.375 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.861705+0000", 2024-03-30T14:01:41.375 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5334000000000007e-05 2024-03-30T14:01:41.375 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.376 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.376 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,5", 2024-03-30T14:01:41.376 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.861802+0000", 2024-03-30T14:01:41.376 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.6874999999999997e-05 2024-03-30T14:01:41.376 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.376 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.376 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:41.376 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.865102+0000", 2024-03-30T14:01:41.376 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0032999829999999998 2024-03-30T14:01:41.376 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.376 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.376 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.376 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.867205+0000", 2024-03-30T14:01:41.376 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0021024609999999999 2024-03-30T14:01:41.376 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.377 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.377 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.377 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.867456+0000", 2024-03-30T14:01:41.377 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00025083800000000003 2024-03-30T14:01:41.377 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.377 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.377 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:41.377 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.867508+0000", 2024-03-30T14:01:41.377 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2096999999999999e-05 2024-03-30T14:01:41.377 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.377 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.377 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:41.377 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.867547+0000", 2024-03-30T14:01:41.377 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.9589000000000003e-05 2024-03-30T14:01:41.377 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.377 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.378 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.378 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.378 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.378 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:260402 10.7 10:ee21a1bd:::benchmark_data_smithi043_169264_object98175:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-30T14:01:41.378 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:55.895475+0000", 2024-03-30T14:01:41.378 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 105.44950915699999, 2024-03-30T14:01:41.378 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.97641977400000002, 2024-03-30T14:01:41.378 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:41.378 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:41.378 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:41.378 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:41.378 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:01:41.378 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:01:41.378 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 260402 2024-03-30T14:01:41.378 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.379 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:41.379 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.379 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:41.379 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.895475+0000", 2024-03-30T14:01:41.379 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.379 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.379 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.379 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:41.379 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.895475+0000", 2024-03-30T14:01:41.379 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.379 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.379 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.379 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:41.379 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.895476+0000", 2024-03-30T14:01:41.379 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2699999999999999e-06 2024-03-30T14:01:41.379 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.380 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.380 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:41.380 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.895481+0000", 2024-03-30T14:01:41.380 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3499999999999999e-06 2024-03-30T14:01:41.380 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.380 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.380 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:41.380 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.895491+0000", 2024-03-30T14:01:41.380 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0616e-05 2024-03-30T14:01:41.380 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.380 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.380 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:41.380 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.895497+0000", 2024-03-30T14:01:41.381 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1020000000000002e-06 2024-03-30T14:01:41.381 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.381 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.381 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.381 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.895549+0000", 2024-03-30T14:01:41.381 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1609e-05 2024-03-30T14:01:41.381 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.381 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.381 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.381 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.895559+0000", 2024-03-30T14:01:41.381 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0079e-05 2024-03-30T14:01:41.381 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.381 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.381 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.381 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.863392+0000", 2024-03-30T14:01:41.382 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.96783282999999998 2024-03-30T14:01:41.382 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.382 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.382 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:41.382 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.863481+0000", 2024-03-30T14:01:41.382 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.8752999999999995e-05 2024-03-30T14:01:41.382 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.382 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.382 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,5", 2024-03-30T14:01:41.382 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.863559+0000", 2024-03-30T14:01:41.382 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8676000000000004e-05 2024-03-30T14:01:41.382 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.382 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.382 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:41.382 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.867414+0000", 2024-03-30T14:01:41.383 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0038545160000000001 2024-03-30T14:01:41.383 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.383 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.383 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.383 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.867604+0000", 2024-03-30T14:01:41.383 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00018992400000000001 2024-03-30T14:01:41.383 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.383 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.383 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.383 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.871789+0000", 2024-03-30T14:01:41.383 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0041857259999999999 2024-03-30T14:01:41.383 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.383 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.383 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:41.383 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.871864+0000", 2024-03-30T14:01:41.383 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.5081999999999997e-05 2024-03-30T14:01:41.384 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.384 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.384 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:41.384 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.871895+0000", 2024-03-30T14:01:41.384 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.0241000000000001e-05 2024-03-30T14:01:41.384 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.384 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.384 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.384 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.384 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.384 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:260404 10.7 10:effbd01f:::benchmark_data_smithi043_169264_object98177:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-30T14:01:41.384 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:55.895860+0000", 2024-03-30T14:01:41.384 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 105.449124266, 2024-03-30T14:01:41.384 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.97613881400000002, 2024-03-30T14:01:41.384 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:41.385 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:41.385 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:41.385 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:41.385 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:01:41.385 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:01:41.385 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 260404 2024-03-30T14:01:41.385 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.385 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:41.385 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.385 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:41.385 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.895860+0000", 2024-03-30T14:01:41.385 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.385 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.385 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.386 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:41.386 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.895860+0000", 2024-03-30T14:01:41.386 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.386 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.386 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.386 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:41.386 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.895861+0000", 2024-03-30T14:01:41.386 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5099999999999998e-07 2024-03-30T14:01:41.386 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.386 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.386 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:41.386 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.895865+0000", 2024-03-30T14:01:41.386 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6469999999999997e-06 2024-03-30T14:01:41.387 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.387 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.387 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:41.387 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.895876+0000", 2024-03-30T14:01:41.387 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1143e-05 2024-03-30T14:01:41.387 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.387 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.387 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:41.387 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.895882+0000", 2024-03-30T14:01:41.387 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3700000000000003e-06 2024-03-30T14:01:41.387 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.387 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.387 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.387 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.895921+0000", 2024-03-30T14:01:41.387 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.9014000000000003e-05 2024-03-30T14:01:41.387 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.387 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.388 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.388 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.895929+0000", 2024-03-30T14:01:41.388 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.79e-06 2024-03-30T14:01:41.388 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.388 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.388 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.388 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.865135+0000", 2024-03-30T14:01:41.388 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.96920621600000001 2024-03-30T14:01:41.388 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.388 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.388 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:41.388 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.865209+0000", 2024-03-30T14:01:41.388 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.3646000000000006e-05 2024-03-30T14:01:41.388 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.388 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.388 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,5", 2024-03-30T14:01:41.389 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.865285+0000", 2024-03-30T14:01:41.389 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.6379999999999997e-05 2024-03-30T14:01:41.389 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.389 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.389 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:41.389 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.867559+0000", 2024-03-30T14:01:41.389 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0022743400000000001 2024-03-30T14:01:41.389 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.389 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.389 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.389 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.867653+0000", 2024-03-30T14:01:41.389 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4153000000000005e-05 2024-03-30T14:01:41.389 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.389 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.389 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.389 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.871936+0000", 2024-03-30T14:01:41.390 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0042828559999999998 2024-03-30T14:01:41.390 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.390 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.390 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:41.390 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.871965+0000", 2024-03-30T14:01:41.390 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.9088999999999999e-05 2024-03-30T14:01:41.390 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.390 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.390 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:41.390 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.871999+0000", 2024-03-30T14:01:41.390 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.3318999999999997e-05 2024-03-30T14:01:41.390 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.390 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.390 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.391 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.391 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.391 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:260432 10.7 10:e72207fe:::benchmark_data_smithi043_169264_object98205:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-30T14:01:41.391 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:55.915726+0000", 2024-03-30T14:01:41.391 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 105.42925805, 2024-03-30T14:01:41.391 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.96085714, 2024-03-30T14:01:41.391 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:41.391 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:41.391 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:41.391 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:41.391 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:01:41.391 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:01:41.391 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 260432 2024-03-30T14:01:41.391 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.391 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:41.391 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.392 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:41.392 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.915726+0000", 2024-03-30T14:01:41.392 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.392 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.392 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.392 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:41.392 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.915726+0000", 2024-03-30T14:01:41.392 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.392 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.392 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.392 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:41.392 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.915728+0000", 2024-03-30T14:01:41.392 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.652e-06 2024-03-30T14:01:41.392 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.392 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.393 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:41.393 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.915733+0000", 2024-03-30T14:01:41.393 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3469999999999998e-06 2024-03-30T14:01:41.393 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.393 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.393 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:41.393 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.915749+0000", 2024-03-30T14:01:41.393 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.6201999999999998e-05 2024-03-30T14:01:41.393 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.393 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:41.395 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:41.395 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.915758+0000", 2024-03-30T14:01:41.395 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5520000000000005e-06 2024-03-30T14:01:41.395 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.395 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.395 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.395 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.915885+0000", 2024-03-30T14:01:41.395 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012688200000000001 2024-03-30T14:01:41.395 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.395 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.396 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.396 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.915893+0000", 2024-03-30T14:01:41.396 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.7220000000000005e-06 2024-03-30T14:01:41.396 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.396 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.396 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.396 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.867694+0000", 2024-03-30T14:01:41.396 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.95180089800000001 2024-03-30T14:01:41.396 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.396 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.396 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:41.396 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.867807+0000", 2024-03-30T14:01:41.396 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011309100000000001 2024-03-30T14:01:41.396 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.396 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.396 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,5", 2024-03-30T14:01:41.397 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.867908+0000", 2024-03-30T14:01:41.397 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00010021700000000001 2024-03-30T14:01:41.397 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.397 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.397 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.397 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.872047+0000", 2024-03-30T14:01:41.397 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0041396130000000003 2024-03-30T14:01:41.397 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.397 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.397 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:41.397 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.872060+0000", 2024-03-30T14:01:41.397 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2585999999999999e-05 2024-03-30T14:01:41.397 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.397 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.398 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.398 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.876517+0000", 2024-03-30T14:01:41.398 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0044575700000000001 2024-03-30T14:01:41.398 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.398 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.398 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:41.398 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.876550+0000", 2024-03-30T14:01:41.398 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2261000000000002e-05 2024-03-30T14:01:41.398 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.398 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.398 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:41.398 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.876583+0000", 2024-03-30T14:01:41.398 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.3547000000000003e-05 2024-03-30T14:01:41.398 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.399 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.399 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.399 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.399 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.399 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:260461 10.7 10:ec40998f:::benchmark_data_smithi043_169264_object98234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-30T14:01:41.399 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:55.939273+0000", 2024-03-30T14:01:41.399 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 105.405710971, 2024-03-30T14:01:41.399 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.93747376599999999, 2024-03-30T14:01:41.399 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:41.399 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:41.399 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:41.399 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:41.399 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:01:41.399 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:01:41.399 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 260461 2024-03-30T14:01:41.399 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.400 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:41.400 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.400 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:41.400 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.939273+0000", 2024-03-30T14:01:41.400 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.400 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.400 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.400 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:41.400 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.939273+0000", 2024-03-30T14:01:41.400 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.400 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.400 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.400 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:41.400 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.939274+0000", 2024-03-30T14:01:41.400 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.043e-06 2024-03-30T14:01:41.401 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.401 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.401 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:41.401 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.939279+0000", 2024-03-30T14:01:41.401 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2519999999999999e-06 2024-03-30T14:01:41.401 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.401 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.401 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:41.401 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.939295+0000", 2024-03-30T14:01:41.401 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5926999999999998e-05 2024-03-30T14:01:41.401 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.401 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.401 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:41.401 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.939305+0000", 2024-03-30T14:01:41.401 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0074999999999999e-05 2024-03-30T14:01:41.402 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.402 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.402 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.402 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.939377+0000", 2024-03-30T14:01:41.402 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1631999999999995e-05 2024-03-30T14:01:41.402 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.402 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.402 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.402 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.939384+0000", 2024-03-30T14:01:41.402 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.3329999999999999e-06 2024-03-30T14:01:41.402 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.402 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.402 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.402 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.869685+0000", 2024-03-30T14:01:41.402 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.93030091500000001 2024-03-30T14:01:41.403 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.403 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.403 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:41.403 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.869795+0000", 2024-03-30T14:01:41.403 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000109251 2024-03-30T14:01:41.403 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.403 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.403 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,5", 2024-03-30T14:01:41.403 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.869915+0000", 2024-03-30T14:01:41.403 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012059500000000001 2024-03-30T14:01:41.403 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.403 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.403 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:41.403 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.872067+0000", 2024-03-30T14:01:41.403 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0021515190000000002 2024-03-30T14:01:41.404 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.404 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.404 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.404 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.872096+0000", 2024-03-30T14:01:41.404 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.9292999999999999e-05 2024-03-30T14:01:41.404 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.404 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.404 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.404 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.876643+0000", 2024-03-30T14:01:41.404 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0045473639999999999 2024-03-30T14:01:41.404 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.404 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.404 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:41.404 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.876699+0000", 2024-03-30T14:01:41.404 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5447e-05 2024-03-30T14:01:41.404 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.405 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.405 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:41.405 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.876747+0000", 2024-03-30T14:01:41.405 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.812e-05 2024-03-30T14:01:41.405 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.405 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.405 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.405 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.405 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.405 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:260473 10.7 10:e491a6fc:::benchmark_data_smithi043_169264_object98246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-30T14:01:41.405 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:55.947715+0000", 2024-03-30T14:01:41.405 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 105.397268919, 2024-03-30T14:01:41.405 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.92924536499999999, 2024-03-30T14:01:41.405 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:41.406 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:41.406 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:41.406 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:41.406 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:01:41.406 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:01:41.406 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 260473 2024-03-30T14:01:41.406 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.406 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:41.406 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.406 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:41.406 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.947715+0000", 2024-03-30T14:01:41.406 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.406 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.407 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.407 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:41.407 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.947715+0000", 2024-03-30T14:01:41.407 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.407 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.407 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.407 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:41.407 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.947716+0000", 2024-03-30T14:01:41.407 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7299999999999996e-07 2024-03-30T14:01:41.407 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.407 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.407 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:41.407 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.947719+0000", 2024-03-30T14:01:41.407 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.8820000000000001e-06 2024-03-30T14:01:41.408 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.408 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.408 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:41.408 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.947726+0000", 2024-03-30T14:01:41.408 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.537e-06 2024-03-30T14:01:41.408 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.408 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.408 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:41.408 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.947732+0000", 2024-03-30T14:01:41.408 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7010000000000001e-06 2024-03-30T14:01:41.408 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.408 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.408 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.408 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.947774+0000", 2024-03-30T14:01:41.409 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1746999999999999e-05 2024-03-30T14:01:41.409 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.409 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.409 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.409 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:55.947779+0000", 2024-03-30T14:01:41.409 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9570000000000001e-06 2024-03-30T14:01:41.409 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.409 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.409 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.409 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.872128+0000", 2024-03-30T14:01:41.409 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.92434952699999995 2024-03-30T14:01:41.409 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.409 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.409 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:41.409 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.872246+0000", 2024-03-30T14:01:41.410 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011767799999999999 2024-03-30T14:01:41.410 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.410 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.410 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,5", 2024-03-30T14:01:41.410 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.872368+0000", 2024-03-30T14:01:41.410 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012173800000000001 2024-03-30T14:01:41.410 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.410 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.410 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:41.410 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.876767+0000", 2024-03-30T14:01:41.410 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.004399605 2024-03-30T14:01:41.410 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.410 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.410 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.411 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.876814+0000", 2024-03-30T14:01:41.411 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7052999999999998e-05 2024-03-30T14:01:41.411 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.411 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.411 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.411 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.876877+0000", 2024-03-30T14:01:41.411 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.2589000000000007e-05 2024-03-30T14:01:41.411 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.411 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.411 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:41.411 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.876920+0000", 2024-03-30T14:01:41.411 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.2953e-05 2024-03-30T14:01:41.411 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.411 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.412 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:41.412 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:56.876961+0000", 2024-03-30T14:01:41.412 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.0825000000000003e-05 2024-03-30T14:01:41.412 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.412 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.412 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.412 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.412 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.412 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:32449 11.4 11:2fd48be7:::benchmark_data_smithi043_197439_object32448:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-03-30T14:01:41.412 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:01:39.997826+0000", 2024-03-30T14:01:41.412 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.347157766, 2024-03-30T14:01:41.412 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.99870248100000003, 2024-03-30T14:01:41.412 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:41.412 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:41.412 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:41.413 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:41.413 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:01:41.413 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:01:41.413 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 32449 2024-03-30T14:01:41.413 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.413 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:41.413 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.413 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:41.413 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:39.997826+0000", 2024-03-30T14:01:41.413 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.413 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.413 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.413 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:41.413 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:39.997826+0000", 2024-03-30T14:01:41.413 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.414 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.414 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.414 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:41.414 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:39.997827+0000", 2024-03-30T14:01:41.414 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.4600000000000004e-07 2024-03-30T14:01:41.414 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.414 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.414 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:41.414 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:39.997858+0000", 2024-03-30T14:01:41.414 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.0765000000000001e-05 2024-03-30T14:01:41.414 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.414 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.414 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:41.414 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:39.997868+0000", 2024-03-30T14:01:41.415 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.04e-05 2024-03-30T14:01:41.415 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.415 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.415 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:41.415 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:39.997875+0000", 2024-03-30T14:01:41.415 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6540000000000002e-06 2024-03-30T14:01:41.415 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.415 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.415 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.415 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:39.997960+0000", 2024-03-30T14:01:41.415 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5270000000000002e-05 2024-03-30T14:01:41.415 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.415 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.415 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.416 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:39.997970+0000", 2024-03-30T14:01:41.416 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0214000000000001e-05 2024-03-30T14:01:41.416 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.416 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.416 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.416 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.980900+0000", 2024-03-30T14:01:41.416 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.98292963899999997 2024-03-30T14:01:41.416 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.416 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.416 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:41.416 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.981017+0000", 2024-03-30T14:01:41.416 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011705900000000001 2024-03-30T14:01:41.416 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.416 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.417 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,2", 2024-03-30T14:01:41.417 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.981118+0000", 2024-03-30T14:01:41.417 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00010137300000000001 2024-03-30T14:01:41.417 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.417 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.417 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:41.417 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.984862+0000", 2024-03-30T14:01:41.417 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0037436819999999999 2024-03-30T14:01:41.417 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.417 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.417 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.417 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.989002+0000", 2024-03-30T14:01:41.417 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0041395759999999998 2024-03-30T14:01:41.417 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.417 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.418 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.418 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.996417+0000", 2024-03-30T14:01:41.418 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0074150759999999996 2024-03-30T14:01:41.418 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.418 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.418 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:41.418 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.996477+0000", 2024-03-30T14:01:41.418 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0333e-05 2024-03-30T14:01:41.418 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.418 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.418 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:41.418 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.996529+0000", 2024-03-30T14:01:41.419 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1793999999999999e-05 2024-03-30T14:01:41.419 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.419 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.419 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.419 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.419 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.419 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:32462 11.4 11:207ccde6:::benchmark_data_smithi043_197439_object32461:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-03-30T14:01:41.419 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:01:40.008799+0000", 2024-03-30T14:01:41.419 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.33618543, 2024-03-30T14:01:41.419 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.98786624700000003, 2024-03-30T14:01:41.419 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:41.419 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:41.419 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:41.419 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:41.419 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:01:41.420 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:01:41.420 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 32462 2024-03-30T14:01:41.420 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.420 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:41.420 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.420 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:41.420 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.008799+0000", 2024-03-30T14:01:41.420 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.420 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.420 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.420 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:41.420 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.008799+0000", 2024-03-30T14:01:41.420 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.420 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.420 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.421 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:41.421 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.008801+0000", 2024-03-30T14:01:41.421 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.88e-06 2024-03-30T14:01:41.421 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.421 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.421 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:41.421 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.008852+0000", 2024-03-30T14:01:41.421 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1826999999999997e-05 2024-03-30T14:01:41.421 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.421 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.421 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:41.421 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.008868+0000", 2024-03-30T14:01:41.421 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5160000000000001e-05 2024-03-30T14:01:41.421 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.421 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.422 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:41.422 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.008876+0000", 2024-03-30T14:01:41.422 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.7930000000000003e-06 2024-03-30T14:01:41.422 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.422 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.422 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.422 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.008956+0000", 2024-03-30T14:01:41.422 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.9393000000000004e-05 2024-03-30T14:01:41.422 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.422 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.422 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.422 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.009962+0000", 2024-03-30T14:01:41.422 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0010065860000000001 2024-03-30T14:01:41.422 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.423 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.423 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.423 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.982351+0000", 2024-03-30T14:01:41.423 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.97238893100000001 2024-03-30T14:01:41.423 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.423 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.423 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:41.423 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.982457+0000", 2024-03-30T14:01:41.423 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00010556300000000001 2024-03-30T14:01:41.423 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.423 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.423 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,2", 2024-03-30T14:01:41.423 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.982545+0000", 2024-03-30T14:01:41.423 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.8578999999999993e-05 2024-03-30T14:01:41.423 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.424 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.424 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:41.424 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.986773+0000", 2024-03-30T14:01:41.424 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0042275100000000003 2024-03-30T14:01:41.424 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.424 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.424 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.424 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.989038+0000", 2024-03-30T14:01:41.424 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.002264725 2024-03-30T14:01:41.424 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.424 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.424 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.424 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.996574+0000", 2024-03-30T14:01:41.424 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0075361919999999997 2024-03-30T14:01:41.424 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.424 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.425 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:41.425 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.996629+0000", 2024-03-30T14:01:41.425 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4876000000000001e-05 2024-03-30T14:01:41.425 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.425 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.425 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:41.425 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.996665+0000", 2024-03-30T14:01:41.425 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.6232000000000001e-05 2024-03-30T14:01:41.425 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.425 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.425 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.425 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.425 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.425 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:32475 11.4 11:21656ee6:::benchmark_data_smithi043_197439_object32474:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-03-30T14:01:41.425 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:01:40.018245+0000", 2024-03-30T14:01:41.426 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.326738797, 2024-03-30T14:01:41.426 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.97851027800000001, 2024-03-30T14:01:41.426 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:41.426 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:41.426 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:41.426 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:41.426 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:01:41.426 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:01:41.426 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 32475 2024-03-30T14:01:41.426 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.426 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:41.426 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.426 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:41.426 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.018245+0000", 2024-03-30T14:01:41.427 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.427 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.427 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.427 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:41.427 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.018245+0000", 2024-03-30T14:01:41.427 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.427 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.427 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.427 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:41.427 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.018246+0000", 2024-03-30T14:01:41.427 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8899999999999999e-07 2024-03-30T14:01:41.427 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.427 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.427 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:41.427 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.018275+0000", 2024-03-30T14:01:41.427 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.9257000000000001e-05 2024-03-30T14:01:41.428 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.428 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.428 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:41.428 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.018281+0000", 2024-03-30T14:01:41.428 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4110000000000004e-06 2024-03-30T14:01:41.428 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.428 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.428 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:41.428 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.018285+0000", 2024-03-30T14:01:41.428 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.0319999999999997e-06 2024-03-30T14:01:41.428 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.428 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.428 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.428 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.018372+0000", 2024-03-30T14:01:41.428 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.7620000000000005e-05 2024-03-30T14:01:41.429 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.429 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.429 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.429 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.018380+0000", 2024-03-30T14:01:41.429 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.875e-06 2024-03-30T14:01:41.429 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.429 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.429 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.429 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.983316+0000", 2024-03-30T14:01:41.429 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.96493569499999998 2024-03-30T14:01:41.429 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.429 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.429 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:41.429 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.983391+0000", 2024-03-30T14:01:41.430 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.5103000000000002e-05 2024-03-30T14:01:41.430 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.430 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.430 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,2", 2024-03-30T14:01:41.430 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.983480+0000", 2024-03-30T14:01:41.430 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.9247999999999997e-05 2024-03-30T14:01:41.430 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.430 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.430 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:41.430 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.986779+0000", 2024-03-30T14:01:41.430 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0032988409999999998 2024-03-30T14:01:41.430 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.431 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.431 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.431 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.989068+0000", 2024-03-30T14:01:41.431 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.002288787 2024-03-30T14:01:41.431 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.431 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.431 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.431 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.996693+0000", 2024-03-30T14:01:41.431 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0076249810000000003 2024-03-30T14:01:41.431 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.431 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.431 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:41.431 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.996723+0000", 2024-03-30T14:01:41.431 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.0043999999999999e-05 2024-03-30T14:01:41.432 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.432 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.432 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:41.432 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.996756+0000", 2024-03-30T14:01:41.432 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2795e-05 2024-03-30T14:01:41.432 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.432 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.432 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.432 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.432 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.432 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:32478 11.4 11:2d59cc6a:::benchmark_data_smithi043_197439_object32477:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-03-30T14:01:41.432 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:01:40.018296+0000", 2024-03-30T14:01:41.432 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.326688401, 2024-03-30T14:01:41.433 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.97861082899999996, 2024-03-30T14:01:41.433 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:41.433 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:41.433 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:41.433 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:41.433 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:01:41.433 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:01:41.433 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 32478 2024-03-30T14:01:41.433 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.433 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:41.433 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.433 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:41.433 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.018296+0000", 2024-03-30T14:01:41.433 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.433 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.434 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.434 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:41.434 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.018296+0000", 2024-03-30T14:01:41.434 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.434 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.434 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.434 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:41.434 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.018296+0000", 2024-03-30T14:01:41.434 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6400000000000003e-07 2024-03-30T14:01:41.434 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.434 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.434 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:41.434 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.018319+0000", 2024-03-30T14:01:41.435 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.2816999999999999e-05 2024-03-30T14:01:41.435 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.435 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.435 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:41.435 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.018324+0000", 2024-03-30T14:01:41.435 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6410000000000001e-06 2024-03-30T14:01:41.435 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.435 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.435 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:41.435 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.018327+0000", 2024-03-30T14:01:41.435 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2830000000000002e-06 2024-03-30T14:01:41.435 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.435 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.435 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.436 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.018414+0000", 2024-03-30T14:01:41.436 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6776999999999999e-05 2024-03-30T14:01:41.436 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.436 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.436 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.436 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.018419+0000", 2024-03-30T14:01:41.436 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7660000000000004e-06 2024-03-30T14:01:41.436 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.436 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.436 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.436 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.984124+0000", 2024-03-30T14:01:41.436 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.96570492299999999 2024-03-30T14:01:41.436 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.437 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.437 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:41.437 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.984202+0000", 2024-03-30T14:01:41.437 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.7917999999999995e-05 2024-03-30T14:01:41.437 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.437 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.437 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,2", 2024-03-30T14:01:41.437 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.984294+0000", 2024-03-30T14:01:41.437 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.1661000000000002e-05 2024-03-30T14:01:41.437 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.437 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.437 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:41.437 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.988210+0000", 2024-03-30T14:01:41.437 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0039162809999999998 2024-03-30T14:01:41.437 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.438 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.438 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.438 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.989100+0000", 2024-03-30T14:01:41.438 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00088995399999999998 2024-03-30T14:01:41.438 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.438 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.438 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.438 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.996806+0000", 2024-03-30T14:01:41.438 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0077062650000000003 2024-03-30T14:01:41.438 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.438 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.438 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:41.438 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.996865+0000", 2024-03-30T14:01:41.438 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8643000000000001e-05 2024-03-30T14:01:41.438 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.439 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.439 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:41.439 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.996907+0000", 2024-03-30T14:01:41.439 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1436000000000001e-05 2024-03-30T14:01:41.439 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.439 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.439 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.439 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.439 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.439 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:32482 11.4 11:20083e2a:::benchmark_data_smithi043_197439_object32481:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-03-30T14:01:41.439 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:01:40.022333+0000", 2024-03-30T14:01:41.439 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.322651354, 2024-03-30T14:01:41.439 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.97467189399999998, 2024-03-30T14:01:41.439 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:41.439 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:41.439 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:41.440 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:41.440 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:01:41.440 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:01:41.440 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 32482 2024-03-30T14:01:41.440 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.440 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:41.440 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.440 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:41.440 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.022333+0000", 2024-03-30T14:01:41.440 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.440 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.440 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.440 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:41.440 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.022333+0000", 2024-03-30T14:01:41.440 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.440 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.441 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.441 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:41.441 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.022333+0000", 2024-03-30T14:01:41.441 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.0900000000000001e-07 2024-03-30T14:01:41.441 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.441 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.441 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:41.441 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.022397+0000", 2024-03-30T14:01:41.441 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3226999999999996e-05 2024-03-30T14:01:41.441 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.441 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.441 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:41.441 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.022405+0000", 2024-03-30T14:01:41.441 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.1629999999999996e-06 2024-03-30T14:01:41.441 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.442 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.442 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:41.442 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.022409+0000", 2024-03-30T14:01:41.442 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3100000000000002e-06 2024-03-30T14:01:41.442 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.442 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.442 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.442 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.022463+0000", 2024-03-30T14:01:41.442 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3798000000000002e-05 2024-03-30T14:01:41.442 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.442 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.442 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.442 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.022470+0000", 2024-03-30T14:01:41.442 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.7070000000000003e-06 2024-03-30T14:01:41.442 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.443 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.443 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.443 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.984914+0000", 2024-03-30T14:01:41.443 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.96244460300000001 2024-03-30T14:01:41.443 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.443 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.443 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:41.443 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.985003+0000", 2024-03-30T14:01:41.443 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.9239000000000004e-05 2024-03-30T14:01:41.443 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.443 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.443 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,2", 2024-03-30T14:01:41.443 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.985088+0000", 2024-03-30T14:01:41.443 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.4995999999999997e-05 2024-03-30T14:01:41.443 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.443 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.444 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:41.444 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.988219+0000", 2024-03-30T14:01:41.444 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.003130608 2024-03-30T14:01:41.444 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.444 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.444 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.444 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.989129+0000", 2024-03-30T14:01:41.444 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00090972700000000004 2024-03-30T14:01:41.444 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.444 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.444 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.444 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.996945+0000", 2024-03-30T14:01:41.444 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0078157209999999994 2024-03-30T14:01:41.444 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.444 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.445 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:41.445 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.996974+0000", 2024-03-30T14:01:41.445 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.9329e-05 2024-03-30T14:01:41.445 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.445 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.445 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:41.445 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.997005+0000", 2024-03-30T14:01:41.445 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.0756999999999997e-05 2024-03-30T14:01:41.445 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.445 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.445 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.445 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.445 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.445 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:32484 11.4 11:2585f3dd:::benchmark_data_smithi043_197439_object32483:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-03-30T14:01:41.445 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:01:40.024566+0000", 2024-03-30T14:01:41.445 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.320417985, 2024-03-30T14:01:41.446 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.97257871799999995, 2024-03-30T14:01:41.446 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:41.446 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:41.446 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:41.446 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:41.446 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:01:41.446 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:01:41.446 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 32484 2024-03-30T14:01:41.446 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.446 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:41.446 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.446 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:41.447 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.024566+0000", 2024-03-30T14:01:41.447 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.447 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.447 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.447 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:41.447 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.024566+0000", 2024-03-30T14:01:41.447 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.447 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.447 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.447 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:41.447 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.024567+0000", 2024-03-30T14:01:41.447 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.7000000000000004e-07 2024-03-30T14:01:41.447 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.447 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.447 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:41.448 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.024610+0000", 2024-03-30T14:01:41.448 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3269999999999997e-05 2024-03-30T14:01:41.448 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.448 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.448 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:41.448 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.024621+0000", 2024-03-30T14:01:41.448 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0844e-05 2024-03-30T14:01:41.448 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.448 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.448 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:41.448 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.024627+0000", 2024-03-30T14:01:41.448 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.37e-06 2024-03-30T14:01:41.448 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.448 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.448 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.449 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.024683+0000", 2024-03-30T14:01:41.449 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5442999999999998e-05 2024-03-30T14:01:41.449 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.449 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.449 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.449 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.024689+0000", 2024-03-30T14:01:41.449 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5590000000000002e-06 2024-03-30T14:01:41.449 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.449 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.449 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.449 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.985783+0000", 2024-03-30T14:01:41.449 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.96109367499999998 2024-03-30T14:01:41.449 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.449 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.449 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:41.449 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.985913+0000", 2024-03-30T14:01:41.450 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000130013 2024-03-30T14:01:41.450 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.450 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.450 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,2", 2024-03-30T14:01:41.450 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.986043+0000", 2024-03-30T14:01:41.450 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012965900000000001 2024-03-30T14:01:41.450 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.450 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.450 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:41.450 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.989012+0000", 2024-03-30T14:01:41.450 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.002969087 2024-03-30T14:01:41.450 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.450 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.450 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.451 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.990064+0000", 2024-03-30T14:01:41.451 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0010525269999999999 2024-03-30T14:01:41.451 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.451 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.451 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.451 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.997048+0000", 2024-03-30T14:01:41.451 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0069834670000000001 2024-03-30T14:01:41.451 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.451 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.451 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:41.451 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.997102+0000", 2024-03-30T14:01:41.451 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4394999999999997e-05 2024-03-30T14:01:41.451 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.451 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.451 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:41.452 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.997145+0000", 2024-03-30T14:01:41.452 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.2639000000000003e-05 2024-03-30T14:01:41.452 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.452 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.452 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.452 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.452 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.452 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:32492 11.4 11:23e4c540:::benchmark_data_smithi043_197439_object32491:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-03-30T14:01:41.452 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:01:40.032574+0000", 2024-03-30T14:01:41.452 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.3124097379999999, 2024-03-30T14:01:41.452 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.96466541400000005, 2024-03-30T14:01:41.452 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:41.452 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:41.452 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:41.452 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:41.453 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:01:41.453 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:01:41.453 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 32492 2024-03-30T14:01:41.453 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.453 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:41.453 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.453 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:41.453 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.032574+0000", 2024-03-30T14:01:41.453 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.453 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.453 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.453 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:41.453 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.032574+0000", 2024-03-30T14:01:41.453 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.453 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.453 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.454 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:41.454 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.032575+0000", 2024-03-30T14:01:41.454 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0950000000000001e-06 2024-03-30T14:01:41.454 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.454 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.454 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:41.454 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.032598+0000", 2024-03-30T14:01:41.454 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.2688999999999999e-05 2024-03-30T14:01:41.454 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.454 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.454 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:41.454 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.032609+0000", 2024-03-30T14:01:41.454 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0481e-05 2024-03-30T14:01:41.454 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.455 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.455 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:41.455 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.032616+0000", 2024-03-30T14:01:41.455 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.306e-06 2024-03-30T14:01:41.455 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.455 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.455 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.455 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.032684+0000", 2024-03-30T14:01:41.455 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8015000000000005e-05 2024-03-30T14:01:41.455 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.455 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.455 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.455 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.032690+0000", 2024-03-30T14:01:41.455 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5420000000000002e-06 2024-03-30T14:01:41.455 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.456 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.456 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.456 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.986810+0000", 2024-03-30T14:01:41.456 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.95411911699999996 2024-03-30T14:01:41.456 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.456 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.456 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:41.456 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.987140+0000", 2024-03-30T14:01:41.456 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00033019800000000003 2024-03-30T14:01:41.456 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.456 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.456 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,2", 2024-03-30T14:01:41.456 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.987296+0000", 2024-03-30T14:01:41.456 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000155823 2024-03-30T14:01:41.457 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.457 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.457 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:41.457 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.989076+0000", 2024-03-30T14:01:41.457 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0017808380000000001 2024-03-30T14:01:41.457 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.457 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.457 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.457 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.992399+0000", 2024-03-30T14:01:41.457 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0033226420000000002 2024-03-30T14:01:41.457 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.457 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.457 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.457 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.997173+0000", 2024-03-30T14:01:41.457 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0047740580000000003 2024-03-30T14:01:41.457 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.458 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.458 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:41.458 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.997201+0000", 2024-03-30T14:01:41.458 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.7515999999999999e-05 2024-03-30T14:01:41.458 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.458 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.458 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:41.458 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.997240+0000", 2024-03-30T14:01:41.458 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.9094000000000002e-05 2024-03-30T14:01:41.458 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.458 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.458 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.458 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.458 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.459 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:32494 11.4 11:22f494b1:::benchmark_data_smithi043_197439_object32493:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-03-30T14:01:41.459 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:01:40.037871+0000", 2024-03-30T14:01:41.459 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.307113065, 2024-03-30T14:01:41.459 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.961996823, 2024-03-30T14:01:41.459 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:41.459 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:41.459 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:41.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:41.460 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:41.460 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:41.460 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:01:41.460 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:01:41.460 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 32494 2024-03-30T14:01:41.460 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.460 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:41.460 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.460 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:41.460 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.037871+0000", 2024-03-30T14:01:41.460 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.460 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.460 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.460 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:41.460 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.037871+0000", 2024-03-30T14:01:41.461 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.461 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.461 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.461 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:41.461 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.037873+0000", 2024-03-30T14:01:41.461 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.4099999999999998e-06 2024-03-30T14:01:41.461 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.461 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.461 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:41.461 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.037958+0000", 2024-03-30T14:01:41.461 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.4775000000000001e-05 2024-03-30T14:01:41.461 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.461 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.461 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:41.461 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.037983+0000", 2024-03-30T14:01:41.462 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.463e-05 2024-03-30T14:01:41.462 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.462 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.462 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:41.462 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.038002+0000", 2024-03-30T14:01:41.462 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.8995999999999999e-05 2024-03-30T14:01:41.462 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.462 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.462 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.462 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.038126+0000", 2024-03-30T14:01:41.462 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012408800000000001 2024-03-30T14:01:41.462 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.462 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.463 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.463 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.038145+0000", 2024-03-30T14:01:41.463 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.8859e-05 2024-03-30T14:01:41.463 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.463 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.463 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.463 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.988251+0000", 2024-03-30T14:01:41.463 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.95010622600000005 2024-03-30T14:01:41.463 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.463 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.463 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:41.463 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.988338+0000", 2024-03-30T14:01:41.463 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6586000000000001e-05 2024-03-30T14:01:41.463 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.463 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.463 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,2", 2024-03-30T14:01:41.464 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.988420+0000", 2024-03-30T14:01:41.464 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.2817000000000003e-05 2024-03-30T14:01:41.464 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.464 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.464 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:41.464 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.989139+0000", 2024-03-30T14:01:41.464 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000718723 2024-03-30T14:01:41.464 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.464 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.464 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.464 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.992436+0000", 2024-03-30T14:01:41.464 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0032967299999999999 2024-03-30T14:01:41.464 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.464 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.464 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.465 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.999768+0000", 2024-03-30T14:01:41.465 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0073323240000000003 2024-03-30T14:01:41.465 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.465 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.465 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:41.465 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.999825+0000", 2024-03-30T14:01:41.465 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6867999999999998e-05 2024-03-30T14:01:41.465 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.465 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.465 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:41.465 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.999868+0000", 2024-03-30T14:01:41.465 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.2790999999999997e-05 2024-03-30T14:01:41.465 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.465 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.466 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.466 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.466 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.466 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:32518 11.4 11:215b9086:::benchmark_data_smithi043_197439_object32517:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-03-30T14:01:41.466 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:01:40.080207+0000", 2024-03-30T14:01:41.466 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 1.2647769440000001, 2024-03-30T14:01:41.466 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.93162851400000002, 2024-03-30T14:01:41.466 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:01:41.466 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:01:41.466 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:01:41.466 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:01:41.466 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:01:41.466 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:01:41.467 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 32518 2024-03-30T14:01:41.467 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.467 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:01:41.467 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.467 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:01:41.467 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.080207+0000", 2024-03-30T14:01:41.467 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.467 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.467 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.467 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:01:41.467 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.080207+0000", 2024-03-30T14:01:41.467 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:01:41.467 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.467 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.467 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:01:41.468 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.080208+0000", 2024-03-30T14:01:41.468 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.78e-07 2024-03-30T14:01:41.468 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.468 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.468 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:01:41.468 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.080256+0000", 2024-03-30T14:01:41.468 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7438999999999998e-05 2024-03-30T14:01:41.468 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.468 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.468 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:01:41.468 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.080267+0000", 2024-03-30T14:01:41.468 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1901e-05 2024-03-30T14:01:41.468 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.468 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.468 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:01:41.468 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.080274+0000", 2024-03-30T14:01:41.469 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1199999999999999e-06 2024-03-30T14:01:41.469 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.469 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.469 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.469 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.080335+0000", 2024-03-30T14:01:41.469 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1853999999999994e-05 2024-03-30T14:01:41.469 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.469 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.469 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:01:41.469 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.080348+0000", 2024-03-30T14:01:41.469 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2123999999999999e-05 2024-03-30T14:01:41.469 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.469 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.469 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:01:41.469 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.989153+0000", 2024-03-30T14:01:41.469 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.90880508199999999 2024-03-30T14:01:41.470 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.470 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.470 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:01:41.470 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.989237+0000", 2024-03-30T14:01:41.470 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.4122999999999994e-05 2024-03-30T14:01:41.470 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.470 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.470 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,2", 2024-03-30T14:01:41.470 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.989317+0000", 2024-03-30T14:01:41.470 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.0487000000000004e-05 2024-03-30T14:01:41.470 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.470 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.470 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:01:41.470 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.989948+0000", 2024-03-30T14:01:41.471 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00063051399999999999 2024-03-30T14:01:41.471 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.471 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.471 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.471 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:40.992558+0000", 2024-03-30T14:01:41.471 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.002610724 2024-03-30T14:01:41.471 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.471 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.471 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:01:41.471 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:41.011768+0000", 2024-03-30T14:01:41.471 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.019209028999999999 2024-03-30T14:01:41.471 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.471 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.471 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:01:41.471 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:41.011803+0000", 2024-03-30T14:01:41.471 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.5281999999999999e-05 2024-03-30T14:01:41.471 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:01:41.472 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:01:41.472 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:01:41.472 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:01:41.011836+0000", 2024-03-30T14:01:41.472 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2857000000000001e-05 2024-03-30T14:01:41.472 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.472 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.472 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.472 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:01:41.472 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:01:41.472 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T14:01:41.560 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:41.601 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:41.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:41.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:41.703 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:41.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:41.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:41.853 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:41.882 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:41.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:41.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:41.983 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:42.012 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:42.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:42.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:42.056 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 22 16 34053 34037 96.681 90.125 0.0117457 0.0103371 2024-03-30T14:01:42.114 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:42.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:41 smithi043 ceph-mon[29963]: osdmap e396: 8 total, 8 up, 7 in 2024-03-30T14:01:42.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:41 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3987875144' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T14:01:42.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:41 smithi043 ceph-mon[42099]: osdmap e396: 8 total, 8 up, 7 in 2024-03-30T14:01:42.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:41 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3987875144' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T14:01:42.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:42.143 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:42.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:42.244 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:42.285 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:42.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:42.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:42.387 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:42.414 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:42.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:42.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:42.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:41 smithi113 ceph-mon[34341]: osdmap e396: 8 total, 8 up, 7 in 2024-03-30T14:01:42.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:41 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3987875144' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T14:01:42.515 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:42.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:42.559 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:42.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:42.660 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:42.690 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:42.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:42.791 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:42.840 INFO:tasks.thrashosds.thrasher:j is {'epoch': 396, 'fsid': '7ee932fa-ee9a-11ee-b647-cb9ed24678a4', 'created': '2024-03-30T13:37:07.346020+0000', 'modified': '2024-03-30T14:01:40.963485+0000', 'last_up_change': '2024-03-30T14:01:27.027194+0000', 'last_in_change': '2024-03-30T13:58:46.155265+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T13:40:27.658979+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '371', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '368', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7100000381469727, 'score_stable': 1.7100000381469727, 'optimal_score': 1.1699999570846558, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-30T13:42:42.989975+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-30T13:42:43.910306+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8700000047683716, 'score_stable': 1.8700000047683716, 'optimal_score': 1.1699999570846558, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-30T13:42:45.633132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6799999475479126, 'score_stable': 1.6799999475479126, 'optimal_score': 1.1699999570846558, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-30T13:42:47.570084+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6799999475479126, 'score_stable': 1.6799999475479126, 'optimal_score': 1.1699999570846558, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-30T13:42:49.637189+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1.1699999570846558, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-30T14:01:12.894591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '384', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8700000047683716, 'score_stable': 1.8700000047683716, 'optimal_score': 1.1699999570846558, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}], 'osds': [{'osd': 0, 'uuid': '94ea083e-6a27-41db-83af-2f7d3cae1956', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 377, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6802', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6803', 'nonce': 1144592572}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6804', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6805', 'nonce': 1144592572}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6808', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6809', 'nonce': 1144592572}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6806', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6807', 'nonce': 1144592572}]}, 'public_addr': '172.21.15.43:6803/1144592572', 'cluster_addr': '172.21.15.43:6805/1144592572', 'heartbeat_back_addr': '172.21.15.43:6809/1144592572', 'heartbeat_front_addr': '172.21.15.43:6807/1144592572', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 395, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6810', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6811', 'nonce': 3901165928}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6812', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6813', 'nonce': 3901165928}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6816', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6817', 'nonce': 3901165928}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6814', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6815', 'nonce': 3901165928}]}, 'public_addr': '172.21.15.43:6811/3901165928', 'cluster_addr': '172.21.15.43:6813/3901165928', 'heartbeat_back_addr': '172.21.15.43:6817/3901165928', 'heartbeat_front_addr': '172.21.15.43:6815/3901165928', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6db6782e-5889-443e-bc6c-df47a1f93bc8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 18, 'last_clean_end': 166, 'up_from': 388, 'up_thru': 98, 'down_at': 167, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6818', 'nonce': 886354154}, {'type': 'v1', 'addr': '172.21.15.43:6819', 'nonce': 886354154}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6820', 'nonce': 886354154}, {'type': 'v1', 'addr': '172.21.15.43:6821', 'nonce': 886354154}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6824', 'nonce': 886354154}, {'type': 'v1', 'addr': '172.21.15.43:6825', 'nonce': 886354154}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6822', 'nonce': 886354154}, {'type': 'v1', 'addr': '172.21.15.43:6823', 'nonce': 886354154}]}, 'public_addr': '172.21.15.43:6819/886354154', 'cluster_addr': '172.21.15.43:6821/886354154', 'heartbeat_back_addr': '172.21.15.43:6825/886354154', 'heartbeat_front_addr': '172.21.15.43:6823/886354154', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '7048ff46-2cba-4d18-9835-95ecbddb2bb6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 391, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6826', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6827', 'nonce': 2933589376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6828', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6829', 'nonce': 2933589376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6832', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6833', 'nonce': 2933589376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6830', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6831', 'nonce': 2933589376}]}, 'public_addr': '172.21.15.43:6827/2933589376', 'cluster_addr': '172.21.15.43:6829/2933589376', 'heartbeat_back_addr': '172.21.15.43:6833/2933589376', 'heartbeat_front_addr': '172.21.15.43:6831/2933589376', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '42da30e6-9eec-4ed5-8c73-9ce78aa8ff63', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 394, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6800', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6801', 'nonce': 1945833081}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6802', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6803', 'nonce': 1945833081}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6806', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6807', 'nonce': 1945833081}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6804', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6805', 'nonce': 1945833081}]}, 'public_addr': '172.21.15.113:6801/1945833081', 'cluster_addr': '172.21.15.113:6803/1945833081', 'heartbeat_back_addr': '172.21.15.113:6807/1945833081', 'heartbeat_front_addr': '172.21.15.113:6805/1945833081', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '99a11b81-0df7-4b1d-946a-4297ba62e263', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 391, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6808', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6809', 'nonce': 3426360325}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6810', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6811', 'nonce': 3426360325}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6814', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6815', 'nonce': 3426360325}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6812', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6813', 'nonce': 3426360325}]}, 'public_addr': '172.21.15.113:6809/3426360325', 'cluster_addr': '172.21.15.113:6811/3426360325', 'heartbeat_back_addr': '172.21.15.113:6815/3426360325', 'heartbeat_front_addr': '172.21.15.113:6813/3426360325', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'cafd0e26-78e9-4164-a43a-96ca8382fefe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 395, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6816', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6817', 'nonce': 1527017698}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6818', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6819', 'nonce': 1527017698}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6822', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6823', 'nonce': 1527017698}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6820', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6821', 'nonce': 1527017698}]}, 'public_addr': '172.21.15.113:6817/1527017698', 'cluster_addr': '172.21.15.113:6819/1527017698', 'heartbeat_back_addr': '172.21.15.113:6823/1527017698', 'heartbeat_front_addr': '172.21.15.113:6821/1527017698', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e9d59662-9132-48f4-9e95-842a3cfbd79f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 228, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6824', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6825', 'nonce': 3182476015}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6826', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6827', 'nonce': 3182476015}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6830', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6831', 'nonce': 3182476015}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6828', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6829', 'nonce': 3182476015}]}, 'public_addr': '172.21.15.113:6825/3182476015', 'cluster_addr': '172.21.15.113:6827/3182476015', 'heartbeat_back_addr': '172.21.15.113:6831/3182476015', 'heartbeat_front_addr': '172.21.15.113:6829/3182476015', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T14:01:30.245122+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T14:01:13.621213+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T14:01:26.644187+0000', 'dead_epoch': 166}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T14:01:22.343676+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T14:01:26.628194+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T14:00:23.753024+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T14:01:37.799242+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-30T14:01:23.807351+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '11.2', 'osds': [3, 0]}, {'pgid': '11.6', 'osds': [1, 0]}, {'pgid': '11.f', 'osds': [1, 0]}], 'primary_temp': [], 'blocklist': {'172.21.15.43:0/538304234': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:6801/3411723337': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/3656823679': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/2618817647': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:6800/3411723337': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/1206218784': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/886631486': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:6800/1339921781': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:6801/1339921781': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:0/2794494528': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:0/564868778': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:6800/3033707884': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/3539398453': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/2414364740': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:6801/3033707884': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/3923597429': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/2524269526': '2024-03-31T13:43:27.253365+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-30T14:01:42.840 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T14:01:42.852 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:42.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:42.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:42.954 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:42.984 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:42.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:42.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:43.057 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 23 16 35432 35416 96.2245 86.1875 0.0160827 0.0103855 2024-03-30T14:01:43.086 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:43.097 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-30T14:01:43.117 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:43.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:43.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:42 smithi043 ceph-mon[29963]: pgmap v865: 168 pgs: 163 active+clean, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped; 2.0 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 2992/96645 objects degraded (3.096%); 37 MiB/s, 591 objects/s recovering 2024-03-30T14:01:43.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:42 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 2992/96645 objects degraded (3.096%), 4 pgs degraded (PG_DEGRADED) 2024-03-30T14:01:43.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:42 smithi043 ceph-mon[42099]: pgmap v865: 168 pgs: 163 active+clean, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped; 2.0 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 2992/96645 objects degraded (3.096%); 37 MiB/s, 591 objects/s recovering 2024-03-30T14:01:43.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:42 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 2992/96645 objects degraded (3.096%), 4 pgs degraded (PG_DEGRADED) 2024-03-30T14:01:43.220 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:43.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:43.272 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:43.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:43.373 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:43.404 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:43.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:43.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:43.424 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:01:43.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:42 smithi113 ceph-mon[34341]: pgmap v865: 168 pgs: 163 active+clean, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped; 2.0 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 2992/96645 objects degraded (3.096%); 37 MiB/s, 591 objects/s recovering 2024-03-30T14:01:43.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:42 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 2992/96645 objects degraded (3.096%), 4 pgs degraded (PG_DEGRADED) 2024-03-30T14:01:43.495 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:01:43.506 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:43.534 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:43.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:43.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:43.553 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T14:01:43.636 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:43.705 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:43.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:43.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:43.807 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:43.855 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:43.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:43.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:43.959 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:43.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:43.991 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:43.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:44.057 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 24 16 36721 36705 95.5679 80.5625 0.013402 0.0104562 2024-03-30T14:01:44.092 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:44.140 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:44.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:44.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:44.242 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:44.270 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:44.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:44.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:44.298 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T14:01:44.367 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T14:01:44.372 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:44.400 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:44.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:44.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:44.411 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T14:01:44.465 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T14:01:44.503 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:44.512 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T14:01:44.546 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:44.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:44.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:44.556 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T14:01:44.600 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T14:01:44.642 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T14:01:44.649 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:44.649 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:01:44.650 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:01:44.650 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:01:44.650 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:01:44.650 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:01:44.650 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:01:44.650 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:01:44.650 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:01:44.679 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:44.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:44.781 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:44.819 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:44.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:44.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:44.921 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:44.963 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:44.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:44.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:45.058 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 25 16 38019 38003 94.9899 81.125 0.00881301 0.0105219 2024-03-30T14:01:45.066 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:45.090 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:44 smithi113 ceph-mon[34341]: pgmap v866: 168 pgs: 163 active+clean, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped; 2.0 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 881 B/s rd, 75 MiB/s wr, 1.20k op/s; 2262/100860 objects degraded (2.243%); 33 MiB/s, 533 objects/s recovering 2024-03-30T14:01:45.090 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:44 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3827486045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:01:45.090 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:44 smithi113 ceph-mon[34341]: 4.5 scrub starts 2024-03-30T14:01:45.090 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:44 smithi113 ceph-mon[34341]: 4.5 scrub ok 2024-03-30T14:01:45.090 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:44 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:45.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:45.098 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:45.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:45.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:44 smithi043 ceph-mon[29963]: pgmap v866: 168 pgs: 163 active+clean, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped; 2.0 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 881 B/s rd, 75 MiB/s wr, 1.20k op/s; 2262/100860 objects degraded (2.243%); 33 MiB/s, 533 objects/s recovering 2024-03-30T14:01:45.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:44 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3827486045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:01:45.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:44 smithi043 ceph-mon[29963]: 4.5 scrub starts 2024-03-30T14:01:45.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:44 smithi043 ceph-mon[29963]: 4.5 scrub ok 2024-03-30T14:01:45.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:44 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:45.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:44 smithi043 ceph-mon[42099]: pgmap v866: 168 pgs: 163 active+clean, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped; 2.0 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 881 B/s rd, 75 MiB/s wr, 1.20k op/s; 2262/100860 objects degraded (2.243%); 33 MiB/s, 533 objects/s recovering 2024-03-30T14:01:45.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:44 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3827486045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:01:45.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:44 smithi043 ceph-mon[42099]: 4.5 scrub starts 2024-03-30T14:01:45.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:44 smithi043 ceph-mon[42099]: 4.5 scrub ok 2024-03-30T14:01:45.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:44 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:45.199 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:45.228 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:45.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:45.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:45.329 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:45.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:45.369 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:45.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:45.423 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T14:01:45.423 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:01:45.471 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:45.512 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:45.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:45.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:45.614 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:45.644 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:45.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:45.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:45.747 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:45.778 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:45.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:45.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:45.880 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:45.908 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:45.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:45.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:46.009 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:46.050 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:46.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:46.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:46.057 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 26 16 39264 39248 94.3289 77.8125 0.0261755 0.0105959 2024-03-30T14:01:46.152 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:46.161 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:45 smithi113 ceph-mon[34341]: 5.1f scrub starts 2024-03-30T14:01:46.162 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:45 smithi113 ceph-mon[34341]: 5.1f scrub ok 2024-03-30T14:01:46.181 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:46.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:46.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:46.192 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-03-30T14:01:46.283 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:46.312 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:46.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:46.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:46.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:45 smithi043 ceph-mon[29963]: 5.1f scrub starts 2024-03-30T14:01:46.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:45 smithi043 ceph-mon[29963]: 5.1f scrub ok 2024-03-30T14:01:46.392 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:01:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:01:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:01:46.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:45 smithi043 ceph-mon[42099]: 5.1f scrub starts 2024-03-30T14:01:46.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:45 smithi043 ceph-mon[42099]: 5.1f scrub ok 2024-03-30T14:01:46.414 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:46.447 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:46.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:46.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:46.549 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:01:46.549 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:46.584 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:46.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:46.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:46.686 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:46.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:46.727 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:46.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:46.828 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:46.861 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:46.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:46.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:46.963 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:47.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:47.006 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:47.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:47.058 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 27 15 40409 40394 93.4879 71.625 0.0163782 0.0106916 2024-03-30T14:01:47.108 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:47.139 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:47.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:47.241 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:47.271 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:47.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:47.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:47.373 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:47.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:47 smithi043 ceph-mon[29963]: pgmap v867: 168 pgs: 164 active+clean, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 753 B/s rd, 50 MiB/s wr, 808 op/s; 2052/106701 objects degraded (1.923%); 24 MiB/s, 389 objects/s recovering 2024-03-30T14:01:47.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:47 smithi043 ceph-mon[29963]: 5.16 scrub starts 2024-03-30T14:01:47.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:47 smithi043 ceph-mon[29963]: 5.16 scrub ok 2024-03-30T14:01:47.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:47 smithi043 ceph-mon[42099]: pgmap v867: 168 pgs: 164 active+clean, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 753 B/s rd, 50 MiB/s wr, 808 op/s; 2052/106701 objects degraded (1.923%); 24 MiB/s, 389 objects/s recovering 2024-03-30T14:01:47.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:47 smithi043 ceph-mon[42099]: 5.16 scrub starts 2024-03-30T14:01:47.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:47 smithi043 ceph-mon[42099]: 5.16 scrub ok 2024-03-30T14:01:47.399 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:47.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:47.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:47.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:47 smithi113 ceph-mon[34341]: pgmap v867: 168 pgs: 164 active+clean, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 753 B/s rd, 50 MiB/s wr, 808 op/s; 2052/106701 objects degraded (1.923%); 24 MiB/s, 389 objects/s recovering 2024-03-30T14:01:47.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:47 smithi113 ceph-mon[34341]: 5.16 scrub starts 2024-03-30T14:01:47.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:47 smithi113 ceph-mon[34341]: 5.16 scrub ok 2024-03-30T14:01:47.502 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:47.533 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:47.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:47.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:47.635 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:47.679 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:47.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:47.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:47.781 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:47.810 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:47.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:47.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:47.911 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:47.947 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:47.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:47.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:48.049 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:48.058 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 28 16 41780 41764 93.2068 85.625 0.00321046 0.0106965 2024-03-30T14:01:48.073 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:48.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:48.175 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:48.203 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:48.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:48.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:48.305 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:48.323 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:48 smithi113 ceph-mon[34341]: Health check update: Degraded data redundancy: 2052/106701 objects degraded (1.923%), 3 pgs degraded (PG_DEGRADED) 2024-03-30T14:01:48.323 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:48 smithi113 ceph-mon[34341]: osdmap e397: 8 total, 8 up, 7 in 2024-03-30T14:01:48.330 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:48.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:48.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:48.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:48 smithi043 ceph-mon[29963]: Health check update: Degraded data redundancy: 2052/106701 objects degraded (1.923%), 3 pgs degraded (PG_DEGRADED) 2024-03-30T14:01:48.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:48 smithi043 ceph-mon[29963]: osdmap e397: 8 total, 8 up, 7 in 2024-03-30T14:01:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:48 smithi043 ceph-mon[42099]: Health check update: Degraded data redundancy: 2052/106701 objects degraded (1.923%), 3 pgs degraded (PG_DEGRADED) 2024-03-30T14:01:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:48 smithi043 ceph-mon[42099]: osdmap e397: 8 total, 8 up, 7 in 2024-03-30T14:01:48.432 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:48.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:48.459 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:48.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:48.560 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:48.587 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:48.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:48.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:48.689 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:48.716 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:48.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:48.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:48.767 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T14:01:48.767 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:01:48.818 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:48.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:48.843 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:48.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:48.944 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:48.969 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:48.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:48.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:49.058 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 29 16 41934 41918 90.3249 9.625 0.00341882 0.0110532 2024-03-30T14:01:49.070 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:49.095 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:49.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:49.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:49.197 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:49.221 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:49.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:49.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:49.322 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:49.347 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:49.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:49.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:49.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:49 smithi043 ceph-mon[29963]: pgmap v868: 168 pgs: 164 active+clean, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 2.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 753 op/s; 1397/111762 objects degraded (1.250%); 22 MiB/s, 354 objects/s recovering 2024-03-30T14:01:49.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:49 smithi043 ceph-mon[29963]: 5.11 scrub starts 2024-03-30T14:01:49.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:49 smithi043 ceph-mon[29963]: 5.11 scrub ok 2024-03-30T14:01:49.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:49 smithi043 ceph-mon[29963]: 3.c scrub starts 2024-03-30T14:01:49.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:49 smithi043 ceph-mon[29963]: 3.c scrub ok 2024-03-30T14:01:49.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:49 smithi043 ceph-mon[29963]: osdmap e398: 8 total, 8 up, 7 in 2024-03-30T14:01:49.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:49 smithi043 ceph-mon[42099]: pgmap v868: 168 pgs: 164 active+clean, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 2.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 753 op/s; 1397/111762 objects degraded (1.250%); 22 MiB/s, 354 objects/s recovering 2024-03-30T14:01:49.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:49 smithi043 ceph-mon[42099]: 5.11 scrub starts 2024-03-30T14:01:49.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:49 smithi043 ceph-mon[42099]: 5.11 scrub ok 2024-03-30T14:01:49.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:49 smithi043 ceph-mon[42099]: 3.c scrub starts 2024-03-30T14:01:49.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:49 smithi043 ceph-mon[42099]: 3.c scrub ok 2024-03-30T14:01:49.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:49 smithi043 ceph-mon[42099]: osdmap e398: 8 total, 8 up, 7 in 2024-03-30T14:01:49.413 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:49 smithi113 ceph-mon[34341]: pgmap v868: 168 pgs: 164 active+clean, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 2.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 753 op/s; 1397/111762 objects degraded (1.250%); 22 MiB/s, 354 objects/s recovering 2024-03-30T14:01:49.414 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:49 smithi113 ceph-mon[34341]: 5.11 scrub starts 2024-03-30T14:01:49.414 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:49 smithi113 ceph-mon[34341]: 5.11 scrub ok 2024-03-30T14:01:49.414 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:49 smithi113 ceph-mon[34341]: 3.c scrub starts 2024-03-30T14:01:49.414 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:49 smithi113 ceph-mon[34341]: 3.c scrub ok 2024-03-30T14:01:49.414 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:49 smithi113 ceph-mon[34341]: osdmap e398: 8 total, 8 up, 7 in 2024-03-30T14:01:49.448 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:49.472 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:49.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:49.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:49.575 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:49.599 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:49.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:49.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:49.701 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:49.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:01:49 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:01:49.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:49.725 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:49.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:49.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:49.852 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:49.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:49.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:49.890 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-03-30T14:01:49.954 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:49.990 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:49.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:49.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:50.058 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 30 16 42132 42116 87.7268 12.375 0.00509698 0.0113845 2024-03-30T14:01:50.094 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:50.124 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:50.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:50.175 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:01:50.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:50 smithi113 ceph-mon[34341]: 6.2 scrub starts 2024-03-30T14:01:50.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:50 smithi113 ceph-mon[34341]: 6.2 scrub ok 2024-03-30T14:01:50.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:50 smithi113 ceph-mon[34341]: 4.15 scrub starts 2024-03-30T14:01:50.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:50 smithi113 ceph-mon[34341]: 4.15 scrub ok 2024-03-30T14:01:50.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:50 smithi113 ceph-mon[34341]: 4.4 scrub starts 2024-03-30T14:01:50.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:50 smithi113 ceph-mon[34341]: 4.4 scrub ok 2024-03-30T14:01:50.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:50 smithi113 ceph-mon[34341]: osdmap e399: 8 total, 8 up, 7 in 2024-03-30T14:01:50.226 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:50.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:50.250 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:50.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:50.352 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:50.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:50.377 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:50.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:50.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:50 smithi043 ceph-mon[29963]: 6.2 scrub starts 2024-03-30T14:01:50.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:50 smithi043 ceph-mon[29963]: 6.2 scrub ok 2024-03-30T14:01:50.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:50 smithi043 ceph-mon[29963]: 4.15 scrub starts 2024-03-30T14:01:50.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:50 smithi043 ceph-mon[29963]: 4.15 scrub ok 2024-03-30T14:01:50.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:50 smithi043 ceph-mon[29963]: 4.4 scrub starts 2024-03-30T14:01:50.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:50 smithi043 ceph-mon[29963]: 4.4 scrub ok 2024-03-30T14:01:50.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:50 smithi043 ceph-mon[29963]: osdmap e399: 8 total, 8 up, 7 in 2024-03-30T14:01:50.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:50 smithi043 ceph-mon[42099]: 6.2 scrub starts 2024-03-30T14:01:50.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:50 smithi043 ceph-mon[42099]: 6.2 scrub ok 2024-03-30T14:01:50.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:50 smithi043 ceph-mon[42099]: 4.15 scrub starts 2024-03-30T14:01:50.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:50 smithi043 ceph-mon[42099]: 4.15 scrub ok 2024-03-30T14:01:50.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:50 smithi043 ceph-mon[42099]: 4.4 scrub starts 2024-03-30T14:01:50.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:50 smithi043 ceph-mon[42099]: 4.4 scrub ok 2024-03-30T14:01:50.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:50 smithi043 ceph-mon[42099]: osdmap e399: 8 total, 8 up, 7 in 2024-03-30T14:01:50.478 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:50.501 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:50.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:50.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:50.603 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:50.628 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:50.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:50.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:50.730 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:50.754 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:50.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:50.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:50.856 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:50.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:50.880 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:50.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:50.981 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:51.004 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:51.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:51.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:51.059 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 31 16 42291 42275 85.2175 9.9375 0.00229203 0.0116957 2024-03-30T14:01:51.106 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:51.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:51.152 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:51.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:51.166 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:51 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:51.166 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:51 smithi113 ceph-mon[34341]: pgmap v871: 168 pgs: 1 peering, 165 active+clean, 2 active+recovering+undersized+degraded+remapped; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.15k op/s; 1043/124179 objects degraded (0.840%); 15 MiB/s, 243 objects/s recovering 2024-03-30T14:01:51.166 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:51 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:01:51.167 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:51 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:01:51.167 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:51 smithi113 ceph-mon[34341]: 6.a scrub starts 2024-03-30T14:01:51.167 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:51 smithi113 ceph-mon[34341]: 6.a scrub ok 2024-03-30T14:01:51.167 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:51 smithi113 ceph-mon[34341]: osdmap e400: 8 total, 8 up, 7 in 2024-03-30T14:01:51.253 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:51.297 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:51.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:51.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:51 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:51 smithi043 ceph-mon[29963]: pgmap v871: 168 pgs: 1 peering, 165 active+clean, 2 active+recovering+undersized+degraded+remapped; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.15k op/s; 1043/124179 objects degraded (0.840%); 15 MiB/s, 243 objects/s recovering 2024-03-30T14:01:51.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:51 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:01:51.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:51 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:01:51.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:51 smithi043 ceph-mon[29963]: 6.a scrub starts 2024-03-30T14:01:51.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:51 smithi043 ceph-mon[29963]: 6.a scrub ok 2024-03-30T14:01:51.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:51 smithi043 ceph-mon[29963]: osdmap e400: 8 total, 8 up, 7 in 2024-03-30T14:01:51.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:51 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:51.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:51 smithi043 ceph-mon[42099]: pgmap v871: 168 pgs: 1 peering, 165 active+clean, 2 active+recovering+undersized+degraded+remapped; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.15k op/s; 1043/124179 objects degraded (0.840%); 15 MiB/s, 243 objects/s recovering 2024-03-30T14:01:51.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:51 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:01:51.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:51 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:01:51.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:51 smithi043 ceph-mon[42099]: 6.a scrub starts 2024-03-30T14:01:51.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:51 smithi043 ceph-mon[42099]: 6.a scrub ok 2024-03-30T14:01:51.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:51 smithi043 ceph-mon[42099]: osdmap e400: 8 total, 8 up, 7 in 2024-03-30T14:01:51.400 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:51.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:51.441 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:51.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:51.542 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:51.582 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:51.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:51.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:51.684 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:51.722 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:51.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:51.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:51.824 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:51.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:51.861 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:51.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:51.962 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:52.004 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:52.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:52.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:52.058 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 32 16 43749 43733 85.4018 91.125 0.00912039 0.0117035 2024-03-30T14:01:52.107 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:52.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:52.132 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:52.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:52.213 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:52 smithi113 ceph-mon[34341]: 5.15 scrub starts 2024-03-30T14:01:52.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:52 smithi113 ceph-mon[34341]: 5.15 scrub ok 2024-03-30T14:01:52.233 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:52.262 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T14:01:52.262 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:01:52.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:52.273 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:52.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:52.375 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:52.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:52 smithi043 ceph-mon[29963]: 5.15 scrub starts 2024-03-30T14:01:52.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:52 smithi043 ceph-mon[29963]: 5.15 scrub ok 2024-03-30T14:01:52.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:52 smithi043 ceph-mon[42099]: 5.15 scrub starts 2024-03-30T14:01:52.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:52 smithi043 ceph-mon[42099]: 5.15 scrub ok 2024-03-30T14:01:52.414 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:52.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:52.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:52.516 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:52.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:52.554 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:52.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:52.655 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:52.678 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:52.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:52.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:52.780 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:52.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:52.818 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:52.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:52.920 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:52.943 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:52.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:52.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:52.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-30T14:01:53.044 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:53.058 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 33 16 45007 44991 85.1963 78.625 0.0115346 0.0117317 2024-03-30T14:01:53.077 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:53.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:53.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:53.179 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:53.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:53 smithi113 ceph-mon[34341]: pgmap v874: 168 pgs: 1 peering, 167 active+clean; 2.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 22 MiB/s, 349 objects/s recovering 2024-03-30T14:01:53.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:53 smithi113 ceph-mon[34341]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1043/124179 objects degraded (0.840%), 2 pgs degraded) 2024-03-30T14:01:53.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:53 smithi113 ceph-mon[34341]: 4.19 scrub starts 2024-03-30T14:01:53.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:53 smithi113 ceph-mon[34341]: 4.19 scrub ok 2024-03-30T14:01:53.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:53.223 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:53.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:53.324 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:53.369 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:53.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:53.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:53.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:53 smithi043 ceph-mon[29963]: pgmap v874: 168 pgs: 1 peering, 167 active+clean; 2.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 22 MiB/s, 349 objects/s recovering 2024-03-30T14:01:53.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:53 smithi043 ceph-mon[29963]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1043/124179 objects degraded (0.840%), 2 pgs degraded) 2024-03-30T14:01:53.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:53 smithi043 ceph-mon[29963]: 4.19 scrub starts 2024-03-30T14:01:53.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:53 smithi043 ceph-mon[29963]: 4.19 scrub ok 2024-03-30T14:01:53.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:53 smithi043 ceph-mon[42099]: pgmap v874: 168 pgs: 1 peering, 167 active+clean; 2.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 22 MiB/s, 349 objects/s recovering 2024-03-30T14:01:53.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:53 smithi043 ceph-mon[42099]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1043/124179 objects degraded (0.840%), 2 pgs degraded) 2024-03-30T14:01:53.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:53 smithi043 ceph-mon[42099]: 4.19 scrub starts 2024-03-30T14:01:53.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:53 smithi043 ceph-mon[42099]: 4.19 scrub ok 2024-03-30T14:01:53.471 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:53.501 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:53.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:53.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:53.603 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:53.643 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:53.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:53.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:53.745 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:53.805 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:53.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:53.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:53.907 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:53.938 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:53.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:54.040 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:54.058 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 34 16 46335 46319 85.1314 83 0.0075385 0.0117401 2024-03-30T14:01:54.083 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:54.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:54.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:54.185 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:54.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:54.218 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:54.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:54.319 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:54.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:54.351 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:54.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:54.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:54 smithi043 ceph-mon[29963]: 5.1b scrub starts 2024-03-30T14:01:54.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:54 smithi043 ceph-mon[29963]: 5.1b scrub ok 2024-03-30T14:01:54.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:54 smithi043 ceph-mon[29963]: 6.f scrub starts 2024-03-30T14:01:54.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:54 smithi043 ceph-mon[42099]: 5.1b scrub starts 2024-03-30T14:01:54.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:54 smithi043 ceph-mon[42099]: 5.1b scrub ok 2024-03-30T14:01:54.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:54 smithi043 ceph-mon[42099]: 6.f scrub starts 2024-03-30T14:01:54.453 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:54.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:54 smithi113 ceph-mon[34341]: 5.1b scrub starts 2024-03-30T14:01:54.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:54 smithi113 ceph-mon[34341]: 5.1b scrub ok 2024-03-30T14:01:54.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:54 smithi113 ceph-mon[34341]: 6.f scrub starts 2024-03-30T14:01:54.481 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:54.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:54.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:54.586 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:54.612 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:54.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:54.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:54.714 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:54.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:54.754 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:54.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:54.855 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:54.879 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:54.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:54.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:54.981 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:55.012 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:55.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:55.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:55.058 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 35 16 47695 47679 85.1274 85 0.00667141 0.0117241 2024-03-30T14:01:55.113 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:55.144 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:55.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:55.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:55.246 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:55.288 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:55.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:55.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:55.390 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:55 smithi043 ceph-mon[29963]: 6.f scrub ok 2024-03-30T14:01:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:55 smithi043 ceph-mon[29963]: pgmap v875: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 876 B/s rd, 66 MiB/s wr, 1.05k op/s; 15 MiB/s, 239 objects/s recovering 2024-03-30T14:01:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:55 smithi043 ceph-mon[29963]: 5.2 scrub starts 2024-03-30T14:01:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:55 smithi043 ceph-mon[29963]: 5.2 scrub ok 2024-03-30T14:01:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:55 smithi043 ceph-mon[29963]: 4.1c scrub starts 2024-03-30T14:01:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:55 smithi043 ceph-mon[29963]: 4.1c scrub ok 2024-03-30T14:01:55.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:55 smithi043 ceph-mon[42099]: 6.f scrub ok 2024-03-30T14:01:55.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:55 smithi043 ceph-mon[42099]: pgmap v875: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 876 B/s rd, 66 MiB/s wr, 1.05k op/s; 15 MiB/s, 239 objects/s recovering 2024-03-30T14:01:55.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:55 smithi043 ceph-mon[42099]: 5.2 scrub starts 2024-03-30T14:01:55.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:55 smithi043 ceph-mon[42099]: 5.2 scrub ok 2024-03-30T14:01:55.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:55 smithi043 ceph-mon[42099]: 4.1c scrub starts 2024-03-30T14:01:55.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:55 smithi043 ceph-mon[42099]: 4.1c scrub ok 2024-03-30T14:01:55.433 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:55.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:55.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:55.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:55 smithi113 ceph-mon[34341]: 6.f scrub ok 2024-03-30T14:01:55.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:55 smithi113 ceph-mon[34341]: pgmap v875: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 876 B/s rd, 66 MiB/s wr, 1.05k op/s; 15 MiB/s, 239 objects/s recovering 2024-03-30T14:01:55.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:55 smithi113 ceph-mon[34341]: 5.2 scrub starts 2024-03-30T14:01:55.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:55 smithi113 ceph-mon[34341]: 5.2 scrub ok 2024-03-30T14:01:55.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:55 smithi113 ceph-mon[34341]: 4.1c scrub starts 2024-03-30T14:01:55.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:55 smithi113 ceph-mon[34341]: 4.1c scrub ok 2024-03-30T14:01:55.535 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:55.563 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:55.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:55.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:55.665 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:55.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:55.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:55.807 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:55.845 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:55.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:55.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:55.933 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:01:55.947 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:56.005 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:01:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:56.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:56.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:56.060 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 36 16 49048 49032 85.1078 84.5625 0.00524132 0.0117436 2024-03-30T14:01:56.145 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:56.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:56.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:56.292 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:56.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:56.351 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:56.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:56.379 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:56 smithi043 ceph-mon[29963]: 5.1c scrub starts 2024-03-30T14:01:56.379 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:56 smithi043 ceph-mon[29963]: 5.1c scrub ok 2024-03-30T14:01:56.379 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:01:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:01:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:01:56.379 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:56 smithi043 ceph-mon[42099]: 5.1c scrub starts 2024-03-30T14:01:56.379 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:56 smithi043 ceph-mon[42099]: 5.1c scrub ok 2024-03-30T14:01:56.453 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:56.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:56 smithi113 ceph-mon[34341]: 5.1c scrub starts 2024-03-30T14:01:56.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:56 smithi113 ceph-mon[34341]: 5.1c scrub ok 2024-03-30T14:01:56.481 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:56.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:56.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:56.583 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:56.616 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:56.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:56.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:56.718 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:56.745 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:56.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:56.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:56.847 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:56.875 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:56.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:56.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:56.977 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:57.008 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:57.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:57.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:57.060 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 37 16 50537 50521 85.3226 93.0625 0.0123846 0.0117145 2024-03-30T14:01:57.110 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:57.155 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:57.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:57.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:57.257 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:57.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:57.302 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:57.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:57.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:57 smithi043 ceph-mon[29963]: pgmap v876: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 747 B/s rd, 45 MiB/s wr, 713 op/s; 9.5 MiB/s, 152 objects/s recovering 2024-03-30T14:01:57.344 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:57 smithi043 ceph-mon[42099]: pgmap v876: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 747 B/s rd, 45 MiB/s wr, 713 op/s; 9.5 MiB/s, 152 objects/s recovering 2024-03-30T14:01:57.403 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:57.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:57.450 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:57.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:57.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:57 smithi113 ceph-mon[34341]: pgmap v876: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 747 B/s rd, 45 MiB/s wr, 713 op/s; 9.5 MiB/s, 152 objects/s recovering 2024-03-30T14:01:57.551 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:57.602 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:57.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:57.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:57.704 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:01:57.733 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:57.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:57.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:01:57.776 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T14:01:57.835 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:57.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:57.891 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:57.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:57.923 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T14:01:57.932 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-30T14:01:57.992 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:58.060 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 38 16 51760 51744 85.0886 76.4375 0.0149706 0.0117465 2024-03-30T14:01:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:58.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:58.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:58.080 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:58 smithi043 ceph-mon[29963]: 3.1e scrub starts 2024-03-30T14:01:58.080 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:58 smithi043 ceph-mon[29963]: 3.1e scrub ok 2024-03-30T14:01:58.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:01:58.213 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:58.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:58.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:01:58.315 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:58.356 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:58.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:58.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:58.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:58 smithi043 ceph-mon[42099]: 3.1e scrub starts 2024-03-30T14:01:58.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:58 smithi043 ceph-mon[42099]: 3.1e scrub ok 2024-03-30T14:01:58.457 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:58.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:58 smithi113 ceph-mon[34341]: 3.1e scrub starts 2024-03-30T14:01:58.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:58 smithi113 ceph-mon[34341]: 3.1e scrub ok 2024-03-30T14:01:58.488 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:58.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:58.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:58.590 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:58.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:58.632 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:58.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:58.733 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:58.760 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:58.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:58.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:58.862 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:58.904 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:58.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:58.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:59.005 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:01:59.035 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:59.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:59.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:01:59.060 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 39 16 53133 53117 85.1069 85.8125 0.0060578 0.0117435 2024-03-30T14:01:59.137 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:01:59.164 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:59.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:59.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:01:59.266 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:01:59.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:59.307 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:59.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:01:59.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:59 smithi043 ceph-mon[29963]: pgmap v877: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 2.9 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 689 op/s; 8.1 MiB/s, 130 objects/s recovering 2024-03-30T14:01:59.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:59 smithi043 ceph-mon[29963]: from='client.15843 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:01:59.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:59 smithi043 ceph-mon[29963]: 2.1 scrub starts 2024-03-30T14:01:59.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:59 smithi043 ceph-mon[29963]: 2.1 scrub ok 2024-03-30T14:01:59.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:59 smithi043 ceph-mon[29963]: 3.0 scrub starts 2024-03-30T14:01:59.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:01:59 smithi043 ceph-mon[29963]: 3.0 scrub ok 2024-03-30T14:01:59.345 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:59 smithi043 ceph-mon[42099]: pgmap v877: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 2.9 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 689 op/s; 8.1 MiB/s, 130 objects/s recovering 2024-03-30T14:01:59.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:59 smithi043 ceph-mon[42099]: from='client.15843 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:01:59.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:59 smithi043 ceph-mon[42099]: 2.1 scrub starts 2024-03-30T14:01:59.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:59 smithi043 ceph-mon[42099]: 2.1 scrub ok 2024-03-30T14:01:59.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:59 smithi043 ceph-mon[42099]: 3.0 scrub starts 2024-03-30T14:01:59.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:01:59 smithi043 ceph-mon[42099]: 3.0 scrub ok 2024-03-30T14:01:59.408 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:59.424 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:59 smithi113 ceph-mon[34341]: pgmap v877: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 2.9 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 689 op/s; 8.1 MiB/s, 130 objects/s recovering 2024-03-30T14:01:59.425 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:59 smithi113 ceph-mon[34341]: from='client.15843 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:01:59.425 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:59 smithi113 ceph-mon[34341]: 2.1 scrub starts 2024-03-30T14:01:59.425 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:59 smithi113 ceph-mon[34341]: 2.1 scrub ok 2024-03-30T14:01:59.425 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:59 smithi113 ceph-mon[34341]: 3.0 scrub starts 2024-03-30T14:01:59.425 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:01:59 smithi113 ceph-mon[34341]: 3.0 scrub ok 2024-03-30T14:01:59.449 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:59.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:59.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:59.551 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:01:59.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:59.607 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:59.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:01:59.708 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:01:59.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:01:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:01:59.741 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:01:59.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:59.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:01:59.843 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:01:59.881 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:01:59.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:01:59.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:01:59.983 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:00.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:00.023 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:00.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:00.060 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-03-30T14:02:00.067323+0000 min lat: 0.00121029 max lat: 2.12887 avg lat: 0.0117574 2024-03-30T14:02:00.061 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T14:02:00.061 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 40 16 54433 54417 85.0104 81.25 0.0116755 0.0117574 2024-03-30T14:02:00.124 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:00.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:00.174 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:00.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:00.275 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:00.303 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:00.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:00.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:00.405 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:00.407 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-03-30T14:02:00.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:00.450 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:00.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:00.551 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:00.581 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:00.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:00.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:00 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:02:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:00 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:00 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:00 smithi043 ceph-mon[29963]: 5.8 scrub starts 2024-03-30T14:02:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:00 smithi043 ceph-mon[29963]: 5.8 scrub ok 2024-03-30T14:02:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:00 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/343446800' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:02:00.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:00 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:02:00.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:00 smithi043 ceph-mon[29963]: pgmap v878: 168 pgs: 168 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 67 MiB/s wr, 1.08k op/s; 6.6 MiB/s, 105 objects/s recovering 2024-03-30T14:02:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:00 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:02:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:00 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:00 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:00 smithi043 ceph-mon[42099]: 5.8 scrub starts 2024-03-30T14:02:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:00 smithi043 ceph-mon[42099]: 5.8 scrub ok 2024-03-30T14:02:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:00 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/343446800' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:02:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:00 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:02:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:00 smithi043 ceph-mon[42099]: pgmap v878: 168 pgs: 168 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 67 MiB/s wr, 1.08k op/s; 6.6 MiB/s, 105 objects/s recovering 2024-03-30T14:02:00.683 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:00.710 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:00.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:00.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:00.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:00 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:02:00.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:00 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:00.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:00 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:00.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:00 smithi113 ceph-mon[34341]: 5.8 scrub starts 2024-03-30T14:02:00.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:00 smithi113 ceph-mon[34341]: 5.8 scrub ok 2024-03-30T14:02:00.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:00 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/343446800' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:02:00.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:00 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:02:00.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:00 smithi113 ceph-mon[34341]: pgmap v878: 168 pgs: 168 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 67 MiB/s wr, 1.08k op/s; 6.6 MiB/s, 105 objects/s recovering 2024-03-30T14:02:00.812 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:00.852 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:00.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:00.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:00.954 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:00.996 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:00.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:00.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:01.060 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 41 16 55814 55798 85.0419 86.3125 0.00888952 0.0117531 2024-03-30T14:02:01.098 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:01.127 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:01.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:01.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:01.229 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:01.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:01.261 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:01.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:01.362 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:01.416 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:01.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:01.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:01.501 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":877,"stamp":"2024-03-30T14:01:57.760852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3061258590,"num_objects":46900,"num_object_clones":0,"num_object_copies":140700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46900,"num_whiteouts":0,"num_read":1634,"num_read_kb":1386,"num_write":47424,"num_write_kb":2988973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7277,"num_bytes_recovered":464196710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50244,"ondisk_log_size":50244,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":505,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13970688,"kb_used_data":9034120,"kb_used_omap":178,"kb_used_meta":4936333,"kb_avail":642245376,"statfs":{"total":671965249536,"available":657659265024,"internally_reserved":0,"allocated":9250938880,"data_stored":9237968775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":182415,"internal_metadata":5054805873},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":360906752,"num_objects":5507,"num_object_clones":0,"num_object_copies":16521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5507,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":5507,"num_write_kb":352448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1043,"num_bytes_recovered":68354048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004566"},"pg_stats":[{"pgid":"11.3","version":"400'2890","reported_seq":2939,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:53.946298+0000","last_change":"2024-03-30T14:01:13.913478+0000","last_active":"2024-03-30T14:01:53.946298+0000","last_peered":"2024-03-30T14:01:53.946298+0000","last_clean":"2024-03-30T14:01:53.946298+0000","last_became_active":"2024-03-30T14:01:13.913090+0000","last_became_peered":"2024-03-30T14:01:13.913090+0000","last_unstale":"2024-03-30T14:01:53.946298+0000","last_undegraded":"2024-03-30T14:01:53.946298+0000","last_fullsized":"2024-03-30T14:01:53.946298+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_clean_scrub_stamp":"2024-03-30T14:01:12.895052+0000","objects_scrubbed":0,"log_size":2890,"log_dups_size":0,"ondisk_log_size":2890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:40.150470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189399040,"num_objects":2890,"num_object_clones":0,"num_object_copies":8670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2890,"num_write_kb":184960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"216'20","reported_seq":762,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916814+0000","last_change":"2024-03-30T13:58:22.508345+0000","last_active":"2024-03-30T14:01:50.916814+0000","last_peered":"2024-03-30T14:01:50.916814+0000","last_clean":"2024-03-30T14:01:50.916814+0000","last_became_active":"2024-03-30T13:42:47.313026+0000","last_became_peered":"2024-03-30T13:42:47.313026+0000","last_unstale":"2024-03-30T14:01:50.916814+0000","last_undegraded":"2024-03-30T14:01:50.916814+0000","last_fullsized":"2024-03-30T14:01:50.916814+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'20","last_scrub_stamp":"2024-03-30T13:58:22.508219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:58:22.508219+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":736,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.917246+0000","last_change":"2024-03-30T14:01:15.319510+0000","last_active":"2024-03-30T14:01:50.917246+0000","last_peered":"2024-03-30T14:01:50.917246+0000","last_clean":"2024-03-30T14:01:50.917246+0000","last_became_active":"2024-03-30T13:42:45.309265+0000","last_became_peered":"2024-03-30T13:42:45.309265+0000","last_unstale":"2024-03-30T14:01:50.917246+0000","last_undegraded":"2024-03-30T14:01:50.917246+0000","last_fullsized":"2024-03-30T14:01:50.917246+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:15.319393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:01:15.319393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:17.687523+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"391'16","reported_seq":215,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.942424+0000","last_change":"2024-03-30T14:01:28.919551+0000","last_active":"2024-03-30T14:01:50.942424+0000","last_peered":"2024-03-30T14:01:50.942424+0000","last_clean":"2024-03-30T14:01:50.942424+0000","last_became_active":"2024-03-30T14:01:28.918955+0000","last_became_peered":"2024-03-30T14:01:28.918955+0000","last_unstale":"2024-03-30T14:01:50.942424+0000","last_undegraded":"2024-03-30T14:01:50.942424+0000","last_fullsized":"2024-03-30T14:01:50.942424+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'14","last_scrub_stamp":"2024-03-30T14:01:15.584080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:01:15.584080+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:23.417096+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":709,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.917517+0000","last_change":"2024-03-30T14:01:27.931894+0000","last_active":"2024-03-30T14:01:50.917517+0000","last_peered":"2024-03-30T14:01:50.917517+0000","last_clean":"2024-03-30T14:01:50.917517+0000","last_became_active":"2024-03-30T14:01:27.931422+0000","last_became_peered":"2024-03-30T14:01:27.931422+0000","last_unstale":"2024-03-30T14:01:50.917517+0000","last_undegraded":"2024-03-30T14:01:50.917517+0000","last_fullsized":"2024-03-30T14:01:50.917517+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:06.154805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:06.154805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":988,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.918462+0000","last_change":"2024-03-30T14:01:31.009054+0000","last_active":"2024-03-30T14:01:50.918462+0000","last_peered":"2024-03-30T14:01:50.918462+0000","last_clean":"2024-03-30T14:01:50.918462+0000","last_became_active":"2024-03-30T14:01:31.008696+0000","last_became_peered":"2024-03-30T14:01:31.008696+0000","last_unstale":"2024-03-30T14:01:50.918462+0000","last_undegraded":"2024-03-30T14:01:50.918462+0000","last_fullsized":"2024-03-30T14:01:50.918462+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:01:14.284304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:01:14.284304+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:26.597894+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"400'2826","reported_seq":5148,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:53.468742+0000","last_change":"2024-03-30T14:01:48.925269+0000","last_active":"2024-03-30T14:01:53.468742+0000","last_peered":"2024-03-30T14:01:53.468742+0000","last_clean":"2024-03-30T14:01:53.468742+0000","last_became_active":"2024-03-30T14:01:48.922508+0000","last_became_peered":"2024-03-30T14:01:48.922508+0000","last_unstale":"2024-03-30T14:01:53.468742+0000","last_undegraded":"2024-03-30T14:01:53.468742+0000","last_fullsized":"2024-03-30T14:01:53.468742+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_clean_scrub_stamp":"2024-03-30T14:01:12.895052+0000","objects_scrubbed":0,"log_size":2826,"log_dups_size":0,"ondisk_log_size":2826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:15.198580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185204736,"num_objects":2826,"num_object_clones":0,"num_object_copies":8478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2826,"num_write_kb":180864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1132,"num_bytes_recovered":74186752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"217'31","reported_seq":783,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.917213+0000","last_change":"2024-03-30T14:01:27.976934+0000","last_active":"2024-03-30T14:01:50.917213+0000","last_peered":"2024-03-30T14:01:50.917213+0000","last_clean":"2024-03-30T14:01:50.917213+0000","last_became_active":"2024-03-30T14:01:27.931359+0000","last_became_peered":"2024-03-30T14:01:27.931359+0000","last_unstale":"2024-03-30T14:01:50.917213+0000","last_undegraded":"2024-03-30T14:01:50.917213+0000","last_fullsized":"2024-03-30T14:01:50.917213+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-03-30T13:48:27.096261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:48:27.096261+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":757,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.934033+0000","last_change":"2024-03-30T14:01:01.229916+0000","last_active":"2024-03-30T14:01:50.934033+0000","last_peered":"2024-03-30T14:01:50.934033+0000","last_clean":"2024-03-30T14:01:50.934033+0000","last_became_active":"2024-03-30T13:42:45.315251+0000","last_became_peered":"2024-03-30T13:42:45.315251+0000","last_unstale":"2024-03-30T14:01:50.934033+0000","last_undegraded":"2024-03-30T14:01:50.934033+0000","last_fullsized":"2024-03-30T14:01:50.934033+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:01.229786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:57:12.187291+0000","last_clean_scrub_stamp":"2024-03-30T14:01:01.229786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:26.999234+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":751,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.921174+0000","last_change":"2024-03-30T14:00:24.179462+0000","last_active":"2024-03-30T14:01:50.921174+0000","last_peered":"2024-03-30T14:01:50.921174+0000","last_clean":"2024-03-30T14:01:50.921174+0000","last_became_active":"2024-03-30T13:42:49.321360+0000","last_became_peered":"2024-03-30T13:42:49.321360+0000","last_unstale":"2024-03-30T14:01:50.921174+0000","last_undegraded":"2024-03-30T14:01:50.921174+0000","last_fullsized":"2024-03-30T14:01:50.921174+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:24.179354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:59:06.350813+0000","last_clean_scrub_stamp":"2024-03-30T14:00:24.179354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:01:51.395112+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":502,"reported_epoch":400,"state":"active+clean+scrubbing","last_fresh":"2024-03-30T14:01:53.312096+0000","last_change":"2024-03-30T14:01:53.310817+0000","last_active":"2024-03-30T14:01:53.312096+0000","last_peered":"2024-03-30T14:01:53.312096+0000","last_clean":"2024-03-30T14:01:53.312096+0000","last_became_active":"2024-03-30T14:01:27.933449+0000","last_became_peered":"2024-03-30T14:01:27.933449+0000","last_unstale":"2024-03-30T14:01:53.312096+0000","last_undegraded":"2024-03-30T14:01:53.312096+0000","last_fullsized":"2024-03-30T14:01:53.312096+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:09.357378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:48:09.357378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":526,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.941401+0000","last_change":"2024-03-30T14:01:31.012758+0000","last_active":"2024-03-30T14:01:50.941401+0000","last_peered":"2024-03-30T14:01:50.941401+0000","last_clean":"2024-03-30T14:01:50.941401+0000","last_became_active":"2024-03-30T14:01:31.010864+0000","last_became_peered":"2024-03-30T14:01:31.010864+0000","last_unstale":"2024-03-30T14:01:50.941401+0000","last_undegraded":"2024-03-30T14:01:50.941401+0000","last_fullsized":"2024-03-30T14:01:50.941401+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:00:15.789401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:00:15.789401+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"400'2907","reported_seq":2957,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:53.931301+0000","last_change":"2024-03-30T14:01:13.909623+0000","last_active":"2024-03-30T14:01:53.931301+0000","last_peered":"2024-03-30T14:01:53.931301+0000","last_clean":"2024-03-30T14:01:53.931301+0000","last_became_active":"2024-03-30T14:01:13.909359+0000","last_became_peered":"2024-03-30T14:01:13.909359+0000","last_unstale":"2024-03-30T14:01:53.931301+0000","last_undegraded":"2024-03-30T14:01:53.931301+0000","last_fullsized":"2024-03-30T14:01:53.931301+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_clean_scrub_stamp":"2024-03-30T14:01:12.895052+0000","objects_scrubbed":0,"log_size":2907,"log_dups_size":0,"ondisk_log_size":2907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:21.891910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190513152,"num_objects":2907,"num_object_clones":0,"num_object_copies":8721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2907,"num_write_kb":186048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"217'19","reported_seq":771,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.917329+0000","last_change":"2024-03-30T13:58:53.440615+0000","last_active":"2024-03-30T14:01:50.917329+0000","last_peered":"2024-03-30T14:01:50.917329+0000","last_clean":"2024-03-30T14:01:50.917329+0000","last_became_active":"2024-03-30T13:42:47.315667+0000","last_became_peered":"2024-03-30T13:42:47.315667+0000","last_unstale":"2024-03-30T14:01:50.917329+0000","last_undegraded":"2024-03-30T14:01:50.917329+0000","last_fullsized":"2024-03-30T14:01:50.917329+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'19","last_scrub_stamp":"2024-03-30T13:58:53.440486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:58:53.440486+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":741,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916122+0000","last_change":"2024-03-30T14:01:28.927064+0000","last_active":"2024-03-30T14:01:50.916122+0000","last_peered":"2024-03-30T14:01:50.916122+0000","last_clean":"2024-03-30T14:01:50.916122+0000","last_became_active":"2024-03-30T14:01:28.926504+0000","last_became_peered":"2024-03-30T14:01:28.926504+0000","last_unstale":"2024-03-30T14:01:50.916122+0000","last_undegraded":"2024-03-30T14:01:50.916122+0000","last_fullsized":"2024-03-30T14:01:50.916122+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:21.110376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:00:21.110376+0000","last_clean_scrub_stamp":"2024-03-30T14:00:21.110376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":513,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.943110+0000","last_change":"2024-03-30T13:58:48.216887+0000","last_active":"2024-03-30T14:01:50.943110+0000","last_peered":"2024-03-30T14:01:50.943110+0000","last_clean":"2024-03-30T14:01:50.943110+0000","last_became_active":"2024-03-30T13:58:48.216542+0000","last_became_peered":"2024-03-30T13:58:48.216542+0000","last_unstale":"2024-03-30T14:01:50.943110+0000","last_undegraded":"2024-03-30T14:01:50.943110+0000","last_fullsized":"2024-03-30T14:01:50.943110+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:15.580974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:52.603772+0000","last_clean_scrub_stamp":"2024-03-30T13:58:15.580974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":734,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916422+0000","last_change":"2024-03-30T13:58:47.194433+0000","last_active":"2024-03-30T14:01:50.916422+0000","last_peered":"2024-03-30T14:01:50.916422+0000","last_clean":"2024-03-30T14:01:50.916422+0000","last_became_active":"2024-03-30T13:58:47.193821+0000","last_became_peered":"2024-03-30T13:58:47.193821+0000","last_unstale":"2024-03-30T14:01:50.916422+0000","last_undegraded":"2024-03-30T14:01:50.916422+0000","last_fullsized":"2024-03-30T14:01:50.916422+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:09.195697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:21.081938+0000","last_clean_scrub_stamp":"2024-03-30T13:58:09.195697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"82'128","reported_seq":746,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.920695+0000","last_change":"2024-03-30T14:01:02.759136+0000","last_active":"2024-03-30T14:01:50.920695+0000","last_peered":"2024-03-30T14:01:50.920695+0000","last_clean":"2024-03-30T14:01:50.920695+0000","last_became_active":"2024-03-30T14:01:02.753834+0000","last_became_peered":"2024-03-30T14:01:02.753834+0000","last_unstale":"2024-03-30T14:01:50.920695+0000","last_undegraded":"2024-03-30T14:01:50.920695+0000","last_fullsized":"2024-03-30T14:01:50.920695+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:00:58.121974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:00:58.121974+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:04.213851+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"400'3026","reported_seq":3076,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:56.160352+0000","last_change":"2024-03-30T14:01:13.914144+0000","last_active":"2024-03-30T14:01:56.160352+0000","last_peered":"2024-03-30T14:01:56.160352+0000","last_clean":"2024-03-30T14:01:56.160352+0000","last_became_active":"2024-03-30T14:01:13.913824+0000","last_became_peered":"2024-03-30T14:01:13.913824+0000","last_unstale":"2024-03-30T14:01:56.160352+0000","last_undegraded":"2024-03-30T14:01:56.160352+0000","last_fullsized":"2024-03-30T14:01:56.160352+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_clean_scrub_stamp":"2024-03-30T14:01:12.895052+0000","objects_scrubbed":0,"log_size":3026,"log_dups_size":0,"ondisk_log_size":3026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:26.925663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198311936,"num_objects":3026,"num_object_clones":0,"num_object_copies":9078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3026,"num_write_kb":193664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"217'34","reported_seq":797,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.920456+0000","last_change":"2024-03-30T14:00:02.158918+0000","last_active":"2024-03-30T14:01:50.920456+0000","last_peered":"2024-03-30T14:01:50.920456+0000","last_clean":"2024-03-30T14:01:50.920456+0000","last_became_active":"2024-03-30T13:42:47.324350+0000","last_became_peered":"2024-03-30T13:42:47.324350+0000","last_unstale":"2024-03-30T14:01:50.920456+0000","last_undegraded":"2024-03-30T14:01:50.920456+0000","last_fullsized":"2024-03-30T14:01:50.920456+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'34","last_scrub_stamp":"2024-03-30T14:00:02.158736+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:28.831527+0000","last_clean_scrub_stamp":"2024-03-30T14:00:02.158736+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":734,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916455+0000","last_change":"2024-03-30T14:00:11.446913+0000","last_active":"2024-03-30T14:01:50.916455+0000","last_peered":"2024-03-30T14:01:50.916455+0000","last_clean":"2024-03-30T14:01:50.916455+0000","last_became_active":"2024-03-30T13:53:35.886750+0000","last_became_peered":"2024-03-30T13:53:35.886750+0000","last_unstale":"2024-03-30T14:01:50.916455+0000","last_undegraded":"2024-03-30T14:01:50.916455+0000","last_fullsized":"2024-03-30T14:01:50.916455+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:11.446804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:00:11.446804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"327'13","reported_seq":981,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.917458+0000","last_change":"2024-03-30T13:58:48.205934+0000","last_active":"2024-03-30T14:01:50.917458+0000","last_peered":"2024-03-30T14:01:50.917458+0000","last_clean":"2024-03-30T14:01:50.917458+0000","last_became_active":"2024-03-30T13:58:48.204824+0000","last_became_peered":"2024-03-30T13:58:48.204824+0000","last_unstale":"2024-03-30T14:01:50.917458+0000","last_undegraded":"2024-03-30T14:01:50.917458+0000","last_fullsized":"2024-03-30T14:01:50.917458+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'11","last_scrub_stamp":"2024-03-30T13:58:06.437298+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-30T13:51:15.146255+0000","last_clean_scrub_stamp":"2024-03-30T13:58:06.437298+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":509,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.943265+0000","last_change":"2024-03-30T13:58:48.184750+0000","last_active":"2024-03-30T14:01:50.943265+0000","last_peered":"2024-03-30T14:01:50.943265+0000","last_clean":"2024-03-30T14:01:50.943265+0000","last_became_active":"2024-03-30T13:58:48.183934+0000","last_became_peered":"2024-03-30T13:58:48.183934+0000","last_unstale":"2024-03-30T14:01:50.943265+0000","last_undegraded":"2024-03-30T14:01:50.943265+0000","last_fullsized":"2024-03-30T14:01:50.943265+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:23.126245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:23.126245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":522,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.941992+0000","last_change":"2024-03-30T14:01:31.008701+0000","last_active":"2024-03-30T14:01:50.941992+0000","last_peered":"2024-03-30T14:01:50.941992+0000","last_clean":"2024-03-30T14:01:50.941992+0000","last_became_active":"2024-03-30T14:01:31.008205+0000","last_became_peered":"2024-03-30T14:01:31.008205+0000","last_unstale":"2024-03-30T14:01:50.941992+0000","last_undegraded":"2024-03-30T14:01:50.941992+0000","last_fullsized":"2024-03-30T14:01:50.941992+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:00:55.705199+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T13:56:08.385059+0000","last_clean_scrub_stamp":"2024-03-30T14:00:55.705199+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:21.322215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"400'3043","reported_seq":5508,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:56.182449+0000","last_change":"2024-03-30T14:01:50.926145+0000","last_active":"2024-03-30T14:01:56.182449+0000","last_peered":"2024-03-30T14:01:56.182449+0000","last_clean":"2024-03-30T14:01:56.182449+0000","last_became_active":"2024-03-30T14:01:50.922151+0000","last_became_peered":"2024-03-30T14:01:50.922151+0000","last_unstale":"2024-03-30T14:01:56.182449+0000","last_undegraded":"2024-03-30T14:01:56.182449+0000","last_fullsized":"2024-03-30T14:01:56.182449+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_clean_scrub_stamp":"2024-03-30T14:01:12.895052+0000","objects_scrubbed":0,"log_size":3043,"log_dups_size":0,"ondisk_log_size":3043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:42.840891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199426048,"num_objects":3043,"num_object_clones":0,"num_object_copies":9129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3043,"num_write_kb":194752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1205,"num_bytes_recovered":78970880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":735,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916115+0000","last_change":"2024-03-30T14:01:14.039578+0000","last_active":"2024-03-30T14:01:50.916115+0000","last_peered":"2024-03-30T14:01:50.916115+0000","last_clean":"2024-03-30T14:01:50.916115+0000","last_became_active":"2024-03-30T13:53:35.863124+0000","last_became_peered":"2024-03-30T13:53:35.863124+0000","last_unstale":"2024-03-30T14:01:50.916115+0000","last_undegraded":"2024-03-30T14:01:50.916115+0000","last_fullsized":"2024-03-30T14:01:50.916115+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:14.039501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:01:14.039501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:26.268255+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"216'35","reported_seq":812,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916561+0000","last_change":"2024-03-30T14:01:07.386479+0000","last_active":"2024-03-30T14:01:50.916561+0000","last_peered":"2024-03-30T14:01:50.916561+0000","last_clean":"2024-03-30T14:01:50.916561+0000","last_became_active":"2024-03-30T13:53:35.888652+0000","last_became_peered":"2024-03-30T13:53:35.888652+0000","last_unstale":"2024-03-30T14:01:50.916561+0000","last_undegraded":"2024-03-30T14:01:50.916561+0000","last_fullsized":"2024-03-30T14:01:50.916561+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'35","last_scrub_stamp":"2024-03-30T14:01:07.386369+0000","last_deep_scrub":"216'29","last_deep_scrub_stamp":"2024-03-30T13:52:52.973602+0000","last_clean_scrub_stamp":"2024-03-30T14:01:07.386369+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:11.878905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":748,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.917978+0000","last_change":"2024-03-30T14:00:18.364366+0000","last_active":"2024-03-30T14:01:50.917978+0000","last_peered":"2024-03-30T14:01:50.917978+0000","last_clean":"2024-03-30T14:01:50.917978+0000","last_became_active":"2024-03-30T13:58:47.194093+0000","last_became_peered":"2024-03-30T13:58:47.194093+0000","last_unstale":"2024-03-30T14:01:50.917978+0000","last_undegraded":"2024-03-30T14:01:50.917978+0000","last_fullsized":"2024-03-30T14:01:50.917978+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:18.364248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:53:01.031216+0000","last_clean_scrub_stamp":"2024-03-30T14:00:18.364248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":718,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916336+0000","last_change":"2024-03-30T14:01:48.194439+0000","last_active":"2024-03-30T14:01:50.916336+0000","last_peered":"2024-03-30T14:01:50.916336+0000","last_clean":"2024-03-30T14:01:50.916336+0000","last_became_active":"2024-03-30T14:01:27.931506+0000","last_became_peered":"2024-03-30T14:01:27.931506+0000","last_unstale":"2024-03-30T14:01:50.916336+0000","last_undegraded":"2024-03-30T14:01:50.916336+0000","last_fullsized":"2024-03-30T14:01:50.916336+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:48.194362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:01:48.194362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:03:01.764006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":740,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916854+0000","last_change":"2024-03-30T14:01:31.004056+0000","last_active":"2024-03-30T14:01:50.916854+0000","last_peered":"2024-03-30T14:01:50.916854+0000","last_clean":"2024-03-30T14:01:50.916854+0000","last_became_active":"2024-03-30T14:01:31.003676+0000","last_became_peered":"2024-03-30T14:01:31.003676+0000","last_unstale":"2024-03-30T14:01:50.916854+0000","last_undegraded":"2024-03-30T14:01:50.916854+0000","last_fullsized":"2024-03-30T14:01:50.916854+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:00:56.918551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:00:56.918551+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:06.833715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"400'2911","reported_seq":2961,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:53.866502+0000","last_change":"2024-03-30T14:01:13.911035+0000","last_active":"2024-03-30T14:01:53.866502+0000","last_peered":"2024-03-30T14:01:53.866502+0000","last_clean":"2024-03-30T14:01:53.866502+0000","last_became_active":"2024-03-30T14:01:13.910749+0000","last_became_peered":"2024-03-30T14:01:13.910749+0000","last_unstale":"2024-03-30T14:01:53.866502+0000","last_undegraded":"2024-03-30T14:01:53.866502+0000","last_fullsized":"2024-03-30T14:01:53.866502+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_clean_scrub_stamp":"2024-03-30T14:01:12.895052+0000","objects_scrubbed":0,"log_size":2911,"log_dups_size":0,"ondisk_log_size":2911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:20.172897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190775296,"num_objects":2911,"num_object_clones":0,"num_object_copies":8733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2911,"num_write_kb":186304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":959,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.918200+0000","last_change":"2024-03-30T13:57:22.046830+0000","last_active":"2024-03-30T14:01:50.918200+0000","last_peered":"2024-03-30T14:01:50.918200+0000","last_clean":"2024-03-30T14:01:50.918200+0000","last_became_active":"2024-03-30T13:42:49.321423+0000","last_became_peered":"2024-03-30T13:42:49.321423+0000","last_unstale":"2024-03-30T14:01:50.918200+0000","last_undegraded":"2024-03-30T14:01:50.918200+0000","last_fullsized":"2024-03-30T14:01:50.918200+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-30T13:57:22.046689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:22.046689+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"217'35","reported_seq":834,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916297+0000","last_change":"2024-03-30T14:00:12.151979+0000","last_active":"2024-03-30T14:01:50.916297+0000","last_peered":"2024-03-30T14:01:50.916297+0000","last_clean":"2024-03-30T14:01:50.916297+0000","last_became_active":"2024-03-30T13:58:47.192909+0000","last_became_peered":"2024-03-30T13:58:47.192909+0000","last_unstale":"2024-03-30T14:01:50.916297+0000","last_undegraded":"2024-03-30T14:01:50.916297+0000","last_fullsized":"2024-03-30T14:01:50.916297+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'35","last_scrub_stamp":"2024-03-30T14:00:12.151846+0000","last_deep_scrub":"91'21","last_deep_scrub_stamp":"2024-03-30T13:44:05.917064+0000","last_clean_scrub_stamp":"2024-03-30T14:00:12.151846+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":73,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.920120+0000","last_change":"2024-03-30T14:01:15.192508+0000","last_active":"2024-03-30T14:01:50.920120+0000","last_peered":"2024-03-30T14:01:50.920120+0000","last_clean":"2024-03-30T14:01:50.920120+0000","last_became_active":"2024-03-30T14:01:09.871040+0000","last_became_peered":"2024-03-30T14:01:09.871040+0000","last_unstale":"2024-03-30T14:01:50.920120+0000","last_undegraded":"2024-03-30T14:01:50.920120+0000","last_fullsized":"2024-03-30T14:01:50.920120+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T14:01:15.192409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:01:15.192409+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:33.730260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":610,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.934673+0000","last_change":"2024-03-30T14:01:28.923259+0000","last_active":"2024-03-30T14:01:50.934673+0000","last_peered":"2024-03-30T14:01:50.934673+0000","last_clean":"2024-03-30T14:01:50.934673+0000","last_became_active":"2024-03-30T14:01:28.922910+0000","last_became_peered":"2024-03-30T14:01:28.922910+0000","last_unstale":"2024-03-30T14:01:50.934673+0000","last_undegraded":"2024-03-30T14:01:50.934673+0000","last_fullsized":"2024-03-30T14:01:50.934673+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:12.273724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:31.900908+0000","last_clean_scrub_stamp":"2024-03-30T14:01:12.273724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:13.468554+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":771,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916201+0000","last_change":"2024-03-30T14:00:57.236106+0000","last_active":"2024-03-30T14:01:50.916201+0000","last_peered":"2024-03-30T14:01:50.916201+0000","last_clean":"2024-03-30T14:01:50.916201+0000","last_became_active":"2024-03-30T14:00:56.841088+0000","last_became_peered":"2024-03-30T14:00:56.841088+0000","last_unstale":"2024-03-30T14:01:50.916201+0000","last_undegraded":"2024-03-30T14:01:50.916201+0000","last_fullsized":"2024-03-30T14:01:50.916201+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:00:57.236019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:00:57.236019+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:07.364236+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"400'2925","reported_seq":5399,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:53.829407+0000","last_change":"2024-03-30T14:01:47.541273+0000","last_active":"2024-03-30T14:01:53.829407+0000","last_peered":"2024-03-30T14:01:53.829407+0000","last_clean":"2024-03-30T14:01:53.829407+0000","last_became_active":"2024-03-30T14:01:39.977513+0000","last_became_peered":"2024-03-30T14:01:39.977513+0000","last_unstale":"2024-03-30T14:01:53.829407+0000","last_undegraded":"2024-03-30T14:01:53.829407+0000","last_fullsized":"2024-03-30T14:01:53.829407+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_clean_scrub_stamp":"2024-03-30T14:01:12.895052+0000","objects_scrubbed":0,"log_size":2925,"log_dups_size":0,"ondisk_log_size":2925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:40.390940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191692800,"num_objects":2925,"num_object_clones":0,"num_object_copies":8775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2925,"num_write_kb":187200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1207,"num_bytes_recovered":79101952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"378'12","reported_seq":557,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.934577+0000","last_change":"2024-03-30T14:01:13.174104+0000","last_active":"2024-03-30T14:01:50.934577+0000","last_peered":"2024-03-30T14:01:50.934577+0000","last_clean":"2024-03-30T14:01:50.934577+0000","last_became_active":"2024-03-30T14:01:09.869694+0000","last_became_peered":"2024-03-30T14:01:09.869694+0000","last_unstale":"2024-03-30T14:01:50.934577+0000","last_undegraded":"2024-03-30T14:01:50.934577+0000","last_fullsized":"2024-03-30T14:01:50.934577+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'12","last_scrub_stamp":"2024-03-30T14:01:13.173987+0000","last_deep_scrub":"218'9","last_deep_scrub_stamp":"2024-03-30T13:52:57.873554+0000","last_clean_scrub_stamp":"2024-03-30T14:01:13.173987+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:13.679050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"217'25","reported_seq":825,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.920091+0000","last_change":"2024-03-30T14:00:23.161109+0000","last_active":"2024-03-30T14:01:50.920091+0000","last_peered":"2024-03-30T14:01:50.920091+0000","last_clean":"2024-03-30T14:01:50.920091+0000","last_became_active":"2024-03-30T13:58:47.186574+0000","last_became_peered":"2024-03-30T13:58:47.186574+0000","last_unstale":"2024-03-30T14:01:50.920091+0000","last_undegraded":"2024-03-30T14:01:50.920091+0000","last_fullsized":"2024-03-30T14:01:50.920091+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'25","last_scrub_stamp":"2024-03-30T14:00:23.160941+0000","last_deep_scrub":"217'25","last_deep_scrub_stamp":"2024-03-30T14:00:23.160941+0000","last_clean_scrub_stamp":"2024-03-30T14:00:23.160941+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":529,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.944383+0000","last_change":"2024-03-30T14:01:28.922645+0000","last_active":"2024-03-30T14:01:50.944383+0000","last_peered":"2024-03-30T14:01:50.944383+0000","last_clean":"2024-03-30T14:01:50.944383+0000","last_became_active":"2024-03-30T14:01:28.921918+0000","last_became_peered":"2024-03-30T14:01:28.921918+0000","last_unstale":"2024-03-30T14:01:50.944383+0000","last_undegraded":"2024-03-30T14:01:50.944383+0000","last_fullsized":"2024-03-30T14:01:50.944383+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:21.724810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:55.458779+0000","last_clean_scrub_stamp":"2024-03-30T14:00:21.724810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":293,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.915900+0000","last_change":"2024-03-30T14:01:27.969846+0000","last_active":"2024-03-30T14:01:50.915900+0000","last_peered":"2024-03-30T14:01:50.915900+0000","last_clean":"2024-03-30T14:01:50.915900+0000","last_became_active":"2024-03-30T14:01:27.969345+0000","last_became_peered":"2024-03-30T14:01:27.969345+0000","last_unstale":"2024-03-30T14:01:50.915900+0000","last_undegraded":"2024-03-30T14:01:50.915900+0000","last_fullsized":"2024-03-30T14:01:50.915900+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:28.946303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:41.142941+0000","last_clean_scrub_stamp":"2024-03-30T13:49:28.946303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":750,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.935133+0000","last_change":"2024-03-30T14:00:57.316672+0000","last_active":"2024-03-30T14:01:50.935133+0000","last_peered":"2024-03-30T14:01:50.935133+0000","last_clean":"2024-03-30T14:01:50.935133+0000","last_became_active":"2024-03-30T14:00:56.841284+0000","last_became_peered":"2024-03-30T14:00:56.841284+0000","last_unstale":"2024-03-30T14:01:50.935133+0000","last_undegraded":"2024-03-30T14:01:50.935133+0000","last_fullsized":"2024-03-30T14:01:50.935133+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:00:57.316597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:00:57.316597+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:17.848750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"400'3010","reported_seq":5371,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:56.168922+0000","last_change":"2024-03-30T14:01:40.979240+0000","last_active":"2024-03-30T14:01:56.168922+0000","last_peered":"2024-03-30T14:01:56.168922+0000","last_clean":"2024-03-30T14:01:56.168922+0000","last_became_active":"2024-03-30T14:01:40.976570+0000","last_became_peered":"2024-03-30T14:01:40.976570+0000","last_unstale":"2024-03-30T14:01:56.168922+0000","last_undegraded":"2024-03-30T14:01:56.168922+0000","last_fullsized":"2024-03-30T14:01:56.168922+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_clean_scrub_stamp":"2024-03-30T14:01:12.895052+0000","objects_scrubbed":0,"log_size":3010,"log_dups_size":0,"ondisk_log_size":3010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:29.635010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197263360,"num_objects":3010,"num_object_clones":0,"num_object_copies":9030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3010,"num_write_kb":192640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1151,"num_bytes_recovered":75431936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":761,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.934231+0000","last_change":"2024-03-30T14:00:24.340982+0000","last_active":"2024-03-30T14:01:50.934231+0000","last_peered":"2024-03-30T14:01:50.934231+0000","last_clean":"2024-03-30T14:01:50.934231+0000","last_became_active":"2024-03-30T13:58:47.204999+0000","last_became_peered":"2024-03-30T13:58:47.204999+0000","last_unstale":"2024-03-30T14:01:50.934231+0000","last_undegraded":"2024-03-30T14:01:50.934231+0000","last_fullsized":"2024-03-30T14:01:50.934231+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:24.340873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:52.887014+0000","last_clean_scrub_stamp":"2024-03-30T14:00:24.340873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"216'29","reported_seq":715,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.942661+0000","last_change":"2024-03-30T14:01:09.878949+0000","last_active":"2024-03-30T14:01:50.942661+0000","last_peered":"2024-03-30T14:01:50.942661+0000","last_clean":"2024-03-30T14:01:50.942661+0000","last_became_active":"2024-03-30T14:01:09.878634+0000","last_became_peered":"2024-03-30T14:01:09.878634+0000","last_unstale":"2024-03-30T14:01:50.942661+0000","last_undegraded":"2024-03-30T14:01:50.942661+0000","last_fullsized":"2024-03-30T14:01:50.942661+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'29","last_scrub_stamp":"2024-03-30T13:57:59.935365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:59.935365+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":762,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.921314+0000","last_change":"2024-03-30T14:01:28.998850+0000","last_active":"2024-03-30T14:01:50.921314+0000","last_peered":"2024-03-30T14:01:50.921314+0000","last_clean":"2024-03-30T14:01:50.921314+0000","last_became_active":"2024-03-30T14:01:28.998527+0000","last_became_peered":"2024-03-30T14:01:28.998527+0000","last_unstale":"2024-03-30T14:01:50.921314+0000","last_undegraded":"2024-03-30T14:01:50.921314+0000","last_fullsized":"2024-03-30T14:01:50.921314+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:04.161648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:45.975163+0000","last_clean_scrub_stamp":"2024-03-30T14:00:04.161648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":717,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.919269+0000","last_change":"2024-03-30T14:01:27.931969+0000","last_active":"2024-03-30T14:01:50.919269+0000","last_peered":"2024-03-30T14:01:50.919269+0000","last_clean":"2024-03-30T14:01:50.919269+0000","last_became_active":"2024-03-30T14:01:27.931619+0000","last_became_peered":"2024-03-30T14:01:27.931619+0000","last_unstale":"2024-03-30T14:01:50.919269+0000","last_undegraded":"2024-03-30T14:01:50.919269+0000","last_fullsized":"2024-03-30T14:01:50.919269+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:21.527712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:49:21.527712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":744,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.934859+0000","last_change":"2024-03-30T14:00:59.289917+0000","last_active":"2024-03-30T14:01:50.934859+0000","last_peered":"2024-03-30T14:01:50.934859+0000","last_clean":"2024-03-30T14:01:50.934859+0000","last_became_active":"2024-03-30T14:00:56.843071+0000","last_became_peered":"2024-03-30T14:00:56.843071+0000","last_unstale":"2024-03-30T14:01:50.934859+0000","last_undegraded":"2024-03-30T14:01:50.934859+0000","last_fullsized":"2024-03-30T14:01:50.934859+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:00:59.289832+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T13:57:14.131256+0000","last_clean_scrub_stamp":"2024-03-30T14:00:59.289832+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:05.314422+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"400'2862","reported_seq":2912,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:53.440308+0000","last_change":"2024-03-30T14:01:13.907919+0000","last_active":"2024-03-30T14:01:53.440308+0000","last_peered":"2024-03-30T14:01:53.440308+0000","last_clean":"2024-03-30T14:01:53.440308+0000","last_became_active":"2024-03-30T14:01:13.907626+0000","last_became_peered":"2024-03-30T14:01:13.907626+0000","last_unstale":"2024-03-30T14:01:53.440308+0000","last_undegraded":"2024-03-30T14:01:53.440308+0000","last_fullsized":"2024-03-30T14:01:53.440308+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_clean_scrub_stamp":"2024-03-30T14:01:12.895052+0000","objects_scrubbed":0,"log_size":2862,"log_dups_size":0,"ondisk_log_size":2862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:24.889220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187564032,"num_objects":2862,"num_object_clones":0,"num_object_copies":8586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2862,"num_write_kb":183168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":168,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.942990+0000","last_change":"2024-03-30T14:00:22.722109+0000","last_active":"2024-03-30T14:01:50.942990+0000","last_peered":"2024-03-30T14:01:50.942990+0000","last_clean":"2024-03-30T14:01:50.942990+0000","last_became_active":"2024-03-30T13:58:47.195511+0000","last_became_peered":"2024-03-30T13:58:47.195511+0000","last_unstale":"2024-03-30T14:01:50.942990+0000","last_undegraded":"2024-03-30T14:01:50.942990+0000","last_fullsized":"2024-03-30T14:01:50.942990+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:22.722033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:00:22.722033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"216'30","reported_seq":459,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.920765+0000","last_change":"2024-03-30T14:01:27.978237+0000","last_active":"2024-03-30T14:01:50.920765+0000","last_peered":"2024-03-30T14:01:50.920765+0000","last_clean":"2024-03-30T14:01:50.920765+0000","last_became_active":"2024-03-30T14:01:27.965041+0000","last_became_peered":"2024-03-30T14:01:27.965041+0000","last_unstale":"2024-03-30T14:01:50.920765+0000","last_undegraded":"2024-03-30T14:01:50.920765+0000","last_fullsized":"2024-03-30T14:01:50.920765+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'18","last_scrub_stamp":"2024-03-30T13:49:13.774982+0000","last_deep_scrub":"93'18","last_deep_scrub_stamp":"2024-03-30T13:47:28.872529+0000","last_clean_scrub_stamp":"2024-03-30T13:49:13.774982+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":921,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:55.178593+0000","last_change":"2024-03-30T13:59:10.350921+0000","last_active":"2024-03-30T14:01:55.178593+0000","last_peered":"2024-03-30T14:01:55.178593+0000","last_clean":"2024-03-30T14:01:55.178593+0000","last_became_active":"2024-03-30T13:53:35.889071+0000","last_became_peered":"2024-03-30T13:53:35.889071+0000","last_unstale":"2024-03-30T14:01:55.178593+0000","last_undegraded":"2024-03-30T14:01:55.178593+0000","last_fullsized":"2024-03-30T14:01:55.178593+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-30T13:59:10.350780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:59:10.350780+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0.00126391,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":505,"num_read_kb":500,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":57,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916405+0000","last_change":"2024-03-30T14:01:09.871953+0000","last_active":"2024-03-30T14:01:50.916405+0000","last_peered":"2024-03-30T14:01:50.916405+0000","last_clean":"2024-03-30T14:01:50.916405+0000","last_became_active":"2024-03-30T14:01:09.871602+0000","last_became_peered":"2024-03-30T14:01:09.871602+0000","last_unstale":"2024-03-30T14:01:50.916405+0000","last_undegraded":"2024-03-30T14:01:50.916405+0000","last_fullsized":"2024-03-30T14:01:50.916405+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:25.597831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:00:25.597831+0000","last_clean_scrub_stamp":"2024-03-30T14:00:25.597831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":752,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.920105+0000","last_change":"2024-03-30T14:01:00.115991+0000","last_active":"2024-03-30T14:01:50.920105+0000","last_peered":"2024-03-30T14:01:50.920105+0000","last_clean":"2024-03-30T14:01:50.920105+0000","last_became_active":"2024-03-30T13:58:47.189419+0000","last_became_peered":"2024-03-30T13:58:47.189419+0000","last_unstale":"2024-03-30T14:01:50.920105+0000","last_undegraded":"2024-03-30T14:01:50.920105+0000","last_fullsized":"2024-03-30T14:01:50.920105+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:00.115910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:01:00.115910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:22.561628+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":739,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.921302+0000","last_change":"2024-03-30T14:00:57.153715+0000","last_active":"2024-03-30T14:01:50.921302+0000","last_peered":"2024-03-30T14:01:50.921302+0000","last_clean":"2024-03-30T14:01:50.921302+0000","last_became_active":"2024-03-30T14:00:56.844488+0000","last_became_peered":"2024-03-30T14:00:56.844488+0000","last_unstale":"2024-03-30T14:01:50.921302+0000","last_undegraded":"2024-03-30T14:01:50.921302+0000","last_fullsized":"2024-03-30T14:01:50.921302+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:00:57.153608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:00:57.153608+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:04.371162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"400'2823","reported_seq":2873,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:53.467521+0000","last_change":"2024-03-30T14:01:13.911140+0000","last_active":"2024-03-30T14:01:53.467521+0000","last_peered":"2024-03-30T14:01:53.467521+0000","last_clean":"2024-03-30T14:01:53.467521+0000","last_became_active":"2024-03-30T14:01:13.910817+0000","last_became_peered":"2024-03-30T14:01:13.910817+0000","last_unstale":"2024-03-30T14:01:53.467521+0000","last_undegraded":"2024-03-30T14:01:53.467521+0000","last_fullsized":"2024-03-30T14:01:53.467521+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_clean_scrub_stamp":"2024-03-30T14:01:12.895052+0000","objects_scrubbed":0,"log_size":2823,"log_dups_size":0,"ondisk_log_size":2823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:19.329434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185008128,"num_objects":2823,"num_object_clones":0,"num_object_copies":8469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2823,"num_write_kb":180672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":171,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.941891+0000","last_change":"2024-03-30T14:01:40.481407+0000","last_active":"2024-03-30T14:01:50.941891+0000","last_peered":"2024-03-30T14:01:50.941891+0000","last_clean":"2024-03-30T14:01:50.941891+0000","last_became_active":"2024-03-30T14:01:28.998218+0000","last_became_peered":"2024-03-30T14:01:28.998218+0000","last_unstale":"2024-03-30T14:01:50.941891+0000","last_undegraded":"2024-03-30T14:01:50.941891+0000","last_fullsized":"2024-03-30T14:01:50.941891+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:40.481334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:01:40.481334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:53.394126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"229'29","reported_seq":1048,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.934793+0000","last_change":"2024-03-30T14:01:44.288353+0000","last_active":"2024-03-30T14:01:50.934793+0000","last_peered":"2024-03-30T14:01:50.934793+0000","last_clean":"2024-03-30T14:01:50.934793+0000","last_became_active":"2024-03-30T13:53:35.864416+0000","last_became_peered":"2024-03-30T13:53:35.864416+0000","last_unstale":"2024-03-30T14:01:50.934793+0000","last_undegraded":"2024-03-30T14:01:50.934793+0000","last_fullsized":"2024-03-30T14:01:50.934793+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'29","last_scrub_stamp":"2024-03-30T14:01:44.288207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:01:44.288207+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:50.590203+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":741,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916840+0000","last_change":"2024-03-30T13:58:47.193901+0000","last_active":"2024-03-30T14:01:50.916840+0000","last_peered":"2024-03-30T14:01:50.916840+0000","last_clean":"2024-03-30T14:01:50.916840+0000","last_became_active":"2024-03-30T13:58:47.193538+0000","last_became_peered":"2024-03-30T13:58:47.193538+0000","last_unstale":"2024-03-30T14:01:50.916840+0000","last_undegraded":"2024-03-30T14:01:50.916840+0000","last_fullsized":"2024-03-30T14:01:50.916840+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:16.238202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:12.975288+0000","last_clean_scrub_stamp":"2024-03-30T13:58:16.238202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":359,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.933917+0000","last_change":"2024-03-30T14:01:10.265990+0000","last_active":"2024-03-30T14:01:50.933917+0000","last_peered":"2024-03-30T14:01:50.933917+0000","last_clean":"2024-03-30T14:01:50.933917+0000","last_became_active":"2024-03-30T14:01:09.876485+0000","last_became_peered":"2024-03-30T14:01:09.876485+0000","last_unstale":"2024-03-30T14:01:50.933917+0000","last_undegraded":"2024-03-30T14:01:50.933917+0000","last_fullsized":"2024-03-30T14:01:50.933917+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:01:10.265901+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T14:01:10.265901+0000","last_clean_scrub_stamp":"2024-03-30T14:01:10.265901+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:34.908778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":168,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.942829+0000","last_change":"2024-03-30T14:00:05.877131+0000","last_active":"2024-03-30T14:01:50.942829+0000","last_peered":"2024-03-30T14:01:50.942829+0000","last_clean":"2024-03-30T14:01:50.942829+0000","last_became_active":"2024-03-30T13:58:47.194374+0000","last_became_peered":"2024-03-30T13:58:47.194374+0000","last_unstale":"2024-03-30T14:01:50.942829+0000","last_undegraded":"2024-03-30T14:01:50.942829+0000","last_fullsized":"2024-03-30T14:01:50.942829+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:05.877011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:12.433166+0000","last_clean_scrub_stamp":"2024-03-30T14:00:05.877011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"400'2874","reported_seq":2923,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:53.417516+0000","last_change":"2024-03-30T14:01:13.911072+0000","last_active":"2024-03-30T14:01:53.417516+0000","last_peered":"2024-03-30T14:01:53.417516+0000","last_clean":"2024-03-30T14:01:53.417516+0000","last_became_active":"2024-03-30T14:01:13.910750+0000","last_became_peered":"2024-03-30T14:01:13.910750+0000","last_unstale":"2024-03-30T14:01:53.417516+0000","last_undegraded":"2024-03-30T14:01:53.417516+0000","last_fullsized":"2024-03-30T14:01:53.417516+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_clean_scrub_stamp":"2024-03-30T14:01:12.895052+0000","objects_scrubbed":0,"log_size":2874,"log_dups_size":0,"ondisk_log_size":2874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:32.409301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188350464,"num_objects":2874,"num_object_clones":0,"num_object_copies":8622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2874,"num_write_kb":183936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":171,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.942317+0000","last_change":"2024-03-30T14:01:28.925945+0000","last_active":"2024-03-30T14:01:50.942317+0000","last_peered":"2024-03-30T14:01:50.942317+0000","last_clean":"2024-03-30T14:01:50.942317+0000","last_became_active":"2024-03-30T14:01:28.925617+0000","last_became_peered":"2024-03-30T14:01:28.925617+0000","last_unstale":"2024-03-30T14:01:50.942317+0000","last_undegraded":"2024-03-30T14:01:50.942317+0000","last_fullsized":"2024-03-30T14:01:50.942317+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:00.683688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:28.373277+0000","last_clean_scrub_stamp":"2024-03-30T14:01:00.683688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:02.444769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"216'17","reported_seq":834,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.920564+0000","last_change":"2024-03-30T13:58:59.170542+0000","last_active":"2024-03-30T14:01:50.920564+0000","last_peered":"2024-03-30T14:01:50.920564+0000","last_clean":"2024-03-30T14:01:50.920564+0000","last_became_active":"2024-03-30T13:58:47.192640+0000","last_became_peered":"2024-03-30T13:58:47.192640+0000","last_unstale":"2024-03-30T14:01:50.920564+0000","last_undegraded":"2024-03-30T14:01:50.920564+0000","last_fullsized":"2024-03-30T14:01:50.920564+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'17","last_scrub_stamp":"2024-03-30T13:58:59.170397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:58:59.170397+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":498,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.918679+0000","last_change":"2024-03-30T14:01:27.969842+0000","last_active":"2024-03-30T14:01:50.918679+0000","last_peered":"2024-03-30T14:01:50.918679+0000","last_clean":"2024-03-30T14:01:50.918679+0000","last_became_active":"2024-03-30T14:01:27.969316+0000","last_became_peered":"2024-03-30T14:01:27.969316+0000","last_unstale":"2024-03-30T14:01:50.918679+0000","last_undegraded":"2024-03-30T14:01:50.918679+0000","last_fullsized":"2024-03-30T14:01:50.918679+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:25.398232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:51.919203+0000","last_clean_scrub_stamp":"2024-03-30T13:49:25.398232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":729,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.919625+0000","last_change":"2024-03-30T14:01:27.964107+0000","last_active":"2024-03-30T14:01:50.919625+0000","last_peered":"2024-03-30T14:01:50.919625+0000","last_clean":"2024-03-30T14:01:50.919625+0000","last_became_active":"2024-03-30T14:01:27.963241+0000","last_became_peered":"2024-03-30T14:01:27.963241+0000","last_unstale":"2024-03-30T14:01:50.919625+0000","last_undegraded":"2024-03-30T14:01:50.919625+0000","last_fullsized":"2024-03-30T14:01:50.919625+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:16.464234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:59.810189+0000","last_clean_scrub_stamp":"2024-03-30T13:49:16.464234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":384,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.934823+0000","last_change":"2024-03-30T14:01:28.921463+0000","last_active":"2024-03-30T14:01:50.934823+0000","last_peered":"2024-03-30T14:01:50.934823+0000","last_clean":"2024-03-30T14:01:50.934823+0000","last_became_active":"2024-03-30T14:01:28.921095+0000","last_became_peered":"2024-03-30T14:01:28.921095+0000","last_unstale":"2024-03-30T14:01:50.934823+0000","last_undegraded":"2024-03-30T14:01:50.934823+0000","last_fullsized":"2024-03-30T14:01:50.934823+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:02.296867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:01:02.296867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:05.464331+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":662,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916111+0000","last_change":"2024-03-30T14:01:31.002530+0000","last_active":"2024-03-30T14:01:50.916111+0000","last_peered":"2024-03-30T14:01:50.916111+0000","last_clean":"2024-03-30T14:01:50.916111+0000","last_became_active":"2024-03-30T14:01:31.001849+0000","last_became_peered":"2024-03-30T14:01:31.001849+0000","last_unstale":"2024-03-30T14:01:50.916111+0000","last_undegraded":"2024-03-30T14:01:50.916111+0000","last_fullsized":"2024-03-30T14:01:50.916111+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:01:11.075885+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T14:01:11.075885+0000","last_clean_scrub_stamp":"2024-03-30T14:01:11.075885+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:34.331719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.b","version":"400'2794","reported_seq":2843,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:53.462140+0000","last_change":"2024-03-30T14:01:13.913438+0000","last_active":"2024-03-30T14:01:53.462140+0000","last_peered":"2024-03-30T14:01:53.462140+0000","last_clean":"2024-03-30T14:01:53.462140+0000","last_became_active":"2024-03-30T14:01:13.913091+0000","last_became_peered":"2024-03-30T14:01:13.913091+0000","last_unstale":"2024-03-30T14:01:53.462140+0000","last_undegraded":"2024-03-30T14:01:53.462140+0000","last_fullsized":"2024-03-30T14:01:53.462140+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_clean_scrub_stamp":"2024-03-30T14:01:12.895052+0000","objects_scrubbed":0,"log_size":2794,"log_dups_size":0,"ondisk_log_size":2794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:33.735360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183107584,"num_objects":2794,"num_object_clones":0,"num_object_copies":8382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2794,"num_write_kb":178816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":56,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.920193+0000","last_change":"2024-03-30T14:01:09.876591+0000","last_active":"2024-03-30T14:01:50.920193+0000","last_peered":"2024-03-30T14:01:50.920193+0000","last_clean":"2024-03-30T14:01:50.920193+0000","last_became_active":"2024-03-30T14:01:09.876281+0000","last_became_peered":"2024-03-30T14:01:09.876281+0000","last_unstale":"2024-03-30T14:01:50.920193+0000","last_undegraded":"2024-03-30T14:01:50.920193+0000","last_fullsized":"2024-03-30T14:01:50.920193+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:07.527735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:18.862784+0000","last_clean_scrub_stamp":"2024-03-30T14:01:07.527735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:16.023433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"391'42","reported_seq":1038,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.921120+0000","last_change":"2024-03-30T14:01:49.587111+0000","last_active":"2024-03-30T14:01:50.921120+0000","last_peered":"2024-03-30T14:01:50.921120+0000","last_clean":"2024-03-30T14:01:50.921120+0000","last_became_active":"2024-03-30T14:01:27.966083+0000","last_became_peered":"2024-03-30T14:01:27.966083+0000","last_unstale":"2024-03-30T14:01:50.921120+0000","last_undegraded":"2024-03-30T14:01:50.921120+0000","last_fullsized":"2024-03-30T14:01:50.921120+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'42","last_scrub_stamp":"2024-03-30T14:01:49.586938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:01:49.586938+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:03:16.815661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":729,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916438+0000","last_change":"2024-03-30T13:58:04.478097+0000","last_active":"2024-03-30T14:01:50.916438+0000","last_peered":"2024-03-30T14:01:50.916438+0000","last_clean":"2024-03-30T14:01:50.916438+0000","last_became_active":"2024-03-30T13:42:45.310209+0000","last_became_peered":"2024-03-30T13:42:45.310209+0000","last_unstale":"2024-03-30T14:01:50.916438+0000","last_undegraded":"2024-03-30T14:01:50.916438+0000","last_fullsized":"2024-03-30T14:01:50.916438+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:04.478028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:04.478028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":164,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.941839+0000","last_change":"2024-03-30T13:58:47.215143+0000","last_active":"2024-03-30T14:01:50.941839+0000","last_peered":"2024-03-30T14:01:50.941839+0000","last_clean":"2024-03-30T14:01:50.941839+0000","last_became_active":"2024-03-30T13:58:47.199657+0000","last_became_peered":"2024-03-30T13:58:47.199657+0000","last_unstale":"2024-03-30T14:01:50.941839+0000","last_undegraded":"2024-03-30T14:01:50.941839+0000","last_fullsized":"2024-03-30T14:01:50.941839+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-30T13:58:03.985919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T13:58:03.985919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":722,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.917461+0000","last_change":"2024-03-30T13:57:57.278091+0000","last_active":"2024-03-30T14:01:50.917461+0000","last_peered":"2024-03-30T14:01:50.917461+0000","last_clean":"2024-03-30T14:01:50.917461+0000","last_became_active":"2024-03-30T13:53:35.855186+0000","last_became_peered":"2024-03-30T13:53:35.855186+0000","last_unstale":"2024-03-30T14:01:50.917461+0000","last_undegraded":"2024-03-30T14:01:50.917461+0000","last_fullsized":"2024-03-30T14:01:50.917461+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:57.277970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:57.277970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":535,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.941367+0000","last_change":"2024-03-30T14:01:31.008667+0000","last_active":"2024-03-30T14:01:50.941367+0000","last_peered":"2024-03-30T14:01:50.941367+0000","last_clean":"2024-03-30T14:01:50.941367+0000","last_became_active":"2024-03-30T14:01:31.008225+0000","last_became_peered":"2024-03-30T14:01:31.008225+0000","last_unstale":"2024-03-30T14:01:50.941367+0000","last_undegraded":"2024-03-30T14:01:50.941367+0000","last_fullsized":"2024-03-30T14:01:50.941367+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:00:57.662618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:00:57.662618+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:22.969276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"400'2914","reported_seq":5342,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:53.961631+0000","last_change":"2024-03-30T14:01:40.982240+0000","last_active":"2024-03-30T14:01:53.961631+0000","last_peered":"2024-03-30T14:01:53.961631+0000","last_clean":"2024-03-30T14:01:53.961631+0000","last_became_active":"2024-03-30T14:01:40.979471+0000","last_became_peered":"2024-03-30T14:01:40.979471+0000","last_unstale":"2024-03-30T14:01:53.961631+0000","last_undegraded":"2024-03-30T14:01:53.961631+0000","last_fullsized":"2024-03-30T14:01:53.961631+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_clean_scrub_stamp":"2024-03-30T14:01:12.895052+0000","objects_scrubbed":0,"log_size":2914,"log_dups_size":0,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:19.564803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190971904,"num_objects":2914,"num_object_clones":0,"num_object_copies":8742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":186496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1185,"num_bytes_recovered":77660160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"216'15","reported_seq":472,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.920969+0000","last_change":"2024-03-30T14:01:13.279683+0000","last_active":"2024-03-30T14:01:50.920969+0000","last_peered":"2024-03-30T14:01:50.920969+0000","last_clean":"2024-03-30T14:01:50.920969+0000","last_became_active":"2024-03-30T14:01:09.873532+0000","last_became_peered":"2024-03-30T14:01:09.873532+0000","last_unstale":"2024-03-30T14:01:50.920969+0000","last_undegraded":"2024-03-30T14:01:50.920969+0000","last_fullsized":"2024-03-30T14:01:50.920969+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'15","last_scrub_stamp":"2024-03-30T14:01:13.279579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:01:13.279579+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:27.284322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":168,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.942126+0000","last_change":"2024-03-30T14:01:48.513590+0000","last_active":"2024-03-30T14:01:50.942126+0000","last_peered":"2024-03-30T14:01:50.942126+0000","last_clean":"2024-03-30T14:01:50.942126+0000","last_became_active":"2024-03-30T13:58:47.194288+0000","last_became_peered":"2024-03-30T13:58:47.194288+0000","last_unstale":"2024-03-30T14:01:50.942126+0000","last_undegraded":"2024-03-30T14:01:50.942126+0000","last_fullsized":"2024-03-30T14:01:50.942126+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:48.513515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:01:48.513515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:57.851755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":497,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916563+0000","last_change":"2024-03-30T14:01:27.932340+0000","last_active":"2024-03-30T14:01:50.916563+0000","last_peered":"2024-03-30T14:01:50.916563+0000","last_clean":"2024-03-30T14:01:50.916563+0000","last_became_active":"2024-03-30T14:01:27.931964+0000","last_became_peered":"2024-03-30T14:01:27.931964+0000","last_unstale":"2024-03-30T14:01:50.916563+0000","last_undegraded":"2024-03-30T14:01:50.916563+0000","last_fullsized":"2024-03-30T14:01:50.916563+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:13.399683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:28.225742+0000","last_clean_scrub_stamp":"2024-03-30T13:49:13.399683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":412,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.918505+0000","last_change":"2024-03-30T14:01:28.924315+0000","last_active":"2024-03-30T14:01:50.918505+0000","last_peered":"2024-03-30T14:01:50.918505+0000","last_clean":"2024-03-30T14:01:50.918505+0000","last_became_active":"2024-03-30T14:01:28.923498+0000","last_became_peered":"2024-03-30T14:01:28.923498+0000","last_unstale":"2024-03-30T14:01:50.918505+0000","last_undegraded":"2024-03-30T14:01:50.918505+0000","last_fullsized":"2024-03-30T14:01:50.918505+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:55.598675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:44.095338+0000","last_clean_scrub_stamp":"2024-03-30T14:00:55.598675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:13.642347+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"82'128","reported_seq":376,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916866+0000","last_change":"2024-03-30T14:01:31.003584+0000","last_active":"2024-03-30T14:01:50.916866+0000","last_peered":"2024-03-30T14:01:50.916866+0000","last_clean":"2024-03-30T14:01:50.916866+0000","last_became_active":"2024-03-30T14:01:31.003158+0000","last_became_peered":"2024-03-30T14:01:31.003158+0000","last_unstale":"2024-03-30T14:01:50.916866+0000","last_undegraded":"2024-03-30T14:01:50.916866+0000","last_fullsized":"2024-03-30T14:01:50.916866+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:01:03.967402+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T13:57:14.131256+0000","last_clean_scrub_stamp":"2024-03-30T14:01:03.967402+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:07.040039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"400'2909","reported_seq":2959,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:53.745427+0000","last_change":"2024-03-30T14:01:13.910931+0000","last_active":"2024-03-30T14:01:53.745427+0000","last_peered":"2024-03-30T14:01:53.745427+0000","last_clean":"2024-03-30T14:01:53.745427+0000","last_became_active":"2024-03-30T14:01:13.910607+0000","last_became_peered":"2024-03-30T14:01:13.910607+0000","last_unstale":"2024-03-30T14:01:53.745427+0000","last_undegraded":"2024-03-30T14:01:53.745427+0000","last_fullsized":"2024-03-30T14:01:53.745427+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_clean_scrub_stamp":"2024-03-30T14:01:12.895052+0000","objects_scrubbed":0,"log_size":2909,"log_dups_size":0,"ondisk_log_size":2909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:15.421215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190644224,"num_objects":2909,"num_object_clones":0,"num_object_copies":8727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2909,"num_write_kb":186176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"217'44","reported_seq":869,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.934167+0000","last_change":"2024-03-30T14:01:14.212381+0000","last_active":"2024-03-30T14:01:50.934167+0000","last_peered":"2024-03-30T14:01:50.934167+0000","last_clean":"2024-03-30T14:01:50.934167+0000","last_became_active":"2024-03-30T13:58:47.204577+0000","last_became_peered":"2024-03-30T13:58:47.204577+0000","last_unstale":"2024-03-30T14:01:50.934167+0000","last_undegraded":"2024-03-30T14:01:50.934167+0000","last_fullsized":"2024-03-30T14:01:50.934167+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'44","last_scrub_stamp":"2024-03-30T14:01:14.212266+0000","last_deep_scrub":"108'28","last_deep_scrub_stamp":"2024-03-30T13:45:49.140418+0000","last_clean_scrub_stamp":"2024-03-30T14:01:14.212266+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:17.061215+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":160,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.942934+0000","last_change":"2024-03-30T13:58:47.200041+0000","last_active":"2024-03-30T14:01:50.942934+0000","last_peered":"2024-03-30T14:01:50.942934+0000","last_clean":"2024-03-30T14:01:50.942934+0000","last_became_active":"2024-03-30T13:58:47.199395+0000","last_became_peered":"2024-03-30T13:58:47.199395+0000","last_unstale":"2024-03-30T14:01:50.942934+0000","last_undegraded":"2024-03-30T14:01:50.942934+0000","last_fullsized":"2024-03-30T14:01:50.942934+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:16.087693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:17.366740+0000","last_clean_scrub_stamp":"2024-03-30T13:58:16.087693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":417,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.942662+0000","last_change":"2024-03-30T14:01:28.921192+0000","last_active":"2024-03-30T14:01:50.942662+0000","last_peered":"2024-03-30T14:01:50.942662+0000","last_clean":"2024-03-30T14:01:50.942662+0000","last_became_active":"2024-03-30T14:01:28.920865+0000","last_became_peered":"2024-03-30T14:01:28.920865+0000","last_unstale":"2024-03-30T14:01:50.942662+0000","last_undegraded":"2024-03-30T14:01:50.942662+0000","last_fullsized":"2024-03-30T14:01:50.942662+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:20.674306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:00:20.674306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":603,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.915909+0000","last_change":"2024-03-30T14:01:28.924352+0000","last_active":"2024-03-30T14:01:50.915909+0000","last_peered":"2024-03-30T14:01:50.915909+0000","last_clean":"2024-03-30T14:01:50.915909+0000","last_became_active":"2024-03-30T14:01:28.923943+0000","last_became_peered":"2024-03-30T14:01:28.923943+0000","last_unstale":"2024-03-30T14:01:50.915909+0000","last_undegraded":"2024-03-30T14:01:50.915909+0000","last_fullsized":"2024-03-30T14:01:50.915909+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:03.960362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:03.960362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"82'128","reported_seq":757,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.935021+0000","last_change":"2024-03-30T14:01:02.742991+0000","last_active":"2024-03-30T14:01:50.935021+0000","last_peered":"2024-03-30T14:01:50.935021+0000","last_clean":"2024-03-30T14:01:50.935021+0000","last_became_active":"2024-03-30T14:01:02.741907+0000","last_became_peered":"2024-03-30T14:01:02.741907+0000","last_unstale":"2024-03-30T14:01:50.935021+0000","last_undegraded":"2024-03-30T14:01:50.935021+0000","last_fullsized":"2024-03-30T14:01:50.935021+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:00:58.273670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:00:58.273670+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:06.164821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"400'3074","reported_seq":5530,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:56.142578+0000","last_change":"2024-03-30T14:01:49.924615+0000","last_active":"2024-03-30T14:01:56.142578+0000","last_peered":"2024-03-30T14:01:56.142578+0000","last_clean":"2024-03-30T14:01:56.142578+0000","last_became_active":"2024-03-30T14:01:49.920771+0000","last_became_peered":"2024-03-30T14:01:49.920771+0000","last_unstale":"2024-03-30T14:01:56.142578+0000","last_undegraded":"2024-03-30T14:01:56.142578+0000","last_fullsized":"2024-03-30T14:01:56.142578+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_clean_scrub_stamp":"2024-03-30T14:01:12.895052+0000","objects_scrubbed":0,"log_size":3074,"log_dups_size":0,"ondisk_log_size":3074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:23.118119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201457664,"num_objects":3074,"num_object_clones":0,"num_object_copies":9222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3074,"num_write_kb":196736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1199,"num_bytes_recovered":78577664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":76,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.919444+0000","last_change":"2024-03-30T14:01:27.963599+0000","last_active":"2024-03-30T14:01:50.919444+0000","last_peered":"2024-03-30T14:01:50.919444+0000","last_clean":"2024-03-30T14:01:50.919444+0000","last_became_active":"2024-03-30T14:01:27.932420+0000","last_became_peered":"2024-03-30T14:01:27.932420+0000","last_unstale":"2024-03-30T14:01:50.919444+0000","last_undegraded":"2024-03-30T14:01:50.919444+0000","last_fullsized":"2024-03-30T14:01:50.919444+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:22.401087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:49:22.401087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"216'18","reported_seq":764,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916572+0000","last_change":"2024-03-30T13:58:54.451235+0000","last_active":"2024-03-30T14:01:50.916572+0000","last_peered":"2024-03-30T14:01:50.916572+0000","last_clean":"2024-03-30T14:01:50.916572+0000","last_became_active":"2024-03-30T13:42:47.316187+0000","last_became_peered":"2024-03-30T13:42:47.316187+0000","last_unstale":"2024-03-30T14:01:50.916572+0000","last_undegraded":"2024-03-30T14:01:50.916572+0000","last_fullsized":"2024-03-30T14:01:50.916572+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'18","last_scrub_stamp":"2024-03-30T13:58:54.450973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:58:54.450973+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":364,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.917155+0000","last_change":"2024-03-30T13:58:13.511206+0000","last_active":"2024-03-30T14:01:50.917155+0000","last_peered":"2024-03-30T14:01:50.917155+0000","last_clean":"2024-03-30T14:01:50.917155+0000","last_became_active":"2024-03-30T13:53:35.861370+0000","last_became_peered":"2024-03-30T13:53:35.861370+0000","last_unstale":"2024-03-30T14:01:50.917155+0000","last_undegraded":"2024-03-30T14:01:50.917155+0000","last_fullsized":"2024-03-30T14:01:50.917155+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:13.511094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:13.511094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":722,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.917520+0000","last_change":"2024-03-30T13:58:12.361908+0000","last_active":"2024-03-30T14:01:50.917520+0000","last_peered":"2024-03-30T14:01:50.917520+0000","last_clean":"2024-03-30T14:01:50.917520+0000","last_became_active":"2024-03-30T13:53:35.862056+0000","last_became_peered":"2024-03-30T13:53:35.862056+0000","last_unstale":"2024-03-30T14:01:50.917520+0000","last_undegraded":"2024-03-30T14:01:50.917520+0000","last_fullsized":"2024-03-30T14:01:50.917520+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:12.361782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:12.361782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"82'128","reported_seq":783,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916079+0000","last_change":"2024-03-30T14:01:03.353971+0000","last_active":"2024-03-30T14:01:50.916079+0000","last_peered":"2024-03-30T14:01:50.916079+0000","last_clean":"2024-03-30T14:01:50.916079+0000","last_became_active":"2024-03-30T14:01:02.881640+0000","last_became_peered":"2024-03-30T14:01:02.881640+0000","last_unstale":"2024-03-30T14:01:50.916079+0000","last_undegraded":"2024-03-30T14:01:50.916079+0000","last_fullsized":"2024-03-30T14:01:50.916079+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:01:03.353832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:01:03.353832+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:08.284558+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"400'2914","reported_seq":2963,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:53.755470+0000","last_change":"2024-03-30T14:01:13.911828+0000","last_active":"2024-03-30T14:01:53.755470+0000","last_peered":"2024-03-30T14:01:53.755470+0000","last_clean":"2024-03-30T14:01:53.755470+0000","last_became_active":"2024-03-30T14:01:13.911486+0000","last_became_peered":"2024-03-30T14:01:13.911486+0000","last_unstale":"2024-03-30T14:01:53.755470+0000","last_undegraded":"2024-03-30T14:01:53.755470+0000","last_fullsized":"2024-03-30T14:01:53.755470+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:01:12.895052+0000","last_clean_scrub_stamp":"2024-03-30T14:01:12.895052+0000","objects_scrubbed":0,"log_size":2914,"log_dups_size":0,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:36.148950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190971904,"num_objects":2914,"num_object_clones":0,"num_object_copies":8742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":186496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"376'10","reported_seq":510,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.934560+0000","last_change":"2024-03-30T14:01:08.312215+0000","last_active":"2024-03-30T14:01:50.934560+0000","last_peered":"2024-03-30T14:01:50.934560+0000","last_clean":"2024-03-30T14:01:50.934560+0000","last_became_active":"2024-03-30T13:53:35.862126+0000","last_became_peered":"2024-03-30T13:53:35.862126+0000","last_unstale":"2024-03-30T14:01:50.934560+0000","last_undegraded":"2024-03-30T14:01:50.934560+0000","last_fullsized":"2024-03-30T14:01:50.934560+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'10","last_scrub_stamp":"2024-03-30T14:01:08.312068+0000","last_deep_scrub":"376'10","last_deep_scrub_stamp":"2024-03-30T14:01:08.312068+0000","last_clean_scrub_stamp":"2024-03-30T14:01:08.312068+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:28.360269+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"216'27","reported_seq":176,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.941318+0000","last_change":"2024-03-30T13:58:47.222352+0000","last_active":"2024-03-30T14:01:50.941318+0000","last_peered":"2024-03-30T14:01:50.941318+0000","last_clean":"2024-03-30T14:01:50.941318+0000","last_became_active":"2024-03-30T13:58:47.197778+0000","last_became_peered":"2024-03-30T13:58:47.197778+0000","last_unstale":"2024-03-30T14:01:50.941318+0000","last_undegraded":"2024-03-30T14:01:50.941318+0000","last_fullsized":"2024-03-30T14:01:50.941318+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'27","last_scrub_stamp":"2024-03-30T13:58:08.152048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:58:08.152048+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":379,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916938+0000","last_change":"2024-03-30T13:59:11.278099+0000","last_active":"2024-03-30T14:01:50.916938+0000","last_peered":"2024-03-30T14:01:50.916938+0000","last_clean":"2024-03-30T14:01:50.916938+0000","last_became_active":"2024-03-30T13:53:35.874521+0000","last_became_peered":"2024-03-30T13:53:35.874521+0000","last_unstale":"2024-03-30T14:01:50.916938+0000","last_undegraded":"2024-03-30T14:01:50.916938+0000","last_fullsized":"2024-03-30T14:01:50.916938+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-30T13:59:11.277957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:59:11.277957+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":509,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.942128+0000","last_change":"2024-03-30T14:01:50.565846+0000","last_active":"2024-03-30T14:01:50.942128+0000","last_peered":"2024-03-30T14:01:50.942128+0000","last_clean":"2024-03-30T14:01:50.942128+0000","last_became_active":"2024-03-30T13:58:48.234649+0000","last_became_peered":"2024-03-30T13:58:48.234649+0000","last_unstale":"2024-03-30T14:01:50.942128+0000","last_undegraded":"2024-03-30T14:01:50.942128+0000","last_fullsized":"2024-03-30T14:01:50.942128+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:50.565748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:01:50.565748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:03:17.692340+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"82'128","reported_seq":738,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916746+0000","last_change":"2024-03-30T14:01:31.010756+0000","last_active":"2024-03-30T14:01:50.916746+0000","last_peered":"2024-03-30T14:01:50.916746+0000","last_clean":"2024-03-30T14:01:50.916746+0000","last_became_active":"2024-03-30T14:01:31.010343+0000","last_became_peered":"2024-03-30T14:01:31.010343+0000","last_unstale":"2024-03-30T14:01:50.916746+0000","last_undegraded":"2024-03-30T14:01:50.916746+0000","last_fullsized":"2024-03-30T14:01:50.916746+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:01:02.952369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:01:02.952369+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:15.398438+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"316'15","reported_seq":762,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.918067+0000","last_change":"2024-03-30T14:01:29.008071+0000","last_active":"2024-03-30T14:01:50.918067+0000","last_peered":"2024-03-30T14:01:50.918067+0000","last_clean":"2024-03-30T14:01:50.918067+0000","last_became_active":"2024-03-30T14:01:28.932405+0000","last_became_peered":"2024-03-30T14:01:28.932405+0000","last_unstale":"2024-03-30T14:01:50.918067+0000","last_undegraded":"2024-03-30T14:01:50.918067+0000","last_fullsized":"2024-03-30T14:01:50.918067+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'15","last_scrub_stamp":"2024-03-30T13:58:13.210392+0000","last_deep_scrub":"316'15","last_deep_scrub_stamp":"2024-03-30T13:58:13.210392+0000","last_clean_scrub_stamp":"2024-03-30T13:58:13.210392+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":769,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.933814+0000","last_change":"2024-03-30T14:00:56.272749+0000","last_active":"2024-03-30T14:01:50.933814+0000","last_peered":"2024-03-30T14:01:50.933814+0000","last_clean":"2024-03-30T14:01:50.933814+0000","last_became_active":"2024-03-30T13:58:47.203903+0000","last_became_peered":"2024-03-30T13:58:47.203903+0000","last_unstale":"2024-03-30T14:01:50.933814+0000","last_undegraded":"2024-03-30T14:01:50.933814+0000","last_fullsized":"2024-03-30T14:01:50.933814+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:56.272666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:51:20.175438+0000","last_clean_scrub_stamp":"2024-03-30T14:00:56.272666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:03.626695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":168,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.942420+0000","last_change":"2024-03-30T14:01:46.515617+0000","last_active":"2024-03-30T14:01:50.942420+0000","last_peered":"2024-03-30T14:01:50.942420+0000","last_clean":"2024-03-30T14:01:50.942420+0000","last_became_active":"2024-03-30T13:58:47.193472+0000","last_became_peered":"2024-03-30T13:58:47.193472+0000","last_unstale":"2024-03-30T14:01:50.942420+0000","last_undegraded":"2024-03-30T14:01:50.942420+0000","last_fullsized":"2024-03-30T14:01:50.942420+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:46.515498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:57:09.967517+0000","last_clean_scrub_stamp":"2024-03-30T14:01:46.515498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:03:06.119006+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"72'1","reported_seq":406,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.935239+0000","last_change":"2024-03-30T14:01:40.318398+0000","last_active":"2024-03-30T14:01:50.935239+0000","last_peered":"2024-03-30T14:01:50.935239+0000","last_clean":"2024-03-30T14:01:50.935239+0000","last_became_active":"2024-03-30T13:58:47.198798+0000","last_became_peered":"2024-03-30T13:58:47.198798+0000","last_unstale":"2024-03-30T14:01:50.935239+0000","last_undegraded":"2024-03-30T14:01:50.935239+0000","last_fullsized":"2024-03-30T14:01:50.935239+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T14:01:40.318298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:01:40.318298+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:53.445632+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"82'128","reported_seq":679,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.941091+0000","last_change":"2024-03-30T14:01:03.727604+0000","last_active":"2024-03-30T14:01:50.941091+0000","last_peered":"2024-03-30T14:01:50.941091+0000","last_clean":"2024-03-30T14:01:50.941091+0000","last_became_active":"2024-03-30T14:01:02.778671+0000","last_became_peered":"2024-03-30T14:01:02.778671+0000","last_unstale":"2024-03-30T14:01:50.941091+0000","last_undegraded":"2024-03-30T14:01:50.941091+0000","last_fullsized":"2024-03-30T14:01:50.941091+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:01:03.727524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:01:03.727524+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:21.994417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"82'128","reported_seq":746,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.920787+0000","last_change":"2024-03-30T14:01:03.161717+0000","last_active":"2024-03-30T14:01:50.920787+0000","last_peered":"2024-03-30T14:01:50.920787+0000","last_clean":"2024-03-30T14:01:50.920787+0000","last_became_active":"2024-03-30T14:01:02.682099+0000","last_became_peered":"2024-03-30T14:01:02.682099+0000","last_unstale":"2024-03-30T14:01:50.920787+0000","last_undegraded":"2024-03-30T14:01:50.920787+0000","last_fullsized":"2024-03-30T14:01:50.920787+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:01:03.161599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:01:03.161599+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:11.333281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"218'16","reported_seq":451,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.917362+0000","last_change":"2024-03-30T14:01:29.010714+0000","last_active":"2024-03-30T14:01:50.917362+0000","last_peered":"2024-03-30T14:01:50.917362+0000","last_clean":"2024-03-30T14:01:50.917362+0000","last_became_active":"2024-03-30T14:01:28.926143+0000","last_became_peered":"2024-03-30T14:01:28.926143+0000","last_unstale":"2024-03-30T14:01:50.917362+0000","last_undegraded":"2024-03-30T14:01:50.917362+0000","last_fullsized":"2024-03-30T14:01:50.917362+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'16","last_scrub_stamp":"2024-03-30T14:00:00.547687+0000","last_deep_scrub":"218'16","last_deep_scrub_stamp":"2024-03-30T13:52:59.883027+0000","last_clean_scrub_stamp":"2024-03-30T14:00:00.547687+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":364,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916487+0000","last_change":"2024-03-30T13:58:16.531285+0000","last_active":"2024-03-30T14:01:50.916487+0000","last_peered":"2024-03-30T14:01:50.916487+0000","last_clean":"2024-03-30T14:01:50.916487+0000","last_became_active":"2024-03-30T13:53:35.858261+0000","last_became_peered":"2024-03-30T13:53:35.858261+0000","last_unstale":"2024-03-30T14:01:50.916487+0000","last_undegraded":"2024-03-30T14:01:50.916487+0000","last_fullsized":"2024-03-30T14:01:50.916487+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:16.530762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:16.530762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":725,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.918069+0000","last_change":"2024-03-30T13:58:03.191278+0000","last_active":"2024-03-30T14:01:50.918069+0000","last_peered":"2024-03-30T14:01:50.918069+0000","last_clean":"2024-03-30T14:01:50.918069+0000","last_became_active":"2024-03-30T13:53:35.875783+0000","last_became_peered":"2024-03-30T13:53:35.875783+0000","last_unstale":"2024-03-30T14:01:50.918069+0000","last_undegraded":"2024-03-30T14:01:50.918069+0000","last_fullsized":"2024-03-30T14:01:50.918069+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:03.191052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:58:03.191052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":517,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916992+0000","last_change":"2024-03-30T14:01:28.998706+0000","last_active":"2024-03-30T14:01:50.916992+0000","last_peered":"2024-03-30T14:01:50.916992+0000","last_clean":"2024-03-30T14:01:50.916992+0000","last_became_active":"2024-03-30T14:01:28.998315+0000","last_became_peered":"2024-03-30T14:01:28.998315+0000","last_unstale":"2024-03-30T14:01:50.916992+0000","last_undegraded":"2024-03-30T14:01:50.916992+0000","last_fullsized":"2024-03-30T14:01:50.916992+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:08.050518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:01:08.050518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:24.709848+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"217'15","reported_seq":178,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.942533+0000","last_change":"2024-03-30T14:01:49.545081+0000","last_active":"2024-03-30T14:01:50.942533+0000","last_peered":"2024-03-30T14:01:50.942533+0000","last_clean":"2024-03-30T14:01:50.942533+0000","last_became_active":"2024-03-30T13:58:47.198411+0000","last_became_peered":"2024-03-30T13:58:47.198411+0000","last_unstale":"2024-03-30T14:01:50.942533+0000","last_undegraded":"2024-03-30T14:01:50.942533+0000","last_fullsized":"2024-03-30T14:01:50.942533+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'15","last_scrub_stamp":"2024-03-30T14:01:49.544994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:01:49.544994+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:53.527261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":433,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916637+0000","last_change":"2024-03-30T14:01:09.876748+0000","last_active":"2024-03-30T14:01:50.916637+0000","last_peered":"2024-03-30T14:01:50.916637+0000","last_clean":"2024-03-30T14:01:50.916637+0000","last_became_active":"2024-03-30T14:01:09.876427+0000","last_became_peered":"2024-03-30T14:01:09.876427+0000","last_unstale":"2024-03-30T14:01:50.916637+0000","last_undegraded":"2024-03-30T14:01:50.916637+0000","last_fullsized":"2024-03-30T14:01:50.916637+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:03.861718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:28.734080+0000","last_clean_scrub_stamp":"2024-03-30T13:58:03.861718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"391'13","reported_seq":971,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916683+0000","last_change":"2024-03-30T14:01:28.999207+0000","last_active":"2024-03-30T14:01:50.916683+0000","last_peered":"2024-03-30T14:01:50.916683+0000","last_clean":"2024-03-30T14:01:50.916683+0000","last_became_active":"2024-03-30T14:01:28.998736+0000","last_became_peered":"2024-03-30T14:01:28.998736+0000","last_unstale":"2024-03-30T14:01:50.916683+0000","last_undegraded":"2024-03-30T14:01:50.916683+0000","last_fullsized":"2024-03-30T14:01:50.916683+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'11","last_scrub_stamp":"2024-03-30T13:57:01.991476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:57:01.991476+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":725,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.917105+0000","last_change":"2024-03-30T14:01:28.922980+0000","last_active":"2024-03-30T14:01:50.917105+0000","last_peered":"2024-03-30T14:01:50.917105+0000","last_clean":"2024-03-30T14:01:50.917105+0000","last_became_active":"2024-03-30T14:01:28.922628+0000","last_became_peered":"2024-03-30T14:01:28.922628+0000","last_unstale":"2024-03-30T14:01:50.917105+0000","last_undegraded":"2024-03-30T14:01:50.917105+0000","last_fullsized":"2024-03-30T14:01:50.917105+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:06.727774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:57:06.727774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"82'128","reported_seq":690,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.941077+0000","last_change":"2024-03-30T14:01:02.699046+0000","last_active":"2024-03-30T14:01:50.941077+0000","last_peered":"2024-03-30T14:01:50.941077+0000","last_clean":"2024-03-30T14:01:50.941077+0000","last_became_active":"2024-03-30T14:01:02.698676+0000","last_became_peered":"2024-03-30T14:01:02.698676+0000","last_unstale":"2024-03-30T14:01:50.941077+0000","last_undegraded":"2024-03-30T14:01:50.941077+0000","last_fullsized":"2024-03-30T14:01:50.941077+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:00:58.555385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:00:58.555385+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:03.027305+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"217'20","reported_seq":802,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916578+0000","last_change":"2024-03-30T14:01:38.164695+0000","last_active":"2024-03-30T14:01:50.916578+0000","last_peered":"2024-03-30T14:01:50.916578+0000","last_clean":"2024-03-30T14:01:50.916578+0000","last_became_active":"2024-03-30T14:01:28.998233+0000","last_became_peered":"2024-03-30T14:01:28.998233+0000","last_unstale":"2024-03-30T14:01:50.916578+0000","last_undegraded":"2024-03-30T14:01:50.916578+0000","last_fullsized":"2024-03-30T14:01:50.916578+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'20","last_scrub_stamp":"2024-03-30T14:01:08.434538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:01:08.434538+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:31.397817+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":614,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916686+0000","last_change":"2024-03-30T14:01:28.923004+0000","last_active":"2024-03-30T14:01:50.916686+0000","last_peered":"2024-03-30T14:01:50.916686+0000","last_clean":"2024-03-30T14:01:50.916686+0000","last_became_active":"2024-03-30T14:01:28.921148+0000","last_became_peered":"2024-03-30T14:01:28.921148+0000","last_unstale":"2024-03-30T14:01:50.916686+0000","last_undegraded":"2024-03-30T14:01:50.916686+0000","last_fullsized":"2024-03-30T14:01:50.916686+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:23.112463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:00:23.112463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"326'10","reported_seq":210,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:51.547154+0000","last_change":"2024-03-30T14:01:51.547204+0000","last_active":"2024-03-30T14:01:51.547154+0000","last_peered":"2024-03-30T14:01:51.547154+0000","last_clean":"2024-03-30T14:01:51.547154+0000","last_became_active":"2024-03-30T13:58:47.194143+0000","last_became_peered":"2024-03-30T13:58:47.194143+0000","last_unstale":"2024-03-30T14:01:51.547154+0000","last_undegraded":"2024-03-30T14:01:51.547154+0000","last_fullsized":"2024-03-30T14:01:51.547154+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'10","last_scrub_stamp":"2024-03-30T14:01:51.547045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:01:51.547045+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:56.141114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":429,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.917531+0000","last_change":"2024-03-30T14:01:09.873320+0000","last_active":"2024-03-30T14:01:50.917531+0000","last_peered":"2024-03-30T14:01:50.917531+0000","last_clean":"2024-03-30T14:01:50.917531+0000","last_became_active":"2024-03-30T14:01:09.872913+0000","last_became_peered":"2024-03-30T14:01:09.872913+0000","last_unstale":"2024-03-30T14:01:50.917531+0000","last_undegraded":"2024-03-30T14:01:50.917531+0000","last_fullsized":"2024-03-30T14:01:50.917531+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:59:58.656716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:56.942611+0000","last_clean_scrub_stamp":"2024-03-30T13:59:58.656716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"82'128","reported_seq":511,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.941083+0000","last_change":"2024-03-30T14:01:10.647920+0000","last_active":"2024-03-30T14:01:50.941083+0000","last_peered":"2024-03-30T14:01:50.941083+0000","last_clean":"2024-03-30T14:01:50.941083+0000","last_became_active":"2024-03-30T14:01:09.871424+0000","last_became_peered":"2024-03-30T14:01:09.871424+0000","last_unstale":"2024-03-30T14:01:50.941083+0000","last_undegraded":"2024-03-30T14:01:50.941083+0000","last_fullsized":"2024-03-30T14:01:50.941083+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:01:10.647831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:01:10.647831+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:16.630980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"216'19","reported_seq":786,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916719+0000","last_change":"2024-03-30T14:00:04.206440+0000","last_active":"2024-03-30T14:01:50.916719+0000","last_peered":"2024-03-30T14:01:50.916719+0000","last_clean":"2024-03-30T14:01:50.916719+0000","last_became_active":"2024-03-30T13:59:55.845201+0000","last_became_peered":"2024-03-30T13:59:55.845201+0000","last_unstale":"2024-03-30T14:01:50.916719+0000","last_undegraded":"2024-03-30T14:01:50.916719+0000","last_fullsized":"2024-03-30T14:01:50.916719+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'19","last_scrub_stamp":"2024-03-30T14:00:04.206318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:00:04.206318+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":746,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916691+0000","last_change":"2024-03-30T14:01:28.921534+0000","last_active":"2024-03-30T14:01:50.916691+0000","last_peered":"2024-03-30T14:01:50.916691+0000","last_clean":"2024-03-30T14:01:50.916691+0000","last_became_active":"2024-03-30T14:01:28.921155+0000","last_became_peered":"2024-03-30T14:01:28.921155+0000","last_unstale":"2024-03-30T14:01:50.916691+0000","last_undegraded":"2024-03-30T14:01:50.916691+0000","last_fullsized":"2024-03-30T14:01:50.916691+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:52.481860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:52.481860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":759,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.919990+0000","last_change":"2024-03-30T14:00:13.220615+0000","last_active":"2024-03-30T14:01:50.919990+0000","last_peered":"2024-03-30T14:01:50.919990+0000","last_clean":"2024-03-30T14:01:50.919990+0000","last_became_active":"2024-03-30T13:58:47.185594+0000","last_became_peered":"2024-03-30T13:58:47.185594+0000","last_unstale":"2024-03-30T14:01:50.919990+0000","last_undegraded":"2024-03-30T14:01:50.919990+0000","last_fullsized":"2024-03-30T14:01:50.919990+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:13.220494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:34.815783+0000","last_clean_scrub_stamp":"2024-03-30T14:00:13.220494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":727,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.918566+0000","last_change":"2024-03-30T13:58:47.194818+0000","last_active":"2024-03-30T14:01:50.918566+0000","last_peered":"2024-03-30T14:01:50.918566+0000","last_clean":"2024-03-30T14:01:50.918566+0000","last_became_active":"2024-03-30T13:58:47.194434+0000","last_became_peered":"2024-03-30T13:58:47.194434+0000","last_unstale":"2024-03-30T14:01:50.918566+0000","last_undegraded":"2024-03-30T14:01:50.918566+0000","last_fullsized":"2024-03-30T14:01:50.918566+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:17.058094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:24.598976+0000","last_clean_scrub_stamp":"2024-03-30T13:57:17.058094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"216'15","reported_seq":756,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.920205+0000","last_change":"2024-03-30T14:01:27.975188+0000","last_active":"2024-03-30T14:01:50.920205+0000","last_peered":"2024-03-30T14:01:50.920205+0000","last_clean":"2024-03-30T14:01:50.920205+0000","last_became_active":"2024-03-30T14:01:27.932259+0000","last_became_peered":"2024-03-30T14:01:27.932259+0000","last_unstale":"2024-03-30T14:01:50.920205+0000","last_undegraded":"2024-03-30T14:01:50.920205+0000","last_fullsized":"2024-03-30T14:01:50.920205+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-30T13:49:26.480540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:49:26.480540+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":365,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916571+0000","last_change":"2024-03-30T13:57:19.071948+0000","last_active":"2024-03-30T14:01:50.916571+0000","last_peered":"2024-03-30T14:01:50.916571+0000","last_clean":"2024-03-30T14:01:50.916571+0000","last_became_active":"2024-03-30T13:53:35.859253+0000","last_became_peered":"2024-03-30T13:53:35.859253+0000","last_unstale":"2024-03-30T14:01:50.916571+0000","last_undegraded":"2024-03-30T14:01:50.916571+0000","last_fullsized":"2024-03-30T14:01:50.916571+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:19.071814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:11.701294+0000","last_clean_scrub_stamp":"2024-03-30T13:57:19.071814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":720,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.917334+0000","last_change":"2024-03-30T14:00:01.326908+0000","last_active":"2024-03-30T14:01:50.917334+0000","last_peered":"2024-03-30T14:01:50.917334+0000","last_clean":"2024-03-30T14:01:50.917334+0000","last_became_active":"2024-03-30T13:42:49.325219+0000","last_became_peered":"2024-03-30T13:42:49.325219+0000","last_unstale":"2024-03-30T14:01:50.917334+0000","last_undegraded":"2024-03-30T14:01:50.917334+0000","last_fullsized":"2024-03-30T14:01:50.917334+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:01.326777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:00:01.326777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":631,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.941174+0000","last_change":"2024-03-30T14:01:09.877881+0000","last_active":"2024-03-30T14:01:50.941174+0000","last_peered":"2024-03-30T14:01:50.941174+0000","last_clean":"2024-03-30T14:01:50.941174+0000","last_became_active":"2024-03-30T14:01:09.877500+0000","last_became_peered":"2024-03-30T14:01:09.877500+0000","last_unstale":"2024-03-30T14:01:50.941174+0000","last_undegraded":"2024-03-30T14:01:50.941174+0000","last_fullsized":"2024-03-30T14:01:50.941174+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:02.639129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:28.314391+0000","last_clean_scrub_stamp":"2024-03-30T14:01:02.639129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:18.589747+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"82'128","reported_seq":781,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916567+0000","last_change":"2024-03-30T14:01:31.002849+0000","last_active":"2024-03-30T14:01:50.916567+0000","last_peered":"2024-03-30T14:01:50.916567+0000","last_clean":"2024-03-30T14:01:50.916567+0000","last_became_active":"2024-03-30T14:01:31.002446+0000","last_became_peered":"2024-03-30T14:01:31.002446+0000","last_unstale":"2024-03-30T14:01:50.916567+0000","last_undegraded":"2024-03-30T14:01:50.916567+0000","last_fullsized":"2024-03-30T14:01:50.916567+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:00:58.254858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:00:58.254858+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:15.183660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"217'19","reported_seq":799,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.918855+0000","last_change":"2024-03-30T14:00:24.357354+0000","last_active":"2024-03-30T14:01:50.918855+0000","last_peered":"2024-03-30T14:01:50.918855+0000","last_clean":"2024-03-30T14:01:50.918855+0000","last_became_active":"2024-03-30T13:58:47.193590+0000","last_became_peered":"2024-03-30T13:58:47.193590+0000","last_unstale":"2024-03-30T14:01:50.918855+0000","last_undegraded":"2024-03-30T14:01:50.918855+0000","last_fullsized":"2024-03-30T14:01:50.918855+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'19","last_scrub_stamp":"2024-03-30T14:00:24.357069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:00:24.357069+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":166,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.942181+0000","last_change":"2024-03-30T14:01:08.644461+0000","last_active":"2024-03-30T14:01:50.942181+0000","last_peered":"2024-03-30T14:01:50.942181+0000","last_clean":"2024-03-30T14:01:50.942181+0000","last_became_active":"2024-03-30T13:58:48.187064+0000","last_became_peered":"2024-03-30T13:58:48.187064+0000","last_unstale":"2024-03-30T14:01:50.942181+0000","last_undegraded":"2024-03-30T14:01:50.942181+0000","last_fullsized":"2024-03-30T14:01:50.942181+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:08.644376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:25.494903+0000","last_clean_scrub_stamp":"2024-03-30T14:01:08.644376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:23.766339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":364,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916041+0000","last_change":"2024-03-30T14:00:11.156066+0000","last_active":"2024-03-30T14:01:50.916041+0000","last_peered":"2024-03-30T14:01:50.916041+0000","last_clean":"2024-03-30T14:01:50.916041+0000","last_became_active":"2024-03-30T13:53:35.854002+0000","last_became_peered":"2024-03-30T13:53:35.854002+0000","last_unstale":"2024-03-30T14:01:50.916041+0000","last_undegraded":"2024-03-30T14:01:50.916041+0000","last_fullsized":"2024-03-30T14:01:50.916041+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:11.155966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:42.616155+0000","last_clean_scrub_stamp":"2024-03-30T14:00:11.155966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":716,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.918297+0000","last_change":"2024-03-30T14:00:05.461525+0000","last_active":"2024-03-30T14:01:50.918297+0000","last_peered":"2024-03-30T14:01:50.918297+0000","last_clean":"2024-03-30T14:01:50.918297+0000","last_became_active":"2024-03-30T13:42:51.374304+0000","last_became_peered":"2024-03-30T13:42:51.374304+0000","last_unstale":"2024-03-30T14:01:50.918297+0000","last_undegraded":"2024-03-30T14:01:50.918297+0000","last_fullsized":"2024-03-30T14:01:50.918297+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:05.461415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:00:05.461415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"216'8","reported_seq":736,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916277+0000","last_change":"2024-03-30T13:57:07.952284+0000","last_active":"2024-03-30T14:01:50.916277+0000","last_peered":"2024-03-30T14:01:50.916277+0000","last_clean":"2024-03-30T14:01:50.916277+0000","last_became_active":"2024-03-30T13:42:47.321428+0000","last_became_peered":"2024-03-30T13:42:47.321428+0000","last_unstale":"2024-03-30T14:01:50.916277+0000","last_undegraded":"2024-03-30T14:01:50.916277+0000","last_fullsized":"2024-03-30T14:01:50.916277+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'8","last_scrub_stamp":"2024-03-30T13:57:07.952145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:07.952145+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":651,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.942827+0000","last_change":"2024-03-30T14:01:09.871993+0000","last_active":"2024-03-30T14:01:50.942827+0000","last_peered":"2024-03-30T14:01:50.942827+0000","last_clean":"2024-03-30T14:01:50.942827+0000","last_became_active":"2024-03-30T14:01:09.871701+0000","last_became_peered":"2024-03-30T14:01:09.871701+0000","last_unstale":"2024-03-30T14:01:50.942827+0000","last_undegraded":"2024-03-30T14:01:50.942827+0000","last_fullsized":"2024-03-30T14:01:50.942827+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:16.924797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:16.924797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":763,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.935057+0000","last_change":"2024-03-30T14:01:48.346129+0000","last_active":"2024-03-30T14:01:50.935057+0000","last_peered":"2024-03-30T14:01:50.935057+0000","last_clean":"2024-03-30T14:01:50.935057+0000","last_became_active":"2024-03-30T13:53:35.862467+0000","last_became_peered":"2024-03-30T13:53:35.862467+0000","last_unstale":"2024-03-30T14:01:50.935057+0000","last_undegraded":"2024-03-30T14:01:50.935057+0000","last_fullsized":"2024-03-30T14:01:50.935057+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:48.346052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:54.893653+0000","last_clean_scrub_stamp":"2024-03-30T14:01:48.346052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:53.123366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":603,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916194+0000","last_change":"2024-03-30T14:01:28.923734+0000","last_active":"2024-03-30T14:01:50.916194+0000","last_peered":"2024-03-30T14:01:50.916194+0000","last_clean":"2024-03-30T14:01:50.916194+0000","last_became_active":"2024-03-30T14:01:28.923385+0000","last_became_peered":"2024-03-30T14:01:28.923385+0000","last_unstale":"2024-03-30T14:01:50.916194+0000","last_undegraded":"2024-03-30T14:01:50.916194+0000","last_fullsized":"2024-03-30T14:01:50.916194+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:14.193663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:00:14.193663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"217'19","reported_seq":813,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.935160+0000","last_change":"2024-03-30T14:01:00.273249+0000","last_active":"2024-03-30T14:01:50.935160+0000","last_peered":"2024-03-30T14:01:50.935160+0000","last_clean":"2024-03-30T14:01:50.935160+0000","last_became_active":"2024-03-30T13:58:47.204696+0000","last_became_peered":"2024-03-30T13:58:47.204696+0000","last_unstale":"2024-03-30T14:01:50.935160+0000","last_undegraded":"2024-03-30T14:01:50.935160+0000","last_fullsized":"2024-03-30T14:01:50.935160+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'19","last_scrub_stamp":"2024-03-30T14:01:00.273150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:01:00.273150+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:17.641495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":727,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916030+0000","last_change":"2024-03-30T13:58:14.227217+0000","last_active":"2024-03-30T14:01:50.916030+0000","last_peered":"2024-03-30T14:01:50.916030+0000","last_clean":"2024-03-30T14:01:50.916030+0000","last_became_active":"2024-03-30T13:42:45.310165+0000","last_became_peered":"2024-03-30T13:42:45.310165+0000","last_unstale":"2024-03-30T14:01:50.916030+0000","last_undegraded":"2024-03-30T14:01:50.916030+0000","last_fullsized":"2024-03-30T14:01:50.916030+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:14.224328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:58:14.224328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":425,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.920391+0000","last_change":"2024-03-30T14:01:28.936640+0000","last_active":"2024-03-30T14:01:50.920391+0000","last_peered":"2024-03-30T14:01:50.920391+0000","last_clean":"2024-03-30T14:01:50.920391+0000","last_became_active":"2024-03-30T14:01:28.934796+0000","last_became_peered":"2024-03-30T14:01:28.934796+0000","last_unstale":"2024-03-30T14:01:50.920391+0000","last_undegraded":"2024-03-30T14:01:50.920391+0000","last_fullsized":"2024-03-30T14:01:50.920391+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:09.655585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:00:09.655585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":748,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.919961+0000","last_change":"2024-03-30T14:00:17.296945+0000","last_active":"2024-03-30T14:01:50.919961+0000","last_peered":"2024-03-30T14:01:50.919961+0000","last_clean":"2024-03-30T14:01:50.919961+0000","last_became_active":"2024-03-30T13:58:48.180840+0000","last_became_peered":"2024-03-30T13:58:48.180840+0000","last_unstale":"2024-03-30T14:01:50.919961+0000","last_undegraded":"2024-03-30T14:01:50.919961+0000","last_fullsized":"2024-03-30T14:01:50.919961+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:17.296689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:00:17.296689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"217'25","reported_seq":441,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.917209+0000","last_change":"2024-03-30T14:01:09.877649+0000","last_active":"2024-03-30T14:01:50.917209+0000","last_peered":"2024-03-30T14:01:50.917209+0000","last_clean":"2024-03-30T14:01:50.917209+0000","last_became_active":"2024-03-30T14:01:09.877240+0000","last_became_peered":"2024-03-30T14:01:09.877240+0000","last_unstale":"2024-03-30T14:01:50.917209+0000","last_undegraded":"2024-03-30T14:01:50.917209+0000","last_fullsized":"2024-03-30T14:01:50.917209+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'25","last_scrub_stamp":"2024-03-30T13:59:59.559555+0000","last_deep_scrub":"153'15","last_deep_scrub_stamp":"2024-03-30T13:51:22.252831+0000","last_clean_scrub_stamp":"2024-03-30T13:59:59.559555+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"65'1","reported_seq":751,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.919140+0000","last_change":"2024-03-30T14:00:05.061932+0000","last_active":"2024-03-30T14:01:50.919140+0000","last_peered":"2024-03-30T14:01:50.919140+0000","last_clean":"2024-03-30T14:01:50.919140+0000","last_became_active":"2024-03-30T13:42:45.307832+0000","last_became_peered":"2024-03-30T13:42:45.307832+0000","last_unstale":"2024-03-30T14:01:50.919140+0000","last_undegraded":"2024-03-30T14:01:50.919140+0000","last_fullsized":"2024-03-30T14:01:50.919140+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T14:00:05.061791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:00:05.061791+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"391'12","reported_seq":1007,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.935053+0000","last_change":"2024-03-30T14:01:45.314496+0000","last_active":"2024-03-30T14:01:50.935053+0000","last_peered":"2024-03-30T14:01:50.935053+0000","last_clean":"2024-03-30T14:01:50.935053+0000","last_became_active":"2024-03-30T14:01:28.920262+0000","last_became_peered":"2024-03-30T14:01:28.920262+0000","last_unstale":"2024-03-30T14:01:50.935053+0000","last_undegraded":"2024-03-30T14:01:50.935053+0000","last_fullsized":"2024-03-30T14:01:50.935053+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'12","last_scrub_stamp":"2024-03-30T14:01:45.314385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:01:45.314385+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:59.998600+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":175,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.941898+0000","last_change":"2024-03-30T14:01:28.999486+0000","last_active":"2024-03-30T14:01:50.941898+0000","last_peered":"2024-03-30T14:01:50.941898+0000","last_clean":"2024-03-30T14:01:50.941898+0000","last_became_active":"2024-03-30T14:01:28.999125+0000","last_became_peered":"2024-03-30T14:01:28.999125+0000","last_unstale":"2024-03-30T14:01:50.941898+0000","last_undegraded":"2024-03-30T14:01:50.941898+0000","last_fullsized":"2024-03-30T14:01:50.941898+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T14:00:06.703980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:00:06.703980+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"217'23","reported_seq":768,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916671+0000","last_change":"2024-03-30T14:00:22.339054+0000","last_active":"2024-03-30T14:01:50.916671+0000","last_peered":"2024-03-30T14:01:50.916671+0000","last_clean":"2024-03-30T14:01:50.916671+0000","last_became_active":"2024-03-30T13:42:47.317881+0000","last_became_peered":"2024-03-30T13:42:47.317881+0000","last_unstale":"2024-03-30T14:01:50.916671+0000","last_undegraded":"2024-03-30T14:01:50.916671+0000","last_fullsized":"2024-03-30T14:01:50.916671+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'23","last_scrub_stamp":"2024-03-30T14:00:22.338785+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-03-30T13:43:52.044427+0000","last_clean_scrub_stamp":"2024-03-30T14:00:22.338785+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":720,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916439+0000","last_change":"2024-03-30T14:01:27.932274+0000","last_active":"2024-03-30T14:01:50.916439+0000","last_peered":"2024-03-30T14:01:50.916439+0000","last_clean":"2024-03-30T14:01:50.916439+0000","last_became_active":"2024-03-30T14:01:27.931532+0000","last_became_peered":"2024-03-30T14:01:27.931532+0000","last_unstale":"2024-03-30T14:01:50.916439+0000","last_undegraded":"2024-03-30T14:01:50.916439+0000","last_fullsized":"2024-03-30T14:01:50.916439+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:50:07.261595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T13:50:07.261595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":712,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916908+0000","last_change":"2024-03-30T14:01:27.932773+0000","last_active":"2024-03-30T14:01:50.916908+0000","last_peered":"2024-03-30T14:01:50.916908+0000","last_clean":"2024-03-30T14:01:50.916908+0000","last_became_active":"2024-03-30T14:01:27.932432+0000","last_became_peered":"2024-03-30T14:01:27.932432+0000","last_unstale":"2024-03-30T14:01:50.916908+0000","last_undegraded":"2024-03-30T14:01:50.916908+0000","last_fullsized":"2024-03-30T14:01:50.916908+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:48:25.104610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T13:48:25.104610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":729,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.917858+0000","last_change":"2024-03-30T14:00:09.659366+0000","last_active":"2024-03-30T14:01:50.917858+0000","last_peered":"2024-03-30T14:01:50.917858+0000","last_clean":"2024-03-30T14:01:50.917858+0000","last_became_active":"2024-03-30T13:58:47.194643+0000","last_became_peered":"2024-03-30T13:58:47.194643+0000","last_unstale":"2024-03-30T14:01:50.917858+0000","last_undegraded":"2024-03-30T14:01:50.917858+0000","last_fullsized":"2024-03-30T14:01:50.917858+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:09.659259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:00:09.659259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":220,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"216'27","reported_seq":581,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:54.498730+0000","last_change":"2024-03-30T14:01:54.498799+0000","last_active":"2024-03-30T14:01:54.498730+0000","last_peered":"2024-03-30T14:01:54.498730+0000","last_clean":"2024-03-30T14:01:54.498730+0000","last_became_active":"2024-03-30T14:01:27.965914+0000","last_became_peered":"2024-03-30T14:01:27.965914+0000","last_unstale":"2024-03-30T14:01:54.498730+0000","last_undegraded":"2024-03-30T14:01:54.498730+0000","last_fullsized":"2024-03-30T14:01:54.498730+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'27","last_scrub_stamp":"2024-03-30T14:01:54.498644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:01:54.498644+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:03:00.997976+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":433,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916565+0000","last_change":"2024-03-30T14:01:09.876748+0000","last_active":"2024-03-30T14:01:50.916565+0000","last_peered":"2024-03-30T14:01:50.916565+0000","last_clean":"2024-03-30T14:01:50.916565+0000","last_became_active":"2024-03-30T14:01:09.876437+0000","last_became_peered":"2024-03-30T14:01:09.876437+0000","last_unstale":"2024-03-30T14:01:50.916565+0000","last_undegraded":"2024-03-30T14:01:50.916565+0000","last_fullsized":"2024-03-30T14:01:50.916565+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:57:56.945861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:43.088964+0000","last_clean_scrub_stamp":"2024-03-30T13:57:56.945861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":750,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.920187+0000","last_change":"2024-03-30T14:00:59.075755+0000","last_active":"2024-03-30T14:01:50.920187+0000","last_peered":"2024-03-30T14:01:50.920187+0000","last_clean":"2024-03-30T14:01:50.920187+0000","last_became_active":"2024-03-30T13:42:49.322665+0000","last_became_peered":"2024-03-30T13:42:49.322665+0000","last_unstale":"2024-03-30T14:01:50.920187+0000","last_undegraded":"2024-03-30T14:01:50.920187+0000","last_fullsized":"2024-03-30T14:01:50.920187+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:59.075669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:30.465974+0000","last_clean_scrub_stamp":"2024-03-30T14:00:59.075669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:24.568429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":737,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.917424+0000","last_change":"2024-03-30T13:58:47.199570+0000","last_active":"2024-03-30T14:01:50.917424+0000","last_peered":"2024-03-30T14:01:50.917424+0000","last_clean":"2024-03-30T14:01:50.917424+0000","last_became_active":"2024-03-30T13:58:47.199001+0000","last_became_peered":"2024-03-30T13:58:47.199001+0000","last_unstale":"2024-03-30T14:01:50.917424+0000","last_undegraded":"2024-03-30T14:01:50.917424+0000","last_fullsized":"2024-03-30T14:01:50.917424+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:21.602164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:21.602164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"216'9","reported_seq":754,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916604+0000","last_change":"2024-03-30T13:59:13.285172+0000","last_active":"2024-03-30T14:01:50.916604+0000","last_peered":"2024-03-30T14:01:50.916604+0000","last_clean":"2024-03-30T14:01:50.916604+0000","last_became_active":"2024-03-30T13:42:47.317352+0000","last_became_peered":"2024-03-30T13:42:47.317352+0000","last_unstale":"2024-03-30T14:01:50.916604+0000","last_undegraded":"2024-03-30T14:01:50.916604+0000","last_fullsized":"2024-03-30T14:01:50.916604+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'9","last_scrub_stamp":"2024-03-30T13:59:13.285052+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-30T13:43:50.960980+0000","last_clean_scrub_stamp":"2024-03-30T13:59:13.285052+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"186'2","reported_seq":172,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.942018+0000","last_change":"2024-03-30T14:00:56.698912+0000","last_active":"2024-03-30T14:01:50.942018+0000","last_peered":"2024-03-30T14:01:50.942018+0000","last_clean":"2024-03-30T14:01:50.942018+0000","last_became_active":"2024-03-30T13:58:47.193793+0000","last_became_peered":"2024-03-30T13:58:47.193793+0000","last_unstale":"2024-03-30T14:01:50.942018+0000","last_undegraded":"2024-03-30T14:01:50.942018+0000","last_fullsized":"2024-03-30T14:01:50.942018+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-30T14:00:56.698804+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-30T13:51:15.932076+0000","last_clean_scrub_stamp":"2024-03-30T14:00:56.698804+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:17.834683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":364,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916579+0000","last_change":"2024-03-30T13:59:16.269252+0000","last_active":"2024-03-30T14:01:50.916579+0000","last_peered":"2024-03-30T14:01:50.916579+0000","last_clean":"2024-03-30T14:01:50.916579+0000","last_became_active":"2024-03-30T13:53:35.861405+0000","last_became_peered":"2024-03-30T13:53:35.861405+0000","last_unstale":"2024-03-30T14:01:50.916579+0000","last_undegraded":"2024-03-30T14:01:50.916579+0000","last_fullsized":"2024-03-30T14:01:50.916579+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:59:16.269136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:41.633284+0000","last_clean_scrub_stamp":"2024-03-30T13:59:16.269136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":517,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.941452+0000","last_change":"2024-03-30T14:01:02.837268+0000","last_active":"2024-03-30T14:01:50.941452+0000","last_peered":"2024-03-30T14:01:50.941452+0000","last_clean":"2024-03-30T14:01:50.941452+0000","last_became_active":"2024-03-30T13:58:48.202430+0000","last_became_peered":"2024-03-30T13:58:48.202430+0000","last_unstale":"2024-03-30T14:01:50.941452+0000","last_undegraded":"2024-03-30T14:01:50.941452+0000","last_fullsized":"2024-03-30T14:01:50.941452+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:02.837098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:01:02.837098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:22.790046+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"216'15","reported_seq":756,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916398+0000","last_change":"2024-03-30T13:57:19.638146+0000","last_active":"2024-03-30T14:01:50.916398+0000","last_peered":"2024-03-30T14:01:50.916398+0000","last_clean":"2024-03-30T14:01:50.916398+0000","last_became_active":"2024-03-30T13:42:47.310289+0000","last_became_peered":"2024-03-30T13:42:47.310289+0000","last_unstale":"2024-03-30T14:01:50.916398+0000","last_undegraded":"2024-03-30T14:01:50.916398+0000","last_fullsized":"2024-03-30T14:01:50.916398+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'15","last_scrub_stamp":"2024-03-30T13:57:19.638024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:57:19.638024+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":168,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.943058+0000","last_change":"2024-03-30T13:59:58.707437+0000","last_active":"2024-03-30T14:01:50.943058+0000","last_peered":"2024-03-30T14:01:50.943058+0000","last_clean":"2024-03-30T14:01:50.943058+0000","last_became_active":"2024-03-30T13:58:47.199130+0000","last_became_peered":"2024-03-30T13:58:47.199130+0000","last_unstale":"2024-03-30T14:01:50.943058+0000","last_undegraded":"2024-03-30T14:01:50.943058+0000","last_fullsized":"2024-03-30T14:01:50.943058+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:59:58.707314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:37.513596+0000","last_clean_scrub_stamp":"2024-03-30T13:59:58.707314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":168,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:52.594322+0000","last_change":"2024-03-30T14:01:52.594362+0000","last_active":"2024-03-30T14:01:52.594322+0000","last_peered":"2024-03-30T14:01:52.594322+0000","last_clean":"2024-03-30T14:01:52.594322+0000","last_became_active":"2024-03-30T13:58:47.192207+0000","last_became_peered":"2024-03-30T13:58:47.192207+0000","last_unstale":"2024-03-30T14:01:52.594322+0000","last_undegraded":"2024-03-30T14:01:52.594322+0000","last_fullsized":"2024-03-30T14:01:52.594322+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:52.593958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:20.069864+0000","last_clean_scrub_stamp":"2024-03-30T14:01:52.593958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:03:21.838569+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":425,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.919260+0000","last_change":"2024-03-30T14:01:28.929246+0000","last_active":"2024-03-30T14:01:50.919260+0000","last_peered":"2024-03-30T14:01:50.919260+0000","last_clean":"2024-03-30T14:01:50.919260+0000","last_became_active":"2024-03-30T14:01:28.928931+0000","last_became_peered":"2024-03-30T14:01:28.928931+0000","last_unstale":"2024-03-30T14:01:50.919260+0000","last_undegraded":"2024-03-30T14:01:50.919260+0000","last_fullsized":"2024-03-30T14:01:50.919260+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:10.911245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:27.061281+0000","last_clean_scrub_stamp":"2024-03-30T13:58:10.911245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"217'34","reported_seq":781,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:52.378171+0000","last_change":"2024-03-30T14:01:52.378252+0000","last_active":"2024-03-30T14:01:52.378171+0000","last_peered":"2024-03-30T14:01:52.378171+0000","last_clean":"2024-03-30T14:01:52.378171+0000","last_became_active":"2024-03-30T14:01:27.969354+0000","last_became_peered":"2024-03-30T14:01:27.969354+0000","last_unstale":"2024-03-30T14:01:52.378171+0000","last_undegraded":"2024-03-30T14:01:52.378171+0000","last_fullsized":"2024-03-30T14:01:52.378171+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'34","last_scrub_stamp":"2024-03-30T14:01:52.377847+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:29.151791+0000","last_clean_scrub_stamp":"2024-03-30T14:01:52.377847+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:03:01.206456+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":722,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.917114+0000","last_change":"2024-03-30T14:01:27.933691+0000","last_active":"2024-03-30T14:01:50.917114+0000","last_peered":"2024-03-30T14:01:50.917114+0000","last_clean":"2024-03-30T14:01:50.917114+0000","last_became_active":"2024-03-30T14:01:27.933287+0000","last_became_peered":"2024-03-30T14:01:27.933287+0000","last_unstale":"2024-03-30T14:01:50.917114+0000","last_undegraded":"2024-03-30T14:01:50.917114+0000","last_fullsized":"2024-03-30T14:01:50.917114+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:49:20.499768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:20.499768+0000","last_clean_scrub_stamp":"2024-03-30T13:49:20.499768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":727,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916149+0000","last_change":"2024-03-30T14:01:05.016834+0000","last_active":"2024-03-30T14:01:50.916149+0000","last_peered":"2024-03-30T14:01:50.916149+0000","last_clean":"2024-03-30T14:01:50.916149+0000","last_became_active":"2024-03-30T13:42:49.327124+0000","last_became_peered":"2024-03-30T13:42:49.327124+0000","last_unstale":"2024-03-30T14:01:50.916149+0000","last_undegraded":"2024-03-30T14:01:50.916149+0000","last_fullsized":"2024-03-30T14:01:50.916149+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:01:05.016749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:01:05.016749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:02:08.164217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":721,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.917427+0000","last_change":"2024-03-30T13:58:08.388962+0000","last_active":"2024-03-30T14:01:50.917427+0000","last_peered":"2024-03-30T14:01:50.917427+0000","last_clean":"2024-03-30T14:01:50.917427+0000","last_became_active":"2024-03-30T13:53:35.862085+0000","last_became_peered":"2024-03-30T13:53:35.862085+0000","last_unstale":"2024-03-30T14:01:50.917427+0000","last_undegraded":"2024-03-30T14:01:50.917427+0000","last_fullsized":"2024-03-30T14:01:50.917427+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:58:08.388764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T13:58:08.388764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"216'15","reported_seq":756,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916299+0000","last_change":"2024-03-30T13:59:17.249544+0000","last_active":"2024-03-30T14:01:50.916299+0000","last_peered":"2024-03-30T14:01:50.916299+0000","last_clean":"2024-03-30T14:01:50.916299+0000","last_became_active":"2024-03-30T13:42:47.319638+0000","last_became_peered":"2024-03-30T13:42:47.319638+0000","last_unstale":"2024-03-30T14:01:50.916299+0000","last_undegraded":"2024-03-30T14:01:50.916299+0000","last_fullsized":"2024-03-30T14:01:50.916299+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'15","last_scrub_stamp":"2024-03-30T13:59:17.249429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T13:59:17.249429+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"65'1","reported_seq":68,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.942993+0000","last_change":"2024-03-30T14:01:28.922334+0000","last_active":"2024-03-30T14:01:50.942993+0000","last_peered":"2024-03-30T14:01:50.942993+0000","last_clean":"2024-03-30T14:01:50.942993+0000","last_became_active":"2024-03-30T14:01:28.921979+0000","last_became_peered":"2024-03-30T14:01:28.921979+0000","last_unstale":"2024-03-30T14:01:50.942993+0000","last_undegraded":"2024-03-30T14:01:50.942993+0000","last_fullsized":"2024-03-30T14:01:50.942993+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T14:00:02.574261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:00:02.574261+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":762,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.919168+0000","last_change":"2024-03-30T14:01:28.922221+0000","last_active":"2024-03-30T14:01:50.919168+0000","last_peered":"2024-03-30T14:01:50.919168+0000","last_clean":"2024-03-30T14:01:50.919168+0000","last_became_active":"2024-03-30T14:01:28.921911+0000","last_became_peered":"2024-03-30T14:01:28.921911+0000","last_unstale":"2024-03-30T14:01:50.919168+0000","last_undegraded":"2024-03-30T14:01:50.919168+0000","last_fullsized":"2024-03-30T14:01:50.919168+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:59:14.336241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:17.548567+0000","last_clean_scrub_stamp":"2024-03-30T13:59:14.336241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":741,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-30T14:01:50.916220+0000","last_change":"2024-03-30T14:00:18.152378+0000","last_active":"2024-03-30T14:01:50.916220+0000","last_peered":"2024-03-30T14:01:50.916220+0000","last_clean":"2024-03-30T14:01:50.916220+0000","last_became_active":"2024-03-30T13:58:47.182681+0000","last_became_peered":"2024-03-30T13:58:47.182681+0000","last_unstale":"2024-03-30T14:01:50.916220+0000","last_undegraded":"2024-03-30T14:01:50.916220+0000","last_fullsized":"2024-03-30T14:01:50.916220+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:00:18.152265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:00:18.152265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3060662272,"num_objects":46702,"num_object_clones":0,"num_object_copies":140106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46702,"num_write_kb":2988928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7079,"num_bytes_recovered":463929344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9321775104,"data_stored":9321775104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46702,"ondisk_log_size":46702,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":505,"num_read_kb":500,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1122,"num_read_kb":879,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":3014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":744,"ondisk_log_size":744,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":249108103403,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397743,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2414444,"kb_used_data":1643300,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":91330708,"statfs":{"total":95995035648,"available":93522644992,"internally_reserved":0,"allocated":1682739200,"data_stored":1680914563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":789641670},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575113,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2621188,"kb_used_data":1769532,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":91123964,"statfs":{"total":95995035648,"available":93310939136,"internally_reserved":0,"allocated":1812000768,"data_stored":1810088721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872085957},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738638,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2218052,"kb_used_data":1352508,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":91527100,"statfs":{"total":95995035648,"available":93723750400,"internally_reserved":0,"allocated":1384968192,"data_stored":1383120679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":388,"seq":1666447310855,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1242156,"kb_used_data":1161508,"kb_used_omap":168,"kb_used_meta":80471,"kb_avail":92502996,"statfs":{"total":95995035648,"available":94723067904,"internally_reserved":0,"allocated":1189384192,"data_stored":1187561076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":172848,"internal_metadata":82402512},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182655,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2363796,"kb_used_data":1452428,"kb_used_omap":1,"kb_used_meta":911358,"kb_avail":91381356,"statfs":{"total":95995035648,"available":93574508544,"internally_reserved":0,"allocated":1487286272,"data_stored":1485422746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":933231047},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953721,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1492620,"kb_used_data":731204,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92252532,"statfs":{"total":95995035648,"available":94466592768,"internally_reserved":0,"allocated":748752896,"data_stored":746896494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":779680197},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692084,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1618432,"kb_used_data":923640,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":92126720,"statfs":{"total":95995035648,"available":94337761280,"internally_reserved":0,"allocated":945807360,"data_stored":943964496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":2888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1402142720,"data_stored":1402142720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1809842176,"data_stored":1809842176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1203306496,"data_stored":1203306496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1492320256,"data_stored":1492320256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":757530624,"data_stored":757530624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":945946624,"data_stored":945946624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1710686208,"data_stored":1710686208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T14:02:01.509 INFO:tasks.thrashosds.thrasher:Setting 1.2 to [2, 4, 0, 1, 7, 6] 2024-03-30T14:02:01.509 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.2', '2', '4', '0', '1', '7', '6'] 2024-03-30T14:02:01.509 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pg-upmap-items 1.2 2 4 0 1 7 6 2024-03-30T14:02:01.510 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:02:01.518 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:01.608 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:01.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:01.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:01.640 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:01 smithi043 ceph-mon[29963]: 6.0 scrub starts 2024-03-30T14:02:01.640 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:01 smithi043 ceph-mon[29963]: 6.0 scrub ok 2024-03-30T14:02:01.640 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:01 smithi043 ceph-mon[29963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:02:01.640 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:01 smithi043 ceph-mon[29963]: Cluster is now healthy 2024-03-30T14:02:01.640 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:01 smithi043 ceph-mon[29963]: 4.d scrub starts 2024-03-30T14:02:01.640 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:01 smithi043 ceph-mon[29963]: 4.d scrub ok 2024-03-30T14:02:01.640 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:01 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:02:01.640 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:01 smithi043 ceph-mon[29963]: osdmap e401: 8 total, 8 up, 7 in 2024-03-30T14:02:01.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:01 smithi043 ceph-mon[42099]: 6.0 scrub starts 2024-03-30T14:02:01.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:01 smithi043 ceph-mon[42099]: 6.0 scrub ok 2024-03-30T14:02:01.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:01 smithi043 ceph-mon[42099]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:02:01.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:01 smithi043 ceph-mon[42099]: Cluster is now healthy 2024-03-30T14:02:01.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:01 smithi043 ceph-mon[42099]: 4.d scrub starts 2024-03-30T14:02:01.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:01 smithi043 ceph-mon[42099]: 4.d scrub ok 2024-03-30T14:02:01.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:01 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:02:01.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:01 smithi043 ceph-mon[42099]: osdmap e401: 8 total, 8 up, 7 in 2024-03-30T14:02:01.710 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:01.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:01 smithi113 ceph-mon[34341]: 6.0 scrub starts 2024-03-30T14:02:01.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:01 smithi113 ceph-mon[34341]: 6.0 scrub ok 2024-03-30T14:02:01.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:01 smithi113 ceph-mon[34341]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:02:01.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:01 smithi113 ceph-mon[34341]: Cluster is now healthy 2024-03-30T14:02:01.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:01 smithi113 ceph-mon[34341]: 4.d scrub starts 2024-03-30T14:02:01.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:01 smithi113 ceph-mon[34341]: 4.d scrub ok 2024-03-30T14:02:01.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:01 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:02:01.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:01 smithi113 ceph-mon[34341]: osdmap e401: 8 total, 8 up, 7 in 2024-03-30T14:02:01.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:01.766 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:01.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:01.867 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:01.922 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:01.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:01.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:02.026 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:02.055 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:02.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:02.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:02.062 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 42 16 57085 57069 84.9083 79.4375 0.0151888 0.0117714 2024-03-30T14:02:02.157 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:02.184 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:02.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:02.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:02.204 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T14:02:02.205 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:02:02.286 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:02.337 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:02.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:02.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:02.439 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:02.480 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:02.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:02.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:02.583 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:02.638 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:02.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:02.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:02 smithi043 ceph-mon[29963]: 5.14 scrub starts 2024-03-30T14:02:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:02 smithi043 ceph-mon[29963]: 5.14 scrub ok 2024-03-30T14:02:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:02 smithi043 ceph-mon[29963]: 6.1 scrub starts 2024-03-30T14:02:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:02 smithi043 ceph-mon[29963]: 6.1 scrub ok 2024-03-30T14:02:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:02 smithi043 ceph-mon[29963]: pgmap v880: 168 pgs: 168 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-30T14:02:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:02 smithi043 ceph-mon[42099]: 5.14 scrub starts 2024-03-30T14:02:02.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:02 smithi043 ceph-mon[42099]: 5.14 scrub ok 2024-03-30T14:02:02.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:02 smithi043 ceph-mon[42099]: 6.1 scrub starts 2024-03-30T14:02:02.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:02 smithi043 ceph-mon[42099]: 6.1 scrub ok 2024-03-30T14:02:02.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:02 smithi043 ceph-mon[42099]: pgmap v880: 168 pgs: 168 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-30T14:02:02.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:02 smithi113 ceph-mon[34341]: 5.14 scrub starts 2024-03-30T14:02:02.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:02 smithi113 ceph-mon[34341]: 5.14 scrub ok 2024-03-30T14:02:02.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:02 smithi113 ceph-mon[34341]: 6.1 scrub starts 2024-03-30T14:02:02.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:02 smithi113 ceph-mon[34341]: 6.1 scrub ok 2024-03-30T14:02:02.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:02 smithi113 ceph-mon[34341]: pgmap v880: 168 pgs: 168 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-30T14:02:02.739 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:02.779 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:02.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:02.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:02.881 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:02.910 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:02.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:02.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:03.012 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:03.039 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:03.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:03.061 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 43 16 58363 58347 84.7911 79.875 0.00922913 0.0117885 2024-03-30T14:02:03.141 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:03.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:03.183 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:03.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:03.284 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:03.320 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:03.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:03.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:03.381 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T14:02:03.422 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:03.450 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:03.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:03.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:03.507 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-03-30T14:02:03.551 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:03.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:03.608 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:03.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:03.647 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:03 smithi043 ceph-mon[42099]: 4.6 scrub starts 2024-03-30T14:02:03.647 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:03 smithi043 ceph-mon[42099]: 4.6 scrub ok 2024-03-30T14:02:03.647 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:03 smithi043 ceph-mon[42099]: osdmap e402: 8 total, 8 up, 7 in 2024-03-30T14:02:03.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:03 smithi043 ceph-mon[29963]: 4.6 scrub starts 2024-03-30T14:02:03.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:03 smithi043 ceph-mon[29963]: 4.6 scrub ok 2024-03-30T14:02:03.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:03 smithi043 ceph-mon[29963]: osdmap e402: 8 total, 8 up, 7 in 2024-03-30T14:02:03.709 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:03.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:03 smithi113 ceph-mon[34341]: 4.6 scrub starts 2024-03-30T14:02:03.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:03 smithi113 ceph-mon[34341]: 4.6 scrub ok 2024-03-30T14:02:03.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:03 smithi113 ceph-mon[34341]: osdmap e402: 8 total, 8 up, 7 in 2024-03-30T14:02:03.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:03.739 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:03.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:03.840 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:03.867 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:03.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:03.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:03.969 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:03.997 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:03.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:03.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:04.037 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:02:04.063 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 44 16 59552 59536 84.5528 74.3125 0.0184806 0.0118202 2024-03-30T14:02:04.099 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:04.130 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:04.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:04.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:04.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:04.281 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:04.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:04.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:04.383 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:04.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:04.423 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:04.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:04.464 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:04 smithi043 ceph-mon[42099]: 4.10 scrub starts 2024-03-30T14:02:04.464 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:04 smithi043 ceph-mon[42099]: 4.10 scrub ok 2024-03-30T14:02:04.464 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:04 smithi043 ceph-mon[42099]: 4.12 scrub starts 2024-03-30T14:02:04.464 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:04 smithi043 ceph-mon[42099]: 4.12 scrub ok 2024-03-30T14:02:04.464 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:04 smithi043 ceph-mon[42099]: pgmap v882: 168 pgs: 168 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-30T14:02:04.464 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:04 smithi043 ceph-mon[29963]: 4.10 scrub starts 2024-03-30T14:02:04.464 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:04 smithi043 ceph-mon[29963]: 4.10 scrub ok 2024-03-30T14:02:04.464 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:04 smithi043 ceph-mon[29963]: 4.12 scrub starts 2024-03-30T14:02:04.465 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:04 smithi043 ceph-mon[29963]: 4.12 scrub ok 2024-03-30T14:02:04.465 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:04 smithi043 ceph-mon[29963]: pgmap v882: 168 pgs: 168 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-30T14:02:04.524 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:04.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:04.551 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:04.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:04.562 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:02:04.652 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:04.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:04.707 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:04.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:04.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:04 smithi113 ceph-mon[34341]: 4.10 scrub starts 2024-03-30T14:02:04.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:04 smithi113 ceph-mon[34341]: 4.10 scrub ok 2024-03-30T14:02:04.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:04 smithi113 ceph-mon[34341]: 4.12 scrub starts 2024-03-30T14:02:04.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:04 smithi113 ceph-mon[34341]: 4.12 scrub ok 2024-03-30T14:02:04.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:04 smithi113 ceph-mon[34341]: pgmap v882: 168 pgs: 168 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-30T14:02:04.807 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:04.832 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:04.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:04.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:04.934 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:04.982 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:04.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:04.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:05.061 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 45 16 60646 60630 84.1932 68.375 0.00523599 0.0118716 2024-03-30T14:02:05.084 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:05.140 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:05.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:05.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:05.242 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:05.268 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:05.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:05.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:05.370 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:05.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:05.415 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:05.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:05.516 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:05.539 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:02:05.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:05.576 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:05.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:05.664 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:05 smithi043 ceph-mon[29963]: 3.10 scrub starts 2024-03-30T14:02:05.664 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:05 smithi043 ceph-mon[29963]: 3.10 scrub ok 2024-03-30T14:02:05.664 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:05 smithi043 ceph-mon[29963]: 6.11 scrub starts 2024-03-30T14:02:05.664 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:05 smithi043 ceph-mon[29963]: 6.18 scrub starts 2024-03-30T14:02:05.664 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:05 smithi043 ceph-mon[29963]: 6.18 scrub ok 2024-03-30T14:02:05.666 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:05 smithi043 ceph-mon[42099]: 3.10 scrub starts 2024-03-30T14:02:05.666 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:05 smithi043 ceph-mon[42099]: 3.10 scrub ok 2024-03-30T14:02:05.666 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:05 smithi043 ceph-mon[42099]: 6.11 scrub starts 2024-03-30T14:02:05.666 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:05 smithi043 ceph-mon[42099]: 6.18 scrub starts 2024-03-30T14:02:05.666 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:05 smithi043 ceph-mon[42099]: 6.18 scrub ok 2024-03-30T14:02:05.677 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:05.706 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:05.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:05.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:05.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:05 smithi113 ceph-mon[34341]: 3.10 scrub starts 2024-03-30T14:02:05.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:05 smithi113 ceph-mon[34341]: 3.10 scrub ok 2024-03-30T14:02:05.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:05 smithi113 ceph-mon[34341]: 6.11 scrub starts 2024-03-30T14:02:05.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:05 smithi113 ceph-mon[34341]: 6.18 scrub starts 2024-03-30T14:02:05.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:05 smithi113 ceph-mon[34341]: 6.18 scrub ok 2024-03-30T14:02:05.808 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:05.849 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:05.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:05.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:05.951 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:05.979 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:05.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:05.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:06.061 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 46 16 61764 61748 83.8818 69.875 0.0369646 0.0119168 2024-03-30T14:02:06.081 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:06.125 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:06.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:06.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:06.183 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:02:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:02:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:02:06.227 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:06.255 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:06.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:06.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:06.357 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:06.399 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:06.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:06.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:06.433 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:06 smithi043 ceph-mon[29963]: 6.11 scrub ok 2024-03-30T14:02:06.433 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:06 smithi043 ceph-mon[29963]: pgmap v883: 168 pgs: 168 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-30T14:02:06.433 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:06 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/727194860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:02:06.434 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:06 smithi043 ceph-mon[42099]: 6.11 scrub ok 2024-03-30T14:02:06.434 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:06 smithi043 ceph-mon[42099]: pgmap v883: 168 pgs: 168 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-30T14:02:06.434 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:06 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/727194860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:02:06.502 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:06.530 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:06.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:06.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:06.632 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:06.661 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:06.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:06.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:06.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:06 smithi113 ceph-mon[34341]: 6.11 scrub ok 2024-03-30T14:02:06.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:06 smithi113 ceph-mon[34341]: pgmap v883: 168 pgs: 168 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-30T14:02:06.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:06 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/727194860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:02:06.763 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:06.774 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T14:02:06.774 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T14:02:06.774 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T14:02:06.774 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T14:02:06.774 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T14:02:06.774 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T14:02:06.774 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T14:02:06.774 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T14:02:06.775 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:02:06.775 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:02:06.775 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:02:06.775 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:02:06.775 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:02:06.775 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:02:06.775 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:02:06.775 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:02:06.793 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:06.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:06.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:06.895 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:06.935 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:06.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:06.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:07.037 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:07.061 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 47 16 62949 62933 83.6727 74.0625 0.0105254 0.011946 2024-03-30T14:02:07.078 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:07.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:07.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:07.181 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:07.210 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:07.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:07.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:07.312 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:07.345 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:07.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:07.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:07.447 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:07.486 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:07.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:07.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:07.589 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:07.620 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:07.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:07.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:07.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:07 smithi113 ceph-mon[34341]: 1.6 deep-scrub starts 2024-03-30T14:02:07.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:07 smithi113 ceph-mon[34341]: 1.6 deep-scrub ok 2024-03-30T14:02:07.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:07 smithi113 ceph-mon[34341]: 6.e deep-scrub starts 2024-03-30T14:02:07.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:07 smithi113 ceph-mon[34341]: 6.e deep-scrub ok 2024-03-30T14:02:07.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:07 smithi113 ceph-mon[34341]: 4.3 deep-scrub starts 2024-03-30T14:02:07.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:07 smithi113 ceph-mon[34341]: 4.3 deep-scrub ok 2024-03-30T14:02:07.721 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:07.762 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:07.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:07.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:07.863 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:07.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:07 smithi043 ceph-mon[29963]: 1.6 deep-scrub starts 2024-03-30T14:02:07.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:07 smithi043 ceph-mon[29963]: 1.6 deep-scrub ok 2024-03-30T14:02:07.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:07 smithi043 ceph-mon[29963]: 6.e deep-scrub starts 2024-03-30T14:02:07.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:07 smithi043 ceph-mon[29963]: 6.e deep-scrub ok 2024-03-30T14:02:07.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:07 smithi043 ceph-mon[29963]: 4.3 deep-scrub starts 2024-03-30T14:02:07.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:07 smithi043 ceph-mon[29963]: 4.3 deep-scrub ok 2024-03-30T14:02:07.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:07 smithi043 ceph-mon[42099]: 1.6 deep-scrub starts 2024-03-30T14:02:07.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:07 smithi043 ceph-mon[42099]: 1.6 deep-scrub ok 2024-03-30T14:02:07.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:07 smithi043 ceph-mon[42099]: 6.e deep-scrub starts 2024-03-30T14:02:07.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:07 smithi043 ceph-mon[42099]: 6.e deep-scrub ok 2024-03-30T14:02:07.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:07 smithi043 ceph-mon[42099]: 4.3 deep-scrub starts 2024-03-30T14:02:07.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:07 smithi043 ceph-mon[42099]: 4.3 deep-scrub ok 2024-03-30T14:02:07.896 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:07.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:07.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:07.997 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:08.030 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:08.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:08.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:08.061 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 48 15 64236 64221 83.6065 80.5 0.00966125 0.0119558 2024-03-30T14:02:08.132 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:08.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:08.163 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:08.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:08.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:08.306 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:08.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:08.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:08.410 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:08.450 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:08.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:08.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:08.552 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:08.592 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:08.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:08.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:08.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:08 smithi043 ceph-mon[29963]: 6.5 scrub starts 2024-03-30T14:02:08.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:08 smithi043 ceph-mon[29963]: 6.5 scrub ok 2024-03-30T14:02:08.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:08 smithi043 ceph-mon[29963]: 5.a scrub starts 2024-03-30T14:02:08.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:08 smithi043 ceph-mon[29963]: 5.a scrub ok 2024-03-30T14:02:08.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:08 smithi043 ceph-mon[29963]: 2.2 scrub starts 2024-03-30T14:02:08.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:08 smithi043 ceph-mon[29963]: 2.2 scrub ok 2024-03-30T14:02:08.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:08 smithi043 ceph-mon[29963]: pgmap v884: 168 pgs: 168 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-30T14:02:08.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:08 smithi043 ceph-mon[42099]: 6.5 scrub starts 2024-03-30T14:02:08.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:08 smithi043 ceph-mon[42099]: 6.5 scrub ok 2024-03-30T14:02:08.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:08 smithi043 ceph-mon[42099]: 5.a scrub starts 2024-03-30T14:02:08.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:08 smithi043 ceph-mon[42099]: 5.a scrub ok 2024-03-30T14:02:08.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:08 smithi043 ceph-mon[42099]: 2.2 scrub starts 2024-03-30T14:02:08.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:08 smithi043 ceph-mon[42099]: 2.2 scrub ok 2024-03-30T14:02:08.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:08 smithi043 ceph-mon[42099]: pgmap v884: 168 pgs: 168 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-30T14:02:08.694 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:08.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:08 smithi113 ceph-mon[34341]: 6.5 scrub starts 2024-03-30T14:02:08.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:08 smithi113 ceph-mon[34341]: 6.5 scrub ok 2024-03-30T14:02:08.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:08 smithi113 ceph-mon[34341]: 5.a scrub starts 2024-03-30T14:02:08.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:08 smithi113 ceph-mon[34341]: 5.a scrub ok 2024-03-30T14:02:08.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:08 smithi113 ceph-mon[34341]: 2.2 scrub starts 2024-03-30T14:02:08.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:08 smithi113 ceph-mon[34341]: 2.2 scrub ok 2024-03-30T14:02:08.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:08 smithi113 ceph-mon[34341]: pgmap v884: 168 pgs: 168 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-30T14:02:08.751 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:08.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:08.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:08.855 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:08.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:08.899 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:08.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:09.000 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:09.027 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:09.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:09.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:09.061 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 49 16 65530 65514 83.5493 80.8125 0.0109354 0.0119631 2024-03-30T14:02:09.129 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:09.155 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:09.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:09.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:09.257 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:09.300 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:09.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:09.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:09.401 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:09.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:09.427 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:09.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:09.528 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:09.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:09.573 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:09.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:09.674 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:09.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:09 smithi113 ceph-mon[34341]: 3.1a scrub starts 2024-03-30T14:02:09.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:09 smithi113 ceph-mon[34341]: 3.1a scrub ok 2024-03-30T14:02:09.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:09 smithi113 ceph-mon[34341]: 5.0 scrub starts 2024-03-30T14:02:09.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:09 smithi113 ceph-mon[34341]: 5.0 scrub ok 2024-03-30T14:02:09.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:02:09 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:02:09.716 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:09.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:09.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:09.818 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:09.828 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:09 smithi043 ceph-mon[29963]: 3.1a scrub starts 2024-03-30T14:02:09.828 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:09 smithi043 ceph-mon[29963]: 3.1a scrub ok 2024-03-30T14:02:09.828 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:09 smithi043 ceph-mon[29963]: 5.0 scrub starts 2024-03-30T14:02:09.828 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:09 smithi043 ceph-mon[29963]: 5.0 scrub ok 2024-03-30T14:02:09.829 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:09 smithi043 ceph-mon[42099]: 3.1a scrub starts 2024-03-30T14:02:09.829 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:09 smithi043 ceph-mon[42099]: 3.1a scrub ok 2024-03-30T14:02:09.829 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:09 smithi043 ceph-mon[42099]: 5.0 scrub starts 2024-03-30T14:02:09.829 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:09 smithi043 ceph-mon[42099]: 5.0 scrub ok 2024-03-30T14:02:09.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:09.865 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:09.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:09.966 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:09.995 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:09.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:09.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:10.061 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 50 16 66809 66793 83.4769 79.9375 0.0176897 0.0119739 2024-03-30T14:02:10.097 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:10.124 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:10.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:10.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:10.226 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:10.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:10.258 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:10.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:10.359 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:10.402 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:10.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:10.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:10.504 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:10.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:10.541 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:10.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:10.642 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:10.669 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T14:02:10.675 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:10.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:10.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:10.694 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:10 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:10.695 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:10 smithi043 ceph-mon[29963]: pgmap v885: 168 pgs: 168 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-30T14:02:10.695 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:10 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:10.695 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:10 smithi043 ceph-mon[42099]: pgmap v885: 168 pgs: 168 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-30T14:02:10.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:10 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:10.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:10 smithi113 ceph-mon[34341]: pgmap v885: 168 pgs: 168 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-30T14:02:10.777 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:10.839 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:10.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:10.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:10.941 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:10.967 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:10.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:10.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:11.062 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 51 16 68022 68006 83.3262 75.8125 0.0107368 0.0119957 2024-03-30T14:02:11.069 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:11.097 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:11.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:11.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:11.198 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:11.229 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:11.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:11.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:11.331 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:11.375 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:11.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:11.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:11.477 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:11.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:11.518 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:11.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:11.527 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:11 smithi043 ceph-mon[29963]: 6.8 deep-scrub starts 2024-03-30T14:02:11.527 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:11 smithi043 ceph-mon[29963]: 6.8 deep-scrub ok 2024-03-30T14:02:11.527 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:11 smithi043 ceph-mon[29963]: 1.f scrub starts 2024-03-30T14:02:11.527 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:11 smithi043 ceph-mon[29963]: 1.f scrub ok 2024-03-30T14:02:11.527 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:11 smithi043 ceph-mon[29963]: 6.17 scrub starts 2024-03-30T14:02:11.527 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:11 smithi043 ceph-mon[29963]: 6.17 scrub ok 2024-03-30T14:02:11.528 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:11 smithi043 ceph-mon[42099]: 6.8 deep-scrub starts 2024-03-30T14:02:11.528 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:11 smithi043 ceph-mon[42099]: 6.8 deep-scrub ok 2024-03-30T14:02:11.528 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:11 smithi043 ceph-mon[42099]: 1.f scrub starts 2024-03-30T14:02:11.528 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:11 smithi043 ceph-mon[42099]: 1.f scrub ok 2024-03-30T14:02:11.528 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:11 smithi043 ceph-mon[42099]: 6.17 scrub starts 2024-03-30T14:02:11.529 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:11 smithi043 ceph-mon[42099]: 6.17 scrub ok 2024-03-30T14:02:11.619 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:11.648 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:11.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:11.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:11.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:11 smithi113 ceph-mon[34341]: 6.8 deep-scrub starts 2024-03-30T14:02:11.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:11 smithi113 ceph-mon[34341]: 6.8 deep-scrub ok 2024-03-30T14:02:11.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:11 smithi113 ceph-mon[34341]: 1.f scrub starts 2024-03-30T14:02:11.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:11 smithi113 ceph-mon[34341]: 1.f scrub ok 2024-03-30T14:02:11.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:11 smithi113 ceph-mon[34341]: 6.17 scrub starts 2024-03-30T14:02:11.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:11 smithi113 ceph-mon[34341]: 6.17 scrub ok 2024-03-30T14:02:11.750 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:11.779 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:11.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:11.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:11.881 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:11.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:11.940 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:11.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:12.041 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:12.062 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 52 16 69082 69066 82.997 66.25 0.0148265 0.0120427 2024-03-30T14:02:12.088 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:12.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:12.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:12.190 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:12.216 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:12.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:12.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:12.318 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:12.346 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:12.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:12.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:12.448 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:12.464 INFO:teuthology.orchestra.run.smithi043.stderr:set 1.2 pg_upmap_items mapping to [2->4,0->1,7->6] 2024-03-30T14:02:12.473 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-03-30T14:02:12.475 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:12.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:12.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:12.476 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:02:12.573 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T14:02:12.574 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T14:02:12.574 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-03-30T14:02:12.574 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-03-30T14:02:12.574 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T14:02:12.576 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:12.604 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:12.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:12.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:12.707 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:12.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:12.758 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:12.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:12.800 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:12 smithi043 ceph-mon[29963]: 4.1a scrub starts 2024-03-30T14:02:12.801 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:12 smithi043 ceph-mon[29963]: 4.1a scrub ok 2024-03-30T14:02:12.801 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:12 smithi043 ceph-mon[29963]: pgmap v886: 168 pgs: 168 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-30T14:02:12.801 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:12 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/124606546' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.2", "id": [2, 4, 0, 1, 7, 6]}]: dispatch 2024-03-30T14:02:12.801 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:12 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2089384952' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:02:12.801 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:12 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:02:12.801 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:12 smithi043 ceph-mon[29963]: 3.15 scrub starts 2024-03-30T14:02:12.801 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:12 smithi043 ceph-mon[29963]: 3.15 scrub ok 2024-03-30T14:02:12.801 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:12 smithi043 ceph-mon[42099]: 4.1a scrub starts 2024-03-30T14:02:12.801 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:12 smithi043 ceph-mon[42099]: 4.1a scrub ok 2024-03-30T14:02:12.802 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:12 smithi043 ceph-mon[42099]: pgmap v886: 168 pgs: 168 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-30T14:02:12.802 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:12 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/124606546' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.2", "id": [2, 4, 0, 1, 7, 6]}]: dispatch 2024-03-30T14:02:12.802 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:12 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2089384952' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:02:12.802 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:12 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:02:12.802 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:12 smithi043 ceph-mon[42099]: 3.15 scrub starts 2024-03-30T14:02:12.802 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:12 smithi043 ceph-mon[42099]: 3.15 scrub ok 2024-03-30T14:02:12.859 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:12.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:12.892 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:12.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:12.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:12 smithi113 ceph-mon[34341]: 4.1a scrub starts 2024-03-30T14:02:12.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:12 smithi113 ceph-mon[34341]: 4.1a scrub ok 2024-03-30T14:02:12.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:12 smithi113 ceph-mon[34341]: pgmap v886: 168 pgs: 168 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-30T14:02:12.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:12 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/124606546' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.2", "id": [2, 4, 0, 1, 7, 6]}]: dispatch 2024-03-30T14:02:12.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:12 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2089384952' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:02:12.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:12 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:02:12.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:12 smithi113 ceph-mon[34341]: 3.15 scrub starts 2024-03-30T14:02:12.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:12 smithi113 ceph-mon[34341]: 3.15 scrub ok 2024-03-30T14:02:12.993 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:13.023 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:13.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:13.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:13.062 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 53 16 70206 70190 82.7564 70.25 0.00363203 0.012078 2024-03-30T14:02:13.125 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:13.167 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:13.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:13.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:13.269 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:13.307 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:13.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:13.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:13.409 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:13.448 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:13.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:13.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:13.550 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:13.589 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:13.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:13.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:13.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:13 smithi043 ceph-mon[29963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:02:13.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:13 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/124606546' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.2", "id": [2, 4, 0, 1, 7, 6]}]': finished 2024-03-30T14:02:13.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:13 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:02:13.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:13 smithi043 ceph-mon[29963]: osdmap e403: 8 total, 8 up, 7 in 2024-03-30T14:02:13.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:13 smithi043 ceph-mon[29963]: 4.7 scrub starts 2024-03-30T14:02:13.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:13 smithi043 ceph-mon[29963]: osdmap e404: 8 total, 8 up, 7 in 2024-03-30T14:02:13.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:13 smithi043 ceph-mon[42099]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:02:13.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:13 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/124606546' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.2", "id": [2, 4, 0, 1, 7, 6]}]': finished 2024-03-30T14:02:13.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:13 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:02:13.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:13 smithi043 ceph-mon[42099]: osdmap e403: 8 total, 8 up, 7 in 2024-03-30T14:02:13.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:13 smithi043 ceph-mon[42099]: 4.7 scrub starts 2024-03-30T14:02:13.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:13 smithi043 ceph-mon[42099]: osdmap e404: 8 total, 8 up, 7 in 2024-03-30T14:02:13.692 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:13.720 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:13.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:13.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:13.822 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:13.851 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:13.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:13.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:13.953 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:13.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:13 smithi113 ceph-mon[34341]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:02:13.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:13 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/124606546' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.2", "id": [2, 4, 0, 1, 7, 6]}]': finished 2024-03-30T14:02:13.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:13 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:02:13.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:13 smithi113 ceph-mon[34341]: osdmap e403: 8 total, 8 up, 7 in 2024-03-30T14:02:13.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:13 smithi113 ceph-mon[34341]: 4.7 scrub starts 2024-03-30T14:02:13.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:13 smithi113 ceph-mon[34341]: osdmap e404: 8 total, 8 up, 7 in 2024-03-30T14:02:13.995 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:13.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:13.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:14.062 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 54 16 71479 71463 82.6971 79.5625 0.0146737 0.0120869 2024-03-30T14:02:14.097 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:14.128 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:14.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:14.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:14.229 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-03-30T14:02:14.242 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:14.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:14.294 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:14.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:14.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:14.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:14.437 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:14.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:14.538 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:14.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:14.589 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:14.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:14.690 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:14.718 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:14.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:14.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:14.820 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:14.850 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:14.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:14.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:14.952 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:14.980 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:14.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:14.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:15.062 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 55 16 72679 72663 82.5571 75 0.010532 0.0121068 2024-03-30T14:02:15.082 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:15.110 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:15.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:15.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:15.212 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:15.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:14 smithi113 ceph-mon[34341]: pgmap v889: 168 pgs: 168 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-30T14:02:15.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:14 smithi113 ceph-mon[34341]: osdmap e405: 8 total, 8 up, 7 in 2024-03-30T14:02:15.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:14 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:15.240 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:15.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:15.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:15.342 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:15.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:15.383 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:15.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:15.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:14 smithi043 ceph-mon[42099]: pgmap v889: 168 pgs: 168 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-30T14:02:15.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:14 smithi043 ceph-mon[42099]: osdmap e405: 8 total, 8 up, 7 in 2024-03-30T14:02:15.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:14 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:15.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:14 smithi043 ceph-mon[29963]: pgmap v889: 168 pgs: 168 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-30T14:02:15.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:14 smithi043 ceph-mon[29963]: osdmap e405: 8 total, 8 up, 7 in 2024-03-30T14:02:15.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:14 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:15.484 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:15.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:15.509 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:15.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:15.611 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:15.647 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:15.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:15.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:15.749 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:15.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:15.783 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:15.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:15.884 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:15.915 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:15.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:15.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:16.017 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:16.045 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:16.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:16.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:16.062 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 56 16 73725 73709 82.2501 65.375 0.00605342 0.0121494 2024-03-30T14:02:16.108 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T14:02:16.147 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:16.176 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:16.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:16.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:16.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:15 smithi113 ceph-mon[34341]: osdmap e406: 8 total, 8 up, 7 in 2024-03-30T14:02:16.278 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:16.302 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:16.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:16.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:16.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:15 smithi043 ceph-mon[29963]: osdmap e406: 8 total, 8 up, 7 in 2024-03-30T14:02:16.392 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:02:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:02:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:02:16.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:15 smithi043 ceph-mon[42099]: osdmap e406: 8 total, 8 up, 7 in 2024-03-30T14:02:16.403 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:16.435 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:16.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:16.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:16.537 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:16.565 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:16.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:16.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:16.667 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:16.697 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:16.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:16.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:16.801 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:16.832 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:16.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:16.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:16.933 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:16.963 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:16.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:16.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:17.063 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 57 16 74776 74760 81.9594 65.6875 0.0505634 0.0121916 2024-03-30T14:02:17.064 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:17.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:17.105 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:17.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:17.206 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:17.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:16 smithi113 ceph-mon[34341]: pgmap v892: 168 pgs: 168 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-03-30T14:02:17.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:16 smithi113 ceph-mon[34341]: osdmap e407: 8 total, 8 up, 7 in 2024-03-30T14:02:17.237 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:17.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:17.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:17.339 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:17.386 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:17.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:17.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:17.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:16 smithi043 ceph-mon[29963]: pgmap v892: 168 pgs: 168 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-03-30T14:02:17.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:16 smithi043 ceph-mon[29963]: osdmap e407: 8 total, 8 up, 7 in 2024-03-30T14:02:17.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:16 smithi043 ceph-mon[42099]: pgmap v892: 168 pgs: 168 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-03-30T14:02:17.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:16 smithi043 ceph-mon[42099]: osdmap e407: 8 total, 8 up, 7 in 2024-03-30T14:02:17.489 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:17.527 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:17.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:17.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:17.629 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:17.664 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:17.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:17.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:17.766 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:17.796 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:17.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:17.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:17.898 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:17.924 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:17.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:17.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:18.026 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:18.055 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:18.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:18.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:18.062 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 58 16 75887 75871 81.7434 69.4375 0.00690188 0.0122285 2024-03-30T14:02:18.157 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:18.193 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:18.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:18.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:18.295 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:18.340 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:18.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:18.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:18.442 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:18.471 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:18.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:18.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:18.574 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:18.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:18.617 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:18.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:18.718 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:18.756 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:18.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:18.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:18.858 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:18.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:18.890 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:18.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:18.991 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:19.033 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:19.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:19.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:19.063 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 59 16 77130 77114 81.6746 77.6875 0.0113507 0.012239 2024-03-30T14:02:19.135 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:19.163 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:19.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:19.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:19.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:18 smithi113 ceph-mon[34341]: pgmap v894: 168 pgs: 168 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-30T14:02:19.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:18 smithi113 ceph-mon[34341]: osdmap e408: 8 total, 8 up, 7 in 2024-03-30T14:02:19.265 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:19.295 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:19.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:19.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:19.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:18 smithi043 ceph-mon[29963]: pgmap v894: 168 pgs: 168 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-30T14:02:19.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:18 smithi043 ceph-mon[29963]: osdmap e408: 8 total, 8 up, 7 in 2024-03-30T14:02:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:18 smithi043 ceph-mon[42099]: pgmap v894: 168 pgs: 168 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-30T14:02:19.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:18 smithi043 ceph-mon[42099]: osdmap e408: 8 total, 8 up, 7 in 2024-03-30T14:02:19.397 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:19.429 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:19.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:19.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:19.531 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:19.558 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:19.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:19.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:19.660 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:19.689 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:19.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:19.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:19.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:02:19 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:02:19.790 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:19.824 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:19.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:19.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:19.926 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:19.959 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:19.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:19.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:20.061 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:20.064 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-03-30T14:02:20.069879+0000 min lat: 0.00121029 max lat: 2.12887 avg lat: 0.0122659 2024-03-30T14:02:20.064 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T14:02:20.064 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 60 16 78266 78250 81.4966 71 0.00825912 0.0122659 2024-03-30T14:02:20.098 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:20.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:20.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:20.200 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:20.239 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:20.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:20.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:20.341 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:20.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:20.388 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:20.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:20.489 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:20.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:20.524 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:20.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:20.624 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:20.657 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:20.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:20.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:20.759 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:20.791 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:20.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:20.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:20.893 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:20.925 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:20.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:20.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:21.027 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:21.050 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:21.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:21.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:21.063 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 61 16 79476 79460 81.4002 75.625 0.0492652 0.0122768 2024-03-30T14:02:21.153 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:21.188 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:21.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:21.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:21.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:20 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:21.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:20 smithi113 ceph-mon[34341]: pgmap v896: 168 pgs: 168 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 875 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-03-30T14:02:21.290 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:21.322 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:21.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:21.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:20 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:20 smithi043 ceph-mon[29963]: pgmap v896: 168 pgs: 168 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 875 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-03-30T14:02:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:20 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:20 smithi043 ceph-mon[42099]: pgmap v896: 168 pgs: 168 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 875 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-03-30T14:02:21.425 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:21.454 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:21.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:21.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:21.556 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:21.586 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:21.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:21.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:21.688 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:21.731 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:21.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:21.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:21.833 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:21.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:21.904 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:21.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:22.006 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:22.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:22.037 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:22.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:22.063 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 62 16 80736 80720 81.3568 78.75 0.00877484 0.0122862 2024-03-30T14:02:22.138 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:22.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:22.168 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:22.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:22.270 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:22.315 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:22.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:22.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:22.417 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:22.445 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:22.446 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:02:22.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:22.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:22.505 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:02:22.547 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:22.568 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T14:02:22.596 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:22.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:22.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:22.609 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T14:02:22.642 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T14:02:22.687 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T14:02:22.698 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:22.732 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T14:02:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:22.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:22.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:22.780 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T14:02:22.819 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T14:02:22.844 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:22.867 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T14:02:22.873 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:02:22.873 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:02:22.873 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:02:22.873 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:02:22.873 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:02:22.874 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:02:22.874 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:02:22.874 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:02:22.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:22.882 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:22.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:22.983 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:23.028 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:23.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:23.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:23.063 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 63 16 81819 81803 81.1398 67.6875 0.00712991 0.012319 2024-03-30T14:02:23.112 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:22 smithi043 ceph-mon[29963]: pgmap v897: 168 pgs: 168 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.5 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-30T14:02:23.113 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:22 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1852062456' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:02:23.113 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:22 smithi043 ceph-mon[42099]: pgmap v897: 168 pgs: 168 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.5 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-30T14:02:23.113 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:22 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1852062456' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:02:23.130 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:23.160 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:23.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:23.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:23.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:22 smithi113 ceph-mon[34341]: pgmap v897: 168 pgs: 168 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.5 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-30T14:02:23.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:22 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1852062456' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:02:23.262 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:23.290 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:23.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:23.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:23.392 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:23.419 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:23.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:23.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:23.521 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:23.566 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:23.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:23.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:23.668 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:23.710 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:23.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:23.812 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:23.855 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:23.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:23.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:23.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:23.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:23.998 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:23.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:24.064 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 64 16 83081 83065 81.1043 78.875 0.0115452 0.0123242 2024-03-30T14:02:24.099 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:24.130 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:24.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:24.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:24.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:23 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:02:24.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:23 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:02:24.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:23 smithi113 ceph-mon[34341]: osdmap e409: 8 total, 8 up, 7 in 2024-03-30T14:02:24.231 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:24.262 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:24.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:24.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:24.318 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:23 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:02:24.318 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:23 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:02:24.318 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:23 smithi043 ceph-mon[29963]: osdmap e409: 8 total, 8 up, 7 in 2024-03-30T14:02:24.319 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:23 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:02:24.319 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:23 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:02:24.319 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:23 smithi043 ceph-mon[42099]: osdmap e409: 8 total, 8 up, 7 in 2024-03-30T14:02:24.364 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:24.402 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:24.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:24.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:24.503 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:24.540 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:24.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:24.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:24.642 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:24.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:24.685 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:24.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:24.786 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:24.814 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:24.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:24.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:24.916 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:24.947 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:24.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:24.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:25.008 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:24 smithi043 ceph-mon[29963]: pgmap v899: 168 pgs: 168 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.3 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-30T14:02:25.008 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:24 smithi043 ceph-mon[42099]: pgmap v899: 168 pgs: 168 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.3 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-30T14:02:25.049 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:25.064 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 65 16 84379 84363 81.1045 81.125 0.0126509 0.0123251 2024-03-30T14:02:25.077 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:25.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:25.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:25.178 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:25.209 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:25.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:25.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:25.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:24 smithi113 ceph-mon[34341]: pgmap v899: 168 pgs: 168 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.3 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-30T14:02:25.310 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:25.338 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:25.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:25.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:25.440 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:25.454 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T14:02:25.455 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:02:25.467 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:25.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:25.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:25.569 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:25.609 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:25.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:25.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:25.711 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:25.742 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:25.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:25.843 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:25.880 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:25.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:25.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:25.982 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:26.023 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:26.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:26.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:26.065 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 66 16 85552 85536 80.9848 73.3125 0.0184723 0.0123425 2024-03-30T14:02:26.126 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:26.142 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:02:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:02:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:02:26.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:26.175 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:26.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:26.276 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:26.302 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:26.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:26.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:26.403 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:26.434 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:26.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:26.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:26.536 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:26.588 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:26.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:26.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:26.601 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-03-30T14:02:26.601 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T14:02:26.601 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-03-30T14:02:26.602 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-30T14:02:26.602 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-30T14:02:26.602 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-03-30T14:02:26.689 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:26.730 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:26.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:26.793 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T14:02:26.831 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:26.903 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:26.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:26.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:26.940 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:02:27.005 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:27.033 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:27.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:27.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:27.060 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-30T14:02:27.069 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 67 16 86588 86572 80.7418 64.75 0.0212196 0.0123791 2024-03-30T14:02:27.135 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:27.160 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:27.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:27.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:27.241 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:26 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:02:27.241 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:26 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:02:27.241 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:26 smithi113 ceph-mon[34341]: pgmap v900: 168 pgs: 168 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-30T14:02:27.241 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:26 smithi113 ceph-mon[34341]: osdmap e410: 8 total, 8 up, 7 in 2024-03-30T14:02:27.241 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:26 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:02:27.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:27.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:27.316 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:27.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:27.342 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:26 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:02:27.342 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:26 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:02:27.342 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:26 smithi043 ceph-mon[42099]: pgmap v900: 168 pgs: 168 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-30T14:02:27.342 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:26 smithi043 ceph-mon[42099]: osdmap e410: 8 total, 8 up, 7 in 2024-03-30T14:02:27.342 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:26 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:02:27.343 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:26 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:02:27.343 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:26 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:02:27.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:26 smithi043 ceph-mon[29963]: pgmap v900: 168 pgs: 168 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-30T14:02:27.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:26 smithi043 ceph-mon[29963]: osdmap e410: 8 total, 8 up, 7 in 2024-03-30T14:02:27.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:26 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:02:27.418 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:27.449 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:27.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:27.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:27.551 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:27.604 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:27.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:27.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:27.706 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:27.725 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:02:27.734 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:02:27.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:27.736 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:27.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:27.837 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:27.891 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:27.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:27.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:27.993 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:28.037 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:28.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:28.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:28.066 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 68 16 87556 87540 80.4441 60.5 0.0111714 0.0124257 2024-03-30T14:02:28.139 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:28.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:02:28.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:02:28.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:02:28.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-30T14:02:28.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-30T14:02:28.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-30T14:02:28.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-03-30T14:02:28.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]}]: dispatch 2024-03-30T14:02:28.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]: dispatch 2024-03-30T14:02:28.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 6]}]: dispatch 2024-03-30T14:02:28.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]}]: dispatch 2024-03-30T14:02:28.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:02:28.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:02:28.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:02:28.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-30T14:02:28.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-30T14:02:28.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-30T14:02:28.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-03-30T14:02:28.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]}]: dispatch 2024-03-30T14:02:28.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]: dispatch 2024-03-30T14:02:28.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 6]}]: dispatch 2024-03-30T14:02:28.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]}]: dispatch 2024-03-30T14:02:28.179 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:28.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:28.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:28.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:02:28.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:02:28.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:02:28.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-30T14:02:28.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-30T14:02:28.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-30T14:02:28.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-03-30T14:02:28.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]}]: dispatch 2024-03-30T14:02:28.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]: dispatch 2024-03-30T14:02:28.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 6]}]: dispatch 2024-03-30T14:02:28.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]}]: dispatch 2024-03-30T14:02:28.282 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:28.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:28.310 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:28.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:28.375 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-03-30T14:02:28.411 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:28.440 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:28.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:28.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:28.542 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:28.576 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:28.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:28.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:28.677 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:28.709 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:28.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:28.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:28.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:28.853 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:28.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:28.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:28.922 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T14:02:28.922 INFO:teuthology.orchestra.run.smithi113.stdout: "success": "" 2024-03-30T14:02:28.922 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T14:02:28.955 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:28.987 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:28.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:28.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:29.066 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 69 16 88779 88763 80.3859 76.4375 0.00692132 0.0124351 2024-03-30T14:02:29.089 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:29.114 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[29963]: pgmap v902: 168 pgs: 168 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 928 op/s 2024-03-30T14:02:29.114 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4086670803' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:02:29.114 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:02:29.114 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[29963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:02:29.114 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-30T14:02:29.114 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-30T14:02:29.114 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-30T14:02:29.114 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-03-30T14:02:29.114 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]}]': finished 2024-03-30T14:02:29.114 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]': finished 2024-03-30T14:02:29.114 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 6]}]': finished 2024-03-30T14:02:29.114 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]}]': finished 2024-03-30T14:02:29.115 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:02:29.115 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[29963]: osdmap e411: 8 total, 8 up, 7 in 2024-03-30T14:02:29.115 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[42099]: pgmap v902: 168 pgs: 168 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 928 op/s 2024-03-30T14:02:29.115 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/4086670803' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:02:29.115 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:02:29.115 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[42099]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:02:29.115 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-30T14:02:29.115 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-30T14:02:29.115 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-30T14:02:29.115 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-03-30T14:02:29.115 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]}]': finished 2024-03-30T14:02:29.116 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]': finished 2024-03-30T14:02:29.116 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 6]}]': finished 2024-03-30T14:02:29.116 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]}]': finished 2024-03-30T14:02:29.116 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:02:29.116 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:28 smithi043 ceph-mon[42099]: osdmap e411: 8 total, 8 up, 7 in 2024-03-30T14:02:29.117 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:29.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:29.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:29.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:28 smithi113 ceph-mon[34341]: pgmap v902: 168 pgs: 168 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 928 op/s 2024-03-30T14:02:29.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:28 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/4086670803' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:02:29.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:28 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:02:29.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:28 smithi113 ceph-mon[34341]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:02:29.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-30T14:02:29.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-30T14:02:29.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-30T14:02:29.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-03-30T14:02:29.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]}]': finished 2024-03-30T14:02:29.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]': finished 2024-03-30T14:02:29.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 6]}]': finished 2024-03-30T14:02:29.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]}]': finished 2024-03-30T14:02:29.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:28 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:02:29.216 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:28 smithi113 ceph-mon[34341]: osdmap e411: 8 total, 8 up, 7 in 2024-03-30T14:02:29.220 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:29.265 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:29.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:29.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:29.367 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:29.412 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:29.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:29.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:29.514 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:29.546 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:29.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:29.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:29.589 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:02:29 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:02:29.649 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:29.686 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:29.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:29.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:29.788 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:29.828 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:29.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:29.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:29.930 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:29.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:29.961 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:29.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:30.062 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:30.066 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 70 16 90000 89984 80.3275 76.3125 0.00773322 0.0124436 2024-03-30T14:02:30.094 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:30.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:30.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:30.196 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:30.227 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:29 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:02:30.227 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:29 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:02:30.227 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:29 smithi113 ceph-mon[34341]: osdmap e412: 8 total, 8 up, 7 in 2024-03-30T14:02:30.227 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:29 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:30.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:30.238 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:30.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:30.338 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:29 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:02:30.339 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:29 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:02:30.339 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:29 smithi043 ceph-mon[29963]: osdmap e412: 8 total, 8 up, 7 in 2024-03-30T14:02:30.339 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:29 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:30.339 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:29 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:02:30.340 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:29 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:02:30.340 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:29 smithi043 ceph-mon[42099]: osdmap e412: 8 total, 8 up, 7 in 2024-03-30T14:02:30.340 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:29 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:30.340 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:30.368 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:30.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:30.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:30.470 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:30.496 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:30.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:30.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:30.598 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:30.643 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:30.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:30.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:30.745 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:30.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:30.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:30.891 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:30.921 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:30.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:30.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:31.023 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:31.048 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T14:02:31.049 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:02:31.066 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 71 16 91240 91224 80.2875 77.5 0.00972698 0.0124498 2024-03-30T14:02:31.067 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:31.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:31.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:31.169 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:31.208 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:31.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:31.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:31.310 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:31.340 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:31.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:31.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:30 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:30 smithi043 ceph-mon[29963]: pgmap v905: 168 pgs: 8 peering, 160 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s; 0 B/s, 0 objects/s recovering 2024-03-30T14:02:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:30 smithi043 ceph-mon[29963]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-30T14:02:31.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:30 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:31.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:30 smithi043 ceph-mon[42099]: pgmap v905: 168 pgs: 8 peering, 160 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s; 0 B/s, 0 objects/s recovering 2024-03-30T14:02:31.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:30 smithi043 ceph-mon[42099]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-30T14:02:31.442 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:31.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:30 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:31.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:30 smithi113 ceph-mon[34341]: pgmap v905: 168 pgs: 8 peering, 160 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s; 0 B/s, 0 objects/s recovering 2024-03-30T14:02:31.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:30 smithi113 ceph-mon[34341]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-30T14:02:31.470 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:31.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:31.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:31.571 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:31.597 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:31.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:31.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:31.699 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:31.730 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:31.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:31.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:31.832 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:31.863 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:31.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:31.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:31.965 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:32.008 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:32.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:32.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:32.066 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 72 16 92481 92465 80.2496 77.5625 0.00408631 0.0124559 2024-03-30T14:02:32.112 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:32.142 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:32.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:32.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:32.244 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:32.290 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:32.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:32.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:32.394 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T14:02:32.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:32.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:32.499 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-03-30T14:02:32.535 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:32.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:32.563 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:32.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:32.664 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:32.690 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:32.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:32.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:32.793 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:32.850 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:32.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:32.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:32.952 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:32.984 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:32.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:32.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:33.066 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 73 16 93622 93606 80.1271 71.3125 0.0132016 0.0124756 2024-03-30T14:02:33.086 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:33.116 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:33.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:33.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:33.218 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:33.248 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:33.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:33.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:33.318 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:02:33.333 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:02:33.350 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:33.377 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T14:02:33.378 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:33.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:33.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:33.379 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:32 smithi043 ceph-mon[29963]: pgmap v906: 168 pgs: 8 peering, 160 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 0 B/s, 0 objects/s recovering 2024-03-30T14:02:33.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:32 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:02:33.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:32 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:02:33.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:32 smithi043 ceph-mon[29963]: osdmap e413: 8 total, 8 up, 7 in 2024-03-30T14:02:33.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:32 smithi043 ceph-mon[29963]: osdmap e414: 8 total, 8 up, 7 in 2024-03-30T14:02:33.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:33 smithi043 ceph-mon[42099]: pgmap v906: 168 pgs: 8 peering, 160 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 0 B/s, 0 objects/s recovering 2024-03-30T14:02:33.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:33 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:02:33.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:33 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:02:33.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:33 smithi043 ceph-mon[42099]: osdmap e413: 8 total, 8 up, 7 in 2024-03-30T14:02:33.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:33 smithi043 ceph-mon[42099]: osdmap e414: 8 total, 8 up, 7 in 2024-03-30T14:02:33.424 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T14:02:33.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:32 smithi113 ceph-mon[34341]: pgmap v906: 168 pgs: 8 peering, 160 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 0 B/s, 0 objects/s recovering 2024-03-30T14:02:33.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:32 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:02:33.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:32 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:02:33.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:32 smithi113 ceph-mon[34341]: osdmap e413: 8 total, 8 up, 7 in 2024-03-30T14:02:33.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:32 smithi113 ceph-mon[34341]: osdmap e414: 8 total, 8 up, 7 in 2024-03-30T14:02:33.467 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T14:02:33.480 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:33.508 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T14:02:33.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:33.510 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:33.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:33.550 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T14:02:33.591 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T14:02:33.612 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:33.636 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T14:02:33.668 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:33.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:33.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:33.685 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T14:02:33.692 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:02:33.692 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:02:33.692 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:02:33.692 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:02:33.692 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:02:33.692 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:02:33.692 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:02:33.692 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:02:33.770 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:33.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:33.819 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:33.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:33.920 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:33.970 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:33.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:33.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:34.066 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 74 15 94661 94646 79.9226 65 0.0279792 0.0125073 2024-03-30T14:02:34.072 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:34.099 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:34.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:34.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:34.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:33 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:02:34.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:33 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:02:34.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:34 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3999246095' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:02:34.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:34 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:02:34.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:34 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:02:34.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:34 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3999246095' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:02:34.201 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:34.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:34.353 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:34.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:34.383 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:34.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:34.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:33 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:02:34.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:33 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:02:34.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:33 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3999246095' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:02:34.484 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:34.510 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:34.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:34.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:34.612 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:34.633 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-03-30T14:02:34.633 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T14:02:34.633 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 6, 5] 2024-03-30T14:02:34.634 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd out 3 2024-03-30T14:02:34.651 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:34.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:34.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:34.754 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:34.780 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:34.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:34.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:34.882 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:34.913 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:34.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:34.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:35.015 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:35.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:35.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:35.066 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 75 16 95740 95724 79.7552 67.375 0.0262923 0.0125333 2024-03-30T14:02:35.163 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:35.192 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:35.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:35.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:35.294 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:35.322 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:35.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:35.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:35.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:35 smithi043 ceph-mon[29963]: pgmap v909: 168 pgs: 2 peering, 166 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 944 B/s rd, 96 MiB/s wr, 1.54k op/s; 106 B/s, 4 objects/s recovering 2024-03-30T14:02:35.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:35 smithi043 ceph-mon[29963]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-03-30T14:02:35.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:35 smithi043 ceph-mon[42099]: pgmap v909: 168 pgs: 2 peering, 166 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 944 B/s rd, 96 MiB/s wr, 1.54k op/s; 106 B/s, 4 objects/s recovering 2024-03-30T14:02:35.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:35 smithi043 ceph-mon[42099]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-03-30T14:02:35.425 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:35.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:35.454 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:35.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:35.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:34 smithi113 ceph-mon[34341]: pgmap v909: 168 pgs: 2 peering, 166 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 944 B/s rd, 96 MiB/s wr, 1.54k op/s; 106 B/s, 4 objects/s recovering 2024-03-30T14:02:35.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:34 smithi113 ceph-mon[34341]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-03-30T14:02:35.555 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:35.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:35.597 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:35.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:35.698 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:35.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:35.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:35.844 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:35.881 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:35.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:35.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:35.983 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:36.012 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:36.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:36.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:36.067 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 76 15 96772 96757 79.5553 64.5625 0.0194353 0.0125653 2024-03-30T14:02:36.114 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:36.154 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:36.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:36.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:36.180 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:02:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:02:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:02:36.257 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:36.299 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:36.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:36.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:36.401 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:36.441 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:36.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:36.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:36.543 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:36.571 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:36.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:36.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:36.673 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:36.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:36.717 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:36.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:36.818 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:36.862 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:36.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:36.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:36.963 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:37.015 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:02:37.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:37.036 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:37.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:37.068 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 77 16 97988 97972 79.5073 75.9375 0.00702775 0.0125727 2024-03-30T14:02:37.137 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:37.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:37.191 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:37.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:37.217 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:37 smithi043 ceph-mon[29963]: pgmap v910: 168 pgs: 168 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 797 B/s rd, 58 MiB/s wr, 926 op/s; 175 B/s, 4 objects/s recovering 2024-03-30T14:02:37.218 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:37 smithi043 ceph-mon[42099]: pgmap v910: 168 pgs: 168 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 797 B/s rd, 58 MiB/s wr, 926 op/s; 175 B/s, 4 objects/s recovering 2024-03-30T14:02:37.292 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:37.331 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:37.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:37.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:37.433 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:37.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:37 smithi113 ceph-mon[34341]: pgmap v910: 168 pgs: 168 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 797 B/s rd, 58 MiB/s wr, 926 op/s; 175 B/s, 4 objects/s recovering 2024-03-30T14:02:37.470 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:37.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:37.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:37.572 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:37.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:37.615 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:37.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:37.717 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:37.748 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:37.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:37.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:37.850 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:37.876 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:37.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:37.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:37.972 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T14:02:37.978 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:38.069 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 78 16 99130 99114 79.4011 71.375 0.0113693 0.0125891 2024-03-30T14:02:38.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:38.073 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:38.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:38.174 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:38.202 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:38.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:38.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:38.306 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:38.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:38.335 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:38.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:38.436 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:38.484 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:38.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:38.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:38.586 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:38.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:38.619 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:38.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:38.721 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:38.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:38.765 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:38.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:38.866 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:38.912 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:38.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:38.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:39.014 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:39.046 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:39.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:39.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:39.071 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 79 16 100270 100254 79.2976 71.25 0.0176035 0.0126058 2024-03-30T14:02:39.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:39 smithi043 ceph-mon[29963]: pgmap v911: 168 pgs: 168 active+clean; 5.8 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 914 op/s; 140 B/s, 3 objects/s recovering 2024-03-30T14:02:39.094 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:39 smithi043 ceph-mon[42099]: pgmap v911: 168 pgs: 168 active+clean; 5.8 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 914 op/s; 140 B/s, 3 objects/s recovering 2024-03-30T14:02:39.147 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:39.176 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:39.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:39.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:39.277 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:39.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:39.320 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:39.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:39.399 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T14:02:39.399 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T14:02:39.399 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-03-30T14:02:39.399 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-03-30T14:02:39.399 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T14:02:39.421 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:39.446 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:39.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:39.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:39.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:39 smithi113 ceph-mon[34341]: pgmap v911: 168 pgs: 168 active+clean; 5.8 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 914 op/s; 140 B/s, 3 objects/s recovering 2024-03-30T14:02:39.548 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:39.580 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:39.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:39.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:39.682 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:39.710 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:39.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:39.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:39.812 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:39.841 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:39.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:39.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:39.944 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:39.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:02:39 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:02:39.984 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:39.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:39.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:40.034 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T14:02:40.070 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-03-30T14:02:40.076965+0000 min lat: 0.00121029 max lat: 2.12887 avg lat: 0.0126196 2024-03-30T14:02:40.070 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-30T14:02:40.070 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 80 16 101431 101415 79.2134 72.5625 0.0169104 0.0126196 2024-03-30T14:02:40.085 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:40.113 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:40.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:40.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:40.215 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:40.228 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-03-30T14:02:40.249 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:40.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:40.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:40.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:40 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1525763183' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:02:40.265 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:40 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1525763183' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:02:40.351 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:40.380 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:40.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:40.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:40.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:40 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1525763183' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:02:40.481 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:40.510 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:40.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:40.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:40.612 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:40.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:40.661 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:40.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:40.762 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:40.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:40.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:40.905 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:40.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:40.936 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:40.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:41.037 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:41.069 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:41.070 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 81 16 102657 102641 79.1813 76.625 0.0096835 0.0126245 2024-03-30T14:02:41.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:41.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:41.172 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:41.203 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:41.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:41.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:41.305 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:41.344 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:41.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:41.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:41.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:41 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:41.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:41 smithi043 ceph-mon[29963]: pgmap v912: 168 pgs: 168 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.26k op/s; 140 B/s, 3 objects/s recovering 2024-03-30T14:02:41.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:41 smithi043 ceph-mon[29963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:02:41.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:41 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1525763183' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:02:41.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:41 smithi043 ceph-mon[29963]: osdmap e415: 8 total, 8 up, 7 in 2024-03-30T14:02:41.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:41 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:41.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:41 smithi043 ceph-mon[42099]: pgmap v912: 168 pgs: 168 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.26k op/s; 140 B/s, 3 objects/s recovering 2024-03-30T14:02:41.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:41 smithi043 ceph-mon[42099]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:02:41.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:41 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1525763183' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:02:41.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:41 smithi043 ceph-mon[42099]: osdmap e415: 8 total, 8 up, 7 in 2024-03-30T14:02:41.446 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:41.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:41 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:41.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:41 smithi113 ceph-mon[34341]: pgmap v912: 168 pgs: 168 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.26k op/s; 140 B/s, 3 objects/s recovering 2024-03-30T14:02:41.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:41 smithi113 ceph-mon[34341]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:02:41.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:41 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1525763183' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:02:41.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:41 smithi113 ceph-mon[34341]: osdmap e415: 8 total, 8 up, 7 in 2024-03-30T14:02:41.493 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:41.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:41.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:41.595 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:41.623 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:41.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:41.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:41.725 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:41.768 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:41.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:41.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:41.870 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:41.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:41.912 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:41.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:42.013 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:42.056 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:42.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:42.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:42.070 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 82 16 103786 103770 79.0762 70.5625 0.00882638 0.0126398 2024-03-30T14:02:42.158 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:42.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:42.195 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:42.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:42.297 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:42.322 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:42.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:42.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:42.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:42 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:02:42.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:42 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:02:42.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:42 smithi043 ceph-mon[29963]: 1.2 scrub starts 2024-03-30T14:02:42.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:42 smithi043 ceph-mon[29963]: 1.2 scrub ok 2024-03-30T14:02:42.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:42 smithi043 ceph-mon[29963]: 11.b scrub starts 2024-03-30T14:02:42.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:42 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:02:42.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:42 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:02:42.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:42 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:02:42.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:42 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:02:42.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:42 smithi043 ceph-mon[42099]: 1.2 scrub starts 2024-03-30T14:02:42.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:42 smithi043 ceph-mon[42099]: 1.2 scrub ok 2024-03-30T14:02:42.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:42 smithi043 ceph-mon[42099]: 11.b scrub starts 2024-03-30T14:02:42.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:42 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:02:42.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:42 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:02:42.425 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:42.455 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:42.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:42.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:42.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:42 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:02:42.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:42 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:02:42.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:42 smithi113 ceph-mon[34341]: 1.2 scrub starts 2024-03-30T14:02:42.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:42 smithi113 ceph-mon[34341]: 1.2 scrub ok 2024-03-30T14:02:42.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:42 smithi113 ceph-mon[34341]: 11.b scrub starts 2024-03-30T14:02:42.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:42 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:02:42.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:42 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:02:42.557 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:42.587 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:42.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:42.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:42.689 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:42.717 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:42.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:42.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:42.818 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:42.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:42.844 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:42.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:42.945 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:42.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:42.971 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:42.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:43.070 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 83 16 104583 104567 78.7236 49.8125 0.0115784 0.0126972 2024-03-30T14:02:43.072 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:43.102 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:43.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:43.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:43.204 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:43.233 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:43.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:43.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:43.335 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:43.360 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:43.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:43.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:43 smithi043 ceph-mon[29963]: pgmap v914: 168 pgs: 168 active+clean; 6.1 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 127 B/s, 3 objects/s recovering 2024-03-30T14:02:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:43 smithi043 ceph-mon[29963]: 6.3 scrub starts 2024-03-30T14:02:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:43 smithi043 ceph-mon[29963]: osdmap e416: 8 total, 8 up, 7 in 2024-03-30T14:02:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:43 smithi043 ceph-mon[29963]: 6.3 scrub ok 2024-03-30T14:02:43.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:43 smithi043 ceph-mon[42099]: pgmap v914: 168 pgs: 168 active+clean; 6.1 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 127 B/s, 3 objects/s recovering 2024-03-30T14:02:43.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:43 smithi043 ceph-mon[42099]: 6.3 scrub starts 2024-03-30T14:02:43.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:43 smithi043 ceph-mon[42099]: osdmap e416: 8 total, 8 up, 7 in 2024-03-30T14:02:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:43 smithi043 ceph-mon[42099]: 6.3 scrub ok 2024-03-30T14:02:43.461 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:43.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:43 smithi113 ceph-mon[34341]: pgmap v914: 168 pgs: 168 active+clean; 6.1 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 127 B/s, 3 objects/s recovering 2024-03-30T14:02:43.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:43 smithi113 ceph-mon[34341]: 6.3 scrub starts 2024-03-30T14:02:43.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:43 smithi113 ceph-mon[34341]: osdmap e416: 8 total, 8 up, 7 in 2024-03-30T14:02:43.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:43 smithi113 ceph-mon[34341]: 6.3 scrub ok 2024-03-30T14:02:43.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:43.492 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:43.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:43.593 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:43.631 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:43.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:43.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:43.733 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:43.779 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:43.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:43.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:43.881 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:43.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:43.913 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:43.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:44.014 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:44.059 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:44.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:44.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:44.070 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 84 16 105504 105488 78.4716 57.5625 0.0103794 0.0127386 2024-03-30T14:02:44.161 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:44.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:44.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:44.303 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:44.333 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:44.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:44.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:44.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:44 smithi043 ceph-mon[29963]: 3.1b scrub starts 2024-03-30T14:02:44.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:44 smithi043 ceph-mon[29963]: 3.1b scrub ok 2024-03-30T14:02:44.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:44 smithi043 ceph-mon[42099]: 3.1b scrub starts 2024-03-30T14:02:44.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:44 smithi043 ceph-mon[42099]: 3.1b scrub ok 2024-03-30T14:02:44.435 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:44.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:44 smithi113 ceph-mon[34341]: 3.1b scrub starts 2024-03-30T14:02:44.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:44 smithi113 ceph-mon[34341]: 3.1b scrub ok 2024-03-30T14:02:44.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:44.479 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:44.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:44.580 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:44.622 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:44.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:44.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:44.724 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:44.764 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:44.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:44.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:44.866 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:44.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:44.896 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:44.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:44.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:45.037 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:45.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:45.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:45.070 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 85 16 106700 106684 78.4278 74.75 0.0190404 0.0127455 2024-03-30T14:02:45.139 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:45.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:45.179 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:45.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:45.280 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:45.304 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:45.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:45.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:45.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:45 smithi043 ceph-mon[29963]: 6.d scrub starts 2024-03-30T14:02:45.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:45 smithi043 ceph-mon[29963]: pgmap v916: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-30T14:02:45.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:45 smithi043 ceph-mon[29963]: 6.d scrub ok 2024-03-30T14:02:45.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:45 smithi043 ceph-mon[29963]: 11.b scrub ok 2024-03-30T14:02:45.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:45 smithi043 ceph-mon[29963]: 3.12 scrub starts 2024-03-30T14:02:45.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:45 smithi043 ceph-mon[29963]: 3.12 scrub ok 2024-03-30T14:02:45.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:45 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:45.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:45 smithi043 ceph-mon[42099]: 6.d scrub starts 2024-03-30T14:02:45.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:45 smithi043 ceph-mon[42099]: pgmap v916: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-30T14:02:45.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:45 smithi043 ceph-mon[42099]: 6.d scrub ok 2024-03-30T14:02:45.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:45 smithi043 ceph-mon[42099]: 11.b scrub ok 2024-03-30T14:02:45.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:45 smithi043 ceph-mon[42099]: 3.12 scrub starts 2024-03-30T14:02:45.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:45 smithi043 ceph-mon[42099]: 3.12 scrub ok 2024-03-30T14:02:45.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:45 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:45.406 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:45.433 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:45.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:45.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:45.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:45 smithi113 ceph-mon[34341]: 6.d scrub starts 2024-03-30T14:02:45.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:45 smithi113 ceph-mon[34341]: pgmap v916: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-30T14:02:45.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:45 smithi113 ceph-mon[34341]: 6.d scrub ok 2024-03-30T14:02:45.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:45 smithi113 ceph-mon[34341]: 11.b scrub ok 2024-03-30T14:02:45.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:45 smithi113 ceph-mon[34341]: 3.12 scrub starts 2024-03-30T14:02:45.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:45 smithi113 ceph-mon[34341]: 3.12 scrub ok 2024-03-30T14:02:45.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:45 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:45.535 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:45.574 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:45.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:45.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:45.676 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:45.716 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:45.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:45.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:45.818 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:45.846 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:45.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:45.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:45.948 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:45.976 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:45.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:45.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:46.070 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 86 16 107812 107796 78.3239 69.5 0.0156275 0.0127625 2024-03-30T14:02:46.078 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:46.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:46.115 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:46.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:46.216 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:46.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:46.266 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:46.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:46.313 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:46 smithi043 ceph-mon[29963]: 3.3 scrub starts 2024-03-30T14:02:46.313 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:46 smithi043 ceph-mon[29963]: 3.3 scrub ok 2024-03-30T14:02:46.314 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:02:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:02:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:02:46.314 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:46 smithi043 ceph-mon[42099]: 3.3 scrub starts 2024-03-30T14:02:46.314 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:46 smithi043 ceph-mon[42099]: 3.3 scrub ok 2024-03-30T14:02:46.367 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:46.418 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:46.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:46.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:46.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:46 smithi113 ceph-mon[34341]: 3.3 scrub starts 2024-03-30T14:02:46.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:46 smithi113 ceph-mon[34341]: 3.3 scrub ok 2024-03-30T14:02:46.491 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:02:46.520 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:46.549 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:46.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:46.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:46.651 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:46.698 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:46.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:46.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:46.800 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:46.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:46.830 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:46.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:46.931 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:46.963 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:46.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:46.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:47.065 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:47.070 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 87 16 108933 108917 78.2289 70.0625 0.00852877 0.0127779 2024-03-30T14:02:47.093 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:47.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:47.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:47.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:47 smithi043 ceph-mon[42099]: 3.d scrub starts 2024-03-30T14:02:47.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:47 smithi043 ceph-mon[42099]: 3.d scrub ok 2024-03-30T14:02:47.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:47 smithi043 ceph-mon[42099]: pgmap v917: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-30T14:02:47.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:47 smithi043 ceph-mon[42099]: 2.0 scrub starts 2024-03-30T14:02:47.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:47 smithi043 ceph-mon[42099]: 2.0 scrub ok 2024-03-30T14:02:47.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:47 smithi043 ceph-mon[29963]: 3.d scrub starts 2024-03-30T14:02:47.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:47 smithi043 ceph-mon[29963]: 3.d scrub ok 2024-03-30T14:02:47.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:47 smithi043 ceph-mon[29963]: pgmap v917: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-30T14:02:47.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:47 smithi043 ceph-mon[29963]: 2.0 scrub starts 2024-03-30T14:02:47.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:47 smithi043 ceph-mon[29963]: 2.0 scrub ok 2024-03-30T14:02:47.195 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:47.240 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:47.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:47.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:47.341 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:47.363 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:02:47.369 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:47.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:47.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:47.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:47 smithi113 ceph-mon[34341]: 3.d scrub starts 2024-03-30T14:02:47.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:47 smithi113 ceph-mon[34341]: 3.d scrub ok 2024-03-30T14:02:47.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:47 smithi113 ceph-mon[34341]: pgmap v917: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-30T14:02:47.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:47 smithi113 ceph-mon[34341]: 2.0 scrub starts 2024-03-30T14:02:47.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:47 smithi113 ceph-mon[34341]: 2.0 scrub ok 2024-03-30T14:02:47.470 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:47.523 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:47.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:47.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:47.627 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:47.656 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:47.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:47.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:47.759 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:47.787 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:47.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:47.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:47.889 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:47.917 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:47.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:47.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:48.019 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:48.050 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:48.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:48.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:48.071 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 88 16 110074 110058 78.1502 71.3125 0.00780902 0.0127909 2024-03-30T14:02:48.071 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:47 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a[29940]: 2024-03-30T14:02:47.775+0000 7ff6065d3700 -1 mon.a@0(leader).osd e416 definitely_dead 0 2024-03-30T14:02:48.103 INFO:teuthology.orchestra.run.smithi043.stderr:marked out osd.3. 2024-03-30T14:02:48.152 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:48.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:48.176 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:48.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:48.204 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T14:02:48.277 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:48.301 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:48.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:48.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:48.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:48 smithi043 ceph-mon[29963]: 4.8 scrub starts 2024-03-30T14:02:48.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:48 smithi043 ceph-mon[29963]: 4.8 scrub ok 2024-03-30T14:02:48.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:48 smithi043 ceph-mon[29963]: 3.e scrub starts 2024-03-30T14:02:48.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:48 smithi043 ceph-mon[29963]: 3.e scrub ok 2024-03-30T14:02:48.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:48 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/743408588' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-30T14:02:48.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:48 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-30T14:02:48.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:48 smithi043 ceph-mon[29963]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-30T14:02:48.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:48 smithi043 ceph-mon[42099]: 4.8 scrub starts 2024-03-30T14:02:48.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:48 smithi043 ceph-mon[42099]: 4.8 scrub ok 2024-03-30T14:02:48.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:48 smithi043 ceph-mon[42099]: 3.e scrub starts 2024-03-30T14:02:48.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:48 smithi043 ceph-mon[42099]: 3.e scrub ok 2024-03-30T14:02:48.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:48 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/743408588' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-30T14:02:48.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:48 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-30T14:02:48.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:48 smithi043 ceph-mon[42099]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-30T14:02:48.403 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:48.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:48.427 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:48.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:48.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:48 smithi113 ceph-mon[34341]: 4.8 scrub starts 2024-03-30T14:02:48.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:48 smithi113 ceph-mon[34341]: 4.8 scrub ok 2024-03-30T14:02:48.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:48 smithi113 ceph-mon[34341]: 3.e scrub starts 2024-03-30T14:02:48.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:48 smithi113 ceph-mon[34341]: 3.e scrub ok 2024-03-30T14:02:48.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:48 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/743408588' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-30T14:02:48.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:48 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-30T14:02:48.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:48 smithi113 ceph-mon[34341]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-30T14:02:48.529 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:48.553 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:48.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:48.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:48.655 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:48.679 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:48.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:48.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:48.781 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:48.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:48.805 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:48.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:48.906 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:48.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:48.931 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:48.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:49.032 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:49.057 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:49.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:49.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:49.070 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 89 16 110115 110099 77.301 2.5625 0.00713684 0.012795 2024-03-30T14:02:49.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:49 smithi043 ceph-mon[29963]: 6.6 scrub starts 2024-03-30T14:02:49.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:49 smithi043 ceph-mon[29963]: 6.6 scrub ok 2024-03-30T14:02:49.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:49 smithi043 ceph-mon[29963]: pgmap v918: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 6.5 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 833 op/s 2024-03-30T14:02:49.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:49 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-30T14:02:49.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:49 smithi043 ceph-mon[29963]: osdmap e417: 8 total, 8 up, 6 in 2024-03-30T14:02:49.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:49 smithi043 ceph-mon[29963]: 5.19 scrub starts 2024-03-30T14:02:49.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:49 smithi043 ceph-mon[29963]: 5.19 scrub ok 2024-03-30T14:02:49.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:49 smithi043 ceph-mon[42099]: 6.6 scrub starts 2024-03-30T14:02:49.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:49 smithi043 ceph-mon[42099]: 6.6 scrub ok 2024-03-30T14:02:49.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:49 smithi043 ceph-mon[42099]: pgmap v918: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 6.5 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 833 op/s 2024-03-30T14:02:49.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:49 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-30T14:02:49.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:49 smithi043 ceph-mon[42099]: osdmap e417: 8 total, 8 up, 6 in 2024-03-30T14:02:49.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:49 smithi043 ceph-mon[42099]: 5.19 scrub starts 2024-03-30T14:02:49.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:49 smithi043 ceph-mon[42099]: 5.19 scrub ok 2024-03-30T14:02:49.158 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:49.183 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:49.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:49.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:49.285 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:49.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:49.310 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:49.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:49.366 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:49 smithi113 ceph-mon[34341]: 6.6 scrub starts 2024-03-30T14:02:49.366 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:49 smithi113 ceph-mon[34341]: 6.6 scrub ok 2024-03-30T14:02:49.366 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:49 smithi113 ceph-mon[34341]: pgmap v918: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 6.5 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 833 op/s 2024-03-30T14:02:49.366 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:49 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-30T14:02:49.366 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:49 smithi113 ceph-mon[34341]: osdmap e417: 8 total, 8 up, 6 in 2024-03-30T14:02:49.367 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:49 smithi113 ceph-mon[34341]: 5.19 scrub starts 2024-03-30T14:02:49.367 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:49 smithi113 ceph-mon[34341]: 5.19 scrub ok 2024-03-30T14:02:49.410 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:49.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:49.435 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:49.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:49.536 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:49.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:49.562 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:49.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:49.663 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:49.686 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:49.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:49.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:49.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:02:49 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:02:49.788 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:49.812 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:49.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:49.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:49.914 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:49.937 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:49.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:49.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:50.039 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:50.096 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 90 16 110115 110099 76.4422 0 - 0.012795 2024-03-30T14:02:50.133 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:50.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:50.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:50.235 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:50.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:50.263 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:50.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:50.364 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:50.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:50 smithi043 ceph-mon[29963]: osdmap e418: 8 total, 8 up, 6 in 2024-03-30T14:02:50.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:50 smithi043 ceph-mon[29963]: 3.11 scrub starts 2024-03-30T14:02:50.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:50 smithi043 ceph-mon[29963]: 3.11 scrub ok 2024-03-30T14:02:50.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:50 smithi043 ceph-mon[29963]: osdmap e419: 8 total, 8 up, 6 in 2024-03-30T14:02:50.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:50 smithi043 ceph-mon[42099]: osdmap e418: 8 total, 8 up, 6 in 2024-03-30T14:02:50.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:50 smithi043 ceph-mon[42099]: 3.11 scrub starts 2024-03-30T14:02:50.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:50 smithi043 ceph-mon[42099]: 3.11 scrub ok 2024-03-30T14:02:50.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:50 smithi043 ceph-mon[42099]: osdmap e419: 8 total, 8 up, 6 in 2024-03-30T14:02:50.394 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:02:50.397 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:50.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:50.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:50.459 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T14:02:50.459 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-03-30T14:02:50.459 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-03-30T14:02:50.459 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-03-30T14:02:50.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.459 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:56045 9.9 9:9d125b1d:::benchmark_data_smithi043_145506_object56044:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-30T14:02:50.459 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:27.623224+0000", 2024-03-30T14:02:50.459 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 562.83887154700005, 2024-03-30T14:02:50.459 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.19522012399999999, 2024-03-30T14:02:50.459 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:02:50.459 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:02:50.460 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:02:50.460 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:02:50.460 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:02:50.460 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:02:50.460 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 56045 2024-03-30T14:02:50.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.460 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:02:50.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.460 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:02:50.460 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.623224+0000", 2024-03-30T14:02:50.460 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.460 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:02:50.461 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.623224+0000", 2024-03-30T14:02:50.461 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.461 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:02:50.461 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.623225+0000", 2024-03-30T14:02:50.461 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1400000000000001e-06 2024-03-30T14:02:50.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.461 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:02:50.461 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.623247+0000", 2024-03-30T14:02:50.461 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1265000000000001e-05 2024-03-30T14:02:50.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.462 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:02:50.462 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.623254+0000", 2024-03-30T14:02:50.462 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1350000000000003e-06 2024-03-30T14:02:50.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.462 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:02:50.462 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.623259+0000", 2024-03-30T14:02:50.462 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5659999999999997e-06 2024-03-30T14:02:50.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.462 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.462 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.630735+0000", 2024-03-30T14:02:50.462 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0074759639999999999 2024-03-30T14:02:50.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.463 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:02:50.463 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.630909+0000", 2024-03-30T14:02:50.463 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017374400000000001 2024-03-30T14:02:50.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.463 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T14:02:50.463 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.631087+0000", 2024-03-30T14:02:50.463 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017782799999999999 2024-03-30T14:02:50.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.463 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:02:50.463 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.635848+0000", 2024-03-30T14:02:50.464 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0047612590000000003 2024-03-30T14:02:50.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.464 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.464 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638798+0000", 2024-03-30T14:02:50.464 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0029492820000000001 2024-03-30T14:02:50.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.464 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.464 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.816562+0000", 2024-03-30T14:02:50.464 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.177764483 2024-03-30T14:02:50.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.465 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:02:50.465 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.816647+0000", 2024-03-30T14:02:50.465 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5346000000000006e-05 2024-03-30T14:02:50.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.465 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:02:50.465 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.818444+0000", 2024-03-30T14:02:50.465 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001797112 2024-03-30T14:02:50.465 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.465 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.466 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.466 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:56049 9.9 9:9a2cdab1:::benchmark_data_smithi043_145506_object56048:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-30T14:02:50.466 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:27.626166+0000", 2024-03-30T14:02:50.466 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 562.835930104, 2024-03-30T14:02:50.466 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.194835114, 2024-03-30T14:02:50.466 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:02:50.466 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:02:50.466 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:02:50.466 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:02:50.466 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:02:50.466 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:02:50.467 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 56049 2024-03-30T14:02:50.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.467 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:02:50.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.467 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:02:50.467 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626166+0000", 2024-03-30T14:02:50.467 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.467 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:02:50.467 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626166+0000", 2024-03-30T14:02:50.467 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.467 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:02:50.468 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626168+0000", 2024-03-30T14:02:50.468 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9290000000000001e-06 2024-03-30T14:02:50.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.468 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:02:50.468 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626196+0000", 2024-03-30T14:02:50.468 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.8492000000000001e-05 2024-03-30T14:02:50.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.468 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:02:50.468 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626210+0000", 2024-03-30T14:02:50.468 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3713999999999999e-05 2024-03-30T14:02:50.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.469 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:02:50.469 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626220+0000", 2024-03-30T14:02:50.469 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0210999999999999e-05 2024-03-30T14:02:50.469 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.469 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.469 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.633332+0000", 2024-03-30T14:02:50.469 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0071117130000000004 2024-03-30T14:02:50.469 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.469 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:02:50.469 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.633484+0000", 2024-03-30T14:02:50.469 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015213600000000001 2024-03-30T14:02:50.469 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.470 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T14:02:50.470 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.633660+0000", 2024-03-30T14:02:50.470 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017599299999999999 2024-03-30T14:02:50.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.470 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.470 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638845+0000", 2024-03-30T14:02:50.470 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.005185182 2024-03-30T14:02:50.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.470 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:02:50.470 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.788979+0000", 2024-03-30T14:02:50.470 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.150133868 2024-03-30T14:02:50.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.471 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.471 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.471 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.818506+0000", 2024-03-30T14:02:50.471 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.029527241999999999 2024-03-30T14:02:50.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.471 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.471 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:02:50.471 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.818764+0000", 2024-03-30T14:02:50.471 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00025784799999999997 2024-03-30T14:02:50.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.471 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.471 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:02:50.471 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.821001+0000", 2024-03-30T14:02:50.472 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022367860000000002 2024-03-30T14:02:50.472 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.472 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.472 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.472 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.472 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:56054 9.9 9:94de70d2:::benchmark_data_smithi043_145506_object56053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-30T14:02:50.472 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:27.626948+0000", 2024-03-30T14:02:50.472 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 562.835148033, 2024-03-30T14:02:50.472 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.19715785699999999, 2024-03-30T14:02:50.472 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:02:50.472 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:02:50.472 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:02:50.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:50 smithi113 ceph-mon[34341]: osdmap e418: 8 total, 8 up, 6 in 2024-03-30T14:02:50.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:50 smithi113 ceph-mon[34341]: 3.11 scrub starts 2024-03-30T14:02:50.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:50 smithi113 ceph-mon[34341]: 3.11 scrub ok 2024-03-30T14:02:50.473 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:50 smithi113 ceph-mon[34341]: osdmap e419: 8 total, 8 up, 6 in 2024-03-30T14:02:50.474 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:02:50.474 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:02:50.474 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:02:50.474 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 56054 2024-03-30T14:02:50.474 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.474 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:02:50.474 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.474 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:02:50.475 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626948+0000", 2024-03-30T14:02:50.475 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.475 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.475 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.475 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:02:50.475 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626948+0000", 2024-03-30T14:02:50.475 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.475 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.475 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.475 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:02:50.475 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626949+0000", 2024-03-30T14:02:50.475 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1060000000000001e-06 2024-03-30T14:02:50.475 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.475 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.475 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:02:50.476 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626971+0000", 2024-03-30T14:02:50.476 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1872e-05 2024-03-30T14:02:50.476 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.476 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.476 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:02:50.476 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626983+0000", 2024-03-30T14:02:50.476 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2349e-05 2024-03-30T14:02:50.476 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.476 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.476 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:02:50.476 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.626992+0000", 2024-03-30T14:02:50.476 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3639999999999999e-06 2024-03-30T14:02:50.476 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.476 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.477 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.477 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.636090+0000", 2024-03-30T14:02:50.477 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0090986650000000006 2024-03-30T14:02:50.477 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.477 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.477 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:02:50.477 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.636249+0000", 2024-03-30T14:02:50.477 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015851 2024-03-30T14:02:50.477 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.477 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.477 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T14:02:50.477 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.636486+0000", 2024-03-30T14:02:50.477 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000236958 2024-03-30T14:02:50.477 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.478 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.478 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.478 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638903+0000", 2024-03-30T14:02:50.478 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024170289999999998 2024-03-30T14:02:50.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.478 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.478 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:02:50.478 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.788986+0000", 2024-03-30T14:02:50.478 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.150083314 2024-03-30T14:02:50.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.478 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.478 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.478 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.821064+0000", 2024-03-30T14:02:50.478 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.032077556 2024-03-30T14:02:50.479 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.479 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.479 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:02:50.479 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.821770+0000", 2024-03-30T14:02:50.479 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00070655000000000002 2024-03-30T14:02:50.479 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.479 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.479 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:02:50.479 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.824106+0000", 2024-03-30T14:02:50.479 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0023355839999999999 2024-03-30T14:02:50.479 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.479 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.479 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.479 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.480 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.480 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:56056 9.9 9:90e814ba:::benchmark_data_smithi043_145506_object56055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-30T14:02:50.480 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:27.628943+0000", 2024-03-30T14:02:50.480 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 562.833153305, 2024-03-30T14:02:50.480 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.20046303700000001, 2024-03-30T14:02:50.480 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:02:50.480 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:02:50.480 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:02:50.480 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:02:50.480 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:02:50.480 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:02:50.480 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 56056 2024-03-30T14:02:50.480 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.480 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:02:50.480 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.481 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:02:50.481 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.628943+0000", 2024-03-30T14:02:50.481 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.481 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.481 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.481 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:02:50.481 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.628943+0000", 2024-03-30T14:02:50.481 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.481 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.481 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.481 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:02:50.481 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.628944+0000", 2024-03-30T14:02:50.481 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0300000000000001e-06 2024-03-30T14:02:50.481 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.481 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.482 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:02:50.482 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.628963+0000", 2024-03-30T14:02:50.482 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9885999999999999e-05 2024-03-30T14:02:50.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.482 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.482 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:02:50.482 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.628972+0000", 2024-03-30T14:02:50.482 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2689999999999999e-06 2024-03-30T14:02:50.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.482 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.482 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:02:50.482 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.628978+0000", 2024-03-30T14:02:50.482 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8989999999999998e-06 2024-03-30T14:02:50.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.483 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.483 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.483 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638945+0000", 2024-03-30T14:02:50.483 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0099676839999999992 2024-03-30T14:02:50.483 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.483 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.483 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:02:50.483 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.639093+0000", 2024-03-30T14:02:50.483 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014789 2024-03-30T14:02:50.483 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.483 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.483 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T14:02:50.483 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.639261+0000", 2024-03-30T14:02:50.483 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016739500000000001 2024-03-30T14:02:50.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.484 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.484 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.484 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.788960+0000", 2024-03-30T14:02:50.484 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.149699096 2024-03-30T14:02:50.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.484 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.484 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:02:50.484 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.788993+0000", 2024-03-30T14:02:50.484 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.2978999999999997e-05 2024-03-30T14:02:50.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.484 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.484 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.484 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.824162+0000", 2024-03-30T14:02:50.484 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.035169259000000001 2024-03-30T14:02:50.485 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.485 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.485 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:02:50.485 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.824379+0000", 2024-03-30T14:02:50.485 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00021719600000000001 2024-03-30T14:02:50.485 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.485 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.485 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:02:50.485 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.829406+0000", 2024-03-30T14:02:50.485 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0050264539999999996 2024-03-30T14:02:50.485 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.485 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.485 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.485 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.485 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.486 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:56060 9.9 9:969aa2d0:::benchmark_data_smithi043_145506_object56059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-30T14:02:50.486 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:27.634703+0000", 2024-03-30T14:02:50.486 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 562.82739272399999, 2024-03-30T14:02:50.486 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.20106921799999999, 2024-03-30T14:02:50.486 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:02:50.486 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:02:50.486 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:02:50.486 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:02:50.486 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:02:50.486 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:02:50.486 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 56060 2024-03-30T14:02:50.486 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.486 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:02:50.486 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.487 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:02:50.487 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.634703+0000", 2024-03-30T14:02:50.487 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.487 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.487 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.487 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:02:50.487 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.634703+0000", 2024-03-30T14:02:50.487 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.487 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.487 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.487 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:02:50.487 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.634704+0000", 2024-03-30T14:02:50.487 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2e-07 2024-03-30T14:02:50.487 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.488 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.488 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:02:50.488 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.634723+0000", 2024-03-30T14:02:50.488 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9125000000000001e-05 2024-03-30T14:02:50.488 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.488 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.488 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:02:50.488 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.634784+0000", 2024-03-30T14:02:50.488 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0797000000000001e-05 2024-03-30T14:02:50.488 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.488 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.488 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:02:50.488 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.634790+0000", 2024-03-30T14:02:50.488 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3010000000000003e-06 2024-03-30T14:02:50.488 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.489 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.489 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.489 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.641402+0000", 2024-03-30T14:02:50.489 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0066118039999999998 2024-03-30T14:02:50.489 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.489 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.489 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:02:50.489 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.641547+0000", 2024-03-30T14:02:50.489 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014569 2024-03-30T14:02:50.489 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.489 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.489 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-03-30T14:02:50.489 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.641656+0000", 2024-03-30T14:02:50.489 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000108305 2024-03-30T14:02:50.490 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.490 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.490 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:02:50.490 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.788999+0000", 2024-03-30T14:02:50.490 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14734349499999999 2024-03-30T14:02:50.490 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.490 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.490 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.490 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.802260+0000", 2024-03-30T14:02:50.490 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.013260625999999999 2024-03-30T14:02:50.490 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.490 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.490 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.491 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.829466+0000", 2024-03-30T14:02:50.491 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.027206096999999999 2024-03-30T14:02:50.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.491 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.491 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:02:50.491 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.829525+0000", 2024-03-30T14:02:50.491 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8798999999999997e-05 2024-03-30T14:02:50.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.491 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.491 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:02:50.491 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.835772+0000", 2024-03-30T14:02:50.491 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.006247659 2024-03-30T14:02:50.491 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.492 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.492 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.492 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.492 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:56064 9.5 9:aa87c190:::benchmark_data_smithi043_145506_object56063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-30T14:02:50.492 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:27.638259+0000", 2024-03-30T14:02:50.492 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 562.82383730799995, 2024-03-30T14:02:50.492 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.18948137900000001, 2024-03-30T14:02:50.492 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:02:50.492 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:02:50.492 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:02:50.492 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:02:50.492 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:02:50.492 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:02:50.493 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 56064 2024-03-30T14:02:50.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.493 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:02:50.493 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.493 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:02:50.493 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638259+0000", 2024-03-30T14:02:50.493 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.493 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.493 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:02:50.493 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638259+0000", 2024-03-30T14:02:50.493 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.493 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.493 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:02:50.494 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638259+0000", 2024-03-30T14:02:50.494 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1500000000000005e-07 2024-03-30T14:02:50.494 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.494 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.494 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:02:50.494 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638280+0000", 2024-03-30T14:02:50.494 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1359999999999999e-05 2024-03-30T14:02:50.494 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.494 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.494 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:02:50.494 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638288+0000", 2024-03-30T14:02:50.494 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.571e-06 2024-03-30T14:02:50.494 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.495 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.495 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:02:50.495 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.638294+0000", 2024-03-30T14:02:50.495 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.981e-06 2024-03-30T14:02:50.495 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.495 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.495 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.495 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.796790+0000", 2024-03-30T14:02:50.495 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.15849614300000001 2024-03-30T14:02:50.495 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.495 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.495 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:02:50.495 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.796977+0000", 2024-03-30T14:02:50.495 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018698400000000001 2024-03-30T14:02:50.495 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.496 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.496 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,7", 2024-03-30T14:02:50.496 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.797148+0000", 2024-03-30T14:02:50.496 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000170755 2024-03-30T14:02:50.496 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.496 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.496 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:02:50.496 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.810226+0000", 2024-03-30T14:02:50.496 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.013078485000000001 2024-03-30T14:02:50.496 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.496 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.496 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.496 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.810298+0000", 2024-03-30T14:02:50.496 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1335999999999996e-05 2024-03-30T14:02:50.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.497 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.497 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.497 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.823041+0000", 2024-03-30T14:02:50.497 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.012742974000000001 2024-03-30T14:02:50.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.497 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.497 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:02:50.497 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.823104+0000", 2024-03-30T14:02:50.497 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3792000000000002e-05 2024-03-30T14:02:50.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.497 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.497 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:02:50.497 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:27.827740+0000", 2024-03-30T14:02:50.497 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0046354830000000001 2024-03-30T14:02:50.498 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.498 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.498 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.498 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.498 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61013 9.5 9:abc7156e:::benchmark_data_smithi043_145506_object61012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-30T14:02:50.498 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:33.735921+0000", 2024-03-30T14:02:50.498 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 556.72617517599997, 2024-03-30T14:02:50.498 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.19262681400000001, 2024-03-30T14:02:50.498 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:02:50.498 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:02:50.498 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:02:50.498 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:02:50.498 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:02:50.499 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:02:50.499 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61013 2024-03-30T14:02:50.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.499 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:02:50.499 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.499 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:02:50.499 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.735921+0000", 2024-03-30T14:02:50.499 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.499 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.499 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:02:50.499 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.735921+0000", 2024-03-30T14:02:50.499 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.499 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.500 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:02:50.500 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.735922+0000", 2024-03-30T14:02:50.500 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3030000000000001e-06 2024-03-30T14:02:50.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.500 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.500 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:02:50.500 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.735947+0000", 2024-03-30T14:02:50.500 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5089999999999999e-05 2024-03-30T14:02:50.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.500 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.500 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:02:50.500 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.735961+0000", 2024-03-30T14:02:50.500 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3869999999999999e-05 2024-03-30T14:02:50.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.501 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.501 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:02:50.501 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.735970+0000", 2024-03-30T14:02:50.501 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0580000000000004e-06 2024-03-30T14:02:50.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.501 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.501 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.501 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.738486+0000", 2024-03-30T14:02:50.501 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0025157740000000001 2024-03-30T14:02:50.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.501 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.501 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:02:50.501 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.738632+0000", 2024-03-30T14:02:50.501 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000145983 2024-03-30T14:02:50.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.502 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.502 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,7", 2024-03-30T14:02:50.502 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.738791+0000", 2024-03-30T14:02:50.502 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015888999999999999 2024-03-30T14:02:50.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.502 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.502 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:02:50.502 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.741638+0000", 2024-03-30T14:02:50.502 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0028475689999999999 2024-03-30T14:02:50.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.502 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.502 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.502 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.744616+0000", 2024-03-30T14:02:50.502 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0029772689999999998 2024-03-30T14:02:50.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.503 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.503 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.503 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.926765+0000", 2024-03-30T14:02:50.503 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.182149847 2024-03-30T14:02:50.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.503 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.503 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:02:50.503 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.926825+0000", 2024-03-30T14:02:50.503 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0127000000000002e-05 2024-03-30T14:02:50.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.503 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.503 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:02:50.503 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:33.928548+0000", 2024-03-30T14:02:50.503 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0017220340000000001 2024-03-30T14:02:50.503 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.504 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.504 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.504 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.504 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.14451.0:2653 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94764809514240 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-03-30T14:02:50.504 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:35.846463+0000", 2024-03-30T14:02:50.504 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 554.61563278799997, 2024-03-30T14:02:50.504 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.741426648, 2024-03-30T14:02:50.504 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:02:50.504 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:02:50.504 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T14:02:50.504 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:02:50.504 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.14451", 2024-03-30T14:02:50.504 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T14:02:50.505 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 2653 2024-03-30T14:02:50.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.505 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:02:50.505 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.505 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:02:50.505 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.846463+0000", 2024-03-30T14:02:50.505 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.505 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.505 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:02:50.505 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.846463+0000", 2024-03-30T14:02:50.505 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.505 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.506 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:02:50.506 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.846464+0000", 2024-03-30T14:02:50.506 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.322e-06 2024-03-30T14:02:50.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.506 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.506 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:02:50.506 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.846470+0000", 2024-03-30T14:02:50.506 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8449999999999999e-06 2024-03-30T14:02:50.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.506 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.506 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:02:50.506 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.847878+0000", 2024-03-30T14:02:50.506 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0014079959999999999 2024-03-30T14:02:50.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.507 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:02:50.507 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.847890+0000", 2024-03-30T14:02:50.507 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1647999999999999e-05 2024-03-30T14:02:50.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.507 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.507 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.854592+0000", 2024-03-30T14:02:50.507 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.006701762 2024-03-30T14:02:50.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.507 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:02:50.507 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.854603+0000", 2024-03-30T14:02:50.507 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1771e-05 2024-03-30T14:02:50.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.508 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.508 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.557525+0000", 2024-03-30T14:02:50.508 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.702921425 2024-03-30T14:02:50.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.508 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for degraded object", 2024-03-30T14:02:50.508 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.559277+0000", 2024-03-30T14:02:50.508 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001752219 2024-03-30T14:02:50.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.508 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.508 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.562914+0000", 2024-03-30T14:02:50.508 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003636996 2024-03-30T14:02:50.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.509 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:02:50.509 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.563000+0000", 2024-03-30T14:02:50.509 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5944000000000002e-05 2024-03-30T14:02:50.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.509 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:02:50.509 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.587890+0000", 2024-03-30T14:02:50.509 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.024889720000000001 2024-03-30T14:02:50.509 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.509 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.509 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.510 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61235 9.5 9:a316fbe6:::benchmark_data_smithi043_145506_object61234:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e230)", 2024-03-30T14:02:50.510 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:37.375080+0000", 2024-03-30T14:02:50.510 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 553.08701603600002, 2024-03-30T14:02:50.510 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.21255780299999999, 2024-03-30T14:02:50.510 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:02:50.510 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:02:50.510 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:02:50.510 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:02:50.510 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:02:50.510 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:02:50.510 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61235 2024-03-30T14:02:50.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.511 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:02:50.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.511 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:02:50.511 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.375080+0000", 2024-03-30T14:02:50.511 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.511 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:02:50.511 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.375080+0000", 2024-03-30T14:02:50.511 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.511 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:02:50.512 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.375081+0000", 2024-03-30T14:02:50.512 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.482e-06 2024-03-30T14:02:50.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.512 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:02:50.512 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.375104+0000", 2024-03-30T14:02:50.512 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3110000000000001e-05 2024-03-30T14:02:50.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.512 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:02:50.512 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.375116+0000", 2024-03-30T14:02:50.512 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.137e-05 2024-03-30T14:02:50.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.512 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:02:50.513 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.375125+0000", 2024-03-30T14:02:50.513 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5869999999999999e-06 2024-03-30T14:02:50.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.513 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.513 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.559403+0000", 2024-03-30T14:02:50.513 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.18427763699999999 2024-03-30T14:02:50.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.513 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:02:50.513 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.559556+0000", 2024-03-30T14:02:50.513 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015315199999999999 2024-03-30T14:02:50.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.514 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-03-30T14:02:50.514 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.559772+0000", 2024-03-30T14:02:50.514 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000216135 2024-03-30T14:02:50.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.514 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:50.515 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:02:50.515 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.569599+0000", 2024-03-30T14:02:50.516 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0098271369999999997 2024-03-30T14:02:50.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.516 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.516 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.583927+0000", 2024-03-30T14:02:50.516 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.014327555 2024-03-30T14:02:50.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.516 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.516 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.584054+0000", 2024-03-30T14:02:50.516 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012727300000000001 2024-03-30T14:02:50.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.516 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:02:50.516 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.584116+0000", 2024-03-30T14:02:50.517 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1396999999999995e-05 2024-03-30T14:02:50.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.517 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:02:50.517 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.587638+0000", 2024-03-30T14:02:50.517 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0035219679999999999 2024-03-30T14:02:50.517 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.517 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.517 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.517 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61236 9.5 9:ae283ce4:::benchmark_data_smithi043_145506_object61235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e230)", 2024-03-30T14:02:50.517 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:37.384596+0000", 2024-03-30T14:02:50.517 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 553.07749998099996, 2024-03-30T14:02:50.518 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.21038131199999999, 2024-03-30T14:02:50.518 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:02:50.518 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:02:50.518 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:02:50.518 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:02:50.518 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:02:50.518 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:02:50.518 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61236 2024-03-30T14:02:50.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.518 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:02:50.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.518 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:02:50.518 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.384596+0000", 2024-03-30T14:02:50.519 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.519 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:02:50.519 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.384596+0000", 2024-03-30T14:02:50.519 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.519 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:02:50.519 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.384597+0000", 2024-03-30T14:02:50.519 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4810000000000001e-06 2024-03-30T14:02:50.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.519 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:02:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.384619+0000", 2024-03-30T14:02:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1508000000000001e-05 2024-03-30T14:02:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:02:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.384630+0000", 2024-03-30T14:02:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1474000000000001e-05 2024-03-30T14:02:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:02:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.384639+0000", 2024-03-30T14:02:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3259999999999996e-06 2024-03-30T14:02:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.521 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.563297+0000", 2024-03-30T14:02:50.521 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.178658075 2024-03-30T14:02:50.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.521 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:02:50.521 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.563421+0000", 2024-03-30T14:02:50.521 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000123733 2024-03-30T14:02:50.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.521 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-03-30T14:02:50.521 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.563588+0000", 2024-03-30T14:02:50.521 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016715400000000001 2024-03-30T14:02:50.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.522 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:02:50.522 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.569613+0000", 2024-03-30T14:02:50.522 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0060258239999999999 2024-03-30T14:02:50.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.522 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.522 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.583971+0000", 2024-03-30T14:02:50.522 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.014358006 2024-03-30T14:02:50.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.522 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.522 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.587872+0000", 2024-03-30T14:02:50.522 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0039002189999999999 2024-03-30T14:02:50.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.523 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.523 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:02:50.523 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.587934+0000", 2024-03-30T14:02:50.523 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2324999999999997e-05 2024-03-30T14:02:50.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.523 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.523 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:02:50.523 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.594977+0000", 2024-03-30T14:02:50.523 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0070431870000000002 2024-03-30T14:02:50.523 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.523 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.523 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.523 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.524 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:62357 9.9 9:9a6084bc:::benchmark_data_smithi043_145506_object62356:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-30T14:02:50.524 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:39.499871+0000", 2024-03-30T14:02:50.524 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 550.96222494999995, 2024-03-30T14:02:50.524 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.18574223100000001, 2024-03-30T14:02:50.524 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:02:50.524 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:02:50.524 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:02:50.524 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:02:50.524 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:02:50.524 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:02:50.524 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 62357 2024-03-30T14:02:50.524 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.524 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:02:50.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.524 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:02:50.525 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.499871+0000", 2024-03-30T14:02:50.525 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.525 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.525 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:02:50.525 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.499871+0000", 2024-03-30T14:02:50.525 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.525 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.525 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:02:50.525 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.499873+0000", 2024-03-30T14:02:50.525 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.688e-06 2024-03-30T14:02:50.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.525 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.526 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:02:50.526 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.499922+0000", 2024-03-30T14:02:50.526 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9441999999999999e-05 2024-03-30T14:02:50.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.526 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.526 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:02:50.526 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.499932+0000", 2024-03-30T14:02:50.526 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.01e-05 2024-03-30T14:02:50.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.526 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.526 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:02:50.526 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.499939+0000", 2024-03-30T14:02:50.526 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6579999999999996e-06 2024-03-30T14:02:50.527 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.527 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.527 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.527 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.508463+0000", 2024-03-30T14:02:50.527 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0085245210000000002 2024-03-30T14:02:50.527 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.527 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.527 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:02:50.527 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.508580+0000", 2024-03-30T14:02:50.527 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000116755 2024-03-30T14:02:50.527 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.527 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.527 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,6", 2024-03-30T14:02:50.527 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.508720+0000", 2024-03-30T14:02:50.527 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014024999999999999 2024-03-30T14:02:50.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.528 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.528 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.528 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.654170+0000", 2024-03-30T14:02:50.528 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.145449936 2024-03-30T14:02:50.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.528 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.528 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:02:50.528 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.654221+0000", 2024-03-30T14:02:50.528 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0223999999999999e-05 2024-03-30T14:02:50.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.528 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.528 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.528 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.681398+0000", 2024-03-30T14:02:50.529 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.027177903 2024-03-30T14:02:50.529 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.529 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.529 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:02:50.529 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.681511+0000", 2024-03-30T14:02:50.529 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000112841 2024-03-30T14:02:50.529 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.529 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.529 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:02:50.529 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:39.685613+0000", 2024-03-30T14:02:50.529 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0041019129999999996 2024-03-30T14:02:50.529 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.529 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.529 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.530 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.530 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:62906 9.9 9:9cbd581b:::benchmark_data_smithi043_145506_object62905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-30T14:02:50.530 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:40.799969+0000", 2024-03-30T14:02:50.530 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 549.66212653499997, 2024-03-30T14:02:50.530 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17861787100000001, 2024-03-30T14:02:50.530 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:02:50.530 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:02:50.530 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:02:50.530 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:02:50.530 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:02:50.530 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:02:50.530 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 62906 2024-03-30T14:02:50.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.531 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:02:50.531 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.531 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:02:50.531 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.799969+0000", 2024-03-30T14:02:50.531 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.531 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.531 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:02:50.531 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.799969+0000", 2024-03-30T14:02:50.531 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.531 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.531 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:02:50.532 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.799971+0000", 2024-03-30T14:02:50.532 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6470000000000001e-06 2024-03-30T14:02:50.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.532 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.532 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:02:50.532 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.799993+0000", 2024-03-30T14:02:50.532 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1851000000000001e-05 2024-03-30T14:02:50.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.532 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.532 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:02:50.532 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.800005+0000", 2024-03-30T14:02:50.532 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2471e-05 2024-03-30T14:02:50.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.532 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.532 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:02:50.533 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.800014+0000", 2024-03-30T14:02:50.533 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3760000000000008e-06 2024-03-30T14:02:50.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.533 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.533 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.800077+0000", 2024-03-30T14:02:50.533 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2972999999999995e-05 2024-03-30T14:02:50.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.533 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:02:50.533 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.800250+0000", 2024-03-30T14:02:50.533 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017310000000000001 2024-03-30T14:02:50.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.533 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,6", 2024-03-30T14:02:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.800447+0000", 2024-03-30T14:02:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00019700499999999999 2024-03-30T14:02:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:02:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.804124+0000", 2024-03-30T14:02:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003677699 2024-03-30T14:02:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.806781+0000", 2024-03-30T14:02:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026568709999999999 2024-03-30T14:02:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.972322+0000", 2024-03-30T14:02:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.165540771 2024-03-30T14:02:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:02:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.972508+0000", 2024-03-30T14:02:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000186217 2024-03-30T14:02:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:02:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:40.978587+0000", 2024-03-30T14:02:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00607889 2024-03-30T14:02:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:91174 9.5 9:a8f96676:::benchmark_data_smithi043_145506_object4358:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-03-30T14:02:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:54:30.634645+0000", 2024-03-30T14:02:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 499.82745080900003, 2024-03-30T14:02:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.18161228300000001, 2024-03-30T14:02:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:02:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:02:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:02:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:02:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:02:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:02:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 91174 2024-03-30T14:02:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:02:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:02:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.634645+0000", 2024-03-30T14:02:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:02:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.634645+0000", 2024-03-30T14:02:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:02:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.634646+0000", 2024-03-30T14:02:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3570000000000001e-06 2024-03-30T14:02:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:02:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.634652+0000", 2024-03-30T14:02:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7100000000000004e-06 2024-03-30T14:02:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:02:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.634662+0000", 2024-03-30T14:02:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0210000000000001e-05 2024-03-30T14:02:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.539 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:02:50.539 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.634671+0000", 2024-03-30T14:02:50.539 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.32e-06 2024-03-30T14:02:50.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.539 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.539 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.638241+0000", 2024-03-30T14:02:50.539 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003570705 2024-03-30T14:02:50.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.539 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:02:50.539 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.638413+0000", 2024-03-30T14:02:50.539 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000172113 2024-03-30T14:02:50.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.540 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.540 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-03-30T14:02:50.540 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.638635+0000", 2024-03-30T14:02:50.540 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000221599 2024-03-30T14:02:50.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.540 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.540 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:02:50.540 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.643752+0000", 2024-03-30T14:02:50.540 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0051172329999999997 2024-03-30T14:02:50.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.540 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.540 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.540 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.764456+0000", 2024-03-30T14:02:50.540 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.120703541 2024-03-30T14:02:50.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.541 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.541 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.541 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.811923+0000", 2024-03-30T14:02:50.541 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.047467061999999997 2024-03-30T14:02:50.541 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.541 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.541 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:02:50.541 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.812112+0000", 2024-03-30T14:02:50.541 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000189311 2024-03-30T14:02:50.541 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.541 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.541 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:02:50.541 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.816257+0000", 2024-03-30T14:02:50.541 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0041451220000000002 2024-03-30T14:02:50.542 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.542 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.542 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.542 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:91180 9.5 9:a46b4a42:::benchmark_data_smithi043_145506_object4364:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-03-30T14:02:50.542 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:54:30.640558+0000", 2024-03-30T14:02:50.542 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 499.82153837099997, 2024-03-30T14:02:50.542 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17949796500000001, 2024-03-30T14:02:50.542 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:02:50.542 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:02:50.542 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:02:50.542 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:02:50.542 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:02:50.543 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:02:50.543 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 91180 2024-03-30T14:02:50.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.543 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:02:50.543 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.543 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:02:50.543 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.640558+0000", 2024-03-30T14:02:50.543 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.543 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.543 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:02:50.543 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.640558+0000", 2024-03-30T14:02:50.543 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.544 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.544 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:02:50.544 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.640559+0000", 2024-03-30T14:02:50.544 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.451e-06 2024-03-30T14:02:50.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.544 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.544 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:02:50.544 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.640565+0000", 2024-03-30T14:02:50.544 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9229999999999999e-06 2024-03-30T14:02:50.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.544 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.544 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:02:50.544 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.640577+0000", 2024-03-30T14:02:50.544 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1915e-05 2024-03-30T14:02:50.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.545 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:02:50.545 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.640585+0000", 2024-03-30T14:02:50.545 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3140000000000004e-06 2024-03-30T14:02:50.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.545 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.545 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.641047+0000", 2024-03-30T14:02:50.545 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00046140999999999999 2024-03-30T14:02:50.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.545 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:02:50.545 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.641222+0000", 2024-03-30T14:02:50.546 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017528500000000001 2024-03-30T14:02:50.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.546 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-03-30T14:02:50.546 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.641387+0000", 2024-03-30T14:02:50.546 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016495299999999999 2024-03-30T14:02:50.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.546 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:02:50.546 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.644229+0000", 2024-03-30T14:02:50.546 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002842533 2024-03-30T14:02:50.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.547 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.547 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.764511+0000", 2024-03-30T14:02:50.547 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.12028161499999999 2024-03-30T14:02:50.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.547 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.547 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.547 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.816321+0000", 2024-03-30T14:02:50.547 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.051810114999999997 2024-03-30T14:02:50.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.547 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.547 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:02:50.547 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.816394+0000", 2024-03-30T14:02:50.547 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2795999999999999e-05 2024-03-30T14:02:50.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.548 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:02:50.548 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.820055+0000", 2024-03-30T14:02:50.548 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0036616550000000002 2024-03-30T14:02:50.548 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.548 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.548 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.548 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:91182 9.5 9:aaaddeb6:::benchmark_data_smithi043_145506_object4366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-03-30T14:02:50.548 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:54:30.644353+0000", 2024-03-30T14:02:50.548 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 499.81774243500001, 2024-03-30T14:02:50.548 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17937473400000001, 2024-03-30T14:02:50.548 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:02:50.548 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:02:50.549 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:02:50.549 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:02:50.549 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:02:50.549 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:02:50.549 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 91182 2024-03-30T14:02:50.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.549 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:02:50.549 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.549 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:02:50.549 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.644353+0000", 2024-03-30T14:02:50.549 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.549 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.549 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:02:50.550 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.644353+0000", 2024-03-30T14:02:50.550 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.550 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:02:50.550 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.644355+0000", 2024-03-30T14:02:50.550 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.269e-06 2024-03-30T14:02:50.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.550 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:02:50.550 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.644360+0000", 2024-03-30T14:02:50.550 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6200000000000004e-06 2024-03-30T14:02:50.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.551 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:02:50.551 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.644372+0000", 2024-03-30T14:02:50.551 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1250000000000001e-05 2024-03-30T14:02:50.551 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.551 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.551 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:02:50.551 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.644380+0000", 2024-03-30T14:02:50.551 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9510000000000006e-06 2024-03-30T14:02:50.551 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.551 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.551 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.551 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.644460+0000", 2024-03-30T14:02:50.551 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0020999999999998e-05 2024-03-30T14:02:50.551 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.552 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.552 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:02:50.552 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.644630+0000", 2024-03-30T14:02:50.552 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017029499999999999 2024-03-30T14:02:50.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.552 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.552 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-03-30T14:02:50.552 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.644794+0000", 2024-03-30T14:02:50.552 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000163683 2024-03-30T14:02:50.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.552 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.552 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:02:50.552 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.763230+0000", 2024-03-30T14:02:50.552 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.118436718 2024-03-30T14:02:50.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.553 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.553 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.553 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.764563+0000", 2024-03-30T14:02:50.553 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0013326779999999999 2024-03-30T14:02:50.553 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.553 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.553 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.553 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.820132+0000", 2024-03-30T14:02:50.553 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.055569354000000001 2024-03-30T14:02:50.553 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.553 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.553 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:02:50.553 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.820221+0000", 2024-03-30T14:02:50.553 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8987999999999996e-05 2024-03-30T14:02:50.553 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.554 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.554 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:02:50.554 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.823728+0000", 2024-03-30T14:02:50.554 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0035069070000000001 2024-03-30T14:02:50.554 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.554 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.554 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.554 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.554 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.554 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:91184 9.5 9:aacc9fab:::benchmark_data_smithi043_145506_object4368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-03-30T14:02:50.554 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:54:30.646992+0000", 2024-03-30T14:02:50.554 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 499.81510422000002, 2024-03-30T14:02:50.554 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.18059350800000001, 2024-03-30T14:02:50.554 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:02:50.555 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:02:50.555 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:02:50.555 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:02:50.555 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:02:50.555 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:02:50.555 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 91184 2024-03-30T14:02:50.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.555 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:02:50.555 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.555 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:02:50.555 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.646992+0000", 2024-03-30T14:02:50.555 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.555 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.555 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:02:50.556 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.646992+0000", 2024-03-30T14:02:50.556 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.556 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.556 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.556 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:02:50.556 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.646993+0000", 2024-03-30T14:02:50.556 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.469e-06 2024-03-30T14:02:50.556 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.556 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.556 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:02:50.556 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.646999+0000", 2024-03-30T14:02:50.556 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3180000000000003e-06 2024-03-30T14:02:50.556 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.557 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.557 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:02:50.557 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.647012+0000", 2024-03-30T14:02:50.557 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2227e-05 2024-03-30T14:02:50.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.557 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.557 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:02:50.557 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.647021+0000", 2024-03-30T14:02:50.557 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0420000000000008e-06 2024-03-30T14:02:50.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.557 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.557 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.557 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.647193+0000", 2024-03-30T14:02:50.557 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000172151 2024-03-30T14:02:50.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.558 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:02:50.558 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.647361+0000", 2024-03-30T14:02:50.558 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016841 2024-03-30T14:02:50.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.558 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-03-30T14:02:50.558 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.647545+0000", 2024-03-30T14:02:50.558 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000183327 2024-03-30T14:02:50.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.558 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:02:50.559 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.764471+0000", 2024-03-30T14:02:50.559 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.116926523 2024-03-30T14:02:50.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.559 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.559 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.801712+0000", 2024-03-30T14:02:50.559 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.037241104999999997 2024-03-30T14:02:50.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.559 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.559 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.823794+0000", 2024-03-30T14:02:50.559 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.022081997999999999 2024-03-30T14:02:50.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.559 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:02:50.560 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.823883+0000", 2024-03-30T14:02:50.560 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8485000000000004e-05 2024-03-30T14:02:50.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.560 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:02:50.560 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:30.827585+0000", 2024-03-30T14:02:50.560 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0037024530000000001 2024-03-30T14:02:50.560 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.560 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.560 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.560 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:91783 9.5 9:a80d455a:::benchmark_data_smithi043_145506_object4967:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-03-30T14:02:50.560 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:54:31.738265+0000", 2024-03-30T14:02:50.561 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 498.72383137100002, 2024-03-30T14:02:50.561 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.18641157999999999, 2024-03-30T14:02:50.561 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:02:50.561 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:02:50.561 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:02:50.561 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:02:50.561 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:02:50.561 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:02:50.561 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 91783 2024-03-30T14:02:50.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.561 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:02:50.561 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.561 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:02:50.561 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.738265+0000", 2024-03-30T14:02:50.561 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.562 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.562 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:02:50.562 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.738265+0000", 2024-03-30T14:02:50.562 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.562 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.562 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:02:50.562 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.738265+0000", 2024-03-30T14:02:50.562 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2e-07 2024-03-30T14:02:50.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.562 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.562 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:02:50.562 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.738268+0000", 2024-03-30T14:02:50.563 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3780000000000001e-06 2024-03-30T14:02:50.563 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.563 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.563 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:02:50.563 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.738275+0000", 2024-03-30T14:02:50.563 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8120000000000002e-06 2024-03-30T14:02:50.563 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.563 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.563 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:02:50.563 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.738281+0000", 2024-03-30T14:02:50.563 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4160000000000003e-06 2024-03-30T14:02:50.563 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.563 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.563 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.564 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.743467+0000", 2024-03-30T14:02:50.564 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0051866949999999998 2024-03-30T14:02:50.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.564 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:02:50.564 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.743645+0000", 2024-03-30T14:02:50.564 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017815999999999999 2024-03-30T14:02:50.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.564 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-03-30T14:02:50.564 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.743833+0000", 2024-03-30T14:02:50.564 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018794800000000001 2024-03-30T14:02:50.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.565 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:02:50.565 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.753912+0000", 2024-03-30T14:02:50.565 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010078454000000001 2024-03-30T14:02:50.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.565 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.565 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.565 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.771965+0000", 2024-03-30T14:02:50.565 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.018052673000000002 2024-03-30T14:02:50.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.565 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.565 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.565 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.920757+0000", 2024-03-30T14:02:50.565 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.148792118 2024-03-30T14:02:50.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.565 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.566 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:02:50.566 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.920885+0000", 2024-03-30T14:02:50.566 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000128071 2024-03-30T14:02:50.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.566 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.566 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:02:50.566 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:31.924676+0000", 2024-03-30T14:02:50.566 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0037913349999999998 2024-03-30T14:02:50.566 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.566 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.566 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.566 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.566 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:94772 9.5 9:ad9e1d45:::benchmark_data_smithi043_145506_object7956:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-03-30T14:02:50.567 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:54:36.976455+0000", 2024-03-30T14:02:50.567 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 493.48564123, 2024-03-30T14:02:50.567 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.178140466, 2024-03-30T14:02:50.567 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:02:50.567 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:02:50.567 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:02:50.567 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:02:50.567 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:02:50.567 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:02:50.567 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 94772 2024-03-30T14:02:50.567 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.567 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:02:50.567 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.567 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:02:50.568 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.976455+0000", 2024-03-30T14:02:50.568 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.568 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:02:50.568 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.976455+0000", 2024-03-30T14:02:50.568 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.568 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:02:50.568 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.976456+0000", 2024-03-30T14:02:50.568 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3400000000000001e-06 2024-03-30T14:02:50.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.569 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:02:50.569 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.976462+0000", 2024-03-30T14:02:50.569 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7610000000000004e-06 2024-03-30T14:02:50.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.569 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:02:50.569 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.976476+0000", 2024-03-30T14:02:50.569 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4258e-05 2024-03-30T14:02:50.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.570 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:02:50.570 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.976487+0000", 2024-03-30T14:02:50.570 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0635000000000001e-05 2024-03-30T14:02:50.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.571 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.571 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.985270+0000", 2024-03-30T14:02:50.571 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0087838029999999997 2024-03-30T14:02:50.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.571 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:02:50.571 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.985417+0000", 2024-03-30T14:02:50.571 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000146546 2024-03-30T14:02:50.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.571 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-03-30T14:02:50.571 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.985571+0000", 2024-03-30T14:02:50.572 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015392499999999999 2024-03-30T14:02:50.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.572 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.572 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:02:50.572 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:36.987537+0000", 2024-03-30T14:02:50.572 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0019664919999999998 2024-03-30T14:02:50.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.572 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.572 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.572 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.141591+0000", 2024-03-30T14:02:50.572 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.154053784 2024-03-30T14:02:50.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.572 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.573 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.573 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.151002+0000", 2024-03-30T14:02:50.573 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0094105219999999993 2024-03-30T14:02:50.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.573 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.573 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:02:50.573 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.151086+0000", 2024-03-30T14:02:50.573 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4689000000000002e-05 2024-03-30T14:02:50.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.573 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.573 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:02:50.573 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:54:37.154595+0000", 2024-03-30T14:02:50.573 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0035087109999999999 2024-03-30T14:02:50.573 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.574 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.574 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.574 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.574 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:127718 9.5 9:a4801cd1:::benchmark_data_smithi043_145506_object40902:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-03-30T14:02:50.574 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:55:35.205939+0000", 2024-03-30T14:02:50.574 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 435.256156707, 2024-03-30T14:02:50.574 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.18288607000000001, 2024-03-30T14:02:50.574 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:02:50.574 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:02:50.574 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:02:50.574 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:02:50.574 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:02:50.574 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:02:50.575 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 127718 2024-03-30T14:02:50.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.575 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:02:50.575 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.575 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:02:50.575 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:35.205939+0000", 2024-03-30T14:02:50.575 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.575 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.575 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:02:50.575 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:35.205939+0000", 2024-03-30T14:02:50.575 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.575 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.576 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:02:50.576 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:35.205940+0000", 2024-03-30T14:02:50.576 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7300000000000002e-07 2024-03-30T14:02:50.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.576 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.576 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:02:50.576 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:35.205942+0000", 2024-03-30T14:02:50.576 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.6989999999999999e-06 2024-03-30T14:02:50.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.576 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.576 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:02:50.576 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:35.205948+0000", 2024-03-30T14:02:50.576 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1070000000000001e-06 2024-03-30T14:02:50.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.576 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.577 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:02:50.577 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:35.205954+0000", 2024-03-30T14:02:50.577 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3009999999999997e-06 2024-03-30T14:02:50.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.577 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.577 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.577 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:35.217419+0000", 2024-03-30T14:02:50.577 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011465259 2024-03-30T14:02:50.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.577 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.577 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for degraded object", 2024-03-30T14:02:50.577 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:35.217492+0000", 2024-03-30T14:02:50.577 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3089000000000004e-05 2024-03-30T14:02:50.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.578 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.578 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.578 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:35.289371+0000", 2024-03-30T14:02:50.578 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.071878683999999998 2024-03-30T14:02:50.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.578 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.578 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:02:50.578 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:35.289472+0000", 2024-03-30T14:02:50.578 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010092800000000001 2024-03-30T14:02:50.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.578 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.578 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-03-30T14:02:50.578 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:35.289626+0000", 2024-03-30T14:02:50.578 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015479600000000001 2024-03-30T14:02:50.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.579 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.579 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.579 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:35.307172+0000", 2024-03-30T14:02:50.579 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.017545233 2024-03-30T14:02:50.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.579 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.579 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.579 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:35.307223+0000", 2024-03-30T14:02:50.579 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1069000000000002e-05 2024-03-30T14:02:50.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.579 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.579 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:02:50.579 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:35.387054+0000", 2024-03-30T14:02:50.580 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.079831764999999999 2024-03-30T14:02:50.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.580 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:02:50.580 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:35.387232+0000", 2024-03-30T14:02:50.580 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000177257 2024-03-30T14:02:50.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.580 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:02:50.580 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:35.388825+0000", 2024-03-30T14:02:50.580 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00159351 2024-03-30T14:02:50.580 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.580 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.580 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.581 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:128328 9.9 9:923d99ee:::benchmark_data_smithi043_145506_object41512:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-03-30T14:02:50.581 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:55:36.136971+0000", 2024-03-30T14:02:50.581 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 434.32512458600002, 2024-03-30T14:02:50.581 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.18076672299999999, 2024-03-30T14:02:50.581 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:02:50.581 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:02:50.581 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:02:50.581 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:02:50.581 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:02:50.581 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:02:50.581 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 128328 2024-03-30T14:02:50.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.581 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:02:50.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.582 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:02:50.582 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:36.136971+0000", 2024-03-30T14:02:50.582 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.582 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:02:50.582 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:36.136971+0000", 2024-03-30T14:02:50.582 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:02:50.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.582 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:02:50.582 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:36.136972+0000", 2024-03-30T14:02:50.582 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.167e-06 2024-03-30T14:02:50.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.583 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:02:50.583 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:36.136978+0000", 2024-03-30T14:02:50.583 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3940000000000004e-06 2024-03-30T14:02:50.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.583 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:02:50.583 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:36.136989+0000", 2024-03-30T14:02:50.583 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1151999999999999e-05 2024-03-30T14:02:50.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.583 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:02:50.583 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:36.136997+0000", 2024-03-30T14:02:50.583 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0930000000000003e-06 2024-03-30T14:02:50.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.584 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.584 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:36.139316+0000", 2024-03-30T14:02:50.584 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0023187099999999999 2024-03-30T14:02:50.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.584 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for degraded object", 2024-03-30T14:02:50.584 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:36.139909+0000", 2024-03-30T14:02:50.584 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00059336299999999996 2024-03-30T14:02:50.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.584 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:02:50.584 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:36.205339+0000", 2024-03-30T14:02:50.585 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.065430184000000002 2024-03-30T14:02:50.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.585 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:02:50.585 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:36.205437+0000", 2024-03-30T14:02:50.585 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7312999999999995e-05 2024-03-30T14:02:50.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.585 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,6", 2024-03-30T14:02:50.585 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:36.205617+0000", 2024-03-30T14:02:50.585 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017988300000000001 2024-03-30T14:02:50.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.585 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.585 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:36.216728+0000", 2024-03-30T14:02:50.586 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011111004000000001 2024-03-30T14:02:50.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.586 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:02:50.586 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:36.296945+0000", 2024-03-30T14:02:50.586 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.080217519000000001 2024-03-30T14:02:50.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.586 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:02:50.586 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:36.316004+0000", 2024-03-30T14:02:50.586 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.019058553999999998 2024-03-30T14:02:50.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.586 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:02:50.587 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:36.316058+0000", 2024-03-30T14:02:50.587 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4279e-05 2024-03-30T14:02:50.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:02:50.587 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:02:50.587 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:02:50.587 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:55:36.317738+0000", 2024-03-30T14:02:50.587 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001680108 2024-03-30T14:02:50.587 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.587 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.587 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.587 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:02:50.587 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:02:50.587 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T14:02:50.597 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:50.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:50.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:50.698 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:50.732 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:50.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:50.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:50.834 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:50.860 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:50.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:50.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:50.963 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:50.993 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:50.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:50.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:51.071 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Total time run: 90.3015 2024-03-30T14:02:51.071 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Total writes made: 110115 2024-03-30T14:02:51.071 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Write size: 65536 2024-03-30T14:02:51.071 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Object size: 65536 2024-03-30T14:02:51.071 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Bandwidth (MB/sec): 76.2134 2024-03-30T14:02:51.071 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev Bandwidth: 37.0341 2024-03-30T14:02:51.071 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max bandwidth (MB/sec): 234.312 2024-03-30T14:02:51.071 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min bandwidth (MB/sec): 0 2024-03-30T14:02:51.071 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Average IOPS: 1219 2024-03-30T14:02:51.072 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev IOPS: 592.546 2024-03-30T14:02:51.072 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max IOPS: 3749 2024-03-30T14:02:51.072 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min IOPS: 0 2024-03-30T14:02:51.072 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Average Latency(s): 0.0131026 2024-03-30T14:02:51.072 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev Latency(s): 0.0438189 2024-03-30T14:02:51.072 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max latency(s): 2.26481 2024-03-30T14:02:51.072 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min latency(s): 0.00121029 2024-03-30T14:02:51.078 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Cleaning up (deleting benchmark objects) 2024-03-30T14:02:51.096 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:51.130 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:51.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:51.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:51.232 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:51.265 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:51.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:51.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:51.367 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[29963]: 3.17 scrub starts 2024-03-30T14:02:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[29963]: 3.17 scrub ok 2024-03-30T14:02:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[29963]: pgmap v921: 168 pgs: 48 peering, 6 unknown, 1 active+clean+scrubbing, 113 active+clean; 6.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 396 B/s rd, 67 MiB/s wr, 1.07k op/s; 27 KiB/s, 0 objects/s recovering 2024-03-30T14:02:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[29963]: 11.b starting backfill to osd.4 from (0'0,0'0] MAX to 416'6864 2024-03-30T14:02:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[29963]: 11.0 starting backfill to osd.0 from (0'0,0'0] MAX to 416'6822 2024-03-30T14:02:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[29963]: 11.2 starting backfill to osd.1 from (0'0,0'0] MAX to 416'6823 2024-03-30T14:02:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[29963]: 11.9 starting backfill to osd.1 from (0'0,0'0] MAX to 416'6995 2024-03-30T14:02:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[29963]: 11.8 starting backfill to osd.0 from (0'0,0'0] MAX to 416'6906 2024-03-30T14:02:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[29963]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 416'6883 2024-03-30T14:02:51.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[29963]: 11.5 starting backfill to osd.0 from (0'0,0'0] MAX to 416'6891 2024-03-30T14:02:51.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[29963]: Health check failed: Reduced data availability: 20 pgs inactive, 19 pgs peering (PG_AVAILABILITY) 2024-03-30T14:02:51.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[29963]: 11.1 starting backfill to osd.2 from (0'0,0'0] MAX to 416'6836 2024-03-30T14:02:51.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:51.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[42099]: 3.17 scrub starts 2024-03-30T14:02:51.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[42099]: 3.17 scrub ok 2024-03-30T14:02:51.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[42099]: pgmap v921: 168 pgs: 48 peering, 6 unknown, 1 active+clean+scrubbing, 113 active+clean; 6.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 396 B/s rd, 67 MiB/s wr, 1.07k op/s; 27 KiB/s, 0 objects/s recovering 2024-03-30T14:02:51.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[42099]: 11.b starting backfill to osd.4 from (0'0,0'0] MAX to 416'6864 2024-03-30T14:02:51.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[42099]: 11.0 starting backfill to osd.0 from (0'0,0'0] MAX to 416'6822 2024-03-30T14:02:51.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[42099]: 11.2 starting backfill to osd.1 from (0'0,0'0] MAX to 416'6823 2024-03-30T14:02:51.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[42099]: 11.9 starting backfill to osd.1 from (0'0,0'0] MAX to 416'6995 2024-03-30T14:02:51.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[42099]: 11.8 starting backfill to osd.0 from (0'0,0'0] MAX to 416'6906 2024-03-30T14:02:51.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[42099]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 416'6883 2024-03-30T14:02:51.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[42099]: 11.5 starting backfill to osd.0 from (0'0,0'0] MAX to 416'6891 2024-03-30T14:02:51.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[42099]: Health check failed: Reduced data availability: 20 pgs inactive, 19 pgs peering (PG_AVAILABILITY) 2024-03-30T14:02:51.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:51 smithi043 ceph-mon[42099]: 11.1 starting backfill to osd.2 from (0'0,0'0] MAX to 416'6836 2024-03-30T14:02:51.400 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:51.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:51.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:51.449 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:51 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:51.449 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:51 smithi113 ceph-mon[34341]: 3.17 scrub starts 2024-03-30T14:02:51.449 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:51 smithi113 ceph-mon[34341]: 3.17 scrub ok 2024-03-30T14:02:51.450 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:51 smithi113 ceph-mon[34341]: pgmap v921: 168 pgs: 48 peering, 6 unknown, 1 active+clean+scrubbing, 113 active+clean; 6.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 396 B/s rd, 67 MiB/s wr, 1.07k op/s; 27 KiB/s, 0 objects/s recovering 2024-03-30T14:02:51.450 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:51 smithi113 ceph-mon[34341]: 11.b starting backfill to osd.4 from (0'0,0'0] MAX to 416'6864 2024-03-30T14:02:51.450 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:51 smithi113 ceph-mon[34341]: 11.0 starting backfill to osd.0 from (0'0,0'0] MAX to 416'6822 2024-03-30T14:02:51.450 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:51 smithi113 ceph-mon[34341]: 11.2 starting backfill to osd.1 from (0'0,0'0] MAX to 416'6823 2024-03-30T14:02:51.450 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:51 smithi113 ceph-mon[34341]: 11.9 starting backfill to osd.1 from (0'0,0'0] MAX to 416'6995 2024-03-30T14:02:51.450 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:51 smithi113 ceph-mon[34341]: 11.8 starting backfill to osd.0 from (0'0,0'0] MAX to 416'6906 2024-03-30T14:02:51.450 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:51 smithi113 ceph-mon[34341]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 416'6883 2024-03-30T14:02:51.450 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:51 smithi113 ceph-mon[34341]: 11.5 starting backfill to osd.0 from (0'0,0'0] MAX to 416'6891 2024-03-30T14:02:51.450 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:51 smithi113 ceph-mon[34341]: Health check failed: Reduced data availability: 20 pgs inactive, 19 pgs peering (PG_AVAILABILITY) 2024-03-30T14:02:51.450 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:51 smithi113 ceph-mon[34341]: 11.1 starting backfill to osd.2 from (0'0,0'0] MAX to 416'6836 2024-03-30T14:02:51.502 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:51.535 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:51.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:51.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:51.637 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:51.684 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:51.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:51.786 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:51.832 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:51.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:51.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:51.934 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:51.973 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:51.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:51.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:52.078 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:52.108 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:52.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:52.210 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:52.238 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:52.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:52.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:52.340 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:52.366 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:52.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:52.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:52.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:52 smithi043 ceph-mon[42099]: osdmap e420: 8 total, 8 up, 6 in 2024-03-30T14:02:52.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:52 smithi043 ceph-mon[42099]: 6.1e scrub starts 2024-03-30T14:02:52.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:52 smithi043 ceph-mon[42099]: 6.1e scrub ok 2024-03-30T14:02:52.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:52 smithi043 ceph-mon[29963]: osdmap e420: 8 total, 8 up, 6 in 2024-03-30T14:02:52.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:52 smithi043 ceph-mon[29963]: 6.1e scrub starts 2024-03-30T14:02:52.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:52 smithi043 ceph-mon[29963]: 6.1e scrub ok 2024-03-30T14:02:52.420 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-30T14:02:52.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:52 smithi113 ceph-mon[34341]: osdmap e420: 8 total, 8 up, 6 in 2024-03-30T14:02:52.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:52 smithi113 ceph-mon[34341]: 6.1e scrub starts 2024-03-30T14:02:52.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:52 smithi113 ceph-mon[34341]: 6.1e scrub ok 2024-03-30T14:02:52.468 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:52.509 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:52.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:52.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:52.610 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:52.636 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:52.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:52.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:52.737 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:52.786 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:52.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:52.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:52.888 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:52.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:52.935 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:52.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:53.036 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:53.080 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:53.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:53.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:53.141 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T14:02:53.183 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:53.189 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T14:02:53.211 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:53.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:53.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:53.244 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T14:02:53.293 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T14:02:53.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:53 smithi043 ceph-mon[29963]: 3.2 scrub starts 2024-03-30T14:02:53.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:53 smithi043 ceph-mon[29963]: 3.2 scrub ok 2024-03-30T14:02:53.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:53 smithi043 ceph-mon[29963]: pgmap v924: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 49 peering, 6 unknown, 110 active+clean; 6.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 255 B/s rd, 65 MiB/s wr, 1.07k op/s; 20027/330576 objects misplaced (6.058%); 7.0 MiB/s, 111 objects/s recovering 2024-03-30T14:02:53.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:53 smithi043 ceph-mon[29963]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T14:02:53.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:53 smithi043 ceph-mon[29963]: osdmap e421: 8 total, 8 up, 6 in 2024-03-30T14:02:53.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:53 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3371488629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:02:53.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:53 smithi043 ceph-mon[42099]: 3.2 scrub starts 2024-03-30T14:02:53.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:53 smithi043 ceph-mon[42099]: 3.2 scrub ok 2024-03-30T14:02:53.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:53 smithi043 ceph-mon[42099]: pgmap v924: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 49 peering, 6 unknown, 110 active+clean; 6.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 255 B/s rd, 65 MiB/s wr, 1.07k op/s; 20027/330576 objects misplaced (6.058%); 7.0 MiB/s, 111 objects/s recovering 2024-03-30T14:02:53.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:53 smithi043 ceph-mon[42099]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T14:02:53.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:53 smithi043 ceph-mon[42099]: osdmap e421: 8 total, 8 up, 6 in 2024-03-30T14:02:53.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:53 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3371488629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:02:53.313 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:53.339 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T14:02:53.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:53.365 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:53.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:53.383 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T14:02:53.424 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T14:02:53.456 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T14:02:53.459 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:02:53.459 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:02:53.459 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:02:53.460 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:02:53.460 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:02:53.460 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:02:53.460 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:02:53.460 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:02:53.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:53 smithi113 ceph-mon[34341]: 3.2 scrub starts 2024-03-30T14:02:53.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:53 smithi113 ceph-mon[34341]: 3.2 scrub ok 2024-03-30T14:02:53.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:53 smithi113 ceph-mon[34341]: pgmap v924: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 49 peering, 6 unknown, 110 active+clean; 6.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 255 B/s rd, 65 MiB/s wr, 1.07k op/s; 20027/330576 objects misplaced (6.058%); 7.0 MiB/s, 111 objects/s recovering 2024-03-30T14:02:53.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:53 smithi113 ceph-mon[34341]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T14:02:53.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:53 smithi113 ceph-mon[34341]: osdmap e421: 8 total, 8 up, 6 in 2024-03-30T14:02:53.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:53 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3371488629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:02:53.466 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:53.490 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:53.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:53.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:53.592 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:53.624 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:53.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:53.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:53.726 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:53.769 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:53.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:53.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:53.871 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:53.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:53.906 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:53.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:54.007 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:54.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:54.044 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:54.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:54.145 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:54.174 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:54.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:54.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:54.277 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:54.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:54.309 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:54.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:54.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:54 smithi043 ceph-mon[29963]: 1.4 scrub starts 2024-03-30T14:02:54.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:54 smithi043 ceph-mon[29963]: 1.4 scrub ok 2024-03-30T14:02:54.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:54 smithi043 ceph-mon[42099]: 1.4 scrub starts 2024-03-30T14:02:54.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:54 smithi043 ceph-mon[42099]: 1.4 scrub ok 2024-03-30T14:02:54.410 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:54.440 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:54.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:54.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:54.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:54 smithi113 ceph-mon[34341]: 1.4 scrub starts 2024-03-30T14:02:54.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:54 smithi113 ceph-mon[34341]: 1.4 scrub ok 2024-03-30T14:02:54.483 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 5] out_osds: [7, 3] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-03-30T14:02:54.483 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T14:02:54.483 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-30T14:02:54.483 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-03-30T14:02:54.543 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:54.574 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:54.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:54.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:54.676 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:54.725 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:54.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:54.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:54.827 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:54.857 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:54.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:54.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:54.959 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:54.988 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:54.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:54.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:55.090 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:55.122 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:55.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:55.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:55.224 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:55.260 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:55.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:55.362 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:55 smithi043 ceph-mon[29963]: pgmap v926: 168 pgs: 3 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 30 peering, 4 unknown, 130 active+clean; 6.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 5.7 KiB/s rd, 15 MiB/s wr, 325 op/s; 26761/329766 objects misplaced (8.115%); 5.9 MiB/s, 104 objects/s recovering 2024-03-30T14:02:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:55 smithi043 ceph-mon[29963]: 4.c scrub starts 2024-03-30T14:02:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:55 smithi043 ceph-mon[29963]: 4.c scrub ok 2024-03-30T14:02:55.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:55 smithi043 ceph-mon[42099]: pgmap v926: 168 pgs: 3 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 30 peering, 4 unknown, 130 active+clean; 6.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 5.7 KiB/s rd, 15 MiB/s wr, 325 op/s; 26761/329766 objects misplaced (8.115%); 5.9 MiB/s, 104 objects/s recovering 2024-03-30T14:02:55.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:55 smithi043 ceph-mon[42099]: 4.c scrub starts 2024-03-30T14:02:55.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:55 smithi043 ceph-mon[42099]: 4.c scrub ok 2024-03-30T14:02:55.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:55.394 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:55.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:55.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:55 smithi113 ceph-mon[34341]: pgmap v926: 168 pgs: 3 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 30 peering, 4 unknown, 130 active+clean; 6.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 5.7 KiB/s rd, 15 MiB/s wr, 325 op/s; 26761/329766 objects misplaced (8.115%); 5.9 MiB/s, 104 objects/s recovering 2024-03-30T14:02:55.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:55 smithi113 ceph-mon[34341]: 4.c scrub starts 2024-03-30T14:02:55.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:55 smithi113 ceph-mon[34341]: 4.c scrub ok 2024-03-30T14:02:55.495 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:55.534 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:55.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:55.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:55.636 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:55.667 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:55.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:55.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:55.769 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:55.802 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:55.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:55.903 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:55.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:55.928 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:55.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:56.029 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:56.059 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:56.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:56.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:56.161 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:56.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:56.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:56.305 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:56.334 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:56.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:56.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:56.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:56 smithi043 ceph-mon[29963]: 5.f scrub starts 2024-03-30T14:02:56.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:56 smithi043 ceph-mon[29963]: 5.f scrub ok 2024-03-30T14:02:56.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:56 smithi043 ceph-mon[29963]: 5.e scrub starts 2024-03-30T14:02:56.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:56 smithi043 ceph-mon[29963]: 5.e scrub ok 2024-03-30T14:02:56.392 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:02:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:02:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:02:56.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:56 smithi043 ceph-mon[42099]: 5.f scrub starts 2024-03-30T14:02:56.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:56 smithi043 ceph-mon[42099]: 5.f scrub ok 2024-03-30T14:02:56.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:56 smithi043 ceph-mon[42099]: 5.e scrub starts 2024-03-30T14:02:56.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:56 smithi043 ceph-mon[42099]: 5.e scrub ok 2024-03-30T14:02:56.436 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:56.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:56 smithi113 ceph-mon[34341]: 5.f scrub starts 2024-03-30T14:02:56.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:56 smithi113 ceph-mon[34341]: 5.f scrub ok 2024-03-30T14:02:56.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:56 smithi113 ceph-mon[34341]: 5.e scrub starts 2024-03-30T14:02:56.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:56 smithi113 ceph-mon[34341]: 5.e scrub ok 2024-03-30T14:02:56.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:56.466 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:56.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:56.567 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:56.597 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:56.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:56.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:56.699 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:56.729 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:56.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:56.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:56.831 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:56.861 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:56.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:56.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:56.964 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:57.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:57.012 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:57.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:57.113 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:57.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:57.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:57.266 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:02:57.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:57.319 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:57.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:02:57.329 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:57 smithi043 ceph-mon[29963]: 4.7 scrub starts 2024-03-30T14:02:57.329 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:57 smithi043 ceph-mon[29963]: 4.7 scrub ok 2024-03-30T14:02:57.329 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:57 smithi043 ceph-mon[29963]: pgmap v927: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 peering, 159 active+clean; 6.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 16 KiB/s rd, 12 MiB/s wr, 460 op/s; 51565/326448 objects misplaced (15.796%); 28 MiB/s, 463 objects/s recovering 2024-03-30T14:02:57.329 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:57 smithi043 ceph-mon[29963]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 14 pgs peering) 2024-03-30T14:02:57.330 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:57 smithi043 ceph-mon[29963]: 4.e scrub starts 2024-03-30T14:02:57.330 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:57 smithi043 ceph-mon[29963]: 4.e scrub ok 2024-03-30T14:02:57.330 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:57 smithi043 ceph-mon[29963]: 6.1c scrub starts 2024-03-30T14:02:57.330 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:57 smithi043 ceph-mon[29963]: 6.1c scrub ok 2024-03-30T14:02:57.330 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:57 smithi043 ceph-mon[42099]: 4.7 scrub starts 2024-03-30T14:02:57.330 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:57 smithi043 ceph-mon[42099]: 4.7 scrub ok 2024-03-30T14:02:57.330 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:57 smithi043 ceph-mon[42099]: pgmap v927: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 peering, 159 active+clean; 6.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 16 KiB/s rd, 12 MiB/s wr, 460 op/s; 51565/326448 objects misplaced (15.796%); 28 MiB/s, 463 objects/s recovering 2024-03-30T14:02:57.330 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:57 smithi043 ceph-mon[42099]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 14 pgs peering) 2024-03-30T14:02:57.330 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:57 smithi043 ceph-mon[42099]: 4.e scrub starts 2024-03-30T14:02:57.330 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:57 smithi043 ceph-mon[42099]: 4.e scrub ok 2024-03-30T14:02:57.330 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:57 smithi043 ceph-mon[42099]: 6.1c scrub starts 2024-03-30T14:02:57.331 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:57 smithi043 ceph-mon[42099]: 6.1c scrub ok 2024-03-30T14:02:57.420 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:57.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:57 smithi113 ceph-mon[34341]: 4.7 scrub starts 2024-03-30T14:02:57.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:57 smithi113 ceph-mon[34341]: 4.7 scrub ok 2024-03-30T14:02:57.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:57 smithi113 ceph-mon[34341]: pgmap v927: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 peering, 159 active+clean; 6.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 16 KiB/s rd, 12 MiB/s wr, 460 op/s; 51565/326448 objects misplaced (15.796%); 28 MiB/s, 463 objects/s recovering 2024-03-30T14:02:57.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:57 smithi113 ceph-mon[34341]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 14 pgs peering) 2024-03-30T14:02:57.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:57 smithi113 ceph-mon[34341]: 4.e scrub starts 2024-03-30T14:02:57.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:57 smithi113 ceph-mon[34341]: 4.e scrub ok 2024-03-30T14:02:57.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:57 smithi113 ceph-mon[34341]: 6.1c scrub starts 2024-03-30T14:02:57.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:57 smithi113 ceph-mon[34341]: 6.1c scrub ok 2024-03-30T14:02:57.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:57.472 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:57.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:57.573 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:57.605 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:57.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:57.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:57.707 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:57.739 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:57.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:57.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:57.825 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:02:57.840 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:57.869 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:57.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:57.971 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:58.001 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:58.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:58.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:58.103 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:02:58.149 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:58.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:58.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:02:58.250 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:58.277 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:58.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:58.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:58.313 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:58 smithi043 ceph-mon[29963]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T14:02:58.314 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:58 smithi043 ceph-mon[42099]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T14:02:58.378 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:58.416 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:58.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:58.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:58.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:58 smithi113 ceph-mon[34341]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T14:02:58.523 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:58.568 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:58.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:58.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:58.670 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:02:58.704 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:58.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:58.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:02:58.806 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:58.844 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:58.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:58.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:58.946 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:02:58.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:58.979 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:58.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:02:59.080 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:02:59.093 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:02:59.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:59.128 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:59.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:02:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-03-30T14:02:59.229 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:59.260 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:59.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:59.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:59.362 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:59.411 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:59.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:59.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:59.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:59 smithi113 ceph-mon[34341]: pgmap v928: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 6.5 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 32 KiB/s rd, 9.0 MiB/s wr, 610 op/s; 49921/321414 objects misplaced (15.532%); 29 MiB/s, 473 objects/s recovering 2024-03-30T14:02:59.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:59 smithi113 ceph-mon[34341]: 3.14 scrub starts 2024-03-30T14:02:59.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:59 smithi113 ceph-mon[34341]: 3.14 scrub ok 2024-03-30T14:02:59.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:02:59 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2284380025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:02:59.513 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:59.544 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:59.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:59.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:59.562 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:59 smithi043 ceph-mon[29963]: pgmap v928: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 6.5 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 32 KiB/s rd, 9.0 MiB/s wr, 610 op/s; 49921/321414 objects misplaced (15.532%); 29 MiB/s, 473 objects/s recovering 2024-03-30T14:02:59.563 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:59 smithi043 ceph-mon[29963]: 3.14 scrub starts 2024-03-30T14:02:59.563 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:59 smithi043 ceph-mon[29963]: 3.14 scrub ok 2024-03-30T14:02:59.563 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:02:59 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2284380025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:02:59.563 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:59 smithi043 ceph-mon[42099]: pgmap v928: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 6.5 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 32 KiB/s rd, 9.0 MiB/s wr, 610 op/s; 49921/321414 objects misplaced (15.532%); 29 MiB/s, 473 objects/s recovering 2024-03-30T14:02:59.563 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:59 smithi043 ceph-mon[42099]: 3.14 scrub starts 2024-03-30T14:02:59.563 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:59 smithi043 ceph-mon[42099]: 3.14 scrub ok 2024-03-30T14:02:59.563 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:02:59 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2284380025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:02:59.646 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:02:59.696 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:02:59.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:59.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:02:59.798 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:02:59.826 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:59.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:59.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:02:59.927 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:02:59.958 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:02:59.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:02:59.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:02:59.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:02:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:03:00.061 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:00.105 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:00.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:00.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:00.207 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:00.234 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:00.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:00.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:00.336 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:00.360 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T14:03:00.375 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:00.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:00.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:00.423 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:00 smithi043 ceph-mon[29963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:03:00.423 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:00 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2284380025' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:03:00.424 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:00 smithi043 ceph-mon[29963]: osdmap e422: 8 total, 8 up, 6 in 2024-03-30T14:03:00.424 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:00 smithi043 ceph-mon[29963]: 4.9 scrub starts 2024-03-30T14:03:00.424 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:00 smithi043 ceph-mon[29963]: 4.9 scrub ok 2024-03-30T14:03:00.424 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:00 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:03:00.424 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:00 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:00.424 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:00 smithi043 ceph-mon[42099]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:03:00.424 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:00 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2284380025' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:03:00.424 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:00 smithi043 ceph-mon[42099]: osdmap e422: 8 total, 8 up, 6 in 2024-03-30T14:03:00.424 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:00 smithi043 ceph-mon[42099]: 4.9 scrub starts 2024-03-30T14:03:00.424 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:00 smithi043 ceph-mon[42099]: 4.9 scrub ok 2024-03-30T14:03:00.424 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:00 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:03:00.424 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:00 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:00 smithi113 ceph-mon[34341]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:03:00.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:00 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2284380025' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:03:00.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:00 smithi113 ceph-mon[34341]: osdmap e422: 8 total, 8 up, 6 in 2024-03-30T14:03:00.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:00 smithi113 ceph-mon[34341]: 4.9 scrub starts 2024-03-30T14:03:00.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:00 smithi113 ceph-mon[34341]: 4.9 scrub ok 2024-03-30T14:03:00.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:00 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:03:00.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:00 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:00.477 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:00.540 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:00.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:00.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:00.642 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:00.669 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:00.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:00.771 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:00.804 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:00.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:00.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:00.906 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:00.952 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:00.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:00.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:01.054 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:01.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:01.086 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:01.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:01.187 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:01.219 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:01.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:01.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:01.321 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:01.359 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:01.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:01.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:01.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:01 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:01.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:01 smithi043 ceph-mon[29963]: pgmap v930: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 6.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 54 KiB/s rd, 72 KiB/s wr, 913 op/s; 47001/310863 objects misplaced (15.120%); 35 MiB/s, 561 objects/s recovering 2024-03-30T14:03:01.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:01 smithi043 ceph-mon[29963]: 4.1b scrub starts 2024-03-30T14:03:01.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:01 smithi043 ceph-mon[29963]: 4.1b scrub ok 2024-03-30T14:03:01.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:01 smithi043 ceph-mon[29963]: 3.1f scrub starts 2024-03-30T14:03:01.364 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:01 smithi043 ceph-mon[29963]: 3.1f scrub ok 2024-03-30T14:03:01.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:01 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:01.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:01 smithi043 ceph-mon[42099]: pgmap v930: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 6.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 54 KiB/s rd, 72 KiB/s wr, 913 op/s; 47001/310863 objects misplaced (15.120%); 35 MiB/s, 561 objects/s recovering 2024-03-30T14:03:01.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:01 smithi043 ceph-mon[42099]: 4.1b scrub starts 2024-03-30T14:03:01.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:01 smithi043 ceph-mon[42099]: 4.1b scrub ok 2024-03-30T14:03:01.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:01 smithi043 ceph-mon[42099]: 3.1f scrub starts 2024-03-30T14:03:01.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:01 smithi043 ceph-mon[42099]: 3.1f scrub ok 2024-03-30T14:03:01.460 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:01.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:01 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:01.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:01 smithi113 ceph-mon[34341]: pgmap v930: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 6.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 54 KiB/s rd, 72 KiB/s wr, 913 op/s; 47001/310863 objects misplaced (15.120%); 35 MiB/s, 561 objects/s recovering 2024-03-30T14:03:01.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:01 smithi113 ceph-mon[34341]: 4.1b scrub starts 2024-03-30T14:03:01.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:01 smithi113 ceph-mon[34341]: 4.1b scrub ok 2024-03-30T14:03:01.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:01 smithi113 ceph-mon[34341]: 3.1f scrub starts 2024-03-30T14:03:01.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:01 smithi113 ceph-mon[34341]: 3.1f scrub ok 2024-03-30T14:03:01.506 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:01.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:01.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:01.608 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:01.661 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:01.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:01.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:01.763 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:01.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:01.818 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:01.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:01.919 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:01.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:01.973 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:01.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:02.074 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:02.117 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:02.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:02.219 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:02.245 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:02.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:02.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:02.347 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:02.356 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:02 smithi043 ceph-mon[29963]: 3.f scrub starts 2024-03-30T14:03:02.356 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:02 smithi043 ceph-mon[29963]: 3.f scrub ok 2024-03-30T14:03:02.356 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:02 smithi043 ceph-mon[29963]: pgmap v931: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 46 KiB/s rd, 60 KiB/s wr, 976 op/s; 45628/304497 objects misplaced (14.985%); 31 MiB/s, 507 objects/s recovering 2024-03-30T14:03:02.356 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:02 smithi043 ceph-mon[42099]: 3.f scrub starts 2024-03-30T14:03:02.356 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:02 smithi043 ceph-mon[42099]: 3.f scrub ok 2024-03-30T14:03:02.356 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:02 smithi043 ceph-mon[42099]: pgmap v931: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 46 KiB/s rd, 60 KiB/s wr, 976 op/s; 45628/304497 objects misplaced (14.985%); 31 MiB/s, 507 objects/s recovering 2024-03-30T14:03:02.371 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:02.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:02.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:02.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:02 smithi113 ceph-mon[34341]: 3.f scrub starts 2024-03-30T14:03:02.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:02 smithi113 ceph-mon[34341]: 3.f scrub ok 2024-03-30T14:03:02.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:02 smithi113 ceph-mon[34341]: pgmap v931: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 46 KiB/s rd, 60 KiB/s wr, 976 op/s; 45628/304497 objects misplaced (14.985%); 31 MiB/s, 507 objects/s recovering 2024-03-30T14:03:02.473 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:02.512 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:02.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:02.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:02.546 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T14:03:02.547 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:03:02.614 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:02.643 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:02.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:02.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:02.745 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:02.788 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:02.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:02.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:02.890 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:02.918 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:02.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:03.020 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:03.038 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:03:03.049 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:03.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:03.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:03.151 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:03.179 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:03.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:03.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:03.281 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:03.309 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:03.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:03.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:03.410 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:03.413 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T14:03:03.439 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:03.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:03.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:03.476 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T14:03:03.477 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T14:03:03.477 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 422, 2024-03-30T14:03:03.477 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "7ee932fa-ee9a-11ee-b647-cb9ed24678a4", 2024-03-30T14:03:03.477 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-03-30T13:37:07.346020+0000", 2024-03-30T14:03:03.477 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-03-30T14:02:59.153391+0000", 2024-03-30T14:03:03.477 INFO:teuthology.orchestra.run.smithi043.stdout: "last_up_change": "2024-03-30T14:01:27.027194+0000", 2024-03-30T14:03:03.477 INFO:teuthology.orchestra.run.smithi043.stdout: "last_in_change": "2024-03-30T14:02:48.075504+0000", 2024-03-30T14:03:03.477 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-30T14:03:03.477 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_num": 5799936, 2024-03-30T14:03:03.477 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_set": [ 2024-03-30T14:03:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: "pglog_hardlimit", 2024-03-30T14:03:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: "purged_snapdirs", 2024-03-30T14:03:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: "recovery_deletes", 2024-03-30T14:03:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: "sortbitwise" 2024-03-30T14:03:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T14:03:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_version": 19, 2024-03-30T14:03:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: "full_ratio": 0.94999998807907104, 2024-03-30T14:03:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-30T14:03:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-30T14:03:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_snapshot": "", 2024-03-30T14:03:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_max": 11, 2024-03-30T14:03:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: "max_osd": 8, 2024-03-30T14:03:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: "require_min_compat_client": "luminous", 2024-03-30T14:03:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "min_compat_client": "luminous", 2024-03-30T14:03:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "require_osd_release": "squid", 2024-03-30T14:03:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_crimson": false, 2024-03-30T14:03:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "pools": [ 2024-03-30T14:03:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 1, 2024-03-30T14:03:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".mgr", 2024-03-30T14:03:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:40:27.658979+0000", 2024-03-30T14:03:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T14:03:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T14:03:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:03:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:03:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:03:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:03:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:03:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:03:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:03:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:03:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:03:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:03:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 21, 2024-03-30T14:03:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 21, 2024-03-30T14:03:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 21, 2024-03-30T14:03:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 21, 2024-03-30T14:03:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 21, 2024-03-30T14:03:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:03:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:03:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:03:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:03:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:03:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:03:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:03:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "371", 2024-03-30T14:03:03.482 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:03:03.482 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:03:03.482 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "368", 2024-03-30T14:03:03.482 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:03:03.482 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:03:03.482 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T14:03:03.482 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T14:03:03.482 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:03:03.482 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:03:03.482 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:03:03.483 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:03:03.483 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:03:03.483 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:03:03.483 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:03:03.483 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:03:03.483 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:03:03.483 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:03:03.483 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:03:03.483 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:03:03.483 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:03:03.483 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:03:03.483 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:03:03.484 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:03:03.484 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:03:03.484 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:03:03.484 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:03:03.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.484 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:03:03.484 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:03:03.484 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:03:03.484 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:03:03.484 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:03:03.484 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:03:03.484 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:03:03.484 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:03:03.485 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:03:03.485 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:03:03.485 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:03:03.485 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-03-30T14:03:03.485 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_max": 32, 2024-03-30T14:03:03.485 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_min": 1 2024-03-30T14:03:03.485 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.485 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:03:03.485 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": {} 2024-03-30T14:03:03.485 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.485 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:03:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:03:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.2899999618530273, 2024-03-30T14:03:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.2899999618530273, 2024-03-30T14:03:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1, 2024-03-30T14:03:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2.2899999618530273, 2024-03-30T14:03:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2.2899999618530273, 2024-03-30T14:03:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.75, 2024-03-30T14:03:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:03:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.75 2024-03-30T14:03:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.487 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 2, 2024-03-30T14:03:03.487 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "datapool", 2024-03-30T14:03:03.487 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:42.989975+0000", 2024-03-30T14:03:03.487 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 8193, 2024-03-30T14:03:03.487 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T14:03:03.487 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:03:03.487 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:03:03.487 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:03:03.487 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:03:03.487 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:03:03.487 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:03:03.487 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:03:03.487 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:03:03.487 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:03:03.488 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:03:03.488 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 3, 2024-03-30T14:03:03.488 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 3, 2024-03-30T14:03:03.488 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 3, 2024-03-30T14:03:03.488 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 3, 2024-03-30T14:03:03.488 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 3, 2024-03-30T14:03:03.488 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:03:03.488 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:03:03.488 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:03:03.488 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:03:03.488 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:03:03.488 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:03:03.488 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:03:03.488 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.489 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "68", 2024-03-30T14:03:03.489 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:03:03.489 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:03:03.489 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T14:03:03.489 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:03:03.489 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:03:03.489 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 2, 2024-03-30T14:03:03.489 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 68, 2024-03-30T14:03:03.489 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:03:03.489 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:03:03.489 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:03:03.489 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:03:03.489 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:03:03.490 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:03:03.490 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:03:03.490 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:03:03.490 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:03:03.490 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:03:03.490 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:03:03.490 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:03:03.490 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:03:03.490 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:03:03.490 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:03:03.490 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:03:03.490 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:03:03.490 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:03:03.491 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:03:03.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.491 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:03:03.491 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:03:03.491 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:03:03.491 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:03:03.491 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:03:03.491 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:03:03.491 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:03:03.491 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:03:03.491 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:03:03.491 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:03:03.491 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:03:03.491 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T14:03:03.492 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:03:03.492 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd": {} 2024-03-30T14:03:03.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.492 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:03:03.492 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:03:03.492 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 4.0100002288818359, 2024-03-30T14:03:03.492 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 4.0100002288818359, 2024-03-30T14:03:03.492 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.82999998331069946, 2024-03-30T14:03:03.492 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 3.3299999237060547, 2024-03-30T14:03:03.492 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 3.3299999237060547, 2024-03-30T14:03:03.492 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.75, 2024-03-30T14:03:03.492 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:03:03.492 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.75 2024-03-30T14:03:03.492 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.493 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.493 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 3, 2024-03-30T14:03:03.493 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".rgw.root", 2024-03-30T14:03:03.493 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:43.910306+0000", 2024-03-30T14:03:03.493 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T14:03:03.493 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T14:03:03.493 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:03:03.493 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:03:03.493 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:03:03.493 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:03:03.493 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:03:03.493 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:03:03.493 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:03:03.494 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:03:03.494 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:03:03.494 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:03:03.494 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T14:03:03.494 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T14:03:03.494 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T14:03:03.494 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T14:03:03.494 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T14:03:03.494 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:03:03.494 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:03:03.495 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:03:03.495 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:03:03.495 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:03:03.495 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:03:03.495 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:03:03.495 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.495 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "66", 2024-03-30T14:03:03.496 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:03:03.496 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:03:03.496 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T14:03:03.496 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:03:03.496 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:03:03.496 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T14:03:03.496 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T14:03:03.496 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:03:03.496 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:03:03.496 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:03:03.496 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:03:03.496 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:03:03.497 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:03:03.497 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:03:03.497 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:03:03.497 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:03:03.497 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:03:03.497 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:03:03.497 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:03:03.497 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:03:03.497 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:03:03.497 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:03:03.497 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:03:03.497 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:03:03.497 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:03:03.497 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:03:03.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.498 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:03:03.498 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:03:03.498 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:03:03.498 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:03:03.498 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:03:03.498 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:03:03.498 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:03:03.498 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:03:03.498 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:03:03.498 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:03:03.498 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:03:03.498 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T14:03:03.499 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:03:03.499 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T14:03:03.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.499 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:03:03.499 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:03:03.499 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.25, 2024-03-30T14:03:03.499 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.25, 2024-03-30T14:03:03.499 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1, 2024-03-30T14:03:03.499 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2.25, 2024-03-30T14:03:03.499 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2.25, 2024-03-30T14:03:03.499 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.75, 2024-03-30T14:03:03.499 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:03:03.499 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.75 2024-03-30T14:03:03.499 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.500 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.500 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 4, 2024-03-30T14:03:03.500 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.log", 2024-03-30T14:03:03.500 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:45.633132+0000", 2024-03-30T14:03:03.500 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T14:03:03.500 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T14:03:03.500 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:03:03.500 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:03:03.500 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:03:03.500 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:03:03.500 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:03:03.500 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:03:03.500 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:03:03.501 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:03:03.501 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:03:03.501 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:03:03.501 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T14:03:03.501 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T14:03:03.501 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T14:03:03.501 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T14:03:03.501 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T14:03:03.501 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:03:03.501 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:03:03.501 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:03:03.501 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:03:03.501 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:03:03.501 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:03:03.501 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:03:03.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.502 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "68", 2024-03-30T14:03:03.502 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:03:03.502 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:03:03.502 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T14:03:03.502 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:03:03.502 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:03:03.502 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T14:03:03.502 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T14:03:03.502 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:03:03.502 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:03:03.502 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:03:03.502 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:03:03.503 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:03:03.503 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:03:03.503 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:03:03.503 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:03:03.503 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:03:03.503 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:03:03.503 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:03:03.503 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:03:03.503 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:03:03.503 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:03:03.503 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:03:03.503 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:03:03.503 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:03:03.503 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:03:03.503 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:03:03.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.504 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:03:03.504 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:03:03.504 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:03:03.504 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:03:03.504 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:03:03.504 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:03:03.504 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:03:03.504 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:03:03.504 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:03:03.504 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:03:03.504 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:03:03.504 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T14:03:03.504 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:03:03.505 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T14:03:03.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.505 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:03:03.505 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:03:03.505 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.059999942779541, 2024-03-30T14:03:03.505 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.059999942779541, 2024-03-30T14:03:03.505 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1, 2024-03-30T14:03:03.505 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2.059999942779541, 2024-03-30T14:03:03.505 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2.059999942779541, 2024-03-30T14:03:03.505 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.75, 2024-03-30T14:03:03.505 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:03:03.505 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.75 2024-03-30T14:03:03.505 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.506 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.506 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 5, 2024-03-30T14:03:03.506 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.control", 2024-03-30T14:03:03.506 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:47.570084+0000", 2024-03-30T14:03:03.506 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T14:03:03.506 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T14:03:03.506 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:03:03.506 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:03:03.506 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:03:03.506 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:03:03.506 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:03:03.506 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:03:03.506 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:03:03.507 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:03:03.507 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:03:03.507 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:03:03.507 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T14:03:03.507 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T14:03:03.507 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T14:03:03.507 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T14:03:03.507 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T14:03:03.507 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:03:03.507 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:03:03.507 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:03:03.507 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:03:03.507 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:03:03.508 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:03:03.508 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:03:03.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.508 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "70", 2024-03-30T14:03:03.508 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:03:03.508 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:03:03.508 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T14:03:03.508 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:03:03.508 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:03:03.508 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T14:03:03.508 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T14:03:03.508 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:03:03.508 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:03:03.508 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:03:03.508 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:03:03.509 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:03:03.509 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:03:03.509 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:03:03.509 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:03:03.509 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:03:03.509 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:03:03.509 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:03:03.509 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:03:03.509 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:03:03.509 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:03:03.509 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:03:03.509 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:03:03.509 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:03:03.509 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:03:03.510 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:03:03.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.510 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:03:03.510 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:03:03.510 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:03:03.510 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:03:03.510 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:03:03.510 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:03:03.510 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:03:03.510 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:03:03.510 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:03:03.510 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:03:03.510 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:03:03.511 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T14:03:03.511 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:03:03.511 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T14:03:03.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.511 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:03:03.511 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:03:03.511 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.059999942779541, 2024-03-30T14:03:03.511 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.059999942779541, 2024-03-30T14:03:03.511 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1, 2024-03-30T14:03:03.511 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2.059999942779541, 2024-03-30T14:03:03.511 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2.059999942779541, 2024-03-30T14:03:03.511 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.75, 2024-03-30T14:03:03.511 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:03:03.511 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.75 2024-03-30T14:03:03.512 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.512 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 6, 2024-03-30T14:03:03.512 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.meta", 2024-03-30T14:03:03.512 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T13:42:49.637189+0000", 2024-03-30T14:03:03.512 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T14:03:03.512 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T14:03:03.512 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:03:03.512 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:03:03.512 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:03:03.512 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:03:03.512 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:03:03.512 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:03:03.513 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:03:03.513 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:03:03.513 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:03:03.513 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:03:03.513 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-03-30T14:03:03.513 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-03-30T14:03:03.513 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-03-30T14:03:03.513 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-03-30T14:03:03.513 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-03-30T14:03:03.513 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:03:03.513 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:03:03.513 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:03:03.513 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:03:03.513 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:03:03.514 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:03:03.514 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:03:03.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.514 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "72", 2024-03-30T14:03:03.514 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:03:03.514 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:03:03.514 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T14:03:03.514 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:03:03.514 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:03:03.514 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T14:03:03.514 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T14:03:03.514 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:03:03.514 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:03:03.515 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:03:03.515 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:03:03.515 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:03:03.515 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:03:03.515 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:03:03.515 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:03:03.515 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:03:03.515 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:03:03.515 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:03:03.515 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:03:03.515 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:03:03.515 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:03:03.515 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:03:03.515 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:03:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:03:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:03:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:03:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:03:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:03:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:03:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:03:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:03:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:03:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:03:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:03:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:03:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:03:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:03:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-03-30T14:03:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_bias": 4 2024-03-30T14:03:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:03:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-03-30T14:03:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:03:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:03:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.8799999952316284, 2024-03-30T14:03:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.8799999952316284, 2024-03-30T14:03:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1, 2024-03-30T14:03:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.8799999952316284, 2024-03-30T14:03:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.8799999952316284, 2024-03-30T14:03:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.75, 2024-03-30T14:03:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:03:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.75 2024-03-30T14:03:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 11, 2024-03-30T14:03:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "unique_pool_4", 2024-03-30T14:03:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-03-30T14:01:12.894591+0000", 2024-03-30T14:03:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-30T14:03:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-03-30T14:03:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-03-30T14:03:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-03-30T14:03:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-03-30T14:03:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-03-30T14:03:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-03-30T14:03:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-03-30T14:03:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T14:03:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T14:03:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-03-30T14:03:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-03-30T14:03:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 16, 2024-03-30T14:03:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 16, 2024-03-30T14:03:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 16, 2024-03-30T14:03:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 16, 2024-03-30T14:03:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 16, 2024-03-30T14:03:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-03-30T14:03:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-03-30T14:03:03.520 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-03-30T14:03:03.520 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-03-30T14:03:03.520 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-03-30T14:03:03.520 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-03-30T14:03:03.520 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-03-30T14:03:03.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.520 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "384", 2024-03-30T14:03:03.520 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-03-30T14:03:03.520 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T14:03:03.520 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T14:03:03.520 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-03-30T14:03:03.520 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-03-30T14:03:03.520 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-03-30T14:03:03.520 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-03-30T14:03:03.521 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-03-30T14:03:03.521 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-03-30T14:03:03.521 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-03-30T14:03:03.521 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-03-30T14:03:03.521 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-03-30T14:03:03.521 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-03-30T14:03:03.521 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-03-30T14:03:03.521 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-03-30T14:03:03.521 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-03-30T14:03:03.521 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-03-30T14:03:03.521 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-03-30T14:03:03.521 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T14:03:03.521 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T14:03:03.521 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T14:03:03.522 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-03-30T14:03:03.522 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-03-30T14:03:03.522 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-03-30T14:03:03.522 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-03-30T14:03:03.522 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-03-30T14:03:03.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.522 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-03-30T14:03:03.522 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-03-30T14:03:03.522 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-03-30T14:03:03.522 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-03-30T14:03:03.522 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-03-30T14:03:03.522 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T14:03:03.522 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-03-30T14:03:03.522 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-03-30T14:03:03.523 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-03-30T14:03:03.523 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-03-30T14:03:03.523 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-03-30T14:03:03.523 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-03-30T14:03:03.523 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-03-30T14:03:03.523 INFO:teuthology.orchestra.run.smithi043.stdout: "rados": {} 2024-03-30T14:03:03.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.523 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-03-30T14:03:03.523 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-03-30T14:03:03.523 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 6, 2024-03-30T14:03:03.523 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.25, 2024-03-30T14:03:03.523 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1, 2024-03-30T14:03:03.523 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 6, 2024-03-30T14:03:03.523 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2.25, 2024-03-30T14:03:03.524 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.75, 2024-03-30T14:03:03.524 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-03-30T14:03:03.524 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.75, 2024-03-30T14:03:03.524 INFO:teuthology.orchestra.run.smithi043.stdout: "error_message": "pool 11 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate" 2024-03-30T14:03:03.524 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.524 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.524 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T14:03:03.524 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:03:03.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.524 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-03-30T14:03:03.524 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "94ea083e-6a27-41db-83af-2f7d3cae1956", 2024-03-30T14:03:03.524 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:03:03.524 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T14:03:03.525 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T14:03:03.525 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 0, 2024-03-30T14:03:03.525 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:03:03.525 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:03:03.525 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 8, 2024-03-30T14:03:03.525 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 418, 2024-03-30T14:03:03.525 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:03:03.525 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:03:03.525 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:03:03.525 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.525 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.525 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.525 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6802", 2024-03-30T14:03:03.525 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:03:03.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.526 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.526 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.526 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6803", 2024-03-30T14:03:03.526 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:03:03.526 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.526 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.526 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:03:03.526 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.526 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.526 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.526 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6804", 2024-03-30T14:03:03.526 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:03:03.527 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.527 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.527 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.527 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6805", 2024-03-30T14:03:03.527 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:03:03.527 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.527 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.527 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.527 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:03:03.527 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.527 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.527 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.527 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6808", 2024-03-30T14:03:03.527 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:03:03.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.528 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.528 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.528 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6809", 2024-03-30T14:03:03.528 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:03:03.528 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.528 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.528 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:03:03.528 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.528 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.528 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.528 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6806", 2024-03-30T14:03:03.529 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:03:03.529 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.529 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.529 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.529 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6807", 2024-03-30T14:03:03.529 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1144592572 2024-03-30T14:03:03.529 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.530 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.530 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6803/1144592572", 2024-03-30T14:03:03.530 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6805/1144592572", 2024-03-30T14:03:03.530 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6809/1144592572", 2024-03-30T14:03:03.530 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6807/1144592572", 2024-03-30T14:03:03.530 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:03:03.530 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:03:03.530 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:03:03.530 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.530 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.530 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-03-30T14:03:03.531 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a", 2024-03-30T14:03:03.531 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:03:03.531 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T14:03:03.531 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T14:03:03.531 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T14:03:03.531 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:03:03.531 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:03:03.531 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 13, 2024-03-30T14:03:03.531 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 420, 2024-03-30T14:03:03.531 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:03:03.531 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:03:03.531 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:03:03.531 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.531 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.532 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.532 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6810", 2024-03-30T14:03:03.532 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:03:03.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.532 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.532 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.532 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6811", 2024-03-30T14:03:03.532 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:03:03.532 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.532 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.532 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:03:03.532 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.532 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.532 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.533 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6812", 2024-03-30T14:03:03.533 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:03:03.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.533 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.533 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6813", 2024-03-30T14:03:03.533 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:03:03.533 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.533 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.533 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:03:03.533 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.533 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.534 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6816", 2024-03-30T14:03:03.534 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:03:03.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.534 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.534 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.534 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6817", 2024-03-30T14:03:03.534 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:03:03.534 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.534 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.534 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:03:03.534 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.534 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.535 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.535 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6814", 2024-03-30T14:03:03.535 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:03:03.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.535 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.535 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6815", 2024-03-30T14:03:03.535 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3901165928 2024-03-30T14:03:03.535 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.535 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.535 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6811/3901165928", 2024-03-30T14:03:03.535 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6813/3901165928", 2024-03-30T14:03:03.536 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6817/3901165928", 2024-03-30T14:03:03.536 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6815/3901165928", 2024-03-30T14:03:03.536 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:03:03.536 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:03:03.536 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:03:03.536 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.536 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.536 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-03-30T14:03:03.536 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "6db6782e-5889-443e-bc6c-df47a1f93bc8", 2024-03-30T14:03:03.536 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:03:03.536 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T14:03:03.536 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T14:03:03.537 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 0, 2024-03-30T14:03:03.537 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 18, 2024-03-30T14:03:03.537 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 166, 2024-03-30T14:03:03.537 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 388, 2024-03-30T14:03:03.537 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 98, 2024-03-30T14:03:03.537 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 167, 2024-03-30T14:03:03.537 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:03:03.537 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:03:03.537 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.537 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.537 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.537 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6818", 2024-03-30T14:03:03.537 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 886354154 2024-03-30T14:03:03.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.538 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.538 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.538 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6819", 2024-03-30T14:03:03.538 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 886354154 2024-03-30T14:03:03.538 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.538 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.538 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.538 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:03:03.538 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.538 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.538 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.538 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6820", 2024-03-30T14:03:03.538 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 886354154 2024-03-30T14:03:03.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.539 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.539 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6821", 2024-03-30T14:03:03.539 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 886354154 2024-03-30T14:03:03.539 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.539 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.539 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:03:03.539 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.539 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.539 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6824", 2024-03-30T14:03:03.539 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 886354154 2024-03-30T14:03:03.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.540 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.540 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.540 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6825", 2024-03-30T14:03:03.540 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 886354154 2024-03-30T14:03:03.540 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.540 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.541 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.541 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:03:03.542 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.542 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.542 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6822", 2024-03-30T14:03:03.542 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 886354154 2024-03-30T14:03:03.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.542 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.542 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6823", 2024-03-30T14:03:03.542 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 886354154 2024-03-30T14:03:03.542 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.542 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.543 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6819/886354154", 2024-03-30T14:03:03.543 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6821/886354154", 2024-03-30T14:03:03.543 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6825/886354154", 2024-03-30T14:03:03.543 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6823/886354154", 2024-03-30T14:03:03.543 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:03:03.543 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:03:03.543 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:03:03.543 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.543 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.543 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-03-30T14:03:03.543 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "7048ff46-2cba-4d18-9835-95ecbddb2bb6", 2024-03-30T14:03:03.543 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:03:03.543 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 0, 2024-03-30T14:03:03.544 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-03-30T14:03:03.544 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T14:03:03.544 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:03:03.544 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:03:03.544 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 25, 2024-03-30T14:03:03.544 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 411, 2024-03-30T14:03:03.544 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:03:03.544 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:03:03.544 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:03:03.544 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.544 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.544 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.544 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6826", 2024-03-30T14:03:03.544 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:03:03.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.545 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.545 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6827", 2024-03-30T14:03:03.545 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:03:03.545 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.545 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.545 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:03:03.545 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.545 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.545 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6828", 2024-03-30T14:03:03.545 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:03:03.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.546 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.546 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6829", 2024-03-30T14:03:03.546 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:03:03.546 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.546 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.546 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:03:03.546 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.546 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.546 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6832", 2024-03-30T14:03:03.547 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:03:03.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.547 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.547 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.547 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6833", 2024-03-30T14:03:03.547 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:03:03.547 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.547 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.547 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:03:03.547 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.547 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.547 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.548 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6830", 2024-03-30T14:03:03.548 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:03:03.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.548 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.548 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6831", 2024-03-30T14:03:03.548 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2933589376 2024-03-30T14:03:03.548 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.548 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.548 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6827/2933589376", 2024-03-30T14:03:03.548 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6829/2933589376", 2024-03-30T14:03:03.548 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6833/2933589376", 2024-03-30T14:03:03.548 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6831/2933589376", 2024-03-30T14:03:03.549 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:03:03.549 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:03:03.549 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:03:03.549 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.549 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.549 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-03-30T14:03:03.549 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "42da30e6-9eec-4ed5-8c73-9ce78aa8ff63", 2024-03-30T14:03:03.549 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:03:03.549 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T14:03:03.549 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T14:03:03.549 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T14:03:03.549 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:03:03.549 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:03:03.550 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 32, 2024-03-30T14:03:03.550 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 420, 2024-03-30T14:03:03.550 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:03:03.550 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:03:03.550 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:03:03.550 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.550 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.550 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6800", 2024-03-30T14:03:03.550 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:03:03.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.550 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.551 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6801", 2024-03-30T14:03:03.551 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:03:03.551 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.551 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.551 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.551 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:03:03.551 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.551 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.551 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.551 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6802", 2024-03-30T14:03:03.551 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:03:03.551 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.551 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.552 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.552 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6803", 2024-03-30T14:03:03.552 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:03:03.552 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.552 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.552 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:03:03.552 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.552 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.552 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.552 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6806", 2024-03-30T14:03:03.552 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:03:03.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.553 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.553 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.553 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6807", 2024-03-30T14:03:03.553 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:03:03.553 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.553 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.553 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.553 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:03:03.553 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.553 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.553 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.553 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6804", 2024-03-30T14:03:03.553 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:03:03.554 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.554 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.554 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.554 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6805", 2024-03-30T14:03:03.554 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1945833081 2024-03-30T14:03:03.554 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.554 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.554 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.554 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6801/1945833081", 2024-03-30T14:03:03.554 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6803/1945833081", 2024-03-30T14:03:03.554 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6807/1945833081", 2024-03-30T14:03:03.554 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6805/1945833081", 2024-03-30T14:03:03.554 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:03:03.554 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:03:03.555 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:03:03.555 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.555 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.555 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-03-30T14:03:03.555 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "99a11b81-0df7-4b1d-946a-4297ba62e263", 2024-03-30T14:03:03.555 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:03:03.555 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T14:03:03.555 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T14:03:03.555 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T14:03:03.555 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:03:03.555 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:03:03.555 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 40, 2024-03-30T14:03:03.555 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 420, 2024-03-30T14:03:03.556 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:03:03.556 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:03:03.556 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:03:03.556 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.556 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.556 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.556 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6808", 2024-03-30T14:03:03.556 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:03:03.556 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.556 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.556 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.556 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6809", 2024-03-30T14:03:03.556 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:03:03.556 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.557 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.557 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:03:03.557 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.557 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.557 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.557 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6810", 2024-03-30T14:03:03.557 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:03:03.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.557 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.557 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.557 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6811", 2024-03-30T14:03:03.557 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:03:03.557 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.558 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.558 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:03:03.558 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.558 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.558 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6814", 2024-03-30T14:03:03.558 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:03:03.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.558 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.558 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6815", 2024-03-30T14:03:03.558 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:03:03.559 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.559 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.559 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:03:03.559 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.559 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.559 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6812", 2024-03-30T14:03:03.559 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:03:03.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.559 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.559 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6813", 2024-03-30T14:03:03.559 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426360325 2024-03-30T14:03:03.560 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.560 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.560 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6809/3426360325", 2024-03-30T14:03:03.560 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6811/3426360325", 2024-03-30T14:03:03.560 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6815/3426360325", 2024-03-30T14:03:03.560 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6813/3426360325", 2024-03-30T14:03:03.560 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:03:03.560 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:03:03.560 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:03:03.560 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.560 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-03-30T14:03:03.561 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "cafd0e26-78e9-4164-a43a-96ca8382fefe", 2024-03-30T14:03:03.561 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:03:03.561 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-03-30T14:03:03.561 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-03-30T14:03:03.561 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T14:03:03.561 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:03:03.561 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:03:03.561 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 49, 2024-03-30T14:03:03.561 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 420, 2024-03-30T14:03:03.561 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:03:03.561 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:03:03.561 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:03:03.561 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.562 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.562 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.562 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6816", 2024-03-30T14:03:03.562 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:03:03.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.562 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.562 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.562 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6817", 2024-03-30T14:03:03.562 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:03:03.562 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.562 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.562 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:03:03.562 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.563 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.563 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.563 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6818", 2024-03-30T14:03:03.563 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:03:03.563 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.563 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.563 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.563 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6819", 2024-03-30T14:03:03.563 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:03:03.563 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.563 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.563 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.563 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:03:03.563 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.564 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.564 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6822", 2024-03-30T14:03:03.564 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:03:03.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.564 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.564 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6823", 2024-03-30T14:03:03.564 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:03:03.564 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.564 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.564 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:03:03.564 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.565 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.565 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.565 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6820", 2024-03-30T14:03:03.565 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:03:03.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.565 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.565 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.565 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6821", 2024-03-30T14:03:03.565 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1527017698 2024-03-30T14:03:03.565 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.565 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.565 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6817/1527017698", 2024-03-30T14:03:03.565 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6819/1527017698", 2024-03-30T14:03:03.566 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6823/1527017698", 2024-03-30T14:03:03.566 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6821/1527017698", 2024-03-30T14:03:03.566 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:03:03.566 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:03:03.566 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:03:03.566 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.566 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.566 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-03-30T14:03:03.566 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "e9d59662-9132-48f4-9e95-842a3cfbd79f", 2024-03-30T14:03:03.566 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-03-30T14:03:03.566 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 0, 2024-03-30T14:03:03.566 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-03-30T14:03:03.566 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-03-30T14:03:03.567 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-03-30T14:03:03.567 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-03-30T14:03:03.567 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 58, 2024-03-30T14:03:03.567 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 228, 2024-03-30T14:03:03.567 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-03-30T14:03:03.567 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-03-30T14:03:03.567 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-03-30T14:03:03.567 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.567 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.567 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.567 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:03.569 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6824", 2024-03-30T14:03:03.569 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:03:03.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.569 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.569 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6825", 2024-03-30T14:03:03.569 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:03:03.569 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.569 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.570 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-03-30T14:03:03.570 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.570 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.570 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6826", 2024-03-30T14:03:03.570 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:03:03.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.570 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.570 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6827", 2024-03-30T14:03:03.570 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:03:03.570 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.571 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.571 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-03-30T14:03:03.571 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.571 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.571 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6830", 2024-03-30T14:03:03.571 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:03:03.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.571 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.571 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6831", 2024-03-30T14:03:03.571 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:03:03.571 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.572 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.572 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-03-30T14:03:03.572 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-30T14:03:03.572 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.572 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-30T14:03:03.572 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6828", 2024-03-30T14:03:03.572 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:03:03.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.572 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.572 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-30T14:03:03.572 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.113:6829", 2024-03-30T14:03:03.572 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3182476015 2024-03-30T14:03:03.572 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.573 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.573 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.113:6825/3182476015", 2024-03-30T14:03:03.573 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.113:6827/3182476015", 2024-03-30T14:03:03.573 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.113:6831/3182476015", 2024-03-30T14:03:03.573 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.113:6829/3182476015", 2024-03-30T14:03:03.573 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-03-30T14:03:03.573 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-03-30T14:03:03.573 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-03-30T14:03:03.573 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.573 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.573 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T14:03:03.573 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_xinfo": [ 2024-03-30T14:03:03.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.574 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-03-30T14:03:03.574 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:03:03.574 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:03:03.574 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:03:03.574 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:03:03.574 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T14:03:03.574 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T14:02:32.250706+0000", 2024-03-30T14:03:03.574 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:03:03.574 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.574 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-03-30T14:03:03.574 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:03:03.574 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:03:03.575 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:03:03.575 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:03:03.575 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T14:03:03.575 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T14:02:22.586995+0000", 2024-03-30T14:03:03.575 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:03:03.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.575 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.575 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-03-30T14:03:03.575 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:03:03.575 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:03:03.575 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:03:03.575 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:03:03.575 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T14:03:03.575 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T14:02:41.533018+0000", 2024-03-30T14:03:03.576 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 166 2024-03-30T14:03:03.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.576 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.576 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-03-30T14:03:03.576 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:03:03.576 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:03:03.576 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:03:03.576 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:03:03.576 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 65536, 2024-03-30T14:03:03.576 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T14:02:31.974551+0000", 2024-03-30T14:03:03.576 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:03:03.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.576 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.576 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-03-30T14:03:03.577 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:03:03.577 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:03:03.577 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:03:03.577 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:03:03.577 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T14:03:03.577 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T14:02:28.170275+0000", 2024-03-30T14:03:03.577 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:03:03.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.577 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.577 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-03-30T14:03:03.577 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:03:03.577 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:03:03.577 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:03:03.577 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:03:03.578 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T14:03:03.578 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T14:01:50.085517+0000", 2024-03-30T14:03:03.578 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:03:03.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.578 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.578 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-03-30T14:03:03.578 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:03:03.578 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:03:03.578 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:03:03.578 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:03:03.578 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-03-30T14:03:03.578 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T14:02:41.285874+0000", 2024-03-30T14:03:03.578 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:03:03.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.579 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.579 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-03-30T14:03:03.579 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-03-30T14:03:03.579 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-03-30T14:03:03.579 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-03-30T14:03:03.579 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-03-30T14:03:03.579 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 65536, 2024-03-30T14:03:03.579 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-03-30T14:02:25.717578+0000", 2024-03-30T14:03:03.579 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-03-30T14:03:03.579 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.579 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T14:03:03.579 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap": [], 2024-03-30T14:03:03.579 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_items": [ 2024-03-30T14:03:03.579 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.579 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "1.2", 2024-03-30T14:03:03.580 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-03-30T14:03:03.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.580 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 2, 2024-03-30T14:03:03.580 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 4 2024-03-30T14:03:03.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.580 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 0, 2024-03-30T14:03:03.580 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 1 2024-03-30T14:03:03.580 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.580 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.580 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.4", 2024-03-30T14:03:03.580 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-03-30T14:03:03.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.581 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-03-30T14:03:03.581 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 5 2024-03-30T14:03:03.581 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.581 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.581 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.7", 2024-03-30T14:03:03.581 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-03-30T14:03:03.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.581 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-03-30T14:03:03.581 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-03-30T14:03:03.581 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.581 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.582 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.9", 2024-03-30T14:03:03.582 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-03-30T14:03:03.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.582 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-03-30T14:03:03.582 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-03-30T14:03:03.582 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.582 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.582 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.b", 2024-03-30T14:03:03.582 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-03-30T14:03:03.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.583 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-03-30T14:03:03.583 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 5 2024-03-30T14:03:03.583 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.583 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.583 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.14", 2024-03-30T14:03:03.583 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-03-30T14:03:03.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.583 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-03-30T14:03:03.583 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-03-30T14:03:03.583 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.584 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.584 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.1b", 2024-03-30T14:03:03.584 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-03-30T14:03:03.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.584 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-03-30T14:03:03.584 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 0 2024-03-30T14:03:03.584 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.584 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.584 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.15", 2024-03-30T14:03:03.584 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-03-30T14:03:03.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.585 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 0, 2024-03-30T14:03:03.585 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 6 2024-03-30T14:03:03.585 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.585 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.585 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.585 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T14:03:03.585 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_primaries": [], 2024-03-30T14:03:03.585 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_temp": [ 2024-03-30T14:03:03.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.585 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "11.0", 2024-03-30T14:03:03.585 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:03:03.585 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T14:03:03.586 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T14:03:03.586 INFO:teuthology.orchestra.run.smithi043.stdout: 3 2024-03-30T14:03:03.586 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.586 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "11.1", 2024-03-30T14:03:03.586 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:03:03.586 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T14:03:03.586 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T14:03:03.586 INFO:teuthology.orchestra.run.smithi043.stdout: 3 2024-03-30T14:03:03.586 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.587 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "11.2", 2024-03-30T14:03:03.587 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:03:03.587 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T14:03:03.587 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-03-30T14:03:03.587 INFO:teuthology.orchestra.run.smithi043.stdout: 3 2024-03-30T14:03:03.587 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.587 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.587 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "11.5", 2024-03-30T14:03:03.587 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:03:03.587 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T14:03:03.587 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T14:03:03.587 INFO:teuthology.orchestra.run.smithi043.stdout: 3 2024-03-30T14:03:03.588 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.588 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "11.8", 2024-03-30T14:03:03.588 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:03:03.588 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T14:03:03.588 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T14:03:03.588 INFO:teuthology.orchestra.run.smithi043.stdout: 3 2024-03-30T14:03:03.588 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.588 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "11.9", 2024-03-30T14:03:03.588 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:03:03.588 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T14:03:03.589 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T14:03:03.589 INFO:teuthology.orchestra.run.smithi043.stdout: 3 2024-03-30T14:03:03.589 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.589 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.589 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "11.a", 2024-03-30T14:03:03.589 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:03:03.589 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-03-30T14:03:03.589 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-03-30T14:03:03.589 INFO:teuthology.orchestra.run.smithi043.stdout: 3 2024-03-30T14:03:03.589 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.589 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:03.590 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "11.b", 2024-03-30T14:03:03.590 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-03-30T14:03:03.590 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-03-30T14:03:03.590 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-03-30T14:03:03.590 INFO:teuthology.orchestra.run.smithi043.stdout: 3 2024-03-30T14:03:03.590 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:03.590 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.590 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-30T14:03:03.590 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_temp": [], 2024-03-30T14:03:03.590 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklist": { 2024-03-30T14:03:03.590 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/538304234": "2024-03-31T13:43:27.253365+0000", 2024-03-30T14:03:03.590 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6801/3411723337": "2024-03-31T13:37:41.733515+0000", 2024-03-30T14:03:03.590 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3656823679": "2024-03-31T13:37:41.733515+0000", 2024-03-30T14:03:03.591 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2618817647": "2024-03-31T13:38:09.321740+0000", 2024-03-30T14:03:03.591 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/3411723337": "2024-03-31T13:37:41.733515+0000", 2024-03-30T14:03:03.591 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1206218784": "2024-03-31T13:37:41.733515+0000", 2024-03-30T14:03:03.591 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/886631486": "2024-03-31T13:37:41.733515+0000", 2024-03-30T14:03:03.591 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/1339921781": "2024-03-31T13:38:09.321740+0000", 2024-03-30T14:03:03.591 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6801/1339921781": "2024-03-31T13:38:09.321740+0000", 2024-03-30T14:03:03.591 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2794494528": "2024-03-31T13:38:09.321740+0000", 2024-03-30T14:03:03.591 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/564868778": "2024-03-31T13:38:09.321740+0000", 2024-03-30T14:03:03.591 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/3033707884": "2024-03-31T13:43:27.253365+0000", 2024-03-30T14:03:03.591 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3539398453": "2024-03-31T13:43:27.253365+0000", 2024-03-30T14:03:03.591 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2414364740": "2024-03-31T13:43:27.253365+0000", 2024-03-30T14:03:03.591 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6801/3033707884": "2024-03-31T13:43:27.253365+0000", 2024-03-30T14:03:03.591 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3923597429": "2024-03-31T13:43:27.253365+0000", 2024-03-30T14:03:03.591 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2524269526": "2024-03-31T13:43:27.253365+0000" 2024-03-30T14:03:03.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.592 INFO:teuthology.orchestra.run.smithi043.stdout: "range_blocklist": {}, 2024-03-30T14:03:03.592 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profiles": { 2024-03-30T14:03:03.592 INFO:teuthology.orchestra.run.smithi043.stdout: "default": { 2024-03-30T14:03:03.592 INFO:teuthology.orchestra.run.smithi043.stdout: "crush-failure-domain": "osd", 2024-03-30T14:03:03.592 INFO:teuthology.orchestra.run.smithi043.stdout: "k": "2", 2024-03-30T14:03:03.592 INFO:teuthology.orchestra.run.smithi043.stdout: "m": "1", 2024-03-30T14:03:03.592 INFO:teuthology.orchestra.run.smithi043.stdout: "plugin": "jerasure", 2024-03-30T14:03:03.592 INFO:teuthology.orchestra.run.smithi043.stdout: "technique": "reed_sol_van" 2024-03-30T14:03:03.592 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:03.592 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps_queue": [], 2024-03-30T14:03:03.592 INFO:teuthology.orchestra.run.smithi043.stdout: "new_removed_snaps": [], 2024-03-30T14:03:03.592 INFO:teuthology.orchestra.run.smithi043.stdout: "new_purged_snaps": [], 2024-03-30T14:03:03.592 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_node_flags": {}, 2024-03-30T14:03:03.592 INFO:teuthology.orchestra.run.smithi043.stdout: "device_class_flags": {}, 2024-03-30T14:03:03.592 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": { 2024-03-30T14:03:03.593 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_enabled": false, 2024-03-30T14:03:03.593 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_bucket_count": 0, 2024-03-30T14:03:03.593 INFO:teuthology.orchestra.run.smithi043.stdout: "degraded_stretch_mode": 0, 2024-03-30T14:03:03.593 INFO:teuthology.orchestra.run.smithi043.stdout: "recovering_stretch_mode": 0, 2024-03-30T14:03:03.593 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_bucket": 0 2024-03-30T14:03:03.593 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:03.593 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T14:03:03.595 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:03.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:03.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:03.641 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-03-30T14:03:03.697 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:03.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:03.762 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:03.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:03.864 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:03.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:03.924 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:03.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:04.026 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:04.054 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:04.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:04.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:04.158 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:04.188 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:04.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:04.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:04.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:03 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1854821215' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T14:03:04.217 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:03 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1854821215' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T14:03:04.218 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:03 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1854821215' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T14:03:04.290 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:04.335 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:04.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:04.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:04.437 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:04.465 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:04.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:04.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:04.568 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:04.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:04.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:04.710 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:04.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:04.738 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:04.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:04.754 INFO:tasks.thrashosds.thrasher:j is {'epoch': 422, 'fsid': '7ee932fa-ee9a-11ee-b647-cb9ed24678a4', 'created': '2024-03-30T13:37:07.346020+0000', 'modified': '2024-03-30T14:02:59.153391+0000', 'last_up_change': '2024-03-30T14:01:27.027194+0000', 'last_in_change': '2024-03-30T14:02:48.075504+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T13:40:27.658979+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '371', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '368', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2899999618530273, 'score_stable': 2.2899999618530273, 'optimal_score': 1, 'raw_score_acting': 2.2899999618530273, 'raw_score_stable': 2.2899999618530273, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-30T13:42:42.989975+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.010000228881836, 'score_stable': 4.010000228881836, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 3.3299999237060547, 'raw_score_stable': 3.3299999237060547, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-30T13:42:43.910306+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-30T13:42:45.633132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.059999942779541, 'score_stable': 2.059999942779541, 'optimal_score': 1, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-30T13:42:47.570084+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.059999942779541, 'score_stable': 2.059999942779541, 'optimal_score': 1, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-30T13:42:49.637189+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-30T14:01:12.894591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '384', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 6, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75, 'error_message': 'pool 11 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate'}}], 'osds': [{'osd': 0, 'uuid': '94ea083e-6a27-41db-83af-2f7d3cae1956', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 418, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6802', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6803', 'nonce': 1144592572}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6804', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6805', 'nonce': 1144592572}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6808', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6809', 'nonce': 1144592572}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6806', 'nonce': 1144592572}, {'type': 'v1', 'addr': '172.21.15.43:6807', 'nonce': 1144592572}]}, 'public_addr': '172.21.15.43:6803/1144592572', 'cluster_addr': '172.21.15.43:6805/1144592572', 'heartbeat_back_addr': '172.21.15.43:6809/1144592572', 'heartbeat_front_addr': '172.21.15.43:6807/1144592572', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 420, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6810', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6811', 'nonce': 3901165928}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6812', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6813', 'nonce': 3901165928}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6816', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6817', 'nonce': 3901165928}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6814', 'nonce': 3901165928}, {'type': 'v1', 'addr': '172.21.15.43:6815', 'nonce': 3901165928}]}, 'public_addr': '172.21.15.43:6811/3901165928', 'cluster_addr': '172.21.15.43:6813/3901165928', 'heartbeat_back_addr': '172.21.15.43:6817/3901165928', 'heartbeat_front_addr': '172.21.15.43:6815/3901165928', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6db6782e-5889-443e-bc6c-df47a1f93bc8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 18, 'last_clean_end': 166, 'up_from': 388, 'up_thru': 98, 'down_at': 167, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6818', 'nonce': 886354154}, {'type': 'v1', 'addr': '172.21.15.43:6819', 'nonce': 886354154}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6820', 'nonce': 886354154}, {'type': 'v1', 'addr': '172.21.15.43:6821', 'nonce': 886354154}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6824', 'nonce': 886354154}, {'type': 'v1', 'addr': '172.21.15.43:6825', 'nonce': 886354154}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6822', 'nonce': 886354154}, {'type': 'v1', 'addr': '172.21.15.43:6823', 'nonce': 886354154}]}, 'public_addr': '172.21.15.43:6819/886354154', 'cluster_addr': '172.21.15.43:6821/886354154', 'heartbeat_back_addr': '172.21.15.43:6825/886354154', 'heartbeat_front_addr': '172.21.15.43:6823/886354154', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '7048ff46-2cba-4d18-9835-95ecbddb2bb6', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 411, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6826', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6827', 'nonce': 2933589376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6828', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6829', 'nonce': 2933589376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6832', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6833', 'nonce': 2933589376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6830', 'nonce': 2933589376}, {'type': 'v1', 'addr': '172.21.15.43:6831', 'nonce': 2933589376}]}, 'public_addr': '172.21.15.43:6827/2933589376', 'cluster_addr': '172.21.15.43:6829/2933589376', 'heartbeat_back_addr': '172.21.15.43:6833/2933589376', 'heartbeat_front_addr': '172.21.15.43:6831/2933589376', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '42da30e6-9eec-4ed5-8c73-9ce78aa8ff63', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 420, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6800', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6801', 'nonce': 1945833081}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6802', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6803', 'nonce': 1945833081}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6806', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6807', 'nonce': 1945833081}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6804', 'nonce': 1945833081}, {'type': 'v1', 'addr': '172.21.15.113:6805', 'nonce': 1945833081}]}, 'public_addr': '172.21.15.113:6801/1945833081', 'cluster_addr': '172.21.15.113:6803/1945833081', 'heartbeat_back_addr': '172.21.15.113:6807/1945833081', 'heartbeat_front_addr': '172.21.15.113:6805/1945833081', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '99a11b81-0df7-4b1d-946a-4297ba62e263', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 420, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6808', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6809', 'nonce': 3426360325}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6810', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6811', 'nonce': 3426360325}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6814', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6815', 'nonce': 3426360325}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6812', 'nonce': 3426360325}, {'type': 'v1', 'addr': '172.21.15.113:6813', 'nonce': 3426360325}]}, 'public_addr': '172.21.15.113:6809/3426360325', 'cluster_addr': '172.21.15.113:6811/3426360325', 'heartbeat_back_addr': '172.21.15.113:6815/3426360325', 'heartbeat_front_addr': '172.21.15.113:6813/3426360325', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'cafd0e26-78e9-4164-a43a-96ca8382fefe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 420, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6816', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6817', 'nonce': 1527017698}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6818', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6819', 'nonce': 1527017698}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6822', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6823', 'nonce': 1527017698}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6820', 'nonce': 1527017698}, {'type': 'v1', 'addr': '172.21.15.113:6821', 'nonce': 1527017698}]}, 'public_addr': '172.21.15.113:6817/1527017698', 'cluster_addr': '172.21.15.113:6819/1527017698', 'heartbeat_back_addr': '172.21.15.113:6823/1527017698', 'heartbeat_front_addr': '172.21.15.113:6821/1527017698', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e9d59662-9132-48f4-9e95-842a3cfbd79f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 228, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6824', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6825', 'nonce': 3182476015}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6826', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6827', 'nonce': 3182476015}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6830', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6831', 'nonce': 3182476015}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6828', 'nonce': 3182476015}, {'type': 'v1', 'addr': '172.21.15.113:6829', 'nonce': 3182476015}]}, 'public_addr': '172.21.15.113:6825/3182476015', 'cluster_addr': '172.21.15.113:6827/3182476015', 'heartbeat_back_addr': '172.21.15.113:6831/3182476015', 'heartbeat_front_addr': '172.21.15.113:6829/3182476015', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T14:02:32.250706+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T14:02:22.586995+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T14:02:41.533018+0000', 'dead_epoch': 166}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-30T14:02:31.974551+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T14:02:28.170275+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T14:01:50.085517+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T14:02:41.285874+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-30T14:02:25.717578+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '1.2', 'mappings': [{'from': 2, 'to': 4}, {'from': 0, 'to': 1}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '5.15', 'mappings': [{'from': 0, 'to': 6}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '11.0', 'osds': [1, 6, 3]}, {'pgid': '11.1', 'osds': [6, 0, 3]}, {'pgid': '11.2', 'osds': [0, 2, 3]}, {'pgid': '11.5', 'osds': [5, 6, 3]}, {'pgid': '11.8', 'osds': [6, 5, 3]}, {'pgid': '11.9', 'osds': [6, 0, 3]}, {'pgid': '11.a', 'osds': [6, 5, 3]}, {'pgid': '11.b', 'osds': [1, 0, 3]}], 'primary_temp': [], 'blocklist': {'172.21.15.43:0/538304234': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:6801/3411723337': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/3656823679': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/2618817647': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:6800/3411723337': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/1206218784': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:0/886631486': '2024-03-31T13:37:41.733515+0000', '172.21.15.43:6800/1339921781': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:6801/1339921781': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:0/2794494528': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:0/564868778': '2024-03-31T13:38:09.321740+0000', '172.21.15.43:6800/3033707884': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/3539398453': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/2414364740': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:6801/3033707884': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/3923597429': '2024-03-31T13:43:27.253365+0000', '172.21.15.43:0/2524269526': '2024-03-31T13:43:27.253365+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-30T14:03:04.754 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.9 2024-03-30T14:03:04.754 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd rm-pg-upmap-items 4.9 2024-03-30T14:03:04.839 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:04.866 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:04.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:04.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:04.968 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:05.000 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:05.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:05.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:05.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:04 smithi043 ceph-mon[29963]: pgmap v932: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 6.1 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 42 KiB/s rd, 58 KiB/s wr, 979 op/s; 45316/302481 objects misplaced (14.981%); 30 MiB/s, 487 objects/s recovering 2024-03-30T14:03:05.045 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:04 smithi043 ceph-mon[42099]: pgmap v932: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 6.1 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 42 KiB/s rd, 58 KiB/s wr, 979 op/s; 45316/302481 objects misplaced (14.981%); 30 MiB/s, 487 objects/s recovering 2024-03-30T14:03:05.102 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:05.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:05.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:04 smithi113 ceph-mon[34341]: pgmap v932: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 6.1 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 42 KiB/s rd, 58 KiB/s wr, 979 op/s; 45316/302481 objects misplaced (14.981%); 30 MiB/s, 487 objects/s recovering 2024-03-30T14:03:05.259 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:05.287 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:05.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:05.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:05.389 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:05.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:05.494 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:05.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:05.596 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:05.622 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:05.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:05.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:05.724 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:05.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:05.781 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:05.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:05.882 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:05.910 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:05.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:05.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:06.012 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:06.047 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:06.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:06.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:06.090 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:05 smithi043 ceph-mon[29963]: 5.1a scrub starts 2024-03-30T14:03:06.090 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:05 smithi043 ceph-mon[29963]: 5.1a scrub ok 2024-03-30T14:03:06.090 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:05 smithi043 ceph-mon[42099]: 5.1a scrub starts 2024-03-30T14:03:06.090 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:05 smithi043 ceph-mon[42099]: 5.1a scrub ok 2024-03-30T14:03:06.149 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:06.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:06.199 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:06.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:06.210 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:03:06.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:05 smithi113 ceph-mon[34341]: 5.1a scrub starts 2024-03-30T14:03:06.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:05 smithi113 ceph-mon[34341]: 5.1a scrub ok 2024-03-30T14:03:06.215 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:03:06.242 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:03:06.300 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:06.326 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:06.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:06.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:06.365 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:03:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:03:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:03:06.428 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:06.456 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:06.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:06.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:06.558 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:06.589 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:06.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:06.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:06.691 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:06.722 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:06.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:06.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:06.824 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:06.864 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:06.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:06.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:06.966 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:06.997 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:06.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:06.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:07.099 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:07.126 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:07.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:07.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:07.228 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:07.233 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:06 smithi043 ceph-mon[29963]: pgmap v933: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 35 KiB/s rd, 0 B/s wr, 1.14k op/s; 42833/294054 objects misplaced (14.566%); 24 MiB/s, 383 objects/s recovering 2024-03-30T14:03:07.233 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:06 smithi043 ceph-mon[29963]: 3.5 scrub starts 2024-03-30T14:03:07.233 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:06 smithi043 ceph-mon[29963]: 3.5 scrub ok 2024-03-30T14:03:07.233 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:06 smithi043 ceph-mon[42099]: pgmap v933: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 35 KiB/s rd, 0 B/s wr, 1.14k op/s; 42833/294054 objects misplaced (14.566%); 24 MiB/s, 383 objects/s recovering 2024-03-30T14:03:07.233 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:06 smithi043 ceph-mon[42099]: 3.5 scrub starts 2024-03-30T14:03:07.233 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:06 smithi043 ceph-mon[42099]: 3.5 scrub ok 2024-03-30T14:03:07.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:07.259 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:07.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:07.360 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:07.398 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:07.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:07.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:07.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:06 smithi113 ceph-mon[34341]: pgmap v933: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 35 KiB/s rd, 0 B/s wr, 1.14k op/s; 42833/294054 objects misplaced (14.566%); 24 MiB/s, 383 objects/s recovering 2024-03-30T14:03:07.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:06 smithi113 ceph-mon[34341]: 3.5 scrub starts 2024-03-30T14:03:07.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:06 smithi113 ceph-mon[34341]: 3.5 scrub ok 2024-03-30T14:03:07.500 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:07.539 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:07.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:07.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:07.641 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:07.667 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:07.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:07.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:07.769 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:07.812 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:07.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:07.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:07.915 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:07.945 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:07.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:07.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:08.047 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:08.092 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:08.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:08.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:08.194 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:08.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:08.242 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:08.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:08.344 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:08.376 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:08.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:08.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:08.478 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:08.508 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:08.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:08.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:08.610 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:08.650 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:08.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:08.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:08.752 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:08.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:08.781 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:08.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:08.882 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:08.923 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:08.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:08.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:09.025 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:09.039 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T14:03:09.050 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:08 smithi043 ceph-mon[29963]: 3.1d scrub starts 2024-03-30T14:03:09.050 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:08 smithi043 ceph-mon[29963]: 3.1d scrub ok 2024-03-30T14:03:09.050 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:08 smithi043 ceph-mon[29963]: pgmap v934: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 5.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 1.13k op/s; 41541/288477 objects misplaced (14.400%); 21 MiB/s, 342 objects/s recovering 2024-03-30T14:03:09.051 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:08 smithi043 ceph-mon[42099]: 3.1d scrub starts 2024-03-30T14:03:09.051 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:08 smithi043 ceph-mon[42099]: 3.1d scrub ok 2024-03-30T14:03:09.051 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:08 smithi043 ceph-mon[42099]: pgmap v934: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 5.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 1.13k op/s; 41541/288477 objects misplaced (14.400%); 21 MiB/s, 342 objects/s recovering 2024-03-30T14:03:09.070 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:09.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:09.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:09.095 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T14:03:09.147 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T14:03:09.172 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:09.189 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T14:03:09.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:09.216 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:09.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:09.229 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T14:03:09.269 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T14:03:09.311 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T14:03:09.317 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:09.350 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T14:03:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:03:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:03:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:03:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:03:09.359 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:03:09.359 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:03:09.359 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:03:09.359 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:03:09.360 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:09.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:09.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:09.462 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:09.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:08 smithi113 ceph-mon[34341]: 3.1d scrub starts 2024-03-30T14:03:09.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:08 smithi113 ceph-mon[34341]: 3.1d scrub ok 2024-03-30T14:03:09.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:08 smithi113 ceph-mon[34341]: pgmap v934: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 5.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 1.13k op/s; 41541/288477 objects misplaced (14.400%); 21 MiB/s, 342 objects/s recovering 2024-03-30T14:03:09.493 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:09.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:09.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:09.595 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:09.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:09.629 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:09.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:09.730 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:09.759 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:09.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:09.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:09.861 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:09.893 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:09.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:09.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:09.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:03:09 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:03:09.995 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:10.028 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:10.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:10.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:10.130 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:10.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:10.180 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:10.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:10.280 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:10.318 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:09 smithi043 ceph-mon[29963]: 5.b scrub starts 2024-03-30T14:03:10.318 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:09 smithi043 ceph-mon[29963]: 5.b scrub ok 2024-03-30T14:03:10.318 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:09 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2757223282' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:03:10.318 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:09 smithi043 ceph-mon[42099]: 5.b scrub starts 2024-03-30T14:03:10.319 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:09 smithi043 ceph-mon[42099]: 5.b scrub ok 2024-03-30T14:03:10.319 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:09 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2757223282' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:03:10.324 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:10.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:10.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:10.426 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:10.455 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:10.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:10.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:10.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:09 smithi113 ceph-mon[34341]: 5.b scrub starts 2024-03-30T14:03:10.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:09 smithi113 ceph-mon[34341]: 5.b scrub ok 2024-03-30T14:03:10.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:09 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2757223282' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:03:10.557 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:10.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:10.588 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:10.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:10.689 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:10.718 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:10.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:10.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:10.820 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:10.848 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:10.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:10.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:10.950 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:10.974 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:10.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:10.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:11.076 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:11.115 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:11.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:11.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:11.218 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:11.247 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:11.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:11.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:11.349 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:11.377 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:11.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:11.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:11.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:10 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:11.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:10 smithi043 ceph-mon[29963]: pgmap v935: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 5.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 964 B/s rd, 0 B/s wr, 1.03k op/s; 38898/278160 objects misplaced (13.984%); 20 MiB/s, 314 objects/s recovering 2024-03-30T14:03:11.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:10 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:11.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:10 smithi043 ceph-mon[42099]: pgmap v935: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 5.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 964 B/s rd, 0 B/s wr, 1.03k op/s; 38898/278160 objects misplaced (13.984%); 20 MiB/s, 314 objects/s recovering 2024-03-30T14:03:11.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:10 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:11.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:10 smithi113 ceph-mon[34341]: pgmap v935: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 5.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 964 B/s rd, 0 B/s wr, 1.03k op/s; 38898/278160 objects misplaced (13.984%); 20 MiB/s, 314 objects/s recovering 2024-03-30T14:03:11.479 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:11.516 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:11.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:11.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:11.618 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:11.648 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:11.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:11.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:11.750 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:11.778 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:11.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:11.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:11.880 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:11.908 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:11.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:11.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:12.010 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:12.037 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:12.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:12.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:12.139 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:12.169 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:12.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:12.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:12.271 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:12.298 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:12.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:12.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:12.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:11 smithi043 ceph-mon[29963]: 4.18 scrub starts 2024-03-30T14:03:12.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:12 smithi043 ceph-mon[29963]: 4.18 scrub ok 2024-03-30T14:03:12.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:11 smithi043 ceph-mon[42099]: 4.18 scrub starts 2024-03-30T14:03:12.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:12 smithi043 ceph-mon[42099]: 4.18 scrub ok 2024-03-30T14:03:12.400 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:12.430 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:12.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:12.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:12.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:11 smithi113 ceph-mon[34341]: 4.18 scrub starts 2024-03-30T14:03:12.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:11 smithi113 ceph-mon[34341]: 4.18 scrub ok 2024-03-30T14:03:12.532 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:12.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:12.573 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:12.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:12.674 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:12.703 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:12.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:12.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:12.805 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:12.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:12.846 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:12.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:12.947 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:12.987 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:12.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:12.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:13.089 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:13.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:13.115 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:13.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:13.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:12 smithi043 ceph-mon[29963]: pgmap v936: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 37594/271776 objects misplaced (13.833%); 19 MiB/s, 311 objects/s recovering 2024-03-30T14:03:13.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:12 smithi043 ceph-mon[29963]: 5.7 scrub starts 2024-03-30T14:03:13.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:12 smithi043 ceph-mon[29963]: 5.7 scrub ok 2024-03-30T14:03:13.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:12 smithi043 ceph-mon[42099]: pgmap v936: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 37594/271776 objects misplaced (13.833%); 19 MiB/s, 311 objects/s recovering 2024-03-30T14:03:13.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:12 smithi043 ceph-mon[42099]: 5.7 scrub starts 2024-03-30T14:03:13.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:13 smithi043 ceph-mon[42099]: 5.7 scrub ok 2024-03-30T14:03:13.216 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:13.254 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:13.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:13.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:13.356 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:13.385 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:13.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:13.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:13.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:12 smithi113 ceph-mon[34341]: pgmap v936: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 37594/271776 objects misplaced (13.833%); 19 MiB/s, 311 objects/s recovering 2024-03-30T14:03:13.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:12 smithi113 ceph-mon[34341]: 5.7 scrub starts 2024-03-30T14:03:13.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:12 smithi113 ceph-mon[34341]: 5.7 scrub ok 2024-03-30T14:03:13.487 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:13.527 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:13.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:13.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:13.629 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:13.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:13.661 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:13.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:13.762 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:13.793 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:13.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:13.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:13.895 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:13.933 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:13.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:13.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:14.035 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:14.078 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:14.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:14.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:14.180 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:14.221 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:14.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:14.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:14.322 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:14.353 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:14.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:14.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:14.376 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:14 smithi043 ceph-mon[29963]: 4.1 scrub starts 2024-03-30T14:03:14.376 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:14 smithi043 ceph-mon[29963]: 4.1 scrub ok 2024-03-30T14:03:14.376 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:14 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:03:14.376 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:14 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:03:14.377 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:14 smithi043 ceph-mon[29963]: 6.7 deep-scrub starts 2024-03-30T14:03:14.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:14 smithi043 ceph-mon[42099]: 4.1 scrub starts 2024-03-30T14:03:14.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:14 smithi043 ceph-mon[42099]: 4.1 scrub ok 2024-03-30T14:03:14.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:14 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:03:14.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:14 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:03:14.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:14 smithi043 ceph-mon[42099]: 6.7 deep-scrub starts 2024-03-30T14:03:14.454 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:14.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:14 smithi113 ceph-mon[34341]: 4.1 scrub starts 2024-03-30T14:03:14.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:14 smithi113 ceph-mon[34341]: 4.1 scrub ok 2024-03-30T14:03:14.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:14 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:03:14.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:14 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:03:14.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:14 smithi113 ceph-mon[34341]: 6.7 deep-scrub starts 2024-03-30T14:03:14.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:14.494 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:14.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:14.595 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:14.625 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:14.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:14.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:14.727 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:14.768 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:14.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:14.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:14.870 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:14.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:14.917 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:14.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:15.019 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:15.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:15.063 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:15.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:15.164 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:15.195 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:15.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:15.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:15.297 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:15.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:15.337 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:15.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:15 smithi043 ceph-mon[29963]: 6.7 deep-scrub ok 2024-03-30T14:03:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:15 smithi043 ceph-mon[29963]: pgmap v937: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 965 op/s; 37274/269766 objects misplaced (13.817%); 17 MiB/s, 276 objects/s recovering 2024-03-30T14:03:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:15 smithi043 ceph-mon[29963]: osdmap e423: 8 total, 8 up, 6 in 2024-03-30T14:03:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:15 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:15.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:15 smithi043 ceph-mon[42099]: 6.7 deep-scrub ok 2024-03-30T14:03:15.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:15 smithi043 ceph-mon[42099]: pgmap v937: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 965 op/s; 37274/269766 objects misplaced (13.817%); 17 MiB/s, 276 objects/s recovering 2024-03-30T14:03:15.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:15 smithi043 ceph-mon[42099]: osdmap e423: 8 total, 8 up, 6 in 2024-03-30T14:03:15.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:15 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:15.438 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:15.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:15 smithi113 ceph-mon[34341]: 6.7 deep-scrub ok 2024-03-30T14:03:15.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:15 smithi113 ceph-mon[34341]: pgmap v937: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 965 op/s; 37274/269766 objects misplaced (13.817%); 17 MiB/s, 276 objects/s recovering 2024-03-30T14:03:15.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:15 smithi113 ceph-mon[34341]: osdmap e423: 8 total, 8 up, 6 in 2024-03-30T14:03:15.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:15 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:15.484 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:15.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:15.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:15.586 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:15.615 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:15.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:15.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:15.893 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:15.938 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:15.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:15.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:16.030 INFO:teuthology.orchestra.run.smithi043.stderr:clear 4.9 pg_upmap_items mapping 2024-03-30T14:03:16.040 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:16.078 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:16.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:16.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:16.180 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:16.209 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:16.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:16.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:16.311 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:16.328 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T14:03:16.328 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T14:03:16.328 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-03-30T14:03:16.328 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-03-30T14:03:16.328 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T14:03:16.329 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:16 smithi043 ceph-mon[29963]: 6.16 deep-scrub starts 2024-03-30T14:03:16.329 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:16 smithi043 ceph-mon[29963]: 3.7 scrub starts 2024-03-30T14:03:16.329 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:16 smithi043 ceph-mon[29963]: 3.7 scrub ok 2024-03-30T14:03:16.329 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:16 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2486403739' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]: dispatch 2024-03-30T14:03:16.329 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:16 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]: dispatch 2024-03-30T14:03:16.329 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:03:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:03:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:03:16.329 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:16 smithi043 ceph-mon[42099]: 6.16 deep-scrub starts 2024-03-30T14:03:16.329 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:16 smithi043 ceph-mon[42099]: 3.7 scrub starts 2024-03-30T14:03:16.330 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:16 smithi043 ceph-mon[42099]: 3.7 scrub ok 2024-03-30T14:03:16.330 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:16 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2486403739' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]: dispatch 2024-03-30T14:03:16.330 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:16 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]: dispatch 2024-03-30T14:03:16.342 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:16.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:16.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:16.444 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:16.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:16 smithi113 ceph-mon[34341]: 6.16 deep-scrub starts 2024-03-30T14:03:16.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:16 smithi113 ceph-mon[34341]: 3.7 scrub starts 2024-03-30T14:03:16.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:16 smithi113 ceph-mon[34341]: 3.7 scrub ok 2024-03-30T14:03:16.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:16 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2486403739' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]: dispatch 2024-03-30T14:03:16.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:16 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]: dispatch 2024-03-30T14:03:16.476 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:16.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:16.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:16.578 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:16.617 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:16.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:16.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:16.719 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:16.747 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:16.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:16.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:16.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:16.895 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:16.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:16.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:16.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:17.039 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:17.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:17.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:17.106 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:17 smithi043 ceph-mon[29963]: 6.16 deep-scrub ok 2024-03-30T14:03:17.106 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:17 smithi043 ceph-mon[29963]: pgmap v939: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 5.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s; 34946/262026 objects misplaced (13.337%); 21 MiB/s, 343 objects/s recovering 2024-03-30T14:03:17.106 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:17 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]': finished 2024-03-30T14:03:17.107 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:17 smithi043 ceph-mon[29963]: osdmap e424: 8 total, 8 up, 6 in 2024-03-30T14:03:17.107 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:17 smithi043 ceph-mon[42099]: 6.16 deep-scrub ok 2024-03-30T14:03:17.107 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:17 smithi043 ceph-mon[42099]: pgmap v939: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 5.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s; 34946/262026 objects misplaced (13.337%); 21 MiB/s, 343 objects/s recovering 2024-03-30T14:03:17.107 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:17 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]': finished 2024-03-30T14:03:17.107 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:17 smithi043 ceph-mon[42099]: osdmap e424: 8 total, 8 up, 6 in 2024-03-30T14:03:17.140 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:17.188 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:17.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:17.290 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:17.319 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:17.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:17.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:17.421 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:17.451 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:17.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:17.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:17.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:17 smithi113 ceph-mon[34341]: 6.16 deep-scrub ok 2024-03-30T14:03:17.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:17 smithi113 ceph-mon[34341]: pgmap v939: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 5.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s; 34946/262026 objects misplaced (13.337%); 21 MiB/s, 343 objects/s recovering 2024-03-30T14:03:17.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:17 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]': finished 2024-03-30T14:03:17.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:17 smithi113 ceph-mon[34341]: osdmap e424: 8 total, 8 up, 6 in 2024-03-30T14:03:17.553 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:17.583 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:17.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:17.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:17.685 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:17.713 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:17.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:17.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:17.721 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-03-30T14:03:17.814 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:17.841 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:17.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:17.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:17.943 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:17.970 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:17.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:18.072 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:18.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:18.100 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:18.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:18.201 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:18.229 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:18.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:18.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:18.331 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:18.375 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:18.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:18.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:18.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:18 smithi043 ceph-mon[29963]: osdmap e425: 8 total, 8 up, 6 in 2024-03-30T14:03:18.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:18 smithi043 ceph-mon[42099]: osdmap e425: 8 total, 8 up, 6 in 2024-03-30T14:03:18.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:18 smithi113 ceph-mon[34341]: osdmap e425: 8 total, 8 up, 6 in 2024-03-30T14:03:18.477 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:18.505 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:18.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:18.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:18.607 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:18.646 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:18.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:18.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:18.748 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:18.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:18.804 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:18.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:18.905 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:18.937 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:18.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:18.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:19.039 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:19.077 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:19.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:19.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:19.179 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:19.208 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:19.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:19.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:19.310 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:19.340 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:19.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:19.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:19.365 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:19 smithi113 ceph-mon[34341]: pgmap v942: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 5.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 871 op/s; 33760/256107 objects misplaced (13.182%); 18 MiB/s, 284 objects/s recovering 2024-03-30T14:03:19.365 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:19 smithi113 ceph-mon[34341]: 5.10 scrub starts 2024-03-30T14:03:19.365 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:19 smithi113 ceph-mon[34341]: 5.10 scrub ok 2024-03-30T14:03:19.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:19 smithi043 ceph-mon[29963]: pgmap v942: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 5.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 871 op/s; 33760/256107 objects misplaced (13.182%); 18 MiB/s, 284 objects/s recovering 2024-03-30T14:03:19.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:19 smithi043 ceph-mon[29963]: 5.10 scrub starts 2024-03-30T14:03:19.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:19 smithi043 ceph-mon[29963]: 5.10 scrub ok 2024-03-30T14:03:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:19 smithi043 ceph-mon[42099]: pgmap v942: 168 pgs: 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 160 active+clean; 5.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 871 op/s; 33760/256107 objects misplaced (13.182%); 18 MiB/s, 284 objects/s recovering 2024-03-30T14:03:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:19 smithi043 ceph-mon[42099]: 5.10 scrub starts 2024-03-30T14:03:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:19 smithi043 ceph-mon[42099]: 5.10 scrub ok 2024-03-30T14:03:19.442 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:19.469 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:19.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:19.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:19.571 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:19.614 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:19.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:19.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:19.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:03:19 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:03:19.716 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:19.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:19.760 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:19.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:19.861 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:19.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T14:03:19.888 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:19.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:19.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:19.991 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:20.016 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:20.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:20.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:20.118 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:20.147 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:20.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:20.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:20.248 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:20.283 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:20.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:20.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:20.385 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:20.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:20.428 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:20.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:20.529 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:20.557 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:20.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:20.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:20.659 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:20.698 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:20.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:20.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:20.800 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:20.813 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:03:20.828 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:20.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:20.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:20.930 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:20.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:20.960 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:20.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:21.013 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:03:21.059 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-03-30T14:03:21.060 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:21.085 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:21.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:21.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:21.187 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:21.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:21.217 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:21.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:21.318 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:21.342 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:21.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:21.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:21 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:21 smithi043 ceph-mon[29963]: pgmap v943: 168 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 159 active+clean; 5.0 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s; 31286/245802 objects misplaced (12.728%); 31 MiB/s, 491 objects/s recovering 2024-03-30T14:03:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:21 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1903655551' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:03:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:21 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:03:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:21 smithi043 ceph-mon[29963]: 1.8 scrub starts 2024-03-30T14:03:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:21 smithi043 ceph-mon[29963]: 1.8 scrub ok 2024-03-30T14:03:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:21 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:21 smithi043 ceph-mon[42099]: pgmap v943: 168 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 159 active+clean; 5.0 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s; 31286/245802 objects misplaced (12.728%); 31 MiB/s, 491 objects/s recovering 2024-03-30T14:03:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:21 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1903655551' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:03:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:21 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:03:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:21 smithi043 ceph-mon[42099]: 1.8 scrub starts 2024-03-30T14:03:21.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:21 smithi043 ceph-mon[42099]: 1.8 scrub ok 2024-03-30T14:03:21.444 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:21.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:21 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:21.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:21 smithi113 ceph-mon[34341]: pgmap v943: 168 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 159 active+clean; 5.0 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s; 31286/245802 objects misplaced (12.728%); 31 MiB/s, 491 objects/s recovering 2024-03-30T14:03:21.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:21 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1903655551' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:03:21.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:21 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:03:21.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:21 smithi113 ceph-mon[34341]: 1.8 scrub starts 2024-03-30T14:03:21.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:21 smithi113 ceph-mon[34341]: 1.8 scrub ok 2024-03-30T14:03:21.477 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:21.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:21.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:21.578 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:21.645 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:21.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:21.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:21.747 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:21.788 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:21.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:21.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:21.890 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:21.918 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:21.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:21.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:22.019 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:22.048 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:22.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:22.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:22.150 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:22.160 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 5] out_osds: [7, 3] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-03-30T14:03:22.160 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T14:03:22.160 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-30T14:03:22.161 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-30T14:03:22.161 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-30T14:03:22.161 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-03-30T14:03:22.177 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:22.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:22.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:22.200 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:22 smithi043 ceph-mon[29963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:03:22.201 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:22 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:03:22.201 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:22 smithi043 ceph-mon[29963]: osdmap e426: 8 total, 8 up, 6 in 2024-03-30T14:03:22.201 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:22 smithi043 ceph-mon[42099]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:03:22.201 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:22 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:03:22.201 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:22 smithi043 ceph-mon[42099]: osdmap e426: 8 total, 8 up, 6 in 2024-03-30T14:03:22.280 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:22.311 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:22.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:22.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:22.413 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:22.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:22.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:22.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:22 smithi113 ceph-mon[34341]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:03:22.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:22 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:03:22.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:22 smithi113 ceph-mon[34341]: osdmap e426: 8 total, 8 up, 6 in 2024-03-30T14:03:22.563 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:22.617 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:22.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:22.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:22.719 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:22.750 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:22.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:22.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:22.852 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:22.893 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:22.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:22.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:22.995 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:23.032 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:23.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:23.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:23.134 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:23.163 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:23.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:23.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:23.264 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:23.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:23.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:23.319 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:23 smithi043 ceph-mon[29963]: pgmap v945: 168 pgs: 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 159 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.23k op/s; 29998/239877 objects misplaced (12.506%); 21 MiB/s, 341 objects/s recovering 2024-03-30T14:03:23.319 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:23 smithi043 ceph-mon[29963]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T14:03:23.319 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:23 smithi043 ceph-mon[29963]: osdmap e427: 8 total, 8 up, 6 in 2024-03-30T14:03:23.319 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:23 smithi043 ceph-mon[42099]: pgmap v945: 168 pgs: 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 159 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.23k op/s; 29998/239877 objects misplaced (12.506%); 21 MiB/s, 341 objects/s recovering 2024-03-30T14:03:23.319 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:23 smithi043 ceph-mon[42099]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T14:03:23.319 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:23 smithi043 ceph-mon[42099]: osdmap e427: 8 total, 8 up, 6 in 2024-03-30T14:03:23.398 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:23.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:23.427 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:23.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:23.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:23 smithi113 ceph-mon[34341]: pgmap v945: 168 pgs: 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 159 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.23k op/s; 29998/239877 objects misplaced (12.506%); 21 MiB/s, 341 objects/s recovering 2024-03-30T14:03:23.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:23 smithi113 ceph-mon[34341]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T14:03:23.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:23 smithi113 ceph-mon[34341]: osdmap e427: 8 total, 8 up, 6 in 2024-03-30T14:03:23.528 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:23.556 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:23.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:23.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:23.658 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:23.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:23.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:23.790 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:23.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:23.819 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:23.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:23.920 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:23.948 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:23.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:23.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:24.050 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:24.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:24.111 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:24.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:24.212 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:24.243 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:24.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:24.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:24.345 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:24.387 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:24.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:24.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:24.489 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:24.534 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:24.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:24.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:24.636 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:24.674 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:24.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:24.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:24.777 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:24.816 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:24.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:24.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:24.918 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:24.946 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:24.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:24.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:25.048 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:25.078 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:25.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:25.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:25.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:24 smithi043 ceph-mon[29963]: pgmap v947: 168 pgs: 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 159 active+clean; 4.8 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 755 B/s rd, 0 B/s wr, 882 op/s; 29714/238194 objects misplaced (12.475%); 16 MiB/s, 251 objects/s recovering 2024-03-30T14:03:25.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:24 smithi043 ceph-mon[29963]: osdmap e428: 8 total, 8 up, 6 in 2024-03-30T14:03:25.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:24 smithi043 ceph-mon[42099]: pgmap v947: 168 pgs: 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 159 active+clean; 4.8 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 755 B/s rd, 0 B/s wr, 882 op/s; 29714/238194 objects misplaced (12.475%); 16 MiB/s, 251 objects/s recovering 2024-03-30T14:03:25.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:24 smithi043 ceph-mon[42099]: osdmap e428: 8 total, 8 up, 6 in 2024-03-30T14:03:25.179 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:25.210 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:25.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:25.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:25.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:24 smithi113 ceph-mon[34341]: pgmap v947: 168 pgs: 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 159 active+clean; 4.8 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 755 B/s rd, 0 B/s wr, 882 op/s; 29714/238194 objects misplaced (12.475%); 16 MiB/s, 251 objects/s recovering 2024-03-30T14:03:25.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:24 smithi113 ceph-mon[34341]: osdmap e428: 8 total, 8 up, 6 in 2024-03-30T14:03:25.312 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:25.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:25.352 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:25.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:25.453 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:25.504 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:25.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:25.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:25.606 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:25.632 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:25.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:25.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:25.734 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T14:03:25.734 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-03-30T14:03:25.734 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-03-30T14:03:25.734 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-03-30T14:03:25.734 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.734 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61015 9.d 9:bd835e52:::benchmark_data_smithi043_145506_object61014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T14:03:25.734 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.849021+0000", 2024-03-30T14:03:25.734 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 590.88578512300001, 2024-03-30T14:03:25.735 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2177895259999998, 2024-03-30T14:03:25.735 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:25.735 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:25.735 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-03-30T14:03:25.735 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:25.735 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:03:25.735 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:03:25.735 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61015 2024-03-30T14:03:25.735 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.735 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:25.735 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.735 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:25.735 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849021+0000", 2024-03-30T14:03:25.735 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.736 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.736 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.736 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:25.736 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849021+0000", 2024-03-30T14:03:25.736 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.736 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.736 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.736 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:25.736 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849022+0000", 2024-03-30T14:03:25.736 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.254e-06 2024-03-30T14:03:25.736 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.736 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.736 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:25.736 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849056+0000", 2024-03-30T14:03:25.737 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4051000000000003e-05 2024-03-30T14:03:25.737 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.737 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.737 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:25.737 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849067+0000", 2024-03-30T14:03:25.737 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0173000000000001e-05 2024-03-30T14:03:25.737 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.737 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.737 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:25.737 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849230+0000", 2024-03-30T14:03:25.737 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000163086 2024-03-30T14:03:25.737 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.737 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.738 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.738 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854000+0000", 2024-03-30T14:03:25.738 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.004770552 2024-03-30T14:03:25.738 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.738 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.738 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.738 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854011+0000", 2024-03-30T14:03:25.738 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0679000000000001e-05 2024-03-30T14:03:25.738 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.738 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.738 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.738 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845267+0000", 2024-03-30T14:03:25.739 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99125657899999997 2024-03-30T14:03:25.739 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.739 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.739 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.739 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845280+0000", 2024-03-30T14:03:25.739 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2313999999999999e-05 2024-03-30T14:03:25.739 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.739 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.739 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.739 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.065485+0000", 2024-03-30T14:03:25.739 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.220205703 2024-03-30T14:03:25.739 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.739 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.739 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:25.740 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.066811+0000", 2024-03-30T14:03:25.740 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0013251350000000001 2024-03-30T14:03:25.740 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.740 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.740 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.740 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.740 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.740 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61042 9.d 9:bd1a2220:::benchmark_data_smithi043_145506_object61041:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T14:03:25.740 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.849245+0000", 2024-03-30T14:03:25.740 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 590.88556088600001, 2024-03-30T14:03:25.740 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4398415450000002, 2024-03-30T14:03:25.740 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:25.740 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:25.740 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:25.741 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:25.741 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:03:25.741 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:03:25.741 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61042 2024-03-30T14:03:25.741 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.741 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:25.741 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.741 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:25.741 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849245+0000", 2024-03-30T14:03:25.741 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.741 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.741 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.741 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:25.742 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849245+0000", 2024-03-30T14:03:25.742 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.742 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.742 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.742 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:25.742 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849246+0000", 2024-03-30T14:03:25.742 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9999999999999997e-07 2024-03-30T14:03:25.742 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.742 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.742 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:25.742 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849272+0000", 2024-03-30T14:03:25.742 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5956999999999999e-05 2024-03-30T14:03:25.742 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.743 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.743 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:25.743 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849281+0000", 2024-03-30T14:03:25.743 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2089999999999994e-06 2024-03-30T14:03:25.743 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.743 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.743 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:25.743 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849288+0000", 2024-03-30T14:03:25.743 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8920000000000003e-06 2024-03-30T14:03:25.743 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.743 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.743 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.743 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854033+0000", 2024-03-30T14:03:25.744 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0047449299999999996 2024-03-30T14:03:25.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.744 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.744 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.744 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854040+0000", 2024-03-30T14:03:25.744 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.737e-06 2024-03-30T14:03:25.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.744 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.744 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.744 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845318+0000", 2024-03-30T14:03:25.744 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.991278561 2024-03-30T14:03:25.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.744 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.744 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.744 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845327+0000", 2024-03-30T14:03:25.744 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.166e-06 2024-03-30T14:03:25.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.745 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.745 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.745 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.233130+0000", 2024-03-30T14:03:25.745 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.38780266 2024-03-30T14:03:25.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.745 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.745 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:25.745 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.233697+0000", 2024-03-30T14:03:25.745 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00056653400000000005 2024-03-30T14:03:25.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.745 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.746 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T14:03:25.746 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.233994+0000", 2024-03-30T14:03:25.746 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00029792300000000001 2024-03-30T14:03:25.746 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.746 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.746 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.746 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.257518+0000", 2024-03-30T14:03:25.746 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.023523143 2024-03-30T14:03:25.746 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.746 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.746 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.746 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.257567+0000", 2024-03-30T14:03:25.746 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8928999999999999e-05 2024-03-30T14:03:25.747 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.747 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.747 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:25.747 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.286372+0000", 2024-03-30T14:03:25.747 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.028805173 2024-03-30T14:03:25.747 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.747 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.747 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:25.747 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.286429+0000", 2024-03-30T14:03:25.747 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7568000000000002e-05 2024-03-30T14:03:25.747 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.747 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.747 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:25.747 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.289087+0000", 2024-03-30T14:03:25.748 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026575629999999999 2024-03-30T14:03:25.748 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.748 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.748 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.748 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.748 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.748 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61052 9.d 9:bc7be88d:::benchmark_data_smithi043_145506_object61051:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T14:03:25.748 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.849296+0000", 2024-03-30T14:03:25.748 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 590.88551051000002, 2024-03-30T14:03:25.748 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4426081989999999, 2024-03-30T14:03:25.748 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:25.748 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:25.748 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:25.748 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:25.749 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:03:25.749 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:03:25.749 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61052 2024-03-30T14:03:25.749 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.749 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:25.749 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.749 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:25.749 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849296+0000", 2024-03-30T14:03:25.749 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.749 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.749 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.750 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:25.750 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849296+0000", 2024-03-30T14:03:25.750 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.750 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.750 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.750 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:25.750 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849296+0000", 2024-03-30T14:03:25.750 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2e-07 2024-03-30T14:03:25.750 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.750 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.750 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:25.750 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849427+0000", 2024-03-30T14:03:25.751 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013042800000000001 2024-03-30T14:03:25.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.751 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.751 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:25.751 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849436+0000", 2024-03-30T14:03:25.751 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2010000000000005e-06 2024-03-30T14:03:25.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.751 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.751 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:25.751 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849443+0000", 2024-03-30T14:03:25.751 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1339999999999998e-06 2024-03-30T14:03:25.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.751 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.751 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.751 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854060+0000", 2024-03-30T14:03:25.752 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0046171880000000004 2024-03-30T14:03:25.752 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.752 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.752 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.752 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854067+0000", 2024-03-30T14:03:25.752 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.72e-06 2024-03-30T14:03:25.752 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.752 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.752 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.752 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845359+0000", 2024-03-30T14:03:25.752 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99129192799999999 2024-03-30T14:03:25.752 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.752 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.753 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.753 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845367+0000", 2024-03-30T14:03:25.753 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1249999999999993e-06 2024-03-30T14:03:25.753 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.753 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.753 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.753 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.250580+0000", 2024-03-30T14:03:25.753 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4052128429999999 2024-03-30T14:03:25.753 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.753 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.753 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:25.753 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.250754+0000", 2024-03-30T14:03:25.753 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017425599999999999 2024-03-30T14:03:25.754 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.754 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.754 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T14:03:25.754 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.250985+0000", 2024-03-30T14:03:25.754 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00023110900000000001 2024-03-30T14:03:25.754 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.754 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.754 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.754 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.257611+0000", 2024-03-30T14:03:25.754 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0066258840000000003 2024-03-30T14:03:25.754 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.755 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.755 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.755 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.257656+0000", 2024-03-30T14:03:25.755 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5240000000000001e-05 2024-03-30T14:03:25.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.755 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.755 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:25.755 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.289106+0000", 2024-03-30T14:03:25.755 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.031449612000000002 2024-03-30T14:03:25.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.755 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.755 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:25.755 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.289162+0000", 2024-03-30T14:03:25.756 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6039999999999999e-05 2024-03-30T14:03:25.756 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.756 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.756 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:25.756 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.291904+0000", 2024-03-30T14:03:25.756 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002742071 2024-03-30T14:03:25.756 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.756 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.756 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.756 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.756 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.756 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61062 9.d 9:be3b0b08:::benchmark_data_smithi043_145506_object61061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T14:03:25.756 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.849454+0000", 2024-03-30T14:03:25.756 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 590.88535224400005, 2024-03-30T14:03:25.757 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4481625989999998, 2024-03-30T14:03:25.757 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:25.757 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:25.757 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:25.757 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:25.757 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:03:25.757 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:03:25.757 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61062 2024-03-30T14:03:25.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.757 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:25.757 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.757 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:25.757 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849454+0000", 2024-03-30T14:03:25.758 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.758 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.758 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:25.758 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849454+0000", 2024-03-30T14:03:25.758 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.758 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.758 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:25.758 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849455+0000", 2024-03-30T14:03:25.758 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5499999999999998e-07 2024-03-30T14:03:25.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.758 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.759 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:25.759 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849502+0000", 2024-03-30T14:03:25.759 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7818999999999997e-05 2024-03-30T14:03:25.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.759 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:25.759 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849510+0000", 2024-03-30T14:03:25.759 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4189999999999996e-06 2024-03-30T14:03:25.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.759 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:25.759 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849516+0000", 2024-03-30T14:03:25.759 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9309999999999996e-06 2024-03-30T14:03:25.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.760 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.760 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854101+0000", 2024-03-30T14:03:25.760 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0045853860000000003 2024-03-30T14:03:25.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.760 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.760 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854108+0000", 2024-03-30T14:03:25.760 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4070000000000004e-06 2024-03-30T14:03:25.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.760 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.760 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845395+0000", 2024-03-30T14:03:25.760 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99128673099999998 2024-03-30T14:03:25.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.761 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.761 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845404+0000", 2024-03-30T14:03:25.761 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4540000000000007e-06 2024-03-30T14:03:25.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.761 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.761 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.257690+0000", 2024-03-30T14:03:25.761 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4122865120000001 2024-03-30T14:03:25.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.761 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:25.761 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.257880+0000", 2024-03-30T14:03:25.761 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000190309 2024-03-30T14:03:25.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.762 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T14:03:25.762 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.258100+0000", 2024-03-30T14:03:25.762 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00021951300000000001 2024-03-30T14:03:25.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.762 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.762 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.762 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:25.763 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.260161+0000", 2024-03-30T14:03:25.763 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0020608380000000002 2024-03-30T14:03:25.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.763 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.764 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.764 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.276325+0000", 2024-03-30T14:03:25.764 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.016164380999999999 2024-03-30T14:03:25.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.764 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.764 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:25.764 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.291984+0000", 2024-03-30T14:03:25.764 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.015658436000000001 2024-03-30T14:03:25.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.764 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.764 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:25.764 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.293663+0000", 2024-03-30T14:03:25.764 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0016796319999999999 2024-03-30T14:03:25.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.765 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.765 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:25.765 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.297617+0000", 2024-03-30T14:03:25.765 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0039532760000000004 2024-03-30T14:03:25.765 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.765 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.765 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.765 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.765 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61080 9.d 9:bc5be45f:::benchmark_data_smithi043_145506_object61079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T14:03:25.765 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.849523+0000", 2024-03-30T14:03:25.765 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 590.885282942, 2024-03-30T14:03:25.765 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4527278949999998, 2024-03-30T14:03:25.765 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:25.765 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:25.766 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:25.766 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:25.766 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:03:25.766 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:03:25.766 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61080 2024-03-30T14:03:25.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.766 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:25.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.766 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:25.766 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849523+0000", 2024-03-30T14:03:25.766 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.767 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:25.767 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849523+0000", 2024-03-30T14:03:25.767 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.767 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:25.767 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849524+0000", 2024-03-30T14:03:25.767 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4400000000000001e-07 2024-03-30T14:03:25.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.767 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:25.767 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.849566+0000", 2024-03-30T14:03:25.767 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1857000000000003e-05 2024-03-30T14:03:25.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.768 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:25.768 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850156+0000", 2024-03-30T14:03:25.768 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00059000000000000003 2024-03-30T14:03:25.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.768 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:25.768 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850167+0000", 2024-03-30T14:03:25.768 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1904e-05 2024-03-30T14:03:25.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.768 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.768 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854131+0000", 2024-03-30T14:03:25.769 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0039639139999999998 2024-03-30T14:03:25.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.769 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.769 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854139+0000", 2024-03-30T14:03:25.769 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3780000000000003e-06 2024-03-30T14:03:25.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.769 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.769 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845432+0000", 2024-03-30T14:03:25.769 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99129344100000005 2024-03-30T14:03:25.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.769 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.770 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845441+0000", 2024-03-30T14:03:25.770 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2839999999999995e-06 2024-03-30T14:03:25.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.770 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.770 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.770 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.260197+0000", 2024-03-30T14:03:25.770 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.414755948 2024-03-30T14:03:25.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.770 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.770 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:25.770 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.260372+0000", 2024-03-30T14:03:25.771 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000174864 2024-03-30T14:03:25.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.771 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T14:03:25.771 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.260584+0000", 2024-03-30T14:03:25.771 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000212031 2024-03-30T14:03:25.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.771 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.771 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.276519+0000", 2024-03-30T14:03:25.771 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.015934308000000001 2024-03-30T14:03:25.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.771 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.772 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.276685+0000", 2024-03-30T14:03:25.772 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000166238 2024-03-30T14:03:25.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.772 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:25.772 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.297637+0000", 2024-03-30T14:03:25.772 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.020952028000000001 2024-03-30T14:03:25.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.772 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:25.772 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.297704+0000", 2024-03-30T14:03:25.772 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7503999999999996e-05 2024-03-30T14:03:25.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.772 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:25.773 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.302251+0000", 2024-03-30T14:03:25.773 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0045467520000000003 2024-03-30T14:03:25.773 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.773 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.773 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.773 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61090 9.d 9:b53a3867:::benchmark_data_smithi043_145506_object61089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T14:03:25.773 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850180+0000", 2024-03-30T14:03:25.773 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 590.88462607700001, 2024-03-30T14:03:25.773 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4542226359999999, 2024-03-30T14:03:25.774 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:25.774 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:25.774 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:25.774 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:25.774 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:03:25.774 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:03:25.774 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61090 2024-03-30T14:03:25.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.774 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:25.774 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.774 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:25.774 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850180+0000", 2024-03-30T14:03:25.774 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.775 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:25.775 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850180+0000", 2024-03-30T14:03:25.775 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.775 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:25.775 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850181+0000", 2024-03-30T14:03:25.775 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4099999999999998e-07 2024-03-30T14:03:25.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.775 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:25.775 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850236+0000", 2024-03-30T14:03:25.776 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5337999999999998e-05 2024-03-30T14:03:25.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.776 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:25.776 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850245+0000", 2024-03-30T14:03:25.776 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0699999999999996e-06 2024-03-30T14:03:25.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.776 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:25.776 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850251+0000", 2024-03-30T14:03:25.777 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8819999999999998e-06 2024-03-30T14:03:25.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.777 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.777 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.777 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854162+0000", 2024-03-30T14:03:25.777 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0039110409999999996 2024-03-30T14:03:25.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.778 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.778 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.778 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854169+0000", 2024-03-30T14:03:25.778 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2869999999999999e-06 2024-03-30T14:03:25.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.779 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.779 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845471+0000", 2024-03-30T14:03:25.779 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99130147599999996 2024-03-30T14:03:25.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.780 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.780 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845479+0000", 2024-03-30T14:03:25.780 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3119999999999994e-06 2024-03-30T14:03:25.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.780 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.780 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.781 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.266915+0000", 2024-03-30T14:03:25.781 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.421435687 2024-03-30T14:03:25.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.781 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.781 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:25.781 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.267087+0000", 2024-03-30T14:03:25.781 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017207799999999999 2024-03-30T14:03:25.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.782 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.782 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T14:03:25.782 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.267313+0000", 2024-03-30T14:03:25.782 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00022567400000000001 2024-03-30T14:03:25.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.783 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.783 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.783 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.276854+0000", 2024-03-30T14:03:25.783 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0095411190000000007 2024-03-30T14:03:25.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.783 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.783 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.784 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.277083+0000", 2024-03-30T14:03:25.784 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00022885599999999999 2024-03-30T14:03:25.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.784 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.784 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:25.784 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.302266+0000", 2024-03-30T14:03:25.785 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.025182942999999999 2024-03-30T14:03:25.785 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.785 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.785 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:25.785 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.302375+0000", 2024-03-30T14:03:25.785 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000109098 2024-03-30T14:03:25.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.786 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.786 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:25.786 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.304403+0000", 2024-03-30T14:03:25.787 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002028134 2024-03-30T14:03:25.787 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.787 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.787 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.787 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.787 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.787 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61125 9.d 9:bd41aaa2:::benchmark_data_smithi043_145506_object61124:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T14:03:25.788 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850259+0000", 2024-03-30T14:03:25.788 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 590.88454676599997, 2024-03-30T14:03:25.788 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4559017920000001, 2024-03-30T14:03:25.788 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:25.788 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:25.788 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:25.789 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:25.789 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:03:25.789 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:03:25.789 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61125 2024-03-30T14:03:25.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.789 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:25.790 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.790 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:25.790 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850259+0000", 2024-03-30T14:03:25.790 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.790 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.791 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:25.791 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850259+0000", 2024-03-30T14:03:25.791 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.791 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.791 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:25.791 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850260+0000", 2024-03-30T14:03:25.792 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.39e-07 2024-03-30T14:03:25.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.792 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.792 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:25.792 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850303+0000", 2024-03-30T14:03:25.792 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3279999999999999e-05 2024-03-30T14:03:25.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.793 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:25.793 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850310+0000", 2024-03-30T14:03:25.793 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0589999999999997e-06 2024-03-30T14:03:25.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.794 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:25.794 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850316+0000", 2024-03-30T14:03:25.794 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3859999999999998e-06 2024-03-30T14:03:25.794 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.794 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.795 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.795 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854192+0000", 2024-03-30T14:03:25.795 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003876759 2024-03-30T14:03:25.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.795 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.795 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.796 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854199+0000", 2024-03-30T14:03:25.796 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.109e-06 2024-03-30T14:03:25.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.796 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.796 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.796 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845507+0000", 2024-03-30T14:03:25.797 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99130747100000005 2024-03-30T14:03:25.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.797 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.797 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845515+0000", 2024-03-30T14:03:25.797 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9079999999999995e-06 2024-03-30T14:03:25.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.798 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.798 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.277120+0000", 2024-03-30T14:03:25.798 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4316047009999999 2024-03-30T14:03:25.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.799 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.799 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:25.799 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.277293+0000", 2024-03-30T14:03:25.799 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017354 2024-03-30T14:03:25.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.799 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.800 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T14:03:25.800 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.277471+0000", 2024-03-30T14:03:25.800 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017787799999999999 2024-03-30T14:03:25.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.800 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.801 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.291965+0000", 2024-03-30T14:03:25.801 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.014493852999999999 2024-03-30T14:03:25.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.801 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:25.802 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.304419+0000", 2024-03-30T14:03:25.802 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.012453697 2024-03-30T14:03:25.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.802 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.802 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.802 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.304796+0000", 2024-03-30T14:03:25.803 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00037768200000000001 2024-03-30T14:03:25.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.803 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:25.803 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.304833+0000", 2024-03-30T14:03:25.803 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7119000000000002e-05 2024-03-30T14:03:25.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.804 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:25.804 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.306161+0000", 2024-03-30T14:03:25.804 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001327911 2024-03-30T14:03:25.804 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.805 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.805 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.805 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61137 9.d 9:b639fb2d:::benchmark_data_smithi043_145506_object61136:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T14:03:25.806 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850323+0000", 2024-03-30T14:03:25.806 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 590.88448328200002, 2024-03-30T14:03:25.806 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4574525619999998, 2024-03-30T14:03:25.806 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:25.806 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:25.806 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:25.807 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:25.807 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:03:25.807 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:03:25.807 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61137 2024-03-30T14:03:25.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.807 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:25.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.808 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:25.808 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850323+0000", 2024-03-30T14:03:25.808 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.808 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.809 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:25.809 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850323+0000", 2024-03-30T14:03:25.809 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.809 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.809 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:25.810 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850323+0000", 2024-03-30T14:03:25.812 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.65e-07 2024-03-30T14:03:25.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.812 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:25.812 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850366+0000", 2024-03-30T14:03:25.812 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2586000000000002e-05 2024-03-30T14:03:25.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.813 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:25.813 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850373+0000", 2024-03-30T14:03:25.813 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9079999999999998e-06 2024-03-30T14:03:25.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.814 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.814 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:25.814 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850378+0000", 2024-03-30T14:03:25.814 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6860000000000003e-06 2024-03-30T14:03:25.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.814 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.815 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.815 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854221+0000", 2024-03-30T14:03:25.815 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0038425569999999999 2024-03-30T14:03:25.815 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.815 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.815 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.815 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854228+0000", 2024-03-30T14:03:25.816 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.861e-06 2024-03-30T14:03:25.816 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.816 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.816 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.816 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845543+0000", 2024-03-30T14:03:25.816 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99131543300000002 2024-03-30T14:03:25.816 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.816 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.816 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.816 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845552+0000", 2024-03-30T14:03:25.817 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3520000000000007e-06 2024-03-30T14:03:25.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.817 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.817 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.817 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.279430+0000", 2024-03-30T14:03:25.817 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4338782210000001 2024-03-30T14:03:25.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.817 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.817 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:25.818 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.279622+0000", 2024-03-30T14:03:25.818 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00019243299999999999 2024-03-30T14:03:25.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.818 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.818 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T14:03:25.818 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.280952+0000", 2024-03-30T14:03:25.818 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0013300860000000001 2024-03-30T14:03:25.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.818 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.819 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:25.819 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.304430+0000", 2024-03-30T14:03:25.819 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.023477554000000001 2024-03-30T14:03:25.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.819 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.819 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.819 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.304490+0000", 2024-03-30T14:03:25.820 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0494000000000001e-05 2024-03-30T14:03:25.820 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.820 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.820 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.820 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.306238+0000", 2024-03-30T14:03:25.820 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001747342 2024-03-30T14:03:25.820 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.820 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.820 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:25.820 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.306342+0000", 2024-03-30T14:03:25.820 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010447500000000001 2024-03-30T14:03:25.820 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.820 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.821 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:25.821 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.307775+0000", 2024-03-30T14:03:25.821 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001433209 2024-03-30T14:03:25.821 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.821 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.821 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.821 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.821 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61140 9.d 9:b67c2d0c:::benchmark_data_smithi043_145506_object61139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T14:03:25.821 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850385+0000", 2024-03-30T14:03:25.821 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 590.88442077100001, 2024-03-30T14:03:25.821 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4594105769999999, 2024-03-30T14:03:25.821 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:25.822 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:25.822 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:25.822 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:25.822 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:03:25.822 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:03:25.822 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61140 2024-03-30T14:03:25.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.822 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:25.822 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.822 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:25.822 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850385+0000", 2024-03-30T14:03:25.822 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.822 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.823 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:25.823 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850385+0000", 2024-03-30T14:03:25.823 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.823 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.823 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:25.823 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850386+0000", 2024-03-30T14:03:25.823 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4999999999999998e-07 2024-03-30T14:03:25.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.823 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.823 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:25.823 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850432+0000", 2024-03-30T14:03:25.823 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6236999999999998e-05 2024-03-30T14:03:25.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.823 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.824 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:25.824 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850439+0000", 2024-03-30T14:03:25.824 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7530000000000004e-06 2024-03-30T14:03:25.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.824 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.824 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:25.824 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850444+0000", 2024-03-30T14:03:25.824 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.169e-06 2024-03-30T14:03:25.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.824 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.824 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.824 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854251+0000", 2024-03-30T14:03:25.824 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003806781 2024-03-30T14:03:25.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.825 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.825 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854258+0000", 2024-03-30T14:03:25.825 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4000000000000003e-06 2024-03-30T14:03:25.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.825 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.825 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845580+0000", 2024-03-30T14:03:25.825 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99132166899999996 2024-03-30T14:03:25.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.825 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.825 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845588+0000", 2024-03-30T14:03:25.825 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3669999999999997e-06 2024-03-30T14:03:25.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.826 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.826 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.283446+0000", 2024-03-30T14:03:25.826 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4378582849999999 2024-03-30T14:03:25.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.826 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:25.826 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.283622+0000", 2024-03-30T14:03:25.826 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017582700000000001 2024-03-30T14:03:25.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.826 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T14:03:25.827 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.283835+0000", 2024-03-30T14:03:25.827 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00021268799999999999 2024-03-30T14:03:25.827 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.827 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.827 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:25.827 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.304605+0000", 2024-03-30T14:03:25.827 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.020769823999999999 2024-03-30T14:03:25.827 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.827 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.827 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.828 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.304753+0000", 2024-03-30T14:03:25.828 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014781700000000001 2024-03-30T14:03:25.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.828 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.828 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.307918+0000", 2024-03-30T14:03:25.828 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0031653660000000002 2024-03-30T14:03:25.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.828 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:25.828 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.308027+0000", 2024-03-30T14:03:25.828 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000108789 2024-03-30T14:03:25.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.828 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:25.828 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.309796+0000", 2024-03-30T14:03:25.829 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001769255 2024-03-30T14:03:25.829 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.829 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.829 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.829 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61145 9.d 9:ba09f9de:::benchmark_data_smithi043_145506_object61144:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T14:03:25.829 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850451+0000", 2024-03-30T14:03:25.829 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 590.88435548100006, 2024-03-30T14:03:25.829 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4658662370000002, 2024-03-30T14:03:25.829 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:25.829 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:25.829 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:25.830 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:25.830 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:03:25.830 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:03:25.830 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61145 2024-03-30T14:03:25.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.830 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:25.830 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.830 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:25.830 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850451+0000", 2024-03-30T14:03:25.830 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.830 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.831 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:25.831 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850451+0000", 2024-03-30T14:03:25.831 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.831 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:25.831 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850451+0000", 2024-03-30T14:03:25.831 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7e-07 2024-03-30T14:03:25.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.831 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:25.831 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850494+0000", 2024-03-30T14:03:25.831 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2991000000000002e-05 2024-03-30T14:03:25.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.832 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:25.832 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850501+0000", 2024-03-30T14:03:25.832 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8310000000000003e-06 2024-03-30T14:03:25.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.832 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:25.832 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850506+0000", 2024-03-30T14:03:25.832 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.485e-06 2024-03-30T14:03:25.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.832 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.832 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854282+0000", 2024-03-30T14:03:25.832 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0037756550000000002 2024-03-30T14:03:25.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.833 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.833 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854290+0000", 2024-03-30T14:03:25.833 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5490000000000001e-06 2024-03-30T14:03:25.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.833 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.833 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845616+0000", 2024-03-30T14:03:25.833 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.991326343 2024-03-30T14:03:25.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.833 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.833 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845624+0000", 2024-03-30T14:03:25.833 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9410000000000007e-06 2024-03-30T14:03:25.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.834 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.834 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.309839+0000", 2024-03-30T14:03:25.834 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4642154810000001 2024-03-30T14:03:25.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.834 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:25.834 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.310036+0000", 2024-03-30T14:03:25.834 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000196687 2024-03-30T14:03:25.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.834 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T14:03:25.834 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.310262+0000", 2024-03-30T14:03:25.835 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00022566700000000001 2024-03-30T14:03:25.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.835 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.835 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.314606+0000", 2024-03-30T14:03:25.835 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0043444149999999999 2024-03-30T14:03:25.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.835 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.835 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.314671+0000", 2024-03-30T14:03:25.835 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4560000000000005e-05 2024-03-30T14:03:25.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.835 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:25.836 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.314689+0000", 2024-03-30T14:03:25.836 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8709999999999999e-05 2024-03-30T14:03:25.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.836 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:25.836 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.314804+0000", 2024-03-30T14:03:25.836 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000114811 2024-03-30T14:03:25.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.836 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:25.836 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.316317+0000", 2024-03-30T14:03:25.836 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001512741 2024-03-30T14:03:25.836 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.836 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.836 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.837 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61147 9.d 9:b6510d4c:::benchmark_data_smithi043_145506_object61146:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T14:03:25.837 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850513+0000", 2024-03-30T14:03:25.837 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 590.88429312100004, 2024-03-30T14:03:25.837 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4759084520000001, 2024-03-30T14:03:25.837 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:25.837 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:25.837 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:25.837 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:25.837 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:03:25.837 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:03:25.837 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61147 2024-03-30T14:03:25.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.838 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:25.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.838 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:25.838 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850513+0000", 2024-03-30T14:03:25.838 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.838 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:25.838 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850513+0000", 2024-03-30T14:03:25.838 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.839 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:25.839 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850513+0000", 2024-03-30T14:03:25.839 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4499999999999998e-07 2024-03-30T14:03:25.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.839 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:25.839 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850554+0000", 2024-03-30T14:03:25.839 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0902999999999998e-05 2024-03-30T14:03:25.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.839 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:25.839 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850561+0000", 2024-03-30T14:03:25.839 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3280000000000002e-06 2024-03-30T14:03:25.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.841 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:25.841 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850566+0000", 2024-03-30T14:03:25.842 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9219999999999996e-06 2024-03-30T14:03:25.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.842 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.842 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.842 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854312+0000", 2024-03-30T14:03:25.842 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0037467680000000001 2024-03-30T14:03:25.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.842 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.842 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.842 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854320+0000", 2024-03-30T14:03:25.842 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3150000000000003e-06 2024-03-30T14:03:25.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.843 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.843 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845655+0000", 2024-03-30T14:03:25.843 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99133570199999999 2024-03-30T14:03:25.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.843 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.843 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845664+0000", 2024-03-30T14:03:25.843 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.9020000000000005e-06 2024-03-30T14:03:25.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.843 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.843 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.316361+0000", 2024-03-30T14:03:25.844 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4706968279999999 2024-03-30T14:03:25.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.844 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:25.844 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.316547+0000", 2024-03-30T14:03:25.844 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000186059 2024-03-30T14:03:25.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.844 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T14:03:25.844 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.316770+0000", 2024-03-30T14:03:25.844 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000223119 2024-03-30T14:03:25.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.844 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.845 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.321387+0000", 2024-03-30T14:03:25.845 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0046164140000000001 2024-03-30T14:03:25.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.845 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:25.845 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.321405+0000", 2024-03-30T14:03:25.845 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8094999999999999e-05 2024-03-30T14:03:25.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.845 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.845 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.324536+0000", 2024-03-30T14:03:25.845 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0031307269999999998 2024-03-30T14:03:25.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.846 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:25.846 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.324576+0000", 2024-03-30T14:03:25.846 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0374999999999999e-05 2024-03-30T14:03:25.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.846 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.846 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:25.846 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.326422+0000", 2024-03-30T14:03:25.846 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0018456499999999999 2024-03-30T14:03:25.846 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.846 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.846 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.847 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.847 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.847 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61150 9.d 9:baf726bb:::benchmark_data_smithi043_145506_object61149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T14:03:25.847 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850572+0000", 2024-03-30T14:03:25.847 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 590.88423388499996, 2024-03-30T14:03:25.847 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4862311400000001, 2024-03-30T14:03:25.847 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:25.847 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:25.847 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:25.847 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:25.847 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:03:25.847 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:03:25.847 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61150 2024-03-30T14:03:25.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.848 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:25.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.848 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:25.848 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850572+0000", 2024-03-30T14:03:25.848 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.848 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:25.848 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850572+0000", 2024-03-30T14:03:25.848 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.848 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:25.849 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850573+0000", 2024-03-30T14:03:25.849 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4900000000000001e-07 2024-03-30T14:03:25.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.849 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:25.849 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850613+0000", 2024-03-30T14:03:25.849 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0605000000000002e-05 2024-03-30T14:03:25.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.849 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:25.849 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850620+0000", 2024-03-30T14:03:25.849 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4749999999999998e-06 2024-03-30T14:03:25.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.850 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.850 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:25.850 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850625+0000", 2024-03-30T14:03:25.850 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0259999999999998e-06 2024-03-30T14:03:25.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.850 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.850 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.850 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854345+0000", 2024-03-30T14:03:25.850 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00372048 2024-03-30T14:03:25.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.851 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.851 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854353+0000", 2024-03-30T14:03:25.851 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3549999999999999e-06 2024-03-30T14:03:25.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.851 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.851 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845694+0000", 2024-03-30T14:03:25.851 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99134108600000004 2024-03-30T14:03:25.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.851 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.851 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845702+0000", 2024-03-30T14:03:25.851 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0770000000000008e-06 2024-03-30T14:03:25.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.852 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.852 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.318701+0000", 2024-03-30T14:03:25.852 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4729991899999999 2024-03-30T14:03:25.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.852 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:25.852 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.318816+0000", 2024-03-30T14:03:25.852 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011537899999999999 2024-03-30T14:03:25.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.852 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T14:03:25.853 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.318948+0000", 2024-03-30T14:03:25.853 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000132037 2024-03-30T14:03:25.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.853 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.853 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:25.853 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.321413+0000", 2024-03-30T14:03:25.853 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024647940000000002 2024-03-30T14:03:25.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.853 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.853 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.853 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.324424+0000", 2024-03-30T14:03:25.853 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003011208 2024-03-30T14:03:25.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.853 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.854 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.854 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.334395+0000", 2024-03-30T14:03:25.854 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0099709849999999999 2024-03-30T14:03:25.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.854 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:25.854 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.334467+0000", 2024-03-30T14:03:25.854 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2028999999999998e-05 2024-03-30T14:03:25.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.854 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:25.854 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.336803+0000", 2024-03-30T14:03:25.854 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002336065 2024-03-30T14:03:25.855 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.855 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.855 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.855 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61169 9.d 9:b3588fe6:::benchmark_data_smithi043_145506_object61168:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T14:03:25.855 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850631+0000", 2024-03-30T14:03:25.855 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 590.88417503999995, 2024-03-30T14:03:25.855 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4878158689999998, 2024-03-30T14:03:25.855 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:25.855 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:25.855 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:25.855 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:25.855 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:03:25.855 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:03:25.856 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61169 2024-03-30T14:03:25.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.856 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:25.856 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.856 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:25.856 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850631+0000", 2024-03-30T14:03:25.856 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.856 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.856 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:25.856 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850631+0000", 2024-03-30T14:03:25.856 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.856 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.857 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:25.857 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850632+0000", 2024-03-30T14:03:25.857 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4900000000000001e-07 2024-03-30T14:03:25.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.857 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.857 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:25.857 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850678+0000", 2024-03-30T14:03:25.857 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6363999999999999e-05 2024-03-30T14:03:25.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.857 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.857 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:25.857 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850684+0000", 2024-03-30T14:03:25.857 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5289999999999997e-06 2024-03-30T14:03:25.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.857 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.857 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:25.858 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850689+0000", 2024-03-30T14:03:25.858 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8060000000000002e-06 2024-03-30T14:03:25.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.858 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.858 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.858 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854375+0000", 2024-03-30T14:03:25.858 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0036859369999999998 2024-03-30T14:03:25.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.858 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.858 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.858 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854383+0000", 2024-03-30T14:03:25.858 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.43e-06 2024-03-30T14:03:25.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.859 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.859 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845730+0000", 2024-03-30T14:03:25.859 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99134741999999998 2024-03-30T14:03:25.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.859 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.859 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845738+0000", 2024-03-30T14:03:25.859 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1599999999999998e-06 2024-03-30T14:03:25.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.859 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.859 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.319984+0000", 2024-03-30T14:03:25.859 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4742460390000001 2024-03-30T14:03:25.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.860 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:25.860 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.320135+0000", 2024-03-30T14:03:25.860 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015038000000000001 2024-03-30T14:03:25.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.860 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T14:03:25.860 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.320335+0000", 2024-03-30T14:03:25.860 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00020085799999999999 2024-03-30T14:03:25.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.860 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:25.860 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.324379+0000", 2024-03-30T14:03:25.860 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0040431310000000002 2024-03-30T14:03:25.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.861 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.861 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.324479+0000", 2024-03-30T14:03:25.861 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000100487 2024-03-30T14:03:25.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.861 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.861 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.336884+0000", 2024-03-30T14:03:25.861 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.012405143 2024-03-30T14:03:25.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.861 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:25.861 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.336961+0000", 2024-03-30T14:03:25.861 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7127999999999997e-05 2024-03-30T14:03:25.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.862 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:25.862 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.338447+0000", 2024-03-30T14:03:25.862 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0014857080000000001 2024-03-30T14:03:25.862 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.862 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.862 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.862 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61174 9.d 9:b691a112:::benchmark_data_smithi043_145506_object61173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T14:03:25.862 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850696+0000", 2024-03-30T14:03:25.862 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 590.88411022299999, 2024-03-30T14:03:25.862 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4896642509999998, 2024-03-30T14:03:25.863 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:25.863 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:25.863 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:25.863 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:25.863 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:03:25.863 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:03:25.863 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61174 2024-03-30T14:03:25.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.863 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:25.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.863 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:25.863 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850696+0000", 2024-03-30T14:03:25.863 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.864 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:25.864 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850696+0000", 2024-03-30T14:03:25.864 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.864 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:25.864 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850696+0000", 2024-03-30T14:03:25.864 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3599999999999999e-07 2024-03-30T14:03:25.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.864 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:25.864 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850738+0000", 2024-03-30T14:03:25.864 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1473000000000001e-05 2024-03-30T14:03:25.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.865 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:25.865 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850744+0000", 2024-03-30T14:03:25.865 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6379999999999998e-06 2024-03-30T14:03:25.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.865 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.865 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:25.865 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850749+0000", 2024-03-30T14:03:25.865 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8690000000000003e-06 2024-03-30T14:03:25.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.865 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.865 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.865 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854408+0000", 2024-03-30T14:03:25.865 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0036582339999999998 2024-03-30T14:03:25.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.866 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.866 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.866 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854415+0000", 2024-03-30T14:03:25.866 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.537e-06 2024-03-30T14:03:25.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.866 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.866 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.866 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845768+0000", 2024-03-30T14:03:25.866 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99135278699999996 2024-03-30T14:03:25.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.866 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.866 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.866 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845776+0000", 2024-03-30T14:03:25.867 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3100000000000001e-06 2024-03-30T14:03:25.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.867 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.867 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.321437+0000", 2024-03-30T14:03:25.867 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.475661302 2024-03-30T14:03:25.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.867 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:25.867 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.321525+0000", 2024-03-30T14:03:25.867 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7298000000000003e-05 2024-03-30T14:03:25.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.868 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T14:03:25.868 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.321632+0000", 2024-03-30T14:03:25.868 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000107734 2024-03-30T14:03:25.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.868 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.868 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:25.868 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.324496+0000", 2024-03-30T14:03:25.868 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0028631220000000001 2024-03-30T14:03:25.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.868 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.868 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.868 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.326464+0000", 2024-03-30T14:03:25.868 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0019687020000000001 2024-03-30T14:03:25.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.869 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.869 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.869 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.338506+0000", 2024-03-30T14:03:25.869 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.012041530999999999 2024-03-30T14:03:25.869 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.869 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.869 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:25.869 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.338570+0000", 2024-03-30T14:03:25.869 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4144000000000001e-05 2024-03-30T14:03:25.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.870 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.870 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:25.870 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.340360+0000", 2024-03-30T14:03:25.870 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0017902339999999999 2024-03-30T14:03:25.870 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61190 9.d 9:ba843ea7:::benchmark_data_smithi043_145506_object61189:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T14:03:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850756+0000", 2024-03-30T14:03:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 590.88405046699995, 2024-03-30T14:03:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4909397279999999, 2024-03-30T14:03:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:03:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:03:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61190 2024-03-30T14:03:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850756+0000", 2024-03-30T14:03:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:25.873 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850756+0000", 2024-03-30T14:03:25.873 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.873 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.873 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.873 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:25.875 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850756+0000", 2024-03-30T14:03:25.875 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4499999999999998e-07 2024-03-30T14:03:25.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.875 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.875 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850801+0000", 2024-03-30T14:03:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4878e-05 2024-03-30T14:03:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850808+0000", 2024-03-30T14:03:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.72e-06 2024-03-30T14:03:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850813+0000", 2024-03-30T14:03:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9389999999999996e-06 2024-03-30T14:03:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854440+0000", 2024-03-30T14:03:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003627052 2024-03-30T14:03:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854447+0000", 2024-03-30T14:03:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7049999999999999e-06 2024-03-30T14:03:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845805+0000", 2024-03-30T14:03:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99135801000000001 2024-03-30T14:03:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.845814+0000", 2024-03-30T14:03:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4840000000000004e-06 2024-03-30T14:03:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.323021+0000", 2024-03-30T14:03:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.477207148 2024-03-30T14:03:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.323113+0000", 2024-03-30T14:03:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1707999999999997e-05 2024-03-30T14:03:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T14:03:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.323248+0000", 2024-03-30T14:03:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013484 2024-03-30T14:03:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.326474+0000", 2024-03-30T14:03:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0032268739999999998 2024-03-30T14:03:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.326510+0000", 2024-03-30T14:03:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5444000000000001e-05 2024-03-30T14:03:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.340411+0000", 2024-03-30T14:03:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.013900859999999999 2024-03-30T14:03:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.340451+0000", 2024-03-30T14:03:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.032e-05 2024-03-30T14:03:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.341695+0000", 2024-03-30T14:03:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001244401 2024-03-30T14:03:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15354.0:61214 9.d 9:bfa3a72f:::benchmark_data_smithi043_145506_object61213:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T14:03:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T13:53:34.850819+0000", 2024-03-30T14:03:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 590.88398680900002, 2024-03-30T14:03:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4922764219999998, 2024-03-30T14:03:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15354", 2024-03-30T14:03:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/1195819627", 2024-03-30T14:03:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 61214 2024-03-30T14:03:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850819+0000", 2024-03-30T14:03:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850819+0000", 2024-03-30T14:03:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850820+0000", 2024-03-30T14:03:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2300000000000002e-07 2024-03-30T14:03:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850877+0000", 2024-03-30T14:03:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6759000000000003e-05 2024-03-30T14:03:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850884+0000", 2024-03-30T14:03:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5619999999999998e-06 2024-03-30T14:03:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.850890+0000", 2024-03-30T14:03:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4750000000000001e-06 2024-03-30T14:03:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854471+0000", 2024-03-30T14:03:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0035816419999999999 2024-03-30T14:03:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:34.854479+0000", 2024-03-30T14:03:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4660000000000002e-06 2024-03-30T14:03:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.847038+0000", 2024-03-30T14:03:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99255926400000005 2024-03-30T14:03:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:35.847052+0000", 2024-03-30T14:03:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.447e-05 2024-03-30T14:03:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.326536+0000", 2024-03-30T14:03:25.887 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4794833030000001 2024-03-30T14:03:25.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.326641+0000", 2024-03-30T14:03:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000105073 2024-03-30T14:03:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-03-30T14:03:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.326758+0000", 2024-03-30T14:03:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000116749 2024-03-30T14:03:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.328416+0000", 2024-03-30T14:03:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001658358 2024-03-30T14:03:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.334339+0000", 2024-03-30T14:03:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0059233610000000003 2024-03-30T14:03:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.341763+0000", 2024-03-30T14:03:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0074236479999999997 2024-03-30T14:03:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.341826+0000", 2024-03-30T14:03:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3196999999999998e-05 2024-03-30T14:03:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T13:53:37.343096+0000", 2024-03-30T14:03:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0012696719999999999 2024-03-30T14:03:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25685.0:110091 11.0 11:02ee931f:::benchmark_data_smithi043_197439_object110090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-03-30T14:03:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:02:48.097987+0000", 2024-03-30T14:03:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 37.636818736999999, 2024-03-30T14:03:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2625925520000001, 2024-03-30T14:03:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-03-30T14:03:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25685", 2024-03-30T14:03:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:03:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 110091 2024-03-30T14:03:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.097987+0000", 2024-03-30T14:03:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.097987+0000", 2024-03-30T14:03:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.097989+0000", 2024-03-30T14:03:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.514e-06 2024-03-30T14:03:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.893 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:25.893 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.098018+0000", 2024-03-30T14:03:25.893 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.8799e-05 2024-03-30T14:03:25.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.893 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:25.893 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.098036+0000", 2024-03-30T14:03:25.893 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7819999999999999e-05 2024-03-30T14:03:25.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.893 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:25.893 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.098046+0000", 2024-03-30T14:03:25.893 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0389e-05 2024-03-30T14:03:25.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.894 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.894 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.098881+0000", 2024-03-30T14:03:25.894 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000835143 2024-03-30T14:03:25.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.894 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:25.894 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.099107+0000", 2024-03-30T14:03:25.894 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00022566700000000001 2024-03-30T14:03:25.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.894 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,6", 2024-03-30T14:03:25.894 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.099273+0000", 2024-03-30T14:03:25.895 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000165759 2024-03-30T14:03:25.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.895 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:25.895 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.115950+0000", 2024-03-30T14:03:25.895 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.016677569 2024-03-30T14:03:25.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.895 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.895 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.135334+0000", 2024-03-30T14:03:25.895 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.019383648 2024-03-30T14:03:25.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.895 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.896 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.135348+0000", 2024-03-30T14:03:25.896 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3978000000000001e-05 2024-03-30T14:03:25.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.896 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.896 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:49.103167+0000", 2024-03-30T14:03:25.896 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.96781961000000005 2024-03-30T14:03:25.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.896 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.896 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:49.103182+0000", 2024-03-30T14:03:25.896 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4754e-05 2024-03-30T14:03:25.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.897 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.897 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.251492+0000", 2024-03-30T14:03:25.897 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.148309781 2024-03-30T14:03:25.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.897 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for degraded object", 2024-03-30T14:03:25.897 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.251978+0000", 2024-03-30T14:03:25.897 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00048650800000000003 2024-03-30T14:03:25.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.897 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.897 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.360506+0000", 2024-03-30T14:03:25.897 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.10852718 2024-03-30T14:03:25.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.898 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:25.898 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.360580+0000", 2024-03-30T14:03:25.898 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4432999999999997e-05 2024-03-30T14:03:25.898 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.898 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.898 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.898 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25685.0:110094 11.b 11:d7571555:::benchmark_data_smithi043_197439_object110093:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e417)", 2024-03-30T14:03:25.898 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:02:48.117798+0000", 2024-03-30T14:03:25.898 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 37.617007733999998, 2024-03-30T14:03:25.898 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0927753849999999, 2024-03-30T14:03:25.898 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:25.899 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:25.899 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:25.899 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:25.899 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25685", 2024-03-30T14:03:25.899 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:03:25.899 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 110094 2024-03-30T14:03:25.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.899 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:25.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.899 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:25.899 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.117798+0000", 2024-03-30T14:03:25.899 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.900 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.900 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:25.900 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.117798+0000", 2024-03-30T14:03:25.900 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.900 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.900 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:25.900 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.117799+0000", 2024-03-30T14:03:25.900 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8599999999999998e-07 2024-03-30T14:03:25.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.900 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.900 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:25.900 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.117836+0000", 2024-03-30T14:03:25.901 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6587000000000001e-05 2024-03-30T14:03:25.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.901 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.901 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:25.901 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.117845+0000", 2024-03-30T14:03:25.901 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0089999999999996e-06 2024-03-30T14:03:25.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.901 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.901 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:25.901 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.117852+0000", 2024-03-30T14:03:25.901 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0010000000000004e-06 2024-03-30T14:03:25.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.901 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.901 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.901 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.128172+0000", 2024-03-30T14:03:25.902 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010319975 2024-03-30T14:03:25.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.902 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.902 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.902 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.128183+0000", 2024-03-30T14:03:25.902 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1449e-05 2024-03-30T14:03:25.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.902 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.902 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.902 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:49.096002+0000", 2024-03-30T14:03:25.902 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.96781855999999999 2024-03-30T14:03:25.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.902 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.902 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.903 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:49.096015+0000", 2024-03-30T14:03:25.903 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3424999999999999e-05 2024-03-30T14:03:25.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.903 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.903 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.903 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.172711+0000", 2024-03-30T14:03:25.903 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.076695776 2024-03-30T14:03:25.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.903 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.903 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:25.903 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.174732+0000", 2024-03-30T14:03:25.903 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0020208489999999999 2024-03-30T14:03:25.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.903 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.904 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,3,4", 2024-03-30T14:03:25.904 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.175064+0000", 2024-03-30T14:03:25.904 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00033187400000000001 2024-03-30T14:03:25.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.904 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:25.904 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.186850+0000", 2024-03-30T14:03:25.904 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011785914 2024-03-30T14:03:25.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.904 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.904 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.186901+0000", 2024-03-30T14:03:25.904 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1861999999999999e-05 2024-03-30T14:03:25.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.905 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.905 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.201527+0000", 2024-03-30T14:03:25.905 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.014625188000000001 2024-03-30T14:03:25.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.905 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.905 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.210385+0000", 2024-03-30T14:03:25.905 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0088582350000000008 2024-03-30T14:03:25.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.905 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:25.905 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.210454+0000", 2024-03-30T14:03:25.906 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9499000000000007e-05 2024-03-30T14:03:25.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.906 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:25.906 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.210574+0000", 2024-03-30T14:03:25.906 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000119496 2024-03-30T14:03:25.906 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.906 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.906 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.906 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25685.0:110111 11.0 11:0cb3e7e7:::benchmark_data_smithi043_197439_object110110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-30T14:03:25.906 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:02:48.150032+0000", 2024-03-30T14:03:25.906 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 37.584774359000001, 2024-03-30T14:03:25.907 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2108067060000001, 2024-03-30T14:03:25.907 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:25.907 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:25.907 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:25.907 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:25.907 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25685", 2024-03-30T14:03:25.907 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:03:25.907 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 110111 2024-03-30T14:03:25.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.907 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:25.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.907 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:25.907 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.150032+0000", 2024-03-30T14:03:25.907 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.908 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:25.908 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.150032+0000", 2024-03-30T14:03:25.908 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.908 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:25.908 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.150033+0000", 2024-03-30T14:03:25.908 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.8899999999999998e-07 2024-03-30T14:03:25.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.908 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:25.908 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.150052+0000", 2024-03-30T14:03:25.908 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8828e-05 2024-03-30T14:03:25.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.909 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:25.909 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.150061+0000", 2024-03-30T14:03:25.909 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.9819999999999997e-06 2024-03-30T14:03:25.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.909 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:25.909 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.150066+0000", 2024-03-30T14:03:25.909 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3369999999999999e-06 2024-03-30T14:03:25.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.909 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.909 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.150136+0000", 2024-03-30T14:03:25.910 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9967000000000003e-05 2024-03-30T14:03:25.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.910 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.910 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.910 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.150227+0000", 2024-03-30T14:03:25.910 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1207999999999998e-05 2024-03-30T14:03:25.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.910 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.910 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.910 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:49.103210+0000", 2024-03-30T14:03:25.910 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95298282999999995 2024-03-30T14:03:25.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.910 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.911 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.911 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:49.103218+0000", 2024-03-30T14:03:25.911 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1370000000000002e-06 2024-03-30T14:03:25.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.911 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.911 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.252313+0000", 2024-03-30T14:03:25.911 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1490949880000001 2024-03-30T14:03:25.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.911 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:25.911 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.252465+0000", 2024-03-30T14:03:25.911 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015152199999999999 2024-03-30T14:03:25.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.912 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.912 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,3,6", 2024-03-30T14:03:25.912 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.252906+0000", 2024-03-30T14:03:25.912 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000441586 2024-03-30T14:03:25.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.912 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.912 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.912 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.264205+0000", 2024-03-30T14:03:25.912 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011299241 2024-03-30T14:03:25.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.912 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.912 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:25.912 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.264221+0000", 2024-03-30T14:03:25.913 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5843999999999999e-05 2024-03-30T14:03:25.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.913 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.913 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.264269+0000", 2024-03-30T14:03:25.913 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7496000000000001e-05 2024-03-30T14:03:25.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.913 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.913 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.360649+0000", 2024-03-30T14:03:25.913 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.096379873000000005 2024-03-30T14:03:25.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.913 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:25.914 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.360705+0000", 2024-03-30T14:03:25.914 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6597e-05 2024-03-30T14:03:25.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.914 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.914 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:25.914 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.360839+0000", 2024-03-30T14:03:25.914 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013328099999999999 2024-03-30T14:03:25.914 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.914 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.914 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.914 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.914 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25685.0:110115 11.0 11:0a5ec0c5:::benchmark_data_smithi043_197439_object110114:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-30T14:03:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:02:48.152019+0000", 2024-03-30T14:03:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 37.582786912000003, 2024-03-30T14:03:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.209045626, 2024-03-30T14:03:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25685", 2024-03-30T14:03:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:03:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 110115 2024-03-30T14:03:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.152019+0000", 2024-03-30T14:03:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.152019+0000", 2024-03-30T14:03:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.152021+0000", 2024-03-30T14:03:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8300000000000001e-06 2024-03-30T14:03:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.152043+0000", 2024-03-30T14:03:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1809000000000001e-05 2024-03-30T14:03:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.152055+0000", 2024-03-30T14:03:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1871e-05 2024-03-30T14:03:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.152062+0000", 2024-03-30T14:03:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4590000000000001e-06 2024-03-30T14:03:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.152133+0000", 2024-03-30T14:03:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0426999999999995e-05 2024-03-30T14:03:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:48.152145+0000", 2024-03-30T14:03:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2281e-05 2024-03-30T14:03:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:49.103243+0000", 2024-03-30T14:03:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95109754300000005 2024-03-30T14:03:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-03-30T14:03:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:49.103251+0000", 2024-03-30T14:03:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4489999999999999e-06 2024-03-30T14:03:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.260932+0000", 2024-03-30T14:03:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.157680579 2024-03-30T14:03:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.261087+0000", 2024-03-30T14:03:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015555699999999999 2024-03-30T14:03:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,3,6", 2024-03-30T14:03:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.261266+0000", 2024-03-30T14:03:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017848500000000001 2024-03-30T14:03:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.264227+0000", 2024-03-30T14:03:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002961722 2024-03-30T14:03:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.264315+0000", 2024-03-30T14:03:25.921 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7717000000000001e-05 2024-03-30T14:03:25.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.921 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.921 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.921 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.264360+0000", 2024-03-30T14:03:25.921 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4771999999999998e-05 2024-03-30T14:03:25.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.921 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.921 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:25.921 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.360894+0000", 2024-03-30T14:03:25.921 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.096534597999999999 2024-03-30T14:03:25.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.921 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.921 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:25.921 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.360961+0000", 2024-03-30T14:03:25.922 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6229000000000006e-05 2024-03-30T14:03:25.922 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:25.922 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:25.922 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:25.922 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:50.361065+0000", 2024-03-30T14:03:25.922 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000104298 2024-03-30T14:03:25.922 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.922 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.922 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.922 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:25.922 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:25.922 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T14:03:25.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:25.924 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:25.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:26.025 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:26.067 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:26.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:26.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:26.169 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:26.198 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:26.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:26.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:26.300 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:26.342 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:26.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:26.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:26.391 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:03:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:03:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:03:26.408 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:03:26.444 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:26.469 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T14:03:26.498 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:26.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:26.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:26.517 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T14:03:26.565 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T14:03:26.601 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:26.615 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T14:03:26.656 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T14:03:26.657 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:26.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:26.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:26.700 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T14:03:26.740 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T14:03:26.759 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:26.766 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T14:03:26.778 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T14:03:26.782 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:03:26.782 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:03:26.782 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:03:26.783 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:03:26.783 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:03:26.783 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:03:26.783 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:03:26.783 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:03:26.812 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:26.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:26.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:26.914 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:26.966 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:26.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:26.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:27.068 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:27.097 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:27.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:27.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:27.199 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:27.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:26 smithi113 ceph-mon[34341]: pgmap v949: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 4.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 780 op/s; 26841/231771 objects misplaced (11.581%); 28 MiB/s, 447 objects/s recovering 2024-03-30T14:03:27.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:26 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1219925720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:03:27.243 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:27.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:27.345 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:27.390 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:27.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:27.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:26 smithi043 ceph-mon[29963]: pgmap v949: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 4.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 780 op/s; 26841/231771 objects misplaced (11.581%); 28 MiB/s, 447 objects/s recovering 2024-03-30T14:03:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:26 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1219925720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:03:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:26 smithi043 ceph-mon[42099]: pgmap v949: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 4.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 780 op/s; 26841/231771 objects misplaced (11.581%); 28 MiB/s, 447 objects/s recovering 2024-03-30T14:03:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:26 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1219925720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:03:27.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:27.537 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:27.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:27.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:27.639 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:27.672 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:27.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:27.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:27.774 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:27.810 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:27.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:27.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:27.912 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:27.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:27.946 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:27.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:28.047 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:28.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:28.097 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:28.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:28.198 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:28.221 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:28.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:28.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:28.323 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:28.354 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:28.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:28.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:28.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:27 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:03:28.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:27 smithi043 ceph-mon[29963]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T14:03:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:27 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:03:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:27 smithi043 ceph-mon[42099]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T14:03:28.456 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:28.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:27 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:03:28.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:27 smithi113 ceph-mon[34341]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T14:03:28.486 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:28.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:28.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:28.588 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:28.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:28.629 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:28.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:28.730 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:28.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:28.775 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:28.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:28.876 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:28.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:28.906 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:28.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:29.000 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-30T14:03:29.007 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:29.036 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:29.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:29.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:29.138 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:29.163 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:29.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:29.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:29.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:28 smithi043 ceph-mon[42099]: pgmap v950: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 890 op/s; 24333/226722 objects misplaced (10.733%); 39 MiB/s, 617 objects/s recovering 2024-03-30T14:03:29.240 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:28 smithi043 ceph-mon[29963]: pgmap v950: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 890 op/s; 24333/226722 objects misplaced (10.733%); 39 MiB/s, 617 objects/s recovering 2024-03-30T14:03:29.265 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:29.292 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:29.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:29.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:29.393 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:29.421 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:29.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:29.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:29.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:28 smithi113 ceph-mon[34341]: pgmap v950: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 890 op/s; 24333/226722 objects misplaced (10.733%); 39 MiB/s, 617 objects/s recovering 2024-03-30T14:03:29.523 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:29.553 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:29.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:29.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:29.655 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:29.697 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:29.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:29.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:29.799 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:29.829 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:29.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:29.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:29.931 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:29.962 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:29.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:29.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:29.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:03:29 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:03:30.065 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:30.097 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:30.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:30.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:30.200 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:30.239 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:30.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:30.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:30.340 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:30.386 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:30.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:30.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:30.488 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:30.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:30.531 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:30.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:30.632 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:30.661 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:30.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:30.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:30.763 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:30.796 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:30.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:30.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:30.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:30 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:30.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:03:30.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:30.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:30 smithi043 ceph-mon[29963]: pgmap v951: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 4.4 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 960 op/s; 22279/216831 objects misplaced (10.275%); 40 MiB/s, 645 objects/s recovering 2024-03-30T14:03:30.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:30 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:30.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:03:30.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:30.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:30 smithi043 ceph-mon[42099]: pgmap v951: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 4.4 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 960 op/s; 22279/216831 objects misplaced (10.275%); 40 MiB/s, 645 objects/s recovering 2024-03-30T14:03:30.899 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:30.928 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:30.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:30.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:30.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:30 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:30.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:03:30.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:30.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:30 smithi113 ceph-mon[34341]: pgmap v951: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 4.4 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 960 op/s; 22279/216831 objects misplaced (10.275%); 40 MiB/s, 645 objects/s recovering 2024-03-30T14:03:31.030 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:31.061 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:31.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:31.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:31.162 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:31.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:31.194 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:31.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:31.295 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:31.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:31.321 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:31.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:31.422 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:31.461 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:31.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:31.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:31.563 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:31.591 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:31.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:31.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:31.693 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:31.721 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:31.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:31.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:31.822 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:31.852 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:31.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:31.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:31.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:31 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:03:31.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:31 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:03:31.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:31 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:03:31.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:31 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:03:31.954 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:31.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:31 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:03:31.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:31 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:03:31.993 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:31.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:31.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:32.095 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:32.125 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:32.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:32.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:32.228 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:32.256 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:32.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:32.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:32.358 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:32.393 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:32.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:32.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:32.495 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:32.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:32.540 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:32.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:32.641 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:32.678 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:32.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:32.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:32.780 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:32.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:32.811 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:32.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:32.873 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:32 smithi043 ceph-mon[29963]: osdmap e429: 8 total, 8 up, 6 in 2024-03-30T14:03:32.873 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:32 smithi043 ceph-mon[29963]: pgmap v953: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 4.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 20684/210861 objects misplaced (9.809%); 48 MiB/s, 762 objects/s recovering 2024-03-30T14:03:32.873 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:32 smithi043 ceph-mon[42099]: osdmap e429: 8 total, 8 up, 6 in 2024-03-30T14:03:32.873 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:32 smithi043 ceph-mon[42099]: pgmap v953: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 4.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 20684/210861 objects misplaced (9.809%); 48 MiB/s, 762 objects/s recovering 2024-03-30T14:03:32.913 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:32.945 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:32.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:32.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:32.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:32 smithi113 ceph-mon[34341]: osdmap e429: 8 total, 8 up, 6 in 2024-03-30T14:03:32.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:32 smithi113 ceph-mon[34341]: pgmap v953: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 4.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 20684/210861 objects misplaced (9.809%); 48 MiB/s, 762 objects/s recovering 2024-03-30T14:03:33.047 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:33.090 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:33.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:33.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:33.192 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:33.260 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:33.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:33.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:33.362 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:33.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:33.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:33.511 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:33.541 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:33.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:33.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:33.643 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:33.683 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:33.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:33.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:33.732 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:03:33.785 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:33.811 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:33.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:33.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:03:33.913 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:33.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:33.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:34.061 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:34.077 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:33 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:03:34.077 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:33 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:03:34.077 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:33 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:03:34.077 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:33 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:03:34.077 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:33 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:03:34.077 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:33 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:03:34.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:34.122 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:34.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:34.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:33 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:03:34.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:33 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:03:34.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:33 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:03:34.223 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:34.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:34.271 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:34.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:34.373 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:34.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:34.403 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:34.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:34.504 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:34.535 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:34.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:34.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:34.637 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:34.677 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:34.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:34.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:34.777 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T14:03:34.779 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:34.823 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:34.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:34.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:34.925 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:34.953 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:34.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:34.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:35.055 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:35.111 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:35.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:35.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:35.213 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:35.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:34 smithi113 ceph-mon[34341]: pgmap v954: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 4.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 940 op/s; 20339/208797 objects misplaced (9.741%); 39 MiB/s, 619 objects/s recovering 2024-03-30T14:03:35.244 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:35.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:35.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:35.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:34 smithi043 ceph-mon[29963]: pgmap v954: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 4.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 940 op/s; 20339/208797 objects misplaced (9.741%); 39 MiB/s, 619 objects/s recovering 2024-03-30T14:03:35.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:34 smithi043 ceph-mon[42099]: pgmap v954: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 4.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 940 op/s; 20339/208797 objects misplaced (9.741%); 39 MiB/s, 619 objects/s recovering 2024-03-30T14:03:35.345 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:35.382 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:35.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:35.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:35.484 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:35.514 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:35.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:35.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:35.616 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:35.643 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:35.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:35.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:35.745 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:35.775 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:35.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:35.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:35.877 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:35.920 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:35.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:35.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:36.021 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:36.061 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:36.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:36.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:36.163 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:36.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:36.203 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:36.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:36.297 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T14:03:36.297 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "" 2024-03-30T14:03:36.297 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T14:03:36.303 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:36.344 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:36.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:36.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:36.391 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:03:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:03:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:03:36.447 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:36.476 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:36.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:36.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:36.578 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:36.625 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:36.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:36.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:36.726 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:36.769 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:36.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:36.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:36.871 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:36.898 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:36.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:36.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:36.999 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:37.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:37.043 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:37.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:37.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:36 smithi043 ceph-mon[29963]: pgmap v955: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 4.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 18634/200349 objects misplaced (9.301%); 31 MiB/s, 494 objects/s recovering 2024-03-30T14:03:37.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:36 smithi043 ceph-mon[42099]: pgmap v955: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 4.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 18634/200349 objects misplaced (9.301%); 31 MiB/s, 494 objects/s recovering 2024-03-30T14:03:37.144 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:37.173 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:37.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:37.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:37.274 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:37.310 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:37.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:37.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:37.412 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:37.438 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:37.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:37.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:37.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:36 smithi113 ceph-mon[34341]: pgmap v955: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 4.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 18634/200349 objects misplaced (9.301%); 31 MiB/s, 494 objects/s recovering 2024-03-30T14:03:37.543 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:37.569 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:37.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:37.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:37.670 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:37.697 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:37.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:37.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:37.800 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:37.828 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:37.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:37.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:37.930 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:37.960 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:37.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:37.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:38.034 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:37 smithi043 ceph-mon[29963]: osdmap e430: 8 total, 8 up, 6 in 2024-03-30T14:03:38.034 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:37 smithi043 ceph-mon[29963]: osdmap e431: 8 total, 8 up, 6 in 2024-03-30T14:03:38.034 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:37 smithi043 ceph-mon[42099]: osdmap e430: 8 total, 8 up, 6 in 2024-03-30T14:03:38.034 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:37 smithi043 ceph-mon[42099]: osdmap e431: 8 total, 8 up, 6 in 2024-03-30T14:03:38.063 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:38.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:38.109 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:38.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:38.210 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:38.236 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:38.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:38.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:38.338 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:38.363 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:38.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:38.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:38.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:37 smithi113 ceph-mon[34341]: osdmap e430: 8 total, 8 up, 6 in 2024-03-30T14:03:38.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:37 smithi113 ceph-mon[34341]: osdmap e431: 8 total, 8 up, 6 in 2024-03-30T14:03:38.465 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:38.483 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:03:38.490 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:38.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:38.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:38.591 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:38.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:38.617 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:38.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:38.718 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:38.743 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:38.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:38.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:38.769 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-03-30T14:03:38.770 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:03:38.845 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:38.875 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:38.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:38.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:38.956 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-03-30T14:03:38.977 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:39.001 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:39.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:39.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:39.103 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:39.151 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:39.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:39.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:39.252 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:39.292 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:39.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:39.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:39.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:38 smithi043 ceph-mon[29963]: pgmap v957: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 3.9 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 940 op/s; 17095/194283 objects misplaced (8.799%); 23 MiB/s, 372 objects/s recovering 2024-03-30T14:03:39.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:38 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1179881944' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:03:39.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:38 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:03:39.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:38 smithi043 ceph-mon[29963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:03:39.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:38 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:03:39.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:38 smithi043 ceph-mon[29963]: osdmap e432: 8 total, 8 up, 6 in 2024-03-30T14:03:39.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:38 smithi043 ceph-mon[42099]: pgmap v957: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 3.9 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 940 op/s; 17095/194283 objects misplaced (8.799%); 23 MiB/s, 372 objects/s recovering 2024-03-30T14:03:39.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:38 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1179881944' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:03:39.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:38 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:03:39.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:38 smithi043 ceph-mon[42099]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:03:39.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:38 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:03:39.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:38 smithi043 ceph-mon[42099]: osdmap e432: 8 total, 8 up, 6 in 2024-03-30T14:03:39.394 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:39.427 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:39.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:39.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:39.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:38 smithi113 ceph-mon[34341]: pgmap v957: 168 pgs: 3 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 161 active+clean; 3.9 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 940 op/s; 17095/194283 objects misplaced (8.799%); 23 MiB/s, 372 objects/s recovering 2024-03-30T14:03:39.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:38 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1179881944' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:03:39.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:38 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:03:39.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:38 smithi113 ceph-mon[34341]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:03:39.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:38 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:03:39.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:38 smithi113 ceph-mon[34341]: osdmap e432: 8 total, 8 up, 6 in 2024-03-30T14:03:39.528 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:39.567 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:39.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:39.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:39.669 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:39.705 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:39.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:39.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:39.807 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:39.842 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:39.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:39.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:39.943 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:39.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:03:39 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:03:39.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:39.972 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:39.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:40.073 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:40.100 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:40.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:40.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:40.201 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:40.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:40.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:40.346 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:40.384 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:40.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:40.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:40.486 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:40.516 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:40.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:40.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:40.618 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:40.659 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:40.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:40.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:40.761 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:40.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:40.791 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:40.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:40.892 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:40.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:40.936 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:40.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:40.981 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-03-30T14:03:41.037 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:41.066 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:41.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:41.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:41.168 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:41.198 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:41.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:41.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:41.300 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:41.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:41.347 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:41.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:41.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:41 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:41.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:41 smithi043 ceph-mon[29963]: pgmap v960: 168 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 3.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.23k op/s; 14898/186699 objects misplaced (7.980%); 38 MiB/s, 604 objects/s recovering 2024-03-30T14:03:41.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:41 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:41.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:41 smithi043 ceph-mon[42099]: pgmap v960: 168 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 3.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.23k op/s; 14898/186699 objects misplaced (7.980%); 38 MiB/s, 604 objects/s recovering 2024-03-30T14:03:41.448 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:41.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:41 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:41.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:41 smithi113 ceph-mon[34341]: pgmap v960: 168 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 3.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.23k op/s; 14898/186699 objects misplaced (7.980%); 38 MiB/s, 604 objects/s recovering 2024-03-30T14:03:41.507 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:41.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:41.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:41.536 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:03:41.609 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:41.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:41.663 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:41.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:41.764 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:41.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:41.796 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:41.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:41.897 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:41.925 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 6, 5] out_osds: [7, 3] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-03-30T14:03:41.925 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T14:03:41.926 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 4, 6, 5] 2024-03-30T14:03:41.926 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd out 2 2024-03-30T14:03:41.935 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:41.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:41.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:42.024 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T14:03:42.037 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:42.159 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:42.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:42.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:42.261 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:42.292 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:42.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:42.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:42.394 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:42.456 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:42.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:42.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:42.557 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:42.586 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:42.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:42.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:42.688 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:42.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:42.724 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:42.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:42.825 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:42.856 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:42.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:42.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:42.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:42.998 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:42.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:42.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:43.100 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:43.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:43.155 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:43.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:43.255 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:43.298 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:43 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:03:43.299 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:43 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:03:43.299 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:43 smithi043 ceph-mon[29963]: pgmap v961: 168 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 163 active+clean; 3.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.04k op/s; 12695/181560 objects misplaced (6.992%); 46 MiB/s, 733 objects/s recovering 2024-03-30T14:03:43.299 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:43 smithi043 ceph-mon[29963]: osdmap e433: 8 total, 8 up, 6 in 2024-03-30T14:03:43.299 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:43 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:03:43.299 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:43 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:03:43.299 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:43 smithi043 ceph-mon[42099]: pgmap v961: 168 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 163 active+clean; 3.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.04k op/s; 12695/181560 objects misplaced (6.992%); 46 MiB/s, 733 objects/s recovering 2024-03-30T14:03:43.299 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:43 smithi043 ceph-mon[42099]: osdmap e433: 8 total, 8 up, 6 in 2024-03-30T14:03:43.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:43.300 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:43.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:43.401 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:43.433 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T14:03:43.446 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:43.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:43.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:43.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:43 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:03:43.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:43 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:03:43.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:43 smithi113 ceph-mon[34341]: pgmap v961: 168 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 163 active+clean; 3.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.04k op/s; 12695/181560 objects misplaced (6.992%); 46 MiB/s, 733 objects/s recovering 2024-03-30T14:03:43.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:43 smithi113 ceph-mon[34341]: osdmap e433: 8 total, 8 up, 6 in 2024-03-30T14:03:43.485 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T14:03:43.528 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T14:03:43.548 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:43.570 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T14:03:43.578 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:43.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:43.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:43.609 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T14:03:43.653 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T14:03:43.680 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:43.700 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T14:03:43.724 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:43.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:43.745 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T14:03:43.752 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:03:43.752 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:03:43.752 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:03:43.752 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:03:43.752 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:03:43.752 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:03:43.752 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:03:43.752 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:03:43.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:43.864 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:43.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:43.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:43.967 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:44.003 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:44.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:44.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:44.105 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:44.134 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:44.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:44.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:44.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:44 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/755424328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:03:44.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:44 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/755424328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:03:44.236 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:44.266 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:44.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:44.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:44.368 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:44.400 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:44.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:44.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:44.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:44 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/755424328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:03:44.497 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T14:03:44.497 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T14:03:44.497 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-03-30T14:03:44.497 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-03-30T14:03:44.498 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T14:03:44.502 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:44.540 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:44.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:44.641 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:44.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:44.683 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:44.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:44.706 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:03:44.743 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:03:44.785 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:44.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:44.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:44.943 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:44.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:44.987 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:44.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:45.087 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:45.122 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:45.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:45.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:45.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:45 smithi043 ceph-mon[29963]: pgmap v963: 168 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 163 active+clean; 3.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 821 op/s; 12352/179490 objects misplaced (6.882%); 36 MiB/s, 571 objects/s recovering 2024-03-30T14:03:45.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:45 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:03:45.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:45 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:03:45.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:45 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:45.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:45 smithi043 ceph-mon[29963]: osdmap e434: 8 total, 8 up, 6 in 2024-03-30T14:03:45.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:45 smithi043 ceph-mon[42099]: pgmap v963: 168 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 163 active+clean; 3.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 821 op/s; 12352/179490 objects misplaced (6.882%); 36 MiB/s, 571 objects/s recovering 2024-03-30T14:03:45.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:45 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:03:45.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:45 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:03:45.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:45 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:45.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:45 smithi043 ceph-mon[42099]: osdmap e434: 8 total, 8 up, 6 in 2024-03-30T14:03:45.225 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:45.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:45.255 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:45.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:45.328 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-03-30T14:03:45.356 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:45.382 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:45.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:45.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:45 smithi113 ceph-mon[34341]: pgmap v963: 168 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 163 active+clean; 3.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 821 op/s; 12352/179490 objects misplaced (6.882%); 36 MiB/s, 571 objects/s recovering 2024-03-30T14:03:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:45 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:03:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:45 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:03:45.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:45 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:45.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:45 smithi113 ceph-mon[34341]: osdmap e434: 8 total, 8 up, 6 in 2024-03-30T14:03:45.484 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:45.512 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:45.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:45.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:45.613 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:45.641 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:45.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:45.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:45.743 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:45.773 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:45.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:45.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:45.875 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:45.907 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:45.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:45.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:46.009 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:46.045 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:46.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:46.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:46.147 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:46.171 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:46.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:46.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:46.272 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:46.297 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:46.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:46.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:46 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:03:46.606 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:46 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:03:46.606 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:03:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:03:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:03:46.607 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:46 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:03:46.607 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:46 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:03:46.607 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:46.608 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:46 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:03:46.608 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:46 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:03:46.632 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:46.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:46.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:46.734 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:46.758 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:46.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:46.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:46.860 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:46.886 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:46.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:46.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:46.987 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:47.013 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:47.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:47.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:47.114 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:47.146 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:47.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:47.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:47.248 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:47.279 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:47.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:47.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:47.380 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:47.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:47 smithi043 ceph-mon[29963]: pgmap v965: 168 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 163 active+clean; 3.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 746 B/s rd, 0 B/s wr, 1.01k op/s; 11299/170364 objects misplaced (6.632%); 37 MiB/s, 589 objects/s recovering 2024-03-30T14:03:47.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:47 smithi043 ceph-mon[29963]: osdmap e435: 8 total, 8 up, 6 in 2024-03-30T14:03:47.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:47 smithi043 ceph-mon[42099]: pgmap v965: 168 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 163 active+clean; 3.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 746 B/s rd, 0 B/s wr, 1.01k op/s; 11299/170364 objects misplaced (6.632%); 37 MiB/s, 589 objects/s recovering 2024-03-30T14:03:47.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:47 smithi043 ceph-mon[42099]: osdmap e435: 8 total, 8 up, 6 in 2024-03-30T14:03:47.411 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:47.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:47.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:47.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:47 smithi113 ceph-mon[34341]: pgmap v965: 168 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 163 active+clean; 3.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 746 B/s rd, 0 B/s wr, 1.01k op/s; 11299/170364 objects misplaced (6.632%); 37 MiB/s, 589 objects/s recovering 2024-03-30T14:03:47.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:47 smithi113 ceph-mon[34341]: osdmap e435: 8 total, 8 up, 6 in 2024-03-30T14:03:47.513 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:47.549 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:47.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:47.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:47.651 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:47.681 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:47.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:47.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:47.782 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:47.810 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:47.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:47.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:47.911 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:47.949 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:47.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:47.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:48.051 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:48.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:48.087 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:48.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:48.123 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T14:03:48.187 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:48.224 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:48.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:48.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:48.326 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:48.355 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:48.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:48.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:48.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:48 smithi043 ceph-mon[29963]: osdmap e436: 8 total, 8 up, 6 in 2024-03-30T14:03:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:48 smithi043 ceph-mon[42099]: osdmap e436: 8 total, 8 up, 6 in 2024-03-30T14:03:48.456 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:48.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:48 smithi113 ceph-mon[34341]: osdmap e436: 8 total, 8 up, 6 in 2024-03-30T14:03:48.489 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:48.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:48.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:48.591 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:48.621 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:48.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:48.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:48.723 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:48.753 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:48.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:48.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:48.855 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:48.895 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:48.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:48.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:48.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:49.027 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:49.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:49.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:49.129 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:49.160 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:49.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:49.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:49.261 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:49.297 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:49.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:49.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:49.365 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:49 smithi113 ceph-mon[34341]: pgmap v968: 168 pgs: 1 peering, 3 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 163 active+clean; 3.3 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 883 B/s rd, 0 B/s wr, 996 op/s; 10476/164250 objects misplaced (6.378%); 11 MiB/s, 181 objects/s recovering 2024-03-30T14:03:49.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:49 smithi043 ceph-mon[29963]: pgmap v968: 168 pgs: 1 peering, 3 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 163 active+clean; 3.3 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 883 B/s rd, 0 B/s wr, 996 op/s; 10476/164250 objects misplaced (6.378%); 11 MiB/s, 181 objects/s recovering 2024-03-30T14:03:49.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:49 smithi043 ceph-mon[42099]: pgmap v968: 168 pgs: 1 peering, 3 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 163 active+clean; 3.3 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 883 B/s rd, 0 B/s wr, 996 op/s; 10476/164250 objects misplaced (6.378%); 11 MiB/s, 181 objects/s recovering 2024-03-30T14:03:49.398 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:49.428 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:49.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:49.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:49.529 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:49.570 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:49.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:49.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:49.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:49.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:49.713 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:49.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:49.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:03:49 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:03:49.813 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:49.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:49.837 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:49.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:49.938 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:49.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:49.970 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:49.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:50.071 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:50.101 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:50.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:50.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:50.202 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:50.228 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:50.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:50.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:50.330 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:50.362 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:50.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:50.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:50.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:50 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:03:50.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:50 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:03:50.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:50 smithi043 ceph-mon[29963]: osdmap e437: 8 total, 8 up, 6 in 2024-03-30T14:03:50.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:50 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:03:50.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:50 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:03:50.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:50 smithi043 ceph-mon[42099]: osdmap e437: 8 total, 8 up, 6 in 2024-03-30T14:03:50.463 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:50.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:50 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:03:50.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:50 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:03:50.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:50 smithi113 ceph-mon[34341]: osdmap e437: 8 total, 8 up, 6 in 2024-03-30T14:03:50.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:50.500 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:50.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:50.601 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:50.632 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:50.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:50.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:50.734 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:50.765 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:50.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:50.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:50.867 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:50.904 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:50.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:50.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:51.006 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:51.044 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:51.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:51.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:51.146 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:51.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:51.179 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:51.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:51.280 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:51.314 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:51.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:51.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:51.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:51 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:51.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:51 smithi043 ceph-mon[29963]: pgmap v970: 168 pgs: 1 peering, 3 active+remapped+backfill_toofull, 164 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.10k op/s; 9684/154530 objects misplaced (6.267%); 7.7 MiB/s, 123 objects/s recovering 2024-03-30T14:03:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:51 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:51 smithi043 ceph-mon[42099]: pgmap v970: 168 pgs: 1 peering, 3 active+remapped+backfill_toofull, 164 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.10k op/s; 9684/154530 objects misplaced (6.267%); 7.7 MiB/s, 123 objects/s recovering 2024-03-30T14:03:51.416 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:51.444 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:51.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:51.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:51.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:51 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:51.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:51 smithi113 ceph-mon[34341]: pgmap v970: 168 pgs: 1 peering, 3 active+remapped+backfill_toofull, 164 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.10k op/s; 9684/154530 objects misplaced (6.267%); 7.7 MiB/s, 123 objects/s recovering 2024-03-30T14:03:51.547 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:51.575 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:51.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:51.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:51.676 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:51.705 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:51.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:51.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:51.807 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:51.836 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:51.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:51.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:51.938 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:51.961 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:51.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:51.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:52.062 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:52.092 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:52.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:52.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:52.194 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:52.224 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:52.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:52.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:52.326 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:52.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:52.379 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:52.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:52.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:52 smithi043 ceph-mon[42099]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T14:03:52.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:52 smithi043 ceph-mon[29963]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T14:03:52.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:52 smithi113 ceph-mon[34341]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T14:03:52.480 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:52.503 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:52.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:52.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:52.605 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:52.633 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:52.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:52.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:52.735 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:52.772 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:52.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:52.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:52.874 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:52.904 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:52.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:52.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:53.005 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:53.030 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:53.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:53.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:53.132 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:53.170 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:53.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:53.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:53.272 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:53.297 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:53.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:53.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:53 smithi043 ceph-mon[29963]: pgmap v971: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 3.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.26k op/s; 9198/147669 objects misplaced (6.229%); 7.1 MiB/s, 113 objects/s recovering 2024-03-30T14:03:53.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:53 smithi043 ceph-mon[42099]: pgmap v971: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 3.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.26k op/s; 9198/147669 objects misplaced (6.229%); 7.1 MiB/s, 113 objects/s recovering 2024-03-30T14:03:53.399 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:53.441 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:53.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:53.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:53.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:53 smithi113 ceph-mon[34341]: pgmap v971: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 3.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.26k op/s; 9198/147669 objects misplaced (6.229%); 7.1 MiB/s, 113 objects/s recovering 2024-03-30T14:03:53.542 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:53.582 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:53.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:53.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:53.684 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:53.712 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:53.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:53.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:53.814 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:53.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:53.851 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:53.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:53.952 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:53.984 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:53.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:53.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:54.086 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:54.116 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:54.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:54.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:54.218 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:54.255 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:54.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:54.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:54.358 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:54.390 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:54.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:54.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:54.493 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:54.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:54.526 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:54.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:54.627 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:54.654 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:54.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:54.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:54.756 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:54.790 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:54.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:54.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:54.892 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:54.923 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:54.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:54.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:55.024 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:55.056 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:55.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:55.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:55.157 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:55.202 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:55.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:55.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:55.304 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:55.344 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:55.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:55.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:55.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:55 smithi043 ceph-mon[29963]: pgmap v972: 168 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 165 active+clean; 2.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.07k op/s; 8199/145389 objects misplaced (5.639%); 11 MiB/s, 169 objects/s recovering 2024-03-30T14:03:55.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:55 smithi043 ceph-mon[42099]: pgmap v972: 168 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 165 active+clean; 2.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.07k op/s; 8199/145389 objects misplaced (5.639%); 11 MiB/s, 169 objects/s recovering 2024-03-30T14:03:55.447 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:55.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:55 smithi113 ceph-mon[34341]: pgmap v972: 168 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 165 active+clean; 2.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.07k op/s; 8199/145389 objects misplaced (5.639%); 11 MiB/s, 169 objects/s recovering 2024-03-30T14:03:55.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:55.479 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:55.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:55.580 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:55.611 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:55.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:55.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:55.713 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:55.747 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:55.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:55.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:55.754 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:03:55.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:55.898 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:55.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:55.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:56.000 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:56.028 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:56.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:56.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:56.130 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:56.160 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T14:03:56.180 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:56.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:56.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:56.258 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:56 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2186903962' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:03:56.259 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:03:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:03:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:03:56.259 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:56 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2186903962' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:03:56.281 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:56.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:56.332 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:56.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:56.433 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:56.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:56 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2186903962' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:03:56.471 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:56.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:56.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:56.572 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:56.601 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:56.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:56.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:56.702 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:56.732 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:56.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:56.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:56.834 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:56.864 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:56.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:56.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:56.965 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:56.995 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:56.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:56.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:57.097 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:57.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:57.137 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:57.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:57.238 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:57.280 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:57.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:57.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:57.382 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:57.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:57 smithi043 ceph-mon[29963]: pgmap v973: 168 pgs: 3 active+remapped+backfilling, 165 active+clean; 2.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 583 B/s rd, 0 B/s wr, 1.06k op/s; 7445/136209 objects misplaced (5.466%); 9.2 MiB/s, 146 objects/s recovering 2024-03-30T14:03:57.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:57 smithi043 ceph-mon[29963]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-30T14:03:57.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:57 smithi043 ceph-mon[29963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:03:57.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:57 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2186903962' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:03:57.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:57 smithi043 ceph-mon[29963]: osdmap e438: 8 total, 8 up, 6 in 2024-03-30T14:03:57.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:57 smithi043 ceph-mon[42099]: pgmap v973: 168 pgs: 3 active+remapped+backfilling, 165 active+clean; 2.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 583 B/s rd, 0 B/s wr, 1.06k op/s; 7445/136209 objects misplaced (5.466%); 9.2 MiB/s, 146 objects/s recovering 2024-03-30T14:03:57.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:57 smithi043 ceph-mon[42099]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-30T14:03:57.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:57 smithi043 ceph-mon[42099]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:03:57.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:57 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2186903962' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:03:57.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:57 smithi043 ceph-mon[42099]: osdmap e438: 8 total, 8 up, 6 in 2024-03-30T14:03:57.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:57.422 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:57.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:57.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:57 smithi113 ceph-mon[34341]: pgmap v973: 168 pgs: 3 active+remapped+backfilling, 165 active+clean; 2.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 583 B/s rd, 0 B/s wr, 1.06k op/s; 7445/136209 objects misplaced (5.466%); 9.2 MiB/s, 146 objects/s recovering 2024-03-30T14:03:57.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:57 smithi113 ceph-mon[34341]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-30T14:03:57.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:57 smithi113 ceph-mon[34341]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:03:57.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:57 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2186903962' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:03:57.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:57 smithi113 ceph-mon[34341]: osdmap e438: 8 total, 8 up, 6 in 2024-03-30T14:03:57.522 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:57.559 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:57.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:57.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:57.661 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:57.689 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:57.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:57.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:57.757 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:03:57.791 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:57.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:57.849 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:57.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:57.950 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:58.002 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:58.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:58.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:58.103 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:58.132 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:58.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:58.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:58.233 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:58.278 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:58.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:58.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:58.380 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:03:58.409 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:58.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:58.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:03:58.438 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:58 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:03:58.438 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:58 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:03:58.438 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:58 smithi043 ceph-mon[29963]: osdmap e439: 8 total, 8 up, 6 in 2024-03-30T14:03:58.438 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:58 smithi043 ceph-mon[29963]: 3.9 scrub starts 2024-03-30T14:03:58.438 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:58 smithi043 ceph-mon[29963]: 3.9 scrub ok 2024-03-30T14:03:58.439 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:58 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a[29940]: 2024-03-30T14:03:58.264+0000 7ff6065d3700 -1 mon.a@0(leader).osd e439 definitely_dead 0 2024-03-30T14:03:58.439 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:58 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:03:58.439 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:58 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:03:58.439 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:58 smithi043 ceph-mon[42099]: osdmap e439: 8 total, 8 up, 6 in 2024-03-30T14:03:58.439 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:58 smithi043 ceph-mon[42099]: 3.9 scrub starts 2024-03-30T14:03:58.439 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:58 smithi043 ceph-mon[42099]: 3.9 scrub ok 2024-03-30T14:03:58.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:58 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:03:58.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:58 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:03:58.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:58 smithi113 ceph-mon[34341]: osdmap e439: 8 total, 8 up, 6 in 2024-03-30T14:03:58.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:58 smithi113 ceph-mon[34341]: 3.9 scrub starts 2024-03-30T14:03:58.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:58 smithi113 ceph-mon[34341]: 3.9 scrub ok 2024-03-30T14:03:58.511 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:58.551 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:58.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:58.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:58.652 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:03:58.699 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:58.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:58.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:03:58.801 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:58.830 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:58.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:58.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:58.932 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:03:58.960 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:58.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:58.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:03:59.062 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:03:59.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:59.106 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:59.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:03:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:marked out osd.2. 2024-03-30T14:03:59.206 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:03:59.215 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:59 smithi043 ceph-mon[29963]: pgmap v976: 168 pgs: 3 active+remapped+backfilling, 165 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 6644/129891 objects misplaced (5.115%); 13 MiB/s, 204 objects/s recovering 2024-03-30T14:03:59.215 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:59 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3307470183' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-30T14:03:59.216 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:03:59 smithi043 ceph-mon[29963]: Client client.admin marked osd.2 out, while it was still marked up 2024-03-30T14:03:59.216 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:59 smithi043 ceph-mon[42099]: pgmap v976: 168 pgs: 3 active+remapped+backfilling, 165 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 6644/129891 objects misplaced (5.115%); 13 MiB/s, 204 objects/s recovering 2024-03-30T14:03:59.216 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:59 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3307470183' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-30T14:03:59.216 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:03:59 smithi043 ceph-mon[42099]: Client client.admin marked osd.2 out, while it was still marked up 2024-03-30T14:03:59.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:59.234 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:59.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:03:59.335 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:59.361 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:59.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:59.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:59.463 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:59.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:59 smithi113 ceph-mon[34341]: pgmap v976: 168 pgs: 3 active+remapped+backfilling, 165 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 6644/129891 objects misplaced (5.115%); 13 MiB/s, 204 objects/s recovering 2024-03-30T14:03:59.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:59 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3307470183' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-30T14:03:59.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:03:59 smithi113 ceph-mon[34341]: Client client.admin marked osd.2 out, while it was still marked up 2024-03-30T14:03:59.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:59.490 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:59.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:59.509 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T14:03:59.591 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:03:59.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:59.618 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:03:59.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:03:59.718 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:03:59.752 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:59.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:03:59.794 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T14:03:59.795 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-03-30T14:03:59.795 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-03-30T14:03:59.795 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-03-30T14:03:59.795 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.795 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25685.0:31175 11.8 11:1d0b6491:::benchmark_data_smithi043_197439_object31174:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-03-30T14:03:59.795 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:01:38.014055+0000", 2024-03-30T14:03:59.795 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 141.270829144, 2024-03-30T14:03:59.795 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.289479176, 2024-03-30T14:03:59.795 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:59.795 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:59.795 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:59.795 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:59.795 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25685", 2024-03-30T14:03:59.796 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:03:59.796 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 31175 2024-03-30T14:03:59.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.796 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:59.796 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.796 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:59.796 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.014055+0000", 2024-03-30T14:03:59.796 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.796 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.796 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:59.796 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.014055+0000", 2024-03-30T14:03:59.796 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.797 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:59.797 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.014055+0000", 2024-03-30T14:03:59.797 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4900000000000004e-07 2024-03-30T14:03:59.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.797 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:59.797 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.014198+0000", 2024-03-30T14:03:59.797 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000142433 2024-03-30T14:03:59.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.797 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:59.798 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.014211+0000", 2024-03-30T14:03:59.798 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3059e-05 2024-03-30T14:03:59.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.798 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:59.798 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.014220+0000", 2024-03-30T14:03:59.798 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.935e-06 2024-03-30T14:03:59.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.798 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:59.798 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.014343+0000", 2024-03-30T14:03:59.798 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000122982 2024-03-30T14:03:59.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.799 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.799 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:59.799 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.014483+0000", 2024-03-30T14:03:59.799 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014018599999999999 2024-03-30T14:03:59.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.799 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.799 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 5,6", 2024-03-30T14:03:59.799 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.014757+0000", 2024-03-30T14:03:59.799 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00027342000000000002 2024-03-30T14:03:59.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.799 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.799 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:59.799 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.018692+0000", 2024-03-30T14:03:59.800 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0039357289999999998 2024-03-30T14:03:59.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.800 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:59.800 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.018849+0000", 2024-03-30T14:03:59.800 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015655199999999999 2024-03-30T14:03:59.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.800 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:59.800 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.303412+0000", 2024-03-30T14:03:59.800 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.28456342800000001 2024-03-30T14:03:59.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.801 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:59.801 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.303478+0000", 2024-03-30T14:03:59.801 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6030000000000003e-05 2024-03-30T14:03:59.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.801 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:59.801 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.303534+0000", 2024-03-30T14:03:59.801 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5473000000000002e-05 2024-03-30T14:03:59.801 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.801 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.801 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.802 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25685.0:31178 11.8 11:15e37858:::benchmark_data_smithi043_197439_object31177:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-03-30T14:03:59.802 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:01:38.015722+0000", 2024-03-30T14:03:59.802 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 141.26916183899999, 2024-03-30T14:03:59.802 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.29151293700000003, 2024-03-30T14:03:59.802 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:59.802 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:59.802 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:59.802 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:59.802 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25685", 2024-03-30T14:03:59.802 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:03:59.802 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 31178 2024-03-30T14:03:59.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.802 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:59.802 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.802 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.015722+0000", 2024-03-30T14:03:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.015722+0000", 2024-03-30T14:03:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.015723+0000", 2024-03-30T14:03:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9900000000000009e-07 2024-03-30T14:03:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.015744+0000", 2024-03-30T14:03:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0990000000000001e-05 2024-03-30T14:03:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.015757+0000", 2024-03-30T14:03:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3379e-05 2024-03-30T14:03:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.015765+0000", 2024-03-30T14:03:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9279999999999993e-06 2024-03-30T14:03:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.016774+0000", 2024-03-30T14:03:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0010084180000000001 2024-03-30T14:03:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.016939+0000", 2024-03-30T14:03:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016539899999999999 2024-03-30T14:03:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 5,6", 2024-03-30T14:03:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.017168+0000", 2024-03-30T14:03:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00022949999999999999 2024-03-30T14:03:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.019212+0000", 2024-03-30T14:03:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0020437749999999998 2024-03-30T14:03:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.020816+0000", 2024-03-30T14:03:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001603651 2024-03-30T14:03:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.807 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:59.807 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.307133+0000", 2024-03-30T14:03:59.807 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.28631678999999999 2024-03-30T14:03:59.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.807 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:59.807 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.307189+0000", 2024-03-30T14:03:59.807 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6348999999999999e-05 2024-03-30T14:03:59.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.807 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:59.807 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.307235+0000", 2024-03-30T14:03:59.807 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5759e-05 2024-03-30T14:03:59.808 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.808 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.808 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.808 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.808 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25685.0:31180 11.9 11:98c57f37:::benchmark_data_smithi043_197439_object31179:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-03-30T14:03:59.808 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:01:38.015780+0000", 2024-03-30T14:03:59.808 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 141.26910417400001, 2024-03-30T14:03:59.808 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.29361259699999998, 2024-03-30T14:03:59.808 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:59.808 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:59.808 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:59.808 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:59.808 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25685", 2024-03-30T14:03:59.809 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:03:59.809 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 31180 2024-03-30T14:03:59.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.809 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:59.809 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.809 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:59.809 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.015780+0000", 2024-03-30T14:03:59.809 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.809 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.809 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:59.809 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.015780+0000", 2024-03-30T14:03:59.809 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.809 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.810 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:59.810 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.015780+0000", 2024-03-30T14:03:59.810 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8100000000000003e-07 2024-03-30T14:03:59.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.810 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.810 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:59.810 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.015805+0000", 2024-03-30T14:03:59.810 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4782000000000001e-05 2024-03-30T14:03:59.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.810 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.810 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:59.810 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.015814+0000", 2024-03-30T14:03:59.810 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5249999999999994e-06 2024-03-30T14:03:59.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.810 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.810 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:59.811 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.015821+0000", 2024-03-30T14:03:59.811 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4860000000000002e-06 2024-03-30T14:03:59.811 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.811 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:59.811 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.015874+0000", 2024-03-30T14:03:59.811 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2784000000000001e-05 2024-03-30T14:03:59.811 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.811 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:59.811 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.016019+0000", 2024-03-30T14:03:59.811 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014507699999999999 2024-03-30T14:03:59.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.812 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,6", 2024-03-30T14:03:59.812 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.016159+0000", 2024-03-30T14:03:59.812 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014039200000000001 2024-03-30T14:03:59.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.812 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:59.812 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.018123+0000", 2024-03-30T14:03:59.812 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0019639919999999999 2024-03-30T14:03:59.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.812 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:59.813 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.020577+0000", 2024-03-30T14:03:59.813 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024544189999999998 2024-03-30T14:03:59.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.813 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:59.813 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.309295+0000", 2024-03-30T14:03:59.813 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.28871750699999998 2024-03-30T14:03:59.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.813 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:59.813 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.309351+0000", 2024-03-30T14:03:59.813 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6403000000000002e-05 2024-03-30T14:03:59.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.814 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:59.814 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.309392+0000", 2024-03-30T14:03:59.814 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0748999999999999e-05 2024-03-30T14:03:59.814 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.814 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.814 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.814 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.814 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25685.0:31194 11.a 11:54646b35:::benchmark_data_smithi043_197439_object31193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-03-30T14:03:59.814 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:01:38.028286+0000", 2024-03-30T14:03:59.814 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 141.25659800899999, 2024-03-30T14:03:59.814 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.28005552500000003, 2024-03-30T14:03:59.814 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:59.814 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:59.814 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:59.815 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:59.815 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25685", 2024-03-30T14:03:59.815 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:03:59.815 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 31194 2024-03-30T14:03:59.815 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.815 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:59.815 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.815 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:59.815 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.028286+0000", 2024-03-30T14:03:59.815 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.815 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.815 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.815 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:59.815 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.028286+0000", 2024-03-30T14:03:59.815 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.816 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.816 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.816 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:59.816 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.028287+0000", 2024-03-30T14:03:59.816 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6330000000000001e-06 2024-03-30T14:03:59.816 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.816 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.816 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:59.816 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.028320+0000", 2024-03-30T14:03:59.816 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.2863e-05 2024-03-30T14:03:59.816 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.816 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.816 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:59.816 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.028338+0000", 2024-03-30T14:03:59.816 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7652e-05 2024-03-30T14:03:59.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.817 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.817 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:59.817 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.028346+0000", 2024-03-30T14:03:59.817 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1119999999999996e-06 2024-03-30T14:03:59.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.817 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.817 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:59.817 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.028414+0000", 2024-03-30T14:03:59.817 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7656000000000004e-05 2024-03-30T14:03:59.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.817 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.817 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:59.817 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.028596+0000", 2024-03-30T14:03:59.817 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018267000000000001 2024-03-30T14:03:59.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.818 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.818 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 5,6", 2024-03-30T14:03:59.818 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.028703+0000", 2024-03-30T14:03:59.818 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000106772 2024-03-30T14:03:59.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.818 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.818 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:59.818 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.030126+0000", 2024-03-30T14:03:59.818 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001423105 2024-03-30T14:03:59.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.818 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.818 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:59.818 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.030165+0000", 2024-03-30T14:03:59.818 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8714999999999998e-05 2024-03-30T14:03:59.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.819 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.819 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:59.819 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.308251+0000", 2024-03-30T14:03:59.819 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.278086051 2024-03-30T14:03:59.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.819 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.819 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:59.819 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.308302+0000", 2024-03-30T14:03:59.819 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1165000000000002e-05 2024-03-30T14:03:59.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.819 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.819 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:59.819 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.308341+0000", 2024-03-30T14:03:59.820 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9131000000000002e-05 2024-03-30T14:03:59.820 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.820 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.820 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.820 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.820 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.820 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25685.0:31195 11.8 11:13be55ef:::benchmark_data_smithi043_197439_object31194:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-03-30T14:03:59.820 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:01:38.028998+0000", 2024-03-30T14:03:59.820 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 141.25588584600001, 2024-03-30T14:03:59.820 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.27845944099999997, 2024-03-30T14:03:59.820 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:59.820 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:59.820 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:59.820 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:59.821 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25685", 2024-03-30T14:03:59.821 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:03:59.821 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 31195 2024-03-30T14:03:59.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.821 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:59.821 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.821 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:59.821 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.028998+0000", 2024-03-30T14:03:59.821 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.821 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.821 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:59.821 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.028998+0000", 2024-03-30T14:03:59.821 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.822 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.822 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:59.822 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.028999+0000", 2024-03-30T14:03:59.822 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3099999999999996e-07 2024-03-30T14:03:59.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.822 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.029023+0000", 2024-03-30T14:03:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3798000000000002e-05 2024-03-30T14:03:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.029038+0000", 2024-03-30T14:03:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5815999999999999e-05 2024-03-30T14:03:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.029047+0000", 2024-03-30T14:03:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7669999999999992e-06 2024-03-30T14:03:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.029115+0000", 2024-03-30T14:03:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7403000000000005e-05 2024-03-30T14:03:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.029481+0000", 2024-03-30T14:03:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000365986 2024-03-30T14:03:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 5,6", 2024-03-30T14:03:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.029632+0000", 2024-03-30T14:03:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015149099999999999 2024-03-30T14:03:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.032383+0000", 2024-03-30T14:03:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0027512499999999998 2024-03-30T14:03:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.032414+0000", 2024-03-30T14:03:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.0808e-05 2024-03-30T14:03:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.307370+0000", 2024-03-30T14:03:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.274956164 2024-03-30T14:03:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.307420+0000", 2024-03-30T14:03:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0163000000000001e-05 2024-03-30T14:03:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.307457+0000", 2024-03-30T14:03:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6863999999999998e-05 2024-03-30T14:03:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25685.0:31196 11.8 11:16e64130:::benchmark_data_smithi043_197439_object31195:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-03-30T14:03:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:01:38.029412+0000", 2024-03-30T14:03:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 141.25547203100001, 2024-03-30T14:03:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.27815925699999999, 2024-03-30T14:03:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-03-30T14:03:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25685", 2024-03-30T14:03:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:03:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 31196 2024-03-30T14:03:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.029412+0000", 2024-03-30T14:03:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.029412+0000", 2024-03-30T14:03:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.029413+0000", 2024-03-30T14:03:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3900000000000004e-07 2024-03-30T14:03:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.029434+0000", 2024-03-30T14:03:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1341000000000001e-05 2024-03-30T14:03:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.029446+0000", 2024-03-30T14:03:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1788e-05 2024-03-30T14:03:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.029454+0000", 2024-03-30T14:03:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.524e-06 2024-03-30T14:03:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.031120+0000", 2024-03-30T14:03:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001666091 2024-03-30T14:03:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.031265+0000", 2024-03-30T14:03:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014479 2024-03-30T14:03:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 5,6", 2024-03-30T14:03:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.031415+0000", 2024-03-30T14:03:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015001200000000001 2024-03-30T14:03:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.032466+0000", 2024-03-30T14:03:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00105061 2024-03-30T14:03:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-03-30T14:03:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.032554+0000", 2024-03-30T14:03:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8299000000000003e-05 2024-03-30T14:03:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:03:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.307493+0000", 2024-03-30T14:03:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.274939498 2024-03-30T14:03:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.307538+0000", 2024-03-30T14:03:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4183e-05 2024-03-30T14:03:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:38.307571+0000", 2024-03-30T14:03:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3281999999999997e-05 2024-03-30T14:03:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:111345 11.9 e421/418 11:9217ca30:::benchmark_data_smithi043_197439_object1227:head v 421'7073, mlcod=420'7068)", 2024-03-30T14:03:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:02:52.234478+0000", 2024-03-30T14:03:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 67.050405447000003, 2024-03-30T14:03:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.379819032, 2024-03-30T14:03:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.234478+0000", 2024-03-30T14:03:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.234478+0000", 2024-03-30T14:03:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.234479+0000", 2024-03-30T14:03:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0399999999999995e-07 2024-03-30T14:03:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.234486+0000", 2024-03-30T14:03:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.781e-06 2024-03-30T14:03:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.234500+0000", 2024-03-30T14:03:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4335000000000001e-05 2024-03-30T14:03:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.234508+0000", 2024-03-30T14:03:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1620000000000008e-06 2024-03-30T14:03:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.308555+0000", 2024-03-30T14:03:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.074046773999999996 2024-03-30T14:03:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.308578+0000", 2024-03-30T14:03:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2929e-05 2024-03-30T14:03:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.614208+0000", 2024-03-30T14:03:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.305629867 2024-03-30T14:03:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.614297+0000", 2024-03-30T14:03:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.9480000000000004e-05 2024-03-30T14:03:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:111347 11.5 e421/418 11:af88d375:::benchmark_data_smithi043_197439_object1229:head v 421'6993, mlcod=421'6989)", 2024-03-30T14:03:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:02:52.238874+0000", 2024-03-30T14:03:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 67.046009201000004, 2024-03-30T14:03:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.50148895199999999, 2024-03-30T14:03:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.238874+0000", 2024-03-30T14:03:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.238874+0000", 2024-03-30T14:03:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.238876+0000", 2024-03-30T14:03:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2190000000000001e-06 2024-03-30T14:03:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.238885+0000", 2024-03-30T14:03:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7189999999999996e-06 2024-03-30T14:03:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.238901+0000", 2024-03-30T14:03:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5736e-05 2024-03-30T14:03:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.238909+0000", 2024-03-30T14:03:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0760000000000003e-06 2024-03-30T14:03:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.346490+0000", 2024-03-30T14:03:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.107581221 2024-03-30T14:03:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.346732+0000", 2024-03-30T14:03:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00024158400000000001 2024-03-30T14:03:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.740317+0000", 2024-03-30T14:03:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.39358538700000001 2024-03-30T14:03:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.740363+0000", 2024-03-30T14:03:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6010000000000002e-05 2024-03-30T14:03:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:111360 11.1 e421/418 11:83e1862c:::benchmark_data_smithi043_197439_object1242:head v 421'6905, mlcod=420'6901)", 2024-03-30T14:03:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:02:52.313109+0000", 2024-03-30T14:03:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 66.971774753999995, 2024-03-30T14:03:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.42933832999999999, 2024-03-30T14:03:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.313109+0000", 2024-03-30T14:03:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.313109+0000", 2024-03-30T14:03:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.313110+0000", 2024-03-30T14:03:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.048e-06 2024-03-30T14:03:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.313117+0000", 2024-03-30T14:03:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3309999999999998e-06 2024-03-30T14:03:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.313160+0000", 2024-03-30T14:03:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2658999999999999e-05 2024-03-30T14:03:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.313184+0000", 2024-03-30T14:03:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3864999999999999e-05 2024-03-30T14:03:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.346382+0000", 2024-03-30T14:03:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.033198397999999997 2024-03-30T14:03:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:59.846 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.346410+0000", 2024-03-30T14:03:59.846 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.8248000000000001e-05 2024-03-30T14:03:59.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.846 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.846 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:59.846 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.742394+0000", 2024-03-30T14:03:59.846 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.39598308900000001 2024-03-30T14:03:59.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.846 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.846 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:59.846 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.742447+0000", 2024-03-30T14:03:59.846 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3692e-05 2024-03-30T14:03:59.847 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.847 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.847 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.847 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.847 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.847 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:111363 11.5 e421/418 11:a298136d:::benchmark_data_smithi043_197439_object1245:head v 421'6994, mlcod=421'6989)", 2024-03-30T14:03:59.847 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:02:52.347549+0000", 2024-03-30T14:03:59.847 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 66.937334370000002, 2024-03-30T14:03:59.847 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.39284452199999997, 2024-03-30T14:03:59.847 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:59.847 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:59.847 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:59.847 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:59.847 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.848 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:59.848 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.347549+0000", 2024-03-30T14:03:59.848 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.848 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:59.848 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.347549+0000", 2024-03-30T14:03:59.848 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.848 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:59.848 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.347550+0000", 2024-03-30T14:03:59.848 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3600000000000002e-07 2024-03-30T14:03:59.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.849 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:59.849 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.347557+0000", 2024-03-30T14:03:59.849 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1840000000000002e-06 2024-03-30T14:03:59.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.849 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:59.849 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.347570+0000", 2024-03-30T14:03:59.849 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2394e-05 2024-03-30T14:03:59.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.849 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:59.849 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.347578+0000", 2024-03-30T14:03:59.850 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.729e-06 2024-03-30T14:03:59.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.850 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.850 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:59.850 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.465215+0000", 2024-03-30T14:03:59.850 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.117637914 2024-03-30T14:03:59.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.850 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.850 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:59.850 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.465251+0000", 2024-03-30T14:03:59.850 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5312999999999999e-05 2024-03-30T14:03:59.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.850 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.851 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:59.851 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.740369+0000", 2024-03-30T14:03:59.851 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.27511786900000001 2024-03-30T14:03:59.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.851 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:59.851 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.740394+0000", 2024-03-30T14:03:59.851 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5182999999999999e-05 2024-03-30T14:03:59.851 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.851 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.851 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.851 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:111367 11.a e421/418 11:56c0746a:::benchmark_data_smithi043_197439_object1249:head v 421'6962, mlcod=421'6960)", 2024-03-30T14:03:59.852 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:02:52.349366+0000", 2024-03-30T14:03:59.852 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 66.935518121000001, 2024-03-30T14:03:59.852 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.388989313, 2024-03-30T14:03:59.852 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:59.852 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:59.852 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:59.852 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:59.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.852 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:59.852 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.349366+0000", 2024-03-30T14:03:59.852 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.852 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:59.853 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.349366+0000", 2024-03-30T14:03:59.853 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.853 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.853 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:59.853 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.349367+0000", 2024-03-30T14:03:59.853 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4399999999999998e-07 2024-03-30T14:03:59.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.853 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.853 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:59.853 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.349374+0000", 2024-03-30T14:03:59.853 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.323e-06 2024-03-30T14:03:59.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.854 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:59.854 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.349389+0000", 2024-03-30T14:03:59.854 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5603e-05 2024-03-30T14:03:59.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.854 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:59.854 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.349397+0000", 2024-03-30T14:03:59.854 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7619999999999995e-06 2024-03-30T14:03:59.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.854 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:59.854 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.349466+0000", 2024-03-30T14:03:59.855 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9177000000000005e-05 2024-03-30T14:03:59.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.855 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:59.855 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.349491+0000", 2024-03-30T14:03:59.855 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4439000000000001e-05 2024-03-30T14:03:59.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.855 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:59.855 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.738301+0000", 2024-03-30T14:03:59.855 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.38880977799999999 2024-03-30T14:03:59.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.855 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:59.856 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.738355+0000", 2024-03-30T14:03:59.856 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4286999999999997e-05 2024-03-30T14:03:59.856 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.856 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.856 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.856 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.856 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:111368 11.2 e421/418 11:42ee6a94:::benchmark_data_smithi043_197439_object1250:head v 421'6890, mlcod=421'6888)", 2024-03-30T14:03:59.856 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:02:52.352289+0000", 2024-03-30T14:03:59.856 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 66.932594351000006, 2024-03-30T14:03:59.856 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.26197097699999999, 2024-03-30T14:03:59.856 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:59.856 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:59.856 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:59.857 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:59.857 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.857 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:59.857 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.352289+0000", 2024-03-30T14:03:59.857 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.857 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.857 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:59.857 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.352289+0000", 2024-03-30T14:03:59.857 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.857 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.857 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:59.857 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.352290+0000", 2024-03-30T14:03:59.858 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0950000000000001e-06 2024-03-30T14:03:59.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.858 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.858 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:59.858 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.352299+0000", 2024-03-30T14:03:59.858 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4889999999999995e-06 2024-03-30T14:03:59.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.858 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.858 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:59.858 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.352312+0000", 2024-03-30T14:03:59.858 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2645e-05 2024-03-30T14:03:59.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.858 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.859 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:59.859 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.352320+0000", 2024-03-30T14:03:59.859 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2290000000000003e-06 2024-03-30T14:03:59.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.859 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:59.859 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.352382+0000", 2024-03-30T14:03:59.859 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1962000000000001e-05 2024-03-30T14:03:59.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.859 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:59.859 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.352407+0000", 2024-03-30T14:03:59.859 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.516e-05 2024-03-30T14:03:59.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.614213+0000", 2024-03-30T14:03:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.26180580399999998 2024-03-30T14:03:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.614260+0000", 2024-03-30T14:03:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7593000000000003e-05 2024-03-30T14:03:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:111370 11.9 e421/418 11:90b0e587:::benchmark_data_smithi043_197439_object1252:head v 421'7074, mlcod=421'7071)", 2024-03-30T14:03:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:02:52.357423+0000", 2024-03-30T14:03:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 66.927460492999998, 2024-03-30T14:03:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.38505979000000001, 2024-03-30T14:03:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.357423+0000", 2024-03-30T14:03:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.357423+0000", 2024-03-30T14:03:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.357424+0000", 2024-03-30T14:03:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.15e-06 2024-03-30T14:03:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.357433+0000", 2024-03-30T14:03:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0459999999999994e-06 2024-03-30T14:03:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.357452+0000", 2024-03-30T14:03:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9079000000000001e-05 2024-03-30T14:03:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.357462+0000", 2024-03-30T14:03:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1530000000000003e-06 2024-03-30T14:03:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.464846+0000", 2024-03-30T14:03:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.10738420899999999 2024-03-30T14:03:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.464884+0000", 2024-03-30T14:03:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8229000000000003e-05 2024-03-30T14:03:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.742452+0000", 2024-03-30T14:03:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.27756815699999998 2024-03-30T14:03:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.742483+0000", 2024-03-30T14:03:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.0766999999999999e-05 2024-03-30T14:03:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:111374 11.2 e421/418 11:4edbfb88:::benchmark_data_smithi043_197439_object1256:head v 421'6891, mlcod=421'6888)", 2024-03-30T14:03:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:02:52.366840+0000", 2024-03-30T14:03:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 66.918043757000007, 2024-03-30T14:03:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.37155924200000001, 2024-03-30T14:03:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.366840+0000", 2024-03-30T14:03:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.366840+0000", 2024-03-30T14:03:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.366841+0000", 2024-03-30T14:03:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.209e-06 2024-03-30T14:03:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.366851+0000", 2024-03-30T14:03:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9170000000000001e-06 2024-03-30T14:03:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.366878+0000", 2024-03-30T14:03:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.6483e-05 2024-03-30T14:03:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.366889+0000", 2024-03-30T14:03:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.137e-05 2024-03-30T14:03:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.464849+0000", 2024-03-30T14:03:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.097960551000000007 2024-03-30T14:03:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.464887+0000", 2024-03-30T14:03:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7586999999999998e-05 2024-03-30T14:03:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.738361+0000", 2024-03-30T14:03:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.27347376800000001 2024-03-30T14:03:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.738399+0000", 2024-03-30T14:03:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8356999999999999e-05 2024-03-30T14:03:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:111382 11.5 e421/418 11:afa63c67:::benchmark_data_smithi043_197439_object1264:head v 421'6995, mlcod=421'6991)", 2024-03-30T14:03:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:02:52.471315+0000", 2024-03-30T14:03:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 66.813569004000001, 2024-03-30T14:03:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.26910358000000001, 2024-03-30T14:03:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.471315+0000", 2024-03-30T14:03:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.471315+0000", 2024-03-30T14:03:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.471316+0000", 2024-03-30T14:03:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2869999999999999e-06 2024-03-30T14:03:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.471325+0000", 2024-03-30T14:03:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1679999999999993e-06 2024-03-30T14:03:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.471344+0000", 2024-03-30T14:03:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8899999999999999e-05 2024-03-30T14:03:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.471353+0000", 2024-03-30T14:03:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4550000000000001e-06 2024-03-30T14:03:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.614247+0000", 2024-03-30T14:03:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14289317700000001 2024-03-30T14:03:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.614272+0000", 2024-03-30T14:03:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5800999999999999e-05 2024-03-30T14:03:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.740396+0000", 2024-03-30T14:03:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.126123597 2024-03-30T14:03:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.740418+0000", 2024-03-30T14:03:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2195e-05 2024-03-30T14:03:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:111385 11.5 e421/418 11:abbc268a:::benchmark_data_smithi043_197439_object1267:head v 421'6996, mlcod=421'6991)", 2024-03-30T14:03:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:02:52.481526+0000", 2024-03-30T14:03:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 66.803357825999996, 2024-03-30T14:03:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.25905530599999999, 2024-03-30T14:03:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.481526+0000", 2024-03-30T14:03:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.481526+0000", 2024-03-30T14:03:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.481527+0000", 2024-03-30T14:03:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.331e-06 2024-03-30T14:03:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.481537+0000", 2024-03-30T14:03:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7040000000000006e-06 2024-03-30T14:03:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.481560+0000", 2024-03-30T14:03:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2699000000000001e-05 2024-03-30T14:03:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.481569+0000", 2024-03-30T14:03:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7569999999999999e-06 2024-03-30T14:03:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.738323+0000", 2024-03-30T14:03:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.25675404800000001 2024-03-30T14:03:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.738495+0000", 2024-03-30T14:03:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017201599999999999 2024-03-30T14:03:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.740538+0000", 2024-03-30T14:03:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00204294 2024-03-30T14:03:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.740581+0000", 2024-03-30T14:03:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2811000000000001e-05 2024-03-30T14:03:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:111388 11.1 e421/418 11:867604fa:::benchmark_data_smithi043_197439_object1270:head v 421'6906, mlcod=421'6903)", 2024-03-30T14:03:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:02:52.483480+0000", 2024-03-30T14:03:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 66.801403319000002, 2024-03-30T14:03:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.259032076, 2024-03-30T14:03:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.483480+0000", 2024-03-30T14:03:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.483480+0000", 2024-03-30T14:03:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.483481+0000", 2024-03-30T14:03:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.2899999999999999e-07 2024-03-30T14:03:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.483485+0000", 2024-03-30T14:03:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5510000000000001e-06 2024-03-30T14:03:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.483492+0000", 2024-03-30T14:03:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3799999999999999e-06 2024-03-30T14:03:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.483497+0000", 2024-03-30T14:03:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.648e-06 2024-03-30T14:03:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:59.882 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.614114+0000", 2024-03-30T14:03:59.882 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13061687999999999 2024-03-30T14:03:59.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.882 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.882 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:59.882 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.614142+0000", 2024-03-30T14:03:59.882 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7387000000000001e-05 2024-03-30T14:03:59.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.882 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.882 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:59.882 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.742485+0000", 2024-03-30T14:03:59.882 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.12834387799999999 2024-03-30T14:03:59.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.883 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:59.883 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:52.742512+0000", 2024-03-30T14:03:59.883 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7022999999999999e-05 2024-03-30T14:03:59.883 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.883 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.883 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.883 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:166728 11.8 e434/418 11:14b58e6d:::benchmark_data_smithi043_197439_object56610:head v 434'10470, mlcod=434'10468)", 2024-03-30T14:03:59.883 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:03:45.617931+0000", 2024-03-30T14:03:59.883 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 13.666952725, 2024-03-30T14:03:59.883 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.020181681, 2024-03-30T14:03:59.884 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:59.884 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:59.884 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:59.884 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:59.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.884 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:59.884 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:45.617931+0000", 2024-03-30T14:03:59.884 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.884 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:59.884 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:45.617931+0000", 2024-03-30T14:03:59.884 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.885 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:59.885 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:45.617932+0000", 2024-03-30T14:03:59.885 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4600000000000003e-07 2024-03-30T14:03:59.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.885 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:59.885 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:45.617939+0000", 2024-03-30T14:03:59.885 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9940000000000003e-06 2024-03-30T14:03:59.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.885 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:59.885 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:45.617954+0000", 2024-03-30T14:03:59.885 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4994000000000001e-05 2024-03-30T14:03:59.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.886 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:59.886 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:45.617962+0000", 2024-03-30T14:03:59.886 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9179999999999994e-06 2024-03-30T14:03:59.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.886 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:59.886 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:45.619488+0000", 2024-03-30T14:03:59.886 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00152633 2024-03-30T14:03:59.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.886 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:59.886 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:45.619624+0000", 2024-03-30T14:03:59.887 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013570800000000001 2024-03-30T14:03:59.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.887 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:59.887 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:45.637956+0000", 2024-03-30T14:03:59.887 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.018331766999999999 2024-03-30T14:03:59.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.887 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:59.887 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:45.638113+0000", 2024-03-30T14:03:59.887 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000157124 2024-03-30T14:03:59.887 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.887 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.887 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.888 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:166965 11.0 e434/418 11:0369bff5:::benchmark_data_smithi043_197439_object56847:head v 434'10358, mlcod=434'10355)", 2024-03-30T14:03:59.888 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:03:45.806589+0000", 2024-03-30T14:03:59.888 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 13.478295129999999, 2024-03-30T14:03:59.888 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.018613856000000002, 2024-03-30T14:03:59.888 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:59.888 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:59.888 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:59.888 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:59.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.888 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:59.888 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:45.806589+0000", 2024-03-30T14:03:59.889 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.889 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:59.889 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:45.806589+0000", 2024-03-30T14:03:59.889 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.889 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:59.889 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:45.806589+0000", 2024-03-30T14:03:59.889 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.23e-07 2024-03-30T14:03:59.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.889 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:59.890 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:45.806592+0000", 2024-03-30T14:03:59.890 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3220000000000001e-06 2024-03-30T14:03:59.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.890 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:59.890 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:45.806597+0000", 2024-03-30T14:03:59.890 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9439999999999996e-06 2024-03-30T14:03:59.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.890 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:59.890 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:45.806601+0000", 2024-03-30T14:03:59.890 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0219999999999998e-06 2024-03-30T14:03:59.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.891 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:59.891 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:45.815660+0000", 2024-03-30T14:03:59.891 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0090589380000000008 2024-03-30T14:03:59.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.891 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:59.891 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:45.815782+0000", 2024-03-30T14:03:59.891 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012161 2024-03-30T14:03:59.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.891 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:59.891 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:45.825034+0000", 2024-03-30T14:03:59.891 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.009252751 2024-03-30T14:03:59.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.892 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:59.892 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:45.825202+0000", 2024-03-30T14:03:59.892 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016804600000000001 2024-03-30T14:03:59.892 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.892 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.892 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.892 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:177060 11.5 e437/418 11:ac0b412b:::benchmark_data_smithi043_197439_object66942:head v 437'11092, mlcod=437'11090)", 2024-03-30T14:03:59.892 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:03:54.847522+0000", 2024-03-30T14:03:59.892 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 4.4373620169999999, 2024-03-30T14:03:59.892 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.021225489, 2024-03-30T14:03:59.892 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:03:59.892 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:03:59.893 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:03:59.893 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:03:59.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.893 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:03:59.893 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:54.847522+0000", 2024-03-30T14:03:59.893 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.893 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:03:59.893 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:54.847522+0000", 2024-03-30T14:03:59.893 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:03:59.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.894 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:03:59.894 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:54.847523+0000", 2024-03-30T14:03:59.894 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2950000000000001e-06 2024-03-30T14:03:59.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.894 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:03:59.894 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:54.847532+0000", 2024-03-30T14:03:59.894 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6060000000000003e-06 2024-03-30T14:03:59.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.894 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:03:59.894 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:54.847549+0000", 2024-03-30T14:03:59.894 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6974999999999999e-05 2024-03-30T14:03:59.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.895 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:03:59.895 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:54.847558+0000", 2024-03-30T14:03:59.895 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2140000000000002e-06 2024-03-30T14:03:59.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.895 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:03:59.895 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:54.847751+0000", 2024-03-30T14:03:59.895 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00019364 2024-03-30T14:03:59.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.895 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:03:59.895 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:54.847808+0000", 2024-03-30T14:03:59.895 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6926999999999999e-05 2024-03-30T14:03:59.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.896 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:03:59.896 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:54.868495+0000", 2024-03-30T14:03:59.896 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.020686467 2024-03-30T14:03:59.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:03:59.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:03:59.896 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:03:59.896 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:54.868747+0000", 2024-03-30T14:03:59.896 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000252365 2024-03-30T14:03:59.896 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.896 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.896 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.897 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:03:59.897 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:03:59.897 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T14:03:59.897 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:03:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:03:59.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:03:59.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:03:59.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:03:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:04:00.028 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:00.054 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:00.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:00.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:00.156 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:00.186 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:00.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:00.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:00.288 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:00.313 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:00.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:00.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:00.414 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:00.440 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:00.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:00 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3307470183' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-03-30T14:04:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:00 smithi113 ceph-mon[34341]: osdmap e440: 8 total, 8 up, 5 in 2024-03-30T14:04:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:00 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:04:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:00 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:04:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:00 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:00 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:04:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:00 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:00 smithi113 ceph-mon[34341]: 6.1a scrub starts 2024-03-30T14:04:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:00 smithi113 ceph-mon[34341]: 6.1a scrub ok 2024-03-30T14:04:00.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:00 smithi113 ceph-mon[34341]: pgmap v978: 168 pgs: 3 active+remapped+backfilling, 165 active+clean; 2.4 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.52k op/s; 4721/117948 objects misplaced (4.003%); 24 MiB/s, 376 objects/s recovering 2024-03-30T14:04:00.542 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:00.568 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:00.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:00.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:00.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:00 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3307470183' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-03-30T14:04:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:00 smithi043 ceph-mon[29963]: osdmap e440: 8 total, 8 up, 5 in 2024-03-30T14:04:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:00 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:04:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:00 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:04:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:00 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:00 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:04:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:00 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:00 smithi043 ceph-mon[29963]: 6.1a scrub starts 2024-03-30T14:04:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:00 smithi043 ceph-mon[29963]: 6.1a scrub ok 2024-03-30T14:04:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:00 smithi043 ceph-mon[29963]: pgmap v978: 168 pgs: 3 active+remapped+backfilling, 165 active+clean; 2.4 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.52k op/s; 4721/117948 objects misplaced (4.003%); 24 MiB/s, 376 objects/s recovering 2024-03-30T14:04:00.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:00 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3307470183' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-03-30T14:04:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:00 smithi043 ceph-mon[42099]: osdmap e440: 8 total, 8 up, 5 in 2024-03-30T14:04:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:00 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:04:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:00 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:04:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:00 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:00 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:04:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:00 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:00 smithi043 ceph-mon[42099]: 6.1a scrub starts 2024-03-30T14:04:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:00 smithi043 ceph-mon[42099]: 6.1a scrub ok 2024-03-30T14:04:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:00 smithi043 ceph-mon[42099]: pgmap v978: 168 pgs: 3 active+remapped+backfilling, 165 active+clean; 2.4 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.52k op/s; 4721/117948 objects misplaced (4.003%); 24 MiB/s, 376 objects/s recovering 2024-03-30T14:04:00.670 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:00.700 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:00.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:00.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:00.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:00.830 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:00.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:00.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:00.932 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:00.958 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:00.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:00.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:01.060 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:01.084 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:01.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:01.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:01.186 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:01.219 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:01.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:01.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:01.320 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:01.353 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:01.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:01.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:01.455 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:01.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:01 smithi113 ceph-mon[34341]: osdmap e441: 8 total, 8 up, 5 in 2024-03-30T14:04:01.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:01 smithi113 ceph-mon[34341]: 6.12 scrub starts 2024-03-30T14:04:01.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:01 smithi113 ceph-mon[34341]: 6.12 scrub ok 2024-03-30T14:04:01.488 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:01.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:01.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:01.589 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:01.629 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:01.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:01.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:01.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:01 smithi043 ceph-mon[29963]: osdmap e441: 8 total, 8 up, 5 in 2024-03-30T14:04:01.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:01 smithi043 ceph-mon[29963]: 6.12 scrub starts 2024-03-30T14:04:01.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:01 smithi043 ceph-mon[29963]: 6.12 scrub ok 2024-03-30T14:04:01.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:01 smithi043 ceph-mon[42099]: osdmap e441: 8 total, 8 up, 5 in 2024-03-30T14:04:01.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:01 smithi043 ceph-mon[42099]: 6.12 scrub starts 2024-03-30T14:04:01.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:01 smithi043 ceph-mon[42099]: 6.12 scrub ok 2024-03-30T14:04:01.730 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:01.761 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:01.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:01.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:01.863 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:01.895 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:01.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:01.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:01.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:02.027 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:02.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:02.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:02.129 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:02.160 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:02.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:02.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:02.261 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:02.295 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:02.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:02.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:02.397 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:02.439 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:02.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:02.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:02.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:02 smithi113 ceph-mon[34341]: 11.d starting backfill to osd.1 from (0'0,0'0] MAX to 439'11570 2024-03-30T14:04:02.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:02 smithi113 ceph-mon[34341]: 11.6 starting backfill to osd.4 from (0'0,0'0] MAX to 439'11384 2024-03-30T14:04:02.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:02 smithi113 ceph-mon[34341]: osdmap e442: 8 total, 8 up, 5 in 2024-03-30T14:04:02.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:02 smithi113 ceph-mon[34341]: 11.1 starting backfill to osd.4 from (0'0,0'0] MAX to 439'11335 2024-03-30T14:04:02.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:02 smithi113 ceph-mon[34341]: 11.4 starting backfill to osd.0 from (0'0,0'0] MAX to 439'11621 2024-03-30T14:04:02.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:02 smithi113 ceph-mon[34341]: 11.c starting backfill to osd.5 from (0'0,0'0] MAX to 439'11172 2024-03-30T14:04:02.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:02 smithi113 ceph-mon[34341]: 11.2 starting backfill to osd.5 from (0'0,0'0] MAX to 439'11265 2024-03-30T14:04:02.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:02 smithi113 ceph-mon[34341]: 11.f starting backfill to osd.4 from (0'0,0'0] MAX to 439'11305 2024-03-30T14:04:02.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:02 smithi113 ceph-mon[34341]: pgmap v981: 168 pgs: 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 5 active+remapped+backfilling, 160 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 659 B/s rd, 0 B/s wr, 1.13k op/s; 12133/114069 objects misplaced (10.637%); 41 MiB/s, 667 objects/s recovering 2024-03-30T14:04:02.540 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:02.576 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:02.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:02.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:02 smithi043 ceph-mon[29963]: 11.d starting backfill to osd.1 from (0'0,0'0] MAX to 439'11570 2024-03-30T14:04:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:02 smithi043 ceph-mon[29963]: 11.6 starting backfill to osd.4 from (0'0,0'0] MAX to 439'11384 2024-03-30T14:04:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:02 smithi043 ceph-mon[29963]: osdmap e442: 8 total, 8 up, 5 in 2024-03-30T14:04:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:02 smithi043 ceph-mon[29963]: 11.1 starting backfill to osd.4 from (0'0,0'0] MAX to 439'11335 2024-03-30T14:04:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:02 smithi043 ceph-mon[29963]: 11.4 starting backfill to osd.0 from (0'0,0'0] MAX to 439'11621 2024-03-30T14:04:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:02 smithi043 ceph-mon[29963]: 11.c starting backfill to osd.5 from (0'0,0'0] MAX to 439'11172 2024-03-30T14:04:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:02 smithi043 ceph-mon[29963]: 11.2 starting backfill to osd.5 from (0'0,0'0] MAX to 439'11265 2024-03-30T14:04:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:02 smithi043 ceph-mon[29963]: 11.f starting backfill to osd.4 from (0'0,0'0] MAX to 439'11305 2024-03-30T14:04:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:02 smithi043 ceph-mon[29963]: pgmap v981: 168 pgs: 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 5 active+remapped+backfilling, 160 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 659 B/s rd, 0 B/s wr, 1.13k op/s; 12133/114069 objects misplaced (10.637%); 41 MiB/s, 667 objects/s recovering 2024-03-30T14:04:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:02 smithi043 ceph-mon[42099]: 11.d starting backfill to osd.1 from (0'0,0'0] MAX to 439'11570 2024-03-30T14:04:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:02 smithi043 ceph-mon[42099]: 11.6 starting backfill to osd.4 from (0'0,0'0] MAX to 439'11384 2024-03-30T14:04:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:02 smithi043 ceph-mon[42099]: osdmap e442: 8 total, 8 up, 5 in 2024-03-30T14:04:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:02 smithi043 ceph-mon[42099]: 11.1 starting backfill to osd.4 from (0'0,0'0] MAX to 439'11335 2024-03-30T14:04:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:02 smithi043 ceph-mon[42099]: 11.4 starting backfill to osd.0 from (0'0,0'0] MAX to 439'11621 2024-03-30T14:04:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:02 smithi043 ceph-mon[42099]: 11.c starting backfill to osd.5 from (0'0,0'0] MAX to 439'11172 2024-03-30T14:04:02.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:02 smithi043 ceph-mon[42099]: 11.2 starting backfill to osd.5 from (0'0,0'0] MAX to 439'11265 2024-03-30T14:04:02.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:02 smithi043 ceph-mon[42099]: 11.f starting backfill to osd.4 from (0'0,0'0] MAX to 439'11305 2024-03-30T14:04:02.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:02 smithi043 ceph-mon[42099]: pgmap v981: 168 pgs: 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 5 active+remapped+backfilling, 160 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 659 B/s rd, 0 B/s wr, 1.13k op/s; 12133/114069 objects misplaced (10.637%); 41 MiB/s, 667 objects/s recovering 2024-03-30T14:04:02.678 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:02.714 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:02.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:02.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:02.817 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:02.849 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:02.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:02.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:02.951 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:02.993 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:02.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:02.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:03.094 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:03.127 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:03.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:03.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:03.229 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:03.253 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:03.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:03.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:03.355 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:03.393 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:03.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:03.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:03.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:03 smithi113 ceph-mon[34341]: osdmap e443: 8 total, 8 up, 5 in 2024-03-30T14:04:03.466 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:03 smithi113 ceph-mon[34341]: osdmap e444: 8 total, 8 up, 5 in 2024-03-30T14:04:03.494 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:03.527 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:03.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:03.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:03.629 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:03.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:03 smithi043 ceph-mon[29963]: osdmap e443: 8 total, 8 up, 5 in 2024-03-30T14:04:03.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:03 smithi043 ceph-mon[29963]: osdmap e444: 8 total, 8 up, 5 in 2024-03-30T14:04:03.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:03 smithi043 ceph-mon[42099]: osdmap e443: 8 total, 8 up, 5 in 2024-03-30T14:04:03.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:03 smithi043 ceph-mon[42099]: osdmap e444: 8 total, 8 up, 5 in 2024-03-30T14:04:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:03.669 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:03.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:03.770 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:03.810 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:03.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:03.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:03.911 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:03.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:03.950 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:03.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:04.051 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:04.076 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:04.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:04.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:04.178 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:04.211 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:04.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:04.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:04.313 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:04.347 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:04.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:04.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:04.448 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:04.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:04 smithi113 ceph-mon[34341]: pgmap v984: 168 pgs: 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 4 active+remapped+backfilling, 161 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 767 B/s rd, 0 B/s wr, 396 op/s; 10962/113196 objects misplaced (9.684%); 45 MiB/s, 732 objects/s recovering 2024-03-30T14:04:04.479 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:04.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:04.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:04.581 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:04.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:04.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:04.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:04 smithi043 ceph-mon[29963]: pgmap v984: 168 pgs: 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 4 active+remapped+backfilling, 161 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 767 B/s rd, 0 B/s wr, 396 op/s; 10962/113196 objects misplaced (9.684%); 45 MiB/s, 732 objects/s recovering 2024-03-30T14:04:04.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:04 smithi043 ceph-mon[42099]: pgmap v984: 168 pgs: 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 4 active+remapped+backfilling, 161 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 767 B/s rd, 0 B/s wr, 396 op/s; 10962/113196 objects misplaced (9.684%); 45 MiB/s, 732 objects/s recovering 2024-03-30T14:04:04.711 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:04.740 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:04.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:04.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:04.842 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:04.873 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:04.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:04.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:04.975 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:05.002 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:05.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:05.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:05.104 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:05.132 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:05.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:05.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:05.234 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:05.265 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:05.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:05.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:05.366 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:05.401 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:05.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:05.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:05.504 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:05.540 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:05.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:05.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:05.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:05 smithi043 ceph-mon[29963]: osdmap e445: 8 total, 8 up, 5 in 2024-03-30T14:04:05.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:05 smithi043 ceph-mon[42099]: osdmap e445: 8 total, 8 up, 5 in 2024-03-30T14:04:05.643 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:05.683 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:05.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:05.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:05 smithi113 ceph-mon[34341]: osdmap e445: 8 total, 8 up, 5 in 2024-03-30T14:04:05.783 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:05.815 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:05.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:05.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:05.916 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:05.953 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:05.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:05.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:05.971 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6, 5] out_osds: [7, 3, 2] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-03-30T14:04:05.971 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T14:04:05.971 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-30T14:04:05.971 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.000000 2024-03-30T14:04:05.972 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 4 0 2024-03-30T14:04:06.030 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-30T14:04:06.056 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:06.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:06.111 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:06.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:06.212 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:06.230 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:04:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:04:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:04:06.231 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:06 smithi043 ceph-mon[29963]: osdmap e446: 8 total, 8 up, 5 in 2024-03-30T14:04:06.231 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:06 smithi043 ceph-mon[29963]: pgmap v987: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 5 active+remapped+backfilling, 158 active+clean; 2.2 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 383 op/s; 14515/109470 objects misplaced (13.259%); 69 MiB/s, 1.10k objects/s recovering 2024-03-30T14:04:06.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:06.252 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:06.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:06.353 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:06.398 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:04:06.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:06.399 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:06.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:06.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:06 smithi113 ceph-mon[34341]: osdmap e446: 8 total, 8 up, 5 in 2024-03-30T14:04:06.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:06 smithi113 ceph-mon[34341]: pgmap v987: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 5 active+remapped+backfilling, 158 active+clean; 2.2 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 383 op/s; 14515/109470 objects misplaced (13.259%); 69 MiB/s, 1.10k objects/s recovering 2024-03-30T14:04:06.496 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:06 smithi043 ceph-mon[42099]: osdmap e446: 8 total, 8 up, 5 in 2024-03-30T14:04:06.496 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:06 smithi043 ceph-mon[42099]: pgmap v987: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 5 active+remapped+backfilling, 158 active+clean; 2.2 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 383 op/s; 14515/109470 objects misplaced (13.259%); 69 MiB/s, 1.10k objects/s recovering 2024-03-30T14:04:06.500 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:06.535 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:06.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:06.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:06.637 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:06.681 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:06.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:06.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:06.783 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:06.824 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:06.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:06.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:07.003 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:04:07.049 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:07.079 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:04:07.083 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:07.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:07.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:07.185 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:07.220 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:07.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:07.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:07.322 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:07.357 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:07.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:07.458 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:07.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:07.495 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:07.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:07.572 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:07 smithi043 ceph-mon[42099]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T14:04:07.574 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:07 smithi043 ceph-mon[29963]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T14:04:07.596 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:07.631 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:07.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:07.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:07.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:07 smithi113 ceph-mon[34341]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-30T14:04:07.733 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:07.765 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:07.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:07.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:07.867 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:07.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:07.913 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:07.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:08.014 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:08.052 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:08.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:08.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:08.154 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:08.186 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:08.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:08.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:08.288 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:08.318 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:08.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:08.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:08.420 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:08.449 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:08.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:08.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:08.549 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T14:04:08.550 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:08.577 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T14:04:08.578 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:08.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:08.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:08.605 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T14:04:08.634 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T14:04:08.662 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T14:04:08.680 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:08.692 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T14:04:08.709 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:08.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:08.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:08.721 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T14:04:08.752 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T14:04:08.754 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:04:08.754 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:04:08.754 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:04:08.754 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:04:08.754 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:04:08.754 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:04:08.754 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:04:08.755 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:04:08.810 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:08.839 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:08.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:08.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:08.941 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:08.972 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:08.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:08.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:09.073 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:09.097 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:09.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:09.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:09.199 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:09.225 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:08 smithi043 ceph-mon[29963]: pgmap v988: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 159 active+clean; 2.1 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 911 B/s rd, 0 B/s wr, 557 op/s; 10229/104688 objects misplaced (9.771%); 87 MiB/s, 1.40k objects/s recovering 2024-03-30T14:04:09.225 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:08 smithi043 ceph-mon[29963]: osdmap e447: 8 total, 8 up, 5 in 2024-03-30T14:04:09.225 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:08 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2429137269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:04:09.225 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:08 smithi043 ceph-mon[42099]: pgmap v988: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 159 active+clean; 2.1 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 911 B/s rd, 0 B/s wr, 557 op/s; 10229/104688 objects misplaced (9.771%); 87 MiB/s, 1.40k objects/s recovering 2024-03-30T14:04:09.225 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:08 smithi043 ceph-mon[42099]: osdmap e447: 8 total, 8 up, 5 in 2024-03-30T14:04:09.225 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:08 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2429137269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:04:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:09.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:09.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:09.292 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:08 smithi113 ceph-mon[34341]: pgmap v988: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 2 active+remapped+backfill_wait, 5 active+remapped+backfilling, 159 active+clean; 2.1 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 911 B/s rd, 0 B/s wr, 557 op/s; 10229/104688 objects misplaced (9.771%); 87 MiB/s, 1.40k objects/s recovering 2024-03-30T14:04:09.292 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:08 smithi113 ceph-mon[34341]: osdmap e447: 8 total, 8 up, 5 in 2024-03-30T14:04:09.293 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:08 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2429137269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:04:09.340 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:09.372 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:09.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:09.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:09.474 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:09.507 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:09.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:09.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:09.609 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:09.638 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:09.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:09.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:09.740 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:09.779 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:09.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:09.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:09.881 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:09.920 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:09.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:09.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:04:09 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:04:10.022 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:10.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:10.069 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:10.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:10.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:09 smithi043 ceph-mon[29963]: osdmap e448: 8 total, 8 up, 5 in 2024-03-30T14:04:10.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:09 smithi043 ceph-mon[42099]: osdmap e448: 8 total, 8 up, 5 in 2024-03-30T14:04:10.170 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:10.201 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:10.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:10.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:10.303 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:10.312 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:09 smithi113 ceph-mon[34341]: osdmap e448: 8 total, 8 up, 5 in 2024-03-30T14:04:10.345 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:10.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:10.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:10.447 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:10.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:10.483 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:10.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:10.584 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:10.621 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:10.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:10.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:10.722 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:10.773 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:10.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:10.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:10.874 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:10.900 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:10.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:10.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:11.002 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:11.011 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-03-30T14:04:11.043 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:11.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:11.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:11.146 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:11.170 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:11.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:11.272 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:11.297 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:11.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:11.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:11.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:10 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:11.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:10 smithi043 ceph-mon[29963]: pgmap v991: 168 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 4 active+remapped+backfilling, 161 active+clean; 2.0 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 913 B/s rd, 0 B/s wr, 768 op/s; 7300/96567 objects misplaced (7.560%); 66 MiB/s, 1.06k objects/s recovering 2024-03-30T14:04:11.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:10 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3327161585' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:04:11.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:10 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:04:11.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:10 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:11.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:10 smithi043 ceph-mon[42099]: pgmap v991: 168 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 4 active+remapped+backfilling, 161 active+clean; 2.0 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 913 B/s rd, 0 B/s wr, 768 op/s; 7300/96567 objects misplaced (7.560%); 66 MiB/s, 1.06k objects/s recovering 2024-03-30T14:04:11.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:10 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3327161585' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:04:11.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:10 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:04:11.399 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:11.422 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:11.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:11.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:11.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:10 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:11.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:10 smithi113 ceph-mon[34341]: pgmap v991: 168 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 4 active+remapped+backfilling, 161 active+clean; 2.0 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 913 B/s rd, 0 B/s wr, 768 op/s; 7300/96567 objects misplaced (7.560%); 66 MiB/s, 1.06k objects/s recovering 2024-03-30T14:04:11.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:10 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3327161585' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:04:11.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:10 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:04:11.524 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:11.548 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:11.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:11.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:11.650 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:11.674 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:11.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:11.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:11.751 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T14:04:11.751 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:04:11.777 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:11.800 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:11.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:11.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:11.902 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:11.928 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:11.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:11.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:12.030 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:12.060 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:12.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:12.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:12.162 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:12.187 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:12.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:12.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:12.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:11 smithi113 ceph-mon[34341]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:04:12.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:11 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:04:12.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:11 smithi113 ceph-mon[34341]: osdmap e449: 8 total, 8 up, 5 in 2024-03-30T14:04:12.290 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:12.316 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:12.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:12.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:12.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:11 smithi043 ceph-mon[29963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:04:12.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:11 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:04:12.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:11 smithi043 ceph-mon[29963]: osdmap e449: 8 total, 8 up, 5 in 2024-03-30T14:04:12.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:11 smithi043 ceph-mon[42099]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:04:12.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:11 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:04:12.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:11 smithi043 ceph-mon[42099]: osdmap e449: 8 total, 8 up, 5 in 2024-03-30T14:04:12.417 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:12.441 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:12.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:12.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:12.543 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:12.567 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:12.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:12.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:12.669 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:12.695 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:12.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:12.796 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:12.820 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:12.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:12.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:12.922 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:12.946 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:12.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:12.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:13.048 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:13.086 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:13.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:13.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:13.188 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:13.223 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:13.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:13.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:13.325 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:13.362 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:13.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:13.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:13.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:13 smithi043 ceph-mon[29963]: pgmap v993: 168 pgs: 3 peering, 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 161 active+clean; 1.9 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 980 op/s; 3393/91836 objects misplaced (3.695%); 98 MiB/s, 1.56k objects/s recovering 2024-03-30T14:04:13.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:13 smithi043 ceph-mon[29963]: osdmap e450: 8 total, 8 up, 5 in 2024-03-30T14:04:13.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:13 smithi043 ceph-mon[29963]: osdmap e451: 8 total, 8 up, 5 in 2024-03-30T14:04:13.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:13 smithi043 ceph-mon[42099]: pgmap v993: 168 pgs: 3 peering, 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 161 active+clean; 1.9 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 980 op/s; 3393/91836 objects misplaced (3.695%); 98 MiB/s, 1.56k objects/s recovering 2024-03-30T14:04:13.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:13 smithi043 ceph-mon[42099]: osdmap e450: 8 total, 8 up, 5 in 2024-03-30T14:04:13.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:13 smithi043 ceph-mon[42099]: osdmap e451: 8 total, 8 up, 5 in 2024-03-30T14:04:13.464 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:13.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:13 smithi113 ceph-mon[34341]: pgmap v993: 168 pgs: 3 peering, 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 161 active+clean; 1.9 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 980 op/s; 3393/91836 objects misplaced (3.695%); 98 MiB/s, 1.56k objects/s recovering 2024-03-30T14:04:13.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:13 smithi113 ceph-mon[34341]: osdmap e450: 8 total, 8 up, 5 in 2024-03-30T14:04:13.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:13 smithi113 ceph-mon[34341]: osdmap e451: 8 total, 8 up, 5 in 2024-03-30T14:04:13.503 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:13.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:13.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:13.605 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:13.640 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:13.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:13.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:13.742 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:13.781 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:13.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:13.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:13.883 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:13.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:13.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:13.909 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-03-30T14:04:14.008 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:14.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:14.044 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:14.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:14.145 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:14.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:14.187 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:14.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:14.288 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:14.298 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:04:14.325 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:14.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:14.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:14.427 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:14.472 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:14.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:14.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:14.574 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:14.611 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:14.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:14.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:14.713 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:14.751 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:14.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:14.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:14.852 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:14.889 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:14.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:14.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:14.991 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:15.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:15.041 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:15.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:15.142 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:15.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:15.180 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:15.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:15.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:15 smithi113 ceph-mon[34341]: pgmap v996: 168 pgs: 3 peering, 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 161 active+clean; 1.8 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 751 op/s; 3393/90429 objects misplaced (3.752%); 77 MiB/s, 1.23k objects/s recovering 2024-03-30T14:04:15.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:15 smithi113 ceph-mon[34341]: 6.1b scrub starts 2024-03-30T14:04:15.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:15 smithi113 ceph-mon[34341]: 6.1b scrub ok 2024-03-30T14:04:15.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:15 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:04:15.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:15 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:15.280 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:15.314 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:15.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:15.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:15.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:15 smithi043 ceph-mon[42099]: pgmap v996: 168 pgs: 3 peering, 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 161 active+clean; 1.8 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 751 op/s; 3393/90429 objects misplaced (3.752%); 77 MiB/s, 1.23k objects/s recovering 2024-03-30T14:04:15.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:15 smithi043 ceph-mon[42099]: 6.1b scrub starts 2024-03-30T14:04:15.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:15 smithi043 ceph-mon[42099]: 6.1b scrub ok 2024-03-30T14:04:15.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:15 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:04:15.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:15 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:15 smithi043 ceph-mon[29963]: pgmap v996: 168 pgs: 3 peering, 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 161 active+clean; 1.8 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 751 op/s; 3393/90429 objects misplaced (3.752%); 77 MiB/s, 1.23k objects/s recovering 2024-03-30T14:04:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:15 smithi043 ceph-mon[29963]: 6.1b scrub starts 2024-03-30T14:04:15.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:15 smithi043 ceph-mon[29963]: 6.1b scrub ok 2024-03-30T14:04:15.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:15 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:04:15.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:15 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:15.416 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:15.447 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:15.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:15.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:15.549 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:15.586 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:15.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:15.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:15.688 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:15.711 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:15.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:15.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:15.813 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:15.849 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:15.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:15.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:15.950 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:15.988 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:15.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:15.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:16.090 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:16.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:16.132 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:16.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:16.233 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:16.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:16.258 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:16.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:16.335 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:16 smithi113 ceph-mon[34341]: 1.c deep-scrub starts 2024-03-30T14:04:16.335 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:16 smithi113 ceph-mon[34341]: 1.c deep-scrub ok 2024-03-30T14:04:16.335 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:16 smithi113 ceph-mon[34341]: 1.10 scrub starts 2024-03-30T14:04:16.335 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:16 smithi113 ceph-mon[34341]: 1.10 scrub ok 2024-03-30T14:04:16.359 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:16.391 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:04:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:04:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:04:16.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:16 smithi043 ceph-mon[42099]: 1.c deep-scrub starts 2024-03-30T14:04:16.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:16 smithi043 ceph-mon[42099]: 1.c deep-scrub ok 2024-03-30T14:04:16.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:16 smithi043 ceph-mon[42099]: 1.10 scrub starts 2024-03-30T14:04:16.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:16 smithi043 ceph-mon[42099]: 1.10 scrub ok 2024-03-30T14:04:16.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:16 smithi043 ceph-mon[29963]: 1.c deep-scrub starts 2024-03-30T14:04:16.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:16 smithi043 ceph-mon[29963]: 1.c deep-scrub ok 2024-03-30T14:04:16.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:16 smithi043 ceph-mon[29963]: 1.10 scrub starts 2024-03-30T14:04:16.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:16 smithi043 ceph-mon[29963]: 1.10 scrub ok 2024-03-30T14:04:16.392 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:16.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:16.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:16.399 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T14:04:16.399 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:04:16.494 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:16.529 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:16.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:16.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:16.631 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:16.668 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:16.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:16.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:16.770 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:16.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:16.805 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:16.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:16.906 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:16.946 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:16.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:16.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:17.048 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:17.085 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:17.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:17.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:17.187 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:17.218 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:17.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:17.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:17.258 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-03-30T14:04:17.283 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:17 smithi113 ceph-mon[34341]: pgmap v997: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 3 peering, 1 active+clean+remapped, 1 active+remapped+backfilling, 161 active+clean; 1.7 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 687 op/s; 2938/84201 objects misplaced (3.489%); 38 MiB/s, 608 objects/s recovering 2024-03-30T14:04:17.283 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:17 smithi113 ceph-mon[34341]: 6.c scrub starts 2024-03-30T14:04:17.283 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:17 smithi113 ceph-mon[34341]: 6.c scrub ok 2024-03-30T14:04:17.283 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:17 smithi113 ceph-mon[34341]: 6.19 scrub starts 2024-03-30T14:04:17.283 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:17 smithi113 ceph-mon[34341]: 6.19 scrub ok 2024-03-30T14:04:17.320 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:17.343 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:17.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:17.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:17.378 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:17 smithi043 ceph-mon[29963]: pgmap v997: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 3 peering, 1 active+clean+remapped, 1 active+remapped+backfilling, 161 active+clean; 1.7 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 687 op/s; 2938/84201 objects misplaced (3.489%); 38 MiB/s, 608 objects/s recovering 2024-03-30T14:04:17.378 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:17 smithi043 ceph-mon[29963]: 6.c scrub starts 2024-03-30T14:04:17.378 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:17 smithi043 ceph-mon[29963]: 6.c scrub ok 2024-03-30T14:04:17.378 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:17 smithi043 ceph-mon[29963]: 6.19 scrub starts 2024-03-30T14:04:17.378 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:17 smithi043 ceph-mon[29963]: 6.19 scrub ok 2024-03-30T14:04:17.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:17 smithi043 ceph-mon[42099]: pgmap v997: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 3 peering, 1 active+clean+remapped, 1 active+remapped+backfilling, 161 active+clean; 1.7 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 687 op/s; 2938/84201 objects misplaced (3.489%); 38 MiB/s, 608 objects/s recovering 2024-03-30T14:04:17.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:17 smithi043 ceph-mon[42099]: 6.c scrub starts 2024-03-30T14:04:17.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:17 smithi043 ceph-mon[42099]: 6.c scrub ok 2024-03-30T14:04:17.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:17 smithi043 ceph-mon[42099]: 6.19 scrub starts 2024-03-30T14:04:17.379 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:17 smithi043 ceph-mon[42099]: 6.19 scrub ok 2024-03-30T14:04:17.445 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:17.468 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:17.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:17.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:17.529 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:04:17.569 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:17.594 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:17.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:17.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:17.696 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:17.721 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:17.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:17.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:17.822 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:17.847 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:17.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:17.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:17.948 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:17.975 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:17.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:17.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:18.077 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:18.109 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:18.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:18.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:18.210 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:18.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:18 smithi113 ceph-mon[34341]: 5.17 scrub starts 2024-03-30T14:04:18.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:18 smithi113 ceph-mon[34341]: 5.17 scrub ok 2024-03-30T14:04:18.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:18 smithi113 ceph-mon[34341]: osdmap e452: 8 total, 8 up, 5 in 2024-03-30T14:04:18.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:18 smithi113 ceph-mon[34341]: 5.6 scrub starts 2024-03-30T14:04:18.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:18 smithi113 ceph-mon[34341]: 5.6 scrub ok 2024-03-30T14:04:18.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:18 smithi113 ceph-mon[34341]: osdmap e453: 8 total, 8 up, 5 in 2024-03-30T14:04:18.247 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:18.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:18.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:18.349 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:18.383 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:18.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:18.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:18.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:18 smithi043 ceph-mon[29963]: 5.17 scrub starts 2024-03-30T14:04:18.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:18 smithi043 ceph-mon[29963]: 5.17 scrub ok 2024-03-30T14:04:18.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:18 smithi043 ceph-mon[29963]: osdmap e452: 8 total, 8 up, 5 in 2024-03-30T14:04:18.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:18 smithi043 ceph-mon[29963]: 5.6 scrub starts 2024-03-30T14:04:18.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:18 smithi043 ceph-mon[29963]: 5.6 scrub ok 2024-03-30T14:04:18.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:18 smithi043 ceph-mon[29963]: osdmap e453: 8 total, 8 up, 5 in 2024-03-30T14:04:18.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:18 smithi043 ceph-mon[42099]: 5.17 scrub starts 2024-03-30T14:04:18.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:18 smithi043 ceph-mon[42099]: 5.17 scrub ok 2024-03-30T14:04:18.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:18 smithi043 ceph-mon[42099]: osdmap e452: 8 total, 8 up, 5 in 2024-03-30T14:04:18.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:18 smithi043 ceph-mon[42099]: 5.6 scrub starts 2024-03-30T14:04:18.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:18 smithi043 ceph-mon[42099]: 5.6 scrub ok 2024-03-30T14:04:18.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:18 smithi043 ceph-mon[42099]: osdmap e453: 8 total, 8 up, 5 in 2024-03-30T14:04:18.484 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:18.520 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:18.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:18.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:18.622 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:18.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:18.659 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:18.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:18.759 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:18.794 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T14:04:18.797 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:18.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:18.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:18.899 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:18.942 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:18.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:18.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:19.043 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:19.061 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.4 primary-affinity to 0 (802) 2024-03-30T14:04:19.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:19.090 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:19.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:19.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:19 smithi043 ceph-mon[29963]: 4.17 scrub starts 2024-03-30T14:04:19.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:19 smithi043 ceph-mon[29963]: 4.17 scrub ok 2024-03-30T14:04:19.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:19 smithi043 ceph-mon[29963]: 4.13 scrub starts 2024-03-30T14:04:19.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:19 smithi043 ceph-mon[29963]: 4.13 scrub ok 2024-03-30T14:04:19.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:19 smithi043 ceph-mon[29963]: pgmap v999: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 165 active+clean; 1.6 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 795 op/s; 1734/77535 objects misplaced (2.236%); 14 MiB/s, 225 objects/s recovering 2024-03-30T14:04:19.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:19 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4111132733' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-03-30T14:04:19.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:19 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-03-30T14:04:19.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:19 smithi043 ceph-mon[42099]: 4.17 scrub starts 2024-03-30T14:04:19.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:19 smithi043 ceph-mon[42099]: 4.17 scrub ok 2024-03-30T14:04:19.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:19 smithi043 ceph-mon[42099]: 4.13 scrub starts 2024-03-30T14:04:19.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:19 smithi043 ceph-mon[42099]: 4.13 scrub ok 2024-03-30T14:04:19.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:19 smithi043 ceph-mon[42099]: pgmap v999: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 165 active+clean; 1.6 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 795 op/s; 1734/77535 objects misplaced (2.236%); 14 MiB/s, 225 objects/s recovering 2024-03-30T14:04:19.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:19 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/4111132733' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-03-30T14:04:19.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:19 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-03-30T14:04:19.191 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:19.211 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:19 smithi113 ceph-mon[34341]: 4.17 scrub starts 2024-03-30T14:04:19.211 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:19 smithi113 ceph-mon[34341]: 4.17 scrub ok 2024-03-30T14:04:19.211 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:19 smithi113 ceph-mon[34341]: 4.13 scrub starts 2024-03-30T14:04:19.211 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:19 smithi113 ceph-mon[34341]: 4.13 scrub ok 2024-03-30T14:04:19.211 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:19 smithi113 ceph-mon[34341]: pgmap v999: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 165 active+clean; 1.6 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 795 op/s; 1734/77535 objects misplaced (2.236%); 14 MiB/s, 225 objects/s recovering 2024-03-30T14:04:19.212 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:19 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/4111132733' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-03-30T14:04:19.212 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:19 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-03-30T14:04:19.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:19.215 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:19.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:19.316 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:19.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:19.342 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:19.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:19.401 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T14:04:19.442 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:19.468 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:19.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:19.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:19.569 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:19.594 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:19.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:19.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:19.673 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T14:04:19.673 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:04:19.695 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:19.720 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:19.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:19.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:19.821 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:19.847 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:19.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:19.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:19.884 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:04:19.897 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:04:19.949 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:19.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:04:19 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:04:19.973 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:19.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:19.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:20.075 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:20.103 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:20.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:20.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:20.205 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:20.231 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:20.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:20.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:20.332 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:20.355 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:20.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:20.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:20.371 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:20 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-03-30T14:04:20.371 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:20 smithi043 ceph-mon[29963]: osdmap e454: 8 total, 8 up, 5 in 2024-03-30T14:04:20.371 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:20 smithi043 ceph-mon[29963]: 1.9 scrub starts 2024-03-30T14:04:20.371 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:20 smithi043 ceph-mon[29963]: 1.9 scrub ok 2024-03-30T14:04:20.371 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:20 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-03-30T14:04:20.371 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:20 smithi043 ceph-mon[42099]: osdmap e454: 8 total, 8 up, 5 in 2024-03-30T14:04:20.371 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:20 smithi043 ceph-mon[42099]: 1.9 scrub starts 2024-03-30T14:04:20.371 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:20 smithi043 ceph-mon[42099]: 1.9 scrub ok 2024-03-30T14:04:20.435 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:20 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-03-30T14:04:20.435 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:20 smithi113 ceph-mon[34341]: osdmap e454: 8 total, 8 up, 5 in 2024-03-30T14:04:20.435 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:20 smithi113 ceph-mon[34341]: 1.9 scrub starts 2024-03-30T14:04:20.436 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:20 smithi113 ceph-mon[34341]: 1.9 scrub ok 2024-03-30T14:04:20.457 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:20.461 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-30T14:04:20.481 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:20.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:20.583 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:20.607 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:20.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:20.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:20.709 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:20.712 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:04:20.735 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:20.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:20.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:20.837 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:20.861 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:20.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:20.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:20.963 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:20.986 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:20.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:20.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:21.088 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:21.115 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:21.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:21.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:21.217 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:21.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:21.258 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:21.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:21.359 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:21 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:21 smithi043 ceph-mon[29963]: pgmap v1002: 168 pgs: 17 peering, 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 149 active+clean; 1.3 GiB data, 8.4 GiB used, 439 GiB / 447 GiB avail; 2.3 KiB/s rd, 18 KiB/s wr, 1.46k op/s; 1309/64218 objects misplaced (2.038%); 14 MiB/s, 224 objects/s recovering 2024-03-30T14:04:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:21 smithi043 ceph-mon[29963]: Health check failed: Reduced data availability: 7 pgs inactive, 12 pgs peering (PG_AVAILABILITY) 2024-03-30T14:04:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:21 smithi043 ceph-mon[29963]: osdmap e455: 8 total, 8 up, 5 in 2024-03-30T14:04:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:21 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:21 smithi043 ceph-mon[42099]: pgmap v1002: 168 pgs: 17 peering, 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 149 active+clean; 1.3 GiB data, 8.4 GiB used, 439 GiB / 447 GiB avail; 2.3 KiB/s rd, 18 KiB/s wr, 1.46k op/s; 1309/64218 objects misplaced (2.038%); 14 MiB/s, 224 objects/s recovering 2024-03-30T14:04:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:21 smithi043 ceph-mon[42099]: Health check failed: Reduced data availability: 7 pgs inactive, 12 pgs peering (PG_AVAILABILITY) 2024-03-30T14:04:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:21 smithi043 ceph-mon[42099]: osdmap e455: 8 total, 8 up, 5 in 2024-03-30T14:04:21.394 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:21.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:21.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:21.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:21 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:21.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:21 smithi113 ceph-mon[34341]: pgmap v1002: 168 pgs: 17 peering, 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 149 active+clean; 1.3 GiB data, 8.4 GiB used, 439 GiB / 447 GiB avail; 2.3 KiB/s rd, 18 KiB/s wr, 1.46k op/s; 1309/64218 objects misplaced (2.038%); 14 MiB/s, 224 objects/s recovering 2024-03-30T14:04:21.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:21 smithi113 ceph-mon[34341]: Health check failed: Reduced data availability: 7 pgs inactive, 12 pgs peering (PG_AVAILABILITY) 2024-03-30T14:04:21.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:21 smithi113 ceph-mon[34341]: osdmap e455: 8 total, 8 up, 5 in 2024-03-30T14:04:21.496 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:21.531 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:21.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:21.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:21.633 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:21.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:21.675 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:21.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:21.776 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:21.812 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:21.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:21.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:21.914 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:21.950 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:21.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:21.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:22.052 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:22.094 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:22.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:22.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:22.196 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:22.300 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:22.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:22.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:22.352 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:22 smithi113 ceph-mon[34341]: 1.d scrub starts 2024-03-30T14:04:22.352 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:22 smithi113 ceph-mon[34341]: 1.d scrub ok 2024-03-30T14:04:22.352 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:22 smithi113 ceph-mon[34341]: 1.5 deep-scrub starts 2024-03-30T14:04:22.352 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:22 smithi113 ceph-mon[34341]: 1.5 deep-scrub ok 2024-03-30T14:04:22.352 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:22 smithi113 ceph-mon[34341]: 11.d continuing backfill to osd.1 from (429'9670,453'12678] MIN to 453'12678 2024-03-30T14:04:22.352 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:22 smithi113 ceph-mon[34341]: osdmap e456: 8 total, 8 up, 5 in 2024-03-30T14:04:22.352 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 14:04:22 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=cleanup t=2024-03-30T14:04:22.124457663Z level=info msg="Completed cleanup jobs" duration=5.99315ms 2024-03-30T14:04:22.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:22 smithi043 ceph-mon[29963]: 1.d scrub starts 2024-03-30T14:04:22.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:22 smithi043 ceph-mon[29963]: 1.d scrub ok 2024-03-30T14:04:22.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:22 smithi043 ceph-mon[29963]: 1.5 deep-scrub starts 2024-03-30T14:04:22.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:22 smithi043 ceph-mon[29963]: 1.5 deep-scrub ok 2024-03-30T14:04:22.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:22 smithi043 ceph-mon[29963]: 11.d continuing backfill to osd.1 from (429'9670,453'12678] MIN to 453'12678 2024-03-30T14:04:22.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:22 smithi043 ceph-mon[29963]: osdmap e456: 8 total, 8 up, 5 in 2024-03-30T14:04:22.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:22 smithi043 ceph-mon[42099]: 1.d scrub starts 2024-03-30T14:04:22.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:22 smithi043 ceph-mon[42099]: 1.d scrub ok 2024-03-30T14:04:22.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:22 smithi043 ceph-mon[42099]: 1.5 deep-scrub starts 2024-03-30T14:04:22.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:22 smithi043 ceph-mon[42099]: 1.5 deep-scrub ok 2024-03-30T14:04:22.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:22 smithi043 ceph-mon[42099]: 11.d continuing backfill to osd.1 from (429'9670,453'12678] MIN to 453'12678 2024-03-30T14:04:22.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:22 smithi043 ceph-mon[42099]: osdmap e456: 8 total, 8 up, 5 in 2024-03-30T14:04:22.402 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:22.440 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:22.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:22.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:22.542 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:22.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:22.583 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:22.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:22.684 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:22.719 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:22.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:22.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:22.761 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T14:04:22.761 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:04:22.821 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:22.845 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:22.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:22.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:22.946 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:22.982 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:22.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:22.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:23.084 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:23.123 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:23.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:23.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:23.225 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:23.262 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:23.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:23.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:23.364 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:23 smithi043 ceph-mon[29963]: 6.b deep-scrub starts 2024-03-30T14:04:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:23 smithi043 ceph-mon[29963]: 6.b deep-scrub ok 2024-03-30T14:04:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:23 smithi043 ceph-mon[29963]: pgmap v1005: 168 pgs: 17 peering, 1 active+remapped+backfill_toofull, 150 active+clean; 1.2 GiB data, 8.0 GiB used, 439 GiB / 447 GiB avail; 3.2 KiB/s rd, 27 KiB/s wr, 1.53k op/s; 1309/59223 objects misplaced (2.210%) 2024-03-30T14:04:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:23 smithi043 ceph-mon[29963]: 4.16 scrub starts 2024-03-30T14:04:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:23 smithi043 ceph-mon[29963]: 4.16 scrub ok 2024-03-30T14:04:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:23 smithi043 ceph-mon[29963]: 3.1 scrub starts 2024-03-30T14:04:23.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:23 smithi043 ceph-mon[29963]: 3.1 scrub ok 2024-03-30T14:04:23.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:23 smithi043 ceph-mon[29963]: osdmap e457: 8 total, 8 up, 5 in 2024-03-30T14:04:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:23 smithi043 ceph-mon[42099]: 6.b deep-scrub starts 2024-03-30T14:04:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:23 smithi043 ceph-mon[42099]: 6.b deep-scrub ok 2024-03-30T14:04:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:23 smithi043 ceph-mon[42099]: pgmap v1005: 168 pgs: 17 peering, 1 active+remapped+backfill_toofull, 150 active+clean; 1.2 GiB data, 8.0 GiB used, 439 GiB / 447 GiB avail; 3.2 KiB/s rd, 27 KiB/s wr, 1.53k op/s; 1309/59223 objects misplaced (2.210%) 2024-03-30T14:04:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:23 smithi043 ceph-mon[42099]: 4.16 scrub starts 2024-03-30T14:04:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:23 smithi043 ceph-mon[42099]: 4.16 scrub ok 2024-03-30T14:04:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:23 smithi043 ceph-mon[42099]: 3.1 scrub starts 2024-03-30T14:04:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:23 smithi043 ceph-mon[42099]: 3.1 scrub ok 2024-03-30T14:04:23.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:23 smithi043 ceph-mon[42099]: osdmap e457: 8 total, 8 up, 5 in 2024-03-30T14:04:23.408 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:23.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:23.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:23.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:23 smithi113 ceph-mon[34341]: 6.b deep-scrub starts 2024-03-30T14:04:23.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:23 smithi113 ceph-mon[34341]: 6.b deep-scrub ok 2024-03-30T14:04:23.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:23 smithi113 ceph-mon[34341]: pgmap v1005: 168 pgs: 17 peering, 1 active+remapped+backfill_toofull, 150 active+clean; 1.2 GiB data, 8.0 GiB used, 439 GiB / 447 GiB avail; 3.2 KiB/s rd, 27 KiB/s wr, 1.53k op/s; 1309/59223 objects misplaced (2.210%) 2024-03-30T14:04:23.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:23 smithi113 ceph-mon[34341]: 4.16 scrub starts 2024-03-30T14:04:23.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:23 smithi113 ceph-mon[34341]: 4.16 scrub ok 2024-03-30T14:04:23.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:23 smithi113 ceph-mon[34341]: 3.1 scrub starts 2024-03-30T14:04:23.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:23 smithi113 ceph-mon[34341]: 3.1 scrub ok 2024-03-30T14:04:23.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:23 smithi113 ceph-mon[34341]: osdmap e457: 8 total, 8 up, 5 in 2024-03-30T14:04:23.511 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:23.558 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-03-30T14:04:23.577 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:23.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:23.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:23.680 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:23.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:23.717 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:23.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:23.818 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:23.856 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:23.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:23.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:23.958 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:23.959 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:04:24.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:24.001 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:24.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:24.102 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:24.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:24.141 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:24.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:24.242 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:24.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:24.281 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:24.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:24.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:24 smithi043 ceph-mon[29963]: 3.18 scrub starts 2024-03-30T14:04:24.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:24 smithi043 ceph-mon[29963]: 3.18 scrub ok 2024-03-30T14:04:24.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:24 smithi043 ceph-mon[29963]: 3.1c scrub starts 2024-03-30T14:04:24.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:24 smithi043 ceph-mon[29963]: 3.1c scrub ok 2024-03-30T14:04:24.381 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:24 smithi043 ceph-mon[42099]: 3.18 scrub starts 2024-03-30T14:04:24.381 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:24 smithi043 ceph-mon[42099]: 3.18 scrub ok 2024-03-30T14:04:24.381 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:24 smithi043 ceph-mon[42099]: 3.1c scrub starts 2024-03-30T14:04:24.381 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:24 smithi043 ceph-mon[42099]: 3.1c scrub ok 2024-03-30T14:04:24.382 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:24.413 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:24.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:24.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:24.427 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:24 smithi113 ceph-mon[34341]: 3.18 scrub starts 2024-03-30T14:04:24.427 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:24 smithi113 ceph-mon[34341]: 3.18 scrub ok 2024-03-30T14:04:24.428 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:24 smithi113 ceph-mon[34341]: 3.1c scrub starts 2024-03-30T14:04:24.428 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:24 smithi113 ceph-mon[34341]: 3.1c scrub ok 2024-03-30T14:04:24.515 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:24.552 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:24.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:24.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:24.654 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:24.678 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:24.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:24.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:24.780 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:24.794 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6, 5] out_osds: [7, 3, 2] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-03-30T14:04:24.794 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T14:04:24.794 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-30T14:04:24.794 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 1.000000 2024-03-30T14:04:24.794 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 4 1 2024-03-30T14:04:24.815 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:24.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:24.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:24.917 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:24.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:24.961 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:24.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:25.062 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:25.101 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T14:04:25.109 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:25.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:25.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:25.116 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-03-30T14:04:25.156 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T14:04:25.195 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T14:04:25.212 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:25.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:25 smithi113 ceph-mon[34341]: 5.13 scrub starts 2024-03-30T14:04:25.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:25 smithi113 ceph-mon[34341]: 5.13 scrub ok 2024-03-30T14:04:25.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:25 smithi113 ceph-mon[34341]: pgmap v1007: 168 pgs: 17 peering, 1 active+remapped+backfill_toofull, 150 active+clean; 1.2 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 2.5 KiB/s rd, 23 KiB/s wr, 1.07k op/s; 1309/57135 objects misplaced (2.291%) 2024-03-30T14:04:25.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:25 smithi113 ceph-mon[34341]: 6.13 scrub starts 2024-03-30T14:04:25.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:25 smithi113 ceph-mon[34341]: 6.13 scrub ok 2024-03-30T14:04:25.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:25 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/872238364' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:04:25.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:25 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:04:25.215 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:25 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1192920222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:04:25.239 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T14:04:25.254 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:25.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:25.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:25.283 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T14:04:25.321 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T14:04:25.355 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:25.368 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T14:04:25.369 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:25 smithi043 ceph-mon[29963]: 5.13 scrub starts 2024-03-30T14:04:25.369 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:25 smithi043 ceph-mon[29963]: 5.13 scrub ok 2024-03-30T14:04:25.369 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:25 smithi043 ceph-mon[29963]: pgmap v1007: 168 pgs: 17 peering, 1 active+remapped+backfill_toofull, 150 active+clean; 1.2 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 2.5 KiB/s rd, 23 KiB/s wr, 1.07k op/s; 1309/57135 objects misplaced (2.291%) 2024-03-30T14:04:25.369 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:25 smithi043 ceph-mon[29963]: 6.13 scrub starts 2024-03-30T14:04:25.369 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:25 smithi043 ceph-mon[29963]: 6.13 scrub ok 2024-03-30T14:04:25.369 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:25 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/872238364' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:04:25.369 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:25 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:04:25.369 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:25 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1192920222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:04:25.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:25 smithi043 ceph-mon[42099]: 5.13 scrub starts 2024-03-30T14:04:25.370 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:25 smithi043 ceph-mon[42099]: 5.13 scrub ok 2024-03-30T14:04:25.370 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:25 smithi043 ceph-mon[42099]: pgmap v1007: 168 pgs: 17 peering, 1 active+remapped+backfill_toofull, 150 active+clean; 1.2 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 2.5 KiB/s rd, 23 KiB/s wr, 1.07k op/s; 1309/57135 objects misplaced (2.291%) 2024-03-30T14:04:25.370 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:25 smithi043 ceph-mon[42099]: 6.13 scrub starts 2024-03-30T14:04:25.370 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:25 smithi043 ceph-mon[42099]: 6.13 scrub ok 2024-03-30T14:04:25.370 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:25 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/872238364' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:04:25.370 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:25 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:04:25.370 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:25 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1192920222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:04:25.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:25.411 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:25.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:25.414 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T14:04:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:04:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:04:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:04:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:04:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:04:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:04:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:04:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:04:25.512 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:25.549 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:25.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:25.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:25.651 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:25.687 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:25.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:25.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:25.790 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:25.827 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:25.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:25.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:25.929 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:25.977 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:25.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:25.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:26.068 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T14:04:26.069 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [], 2024-03-30T14:04:26.069 INFO:teuthology.orchestra.run.smithi113.stdout: "complaint_time": 30, 2024-03-30T14:04:26.069 INFO:teuthology.orchestra.run.smithi113.stdout: "num_blocked_ops": 0 2024-03-30T14:04:26.069 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T14:04:26.079 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:26.115 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:26.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:26.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:26.199 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:04:26.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:26 smithi113 ceph-mon[34341]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:04:26.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:26 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:04:26.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:26 smithi113 ceph-mon[34341]: osdmap e458: 8 total, 8 up, 5 in 2024-03-30T14:04:26.218 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:26.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:26.253 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:26.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:26.337 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:26 smithi043 ceph-mon[29963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:04:26.337 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:26 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:04:26.337 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:26 smithi043 ceph-mon[29963]: osdmap e458: 8 total, 8 up, 5 in 2024-03-30T14:04:26.338 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:04:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:04:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:04:26.338 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:26 smithi043 ceph-mon[42099]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:04:26.338 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:26 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:04:26.338 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:26 smithi043 ceph-mon[42099]: osdmap e458: 8 total, 8 up, 5 in 2024-03-30T14:04:26.355 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:26.390 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:26.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:26.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:26.491 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:26.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:26.522 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:26.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:26.624 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:26.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:26.658 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:26.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:26.759 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:26.794 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:26.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:26.896 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:26.930 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:26.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:26.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:27.032 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:27.072 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:27.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:27.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:27.174 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:27.214 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:27.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:27.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:27.283 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-03-30T14:04:27.305 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:27 smithi043 ceph-mon[29963]: pgmap v1009: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.0 GiB data, 7.5 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 823 op/s; 909/49947 objects misplaced (1.820%) 2024-03-30T14:04:27.305 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:27 smithi043 ceph-mon[29963]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 12 pgs peering) 2024-03-30T14:04:27.305 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:27 smithi043 ceph-mon[42099]: pgmap v1009: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.0 GiB data, 7.5 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 823 op/s; 909/49947 objects misplaced (1.820%) 2024-03-30T14:04:27.305 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:27 smithi043 ceph-mon[42099]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 12 pgs peering) 2024-03-30T14:04:27.316 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:27.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:27.376 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:27.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:27.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:27 smithi113 ceph-mon[34341]: pgmap v1009: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.0 GiB data, 7.5 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 823 op/s; 909/49947 objects misplaced (1.820%) 2024-03-30T14:04:27.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:27 smithi113 ceph-mon[34341]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 12 pgs peering) 2024-03-30T14:04:27.478 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:27.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:27.514 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:27.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:27.615 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:27.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:27.651 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:27.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:27.688 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:04:27.752 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:27.793 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:27.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:27.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:27.894 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:27.926 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:27.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:27.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:27.970 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T14:04:28.028 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:28.066 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:28.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:28.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:28.169 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:28.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:28.214 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:28.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:28.315 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:28.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:28.347 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:28.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:28.448 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:28.482 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:28.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:28.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:28.584 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:28.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:28.620 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:28.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:28.720 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:28.763 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:28.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:28.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:28.779 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T14:04:28.866 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:28.900 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:28.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:28.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:29.002 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:29.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:29.044 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:29.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:29.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:28 smithi043 ceph-mon[29963]: pgmap v1010: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 801 MiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.24k op/s; 909/39033 objects misplaced (2.329%) 2024-03-30T14:04:29.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:28 smithi043 ceph-mon[29963]: osdmap e459: 8 total, 8 up, 5 in 2024-03-30T14:04:29.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.d", "id": [4, 0]}]: dispatch 2024-03-30T14:04:29.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [4, 6]}]: dispatch 2024-03-30T14:04:29.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:28 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [1, 6]}]: dispatch 2024-03-30T14:04:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:28 smithi043 ceph-mon[42099]: pgmap v1010: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 801 MiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.24k op/s; 909/39033 objects misplaced (2.329%) 2024-03-30T14:04:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:28 smithi043 ceph-mon[42099]: osdmap e459: 8 total, 8 up, 5 in 2024-03-30T14:04:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.d", "id": [4, 0]}]: dispatch 2024-03-30T14:04:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [4, 6]}]: dispatch 2024-03-30T14:04:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:28 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [1, 6]}]: dispatch 2024-03-30T14:04:29.145 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:29.178 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:29.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:29.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:29.280 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:29.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:29.323 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:29.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:29.398 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:28 smithi113 ceph-mon[34341]: pgmap v1010: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 801 MiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.24k op/s; 909/39033 objects misplaced (2.329%) 2024-03-30T14:04:29.398 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:28 smithi113 ceph-mon[34341]: osdmap e459: 8 total, 8 up, 5 in 2024-03-30T14:04:29.398 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.d", "id": [4, 0]}]: dispatch 2024-03-30T14:04:29.398 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [4, 6]}]: dispatch 2024-03-30T14:04:29.398 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:28 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [1, 6]}]: dispatch 2024-03-30T14:04:29.424 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:29.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:29.461 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:29.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:29.562 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:29.600 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:29.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:29.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:29.702 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:29.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:04:29 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:04:29.736 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:29.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:29.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:29.838 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:29.875 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:29.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:29.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:29.976 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:30.011 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:30.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:30.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:30.113 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:30.114 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:04:30.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:30.154 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:30.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:30.189 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:04:30.222 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T14:04:30.222 INFO:teuthology.orchestra.run.smithi113.stdout: "size": 20, 2024-03-30T14:04:30.222 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 600, 2024-03-30T14:04:30.222 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [ 2024-03-30T14:04:30.222 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.222 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:262248 10.6 10:69141517:::benchmark_data_smithi043_169264_object100021:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e351)", 2024-03-30T14:04:30.222 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:58.662488+0000", 2024-03-30T14:04:30.222 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 271.56214335700002, 2024-03-30T14:04:30.222 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.015917921, 2024-03-30T14:04:30.222 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:30.222 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:30.223 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "reached pg", 2024-03-30T14:04:30.223 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:30.223 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:04:30.223 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:04:30.223 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 262248 2024-03-30T14:04:30.223 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.223 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:30.223 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.223 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:30.223 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662488+0000", 2024-03-30T14:04:30.223 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.223 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.223 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.223 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:30.223 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662488+0000", 2024-03-30T14:04:30.223 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.224 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.224 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.224 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:30.224 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662489+0000", 2024-03-30T14:04:30.224 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.5200000000000002e-07 2024-03-30T14:04:30.224 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.224 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.224 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:30.224 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662491+0000", 2024-03-30T14:04:30.224 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.4310000000000001e-06 2024-03-30T14:04:30.224 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.224 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.224 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:30.224 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662498+0000", 2024-03-30T14:04:30.225 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3899999999999998e-06 2024-03-30T14:04:30.225 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.225 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.225 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:30.225 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662503+0000", 2024-03-30T14:04:30.225 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6829999999999996e-06 2024-03-30T14:04:30.225 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.225 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.225 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.225 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662552+0000", 2024-03-30T14:04:30.225 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8479000000000001e-05 2024-03-30T14:04:30.225 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.225 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.225 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op must wait for map", 2024-03-30T14:04:30.225 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662558+0000", 2024-03-30T14:04:30.226 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.2349999999999996e-06 2024-03-30T14:04:30.226 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.226 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.226 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.226 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.664180+0000", 2024-03-30T14:04:30.226 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0016223240000000001 2024-03-30T14:04:30.226 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.226 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.226 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.226 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.664189+0000", 2024-03-30T14:04:30.226 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.507e-06 2024-03-30T14:04:30.226 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.226 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.227 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.227 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.678317+0000", 2024-03-30T14:04:30.227 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.014127759 2024-03-30T14:04:30.227 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.227 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.227 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:30.227 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.678406+0000", 2024-03-30T14:04:30.227 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.9660999999999994e-05 2024-03-30T14:04:30.227 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.227 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.227 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.227 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.227 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.227 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:262255 10.6 10:6bbf16dc:::benchmark_data_smithi043_169264_object100028:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e351)", 2024-03-30T14:04:30.228 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:58.662522+0000", 2024-03-30T14:04:30.228 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 271.56210932300002, 2024-03-30T14:04:30.228 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0162618560000001, 2024-03-30T14:04:30.228 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:30.228 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:30.228 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "reached pg", 2024-03-30T14:04:30.228 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:30.228 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:04:30.228 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:04:30.228 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 262255 2024-03-30T14:04:30.228 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.228 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:30.228 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.229 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:30.229 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662522+0000", 2024-03-30T14:04:30.229 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.229 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.229 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.229 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:30.229 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662522+0000", 2024-03-30T14:04:30.229 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.229 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.229 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.229 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:30.229 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662523+0000", 2024-03-30T14:04:30.229 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.2199999999999999e-07 2024-03-30T14:04:30.229 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.229 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.229 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:30.230 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662525+0000", 2024-03-30T14:04:30.230 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.4480000000000001e-06 2024-03-30T14:04:30.230 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.230 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.230 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:30.230 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662532+0000", 2024-03-30T14:04:30.230 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.832e-06 2024-03-30T14:04:30.230 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.230 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.230 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:30.230 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662539+0000", 2024-03-30T14:04:30.230 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3799999999999999e-06 2024-03-30T14:04:30.230 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.230 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.231 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.231 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662640+0000", 2024-03-30T14:04:30.231 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000101273 2024-03-30T14:04:30.231 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.231 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.231 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting_for_map not empty", 2024-03-30T14:04:30.231 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662646+0000", 2024-03-30T14:04:30.231 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6490000000000002e-06 2024-03-30T14:04:30.231 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.231 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.231 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.231 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.664555+0000", 2024-03-30T14:04:30.231 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001908477 2024-03-30T14:04:30.231 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.231 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.231 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.232 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.664562+0000", 2024-03-30T14:04:30.232 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.5129999999999999e-06 2024-03-30T14:04:30.232 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.232 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.232 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.232 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.678721+0000", 2024-03-30T14:04:30.232 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.01415827 2024-03-30T14:04:30.232 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.232 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.232 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:30.232 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.678784+0000", 2024-03-30T14:04:30.232 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3591999999999997e-05 2024-03-30T14:04:30.232 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.232 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.232 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.233 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.233 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.233 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:262259 10.6 10:63ee870a:::benchmark_data_smithi043_169264_object100032:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e351)", 2024-03-30T14:04:30.233 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:58.662546+0000", 2024-03-30T14:04:30.233 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 271.56208570699999, 2024-03-30T14:04:30.233 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0232300519999999, 2024-03-30T14:04:30.233 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:30.233 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:30.233 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:30.233 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:30.233 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:04:30.233 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:04:30.233 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 262259 2024-03-30T14:04:30.233 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.233 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:30.233 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.234 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:30.234 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662546+0000", 2024-03-30T14:04:30.234 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.234 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.234 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.234 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:30.234 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662546+0000", 2024-03-30T14:04:30.234 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.234 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.234 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.234 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:30.234 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662546+0000", 2024-03-30T14:04:30.234 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.9099999999999999e-07 2024-03-30T14:04:30.234 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.234 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.235 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:30.235 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662549+0000", 2024-03-30T14:04:30.235 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.1509999999999998e-06 2024-03-30T14:04:30.235 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.235 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.236 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:30.236 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662555+0000", 2024-03-30T14:04:30.236 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1600000000000003e-06 2024-03-30T14:04:30.236 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.236 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.236 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:30.236 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662560+0000", 2024-03-30T14:04:30.236 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2429999999999996e-06 2024-03-30T14:04:30.236 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.236 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.236 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.236 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662684+0000", 2024-03-30T14:04:30.236 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012439499999999999 2024-03-30T14:04:30.236 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.236 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.237 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting_for_map not empty", 2024-03-30T14:04:30.237 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662690+0000", 2024-03-30T14:04:30.237 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5430000000000001e-06 2024-03-30T14:04:30.237 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.237 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.237 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.237 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.664588+0000", 2024-03-30T14:04:30.237 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0018977810000000001 2024-03-30T14:04:30.237 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.237 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.237 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.237 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.664595+0000", 2024-03-30T14:04:30.237 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8179999999999998e-06 2024-03-30T14:04:30.237 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.237 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.238 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.238 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.678819+0000", 2024-03-30T14:04:30.238 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0142241409999999 2024-03-30T14:04:30.238 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.238 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.238 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:30.238 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.678919+0000", 2024-03-30T14:04:30.238 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00010043999999999999 2024-03-30T14:04:30.238 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.238 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.238 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,4", 2024-03-30T14:04:30.238 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.679004+0000", 2024-03-30T14:04:30.238 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.4767999999999999e-05 2024-03-30T14:04:30.239 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.239 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.239 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.239 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.685455+0000", 2024-03-30T14:04:30.239 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0064515140000000002 2024-03-30T14:04:30.239 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.239 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.239 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:30.239 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.685589+0000", 2024-03-30T14:04:30.239 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013370299999999999 2024-03-30T14:04:30.239 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.239 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.239 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.239 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.685647+0000", 2024-03-30T14:04:30.240 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.77e-05 2024-03-30T14:04:30.240 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.240 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.240 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:30.240 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.685729+0000", 2024-03-30T14:04:30.240 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.2508000000000004e-05 2024-03-30T14:04:30.240 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.240 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.240 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:30.240 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.685776+0000", 2024-03-30T14:04:30.240 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6795999999999997e-05 2024-03-30T14:04:30.240 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.240 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.240 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.240 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.241 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.241 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:262274 10.6 10:6561d21c:::benchmark_data_smithi043_169264_object100047:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e351)", 2024-03-30T14:04:30.241 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:58.662566+0000", 2024-03-30T14:04:30.241 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 271.562065717, 2024-03-30T14:04:30.241 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0234275960000001, 2024-03-30T14:04:30.241 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:30.241 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:30.241 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:30.241 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:30.241 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:04:30.241 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:04:30.241 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 262274 2024-03-30T14:04:30.241 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.241 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:30.242 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.242 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:30.242 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662566+0000", 2024-03-30T14:04:30.242 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.242 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.242 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.242 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:30.242 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662566+0000", 2024-03-30T14:04:30.242 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.242 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.242 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.242 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:30.242 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662566+0000", 2024-03-30T14:04:30.243 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.0499999999999999e-07 2024-03-30T14:04:30.243 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.243 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.243 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:30.243 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662568+0000", 2024-03-30T14:04:30.243 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.838e-06 2024-03-30T14:04:30.243 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.243 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.243 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:30.243 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662575+0000", 2024-03-30T14:04:30.243 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1420000000000004e-06 2024-03-30T14:04:30.243 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.243 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.243 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:30.243 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662580+0000", 2024-03-30T14:04:30.244 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8790000000000002e-06 2024-03-30T14:04:30.244 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.244 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.244 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.244 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662732+0000", 2024-03-30T14:04:30.244 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00015133400000000001 2024-03-30T14:04:30.244 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.244 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.244 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting_for_map not empty", 2024-03-30T14:04:30.244 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662737+0000", 2024-03-30T14:04:30.244 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6370000000000004e-06 2024-03-30T14:04:30.244 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.244 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.244 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.244 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.664621+0000", 2024-03-30T14:04:30.245 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0018837179999999999 2024-03-30T14:04:30.245 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.245 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.245 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.245 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.664627+0000", 2024-03-30T14:04:30.245 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0809999999999999e-06 2024-03-30T14:04:30.245 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.245 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.245 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.245 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.680967+0000", 2024-03-30T14:04:30.245 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0163402560000001 2024-03-30T14:04:30.245 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.245 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.245 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:30.246 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.681077+0000", 2024-03-30T14:04:30.246 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000109879 2024-03-30T14:04:30.246 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.246 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.246 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,4", 2024-03-30T14:04:30.246 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.681204+0000", 2024-03-30T14:04:30.246 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012643600000000001 2024-03-30T14:04:30.246 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.246 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.246 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.246 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.685501+0000", 2024-03-30T14:04:30.246 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0042971290000000002 2024-03-30T14:04:30.246 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.247 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.247 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:30.247 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.685595+0000", 2024-03-30T14:04:30.247 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4049999999999996e-05 2024-03-30T14:04:30.247 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.247 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.247 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.247 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.685902+0000", 2024-03-30T14:04:30.247 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00030740800000000001 2024-03-30T14:04:30.247 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.247 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.247 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:30.247 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.685954+0000", 2024-03-30T14:04:30.247 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2193e-05 2024-03-30T14:04:30.247 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.248 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.248 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:30.248 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.685994+0000", 2024-03-30T14:04:30.248 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.9310999999999997e-05 2024-03-30T14:04:30.248 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.248 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.248 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.248 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.248 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.248 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:262297 10.6 10:6424c1fc:::benchmark_data_smithi043_169264_object100070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-30T14:04:30.248 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:58.662679+0000", 2024-03-30T14:04:30.248 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 271.56195268099998, 2024-03-30T14:04:30.248 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0257888340000001, 2024-03-30T14:04:30.248 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:30.248 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:30.249 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:30.249 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:30.249 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:04:30.249 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:04:30.249 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 262297 2024-03-30T14:04:30.249 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.249 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:30.249 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.249 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:30.249 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662679+0000", 2024-03-30T14:04:30.249 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.249 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.249 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.249 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:30.249 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662679+0000", 2024-03-30T14:04:30.250 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.250 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.250 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.250 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:30.250 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662679+0000", 2024-03-30T14:04:30.250 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.9700000000000003e-07 2024-03-30T14:04:30.250 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.250 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.250 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:30.250 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662681+0000", 2024-03-30T14:04:30.250 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.8330000000000001e-06 2024-03-30T14:04:30.250 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.250 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.250 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:30.250 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662686+0000", 2024-03-30T14:04:30.251 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2259999999999996e-06 2024-03-30T14:04:30.251 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.251 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.251 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:30.251 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662691+0000", 2024-03-30T14:04:30.251 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.357e-06 2024-03-30T14:04:30.251 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.251 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.251 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.251 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662782+0000", 2024-03-30T14:04:30.251 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.0766000000000005e-05 2024-03-30T14:04:30.251 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.251 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.251 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting_for_map not empty", 2024-03-30T14:04:30.251 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.662787+0000", 2024-03-30T14:04:30.251 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7250000000000002e-06 2024-03-30T14:04:30.252 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.252 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.252 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.252 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.664651+0000", 2024-03-30T14:04:30.252 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001863979 2024-03-30T14:04:30.252 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.252 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.252 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.252 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.664657+0000", 2024-03-30T14:04:30.252 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1539999999999999e-06 2024-03-30T14:04:30.252 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.252 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.252 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.252 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.683158+0000", 2024-03-30T14:04:30.252 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0185001309999999 2024-03-30T14:04:30.253 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.253 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.253 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:30.253 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.683269+0000", 2024-03-30T14:04:30.253 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000111452 2024-03-30T14:04:30.253 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.253 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.253 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,4", 2024-03-30T14:04:30.253 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.683394+0000", 2024-03-30T14:04:30.253 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012504399999999999 2024-03-30T14:04:30.253 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.253 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.253 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.253 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.685836+0000", 2024-03-30T14:04:30.253 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0024414620000000001 2024-03-30T14:04:30.254 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.254 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.254 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:30.254 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.685849+0000", 2024-03-30T14:04:30.254 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3621000000000001e-05 2024-03-30T14:04:30.254 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.254 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.254 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.254 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.688362+0000", 2024-03-30T14:04:30.254 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0025127980000000001 2024-03-30T14:04:30.254 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.254 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.254 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:30.254 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.688423+0000", 2024-03-30T14:04:30.254 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1512000000000003e-05 2024-03-30T14:04:30.255 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.255 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.255 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:30.255 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.688468+0000", 2024-03-30T14:04:30.255 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.4477000000000001e-05 2024-03-30T14:04:30.255 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.255 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.255 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.255 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.255 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.255 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:262307 10.6 10:6cd40d48:::benchmark_data_smithi043_169264_object100080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-30T14:04:30.255 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:58.671201+0000", 2024-03-30T14:04:30.255 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 271.55343087, 2024-03-30T14:04:30.255 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0273101579999999, 2024-03-30T14:04:30.256 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:30.256 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:30.256 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:30.256 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:30.256 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:04:30.256 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:04:30.256 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 262307 2024-03-30T14:04:30.256 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.256 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:30.256 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.256 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:30.256 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.671201+0000", 2024-03-30T14:04:30.256 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.256 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.256 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.256 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:30.257 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.671201+0000", 2024-03-30T14:04:30.257 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.257 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.257 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.257 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:30.257 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.671202+0000", 2024-03-30T14:04:30.257 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3510000000000001e-06 2024-03-30T14:04:30.257 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.257 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.257 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:30.257 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.671207+0000", 2024-03-30T14:04:30.257 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8609999999999997e-06 2024-03-30T14:04:30.257 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.257 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.257 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:30.258 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.671216+0000", 2024-03-30T14:04:30.258 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.8720000000000008e-06 2024-03-30T14:04:30.258 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.258 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.258 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:30.258 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.671229+0000", 2024-03-30T14:04:30.258 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.276e-05 2024-03-30T14:04:30.258 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.258 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.258 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.258 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.671279+0000", 2024-03-30T14:04:30.258 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0244999999999998e-05 2024-03-30T14:04:30.258 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.258 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.258 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.259 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.671288+0000", 2024-03-30T14:04:30.259 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.8000000000000004e-06 2024-03-30T14:04:30.259 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.259 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.259 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.259 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.686041+0000", 2024-03-30T14:04:30.259 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.014752911 2024-03-30T14:04:30.259 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.259 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.259 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:30.259 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.686158+0000", 2024-03-30T14:04:30.259 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000117087 2024-03-30T14:04:30.259 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.259 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.259 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,4", 2024-03-30T14:04:30.260 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.686268+0000", 2024-03-30T14:04:30.260 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00010987799999999999 2024-03-30T14:04:30.260 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.260 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.260 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.260 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.695310+0000", 2024-03-30T14:04:30.260 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0090422390000000005 2024-03-30T14:04:30.260 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.260 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.260 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:30.260 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698387+0000", 2024-03-30T14:04:30.260 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0030769159999999999 2024-03-30T14:04:30.260 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.260 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.260 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.261 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698432+0000", 2024-03-30T14:04:30.261 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.5184e-05 2024-03-30T14:04:30.261 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.261 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.261 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:30.261 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698464+0000", 2024-03-30T14:04:30.261 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2116000000000002e-05 2024-03-30T14:04:30.261 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.261 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.261 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:30.261 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698511+0000", 2024-03-30T14:04:30.261 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6937999999999997e-05 2024-03-30T14:04:30.261 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.261 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.261 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.261 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.262 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.262 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:262315 10.6 10:6e43c2d3:::benchmark_data_smithi043_169264_object100088:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-30T14:04:30.262 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:58.675200+0000", 2024-03-30T14:04:30.262 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 271.54943224499999, 2024-03-30T14:04:30.262 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.023458271, 2024-03-30T14:04:30.262 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:30.262 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:30.262 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:30.262 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:30.262 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:04:30.262 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:04:30.262 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 262315 2024-03-30T14:04:30.262 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.262 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:30.263 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.263 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:30.263 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.675200+0000", 2024-03-30T14:04:30.263 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.263 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.263 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.263 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:30.263 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.675200+0000", 2024-03-30T14:04:30.263 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.263 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.263 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.263 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:30.263 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.675200+0000", 2024-03-30T14:04:30.263 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.2600000000000001e-07 2024-03-30T14:04:30.263 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.264 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.264 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:30.264 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.675206+0000", 2024-03-30T14:04:30.264 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.57e-06 2024-03-30T14:04:30.264 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.264 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.264 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:30.264 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.675215+0000", 2024-03-30T14:04:30.265 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.8650000000000007e-06 2024-03-30T14:04:30.265 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.265 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.265 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:30.265 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.675229+0000", 2024-03-30T14:04:30.265 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4212e-05 2024-03-30T14:04:30.265 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.265 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.265 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.265 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.675307+0000", 2024-03-30T14:04:30.265 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8391000000000002e-05 2024-03-30T14:04:30.265 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.265 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.265 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.265 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.675315+0000", 2024-03-30T14:04:30.266 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.9070000000000007e-06 2024-03-30T14:04:30.266 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.266 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.266 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.266 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.688501+0000", 2024-03-30T14:04:30.266 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.013185644 2024-03-30T14:04:30.266 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.266 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.266 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:30.266 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.688600+0000", 2024-03-30T14:04:30.266 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.9034000000000001e-05 2024-03-30T14:04:30.266 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.266 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.267 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,4", 2024-03-30T14:04:30.267 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.688704+0000", 2024-03-30T14:04:30.267 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000104406 2024-03-30T14:04:30.267 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.267 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.267 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.267 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.695345+0000", 2024-03-30T14:04:30.267 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0066404639999999996 2024-03-30T14:04:30.267 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.267 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.267 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:30.267 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698393+0000", 2024-03-30T14:04:30.267 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.003047784 2024-03-30T14:04:30.267 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.267 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.267 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.268 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698597+0000", 2024-03-30T14:04:30.268 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00020441799999999999 2024-03-30T14:04:30.268 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.268 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.268 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:30.268 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698626+0000", 2024-03-30T14:04:30.268 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.8506000000000001e-05 2024-03-30T14:04:30.268 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.268 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.268 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:30.268 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698658+0000", 2024-03-30T14:04:30.268 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2243999999999998e-05 2024-03-30T14:04:30.268 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.268 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.268 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.269 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.269 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.269 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:262350 10.6 10:669fcee6:::benchmark_data_smithi043_169264_object100123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-30T14:04:30.269 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:58.702039+0000", 2024-03-30T14:04:30.269 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 271.522592944, 2024-03-30T14:04:30.269 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.99671496699999995, 2024-03-30T14:04:30.269 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:30.269 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:30.269 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:30.269 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:30.269 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:04:30.269 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:04:30.269 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 262350 2024-03-30T14:04:30.269 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.270 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:30.270 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.270 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:30.270 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.702039+0000", 2024-03-30T14:04:30.270 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.270 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.270 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.270 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:30.270 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.702039+0000", 2024-03-30T14:04:30.270 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.270 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.270 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.270 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:30.270 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.702040+0000", 2024-03-30T14:04:30.270 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2270000000000001e-06 2024-03-30T14:04:30.271 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.271 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.271 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:30.271 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.702046+0000", 2024-03-30T14:04:30.271 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8100000000000003e-06 2024-03-30T14:04:30.271 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.271 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.271 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:30.271 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.702060+0000", 2024-03-30T14:04:30.271 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4564e-05 2024-03-30T14:04:30.271 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.271 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.271 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:30.271 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.702069+0000", 2024-03-30T14:04:30.271 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.3809999999999999e-06 2024-03-30T14:04:30.271 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.272 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.272 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.272 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.702175+0000", 2024-03-30T14:04:30.272 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00010631100000000001 2024-03-30T14:04:30.272 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.272 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.272 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.272 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.702184+0000", 2024-03-30T14:04:30.272 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.3149999999999998e-06 2024-03-30T14:04:30.272 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.272 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.272 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.272 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.690766+0000", 2024-03-30T14:04:30.272 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.98858190700000004 2024-03-30T14:04:30.272 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.273 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.273 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:30.273 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.690874+0000", 2024-03-30T14:04:30.273 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000107321 2024-03-30T14:04:30.273 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.273 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.273 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,4", 2024-03-30T14:04:30.273 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.690981+0000", 2024-03-30T14:04:30.273 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000107599 2024-03-30T14:04:30.273 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.273 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.273 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.273 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.695377+0000", 2024-03-30T14:04:30.273 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.004395604 2024-03-30T14:04:30.273 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.273 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.274 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:30.274 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698397+0000", 2024-03-30T14:04:30.274 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0030201519999999999 2024-03-30T14:04:30.274 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.274 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.274 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.274 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698689+0000", 2024-03-30T14:04:30.274 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00029230599999999999 2024-03-30T14:04:30.274 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.274 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.274 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:30.274 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698719+0000", 2024-03-30T14:04:30.275 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:30.275 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.9468999999999999e-05 2024-03-30T14:04:30.276 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.276 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.276 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:30.276 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698754+0000", 2024-03-30T14:04:30.276 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.5000999999999999e-05 2024-03-30T14:04:30.276 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.276 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.276 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.276 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.276 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.276 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:262353 10.6 10:6c918e99:::benchmark_data_smithi043_169264_object100126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-30T14:04:30.276 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:58.704171+0000", 2024-03-30T14:04:30.276 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 271.52046110200001, 2024-03-30T14:04:30.276 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.99468822400000001, 2024-03-30T14:04:30.277 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:30.277 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:30.277 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:30.277 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:30.277 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:04:30.277 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:04:30.277 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 262353 2024-03-30T14:04:30.277 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.277 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:30.277 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.277 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:30.277 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.704171+0000", 2024-03-30T14:04:30.277 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.277 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.277 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.278 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:30.278 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.704171+0000", 2024-03-30T14:04:30.278 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.278 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.278 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.278 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:30.278 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.704172+0000", 2024-03-30T14:04:30.278 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.6500000000000008e-07 2024-03-30T14:04:30.278 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.278 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.278 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:30.278 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.704177+0000", 2024-03-30T14:04:30.278 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9019999999999998e-06 2024-03-30T14:04:30.279 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.279 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.279 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:30.279 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.704186+0000", 2024-03-30T14:04:30.279 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.448e-06 2024-03-30T14:04:30.279 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.279 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.279 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:30.279 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.704193+0000", 2024-03-30T14:04:30.279 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.0910000000000004e-06 2024-03-30T14:04:30.279 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.279 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.279 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.279 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.704252+0000", 2024-03-30T14:04:30.280 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8519e-05 2024-03-30T14:04:30.280 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.280 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.280 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.280 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.704260+0000", 2024-03-30T14:04:30.280 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.7839999999999992e-06 2024-03-30T14:04:30.280 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.280 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.280 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.280 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.692725+0000", 2024-03-30T14:04:30.280 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.98846424600000005 2024-03-30T14:04:30.280 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.280 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.280 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:30.280 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.692793+0000", 2024-03-30T14:04:30.281 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8623999999999999e-05 2024-03-30T14:04:30.281 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.281 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.281 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,4", 2024-03-30T14:04:30.281 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.692854+0000", 2024-03-30T14:04:30.281 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1212999999999998e-05 2024-03-30T14:04:30.281 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.281 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.281 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.281 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698318+0000", 2024-03-30T14:04:30.281 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00546388 2024-03-30T14:04:30.281 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.281 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.282 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:30.282 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698402+0000", 2024-03-30T14:04:30.282 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.3524999999999997e-05 2024-03-30T14:04:30.282 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.282 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.282 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.282 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698799+0000", 2024-03-30T14:04:30.282 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000396759 2024-03-30T14:04:30.282 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.282 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.282 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:30.282 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698826+0000", 2024-03-30T14:04:30.282 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.7157000000000001e-05 2024-03-30T14:04:30.282 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.283 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.283 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:30.283 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698859+0000", 2024-03-30T14:04:30.283 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.3111000000000002e-05 2024-03-30T14:04:30.283 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.283 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.283 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.283 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.283 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.283 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:262369 10.6 10:6bf48805:::benchmark_data_smithi043_169264_object100142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-30T14:04:30.283 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:58.716055+0000", 2024-03-30T14:04:30.283 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 271.50857705599998, 2024-03-30T14:04:30.283 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.98290215599999997, 2024-03-30T14:04:30.283 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:30.284 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:30.284 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:30.284 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:30.284 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:04:30.284 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:04:30.284 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 262369 2024-03-30T14:04:30.284 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.284 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:30.284 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.284 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:30.284 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.716055+0000", 2024-03-30T14:04:30.284 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.284 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.284 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.284 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:30.285 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.716055+0000", 2024-03-30T14:04:30.285 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.285 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.285 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.285 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:30.285 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.716055+0000", 2024-03-30T14:04:30.285 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6000000000000003e-07 2024-03-30T14:04:30.285 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.285 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.285 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:30.285 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.716059+0000", 2024-03-30T14:04:30.285 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.7249999999999999e-06 2024-03-30T14:04:30.285 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.285 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.285 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:30.286 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.716068+0000", 2024-03-30T14:04:30.286 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.4319999999999999e-06 2024-03-30T14:04:30.286 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.286 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.286 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:30.286 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.716073+0000", 2024-03-30T14:04:30.286 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2050000000000001e-06 2024-03-30T14:04:30.286 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.286 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.286 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.286 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.716120+0000", 2024-03-30T14:04:30.286 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7281999999999999e-05 2024-03-30T14:04:30.286 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.286 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.287 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.287 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.716129+0000", 2024-03-30T14:04:30.287 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6999999999999997e-06 2024-03-30T14:04:30.287 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.287 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.287 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.287 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.693951+0000", 2024-03-30T14:04:30.287 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.97782257699999997 2024-03-30T14:04:30.287 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.287 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.287 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:30.287 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.694057+0000", 2024-03-30T14:04:30.287 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000105733 2024-03-30T14:04:30.287 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.287 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.288 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,4", 2024-03-30T14:04:30.288 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.694163+0000", 2024-03-30T14:04:30.288 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00010616999999999999 2024-03-30T14:04:30.288 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.288 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.288 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.288 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698371+0000", 2024-03-30T14:04:30.288 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0042079930000000001 2024-03-30T14:04:30.288 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.288 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.288 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:30.288 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698407+0000", 2024-03-30T14:04:30.288 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.5370000000000002e-05 2024-03-30T14:04:30.288 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.288 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.288 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.289 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698896+0000", 2024-03-30T14:04:30.289 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00048906700000000004 2024-03-30T14:04:30.289 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.289 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.289 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:30.289 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698922+0000", 2024-03-30T14:04:30.289 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.6797000000000001e-05 2024-03-30T14:04:30.289 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.289 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.289 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:30.289 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698957+0000", 2024-03-30T14:04:30.289 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.4445e-05 2024-03-30T14:04:30.289 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.289 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.290 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.290 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.290 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.290 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:262398 10.6 10:671fc759:::benchmark_data_smithi043_169264_object100171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-30T14:04:30.290 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:58.741437+0000", 2024-03-30T14:04:30.290 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 271.48319515399999, 2024-03-30T14:04:30.290 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.95764375899999998, 2024-03-30T14:04:30.290 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:30.290 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:30.290 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:30.290 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:30.290 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:04:30.290 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:04:30.291 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 262398 2024-03-30T14:04:30.291 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.291 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:30.291 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.291 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:30.291 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.741437+0000", 2024-03-30T14:04:30.291 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.291 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.291 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.291 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:30.291 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.741437+0000", 2024-03-30T14:04:30.291 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.291 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.292 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.292 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:30.292 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.741438+0000", 2024-03-30T14:04:30.292 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3510000000000001e-06 2024-03-30T14:04:30.292 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.292 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.292 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:30.292 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.741444+0000", 2024-03-30T14:04:30.292 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.387e-06 2024-03-30T14:04:30.292 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.292 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.292 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:30.292 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.741461+0000", 2024-03-30T14:04:30.292 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.6884999999999998e-05 2024-03-30T14:04:30.293 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.293 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.293 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:30.293 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.741471+0000", 2024-03-30T14:04:30.293 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.9750000000000002e-06 2024-03-30T14:04:30.293 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.293 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.293 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.293 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.741534+0000", 2024-03-30T14:04:30.293 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.2749000000000005e-05 2024-03-30T14:04:30.293 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.293 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.293 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.294 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.741545+0000", 2024-03-30T14:04:30.294 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0618999999999999e-05 2024-03-30T14:04:30.294 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.294 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.294 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.294 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.695402+0000", 2024-03-30T14:04:30.294 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.95385717999999997 2024-03-30T14:04:30.294 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.294 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.294 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:30.294 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.695464+0000", 2024-03-30T14:04:30.294 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.2657999999999995e-05 2024-03-30T14:04:30.294 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.295 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.295 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,4", 2024-03-30T14:04:30.295 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.695531+0000", 2024-03-30T14:04:30.295 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6495000000000006e-05 2024-03-30T14:04:30.295 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.295 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.295 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:30.295 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698410+0000", 2024-03-30T14:04:30.295 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0028790539999999998 2024-03-30T14:04:30.295 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.295 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.295 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.295 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698565+0000", 2024-03-30T14:04:30.295 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000155019 2024-03-30T14:04:30.296 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.296 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.296 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.296 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.699002+0000", 2024-03-30T14:04:30.296 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000437241 2024-03-30T14:04:30.296 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.296 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.296 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:30.296 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.699046+0000", 2024-03-30T14:04:30.296 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3993999999999999e-05 2024-03-30T14:04:30.296 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.296 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.296 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:30.296 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.699080+0000", 2024-03-30T14:04:30.297 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.4152000000000001e-05 2024-03-30T14:04:30.297 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.297 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.297 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.297 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.297 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.297 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:262442 10.6 10:6c3ec934:::benchmark_data_smithi043_169264_object100215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-30T14:04:30.297 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:58.783951+0000", 2024-03-30T14:04:30.297 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 271.44068114599997, 2024-03-30T14:04:30.297 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.92416518599999997, 2024-03-30T14:04:30.297 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:30.297 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:30.297 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:30.297 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:30.298 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:04:30.298 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:04:30.298 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 262442 2024-03-30T14:04:30.298 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.298 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:30.298 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.298 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:30.298 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.783951+0000", 2024-03-30T14:04:30.298 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.298 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.298 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.298 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:30.298 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.783951+0000", 2024-03-30T14:04:30.298 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.298 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.299 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.299 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:30.299 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.783952+0000", 2024-03-30T14:04:30.299 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.229e-06 2024-03-30T14:04:30.299 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.299 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.299 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:30.299 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.783957+0000", 2024-03-30T14:04:30.299 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9250000000000003e-06 2024-03-30T14:04:30.299 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.299 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.299 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:30.299 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.783969+0000", 2024-03-30T14:04:30.299 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2170999999999999e-05 2024-03-30T14:04:30.299 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.300 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.300 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:30.300 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.783977+0000", 2024-03-30T14:04:30.300 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.9289999999999998e-06 2024-03-30T14:04:30.300 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.300 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.300 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.300 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.784033+0000", 2024-03-30T14:04:30.300 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6314000000000003e-05 2024-03-30T14:04:30.300 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.300 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.300 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.300 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.784042+0000", 2024-03-30T14:04:30.300 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6840000000000002e-06 2024-03-30T14:04:30.300 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.301 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.301 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.301 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.696778+0000", 2024-03-30T14:04:30.301 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.91273620499999997 2024-03-30T14:04:30.301 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.301 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.301 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:30.301 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.696885+0000", 2024-03-30T14:04:30.301 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000106572 2024-03-30T14:04:30.301 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.301 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.301 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,4", 2024-03-30T14:04:30.301 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.696964+0000", 2024-03-30T14:04:30.301 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.9204999999999998e-05 2024-03-30T14:04:30.302 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.302 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.302 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:30.302 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.698521+0000", 2024-03-30T14:04:30.302 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001557342 2024-03-30T14:04:30.302 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.302 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.302 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.302 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.703466+0000", 2024-03-30T14:04:30.302 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.004944454 2024-03-30T14:04:30.302 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.302 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.303 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.d", "id": [4, 0]}]': finished 2024-03-30T14:04:30.303 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [4, 6]}]': finished 2024-03-30T14:04:30.303 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [1, 6]}]': finished 2024-03-30T14:04:30.303 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:30 smithi113 ceph-mon[34341]: osdmap e460: 8 total, 8 up, 5 in 2024-03-30T14:04:30.303 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:04:30.303 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:30.303 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.303 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.708027+0000", 2024-03-30T14:04:30.304 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0045613370000000004 2024-03-30T14:04:30.304 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.304 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.304 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:30.304 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.708074+0000", 2024-03-30T14:04:30.304 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7114000000000003e-05 2024-03-30T14:04:30.304 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.304 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.304 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:30.304 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.708116+0000", 2024-03-30T14:04:30.304 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1705000000000001e-05 2024-03-30T14:04:30.304 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.304 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.304 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.304 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.305 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.305 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:262457 10.6 10:65568f45:::benchmark_data_smithi043_169264_object100230:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-30T14:04:30.305 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:58.803017+0000", 2024-03-30T14:04:30.305 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 271.421614358, 2024-03-30T14:04:30.305 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.90526345500000005, 2024-03-30T14:04:30.305 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:30.305 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:30.305 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:30.305 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:30.305 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:04:30.305 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:04:30.305 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 262457 2024-03-30T14:04:30.305 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.305 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:30.306 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.306 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:30.306 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.803017+0000", 2024-03-30T14:04:30.306 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.306 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.306 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.306 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:30.306 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.803017+0000", 2024-03-30T14:04:30.306 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.306 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.306 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.306 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:30.306 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.803019+0000", 2024-03-30T14:04:30.306 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4810000000000001e-06 2024-03-30T14:04:30.307 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.307 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.307 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:30.307 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.803025+0000", 2024-03-30T14:04:30.307 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9490000000000001e-06 2024-03-30T14:04:30.307 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.307 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.307 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:30.307 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.803039+0000", 2024-03-30T14:04:30.307 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3763e-05 2024-03-30T14:04:30.307 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.307 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.307 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:30.307 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.803046+0000", 2024-03-30T14:04:30.307 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1810000000000004e-06 2024-03-30T14:04:30.308 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.308 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.308 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.308 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.803092+0000", 2024-03-30T14:04:30.308 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.5880000000000001e-05 2024-03-30T14:04:30.308 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.308 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.308 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.308 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.803098+0000", 2024-03-30T14:04:30.308 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6630000000000004e-06 2024-03-30T14:04:30.308 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.308 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.309 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.309 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.699131+0000", 2024-03-30T14:04:30.309 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.89603241700000003 2024-03-30T14:04:30.309 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.309 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.309 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:30.309 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.699225+0000", 2024-03-30T14:04:30.309 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4483999999999999e-05 2024-03-30T14:04:30.309 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.309 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.309 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,4", 2024-03-30T14:04:30.309 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.699331+0000", 2024-03-30T14:04:30.310 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000105319 2024-03-30T14:04:30.310 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.310 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.310 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:30.310 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.707985+0000", 2024-03-30T14:04:30.310 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0086540269999999999 2024-03-30T14:04:30.310 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.310 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.310 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.310 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.708160+0000", 2024-03-30T14:04:30.310 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00017568 2024-03-30T14:04:30.310 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.310 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.310 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.311 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.708196+0000", 2024-03-30T14:04:30.311 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.6106999999999998e-05 2024-03-30T14:04:30.311 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.311 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.311 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:30.311 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.708247+0000", 2024-03-30T14:04:30.311 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0439999999999998e-05 2024-03-30T14:04:30.311 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.311 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.311 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:30.311 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.708281+0000", 2024-03-30T14:04:30.311 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.4063999999999998e-05 2024-03-30T14:04:30.312 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.312 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.312 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.312 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.312 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.312 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:262467 10.6 10:6639d4a5:::benchmark_data_smithi043_169264_object100240:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-30T14:04:30.312 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:58.815122+0000", 2024-03-30T14:04:30.312 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 271.409509974, 2024-03-30T14:04:30.312 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.90076515800000001, 2024-03-30T14:04:30.312 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:30.312 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:30.312 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:30.312 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:30.313 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:04:30.313 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:04:30.313 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 262467 2024-03-30T14:04:30.313 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.313 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:30.313 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.313 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:30.313 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.815122+0000", 2024-03-30T14:04:30.313 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.313 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.313 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.313 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:30.313 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.815122+0000", 2024-03-30T14:04:30.314 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.314 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.314 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.314 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:30.314 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.815123+0000", 2024-03-30T14:04:30.314 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.9199999999999999e-07 2024-03-30T14:04:30.314 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.314 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.314 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:30.314 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.815128+0000", 2024-03-30T14:04:30.314 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8269999999999997e-06 2024-03-30T14:04:30.314 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.314 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.315 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:30.315 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.815140+0000", 2024-03-30T14:04:30.315 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2943e-05 2024-03-30T14:04:30.315 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.315 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.315 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:30.315 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.815148+0000", 2024-03-30T14:04:30.315 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.5050000000000001e-06 2024-03-30T14:04:30.315 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.315 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.315 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.315 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.815205+0000", 2024-03-30T14:04:30.315 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6722999999999999e-05 2024-03-30T14:04:30.316 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.316 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.316 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.316 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.815215+0000", 2024-03-30T14:04:30.316 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0417999999999999e-05 2024-03-30T14:04:30.316 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.316 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.316 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.316 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.701244+0000", 2024-03-30T14:04:30.316 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.88602924100000002 2024-03-30T14:04:30.316 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.316 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.316 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:30.317 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.701341+0000", 2024-03-30T14:04:30.317 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.6830000000000006e-05 2024-03-30T14:04:30.317 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.317 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.317 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,4", 2024-03-30T14:04:30.317 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.701436+0000", 2024-03-30T14:04:30.317 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4648000000000006e-05 2024-03-30T14:04:30.317 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.317 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.317 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:30.317 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.707990+0000", 2024-03-30T14:04:30.317 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0065538480000000001 2024-03-30T14:04:30.317 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.317 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.317 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.318 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.708317+0000", 2024-03-30T14:04:30.318 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00032749800000000001 2024-03-30T14:04:30.318 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.318 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.318 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.318 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.715788+0000", 2024-03-30T14:04:30.318 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0074704810000000002 2024-03-30T14:04:30.318 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.318 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.318 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:30.318 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.715840+0000", 2024-03-30T14:04:30.318 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2734e-05 2024-03-30T14:04:30.318 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.319 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.319 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:30.319 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.715887+0000", 2024-03-30T14:04:30.319 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6570000000000003e-05 2024-03-30T14:04:30.319 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.319 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.319 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.319 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.319 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.319 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:262480 10.6 10:6d87e777:::benchmark_data_smithi043_169264_object100253:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-30T14:04:30.319 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:58.842763+0000", 2024-03-30T14:04:30.319 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 271.38186858300003, 2024-03-30T14:04:30.319 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.87325768999999998, 2024-03-30T14:04:30.319 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:30.320 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:30.320 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:30.320 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:30.320 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:04:30.320 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:04:30.320 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 262480 2024-03-30T14:04:30.320 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.320 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:30.320 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.320 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:30.320 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.842763+0000", 2024-03-30T14:04:30.320 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.320 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.320 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.321 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:30.321 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.842763+0000", 2024-03-30T14:04:30.321 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.321 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.321 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.321 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:30.321 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.842764+0000", 2024-03-30T14:04:30.321 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.7900000000000007e-07 2024-03-30T14:04:30.321 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.321 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.321 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:30.321 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.842770+0000", 2024-03-30T14:04:30.321 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1820000000000003e-06 2024-03-30T14:04:30.321 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.321 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.322 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:30.322 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.842781+0000", 2024-03-30T14:04:30.322 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0882e-05 2024-03-30T14:04:30.322 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.322 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.322 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:30.322 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.842790+0000", 2024-03-30T14:04:30.322 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.037e-06 2024-03-30T14:04:30.322 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.322 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.322 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.322 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.842837+0000", 2024-03-30T14:04:30.322 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6462999999999999e-05 2024-03-30T14:04:30.323 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.323 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.323 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.323 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.842845+0000", 2024-03-30T14:04:30.323 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8890000000000002e-06 2024-03-30T14:04:30.323 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.323 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.323 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.323 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.703503+0000", 2024-03-30T14:04:30.323 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.86065884000000004 2024-03-30T14:04:30.323 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.323 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.323 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:30.323 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.703615+0000", 2024-03-30T14:04:30.323 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000111943 2024-03-30T14:04:30.324 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.324 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.324 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,4", 2024-03-30T14:04:30.324 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.703725+0000", 2024-03-30T14:04:30.324 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000109209 2024-03-30T14:04:30.324 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.324 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.324 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:30.324 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.708127+0000", 2024-03-30T14:04:30.324 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0044022389999999996 2024-03-30T14:04:30.324 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.324 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.324 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.324 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.708359+0000", 2024-03-30T14:04:30.325 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000231733 2024-03-30T14:04:30.325 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.325 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.325 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.325 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.715932+0000", 2024-03-30T14:04:30.325 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0075736639999999999 2024-03-30T14:04:30.325 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.325 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.325 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:30.325 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.715982+0000", 2024-03-30T14:04:30.325 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9984000000000001e-05 2024-03-30T14:04:30.325 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.325 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.325 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:30.325 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.716021+0000", 2024-03-30T14:04:30.326 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.8646000000000002e-05 2024-03-30T14:04:30.326 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.326 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.326 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.326 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.326 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.326 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.15531.0:262500 10.6 10:67ef04ad:::benchmark_data_smithi043_169264_object100273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-30T14:04:30.326 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:59:58.901217+0000", 2024-03-30T14:04:30.326 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 271.32341503700002, 2024-03-30T14:04:30.326 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.81506029999999996, 2024-03-30T14:04:30.326 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:30.326 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:30.326 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:30.326 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:30.327 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.15531", 2024-03-30T14:04:30.327 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/741132648", 2024-03-30T14:04:30.327 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 262500 2024-03-30T14:04:30.327 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.327 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:30.327 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.327 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:30.327 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.901217+0000", 2024-03-30T14:04:30.327 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.327 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.327 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.327 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:30.327 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.901217+0000", 2024-03-30T14:04:30.327 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.327 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.328 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.328 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:30.328 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.901218+0000", 2024-03-30T14:04:30.328 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6400000000000001e-07 2024-03-30T14:04:30.328 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.328 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.328 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:30.328 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.901231+0000", 2024-03-30T14:04:30.328 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3128e-05 2024-03-30T14:04:30.328 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.328 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.328 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:30.328 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.901242+0000", 2024-03-30T14:04:30.328 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0879999999999999e-05 2024-03-30T14:04:30.328 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.329 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.329 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:30.329 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.901248+0000", 2024-03-30T14:04:30.329 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5579999999999997e-06 2024-03-30T14:04:30.329 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.329 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.329 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.329 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.901303+0000", 2024-03-30T14:04:30.329 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5071999999999997e-05 2024-03-30T14:04:30.329 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.329 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.329 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.329 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:58.901314+0000", 2024-03-30T14:04:30.329 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0783e-05 2024-03-30T14:04:30.329 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.329 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.330 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.330 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.705857+0000", 2024-03-30T14:04:30.330 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.80454287199999996 2024-03-30T14:04:30.330 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.330 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.330 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:30.330 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.705951+0000", 2024-03-30T14:04:30.330 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.3629999999999996e-05 2024-03-30T14:04:30.330 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.330 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.330 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,4", 2024-03-30T14:04:30.330 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.706045+0000", 2024-03-30T14:04:30.330 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4492000000000003e-05 2024-03-30T14:04:30.330 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.330 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.331 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:30.331 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.708327+0000", 2024-03-30T14:04:30.331 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0022822430000000002 2024-03-30T14:04:30.331 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.331 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.331 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.331 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.715743+0000", 2024-03-30T14:04:30.331 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0074156179999999997 2024-03-30T14:04:30.331 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.331 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.331 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.331 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.716180+0000", 2024-03-30T14:04:30.331 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00043732599999999998 2024-03-30T14:04:30.331 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.332 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.332 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:30.332 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.716239+0000", 2024-03-30T14:04:30.332 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9255000000000001e-05 2024-03-30T14:04:30.332 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.332 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.332 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:30.332 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:59:59.716277+0000", 2024-03-30T14:04:30.332 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.7579000000000001e-05 2024-03-30T14:04:30.332 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.332 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.332 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.332 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.332 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.332 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:110087 11.5 11:ae1ed18c:::benchmark_data_smithi043_197439_object110086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-03-30T14:04:30.332 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:02:48.089246+0000", 2024-03-30T14:04:30.333 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 102.135385591, 2024-03-30T14:04:30.333 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.1289741700000002, 2024-03-30T14:04:30.333 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:30.333 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:30.333 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "reached pg", 2024-03-30T14:04:30.333 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:30.333 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:30.333 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:30.333 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 110087 2024-03-30T14:04:30.333 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.333 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:30.333 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.333 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:30.333 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.089246+0000", 2024-03-30T14:04:30.333 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.334 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.334 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.334 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:30.334 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.089246+0000", 2024-03-30T14:04:30.334 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.334 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.334 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.334 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:30.334 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.089247+0000", 2024-03-30T14:04:30.334 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0189999999999999e-06 2024-03-30T14:04:30.334 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.334 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.334 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:30.334 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.089323+0000", 2024-03-30T14:04:30.335 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.6025000000000005e-05 2024-03-30T14:04:30.335 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.335 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.335 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:30.335 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.089334+0000", 2024-03-30T14:04:30.335 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0533000000000001e-05 2024-03-30T14:04:30.335 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.335 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.335 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:30.335 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.089339+0000", 2024-03-30T14:04:30.335 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6710000000000004e-06 2024-03-30T14:04:30.335 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.335 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.335 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.336 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.089389+0000", 2024-03-30T14:04:30.336 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9976999999999999e-05 2024-03-30T14:04:30.336 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.336 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.336 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:30.336 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.089496+0000", 2024-03-30T14:04:30.336 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000106217 2024-03-30T14:04:30.336 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.336 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.336 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,6", 2024-03-30T14:04:30.336 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.089634+0000", 2024-03-30T14:04:30.336 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013857999999999999 2024-03-30T14:04:30.336 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.336 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.336 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.337 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.093464+0000", 2024-03-30T14:04:30.337 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0038300439999999999 2024-03-30T14:04:30.337 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.337 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.337 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:30.337 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.093476+0000", 2024-03-30T14:04:30.337 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2118e-05 2024-03-30T14:04:30.337 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.337 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.337 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.337 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.110150+0000", 2024-03-30T14:04:30.337 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.016673713999999999 2024-03-30T14:04:30.337 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.338 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.338 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.338 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.110165+0000", 2024-03-30T14:04:30.338 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5328e-05 2024-03-30T14:04:30.338 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.338 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.338 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.338 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.112344+0000", 2024-03-30T14:04:30.338 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0021785670000001 2024-03-30T14:04:30.338 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.338 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.338 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.338 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.112359+0000", 2024-03-30T14:04:30.338 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5345e-05 2024-03-30T14:04:30.339 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.339 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.339 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.339 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.218123+0000", 2024-03-30T14:04:30.339 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1057640879999999 2024-03-30T14:04:30.339 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.339 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.339 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:30.339 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.218220+0000", 2024-03-30T14:04:30.339 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.6943999999999999e-05 2024-03-30T14:04:30.339 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.339 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.339 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.339 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.339 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.340 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:110088 11.5 11:a0bfdddf:::benchmark_data_smithi043_197439_object110087:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-03-30T14:04:30.340 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:02:48.092837+0000", 2024-03-30T14:04:30.340 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 102.131794439, 2024-03-30T14:04:30.340 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.1350299430000002, 2024-03-30T14:04:30.340 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:30.340 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:30.340 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "reached pg", 2024-03-30T14:04:30.340 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:30.340 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:30.340 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:30.340 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 110088 2024-03-30T14:04:30.340 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.340 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:30.340 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.340 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:30.341 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.092837+0000", 2024-03-30T14:04:30.341 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.341 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.341 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.341 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:30.341 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.092837+0000", 2024-03-30T14:04:30.341 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.341 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.341 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.341 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:30.341 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.092839+0000", 2024-03-30T14:04:30.341 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.451e-06 2024-03-30T14:04:30.341 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.341 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.341 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:30.342 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.092892+0000", 2024-03-30T14:04:30.342 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3514999999999998e-05 2024-03-30T14:04:30.342 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.342 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.342 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:30.342 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.092908+0000", 2024-03-30T14:04:30.342 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5721000000000001e-05 2024-03-30T14:04:30.342 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.342 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.342 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:30.342 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.092917+0000", 2024-03-30T14:04:30.342 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5120000000000008e-06 2024-03-30T14:04:30.342 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.342 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.343 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.343 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.093499+0000", 2024-03-30T14:04:30.343 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00058198899999999999 2024-03-30T14:04:30.343 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.343 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.343 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:30.343 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.093634+0000", 2024-03-30T14:04:30.343 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000135697 2024-03-30T14:04:30.343 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.343 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.343 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 3,6", 2024-03-30T14:04:30.343 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.093778+0000", 2024-03-30T14:04:30.343 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000143404 2024-03-30T14:04:30.343 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.343 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.344 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.344 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.110202+0000", 2024-03-30T14:04:30.345 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.016424613000000001 2024-03-30T14:04:30.345 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.345 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.345 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.345 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.110212+0000", 2024-03-30T14:04:30.345 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4970000000000007e-06 2024-03-30T14:04:30.345 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.345 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.345 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.345 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.112396+0000", 2024-03-30T14:04:30.345 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0021846619999999 2024-03-30T14:04:30.345 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.346 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.346 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.346 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.112407+0000", 2024-03-30T14:04:30.346 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0122999999999999e-05 2024-03-30T14:04:30.346 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.346 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.346 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.346 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.218554+0000", 2024-03-30T14:04:30.346 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1061477639999999 2024-03-30T14:04:30.346 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.346 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.346 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for degraded object", 2024-03-30T14:04:30.346 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.219318+0000", 2024-03-30T14:04:30.347 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00076344100000000001 2024-03-30T14:04:30.347 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.347 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.347 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.347 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.227818+0000", 2024-03-30T14:04:30.347 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0085005740000000003 2024-03-30T14:04:30.347 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.347 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.347 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:30.347 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.227867+0000", 2024-03-30T14:04:30.347 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8980000000000002e-05 2024-03-30T14:04:30.347 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.347 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.348 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.348 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.348 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.348 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.14451.0:3793 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94764809516544 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-30T14:04:30.348 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:02:48.092921+0000", 2024-03-30T14:04:30.348 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 102.131710755, 2024-03-30T14:04:30.348 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0417629370000001, 2024-03-30T14:04:30.348 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:30.348 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:30.348 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T14:04:30.348 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:30.348 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.14451", 2024-03-30T14:04:30.348 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/3211716515", 2024-03-30T14:04:30.348 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 3793 2024-03-30T14:04:30.349 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.349 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:30.349 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.349 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:30.349 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.092921+0000", 2024-03-30T14:04:30.349 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.349 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.349 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.349 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:30.349 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.092921+0000", 2024-03-30T14:04:30.349 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.349 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.349 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.349 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:30.350 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.092922+0000", 2024-03-30T14:04:30.350 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.9500000000000001e-07 2024-03-30T14:04:30.350 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.350 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.350 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:30.350 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.092927+0000", 2024-03-30T14:04:30.350 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4140000000000002e-06 2024-03-30T14:04:30.350 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.350 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.350 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:30.350 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.092938+0000", 2024-03-30T14:04:30.350 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1005999999999999e-05 2024-03-30T14:04:30.350 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.350 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.351 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:30.351 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.092950+0000", 2024-03-30T14:04:30.351 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1457000000000001e-05 2024-03-30T14:04:30.351 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.351 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.351 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.351 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.093012+0000", 2024-03-30T14:04:30.351 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.2317000000000007e-05 2024-03-30T14:04:30.351 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.351 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.351 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op must wait for map", 2024-03-30T14:04:30.351 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.093019+0000", 2024-03-30T14:04:30.351 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6120000000000004e-06 2024-03-30T14:04:30.351 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.351 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.352 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.352 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.106813+0000", 2024-03-30T14:04:30.352 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.013794034 2024-03-30T14:04:30.352 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.352 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.352 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.352 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.106824+0000", 2024-03-30T14:04:30.352 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1325000000000001e-05 2024-03-30T14:04:30.352 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.352 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.352 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.352 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.127679+0000", 2024-03-30T14:04:30.352 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0208552980000001 2024-03-30T14:04:30.352 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.352 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.353 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for degraded object", 2024-03-30T14:04:30.353 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.128108+0000", 2024-03-30T14:04:30.353 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000429064 2024-03-30T14:04:30.353 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.353 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.353 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.353 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.130501+0000", 2024-03-30T14:04:30.353 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0023926730000000001 2024-03-30T14:04:30.353 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.353 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.353 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:30.353 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.130556+0000", 2024-03-30T14:04:30.353 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5151000000000001e-05 2024-03-30T14:04:30.353 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.353 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.354 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:30.354 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.134684+0000", 2024-03-30T14:04:30.354 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0041276910000000002 2024-03-30T14:04:30.354 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.354 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.354 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.354 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.354 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.354 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:182534 11.5 11:a141f006:::benchmark_data_smithi043_197439_object72416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-03-30T14:04:30.354 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:04:01.440170+0000", 2024-03-30T14:04:30.354 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 28.784461832000002, 2024-03-30T14:04:30.354 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.817464996, 2024-03-30T14:04:30.354 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:30.354 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:30.355 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:30.355 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:30.355 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:30.355 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:30.355 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 182534 2024-03-30T14:04:30.355 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.355 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:30.355 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.355 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:30.355 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.440170+0000", 2024-03-30T14:04:30.355 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.355 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.355 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.355 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:30.355 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.440170+0000", 2024-03-30T14:04:30.355 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:30.356 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.356 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.356 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:30.356 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.440170+0000", 2024-03-30T14:04:30.356 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.89e-07 2024-03-30T14:04:30.356 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.356 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.356 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:30.356 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.440173+0000", 2024-03-30T14:04:30.356 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.7099999999999999e-06 2024-03-30T14:04:30.356 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.356 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.356 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:30.356 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.440179+0000", 2024-03-30T14:04:30.356 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9259999999999997e-06 2024-03-30T14:04:30.356 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.357 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.357 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:30.357 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.440184+0000", 2024-03-30T14:04:30.357 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1440000000000002e-06 2024-03-30T14:04:30.357 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.357 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.357 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.357 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.441700+0000", 2024-03-30T14:04:30.357 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0015162190000000001 2024-03-30T14:04:30.357 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.357 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.357 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:30.357 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.441709+0000", 2024-03-30T14:04:30.357 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.037e-06 2024-03-30T14:04:30.357 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.358 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.358 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:30.358 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:02.247732+0000", 2024-03-30T14:04:30.358 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.80602234299999997 2024-03-30T14:04:30.358 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.358 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.358 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:30.358 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:02.247871+0000", 2024-03-30T14:04:30.358 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013969699999999999 2024-03-30T14:04:30.358 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.358 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.358 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,6", 2024-03-30T14:04:30.358 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:02.248219+0000", 2024-03-30T14:04:30.358 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00034716 2024-03-30T14:04:30.359 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.359 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.359 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:30.359 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:02.257375+0000", 2024-03-30T14:04:30.359 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0091560059999999995 2024-03-30T14:04:30.359 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.359 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.359 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.359 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:02.257494+0000", 2024-03-30T14:04:30.359 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00011948899999999999 2024-03-30T14:04:30.359 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.359 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.359 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:30.359 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:02.257535+0000", 2024-03-30T14:04:30.359 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1007999999999998e-05 2024-03-30T14:04:30.359 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.360 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.360 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:30.360 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:02.257588+0000", 2024-03-30T14:04:30.360 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2726999999999998e-05 2024-03-30T14:04:30.360 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:30.360 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:30.360 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:30.360 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:02.257635+0000", 2024-03-30T14:04:30.360 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7141000000000001e-05 2024-03-30T14:04:30.360 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.360 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.360 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.360 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:30.360 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:30.360 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T14:04:30.386 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:30.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:30.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:30.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.d", "id": [4, 0]}]': finished 2024-03-30T14:04:30.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [4, 6]}]': finished 2024-03-30T14:04:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [1, 6]}]': finished 2024-03-30T14:04:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:30 smithi043 ceph-mon[29963]: osdmap e460: 8 total, 8 up, 5 in 2024-03-30T14:04:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:04:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:30.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.d", "id": [4, 0]}]': finished 2024-03-30T14:04:30.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [4, 6]}]': finished 2024-03-30T14:04:30.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [1, 6]}]': finished 2024-03-30T14:04:30.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:30 smithi043 ceph-mon[42099]: osdmap e460: 8 total, 8 up, 5 in 2024-03-30T14:04:30.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:04:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:30.487 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:30.520 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:30.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:30.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:30.622 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:30.659 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:30.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:30.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:30.760 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:30.805 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:30.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:30.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:30.906 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:30.944 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:30.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:30.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:31.046 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:31.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:31.084 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:31.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:31.141 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T14:04:31.185 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:31.215 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:31.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:31.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:31.317 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:31.340 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:31.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:31.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:31.413 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:31 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:31.413 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:31 smithi113 ceph-mon[34341]: pgmap v1013: 168 pgs: 1 active+remapped+backfill_toofull, 1 peering, 166 active+clean; 495 MiB data, 6.0 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 1.82k op/s; 386/24321 objects misplaced (1.587%); 0 B/s, 0 objects/s recovering 2024-03-30T14:04:31.413 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:31 smithi113 ceph-mon[34341]: osdmap e461: 8 total, 8 up, 5 in 2024-03-30T14:04:31.413 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:31 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/4289864239' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T14:04:31.413 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:31 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/4289864239' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-30T14:04:31.413 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:31 smithi113 ceph-mon[34341]: osdmap e462: 8 total, 8 up, 5 in 2024-03-30T14:04:31.439 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-30T14:04:31.442 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:31.504 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:31.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:31.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:31.606 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:31.630 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:31.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:31.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:31.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:31 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:31.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:31 smithi043 ceph-mon[29963]: pgmap v1013: 168 pgs: 1 active+remapped+backfill_toofull, 1 peering, 166 active+clean; 495 MiB data, 6.0 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 1.82k op/s; 386/24321 objects misplaced (1.587%); 0 B/s, 0 objects/s recovering 2024-03-30T14:04:31.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:31 smithi043 ceph-mon[29963]: osdmap e461: 8 total, 8 up, 5 in 2024-03-30T14:04:31.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:31 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4289864239' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T14:04:31.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:31 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/4289864239' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-30T14:04:31.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:31 smithi043 ceph-mon[29963]: osdmap e462: 8 total, 8 up, 5 in 2024-03-30T14:04:31.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:31 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:31.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:31 smithi043 ceph-mon[42099]: pgmap v1013: 168 pgs: 1 active+remapped+backfill_toofull, 1 peering, 166 active+clean; 495 MiB data, 6.0 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 1.82k op/s; 386/24321 objects misplaced (1.587%); 0 B/s, 0 objects/s recovering 2024-03-30T14:04:31.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:31 smithi043 ceph-mon[42099]: osdmap e461: 8 total, 8 up, 5 in 2024-03-30T14:04:31.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:31 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/4289864239' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T14:04:31.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:31 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/4289864239' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-30T14:04:31.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:31 smithi043 ceph-mon[42099]: osdmap e462: 8 total, 8 up, 5 in 2024-03-30T14:04:31.679 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:04:31.731 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:31.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:31.757 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:31.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:31.858 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:31.882 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:31.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:31.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:31.984 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:32.009 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:32.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:32.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:32.111 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:32.139 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:32.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:32.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:32.240 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:32.271 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:32.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:32.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:32.372 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:32.396 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:32.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:32.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:32.498 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:32.620 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:32.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:32.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:32.722 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:32.750 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:32.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:32.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:32.851 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:32.878 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:32.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:32.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:32.979 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:33.007 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:33.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:33.109 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:33.133 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:33.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:33.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:33.235 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:33.260 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:33.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:33.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:33.361 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:33.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:33 smithi043 ceph-mon[29963]: pgmap v1016: 168 pgs: 1 active+undersized+remapped, 1 active+remapped+backfill_toofull, 1 peering, 165 active+clean; 306 MiB data, 5.3 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.98k op/s; 386/15270 objects misplaced (2.528%) 2024-03-30T14:04:33.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:33 smithi043 ceph-mon[29963]: osdmap e463: 8 total, 8 up, 5 in 2024-03-30T14:04:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:33 smithi043 ceph-mon[42099]: pgmap v1016: 168 pgs: 1 active+undersized+remapped, 1 active+remapped+backfill_toofull, 1 peering, 165 active+clean; 306 MiB data, 5.3 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.98k op/s; 386/15270 objects misplaced (2.528%) 2024-03-30T14:04:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:33 smithi043 ceph-mon[42099]: osdmap e463: 8 total, 8 up, 5 in 2024-03-30T14:04:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:33.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:33.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:33.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:33 smithi113 ceph-mon[34341]: pgmap v1016: 168 pgs: 1 active+undersized+remapped, 1 active+remapped+backfill_toofull, 1 peering, 165 active+clean; 306 MiB data, 5.3 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.98k op/s; 386/15270 objects misplaced (2.528%) 2024-03-30T14:04:33.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:33 smithi113 ceph-mon[34341]: osdmap e463: 8 total, 8 up, 5 in 2024-03-30T14:04:33.498 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:33.537 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:33.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:33.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:33.639 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:33.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:33.681 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:33.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:33.782 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:33.806 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:33.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:33.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:33.908 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:33.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:33.949 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:33.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:34.050 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:34.074 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:34.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:34.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:34.176 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:34.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:34 smithi113 ceph-mon[34341]: osdmap e464: 8 total, 8 up, 5 in 2024-03-30T14:04:34.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:34 smithi113 ceph-mon[34341]: 11.d continuing backfill to osd.1 from (437'10795,461'13810] MIN to 461'13810 2024-03-30T14:04:34.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:34 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:04:34.215 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:34.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:34.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:34.317 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:34.359 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:34.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:34.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:34.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:34 smithi043 ceph-mon[29963]: osdmap e464: 8 total, 8 up, 5 in 2024-03-30T14:04:34.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:34 smithi043 ceph-mon[29963]: 11.d continuing backfill to osd.1 from (437'10795,461'13810] MIN to 461'13810 2024-03-30T14:04:34.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:34 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:04:34.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:34 smithi043 ceph-mon[42099]: osdmap e464: 8 total, 8 up, 5 in 2024-03-30T14:04:34.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:34 smithi043 ceph-mon[42099]: 11.d continuing backfill to osd.1 from (437'10795,461'13810] MIN to 461'13810 2024-03-30T14:04:34.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:34 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:04:34.462 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:34.493 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:34.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:34.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:34.513 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T14:04:34.554 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T14:04:34.595 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:34.603 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T14:04:34.624 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:34.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:34.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:34.647 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T14:04:34.695 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T14:04:34.725 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:34.741 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T14:04:34.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:34.762 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:34.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:34.793 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T14:04:34.841 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T14:04:34.847 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:04:34.848 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:04:34.848 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:04:34.848 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:04:34.848 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:04:34.848 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:04:34.848 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:04:34.848 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:04:34.863 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:34.891 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:34.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:34.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:34.993 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:35.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:35.018 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:35.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:35.043 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Removed 110115 objects 2024-03-30T14:04:35.043 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Clean up completed and total clean up time :103.965 2024-03-30T14:04:35.119 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:35.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:35.144 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:35.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:35.164 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-03-30T14:04:35.245 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:35.268 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:35.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:35.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:35.370 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:35.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:35.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:35.419 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-03-30T14:04:35.419 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-03-30T14:04:35.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:35 smithi043 ceph-mon[42099]: pgmap v1019: 168 pgs: 1 active+undersized+remapped, 1 active+remapped+backfill_toofull, 1 peering, 165 active+clean; 268 MiB data, 5.2 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 908 op/s; 386/13434 objects misplaced (2.873%) 2024-03-30T14:04:35.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:35 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3232937994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:04:35.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:35 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2822492315' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:04:35.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:35 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:04:35.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:35 smithi043 ceph-mon[29963]: pgmap v1019: 168 pgs: 1 active+undersized+remapped, 1 active+remapped+backfill_toofull, 1 peering, 165 active+clean; 268 MiB data, 5.2 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 908 op/s; 386/13434 objects misplaced (2.873%) 2024-03-30T14:04:35.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:35 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3232937994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:04:35.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:35 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2822492315' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:04:35.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:35 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:04:35.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:35 smithi113 ceph-mon[34341]: pgmap v1019: 168 pgs: 1 active+undersized+remapped, 1 active+remapped+backfill_toofull, 1 peering, 165 active+clean; 268 MiB data, 5.2 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 908 op/s; 386/13434 objects misplaced (2.873%) 2024-03-30T14:04:35.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:35 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3232937994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:04:35.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:35 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2822492315' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:04:35.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:35 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:04:35.497 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:35.520 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:35.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:35.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:35.622 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:35.650 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:35.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:35.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:35.751 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:35.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:35.779 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:35.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:35.880 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:35.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:35.904 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:35.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:36.005 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:36.029 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:36.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:36.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:36.131 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:36.198 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:36.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:36.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:36.217 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:04:36.301 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:36.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:36.325 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:36.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:36.338 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:04:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:04:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:04:36.339 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:36 smithi043 ceph-mon[29963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:04:36.339 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:36 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:04:36.339 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:36 smithi043 ceph-mon[29963]: osdmap e465: 8 total, 8 up, 5 in 2024-03-30T14:04:36.339 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:36 smithi043 ceph-mon[42099]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:04:36.339 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:36 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:04:36.339 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:36 smithi043 ceph-mon[42099]: osdmap e465: 8 total, 8 up, 5 in 2024-03-30T14:04:36.426 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:36.455 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:36.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:36.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:36.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:36 smithi113 ceph-mon[34341]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:04:36.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:36 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:04:36.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:36 smithi113 ceph-mon[34341]: osdmap e465: 8 total, 8 up, 5 in 2024-03-30T14:04:36.558 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:36.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:36.586 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:36.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:36.687 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:36.715 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:36.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:36.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:36.817 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:36.841 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:36.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:36.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:36.943 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:36.972 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:36.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:36.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:37.073 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:04:37.099 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:37.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:04:37.200 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:37.224 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:37.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:37.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:37.325 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:37.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:37.350 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:37.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:37.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:37 smithi043 ceph-mon[29963]: pgmap v1021: 168 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 166 active+clean; 102 MiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 1.34k op/s; 33/5481 objects misplaced (0.602%); 941 KiB/s, 19 objects/s recovering 2024-03-30T14:04:37.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:37 smithi043 ceph-mon[29963]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-30T14:04:37.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:37 smithi043 ceph-mon[29963]: osdmap e466: 8 total, 8 up, 5 in 2024-03-30T14:04:37.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:37 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:04:37.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:37 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:04:37.383 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:37 smithi043 ceph-mon[42099]: pgmap v1021: 168 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 166 active+clean; 102 MiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 1.34k op/s; 33/5481 objects misplaced (0.602%); 941 KiB/s, 19 objects/s recovering 2024-03-30T14:04:37.383 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:37 smithi043 ceph-mon[42099]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-30T14:04:37.383 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:37 smithi043 ceph-mon[42099]: osdmap e466: 8 total, 8 up, 5 in 2024-03-30T14:04:37.383 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:37 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:04:37.383 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:37 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:04:37.450 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:37.458 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:37 smithi113 ceph-mon[34341]: pgmap v1021: 168 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 166 active+clean; 102 MiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 1.34k op/s; 33/5481 objects misplaced (0.602%); 941 KiB/s, 19 objects/s recovering 2024-03-30T14:04:37.458 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:37 smithi113 ceph-mon[34341]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-30T14:04:37.458 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:37 smithi113 ceph-mon[34341]: osdmap e466: 8 total, 8 up, 5 in 2024-03-30T14:04:37.458 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:37 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:04:37.458 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:37 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:04:37.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:37.478 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:37.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:37.579 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:37.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:37.605 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:37.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:37.706 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:37.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:37.730 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:37.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:37.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T14:04:37.832 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:37.859 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:37.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:37.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:37.960 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:37.989 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:37.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:37.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:38.025 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6, 5] out_osds: [7, 3, 2] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-03-30T14:04:38.026 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T14:04:38.026 INFO:tasks.thrashosds.thrasher:Reweighting osd 5 to 0.3311352474105644 2024-03-30T14:04:38.026 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd reweight 5 0.3311352474105644 2024-03-30T14:04:38.036 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:04:38.091 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:38.114 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:38.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:38.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:38.134 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T14:04:38.216 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:38.243 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:38.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:38.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:38.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:38 smithi043 ceph-mon[29963]: osdmap e467: 8 total, 8 up, 5 in 2024-03-30T14:04:38.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:38 smithi043 ceph-mon[42099]: osdmap e467: 8 total, 8 up, 5 in 2024-03-30T14:04:38.344 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:38.371 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:38.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:38.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:38.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:38 smithi113 ceph-mon[34341]: osdmap e467: 8 total, 8 up, 5 in 2024-03-30T14:04:38.473 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:04:38.501 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:38.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:38.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:04:38.603 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:38.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:38.628 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:38.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:38.729 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:38.752 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:38.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:38.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:38.854 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:04:38.877 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:38.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:38.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:04:38.979 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:39.002 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:39.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:39.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:39.103 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:04:39.127 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:39.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:39.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:04:39.179 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'unique_pool_4' removed 2024-03-30T14:04:39.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:39 smithi043 ceph-mon[29963]: pgmap v1024: 168 pgs: 1 active+remapped+backfilling, 167 active+clean; 49 MiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 879 op/s; 33/2922 objects misplaced (1.129%); 942 KiB/s, 19 objects/s recovering 2024-03-30T14:04:39.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:39 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1317403266' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:04:39.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:39 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:04:39.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:39 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:04:39.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:39 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:04:39.196 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:39 smithi043 ceph-mon[42099]: pgmap v1024: 168 pgs: 1 active+remapped+backfilling, 167 active+clean; 49 MiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 879 op/s; 33/2922 objects misplaced (1.129%); 942 KiB/s, 19 objects/s recovering 2024-03-30T14:04:39.196 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:39 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1317403266' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:04:39.196 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:39 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:04:39.196 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:39 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:04:39.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:39 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:04:39.229 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:04:39.261 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:39.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:04:39.340 INFO:teuthology.orchestra.run.smithi113.stdout:ERROR: (22) Invalid argument 2024-03-30T14:04:39.340 INFO:teuthology.orchestra.run.smithi113.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:04:39.340 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:39 smithi113 ceph-mon[34341]: pgmap v1024: 168 pgs: 1 active+remapped+backfilling, 167 active+clean; 49 MiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 879 op/s; 33/2922 objects misplaced (1.129%); 942 KiB/s, 19 objects/s recovering 2024-03-30T14:04:39.341 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:39 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1317403266' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:04:39.341 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:39 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:04:39.341 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:39 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:04:39.341 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:39 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:04:39.363 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:04:39.388 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:39.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:39.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:04:39.489 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:39.513 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:39.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:39.615 DEBUG:teuthology.orchestra.run.smithi113:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:04:39.639 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: docker: command not found 2024-03-30T14:04:39.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:39.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi113 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:04:39.740 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:04:39.764 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-03-30T14:04:39.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:04:39.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:04:39.842 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-30T14:04:39.852 INFO:tasks.thrashosds:joining thrashosds 2024-03-30T14:04:39.900 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:04:39 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:04:39.925 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-03-30T14:04:39.945 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:04:40.185 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:04:40.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:40 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1317403266' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T14:04:40.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:40 smithi043 ceph-mon[29963]: osdmap e468: 8 total, 8 up, 5 in 2024-03-30T14:04:40.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:40 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1317403266' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T14:04:40.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:40 smithi043 ceph-mon[42099]: osdmap e468: 8 total, 8 up, 5 in 2024-03-30T14:04:40.453 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:40 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1317403266' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T14:04:40.453 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:40 smithi113 ceph-mon[34341]: osdmap e468: 8 total, 8 up, 5 in 2024-03-30T14:04:41.098 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-03-30T14:04:41.098 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-03-30T14:04:41.098 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-03-30T14:04:41.098 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-03-30T14:04:41.098 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-03-30T14:04:41.098 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-03-30T14:04:41.098 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-03-30T14:04:41.098 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-03-30T14:04:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:04:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:04:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:04:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:04:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:04:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:04:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:04:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:04:41.366 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:41 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:41.366 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:41 smithi113 ceph-mon[34341]: pgmap v1026: 152 pgs: 152 active+clean; 582 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 877 B/s rd, 0 op/s 2024-03-30T14:04:41.366 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:41 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1976709878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:04:41.376 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:41 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:41.376 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:41 smithi043 ceph-mon[29963]: pgmap v1026: 152 pgs: 152 active+clean; 582 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 877 B/s rd, 0 op/s 2024-03-30T14:04:41.376 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:41 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1976709878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:04:41.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:41 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:41.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:41 smithi043 ceph-mon[42099]: pgmap v1026: 152 pgs: 152 active+clean; 582 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 877 B/s rd, 0 op/s 2024-03-30T14:04:41.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:41 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1976709878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:04:41.895 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T14:04:41.895 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [], 2024-03-30T14:04:41.895 INFO:teuthology.orchestra.run.smithi113.stdout: "complaint_time": 30, 2024-03-30T14:04:41.895 INFO:teuthology.orchestra.run.smithi113.stdout: "num_blocked_ops": 0 2024-03-30T14:04:41.895 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T14:04:42.253 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:04:42.560 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-03-30T14:04:42.799 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:04:43.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:43 smithi113 ceph-mon[34341]: pgmap v1027: 152 pgs: 152 active+clean; 582 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T14:04:43.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:43 smithi113 ceph-mon[34341]: osdmap e469: 8 total, 8 up, 5 in 2024-03-30T14:04:43.465 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:43 smithi043 ceph-mon[29963]: pgmap v1027: 152 pgs: 152 active+clean; 582 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T14:04:43.465 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:43 smithi043 ceph-mon[29963]: osdmap e469: 8 total, 8 up, 5 in 2024-03-30T14:04:43.465 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:43 smithi043 ceph-mon[42099]: pgmap v1027: 152 pgs: 152 active+clean; 582 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T14:04:43.465 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:43 smithi043 ceph-mon[42099]: osdmap e469: 8 total, 8 up, 5 in 2024-03-30T14:04:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.5 to 0.331135 (54c5) 2024-03-30T14:04:44.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:44 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1170658266' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 0.3311352474105644}]: dispatch 2024-03-30T14:04:44.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:44 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1170658266' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 0.3311352474105644}]: dispatch 2024-03-30T14:04:44.434 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:44 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1170658266' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 0.3311352474105644}]: dispatch 2024-03-30T14:04:44.461 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T14:04:44.462 INFO:teuthology.orchestra.run.smithi113.stdout: "size": 20, 2024-03-30T14:04:44.462 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 600, 2024-03-30T14:04:44.462 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [ 2024-03-30T14:04:44.462 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.462 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:110097 11.9 11:9530eb54:::benchmark_data_smithi043_197439_object110096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-30T14:04:44.462 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:02:48.117700+0000", 2024-03-30T14:04:44.462 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 116.348837493, 2024-03-30T14:04:44.462 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.1339403539999999, 2024-03-30T14:04:44.462 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:44.462 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:44.462 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:44.463 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:44.463 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:44.463 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:44.463 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 110097 2024-03-30T14:04:44.463 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.463 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:44.463 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.463 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:44.463 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.117700+0000", 2024-03-30T14:04:44.463 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.463 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.463 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.463 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:44.463 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.117700+0000", 2024-03-30T14:04:44.463 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.464 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.464 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.464 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:44.464 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.117701+0000", 2024-03-30T14:04:44.464 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2419999999999999e-06 2024-03-30T14:04:44.464 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.464 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.464 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:44.464 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.117740+0000", 2024-03-30T14:04:44.464 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.8609000000000003e-05 2024-03-30T14:04:44.464 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.464 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.464 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:44.464 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.117752+0000", 2024-03-30T14:04:44.464 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1549000000000001e-05 2024-03-30T14:04:44.465 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.465 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.465 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:44.465 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.117758+0000", 2024-03-30T14:04:44.465 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9989999999999997e-06 2024-03-30T14:04:44.465 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.465 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.465 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.465 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.117831+0000", 2024-03-30T14:04:44.465 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.3145999999999994e-05 2024-03-30T14:04:44.465 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.465 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.465 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.465 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.117846+0000", 2024-03-30T14:04:44.465 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.4844000000000001e-05 2024-03-30T14:04:44.466 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.466 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.466 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.466 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.115023+0000", 2024-03-30T14:04:44.466 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.997177113 2024-03-30T14:04:44.466 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.466 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.466 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.466 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.115033+0000", 2024-03-30T14:04:44.466 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0818e-05 2024-03-30T14:04:44.466 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.466 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.466 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.466 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.229725+0000", 2024-03-30T14:04:44.466 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.114692016 2024-03-30T14:04:44.467 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.467 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.467 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:44.467 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.231013+0000", 2024-03-30T14:04:44.467 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0012879790000000001 2024-03-30T14:04:44.467 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.467 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.467 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1,3", 2024-03-30T14:04:44.467 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.231317+0000", 2024-03-30T14:04:44.467 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000303558 2024-03-30T14:04:44.467 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.467 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.467 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:44.467 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.235930+0000", 2024-03-30T14:04:44.468 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0046133260000000001 2024-03-30T14:04:44.468 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.468 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.468 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.468 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.240281+0000", 2024-03-30T14:04:44.468 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0043507909999999997 2024-03-30T14:04:44.468 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.468 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.468 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.468 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.240348+0000", 2024-03-30T14:04:44.468 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.7206000000000007e-05 2024-03-30T14:04:44.468 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.468 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.469 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.469 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.251447+0000", 2024-03-30T14:04:44.469 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.011098238999999999 2024-03-30T14:04:44.469 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.469 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.469 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:44.469 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.251511+0000", 2024-03-30T14:04:44.469 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.4863999999999994e-05 2024-03-30T14:04:44.469 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.469 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.469 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:44.469 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.251641+0000", 2024-03-30T14:04:44.469 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012905499999999999 2024-03-30T14:04:44.469 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.470 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.470 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.470 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.470 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.470 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:110099 11.a 11:5820e717:::benchmark_data_smithi043_197439_object110098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-30T14:04:44.470 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:02:48.127543+0000", 2024-03-30T14:04:44.470 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 116.338994187, 2024-03-30T14:04:44.470 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.1017011659999998, 2024-03-30T14:04:44.470 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:44.470 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:44.470 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:44.470 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:44.470 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:44.470 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:44.471 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 110099 2024-03-30T14:04:44.471 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.471 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:44.471 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.471 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:44.471 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.127543+0000", 2024-03-30T14:04:44.471 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.471 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.471 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.471 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:44.471 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.127543+0000", 2024-03-30T14:04:44.471 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.471 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.471 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.471 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:44.471 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.127545+0000", 2024-03-30T14:04:44.472 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2139999999999999e-06 2024-03-30T14:04:44.472 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.472 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.472 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:44.472 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.127601+0000", 2024-03-30T14:04:44.472 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.6375000000000001e-05 2024-03-30T14:04:44.472 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.472 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.472 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:44.472 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.127613+0000", 2024-03-30T14:04:44.472 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2349e-05 2024-03-30T14:04:44.472 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.472 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.472 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:44.472 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.127621+0000", 2024-03-30T14:04:44.472 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.9880000000000004e-06 2024-03-30T14:04:44.473 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.473 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.473 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.473 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.127718+0000", 2024-03-30T14:04:44.473 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.7041999999999997e-05 2024-03-30T14:04:44.473 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.473 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.473 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.473 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.127730+0000", 2024-03-30T14:04:44.473 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1990999999999999e-05 2024-03-30T14:04:44.473 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.473 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.473 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.473 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.104862+0000", 2024-03-30T14:04:44.474 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.977131533 2024-03-30T14:04:44.474 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.474 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.474 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.474 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.104875+0000", 2024-03-30T14:04:44.474 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3271000000000001e-05 2024-03-30T14:04:44.474 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.474 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.474 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.474 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.214492+0000", 2024-03-30T14:04:44.474 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1096171589999999 2024-03-30T14:04:44.474 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.474 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.474 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:44.474 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.215530+0000", 2024-03-30T14:04:44.475 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0010377489999999999 2024-03-30T14:04:44.475 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.475 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.475 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3,5", 2024-03-30T14:04:44.476 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.215722+0000", 2024-03-30T14:04:44.476 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00019179099999999999 2024-03-30T14:04:44.476 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.476 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.476 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:44.476 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.223332+0000", 2024-03-30T14:04:44.476 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0076096089999999998 2024-03-30T14:04:44.476 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.476 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.476 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.476 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.223522+0000", 2024-03-30T14:04:44.476 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00019093199999999999 2024-03-30T14:04:44.476 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.476 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.477 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.477 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.228426+0000", 2024-03-30T14:04:44.477 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0049033109999999996 2024-03-30T14:04:44.477 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.477 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.477 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.477 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.229099+0000", 2024-03-30T14:04:44.477 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00067363999999999998 2024-03-30T14:04:44.477 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.477 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.477 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:44.478 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.229154+0000", 2024-03-30T14:04:44.478 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4734000000000001e-05 2024-03-30T14:04:44.478 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.478 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.478 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:44.478 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.229245+0000", 2024-03-30T14:04:44.478 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.0477999999999997e-05 2024-03-30T14:04:44.478 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.478 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.478 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.478 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.478 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.479 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:110102 11.1 11:899c6fe4:::benchmark_data_smithi043_197439_object110101:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-30T14:04:44.479 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:02:48.132563+0000", 2024-03-30T14:04:44.479 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 116.33397473700001, 2024-03-30T14:04:44.479 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.1185458189999999, 2024-03-30T14:04:44.479 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:44.479 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:44.479 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:44.479 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:44.479 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:44.479 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:44.479 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 110102 2024-03-30T14:04:44.479 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.480 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:44.480 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.480 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:44.480 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.132563+0000", 2024-03-30T14:04:44.480 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.480 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.480 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.480 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:44.480 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.132563+0000", 2024-03-30T14:04:44.480 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.480 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.480 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.480 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:44.480 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.132564+0000", 2024-03-30T14:04:44.480 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.248e-06 2024-03-30T14:04:44.481 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.481 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.481 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:44.481 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.132602+0000", 2024-03-30T14:04:44.481 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.8234e-05 2024-03-30T14:04:44.481 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.481 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.481 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:44.481 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.132611+0000", 2024-03-30T14:04:44.481 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.4149999999999999e-06 2024-03-30T14:04:44.481 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.481 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.481 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:44.481 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.132617+0000", 2024-03-30T14:04:44.482 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.4269999999999996e-06 2024-03-30T14:04:44.482 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.482 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.482 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.482 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.132658+0000", 2024-03-30T14:04:44.482 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.0354e-05 2024-03-30T14:04:44.482 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.482 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.482 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.482 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.132665+0000", 2024-03-30T14:04:44.482 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1770000000000001e-06 2024-03-30T14:04:44.482 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.482 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.483 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.483 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.117983+0000", 2024-03-30T14:04:44.483 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.985318154 2024-03-30T14:04:44.483 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.483 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.483 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.483 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.117996+0000", 2024-03-30T14:04:44.483 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2639e-05 2024-03-30T14:04:44.483 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.483 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.483 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.483 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.216601+0000", 2024-03-30T14:04:44.483 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0986056980000001 2024-03-30T14:04:44.483 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.483 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.484 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:44.484 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.216756+0000", 2024-03-30T14:04:44.484 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000154537 2024-03-30T14:04:44.484 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.484 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.484 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2,3", 2024-03-30T14:04:44.484 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.217186+0000", 2024-03-30T14:04:44.484 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000430042 2024-03-30T14:04:44.484 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.484 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.484 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:44.484 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.225353+0000", 2024-03-30T14:04:44.484 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0081673660000000006 2024-03-30T14:04:44.485 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.485 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.485 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.485 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.235487+0000", 2024-03-30T14:04:44.485 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.010134071 2024-03-30T14:04:44.485 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.485 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.485 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.485 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.235805+0000", 2024-03-30T14:04:44.485 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000317929 2024-03-30T14:04:44.485 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.485 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.486 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.486 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.250886+0000", 2024-03-30T14:04:44.486 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.015081218 2024-03-30T14:04:44.486 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.486 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.486 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:44.486 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.250962+0000", 2024-03-30T14:04:44.486 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.5275e-05 2024-03-30T14:04:44.486 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.486 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.486 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:44.487 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.251109+0000", 2024-03-30T14:04:44.487 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000147035 2024-03-30T14:04:44.487 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.487 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.487 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.487 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.487 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.487 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:110103 11.1 11:865d33e0:::benchmark_data_smithi043_197439_object110102:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-30T14:04:44.487 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:02:48.137664+0000", 2024-03-30T14:04:44.487 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 116.328873724, 2024-03-30T14:04:44.487 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.1137233370000001, 2024-03-30T14:04:44.488 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:44.488 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:44.488 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:44.488 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:44.488 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:44.488 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:44.488 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 110103 2024-03-30T14:04:44.488 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.488 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:44.488 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.488 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:44.488 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.137664+0000", 2024-03-30T14:04:44.488 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.488 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.489 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.489 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:44.489 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.137664+0000", 2024-03-30T14:04:44.489 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.489 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.489 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.489 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:44.489 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.137665+0000", 2024-03-30T14:04:44.489 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.1900000000000001e-07 2024-03-30T14:04:44.489 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.489 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.489 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:44.489 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.137720+0000", 2024-03-30T14:04:44.489 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5290000000000001e-05 2024-03-30T14:04:44.490 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.490 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.490 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:44.490 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.137730+0000", 2024-03-30T14:04:44.490 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.9890000000000004e-06 2024-03-30T14:04:44.490 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.490 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.490 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:44.490 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.137735+0000", 2024-03-30T14:04:44.490 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2159999999999997e-06 2024-03-30T14:04:44.490 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.490 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.490 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.490 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.137777+0000", 2024-03-30T14:04:44.491 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1841000000000001e-05 2024-03-30T14:04:44.491 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.491 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.491 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.491 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.137785+0000", 2024-03-30T14:04:44.491 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.3370000000000002e-06 2024-03-30T14:04:44.491 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.491 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.491 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.491 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.118023+0000", 2024-03-30T14:04:44.491 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.98023857199999997 2024-03-30T14:04:44.491 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.491 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.491 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.492 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.118032+0000", 2024-03-30T14:04:44.492 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.0480000000000004e-06 2024-03-30T14:04:44.492 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.492 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.492 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.492 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.223879+0000", 2024-03-30T14:04:44.492 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1058468340000001 2024-03-30T14:04:44.492 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.492 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.492 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:44.492 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.223974+0000", 2024-03-30T14:04:44.492 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.5223999999999995e-05 2024-03-30T14:04:44.492 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.492 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.492 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2,3", 2024-03-30T14:04:44.493 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.224074+0000", 2024-03-30T14:04:44.493 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.9784999999999994e-05 2024-03-30T14:04:44.493 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.493 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.493 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:44.493 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.226432+0000", 2024-03-30T14:04:44.493 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0023585250000000002 2024-03-30T14:04:44.493 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.493 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.493 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.493 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.235694+0000", 2024-03-30T14:04:44.493 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0092616049999999991 2024-03-30T14:04:44.493 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.493 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.493 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.493 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.235858+0000", 2024-03-30T14:04:44.494 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00016362599999999999 2024-03-30T14:04:44.494 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.494 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.494 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.494 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.251165+0000", 2024-03-30T14:04:44.494 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0153075 2024-03-30T14:04:44.494 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.494 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.494 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:44.494 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.251241+0000", 2024-03-30T14:04:44.494 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.6031000000000004e-05 2024-03-30T14:04:44.494 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.494 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.494 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:44.494 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.251387+0000", 2024-03-30T14:04:44.495 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000145995 2024-03-30T14:04:44.495 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.495 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.495 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.495 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.495 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.495 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:110105 11.9 11:9ba93213:::benchmark_data_smithi043_197439_object110104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-30T14:04:44.495 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:02:48.138287+0000", 2024-03-30T14:04:44.495 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 116.32825063, 2024-03-30T14:04:44.495 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.113591848, 2024-03-30T14:04:44.495 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:44.495 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:44.495 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:44.495 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:44.495 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:44.495 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:44.496 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 110105 2024-03-30T14:04:44.496 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.496 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:44.496 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.496 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:44.496 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.138287+0000", 2024-03-30T14:04:44.496 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.496 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.496 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.496 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:44.496 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.138287+0000", 2024-03-30T14:04:44.496 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.496 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.496 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.496 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:44.497 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.138288+0000", 2024-03-30T14:04:44.497 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1500000000000005e-07 2024-03-30T14:04:44.497 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.497 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.497 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:44.497 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.138355+0000", 2024-03-30T14:04:44.497 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6983000000000006e-05 2024-03-30T14:04:44.497 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.497 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.497 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:44.497 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.138361+0000", 2024-03-30T14:04:44.497 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8040000000000004e-06 2024-03-30T14:04:44.497 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.497 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.497 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:44.497 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.138365+0000", 2024-03-30T14:04:44.498 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1860000000000002e-06 2024-03-30T14:04:44.498 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.498 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.498 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.498 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.138406+0000", 2024-03-30T14:04:44.498 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.0710000000000002e-05 2024-03-30T14:04:44.498 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.498 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.498 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.498 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.138414+0000", 2024-03-30T14:04:44.498 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.7440000000000007e-06 2024-03-30T14:04:44.498 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.498 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.498 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.498 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.115075+0000", 2024-03-30T14:04:44.499 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.97666148399999997 2024-03-30T14:04:44.499 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.499 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.499 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.499 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.115084+0000", 2024-03-30T14:04:44.499 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6510000000000007e-06 2024-03-30T14:04:44.499 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.499 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.499 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.499 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.236152+0000", 2024-03-30T14:04:44.499 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.121068366 2024-03-30T14:04:44.499 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.499 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.499 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:44.500 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.236311+0000", 2024-03-30T14:04:44.500 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00015839200000000001 2024-03-30T14:04:44.500 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.500 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.500 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1,3", 2024-03-30T14:04:44.500 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.236531+0000", 2024-03-30T14:04:44.500 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00022019100000000001 2024-03-30T14:04:44.500 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.500 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.500 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.500 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.242403+0000", 2024-03-30T14:04:44.500 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0058721650000000004 2024-03-30T14:04:44.500 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.500 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.501 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.501 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.246781+0000", 2024-03-30T14:04:44.501 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0043782919999999998 2024-03-30T14:04:44.501 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.501 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.501 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:44.501 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.247471+0000", 2024-03-30T14:04:44.501 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00068995500000000002 2024-03-30T14:04:44.501 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.501 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.501 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.501 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.251687+0000", 2024-03-30T14:04:44.501 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0042156520000000003 2024-03-30T14:04:44.501 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.501 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.502 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:44.502 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.251757+0000", 2024-03-30T14:04:44.502 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.0210999999999996e-05 2024-03-30T14:04:44.502 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.502 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.502 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:44.502 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.251879+0000", 2024-03-30T14:04:44.502 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012154699999999999 2024-03-30T14:04:44.502 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.502 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.502 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.502 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.502 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.502 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:110107 11.a 11:57844183:::benchmark_data_smithi043_197439_object110106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-30T14:04:44.503 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:02:48.144878+0000", 2024-03-30T14:04:44.503 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 116.321659708, 2024-03-30T14:04:44.503 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.0845247649999998, 2024-03-30T14:04:44.503 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:44.503 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:44.503 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:44.503 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:44.503 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:44.503 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:44.503 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 110107 2024-03-30T14:04:44.503 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.503 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:44.503 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.503 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:44.503 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.144878+0000", 2024-03-30T14:04:44.504 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.504 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.504 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.504 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:44.504 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.144878+0000", 2024-03-30T14:04:44.504 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.504 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.504 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.504 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:44.504 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.144879+0000", 2024-03-30T14:04:44.504 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.7499999999999999e-07 2024-03-30T14:04:44.504 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.504 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.504 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:44.504 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.144927+0000", 2024-03-30T14:04:44.505 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8639e-05 2024-03-30T14:04:44.505 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.505 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.505 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:44.505 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.144939+0000", 2024-03-30T14:04:44.505 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1259e-05 2024-03-30T14:04:44.505 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.505 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.505 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:44.505 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.144944+0000", 2024-03-30T14:04:44.505 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8050000000000003e-06 2024-03-30T14:04:44.505 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.505 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.505 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.506 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.145002+0000", 2024-03-30T14:04:44.506 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8087000000000001e-05 2024-03-30T14:04:44.506 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.506 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.506 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.506 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.145013+0000", 2024-03-30T14:04:44.506 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0169999999999999e-05 2024-03-30T14:04:44.506 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.506 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.506 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.506 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.104911+0000", 2024-03-30T14:04:44.506 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.95989794900000003 2024-03-30T14:04:44.506 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.507 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.507 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.507 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.104921+0000", 2024-03-30T14:04:44.507 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0519e-05 2024-03-30T14:04:44.507 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.507 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.507 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.507 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.219171+0000", 2024-03-30T14:04:44.507 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1142501739999999 2024-03-30T14:04:44.507 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.507 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.507 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:44.507 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.219296+0000", 2024-03-30T14:04:44.508 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012501799999999999 2024-03-30T14:04:44.508 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.508 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.508 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3,5", 2024-03-30T14:04:44.508 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.219460+0000", 2024-03-30T14:04:44.508 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00016398399999999999 2024-03-30T14:04:44.508 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.508 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.508 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:44.508 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.223540+0000", 2024-03-30T14:04:44.508 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0040801530000000004 2024-03-30T14:04:44.508 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.509 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.509 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.509 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.223570+0000", 2024-03-30T14:04:44.509 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.9121000000000001e-05 2024-03-30T14:04:44.509 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.509 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.509 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.509 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.228825+0000", 2024-03-30T14:04:44.509 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0052557810000000002 2024-03-30T14:04:44.509 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.509 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.509 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.509 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.229288+0000", 2024-03-30T14:04:44.509 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000462743 2024-03-30T14:04:44.510 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.510 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.510 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:44.510 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.229337+0000", 2024-03-30T14:04:44.510 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8906999999999998e-05 2024-03-30T14:04:44.510 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.510 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.510 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:44.510 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.229403+0000", 2024-03-30T14:04:44.510 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5680999999999997e-05 2024-03-30T14:04:44.510 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.510 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.510 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.511 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.511 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.511 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:110112 11.1 11:8500eba9:::benchmark_data_smithi043_197439_object110111:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-30T14:04:44.511 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:02:48.150126+0000", 2024-03-30T14:04:44.511 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 116.316412003, 2024-03-30T14:04:44.511 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.1015109609999998, 2024-03-30T14:04:44.511 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:44.511 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:44.511 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:44.511 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:44.511 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:44.511 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:44.511 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 110112 2024-03-30T14:04:44.511 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.511 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:44.512 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.512 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:44.512 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150126+0000", 2024-03-30T14:04:44.512 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.512 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.512 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.512 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:44.512 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150126+0000", 2024-03-30T14:04:44.512 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.512 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.512 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.512 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:44.512 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150126+0000", 2024-03-30T14:04:44.512 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8199999999999999e-07 2024-03-30T14:04:44.512 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.513 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.513 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:44.513 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150152+0000", 2024-03-30T14:04:44.513 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.5491999999999999e-05 2024-03-30T14:04:44.513 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.513 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.513 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:44.513 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150162+0000", 2024-03-30T14:04:44.513 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.8609999999999993e-06 2024-03-30T14:04:44.513 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.513 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.513 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:44.513 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150167+0000", 2024-03-30T14:04:44.514 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5809999999999996e-06 2024-03-30T14:04:44.514 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.514 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.514 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.514 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150209+0000", 2024-03-30T14:04:44.514 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1337000000000001e-05 2024-03-30T14:04:44.514 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.514 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.514 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.514 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150216+0000", 2024-03-30T14:04:44.514 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1919999999999999e-06 2024-03-30T14:04:44.514 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.514 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.514 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.515 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.118060+0000", 2024-03-30T14:04:44.515 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.96784374200000001 2024-03-30T14:04:44.515 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.515 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.515 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.515 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.118069+0000", 2024-03-30T14:04:44.515 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.6099999999999995e-06 2024-03-30T14:04:44.515 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.515 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.515 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.515 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.225375+0000", 2024-03-30T14:04:44.515 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.107305515 2024-03-30T14:04:44.515 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.515 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.515 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:44.516 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.225460+0000", 2024-03-30T14:04:44.516 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.5685000000000004e-05 2024-03-30T14:04:44.516 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.516 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.516 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2,3", 2024-03-30T14:04:44.516 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.225561+0000", 2024-03-30T14:04:44.516 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000100548 2024-03-30T14:04:44.516 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.516 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.516 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:44.516 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.226550+0000", 2024-03-30T14:04:44.516 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00098938200000000002 2024-03-30T14:04:44.516 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.516 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.516 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.517 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.235754+0000", 2024-03-30T14:04:44.517 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0092036410000000003 2024-03-30T14:04:44.517 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.517 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.517 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.517 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.235916+0000", 2024-03-30T14:04:44.517 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000161862 2024-03-30T14:04:44.517 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.517 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.517 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.517 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.251457+0000", 2024-03-30T14:04:44.517 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.015540867 2024-03-30T14:04:44.517 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.517 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.518 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:44.518 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.251518+0000", 2024-03-30T14:04:44.518 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.1443000000000001e-05 2024-03-30T14:04:44.518 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.518 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.518 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:44.518 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.251637+0000", 2024-03-30T14:04:44.518 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000118421 2024-03-30T14:04:44.518 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.518 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.518 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.518 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.519 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.519 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:110113 11.a 11:5a34f9c9:::benchmark_data_smithi043_197439_object110112:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-30T14:04:44.519 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:02:48.150260+0000", 2024-03-30T14:04:44.519 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 116.316277755, 2024-03-30T14:04:44.519 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.0793579310000001, 2024-03-30T14:04:44.519 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:44.519 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:44.519 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:44.519 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:44.519 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:44.519 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:44.519 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 110113 2024-03-30T14:04:44.519 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.519 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:44.520 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.520 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:44.520 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150260+0000", 2024-03-30T14:04:44.520 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.520 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.520 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.520 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:44.520 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150260+0000", 2024-03-30T14:04:44.520 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.520 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.520 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.520 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:44.520 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150260+0000", 2024-03-30T14:04:44.520 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.7000000000000005e-07 2024-03-30T14:04:44.521 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.521 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.521 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:44.521 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150319+0000", 2024-03-30T14:04:44.521 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.8913000000000003e-05 2024-03-30T14:04:44.521 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.521 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.521 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:44.521 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150326+0000", 2024-03-30T14:04:44.521 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.5370000000000003e-06 2024-03-30T14:04:44.521 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.521 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.521 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:44.521 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150330+0000", 2024-03-30T14:04:44.521 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1320000000000004e-06 2024-03-30T14:04:44.522 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.522 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.522 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.522 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150364+0000", 2024-03-30T14:04:44.522 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.3491999999999997e-05 2024-03-30T14:04:44.522 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.522 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.522 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.522 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150368+0000", 2024-03-30T14:04:44.522 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8960000000000002e-06 2024-03-30T14:04:44.522 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.522 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.522 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.522 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.105244+0000", 2024-03-30T14:04:44.522 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.95487507800000004 2024-03-30T14:04:44.523 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.523 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.523 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.523 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.105254+0000", 2024-03-30T14:04:44.523 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0207000000000001e-05 2024-03-30T14:04:44.523 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.523 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.523 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.523 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.221001+0000", 2024-03-30T14:04:44.523 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.115747555 2024-03-30T14:04:44.523 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.523 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.523 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:44.523 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.221160+0000", 2024-03-30T14:04:44.523 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00015888600000000001 2024-03-30T14:04:44.523 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.524 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.524 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,3,5", 2024-03-30T14:04:44.524 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.221357+0000", 2024-03-30T14:04:44.524 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000196417 2024-03-30T14:04:44.524 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.524 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.524 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.524 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.223610+0000", 2024-03-30T14:04:44.524 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0022538609999999998 2024-03-30T14:04:44.524 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.524 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.524 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:44.524 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.223626+0000", 2024-03-30T14:04:44.524 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5888000000000001e-05 2024-03-30T14:04:44.524 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.525 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.525 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.525 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.228868+0000", 2024-03-30T14:04:44.525 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0052413970000000001 2024-03-30T14:04:44.525 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.525 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.525 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.525 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.229449+0000", 2024-03-30T14:04:44.525 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00058150400000000005 2024-03-30T14:04:44.525 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.525 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.525 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:44.525 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.229510+0000", 2024-03-30T14:04:44.525 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0986000000000002e-05 2024-03-30T14:04:44.525 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.525 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.526 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:44.526 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.229618+0000", 2024-03-30T14:04:44.526 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000107612 2024-03-30T14:04:44.526 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.526 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.526 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.526 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.526 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.526 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:110114 11.9 11:92927427:::benchmark_data_smithi043_197439_object110113:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-30T14:04:44.526 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:02:48.150427+0000", 2024-03-30T14:04:44.526 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 116.316110892, 2024-03-30T14:04:44.526 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.101683693, 2024-03-30T14:04:44.526 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:44.526 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:44.526 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:44.527 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:44.527 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:44.527 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:44.527 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 110114 2024-03-30T14:04:44.527 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.527 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:44.527 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.527 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:44.527 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150427+0000", 2024-03-30T14:04:44.527 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.527 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.527 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.527 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:44.527 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150427+0000", 2024-03-30T14:04:44.527 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.527 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.528 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.528 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:44.528 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150427+0000", 2024-03-30T14:04:44.528 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7700000000000005e-07 2024-03-30T14:04:44.528 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.528 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.528 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:44.528 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150490+0000", 2024-03-30T14:04:44.528 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.2645999999999996e-05 2024-03-30T14:04:44.528 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.528 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.528 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:44.529 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150496+0000", 2024-03-30T14:04:44.530 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9649999999999996e-06 2024-03-30T14:04:44.530 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.530 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.530 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:44.530 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150500+0000", 2024-03-30T14:04:44.530 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1479999999999999e-06 2024-03-30T14:04:44.530 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.530 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.530 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.531 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150537+0000", 2024-03-30T14:04:44.531 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.7443e-05 2024-03-30T14:04:44.531 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.531 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.531 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.531 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:48.150543+0000", 2024-03-30T14:04:44.531 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0270000000000001e-06 2024-03-30T14:04:44.531 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.531 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.531 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.531 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.115110+0000", 2024-03-30T14:04:44.531 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.96456676500000005 2024-03-30T14:04:44.531 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.532 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.532 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.532 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:49.115118+0000", 2024-03-30T14:04:44.532 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.9559999999999997e-06 2024-03-30T14:04:44.532 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.532 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.532 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.532 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.238165+0000", 2024-03-30T14:04:44.532 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1230468689999999 2024-03-30T14:04:44.532 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.532 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.532 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:44.532 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.238311+0000", 2024-03-30T14:04:44.533 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000146138 2024-03-30T14:04:44.533 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.533 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.533 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1,3", 2024-03-30T14:04:44.533 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.238491+0000", 2024-03-30T14:04:44.533 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00018009800000000001 2024-03-30T14:04:44.533 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.533 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.533 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.533 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.244541+0000", 2024-03-30T14:04:44.533 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0060499530000000003 2024-03-30T14:04:44.533 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.533 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.533 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.534 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.246833+0000", 2024-03-30T14:04:44.534 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0022913370000000001 2024-03-30T14:04:44.534 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.534 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.534 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:44.534 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.247479+0000", 2024-03-30T14:04:44.534 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00064684200000000001 2024-03-30T14:04:44.534 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.534 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.534 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.534 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.251936+0000", 2024-03-30T14:04:44.534 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0044561949999999996 2024-03-30T14:04:44.534 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.534 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.535 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:44.535 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.251991+0000", 2024-03-30T14:04:44.535 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.5368999999999998e-05 2024-03-30T14:04:44.535 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.535 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.535 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:44.535 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:02:50.252110+0000", 2024-03-30T14:04:44.535 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000119465 2024-03-30T14:04:44.535 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.535 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.535 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.535 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.535 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.535 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:182437 11.4 11:204a0dee:::benchmark_data_smithi043_197439_object72319:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-03-30T14:04:44.535 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:03:59.167902+0000", 2024-03-30T14:04:44.536 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 45.298635945999997, 2024-03-30T14:04:44.536 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.128523918, 2024-03-30T14:04:44.536 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:44.536 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:44.536 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "reached pg", 2024-03-30T14:04:44.536 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:44.536 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:44.536 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:44.536 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 182437 2024-03-30T14:04:44.536 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.536 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:44.536 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.536 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:44.536 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.167902+0000", 2024-03-30T14:04:44.536 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.537 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.537 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.537 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:44.537 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.167902+0000", 2024-03-30T14:04:44.537 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.537 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.537 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.537 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:44.537 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.167902+0000", 2024-03-30T14:04:44.537 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.5100000000000001e-07 2024-03-30T14:04:44.537 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.537 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.537 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:44.537 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.167904+0000", 2024-03-30T14:04:44.537 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.6020000000000001e-06 2024-03-30T14:04:44.537 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.538 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.538 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:44.538 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.167908+0000", 2024-03-30T14:04:44.538 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.391e-06 2024-03-30T14:04:44.538 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.538 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.538 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:44.538 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.167911+0000", 2024-03-30T14:04:44.538 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.1939999999999998e-06 2024-03-30T14:04:44.538 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.538 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.538 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.538 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.167943+0000", 2024-03-30T14:04:44.538 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.1634999999999998e-05 2024-03-30T14:04:44.538 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.539 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.539 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:44.539 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.168022+0000", 2024-03-30T14:04:44.539 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.9073999999999995e-05 2024-03-30T14:04:44.539 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.539 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.539 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,2", 2024-03-30T14:04:44.539 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.168116+0000", 2024-03-30T14:04:44.539 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4424000000000003e-05 2024-03-30T14:04:44.539 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.539 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.539 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:44.539 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.169657+0000", 2024-03-30T14:04:44.539 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0015411750000000001 2024-03-30T14:04:44.539 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.539 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.540 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.540 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.180673+0000", 2024-03-30T14:04:44.540 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.011015969 2024-03-30T14:04:44.540 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.540 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.540 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.540 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.211113+0000", 2024-03-30T14:04:44.540 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.030439919999999999 2024-03-30T14:04:44.540 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.540 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.540 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.540 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.211121+0000", 2024-03-30T14:04:44.540 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.2540000000000003e-06 2024-03-30T14:04:44.540 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.541 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.541 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.541 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:00.199520+0000", 2024-03-30T14:04:44.541 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.98839983099999995 2024-03-30T14:04:44.541 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.541 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.541 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.541 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:00.199534+0000", 2024-03-30T14:04:44.541 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.3927000000000001e-05 2024-03-30T14:04:44.541 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.541 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.541 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.541 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.296337+0000", 2024-03-30T14:04:44.541 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0968030790000001 2024-03-30T14:04:44.541 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.541 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.542 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:44.542 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.296426+0000", 2024-03-30T14:04:44.542 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.8191999999999999e-05 2024-03-30T14:04:44.542 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.542 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.542 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.542 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.542 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.542 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:182463 11.1 11:8f2bd304:::benchmark_data_smithi043_197439_object72345:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-03-30T14:04:44.542 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:03:59.203940+0000", 2024-03-30T14:04:44.542 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 45.262597589999999, 2024-03-30T14:04:44.542 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.1203813249999999, 2024-03-30T14:04:44.542 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:44.542 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:44.542 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:44.543 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:44.543 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:44.543 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:44.543 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 182463 2024-03-30T14:04:44.543 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.543 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:44.543 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.543 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:44.543 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.203940+0000", 2024-03-30T14:04:44.543 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.543 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.543 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.543 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:44.543 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.203940+0000", 2024-03-30T14:04:44.543 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.544 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.544 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.544 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:44.544 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.203941+0000", 2024-03-30T14:04:44.544 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.5200000000000002e-07 2024-03-30T14:04:44.544 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.544 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.544 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:44.544 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.203943+0000", 2024-03-30T14:04:44.544 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.8760000000000001e-06 2024-03-30T14:04:44.544 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.544 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.544 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:44.545 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.203950+0000", 2024-03-30T14:04:44.545 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.545e-06 2024-03-30T14:04:44.545 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.545 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.545 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:44.545 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.203955+0000", 2024-03-30T14:04:44.545 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2240000000000003e-06 2024-03-30T14:04:44.545 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.545 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.545 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.545 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.209702+0000", 2024-03-30T14:04:44.545 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0057466219999999998 2024-03-30T14:04:44.545 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.545 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.545 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.546 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.209723+0000", 2024-03-30T14:04:44.546 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.1376e-05 2024-03-30T14:04:44.546 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.546 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.546 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.546 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:00.199573+0000", 2024-03-30T14:04:44.546 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.98984973700000001 2024-03-30T14:04:44.546 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.546 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.546 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.546 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:00.199582+0000", 2024-03-30T14:04:44.546 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.5689999999999994e-06 2024-03-30T14:04:44.546 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.547 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.547 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.547 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.307742+0000", 2024-03-30T14:04:44.547 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1081594770000001 2024-03-30T14:04:44.547 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.547 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.547 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:44.547 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.307877+0000", 2024-03-30T14:04:44.547 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000135007 2024-03-30T14:04:44.547 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.547 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.547 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,2,4", 2024-03-30T14:04:44.547 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.308269+0000", 2024-03-30T14:04:44.547 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00039242500000000001 2024-03-30T14:04:44.547 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.548 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.548 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.548 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.314142+0000", 2024-03-30T14:04:44.548 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.005873016 2024-03-30T14:04:44.548 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.548 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.548 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:44.548 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.317841+0000", 2024-03-30T14:04:44.548 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0036985429999999999 2024-03-30T14:04:44.548 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.548 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.548 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.548 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.317883+0000", 2024-03-30T14:04:44.548 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1780000000000003e-05 2024-03-30T14:04:44.549 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.549 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.549 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.549 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.324123+0000", 2024-03-30T14:04:44.549 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0062404410000000002 2024-03-30T14:04:44.549 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.549 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.549 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:44.549 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.324183+0000", 2024-03-30T14:04:44.549 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0299e-05 2024-03-30T14:04:44.549 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.549 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.549 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:44.549 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.324321+0000", 2024-03-30T14:04:44.549 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013793599999999999 2024-03-30T14:04:44.549 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.550 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.550 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.550 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.550 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.550 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:182467 11.4 11:2ef1e35a:::benchmark_data_smithi043_197439_object72349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-03-30T14:04:44.550 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:03:59.207480+0000", 2024-03-30T14:04:44.550 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 45.259057542999997, 2024-03-30T14:04:44.550 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.137555366, 2024-03-30T14:04:44.550 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:44.550 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:44.550 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:44.550 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:44.550 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:44.551 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:44.551 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 182467 2024-03-30T14:04:44.551 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.551 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:44.551 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.551 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:44.551 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.207480+0000", 2024-03-30T14:04:44.551 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.551 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.551 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.551 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:44.551 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.207480+0000", 2024-03-30T14:04:44.551 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.551 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.551 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.551 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:44.552 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.207481+0000", 2024-03-30T14:04:44.552 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.167e-06 2024-03-30T14:04:44.552 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.552 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.552 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:44.552 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.207486+0000", 2024-03-30T14:04:44.552 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.3089999999999997e-06 2024-03-30T14:04:44.552 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.552 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.552 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:44.552 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.207495+0000", 2024-03-30T14:04:44.552 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4390000000000006e-06 2024-03-30T14:04:44.552 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.552 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.552 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:44.552 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.207526+0000", 2024-03-30T14:04:44.553 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.1129e-05 2024-03-30T14:04:44.553 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.553 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.553 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.553 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.211315+0000", 2024-03-30T14:04:44.553 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0037886510000000001 2024-03-30T14:04:44.553 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.553 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.553 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.553 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.211324+0000", 2024-03-30T14:04:44.553 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.6409999999999997e-06 2024-03-30T14:04:44.553 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.553 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.553 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.553 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:00.199565+0000", 2024-03-30T14:04:44.554 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.98824088200000004 2024-03-30T14:04:44.554 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.554 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.554 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.554 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:00.199574+0000", 2024-03-30T14:04:44.554 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.4169999999999992e-06 2024-03-30T14:04:44.554 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.554 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.554 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.554 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.296634+0000", 2024-03-30T14:04:44.554 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0970597930000001 2024-03-30T14:04:44.554 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.554 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.554 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:44.554 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.296767+0000", 2024-03-30T14:04:44.555 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000132986 2024-03-30T14:04:44.555 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.555 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.555 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1,2", 2024-03-30T14:04:44.555 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.297141+0000", 2024-03-30T14:04:44.555 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00037475900000000001 2024-03-30T14:04:44.555 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.555 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.555 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.555 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.305460+0000", 2024-03-30T14:04:44.555 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0083189289999999992 2024-03-30T14:04:44.555 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.555 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.555 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:44.555 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.305469+0000", 2024-03-30T14:04:44.555 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.9949999999999994e-06 2024-03-30T14:04:44.556 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.556 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.556 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.556 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.308013+0000", 2024-03-30T14:04:44.556 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0025435010000000001 2024-03-30T14:04:44.556 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.556 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.556 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.556 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.344848+0000", 2024-03-30T14:04:44.556 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.036835589000000002 2024-03-30T14:04:44.556 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.556 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.556 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:44.556 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.344908+0000", 2024-03-30T14:04:44.556 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.9222000000000003e-05 2024-03-30T14:04:44.557 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.557 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.557 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:44.557 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.345035+0000", 2024-03-30T14:04:44.557 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00012795699999999999 2024-03-30T14:04:44.557 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.557 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.557 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.557 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.557 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.557 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:182485 11.4 11:2f4c1cd1:::benchmark_data_smithi043_197439_object72367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-03-30T14:04:44.557 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:03:59.255375+0000", 2024-03-30T14:04:44.557 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 45.211162561999998, 2024-03-30T14:04:44.557 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.092445412, 2024-03-30T14:04:44.557 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:44.557 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:44.558 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:44.558 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:44.558 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:44.558 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:44.558 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 182485 2024-03-30T14:04:44.558 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.558 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:44.558 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.558 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:44.558 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.255375+0000", 2024-03-30T14:04:44.558 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.558 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.558 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.558 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:44.558 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.255375+0000", 2024-03-30T14:04:44.559 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.559 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.559 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.559 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:44.559 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.255376+0000", 2024-03-30T14:04:44.559 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.78e-07 2024-03-30T14:04:44.559 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.559 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.559 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:44.559 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.255380+0000", 2024-03-30T14:04:44.559 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.873e-06 2024-03-30T14:04:44.559 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.559 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.559 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:44.559 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.255389+0000", 2024-03-30T14:04:44.559 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4050000000000006e-06 2024-03-30T14:04:44.560 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.560 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.560 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:44.560 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.255395+0000", 2024-03-30T14:04:44.560 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.4489999999999998e-06 2024-03-30T14:04:44.560 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.560 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.560 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.560 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.255449+0000", 2024-03-30T14:04:44.560 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3733000000000002e-05 2024-03-30T14:04:44.560 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.560 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.560 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.560 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.255456+0000", 2024-03-30T14:04:44.560 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.593e-06 2024-03-30T14:04:44.561 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.561 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.561 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.561 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:00.199603+0000", 2024-03-30T14:04:44.561 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.94414686599999997 2024-03-30T14:04:44.561 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.561 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.561 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.561 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:00.199612+0000", 2024-03-30T14:04:44.561 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.7010000000000002e-06 2024-03-30T14:04:44.561 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.561 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.561 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.561 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.302425+0000", 2024-03-30T14:04:44.561 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1028133419999999 2024-03-30T14:04:44.562 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.562 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.562 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:44.562 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.302512+0000", 2024-03-30T14:04:44.562 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.7447000000000005e-05 2024-03-30T14:04:44.562 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.562 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.562 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1,2", 2024-03-30T14:04:44.562 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.302599+0000", 2024-03-30T14:04:44.562 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6716000000000001e-05 2024-03-30T14:04:44.562 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.562 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.562 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:44.562 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.305473+0000", 2024-03-30T14:04:44.563 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0028734469999999999 2024-03-30T14:04:44.563 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.563 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.563 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.563 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.306475+0000", 2024-03-30T14:04:44.563 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0010022029999999999 2024-03-30T14:04:44.563 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.563 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.563 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.563 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.308056+0000", 2024-03-30T14:04:44.563 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001581378 2024-03-30T14:04:44.563 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.563 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.563 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.564 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.347626+0000", 2024-03-30T14:04:44.564 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.039569629000000002 2024-03-30T14:04:44.564 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.564 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.564 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:44.564 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.347690+0000", 2024-03-30T14:04:44.564 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.3954999999999993e-05 2024-03-30T14:04:44.564 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.564 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.564 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:44.564 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.347820+0000", 2024-03-30T14:04:44.564 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013069700000000001 2024-03-30T14:04:44.564 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.565 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.565 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.565 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.565 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.565 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:182497 11.4 11:2002e838:::benchmark_data_smithi043_197439_object72379:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-03-30T14:04:44.565 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:03:59.278427+0000", 2024-03-30T14:04:44.565 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 45.188110795999997, 2024-03-30T14:04:44.565 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.084599018, 2024-03-30T14:04:44.565 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:44.565 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:44.565 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:04:44.565 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:44.565 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:44.566 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:44.566 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 182497 2024-03-30T14:04:44.566 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.566 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:44.566 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.566 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:44.566 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.278427+0000", 2024-03-30T14:04:44.566 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.566 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.566 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.566 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:44.566 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.278427+0000", 2024-03-30T14:04:44.566 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.566 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.567 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.567 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:44.567 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.278428+0000", 2024-03-30T14:04:44.567 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.8100000000000001e-07 2024-03-30T14:04:44.567 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.567 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.567 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:44.567 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.278432+0000", 2024-03-30T14:04:44.567 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.0579999999999999e-06 2024-03-30T14:04:44.567 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.567 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.567 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:44.567 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.278443+0000", 2024-03-30T14:04:44.567 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1469e-05 2024-03-30T14:04:44.568 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.568 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.568 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:44.568 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.278453+0000", 2024-03-30T14:04:44.568 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4439999999999997e-06 2024-03-30T14:04:44.568 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.568 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.568 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.568 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.278498+0000", 2024-03-30T14:04:44.568 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.5664000000000002e-05 2024-03-30T14:04:44.568 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.568 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.568 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.568 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:03:59.278506+0000", 2024-03-30T14:04:44.568 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.2169999999999997e-06 2024-03-30T14:04:44.569 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.569 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.569 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.569 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:00.199639+0000", 2024-03-30T14:04:44.569 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.92113316300000003 2024-03-30T14:04:44.569 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.569 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.569 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.569 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:00.199647+0000", 2024-03-30T14:04:44.569 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.4319999999999999e-06 2024-03-30T14:04:44.569 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.569 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.569 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.570 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.304023+0000", 2024-03-30T14:04:44.570 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.104375804 2024-03-30T14:04:44.570 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.570 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.570 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:44.570 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.304133+0000", 2024-03-30T14:04:44.570 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00010961800000000001 2024-03-30T14:04:44.570 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.570 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.570 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 0,1,2", 2024-03-30T14:04:44.570 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.304211+0000", 2024-03-30T14:04:44.570 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8103999999999996e-05 2024-03-30T14:04:44.570 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.570 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.570 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:44.571 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.305478+0000", 2024-03-30T14:04:44.571 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0012671609999999999 2024-03-30T14:04:44.571 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.571 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.571 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.571 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.306529+0000", 2024-03-30T14:04:44.571 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0010507590000000001 2024-03-30T14:04:44.571 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.571 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.571 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.571 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.308104+0000", 2024-03-30T14:04:44.571 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001575024 2024-03-30T14:04:44.571 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.571 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.572 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:04:44.572 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.362834+0000", 2024-03-30T14:04:44.572 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.054730503 2024-03-30T14:04:44.572 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.572 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.572 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "commit_sent", 2024-03-30T14:04:44.572 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.362894+0000", 2024-03-30T14:04:44.572 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.0248000000000003e-05 2024-03-30T14:04:44.572 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.572 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.572 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:44.572 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:01.363026+0000", 2024-03-30T14:04:44.572 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000131369 2024-03-30T14:04:44.572 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.573 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.573 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.573 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.573 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.573 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:217749 11.d 11:b8424846:::benchmark_data_smithi043_197439_object107631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-03-30T14:04:44.573 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:04:31.135015+0000", 2024-03-30T14:04:44.573 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 13.331522548000001, 2024-03-30T14:04:44.573 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.087867876, 2024-03-30T14:04:44.573 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:44.573 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:44.573 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "reached pg", 2024-03-30T14:04:44.573 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:44.573 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:44.573 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:44.573 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 217749 2024-03-30T14:04:44.574 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.574 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:44.574 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.574 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:44.574 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.135015+0000", 2024-03-30T14:04:44.574 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.574 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.574 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.574 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:44.574 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.135015+0000", 2024-03-30T14:04:44.574 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.574 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.575 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.575 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:44.575 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.135016+0000", 2024-03-30T14:04:44.575 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1399999999999997e-07 2024-03-30T14:04:44.575 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.575 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.575 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:44.575 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.135018+0000", 2024-03-30T14:04:44.575 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.39e-06 2024-03-30T14:04:44.575 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.575 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.575 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:44.575 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.135025+0000", 2024-03-30T14:04:44.575 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.4410000000000004e-06 2024-03-30T14:04:44.575 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.576 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.576 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:44.576 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.135031+0000", 2024-03-30T14:04:44.576 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.3720000000000004e-06 2024-03-30T14:04:44.576 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.576 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.576 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.576 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.138370+0000", 2024-03-30T14:04:44.576 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0033396110000000001 2024-03-30T14:04:44.576 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.576 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.576 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:44.576 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.138508+0000", 2024-03-30T14:04:44.576 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013712199999999999 2024-03-30T14:04:44.577 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.577 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.577 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,2,4", 2024-03-30T14:04:44.577 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.138641+0000", 2024-03-30T14:04:44.577 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00013376100000000001 2024-03-30T14:04:44.577 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.577 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.577 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op_commit", 2024-03-30T14:04:44.577 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.143117+0000", 2024-03-30T14:04:44.577 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0044760030000000001 2024-03-30T14:04:44.577 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.577 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.577 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.577 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.150017+0000", 2024-03-30T14:04:44.578 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0068997279999999999 2024-03-30T14:04:44.578 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.578 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.578 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.578 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.150045+0000", 2024-03-30T14:04:44.578 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.7523999999999999e-05 2024-03-30T14:04:44.578 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.578 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.578 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.578 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:32.222867+0000", 2024-03-30T14:04:44.578 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0728221870000001 2024-03-30T14:04:44.578 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.579 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.579 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:44.579 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:32.222883+0000", 2024-03-30T14:04:44.579 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.6223e-05 2024-03-30T14:04:44.579 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.579 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.579 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.579 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.579 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.579 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:217751 11.d 11:bbd87cba:::benchmark_data_smithi043_197439_object107633:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-03-30T14:04:44.579 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:04:31.135155+0000", 2024-03-30T14:04:44.579 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 13.331382441000001, 2024-03-30T14:04:44.579 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0877582260000001, 2024-03-30T14:04:44.580 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:44.580 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:44.580 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "reached pg", 2024-03-30T14:04:44.580 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:44.580 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:44.580 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:44.580 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 217751 2024-03-30T14:04:44.580 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.580 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:44.580 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.580 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:44.580 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.135155+0000", 2024-03-30T14:04:44.581 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.581 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.581 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.581 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:44.581 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.135155+0000", 2024-03-30T14:04:44.581 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.581 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.581 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.581 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:44.581 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.135156+0000", 2024-03-30T14:04:44.581 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.7199999999999999e-07 2024-03-30T14:04:44.581 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.582 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.582 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:44.582 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.135158+0000", 2024-03-30T14:04:44.582 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.7650000000000002e-06 2024-03-30T14:04:44.582 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.582 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.582 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:44.582 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.135165+0000", 2024-03-30T14:04:44.582 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.7379999999999997e-06 2024-03-30T14:04:44.582 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.582 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.582 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:44.582 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.135170+0000", 2024-03-30T14:04:44.582 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2739999999999998e-06 2024-03-30T14:04:44.583 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.583 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.583 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.583 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.140827+0000", 2024-03-30T14:04:44.583 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0056561299999999997 2024-03-30T14:04:44.583 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.583 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.583 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:44.583 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.140970+0000", 2024-03-30T14:04:44.583 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.000142977 2024-03-30T14:04:44.583 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.583 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.583 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for subops from 1,2,4", 2024-03-30T14:04:44.584 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.141133+0000", 2024-03-30T14:04:44.584 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00016374 2024-03-30T14:04:44.584 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.584 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.584 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.584 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.162708+0000", 2024-03-30T14:04:44.584 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.021574646999999999 2024-03-30T14:04:44.584 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.584 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.584 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.584 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.162719+0000", 2024-03-30T14:04:44.584 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.1404e-05 2024-03-30T14:04:44.584 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.584 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.584 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.584 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:32.222907+0000", 2024-03-30T14:04:44.585 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.060187255 2024-03-30T14:04:44.585 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.585 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.585 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:44.585 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:32.222913+0000", 2024-03-30T14:04:44.585 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.8240000000000002e-06 2024-03-30T14:04:44.585 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.585 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.585 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.585 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.585 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.585 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:217766 11.d 11.1cccebad (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-03-30T14:04:44.585 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:04:31.149790+0000", 2024-03-30T14:04:44.585 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 13.316747696, 2024-03-30T14:04:44.585 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0731429210000001, 2024-03-30T14:04:44.586 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:44.586 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:44.586 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "reached pg", 2024-03-30T14:04:44.586 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:44.586 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:44.586 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:44.586 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 217766 2024-03-30T14:04:44.586 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.586 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:44.586 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.586 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:44.586 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.149790+0000", 2024-03-30T14:04:44.586 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.586 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.586 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.587 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:44.587 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.149790+0000", 2024-03-30T14:04:44.587 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.587 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.587 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.587 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:44.587 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.149790+0000", 2024-03-30T14:04:44.587 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.6699999999999999e-07 2024-03-30T14:04:44.587 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.587 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.587 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:44.587 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.149793+0000", 2024-03-30T14:04:44.587 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.8729999999999999e-06 2024-03-30T14:04:44.587 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.587 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.588 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:44.588 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.149800+0000", 2024-03-30T14:04:44.588 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1069999999999999e-06 2024-03-30T14:04:44.588 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.588 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.588 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:44.588 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.149807+0000", 2024-03-30T14:04:44.588 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.2419999999999997e-06 2024-03-30T14:04:44.588 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.588 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.588 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.588 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.162798+0000", 2024-03-30T14:04:44.588 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.012991758000000001 2024-03-30T14:04:44.588 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.589 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.589 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.589 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.162806+0000", 2024-03-30T14:04:44.589 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.8399999999999995e-06 2024-03-30T14:04:44.589 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.589 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.589 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.589 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:32.222928+0000", 2024-03-30T14:04:44.589 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0601213979999999 2024-03-30T14:04:44.589 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.589 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.589 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:44.589 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:32.222933+0000", 2024-03-30T14:04:44.589 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2360000000000004e-06 2024-03-30T14:04:44.589 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.589 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.590 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.590 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.590 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.590 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:217770 11.d 11.df5adb8d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-03-30T14:04:44.590 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:04:31.153377+0000", 2024-03-30T14:04:44.590 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 13.313160183000001, 2024-03-30T14:04:44.590 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.069573248, 2024-03-30T14:04:44.590 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:44.590 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:44.590 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "reached pg", 2024-03-30T14:04:44.590 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:44.590 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:44.590 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:44.590 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 217770 2024-03-30T14:04:44.591 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.591 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:44.591 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.591 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:44.591 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.153377+0000", 2024-03-30T14:04:44.591 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.591 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.591 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.591 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:44.591 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.153377+0000", 2024-03-30T14:04:44.591 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.591 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.591 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.592 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:44.592 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.153379+0000", 2024-03-30T14:04:44.592 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.074e-06 2024-03-30T14:04:44.592 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.592 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.592 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:44.592 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.153384+0000", 2024-03-30T14:04:44.592 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.1250000000000001e-06 2024-03-30T14:04:44.592 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.592 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.592 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:44.592 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.153394+0000", 2024-03-30T14:04:44.592 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0529e-05 2024-03-30T14:04:44.592 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.592 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.592 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:44.593 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.153402+0000", 2024-03-30T14:04:44.593 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.0569999999999993e-06 2024-03-30T14:04:44.593 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.593 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.593 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.593 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.162830+0000", 2024-03-30T14:04:44.593 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.009428021 2024-03-30T14:04:44.593 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.593 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.593 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.593 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.162837+0000", 2024-03-30T14:04:44.593 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.2010000000000002e-06 2024-03-30T14:04:44.593 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.593 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.593 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.594 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:32.222946+0000", 2024-03-30T14:04:44.594 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.060108346 2024-03-30T14:04:44.594 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.594 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.594 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:44.594 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:32.222951+0000", 2024-03-30T14:04:44.594 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.8949999999999997e-06 2024-03-30T14:04:44.594 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.594 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.594 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.594 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.594 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.594 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:217777 11.d 11.ed69efcd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-03-30T14:04:44.594 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:04:31.169206+0000", 2024-03-30T14:04:44.594 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 13.297331295999999, 2024-03-30T14:04:44.595 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0537664499999999, 2024-03-30T14:04:44.595 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:44.595 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:44.595 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "reached pg", 2024-03-30T14:04:44.595 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:44.595 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:44.595 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:44.595 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 217777 2024-03-30T14:04:44.595 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.595 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:44.595 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.595 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:44.595 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.169206+0000", 2024-03-30T14:04:44.595 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.596 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.596 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.596 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:44.596 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.169206+0000", 2024-03-30T14:04:44.596 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.596 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.596 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.596 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:44.596 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.169207+0000", 2024-03-30T14:04:44.596 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.5099999999999998e-07 2024-03-30T14:04:44.596 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.596 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.596 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:44.596 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.169212+0000", 2024-03-30T14:04:44.596 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0300000000000001e-06 2024-03-30T14:04:44.597 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.597 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.597 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:44.597 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.169228+0000", 2024-03-30T14:04:44.597 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5242000000000001e-05 2024-03-30T14:04:44.597 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.597 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.597 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:44.597 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.169235+0000", 2024-03-30T14:04:44.597 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1999999999999997e-06 2024-03-30T14:04:44.597 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.597 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.597 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.597 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.169302+0000", 2024-03-30T14:04:44.597 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6921000000000005e-05 2024-03-30T14:04:44.597 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.598 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.598 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.598 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.169312+0000", 2024-03-30T14:04:44.598 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0026e-05 2024-03-30T14:04:44.598 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.598 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.598 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.598 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:32.222964+0000", 2024-03-30T14:04:44.598 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.053652394 2024-03-30T14:04:44.598 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.598 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.598 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:44.599 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:32.222973+0000", 2024-03-30T14:04:44.599 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.6859999999999995e-06 2024-03-30T14:04:44.599 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.599 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.599 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.599 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.599 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.599 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "osd_op(client.25685.0:217779 11.d 11.bebaa49d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-03-30T14:04:44.599 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:04:31.169602+0000", 2024-03-30T14:04:44.599 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 13.296935781, 2024-03-30T14:04:44.599 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0534038640000001, 2024-03-30T14:04:44.599 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:44.600 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:44.600 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "reached pg", 2024-03-30T14:04:44.600 INFO:teuthology.orchestra.run.smithi113.stdout: "client_info": { 2024-03-30T14:04:44.600 INFO:teuthology.orchestra.run.smithi113.stdout: "client": "client.25685", 2024-03-30T14:04:44.600 INFO:teuthology.orchestra.run.smithi113.stdout: "client_addr": "172.21.15.43:0/2184404996", 2024-03-30T14:04:44.600 INFO:teuthology.orchestra.run.smithi113.stdout: "tid": 217779 2024-03-30T14:04:44.600 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.600 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:44.600 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.600 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:44.600 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.169602+0000", 2024-03-30T14:04:44.600 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.600 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.600 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.600 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:44.601 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.169602+0000", 2024-03-30T14:04:44.601 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:44.601 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.601 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.601 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:44.601 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.169603+0000", 2024-03-30T14:04:44.601 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 8.1200000000000002e-07 2024-03-30T14:04:44.601 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.601 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.601 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:44.601 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.169607+0000", 2024-03-30T14:04:44.601 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.1989999999999999e-06 2024-03-30T14:04:44.601 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.602 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.602 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:44.602 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.169616+0000", 2024-03-30T14:04:44.602 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.4080000000000004e-06 2024-03-30T14:04:44.602 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.602 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.602 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:44.602 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.169623+0000", 2024-03-30T14:04:44.602 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.6950000000000003e-06 2024-03-30T14:04:44.602 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.602 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.602 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.602 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.169673+0000", 2024-03-30T14:04:44.603 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.0198999999999998e-05 2024-03-30T14:04:44.603 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.603 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.603 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "waiting for peered", 2024-03-30T14:04:44.603 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:31.169706+0000", 2024-03-30T14:04:44.603 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 3.2333999999999999e-05 2024-03-30T14:04:44.603 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.603 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.603 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:44.603 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:32.222996+0000", 2024-03-30T14:04:44.603 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.053290251 2024-03-30T14:04:44.603 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:44.603 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:44.603 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:44.603 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:04:32.223006+0000", 2024-03-30T14:04:44.604 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 9.9660000000000008e-06 2024-03-30T14:04:44.604 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.604 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.604 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.604 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:44.604 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:44.604 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T14:04:45.199 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T14:04:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:45 smithi113 ceph-mon[34341]: pgmap v1029: 152 pgs: 152 active+clean; 582 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 614 B/s rd, 0 op/s 2024-03-30T14:04:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:45 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1170658266' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 0.3311352474105644}]': finished 2024-03-30T14:04:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:45 smithi113 ceph-mon[34341]: osdmap e470: 8 total, 8 up, 5 in 2024-03-30T14:04:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:45 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2926649829' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:04:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:45 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:04:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:45 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:45.586 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:45 smithi043 ceph-mon[29963]: pgmap v1029: 152 pgs: 152 active+clean; 582 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 614 B/s rd, 0 op/s 2024-03-30T14:04:45.586 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:45 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1170658266' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 0.3311352474105644}]': finished 2024-03-30T14:04:45.586 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:45 smithi043 ceph-mon[29963]: osdmap e470: 8 total, 8 up, 5 in 2024-03-30T14:04:45.586 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:45 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2926649829' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:04:45.586 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:45 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:04:45.586 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:45 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:45.586 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:45 smithi043 ceph-mon[42099]: pgmap v1029: 152 pgs: 152 active+clean; 582 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 614 B/s rd, 0 op/s 2024-03-30T14:04:45.586 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:45 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1170658266' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 0.3311352474105644}]': finished 2024-03-30T14:04:45.586 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:45 smithi043 ceph-mon[42099]: osdmap e470: 8 total, 8 up, 5 in 2024-03-30T14:04:45.587 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:45 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2926649829' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:04:45.587 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:45 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:04:45.587 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:45 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:45.603 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-30T14:04:45.848 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:04:46.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:46 smithi113 ceph-mon[34341]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:04:46.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:46 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2926649829' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:04:46.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:46 smithi113 ceph-mon[34341]: osdmap e471: 8 total, 8 up, 5 in 2024-03-30T14:04:46.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:46 smithi113 ceph-mon[34341]: pgmap v1032: 152 pgs: 152 active+clean; 582 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:04:46.391 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:04:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:04:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:04:46.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:46 smithi043 ceph-mon[29963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:04:46.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:46 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2926649829' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:04:46.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:46 smithi043 ceph-mon[29963]: osdmap e471: 8 total, 8 up, 5 in 2024-03-30T14:04:46.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:46 smithi043 ceph-mon[29963]: pgmap v1032: 152 pgs: 152 active+clean; 582 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:04:46.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:46 smithi043 ceph-mon[42099]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:04:46.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:46 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2926649829' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:04:46.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:46 smithi043 ceph-mon[42099]: osdmap e471: 8 total, 8 up, 5 in 2024-03-30T14:04:46.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:46 smithi043 ceph-mon[42099]: pgmap v1032: 152 pgs: 152 active+clean; 582 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:04:47.302 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:47 smithi113 ceph-mon[34341]: osdmap e472: 8 total, 8 up, 5 in 2024-03-30T14:04:47.539 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T14:04:47.540 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [], 2024-03-30T14:04:47.540 INFO:teuthology.orchestra.run.smithi113.stdout: "num_ops": 0 2024-03-30T14:04:47.540 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T14:04:47.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:47 smithi043 ceph-mon[29963]: osdmap e472: 8 total, 8 up, 5 in 2024-03-30T14:04:47.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:47 smithi043 ceph-mon[42099]: osdmap e472: 8 total, 8 up, 5 in 2024-03-30T14:04:48.128 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-03-30T14:04:48.169 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T14:04:48.379 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:04:48.438 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:04:48.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:48 smithi113 ceph-mon[34341]: 6.1f scrub starts 2024-03-30T14:04:48.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:48 smithi113 ceph-mon[34341]: 6.1f scrub ok 2024-03-30T14:04:48.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:48 smithi113 ceph-mon[34341]: osdmap e473: 8 total, 8 up, 5 in 2024-03-30T14:04:48.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:48 smithi113 ceph-mon[34341]: pgmap v1035: 152 pgs: 3 active+undersized+remapped, 149 active+clean; 582 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 511 B/s rd, 0 op/s; 289 B/s, 9 objects/s recovering 2024-03-30T14:04:48.560 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:48 smithi043 ceph-mon[29963]: 6.1f scrub starts 2024-03-30T14:04:48.561 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:48 smithi043 ceph-mon[29963]: 6.1f scrub ok 2024-03-30T14:04:48.561 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:48 smithi043 ceph-mon[29963]: osdmap e473: 8 total, 8 up, 5 in 2024-03-30T14:04:48.561 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:48 smithi043 ceph-mon[29963]: pgmap v1035: 152 pgs: 3 active+undersized+remapped, 149 active+clean; 582 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 511 B/s rd, 0 op/s; 289 B/s, 9 objects/s recovering 2024-03-30T14:04:48.561 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:48 smithi043 ceph-mon[42099]: 6.1f scrub starts 2024-03-30T14:04:48.561 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:48 smithi043 ceph-mon[42099]: 6.1f scrub ok 2024-03-30T14:04:48.561 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:48 smithi043 ceph-mon[42099]: osdmap e473: 8 total, 8 up, 5 in 2024-03-30T14:04:48.561 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:48 smithi043 ceph-mon[42099]: pgmap v1035: 152 pgs: 3 active+undersized+remapped, 149 active+clean; 582 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 511 B/s rd, 0 op/s; 289 B/s, 9 objects/s recovering 2024-03-30T14:04:49.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:49 smithi113 ceph-mon[34341]: 5.d scrub starts 2024-03-30T14:04:49.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:49 smithi113 ceph-mon[34341]: 5.d scrub ok 2024-03-30T14:04:49.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:49 smithi113 ceph-mon[34341]: osdmap e474: 8 total, 8 up, 5 in 2024-03-30T14:04:49.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:49 smithi113 ceph-mon[34341]: 1.12 scrub starts 2024-03-30T14:04:49.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:49 smithi113 ceph-mon[34341]: 1.12 scrub ok 2024-03-30T14:04:49.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:49 smithi043 ceph-mon[29963]: 5.d scrub starts 2024-03-30T14:04:49.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:49 smithi043 ceph-mon[29963]: 5.d scrub ok 2024-03-30T14:04:49.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:49 smithi043 ceph-mon[29963]: osdmap e474: 8 total, 8 up, 5 in 2024-03-30T14:04:49.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:49 smithi043 ceph-mon[29963]: 1.12 scrub starts 2024-03-30T14:04:49.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:49 smithi043 ceph-mon[29963]: 1.12 scrub ok 2024-03-30T14:04:49.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:49 smithi043 ceph-mon[42099]: 5.d scrub starts 2024-03-30T14:04:49.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:49 smithi043 ceph-mon[42099]: 5.d scrub ok 2024-03-30T14:04:49.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:49 smithi043 ceph-mon[42099]: osdmap e474: 8 total, 8 up, 5 in 2024-03-30T14:04:49.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:49 smithi043 ceph-mon[42099]: 1.12 scrub starts 2024-03-30T14:04:49.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:49 smithi043 ceph-mon[42099]: 1.12 scrub ok 2024-03-30T14:04:49.807 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T14:04:49.807 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-03-30T14:04:49.807 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd in 2 2024-03-30T14:04:49.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:04:49 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:04:50.199 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T14:04:50.199 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [], 2024-03-30T14:04:50.199 INFO:teuthology.orchestra.run.smithi113.stdout: "complaint_time": 30, 2024-03-30T14:04:50.199 INFO:teuthology.orchestra.run.smithi113.stdout: "num_blocked_ops": 0 2024-03-30T14:04:50.199 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T14:04:50.263 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:50 smithi043 ceph-mon[29963]: 1.4 scrub starts 2024-03-30T14:04:50.263 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:50 smithi043 ceph-mon[29963]: 1.4 scrub ok 2024-03-30T14:04:50.263 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:50 smithi043 ceph-mon[29963]: 1.1 scrub starts 2024-03-30T14:04:50.263 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:50 smithi043 ceph-mon[29963]: 1.1 scrub ok 2024-03-30T14:04:50.263 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:50 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:50.263 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:50 smithi043 ceph-mon[29963]: pgmap v1037: 152 pgs: 3 active+undersized+remapped, 149 active+clean; 609 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 3.0 KiB/s rd, 24 KiB/s wr, 3 op/s; 0 B/s, 12 objects/s recovering 2024-03-30T14:04:50.263 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:50 smithi043 ceph-mon[29963]: 3.13 scrub starts 2024-03-30T14:04:50.263 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:50 smithi043 ceph-mon[29963]: 3.13 scrub ok 2024-03-30T14:04:50.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:50 smithi043 ceph-mon[42099]: 1.4 scrub starts 2024-03-30T14:04:50.264 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:50 smithi043 ceph-mon[42099]: 1.4 scrub ok 2024-03-30T14:04:50.264 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:50 smithi043 ceph-mon[42099]: 1.1 scrub starts 2024-03-30T14:04:50.264 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:50 smithi043 ceph-mon[42099]: 1.1 scrub ok 2024-03-30T14:04:50.264 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:50 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:50.264 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:50 smithi043 ceph-mon[42099]: pgmap v1037: 152 pgs: 3 active+undersized+remapped, 149 active+clean; 609 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 3.0 KiB/s rd, 24 KiB/s wr, 3 op/s; 0 B/s, 12 objects/s recovering 2024-03-30T14:04:50.264 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:50 smithi043 ceph-mon[42099]: 3.13 scrub starts 2024-03-30T14:04:50.264 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:50 smithi043 ceph-mon[42099]: 3.13 scrub ok 2024-03-30T14:04:50.389 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:04:50.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:50 smithi113 ceph-mon[34341]: 1.4 scrub starts 2024-03-30T14:04:50.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:50 smithi113 ceph-mon[34341]: 1.4 scrub ok 2024-03-30T14:04:50.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:50 smithi113 ceph-mon[34341]: 1.1 scrub starts 2024-03-30T14:04:50.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:50 smithi113 ceph-mon[34341]: 1.1 scrub ok 2024-03-30T14:04:50.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:50 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:50.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:50 smithi113 ceph-mon[34341]: pgmap v1037: 152 pgs: 3 active+undersized+remapped, 149 active+clean; 609 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 3.0 KiB/s rd, 24 KiB/s wr, 3 op/s; 0 B/s, 12 objects/s recovering 2024-03-30T14:04:50.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:50 smithi113 ceph-mon[34341]: 3.13 scrub starts 2024-03-30T14:04:50.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:50 smithi113 ceph-mon[34341]: 3.13 scrub ok 2024-03-30T14:04:50.886 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-03-30T14:04:51.143 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:04:51.213 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:51 smithi113 ceph-mon[34341]: 1.10 scrub starts 2024-03-30T14:04:51.213 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:51 smithi113 ceph-mon[34341]: 1.10 scrub ok 2024-03-30T14:04:51.213 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:51 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2199684654' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:04:51.265 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-03-30T14:04:51.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:51 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:04:51.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:51 smithi113 ceph-mon[34341]: 6.14 scrub starts 2024-03-30T14:04:51.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:51 smithi113 ceph-mon[34341]: 6.14 scrub ok 2024-03-30T14:04:51.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:51 smithi043 ceph-mon[29963]: 1.10 scrub starts 2024-03-30T14:04:51.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:51 smithi043 ceph-mon[29963]: 1.10 scrub ok 2024-03-30T14:04:51.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:51 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2199684654' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:04:51.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:51 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:04:51.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:51 smithi043 ceph-mon[29963]: 6.14 scrub starts 2024-03-30T14:04:51.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:51 smithi043 ceph-mon[29963]: 6.14 scrub ok 2024-03-30T14:04:51.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:51 smithi043 ceph-mon[42099]: 1.10 scrub starts 2024-03-30T14:04:51.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:51 smithi043 ceph-mon[42099]: 1.10 scrub ok 2024-03-30T14:04:51.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:51 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2199684654' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:04:51.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:51 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:04:51.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:51 smithi043 ceph-mon[42099]: 6.14 scrub starts 2024-03-30T14:04:51.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:51 smithi043 ceph-mon[42099]: 6.14 scrub ok 2024-03-30T14:04:52.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:52 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:04:52.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:52 smithi043 ceph-mon[29963]: osdmap e475: 8 total, 8 up, 5 in 2024-03-30T14:04:52.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:52 smithi043 ceph-mon[29963]: pgmap v1039: 152 pgs: 152 active+clean; 609 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 3.4 KiB/s rd, 20 KiB/s wr, 4 op/s; 0 B/s, 10 objects/s recovering 2024-03-30T14:04:52.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:52 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:04:52.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:52 smithi043 ceph-mon[42099]: osdmap e475: 8 total, 8 up, 5 in 2024-03-30T14:04:52.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:52 smithi043 ceph-mon[42099]: pgmap v1039: 152 pgs: 152 active+clean; 609 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 3.4 KiB/s rd, 20 KiB/s wr, 4 op/s; 0 B/s, 10 objects/s recovering 2024-03-30T14:04:52.571 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:52 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:04:52.571 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:52 smithi113 ceph-mon[34341]: osdmap e475: 8 total, 8 up, 5 in 2024-03-30T14:04:52.571 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:52 smithi113 ceph-mon[34341]: pgmap v1039: 152 pgs: 152 active+clean; 609 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 3.4 KiB/s rd, 20 KiB/s wr, 4 op/s; 0 B/s, 10 objects/s recovering 2024-03-30T14:04:52.848 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-30T14:04:52.848 INFO:teuthology.orchestra.run.smithi113.stdout: "size": 20, 2024-03-30T14:04:52.849 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 600, 2024-03-30T14:04:52.849 INFO:teuthology.orchestra.run.smithi113.stdout: "ops": [ 2024-03-30T14:04:52.849 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.849 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "MOSDScrubReserve(4.3,QREQUEST,e:311,from:0,reservation_nonce:3)", 2024-03-30T14:04:52.849 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T13:57:59.929685+0000", 2024-03-30T14:04:52.849 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 412.92526928199999, 2024-03-30T14:04:52.849 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.001587457, 2024-03-30T14:04:52.849 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:52.849 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:52.849 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T14:04:52.849 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:52.849 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.849 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:52.849 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:57:59.929685+0000", 2024-03-30T14:04:52.849 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:52.850 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:52.850 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.850 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:52.850 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:57:59.929685+0000", 2024-03-30T14:04:52.850 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:52.850 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:52.850 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.850 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:52.850 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:57:59.929686+0000", 2024-03-30T14:04:52.850 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.0380000000000001e-06 2024-03-30T14:04:52.850 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:52.850 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.850 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:52.850 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:57:59.929692+0000", 2024-03-30T14:04:52.851 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 5.2089999999999996e-06 2024-03-30T14:04:52.851 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:52.851 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.851 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:52.851 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:57:59.929706+0000", 2024-03-30T14:04:52.851 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.395e-05 2024-03-30T14:04:52.851 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:52.851 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.851 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:52.851 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:57:59.929713+0000", 2024-03-30T14:04:52.851 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.5689999999999999e-06 2024-03-30T14:04:52.851 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:52.851 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.851 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:52.851 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:57:59.929784+0000", 2024-03-30T14:04:52.852 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.1143999999999995e-05 2024-03-30T14:04:52.852 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:52.852 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.852 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "op must wait for map", 2024-03-30T14:04:52.852 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:57:59.929789+0000", 2024-03-30T14:04:52.852 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 4.9309999999999999e-06 2024-03-30T14:04:52.852 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:52.852 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.852 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:52.852 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:57:59.931190+0000", 2024-03-30T14:04:52.852 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.0014006019999999999 2024-03-30T14:04:52.852 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:52.852 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.852 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:52.853 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:57:59.931209+0000", 2024-03-30T14:04:52.853 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.8924999999999999e-05 2024-03-30T14:04:52.853 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:52.853 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.853 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:52.853 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T13:57:59.931273+0000", 2024-03-30T14:04:52.853 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 6.4089000000000003e-05 2024-03-30T14:04:52.853 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:52.853 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:52.853 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:52.853 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:52.853 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.853 INFO:teuthology.orchestra.run.smithi113.stdout: "description": "replica_scrub(pg: 6.9,from:0'0,to:0'0,epoch:367/229,start:6:90000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-03-30T14:04:52.853 INFO:teuthology.orchestra.run.smithi113.stdout: "initiated_at": "2024-03-30T14:00:55.559466+0000", 2024-03-30T14:04:52.853 INFO:teuthology.orchestra.run.smithi113.stdout: "age": 237.29548902299999, 2024-03-30T14:04:52.854 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00033753599999999999, 2024-03-30T14:04:52.854 INFO:teuthology.orchestra.run.smithi113.stdout: "continuous": false, 2024-03-30T14:04:52.854 INFO:teuthology.orchestra.run.smithi113.stdout: "type_data": { 2024-03-30T14:04:52.854 INFO:teuthology.orchestra.run.smithi113.stdout: "flag_point": "started", 2024-03-30T14:04:52.854 INFO:teuthology.orchestra.run.smithi113.stdout: "events": [ 2024-03-30T14:04:52.854 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.854 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "initiated", 2024-03-30T14:04:52.854 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:00:55.559466+0000", 2024-03-30T14:04:52.854 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:52.854 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:52.854 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.854 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "header_read", 2024-03-30T14:04:52.855 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:00:55.559466+0000", 2024-03-30T14:04:52.855 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0 2024-03-30T14:04:52.855 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:52.855 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.855 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "throttled", 2024-03-30T14:04:52.855 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:00:55.559467+0000", 2024-03-30T14:04:52.855 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.5260000000000001e-06 2024-03-30T14:04:52.855 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:52.855 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.855 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "all_read", 2024-03-30T14:04:52.855 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:00:55.559479+0000", 2024-03-30T14:04:52.855 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.2057e-05 2024-03-30T14:04:52.855 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:52.855 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.855 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "dispatched", 2024-03-30T14:04:52.856 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:00:55.559507+0000", 2024-03-30T14:04:52.856 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.7455999999999999e-05 2024-03-30T14:04:52.856 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:52.856 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.856 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "queued_for_pg", 2024-03-30T14:04:52.856 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:00:55.559524+0000", 2024-03-30T14:04:52.856 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 1.7476e-05 2024-03-30T14:04:52.856 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:52.856 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.856 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "reached_pg", 2024-03-30T14:04:52.856 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:00:55.559709+0000", 2024-03-30T14:04:52.856 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 0.00018507699999999999 2024-03-30T14:04:52.856 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:52.856 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.856 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "started", 2024-03-30T14:04:52.857 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:00:55.559733+0000", 2024-03-30T14:04:52.857 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 2.3363e-05 2024-03-30T14:04:52.857 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-30T14:04:52.857 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-03-30T14:04:52.857 INFO:teuthology.orchestra.run.smithi113.stdout: "event": "done", 2024-03-30T14:04:52.857 INFO:teuthology.orchestra.run.smithi113.stdout: "time": "2024-03-30T14:00:55.559803+0000", 2024-03-30T14:04:52.857 INFO:teuthology.orchestra.run.smithi113.stdout: "duration": 7.0580999999999994e-05 2024-03-30T14:04:52.857 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:52.857 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:52.857 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:52.857 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-30T14:04:52.857 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-30T14:04:52.857 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-30T14:04:52.987 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:52 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a[29940]: 2024-03-30T14:04:52.849+0000 7ff6065d3700 -1 mon.a@0(leader).osd e475 definitely_dead 0 2024-03-30T14:04:53.271 INFO:teuthology.orchestra.run.smithi043.stderr:marked in osd.2. 2024-03-30T14:04:53.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T14:04:53.591 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:53 smithi113 ceph-mon[34341]: 4.11 scrub starts 2024-03-30T14:04:53.592 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:53 smithi113 ceph-mon[34341]: 4.11 scrub ok 2024-03-30T14:04:53.592 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:53 smithi113 ceph-mon[34341]: 3.4 scrub starts 2024-03-30T14:04:53.592 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:53 smithi113 ceph-mon[34341]: 3.4 scrub ok 2024-03-30T14:04:53.592 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:53 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1525890602' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-03-30T14:04:53.592 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:53 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:04:53.592 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:53 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:04:53.618 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-30T14:04:53.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:53 smithi043 ceph-mon[29963]: 4.11 scrub starts 2024-03-30T14:04:53.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:53 smithi043 ceph-mon[29963]: 4.11 scrub ok 2024-03-30T14:04:53.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:53 smithi043 ceph-mon[29963]: 3.4 scrub starts 2024-03-30T14:04:53.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:53 smithi043 ceph-mon[29963]: 3.4 scrub ok 2024-03-30T14:04:53.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:53 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1525890602' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-03-30T14:04:53.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:53 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:04:53.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:53 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:04:53.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:53 smithi043 ceph-mon[42099]: 4.11 scrub starts 2024-03-30T14:04:53.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:53 smithi043 ceph-mon[42099]: 4.11 scrub ok 2024-03-30T14:04:53.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:53 smithi043 ceph-mon[42099]: 3.4 scrub starts 2024-03-30T14:04:53.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:53 smithi043 ceph-mon[42099]: 3.4 scrub ok 2024-03-30T14:04:53.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:53 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1525890602' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-03-30T14:04:53.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:53 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:04:53.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:53 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:04:53.642 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:04:54.120 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-03-30T14:04:54.120 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T14:04:54.121 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-03-30T14:04:54.121 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd in 3 2024-03-30T14:04:54.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:54 smithi043 ceph-mon[29963]: 6.9 scrub starts 2024-03-30T14:04:54.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:54 smithi043 ceph-mon[29963]: 6.9 scrub ok 2024-03-30T14:04:54.353 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:54 smithi043 ceph-mon[29963]: 1.e scrub starts 2024-03-30T14:04:54.353 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:54 smithi043 ceph-mon[29963]: 1.e scrub ok 2024-03-30T14:04:54.353 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:54 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1525890602' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-03-30T14:04:54.353 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:54 smithi043 ceph-mon[29963]: osdmap e476: 8 total, 8 up, 6 in 2024-03-30T14:04:54.353 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:54 smithi043 ceph-mon[29963]: pgmap v1041: 152 pgs: 6 peering, 146 active+clean; 609 KiB data, 4.5 GiB used, 443 GiB / 447 GiB avail; 2.8 KiB/s rd, 18 KiB/s wr, 3 op/s; 0 B/s, 3 objects/s recovering 2024-03-30T14:04:54.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:54 smithi043 ceph-mon[42099]: 6.9 scrub starts 2024-03-30T14:04:54.354 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:54 smithi043 ceph-mon[42099]: 6.9 scrub ok 2024-03-30T14:04:54.354 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:54 smithi043 ceph-mon[42099]: 1.e scrub starts 2024-03-30T14:04:54.354 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:54 smithi043 ceph-mon[42099]: 1.e scrub ok 2024-03-30T14:04:54.354 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:54 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1525890602' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-03-30T14:04:54.354 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:54 smithi043 ceph-mon[42099]: osdmap e476: 8 total, 8 up, 6 in 2024-03-30T14:04:54.354 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:54 smithi043 ceph-mon[42099]: pgmap v1041: 152 pgs: 6 peering, 146 active+clean; 609 KiB data, 4.5 GiB used, 443 GiB / 447 GiB avail; 2.8 KiB/s rd, 18 KiB/s wr, 3 op/s; 0 B/s, 3 objects/s recovering 2024-03-30T14:04:54.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:54 smithi113 ceph-mon[34341]: 6.9 scrub starts 2024-03-30T14:04:54.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:54 smithi113 ceph-mon[34341]: 6.9 scrub ok 2024-03-30T14:04:54.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:54 smithi113 ceph-mon[34341]: 1.e scrub starts 2024-03-30T14:04:54.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:54 smithi113 ceph-mon[34341]: 1.e scrub ok 2024-03-30T14:04:54.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:54 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1525890602' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-03-30T14:04:54.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:54 smithi113 ceph-mon[34341]: osdmap e476: 8 total, 8 up, 6 in 2024-03-30T14:04:54.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:54 smithi113 ceph-mon[34341]: pgmap v1041: 152 pgs: 6 peering, 146 active+clean; 609 KiB data, 4.5 GiB used, 443 GiB / 447 GiB avail; 2.8 KiB/s rd, 18 KiB/s wr, 3 op/s; 0 B/s, 3 objects/s recovering 2024-03-30T14:04:54.799 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:04:55.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:55 smithi043 ceph-mon[29963]: 4.a scrub starts 2024-03-30T14:04:55.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:55 smithi043 ceph-mon[29963]: 4.a scrub ok 2024-03-30T14:04:55.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:55 smithi043 ceph-mon[29963]: 5.18 scrub starts 2024-03-30T14:04:55.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:55 smithi043 ceph-mon[29963]: 5.18 scrub ok 2024-03-30T14:04:55.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:55 smithi043 ceph-mon[29963]: osdmap e477: 8 total, 8 up, 6 in 2024-03-30T14:04:55.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:55 smithi043 ceph-mon[29963]: osdmap e478: 8 total, 8 up, 6 in 2024-03-30T14:04:55.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:55 smithi043 ceph-mon[42099]: 4.a scrub starts 2024-03-30T14:04:55.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:55 smithi043 ceph-mon[42099]: 4.a scrub ok 2024-03-30T14:04:55.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:55 smithi043 ceph-mon[42099]: 5.18 scrub starts 2024-03-30T14:04:55.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:55 smithi043 ceph-mon[42099]: 5.18 scrub ok 2024-03-30T14:04:55.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:55 smithi043 ceph-mon[42099]: osdmap e477: 8 total, 8 up, 6 in 2024-03-30T14:04:55.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:55 smithi043 ceph-mon[42099]: osdmap e478: 8 total, 8 up, 6 in 2024-03-30T14:04:55.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:55 smithi113 ceph-mon[34341]: 4.a scrub starts 2024-03-30T14:04:55.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:55 smithi113 ceph-mon[34341]: 4.a scrub ok 2024-03-30T14:04:55.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:55 smithi113 ceph-mon[34341]: 5.18 scrub starts 2024-03-30T14:04:55.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:55 smithi113 ceph-mon[34341]: 5.18 scrub ok 2024-03-30T14:04:55.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:55 smithi113 ceph-mon[34341]: osdmap e477: 8 total, 8 up, 6 in 2024-03-30T14:04:55.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:55 smithi113 ceph-mon[34341]: osdmap e478: 8 total, 8 up, 6 in 2024-03-30T14:04:56.289 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:04:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:04:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:04:56.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:56 smithi043 ceph-mon[29963]: 5.1 scrub starts 2024-03-30T14:04:56.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:56 smithi043 ceph-mon[29963]: 5.1 scrub ok 2024-03-30T14:04:56.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:56 smithi043 ceph-mon[29963]: 4.b scrub starts 2024-03-30T14:04:56.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:56 smithi043 ceph-mon[29963]: 4.b scrub ok 2024-03-30T14:04:56.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:56 smithi043 ceph-mon[29963]: pgmap v1044: 152 pgs: 45 peering, 107 active+clean; 582 KiB data, 4.5 GiB used, 443 GiB / 447 GiB avail 2024-03-30T14:04:56.627 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:56 smithi043 ceph-mon[42099]: 5.1 scrub starts 2024-03-30T14:04:56.627 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:56 smithi043 ceph-mon[42099]: 5.1 scrub ok 2024-03-30T14:04:56.627 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:56 smithi043 ceph-mon[42099]: 4.b scrub starts 2024-03-30T14:04:56.627 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:56 smithi043 ceph-mon[42099]: 4.b scrub ok 2024-03-30T14:04:56.627 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:56 smithi043 ceph-mon[42099]: pgmap v1044: 152 pgs: 45 peering, 107 active+clean; 582 KiB data, 4.5 GiB used, 443 GiB / 447 GiB avail 2024-03-30T14:04:56.640 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:04:56.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:56 smithi113 ceph-mon[34341]: 5.1 scrub starts 2024-03-30T14:04:56.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:56 smithi113 ceph-mon[34341]: 5.1 scrub ok 2024-03-30T14:04:56.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:56 smithi113 ceph-mon[34341]: 4.b scrub starts 2024-03-30T14:04:56.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:56 smithi113 ceph-mon[34341]: 4.b scrub ok 2024-03-30T14:04:56.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:56 smithi113 ceph-mon[34341]: pgmap v1044: 152 pgs: 45 peering, 107 active+clean; 582 KiB data, 4.5 GiB used, 443 GiB / 447 GiB avail 2024-03-30T14:04:57.387 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:57 smithi043 ceph-mon[29963]: 4.1f scrub starts 2024-03-30T14:04:57.387 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:57 smithi043 ceph-mon[29963]: 4.1f scrub ok 2024-03-30T14:04:57.387 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:57 smithi043 ceph-mon[29963]: 3.b scrub starts 2024-03-30T14:04:57.387 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:57 smithi043 ceph-mon[29963]: 3.b scrub ok 2024-03-30T14:04:57.387 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:57 smithi043 ceph-mon[29963]: osdmap e479: 8 total, 8 up, 6 in 2024-03-30T14:04:57.387 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:57 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:04:57.387 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:57 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:04:57.388 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:57 smithi043 ceph-mon[42099]: 4.1f scrub starts 2024-03-30T14:04:57.388 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:57 smithi043 ceph-mon[42099]: 4.1f scrub ok 2024-03-30T14:04:57.388 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:57 smithi043 ceph-mon[42099]: 3.b scrub starts 2024-03-30T14:04:57.388 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:57 smithi043 ceph-mon[42099]: 3.b scrub ok 2024-03-30T14:04:57.388 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:57 smithi043 ceph-mon[42099]: osdmap e479: 8 total, 8 up, 6 in 2024-03-30T14:04:57.388 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:57 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:04:57.388 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:57 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:04:57.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:57 smithi113 ceph-mon[34341]: 4.1f scrub starts 2024-03-30T14:04:57.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:57 smithi113 ceph-mon[34341]: 4.1f scrub ok 2024-03-30T14:04:57.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:57 smithi113 ceph-mon[34341]: 3.b scrub starts 2024-03-30T14:04:57.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:57 smithi113 ceph-mon[34341]: 3.b scrub ok 2024-03-30T14:04:57.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:57 smithi113 ceph-mon[34341]: osdmap e479: 8 total, 8 up, 6 in 2024-03-30T14:04:57.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:57 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:04:57.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:57 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:04:57.976 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-03-30T14:04:58.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:58 smithi043 ceph-mon[29963]: osdmap e480: 8 total, 8 up, 6 in 2024-03-30T14:04:58.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:58 smithi043 ceph-mon[29963]: 4.f scrub starts 2024-03-30T14:04:58.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:58 smithi043 ceph-mon[29963]: 4.f scrub ok 2024-03-30T14:04:58.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:58 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2813526124' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:04:58.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:58 smithi043 ceph-mon[29963]: pgmap v1047: 152 pgs: 49 peering, 103 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 0 B/s, 11 objects/s recovering 2024-03-30T14:04:58.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:58 smithi043 ceph-mon[29963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:04:58.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:58 smithi043 ceph-mon[29963]: Cluster is now healthy 2024-03-30T14:04:58.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:58 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2813526124' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:04:58.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:58 smithi043 ceph-mon[29963]: osdmap e481: 8 total, 8 up, 6 in 2024-03-30T14:04:58.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:58 smithi043 ceph-mon[42099]: osdmap e480: 8 total, 8 up, 6 in 2024-03-30T14:04:58.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:58 smithi043 ceph-mon[42099]: 4.f scrub starts 2024-03-30T14:04:58.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:58 smithi043 ceph-mon[42099]: 4.f scrub ok 2024-03-30T14:04:58.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:58 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2813526124' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:04:58.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:58 smithi043 ceph-mon[42099]: pgmap v1047: 152 pgs: 49 peering, 103 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 0 B/s, 11 objects/s recovering 2024-03-30T14:04:58.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:58 smithi043 ceph-mon[42099]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:04:58.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:58 smithi043 ceph-mon[42099]: Cluster is now healthy 2024-03-30T14:04:58.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:58 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2813526124' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:04:58.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:58 smithi043 ceph-mon[42099]: osdmap e481: 8 total, 8 up, 6 in 2024-03-30T14:04:58.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:58 smithi113 ceph-mon[34341]: osdmap e480: 8 total, 8 up, 6 in 2024-03-30T14:04:58.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:58 smithi113 ceph-mon[34341]: 4.f scrub starts 2024-03-30T14:04:58.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:58 smithi113 ceph-mon[34341]: 4.f scrub ok 2024-03-30T14:04:58.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:58 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2813526124' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:04:58.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:58 smithi113 ceph-mon[34341]: pgmap v1047: 152 pgs: 49 peering, 103 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 0 B/s, 11 objects/s recovering 2024-03-30T14:04:58.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:58 smithi113 ceph-mon[34341]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:04:58.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:58 smithi113 ceph-mon[34341]: Cluster is now healthy 2024-03-30T14:04:58.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:58 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2813526124' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:04:58.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:58 smithi113 ceph-mon[34341]: osdmap e481: 8 total, 8 up, 6 in 2024-03-30T14:04:59.475 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T14:04:59.476 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T14:04:59.476 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 0 2024-03-30T14:04:59.476 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T14:04:59.476 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:59 smithi043 ceph-mon[29963]: 1.a scrub starts 2024-03-30T14:04:59.476 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:59 smithi043 ceph-mon[29963]: 1.a scrub ok 2024-03-30T14:04:59.476 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:59 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:04:59.476 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:04:59 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:04:59.477 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:59 smithi043 ceph-mon[42099]: 1.a scrub starts 2024-03-30T14:04:59.477 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:59 smithi043 ceph-mon[42099]: 1.a scrub ok 2024-03-30T14:04:59.477 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:59 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:04:59.477 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:04:59 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:04:59.613 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:59 smithi113 ceph-mon[34341]: 1.a scrub starts 2024-03-30T14:04:59.613 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:59 smithi113 ceph-mon[34341]: 1.a scrub ok 2024-03-30T14:04:59.614 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:59 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:04:59.614 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:04:59 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:04:59.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:04:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:05:00.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:00 smithi043 ceph-mon[42099]: 1.7 scrub starts 2024-03-30T14:05:00.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:00 smithi043 ceph-mon[42099]: 1.7 scrub ok 2024-03-30T14:05:00.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:00 smithi043 ceph-mon[42099]: 1.14 scrub starts 2024-03-30T14:05:00.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:00 smithi043 ceph-mon[42099]: 1.14 scrub ok 2024-03-30T14:05:00.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:00 smithi043 ceph-mon[42099]: osdmap e482: 8 total, 8 up, 6 in 2024-03-30T14:05:00.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:00 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:00.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:00 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:05:00.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:00 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:00.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:00 smithi043 ceph-mon[42099]: pgmap v1050: 152 pgs: 4 peering, 148 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 65 KiB/s, 20 objects/s recovering 2024-03-30T14:05:00.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:00 smithi043 ceph-mon[29963]: 1.7 scrub starts 2024-03-30T14:05:00.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:00 smithi043 ceph-mon[29963]: 1.7 scrub ok 2024-03-30T14:05:00.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:00 smithi043 ceph-mon[29963]: 1.14 scrub starts 2024-03-30T14:05:00.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:00 smithi043 ceph-mon[29963]: 1.14 scrub ok 2024-03-30T14:05:00.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:00 smithi043 ceph-mon[29963]: osdmap e482: 8 total, 8 up, 6 in 2024-03-30T14:05:00.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:00 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:00.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:00 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:05:00.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:00 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:00.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:00 smithi043 ceph-mon[29963]: pgmap v1050: 152 pgs: 4 peering, 148 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 65 KiB/s, 20 objects/s recovering 2024-03-30T14:05:00.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:00 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a[29940]: 2024-03-30T14:05:00.344+0000 7ff6065d3700 -1 mon.a@0(leader).osd e482 definitely_dead 0 2024-03-30T14:05:00.664 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-03-30T14:05:00.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:00 smithi113 ceph-mon[34341]: 1.7 scrub starts 2024-03-30T14:05:00.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:00 smithi113 ceph-mon[34341]: 1.7 scrub ok 2024-03-30T14:05:00.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:00 smithi113 ceph-mon[34341]: 1.14 scrub starts 2024-03-30T14:05:00.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:00 smithi113 ceph-mon[34341]: 1.14 scrub ok 2024-03-30T14:05:00.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:00 smithi113 ceph-mon[34341]: osdmap e482: 8 total, 8 up, 6 in 2024-03-30T14:05:00.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:00 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:00.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:00 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:05:00.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:00 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:00.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:00 smithi113 ceph-mon[34341]: pgmap v1050: 152 pgs: 4 peering, 148 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 65 KiB/s, 20 objects/s recovering 2024-03-30T14:05:00.913 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:05:01.334 INFO:teuthology.orchestra.run.smithi043.stderr:marked in osd.3. 2024-03-30T14:05:01.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:01 smithi043 ceph-mon[29963]: 3.a scrub starts 2024-03-30T14:05:01.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:01 smithi043 ceph-mon[29963]: 3.a scrub ok 2024-03-30T14:05:01.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:01 smithi043 ceph-mon[29963]: 6.f scrub starts 2024-03-30T14:05:01.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:01 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3091890951' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-30T14:05:01.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:01 smithi043 ceph-mon[29963]: 6.f scrub ok 2024-03-30T14:05:01.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:01 smithi043 ceph-mon[42099]: 3.a scrub starts 2024-03-30T14:05:01.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:01 smithi043 ceph-mon[42099]: 3.a scrub ok 2024-03-30T14:05:01.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:01 smithi043 ceph-mon[42099]: 6.f scrub starts 2024-03-30T14:05:01.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:01 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3091890951' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-30T14:05:01.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:01 smithi043 ceph-mon[42099]: 6.f scrub ok 2024-03-30T14:05:01.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:01 smithi113 ceph-mon[34341]: 3.a scrub starts 2024-03-30T14:05:01.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:01 smithi113 ceph-mon[34341]: 3.a scrub ok 2024-03-30T14:05:01.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:01 smithi113 ceph-mon[34341]: 6.f scrub starts 2024-03-30T14:05:01.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:01 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3091890951' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-30T14:05:01.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:01 smithi113 ceph-mon[34341]: 6.f scrub ok 2024-03-30T14:05:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:02 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:05:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:02 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:05:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:02 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3091890951' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-30T14:05:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:02 smithi043 ceph-mon[29963]: osdmap e483: 8 total, 8 up, 7 in 2024-03-30T14:05:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:02 smithi043 ceph-mon[29963]: 5.1e scrub starts 2024-03-30T14:05:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:02 smithi043 ceph-mon[29963]: 5.1e scrub ok 2024-03-30T14:05:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:02 smithi043 ceph-mon[29963]: pgmap v1052: 152 pgs: 12 peering, 140 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 901 B/s rd, 0 op/s; 57 KiB/s, 7 objects/s recovering 2024-03-30T14:05:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:02 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:05:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:02 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:05:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:02 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3091890951' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-30T14:05:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:02 smithi043 ceph-mon[42099]: osdmap e483: 8 total, 8 up, 7 in 2024-03-30T14:05:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:02 smithi043 ceph-mon[42099]: 5.1e scrub starts 2024-03-30T14:05:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:02 smithi043 ceph-mon[42099]: 5.1e scrub ok 2024-03-30T14:05:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:02 smithi043 ceph-mon[42099]: pgmap v1052: 152 pgs: 12 peering, 140 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 901 B/s rd, 0 op/s; 57 KiB/s, 7 objects/s recovering 2024-03-30T14:05:02.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:02 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:05:02.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:02 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:05:02.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:02 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3091890951' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-30T14:05:02.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:02 smithi113 ceph-mon[34341]: osdmap e483: 8 total, 8 up, 7 in 2024-03-30T14:05:02.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:02 smithi113 ceph-mon[34341]: 5.1e scrub starts 2024-03-30T14:05:02.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:02 smithi113 ceph-mon[34341]: 5.1e scrub ok 2024-03-30T14:05:02.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:02 smithi113 ceph-mon[34341]: pgmap v1052: 152 pgs: 12 peering, 140 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 901 B/s rd, 0 op/s; 57 KiB/s, 7 objects/s recovering 2024-03-30T14:05:03.430 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:03 smithi043 ceph-mon[29963]: 5.9 scrub starts 2024-03-30T14:05:03.431 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:03 smithi043 ceph-mon[29963]: 5.9 scrub ok 2024-03-30T14:05:03.431 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:03 smithi043 ceph-mon[29963]: osdmap e484: 8 total, 8 up, 7 in 2024-03-30T14:05:03.431 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:03 smithi043 ceph-mon[29963]: osdmap e485: 8 total, 8 up, 7 in 2024-03-30T14:05:03.431 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:03 smithi043 ceph-mon[42099]: 5.9 scrub starts 2024-03-30T14:05:03.431 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:03 smithi043 ceph-mon[42099]: 5.9 scrub ok 2024-03-30T14:05:03.431 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:03 smithi043 ceph-mon[42099]: osdmap e484: 8 total, 8 up, 7 in 2024-03-30T14:05:03.431 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:03 smithi043 ceph-mon[42099]: osdmap e485: 8 total, 8 up, 7 in 2024-03-30T14:05:03.558 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T14:05:03.558 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-03-30T14:05:03.558 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-03-30T14:05:03.558 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-03-30T14:05:03.558 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T14:05:03.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:03 smithi113 ceph-mon[34341]: 5.9 scrub starts 2024-03-30T14:05:03.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:03 smithi113 ceph-mon[34341]: 5.9 scrub ok 2024-03-30T14:05:03.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:03 smithi113 ceph-mon[34341]: osdmap e484: 8 total, 8 up, 7 in 2024-03-30T14:05:03.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:03 smithi113 ceph-mon[34341]: osdmap e485: 8 total, 8 up, 7 in 2024-03-30T14:05:04.267 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-03-30T14:05:04.267 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T14:05:04.268 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-03-30T14:05:04.268 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd in 7 2024-03-30T14:05:04.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:04 smithi043 ceph-mon[29963]: 6.2 scrub starts 2024-03-30T14:05:04.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:04 smithi043 ceph-mon[29963]: 6.2 scrub ok 2024-03-30T14:05:04.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:04 smithi043 ceph-mon[29963]: 5.4 scrub starts 2024-03-30T14:05:04.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:04 smithi043 ceph-mon[29963]: 5.4 scrub ok 2024-03-30T14:05:04.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:04 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:05:04.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:04 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:05:04.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:04 smithi043 ceph-mon[29963]: 3.19 scrub starts 2024-03-30T14:05:04.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:04 smithi043 ceph-mon[29963]: 3.19 scrub ok 2024-03-30T14:05:04.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:04 smithi043 ceph-mon[29963]: pgmap v1055: 152 pgs: 12 peering, 140 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 233 B/s, 5 objects/s recovering 2024-03-30T14:05:04.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:04 smithi043 ceph-mon[29963]: osdmap e486: 8 total, 8 up, 7 in 2024-03-30T14:05:04.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:04 smithi043 ceph-mon[42099]: 6.2 scrub starts 2024-03-30T14:05:04.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:04 smithi043 ceph-mon[42099]: 6.2 scrub ok 2024-03-30T14:05:04.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:04 smithi043 ceph-mon[42099]: 5.4 scrub starts 2024-03-30T14:05:04.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:04 smithi043 ceph-mon[42099]: 5.4 scrub ok 2024-03-30T14:05:04.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:04 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:05:04.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:04 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:05:04.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:04 smithi043 ceph-mon[42099]: 3.19 scrub starts 2024-03-30T14:05:04.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:04 smithi043 ceph-mon[42099]: 3.19 scrub ok 2024-03-30T14:05:04.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:04 smithi043 ceph-mon[42099]: pgmap v1055: 152 pgs: 12 peering, 140 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 233 B/s, 5 objects/s recovering 2024-03-30T14:05:04.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:04 smithi043 ceph-mon[42099]: osdmap e486: 8 total, 8 up, 7 in 2024-03-30T14:05:04.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:04 smithi113 ceph-mon[34341]: 6.2 scrub starts 2024-03-30T14:05:04.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:04 smithi113 ceph-mon[34341]: 6.2 scrub ok 2024-03-30T14:05:04.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:04 smithi113 ceph-mon[34341]: 5.4 scrub starts 2024-03-30T14:05:04.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:04 smithi113 ceph-mon[34341]: 5.4 scrub ok 2024-03-30T14:05:04.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:04 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:05:04.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:04 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:05:04.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:04 smithi113 ceph-mon[34341]: 3.19 scrub starts 2024-03-30T14:05:04.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:04 smithi113 ceph-mon[34341]: 3.19 scrub ok 2024-03-30T14:05:04.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:04 smithi113 ceph-mon[34341]: pgmap v1055: 152 pgs: 12 peering, 140 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 233 B/s, 5 objects/s recovering 2024-03-30T14:05:04.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:04 smithi113 ceph-mon[34341]: osdmap e486: 8 total, 8 up, 7 in 2024-03-30T14:05:04.919 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-03-30T14:05:04.958 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:05:05.444 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:05 smithi043 ceph-mon[29963]: 3.c deep-scrub starts 2024-03-30T14:05:05.445 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:05 smithi043 ceph-mon[29963]: 3.c deep-scrub ok 2024-03-30T14:05:05.445 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:05 smithi043 ceph-mon[29963]: osdmap e487: 8 total, 8 up, 7 in 2024-03-30T14:05:05.445 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:05 smithi043 ceph-mon[42099]: 3.c deep-scrub starts 2024-03-30T14:05:05.445 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:05 smithi043 ceph-mon[42099]: 3.c deep-scrub ok 2024-03-30T14:05:05.445 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:05 smithi043 ceph-mon[42099]: osdmap e487: 8 total, 8 up, 7 in 2024-03-30T14:05:05.659 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:05:05.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:05 smithi113 ceph-mon[34341]: 3.c deep-scrub starts 2024-03-30T14:05:05.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:05 smithi113 ceph-mon[34341]: 3.c deep-scrub ok 2024-03-30T14:05:05.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:05 smithi113 ceph-mon[34341]: osdmap e487: 8 total, 8 up, 7 in 2024-03-30T14:05:06.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:06 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:05:06.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:06 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:05:06.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:06 smithi043 ceph-mon[29963]: 1.4 scrub starts 2024-03-30T14:05:06.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:06 smithi043 ceph-mon[29963]: 1.4 scrub ok 2024-03-30T14:05:06.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:06 smithi043 ceph-mon[29963]: pgmap v1058: 152 pgs: 1 active+undersized+remapped, 14 peering, 137 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 401 B/s, 12 objects/s recovering 2024-03-30T14:05:06.392 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:05:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:05:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:05:06.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:06 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:05:06.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:06 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:05:06.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:06 smithi043 ceph-mon[42099]: 1.4 scrub starts 2024-03-30T14:05:06.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:06 smithi043 ceph-mon[42099]: 1.4 scrub ok 2024-03-30T14:05:06.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:06 smithi043 ceph-mon[42099]: pgmap v1058: 152 pgs: 1 active+undersized+remapped, 14 peering, 137 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 401 B/s, 12 objects/s recovering 2024-03-30T14:05:06.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:06 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:05:06.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:06 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:05:06.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:06 smithi113 ceph-mon[34341]: 1.4 scrub starts 2024-03-30T14:05:06.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:06 smithi113 ceph-mon[34341]: 1.4 scrub ok 2024-03-30T14:05:06.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:06 smithi113 ceph-mon[34341]: pgmap v1058: 152 pgs: 1 active+undersized+remapped, 14 peering, 137 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 401 B/s, 12 objects/s recovering 2024-03-30T14:05:07.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:07 smithi043 ceph-mon[29963]: 1.c scrub starts 2024-03-30T14:05:07.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:07 smithi043 ceph-mon[29963]: 1.c scrub ok 2024-03-30T14:05:07.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:07 smithi043 ceph-mon[42099]: 1.c scrub starts 2024-03-30T14:05:07.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:07 smithi043 ceph-mon[42099]: 1.c scrub ok 2024-03-30T14:05:07.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:07 smithi113 ceph-mon[34341]: 1.c scrub starts 2024-03-30T14:05:07.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:07 smithi113 ceph-mon[34341]: 1.c scrub ok 2024-03-30T14:05:08.452 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-30T14:05:08.452 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-03-30T14:05:08.452 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-03-30T14:05:08.452 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-03-30T14:05:08.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.452 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "MOSDPGPush(11.4 392/389 [PushOp(11:2760df5b:::benchmark_data_smithi043_197439_object6955:head, version: 384'429, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(11:2760df5b:::benchmark_data_smithi043_197439_object6955:head@384'429, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-30T14:05:08.452 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:01:33.020921+0000", 2024-03-30T14:05:08.452 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 215.43637648399999, 2024-03-30T14:05:08.452 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.085514931000000002, 2024-03-30T14:05:08.452 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:05:08.453 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:05:08.453 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T14:05:08.453 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:05:08.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.453 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:05:08.453 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:33.020921+0000", 2024-03-30T14:05:08.453 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.453 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:05:08.453 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:33.020921+0000", 2024-03-30T14:05:08.453 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.454 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:05:08.454 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:33.020921+0000", 2024-03-30T14:05:08.454 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1699999999999997e-07 2024-03-30T14:05:08.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.454 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.454 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:05:08.454 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:33.020988+0000", 2024-03-30T14:05:08.454 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6248999999999995e-05 2024-03-30T14:05:08.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.454 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.454 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:05:08.454 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:33.021003+0000", 2024-03-30T14:05:08.454 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.558e-05 2024-03-30T14:05:08.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.455 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:05:08.455 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:33.021013+0000", 2024-03-30T14:05:08.455 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0335e-05 2024-03-30T14:05:08.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.455 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:05:08.455 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:33.021072+0000", 2024-03-30T14:05:08.455 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8891999999999998e-05 2024-03-30T14:05:08.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.455 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:05:08.455 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:33.021081+0000", 2024-03-30T14:05:08.455 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4279999999999996e-06 2024-03-30T14:05:08.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.456 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.456 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:05:08.456 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:33.106435+0000", 2024-03-30T14:05:08.456 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.085354730000000004 2024-03-30T14:05:08.456 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.456 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.456 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.456 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.456 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "MOSDPGPush(11.d 392/389 [PushOp(11:bbaee752:::benchmark_data_smithi043_197439_object7646:head, version: 384'504, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(11:bbaee752:::benchmark_data_smithi043_197439_object7646:head@384'504, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-30T14:05:08.456 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:01:33.020939+0000", 2024-03-30T14:05:08.456 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 215.43635837799999, 2024-03-30T14:05:08.456 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.089034685000000002, 2024-03-30T14:05:08.456 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:05:08.457 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:05:08.457 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T14:05:08.457 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:05:08.457 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.457 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:05:08.457 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:33.020939+0000", 2024-03-30T14:05:08.457 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.457 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.457 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:05:08.457 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:33.020939+0000", 2024-03-30T14:05:08.457 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.457 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.457 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:05:08.458 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:33.020939+0000", 2024-03-30T14:05:08.458 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6899999999999998e-07 2024-03-30T14:05:08.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.458 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:05:08.458 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:33.021000+0000", 2024-03-30T14:05:08.458 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1346000000000005e-05 2024-03-30T14:05:08.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.458 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:05:08.458 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:33.021014+0000", 2024-03-30T14:05:08.458 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.397e-05 2024-03-30T14:05:08.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.459 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:05:08.459 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:33.021025+0000", 2024-03-30T14:05:08.459 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0314e-05 2024-03-30T14:05:08.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.459 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:05:08.459 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:33.021093+0000", 2024-03-30T14:05:08.459 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8070000000000004e-05 2024-03-30T14:05:08.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.459 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:05:08.459 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:33.021100+0000", 2024-03-30T14:05:08.459 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5769999999999996e-06 2024-03-30T14:05:08.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.459 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:05:08.460 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:01:33.109973+0000", 2024-03-30T14:05:08.460 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.088872938999999998 2024-03-30T14:05:08.460 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.460 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.460 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.460 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:77555 11.c e408/395 11:38191c9c:::benchmark_data_smithi043_197439_object77554:head v 408'4749, mlcod=408'4746)", 2024-03-30T14:05:08.460 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:02:19.451514+0000", 2024-03-30T14:05:08.460 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 169.00578339399999, 2024-03-30T14:05:08.460 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.084284840999999999, 2024-03-30T14:05:08.460 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:05:08.460 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:05:08.460 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:05:08.460 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:05:08.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:05:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:19.451514+0000", 2024-03-30T14:05:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:05:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:19.451514+0000", 2024-03-30T14:05:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:05:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:19.451514+0000", 2024-03-30T14:05:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4499999999999997e-07 2024-03-30T14:05:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:05:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:19.451536+0000", 2024-03-30T14:05:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2135000000000001e-05 2024-03-30T14:05:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:05:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:19.451548+0000", 2024-03-30T14:05:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2174999999999999e-05 2024-03-30T14:05:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:05:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:19.451555+0000", 2024-03-30T14:05:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1299999999999998e-06 2024-03-30T14:05:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:05:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:19.453590+0000", 2024-03-30T14:05:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0020352880000000001 2024-03-30T14:05:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:05:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:19.453610+0000", 2024-03-30T14:05:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0356e-05 2024-03-30T14:05:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:05:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:19.535760+0000", 2024-03-30T14:05:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.082149538999999994 2024-03-30T14:05:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:05:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:19.535798+0000", 2024-03-30T14:05:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8773000000000003e-05 2024-03-30T14:05:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:77556 11.c e408/395 11:3bedc0d4:::benchmark_data_smithi043_197439_object77555:head v 408'4750, mlcod=408'4746)", 2024-03-30T14:05:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:02:19.453550+0000", 2024-03-30T14:05:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 169.00374730499999, 2024-03-30T14:05:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.089769069000000007, 2024-03-30T14:05:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:05:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:05:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:05:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:05:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:05:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:19.453550+0000", 2024-03-30T14:05:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:05:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:19.453550+0000", 2024-03-30T14:05:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:05:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:19.453551+0000", 2024-03-30T14:05:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0500000000000002e-07 2024-03-30T14:05:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:05:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:19.453600+0000", 2024-03-30T14:05:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9851000000000001e-05 2024-03-30T14:05:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:05:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:19.453612+0000", 2024-03-30T14:05:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1144e-05 2024-03-30T14:05:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:05:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:19.453619+0000", 2024-03-30T14:05:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8180000000000004e-06 2024-03-30T14:05:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:05:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:19.455660+0000", 2024-03-30T14:05:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0020410570000000002 2024-03-30T14:05:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:05:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:19.455679+0000", 2024-03-30T14:05:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8822e-05 2024-03-30T14:05:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:05:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:19.543046+0000", 2024-03-30T14:05:08.468 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.087366497000000001 2024-03-30T14:05:08.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.468 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:05:08.468 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:02:19.543319+0000", 2024-03-30T14:05:08.468 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00027307500000000001 2024-03-30T14:05:08.468 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.468 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.468 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.468 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "MOSDPGPush(11.1 422/418 [PushOp(11:89ed9e86:::benchmark_data_smithi043_197439_object96540:head, version: 414'6004, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(11:89ed9e86:::benchmark_data_smithi043_197439_object96540:head@414'6004, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-30T14:05:08.468 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:03:11.398344+0000", 2024-03-30T14:05:08.468 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 117.058952851, 2024-03-30T14:05:08.469 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.084948416999999998, 2024-03-30T14:05:08.469 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:05:08.469 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:05:08.469 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T14:05:08.469 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:05:08.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.469 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:05:08.469 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:11.398344+0000", 2024-03-30T14:05:08.469 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.469 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.469 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:05:08.469 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:11.398344+0000", 2024-03-30T14:05:08.469 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.470 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:05:08.470 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:11.398346+0000", 2024-03-30T14:05:08.470 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4729999999999999e-06 2024-03-30T14:05:08.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.470 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:05:08.470 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:11.398403+0000", 2024-03-30T14:05:08.470 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7292e-05 2024-03-30T14:05:08.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.470 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:05:08.470 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:11.398441+0000", 2024-03-30T14:05:08.470 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7883999999999998e-05 2024-03-30T14:05:08.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.471 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.471 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:05:08.471 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:11.398470+0000", 2024-03-30T14:05:08.471 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9428999999999999e-05 2024-03-30T14:05:08.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.471 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.471 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:05:08.471 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:11.398594+0000", 2024-03-30T14:05:08.471 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000123564 2024-03-30T14:05:08.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.471 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.471 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:05:08.471 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:11.398607+0000", 2024-03-30T14:05:08.471 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3315e-05 2024-03-30T14:05:08.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.472 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.472 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:05:08.472 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:11.483293+0000", 2024-03-30T14:05:08.472 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.084685460000000004 2024-03-30T14:05:08.472 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.472 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.472 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.472 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.472 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "pg_backfill(progress 11.1 e 422/422 lb 11:89ed7e52:::benchmark_data_smithi043_197439_object31944:head)", 2024-03-30T14:05:08.472 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:03:11.398517+0000", 2024-03-30T14:05:08.472 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 117.058780063, 2024-03-30T14:05:08.472 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.086708874000000005, 2024-03-30T14:05:08.472 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:05:08.473 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:05:08.473 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-03-30T14:05:08.473 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:05:08.473 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.473 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:05:08.473 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:11.398517+0000", 2024-03-30T14:05:08.473 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.473 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.473 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:05:08.473 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:11.398517+0000", 2024-03-30T14:05:08.473 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.473 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.473 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:05:08.474 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:11.398518+0000", 2024-03-30T14:05:08.474 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.9599999999999998e-07 2024-03-30T14:05:08.474 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.474 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.474 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:05:08.474 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:11.398538+0000", 2024-03-30T14:05:08.474 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0069e-05 2024-03-30T14:05:08.474 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.474 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.474 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:05:08.474 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:11.398545+0000", 2024-03-30T14:05:08.474 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0999999999999998e-06 2024-03-30T14:05:08.475 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.475 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.475 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:05:08.475 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:11.398550+0000", 2024-03-30T14:05:08.475 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3249999999999998e-06 2024-03-30T14:05:08.475 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.475 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.475 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:05:08.475 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:11.485175+0000", 2024-03-30T14:05:08.475 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.086625118000000001 2024-03-30T14:05:08.475 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.475 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.475 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:05:08.476 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:11.485191+0000", 2024-03-30T14:05:08.476 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5756999999999998e-05 2024-03-30T14:05:08.476 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.476 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.476 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:05:08.477 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:03:11.485226+0000", 2024-03-30T14:05:08.477 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4609e-05 2024-03-30T14:05:08.477 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.477 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.477 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.477 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.477 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.477 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "MOSDPGPull(4.7 e441/440 cost 1049576)", 2024-03-30T14:05:08.477 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:04:00.250422+0000", 2024-03-30T14:05:08.477 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 68.206874843999998, 2024-03-30T14:05:08.477 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.11102833600000001, 2024-03-30T14:05:08.477 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:05:08.477 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:05:08.478 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-03-30T14:05:08.478 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:05:08.478 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.478 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:05:08.478 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:00.250422+0000", 2024-03-30T14:05:08.478 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.478 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.478 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:05:08.478 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:00.250422+0000", 2024-03-30T14:05:08.478 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.478 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.478 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:05:08.479 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:00.250423+0000", 2024-03-30T14:05:08.479 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5700000000000002e-07 2024-03-30T14:05:08.479 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.479 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.479 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:05:08.479 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:00.250434+0000", 2024-03-30T14:05:08.479 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0998999999999999e-05 2024-03-30T14:05:08.479 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.479 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.479 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:05:08.479 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:00.250519+0000", 2024-03-30T14:05:08.479 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5409999999999997e-05 2024-03-30T14:05:08.479 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.480 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.480 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:05:08.480 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:00.250527+0000", 2024-03-30T14:05:08.480 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8730000000000007e-06 2024-03-30T14:05:08.480 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.480 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.480 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:05:08.480 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:00.361252+0000", 2024-03-30T14:05:08.480 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.110725251 2024-03-30T14:05:08.480 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.480 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.480 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:05:08.480 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:00.361451+0000", 2024-03-30T14:05:08.480 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00019824600000000001 2024-03-30T14:05:08.480 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.481 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.481 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.481 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.481 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.481 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "MOSDPGPull(4.7 e441/440 cost 1049576)", 2024-03-30T14:05:08.481 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:04:00.362759+0000", 2024-03-30T14:05:08.481 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 68.094538413999999, 2024-03-30T14:05:08.481 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.086688084999999998, 2024-03-30T14:05:08.481 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:05:08.481 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:05:08.481 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-03-30T14:05:08.481 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:05:08.481 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.481 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:05:08.482 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:00.362759+0000", 2024-03-30T14:05:08.482 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.482 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.482 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:05:08.482 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:00.362759+0000", 2024-03-30T14:05:08.482 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.482 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.482 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:05:08.482 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:00.362760+0000", 2024-03-30T14:05:08.482 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0500000000000002e-07 2024-03-30T14:05:08.483 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.483 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.483 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:05:08.483 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:00.362776+0000", 2024-03-30T14:05:08.483 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6611000000000001e-05 2024-03-30T14:05:08.483 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.483 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.483 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:05:08.483 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:00.362805+0000", 2024-03-30T14:05:08.483 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9337000000000001e-05 2024-03-30T14:05:08.483 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.483 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.483 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:05:08.483 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:00.362814+0000", 2024-03-30T14:05:08.484 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2670000000000006e-06 2024-03-30T14:05:08.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.484 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.484 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:05:08.484 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:00.449281+0000", 2024-03-30T14:05:08.484 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.086467701999999994 2024-03-30T14:05:08.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.484 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.484 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:05:08.484 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:00.449447+0000", 2024-03-30T14:05:08.484 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016526299999999999 2024-03-30T14:05:08.484 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.484 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.484 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.485 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.485 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.485 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:183701 11.1 e444/441 11:82ba9e72:::benchmark_data_smithi043_197439_object73583:head v 444'11413, mlcod=443'11411)", 2024-03-30T14:05:08.485 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:04:02.992568+0000", 2024-03-30T14:05:08.485 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 65.464729360000007, 2024-03-30T14:05:08.485 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.118190172, 2024-03-30T14:05:08.485 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:05:08.485 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:05:08.485 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:05:08.485 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:05:08.485 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.485 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:05:08.485 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:02.992568+0000", 2024-03-30T14:05:08.485 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.486 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.486 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.486 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:05:08.486 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:02.992568+0000", 2024-03-30T14:05:08.486 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.486 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.486 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.486 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:05:08.486 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:02.992568+0000", 2024-03-30T14:05:08.486 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0800000000000004e-07 2024-03-30T14:05:08.486 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.486 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.487 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:05:08.487 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:02.992575+0000", 2024-03-30T14:05:08.487 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0690000000000004e-06 2024-03-30T14:05:08.487 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.487 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.487 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:05:08.487 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:02.992586+0000", 2024-03-30T14:05:08.487 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0193999999999999e-05 2024-03-30T14:05:08.487 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.487 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.487 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:05:08.487 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:02.992593+0000", 2024-03-30T14:05:08.487 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5599999999999996e-06 2024-03-30T14:05:08.487 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.488 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.488 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:05:08.488 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:02.996626+0000", 2024-03-30T14:05:08.488 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0040326839999999999 2024-03-30T14:05:08.488 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.488 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.488 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:05:08.488 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:02.996661+0000", 2024-03-30T14:05:08.488 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5123000000000002e-05 2024-03-30T14:05:08.488 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.488 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.488 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:05:08.488 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.110710+0000", 2024-03-30T14:05:08.489 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.114048761 2024-03-30T14:05:08.489 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.489 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.489 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:05:08.489 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.110758+0000", 2024-03-30T14:05:08.489 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8072999999999999e-05 2024-03-30T14:05:08.489 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.489 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.489 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.489 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.489 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.489 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:183712 11.2 e444/441 11:432c06a2:::benchmark_data_smithi043_197439_object73594:head v 444'11352, mlcod=443'11350)", 2024-03-30T14:05:08.489 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:04:03.084107+0000", 2024-03-30T14:05:08.490 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 65.373189707999998, 2024-03-30T14:05:08.490 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.111041722, 2024-03-30T14:05:08.490 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:05:08.490 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:05:08.490 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:05:08.490 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:05:08.490 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.490 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:05:08.490 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.084107+0000", 2024-03-30T14:05:08.490 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.490 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.490 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.490 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:05:08.490 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.084107+0000", 2024-03-30T14:05:08.491 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.491 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.491 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:05:08.491 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.084108+0000", 2024-03-30T14:05:08.491 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.23e-07 2024-03-30T14:05:08.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.491 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.491 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:05:08.491 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.084115+0000", 2024-03-30T14:05:08.491 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1330000000000001e-06 2024-03-30T14:05:08.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.491 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.491 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:05:08.492 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.084205+0000", 2024-03-30T14:05:08.492 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.9413000000000006e-05 2024-03-30T14:05:08.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.492 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.492 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:05:08.492 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.084213+0000", 2024-03-30T14:05:08.492 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7309999999999999e-06 2024-03-30T14:05:08.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.492 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.492 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:05:08.492 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.086306+0000", 2024-03-30T14:05:08.492 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002092558 2024-03-30T14:05:08.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.492 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.492 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:05:08.493 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.086333+0000", 2024-03-30T14:05:08.493 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7458999999999999e-05 2024-03-30T14:05:08.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.493 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.493 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:05:08.493 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.194935+0000", 2024-03-30T14:05:08.493 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.10860141600000001 2024-03-30T14:05:08.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.493 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.493 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:05:08.493 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.195149+0000", 2024-03-30T14:05:08.493 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00021418899999999999 2024-03-30T14:05:08.493 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.493 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.493 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.494 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.494 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:183713 11.2 e444/441 11:4afaf2bf:::benchmark_data_smithi043_197439_object73595:head v 444'11353, mlcod=443'11350)", 2024-03-30T14:05:08.494 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:04:03.085097+0000", 2024-03-30T14:05:08.494 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 65.372200497999998, 2024-03-30T14:05:08.494 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.110088647, 2024-03-30T14:05:08.494 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:05:08.494 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:05:08.494 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:05:08.494 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:05:08.494 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.494 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:05:08.494 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.085097+0000", 2024-03-30T14:05:08.494 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.495 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.495 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.495 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:05:08.495 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.085097+0000", 2024-03-30T14:05:08.495 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.495 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.495 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.495 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:05:08.495 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.085097+0000", 2024-03-30T14:05:08.495 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1600000000000001e-07 2024-03-30T14:05:08.495 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.495 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.495 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:05:08.495 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.085104+0000", 2024-03-30T14:05:08.496 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.844e-06 2024-03-30T14:05:08.496 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.496 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.496 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:05:08.496 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.085115+0000", 2024-03-30T14:05:08.496 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0669000000000001e-05 2024-03-30T14:05:08.496 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.496 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.496 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:05:08.496 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.085122+0000", 2024-03-30T14:05:08.496 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.272e-06 2024-03-30T14:05:08.496 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.496 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.496 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:05:08.497 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.105726+0000", 2024-03-30T14:05:08.497 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.020604346999999999 2024-03-30T14:05:08.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.497 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.497 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:05:08.497 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.105750+0000", 2024-03-30T14:05:08.497 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3407000000000001e-05 2024-03-30T14:05:08.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.497 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.497 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:05:08.497 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.195155+0000", 2024-03-30T14:05:08.497 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.089404997 2024-03-30T14:05:08.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.497 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.498 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:05:08.498 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.195185+0000", 2024-03-30T14:05:08.498 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.0494999999999999e-05 2024-03-30T14:05:08.498 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.498 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.498 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.498 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.498 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:183728 11.1 e444/441 11:8a07049c:::benchmark_data_smithi043_197439_object73610:head v 444'11414, mlcod=443'11411)", 2024-03-30T14:05:08.498 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:04:03.090445+0000", 2024-03-30T14:05:08.498 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 65.366851928000003, 2024-03-30T14:05:08.498 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14000838400000001, 2024-03-30T14:05:08.498 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:05:08.499 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:05:08.499 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:05:08.499 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:05:08.499 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.499 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:05:08.499 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.090445+0000", 2024-03-30T14:05:08.499 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.499 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.499 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:05:08.499 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.090445+0000", 2024-03-30T14:05:08.499 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.499 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.499 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:05:08.500 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.090446+0000", 2024-03-30T14:05:08.500 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2300000000000001e-07 2024-03-30T14:05:08.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.500 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.500 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:05:08.500 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.090451+0000", 2024-03-30T14:05:08.500 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6559999999999997e-06 2024-03-30T14:05:08.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.500 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.500 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:05:08.500 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.090483+0000", 2024-03-30T14:05:08.500 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.1440999999999999e-05 2024-03-30T14:05:08.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.500 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.500 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:05:08.501 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.090490+0000", 2024-03-30T14:05:08.501 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6329999999999996e-06 2024-03-30T14:05:08.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.501 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.501 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:05:08.501 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.091708+0000", 2024-03-30T14:05:08.501 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0012173080000000001 2024-03-30T14:05:08.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.501 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.501 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:05:08.501 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.091732+0000", 2024-03-30T14:05:08.501 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4385000000000001e-05 2024-03-30T14:05:08.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.501 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.501 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:05:08.501 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.230407+0000", 2024-03-30T14:05:08.502 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13867526899999999 2024-03-30T14:05:08.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.502 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.502 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:05:08.502 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.230453+0000", 2024-03-30T14:05:08.502 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6168999999999998e-05 2024-03-30T14:05:08.502 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.502 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.502 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.502 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.502 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:183736 11.2 e444/441 11:4e6e3038:::benchmark_data_smithi043_197439_object73618:head v 444'11354, mlcod=443'11350)", 2024-03-30T14:05:08.503 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:04:03.100768+0000", 2024-03-30T14:05:08.503 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 65.356529316000007, 2024-03-30T14:05:08.503 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13087163900000001, 2024-03-30T14:05:08.503 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:05:08.503 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:05:08.503 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:05:08.503 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:05:08.503 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.503 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:05:08.503 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.100768+0000", 2024-03-30T14:05:08.503 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.503 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.503 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:05:08.503 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.100768+0000", 2024-03-30T14:05:08.504 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.504 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.504 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:05:08.504 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.100769+0000", 2024-03-30T14:05:08.504 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.08e-06 2024-03-30T14:05:08.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.504 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.504 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:05:08.504 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.100779+0000", 2024-03-30T14:05:08.504 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0618999999999999e-05 2024-03-30T14:05:08.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.504 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.504 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:05:08.504 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.100887+0000", 2024-03-30T14:05:08.505 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010773299999999999 2024-03-30T14:05:08.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.505 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.505 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:05:08.505 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.100898+0000", 2024-03-30T14:05:08.505 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1189000000000001e-05 2024-03-30T14:05:08.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.505 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.505 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:05:08.505 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.143399+0000", 2024-03-30T14:05:08.505 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.042501017000000002 2024-03-30T14:05:08.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.505 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.505 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:05:08.506 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.143445+0000", 2024-03-30T14:05:08.506 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6124000000000001e-05 2024-03-30T14:05:08.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.506 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.506 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:05:08.506 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.231608+0000", 2024-03-30T14:05:08.506 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.088162241000000002 2024-03-30T14:05:08.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.506 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.506 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:05:08.506 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.231639+0000", 2024-03-30T14:05:08.506 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.1636e-05 2024-03-30T14:05:08.506 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.507 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.507 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.507 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:183737 11.2 e444/441 11:4cbb1469:::benchmark_data_smithi043_197439_object73619:head v 444'11355, mlcod=443'11350)", 2024-03-30T14:05:08.507 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:04:03.103678+0000", 2024-03-30T14:05:08.507 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 65.353619015000007, 2024-03-30T14:05:08.507 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.12798309099999999, 2024-03-30T14:05:08.507 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:05:08.507 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:05:08.507 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:05:08.507 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:05:08.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.507 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:05:08.508 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.103678+0000", 2024-03-30T14:05:08.508 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.508 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:05:08.508 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.103678+0000", 2024-03-30T14:05:08.508 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.508 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:05:08.508 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.103679+0000", 2024-03-30T14:05:08.508 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0400000000000005e-07 2024-03-30T14:05:08.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.508 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:05:08.508 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.103686+0000", 2024-03-30T14:05:08.509 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4510000000000003e-06 2024-03-30T14:05:08.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.509 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:05:08.509 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.103773+0000", 2024-03-30T14:05:08.509 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6675999999999995e-05 2024-03-30T14:05:08.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.509 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:05:08.509 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.103781+0000", 2024-03-30T14:05:08.509 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9019999999999999e-06 2024-03-30T14:05:08.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.509 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:05:08.509 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.195214+0000", 2024-03-30T14:05:08.510 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.091433070000000005 2024-03-30T14:05:08.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.510 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:05:08.510 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.195239+0000", 2024-03-30T14:05:08.510 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4825999999999999e-05 2024-03-30T14:05:08.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.510 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:05:08.510 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.231645+0000", 2024-03-30T14:05:08.510 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.036406433000000002 2024-03-30T14:05:08.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.510 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:05:08.511 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.231661+0000", 2024-03-30T14:05:08.511 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5928999999999999e-05 2024-03-30T14:05:08.511 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.511 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.511 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.511 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:183742 11.f e444/441 11:f6a09116:::benchmark_data_smithi043_197439_object73624:head v 444'11380, mlcod=444'11378)", 2024-03-30T14:05:08.511 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:04:03.105366+0000", 2024-03-30T14:05:08.511 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 65.351930761999995, 2024-03-30T14:05:08.511 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.089649818000000006, 2024-03-30T14:05:08.511 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:05:08.511 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:05:08.511 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:05:08.511 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:05:08.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.512 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:05:08.512 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.105366+0000", 2024-03-30T14:05:08.512 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.512 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:05:08.512 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.105366+0000", 2024-03-30T14:05:08.512 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.512 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:05:08.512 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.105377+0000", 2024-03-30T14:05:08.512 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.063e-05 2024-03-30T14:05:08.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.513 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:05:08.513 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.105384+0000", 2024-03-30T14:05:08.513 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1760000000000004e-06 2024-03-30T14:05:08.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.513 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:05:08.513 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.105401+0000", 2024-03-30T14:05:08.513 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6668999999999999e-05 2024-03-30T14:05:08.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.513 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:05:08.513 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.105409+0000", 2024-03-30T14:05:08.513 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4290000000000001e-06 2024-03-30T14:05:08.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.514 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:05:08.514 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.105850+0000", 2024-03-30T14:05:08.514 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00044104000000000002 2024-03-30T14:05:08.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.514 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:05:08.514 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.105899+0000", 2024-03-30T14:05:08.514 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9061999999999999e-05 2024-03-30T14:05:08.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.514 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:05:08.514 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.194985+0000", 2024-03-30T14:05:08.514 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.089085551999999998 2024-03-30T14:05:08.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.515 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:05:08.515 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.195016+0000", 2024-03-30T14:05:08.515 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.1260000000000002e-05 2024-03-30T14:05:08.515 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.515 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.515 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.515 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:183743 11.2 e444/441 11:409ba4ec:::benchmark_data_smithi043_197439_object73625:head v 444'11356, mlcod=443'11350)", 2024-03-30T14:05:08.515 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:04:03.105579+0000", 2024-03-30T14:05:08.515 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 65.351718461999994, 2024-03-30T14:05:08.515 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.12610089699999999, 2024-03-30T14:05:08.515 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:05:08.516 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:05:08.516 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:05:08.516 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:05:08.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.516 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:05:08.516 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.105579+0000", 2024-03-30T14:05:08.516 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.516 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:05:08.516 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.105579+0000", 2024-03-30T14:05:08.516 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.516 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:05:08.517 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.105579+0000", 2024-03-30T14:05:08.517 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7800000000000001e-07 2024-03-30T14:05:08.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.517 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:05:08.517 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.105585+0000", 2024-03-30T14:05:08.517 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0209999999999996e-06 2024-03-30T14:05:08.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.517 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:05:08.517 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.105654+0000", 2024-03-30T14:05:08.517 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8323000000000003e-05 2024-03-30T14:05:08.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.517 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:05:08.517 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.105816+0000", 2024-03-30T14:05:08.518 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000162168 2024-03-30T14:05:08.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.518 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:05:08.518 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.228897+0000", 2024-03-30T14:05:08.518 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.12308137199999999 2024-03-30T14:05:08.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.518 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:05:08.518 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.228922+0000", 2024-03-30T14:05:08.518 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4871e-05 2024-03-30T14:05:08.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.519 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:05:08.519 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.231663+0000", 2024-03-30T14:05:08.519 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0027414789999999998 2024-03-30T14:05:08.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.519 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:05:08.519 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.231679+0000", 2024-03-30T14:05:08.519 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5985e-05 2024-03-30T14:05:08.519 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.519 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.519 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.520 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:183745 11.1 e444/441 11:83f9368d:::benchmark_data_smithi043_197439_object73627:head v 444'11415, mlcod=443'11411)", 2024-03-30T14:05:08.520 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:04:03.107256+0000", 2024-03-30T14:05:08.520 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 65.350041067000006, 2024-03-30T14:05:08.520 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.123300702, 2024-03-30T14:05:08.520 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:05:08.520 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:05:08.520 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:05:08.520 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:05:08.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.520 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:05:08.520 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.107256+0000", 2024-03-30T14:05:08.520 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.521 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:05:08.521 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.107256+0000", 2024-03-30T14:05:08.521 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.521 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:05:08.521 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.107257+0000", 2024-03-30T14:05:08.521 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0599999999999999e-07 2024-03-30T14:05:08.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.521 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:05:08.521 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.107265+0000", 2024-03-30T14:05:08.522 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1310000000000006e-06 2024-03-30T14:05:08.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.522 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:05:08.522 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.107277+0000", 2024-03-30T14:05:08.522 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2408e-05 2024-03-30T14:05:08.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.522 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:05:08.522 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.107285+0000", 2024-03-30T14:05:08.522 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7610000000000007e-06 2024-03-30T14:05:08.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.523 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:05:08.523 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.107358+0000", 2024-03-30T14:05:08.523 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2590999999999997e-05 2024-03-30T14:05:08.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.523 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.523 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:05:08.523 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.107380+0000", 2024-03-30T14:05:08.523 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1931999999999999e-05 2024-03-30T14:05:08.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.523 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.523 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:05:08.523 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.230457+0000", 2024-03-30T14:05:08.523 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.12307699499999999 2024-03-30T14:05:08.524 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.524 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:05:08.524 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.230557+0000", 2024-03-30T14:05:08.524 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9977999999999997e-05 2024-03-30T14:05:08.524 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.524 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.524 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.524 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.524 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:183752 11.1 e444/441 11:89e4e2df:::benchmark_data_smithi043_197439_object73634:head v 444'11416, mlcod=443'11412)", 2024-03-30T14:05:08.524 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:04:03.114274+0000", 2024-03-30T14:05:08.524 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 65.343022529999999, 2024-03-30T14:05:08.524 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.116316583, 2024-03-30T14:05:08.525 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:05:08.525 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:05:08.525 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:05:08.525 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:05:08.525 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.525 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:05:08.525 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.114274+0000", 2024-03-30T14:05:08.525 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.525 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.525 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:05:08.525 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.114274+0000", 2024-03-30T14:05:08.526 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.526 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.526 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:05:08.526 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.114275+0000", 2024-03-30T14:05:08.526 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1699999999999997e-07 2024-03-30T14:05:08.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.526 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.526 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:05:08.526 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.114288+0000", 2024-03-30T14:05:08.526 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2354000000000001e-05 2024-03-30T14:05:08.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.526 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.526 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:05:08.527 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.114298+0000", 2024-03-30T14:05:08.527 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0131e-05 2024-03-30T14:05:08.527 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.527 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.527 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:05:08.527 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.114304+0000", 2024-03-30T14:05:08.527 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5089999999999999e-06 2024-03-30T14:05:08.527 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.527 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.527 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:05:08.527 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.114370+0000", 2024-03-30T14:05:08.527 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5674999999999997e-05 2024-03-30T14:05:08.527 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.528 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.528 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:05:08.528 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.114393+0000", 2024-03-30T14:05:08.528 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2708999999999999e-05 2024-03-30T14:05:08.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.528 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.528 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:05:08.528 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.230560+0000", 2024-03-30T14:05:08.528 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.116167323 2024-03-30T14:05:08.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.528 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.528 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:05:08.528 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.230591+0000", 2024-03-30T14:05:08.528 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.1164999999999997e-05 2024-03-30T14:05:08.529 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.529 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.529 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.529 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.529 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.529 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:183751 11.2 e444/441 11:4b7628f8:::benchmark_data_smithi043_197439_object73633:head v 444'11357, mlcod=443'11350)", 2024-03-30T14:05:08.529 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:04:03.115834+0000", 2024-03-30T14:05:08.529 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 65.341463153999996, 2024-03-30T14:05:08.529 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.116787077, 2024-03-30T14:05:08.529 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:05:08.529 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:05:08.529 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:05:08.529 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:05:08.529 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.529 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:05:08.530 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.115834+0000", 2024-03-30T14:05:08.530 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.530 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.530 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:05:08.530 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.115834+0000", 2024-03-30T14:05:08.530 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.530 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.530 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:05:08.530 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.115835+0000", 2024-03-30T14:05:08.530 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5199999999999996e-07 2024-03-30T14:05:08.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.531 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.531 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:05:08.531 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.115841+0000", 2024-03-30T14:05:08.531 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8809999999999998e-06 2024-03-30T14:05:08.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.531 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.531 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:05:08.531 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.115852+0000", 2024-03-30T14:05:08.531 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0166999999999999e-05 2024-03-30T14:05:08.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.531 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.531 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:05:08.531 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.115858+0000", 2024-03-30T14:05:08.531 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7120000000000003e-06 2024-03-30T14:05:08.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.532 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.532 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:05:08.532 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.230425+0000", 2024-03-30T14:05:08.532 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.11456665000000001 2024-03-30T14:05:08.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.532 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.532 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:05:08.532 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.230449+0000", 2024-03-30T14:05:08.532 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3862e-05 2024-03-30T14:05:08.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.532 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.532 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:05:08.532 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.232576+0000", 2024-03-30T14:05:08.533 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0021272959999999999 2024-03-30T14:05:08.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.533 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:05:08.533 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.232621+0000", 2024-03-30T14:05:08.533 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4857000000000001e-05 2024-03-30T14:05:08.533 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.533 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.533 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.533 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25685.0:183754 11.1 e444/441 11:82383151:::benchmark_data_smithi043_197439_object73636:head v 444'11417, mlcod=443'11412)", 2024-03-30T14:05:08.533 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-03-30T14:04:03.120258+0000", 2024-03-30T14:05:08.533 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 65.337039239999996, 2024-03-30T14:05:08.533 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.110364957, 2024-03-30T14:05:08.534 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-03-30T14:05:08.534 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-03-30T14:05:08.534 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:05:08.534 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-03-30T14:05:08.534 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.534 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-03-30T14:05:08.534 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.120258+0000", 2024-03-30T14:05:08.534 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.534 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.534 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-03-30T14:05:08.534 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.120258+0000", 2024-03-30T14:05:08.534 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-03-30T14:05:08.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.535 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-03-30T14:05:08.535 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.120259+0000", 2024-03-30T14:05:08.535 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7100000000000001e-07 2024-03-30T14:05:08.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.535 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-03-30T14:05:08.535 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.120266+0000", 2024-03-30T14:05:08.535 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9189999999999999e-06 2024-03-30T14:05:08.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.535 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-03-30T14:05:08.536 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.120285+0000", 2024-03-30T14:05:08.536 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8165e-05 2024-03-30T14:05:08.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.536 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.536 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-03-30T14:05:08.536 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.120293+0000", 2024-03-30T14:05:08.536 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6950000000000006e-06 2024-03-30T14:05:08.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.536 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.536 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-03-30T14:05:08.536 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.142111+0000", 2024-03-30T14:05:08.536 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.021817861000000001 2024-03-30T14:05:08.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.536 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.537 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-03-30T14:05:08.537 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.142134+0000", 2024-03-30T14:05:08.537 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2917000000000001e-05 2024-03-30T14:05:08.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.537 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.537 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-03-30T14:05:08.537 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.230593+0000", 2024-03-30T14:05:08.537 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.088459206999999998 2024-03-30T14:05:08.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-30T14:05:08.537 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-30T14:05:08.537 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-03-30T14:05:08.537 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-03-30T14:04:03.230623+0000", 2024-03-30T14:05:08.537 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9422000000000001e-05 2024-03-30T14:05:08.537 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.538 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.538 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.538 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-30T14:05:08.538 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-30T14:05:08.538 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-30T14:05:08.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:08 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a[29940]: 2024-03-30T14:05:08.421+0000 7ff6065d3700 -1 mon.a@0(leader).osd e487 definitely_dead 0 2024-03-30T14:05:09.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:08 smithi043 ceph-mon[29963]: pgmap v1059: 152 pgs: 1 active+undersized+remapped, 3 peering, 148 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 470 B/s, 13 objects/s recovering 2024-03-30T14:05:09.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:08 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/668047538' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-30T14:05:09.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:08 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-30T14:05:09.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:08 smithi043 ceph-mon[42099]: pgmap v1059: 152 pgs: 1 active+undersized+remapped, 3 peering, 148 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 470 B/s, 13 objects/s recovering 2024-03-30T14:05:09.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:08 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/668047538' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-30T14:05:09.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:08 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-30T14:05:09.349 INFO:teuthology.orchestra.run.smithi043.stderr:marked in osd.7. 2024-03-30T14:05:09.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:08 smithi113 ceph-mon[34341]: pgmap v1059: 152 pgs: 1 active+undersized+remapped, 3 peering, 148 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 470 B/s, 13 objects/s recovering 2024-03-30T14:05:09.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:08 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/668047538' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-30T14:05:09.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:08 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-30T14:05:09.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:05:09 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:05:09.974 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-03-30T14:05:09.975 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-30T14:05:09.975 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd pool set .mgr pgp_num 21 2024-03-30T14:05:10.231 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:05:10.343 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:10 smithi043 ceph-mon[29963]: 1.0 scrub starts 2024-03-30T14:05:10.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:10 smithi043 ceph-mon[29963]: 1.0 scrub ok 2024-03-30T14:05:10.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:10 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-03-30T14:05:10.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:10 smithi043 ceph-mon[29963]: osdmap e488: 8 total, 8 up, 8 in 2024-03-30T14:05:10.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:10 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:10.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:10 smithi043 ceph-mon[29963]: pgmap v1061: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 255 B/s, 7 objects/s recovering 2024-03-30T14:05:10.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:10 smithi043 ceph-mon[42099]: 1.0 scrub starts 2024-03-30T14:05:10.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:10 smithi043 ceph-mon[42099]: 1.0 scrub ok 2024-03-30T14:05:10.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:10 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-03-30T14:05:10.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:10 smithi043 ceph-mon[42099]: osdmap e488: 8 total, 8 up, 8 in 2024-03-30T14:05:10.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:10 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:10.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:10 smithi043 ceph-mon[42099]: pgmap v1061: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 255 B/s, 7 objects/s recovering 2024-03-30T14:05:10.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:10 smithi113 ceph-mon[34341]: 1.0 scrub starts 2024-03-30T14:05:10.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:10 smithi113 ceph-mon[34341]: 1.0 scrub ok 2024-03-30T14:05:10.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:10 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-03-30T14:05:10.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:10 smithi113 ceph-mon[34341]: osdmap e488: 8 total, 8 up, 8 in 2024-03-30T14:05:10.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:10 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:10.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:10 smithi113 ceph-mon[34341]: pgmap v1061: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 255 B/s, 7 objects/s recovering 2024-03-30T14:05:11.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:11 smithi043 ceph-mon[29963]: 4.1d scrub starts 2024-03-30T14:05:11.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:11 smithi043 ceph-mon[29963]: 4.1d scrub ok 2024-03-30T14:05:11.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:11 smithi043 ceph-mon[29963]: 4.4 scrub starts 2024-03-30T14:05:11.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:11 smithi043 ceph-mon[29963]: 4.4 scrub ok 2024-03-30T14:05:11.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:11 smithi043 ceph-mon[29963]: osdmap e489: 8 total, 8 up, 8 in 2024-03-30T14:05:11.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:11 smithi043 ceph-mon[42099]: 4.1d scrub starts 2024-03-30T14:05:11.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:11 smithi043 ceph-mon[42099]: 4.1d scrub ok 2024-03-30T14:05:11.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:11 smithi043 ceph-mon[42099]: 4.4 scrub starts 2024-03-30T14:05:11.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:11 smithi043 ceph-mon[42099]: 4.4 scrub ok 2024-03-30T14:05:11.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:11 smithi043 ceph-mon[42099]: osdmap e489: 8 total, 8 up, 8 in 2024-03-30T14:05:11.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:11 smithi113 ceph-mon[34341]: 4.1d scrub starts 2024-03-30T14:05:11.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:11 smithi113 ceph-mon[34341]: 4.1d scrub ok 2024-03-30T14:05:11.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:11 smithi113 ceph-mon[34341]: 4.4 scrub starts 2024-03-30T14:05:11.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:11 smithi113 ceph-mon[34341]: 4.4 scrub ok 2024-03-30T14:05:11.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:11 smithi113 ceph-mon[34341]: osdmap e489: 8 total, 8 up, 8 in 2024-03-30T14:05:12.364 INFO:teuthology.orchestra.run.smithi043.stderr:set pool 1 pgp_num to 21 2024-03-30T14:05:12.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:12 smithi043 ceph-mon[29963]: 5.a scrub starts 2024-03-30T14:05:12.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:12 smithi043 ceph-mon[29963]: 5.a scrub ok 2024-03-30T14:05:12.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:12 smithi043 ceph-mon[29963]: osdmap e490: 8 total, 8 up, 8 in 2024-03-30T14:05:12.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:12 smithi043 ceph-mon[29963]: pgmap v1064: 152 pgs: 2 active+undersized+remapped, 4 remapped+peering, 1 active+remapped, 145 active+clean; 609 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 3.0 KiB/s rd, 18 KiB/s wr, 3 op/s; 13 KiB/s, 4 objects/s recovering 2024-03-30T14:05:12.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:12 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2975406517' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-03-30T14:05:12.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:12 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-03-30T14:05:12.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:12 smithi043 ceph-mon[42099]: 5.a scrub starts 2024-03-30T14:05:12.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:12 smithi043 ceph-mon[42099]: 5.a scrub ok 2024-03-30T14:05:12.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:12 smithi043 ceph-mon[42099]: osdmap e490: 8 total, 8 up, 8 in 2024-03-30T14:05:12.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:12 smithi043 ceph-mon[42099]: pgmap v1064: 152 pgs: 2 active+undersized+remapped, 4 remapped+peering, 1 active+remapped, 145 active+clean; 609 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 3.0 KiB/s rd, 18 KiB/s wr, 3 op/s; 13 KiB/s, 4 objects/s recovering 2024-03-30T14:05:12.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:12 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2975406517' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-03-30T14:05:12.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:12 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-03-30T14:05:12.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:12 smithi113 ceph-mon[34341]: 5.a scrub starts 2024-03-30T14:05:12.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:12 smithi113 ceph-mon[34341]: 5.a scrub ok 2024-03-30T14:05:12.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:12 smithi113 ceph-mon[34341]: osdmap e490: 8 total, 8 up, 8 in 2024-03-30T14:05:12.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:12 smithi113 ceph-mon[34341]: pgmap v1064: 152 pgs: 2 active+undersized+remapped, 4 remapped+peering, 1 active+remapped, 145 active+clean; 609 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 3.0 KiB/s rd, 18 KiB/s wr, 3 op/s; 13 KiB/s, 4 objects/s recovering 2024-03-30T14:05:12.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:12 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2975406517' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-03-30T14:05:12.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:12 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-03-30T14:05:13.074 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-30T14:05:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:05:13.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:13 smithi043 ceph-mon[29963]: 5.3 scrub starts 2024-03-30T14:05:13.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:13 smithi043 ceph-mon[29963]: 5.3 scrub ok 2024-03-30T14:05:13.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:13 smithi043 ceph-mon[29963]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-30T14:05:13.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:13 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-03-30T14:05:13.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:13 smithi043 ceph-mon[29963]: 5.12 scrub starts 2024-03-30T14:05:13.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:13 smithi043 ceph-mon[29963]: 5.12 scrub ok 2024-03-30T14:05:13.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:13 smithi043 ceph-mon[29963]: osdmap e491: 8 total, 8 up, 8 in 2024-03-30T14:05:13.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:13 smithi043 ceph-mon[42099]: 5.3 scrub starts 2024-03-30T14:05:13.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:13 smithi043 ceph-mon[42099]: 5.3 scrub ok 2024-03-30T14:05:13.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:13 smithi043 ceph-mon[42099]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-30T14:05:13.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:13 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-03-30T14:05:13.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:13 smithi043 ceph-mon[42099]: 5.12 scrub starts 2024-03-30T14:05:13.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:13 smithi043 ceph-mon[42099]: 5.12 scrub ok 2024-03-30T14:05:13.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:13 smithi043 ceph-mon[42099]: osdmap e491: 8 total, 8 up, 8 in 2024-03-30T14:05:13.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:13 smithi113 ceph-mon[34341]: 5.3 scrub starts 2024-03-30T14:05:13.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:13 smithi113 ceph-mon[34341]: 5.3 scrub ok 2024-03-30T14:05:13.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:13 smithi113 ceph-mon[34341]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-30T14:05:13.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:13 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-03-30T14:05:13.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:13 smithi113 ceph-mon[34341]: 5.12 scrub starts 2024-03-30T14:05:13.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:13 smithi113 ceph-mon[34341]: 5.12 scrub ok 2024-03-30T14:05:13.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:13 smithi113 ceph-mon[34341]: osdmap e491: 8 total, 8 up, 8 in 2024-03-30T14:05:14.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:14 smithi043 ceph-mon[29963]: osdmap e492: 8 total, 8 up, 8 in 2024-03-30T14:05:14.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:14 smithi043 ceph-mon[29963]: 1.10 scrub starts 2024-03-30T14:05:14.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:14 smithi043 ceph-mon[29963]: 1.10 scrub ok 2024-03-30T14:05:14.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:14 smithi043 ceph-mon[29963]: pgmap v1067: 152 pgs: 1 active+undersized+remapped, 151 active+clean; 609 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 27 KiB/s wr, 1 op/s; 19 KiB/s, 4 objects/s recovering 2024-03-30T14:05:14.602 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:14 smithi043 ceph-mon[42099]: osdmap e492: 8 total, 8 up, 8 in 2024-03-30T14:05:14.603 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:14 smithi043 ceph-mon[42099]: 1.10 scrub starts 2024-03-30T14:05:14.603 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:14 smithi043 ceph-mon[42099]: 1.10 scrub ok 2024-03-30T14:05:14.603 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:14 smithi043 ceph-mon[42099]: pgmap v1067: 152 pgs: 1 active+undersized+remapped, 151 active+clean; 609 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 27 KiB/s wr, 1 op/s; 19 KiB/s, 4 objects/s recovering 2024-03-30T14:05:14.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:14 smithi113 ceph-mon[34341]: osdmap e492: 8 total, 8 up, 8 in 2024-03-30T14:05:14.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:14 smithi113 ceph-mon[34341]: 1.10 scrub starts 2024-03-30T14:05:14.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:14 smithi113 ceph-mon[34341]: 1.10 scrub ok 2024-03-30T14:05:14.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:14 smithi113 ceph-mon[34341]: pgmap v1067: 152 pgs: 1 active+undersized+remapped, 151 active+clean; 609 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 27 KiB/s wr, 1 op/s; 19 KiB/s, 4 objects/s recovering 2024-03-30T14:05:15.173 INFO:teuthology.orchestra.run.smithi043.stdout:mon.a: {} 2024-03-30T14:05:15.184 INFO:teuthology.orchestra.run.smithi043.stdout:mon.b: {} 2024-03-30T14:05:15.194 INFO:teuthology.orchestra.run.smithi043.stdout:mon.c: {} 2024-03-30T14:05:15.195 INFO:teuthology.orchestra.run.smithi043.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-30T14:05:15.195 INFO:teuthology.orchestra.run.smithi043.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-30T14:05:15.195 INFO:teuthology.orchestra.run.smithi043.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-30T14:05:15.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:15 smithi043 ceph-mon[29963]: 3.8 scrub starts 2024-03-30T14:05:15.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:15 smithi043 ceph-mon[29963]: 3.8 scrub ok 2024-03-30T14:05:15.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:15 smithi043 ceph-mon[29963]: 5.1c scrub starts 2024-03-30T14:05:15.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:15 smithi043 ceph-mon[29963]: 5.1c scrub ok 2024-03-30T14:05:15.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:15 smithi043 ceph-mon[29963]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-30T14:05:15.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:15 smithi043 ceph-mon[29963]: Cluster is now healthy 2024-03-30T14:05:15.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:15 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:15.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:15 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3389496412' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:05:15.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:15 smithi043 ceph-mon[42099]: 3.8 scrub starts 2024-03-30T14:05:15.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:15 smithi043 ceph-mon[42099]: 3.8 scrub ok 2024-03-30T14:05:15.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:15 smithi043 ceph-mon[42099]: 5.1c scrub starts 2024-03-30T14:05:15.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:15 smithi043 ceph-mon[42099]: 5.1c scrub ok 2024-03-30T14:05:15.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:15 smithi043 ceph-mon[42099]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-30T14:05:15.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:15 smithi043 ceph-mon[42099]: Cluster is now healthy 2024-03-30T14:05:15.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:15 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:15.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:15 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3389496412' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:05:15.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:15 smithi113 ceph-mon[34341]: 3.8 scrub starts 2024-03-30T14:05:15.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:15 smithi113 ceph-mon[34341]: 3.8 scrub ok 2024-03-30T14:05:15.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:15 smithi113 ceph-mon[34341]: 5.1c scrub starts 2024-03-30T14:05:15.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:15 smithi113 ceph-mon[34341]: 5.1c scrub ok 2024-03-30T14:05:15.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:15 smithi113 ceph-mon[34341]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-30T14:05:15.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:15 smithi113 ceph-mon[34341]: Cluster is now healthy 2024-03-30T14:05:15.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:15 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:15.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:15 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3389496412' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:05:15.972 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd reweight 0 1 2024-03-30T14:05:16.221 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:05:16.343 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:05:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:05:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:05:16.621 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:16 smithi043 ceph-mon[29963]: 5.11 scrub starts 2024-03-30T14:05:16.621 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:16 smithi043 ceph-mon[29963]: 5.11 scrub ok 2024-03-30T14:05:16.621 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:16 smithi043 ceph-mon[29963]: 4.1e deep-scrub starts 2024-03-30T14:05:16.621 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:16 smithi043 ceph-mon[29963]: 4.1e deep-scrub ok 2024-03-30T14:05:16.621 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:16 smithi043 ceph-mon[29963]: pgmap v1068: 152 pgs: 1 active+undersized+remapped, 151 active+clean; 609 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 20 KiB/s wr, 1 op/s; 14 KiB/s, 6 objects/s recovering 2024-03-30T14:05:16.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:16 smithi043 ceph-mon[42099]: 5.11 scrub starts 2024-03-30T14:05:16.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:16 smithi043 ceph-mon[42099]: 5.11 scrub ok 2024-03-30T14:05:16.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:16 smithi043 ceph-mon[42099]: 4.1e deep-scrub starts 2024-03-30T14:05:16.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:16 smithi043 ceph-mon[42099]: 4.1e deep-scrub ok 2024-03-30T14:05:16.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:16 smithi043 ceph-mon[42099]: pgmap v1068: 152 pgs: 1 active+undersized+remapped, 151 active+clean; 609 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 20 KiB/s wr, 1 op/s; 14 KiB/s, 6 objects/s recovering 2024-03-30T14:05:16.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:16 smithi113 ceph-mon[34341]: 5.11 scrub starts 2024-03-30T14:05:16.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:16 smithi113 ceph-mon[34341]: 5.11 scrub ok 2024-03-30T14:05:16.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:16 smithi113 ceph-mon[34341]: 4.1e deep-scrub starts 2024-03-30T14:05:16.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:16 smithi113 ceph-mon[34341]: 4.1e deep-scrub ok 2024-03-30T14:05:16.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:16 smithi113 ceph-mon[34341]: pgmap v1068: 152 pgs: 1 active+undersized+remapped, 151 active+clean; 609 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 20 KiB/s wr, 1 op/s; 14 KiB/s, 6 objects/s recovering 2024-03-30T14:05:17.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:17 smithi043 ceph-mon[29963]: 3.1a scrub starts 2024-03-30T14:05:17.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:17 smithi043 ceph-mon[29963]: 3.1a scrub ok 2024-03-30T14:05:17.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:17 smithi043 ceph-mon[29963]: 2.2 scrub starts 2024-03-30T14:05:17.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:17 smithi043 ceph-mon[29963]: 2.2 scrub ok 2024-03-30T14:05:17.639 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:17 smithi043 ceph-mon[42099]: 3.1a scrub starts 2024-03-30T14:05:17.639 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:17 smithi043 ceph-mon[42099]: 3.1a scrub ok 2024-03-30T14:05:17.640 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:17 smithi043 ceph-mon[42099]: 2.2 scrub starts 2024-03-30T14:05:17.640 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:17 smithi043 ceph-mon[42099]: 2.2 scrub ok 2024-03-30T14:05:17.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:17 smithi113 ceph-mon[34341]: 3.1a scrub starts 2024-03-30T14:05:17.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:17 smithi113 ceph-mon[34341]: 3.1a scrub ok 2024-03-30T14:05:17.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:17 smithi113 ceph-mon[34341]: 2.2 scrub starts 2024-03-30T14:05:17.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:17 smithi113 ceph-mon[34341]: 2.2 scrub ok 2024-03-30T14:05:18.994 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T14:05:19.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:18 smithi043 ceph-mon[29963]: pgmap v1069: 152 pgs: 152 active+clean; 609 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 5 objects/s recovering 2024-03-30T14:05:19.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:18 smithi043 ceph-mon[29963]: osdmap e493: 8 total, 8 up, 8 in 2024-03-30T14:05:19.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:18 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2852655188' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T14:05:19.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:18 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T14:05:19.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:18 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:05:19.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:18 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:05:19.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:18 smithi043 ceph-mon[29963]: 5.1f scrub starts 2024-03-30T14:05:19.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:18 smithi043 ceph-mon[29963]: 5.1f scrub ok 2024-03-30T14:05:19.382 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:18 smithi043 ceph-mon[42099]: pgmap v1069: 152 pgs: 152 active+clean; 609 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 5 objects/s recovering 2024-03-30T14:05:19.382 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:18 smithi043 ceph-mon[42099]: osdmap e493: 8 total, 8 up, 8 in 2024-03-30T14:05:19.383 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:18 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2852655188' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T14:05:19.383 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:18 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T14:05:19.383 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:18 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:05:19.383 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:18 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:05:19.383 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:18 smithi043 ceph-mon[42099]: 5.1f scrub starts 2024-03-30T14:05:19.383 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:18 smithi043 ceph-mon[42099]: 5.1f scrub ok 2024-03-30T14:05:19.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:18 smithi113 ceph-mon[34341]: pgmap v1069: 152 pgs: 152 active+clean; 609 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 5 objects/s recovering 2024-03-30T14:05:19.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:18 smithi113 ceph-mon[34341]: osdmap e493: 8 total, 8 up, 8 in 2024-03-30T14:05:19.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:18 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2852655188' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T14:05:19.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:18 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T14:05:19.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:18 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:05:19.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:18 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:05:19.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:18 smithi113 ceph-mon[34341]: 5.1f scrub starts 2024-03-30T14:05:19.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:18 smithi113 ceph-mon[34341]: 5.1f scrub ok 2024-03-30T14:05:19.733 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:05:19 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:05:19.752 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 0 1 2024-03-30T14:05:20.000 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:05:20.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:19 smithi043 ceph-mon[29963]: 3.16 scrub starts 2024-03-30T14:05:20.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:19 smithi043 ceph-mon[29963]: 3.16 scrub ok 2024-03-30T14:05:20.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:19 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-30T14:05:20.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:19 smithi043 ceph-mon[29963]: osdmap e494: 8 total, 8 up, 8 in 2024-03-30T14:05:20.245 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:19 smithi043 ceph-mon[42099]: 3.16 scrub starts 2024-03-30T14:05:20.245 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:19 smithi043 ceph-mon[42099]: 3.16 scrub ok 2024-03-30T14:05:20.245 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:19 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-30T14:05:20.245 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:19 smithi043 ceph-mon[42099]: osdmap e494: 8 total, 8 up, 8 in 2024-03-30T14:05:20.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:19 smithi113 ceph-mon[34341]: 3.16 scrub starts 2024-03-30T14:05:20.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:19 smithi113 ceph-mon[34341]: 3.16 scrub ok 2024-03-30T14:05:20.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:19 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-30T14:05:20.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:19 smithi113 ceph-mon[34341]: osdmap e494: 8 total, 8 up, 8 in 2024-03-30T14:05:21.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:20 smithi043 ceph-mon[29963]: 6.1d deep-scrub starts 2024-03-30T14:05:21.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:20 smithi043 ceph-mon[29963]: 6.1d deep-scrub ok 2024-03-30T14:05:21.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:20 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:21.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:20 smithi043 ceph-mon[29963]: 6.15 deep-scrub starts 2024-03-30T14:05:21.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:20 smithi043 ceph-mon[29963]: pgmap v1072: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 5 objects/s recovering 2024-03-30T14:05:21.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:20 smithi043 ceph-mon[29963]: 6.15 deep-scrub ok 2024-03-30T14:05:21.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:20 smithi043 ceph-mon[29963]: 4.14 scrub starts 2024-03-30T14:05:21.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:20 smithi043 ceph-mon[29963]: 4.14 scrub ok 2024-03-30T14:05:21.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:20 smithi043 ceph-mon[42099]: 6.1d deep-scrub starts 2024-03-30T14:05:21.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:20 smithi043 ceph-mon[42099]: 6.1d deep-scrub ok 2024-03-30T14:05:21.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:20 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:21.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:20 smithi043 ceph-mon[42099]: 6.15 deep-scrub starts 2024-03-30T14:05:21.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:20 smithi043 ceph-mon[42099]: pgmap v1072: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 5 objects/s recovering 2024-03-30T14:05:21.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:20 smithi043 ceph-mon[42099]: 6.15 deep-scrub ok 2024-03-30T14:05:21.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:20 smithi043 ceph-mon[42099]: 4.14 scrub starts 2024-03-30T14:05:21.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:20 smithi043 ceph-mon[42099]: 4.14 scrub ok 2024-03-30T14:05:21.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:20 smithi113 ceph-mon[34341]: 6.1d deep-scrub starts 2024-03-30T14:05:21.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:20 smithi113 ceph-mon[34341]: 6.1d deep-scrub ok 2024-03-30T14:05:21.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:20 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:21.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:20 smithi113 ceph-mon[34341]: 6.15 deep-scrub starts 2024-03-30T14:05:21.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:20 smithi113 ceph-mon[34341]: pgmap v1072: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 5 objects/s recovering 2024-03-30T14:05:21.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:20 smithi113 ceph-mon[34341]: 6.15 deep-scrub ok 2024-03-30T14:05:21.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:20 smithi113 ceph-mon[34341]: 4.14 scrub starts 2024-03-30T14:05:21.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:20 smithi113 ceph-mon[34341]: 4.14 scrub ok 2024-03-30T14:05:22.003 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T14:05:22.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:22 smithi043 ceph-mon[29963]: 4.5 scrub starts 2024-03-30T14:05:22.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:22 smithi043 ceph-mon[29963]: 4.5 scrub ok 2024-03-30T14:05:22.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:22 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/960392419' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T14:05:22.322 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:22 smithi043 ceph-mon[42099]: 4.5 scrub starts 2024-03-30T14:05:22.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:22 smithi043 ceph-mon[42099]: 4.5 scrub ok 2024-03-30T14:05:22.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:22 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/960392419' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T14:05:22.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:22 smithi113 ceph-mon[34341]: 4.5 scrub starts 2024-03-30T14:05:22.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:22 smithi113 ceph-mon[34341]: 4.5 scrub ok 2024-03-30T14:05:22.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:22 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/960392419' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T14:05:22.727 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd reweight 1 1 2024-03-30T14:05:22.974 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:05:23.011 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:23 smithi043 ceph-mon[29963]: 4.d scrub starts 2024-03-30T14:05:23.011 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:23 smithi043 ceph-mon[29963]: 4.d scrub ok 2024-03-30T14:05:23.011 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:23 smithi043 ceph-mon[29963]: pgmap v1073: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 2 objects/s recovering 2024-03-30T14:05:23.011 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:23 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/960392419' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-30T14:05:23.011 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:23 smithi043 ceph-mon[29963]: osdmap e495: 8 total, 8 up, 8 in 2024-03-30T14:05:23.286 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:23 smithi043 ceph-mon[42099]: 4.d scrub starts 2024-03-30T14:05:23.286 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:23 smithi043 ceph-mon[42099]: 4.d scrub ok 2024-03-30T14:05:23.286 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:23 smithi043 ceph-mon[42099]: pgmap v1073: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 2 objects/s recovering 2024-03-30T14:05:23.286 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:23 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/960392419' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-30T14:05:23.286 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:23 smithi043 ceph-mon[42099]: osdmap e495: 8 total, 8 up, 8 in 2024-03-30T14:05:23.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:23 smithi113 ceph-mon[34341]: 4.d scrub starts 2024-03-30T14:05:23.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:23 smithi113 ceph-mon[34341]: 4.d scrub ok 2024-03-30T14:05:23.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:23 smithi113 ceph-mon[34341]: pgmap v1073: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 2 objects/s recovering 2024-03-30T14:05:23.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:23 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/960392419' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-30T14:05:23.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:23 smithi113 ceph-mon[34341]: osdmap e495: 8 total, 8 up, 8 in 2024-03-30T14:05:24.255 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:24 smithi043 ceph-mon[29963]: 4.15 scrub starts 2024-03-30T14:05:24.255 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:24 smithi043 ceph-mon[29963]: 4.15 scrub ok 2024-03-30T14:05:24.255 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:24 smithi043 ceph-mon[29963]: osdmap e496: 8 total, 8 up, 8 in 2024-03-30T14:05:24.255 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:24 smithi043 ceph-mon[42099]: 4.15 scrub starts 2024-03-30T14:05:24.255 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:24 smithi043 ceph-mon[42099]: 4.15 scrub ok 2024-03-30T14:05:24.255 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:24 smithi043 ceph-mon[42099]: osdmap e496: 8 total, 8 up, 8 in 2024-03-30T14:05:24.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:24 smithi113 ceph-mon[34341]: 4.15 scrub starts 2024-03-30T14:05:24.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:24 smithi113 ceph-mon[34341]: 4.15 scrub ok 2024-03-30T14:05:24.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:24 smithi113 ceph-mon[34341]: osdmap e496: 8 total, 8 up, 8 in 2024-03-30T14:05:25.071 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T14:05:25.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:25 smithi043 ceph-mon[29963]: 4.2 scrub starts 2024-03-30T14:05:25.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:25 smithi043 ceph-mon[29963]: 4.2 scrub ok 2024-03-30T14:05:25.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:25 smithi043 ceph-mon[29963]: 1.1 scrub starts 2024-03-30T14:05:25.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:25 smithi043 ceph-mon[29963]: 1.1 scrub ok 2024-03-30T14:05:25.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:25 smithi043 ceph-mon[29963]: 6.5 scrub starts 2024-03-30T14:05:25.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:25 smithi043 ceph-mon[29963]: 6.5 scrub ok 2024-03-30T14:05:25.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:25 smithi043 ceph-mon[29963]: pgmap v1076: 152 pgs: 152 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-30T14:05:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:25 smithi043 ceph-mon[42099]: 4.2 scrub starts 2024-03-30T14:05:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:25 smithi043 ceph-mon[42099]: 4.2 scrub ok 2024-03-30T14:05:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:25 smithi043 ceph-mon[42099]: 1.1 scrub starts 2024-03-30T14:05:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:25 smithi043 ceph-mon[42099]: 1.1 scrub ok 2024-03-30T14:05:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:25 smithi043 ceph-mon[42099]: 6.5 scrub starts 2024-03-30T14:05:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:25 smithi043 ceph-mon[42099]: 6.5 scrub ok 2024-03-30T14:05:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:25 smithi043 ceph-mon[42099]: pgmap v1076: 152 pgs: 152 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-30T14:05:25.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:25 smithi113 ceph-mon[34341]: 4.2 scrub starts 2024-03-30T14:05:25.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:25 smithi113 ceph-mon[34341]: 4.2 scrub ok 2024-03-30T14:05:25.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:25 smithi113 ceph-mon[34341]: 1.1 scrub starts 2024-03-30T14:05:25.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:25 smithi113 ceph-mon[34341]: 1.1 scrub ok 2024-03-30T14:05:25.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:25 smithi113 ceph-mon[34341]: 6.5 scrub starts 2024-03-30T14:05:25.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:25 smithi113 ceph-mon[34341]: 6.5 scrub ok 2024-03-30T14:05:25.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:25 smithi113 ceph-mon[34341]: pgmap v1076: 152 pgs: 152 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-30T14:05:25.850 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 1 1 2024-03-30T14:05:26.087 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:26 smithi043 ceph-mon[29963]: 3.15 scrub starts 2024-03-30T14:05:26.087 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:26 smithi043 ceph-mon[29963]: 3.15 scrub ok 2024-03-30T14:05:26.088 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:26 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3129689102' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T14:05:26.088 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:26 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3129689102' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-30T14:05:26.088 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:26 smithi043 ceph-mon[29963]: osdmap e497: 8 total, 8 up, 8 in 2024-03-30T14:05:26.088 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:26 smithi043 ceph-mon[42099]: 3.15 scrub starts 2024-03-30T14:05:26.088 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:26 smithi043 ceph-mon[42099]: 3.15 scrub ok 2024-03-30T14:05:26.088 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:26 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3129689102' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T14:05:26.088 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:26 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3129689102' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-30T14:05:26.088 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:26 smithi043 ceph-mon[42099]: osdmap e497: 8 total, 8 up, 8 in 2024-03-30T14:05:26.097 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:05:26.342 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:05:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:05:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:05:26.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:26 smithi113 ceph-mon[34341]: 3.15 scrub starts 2024-03-30T14:05:26.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:26 smithi113 ceph-mon[34341]: 3.15 scrub ok 2024-03-30T14:05:26.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:26 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3129689102' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T14:05:26.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:26 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3129689102' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-30T14:05:26.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:26 smithi113 ceph-mon[34341]: osdmap e497: 8 total, 8 up, 8 in 2024-03-30T14:05:27.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:27 smithi043 ceph-mon[29963]: pgmap v1078: 152 pgs: 152 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:05:27.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:27 smithi043 ceph-mon[29963]: 6.8 deep-scrub starts 2024-03-30T14:05:27.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:27 smithi043 ceph-mon[29963]: 6.8 deep-scrub ok 2024-03-30T14:05:27.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:27 smithi043 ceph-mon[29963]: 3.1d scrub starts 2024-03-30T14:05:27.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:27 smithi043 ceph-mon[29963]: 3.1d scrub ok 2024-03-30T14:05:27.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:27 smithi043 ceph-mon[29963]: 6.e scrub starts 2024-03-30T14:05:27.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:27 smithi043 ceph-mon[29963]: 6.e scrub ok 2024-03-30T14:05:27.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:27 smithi043 ceph-mon[42099]: pgmap v1078: 152 pgs: 152 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:05:27.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:27 smithi043 ceph-mon[42099]: 6.8 deep-scrub starts 2024-03-30T14:05:27.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:27 smithi043 ceph-mon[42099]: 6.8 deep-scrub ok 2024-03-30T14:05:27.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:27 smithi043 ceph-mon[42099]: 3.1d scrub starts 2024-03-30T14:05:27.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:27 smithi043 ceph-mon[42099]: 3.1d scrub ok 2024-03-30T14:05:27.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:27 smithi043 ceph-mon[42099]: 6.e scrub starts 2024-03-30T14:05:27.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:27 smithi043 ceph-mon[42099]: 6.e scrub ok 2024-03-30T14:05:27.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:27 smithi113 ceph-mon[34341]: pgmap v1078: 152 pgs: 152 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:05:27.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:27 smithi113 ceph-mon[34341]: 6.8 deep-scrub starts 2024-03-30T14:05:27.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:27 smithi113 ceph-mon[34341]: 6.8 deep-scrub ok 2024-03-30T14:05:27.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:27 smithi113 ceph-mon[34341]: 3.1d scrub starts 2024-03-30T14:05:27.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:27 smithi113 ceph-mon[34341]: 3.1d scrub ok 2024-03-30T14:05:27.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:27 smithi113 ceph-mon[34341]: 6.e scrub starts 2024-03-30T14:05:27.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:27 smithi113 ceph-mon[34341]: 6.e scrub ok 2024-03-30T14:05:28.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:28 smithi043 ceph-mon[29963]: 5.c deep-scrub starts 2024-03-30T14:05:28.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:28 smithi043 ceph-mon[29963]: 5.c deep-scrub ok 2024-03-30T14:05:28.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:28 smithi043 ceph-mon[29963]: 6.a scrub starts 2024-03-30T14:05:28.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:28 smithi043 ceph-mon[29963]: 6.a scrub ok 2024-03-30T14:05:28.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:28 smithi043 ceph-mon[42099]: 5.c deep-scrub starts 2024-03-30T14:05:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:28 smithi043 ceph-mon[42099]: 5.c deep-scrub ok 2024-03-30T14:05:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:28 smithi043 ceph-mon[42099]: 6.a scrub starts 2024-03-30T14:05:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:28 smithi043 ceph-mon[42099]: 6.a scrub ok 2024-03-30T14:05:28.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:28 smithi113 ceph-mon[34341]: 5.c deep-scrub starts 2024-03-30T14:05:28.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:28 smithi113 ceph-mon[34341]: 5.c deep-scrub ok 2024-03-30T14:05:28.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:28 smithi113 ceph-mon[34341]: 6.a scrub starts 2024-03-30T14:05:28.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:28 smithi113 ceph-mon[34341]: 6.a scrub ok 2024-03-30T14:05:29.083 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T14:05:29.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:29 smithi043 ceph-mon[29963]: 6.10 scrub starts 2024-03-30T14:05:29.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:29 smithi043 ceph-mon[29963]: 6.10 scrub ok 2024-03-30T14:05:29.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:29 smithi043 ceph-mon[29963]: pgmap v1079: 152 pgs: 152 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:05:29.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:29 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2857536531' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T14:05:29.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:29 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T14:05:29.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:29 smithi043 ceph-mon[42099]: 6.10 scrub starts 2024-03-30T14:05:29.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:29 smithi043 ceph-mon[42099]: 6.10 scrub ok 2024-03-30T14:05:29.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:29 smithi043 ceph-mon[42099]: pgmap v1079: 152 pgs: 152 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:05:29.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:29 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2857536531' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T14:05:29.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:29 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T14:05:29.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:29 smithi113 ceph-mon[34341]: 6.10 scrub starts 2024-03-30T14:05:29.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:29 smithi113 ceph-mon[34341]: 6.10 scrub ok 2024-03-30T14:05:29.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:29 smithi113 ceph-mon[34341]: pgmap v1079: 152 pgs: 152 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:05:29.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:29 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2857536531' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T14:05:29.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:29 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T14:05:29.797 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd reweight 3 1 2024-03-30T14:05:29.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:05:29 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:05:30.042 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:05:30.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-30T14:05:30.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[29963]: osdmap e498: 8 total, 8 up, 8 in 2024-03-30T14:05:30.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 2]}]: dispatch 2024-03-30T14:05:30.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [4, 2]}]: dispatch 2024-03-30T14:05:30.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-30T14:05:30.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-30T14:05:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 2]}]: dispatch 2024-03-30T14:05:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 0]}]: dispatch 2024-03-30T14:05:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 7]}]: dispatch 2024-03-30T14:05:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[29963]: 5.f scrub starts 2024-03-30T14:05:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[29963]: 5.f scrub ok 2024-03-30T14:05:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[29963]: 6.17 scrub starts 2024-03-30T14:05:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[29963]: 6.17 scrub ok 2024-03-30T14:05:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:30.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-30T14:05:30.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[42099]: osdmap e498: 8 total, 8 up, 8 in 2024-03-30T14:05:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 2]}]: dispatch 2024-03-30T14:05:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [4, 2]}]: dispatch 2024-03-30T14:05:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-30T14:05:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-30T14:05:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 2]}]: dispatch 2024-03-30T14:05:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 0]}]: dispatch 2024-03-30T14:05:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 7]}]: dispatch 2024-03-30T14:05:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[42099]: 5.f scrub starts 2024-03-30T14:05:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[42099]: 5.f scrub ok 2024-03-30T14:05:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[42099]: 6.17 scrub starts 2024-03-30T14:05:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[42099]: 6.17 scrub ok 2024-03-30T14:05:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:30.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:30 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-30T14:05:30.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:30 smithi113 ceph-mon[34341]: osdmap e498: 8 total, 8 up, 8 in 2024-03-30T14:05:30.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 2]}]: dispatch 2024-03-30T14:05:30.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [4, 2]}]: dispatch 2024-03-30T14:05:30.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-30T14:05:30.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-30T14:05:30.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 2]}]: dispatch 2024-03-30T14:05:30.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 0]}]: dispatch 2024-03-30T14:05:30.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 7]}]: dispatch 2024-03-30T14:05:30.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:30 smithi113 ceph-mon[34341]: 5.f scrub starts 2024-03-30T14:05:30.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:30 smithi113 ceph-mon[34341]: 5.f scrub ok 2024-03-30T14:05:30.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:30 smithi113 ceph-mon[34341]: 6.17 scrub starts 2024-03-30T14:05:30.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:30 smithi113 ceph-mon[34341]: 6.17 scrub ok 2024-03-30T14:05:30.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:31.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[29963]: 6.4 scrub starts 2024-03-30T14:05:31.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[29963]: 6.4 scrub ok 2024-03-30T14:05:31.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[29963]: pgmap v1081: 152 pgs: 152 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-03-30T14:05:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[29963]: 1.2 deep-scrub starts 2024-03-30T14:05:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[29963]: 1.2 deep-scrub ok 2024-03-30T14:05:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 2]}]': finished 2024-03-30T14:05:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [4, 2]}]': finished 2024-03-30T14:05:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-30T14:05:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-30T14:05:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 2]}]': finished 2024-03-30T14:05:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 0]}]': finished 2024-03-30T14:05:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 7]}]': finished 2024-03-30T14:05:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[29963]: osdmap e499: 8 total, 8 up, 8 in 2024-03-30T14:05:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[29963]: 6.7 scrub starts 2024-03-30T14:05:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[29963]: 4.0 scrub starts 2024-03-30T14:05:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[29963]: 4.0 scrub ok 2024-03-30T14:05:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[29963]: 6.7 scrub ok 2024-03-30T14:05:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[42099]: 6.4 scrub starts 2024-03-30T14:05:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[42099]: 6.4 scrub ok 2024-03-30T14:05:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[42099]: pgmap v1081: 152 pgs: 152 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-03-30T14:05:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[42099]: 1.2 deep-scrub starts 2024-03-30T14:05:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[42099]: 1.2 deep-scrub ok 2024-03-30T14:05:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 2]}]': finished 2024-03-30T14:05:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [4, 2]}]': finished 2024-03-30T14:05:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-30T14:05:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-30T14:05:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 2]}]': finished 2024-03-30T14:05:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 0]}]': finished 2024-03-30T14:05:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 7]}]': finished 2024-03-30T14:05:31.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[42099]: osdmap e499: 8 total, 8 up, 8 in 2024-03-30T14:05:31.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[42099]: 6.7 scrub starts 2024-03-30T14:05:31.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[42099]: 4.0 scrub starts 2024-03-30T14:05:31.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[42099]: 4.0 scrub ok 2024-03-30T14:05:31.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:31 smithi043 ceph-mon[42099]: 6.7 scrub ok 2024-03-30T14:05:31.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:31 smithi113 ceph-mon[34341]: 6.4 scrub starts 2024-03-30T14:05:31.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:31 smithi113 ceph-mon[34341]: 6.4 scrub ok 2024-03-30T14:05:31.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:31 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:31.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:31 smithi113 ceph-mon[34341]: pgmap v1081: 152 pgs: 152 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-03-30T14:05:31.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:31 smithi113 ceph-mon[34341]: 1.2 deep-scrub starts 2024-03-30T14:05:31.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:31 smithi113 ceph-mon[34341]: 1.2 deep-scrub ok 2024-03-30T14:05:31.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:31 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 2]}]': finished 2024-03-30T14:05:31.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:31 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [4, 2]}]': finished 2024-03-30T14:05:31.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:31 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-30T14:05:31.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:31 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-30T14:05:31.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:31 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 2]}]': finished 2024-03-30T14:05:31.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:31 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 0]}]': finished 2024-03-30T14:05:31.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:31 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 7]}]': finished 2024-03-30T14:05:31.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:31 smithi113 ceph-mon[34341]: osdmap e499: 8 total, 8 up, 8 in 2024-03-30T14:05:31.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:31 smithi113 ceph-mon[34341]: 6.7 scrub starts 2024-03-30T14:05:31.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:31 smithi113 ceph-mon[34341]: 4.0 scrub starts 2024-03-30T14:05:31.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:31 smithi113 ceph-mon[34341]: 4.0 scrub ok 2024-03-30T14:05:31.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:31 smithi113 ceph-mon[34341]: 6.7 scrub ok 2024-03-30T14:05:32.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:32 smithi043 ceph-mon[29963]: 3.6 deep-scrub starts 2024-03-30T14:05:32.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:32 smithi043 ceph-mon[29963]: 3.6 deep-scrub ok 2024-03-30T14:05:32.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:32 smithi043 ceph-mon[29963]: osdmap e500: 8 total, 8 up, 8 in 2024-03-30T14:05:32.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:32 smithi043 ceph-mon[42099]: 3.6 deep-scrub starts 2024-03-30T14:05:32.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:32 smithi043 ceph-mon[42099]: 3.6 deep-scrub ok 2024-03-30T14:05:32.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:32 smithi043 ceph-mon[42099]: osdmap e500: 8 total, 8 up, 8 in 2024-03-30T14:05:32.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:32 smithi113 ceph-mon[34341]: 3.6 deep-scrub starts 2024-03-30T14:05:32.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:32 smithi113 ceph-mon[34341]: 3.6 deep-scrub ok 2024-03-30T14:05:32.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:32 smithi113 ceph-mon[34341]: osdmap e500: 8 total, 8 up, 8 in 2024-03-30T14:05:32.988 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T14:05:33.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[29963]: 1.b scrub starts 2024-03-30T14:05:33.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[29963]: 1.b scrub ok 2024-03-30T14:05:33.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[29963]: 5.5 scrub starts 2024-03-30T14:05:33.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[29963]: 5.5 scrub ok 2024-03-30T14:05:33.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[29963]: pgmap v1084: 152 pgs: 1 peering, 151 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:05:33.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[29963]: 3.d scrub starts 2024-03-30T14:05:33.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[29963]: 3.d scrub ok 2024-03-30T14:05:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2808059695' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T14:05:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[29963]: 4.1a scrub starts 2024-03-30T14:05:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[29963]: 4.1a scrub ok 2024-03-30T14:05:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2808059695' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-30T14:05:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[29963]: osdmap e501: 8 total, 8 up, 8 in 2024-03-30T14:05:33.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[42099]: 1.b scrub starts 2024-03-30T14:05:33.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[42099]: 1.b scrub ok 2024-03-30T14:05:33.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[42099]: 5.5 scrub starts 2024-03-30T14:05:33.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[42099]: 5.5 scrub ok 2024-03-30T14:05:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[42099]: pgmap v1084: 152 pgs: 1 peering, 151 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:05:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[42099]: 3.d scrub starts 2024-03-30T14:05:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[42099]: 3.d scrub ok 2024-03-30T14:05:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2808059695' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T14:05:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[42099]: 4.1a scrub starts 2024-03-30T14:05:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[42099]: 4.1a scrub ok 2024-03-30T14:05:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2808059695' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-30T14:05:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:33 smithi043 ceph-mon[42099]: osdmap e501: 8 total, 8 up, 8 in 2024-03-30T14:05:33.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:33 smithi113 ceph-mon[34341]: 1.b scrub starts 2024-03-30T14:05:33.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:33 smithi113 ceph-mon[34341]: 1.b scrub ok 2024-03-30T14:05:33.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:33 smithi113 ceph-mon[34341]: 5.5 scrub starts 2024-03-30T14:05:33.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:33 smithi113 ceph-mon[34341]: 5.5 scrub ok 2024-03-30T14:05:33.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:33 smithi113 ceph-mon[34341]: pgmap v1084: 152 pgs: 1 peering, 151 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:05:33.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:33 smithi113 ceph-mon[34341]: 3.d scrub starts 2024-03-30T14:05:33.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:33 smithi113 ceph-mon[34341]: 3.d scrub ok 2024-03-30T14:05:33.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:33 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2808059695' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T14:05:33.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:33 smithi113 ceph-mon[34341]: 4.1a scrub starts 2024-03-30T14:05:33.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:33 smithi113 ceph-mon[34341]: 4.1a scrub ok 2024-03-30T14:05:33.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:33 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2808059695' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-30T14:05:33.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:33 smithi113 ceph-mon[34341]: osdmap e501: 8 total, 8 up, 8 in 2024-03-30T14:05:33.944 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 3 1 2024-03-30T14:05:34.216 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:05:34.337 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:34 smithi043 ceph-mon[29963]: 1.11 scrub starts 2024-03-30T14:05:34.337 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:34 smithi043 ceph-mon[29963]: 1.11 scrub ok 2024-03-30T14:05:34.338 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:34 smithi043 ceph-mon[29963]: 5.2 scrub starts 2024-03-30T14:05:34.338 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:34 smithi043 ceph-mon[29963]: 5.2 scrub ok 2024-03-30T14:05:34.338 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:34 smithi043 ceph-mon[42099]: 1.11 scrub starts 2024-03-30T14:05:34.338 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:34 smithi043 ceph-mon[42099]: 1.11 scrub ok 2024-03-30T14:05:34.338 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:34 smithi043 ceph-mon[42099]: 5.2 scrub starts 2024-03-30T14:05:34.338 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:34 smithi043 ceph-mon[42099]: 5.2 scrub ok 2024-03-30T14:05:34.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:34 smithi113 ceph-mon[34341]: 1.11 scrub starts 2024-03-30T14:05:34.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:34 smithi113 ceph-mon[34341]: 1.11 scrub ok 2024-03-30T14:05:34.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:34 smithi113 ceph-mon[34341]: 5.2 scrub starts 2024-03-30T14:05:34.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:34 smithi113 ceph-mon[34341]: 5.2 scrub ok 2024-03-30T14:05:35.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:35 smithi043 ceph-mon[29963]: 1.3 deep-scrub starts 2024-03-30T14:05:35.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:35 smithi043 ceph-mon[29963]: 1.3 deep-scrub ok 2024-03-30T14:05:35.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:35 smithi043 ceph-mon[29963]: pgmap v1086: 152 pgs: 1 peering, 151 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:05:35.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:35 smithi043 ceph-mon[29963]: 4.8 scrub starts 2024-03-30T14:05:35.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:35 smithi043 ceph-mon[29963]: 4.8 scrub ok 2024-03-30T14:05:35.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:35 smithi043 ceph-mon[29963]: 3.3 scrub starts 2024-03-30T14:05:35.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:35 smithi043 ceph-mon[29963]: 3.3 scrub ok 2024-03-30T14:05:35.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:35 smithi043 ceph-mon[29963]: 3.1e scrub starts 2024-03-30T14:05:35.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:35 smithi043 ceph-mon[29963]: 3.1e scrub ok 2024-03-30T14:05:35.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:35 smithi043 ceph-mon[42099]: 1.3 deep-scrub starts 2024-03-30T14:05:35.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:35 smithi043 ceph-mon[42099]: 1.3 deep-scrub ok 2024-03-30T14:05:35.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:35 smithi043 ceph-mon[42099]: pgmap v1086: 152 pgs: 1 peering, 151 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:05:35.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:35 smithi043 ceph-mon[42099]: 4.8 scrub starts 2024-03-30T14:05:35.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:35 smithi043 ceph-mon[42099]: 4.8 scrub ok 2024-03-30T14:05:35.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:35 smithi043 ceph-mon[42099]: 3.3 scrub starts 2024-03-30T14:05:35.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:35 smithi043 ceph-mon[42099]: 3.3 scrub ok 2024-03-30T14:05:35.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:35 smithi043 ceph-mon[42099]: 3.1e scrub starts 2024-03-30T14:05:35.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:35 smithi043 ceph-mon[42099]: 3.1e scrub ok 2024-03-30T14:05:35.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:35 smithi113 ceph-mon[34341]: 1.3 deep-scrub starts 2024-03-30T14:05:35.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:35 smithi113 ceph-mon[34341]: 1.3 deep-scrub ok 2024-03-30T14:05:35.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:35 smithi113 ceph-mon[34341]: pgmap v1086: 152 pgs: 1 peering, 151 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:05:35.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:35 smithi113 ceph-mon[34341]: 4.8 scrub starts 2024-03-30T14:05:35.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:35 smithi113 ceph-mon[34341]: 4.8 scrub ok 2024-03-30T14:05:35.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:35 smithi113 ceph-mon[34341]: 3.3 scrub starts 2024-03-30T14:05:35.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:35 smithi113 ceph-mon[34341]: 3.3 scrub ok 2024-03-30T14:05:35.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:35 smithi113 ceph-mon[34341]: 3.1e scrub starts 2024-03-30T14:05:35.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:35 smithi113 ceph-mon[34341]: 3.1e scrub ok 2024-03-30T14:05:36.391 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:05:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:05:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:05:36.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:36 smithi043 ceph-mon[29963]: 2.0 scrub starts 2024-03-30T14:05:36.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:36 smithi043 ceph-mon[29963]: 2.0 scrub ok 2024-03-30T14:05:36.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:36 smithi043 ceph-mon[29963]: 5.e scrub starts 2024-03-30T14:05:36.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:36 smithi043 ceph-mon[29963]: 5.e scrub ok 2024-03-30T14:05:36.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:36 smithi043 ceph-mon[42099]: 2.0 scrub starts 2024-03-30T14:05:36.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:36 smithi043 ceph-mon[42099]: 2.0 scrub ok 2024-03-30T14:05:36.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:36 smithi043 ceph-mon[42099]: 5.e scrub starts 2024-03-30T14:05:36.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:36 smithi043 ceph-mon[42099]: 5.e scrub ok 2024-03-30T14:05:36.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:36 smithi113 ceph-mon[34341]: 2.0 scrub starts 2024-03-30T14:05:36.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:36 smithi113 ceph-mon[34341]: 2.0 scrub ok 2024-03-30T14:05:36.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:36 smithi113 ceph-mon[34341]: 5.e scrub starts 2024-03-30T14:05:36.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:36 smithi113 ceph-mon[34341]: 5.e scrub ok 2024-03-30T14:05:37.118 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T14:05:37.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:37 smithi043 ceph-mon[29963]: 5.1b scrub starts 2024-03-30T14:05:37.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:37 smithi043 ceph-mon[29963]: 5.1b scrub ok 2024-03-30T14:05:37.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:37 smithi043 ceph-mon[29963]: pgmap v1087: 152 pgs: 1 peering, 151 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 80 B/s, 3 objects/s recovering 2024-03-30T14:05:37.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:37 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3950904248' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T14:05:37.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:37 smithi043 ceph-mon[29963]: 4.9 scrub starts 2024-03-30T14:05:37.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:37 smithi043 ceph-mon[29963]: 4.9 scrub ok 2024-03-30T14:05:37.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:37 smithi043 ceph-mon[42099]: 5.1b scrub starts 2024-03-30T14:05:37.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:37 smithi043 ceph-mon[42099]: 5.1b scrub ok 2024-03-30T14:05:37.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:37 smithi043 ceph-mon[42099]: pgmap v1087: 152 pgs: 1 peering, 151 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 80 B/s, 3 objects/s recovering 2024-03-30T14:05:37.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:37 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3950904248' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T14:05:37.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:37 smithi043 ceph-mon[42099]: 4.9 scrub starts 2024-03-30T14:05:37.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:37 smithi043 ceph-mon[42099]: 4.9 scrub ok 2024-03-30T14:05:37.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:37 smithi113 ceph-mon[34341]: 5.1b scrub starts 2024-03-30T14:05:37.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:37 smithi113 ceph-mon[34341]: 5.1b scrub ok 2024-03-30T14:05:37.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:37 smithi113 ceph-mon[34341]: pgmap v1087: 152 pgs: 1 peering, 151 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 80 B/s, 3 objects/s recovering 2024-03-30T14:05:37.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:37 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3950904248' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T14:05:37.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:37 smithi113 ceph-mon[34341]: 4.9 scrub starts 2024-03-30T14:05:37.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:37 smithi113 ceph-mon[34341]: 4.9 scrub ok 2024-03-30T14:05:37.912 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd reweight 4 1 2024-03-30T14:05:38.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:38 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3950904248' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-30T14:05:38.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:38 smithi043 ceph-mon[42099]: osdmap e502: 8 total, 8 up, 8 in 2024-03-30T14:05:38.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:38 smithi043 ceph-mon[42099]: osdmap e503: 8 total, 8 up, 8 in 2024-03-30T14:05:38.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:38 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3950904248' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-30T14:05:38.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:38 smithi043 ceph-mon[29963]: osdmap e502: 8 total, 8 up, 8 in 2024-03-30T14:05:38.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:38 smithi043 ceph-mon[29963]: osdmap e503: 8 total, 8 up, 8 in 2024-03-30T14:05:38.177 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:05:38.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:38 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3950904248' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-30T14:05:38.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:38 smithi113 ceph-mon[34341]: osdmap e502: 8 total, 8 up, 8 in 2024-03-30T14:05:38.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:38 smithi113 ceph-mon[34341]: osdmap e503: 8 total, 8 up, 8 in 2024-03-30T14:05:39.360 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:39 smithi043 ceph-mon[29963]: 4.6 scrub starts 2024-03-30T14:05:39.360 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:39 smithi043 ceph-mon[29963]: 4.6 scrub ok 2024-03-30T14:05:39.360 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:39 smithi043 ceph-mon[29963]: pgmap v1089: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 758 B/s rd, 0 op/s; 71 B/s, 3 objects/s recovering 2024-03-30T14:05:39.360 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:39 smithi043 ceph-mon[29963]: 4.10 deep-scrub starts 2024-03-30T14:05:39.360 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:39 smithi043 ceph-mon[29963]: 4.10 deep-scrub ok 2024-03-30T14:05:39.360 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:39 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:05:39.361 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:39 smithi043 ceph-mon[42099]: 4.6 scrub starts 2024-03-30T14:05:39.361 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:39 smithi043 ceph-mon[42099]: 4.6 scrub ok 2024-03-30T14:05:39.361 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:39 smithi043 ceph-mon[42099]: pgmap v1089: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 758 B/s rd, 0 op/s; 71 B/s, 3 objects/s recovering 2024-03-30T14:05:39.361 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:39 smithi043 ceph-mon[42099]: 4.10 deep-scrub starts 2024-03-30T14:05:39.361 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:39 smithi043 ceph-mon[42099]: 4.10 deep-scrub ok 2024-03-30T14:05:39.361 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:39 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:05:39.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:39 smithi113 ceph-mon[34341]: 4.6 scrub starts 2024-03-30T14:05:39.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:39 smithi113 ceph-mon[34341]: 4.6 scrub ok 2024-03-30T14:05:39.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:39 smithi113 ceph-mon[34341]: pgmap v1089: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 758 B/s rd, 0 op/s; 71 B/s, 3 objects/s recovering 2024-03-30T14:05:39.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:39 smithi113 ceph-mon[34341]: 4.10 deep-scrub starts 2024-03-30T14:05:39.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:39 smithi113 ceph-mon[34341]: 4.10 deep-scrub ok 2024-03-30T14:05:39.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:39 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:05:39.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:05:39 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:05:40.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:40 smithi043 ceph-mon[29963]: 3.10 deep-scrub starts 2024-03-30T14:05:40.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:40 smithi043 ceph-mon[29963]: 3.10 deep-scrub ok 2024-03-30T14:05:40.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:40 smithi043 ceph-mon[29963]: 6.11 scrub starts 2024-03-30T14:05:40.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:40 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:05:40.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:40 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:05:40.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:40 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:05:40.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:40 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1034213028' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T14:05:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:40 smithi043 ceph-mon[42099]: 3.10 deep-scrub starts 2024-03-30T14:05:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:40 smithi043 ceph-mon[42099]: 3.10 deep-scrub ok 2024-03-30T14:05:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:40 smithi043 ceph-mon[42099]: 6.11 scrub starts 2024-03-30T14:05:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:40 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:05:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:40 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:05:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:40 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:05:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:40 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1034213028' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T14:05:40.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:40 smithi113 ceph-mon[34341]: 3.10 deep-scrub starts 2024-03-30T14:05:40.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:40 smithi113 ceph-mon[34341]: 3.10 deep-scrub ok 2024-03-30T14:05:40.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:40 smithi113 ceph-mon[34341]: 6.11 scrub starts 2024-03-30T14:05:40.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:40 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:05:40.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:40 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:05:40.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:40 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:05:40.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:40 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1034213028' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T14:05:40.552 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T14:05:41.343 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:41 smithi043 ceph-mon[29963]: 6.11 scrub ok 2024-03-30T14:05:41.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:41 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:41.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:41 smithi043 ceph-mon[29963]: pgmap v1091: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 70 B/s, 4 objects/s recovering 2024-03-30T14:05:41.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:41 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1034213028' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-30T14:05:41.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:41 smithi043 ceph-mon[29963]: osdmap e504: 8 total, 8 up, 8 in 2024-03-30T14:05:41.344 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:41 smithi043 ceph-mon[42099]: 6.11 scrub ok 2024-03-30T14:05:41.344 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:41 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:41.344 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:41 smithi043 ceph-mon[42099]: pgmap v1091: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 70 B/s, 4 objects/s recovering 2024-03-30T14:05:41.344 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:41 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1034213028' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-30T14:05:41.344 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:41 smithi043 ceph-mon[42099]: osdmap e504: 8 total, 8 up, 8 in 2024-03-30T14:05:41.377 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 4 1 2024-03-30T14:05:41.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:41 smithi113 ceph-mon[34341]: 6.11 scrub ok 2024-03-30T14:05:41.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:41 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:41.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:41 smithi113 ceph-mon[34341]: pgmap v1091: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 70 B/s, 4 objects/s recovering 2024-03-30T14:05:41.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:41 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1034213028' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-30T14:05:41.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:41 smithi113 ceph-mon[34341]: osdmap e504: 8 total, 8 up, 8 in 2024-03-30T14:05:41.630 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:05:42.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:42 smithi043 ceph-mon[29963]: 1.6 scrub starts 2024-03-30T14:05:42.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:42 smithi043 ceph-mon[29963]: 1.6 scrub ok 2024-03-30T14:05:42.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:42 smithi043 ceph-mon[29963]: 5.16 scrub starts 2024-03-30T14:05:42.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:42 smithi043 ceph-mon[29963]: 5.16 scrub ok 2024-03-30T14:05:42.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:42 smithi043 ceph-mon[42099]: 1.6 scrub starts 2024-03-30T14:05:42.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:42 smithi043 ceph-mon[42099]: 1.6 scrub ok 2024-03-30T14:05:42.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:42 smithi043 ceph-mon[42099]: 5.16 scrub starts 2024-03-30T14:05:42.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:42 smithi043 ceph-mon[42099]: 5.16 scrub ok 2024-03-30T14:05:42.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:42 smithi113 ceph-mon[34341]: 1.6 scrub starts 2024-03-30T14:05:42.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:42 smithi113 ceph-mon[34341]: 1.6 scrub ok 2024-03-30T14:05:42.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:42 smithi113 ceph-mon[34341]: 5.16 scrub starts 2024-03-30T14:05:42.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:42 smithi113 ceph-mon[34341]: 5.16 scrub ok 2024-03-30T14:05:43.236 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:43 smithi043 ceph-mon[29963]: pgmap v1093: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T14:05:43.236 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:43 smithi043 ceph-mon[29963]: 1.f scrub starts 2024-03-30T14:05:43.237 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:43 smithi043 ceph-mon[29963]: 1.f scrub ok 2024-03-30T14:05:43.237 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:43 smithi043 ceph-mon[29963]: osdmap e505: 8 total, 8 up, 8 in 2024-03-30T14:05:43.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:43 smithi043 ceph-mon[42099]: pgmap v1093: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T14:05:43.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:43 smithi043 ceph-mon[42099]: 1.f scrub starts 2024-03-30T14:05:43.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:43 smithi043 ceph-mon[42099]: 1.f scrub ok 2024-03-30T14:05:43.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:43 smithi043 ceph-mon[42099]: osdmap e505: 8 total, 8 up, 8 in 2024-03-30T14:05:43.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:43 smithi113 ceph-mon[34341]: pgmap v1093: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T14:05:43.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:43 smithi113 ceph-mon[34341]: 1.f scrub starts 2024-03-30T14:05:43.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:43 smithi113 ceph-mon[34341]: 1.f scrub ok 2024-03-30T14:05:43.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:43 smithi113 ceph-mon[34341]: osdmap e505: 8 total, 8 up, 8 in 2024-03-30T14:05:44.142 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T14:05:44.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:44 smithi043 ceph-mon[29963]: 6.18 scrub starts 2024-03-30T14:05:44.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:44 smithi043 ceph-mon[29963]: 6.18 scrub ok 2024-03-30T14:05:44.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:44 smithi043 ceph-mon[29963]: 6.3 deep-scrub starts 2024-03-30T14:05:44.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:44 smithi043 ceph-mon[29963]: 6.3 deep-scrub ok 2024-03-30T14:05:44.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:44 smithi043 ceph-mon[29963]: 4.e scrub starts 2024-03-30T14:05:44.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:44 smithi043 ceph-mon[29963]: 4.e scrub ok 2024-03-30T14:05:44.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:44 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2411986193' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T14:05:44.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:44 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T14:05:44.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:44 smithi043 ceph-mon[42099]: 6.18 scrub starts 2024-03-30T14:05:44.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:44 smithi043 ceph-mon[42099]: 6.18 scrub ok 2024-03-30T14:05:44.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:44 smithi043 ceph-mon[42099]: 6.3 deep-scrub starts 2024-03-30T14:05:44.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:44 smithi043 ceph-mon[42099]: 6.3 deep-scrub ok 2024-03-30T14:05:44.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:44 smithi043 ceph-mon[42099]: 4.e scrub starts 2024-03-30T14:05:44.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:44 smithi043 ceph-mon[42099]: 4.e scrub ok 2024-03-30T14:05:44.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:44 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2411986193' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T14:05:44.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:44 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T14:05:44.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:44 smithi113 ceph-mon[34341]: 6.18 scrub starts 2024-03-30T14:05:44.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:44 smithi113 ceph-mon[34341]: 6.18 scrub ok 2024-03-30T14:05:44.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:44 smithi113 ceph-mon[34341]: 6.3 deep-scrub starts 2024-03-30T14:05:44.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:44 smithi113 ceph-mon[34341]: 6.3 deep-scrub ok 2024-03-30T14:05:44.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:44 smithi113 ceph-mon[34341]: 4.e scrub starts 2024-03-30T14:05:44.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:44 smithi113 ceph-mon[34341]: 4.e scrub ok 2024-03-30T14:05:44.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:44 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2411986193' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T14:05:44.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:44 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T14:05:44.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd reweight 5 1 2024-03-30T14:05:45.098 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:05:45.454 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:45 smithi043 ceph-mon[42099]: 5.14 scrub starts 2024-03-30T14:05:45.454 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:45 smithi043 ceph-mon[42099]: 5.14 scrub ok 2024-03-30T14:05:45.454 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:45 smithi043 ceph-mon[42099]: 5.8 scrub starts 2024-03-30T14:05:45.454 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:45 smithi043 ceph-mon[42099]: 5.8 scrub ok 2024-03-30T14:05:45.454 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:45 smithi043 ceph-mon[42099]: pgmap v1095: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:05:45.455 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:45 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-30T14:05:45.455 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:45 smithi043 ceph-mon[42099]: osdmap e506: 8 total, 8 up, 8 in 2024-03-30T14:05:45.455 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:45 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:45.455 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:45 smithi043 ceph-mon[42099]: 3.11 scrub starts 2024-03-30T14:05:45.455 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:45 smithi043 ceph-mon[42099]: 3.11 scrub ok 2024-03-30T14:05:45.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:45 smithi043 ceph-mon[29963]: 5.14 scrub starts 2024-03-30T14:05:45.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:45 smithi043 ceph-mon[29963]: 5.14 scrub ok 2024-03-30T14:05:45.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:45 smithi043 ceph-mon[29963]: 5.8 scrub starts 2024-03-30T14:05:45.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:45 smithi043 ceph-mon[29963]: 5.8 scrub ok 2024-03-30T14:05:45.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:45 smithi043 ceph-mon[29963]: pgmap v1095: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:05:45.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:45 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-30T14:05:45.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:45 smithi043 ceph-mon[29963]: osdmap e506: 8 total, 8 up, 8 in 2024-03-30T14:05:45.456 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:45 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:45.456 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:45 smithi043 ceph-mon[29963]: 3.11 scrub starts 2024-03-30T14:05:45.456 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:45 smithi043 ceph-mon[29963]: 3.11 scrub ok 2024-03-30T14:05:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:45 smithi113 ceph-mon[34341]: 5.14 scrub starts 2024-03-30T14:05:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:45 smithi113 ceph-mon[34341]: 5.14 scrub ok 2024-03-30T14:05:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:45 smithi113 ceph-mon[34341]: 5.8 scrub starts 2024-03-30T14:05:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:45 smithi113 ceph-mon[34341]: 5.8 scrub ok 2024-03-30T14:05:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:45 smithi113 ceph-mon[34341]: pgmap v1095: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:05:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:45 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-30T14:05:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:45 smithi113 ceph-mon[34341]: osdmap e506: 8 total, 8 up, 8 in 2024-03-30T14:05:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:45 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:45 smithi113 ceph-mon[34341]: 3.11 scrub starts 2024-03-30T14:05:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:45 smithi113 ceph-mon[34341]: 3.11 scrub ok 2024-03-30T14:05:46.325 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:05:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:05:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:05:46.325 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:46 smithi043 ceph-mon[29963]: 4.19 scrub starts 2024-03-30T14:05:46.325 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:46 smithi043 ceph-mon[29963]: 4.19 scrub ok 2024-03-30T14:05:46.325 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:46 smithi043 ceph-mon[29963]: 5.1d scrub starts 2024-03-30T14:05:46.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:46 smithi043 ceph-mon[29963]: 5.1d scrub ok 2024-03-30T14:05:46.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:46 smithi043 ceph-mon[29963]: 3.e scrub starts 2024-03-30T14:05:46.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:46 smithi043 ceph-mon[29963]: 3.e scrub ok 2024-03-30T14:05:46.326 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:46 smithi043 ceph-mon[42099]: 4.19 scrub starts 2024-03-30T14:05:46.326 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:46 smithi043 ceph-mon[42099]: 4.19 scrub ok 2024-03-30T14:05:46.326 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:46 smithi043 ceph-mon[42099]: 5.1d scrub starts 2024-03-30T14:05:46.326 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:46 smithi043 ceph-mon[42099]: 5.1d scrub ok 2024-03-30T14:05:46.326 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:46 smithi043 ceph-mon[42099]: 3.e scrub starts 2024-03-30T14:05:46.326 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:46 smithi043 ceph-mon[42099]: 3.e scrub ok 2024-03-30T14:05:46.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:46 smithi113 ceph-mon[34341]: 4.19 scrub starts 2024-03-30T14:05:46.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:46 smithi113 ceph-mon[34341]: 4.19 scrub ok 2024-03-30T14:05:46.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:46 smithi113 ceph-mon[34341]: 5.1d scrub starts 2024-03-30T14:05:46.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:46 smithi113 ceph-mon[34341]: 5.1d scrub ok 2024-03-30T14:05:46.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:46 smithi113 ceph-mon[34341]: 3.e scrub starts 2024-03-30T14:05:46.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:46 smithi113 ceph-mon[34341]: 3.e scrub ok 2024-03-30T14:05:47.158 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T14:05:47.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:47 smithi113 ceph-mon[34341]: 2.1 scrub starts 2024-03-30T14:05:47.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:47 smithi113 ceph-mon[34341]: 2.1 scrub ok 2024-03-30T14:05:47.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:47 smithi113 ceph-mon[34341]: 6.0 scrub starts 2024-03-30T14:05:47.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:47 smithi113 ceph-mon[34341]: 6.0 scrub ok 2024-03-30T14:05:47.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:47 smithi113 ceph-mon[34341]: pgmap v1097: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:05:47.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:47 smithi113 ceph-mon[34341]: 5.6 scrub starts 2024-03-30T14:05:47.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:47 smithi113 ceph-mon[34341]: 5.6 scrub ok 2024-03-30T14:05:47.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:47 smithi113 ceph-mon[34341]: 1.13 scrub starts 2024-03-30T14:05:47.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:47 smithi113 ceph-mon[34341]: 1.13 scrub ok 2024-03-30T14:05:47.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:47 smithi113 ceph-mon[34341]: 6.1e scrub starts 2024-03-30T14:05:47.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:47 smithi113 ceph-mon[34341]: 6.1e scrub ok 2024-03-30T14:05:47.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:47 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2209749995' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T14:05:47.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:47 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T14:05:47.508 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[29963]: 2.1 scrub starts 2024-03-30T14:05:47.508 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[29963]: 2.1 scrub ok 2024-03-30T14:05:47.508 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[29963]: 6.0 scrub starts 2024-03-30T14:05:47.508 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[29963]: 6.0 scrub ok 2024-03-30T14:05:47.508 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[29963]: pgmap v1097: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:05:47.509 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[29963]: 5.6 scrub starts 2024-03-30T14:05:47.509 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[29963]: 5.6 scrub ok 2024-03-30T14:05:47.509 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[29963]: 1.13 scrub starts 2024-03-30T14:05:47.509 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[29963]: 1.13 scrub ok 2024-03-30T14:05:47.509 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[29963]: 6.1e scrub starts 2024-03-30T14:05:47.509 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[29963]: 6.1e scrub ok 2024-03-30T14:05:47.509 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2209749995' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T14:05:47.509 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T14:05:47.509 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[42099]: 2.1 scrub starts 2024-03-30T14:05:47.509 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[42099]: 2.1 scrub ok 2024-03-30T14:05:47.509 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[42099]: 6.0 scrub starts 2024-03-30T14:05:47.509 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[42099]: 6.0 scrub ok 2024-03-30T14:05:47.510 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[42099]: pgmap v1097: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:05:47.510 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[42099]: 5.6 scrub starts 2024-03-30T14:05:47.510 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[42099]: 5.6 scrub ok 2024-03-30T14:05:47.510 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[42099]: 1.13 scrub starts 2024-03-30T14:05:47.510 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[42099]: 1.13 scrub ok 2024-03-30T14:05:47.510 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[42099]: 6.1e scrub starts 2024-03-30T14:05:47.510 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[42099]: 6.1e scrub ok 2024-03-30T14:05:47.510 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2209749995' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T14:05:47.510 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:47 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T14:05:47.941 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 5 1 2024-03-30T14:05:48.257 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:05:48.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:48 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-30T14:05:48.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:48 smithi113 ceph-mon[34341]: osdmap e507: 8 total, 8 up, 8 in 2024-03-30T14:05:48.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:48 smithi113 ceph-mon[34341]: 4.1c deep-scrub starts 2024-03-30T14:05:48.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:48 smithi113 ceph-mon[34341]: 4.1c deep-scrub ok 2024-03-30T14:05:48.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:48 smithi113 ceph-mon[34341]: 4.1b scrub starts 2024-03-30T14:05:48.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:48 smithi113 ceph-mon[34341]: 4.1b scrub ok 2024-03-30T14:05:48.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:48 smithi113 ceph-mon[34341]: osdmap e508: 8 total, 8 up, 8 in 2024-03-30T14:05:48.493 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:48 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-30T14:05:48.494 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:48 smithi043 ceph-mon[29963]: osdmap e507: 8 total, 8 up, 8 in 2024-03-30T14:05:48.494 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:48 smithi043 ceph-mon[29963]: 4.1c deep-scrub starts 2024-03-30T14:05:48.494 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:48 smithi043 ceph-mon[29963]: 4.1c deep-scrub ok 2024-03-30T14:05:48.494 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:48 smithi043 ceph-mon[29963]: 4.1b scrub starts 2024-03-30T14:05:48.494 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:48 smithi043 ceph-mon[29963]: 4.1b scrub ok 2024-03-30T14:05:48.494 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:48 smithi043 ceph-mon[29963]: osdmap e508: 8 total, 8 up, 8 in 2024-03-30T14:05:48.494 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:48 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-30T14:05:48.494 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:48 smithi043 ceph-mon[42099]: osdmap e507: 8 total, 8 up, 8 in 2024-03-30T14:05:48.494 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:48 smithi043 ceph-mon[42099]: 4.1c deep-scrub starts 2024-03-30T14:05:48.494 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:48 smithi043 ceph-mon[42099]: 4.1c deep-scrub ok 2024-03-30T14:05:48.495 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:48 smithi043 ceph-mon[42099]: 4.1b scrub starts 2024-03-30T14:05:48.495 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:48 smithi043 ceph-mon[42099]: 4.1b scrub ok 2024-03-30T14:05:48.495 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:48 smithi043 ceph-mon[42099]: osdmap e508: 8 total, 8 up, 8 in 2024-03-30T14:05:49.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:49 smithi043 ceph-mon[42099]: 5.0 scrub starts 2024-03-30T14:05:49.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:49 smithi043 ceph-mon[42099]: 5.0 scrub ok 2024-03-30T14:05:49.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:49 smithi043 ceph-mon[42099]: pgmap v1099: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:05:49.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:49 smithi043 ceph-mon[42099]: osdmap e509: 8 total, 8 up, 8 in 2024-03-30T14:05:49.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:49 smithi043 ceph-mon[29963]: 5.0 scrub starts 2024-03-30T14:05:49.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:49 smithi043 ceph-mon[29963]: 5.0 scrub ok 2024-03-30T14:05:49.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:49 smithi043 ceph-mon[29963]: pgmap v1099: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:05:49.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:49 smithi043 ceph-mon[29963]: osdmap e509: 8 total, 8 up, 8 in 2024-03-30T14:05:49.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:49 smithi113 ceph-mon[34341]: 5.0 scrub starts 2024-03-30T14:05:49.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:49 smithi113 ceph-mon[34341]: 5.0 scrub ok 2024-03-30T14:05:49.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:49 smithi113 ceph-mon[34341]: pgmap v1099: 152 pgs: 152 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:05:49.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:49 smithi113 ceph-mon[34341]: osdmap e509: 8 total, 8 up, 8 in 2024-03-30T14:05:49.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:05:49 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:05:50.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:50 smithi113 ceph-mon[34341]: 5.15 deep-scrub starts 2024-03-30T14:05:50.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:50 smithi113 ceph-mon[34341]: 5.15 deep-scrub ok 2024-03-30T14:05:50.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:50 smithi113 ceph-mon[34341]: osdmap e510: 8 total, 8 up, 8 in 2024-03-30T14:05:50.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:50 smithi043 ceph-mon[29963]: 5.15 deep-scrub starts 2024-03-30T14:05:50.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:50 smithi043 ceph-mon[29963]: 5.15 deep-scrub ok 2024-03-30T14:05:50.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:50 smithi043 ceph-mon[29963]: osdmap e510: 8 total, 8 up, 8 in 2024-03-30T14:05:50.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:50 smithi043 ceph-mon[42099]: 5.15 deep-scrub starts 2024-03-30T14:05:50.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:50 smithi043 ceph-mon[42099]: 5.15 deep-scrub ok 2024-03-30T14:05:50.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:50 smithi043 ceph-mon[42099]: osdmap e510: 8 total, 8 up, 8 in 2024-03-30T14:05:50.997 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T14:05:51.340 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[42099]: 6.6 scrub starts 2024-03-30T14:05:51.340 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[42099]: 6.6 scrub ok 2024-03-30T14:05:51.340 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:51.340 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[42099]: pgmap v1102: 152 pgs: 2 remapped+peering, 150 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 896 B/s rd, 0 op/s; 0 B/s, 1 objects/s recovering 2024-03-30T14:05:51.341 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[42099]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-30T14:05:51.341 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/711594378' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T14:05:51.341 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[42099]: 3.14 scrub starts 2024-03-30T14:05:51.341 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[42099]: 3.14 scrub ok 2024-03-30T14:05:51.341 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/711594378' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-30T14:05:51.341 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[42099]: osdmap e511: 8 total, 8 up, 8 in 2024-03-30T14:05:51.341 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[42099]: 5.10 scrub starts 2024-03-30T14:05:51.341 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[42099]: 5.10 scrub ok 2024-03-30T14:05:51.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[29963]: 6.6 scrub starts 2024-03-30T14:05:51.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[29963]: 6.6 scrub ok 2024-03-30T14:05:51.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:51.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[29963]: pgmap v1102: 152 pgs: 2 remapped+peering, 150 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 896 B/s rd, 0 op/s; 0 B/s, 1 objects/s recovering 2024-03-30T14:05:51.342 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[29963]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-30T14:05:51.342 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/711594378' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T14:05:51.342 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[29963]: 3.14 scrub starts 2024-03-30T14:05:51.342 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[29963]: 3.14 scrub ok 2024-03-30T14:05:51.342 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/711594378' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-30T14:05:51.342 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[29963]: osdmap e511: 8 total, 8 up, 8 in 2024-03-30T14:05:51.342 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[29963]: 5.10 scrub starts 2024-03-30T14:05:51.342 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:51 smithi043 ceph-mon[29963]: 5.10 scrub ok 2024-03-30T14:05:51.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:51 smithi113 ceph-mon[34341]: 6.6 scrub starts 2024-03-30T14:05:51.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:51 smithi113 ceph-mon[34341]: 6.6 scrub ok 2024-03-30T14:05:51.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:51 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:51.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:51 smithi113 ceph-mon[34341]: pgmap v1102: 152 pgs: 2 remapped+peering, 150 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 896 B/s rd, 0 op/s; 0 B/s, 1 objects/s recovering 2024-03-30T14:05:51.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:51 smithi113 ceph-mon[34341]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-30T14:05:51.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:51 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/711594378' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T14:05:51.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:51 smithi113 ceph-mon[34341]: 3.14 scrub starts 2024-03-30T14:05:51.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:51 smithi113 ceph-mon[34341]: 3.14 scrub ok 2024-03-30T14:05:51.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:51 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/711594378' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-30T14:05:51.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:51 smithi113 ceph-mon[34341]: osdmap e511: 8 total, 8 up, 8 in 2024-03-30T14:05:51.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:51 smithi113 ceph-mon[34341]: 5.10 scrub starts 2024-03-30T14:05:51.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:51 smithi113 ceph-mon[34341]: 5.10 scrub ok 2024-03-30T14:05:51.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd reweight 6 1 2024-03-30T14:05:52.001 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:05:52.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:52 smithi043 ceph-mon[42099]: 3.2 scrub starts 2024-03-30T14:05:52.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:52 smithi043 ceph-mon[42099]: 3.2 scrub ok 2024-03-30T14:05:52.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:52 smithi043 ceph-mon[42099]: 4.18 scrub starts 2024-03-30T14:05:52.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:52 smithi043 ceph-mon[42099]: 4.18 scrub ok 2024-03-30T14:05:52.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:52 smithi043 ceph-mon[29963]: 3.2 scrub starts 2024-03-30T14:05:52.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:52 smithi043 ceph-mon[29963]: 3.2 scrub ok 2024-03-30T14:05:52.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:52 smithi043 ceph-mon[29963]: 4.18 scrub starts 2024-03-30T14:05:52.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:52 smithi043 ceph-mon[29963]: 4.18 scrub ok 2024-03-30T14:05:52.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:52 smithi113 ceph-mon[34341]: 3.2 scrub starts 2024-03-30T14:05:52.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:52 smithi113 ceph-mon[34341]: 3.2 scrub ok 2024-03-30T14:05:52.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:52 smithi113 ceph-mon[34341]: 4.18 scrub starts 2024-03-30T14:05:52.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:52 smithi113 ceph-mon[34341]: 4.18 scrub ok 2024-03-30T14:05:53.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:53 smithi043 ceph-mon[29963]: 4.3 scrub starts 2024-03-30T14:05:53.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:53 smithi043 ceph-mon[29963]: 4.3 scrub ok 2024-03-30T14:05:53.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:53 smithi043 ceph-mon[29963]: pgmap v1105: 152 pgs: 2 remapped+peering, 150 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 383 B/s, 6 objects/s recovering 2024-03-30T14:05:53.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:53 smithi043 ceph-mon[29963]: 6.d scrub starts 2024-03-30T14:05:53.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:53 smithi043 ceph-mon[29963]: 6.d scrub ok 2024-03-30T14:05:53.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:53 smithi043 ceph-mon[29963]: 6.1c scrub starts 2024-03-30T14:05:53.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:53 smithi043 ceph-mon[29963]: 6.1c scrub ok 2024-03-30T14:05:53.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:53 smithi043 ceph-mon[42099]: 4.3 scrub starts 2024-03-30T14:05:53.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:53 smithi043 ceph-mon[42099]: 4.3 scrub ok 2024-03-30T14:05:53.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:53 smithi043 ceph-mon[42099]: pgmap v1105: 152 pgs: 2 remapped+peering, 150 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 383 B/s, 6 objects/s recovering 2024-03-30T14:05:53.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:53 smithi043 ceph-mon[42099]: 6.d scrub starts 2024-03-30T14:05:53.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:53 smithi043 ceph-mon[42099]: 6.d scrub ok 2024-03-30T14:05:53.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:53 smithi043 ceph-mon[42099]: 6.1c scrub starts 2024-03-30T14:05:53.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:53 smithi043 ceph-mon[42099]: 6.1c scrub ok 2024-03-30T14:05:53.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:53 smithi113 ceph-mon[34341]: 4.3 scrub starts 2024-03-30T14:05:53.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:53 smithi113 ceph-mon[34341]: 4.3 scrub ok 2024-03-30T14:05:53.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:53 smithi113 ceph-mon[34341]: pgmap v1105: 152 pgs: 2 remapped+peering, 150 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 383 B/s, 6 objects/s recovering 2024-03-30T14:05:53.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:53 smithi113 ceph-mon[34341]: 6.d scrub starts 2024-03-30T14:05:53.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:53 smithi113 ceph-mon[34341]: 6.d scrub ok 2024-03-30T14:05:53.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:53 smithi113 ceph-mon[34341]: 6.1c scrub starts 2024-03-30T14:05:53.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:53 smithi113 ceph-mon[34341]: 6.1c scrub ok 2024-03-30T14:05:54.184 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T14:05:54.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:54 smithi113 ceph-mon[34341]: 3.0 scrub starts 2024-03-30T14:05:54.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:54 smithi113 ceph-mon[34341]: 3.0 scrub ok 2024-03-30T14:05:54.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:54 smithi113 ceph-mon[34341]: 4.c scrub starts 2024-03-30T14:05:54.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:54 smithi113 ceph-mon[34341]: 4.c scrub ok 2024-03-30T14:05:54.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:54 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/625022420' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T14:05:54.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:54 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T14:05:54.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:54 smithi043 ceph-mon[29963]: 3.0 scrub starts 2024-03-30T14:05:54.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:54 smithi043 ceph-mon[29963]: 3.0 scrub ok 2024-03-30T14:05:54.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:54 smithi043 ceph-mon[29963]: 4.c scrub starts 2024-03-30T14:05:54.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:54 smithi043 ceph-mon[29963]: 4.c scrub ok 2024-03-30T14:05:54.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:54 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/625022420' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T14:05:54.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:54 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T14:05:54.522 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:54 smithi043 ceph-mon[42099]: 3.0 scrub starts 2024-03-30T14:05:54.522 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:54 smithi043 ceph-mon[42099]: 3.0 scrub ok 2024-03-30T14:05:54.522 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:54 smithi043 ceph-mon[42099]: 4.c scrub starts 2024-03-30T14:05:54.522 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:54 smithi043 ceph-mon[42099]: 4.c scrub ok 2024-03-30T14:05:54.522 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:54 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/625022420' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T14:05:54.522 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:54 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T14:05:54.879 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 6 1 2024-03-30T14:05:55.132 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:05:55.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:55 smithi113 ceph-mon[34341]: pgmap v1106: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 44 KiB/s, 4 objects/s recovering 2024-03-30T14:05:55.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:55 smithi113 ceph-mon[34341]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-30T14:05:55.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:55 smithi113 ceph-mon[34341]: Cluster is now healthy 2024-03-30T14:05:55.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:55 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-30T14:05:55.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:55 smithi113 ceph-mon[34341]: osdmap e512: 8 total, 8 up, 8 in 2024-03-30T14:05:55.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:55 smithi113 ceph-mon[34341]: 4.1 scrub starts 2024-03-30T14:05:55.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:55 smithi113 ceph-mon[34341]: 4.1 scrub ok 2024-03-30T14:05:55.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:55 smithi113 ceph-mon[34341]: 3.f deep-scrub starts 2024-03-30T14:05:55.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:55 smithi113 ceph-mon[34341]: 3.f deep-scrub ok 2024-03-30T14:05:55.533 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:55 smithi043 ceph-mon[29963]: pgmap v1106: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 44 KiB/s, 4 objects/s recovering 2024-03-30T14:05:55.533 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:55 smithi043 ceph-mon[29963]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-30T14:05:55.533 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:55 smithi043 ceph-mon[29963]: Cluster is now healthy 2024-03-30T14:05:55.533 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:55 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-30T14:05:55.534 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:55 smithi043 ceph-mon[29963]: osdmap e512: 8 total, 8 up, 8 in 2024-03-30T14:05:55.534 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:55 smithi043 ceph-mon[29963]: 4.1 scrub starts 2024-03-30T14:05:55.534 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:55 smithi043 ceph-mon[29963]: 4.1 scrub ok 2024-03-30T14:05:55.534 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:55 smithi043 ceph-mon[29963]: 3.f deep-scrub starts 2024-03-30T14:05:55.534 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:55 smithi043 ceph-mon[29963]: 3.f deep-scrub ok 2024-03-30T14:05:55.534 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:55 smithi043 ceph-mon[42099]: pgmap v1106: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 44 KiB/s, 4 objects/s recovering 2024-03-30T14:05:55.534 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:55 smithi043 ceph-mon[42099]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-30T14:05:55.534 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:55 smithi043 ceph-mon[42099]: Cluster is now healthy 2024-03-30T14:05:55.534 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:55 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-30T14:05:55.534 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:55 smithi043 ceph-mon[42099]: osdmap e512: 8 total, 8 up, 8 in 2024-03-30T14:05:55.534 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:55 smithi043 ceph-mon[42099]: 4.1 scrub starts 2024-03-30T14:05:55.534 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:55 smithi043 ceph-mon[42099]: 4.1 scrub ok 2024-03-30T14:05:55.535 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:55 smithi043 ceph-mon[42099]: 3.f deep-scrub starts 2024-03-30T14:05:55.535 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:55 smithi043 ceph-mon[42099]: 3.f deep-scrub ok 2024-03-30T14:05:56.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:05:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:05:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:05:56.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:56 smithi113 ceph-mon[34341]: 3.1b scrub starts 2024-03-30T14:05:56.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:56 smithi113 ceph-mon[34341]: 3.1b scrub ok 2024-03-30T14:05:56.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:56 smithi113 ceph-mon[34341]: 3.7 scrub starts 2024-03-30T14:05:56.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:56 smithi113 ceph-mon[34341]: 3.7 scrub ok 2024-03-30T14:05:56.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:56 smithi113 ceph-mon[34341]: 6.1 scrub starts 2024-03-30T14:05:56.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:56 smithi113 ceph-mon[34341]: 6.1 scrub ok 2024-03-30T14:05:56.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:56 smithi113 ceph-mon[34341]: 5.1a scrub starts 2024-03-30T14:05:56.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:56 smithi113 ceph-mon[34341]: 5.1a scrub ok 2024-03-30T14:05:56.482 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:56 smithi043 ceph-mon[29963]: 3.1b scrub starts 2024-03-30T14:05:56.483 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:56 smithi043 ceph-mon[29963]: 3.1b scrub ok 2024-03-30T14:05:56.483 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:56 smithi043 ceph-mon[29963]: 3.7 scrub starts 2024-03-30T14:05:56.483 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:56 smithi043 ceph-mon[29963]: 3.7 scrub ok 2024-03-30T14:05:56.483 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:56 smithi043 ceph-mon[29963]: 6.1 scrub starts 2024-03-30T14:05:56.483 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:56 smithi043 ceph-mon[29963]: 6.1 scrub ok 2024-03-30T14:05:56.483 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:56 smithi043 ceph-mon[29963]: 5.1a scrub starts 2024-03-30T14:05:56.483 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:56 smithi043 ceph-mon[29963]: 5.1a scrub ok 2024-03-30T14:05:56.483 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:56 smithi043 ceph-mon[42099]: 3.1b scrub starts 2024-03-30T14:05:56.483 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:56 smithi043 ceph-mon[42099]: 3.1b scrub ok 2024-03-30T14:05:56.483 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:56 smithi043 ceph-mon[42099]: 3.7 scrub starts 2024-03-30T14:05:56.483 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:56 smithi043 ceph-mon[42099]: 3.7 scrub ok 2024-03-30T14:05:56.484 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:56 smithi043 ceph-mon[42099]: 6.1 scrub starts 2024-03-30T14:05:56.484 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:56 smithi043 ceph-mon[42099]: 6.1 scrub ok 2024-03-30T14:05:56.484 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:56 smithi043 ceph-mon[42099]: 5.1a scrub starts 2024-03-30T14:05:56.484 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:56 smithi043 ceph-mon[42099]: 5.1a scrub ok 2024-03-30T14:05:57.196 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T14:05:57.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:57 smithi113 ceph-mon[34341]: 3.12 deep-scrub starts 2024-03-30T14:05:57.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:57 smithi113 ceph-mon[34341]: 3.12 deep-scrub ok 2024-03-30T14:05:57.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:57 smithi113 ceph-mon[34341]: pgmap v1108: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 43 KiB/s, 3 objects/s recovering 2024-03-30T14:05:57.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:57 smithi113 ceph-mon[34341]: 6.1a scrub starts 2024-03-30T14:05:57.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:57 smithi113 ceph-mon[34341]: 6.1a scrub ok 2024-03-30T14:05:57.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:57 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/93238429' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T14:05:57.519 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:57 smithi043 ceph-mon[29963]: 3.12 deep-scrub starts 2024-03-30T14:05:57.519 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:57 smithi043 ceph-mon[29963]: 3.12 deep-scrub ok 2024-03-30T14:05:57.519 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:57 smithi043 ceph-mon[29963]: pgmap v1108: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 43 KiB/s, 3 objects/s recovering 2024-03-30T14:05:57.519 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:57 smithi043 ceph-mon[29963]: 6.1a scrub starts 2024-03-30T14:05:57.519 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:57 smithi043 ceph-mon[29963]: 6.1a scrub ok 2024-03-30T14:05:57.519 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:57 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/93238429' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T14:05:57.520 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:57 smithi043 ceph-mon[42099]: 3.12 deep-scrub starts 2024-03-30T14:05:57.520 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:57 smithi043 ceph-mon[42099]: 3.12 deep-scrub ok 2024-03-30T14:05:57.520 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:57 smithi043 ceph-mon[42099]: pgmap v1108: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 43 KiB/s, 3 objects/s recovering 2024-03-30T14:05:57.520 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:57 smithi043 ceph-mon[42099]: 6.1a scrub starts 2024-03-30T14:05:57.520 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:57 smithi043 ceph-mon[42099]: 6.1a scrub ok 2024-03-30T14:05:57.520 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:57 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/93238429' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T14:05:57.912 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd reweight 7 1 2024-03-30T14:05:58.166 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:05:58.447 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:58 smithi043 ceph-mon[42099]: 3.1f scrub starts 2024-03-30T14:05:58.447 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:58 smithi043 ceph-mon[42099]: 3.1f scrub ok 2024-03-30T14:05:58.447 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:58 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/93238429' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-30T14:05:58.448 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:58 smithi043 ceph-mon[42099]: osdmap e513: 8 total, 8 up, 8 in 2024-03-30T14:05:58.448 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:58 smithi043 ceph-mon[42099]: pgmap v1110: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 746 B/s rd, 0 op/s; 38 KiB/s, 3 objects/s recovering 2024-03-30T14:05:58.448 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:58 smithi043 ceph-mon[29963]: 3.1f scrub starts 2024-03-30T14:05:58.448 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:58 smithi043 ceph-mon[29963]: 3.1f scrub ok 2024-03-30T14:05:58.448 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:58 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/93238429' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-30T14:05:58.448 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:58 smithi043 ceph-mon[29963]: osdmap e513: 8 total, 8 up, 8 in 2024-03-30T14:05:58.448 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:58 smithi043 ceph-mon[29963]: pgmap v1110: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 746 B/s rd, 0 op/s; 38 KiB/s, 3 objects/s recovering 2024-03-30T14:05:58.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:58 smithi113 ceph-mon[34341]: 3.1f scrub starts 2024-03-30T14:05:58.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:58 smithi113 ceph-mon[34341]: 3.1f scrub ok 2024-03-30T14:05:58.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:58 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/93238429' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-30T14:05:58.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:58 smithi113 ceph-mon[34341]: osdmap e513: 8 total, 8 up, 8 in 2024-03-30T14:05:58.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:58 smithi113 ceph-mon[34341]: pgmap v1110: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 746 B/s rd, 0 op/s; 38 KiB/s, 3 objects/s recovering 2024-03-30T14:05:59.369 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:59 smithi043 ceph-mon[29963]: 3.9 scrub starts 2024-03-30T14:05:59.369 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:59 smithi043 ceph-mon[29963]: 3.9 scrub ok 2024-03-30T14:05:59.369 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:59 smithi043 ceph-mon[29963]: 1.9 scrub starts 2024-03-30T14:05:59.369 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:05:59 smithi043 ceph-mon[29963]: 1.9 scrub ok 2024-03-30T14:05:59.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:59 smithi043 ceph-mon[42099]: 3.9 scrub starts 2024-03-30T14:05:59.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:59 smithi043 ceph-mon[42099]: 3.9 scrub ok 2024-03-30T14:05:59.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:59 smithi043 ceph-mon[42099]: 1.9 scrub starts 2024-03-30T14:05:59.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:05:59 smithi043 ceph-mon[42099]: 1.9 scrub ok 2024-03-30T14:05:59.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:59 smithi113 ceph-mon[34341]: 3.9 scrub starts 2024-03-30T14:05:59.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:59 smithi113 ceph-mon[34341]: 3.9 scrub ok 2024-03-30T14:05:59.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:59 smithi113 ceph-mon[34341]: 1.9 scrub starts 2024-03-30T14:05:59.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:05:59 smithi113 ceph-mon[34341]: 1.9 scrub ok 2024-03-30T14:05:59.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:05:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:06:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T14:06:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:00 smithi113 ceph-mon[34341]: 3.18 scrub starts 2024-03-30T14:06:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:00 smithi113 ceph-mon[34341]: 3.18 scrub ok 2024-03-30T14:06:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:00 smithi113 ceph-mon[34341]: 3.5 scrub starts 2024-03-30T14:06:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:00 smithi113 ceph-mon[34341]: 3.5 scrub ok 2024-03-30T14:06:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:00 smithi113 ceph-mon[34341]: 4.13 scrub starts 2024-03-30T14:06:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:00 smithi113 ceph-mon[34341]: 4.13 scrub ok 2024-03-30T14:06:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:00 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:00 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:00 smithi113 ceph-mon[34341]: pgmap v1111: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 32 KiB/s, 0 objects/s recovering 2024-03-30T14:06:00.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:00 smithi113 ceph-mon[34341]: 6.12 scrub starts 2024-03-30T14:06:00.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:00 smithi113 ceph-mon[34341]: 6.12 scrub ok 2024-03-30T14:06:00.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:00 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3614979798' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T14:06:00.526 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[29963]: 3.18 scrub starts 2024-03-30T14:06:00.526 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[29963]: 3.18 scrub ok 2024-03-30T14:06:00.526 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[29963]: 3.5 scrub starts 2024-03-30T14:06:00.526 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[29963]: 3.5 scrub ok 2024-03-30T14:06:00.526 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[29963]: 4.13 scrub starts 2024-03-30T14:06:00.526 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[29963]: 4.13 scrub ok 2024-03-30T14:06:00.526 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:00.526 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:00.526 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[29963]: pgmap v1111: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 32 KiB/s, 0 objects/s recovering 2024-03-30T14:06:00.526 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[29963]: 6.12 scrub starts 2024-03-30T14:06:00.526 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[29963]: 6.12 scrub ok 2024-03-30T14:06:00.526 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3614979798' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T14:06:00.527 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[42099]: 3.18 scrub starts 2024-03-30T14:06:00.527 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[42099]: 3.18 scrub ok 2024-03-30T14:06:00.527 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[42099]: 3.5 scrub starts 2024-03-30T14:06:00.527 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[42099]: 3.5 scrub ok 2024-03-30T14:06:00.527 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[42099]: 4.13 scrub starts 2024-03-30T14:06:00.527 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[42099]: 4.13 scrub ok 2024-03-30T14:06:00.527 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:00.527 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:00.527 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[42099]: pgmap v1111: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 32 KiB/s, 0 objects/s recovering 2024-03-30T14:06:00.527 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[42099]: 6.12 scrub starts 2024-03-30T14:06:00.527 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[42099]: 6.12 scrub ok 2024-03-30T14:06:00.527 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:00 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3614979798' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T14:06:00.908 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 7 1 2024-03-30T14:06:01.159 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:01.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:01 smithi113 ceph-mon[34341]: 5.17 scrub starts 2024-03-30T14:06:01.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:01 smithi113 ceph-mon[34341]: 5.17 scrub ok 2024-03-30T14:06:01.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:01 smithi113 ceph-mon[34341]: 4.12 scrub starts 2024-03-30T14:06:01.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:01 smithi113 ceph-mon[34341]: 4.12 scrub ok 2024-03-30T14:06:01.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:01 smithi113 ceph-mon[34341]: 4.a scrub starts 2024-03-30T14:06:01.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:01 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3614979798' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-30T14:06:01.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:01 smithi113 ceph-mon[34341]: 4.a scrub ok 2024-03-30T14:06:01.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:01 smithi113 ceph-mon[34341]: osdmap e514: 8 total, 8 up, 8 in 2024-03-30T14:06:01.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:01 smithi113 ceph-mon[34341]: 1.5 scrub starts 2024-03-30T14:06:01.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:01 smithi113 ceph-mon[34341]: 1.5 scrub ok 2024-03-30T14:06:01.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:01 smithi113 ceph-mon[34341]: 1.d scrub starts 2024-03-30T14:06:01.465 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:01 smithi113 ceph-mon[34341]: 1.d scrub ok 2024-03-30T14:06:01.483 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[29963]: 5.17 scrub starts 2024-03-30T14:06:01.483 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[29963]: 5.17 scrub ok 2024-03-30T14:06:01.483 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[29963]: 4.12 scrub starts 2024-03-30T14:06:01.483 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[29963]: 4.12 scrub ok 2024-03-30T14:06:01.483 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[29963]: 4.a scrub starts 2024-03-30T14:06:01.483 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3614979798' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-30T14:06:01.484 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[29963]: 4.a scrub ok 2024-03-30T14:06:01.484 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[29963]: osdmap e514: 8 total, 8 up, 8 in 2024-03-30T14:06:01.484 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[29963]: 1.5 scrub starts 2024-03-30T14:06:01.484 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[29963]: 1.5 scrub ok 2024-03-30T14:06:01.484 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[29963]: 1.d scrub starts 2024-03-30T14:06:01.484 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[29963]: 1.d scrub ok 2024-03-30T14:06:01.484 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[42099]: 5.17 scrub starts 2024-03-30T14:06:01.484 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[42099]: 5.17 scrub ok 2024-03-30T14:06:01.484 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[42099]: 4.12 scrub starts 2024-03-30T14:06:01.484 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[42099]: 4.12 scrub ok 2024-03-30T14:06:01.485 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[42099]: 4.a scrub starts 2024-03-30T14:06:01.485 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3614979798' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-30T14:06:01.485 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[42099]: 4.a scrub ok 2024-03-30T14:06:01.485 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[42099]: osdmap e514: 8 total, 8 up, 8 in 2024-03-30T14:06:01.485 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[42099]: 1.5 scrub starts 2024-03-30T14:06:01.485 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[42099]: 1.5 scrub ok 2024-03-30T14:06:01.485 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[42099]: 1.d scrub starts 2024-03-30T14:06:01.485 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:01 smithi043 ceph-mon[42099]: 1.d scrub ok 2024-03-30T14:06:02.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:02 smithi043 ceph-mon[29963]: 4.7 scrub starts 2024-03-30T14:06:02.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:02 smithi043 ceph-mon[29963]: 4.7 scrub ok 2024-03-30T14:06:02.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:02 smithi043 ceph-mon[29963]: 1.e scrub starts 2024-03-30T14:06:02.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:02 smithi043 ceph-mon[29963]: 1.e scrub ok 2024-03-30T14:06:02.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:02 smithi043 ceph-mon[29963]: pgmap v1113: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 667 B/s rd, 0 op/s 2024-03-30T14:06:02.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:02 smithi043 ceph-mon[42099]: 4.7 scrub starts 2024-03-30T14:06:02.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:02 smithi043 ceph-mon[42099]: 4.7 scrub ok 2024-03-30T14:06:02.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:02 smithi043 ceph-mon[42099]: 1.e scrub starts 2024-03-30T14:06:02.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:02 smithi043 ceph-mon[42099]: 1.e scrub ok 2024-03-30T14:06:02.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:02 smithi043 ceph-mon[42099]: pgmap v1113: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 667 B/s rd, 0 op/s 2024-03-30T14:06:02.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:02 smithi113 ceph-mon[34341]: 4.7 scrub starts 2024-03-30T14:06:02.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:02 smithi113 ceph-mon[34341]: 4.7 scrub ok 2024-03-30T14:06:02.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:02 smithi113 ceph-mon[34341]: 1.e scrub starts 2024-03-30T14:06:02.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:02 smithi113 ceph-mon[34341]: 1.e scrub ok 2024-03-30T14:06:02.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:02 smithi113 ceph-mon[34341]: pgmap v1113: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 667 B/s rd, 0 op/s 2024-03-30T14:06:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:03 smithi043 ceph-mon[29963]: 5.7 scrub starts 2024-03-30T14:06:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:03 smithi043 ceph-mon[29963]: 5.7 scrub ok 2024-03-30T14:06:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:03 smithi043 ceph-mon[29963]: 5.1 scrub starts 2024-03-30T14:06:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:03 smithi043 ceph-mon[29963]: 5.1 scrub ok 2024-03-30T14:06:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:03 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:06:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:03 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:06:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:03 smithi043 ceph-mon[29963]: osdmap e515: 8 total, 8 up, 8 in 2024-03-30T14:06:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:03 smithi043 ceph-mon[42099]: 5.7 scrub starts 2024-03-30T14:06:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:03 smithi043 ceph-mon[42099]: 5.7 scrub ok 2024-03-30T14:06:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:03 smithi043 ceph-mon[42099]: 5.1 scrub starts 2024-03-30T14:06:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:03 smithi043 ceph-mon[42099]: 5.1 scrub ok 2024-03-30T14:06:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:03 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:06:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:03 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:06:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:03 smithi043 ceph-mon[42099]: osdmap e515: 8 total, 8 up, 8 in 2024-03-30T14:06:03.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:03 smithi113 ceph-mon[34341]: 5.7 scrub starts 2024-03-30T14:06:03.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:03 smithi113 ceph-mon[34341]: 5.7 scrub ok 2024-03-30T14:06:03.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:03 smithi113 ceph-mon[34341]: 5.1 scrub starts 2024-03-30T14:06:03.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:03 smithi113 ceph-mon[34341]: 5.1 scrub ok 2024-03-30T14:06:03.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:03 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:06:03.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:03 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:06:03.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:03 smithi113 ceph-mon[34341]: osdmap e515: 8 total, 8 up, 8 in 2024-03-30T14:06:04.225 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T14:06:04.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:04 smithi043 ceph-mon[29963]: 6.c scrub starts 2024-03-30T14:06:04.692 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:04 smithi043 ceph-mon[29963]: 6.c scrub ok 2024-03-30T14:06:04.692 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:04 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/232117847' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T14:06:04.692 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:04 smithi043 ceph-mon[29963]: pgmap v1115: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T14:06:04.692 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:04 smithi043 ceph-mon[42099]: 6.c scrub starts 2024-03-30T14:06:04.692 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:04 smithi043 ceph-mon[42099]: 6.c scrub ok 2024-03-30T14:06:04.693 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:04 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/232117847' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T14:06:04.693 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:04 smithi043 ceph-mon[42099]: pgmap v1115: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T14:06:04.713 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:04 smithi113 ceph-mon[34341]: 6.c scrub starts 2024-03-30T14:06:04.713 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:04 smithi113 ceph-mon[34341]: 6.c scrub ok 2024-03-30T14:06:04.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:04 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/232117847' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T14:06:04.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:04 smithi113 ceph-mon[34341]: pgmap v1115: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T14:06:05.255 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd reweight 2 1 2024-03-30T14:06:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:05.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:05 smithi043 ceph-mon[29963]: 6.b deep-scrub starts 2024-03-30T14:06:05.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:05 smithi043 ceph-mon[29963]: 6.b deep-scrub ok 2024-03-30T14:06:05.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:05 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/232117847' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-30T14:06:05.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:05 smithi043 ceph-mon[29963]: osdmap e516: 8 total, 8 up, 8 in 2024-03-30T14:06:05.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:05 smithi043 ceph-mon[29963]: 5.9 scrub starts 2024-03-30T14:06:05.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:05 smithi043 ceph-mon[29963]: 5.9 scrub ok 2024-03-30T14:06:05.602 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:05 smithi043 ceph-mon[42099]: 6.b deep-scrub starts 2024-03-30T14:06:05.602 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:05 smithi043 ceph-mon[42099]: 6.b deep-scrub ok 2024-03-30T14:06:05.602 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:05 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/232117847' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-30T14:06:05.602 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:05 smithi043 ceph-mon[42099]: osdmap e516: 8 total, 8 up, 8 in 2024-03-30T14:06:05.603 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:05 smithi043 ceph-mon[42099]: 5.9 scrub starts 2024-03-30T14:06:05.603 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:05 smithi043 ceph-mon[42099]: 5.9 scrub ok 2024-03-30T14:06:05.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:05 smithi113 ceph-mon[34341]: 6.b deep-scrub starts 2024-03-30T14:06:05.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:05 smithi113 ceph-mon[34341]: 6.b deep-scrub ok 2024-03-30T14:06:05.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:05 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/232117847' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-30T14:06:05.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:05 smithi113 ceph-mon[34341]: osdmap e516: 8 total, 8 up, 8 in 2024-03-30T14:06:05.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:05 smithi113 ceph-mon[34341]: 5.9 scrub starts 2024-03-30T14:06:05.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:05 smithi113 ceph-mon[34341]: 5.9 scrub ok 2024-03-30T14:06:06.140 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:06:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:06:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:06:06.634 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:06 smithi043 ceph-mon[29963]: 6.1b scrub starts 2024-03-30T14:06:06.634 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:06 smithi043 ceph-mon[29963]: 6.1b scrub ok 2024-03-30T14:06:06.634 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:06 smithi043 ceph-mon[29963]: 1.8 scrub starts 2024-03-30T14:06:06.634 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:06 smithi043 ceph-mon[29963]: 1.8 scrub ok 2024-03-30T14:06:06.634 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:06 smithi043 ceph-mon[29963]: 5.19 scrub starts 2024-03-30T14:06:06.634 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:06 smithi043 ceph-mon[29963]: 5.19 scrub ok 2024-03-30T14:06:06.634 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:06 smithi043 ceph-mon[29963]: pgmap v1117: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:06:06.635 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:06 smithi043 ceph-mon[42099]: 6.1b scrub starts 2024-03-30T14:06:06.635 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:06 smithi043 ceph-mon[42099]: 6.1b scrub ok 2024-03-30T14:06:06.635 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:06 smithi043 ceph-mon[42099]: 1.8 scrub starts 2024-03-30T14:06:06.635 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:06 smithi043 ceph-mon[42099]: 1.8 scrub ok 2024-03-30T14:06:06.635 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:06 smithi043 ceph-mon[42099]: 5.19 scrub starts 2024-03-30T14:06:06.635 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:06 smithi043 ceph-mon[42099]: 5.19 scrub ok 2024-03-30T14:06:06.635 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:06 smithi043 ceph-mon[42099]: pgmap v1117: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:06:06.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:06 smithi113 ceph-mon[34341]: 6.1b scrub starts 2024-03-30T14:06:06.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:06 smithi113 ceph-mon[34341]: 6.1b scrub ok 2024-03-30T14:06:06.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:06 smithi113 ceph-mon[34341]: 1.8 scrub starts 2024-03-30T14:06:06.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:06 smithi113 ceph-mon[34341]: 1.8 scrub ok 2024-03-30T14:06:06.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:06 smithi113 ceph-mon[34341]: 5.19 scrub starts 2024-03-30T14:06:06.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:06 smithi113 ceph-mon[34341]: 5.19 scrub ok 2024-03-30T14:06:06.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:06 smithi113 ceph-mon[34341]: pgmap v1117: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:06:07.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[29963]: 4.17 scrub starts 2024-03-30T14:06:07.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[29963]: 4.17 scrub ok 2024-03-30T14:06:07.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[29963]: 6.19 scrub starts 2024-03-30T14:06:07.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[29963]: 6.19 scrub ok 2024-03-30T14:06:07.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:06:07.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:06:07.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[29963]: 3.a scrub starts 2024-03-30T14:06:07.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[29963]: 3.a scrub ok 2024-03-30T14:06:07.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[29963]: osdmap e517: 8 total, 8 up, 8 in 2024-03-30T14:06:07.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:06:07.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:06:07.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[29963]: 5.d scrub starts 2024-03-30T14:06:07.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[29963]: 5.d scrub ok 2024-03-30T14:06:07.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[42099]: 4.17 scrub starts 2024-03-30T14:06:07.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[42099]: 4.17 scrub ok 2024-03-30T14:06:07.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[42099]: 6.19 scrub starts 2024-03-30T14:06:07.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[42099]: 6.19 scrub ok 2024-03-30T14:06:07.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:06:07.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:06:07.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[42099]: 3.a scrub starts 2024-03-30T14:06:07.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[42099]: 3.a scrub ok 2024-03-30T14:06:07.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[42099]: osdmap e517: 8 total, 8 up, 8 in 2024-03-30T14:06:07.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:06:07.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:06:07.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[42099]: 5.d scrub starts 2024-03-30T14:06:07.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:07 smithi043 ceph-mon[42099]: 5.d scrub ok 2024-03-30T14:06:07.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:07 smithi113 ceph-mon[34341]: 4.17 scrub starts 2024-03-30T14:06:07.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:07 smithi113 ceph-mon[34341]: 4.17 scrub ok 2024-03-30T14:06:07.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:07 smithi113 ceph-mon[34341]: 6.19 scrub starts 2024-03-30T14:06:07.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:07 smithi113 ceph-mon[34341]: 6.19 scrub ok 2024-03-30T14:06:07.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:07 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:06:07.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:07 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:06:07.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:07 smithi113 ceph-mon[34341]: 3.a scrub starts 2024-03-30T14:06:07.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:07 smithi113 ceph-mon[34341]: 3.a scrub ok 2024-03-30T14:06:07.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:07 smithi113 ceph-mon[34341]: osdmap e517: 8 total, 8 up, 8 in 2024-03-30T14:06:07.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:07 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:06:07.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:07 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:06:07.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:07 smithi113 ceph-mon[34341]: 5.d scrub starts 2024-03-30T14:06:07.715 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:07 smithi113 ceph-mon[34341]: 5.d scrub ok 2024-03-30T14:06:07.999 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T14:06:08.325 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[42099]: 5.13 scrub starts 2024-03-30T14:06:08.325 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[42099]: 5.13 scrub ok 2024-03-30T14:06:08.325 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[42099]: 3.1c scrub starts 2024-03-30T14:06:08.325 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[42099]: 3.1c scrub ok 2024-03-30T14:06:08.325 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[42099]: osdmap e518: 8 total, 8 up, 8 in 2024-03-30T14:06:08.325 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[42099]: 5.18 scrub starts 2024-03-30T14:06:08.325 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[42099]: 5.18 scrub ok 2024-03-30T14:06:08.325 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3028148433' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T14:06:08.325 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[42099]: pgmap v1120: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:06:08.326 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3028148433' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-30T14:06:08.326 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[42099]: osdmap e519: 8 total, 8 up, 8 in 2024-03-30T14:06:08.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[29963]: 5.13 scrub starts 2024-03-30T14:06:08.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[29963]: 5.13 scrub ok 2024-03-30T14:06:08.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[29963]: 3.1c scrub starts 2024-03-30T14:06:08.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[29963]: 3.1c scrub ok 2024-03-30T14:06:08.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[29963]: osdmap e518: 8 total, 8 up, 8 in 2024-03-30T14:06:08.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[29963]: 5.18 scrub starts 2024-03-30T14:06:08.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[29963]: 5.18 scrub ok 2024-03-30T14:06:08.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3028148433' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T14:06:08.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[29963]: pgmap v1120: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:06:08.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3028148433' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-30T14:06:08.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:08 smithi043 ceph-mon[29963]: osdmap e519: 8 total, 8 up, 8 in 2024-03-30T14:06:08.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:08 smithi113 ceph-mon[34341]: 5.13 scrub starts 2024-03-30T14:06:08.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:08 smithi113 ceph-mon[34341]: 5.13 scrub ok 2024-03-30T14:06:08.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:08 smithi113 ceph-mon[34341]: 3.1c scrub starts 2024-03-30T14:06:08.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:08 smithi113 ceph-mon[34341]: 3.1c scrub ok 2024-03-30T14:06:08.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:08 smithi113 ceph-mon[34341]: osdmap e518: 8 total, 8 up, 8 in 2024-03-30T14:06:08.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:08 smithi113 ceph-mon[34341]: 5.18 scrub starts 2024-03-30T14:06:08.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:08 smithi113 ceph-mon[34341]: 5.18 scrub ok 2024-03-30T14:06:08.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:08 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3028148433' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T14:06:08.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:08 smithi113 ceph-mon[34341]: pgmap v1120: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:06:08.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:08 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3028148433' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-30T14:06:08.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:08 smithi113 ceph-mon[34341]: osdmap e519: 8 total, 8 up, 8 in 2024-03-30T14:06:08.725 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 2 1 2024-03-30T14:06:08.975 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:09.689 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:09 smithi113 ceph-mon[34341]: 3.17 scrub starts 2024-03-30T14:06:09.689 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:09 smithi113 ceph-mon[34341]: 3.17 scrub ok 2024-03-30T14:06:09.865 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:09 smithi043 ceph-mon[29963]: 3.17 scrub starts 2024-03-30T14:06:09.865 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:09 smithi043 ceph-mon[29963]: 3.17 scrub ok 2024-03-30T14:06:09.866 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:09 smithi043 ceph-mon[42099]: 3.17 scrub starts 2024-03-30T14:06:09.866 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:09 smithi043 ceph-mon[42099]: 3.17 scrub ok 2024-03-30T14:06:09.963 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:06:09 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:06:10.710 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:10 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:10.710 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:10 smithi043 ceph-mon[29963]: pgmap v1122: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-03-30T14:06:10.711 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:10 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:10.711 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:10 smithi043 ceph-mon[42099]: pgmap v1122: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-03-30T14:06:10.713 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:10 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:10.713 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:10 smithi113 ceph-mon[34341]: pgmap v1122: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-03-30T14:06:11.450 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T14:06:11.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:11 smithi113 ceph-mon[34341]: 1.12 scrub starts 2024-03-30T14:06:11.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:11 smithi113 ceph-mon[34341]: 1.12 scrub ok 2024-03-30T14:06:11.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:11 smithi113 ceph-mon[34341]: 6.2 scrub starts 2024-03-30T14:06:11.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:11 smithi113 ceph-mon[34341]: 6.2 scrub ok 2024-03-30T14:06:11.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:11 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:06:11.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:11 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:06:11.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:11 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/570395118' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T14:06:11.756 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:11 smithi043 ceph-mon[29963]: 1.12 scrub starts 2024-03-30T14:06:11.756 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:11 smithi043 ceph-mon[29963]: 1.12 scrub ok 2024-03-30T14:06:11.756 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:11 smithi043 ceph-mon[29963]: 6.2 scrub starts 2024-03-30T14:06:11.756 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:11 smithi043 ceph-mon[29963]: 6.2 scrub ok 2024-03-30T14:06:11.756 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:11 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:06:11.756 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:11 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:06:11.756 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:11 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/570395118' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T14:06:11.756 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:11 smithi043 ceph-mon[42099]: 1.12 scrub starts 2024-03-30T14:06:11.756 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:11 smithi043 ceph-mon[42099]: 1.12 scrub ok 2024-03-30T14:06:11.756 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:11 smithi043 ceph-mon[42099]: 6.2 scrub starts 2024-03-30T14:06:11.757 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:11 smithi043 ceph-mon[42099]: 6.2 scrub ok 2024-03-30T14:06:11.757 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:11 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:06:11.757 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:11 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:06:11.757 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:11 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/570395118' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T14:06:12.240 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-30T14:06:12.240 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-30T14:06:12.240 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-30T14:06:12.240 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-30T14:06:12.240 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-30T14:06:12.241 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T14:06:12.501 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:12.623 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:12 smithi043 ceph-mon[29963]: 6.13 scrub starts 2024-03-30T14:06:12.624 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:12 smithi043 ceph-mon[29963]: 6.13 scrub ok 2024-03-30T14:06:12.624 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:12 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/570395118' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-30T14:06:12.624 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:12 smithi043 ceph-mon[29963]: osdmap e520: 8 total, 8 up, 8 in 2024-03-30T14:06:12.624 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:12 smithi043 ceph-mon[29963]: 3.13 deep-scrub starts 2024-03-30T14:06:12.624 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:12 smithi043 ceph-mon[29963]: 3.13 deep-scrub ok 2024-03-30T14:06:12.624 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:12 smithi043 ceph-mon[29963]: pgmap v1124: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-03-30T14:06:12.624 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:12 smithi043 ceph-mon[42099]: 6.13 scrub starts 2024-03-30T14:06:12.624 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:12 smithi043 ceph-mon[42099]: 6.13 scrub ok 2024-03-30T14:06:12.624 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:12 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/570395118' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-30T14:06:12.624 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:12 smithi043 ceph-mon[42099]: osdmap e520: 8 total, 8 up, 8 in 2024-03-30T14:06:12.624 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:12 smithi043 ceph-mon[42099]: 3.13 deep-scrub starts 2024-03-30T14:06:12.624 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:12 smithi043 ceph-mon[42099]: 3.13 deep-scrub ok 2024-03-30T14:06:12.625 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:12 smithi043 ceph-mon[42099]: pgmap v1124: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-03-30T14:06:12.713 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:12 smithi113 ceph-mon[34341]: 6.13 scrub starts 2024-03-30T14:06:12.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:12 smithi113 ceph-mon[34341]: 6.13 scrub ok 2024-03-30T14:06:12.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:12 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/570395118' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-30T14:06:12.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:12 smithi113 ceph-mon[34341]: osdmap e520: 8 total, 8 up, 8 in 2024-03-30T14:06:12.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:12 smithi113 ceph-mon[34341]: 3.13 deep-scrub starts 2024-03-30T14:06:12.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:12 smithi113 ceph-mon[34341]: 3.13 deep-scrub ok 2024-03-30T14:06:12.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:12 smithi113 ceph-mon[34341]: pgmap v1124: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-03-30T14:06:13.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:13 smithi043 ceph-mon[29963]: osdmap e521: 8 total, 8 up, 8 in 2024-03-30T14:06:13.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:13 smithi043 ceph-mon[42099]: osdmap e521: 8 total, 8 up, 8 in 2024-03-30T14:06:13.713 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:13 smithi113 ceph-mon[34341]: osdmap e521: 8 total, 8 up, 8 in 2024-03-30T14:06:14.521 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T14:06:14.521 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":521,"fsid":"7ee932fa-ee9a-11ee-b647-cb9ed24678a4","created":"2024-03-30T13:37:07.346020+0000","modified":"2024-03-30T14:06:12.453348+0000","last_up_change":"2024-03-30T14:01:27.027194+0000","last_in_change":"2024-03-30T14:05:09.343313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T13:40:27.658979+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"491","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"368","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8999999761581421,"score_stable":1.8999999761581421,"optimal_score":1,"raw_score_acting":1.8999999761581421,"raw_score_stable":1.8999999761581421,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-30T13:42:42.989975+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":68,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-30T13:42:43.910306+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-30T13:42:45.633132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-30T13:42:47.570084+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-30T13:42:49.637189+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94ea083e-6a27-41db-83af-2f7d3cae1956","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":510,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1144592572},{"type":"v1","addr":"172.21.15.43:6803","nonce":1144592572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1144592572},{"type":"v1","addr":"172.21.15.43:6805","nonce":1144592572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":1144592572},{"type":"v1","addr":"172.21.15.43:6809","nonce":1144592572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1144592572},{"type":"v1","addr":"172.21.15.43:6807","nonce":1144592572}]},"public_addr":"172.21.15.43:6803/1144592572","cluster_addr":"172.21.15.43:6805/1144592572","heartbeat_back_addr":"172.21.15.43:6809/1144592572","heartbeat_front_addr":"172.21.15.43:6807/1144592572","state":["exists","up"]},{"osd":1,"uuid":"8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":510,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":3901165928},{"type":"v1","addr":"172.21.15.43:6811","nonce":3901165928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":3901165928},{"type":"v1","addr":"172.21.15.43:6813","nonce":3901165928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":3901165928},{"type":"v1","addr":"172.21.15.43:6817","nonce":3901165928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":3901165928},{"type":"v1","addr":"172.21.15.43:6815","nonce":3901165928}]},"public_addr":"172.21.15.43:6811/3901165928","cluster_addr":"172.21.15.43:6813/3901165928","heartbeat_back_addr":"172.21.15.43:6817/3901165928","heartbeat_front_addr":"172.21.15.43:6815/3901165928","state":["exists","up"]},{"osd":2,"uuid":"6db6782e-5889-443e-bc6c-df47a1f93bc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":166,"up_from":388,"up_thru":520,"down_at":167,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":886354154},{"type":"v1","addr":"172.21.15.43:6819","nonce":886354154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":886354154},{"type":"v1","addr":"172.21.15.43:6821","nonce":886354154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":886354154},{"type":"v1","addr":"172.21.15.43:6825","nonce":886354154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":886354154},{"type":"v1","addr":"172.21.15.43:6823","nonce":886354154}]},"public_addr":"172.21.15.43:6819/886354154","cluster_addr":"172.21.15.43:6821/886354154","heartbeat_back_addr":"172.21.15.43:6825/886354154","heartbeat_front_addr":"172.21.15.43:6823/886354154","state":["exists","up"]},{"osd":3,"uuid":"7048ff46-2cba-4d18-9835-95ecbddb2bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":510,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2933589376},{"type":"v1","addr":"172.21.15.43:6827","nonce":2933589376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2933589376},{"type":"v1","addr":"172.21.15.43:6829","nonce":2933589376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2933589376},{"type":"v1","addr":"172.21.15.43:6833","nonce":2933589376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2933589376},{"type":"v1","addr":"172.21.15.43:6831","nonce":2933589376}]},"public_addr":"172.21.15.43:6827/2933589376","cluster_addr":"172.21.15.43:6829/2933589376","heartbeat_back_addr":"172.21.15.43:6833/2933589376","heartbeat_front_addr":"172.21.15.43:6831/2933589376","state":["exists","up"]},{"osd":4,"uuid":"42da30e6-9eec-4ed5-8c73-9ce78aa8ff63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1945833081},{"type":"v1","addr":"172.21.15.113:6801","nonce":1945833081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1945833081},{"type":"v1","addr":"172.21.15.113:6803","nonce":1945833081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1945833081},{"type":"v1","addr":"172.21.15.113:6807","nonce":1945833081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1945833081},{"type":"v1","addr":"172.21.15.113:6805","nonce":1945833081}]},"public_addr":"172.21.15.113:6801/1945833081","cluster_addr":"172.21.15.113:6803/1945833081","heartbeat_back_addr":"172.21.15.113:6807/1945833081","heartbeat_front_addr":"172.21.15.113:6805/1945833081","state":["exists","up"]},{"osd":5,"uuid":"99a11b81-0df7-4b1d-946a-4297ba62e263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":510,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3426360325},{"type":"v1","addr":"172.21.15.113:6809","nonce":3426360325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3426360325},{"type":"v1","addr":"172.21.15.113:6811","nonce":3426360325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3426360325},{"type":"v1","addr":"172.21.15.113:6815","nonce":3426360325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3426360325},{"type":"v1","addr":"172.21.15.113:6813","nonce":3426360325}]},"public_addr":"172.21.15.113:6809/3426360325","cluster_addr":"172.21.15.113:6811/3426360325","heartbeat_back_addr":"172.21.15.113:6815/3426360325","heartbeat_front_addr":"172.21.15.113:6813/3426360325","state":["exists","up"]},{"osd":6,"uuid":"cafd0e26-78e9-4164-a43a-96ca8382fefe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":507,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1527017698},{"type":"v1","addr":"172.21.15.113:6817","nonce":1527017698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1527017698},{"type":"v1","addr":"172.21.15.113:6819","nonce":1527017698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1527017698},{"type":"v1","addr":"172.21.15.113:6823","nonce":1527017698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1527017698},{"type":"v1","addr":"172.21.15.113:6821","nonce":1527017698}]},"public_addr":"172.21.15.113:6817/1527017698","cluster_addr":"172.21.15.113:6819/1527017698","heartbeat_back_addr":"172.21.15.113:6823/1527017698","heartbeat_front_addr":"172.21.15.113:6821/1527017698","state":["exists","up"]},{"osd":7,"uuid":"e9d59662-9132-48f4-9e95-842a3cfbd79f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":507,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":3182476015},{"type":"v1","addr":"172.21.15.113:6825","nonce":3182476015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":3182476015},{"type":"v1","addr":"172.21.15.113:6827","nonce":3182476015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":3182476015},{"type":"v1","addr":"172.21.15.113:6831","nonce":3182476015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":3182476015},{"type":"v1","addr":"172.21.15.113:6829","nonce":3182476015}]},"public_addr":"172.21.15.113:6825/3182476015","cluster_addr":"172.21.15.113:6827/3182476015","heartbeat_back_addr":"172.21.15.113:6831/3182476015","heartbeat_front_addr":"172.21.15.113:6829/3182476015","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:06:10.749935+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:04:58.434668+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:05:18.173772+0000","dead_epoch":166},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:05:04.789933+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:06:06.562109+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:06:06.080176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:05:03.449335+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:06:02.971149+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"1.2","mappings":[{"from":0,"to":1}]},{"pgid":"1.d","mappings":[{"from":4,"to":0}]},{"pgid":"3.1d","mappings":[{"from":4,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.5","mappings":[{"from":4,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":3,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":0}]},{"pgid":"4.1a","mappings":[{"from":3,"to":7}]},{"pgid":"4.1b","mappings":[{"from":1,"to":0}]},{"pgid":"5.15","mappings":[{"from":0,"to":6}]},{"pgid":"5.17","mappings":[{"from":1,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/538304234":"2024-03-31T13:43:27.253365+0000","172.21.15.43:6801/3411723337":"2024-03-31T13:37:41.733515+0000","172.21.15.43:0/3656823679":"2024-03-31T13:37:41.733515+0000","172.21.15.43:0/2618817647":"2024-03-31T13:38:09.321740+0000","172.21.15.43:6800/3411723337":"2024-03-31T13:37:41.733515+0000","172.21.15.43:0/1206218784":"2024-03-31T13:37:41.733515+0000","172.21.15.43:0/886631486":"2024-03-31T13:37:41.733515+0000","172.21.15.43:6800/1339921781":"2024-03-31T13:38:09.321740+0000","172.21.15.43:6801/1339921781":"2024-03-31T13:38:09.321740+0000","172.21.15.43:0/2794494528":"2024-03-31T13:38:09.321740+0000","172.21.15.43:0/564868778":"2024-03-31T13:38:09.321740+0000","172.21.15.43:6800/3033707884":"2024-03-31T13:43:27.253365+0000","172.21.15.43:0/3539398453":"2024-03-31T13:43:27.253365+0000","172.21.15.43:0/2414364740":"2024-03-31T13:43:27.253365+0000","172.21.15.43:6801/3033707884":"2024-03-31T13:43:27.253365+0000","172.21.15.43:0/3923597429":"2024-03-31T13:43:27.253365+0000","172.21.15.43:0/2524269526":"2024-03-31T13:43:27.253365+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T14:06:14.713 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:14 smithi113 ceph-mon[34341]: 6.1f scrub starts 2024-03-30T14:06:14.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:14 smithi113 ceph-mon[34341]: 6.1f scrub ok 2024-03-30T14:06:14.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:14 smithi113 ceph-mon[34341]: 6.16 scrub starts 2024-03-30T14:06:14.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:14 smithi113 ceph-mon[34341]: 6.16 scrub ok 2024-03-30T14:06:14.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:14 smithi113 ceph-mon[34341]: pgmap v1126: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:06:14.829 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:14 smithi043 ceph-mon[29963]: 6.1f scrub starts 2024-03-30T14:06:14.829 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:14 smithi043 ceph-mon[29963]: 6.1f scrub ok 2024-03-30T14:06:14.829 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:14 smithi043 ceph-mon[29963]: 6.16 scrub starts 2024-03-30T14:06:14.829 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:14 smithi043 ceph-mon[29963]: 6.16 scrub ok 2024-03-30T14:06:14.829 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:14 smithi043 ceph-mon[29963]: pgmap v1126: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:06:14.829 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:14 smithi043 ceph-mon[42099]: 6.1f scrub starts 2024-03-30T14:06:14.830 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:14 smithi043 ceph-mon[42099]: 6.1f scrub ok 2024-03-30T14:06:14.830 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:14 smithi043 ceph-mon[42099]: 6.16 scrub starts 2024-03-30T14:06:14.830 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:14 smithi043 ceph-mon[42099]: 6.16 scrub ok 2024-03-30T14:06:14.830 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:14 smithi043 ceph-mon[42099]: pgmap v1126: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:06:15.132 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-30T14:06:15.132 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T14:06:15.378 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:15.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:15 smithi113 ceph-mon[34341]: 3.c scrub starts 2024-03-30T14:06:15.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:15 smithi113 ceph-mon[34341]: 3.c scrub ok 2024-03-30T14:06:15.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:15 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3297124862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:06:15.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:15 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:15.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:15 smithi113 ceph-mon[34341]: 5.4 scrub starts 2024-03-30T14:06:15.714 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:15 smithi113 ceph-mon[34341]: 5.4 scrub ok 2024-03-30T14:06:15.815 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:15 smithi043 ceph-mon[29963]: 3.c scrub starts 2024-03-30T14:06:15.816 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:15 smithi043 ceph-mon[29963]: 3.c scrub ok 2024-03-30T14:06:15.816 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:15 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3297124862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:06:15.816 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:15 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:15.816 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:15 smithi043 ceph-mon[29963]: 5.4 scrub starts 2024-03-30T14:06:15.816 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:15 smithi043 ceph-mon[29963]: 5.4 scrub ok 2024-03-30T14:06:15.816 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:15 smithi043 ceph-mon[42099]: 3.c scrub starts 2024-03-30T14:06:15.816 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:15 smithi043 ceph-mon[42099]: 3.c scrub ok 2024-03-30T14:06:15.816 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:15 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3297124862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:06:15.816 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:15 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:15.816 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:15 smithi043 ceph-mon[42099]: 5.4 scrub starts 2024-03-30T14:06:15.817 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:15 smithi043 ceph-mon[42099]: 5.4 scrub ok 2024-03-30T14:06:16.390 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:06:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:06:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:06:16.861 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:16 smithi043 ceph-mon[29963]: 3.4 scrub starts 2024-03-30T14:06:16.861 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:16 smithi043 ceph-mon[29963]: 3.4 scrub ok 2024-03-30T14:06:16.861 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:16 smithi043 ceph-mon[29963]: 4.16 scrub starts 2024-03-30T14:06:16.861 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:16 smithi043 ceph-mon[29963]: 4.16 scrub ok 2024-03-30T14:06:16.861 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:16 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:06:16.861 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:16 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:06:16.862 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:16 smithi043 ceph-mon[29963]: pgmap v1127: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T14:06:16.862 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:16 smithi043 ceph-mon[42099]: 3.4 scrub starts 2024-03-30T14:06:16.862 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:16 smithi043 ceph-mon[42099]: 3.4 scrub ok 2024-03-30T14:06:16.862 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:16 smithi043 ceph-mon[42099]: 4.16 scrub starts 2024-03-30T14:06:16.862 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:16 smithi043 ceph-mon[42099]: 4.16 scrub ok 2024-03-30T14:06:16.862 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:16 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:06:16.862 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:16 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:06:16.862 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:16 smithi043 ceph-mon[42099]: pgmap v1127: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T14:06:16.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:16 smithi113 ceph-mon[34341]: 3.4 scrub starts 2024-03-30T14:06:16.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:16 smithi113 ceph-mon[34341]: 3.4 scrub ok 2024-03-30T14:06:16.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:16 smithi113 ceph-mon[34341]: 4.16 scrub starts 2024-03-30T14:06:16.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:16 smithi113 ceph-mon[34341]: 4.16 scrub ok 2024-03-30T14:06:16.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:16 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:06:16.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:16 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:06:16.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:16 smithi113 ceph-mon[34341]: pgmap v1127: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T14:06:17.423 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T14:06:17.423 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":522,"fsid":"7ee932fa-ee9a-11ee-b647-cb9ed24678a4","created":"2024-03-30T13:37:07.346020+0000","modified":"2024-03-30T14:06:16.465395+0000","last_up_change":"2024-03-30T14:01:27.027194+0000","last_in_change":"2024-03-30T14:05:09.343313+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T13:40:27.658979+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"491","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"368","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8999999761581421,"score_stable":1.8999999761581421,"optimal_score":1,"raw_score_acting":1.8999999761581421,"raw_score_stable":1.8999999761581421,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-30T13:42:42.989975+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":68,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-30T13:42:43.910306+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-30T13:42:45.633132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-30T13:42:47.570084+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-30T13:42:49.637189+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94ea083e-6a27-41db-83af-2f7d3cae1956","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":510,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":1144592572},{"type":"v1","addr":"172.21.15.43:6803","nonce":1144592572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":1144592572},{"type":"v1","addr":"172.21.15.43:6805","nonce":1144592572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":1144592572},{"type":"v1","addr":"172.21.15.43:6809","nonce":1144592572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":1144592572},{"type":"v1","addr":"172.21.15.43:6807","nonce":1144592572}]},"public_addr":"172.21.15.43:6803/1144592572","cluster_addr":"172.21.15.43:6805/1144592572","heartbeat_back_addr":"172.21.15.43:6809/1144592572","heartbeat_front_addr":"172.21.15.43:6807/1144592572","state":["exists","up"]},{"osd":1,"uuid":"8e1c61dc-9ff4-4dbe-aeaa-cbfee0c8804a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":510,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":3901165928},{"type":"v1","addr":"172.21.15.43:6811","nonce":3901165928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":3901165928},{"type":"v1","addr":"172.21.15.43:6813","nonce":3901165928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":3901165928},{"type":"v1","addr":"172.21.15.43:6817","nonce":3901165928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":3901165928},{"type":"v1","addr":"172.21.15.43:6815","nonce":3901165928}]},"public_addr":"172.21.15.43:6811/3901165928","cluster_addr":"172.21.15.43:6813/3901165928","heartbeat_back_addr":"172.21.15.43:6817/3901165928","heartbeat_front_addr":"172.21.15.43:6815/3901165928","state":["exists","up"]},{"osd":2,"uuid":"6db6782e-5889-443e-bc6c-df47a1f93bc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":166,"up_from":388,"up_thru":520,"down_at":167,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":886354154},{"type":"v1","addr":"172.21.15.43:6819","nonce":886354154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":886354154},{"type":"v1","addr":"172.21.15.43:6821","nonce":886354154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":886354154},{"type":"v1","addr":"172.21.15.43:6825","nonce":886354154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":886354154},{"type":"v1","addr":"172.21.15.43:6823","nonce":886354154}]},"public_addr":"172.21.15.43:6819/886354154","cluster_addr":"172.21.15.43:6821/886354154","heartbeat_back_addr":"172.21.15.43:6825/886354154","heartbeat_front_addr":"172.21.15.43:6823/886354154","state":["exists","up"]},{"osd":3,"uuid":"7048ff46-2cba-4d18-9835-95ecbddb2bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":510,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2933589376},{"type":"v1","addr":"172.21.15.43:6827","nonce":2933589376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2933589376},{"type":"v1","addr":"172.21.15.43:6829","nonce":2933589376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2933589376},{"type":"v1","addr":"172.21.15.43:6833","nonce":2933589376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2933589376},{"type":"v1","addr":"172.21.15.43:6831","nonce":2933589376}]},"public_addr":"172.21.15.43:6827/2933589376","cluster_addr":"172.21.15.43:6829/2933589376","heartbeat_back_addr":"172.21.15.43:6833/2933589376","heartbeat_front_addr":"172.21.15.43:6831/2933589376","state":["exists","up"]},{"osd":4,"uuid":"42da30e6-9eec-4ed5-8c73-9ce78aa8ff63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1945833081},{"type":"v1","addr":"172.21.15.113:6801","nonce":1945833081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1945833081},{"type":"v1","addr":"172.21.15.113:6803","nonce":1945833081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1945833081},{"type":"v1","addr":"172.21.15.113:6807","nonce":1945833081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1945833081},{"type":"v1","addr":"172.21.15.113:6805","nonce":1945833081}]},"public_addr":"172.21.15.113:6801/1945833081","cluster_addr":"172.21.15.113:6803/1945833081","heartbeat_back_addr":"172.21.15.113:6807/1945833081","heartbeat_front_addr":"172.21.15.113:6805/1945833081","state":["exists","up"]},{"osd":5,"uuid":"99a11b81-0df7-4b1d-946a-4297ba62e263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":510,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3426360325},{"type":"v1","addr":"172.21.15.113:6809","nonce":3426360325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3426360325},{"type":"v1","addr":"172.21.15.113:6811","nonce":3426360325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3426360325},{"type":"v1","addr":"172.21.15.113:6815","nonce":3426360325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3426360325},{"type":"v1","addr":"172.21.15.113:6813","nonce":3426360325}]},"public_addr":"172.21.15.113:6809/3426360325","cluster_addr":"172.21.15.113:6811/3426360325","heartbeat_back_addr":"172.21.15.113:6815/3426360325","heartbeat_front_addr":"172.21.15.113:6813/3426360325","state":["exists","up"]},{"osd":6,"uuid":"cafd0e26-78e9-4164-a43a-96ca8382fefe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":507,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1527017698},{"type":"v1","addr":"172.21.15.113:6817","nonce":1527017698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1527017698},{"type":"v1","addr":"172.21.15.113:6819","nonce":1527017698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1527017698},{"type":"v1","addr":"172.21.15.113:6823","nonce":1527017698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1527017698},{"type":"v1","addr":"172.21.15.113:6821","nonce":1527017698}]},"public_addr":"172.21.15.113:6817/1527017698","cluster_addr":"172.21.15.113:6819/1527017698","heartbeat_back_addr":"172.21.15.113:6823/1527017698","heartbeat_front_addr":"172.21.15.113:6821/1527017698","state":["exists","up"]},{"osd":7,"uuid":"e9d59662-9132-48f4-9e95-842a3cfbd79f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":507,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":3182476015},{"type":"v1","addr":"172.21.15.113:6825","nonce":3182476015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":3182476015},{"type":"v1","addr":"172.21.15.113:6827","nonce":3182476015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":3182476015},{"type":"v1","addr":"172.21.15.113:6831","nonce":3182476015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":3182476015},{"type":"v1","addr":"172.21.15.113:6829","nonce":3182476015}]},"public_addr":"172.21.15.113:6825/3182476015","cluster_addr":"172.21.15.113:6827/3182476015","heartbeat_back_addr":"172.21.15.113:6831/3182476015","heartbeat_front_addr":"172.21.15.113:6829/3182476015","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:06:10.749935+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:06:15.498692+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:05:18.173772+0000","dead_epoch":166},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:05:04.789933+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:06:06.562109+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:06:06.080176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:05:03.449335+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:06:02.971149+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"1.2","mappings":[{"from":0,"to":1}]},{"pgid":"1.d","mappings":[{"from":4,"to":0}]},{"pgid":"3.1d","mappings":[{"from":4,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.5","mappings":[{"from":4,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":3,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":0}]},{"pgid":"4.1a","mappings":[{"from":3,"to":7}]},{"pgid":"4.1b","mappings":[{"from":1,"to":0}]},{"pgid":"5.15","mappings":[{"from":0,"to":6}]},{"pgid":"5.17","mappings":[{"from":1,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/538304234":"2024-03-31T13:43:27.253365+0000","172.21.15.43:6801/3411723337":"2024-03-31T13:37:41.733515+0000","172.21.15.43:0/3656823679":"2024-03-31T13:37:41.733515+0000","172.21.15.43:0/2618817647":"2024-03-31T13:38:09.321740+0000","172.21.15.43:6800/3411723337":"2024-03-31T13:37:41.733515+0000","172.21.15.43:0/1206218784":"2024-03-31T13:37:41.733515+0000","172.21.15.43:0/886631486":"2024-03-31T13:37:41.733515+0000","172.21.15.43:6800/1339921781":"2024-03-31T13:38:09.321740+0000","172.21.15.43:6801/1339921781":"2024-03-31T13:38:09.321740+0000","172.21.15.43:0/2794494528":"2024-03-31T13:38:09.321740+0000","172.21.15.43:0/564868778":"2024-03-31T13:38:09.321740+0000","172.21.15.43:6800/3033707884":"2024-03-31T13:43:27.253365+0000","172.21.15.43:0/3539398453":"2024-03-31T13:43:27.253365+0000","172.21.15.43:0/2414364740":"2024-03-31T13:43:27.253365+0000","172.21.15.43:6801/3033707884":"2024-03-31T13:43:27.253365+0000","172.21.15.43:0/3923597429":"2024-03-31T13:43:27.253365+0000","172.21.15.43:0/2524269526":"2024-03-31T13:43:27.253365+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T14:06:17.803 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:17 smithi043 ceph-mon[29963]: 5.b scrub starts 2024-03-30T14:06:17.803 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:17 smithi043 ceph-mon[29963]: 5.b scrub ok 2024-03-30T14:06:17.803 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:17 smithi043 ceph-mon[29963]: 3.1 scrub starts 2024-03-30T14:06:17.803 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:17 smithi043 ceph-mon[29963]: 3.1 scrub ok 2024-03-30T14:06:17.803 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:17 smithi043 ceph-mon[29963]: osdmap e522: 8 total, 8 up, 8 in 2024-03-30T14:06:17.803 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:17 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2340563592' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:06:17.803 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:17 smithi043 ceph-mon[42099]: 5.b scrub starts 2024-03-30T14:06:17.804 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:17 smithi043 ceph-mon[42099]: 5.b scrub ok 2024-03-30T14:06:17.804 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:17 smithi043 ceph-mon[42099]: 3.1 scrub starts 2024-03-30T14:06:17.804 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:17 smithi043 ceph-mon[42099]: 3.1 scrub ok 2024-03-30T14:06:17.804 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:17 smithi043 ceph-mon[42099]: osdmap e522: 8 total, 8 up, 8 in 2024-03-30T14:06:17.804 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:17 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2340563592' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:06:17.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:17 smithi113 ceph-mon[34341]: 5.b scrub starts 2024-03-30T14:06:17.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:17 smithi113 ceph-mon[34341]: 5.b scrub ok 2024-03-30T14:06:17.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:17 smithi113 ceph-mon[34341]: 3.1 scrub starts 2024-03-30T14:06:17.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:17 smithi113 ceph-mon[34341]: 3.1 scrub ok 2024-03-30T14:06:17.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:17 smithi113 ceph-mon[34341]: osdmap e522: 8 total, 8 up, 8 in 2024-03-30T14:06:17.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:17 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2340563592' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:06:18.230 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-03-30T14:06:18.231 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-03-30T14:06:18.231 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-03-30T14:06:18.231 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-03-30T14:06:18.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-03-30T14:06:18.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-03-30T14:06:18.233 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-03-30T14:06:18.233 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-03-30T14:06:18.746 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:18.748 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:18.749 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:18.750 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:18.752 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:18.752 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:18.753 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:18.773 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:18.890 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:18 smithi043 ceph-mon[29963]: 1.c deep-scrub starts 2024-03-30T14:06:18.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:18 smithi043 ceph-mon[29963]: 1.c deep-scrub ok 2024-03-30T14:06:18.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:18 smithi043 ceph-mon[29963]: pgmap v1129: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 798 B/s rd, 0 op/s 2024-03-30T14:06:18.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:18 smithi043 ceph-mon[42099]: 1.c deep-scrub starts 2024-03-30T14:06:18.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:18 smithi043 ceph-mon[42099]: 1.c deep-scrub ok 2024-03-30T14:06:18.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:18 smithi043 ceph-mon[42099]: pgmap v1129: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 798 B/s rd, 0 op/s 2024-03-30T14:06:18.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:18 smithi113 ceph-mon[34341]: 1.c deep-scrub starts 2024-03-30T14:06:18.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:18 smithi113 ceph-mon[34341]: 1.c deep-scrub ok 2024-03-30T14:06:18.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:18 smithi113 ceph-mon[34341]: pgmap v1129: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 798 B/s rd, 0 op/s 2024-03-30T14:06:19.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:19 smithi043 ceph-mon[29963]: 6.f scrub starts 2024-03-30T14:06:19.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:19 smithi043 ceph-mon[29963]: 6.f scrub ok 2024-03-30T14:06:19.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:19 smithi043 ceph-mon[29963]: 3.1a scrub starts 2024-03-30T14:06:19.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:19 smithi043 ceph-mon[29963]: 3.1a scrub ok 2024-03-30T14:06:19.830 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:19 smithi043 ceph-mon[42099]: 6.f scrub starts 2024-03-30T14:06:19.830 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:19 smithi043 ceph-mon[42099]: 6.f scrub ok 2024-03-30T14:06:19.830 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:19 smithi043 ceph-mon[42099]: 3.1a scrub starts 2024-03-30T14:06:19.831 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:19 smithi043 ceph-mon[42099]: 3.1a scrub ok 2024-03-30T14:06:19.865 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:19 smithi113 ceph-mon[34341]: 6.f scrub starts 2024-03-30T14:06:19.865 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:19 smithi113 ceph-mon[34341]: 6.f scrub ok 2024-03-30T14:06:19.865 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:19 smithi113 ceph-mon[34341]: 3.1a scrub starts 2024-03-30T14:06:19.865 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:19 smithi113 ceph-mon[34341]: 3.1a scrub ok 2024-03-30T14:06:19.865 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:06:19 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:06:20.824 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:20 smithi043 ceph-mon[29963]: 1.10 scrub starts 2024-03-30T14:06:20.824 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:20 smithi043 ceph-mon[29963]: 1.10 scrub ok 2024-03-30T14:06:20.825 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:20 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:20.825 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:20 smithi043 ceph-mon[29963]: pgmap v1130: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:06:20.825 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:20 smithi043 ceph-mon[42099]: 1.10 scrub starts 2024-03-30T14:06:20.825 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:20 smithi043 ceph-mon[42099]: 1.10 scrub ok 2024-03-30T14:06:20.825 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:20 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:20.825 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:20 smithi043 ceph-mon[42099]: pgmap v1130: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:06:20.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:20 smithi113 ceph-mon[34341]: 1.10 scrub starts 2024-03-30T14:06:20.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:20 smithi113 ceph-mon[34341]: 1.10 scrub ok 2024-03-30T14:06:20.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:20 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:20.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:20 smithi113 ceph-mon[34341]: pgmap v1130: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:06:21.784 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:21 smithi043 ceph-mon[29963]: 1.14 scrub starts 2024-03-30T14:06:21.784 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:21 smithi043 ceph-mon[29963]: 1.14 scrub ok 2024-03-30T14:06:21.784 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:21 smithi043 ceph-mon[42099]: 1.14 scrub starts 2024-03-30T14:06:21.785 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:21 smithi043 ceph-mon[42099]: 1.14 scrub ok 2024-03-30T14:06:21.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:21 smithi113 ceph-mon[34341]: 1.14 scrub starts 2024-03-30T14:06:21.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:21 smithi113 ceph-mon[34341]: 1.14 scrub ok 2024-03-30T14:06:22.785 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:22 smithi043 ceph-mon[42099]: pgmap v1131: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:06:22.785 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:22 smithi043 ceph-mon[29963]: pgmap v1131: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:06:22.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:22 smithi113 ceph-mon[34341]: pgmap v1131: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:06:23.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:23 smithi043 ceph-mon[29963]: 1.a scrub starts 2024-03-30T14:06:23.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:23 smithi043 ceph-mon[29963]: 1.a scrub ok 2024-03-30T14:06:23.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:23 smithi043 ceph-mon[29963]: 4.b scrub starts 2024-03-30T14:06:23.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:23 smithi043 ceph-mon[29963]: 4.b scrub ok 2024-03-30T14:06:23.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:23 smithi043 ceph-mon[29963]: 1.0 deep-scrub starts 2024-03-30T14:06:23.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:23 smithi043 ceph-mon[29963]: 1.0 deep-scrub ok 2024-03-30T14:06:23.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:23 smithi043 ceph-mon[42099]: 1.a scrub starts 2024-03-30T14:06:23.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:23 smithi043 ceph-mon[42099]: 1.a scrub ok 2024-03-30T14:06:23.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:23 smithi043 ceph-mon[42099]: 4.b scrub starts 2024-03-30T14:06:23.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:23 smithi043 ceph-mon[42099]: 4.b scrub ok 2024-03-30T14:06:23.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:23 smithi043 ceph-mon[42099]: 1.0 deep-scrub starts 2024-03-30T14:06:23.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:23 smithi043 ceph-mon[42099]: 1.0 deep-scrub ok 2024-03-30T14:06:23.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:23 smithi113 ceph-mon[34341]: 1.a scrub starts 2024-03-30T14:06:23.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:23 smithi113 ceph-mon[34341]: 1.a scrub ok 2024-03-30T14:06:23.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:23 smithi113 ceph-mon[34341]: 4.b scrub starts 2024-03-30T14:06:23.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:23 smithi113 ceph-mon[34341]: 4.b scrub ok 2024-03-30T14:06:23.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:23 smithi113 ceph-mon[34341]: 1.0 deep-scrub starts 2024-03-30T14:06:23.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:23 smithi113 ceph-mon[34341]: 1.0 deep-scrub ok 2024-03-30T14:06:24.742 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:24 smithi043 ceph-mon[29963]: 3.b scrub starts 2024-03-30T14:06:24.742 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:24 smithi043 ceph-mon[29963]: 3.b scrub ok 2024-03-30T14:06:24.742 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:24 smithi043 ceph-mon[29963]: 6.14 scrub starts 2024-03-30T14:06:24.742 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:24 smithi043 ceph-mon[29963]: 6.14 scrub ok 2024-03-30T14:06:24.742 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:24 smithi043 ceph-mon[29963]: pgmap v1132: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:06:24.742 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:24 smithi043 ceph-mon[42099]: 3.b scrub starts 2024-03-30T14:06:24.743 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:24 smithi043 ceph-mon[42099]: 3.b scrub ok 2024-03-30T14:06:24.743 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:24 smithi043 ceph-mon[42099]: 6.14 scrub starts 2024-03-30T14:06:24.743 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:24 smithi043 ceph-mon[42099]: 6.14 scrub ok 2024-03-30T14:06:24.743 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:24 smithi043 ceph-mon[42099]: pgmap v1132: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:06:24.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:24 smithi113 ceph-mon[34341]: 3.b scrub starts 2024-03-30T14:06:24.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:24 smithi113 ceph-mon[34341]: 3.b scrub ok 2024-03-30T14:06:24.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:24 smithi113 ceph-mon[34341]: 6.14 scrub starts 2024-03-30T14:06:24.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:24 smithi113 ceph-mon[34341]: 6.14 scrub ok 2024-03-30T14:06:24.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:24 smithi113 ceph-mon[34341]: pgmap v1132: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:06:26.390 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:06:26 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:06:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:06:26.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:26 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:06:26.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:26 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:06:26.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:26 smithi043 ceph-mon[29963]: 3.8 deep-scrub starts 2024-03-30T14:06:26.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:26 smithi043 ceph-mon[29963]: 3.8 deep-scrub ok 2024-03-30T14:06:26.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:26 smithi043 ceph-mon[29963]: osdmap e523: 8 total, 8 up, 8 in 2024-03-30T14:06:26.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:26 smithi043 ceph-mon[29963]: pgmap v1134: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:06:26.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:26 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:06:26.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:26 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:06:26.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:26 smithi043 ceph-mon[42099]: 3.8 deep-scrub starts 2024-03-30T14:06:26.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:26 smithi043 ceph-mon[42099]: 3.8 deep-scrub ok 2024-03-30T14:06:26.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:26 smithi043 ceph-mon[42099]: osdmap e523: 8 total, 8 up, 8 in 2024-03-30T14:06:26.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:26 smithi043 ceph-mon[42099]: pgmap v1134: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:06:26.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:26 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:06:26.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:26 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:06:26.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:26 smithi113 ceph-mon[34341]: 3.8 deep-scrub starts 2024-03-30T14:06:26.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:26 smithi113 ceph-mon[34341]: 3.8 deep-scrub ok 2024-03-30T14:06:26.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:26 smithi113 ceph-mon[34341]: osdmap e523: 8 total, 8 up, 8 in 2024-03-30T14:06:26.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:26 smithi113 ceph-mon[34341]: pgmap v1134: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:06:27.725 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:27 smithi043 ceph-mon[29963]: 5.11 deep-scrub starts 2024-03-30T14:06:27.725 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:27 smithi043 ceph-mon[29963]: 5.11 deep-scrub ok 2024-03-30T14:06:27.726 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:27 smithi043 ceph-mon[42099]: 5.11 deep-scrub starts 2024-03-30T14:06:27.726 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:27 smithi043 ceph-mon[42099]: 5.11 deep-scrub ok 2024-03-30T14:06:27.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:27 smithi113 ceph-mon[34341]: 5.11 deep-scrub starts 2024-03-30T14:06:27.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:27 smithi113 ceph-mon[34341]: 5.11 deep-scrub ok 2024-03-30T14:06:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:28 smithi043 ceph-mon[29963]: 2.2 scrub starts 2024-03-30T14:06:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:28 smithi043 ceph-mon[29963]: 2.2 scrub ok 2024-03-30T14:06:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:28 smithi043 ceph-mon[29963]: 4.1f scrub starts 2024-03-30T14:06:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:28 smithi043 ceph-mon[29963]: 4.1f scrub ok 2024-03-30T14:06:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:28 smithi043 ceph-mon[29963]: 3.15 deep-scrub starts 2024-03-30T14:06:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:28 smithi043 ceph-mon[29963]: 3.15 deep-scrub ok 2024-03-30T14:06:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:28 smithi043 ceph-mon[29963]: pgmap v1135: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:06:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:28 smithi043 ceph-mon[42099]: 2.2 scrub starts 2024-03-30T14:06:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:28 smithi043 ceph-mon[42099]: 2.2 scrub ok 2024-03-30T14:06:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:28 smithi043 ceph-mon[42099]: 4.1f scrub starts 2024-03-30T14:06:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:28 smithi043 ceph-mon[42099]: 4.1f scrub ok 2024-03-30T14:06:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:28 smithi043 ceph-mon[42099]: 3.15 deep-scrub starts 2024-03-30T14:06:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:28 smithi043 ceph-mon[42099]: 3.15 deep-scrub ok 2024-03-30T14:06:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:28 smithi043 ceph-mon[42099]: pgmap v1135: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:06:28.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:28 smithi113 ceph-mon[34341]: 2.2 scrub starts 2024-03-30T14:06:28.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:28 smithi113 ceph-mon[34341]: 2.2 scrub ok 2024-03-30T14:06:28.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:28 smithi113 ceph-mon[34341]: 4.1f scrub starts 2024-03-30T14:06:28.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:28 smithi113 ceph-mon[34341]: 4.1f scrub ok 2024-03-30T14:06:28.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:28 smithi113 ceph-mon[34341]: 3.15 deep-scrub starts 2024-03-30T14:06:28.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:28 smithi113 ceph-mon[34341]: 3.15 deep-scrub ok 2024-03-30T14:06:28.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:28 smithi113 ceph-mon[34341]: pgmap v1135: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:06:29.890 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:29 smithi043 ceph-mon[42099]: 5.12 scrub starts 2024-03-30T14:06:29.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:29 smithi043 ceph-mon[42099]: 5.12 scrub ok 2024-03-30T14:06:29.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:29 smithi043 ceph-mon[29963]: 5.12 scrub starts 2024-03-30T14:06:29.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:29 smithi043 ceph-mon[29963]: 5.12 scrub ok 2024-03-30T14:06:29.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:06:29 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:06:29.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:29 smithi113 ceph-mon[34341]: 5.12 scrub starts 2024-03-30T14:06:29.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:29 smithi113 ceph-mon[34341]: 5.12 scrub ok 2024-03-30T14:06:30.000 INFO:teuthology.orchestra.run.smithi043.stdout:55834575167 2024-03-30T14:06:30.001 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-03-30T14:06:30.555 INFO:teuthology.orchestra.run.smithi043.stdout:171798692139 2024-03-30T14:06:30.555 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-03-30T14:06:30.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:30 smithi043 ceph-mon[29963]: 4.14 scrub starts 2024-03-30T14:06:30.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:30 smithi043 ceph-mon[29963]: 4.14 scrub ok 2024-03-30T14:06:30.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:30 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:30.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:30 smithi043 ceph-mon[29963]: 6.5 scrub starts 2024-03-30T14:06:30.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:30 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:30.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:30 smithi043 ceph-mon[29963]: 6.5 scrub ok 2024-03-30T14:06:30.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:30 smithi043 ceph-mon[29963]: pgmap v1136: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:06:30.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:30 smithi043 ceph-mon[29963]: 4.1d deep-scrub starts 2024-03-30T14:06:30.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:30 smithi043 ceph-mon[29963]: 4.1d deep-scrub ok 2024-03-30T14:06:30.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:30 smithi043 ceph-mon[42099]: 4.14 scrub starts 2024-03-30T14:06:30.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:30 smithi043 ceph-mon[42099]: 4.14 scrub ok 2024-03-30T14:06:30.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:30 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:30.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:30 smithi043 ceph-mon[42099]: 6.5 scrub starts 2024-03-30T14:06:30.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:30 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:30.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:30 smithi043 ceph-mon[42099]: 6.5 scrub ok 2024-03-30T14:06:30.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:30 smithi043 ceph-mon[42099]: pgmap v1136: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:06:30.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:30 smithi043 ceph-mon[42099]: 4.1d deep-scrub starts 2024-03-30T14:06:30.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:30 smithi043 ceph-mon[42099]: 4.1d deep-scrub ok 2024-03-30T14:06:30.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:30 smithi113 ceph-mon[34341]: 4.14 scrub starts 2024-03-30T14:06:30.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:30 smithi113 ceph-mon[34341]: 4.14 scrub ok 2024-03-30T14:06:30.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:30 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:30.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:30 smithi113 ceph-mon[34341]: 6.5 scrub starts 2024-03-30T14:06:30.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:30 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:30.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:30 smithi113 ceph-mon[34341]: 6.5 scrub ok 2024-03-30T14:06:30.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:30 smithi113 ceph-mon[34341]: pgmap v1136: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:06:30.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:30 smithi113 ceph-mon[34341]: 4.1d deep-scrub starts 2024-03-30T14:06:30.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:30 smithi113 ceph-mon[34341]: 4.1d deep-scrub ok 2024-03-30T14:06:31.890 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:31 smithi043 ceph-mon[29963]: 4.11 scrub starts 2024-03-30T14:06:31.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:31 smithi043 ceph-mon[29963]: 4.11 scrub ok 2024-03-30T14:06:31.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:31 smithi043 ceph-mon[42099]: 4.11 scrub starts 2024-03-30T14:06:31.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:31 smithi043 ceph-mon[42099]: 4.11 scrub ok 2024-03-30T14:06:31.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:31 smithi113 ceph-mon[34341]: 4.11 scrub starts 2024-03-30T14:06:31.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:31 smithi113 ceph-mon[34341]: 4.11 scrub ok 2024-03-30T14:06:32.139 INFO:teuthology.orchestra.run.smithi043.stdout:107374182710 2024-03-30T14:06:32.139 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-03-30T14:06:32.535 INFO:teuthology.orchestra.run.smithi043.stdout:249108103459 2024-03-30T14:06:32.535 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-03-30T14:06:32.778 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[29963]: 4.4 scrub starts 2024-03-30T14:06:32.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[29963]: 4.4 scrub ok 2024-03-30T14:06:32.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:06:32.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:06:32.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[29963]: 4.f scrub starts 2024-03-30T14:06:32.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[29963]: 4.f scrub ok 2024-03-30T14:06:32.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[29963]: osdmap e524: 8 total, 8 up, 8 in 2024-03-30T14:06:32.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[29963]: pgmap v1138: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:06:32.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[29963]: purged_snaps scrub starts 2024-03-30T14:06:32.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[29963]: purged_snaps scrub ok 2024-03-30T14:06:32.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[29963]: 1.4 scrub starts 2024-03-30T14:06:32.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[29963]: 1.4 scrub ok 2024-03-30T14:06:32.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[42099]: 4.4 scrub starts 2024-03-30T14:06:32.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[42099]: 4.4 scrub ok 2024-03-30T14:06:32.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:06:32.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:06:32.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[42099]: 4.f scrub starts 2024-03-30T14:06:32.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[42099]: 4.f scrub ok 2024-03-30T14:06:32.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[42099]: osdmap e524: 8 total, 8 up, 8 in 2024-03-30T14:06:32.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[42099]: pgmap v1138: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:06:32.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[42099]: purged_snaps scrub starts 2024-03-30T14:06:32.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[42099]: purged_snaps scrub ok 2024-03-30T14:06:32.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[42099]: 1.4 scrub starts 2024-03-30T14:06:32.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:32 smithi043 ceph-mon[42099]: 1.4 scrub ok 2024-03-30T14:06:32.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:32 smithi113 ceph-mon[34341]: 4.4 scrub starts 2024-03-30T14:06:32.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:32 smithi113 ceph-mon[34341]: 4.4 scrub ok 2024-03-30T14:06:32.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:32 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:06:32.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:32 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:06:32.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:32 smithi113 ceph-mon[34341]: 4.f scrub starts 2024-03-30T14:06:32.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:32 smithi113 ceph-mon[34341]: 4.f scrub ok 2024-03-30T14:06:32.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:32 smithi113 ceph-mon[34341]: osdmap e524: 8 total, 8 up, 8 in 2024-03-30T14:06:32.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:32 smithi113 ceph-mon[34341]: pgmap v1138: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:06:32.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:32 smithi113 ceph-mon[34341]: purged_snaps scrub starts 2024-03-30T14:06:32.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:32 smithi113 ceph-mon[34341]: purged_snaps scrub ok 2024-03-30T14:06:32.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:32 smithi113 ceph-mon[34341]: 1.4 scrub starts 2024-03-30T14:06:32.965 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:32 smithi113 ceph-mon[34341]: 1.4 scrub ok 2024-03-30T14:06:33.888 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:33 smithi043 ceph-mon[29963]: 3.16 scrub starts 2024-03-30T14:06:33.888 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:33 smithi043 ceph-mon[29963]: 3.16 scrub ok 2024-03-30T14:06:33.888 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:33 smithi043 ceph-mon[29963]: 6.e scrub starts 2024-03-30T14:06:33.888 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:33 smithi043 ceph-mon[29963]: 6.e scrub ok 2024-03-30T14:06:33.888 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:33 smithi043 ceph-mon[29963]: osdmap e525: 8 total, 8 up, 8 in 2024-03-30T14:06:33.888 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:33 smithi043 ceph-mon[42099]: 3.16 scrub starts 2024-03-30T14:06:33.888 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:33 smithi043 ceph-mon[42099]: 3.16 scrub ok 2024-03-30T14:06:33.888 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:33 smithi043 ceph-mon[42099]: 6.e scrub starts 2024-03-30T14:06:33.888 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:33 smithi043 ceph-mon[42099]: 6.e scrub ok 2024-03-30T14:06:33.888 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:33 smithi043 ceph-mon[42099]: osdmap e525: 8 total, 8 up, 8 in 2024-03-30T14:06:33.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:33 smithi113 ceph-mon[34341]: 3.16 scrub starts 2024-03-30T14:06:33.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:33 smithi113 ceph-mon[34341]: 3.16 scrub ok 2024-03-30T14:06:33.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:33 smithi113 ceph-mon[34341]: 6.e scrub starts 2024-03-30T14:06:33.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:33 smithi113 ceph-mon[34341]: 6.e scrub ok 2024-03-30T14:06:33.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:33 smithi113 ceph-mon[34341]: osdmap e525: 8 total, 8 up, 8 in 2024-03-30T14:06:34.205 INFO:teuthology.orchestra.run.smithi043.stdout:137438953777 2024-03-30T14:06:34.206 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-03-30T14:06:34.382 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:34.844 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:34 smithi043 ceph-mon[29963]: 5.a deep-scrub starts 2024-03-30T14:06:34.844 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:34 smithi043 ceph-mon[29963]: 5.a deep-scrub ok 2024-03-30T14:06:34.845 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:34 smithi043 ceph-mon[29963]: 5.1c deep-scrub starts 2024-03-30T14:06:34.845 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:34 smithi043 ceph-mon[29963]: 5.1c deep-scrub ok 2024-03-30T14:06:34.845 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:34 smithi043 ceph-mon[29963]: 4.2 deep-scrub starts 2024-03-30T14:06:34.845 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:34 smithi043 ceph-mon[29963]: 4.2 deep-scrub ok 2024-03-30T14:06:34.845 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:34 smithi043 ceph-mon[29963]: pgmap v1140: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:06:34.845 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:34 smithi043 ceph-mon[42099]: 5.a deep-scrub starts 2024-03-30T14:06:34.845 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:34 smithi043 ceph-mon[42099]: 5.a deep-scrub ok 2024-03-30T14:06:34.845 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:34 smithi043 ceph-mon[42099]: 5.1c deep-scrub starts 2024-03-30T14:06:34.845 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:34 smithi043 ceph-mon[42099]: 5.1c deep-scrub ok 2024-03-30T14:06:34.845 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:34 smithi043 ceph-mon[42099]: 4.2 deep-scrub starts 2024-03-30T14:06:34.845 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:34 smithi043 ceph-mon[42099]: 4.2 deep-scrub ok 2024-03-30T14:06:34.845 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:34 smithi043 ceph-mon[42099]: pgmap v1140: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:06:34.878 INFO:teuthology.orchestra.run.smithi043.stdout:1666447310911 2024-03-30T14:06:34.879 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-03-30T14:06:34.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:34 smithi113 ceph-mon[34341]: 5.a deep-scrub starts 2024-03-30T14:06:34.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:34 smithi113 ceph-mon[34341]: 5.a deep-scrub ok 2024-03-30T14:06:34.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:34 smithi113 ceph-mon[34341]: 5.1c deep-scrub starts 2024-03-30T14:06:34.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:34 smithi113 ceph-mon[34341]: 5.1c deep-scrub ok 2024-03-30T14:06:34.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:34 smithi113 ceph-mon[34341]: 4.2 deep-scrub starts 2024-03-30T14:06:34.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:34 smithi113 ceph-mon[34341]: 4.2 deep-scrub ok 2024-03-30T14:06:34.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:34 smithi113 ceph-mon[34341]: pgmap v1140: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:06:35.838 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:35 smithi043 ceph-mon[42099]: 5.1f scrub starts 2024-03-30T14:06:35.838 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:35 smithi043 ceph-mon[42099]: 5.1f scrub ok 2024-03-30T14:06:35.838 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:35 smithi043 ceph-mon[42099]: 3.19 scrub starts 2024-03-30T14:06:35.838 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:35 smithi043 ceph-mon[42099]: 3.19 scrub ok 2024-03-30T14:06:35.839 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:35 smithi043 ceph-mon[29963]: 5.1f scrub starts 2024-03-30T14:06:35.839 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:35 smithi043 ceph-mon[29963]: 5.1f scrub ok 2024-03-30T14:06:35.839 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:35 smithi043 ceph-mon[29963]: 3.19 scrub starts 2024-03-30T14:06:35.839 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:35 smithi043 ceph-mon[29963]: 3.19 scrub ok 2024-03-30T14:06:35.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:35 smithi113 ceph-mon[34341]: 5.1f scrub starts 2024-03-30T14:06:35.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:35 smithi113 ceph-mon[34341]: 5.1f scrub ok 2024-03-30T14:06:35.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:35 smithi113 ceph-mon[34341]: 3.19 scrub starts 2024-03-30T14:06:35.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:35 smithi113 ceph-mon[34341]: 3.19 scrub ok 2024-03-30T14:06:36.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:06:36 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:06:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:06:36.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:36 smithi043 ceph-mon[29963]: pgmap v1141: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:06:36.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:36 smithi043 ceph-mon[42099]: pgmap v1141: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:06:36.963 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:36 smithi113 ceph-mon[34341]: pgmap v1141: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:06:37.080 INFO:teuthology.orchestra.run.smithi043.stdout:34359738695 2024-03-30T14:06:37.080 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-03-30T14:06:37.533 INFO:teuthology.orchestra.run.smithi043.stdout:210453397799 2024-03-30T14:06:37.533 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-03-30T14:06:37.604 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:37.605 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:37.607 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:37.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:37 smithi043 ceph-mon[29963]: 1.7 scrub starts 2024-03-30T14:06:37.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:37 smithi043 ceph-mon[29963]: 1.7 scrub ok 2024-03-30T14:06:37.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:37 smithi043 ceph-mon[42099]: 1.7 scrub starts 2024-03-30T14:06:37.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:37 smithi043 ceph-mon[42099]: 1.7 scrub ok 2024-03-30T14:06:37.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:37 smithi113 ceph-mon[34341]: 1.7 scrub starts 2024-03-30T14:06:37.964 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:37 smithi113 ceph-mon[34341]: 1.7 scrub ok 2024-03-30T14:06:38.860 INFO:teuthology.orchestra.run.smithi043.stdout:171798692141 2024-03-30T14:06:39.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:39 smithi043 ceph-mon[29963]: 3.3 scrub starts 2024-03-30T14:06:39.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:39 smithi043 ceph-mon[29963]: 3.3 scrub ok 2024-03-30T14:06:39.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:39 smithi043 ceph-mon[29963]: 5.1e deep-scrub starts 2024-03-30T14:06:39.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:39 smithi043 ceph-mon[29963]: 5.1e deep-scrub ok 2024-03-30T14:06:39.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:39 smithi043 ceph-mon[29963]: pgmap v1142: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:06:39.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:39 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/1140458135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T14:06:39.203 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:39 smithi043 ceph-mon[42099]: 3.3 scrub starts 2024-03-30T14:06:39.203 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:39 smithi043 ceph-mon[42099]: 3.3 scrub ok 2024-03-30T14:06:39.203 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:39 smithi043 ceph-mon[42099]: 5.1e deep-scrub starts 2024-03-30T14:06:39.203 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:39 smithi043 ceph-mon[42099]: 5.1e deep-scrub ok 2024-03-30T14:06:39.203 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:39 smithi043 ceph-mon[42099]: pgmap v1142: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:06:39.203 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:39 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/1140458135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T14:06:39.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:39 smithi113 ceph-mon[34341]: 3.3 scrub starts 2024-03-30T14:06:39.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:39 smithi113 ceph-mon[34341]: 3.3 scrub ok 2024-03-30T14:06:39.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:39 smithi113 ceph-mon[34341]: 5.1e deep-scrub starts 2024-03-30T14:06:39.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:39 smithi113 ceph-mon[34341]: 5.1e deep-scrub ok 2024-03-30T14:06:39.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:39 smithi113 ceph-mon[34341]: pgmap v1142: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:06:39.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:39 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/1140458135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T14:06:39.887 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798692139 got 171798692141 for osd.5 2024-03-30T14:06:39.888 DEBUG:teuthology.parallel:result is None 2024-03-30T14:06:40.012 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:06:39 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:06:40.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:40 smithi043 ceph-mon[29963]: 6.9 deep-scrub starts 2024-03-30T14:06:40.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:40 smithi043 ceph-mon[29963]: 6.9 deep-scrub ok 2024-03-30T14:06:40.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:40 smithi043 ceph-mon[29963]: 1.1 deep-scrub starts 2024-03-30T14:06:40.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:40 smithi043 ceph-mon[29963]: 1.1 deep-scrub ok 2024-03-30T14:06:40.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:40 smithi043 ceph-mon[29963]: 4.5 scrub starts 2024-03-30T14:06:40.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:40 smithi043 ceph-mon[29963]: 4.5 scrub ok 2024-03-30T14:06:40.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:40 smithi043 ceph-mon[29963]: 3.1e scrub starts 2024-03-30T14:06:40.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:40 smithi043 ceph-mon[29963]: 3.1e scrub ok 2024-03-30T14:06:40.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:40 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:06:40.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:40 smithi043 ceph-mon[42099]: 6.9 deep-scrub starts 2024-03-30T14:06:40.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:40 smithi043 ceph-mon[42099]: 6.9 deep-scrub ok 2024-03-30T14:06:40.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:40 smithi043 ceph-mon[42099]: 1.1 deep-scrub starts 2024-03-30T14:06:40.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:40 smithi043 ceph-mon[42099]: 1.1 deep-scrub ok 2024-03-30T14:06:40.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:40 smithi043 ceph-mon[42099]: 4.5 scrub starts 2024-03-30T14:06:40.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:40 smithi043 ceph-mon[42099]: 4.5 scrub ok 2024-03-30T14:06:40.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:40 smithi043 ceph-mon[42099]: 3.1e scrub starts 2024-03-30T14:06:40.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:40 smithi043 ceph-mon[42099]: 3.1e scrub ok 2024-03-30T14:06:40.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:40 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:06:40.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:40 smithi113 ceph-mon[34341]: 6.9 deep-scrub starts 2024-03-30T14:06:40.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:40 smithi113 ceph-mon[34341]: 6.9 deep-scrub ok 2024-03-30T14:06:40.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:40 smithi113 ceph-mon[34341]: 1.1 deep-scrub starts 2024-03-30T14:06:40.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:40 smithi113 ceph-mon[34341]: 1.1 deep-scrub ok 2024-03-30T14:06:40.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:40 smithi113 ceph-mon[34341]: 4.5 scrub starts 2024-03-30T14:06:40.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:40 smithi113 ceph-mon[34341]: 4.5 scrub ok 2024-03-30T14:06:40.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:40 smithi113 ceph-mon[34341]: 3.1e scrub starts 2024-03-30T14:06:40.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:40 smithi113 ceph-mon[34341]: 3.1e scrub ok 2024-03-30T14:06:40.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:40 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:06:41.140 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:41 smithi043 ceph-mon[42099]: 5.3 scrub starts 2024-03-30T14:06:41.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:41 smithi043 ceph-mon[42099]: 5.3 scrub ok 2024-03-30T14:06:41.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:41 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:41.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:41 smithi043 ceph-mon[42099]: pgmap v1143: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:06:41.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:41 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:06:41.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:41 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:06:41.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:41 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:06:41.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:41 smithi043 ceph-mon[29963]: 5.3 scrub starts 2024-03-30T14:06:41.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:41 smithi043 ceph-mon[29963]: 5.3 scrub ok 2024-03-30T14:06:41.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:41 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:41.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:41 smithi043 ceph-mon[29963]: pgmap v1143: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:06:41.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:41 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:06:41.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:41 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:06:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:41 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:06:41.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:41 smithi113 ceph-mon[34341]: 5.3 scrub starts 2024-03-30T14:06:41.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:41 smithi113 ceph-mon[34341]: 5.3 scrub ok 2024-03-30T14:06:41.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:41 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:41.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:41 smithi113 ceph-mon[34341]: pgmap v1143: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:06:41.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:41 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:06:41.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:41 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:06:41.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:41 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:06:41.844 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:41.844 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:42.309 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:42 smithi043 ceph-mon[29963]: 4.1e scrub starts 2024-03-30T14:06:42.309 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:42 smithi043 ceph-mon[29963]: 4.1e scrub ok 2024-03-30T14:06:42.310 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:42 smithi043 ceph-mon[42099]: 4.1e scrub starts 2024-03-30T14:06:42.310 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:42 smithi043 ceph-mon[42099]: 4.1e scrub ok 2024-03-30T14:06:42.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:42 smithi113 ceph-mon[34341]: 4.1e scrub starts 2024-03-30T14:06:42.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:42 smithi113 ceph-mon[34341]: 4.1e scrub ok 2024-03-30T14:06:43.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:43 smithi043 ceph-mon[29963]: 6.17 scrub starts 2024-03-30T14:06:43.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:43 smithi043 ceph-mon[29963]: 5.c scrub starts 2024-03-30T14:06:43.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:43 smithi043 ceph-mon[29963]: 5.c scrub ok 2024-03-30T14:06:43.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:43 smithi043 ceph-mon[29963]: 6.17 scrub ok 2024-03-30T14:06:43.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:43 smithi043 ceph-mon[29963]: pgmap v1144: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:06:43.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:43 smithi043 ceph-mon[42099]: 6.17 scrub starts 2024-03-30T14:06:43.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:43 smithi043 ceph-mon[42099]: 5.c scrub starts 2024-03-30T14:06:43.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:43 smithi043 ceph-mon[42099]: 5.c scrub ok 2024-03-30T14:06:43.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:43 smithi043 ceph-mon[42099]: 6.17 scrub ok 2024-03-30T14:06:43.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:43 smithi043 ceph-mon[42099]: pgmap v1144: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:06:43.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:43 smithi113 ceph-mon[34341]: 6.17 scrub starts 2024-03-30T14:06:43.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:43 smithi113 ceph-mon[34341]: 5.c scrub starts 2024-03-30T14:06:43.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:43 smithi113 ceph-mon[34341]: 5.c scrub ok 2024-03-30T14:06:43.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:43 smithi113 ceph-mon[34341]: 6.17 scrub ok 2024-03-30T14:06:43.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:43 smithi113 ceph-mon[34341]: pgmap v1144: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:06:43.631 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:44.719 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:44.863 INFO:teuthology.orchestra.run.smithi043.stdout:1666447310913 2024-03-30T14:06:45.171 INFO:teuthology.orchestra.run.smithi043.stdout:107374182714 2024-03-30T14:06:45.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:45 smithi043 ceph-mon[29963]: 6.4 scrub starts 2024-03-30T14:06:45.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:45 smithi043 ceph-mon[29963]: 6.4 scrub ok 2024-03-30T14:06:45.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:45 smithi043 ceph-mon[29963]: pgmap v1145: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T14:06:45.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:45 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:45.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:45 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2145819318' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T14:06:45.261 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:45 smithi043 ceph-mon[42099]: 6.4 scrub starts 2024-03-30T14:06:45.261 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:45 smithi043 ceph-mon[42099]: 6.4 scrub ok 2024-03-30T14:06:45.261 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:45 smithi043 ceph-mon[42099]: pgmap v1145: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T14:06:45.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:45 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:45.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:45 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2145819318' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T14:06:45.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:45 smithi113 ceph-mon[34341]: 6.4 scrub starts 2024-03-30T14:06:45.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:45 smithi113 ceph-mon[34341]: 6.4 scrub ok 2024-03-30T14:06:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:45 smithi113 ceph-mon[34341]: pgmap v1145: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T14:06:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:45 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:45.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:45 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2145819318' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T14:06:45.681 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1666447310911 got 1666447310913 for osd.2 2024-03-30T14:06:45.681 DEBUG:teuthology.parallel:result is None 2024-03-30T14:06:46.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:46 smithi043 ceph-mon[29963]: 4.6 scrub starts 2024-03-30T14:06:46.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:46 smithi043 ceph-mon[29963]: 4.6 scrub ok 2024-03-30T14:06:46.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:46 smithi043 ceph-mon[29963]: 3.1d scrub starts 2024-03-30T14:06:46.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:46 smithi043 ceph-mon[29963]: 3.1d scrub ok 2024-03-30T14:06:46.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:46 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2673825837' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T14:06:46.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:46 smithi043 ceph-mon[29963]: 4.10 scrub starts 2024-03-30T14:06:46.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:46 smithi043 ceph-mon[29963]: 4.10 scrub ok 2024-03-30T14:06:46.262 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:06:46 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:06:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:06:46.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:46 smithi043 ceph-mon[42099]: 4.6 scrub starts 2024-03-30T14:06:46.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:46 smithi043 ceph-mon[42099]: 4.6 scrub ok 2024-03-30T14:06:46.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:46 smithi043 ceph-mon[42099]: 3.1d scrub starts 2024-03-30T14:06:46.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:46 smithi043 ceph-mon[42099]: 3.1d scrub ok 2024-03-30T14:06:46.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:46 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2673825837' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T14:06:46.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:46 smithi043 ceph-mon[42099]: 4.10 scrub starts 2024-03-30T14:06:46.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:46 smithi043 ceph-mon[42099]: 4.10 scrub ok 2024-03-30T14:06:46.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:46 smithi113 ceph-mon[34341]: 4.6 scrub starts 2024-03-30T14:06:46.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:46 smithi113 ceph-mon[34341]: 4.6 scrub ok 2024-03-30T14:06:46.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:46 smithi113 ceph-mon[34341]: 3.1d scrub starts 2024-03-30T14:06:46.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:46 smithi113 ceph-mon[34341]: 3.1d scrub ok 2024-03-30T14:06:46.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:46 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2673825837' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T14:06:46.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:46 smithi113 ceph-mon[34341]: 4.10 scrub starts 2024-03-30T14:06:46.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:46 smithi113 ceph-mon[34341]: 4.10 scrub ok 2024-03-30T14:06:47.085 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:47 smithi043 ceph-mon[29963]: pgmap v1146: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:06:47.085 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:47 smithi043 ceph-mon[42099]: pgmap v1146: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:06:47.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:47 smithi113 ceph-mon[34341]: pgmap v1146: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:06:47.475 INFO:teuthology.orchestra.run.smithi043.stdout:249108103462 2024-03-30T14:06:48.122 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:48 smithi043 ceph-mon[29963]: 6.1d scrub starts 2024-03-30T14:06:48.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:48 smithi043 ceph-mon[29963]: 6.10 scrub starts 2024-03-30T14:06:48.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:48 smithi043 ceph-mon[29963]: 6.1d scrub ok 2024-03-30T14:06:48.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:48 smithi043 ceph-mon[29963]: 6.10 scrub ok 2024-03-30T14:06:48.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:48 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2882540157' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T14:06:48.123 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:48 smithi043 ceph-mon[42099]: 6.1d scrub starts 2024-03-30T14:06:48.123 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:48 smithi043 ceph-mon[42099]: 6.10 scrub starts 2024-03-30T14:06:48.123 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:48 smithi043 ceph-mon[42099]: 6.1d scrub ok 2024-03-30T14:06:48.123 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:48 smithi043 ceph-mon[42099]: 6.10 scrub ok 2024-03-30T14:06:48.123 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:48 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2882540157' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T14:06:48.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:48 smithi113 ceph-mon[34341]: 6.1d scrub starts 2024-03-30T14:06:48.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:48 smithi113 ceph-mon[34341]: 6.10 scrub starts 2024-03-30T14:06:48.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:48 smithi113 ceph-mon[34341]: 6.1d scrub ok 2024-03-30T14:06:48.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:48 smithi113 ceph-mon[34341]: 6.10 scrub ok 2024-03-30T14:06:48.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:48 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2882540157' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T14:06:49.140 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:49 smithi043 ceph-mon[29963]: pgmap v1147: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:06:49.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:49 smithi043 ceph-mon[42099]: pgmap v1147: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:06:49.248 INFO:tasks.cephadm.ceph_manager.ceph:need seq 249108103459 got 249108103462 for osd.7 2024-03-30T14:06:49.249 DEBUG:teuthology.parallel:result is None 2024-03-30T14:06:49.365 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:49 smithi113 ceph-mon[34341]: pgmap v1147: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:06:49.924 INFO:teuthology.orchestra.run.smithi043.stdout:137438953780 2024-03-30T14:06:50.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:06:49 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:06:50.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:50 smithi043 ceph-mon[29963]: 1.2 deep-scrub starts 2024-03-30T14:06:50.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:50 smithi043 ceph-mon[29963]: 1.2 deep-scrub ok 2024-03-30T14:06:50.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:50 smithi043 ceph-mon[29963]: 5.1b scrub starts 2024-03-30T14:06:50.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:50 smithi043 ceph-mon[29963]: 5.1b scrub ok 2024-03-30T14:06:50.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:50 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/256064700' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T14:06:50.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:50 smithi043 ceph-mon[42099]: 1.2 deep-scrub starts 2024-03-30T14:06:50.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:50 smithi043 ceph-mon[42099]: 1.2 deep-scrub ok 2024-03-30T14:06:50.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:50 smithi043 ceph-mon[42099]: 5.1b scrub starts 2024-03-30T14:06:50.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:50 smithi043 ceph-mon[42099]: 5.1b scrub ok 2024-03-30T14:06:50.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:50 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/256064700' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T14:06:50.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:50 smithi113 ceph-mon[34341]: 1.2 deep-scrub starts 2024-03-30T14:06:50.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:50 smithi113 ceph-mon[34341]: 1.2 deep-scrub ok 2024-03-30T14:06:50.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:50 smithi113 ceph-mon[34341]: 5.1b scrub starts 2024-03-30T14:06:50.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:50 smithi113 ceph-mon[34341]: 5.1b scrub ok 2024-03-30T14:06:50.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:50 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/256064700' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T14:06:50.692 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953777 got 137438953780 for osd.4 2024-03-30T14:06:50.692 DEBUG:teuthology.parallel:result is None 2024-03-30T14:06:51.114 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182710 got 107374182714 for osd.3 2024-03-30T14:06:51.114 DEBUG:teuthology.parallel:result is None 2024-03-30T14:06:51.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:51 smithi043 ceph-mon[29963]: 4.1a deep-scrub starts 2024-03-30T14:06:51.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:51 smithi043 ceph-mon[29963]: 4.1a deep-scrub ok 2024-03-30T14:06:51.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:51 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:51.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:51 smithi043 ceph-mon[29963]: pgmap v1148: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:06:51.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:51 smithi043 ceph-mon[42099]: 4.1a deep-scrub starts 2024-03-30T14:06:51.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:51 smithi043 ceph-mon[42099]: 4.1a deep-scrub ok 2024-03-30T14:06:51.347 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:51 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:51.347 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:51 smithi043 ceph-mon[42099]: pgmap v1148: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:06:51.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:51 smithi113 ceph-mon[34341]: 4.1a deep-scrub starts 2024-03-30T14:06:51.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:51 smithi113 ceph-mon[34341]: 4.1a deep-scrub ok 2024-03-30T14:06:51.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:51 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:51.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:51 smithi113 ceph-mon[34341]: pgmap v1148: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:06:51.772 INFO:teuthology.orchestra.run.smithi043.stdout:55834575172 2024-03-30T14:06:52.376 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:52 smithi043 ceph-mon[29963]: 6.8 scrub starts 2024-03-30T14:06:52.376 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:52 smithi043 ceph-mon[29963]: 6.8 scrub ok 2024-03-30T14:06:52.376 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:52 smithi043 ceph-mon[29963]: 5.2 scrub starts 2024-03-30T14:06:52.376 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:52 smithi043 ceph-mon[29963]: 5.2 scrub ok 2024-03-30T14:06:52.376 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:52 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/2120919040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T14:06:52.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:52 smithi043 ceph-mon[42099]: 6.8 scrub starts 2024-03-30T14:06:52.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:52 smithi043 ceph-mon[42099]: 6.8 scrub ok 2024-03-30T14:06:52.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:52 smithi043 ceph-mon[42099]: 5.2 scrub starts 2024-03-30T14:06:52.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:52 smithi043 ceph-mon[42099]: 5.2 scrub ok 2024-03-30T14:06:52.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:52 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/2120919040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T14:06:52.407 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575167 got 55834575172 for osd.1 2024-03-30T14:06:52.407 DEBUG:teuthology.parallel:result is None 2024-03-30T14:06:52.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:52 smithi113 ceph-mon[34341]: 6.8 scrub starts 2024-03-30T14:06:52.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:52 smithi113 ceph-mon[34341]: 6.8 scrub ok 2024-03-30T14:06:52.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:52 smithi113 ceph-mon[34341]: 5.2 scrub starts 2024-03-30T14:06:52.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:52 smithi113 ceph-mon[34341]: 5.2 scrub ok 2024-03-30T14:06:52.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:52 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/2120919040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T14:06:53.076 INFO:teuthology.orchestra.run.smithi043.stdout:210453397803 2024-03-30T14:06:53.319 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:53 smithi043 ceph-mon[29963]: pgmap v1149: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:06:53.319 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:53 smithi043 ceph-mon[42099]: pgmap v1149: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:06:53.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:53 smithi113 ceph-mon[34341]: pgmap v1149: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:06:53.501 INFO:teuthology.orchestra.run.smithi043.stdout:34359738698 2024-03-30T14:06:54.149 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397799 got 210453397803 for osd.6 2024-03-30T14:06:54.149 DEBUG:teuthology.parallel:result is None 2024-03-30T14:06:54.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:54 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/3671860361' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T14:06:54.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:54 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/347066918' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T14:06:54.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:54 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/3671860361' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T14:06:54.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:54 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/347066918' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T14:06:54.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:54 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/3671860361' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T14:06:54.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:54 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/347066918' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T14:06:54.602 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738695 got 34359738698 for osd.0 2024-03-30T14:06:54.602 DEBUG:teuthology.parallel:result is None 2024-03-30T14:06:54.603 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T14:06:54.603 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T14:06:54.849 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:55.302 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:55 smithi043 ceph-mon[29963]: 5.e scrub starts 2024-03-30T14:06:55.302 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:55 smithi043 ceph-mon[29963]: 5.e scrub ok 2024-03-30T14:06:55.302 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:55 smithi043 ceph-mon[29963]: 6.15 scrub starts 2024-03-30T14:06:55.302 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:55 smithi043 ceph-mon[29963]: 6.15 scrub ok 2024-03-30T14:06:55.302 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:55 smithi043 ceph-mon[29963]: pgmap v1150: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:06:55.302 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:55 smithi043 ceph-mon[29963]: 3.10 scrub starts 2024-03-30T14:06:55.302 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:55 smithi043 ceph-mon[29963]: 3.10 scrub ok 2024-03-30T14:06:55.303 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:55 smithi043 ceph-mon[42099]: 5.e scrub starts 2024-03-30T14:06:55.303 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:55 smithi043 ceph-mon[42099]: 5.e scrub ok 2024-03-30T14:06:55.303 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:55 smithi043 ceph-mon[42099]: 6.15 scrub starts 2024-03-30T14:06:55.303 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:55 smithi043 ceph-mon[42099]: 6.15 scrub ok 2024-03-30T14:06:55.303 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:55 smithi043 ceph-mon[42099]: pgmap v1150: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:06:55.303 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:55 smithi043 ceph-mon[42099]: 3.10 scrub starts 2024-03-30T14:06:55.303 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:55 smithi043 ceph-mon[42099]: 3.10 scrub ok 2024-03-30T14:06:55.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:55 smithi113 ceph-mon[34341]: 5.e scrub starts 2024-03-30T14:06:55.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:55 smithi113 ceph-mon[34341]: 5.e scrub ok 2024-03-30T14:06:55.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:55 smithi113 ceph-mon[34341]: 6.15 scrub starts 2024-03-30T14:06:55.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:55 smithi113 ceph-mon[34341]: 6.15 scrub ok 2024-03-30T14:06:55.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:55 smithi113 ceph-mon[34341]: pgmap v1150: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:06:55.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:55 smithi113 ceph-mon[34341]: 3.10 scrub starts 2024-03-30T14:06:55.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:55 smithi113 ceph-mon[34341]: 3.10 scrub ok 2024-03-30T14:06:56.329 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:06:56 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:06:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:06:56.834 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T14:06:56.837 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-30T14:06:57.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:57 smithi043 ceph-mon[29963]: 5.8 scrub starts 2024-03-30T14:06:57.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:57 smithi043 ceph-mon[29963]: 5.8 scrub ok 2024-03-30T14:06:57.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:57 smithi043 ceph-mon[29963]: pgmap v1151: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:06:57.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:57 smithi043 ceph-mon[29963]: 1.11 scrub starts 2024-03-30T14:06:57.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:57 smithi043 ceph-mon[29963]: 1.11 scrub ok 2024-03-30T14:06:57.206 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:57 smithi043 ceph-mon[42099]: 5.8 scrub starts 2024-03-30T14:06:57.206 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:57 smithi043 ceph-mon[42099]: 5.8 scrub ok 2024-03-30T14:06:57.206 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:57 smithi043 ceph-mon[42099]: pgmap v1151: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:06:57.206 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:57 smithi043 ceph-mon[42099]: 1.11 scrub starts 2024-03-30T14:06:57.206 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:57 smithi043 ceph-mon[42099]: 1.11 scrub ok 2024-03-30T14:06:57.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:57 smithi113 ceph-mon[34341]: 5.8 scrub starts 2024-03-30T14:06:57.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:57 smithi113 ceph-mon[34341]: 5.8 scrub ok 2024-03-30T14:06:57.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:57 smithi113 ceph-mon[34341]: pgmap v1151: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:06:57.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:57 smithi113 ceph-mon[34341]: 1.11 scrub starts 2024-03-30T14:06:57.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:57 smithi113 ceph-mon[34341]: 1.11 scrub ok 2024-03-30T14:06:57.639 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":1151,"stamp":"2024-03-30T14:06:55.872686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1661,"num_read_kb":1413,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":609,"num_bytes_recovered":1067973,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3896,"ondisk_log_size":3896,"up":456,"acting":456,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":456,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6296444,"kb_used_data":8444,"kb_used_omap":179,"kb_used_meta":6287756,"kb_avail":743664772,"statfs":{"total":767960285184,"available":761512726528,"internally_reserved":0,"allocated":8646656,"data_stored":5881554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":184005,"internal_metadata":6438662459},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003709"},"pg_stats":[{"pgid":"4.c","version":"421'30","reported_seq":1071,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777197+0000","last_change":"2024-03-30T14:05:53.414839+0000","last_active":"2024-03-30T14:06:32.777197+0000","last_peered":"2024-03-30T14:06:32.777197+0000","last_clean":"2024-03-30T14:06:32.777197+0000","last_became_active":"2024-03-30T14:05:02.368055+0000","last_became_peered":"2024-03-30T14:05:02.368055+0000","last_unstale":"2024-03-30T14:06:32.777197+0000","last_undegraded":"2024-03-30T14:06:32.777197+0000","last_fullsized":"2024-03-30T14:06:32.777197+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'30","last_scrub_stamp":"2024-03-30T14:05:53.414675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:05:53.414675+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:04.999571+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":100,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.590072+0000","last_change":"2024-03-30T14:06:22.564184+0000","last_active":"2024-03-30T14:06:32.590072+0000","last_peered":"2024-03-30T14:06:32.590072+0000","last_clean":"2024-03-30T14:06:32.590072+0000","last_became_active":"2024-03-30T14:05:02.374330+0000","last_became_peered":"2024-03-30T14:05:02.374330+0000","last_unstale":"2024-03-30T14:06:32.590072+0000","last_undegraded":"2024-03-30T14:06:32.590072+0000","last_fullsized":"2024-03-30T14:06:32.590072+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:22.564105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:06:22.564105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:43.927593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"521'23","reported_seq":383,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:45.656738+0000","last_change":"2024-03-30T14:06:12.471677+0000","last_active":"2024-03-30T14:06:45.656738+0000","last_peered":"2024-03-30T14:06:45.656738+0000","last_clean":"2024-03-30T14:06:45.656738+0000","last_became_active":"2024-03-30T14:06:12.471241+0000","last_became_peered":"2024-03-30T14:06:12.471241+0000","last_unstale":"2024-03-30T14:06:45.656738+0000","last_undegraded":"2024-03-30T14:06:45.656738+0000","last_fullsized":"2024-03-30T14:06:45.656738+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"510'22","last_scrub_stamp":"2024-03-30T14:06:06.883865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:06:06.883865+0000","objects_scrubbed":1,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:36.262077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":993,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.778214+0000","last_change":"2024-03-30T14:06:32.381500+0000","last_active":"2024-03-30T14:06:32.778214+0000","last_peered":"2024-03-30T14:06:32.778214+0000","last_clean":"2024-03-30T14:06:32.778214+0000","last_became_active":"2024-03-30T14:04:54.315300+0000","last_became_peered":"2024-03-30T14:04:54.315300+0000","last_unstale":"2024-03-30T14:06:32.778214+0000","last_undegraded":"2024-03-30T14:06:32.778214+0000","last_fullsized":"2024-03-30T14:06:32.778214+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:32.381393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:02:06.483964+0000","last_clean_scrub_stamp":"2024-03-30T14:06:32.381393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:44.606428+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":1053,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775728+0000","last_change":"2024-03-30T14:05:59.005586+0000","last_active":"2024-03-30T14:06:32.775728+0000","last_peered":"2024-03-30T14:06:32.775728+0000","last_clean":"2024-03-30T14:06:32.775728+0000","last_became_active":"2024-03-30T14:05:13.374989+0000","last_became_peered":"2024-03-30T14:05:13.374989+0000","last_unstale":"2024-03-30T14:06:32.775728+0000","last_undegraded":"2024-03-30T14:06:32.775728+0000","last_fullsized":"2024-03-30T14:06:32.775728+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:05:59.005483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:05:59.005483+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:28.678058+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"421'45","reported_seq":1110,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777522+0000","last_change":"2024-03-30T14:05:21.376681+0000","last_active":"2024-03-30T14:06:32.777522+0000","last_peered":"2024-03-30T14:06:32.777522+0000","last_clean":"2024-03-30T14:06:32.777522+0000","last_became_active":"2024-03-30T14:04:54.309600+0000","last_became_peered":"2024-03-30T14:04:54.309600+0000","last_unstale":"2024-03-30T14:06:32.777522+0000","last_undegraded":"2024-03-30T14:06:32.777522+0000","last_fullsized":"2024-03-30T14:06:32.777522+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'45","last_scrub_stamp":"2024-03-30T14:05:21.376569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:05:21.376569+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:50.606469+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1026,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776882+0000","last_change":"2024-03-30T14:06:06.136540+0000","last_active":"2024-03-30T14:06:32.776882+0000","last_peered":"2024-03-30T14:06:32.776882+0000","last_clean":"2024-03-30T14:06:32.776882+0000","last_became_active":"2024-03-30T13:42:45.315251+0000","last_became_peered":"2024-03-30T13:42:45.315251+0000","last_unstale":"2024-03-30T14:06:32.776882+0000","last_undegraded":"2024-03-30T14:06:32.776882+0000","last_fullsized":"2024-03-30T14:06:32.776882+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:06.136452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:57:12.187291+0000","last_clean_scrub_stamp":"2024-03-30T14:06:06.136452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:32.859925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1019,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:41.313423+0000","last_change":"2024-03-30T14:06:41.313446+0000","last_active":"2024-03-30T14:06:41.313423+0000","last_peered":"2024-03-30T14:06:41.313423+0000","last_clean":"2024-03-30T14:06:41.313423+0000","last_became_active":"2024-03-30T13:42:49.321360+0000","last_became_peered":"2024-03-30T13:42:49.321360+0000","last_unstale":"2024-03-30T14:06:41.313423+0000","last_undegraded":"2024-03-30T14:06:41.313423+0000","last_fullsized":"2024-03-30T14:06:41.313423+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:41.313355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:05:26.170570+0000","last_clean_scrub_stamp":"2024-03-30T14:06:41.313355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:10.451996+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":618,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589926+0000","last_change":"2024-03-30T14:06:18.063780+0000","last_active":"2024-03-30T14:06:32.589926+0000","last_peered":"2024-03-30T14:06:32.589926+0000","last_clean":"2024-03-30T14:06:32.589926+0000","last_became_active":"2024-03-30T14:06:12.472854+0000","last_became_peered":"2024-03-30T14:06:12.472854+0000","last_unstale":"2024-03-30T14:06:32.589926+0000","last_undegraded":"2024-03-30T14:06:32.589926+0000","last_fullsized":"2024-03-30T14:06:32.589926+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:18.063574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:18.063574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:37.746573+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":652,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.588699+0000","last_change":"2024-03-30T14:06:04.904808+0000","last_active":"2024-03-30T14:06:32.588699+0000","last_peered":"2024-03-30T14:06:32.588699+0000","last_clean":"2024-03-30T14:06:32.588699+0000","last_became_active":"2024-03-30T14:05:51.009740+0000","last_became_peered":"2024-03-30T14:05:51.009740+0000","last_unstale":"2024-03-30T14:06:32.588699+0000","last_undegraded":"2024-03-30T14:06:32.588699+0000","last_fullsized":"2024-03-30T14:06:32.588699+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:04.904721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:06:04.904721+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:07.858677+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1050752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"421'27","reported_seq":1041,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.778076+0000","last_change":"2024-03-30T14:05:43.440570+0000","last_active":"2024-03-30T14:06:32.778076+0000","last_peered":"2024-03-30T14:06:32.778076+0000","last_clean":"2024-03-30T14:06:32.778076+0000","last_became_active":"2024-03-30T14:04:32.238711+0000","last_became_peered":"2024-03-30T14:04:32.238711+0000","last_unstale":"2024-03-30T14:06:32.778076+0000","last_undegraded":"2024-03-30T14:06:32.778076+0000","last_fullsized":"2024-03-30T14:06:32.778076+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'27","last_scrub_stamp":"2024-03-30T14:05:43.440437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:05:43.440437+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:48.556276+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1024,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776281+0000","last_change":"2024-03-30T14:05:58.340520+0000","last_active":"2024-03-30T14:06:32.776281+0000","last_peered":"2024-03-30T14:06:32.776281+0000","last_clean":"2024-03-30T14:06:32.776281+0000","last_became_active":"2024-03-30T14:05:10.381761+0000","last_became_peered":"2024-03-30T14:05:10.381761+0000","last_unstale":"2024-03-30T14:06:32.776281+0000","last_undegraded":"2024-03-30T14:06:32.776281+0000","last_fullsized":"2024-03-30T14:06:32.776281+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:58.340420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:00:21.110376+0000","last_clean_scrub_stamp":"2024-03-30T14:05:58.340420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:27.438884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":781,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589696+0000","last_change":"2024-03-30T14:05:29.344185+0000","last_active":"2024-03-30T14:06:32.589696+0000","last_peered":"2024-03-30T14:06:32.589696+0000","last_clean":"2024-03-30T14:06:32.589696+0000","last_became_active":"2024-03-30T13:58:48.216542+0000","last_became_peered":"2024-03-30T13:58:48.216542+0000","last_unstale":"2024-03-30T14:06:32.589696+0000","last_undegraded":"2024-03-30T14:06:32.589696+0000","last_fullsized":"2024-03-30T14:06:32.589696+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:29.344072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:52.603772+0000","last_clean_scrub_stamp":"2024-03-30T14:05:29.344072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:44.065692+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":112,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.590585+0000","last_change":"2024-03-30T14:06:02.610467+0000","last_active":"2024-03-30T14:06:32.590585+0000","last_peered":"2024-03-30T14:06:32.590585+0000","last_clean":"2024-03-30T14:06:32.590585+0000","last_became_active":"2024-03-30T14:05:48.013542+0000","last_became_peered":"2024-03-30T14:05:48.013542+0000","last_unstale":"2024-03-30T14:06:32.590585+0000","last_undegraded":"2024-03-30T14:06:32.590585+0000","last_fullsized":"2024-03-30T14:06:32.590585+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:02.610385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:21.081938+0000","last_clean_scrub_stamp":"2024-03-30T14:06:02.610385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:21.547464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"82'128","reported_seq":1034,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.774925+0000","last_change":"2024-03-30T14:05:51.010923+0000","last_active":"2024-03-30T14:06:32.774925+0000","last_peered":"2024-03-30T14:06:32.774925+0000","last_clean":"2024-03-30T14:06:32.774925+0000","last_became_active":"2024-03-30T14:05:51.010364+0000","last_became_peered":"2024-03-30T14:05:51.010364+0000","last_unstale":"2024-03-30T14:06:32.774925+0000","last_undegraded":"2024-03-30T14:06:32.774925+0000","last_fullsized":"2024-03-30T14:06:32.774925+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:05:31.157117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:05:31.157117+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:57.606298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"421'46","reported_seq":1171,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776298+0000","last_change":"2024-03-30T14:06:31.465038+0000","last_active":"2024-03-30T14:06:32.776298+0000","last_peered":"2024-03-30T14:06:32.776298+0000","last_clean":"2024-03-30T14:06:32.776298+0000","last_became_active":"2024-03-30T14:05:02.381261+0000","last_became_peered":"2024-03-30T14:05:02.381261+0000","last_unstale":"2024-03-30T14:06:32.776298+0000","last_undegraded":"2024-03-30T14:06:32.776298+0000","last_fullsized":"2024-03-30T14:06:32.776298+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'46","last_scrub_stamp":"2024-03-30T14:06:31.464933+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:28.831527+0000","last_clean_scrub_stamp":"2024-03-30T14:06:31.464933+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:59.116350+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":114,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.588806+0000","last_change":"2024-03-30T14:06:25.534314+0000","last_active":"2024-03-30T14:06:32.588806+0000","last_peered":"2024-03-30T14:06:32.588806+0000","last_clean":"2024-03-30T14:06:32.588806+0000","last_became_active":"2024-03-30T14:05:10.391971+0000","last_became_peered":"2024-03-30T14:05:10.391971+0000","last_unstale":"2024-03-30T14:06:32.588806+0000","last_undegraded":"2024-03-30T14:06:32.588806+0000","last_fullsized":"2024-03-30T14:06:32.588806+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:25.534220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:06:25.534220+0000","last_clean_scrub_stamp":"2024-03-30T14:06:25.534220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:47.374607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"510'18","reported_seq":1334,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:53.319834+0000","last_change":"2024-03-30T14:06:53.319864+0000","last_active":"2024-03-30T14:06:53.319834+0000","last_peered":"2024-03-30T14:06:53.319834+0000","last_clean":"2024-03-30T14:06:53.319834+0000","last_became_active":"2024-03-30T14:05:48.018872+0000","last_became_peered":"2024-03-30T14:05:48.018872+0000","last_unstale":"2024-03-30T14:06:53.319834+0000","last_undegraded":"2024-03-30T14:06:53.319834+0000","last_fullsized":"2024-03-30T14:06:53.319834+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"510'18","last_scrub_stamp":"2024-03-30T14:06:53.319756+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-30T13:51:15.146255+0000","last_clean_scrub_stamp":"2024-03-30T14:06:53.319756+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:21.906843+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":53,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589313+0000","last_change":"2024-03-30T14:05:52.207952+0000","last_active":"2024-03-30T14:06:32.589313+0000","last_peered":"2024-03-30T14:06:32.589313+0000","last_clean":"2024-03-30T14:06:32.589313+0000","last_became_active":"2024-03-30T14:05:48.022206+0000","last_became_peered":"2024-03-30T14:05:48.022206+0000","last_unstale":"2024-03-30T14:06:32.589313+0000","last_undegraded":"2024-03-30T14:06:32.589313+0000","last_fullsized":"2024-03-30T14:06:32.589313+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:52.207826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:05:52.207826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:09.810738+0000","scrub_duration":140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":809,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589101+0000","last_change":"2024-03-30T14:06:21.907236+0000","last_active":"2024-03-30T14:06:32.589101+0000","last_peered":"2024-03-30T14:06:32.589101+0000","last_clean":"2024-03-30T14:06:32.589101+0000","last_became_active":"2024-03-30T14:04:57.309699+0000","last_became_peered":"2024-03-30T14:04:57.309699+0000","last_unstale":"2024-03-30T14:06:32.589101+0000","last_undegraded":"2024-03-30T14:06:32.589101+0000","last_fullsized":"2024-03-30T14:06:32.589101+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:21.907124+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T13:56:08.385059+0000","last_clean_scrub_stamp":"2024-03-30T14:06:21.907124+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:24.997459+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1023,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776116+0000","last_change":"2024-03-30T14:06:02.362759+0000","last_active":"2024-03-30T14:06:32.776116+0000","last_peered":"2024-03-30T14:06:32.776116+0000","last_clean":"2024-03-30T14:06:32.776116+0000","last_became_active":"2024-03-30T14:05:10.369990+0000","last_became_peered":"2024-03-30T14:05:10.369990+0000","last_unstale":"2024-03-30T14:06:32.776116+0000","last_undegraded":"2024-03-30T14:06:32.776116+0000","last_fullsized":"2024-03-30T14:06:32.776116+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:02.362655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:06:02.362655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:17.954023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"421'47","reported_seq":142,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.588123+0000","last_change":"2024-03-30T14:05:30.309172+0000","last_active":"2024-03-30T14:06:32.588123+0000","last_peered":"2024-03-30T14:06:32.588123+0000","last_clean":"2024-03-30T14:06:32.588123+0000","last_became_active":"2024-03-30T14:05:10.391828+0000","last_became_peered":"2024-03-30T14:05:10.391828+0000","last_unstale":"2024-03-30T14:06:32.588123+0000","last_undegraded":"2024-03-30T14:06:32.588123+0000","last_fullsized":"2024-03-30T14:06:32.588123+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'47","last_scrub_stamp":"2024-03-30T14:05:30.309057+0000","last_deep_scrub":"216'29","last_deep_scrub_stamp":"2024-03-30T13:52:52.973602+0000","last_clean_scrub_stamp":"2024-03-30T14:05:30.309057+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:56.831847+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":106,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.591059+0000","last_change":"2024-03-30T14:05:54.644373+0000","last_active":"2024-03-30T14:06:32.591059+0000","last_peered":"2024-03-30T14:06:32.591059+0000","last_clean":"2024-03-30T14:06:32.591059+0000","last_became_active":"2024-03-30T14:05:10.384945+0000","last_became_peered":"2024-03-30T14:05:10.384945+0000","last_unstale":"2024-03-30T14:06:32.591059+0000","last_undegraded":"2024-03-30T14:06:32.591059+0000","last_fullsized":"2024-03-30T14:06:32.591059+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:54.644196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:53:01.031216+0000","last_clean_scrub_stamp":"2024-03-30T14:05:54.644196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:18.604189+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1006,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776229+0000","last_change":"2024-03-30T14:06:10.198636+0000","last_active":"2024-03-30T14:06:32.776229+0000","last_peered":"2024-03-30T14:06:32.776229+0000","last_clean":"2024-03-30T14:06:32.776229+0000","last_became_active":"2024-03-30T14:05:02.355737+0000","last_became_peered":"2024-03-30T14:05:02.355737+0000","last_unstale":"2024-03-30T14:06:32.776229+0000","last_undegraded":"2024-03-30T14:06:32.776229+0000","last_fullsized":"2024-03-30T14:06:32.776229+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:10.198510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:10.198510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:18.919645+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":1009,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776786+0000","last_change":"2024-03-30T14:06:00.329645+0000","last_active":"2024-03-30T14:06:32.776786+0000","last_peered":"2024-03-30T14:06:32.776786+0000","last_clean":"2024-03-30T14:06:32.776786+0000","last_became_active":"2024-03-30T14:04:57.309024+0000","last_became_peered":"2024-03-30T14:04:57.309024+0000","last_unstale":"2024-03-30T14:06:32.776786+0000","last_undegraded":"2024-03-30T14:06:32.776786+0000","last_fullsized":"2024-03-30T14:06:32.776786+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:00.329540+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T14:04:20.952248+0000","last_clean_scrub_stamp":"2024-03-30T14:06:00.329540+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:09.159979+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"487'10","reported_seq":126,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:45.656683+0000","last_change":"2024-03-30T14:05:47.569855+0000","last_active":"2024-03-30T14:06:45.656683+0000","last_peered":"2024-03-30T14:06:45.656683+0000","last_clean":"2024-03-30T14:06:45.656683+0000","last_became_active":"2024-03-30T14:05:02.378061+0000","last_became_peered":"2024-03-30T14:05:02.378061+0000","last_unstale":"2024-03-30T14:06:45.656683+0000","last_undegraded":"2024-03-30T14:06:45.656683+0000","last_fullsized":"2024-03-30T14:06:45.656683+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"487'10","last_scrub_stamp":"2024-03-30T14:05:47.569760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:05:47.569760+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:14.100898+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"421'49","reported_seq":1157,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776379+0000","last_change":"2024-03-30T14:05:54.450308+0000","last_active":"2024-03-30T14:06:32.776379+0000","last_peered":"2024-03-30T14:06:32.776379+0000","last_clean":"2024-03-30T14:06:32.776379+0000","last_became_active":"2024-03-30T14:05:48.013215+0000","last_became_peered":"2024-03-30T14:05:48.013215+0000","last_unstale":"2024-03-30T14:06:32.776379+0000","last_undegraded":"2024-03-30T14:06:32.776379+0000","last_fullsized":"2024-03-30T14:06:32.776379+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'49","last_scrub_stamp":"2024-03-30T14:05:54.450160+0000","last_deep_scrub":"91'21","last_deep_scrub_stamp":"2024-03-30T13:44:05.917064+0000","last_clean_scrub_stamp":"2024-03-30T14:05:54.450160+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:19.068715+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"499'2","reported_seq":773,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775576+0000","last_change":"2024-03-30T14:05:30.754898+0000","last_active":"2024-03-30T14:06:32.775576+0000","last_peered":"2024-03-30T14:06:32.775576+0000","last_clean":"2024-03-30T14:06:32.775576+0000","last_became_active":"2024-03-30T14:05:23.027201+0000","last_became_peered":"2024-03-30T14:05:23.027201+0000","last_unstale":"2024-03-30T14:06:32.775576+0000","last_undegraded":"2024-03-30T14:06:32.775576+0000","last_fullsized":"2024-03-30T14:06:32.775576+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"499'2","last_scrub_stamp":"2024-03-30T14:05:30.754755+0000","last_deep_scrub":"499'2","last_deep_scrub_stamp":"2024-03-30T14:05:30.754755+0000","last_clean_scrub_stamp":"2024-03-30T14:05:30.754755+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:58.196997+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":90,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776338+0000","last_change":"2024-03-30T14:05:43.139037+0000","last_active":"2024-03-30T14:06:32.776338+0000","last_peered":"2024-03-30T14:06:32.776338+0000","last_clean":"2024-03-30T14:06:32.776338+0000","last_became_active":"2024-03-30T14:05:10.389836+0000","last_became_peered":"2024-03-30T14:05:10.389836+0000","last_unstale":"2024-03-30T14:06:32.776338+0000","last_undegraded":"2024-03-30T14:06:32.776338+0000","last_fullsized":"2024-03-30T14:06:32.776338+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:43.138904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:05:43.138904+0000","last_clean_scrub_stamp":"2024-03-30T14:05:43.138904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:48.643259+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":472,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.590482+0000","last_change":"2024-03-30T14:06:32.446539+0000","last_active":"2024-03-30T14:06:32.590482+0000","last_peered":"2024-03-30T14:06:32.590482+0000","last_clean":"2024-03-30T14:06:32.590482+0000","last_became_active":"2024-03-30T14:05:51.010139+0000","last_became_peered":"2024-03-30T14:05:51.010139+0000","last_unstale":"2024-03-30T14:06:32.590482+0000","last_undegraded":"2024-03-30T14:06:32.590482+0000","last_fullsized":"2024-03-30T14:06:32.590482+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:32.446450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:06:32.446450+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:48.178673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"510'18","reported_seq":691,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:45.656666+0000","last_change":"2024-03-30T14:06:39.464075+0000","last_active":"2024-03-30T14:06:45.656666+0000","last_peered":"2024-03-30T14:06:45.656666+0000","last_clean":"2024-03-30T14:06:45.656666+0000","last_became_active":"2024-03-30T14:05:48.024814+0000","last_became_peered":"2024-03-30T14:05:48.024814+0000","last_unstale":"2024-03-30T14:06:45.656666+0000","last_undegraded":"2024-03-30T14:06:45.656666+0000","last_fullsized":"2024-03-30T14:06:45.656666+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"510'18","last_scrub_stamp":"2024-03-30T14:06:39.463949+0000","last_deep_scrub":"218'9","last_deep_scrub_stamp":"2024-03-30T13:52:57.873554+0000","last_clean_scrub_stamp":"2024-03-30T14:06:39.463949+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:40.879509+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"421'35","reported_seq":1144,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:33.463936+0000","last_change":"2024-03-30T14:06:33.463969+0000","last_active":"2024-03-30T14:06:33.463936+0000","last_peered":"2024-03-30T14:06:33.463936+0000","last_clean":"2024-03-30T14:06:33.463936+0000","last_became_active":"2024-03-30T14:05:48.022376+0000","last_became_peered":"2024-03-30T14:05:48.022376+0000","last_unstale":"2024-03-30T14:06:33.463936+0000","last_undegraded":"2024-03-30T14:06:33.463936+0000","last_fullsized":"2024-03-30T14:06:33.463936+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'35","last_scrub_stamp":"2024-03-30T14:06:33.463859+0000","last_deep_scrub":"421'35","last_deep_scrub_stamp":"2024-03-30T14:06:33.463859+0000","last_clean_scrub_stamp":"2024-03-30T14:06:33.463859+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:39.881076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":53,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589391+0000","last_change":"2024-03-30T14:05:59.043577+0000","last_active":"2024-03-30T14:06:32.589391+0000","last_peered":"2024-03-30T14:06:32.589391+0000","last_clean":"2024-03-30T14:06:32.589391+0000","last_became_active":"2024-03-30T14:05:48.022990+0000","last_became_peered":"2024-03-30T14:05:48.022990+0000","last_unstale":"2024-03-30T14:06:32.589391+0000","last_undegraded":"2024-03-30T14:06:32.589391+0000","last_fullsized":"2024-03-30T14:06:32.589391+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:59.043489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:55.458779+0000","last_clean_scrub_stamp":"2024-03-30T14:05:59.043489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:28.883155+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":739,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.774999+0000","last_change":"2024-03-30T14:05:45.745085+0000","last_active":"2024-03-30T14:06:32.774999+0000","last_peered":"2024-03-30T14:06:32.774999+0000","last_clean":"2024-03-30T14:06:32.774999+0000","last_became_active":"2024-03-30T14:05:23.026736+0000","last_became_peered":"2024-03-30T14:05:23.026736+0000","last_unstale":"2024-03-30T14:06:32.774999+0000","last_undegraded":"2024-03-30T14:06:32.774999+0000","last_fullsized":"2024-03-30T14:06:32.774999+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:45.744982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:41.142941+0000","last_clean_scrub_stamp":"2024-03-30T14:05:45.744982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:07.935939+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":1072,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:35.991585+0000","last_change":"2024-03-30T14:06:35.991612+0000","last_active":"2024-03-30T14:06:35.991585+0000","last_peered":"2024-03-30T14:06:35.991585+0000","last_clean":"2024-03-30T14:06:35.991585+0000","last_became_active":"2024-03-30T14:05:13.376299+0000","last_became_peered":"2024-03-30T14:05:13.376299+0000","last_unstale":"2024-03-30T14:06:35.991585+0000","last_undegraded":"2024-03-30T14:06:35.991585+0000","last_fullsized":"2024-03-30T14:06:35.991585+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:35.991460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:06:35.991460+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:50.551910+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1037,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:51.083380+0000","last_change":"2024-03-30T14:06:51.083407+0000","last_active":"2024-03-30T14:06:51.083380+0000","last_peered":"2024-03-30T14:06:51.083380+0000","last_clean":"2024-03-30T14:06:51.083380+0000","last_became_active":"2024-03-30T13:58:47.204999+0000","last_became_peered":"2024-03-30T13:58:47.204999+0000","last_unstale":"2024-03-30T14:06:51.083380+0000","last_undegraded":"2024-03-30T14:06:51.083380+0000","last_fullsized":"2024-03-30T14:06:51.083380+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:51.083308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:52.887014+0000","last_clean_scrub_stamp":"2024-03-30T14:06:51.083308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:57.342668+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"421'47","reported_seq":888,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775756+0000","last_change":"2024-03-30T14:05:51.551210+0000","last_active":"2024-03-30T14:06:32.775756+0000","last_peered":"2024-03-30T14:06:32.775756+0000","last_clean":"2024-03-30T14:06:32.775756+0000","last_became_active":"2024-03-30T14:05:48.010044+0000","last_became_peered":"2024-03-30T14:05:48.010044+0000","last_unstale":"2024-03-30T14:06:32.775756+0000","last_undegraded":"2024-03-30T14:06:32.775756+0000","last_fullsized":"2024-03-30T14:06:32.775756+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'47","last_scrub_stamp":"2024-03-30T14:05:51.551027+0000","last_deep_scrub":"402'37","last_deep_scrub_stamp":"2024-03-30T14:02:06.552075+0000","last_clean_scrub_stamp":"2024-03-30T14:05:51.551027+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:52.768463+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1054,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776459+0000","last_change":"2024-03-30T14:06:15.252088+0000","last_active":"2024-03-30T14:06:32.776459+0000","last_peered":"2024-03-30T14:06:32.776459+0000","last_clean":"2024-03-30T14:06:32.776459+0000","last_became_active":"2024-03-30T14:05:48.009994+0000","last_became_peered":"2024-03-30T14:05:48.009994+0000","last_unstale":"2024-03-30T14:06:32.776459+0000","last_undegraded":"2024-03-30T14:06:32.776459+0000","last_fullsized":"2024-03-30T14:06:32.776459+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:15.252005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:45.975163+0000","last_clean_scrub_stamp":"2024-03-30T14:06:15.252005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:19.851201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":997,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775201+0000","last_change":"2024-03-30T14:05:55.395143+0000","last_active":"2024-03-30T14:06:32.775201+0000","last_peered":"2024-03-30T14:06:32.775201+0000","last_clean":"2024-03-30T14:06:32.775201+0000","last_became_active":"2024-03-30T14:04:54.319778+0000","last_became_peered":"2024-03-30T14:04:54.319778+0000","last_unstale":"2024-03-30T14:06:32.775201+0000","last_undegraded":"2024-03-30T14:06:32.775201+0000","last_fullsized":"2024-03-30T14:06:32.775201+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:55.395044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:05:55.395044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:13.300881+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":1031,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777280+0000","last_change":"2024-03-30T14:05:41.264840+0000","last_active":"2024-03-30T14:06:32.777280+0000","last_peered":"2024-03-30T14:06:32.777280+0000","last_clean":"2024-03-30T14:06:32.777280+0000","last_became_active":"2024-03-30T14:05:05.004102+0000","last_became_peered":"2024-03-30T14:05:05.004102+0000","last_unstale":"2024-03-30T14:06:32.777280+0000","last_undegraded":"2024-03-30T14:06:32.777280+0000","last_fullsized":"2024-03-30T14:06:32.777280+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:05:41.264730+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T14:02:06.183099+0000","last_clean_scrub_stamp":"2024-03-30T14:05:41.264730+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:08.239137+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":53,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589530+0000","last_change":"2024-03-30T14:06:02.061179+0000","last_active":"2024-03-30T14:06:32.589530+0000","last_peered":"2024-03-30T14:06:32.589530+0000","last_clean":"2024-03-30T14:06:32.589530+0000","last_became_active":"2024-03-30T14:05:48.021019+0000","last_became_peered":"2024-03-30T14:05:48.021019+0000","last_unstale":"2024-03-30T14:06:32.589530+0000","last_undegraded":"2024-03-30T14:06:32.589530+0000","last_fullsized":"2024-03-30T14:06:32.589530+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:02.061083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:06:02.061083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:30.280253+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"421'42","reported_seq":631,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:44.412412+0000","last_change":"2024-03-30T14:06:44.412480+0000","last_active":"2024-03-30T14:06:44.412412+0000","last_peered":"2024-03-30T14:06:44.412412+0000","last_clean":"2024-03-30T14:06:44.412412+0000","last_became_active":"2024-03-30T14:05:31.110871+0000","last_became_peered":"2024-03-30T14:05:31.110871+0000","last_unstale":"2024-03-30T14:06:44.412412+0000","last_undegraded":"2024-03-30T14:06:44.412412+0000","last_fullsized":"2024-03-30T14:06:44.412412+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'42","last_scrub_stamp":"2024-03-30T14:06:44.412318+0000","last_deep_scrub":"93'18","last_deep_scrub_stamp":"2024-03-30T13:47:28.872529+0000","last_clean_scrub_stamp":"2024-03-30T14:06:44.412318+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:47.329231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":198,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:52.561380+0000","last_change":"2024-03-30T14:05:34.950515+0000","last_active":"2024-03-30T14:06:52.561380+0000","last_peered":"2024-03-30T14:06:52.561380+0000","last_clean":"2024-03-30T14:06:52.561380+0000","last_became_active":"2024-03-30T14:05:10.390745+0000","last_became_peered":"2024-03-30T14:05:10.390745+0000","last_unstale":"2024-03-30T14:06:52.561380+0000","last_undegraded":"2024-03-30T14:06:52.561380+0000","last_fullsized":"2024-03-30T14:06:52.561380+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-30T14:05:34.950411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T14:05:34.950411+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00126391,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":110,"num_read_kb":105,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":771,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775200+0000","last_change":"2024-03-30T14:06:16.450625+0000","last_active":"2024-03-30T14:06:32.775200+0000","last_peered":"2024-03-30T14:06:32.775200+0000","last_clean":"2024-03-30T14:06:32.775200+0000","last_became_active":"2024-03-30T14:05:23.029617+0000","last_became_peered":"2024-03-30T14:05:23.029617+0000","last_unstale":"2024-03-30T14:06:32.775200+0000","last_undegraded":"2024-03-30T14:06:32.775200+0000","last_fullsized":"2024-03-30T14:06:32.775200+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:16.450479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:00:25.597831+0000","last_clean_scrub_stamp":"2024-03-30T14:06:16.450479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:40.513078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1032,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:43.301180+0000","last_change":"2024-03-30T14:06:43.301202+0000","last_active":"2024-03-30T14:06:43.301180+0000","last_peered":"2024-03-30T14:06:43.301180+0000","last_clean":"2024-03-30T14:06:43.301180+0000","last_became_active":"2024-03-30T14:05:02.375363+0000","last_became_peered":"2024-03-30T14:05:02.375363+0000","last_unstale":"2024-03-30T14:06:43.301180+0000","last_undegraded":"2024-03-30T14:06:43.301180+0000","last_fullsized":"2024-03-30T14:06:43.301180+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:43.301118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:43.301118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:53.234315+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":1000,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776737+0000","last_change":"2024-03-30T14:05:33.136632+0000","last_active":"2024-03-30T14:06:32.776737+0000","last_peered":"2024-03-30T14:06:32.776737+0000","last_clean":"2024-03-30T14:06:32.776737+0000","last_became_active":"2024-03-30T14:00:56.844488+0000","last_became_peered":"2024-03-30T14:00:56.844488+0000","last_unstale":"2024-03-30T14:06:32.776737+0000","last_undegraded":"2024-03-30T14:06:32.776737+0000","last_fullsized":"2024-03-30T14:06:32.776737+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:05:33.136504+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T14:05:33.136504+0000","last_clean_scrub_stamp":"2024-03-30T14:05:33.136504+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:52.647766+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":90,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775896+0000","last_change":"2024-03-30T14:06:14.741843+0000","last_active":"2024-03-30T14:06:32.775896+0000","last_peered":"2024-03-30T14:06:32.775896+0000","last_clean":"2024-03-30T14:06:32.775896+0000","last_became_active":"2024-03-30T14:05:10.385323+0000","last_became_peered":"2024-03-30T14:05:10.385323+0000","last_unstale":"2024-03-30T14:06:32.775896+0000","last_undegraded":"2024-03-30T14:06:32.775896+0000","last_fullsized":"2024-03-30T14:06:32.775896+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:14.741707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:06:14.741707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:28.894718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"500'43","reported_seq":1442,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:45.656952+0000","last_change":"2024-03-30T14:06:39.022189+0000","last_active":"2024-03-30T14:06:45.656952+0000","last_peered":"2024-03-30T14:06:45.656952+0000","last_clean":"2024-03-30T14:06:45.656952+0000","last_became_active":"2024-03-30T14:05:31.112587+0000","last_became_peered":"2024-03-30T14:05:31.112587+0000","last_unstale":"2024-03-30T14:06:45.656952+0000","last_undegraded":"2024-03-30T14:06:45.656952+0000","last_fullsized":"2024-03-30T14:06:45.656952+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'43","last_scrub_stamp":"2024-03-30T14:06:39.022084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:06:39.022084+0000","objects_scrubbed":8,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:59.190264+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":106,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589437+0000","last_change":"2024-03-30T14:05:50.650038+0000","last_active":"2024-03-30T14:06:32.589437+0000","last_peered":"2024-03-30T14:06:32.589437+0000","last_clean":"2024-03-30T14:06:32.589437+0000","last_became_active":"2024-03-30T14:05:48.017537+0000","last_became_peered":"2024-03-30T14:05:48.017537+0000","last_unstale":"2024-03-30T14:06:32.589437+0000","last_undegraded":"2024-03-30T14:06:32.589437+0000","last_fullsized":"2024-03-30T14:06:32.589437+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:50.649954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:12.975288+0000","last_clean_scrub_stamp":"2024-03-30T14:05:50.649954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:04.926746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":728,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775145+0000","last_change":"2024-03-30T14:06:22.775255+0000","last_active":"2024-03-30T14:06:32.775145+0000","last_peered":"2024-03-30T14:06:32.775145+0000","last_clean":"2024-03-30T14:06:32.775145+0000","last_became_active":"2024-03-30T14:05:13.377385+0000","last_became_peered":"2024-03-30T14:05:13.377385+0000","last_unstale":"2024-03-30T14:06:32.775145+0000","last_undegraded":"2024-03-30T14:06:32.775145+0000","last_fullsized":"2024-03-30T14:06:32.775145+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:22.775125+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T14:06:22.775125+0000","last_clean_scrub_stamp":"2024-03-30T14:06:22.775125+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:29.872812+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":448,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589624+0000","last_change":"2024-03-30T14:05:30.383963+0000","last_active":"2024-03-30T14:06:32.589624+0000","last_peered":"2024-03-30T14:06:32.589624+0000","last_clean":"2024-03-30T14:06:32.589624+0000","last_became_active":"2024-03-30T14:05:02.354378+0000","last_became_peered":"2024-03-30T14:05:02.354378+0000","last_unstale":"2024-03-30T14:06:32.589624+0000","last_undegraded":"2024-03-30T14:06:32.589624+0000","last_fullsized":"2024-03-30T14:06:32.589624+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:30.383863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:03:13.730670+0000","last_clean_scrub_stamp":"2024-03-30T14:05:30.383863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:48.038095+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":393,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.590054+0000","last_change":"2024-03-30T14:06:12.470388+0000","last_active":"2024-03-30T14:06:32.590054+0000","last_peered":"2024-03-30T14:06:32.590054+0000","last_clean":"2024-03-30T14:06:32.590054+0000","last_became_active":"2024-03-30T14:06:12.470051+0000","last_became_peered":"2024-03-30T14:06:12.470051+0000","last_unstale":"2024-03-30T14:06:32.590054+0000","last_undegraded":"2024-03-30T14:06:32.590054+0000","last_fullsized":"2024-03-30T14:06:32.590054+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:46.189428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:28.373277+0000","last_clean_scrub_stamp":"2024-03-30T14:05:46.189428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:09.871386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"421'21","reported_seq":272,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776201+0000","last_change":"2024-03-30T14:06:01.182687+0000","last_active":"2024-03-30T14:06:32.776201+0000","last_peered":"2024-03-30T14:06:32.776201+0000","last_clean":"2024-03-30T14:06:32.776201+0000","last_became_active":"2024-03-30T14:05:48.010976+0000","last_became_peered":"2024-03-30T14:05:48.010976+0000","last_unstale":"2024-03-30T14:06:32.776201+0000","last_undegraded":"2024-03-30T14:06:32.776201+0000","last_fullsized":"2024-03-30T14:06:32.776201+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-30T14:06:01.182588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:06:01.182588+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:11.440031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":152,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.588750+0000","last_change":"2024-03-30T14:06:12.469306+0000","last_active":"2024-03-30T14:06:32.588750+0000","last_peered":"2024-03-30T14:06:32.588750+0000","last_clean":"2024-03-30T14:06:32.588750+0000","last_became_active":"2024-03-30T14:06:12.468985+0000","last_became_peered":"2024-03-30T14:06:12.468985+0000","last_unstale":"2024-03-30T14:06:32.588750+0000","last_undegraded":"2024-03-30T14:06:32.588750+0000","last_fullsized":"2024-03-30T14:06:32.588750+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:45.547752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:51.919203+0000","last_clean_scrub_stamp":"2024-03-30T14:05:45.547752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:10.263518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1010,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.774895+0000","last_change":"2024-03-30T14:05:53.286502+0000","last_active":"2024-03-30T14:06:32.774895+0000","last_peered":"2024-03-30T14:06:32.774895+0000","last_clean":"2024-03-30T14:06:32.774895+0000","last_became_active":"2024-03-30T14:04:54.329763+0000","last_became_peered":"2024-03-30T14:04:54.329763+0000","last_unstale":"2024-03-30T14:06:32.774895+0000","last_undegraded":"2024-03-30T14:06:32.774895+0000","last_fullsized":"2024-03-30T14:06:32.774895+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:53.286374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:59.810189+0000","last_clean_scrub_stamp":"2024-03-30T14:05:53.286374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:18.716633+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":98,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776553+0000","last_change":"2024-03-30T14:06:29.796095+0000","last_active":"2024-03-30T14:06:32.776553+0000","last_peered":"2024-03-30T14:06:32.776553+0000","last_clean":"2024-03-30T14:06:32.776553+0000","last_became_active":"2024-03-30T14:05:10.387768+0000","last_became_peered":"2024-03-30T14:05:10.387768+0000","last_unstale":"2024-03-30T14:06:32.776553+0000","last_undegraded":"2024-03-30T14:06:32.776553+0000","last_fullsized":"2024-03-30T14:06:32.776553+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:29.795955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:29.795955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:56.768115+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":921,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:48.800826+0000","last_change":"2024-03-30T14:06:48.800878+0000","last_active":"2024-03-30T14:06:48.800826+0000","last_peered":"2024-03-30T14:06:48.800826+0000","last_clean":"2024-03-30T14:06:48.800826+0000","last_became_active":"2024-03-30T14:05:13.374887+0000","last_became_peered":"2024-03-30T14:05:13.374887+0000","last_unstale":"2024-03-30T14:06:48.800826+0000","last_undegraded":"2024-03-30T14:06:48.800826+0000","last_fullsized":"2024-03-30T14:06:48.800826+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:48.800735+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T14:06:48.800735+0000","last_clean_scrub_stamp":"2024-03-30T14:06:48.800735+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:05.300363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":771,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775605+0000","last_change":"2024-03-30T14:05:31.705349+0000","last_active":"2024-03-30T14:06:32.775605+0000","last_peered":"2024-03-30T14:06:32.775605+0000","last_clean":"2024-03-30T14:06:32.775605+0000","last_became_active":"2024-03-30T14:05:23.027144+0000","last_became_peered":"2024-03-30T14:05:23.027144+0000","last_unstale":"2024-03-30T14:06:32.775605+0000","last_undegraded":"2024-03-30T14:06:32.775605+0000","last_fullsized":"2024-03-30T14:06:32.775605+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:31.705232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:18.862784+0000","last_clean_scrub_stamp":"2024-03-30T14:05:31.705232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:55.636278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"487'60","reported_seq":392,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:50.658053+0000","last_change":"2024-03-30T14:06:30.830561+0000","last_active":"2024-03-30T14:06:50.658053+0000","last_peered":"2024-03-30T14:06:50.658053+0000","last_clean":"2024-03-30T14:06:50.658053+0000","last_became_active":"2024-03-30T14:05:02.351915+0000","last_became_peered":"2024-03-30T14:05:02.351915+0000","last_unstale":"2024-03-30T14:06:50.658053+0000","last_undegraded":"2024-03-30T14:06:50.658053+0000","last_fullsized":"2024-03-30T14:06:50.658053+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"487'60","last_scrub_stamp":"2024-03-30T14:06:30.830372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:06:30.830372+0000","objects_scrubbed":10,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:41.709820+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":1790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":995,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:37.252192+0000","last_change":"2024-03-30T14:06:37.252219+0000","last_active":"2024-03-30T14:06:37.252192+0000","last_peered":"2024-03-30T14:06:37.252192+0000","last_clean":"2024-03-30T14:06:37.252192+0000","last_became_active":"2024-03-30T14:04:32.225606+0000","last_became_peered":"2024-03-30T14:04:32.225606+0000","last_unstale":"2024-03-30T14:06:37.252192+0000","last_undegraded":"2024-03-30T14:06:37.252192+0000","last_fullsized":"2024-03-30T14:06:37.252192+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:37.252116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:06:37.252116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:00.804341+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":440,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.588910+0000","last_change":"2024-03-30T14:06:26.760069+0000","last_active":"2024-03-30T14:06:32.588910+0000","last_peered":"2024-03-30T14:06:32.588910+0000","last_clean":"2024-03-30T14:06:32.588910+0000","last_became_active":"2024-03-30T13:58:47.199657+0000","last_became_peered":"2024-03-30T13:58:47.199657+0000","last_unstale":"2024-03-30T14:06:32.588910+0000","last_undegraded":"2024-03-30T14:06:32.588910+0000","last_fullsized":"2024-03-30T14:06:32.588910+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-30T14:06:26.759968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T14:06:26.759968+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:49.851304+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":106,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589836+0000","last_change":"2024-03-30T14:05:49.672004+0000","last_active":"2024-03-30T14:06:32.589836+0000","last_peered":"2024-03-30T14:06:32.589836+0000","last_clean":"2024-03-30T14:06:32.589836+0000","last_became_active":"2024-03-30T14:05:10.379010+0000","last_became_peered":"2024-03-30T14:05:10.379010+0000","last_unstale":"2024-03-30T14:06:32.589836+0000","last_undegraded":"2024-03-30T14:06:32.589836+0000","last_fullsized":"2024-03-30T14:06:32.589836+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:49.671876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:05:49.671876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:10.705751+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":622,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:38.976634+0000","last_change":"2024-03-30T14:06:38.976661+0000","last_active":"2024-03-30T14:06:38.976634+0000","last_peered":"2024-03-30T14:06:38.976634+0000","last_clean":"2024-03-30T14:06:38.976634+0000","last_became_active":"2024-03-30T14:05:51.010372+0000","last_became_peered":"2024-03-30T14:05:51.010372+0000","last_unstale":"2024-03-30T14:06:38.976634+0000","last_undegraded":"2024-03-30T14:06:38.976634+0000","last_fullsized":"2024-03-30T14:06:38.976634+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:38.976557+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T14:06:38.976557+0000","last_clean_scrub_stamp":"2024-03-30T14:06:38.976557+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:58.695201+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"421'21","reported_seq":124,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775925+0000","last_change":"2024-03-30T14:06:22.485148+0000","last_active":"2024-03-30T14:06:32.775925+0000","last_peered":"2024-03-30T14:06:32.775925+0000","last_clean":"2024-03-30T14:06:32.775925+0000","last_became_active":"2024-03-30T14:05:23.027240+0000","last_became_peered":"2024-03-30T14:05:23.027240+0000","last_unstale":"2024-03-30T14:06:32.775925+0000","last_undegraded":"2024-03-30T14:06:32.775925+0000","last_fullsized":"2024-03-30T14:06:32.775925+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-30T14:06:22.485044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:06:22.485044+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:33.258501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":448,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589440+0000","last_change":"2024-03-30T14:06:13.953901+0000","last_active":"2024-03-30T14:06:32.589440+0000","last_peered":"2024-03-30T14:06:32.589440+0000","last_clean":"2024-03-30T14:06:32.589440+0000","last_became_active":"2024-03-30T14:05:02.351288+0000","last_became_peered":"2024-03-30T14:05:02.351288+0000","last_unstale":"2024-03-30T14:06:32.589440+0000","last_undegraded":"2024-03-30T14:06:32.589440+0000","last_fullsized":"2024-03-30T14:06:32.589440+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:13.953768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:05:04.382303+0000","last_clean_scrub_stamp":"2024-03-30T14:06:13.953768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:28.332449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":707,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.978561+0000","last_change":"2024-03-30T14:06:32.978596+0000","last_active":"2024-03-30T14:06:32.978561+0000","last_peered":"2024-03-30T14:06:32.978561+0000","last_clean":"2024-03-30T14:06:32.978561+0000","last_became_active":"2024-03-30T14:06:12.472079+0000","last_became_peered":"2024-03-30T14:06:12.472079+0000","last_unstale":"2024-03-30T14:06:32.978561+0000","last_undegraded":"2024-03-30T14:06:32.978561+0000","last_fullsized":"2024-03-30T14:06:32.978561+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:32.978474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:06:32.978474+0000","last_clean_scrub_stamp":"2024-03-30T14:06:32.978474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:00.453347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":752,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:38.639044+0000","last_change":"2024-03-30T14:06:38.639069+0000","last_active":"2024-03-30T14:06:38.639044+0000","last_peered":"2024-03-30T14:06:38.639044+0000","last_clean":"2024-03-30T14:06:38.639044+0000","last_became_active":"2024-03-30T14:05:23.032172+0000","last_became_peered":"2024-03-30T14:05:23.032172+0000","last_unstale":"2024-03-30T14:06:38.639044+0000","last_undegraded":"2024-03-30T14:06:38.639044+0000","last_fullsized":"2024-03-30T14:06:38.639044+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:38.638957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:06:38.638957+0000","last_clean_scrub_stamp":"2024-03-30T14:06:38.638957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:08.440754+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"82'128","reported_seq":679,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777209+0000","last_change":"2024-03-30T14:06:01.321053+0000","last_active":"2024-03-30T14:06:32.777209+0000","last_peered":"2024-03-30T14:06:32.777209+0000","last_clean":"2024-03-30T14:06:32.777209+0000","last_became_active":"2024-03-30T14:05:13.375610+0000","last_became_peered":"2024-03-30T14:05:13.375610+0000","last_unstale":"2024-03-30T14:06:32.777209+0000","last_undegraded":"2024-03-30T14:06:32.777209+0000","last_fullsized":"2024-03-30T14:06:32.777209+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:01.320915+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T13:57:14.131256+0000","last_clean_scrub_stamp":"2024-03-30T14:06:01.320915+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:11.501513+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"421'60","reported_seq":1214,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777014+0000","last_change":"2024-03-30T14:06:00.211726+0000","last_active":"2024-03-30T14:06:32.777014+0000","last_peered":"2024-03-30T14:06:32.777014+0000","last_clean":"2024-03-30T14:06:32.777014+0000","last_became_active":"2024-03-30T14:05:31.112101+0000","last_became_peered":"2024-03-30T14:05:31.112101+0000","last_unstale":"2024-03-30T14:06:32.777014+0000","last_undegraded":"2024-03-30T14:06:32.777014+0000","last_fullsized":"2024-03-30T14:06:32.777014+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'60","last_scrub_stamp":"2024-03-30T14:06:00.211598+0000","last_deep_scrub":"108'28","last_deep_scrub_stamp":"2024-03-30T13:45:49.140418+0000","last_clean_scrub_stamp":"2024-03-30T14:06:00.211598+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:14.411804+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":96,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776352+0000","last_change":"2024-03-30T14:05:48.013577+0000","last_active":"2024-03-30T14:06:32.776352+0000","last_peered":"2024-03-30T14:06:32.776352+0000","last_clean":"2024-03-30T14:06:32.776352+0000","last_became_active":"2024-03-30T14:05:48.013164+0000","last_became_peered":"2024-03-30T14:05:48.013164+0000","last_unstale":"2024-03-30T14:06:32.776352+0000","last_undegraded":"2024-03-30T14:06:32.776352+0000","last_fullsized":"2024-03-30T14:06:32.776352+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:31.902862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:17.366740+0000","last_clean_scrub_stamp":"2024-03-30T14:05:31.902862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:46.896777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":587,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589249+0000","last_change":"2024-03-30T14:06:15.900394+0000","last_active":"2024-03-30T14:06:32.589249+0000","last_peered":"2024-03-30T14:06:32.589249+0000","last_clean":"2024-03-30T14:06:32.589249+0000","last_became_active":"2024-03-30T14:06:12.470995+0000","last_became_peered":"2024-03-30T14:06:12.470995+0000","last_unstale":"2024-03-30T14:06:32.589249+0000","last_undegraded":"2024-03-30T14:06:32.589249+0000","last_fullsized":"2024-03-30T14:06:32.589249+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:15.900273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:06:15.900273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:17.378314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":98,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:50.768371+0000","last_change":"2024-03-30T14:06:50.768397+0000","last_active":"2024-03-30T14:06:50.768371+0000","last_peered":"2024-03-30T14:06:50.768371+0000","last_clean":"2024-03-30T14:06:50.768371+0000","last_became_active":"2024-03-30T14:05:10.386648+0000","last_became_peered":"2024-03-30T14:05:10.386648+0000","last_unstale":"2024-03-30T14:06:50.768371+0000","last_undegraded":"2024-03-30T14:06:50.768371+0000","last_fullsized":"2024-03-30T14:06:50.768371+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:50.768296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:05:25.850051+0000","last_clean_scrub_stamp":"2024-03-30T14:06:50.768296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:14.329153+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"82'128","reported_seq":1026,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777169+0000","last_change":"2024-03-30T14:05:42.236006+0000","last_active":"2024-03-30T14:06:32.777169+0000","last_peered":"2024-03-30T14:06:32.777169+0000","last_clean":"2024-03-30T14:06:32.777169+0000","last_became_active":"2024-03-30T14:01:02.741907+0000","last_became_peered":"2024-03-30T14:01:02.741907+0000","last_unstale":"2024-03-30T14:06:32.777169+0000","last_undegraded":"2024-03-30T14:06:32.777169+0000","last_fullsized":"2024-03-30T14:06:32.777169+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:05:42.235898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:05:42.235898+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:56.772347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":465,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.588427+0000","last_change":"2024-03-30T14:06:12.469433+0000","last_active":"2024-03-30T14:06:32.588427+0000","last_peered":"2024-03-30T14:06:32.588427+0000","last_clean":"2024-03-30T14:06:32.588427+0000","last_became_active":"2024-03-30T14:06:12.469099+0000","last_became_peered":"2024-03-30T14:06:12.469099+0000","last_unstale":"2024-03-30T14:06:32.588427+0000","last_undegraded":"2024-03-30T14:06:32.588427+0000","last_fullsized":"2024-03-30T14:06:32.588427+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:43.671179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:05:43.671179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:53.994670+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"421'24","reported_seq":1104,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776562+0000","last_change":"2024-03-30T14:05:36.395782+0000","last_active":"2024-03-30T14:06:32.776562+0000","last_peered":"2024-03-30T14:06:32.776562+0000","last_clean":"2024-03-30T14:06:32.776562+0000","last_became_active":"2024-03-30T14:05:02.367149+0000","last_became_peered":"2024-03-30T14:05:02.367149+0000","last_unstale":"2024-03-30T14:06:32.776562+0000","last_undegraded":"2024-03-30T14:06:32.776562+0000","last_fullsized":"2024-03-30T14:06:32.776562+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'24","last_scrub_stamp":"2024-03-30T14:05:36.395661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:05:36.395661+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:48.653139+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":90,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777283+0000","last_change":"2024-03-30T14:05:46.080898+0000","last_active":"2024-03-30T14:06:32.777283+0000","last_peered":"2024-03-30T14:06:32.777283+0000","last_clean":"2024-03-30T14:06:32.777283+0000","last_became_active":"2024-03-30T14:05:10.386580+0000","last_became_peered":"2024-03-30T14:05:10.386580+0000","last_unstale":"2024-03-30T14:06:32.777283+0000","last_undegraded":"2024-03-30T14:06:32.777283+0000","last_fullsized":"2024-03-30T14:06:32.777283+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:46.080797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:05:46.080797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:59.736296+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":106,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.590153+0000","last_change":"2024-03-30T14:06:03.763278+0000","last_active":"2024-03-30T14:06:32.590153+0000","last_peered":"2024-03-30T14:06:32.590153+0000","last_clean":"2024-03-30T14:06:32.590153+0000","last_became_active":"2024-03-30T14:05:10.392291+0000","last_became_peered":"2024-03-30T14:05:10.392291+0000","last_unstale":"2024-03-30T14:06:32.590153+0000","last_undegraded":"2024-03-30T14:06:32.590153+0000","last_fullsized":"2024-03-30T14:06:32.590153+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:03.763173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:06:03.763173+0000","last_clean_scrub_stamp":"2024-03-30T14:06:03.763173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:21.054140+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"451'129","reported_seq":540,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.590534+0000","last_change":"2024-03-30T14:06:16.562827+0000","last_active":"2024-03-30T14:06:32.590534+0000","last_peered":"2024-03-30T14:06:32.590534+0000","last_clean":"2024-03-30T14:06:32.590534+0000","last_became_active":"2024-03-30T14:05:05.006173+0000","last_became_peered":"2024-03-30T14:05:05.006173+0000","last_unstale":"2024-03-30T14:06:32.590534+0000","last_undegraded":"2024-03-30T14:06:32.590534+0000","last_fullsized":"2024-03-30T14:06:32.590534+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":4,"last_scrub":"451'129","last_scrub_stamp":"2024-03-30T14:06:16.562720+0000","last_deep_scrub":"451'129","last_deep_scrub_stamp":"2024-03-30T14:06:16.562720+0000","last_clean_scrub_stamp":"2024-03-30T14:06:16.562720+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:24.560038+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"489'11","reported_seq":115,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:45.656957+0000","last_change":"2024-03-30T14:06:04.917412+0000","last_active":"2024-03-30T14:06:45.656957+0000","last_peered":"2024-03-30T14:06:45.656957+0000","last_clean":"2024-03-30T14:06:45.656957+0000","last_became_active":"2024-03-30T14:05:10.373980+0000","last_became_peered":"2024-03-30T14:05:10.373980+0000","last_unstale":"2024-03-30T14:06:45.656957+0000","last_undegraded":"2024-03-30T14:06:45.656957+0000","last_fullsized":"2024-03-30T14:06:45.656957+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'11","last_scrub_stamp":"2024-03-30T14:06:04.917261+0000","last_deep_scrub":"376'10","last_deep_scrub_stamp":"2024-03-30T14:01:08.312068+0000","last_clean_scrub_stamp":"2024-03-30T14:06:04.917261+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:30.390464+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"421'39","reported_seq":61,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.588568+0000","last_change":"2024-03-30T14:05:48.033917+0000","last_active":"2024-03-30T14:06:32.588568+0000","last_peered":"2024-03-30T14:06:32.588568+0000","last_clean":"2024-03-30T14:06:32.588568+0000","last_became_active":"2024-03-30T14:05:48.016442+0000","last_became_peered":"2024-03-30T14:05:48.016442+0000","last_unstale":"2024-03-30T14:06:32.588568+0000","last_undegraded":"2024-03-30T14:06:32.588568+0000","last_fullsized":"2024-03-30T14:06:32.588568+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'39","last_scrub_stamp":"2024-03-30T14:05:33.969365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:05:33.969365+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:50.585237+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"512'4","reported_seq":98,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777884+0000","last_change":"2024-03-30T14:05:54.975769+0000","last_active":"2024-03-30T14:06:32.777884+0000","last_peered":"2024-03-30T14:06:32.777884+0000","last_clean":"2024-03-30T14:06:32.777884+0000","last_became_active":"2024-03-30T14:05:10.378426+0000","last_became_peered":"2024-03-30T14:05:10.378426+0000","last_unstale":"2024-03-30T14:06:32.777884+0000","last_undegraded":"2024-03-30T14:06:32.777884+0000","last_fullsized":"2024-03-30T14:06:32.777884+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"512'4","last_scrub_stamp":"2024-03-30T14:05:54.975662+0000","last_deep_scrub":"512'4","last_deep_scrub_stamp":"2024-03-30T14:05:54.975662+0000","last_clean_scrub_stamp":"2024-03-30T14:05:54.975662+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:19.490729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":45,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.588723+0000","last_change":"2024-03-30T14:05:48.019211+0000","last_active":"2024-03-30T14:06:32.588723+0000","last_peered":"2024-03-30T14:06:32.588723+0000","last_clean":"2024-03-30T14:06:32.588723+0000","last_became_active":"2024-03-30T14:05:48.018859+0000","last_became_peered":"2024-03-30T14:05:48.018859+0000","last_unstale":"2024-03-30T14:06:32.588723+0000","last_undegraded":"2024-03-30T14:06:32.588723+0000","last_fullsized":"2024-03-30T14:06:32.588723+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:27.350301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:05:27.350301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:47.723085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"82'128","reported_seq":773,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776110+0000","last_change":"2024-03-30T14:06:00.944621+0000","last_active":"2024-03-30T14:06:32.776110+0000","last_peered":"2024-03-30T14:06:32.776110+0000","last_clean":"2024-03-30T14:06:32.776110+0000","last_became_active":"2024-03-30T14:05:13.375492+0000","last_became_peered":"2024-03-30T14:05:13.375492+0000","last_unstale":"2024-03-30T14:06:32.776110+0000","last_undegraded":"2024-03-30T14:06:32.776110+0000","last_fullsized":"2024-03-30T14:06:32.776110+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:00.944500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:06:00.944500+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:24.213862+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"421'21","reported_seq":122,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.590867+0000","last_change":"2024-03-30T14:06:05.603664+0000","last_active":"2024-03-30T14:06:32.590867+0000","last_peered":"2024-03-30T14:06:32.590867+0000","last_clean":"2024-03-30T14:06:32.590867+0000","last_became_active":"2024-03-30T14:05:10.386775+0000","last_became_peered":"2024-03-30T14:05:10.386775+0000","last_unstale":"2024-03-30T14:06:32.590867+0000","last_undegraded":"2024-03-30T14:06:32.590867+0000","last_fullsized":"2024-03-30T14:06:32.590867+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-30T14:06:05.603574+0000","last_deep_scrub":"316'15","last_deep_scrub_stamp":"2024-03-30T13:58:13.210392+0000","last_clean_scrub_stamp":"2024-03-30T14:06:05.603574+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:22.216211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1069,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:53.981505+0000","last_change":"2024-03-30T14:06:53.981527+0000","last_active":"2024-03-30T14:06:53.981505+0000","last_peered":"2024-03-30T14:06:53.981505+0000","last_clean":"2024-03-30T14:06:53.981505+0000","last_became_active":"2024-03-30T14:05:48.018661+0000","last_became_peered":"2024-03-30T14:05:48.018661+0000","last_unstale":"2024-03-30T14:06:53.981505+0000","last_undegraded":"2024-03-30T14:06:53.981505+0000","last_fullsized":"2024-03-30T14:06:53.981505+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:53.981447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:05:39.330317+0000","last_clean_scrub_stamp":"2024-03-30T14:06:53.981447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:57.421291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":45,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589253+0000","last_change":"2024-03-30T14:05:48.022745+0000","last_active":"2024-03-30T14:06:32.589253+0000","last_peered":"2024-03-30T14:06:32.589253+0000","last_clean":"2024-03-30T14:06:32.589253+0000","last_became_active":"2024-03-30T14:05:48.022330+0000","last_became_peered":"2024-03-30T14:05:48.022330+0000","last_unstale":"2024-03-30T14:06:32.589253+0000","last_undegraded":"2024-03-30T14:06:32.589253+0000","last_fullsized":"2024-03-30T14:06:32.589253+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:41.511271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:57:09.967517+0000","last_clean_scrub_stamp":"2024-03-30T14:05:41.511271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:58.436772+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"494'2","reported_seq":95,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776235+0000","last_change":"2024-03-30T14:05:19.855370+0000","last_active":"2024-03-30T14:06:32.776235+0000","last_peered":"2024-03-30T14:06:32.776235+0000","last_clean":"2024-03-30T14:06:32.776235+0000","last_became_active":"2024-03-30T14:05:10.393013+0000","last_became_peered":"2024-03-30T14:05:10.393013+0000","last_unstale":"2024-03-30T14:06:32.776235+0000","last_undegraded":"2024-03-30T14:06:32.776235+0000","last_fullsized":"2024-03-30T14:06:32.776235+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'2","last_scrub_stamp":"2024-03-30T14:05:19.855273+0000","last_deep_scrub":"494'2","last_deep_scrub_stamp":"2024-03-30T14:05:19.855273+0000","last_clean_scrub_stamp":"2024-03-30T14:05:19.855273+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:44.137966+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"82'128","reported_seq":760,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.588732+0000","last_change":"2024-03-30T14:06:09.972729+0000","last_active":"2024-03-30T14:06:32.588732+0000","last_peered":"2024-03-30T14:06:32.588732+0000","last_clean":"2024-03-30T14:06:32.588732+0000","last_became_active":"2024-03-30T14:05:51.005009+0000","last_became_peered":"2024-03-30T14:05:51.005009+0000","last_unstale":"2024-03-30T14:06:32.588732+0000","last_undegraded":"2024-03-30T14:06:32.588732+0000","last_fullsized":"2024-03-30T14:06:32.588732+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:09.972622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:06:09.972622+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:25.753999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"82'128","reported_seq":1025,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776635+0000","last_change":"2024-03-30T14:05:46.358651+0000","last_active":"2024-03-30T14:06:32.776635+0000","last_peered":"2024-03-30T14:06:32.776635+0000","last_clean":"2024-03-30T14:06:32.776635+0000","last_became_active":"2024-03-30T14:05:05.005735+0000","last_became_peered":"2024-03-30T14:05:05.005735+0000","last_unstale":"2024-03-30T14:06:32.776635+0000","last_undegraded":"2024-03-30T14:06:32.776635+0000","last_fullsized":"2024-03-30T14:06:32.776635+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:05:46.358558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:05:46.358558+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:13.773414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"421'22","reported_seq":112,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775951+0000","last_change":"2024-03-30T14:06:15.495396+0000","last_active":"2024-03-30T14:06:32.775951+0000","last_peered":"2024-03-30T14:06:32.775951+0000","last_clean":"2024-03-30T14:06:32.775951+0000","last_became_active":"2024-03-30T14:05:23.031448+0000","last_became_peered":"2024-03-30T14:05:23.031448+0000","last_unstale":"2024-03-30T14:06:32.775951+0000","last_undegraded":"2024-03-30T14:06:32.775951+0000","last_fullsized":"2024-03-30T14:06:32.775951+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'22","last_scrub_stamp":"2024-03-30T14:06:15.495262+0000","last_deep_scrub":"218'16","last_deep_scrub_stamp":"2024-03-30T13:52:59.883027+0000","last_clean_scrub_stamp":"2024-03-30T14:06:15.495262+0000","objects_scrubbed":4,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:40.696296+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":90,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775969+0000","last_change":"2024-03-30T14:05:45.084438+0000","last_active":"2024-03-30T14:06:32.775969+0000","last_peered":"2024-03-30T14:06:32.775969+0000","last_clean":"2024-03-30T14:06:32.775969+0000","last_became_active":"2024-03-30T14:05:10.372067+0000","last_became_peered":"2024-03-30T14:05:10.372067+0000","last_unstale":"2024-03-30T14:06:32.775969+0000","last_undegraded":"2024-03-30T14:06:32.775969+0000","last_fullsized":"2024-03-30T14:06:32.775969+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:45.083761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:05:45.083761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:47.159548+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":106,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.591023+0000","last_change":"2024-03-30T14:05:59.603064+0000","last_active":"2024-03-30T14:06:32.591023+0000","last_peered":"2024-03-30T14:06:32.591023+0000","last_clean":"2024-03-30T14:06:32.591023+0000","last_became_active":"2024-03-30T14:05:10.385463+0000","last_became_peered":"2024-03-30T14:05:10.385463+0000","last_unstale":"2024-03-30T14:06:32.591023+0000","last_undegraded":"2024-03-30T14:06:32.591023+0000","last_fullsized":"2024-03-30T14:06:32.591023+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:59.602977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:05:59.602977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:07.981574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":731,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589690+0000","last_change":"2024-03-30T14:06:22.915739+0000","last_active":"2024-03-30T14:06:32.589690+0000","last_peered":"2024-03-30T14:06:32.589690+0000","last_clean":"2024-03-30T14:06:32.589690+0000","last_became_active":"2024-03-30T14:06:12.472120+0000","last_became_peered":"2024-03-30T14:06:12.472120+0000","last_unstale":"2024-03-30T14:06:32.589690+0000","last_undegraded":"2024-03-30T14:06:32.589690+0000","last_fullsized":"2024-03-30T14:06:32.589690+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:22.915639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:22.915639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:38.700852+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"421'21","reported_seq":55,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589246+0000","last_change":"2024-03-30T14:05:48.030377+0000","last_active":"2024-03-30T14:06:32.589246+0000","last_peered":"2024-03-30T14:06:32.589246+0000","last_clean":"2024-03-30T14:06:32.589246+0000","last_became_active":"2024-03-30T14:05:48.015202+0000","last_became_peered":"2024-03-30T14:05:48.015202+0000","last_unstale":"2024-03-30T14:06:32.589246+0000","last_undegraded":"2024-03-30T14:06:32.589246+0000","last_fullsized":"2024-03-30T14:06:32.589246+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-30T14:05:22.816090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:05:22.816090+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:51.413393+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":768,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775253+0000","last_change":"2024-03-30T14:05:55.478858+0000","last_active":"2024-03-30T14:06:32.775253+0000","last_peered":"2024-03-30T14:06:32.775253+0000","last_clean":"2024-03-30T14:06:32.775253+0000","last_became_active":"2024-03-30T14:05:23.027423+0000","last_became_peered":"2024-03-30T14:05:23.027423+0000","last_unstale":"2024-03-30T14:06:32.775253+0000","last_undegraded":"2024-03-30T14:06:32.775253+0000","last_fullsized":"2024-03-30T14:06:32.775253+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:55.478773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:05:55.478773+0000","last_clean_scrub_stamp":"2024-03-30T14:05:55.478773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:17.735701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"489'18","reported_seq":136,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:45.656708+0000","last_change":"2024-03-30T14:05:43.545571+0000","last_active":"2024-03-30T14:06:45.656708+0000","last_peered":"2024-03-30T14:06:45.656708+0000","last_clean":"2024-03-30T14:06:45.656708+0000","last_became_active":"2024-03-30T14:05:10.386593+0000","last_became_peered":"2024-03-30T14:05:10.386593+0000","last_unstale":"2024-03-30T14:06:45.656708+0000","last_undegraded":"2024-03-30T14:06:45.656708+0000","last_fullsized":"2024-03-30T14:06:45.656708+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'18","last_scrub_stamp":"2024-03-30T14:05:43.545472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:05:43.545472+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:03.127404+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1015,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:41.461300+0000","last_change":"2024-03-30T14:06:41.461323+0000","last_active":"2024-03-30T14:06:41.461300+0000","last_peered":"2024-03-30T14:06:41.461300+0000","last_clean":"2024-03-30T14:06:41.461300+0000","last_became_active":"2024-03-30T14:05:48.011917+0000","last_became_peered":"2024-03-30T14:05:48.011917+0000","last_unstale":"2024-03-30T14:06:41.461300+0000","last_undegraded":"2024-03-30T14:06:41.461300+0000","last_fullsized":"2024-03-30T14:06:41.461300+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:41.461213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:41.461213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:53.113563+0000","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"82'128","reported_seq":680,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.588570+0000","last_change":"2024-03-30T14:06:18.965794+0000","last_active":"2024-03-30T14:06:32.588570+0000","last_peered":"2024-03-30T14:06:32.588570+0000","last_clean":"2024-03-30T14:06:32.588570+0000","last_became_active":"2024-03-30T14:05:51.005660+0000","last_became_peered":"2024-03-30T14:05:51.005660+0000","last_unstale":"2024-03-30T14:06:32.588570+0000","last_undegraded":"2024-03-30T14:06:32.588570+0000","last_fullsized":"2024-03-30T14:06:32.588570+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:18.965695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:06:18.965695+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:47.725375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"421'30","reported_seq":635,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589213+0000","last_change":"2024-03-30T14:06:28.955171+0000","last_active":"2024-03-30T14:06:32.589213+0000","last_peered":"2024-03-30T14:06:32.589213+0000","last_clean":"2024-03-30T14:06:32.589213+0000","last_became_active":"2024-03-30T14:06:12.469999+0000","last_became_peered":"2024-03-30T14:06:12.469999+0000","last_unstale":"2024-03-30T14:06:32.589213+0000","last_undegraded":"2024-03-30T14:06:32.589213+0000","last_fullsized":"2024-03-30T14:06:32.589213+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'30","last_scrub_stamp":"2024-03-30T14:06:28.955038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:06:28.955038+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:51.311210+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":90,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776215+0000","last_change":"2024-03-30T14:06:11.830739+0000","last_active":"2024-03-30T14:06:32.776215+0000","last_peered":"2024-03-30T14:06:32.776215+0000","last_clean":"2024-03-30T14:06:32.776215+0000","last_became_active":"2024-03-30T14:05:10.380610+0000","last_became_peered":"2024-03-30T14:05:10.380610+0000","last_unstale":"2024-03-30T14:06:32.776215+0000","last_undegraded":"2024-03-30T14:06:32.776215+0000","last_fullsized":"2024-03-30T14:06:32.776215+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:11.830638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:06:11.830638+0000","last_clean_scrub_stamp":"2024-03-30T14:06:11.830638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:31.923697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"510'15","reported_seq":71,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:45.656919+0000","last_change":"2024-03-30T14:05:49.086863+0000","last_active":"2024-03-30T14:06:45.656919+0000","last_peered":"2024-03-30T14:06:45.656919+0000","last_clean":"2024-03-30T14:06:45.656919+0000","last_became_active":"2024-03-30T14:05:48.021934+0000","last_became_peered":"2024-03-30T14:05:48.021934+0000","last_unstale":"2024-03-30T14:06:45.656919+0000","last_undegraded":"2024-03-30T14:06:45.656919+0000","last_fullsized":"2024-03-30T14:06:45.656919+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'14","last_scrub_stamp":"2024-03-30T14:05:49.086752+0000","last_deep_scrub":"509'14","last_deep_scrub_stamp":"2024-03-30T14:05:49.086752+0000","last_clean_scrub_stamp":"2024-03-30T14:05:49.086752+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:03.467752+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":764,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776518+0000","last_change":"2024-03-30T14:06:13.569464+0000","last_active":"2024-03-30T14:06:32.776518+0000","last_peered":"2024-03-30T14:06:32.776518+0000","last_clean":"2024-03-30T14:06:32.776518+0000","last_became_active":"2024-03-30T14:05:23.029918+0000","last_became_peered":"2024-03-30T14:05:23.029918+0000","last_unstale":"2024-03-30T14:06:32.776518+0000","last_undegraded":"2024-03-30T14:06:32.776518+0000","last_fullsized":"2024-03-30T14:06:32.776518+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:13.569359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:03:15.101611+0000","last_clean_scrub_stamp":"2024-03-30T14:06:13.569359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:32.621038+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"82'128","reported_seq":656,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775076+0000","last_change":"2024-03-30T14:05:51.011444+0000","last_active":"2024-03-30T14:06:32.775076+0000","last_peered":"2024-03-30T14:06:32.775076+0000","last_clean":"2024-03-30T14:06:32.775076+0000","last_became_active":"2024-03-30T14:05:51.011025+0000","last_became_peered":"2024-03-30T14:05:51.011025+0000","last_unstale":"2024-03-30T14:06:32.775076+0000","last_undegraded":"2024-03-30T14:06:32.775076+0000","last_fullsized":"2024-03-30T14:06:32.775076+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:05:32.677209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:05:32.677209+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:52.215324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"421'27","reported_seq":1056,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777950+0000","last_change":"2024-03-30T14:05:59.364261+0000","last_active":"2024-03-30T14:06:32.777950+0000","last_peered":"2024-03-30T14:06:32.777950+0000","last_clean":"2024-03-30T14:06:32.777950+0000","last_became_active":"2024-03-30T14:04:32.226780+0000","last_became_peered":"2024-03-30T14:04:32.226780+0000","last_unstale":"2024-03-30T14:06:32.777950+0000","last_undegraded":"2024-03-30T14:06:32.777950+0000","last_fullsized":"2024-03-30T14:06:32.777950+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'27","last_scrub_stamp":"2024-03-30T14:05:59.364079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:05:59.364079+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:17.464204+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1022,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777036+0000","last_change":"2024-03-30T14:05:50.415539+0000","last_active":"2024-03-30T14:06:32.777036+0000","last_peered":"2024-03-30T14:06:32.777036+0000","last_clean":"2024-03-30T14:06:32.777036+0000","last_became_active":"2024-03-30T14:05:10.383734+0000","last_became_peered":"2024-03-30T14:05:10.383734+0000","last_unstale":"2024-03-30T14:06:32.777036+0000","last_undegraded":"2024-03-30T14:06:32.777036+0000","last_fullsized":"2024-03-30T14:06:32.777036+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:50.415461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:05:50.415461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:04.336645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1039,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775831+0000","last_change":"2024-03-30T14:06:28.345633+0000","last_active":"2024-03-30T14:06:32.775831+0000","last_peered":"2024-03-30T14:06:32.775831+0000","last_clean":"2024-03-30T14:06:32.775831+0000","last_became_active":"2024-03-30T14:05:02.377052+0000","last_became_peered":"2024-03-30T14:05:02.377052+0000","last_unstale":"2024-03-30T14:06:32.775831+0000","last_undegraded":"2024-03-30T14:06:32.775831+0000","last_fullsized":"2024-03-30T14:06:32.775831+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:28.345543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:34.815783+0000","last_clean_scrub_stamp":"2024-03-30T14:06:28.345543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:53.918070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":106,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589143+0000","last_change":"2024-03-30T14:05:48.017829+0000","last_active":"2024-03-30T14:06:32.589143+0000","last_peered":"2024-03-30T14:06:32.589143+0000","last_clean":"2024-03-30T14:06:32.589143+0000","last_became_active":"2024-03-30T14:05:48.017431+0000","last_became_peered":"2024-03-30T14:05:48.017431+0000","last_unstale":"2024-03-30T14:06:32.589143+0000","last_undegraded":"2024-03-30T14:06:32.589143+0000","last_fullsized":"2024-03-30T14:06:32.589143+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:39.545440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:24.598976+0000","last_clean_scrub_stamp":"2024-03-30T14:05:39.545440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:06.459494+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"421'21","reported_seq":1065,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775938+0000","last_change":"2024-03-30T14:06:00.204082+0000","last_active":"2024-03-30T14:06:32.775938+0000","last_peered":"2024-03-30T14:06:32.775938+0000","last_clean":"2024-03-30T14:06:32.775938+0000","last_became_active":"2024-03-30T14:04:54.327260+0000","last_became_peered":"2024-03-30T14:04:54.327260+0000","last_unstale":"2024-03-30T14:06:32.775938+0000","last_undegraded":"2024-03-30T14:06:32.775938+0000","last_fullsized":"2024-03-30T14:06:32.775938+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-30T14:06:00.203989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:06:00.203989+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:11.179397+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":98,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776000+0000","last_change":"2024-03-30T14:06:27.814638+0000","last_active":"2024-03-30T14:06:32.776000+0000","last_peered":"2024-03-30T14:06:32.776000+0000","last_clean":"2024-03-30T14:06:32.776000+0000","last_became_active":"2024-03-30T14:05:10.393274+0000","last_became_peered":"2024-03-30T14:05:10.393274+0000","last_unstale":"2024-03-30T14:06:32.776000+0000","last_undegraded":"2024-03-30T14:06:32.776000+0000","last_fullsized":"2024-03-30T14:06:32.776000+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:27.814502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:06:27.814502+0000","last_clean_scrub_stamp":"2024-03-30T14:06:27.814502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:35.697945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":100,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.590236+0000","last_change":"2024-03-30T14:06:06.583759+0000","last_active":"2024-03-30T14:06:32.590236+0000","last_peered":"2024-03-30T14:06:32.590236+0000","last_clean":"2024-03-30T14:06:32.590236+0000","last_became_active":"2024-03-30T14:05:02.373957+0000","last_became_peered":"2024-03-30T14:05:02.373957+0000","last_unstale":"2024-03-30T14:06:32.590236+0000","last_undegraded":"2024-03-30T14:06:32.590236+0000","last_fullsized":"2024-03-30T14:06:32.590236+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:06.583675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:06:06.583675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:18.792149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":784,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:46.502034+0000","last_change":"2024-03-30T14:06:46.502074+0000","last_active":"2024-03-30T14:06:46.502034+0000","last_peered":"2024-03-30T14:06:46.502034+0000","last_clean":"2024-03-30T14:06:46.502034+0000","last_became_active":"2024-03-30T14:05:48.023903+0000","last_became_peered":"2024-03-30T14:05:48.023903+0000","last_unstale":"2024-03-30T14:06:46.502034+0000","last_undegraded":"2024-03-30T14:06:46.502034+0000","last_fullsized":"2024-03-30T14:06:46.502034+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:46.501948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:28.314391+0000","last_clean_scrub_stamp":"2024-03-30T14:06:46.501948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:05.312856+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"82'128","reported_seq":961,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589834+0000","last_change":"2024-03-30T14:06:19.866986+0000","last_active":"2024-03-30T14:06:32.589834+0000","last_peered":"2024-03-30T14:06:32.589834+0000","last_clean":"2024-03-30T14:06:32.589834+0000","last_became_active":"2024-03-30T14:06:12.472272+0000","last_became_peered":"2024-03-30T14:06:12.472272+0000","last_unstale":"2024-03-30T14:06:32.589834+0000","last_undegraded":"2024-03-30T14:06:32.589834+0000","last_fullsized":"2024-03-30T14:06:32.589834+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:19.866782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:06:19.866782+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:43.744359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"421'27","reported_seq":130,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589558+0000","last_change":"2024-03-30T14:06:31.491589+0000","last_active":"2024-03-30T14:06:32.589558+0000","last_peered":"2024-03-30T14:06:32.589558+0000","last_clean":"2024-03-30T14:06:32.589558+0000","last_became_active":"2024-03-30T14:05:10.385458+0000","last_became_peered":"2024-03-30T14:05:10.385458+0000","last_unstale":"2024-03-30T14:06:32.589558+0000","last_undegraded":"2024-03-30T14:06:32.589558+0000","last_fullsized":"2024-03-30T14:06:32.589558+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'27","last_scrub_stamp":"2024-03-30T14:06:31.491495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:06:31.491495+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:46.790429+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":53,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589131+0000","last_change":"2024-03-30T14:06:31.869847+0000","last_active":"2024-03-30T14:06:32.589131+0000","last_peered":"2024-03-30T14:06:32.589131+0000","last_clean":"2024-03-30T14:06:32.589131+0000","last_became_active":"2024-03-30T14:05:48.022381+0000","last_became_peered":"2024-03-30T14:05:48.022381+0000","last_unstale":"2024-03-30T14:06:32.589131+0000","last_undegraded":"2024-03-30T14:06:32.589131+0000","last_fullsized":"2024-03-30T14:06:32.589131+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:31.869751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:25.494903+0000","last_clean_scrub_stamp":"2024-03-30T14:06:31.869751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:58.784022+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":90,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775366+0000","last_change":"2024-03-30T14:05:51.054553+0000","last_active":"2024-03-30T14:06:32.775366+0000","last_peered":"2024-03-30T14:06:32.775366+0000","last_clean":"2024-03-30T14:06:32.775366+0000","last_became_active":"2024-03-30T14:05:10.380531+0000","last_became_peered":"2024-03-30T14:05:10.380531+0000","last_unstale":"2024-03-30T14:06:32.775366+0000","last_undegraded":"2024-03-30T14:06:32.775366+0000","last_fullsized":"2024-03-30T14:06:32.775366+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:51.054480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:42.616155+0000","last_clean_scrub_stamp":"2024-03-30T14:05:51.054480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:59.439677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":100,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589986+0000","last_change":"2024-03-30T14:06:10.742316+0000","last_active":"2024-03-30T14:06:32.589986+0000","last_peered":"2024-03-30T14:06:32.589986+0000","last_clean":"2024-03-30T14:06:32.589986+0000","last_became_active":"2024-03-30T14:05:02.374776+0000","last_became_peered":"2024-03-30T14:05:02.374776+0000","last_unstale":"2024-03-30T14:06:32.589986+0000","last_undegraded":"2024-03-30T14:06:32.589986+0000","last_fullsized":"2024-03-30T14:06:32.589986+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:10.742188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:10.742188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:12.829107+0000","scrub_duration":146,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"503'14","reported_seq":116,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:45.270770+0000","last_change":"2024-03-30T14:06:45.270798+0000","last_active":"2024-03-30T14:06:45.270770+0000","last_peered":"2024-03-30T14:06:45.270770+0000","last_clean":"2024-03-30T14:06:45.270770+0000","last_became_active":"2024-03-30T14:05:02.373527+0000","last_became_peered":"2024-03-30T14:05:02.373527+0000","last_unstale":"2024-03-30T14:06:45.270770+0000","last_undegraded":"2024-03-30T14:06:45.270770+0000","last_fullsized":"2024-03-30T14:06:45.270770+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'14","last_scrub_stamp":"2024-03-30T14:06:45.270708+0000","last_deep_scrub":"503'14","last_deep_scrub_stamp":"2024-03-30T14:05:38.484297+0000","last_clean_scrub_stamp":"2024-03-30T14:06:45.270708+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:45.854051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":788,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775672+0000","last_change":"2024-03-30T14:06:08.523691+0000","last_active":"2024-03-30T14:06:32.775672+0000","last_peered":"2024-03-30T14:06:32.775672+0000","last_clean":"2024-03-30T14:06:32.775672+0000","last_became_active":"2024-03-30T14:05:48.023811+0000","last_became_peered":"2024-03-30T14:05:48.023811+0000","last_unstale":"2024-03-30T14:06:32.775672+0000","last_undegraded":"2024-03-30T14:06:32.775672+0000","last_fullsized":"2024-03-30T14:06:32.775672+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:08.523576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:06:08.523576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:31.810652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1037,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775535+0000","last_change":"2024-03-30T14:06:25.993285+0000","last_active":"2024-03-30T14:06:32.775535+0000","last_peered":"2024-03-30T14:06:32.775535+0000","last_clean":"2024-03-30T14:06:32.775535+0000","last_became_active":"2024-03-30T14:05:10.382829+0000","last_became_peered":"2024-03-30T14:05:10.382829+0000","last_unstale":"2024-03-30T14:06:32.775535+0000","last_undegraded":"2024-03-30T14:06:32.775535+0000","last_fullsized":"2024-03-30T14:06:32.775535+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:25.993110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:06:25.993110+0000","last_clean_scrub_stamp":"2024-03-30T14:06:25.993110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:44.320090+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":90,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775267+0000","last_change":"2024-03-30T14:06:00.002874+0000","last_active":"2024-03-30T14:06:32.775267+0000","last_peered":"2024-03-30T14:06:32.775267+0000","last_clean":"2024-03-30T14:06:32.775267+0000","last_became_active":"2024-03-30T14:05:10.373968+0000","last_became_peered":"2024-03-30T14:05:10.373968+0000","last_unstale":"2024-03-30T14:06:32.775267+0000","last_undegraded":"2024-03-30T14:06:32.775267+0000","last_fullsized":"2024-03-30T14:06:32.775267+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:00.002758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:00.002758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:22.058916+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"421'27","reported_seq":1093,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777543+0000","last_change":"2024-03-30T14:06:27.029077+0000","last_active":"2024-03-30T14:06:32.777543+0000","last_peered":"2024-03-30T14:06:32.777543+0000","last_clean":"2024-03-30T14:06:32.777543+0000","last_became_active":"2024-03-30T13:58:47.204696+0000","last_became_peered":"2024-03-30T13:58:47.204696+0000","last_unstale":"2024-03-30T14:06:32.777543+0000","last_undegraded":"2024-03-30T14:06:32.777543+0000","last_fullsized":"2024-03-30T14:06:32.777543+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'27","last_scrub_stamp":"2024-03-30T14:06:27.028900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:06:27.028900+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:34.843559+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":100,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.588248+0000","last_change":"2024-03-30T14:05:58.604822+0000","last_active":"2024-03-30T14:06:32.588248+0000","last_peered":"2024-03-30T14:06:32.588248+0000","last_clean":"2024-03-30T14:06:32.588248+0000","last_became_active":"2024-03-30T14:05:02.378933+0000","last_became_peered":"2024-03-30T14:05:02.378933+0000","last_unstale":"2024-03-30T14:06:32.588248+0000","last_undegraded":"2024-03-30T14:06:32.588248+0000","last_fullsized":"2024-03-30T14:06:32.588248+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:58.604739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:05:58.604739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:27.526047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":765,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:37.465699+0000","last_change":"2024-03-30T14:06:37.465721+0000","last_active":"2024-03-30T14:06:37.465699+0000","last_peered":"2024-03-30T14:06:37.465699+0000","last_clean":"2024-03-30T14:06:37.465699+0000","last_became_active":"2024-03-30T14:05:23.028766+0000","last_became_peered":"2024-03-30T14:05:23.028766+0000","last_unstale":"2024-03-30T14:06:37.465699+0000","last_undegraded":"2024-03-30T14:06:37.465699+0000","last_fullsized":"2024-03-30T14:06:37.465699+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:37.465634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:06:37.465634+0000","last_clean_scrub_stamp":"2024-03-30T14:06:37.465634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:42.555255+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1016,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:46.497543+0000","last_change":"2024-03-30T14:06:46.497583+0000","last_active":"2024-03-30T14:06:46.497543+0000","last_peered":"2024-03-30T14:06:46.497543+0000","last_clean":"2024-03-30T14:06:46.497543+0000","last_became_active":"2024-03-30T13:58:48.180840+0000","last_became_peered":"2024-03-30T13:58:48.180840+0000","last_unstale":"2024-03-30T14:06:46.497543+0000","last_undegraded":"2024-03-30T14:06:46.497543+0000","last_fullsized":"2024-03-30T14:06:46.497543+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:46.497454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:05:19.446184+0000","last_clean_scrub_stamp":"2024-03-30T14:06:46.497454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:08.898646+0000","scrub_duration":209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"421'35","reported_seq":538,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:40.494189+0000","last_change":"2024-03-30T14:06:40.494221+0000","last_active":"2024-03-30T14:06:40.494189+0000","last_peered":"2024-03-30T14:06:40.494189+0000","last_clean":"2024-03-30T14:06:40.494189+0000","last_became_active":"2024-03-30T14:05:23.032100+0000","last_became_peered":"2024-03-30T14:05:23.032100+0000","last_unstale":"2024-03-30T14:06:40.494189+0000","last_undegraded":"2024-03-30T14:06:40.494189+0000","last_fullsized":"2024-03-30T14:06:40.494189+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'35","last_scrub_stamp":"2024-03-30T14:06:40.494130+0000","last_deep_scrub":"421'35","last_deep_scrub_stamp":"2024-03-30T14:05:15.761820+0000","last_clean_scrub_stamp":"2024-03-30T14:06:40.494130+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:55.727198+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"65'1","reported_seq":1039,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:34.429227+0000","last_change":"2024-03-30T14:06:34.429254+0000","last_active":"2024-03-30T14:06:34.429227+0000","last_peered":"2024-03-30T14:06:34.429227+0000","last_clean":"2024-03-30T14:06:34.429227+0000","last_became_active":"2024-03-30T14:05:02.375804+0000","last_became_peered":"2024-03-30T14:05:02.375804+0000","last_unstale":"2024-03-30T14:06:34.429227+0000","last_undegraded":"2024-03-30T14:06:34.429227+0000","last_fullsized":"2024-03-30T14:06:34.429227+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T14:06:34.429161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:06:34.429161+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:35.338490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"489'15","reported_seq":1365,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:45.656942+0000","last_change":"2024-03-30T14:06:33.980613+0000","last_active":"2024-03-30T14:06:45.656942+0000","last_peered":"2024-03-30T14:06:45.656942+0000","last_clean":"2024-03-30T14:06:45.656942+0000","last_became_active":"2024-03-30T14:05:10.388934+0000","last_became_peered":"2024-03-30T14:05:10.388934+0000","last_unstale":"2024-03-30T14:06:45.656942+0000","last_undegraded":"2024-03-30T14:06:45.656942+0000","last_fullsized":"2024-03-30T14:06:45.656942+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'15","last_scrub_stamp":"2024-03-30T14:06:33.980502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:06:33.980502+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:36.485976+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":94,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775775+0000","last_change":"2024-03-30T14:05:53.028418+0000","last_active":"2024-03-30T14:06:32.775775+0000","last_peered":"2024-03-30T14:06:32.775775+0000","last_clean":"2024-03-30T14:06:32.775775+0000","last_became_active":"2024-03-30T14:05:10.381039+0000","last_became_peered":"2024-03-30T14:05:10.381039+0000","last_unstale":"2024-03-30T14:06:32.775775+0000","last_undegraded":"2024-03-30T14:06:32.775775+0000","last_fullsized":"2024-03-30T14:06:32.775775+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T14:05:53.028321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:05:53.028321+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:02.843553+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1850,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"421'29","reported_seq":120,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.590298+0000","last_change":"2024-03-30T14:06:30.511782+0000","last_active":"2024-03-30T14:06:32.590298+0000","last_peered":"2024-03-30T14:06:32.590298+0000","last_clean":"2024-03-30T14:06:32.590298+0000","last_became_active":"2024-03-30T14:05:02.370465+0000","last_became_peered":"2024-03-30T14:05:02.370465+0000","last_unstale":"2024-03-30T14:06:32.590298+0000","last_undegraded":"2024-03-30T14:06:32.590298+0000","last_fullsized":"2024-03-30T14:06:32.590298+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'29","last_scrub_stamp":"2024-03-30T14:06:30.511684+0000","last_deep_scrub":"421'29","last_deep_scrub_stamp":"2024-03-30T14:06:30.511684+0000","last_clean_scrub_stamp":"2024-03-30T14:06:30.511684+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:56.873539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":998,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776641+0000","last_change":"2024-03-30T14:06:18.317210+0000","last_active":"2024-03-30T14:06:32.776641+0000","last_peered":"2024-03-30T14:06:32.776641+0000","last_clean":"2024-03-30T14:06:32.776641+0000","last_became_active":"2024-03-30T14:04:54.310132+0000","last_became_peered":"2024-03-30T14:04:54.310132+0000","last_unstale":"2024-03-30T14:06:32.776641+0000","last_undegraded":"2024-03-30T14:06:32.776641+0000","last_fullsized":"2024-03-30T14:06:32.776641+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:18.317123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:06:18.317123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:21.307984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1002,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:33.313795+0000","last_change":"2024-03-30T14:06:33.313843+0000","last_active":"2024-03-30T14:06:33.313795+0000","last_peered":"2024-03-30T14:06:33.313795+0000","last_clean":"2024-03-30T14:06:33.313795+0000","last_became_active":"2024-03-30T14:05:02.366337+0000","last_became_peered":"2024-03-30T14:05:02.366337+0000","last_unstale":"2024-03-30T14:06:33.313795+0000","last_undegraded":"2024-03-30T14:06:33.313795+0000","last_fullsized":"2024-03-30T14:06:33.313795+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:33.313697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:06:33.313697+0000","last_clean_scrub_stamp":"2024-03-30T14:06:33.313697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:59.275008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":106,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.590769+0000","last_change":"2024-03-30T14:06:12.823187+0000","last_active":"2024-03-30T14:06:32.590769+0000","last_peered":"2024-03-30T14:06:32.590769+0000","last_clean":"2024-03-30T14:06:32.590769+0000","last_became_active":"2024-03-30T14:05:48.013445+0000","last_became_peered":"2024-03-30T14:05:48.013445+0000","last_unstale":"2024-03-30T14:06:32.590769+0000","last_undegraded":"2024-03-30T14:06:32.590769+0000","last_fullsized":"2024-03-30T14:06:32.590769+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:12.823088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:12.823088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:22.224843+0000","scrub_duration":226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"507'46","reported_seq":926,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589316+0000","last_change":"2024-03-30T14:06:12.468173+0000","last_active":"2024-03-30T14:06:32.589316+0000","last_peered":"2024-03-30T14:06:32.589316+0000","last_clean":"2024-03-30T14:06:32.589316+0000","last_became_active":"2024-03-30T14:06:12.467794+0000","last_became_peered":"2024-03-30T14:06:12.467794+0000","last_unstale":"2024-03-30T14:06:32.589316+0000","last_undegraded":"2024-03-30T14:06:32.589316+0000","last_fullsized":"2024-03-30T14:06:32.589316+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'46","last_scrub_stamp":"2024-03-30T14:05:47.333753+0000","last_deep_scrub":"507'46","last_deep_scrub_stamp":"2024-03-30T14:05:47.333753+0000","last_clean_scrub_stamp":"2024-03-30T14:05:47.333753+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:12.743805+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":768,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775563+0000","last_change":"2024-03-30T14:05:54.490003+0000","last_active":"2024-03-30T14:06:32.775563+0000","last_peered":"2024-03-30T14:06:32.775563+0000","last_clean":"2024-03-30T14:06:32.775563+0000","last_became_active":"2024-03-30T14:05:23.028851+0000","last_became_peered":"2024-03-30T14:05:23.028851+0000","last_unstale":"2024-03-30T14:06:32.775563+0000","last_undegraded":"2024-03-30T14:06:32.775563+0000","last_fullsized":"2024-03-30T14:06:32.775563+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:54.489923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:43.088964+0000","last_clean_scrub_stamp":"2024-03-30T14:05:54.489923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:06.714551+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1010,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776990+0000","last_change":"2024-03-30T14:05:45.378091+0000","last_active":"2024-03-30T14:06:32.776990+0000","last_peered":"2024-03-30T14:06:32.776990+0000","last_clean":"2024-03-30T14:06:32.776990+0000","last_became_active":"2024-03-30T13:42:49.322665+0000","last_became_peered":"2024-03-30T13:42:49.322665+0000","last_unstale":"2024-03-30T14:06:32.776990+0000","last_undegraded":"2024-03-30T14:06:32.776990+0000","last_fullsized":"2024-03-30T14:06:32.776990+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:45.377972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:30.465974+0000","last_clean_scrub_stamp":"2024-03-30T14:05:45.377972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:03.313779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1014,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.778136+0000","last_change":"2024-03-30T14:05:48.017012+0000","last_active":"2024-03-30T14:06:32.778136+0000","last_peered":"2024-03-30T14:06:32.778136+0000","last_clean":"2024-03-30T14:06:32.778136+0000","last_became_active":"2024-03-30T14:05:48.016496+0000","last_became_peered":"2024-03-30T14:05:48.016496+0000","last_unstale":"2024-03-30T14:06:32.778136+0000","last_undegraded":"2024-03-30T14:06:32.778136+0000","last_fullsized":"2024-03-30T14:06:32.778136+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:46.468684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:05:46.468684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:11.981952+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"421'11","reported_seq":1075,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777563+0000","last_change":"2024-03-30T14:05:47.411296+0000","last_active":"2024-03-30T14:06:32.777563+0000","last_peered":"2024-03-30T14:06:32.777563+0000","last_clean":"2024-03-30T14:06:32.777563+0000","last_became_active":"2024-03-30T14:05:02.355441+0000","last_became_peered":"2024-03-30T14:05:02.355441+0000","last_unstale":"2024-03-30T14:06:32.777563+0000","last_undegraded":"2024-03-30T14:06:32.777563+0000","last_fullsized":"2024-03-30T14:06:32.777563+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'11","last_scrub_stamp":"2024-03-30T14:05:47.411182+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-30T13:43:50.960980+0000","last_clean_scrub_stamp":"2024-03-30T14:05:47.411182+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:49.839380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"186'2","reported_seq":440,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589389+0000","last_change":"2024-03-30T14:06:06.924072+0000","last_active":"2024-03-30T14:06:32.589389+0000","last_peered":"2024-03-30T14:06:32.589389+0000","last_clean":"2024-03-30T14:06:32.589389+0000","last_became_active":"2024-03-30T13:58:47.193793+0000","last_became_peered":"2024-03-30T13:58:47.193793+0000","last_unstale":"2024-03-30T14:06:32.589389+0000","last_undegraded":"2024-03-30T14:06:32.589389+0000","last_fullsized":"2024-03-30T14:06:32.589389+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-30T14:06:06.923970+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-30T13:51:15.932076+0000","last_clean_scrub_stamp":"2024-03-30T14:06:06.923970+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:31.937671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":90,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775698+0000","last_change":"2024-03-30T14:05:56.004008+0000","last_active":"2024-03-30T14:06:32.775698+0000","last_peered":"2024-03-30T14:06:32.775698+0000","last_clean":"2024-03-30T14:06:32.775698+0000","last_became_active":"2024-03-30T14:05:10.378262+0000","last_became_peered":"2024-03-30T14:05:10.378262+0000","last_unstale":"2024-03-30T14:06:32.775698+0000","last_undegraded":"2024-03-30T14:06:32.775698+0000","last_fullsized":"2024-03-30T14:06:32.775698+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:56.003910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:41.633284+0000","last_clean_scrub_stamp":"2024-03-30T14:05:56.003910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:04.783154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":53,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589039+0000","last_change":"2024-03-30T14:06:06.031064+0000","last_active":"2024-03-30T14:06:32.589039+0000","last_peered":"2024-03-30T14:06:32.589039+0000","last_clean":"2024-03-30T14:06:32.589039+0000","last_became_active":"2024-03-30T14:05:48.020210+0000","last_became_peered":"2024-03-30T14:05:48.020210+0000","last_unstale":"2024-03-30T14:06:32.589039+0000","last_undegraded":"2024-03-30T14:06:32.589039+0000","last_fullsized":"2024-03-30T14:06:32.589039+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:06.030953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:06.030953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:19.208110+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"525'25","reported_seq":1115,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:49.265722+0000","last_change":"2024-03-30T14:06:49.265774+0000","last_active":"2024-03-30T14:06:49.265722+0000","last_peered":"2024-03-30T14:06:49.265722+0000","last_clean":"2024-03-30T14:06:49.265722+0000","last_became_active":"2024-03-30T14:05:31.110927+0000","last_became_peered":"2024-03-30T14:05:31.110927+0000","last_unstale":"2024-03-30T14:06:49.265722+0000","last_undegraded":"2024-03-30T14:06:49.265722+0000","last_fullsized":"2024-03-30T14:06:49.265722+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"525'25","last_scrub_stamp":"2024-03-30T14:06:49.265649+0000","last_deep_scrub":"525'25","last_deep_scrub_stamp":"2024-03-30T14:06:49.265649+0000","last_clean_scrub_stamp":"2024-03-30T14:06:49.265649+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:50.035575+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":450,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:44.968678+0000","last_change":"2024-03-30T14:06:44.968702+0000","last_active":"2024-03-30T14:06:44.968678+0000","last_peered":"2024-03-30T14:06:44.968678+0000","last_clean":"2024-03-30T14:06:44.968678+0000","last_became_active":"2024-03-30T14:05:31.110665+0000","last_became_peered":"2024-03-30T14:05:31.110665+0000","last_unstale":"2024-03-30T14:06:44.968678+0000","last_undegraded":"2024-03-30T14:06:44.968678+0000","last_fullsized":"2024-03-30T14:06:44.968678+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:44.968609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:37.513596+0000","last_clean_scrub_stamp":"2024-03-30T14:06:44.968609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:12.814206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":53,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:48.968294+0000","last_change":"2024-03-30T14:06:48.968318+0000","last_active":"2024-03-30T14:06:48.968294+0000","last_peered":"2024-03-30T14:06:48.968294+0000","last_clean":"2024-03-30T14:06:48.968294+0000","last_became_active":"2024-03-30T14:05:48.019011+0000","last_became_peered":"2024-03-30T14:05:48.019011+0000","last_unstale":"2024-03-30T14:06:48.968294+0000","last_undegraded":"2024-03-30T14:06:48.968294+0000","last_fullsized":"2024-03-30T14:06:48.968294+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:48.968218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:20.069864+0000","last_clean_scrub_stamp":"2024-03-30T14:06:48.968218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:10.088088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":756,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775310+0000","last_change":"2024-03-30T14:05:42.706598+0000","last_active":"2024-03-30T14:06:32.775310+0000","last_peered":"2024-03-30T14:06:32.775310+0000","last_clean":"2024-03-30T14:06:32.775310+0000","last_became_active":"2024-03-30T14:05:23.029117+0000","last_became_peered":"2024-03-30T14:05:23.029117+0000","last_unstale":"2024-03-30T14:06:32.775310+0000","last_undegraded":"2024-03-30T14:06:32.775310+0000","last_fullsized":"2024-03-30T14:06:32.775310+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:42.706464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:27.061281+0000","last_clean_scrub_stamp":"2024-03-30T14:05:42.706464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:11.966576+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"421'46","reported_seq":116,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589249+0000","last_change":"2024-03-30T14:05:44.527597+0000","last_active":"2024-03-30T14:06:32.589249+0000","last_peered":"2024-03-30T14:06:32.589249+0000","last_clean":"2024-03-30T14:06:32.589249+0000","last_became_active":"2024-03-30T14:05:02.387256+0000","last_became_peered":"2024-03-30T14:05:02.387256+0000","last_unstale":"2024-03-30T14:06:32.589249+0000","last_undegraded":"2024-03-30T14:06:32.589249+0000","last_fullsized":"2024-03-30T14:06:32.589249+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'46","last_scrub_stamp":"2024-03-30T14:05:44.527472+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:29.151791+0000","last_clean_scrub_stamp":"2024-03-30T14:05:44.527472+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:02.849183+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":108,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:39.360721+0000","last_change":"2024-03-30T14:06:39.360743+0000","last_active":"2024-03-30T14:06:39.360721+0000","last_peered":"2024-03-30T14:06:39.360721+0000","last_clean":"2024-03-30T14:06:39.360721+0000","last_became_active":"2024-03-30T14:05:02.385930+0000","last_became_peered":"2024-03-30T14:05:02.385930+0000","last_unstale":"2024-03-30T14:06:39.360721+0000","last_undegraded":"2024-03-30T14:06:39.360721+0000","last_fullsized":"2024-03-30T14:06:39.360721+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:39.360648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:20.499768+0000","last_clean_scrub_stamp":"2024-03-30T14:06:39.360648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:56.678299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":985,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775134+0000","last_change":"2024-03-30T14:06:07.278680+0000","last_active":"2024-03-30T14:06:32.775134+0000","last_peered":"2024-03-30T14:06:32.775134+0000","last_clean":"2024-03-30T14:06:32.775134+0000","last_became_active":"2024-03-30T14:04:32.203392+0000","last_became_peered":"2024-03-30T14:04:32.203392+0000","last_unstale":"2024-03-30T14:06:32.775134+0000","last_undegraded":"2024-03-30T14:06:32.775134+0000","last_fullsized":"2024-03-30T14:06:32.775134+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:07.278519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:06:07.278519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:12.295102+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":106,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589889+0000","last_change":"2024-03-30T14:06:04.777125+0000","last_active":"2024-03-30T14:06:32.589889+0000","last_peered":"2024-03-30T14:06:32.589889+0000","last_clean":"2024-03-30T14:06:32.589889+0000","last_became_active":"2024-03-30T14:05:10.385006+0000","last_became_peered":"2024-03-30T14:05:10.385006+0000","last_unstale":"2024-03-30T14:06:32.589889+0000","last_undegraded":"2024-03-30T14:06:32.589889+0000","last_fullsized":"2024-03-30T14:06:32.589889+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:04.776993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:04.776993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:17.047662+0000","scrub_duration":173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"421'21","reported_seq":1039,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775141+0000","last_change":"2024-03-30T14:05:51.406671+0000","last_active":"2024-03-30T14:06:32.775141+0000","last_peered":"2024-03-30T14:06:32.775141+0000","last_clean":"2024-03-30T14:06:32.775141+0000","last_became_active":"2024-03-30T14:05:31.111349+0000","last_became_peered":"2024-03-30T14:05:31.111349+0000","last_unstale":"2024-03-30T14:06:32.775141+0000","last_undegraded":"2024-03-30T14:06:32.775141+0000","last_fullsized":"2024-03-30T14:06:32.775141+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-30T14:05:51.406568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:05:51.406568+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:20.524986+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"65'1","reported_seq":789,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775438+0000","last_change":"2024-03-30T14:05:56.474530+0000","last_active":"2024-03-30T14:06:32.775438+0000","last_peered":"2024-03-30T14:06:32.775438+0000","last_clean":"2024-03-30T14:06:32.775438+0000","last_became_active":"2024-03-30T14:05:48.023279+0000","last_became_peered":"2024-03-30T14:05:48.023279+0000","last_unstale":"2024-03-30T14:06:32.775438+0000","last_undegraded":"2024-03-30T14:06:32.775438+0000","last_fullsized":"2024-03-30T14:06:32.775438+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T14:05:56.474240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:05:56.474240+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:10.291749+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1048,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775107+0000","last_change":"2024-03-30T14:06:05.161540+0000","last_active":"2024-03-30T14:06:32.775107+0000","last_peered":"2024-03-30T14:06:32.775107+0000","last_clean":"2024-03-30T14:06:32.775107+0000","last_became_active":"2024-03-30T14:05:10.382228+0000","last_became_peered":"2024-03-30T14:05:10.382228+0000","last_unstale":"2024-03-30T14:06:32.775107+0000","last_undegraded":"2024-03-30T14:06:32.775107+0000","last_fullsized":"2024-03-30T14:06:32.775107+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:05.161455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:17.548567+0000","last_clean_scrub_stamp":"2024-03-30T14:06:05.161455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:05.767872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1017,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776236+0000","last_change":"2024-03-30T14:05:56.506831+0000","last_active":"2024-03-30T14:06:32.776236+0000","last_peered":"2024-03-30T14:06:32.776236+0000","last_clean":"2024-03-30T14:06:32.776236+0000","last_became_active":"2024-03-30T14:05:48.009068+0000","last_became_peered":"2024-03-30T14:05:48.009068+0000","last_unstale":"2024-03-30T14:06:32.776236+0000","last_undegraded":"2024-03-30T14:06:32.776236+0000","last_fullsized":"2024-03-30T14:06:32.776236+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:56.506725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:05:56.506725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:58.173557+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":105,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1050752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2689,"ondisk_log_size":2689,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2332,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1537,"num_read_kb":1294,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":565,"num_bytes_recovered":12592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1059,"ondisk_log_size":1059,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1889,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103464,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":347152,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93398000,"statfs":{"total":95995035648,"available":95639552000,"internally_reserved":0,"allocated":860160,"data_stored":513803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":354613706},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397804,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":924984,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92820168,"statfs":{"total":95995035648,"available":95047852032,"internally_reserved":0,"allocated":1163264,"data_stored":840321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":946010566},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575173,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1028824,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":1027966,"kb_avail":92716328,"statfs":{"total":95995035648,"available":94941519872,"internally_reserved":0,"allocated":868352,"data_stored":514013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1052637637},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738698,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1037464,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":1036286,"kb_avail":92707688,"statfs":{"total":95995035648,"available":94932672512,"internally_reserved":0,"allocated":1196032,"data_stored":842197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1061157317},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":388,"seq":1666447310915,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":208064,"kb_used_data":1080,"kb_used_omap":168,"kb_used_meta":206807,"kb_avail":93537088,"statfs":{"total":95995035648,"available":95781978112,"internally_reserved":0,"allocated":1105920,"data_stored":776570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":172848,"internal_metadata":211770576},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182716,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1043288,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":1041854,"kb_avail":92701864,"statfs":{"total":95995035648,"available":94926708736,"internally_reserved":0,"allocated":1458176,"data_stored":1103777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1066858951},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953782,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898324,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":92846828,"statfs":{"total":95995035648,"available":95075151872,"internally_reserved":0,"allocated":864256,"data_stored":513490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":919009733},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692145,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":808344,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92936808,"statfs":{"total":95995035648,"available":95167291392,"internally_reserved":0,"allocated":1130496,"data_stored":777383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T14:06:57.646 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T14:06:57.898 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:06:58.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:58 smithi043 ceph-mon[29963]: 1.3 scrub starts 2024-03-30T14:06:58.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:58 smithi043 ceph-mon[29963]: 1.3 scrub ok 2024-03-30T14:06:58.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:58 smithi043 ceph-mon[29963]: 3.d scrub starts 2024-03-30T14:06:58.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:58 smithi043 ceph-mon[29963]: 3.d scrub ok 2024-03-30T14:06:58.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:58 smithi043 ceph-mon[29963]: from='client.26129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:06:58.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:58 smithi043 ceph-mon[29963]: 4.0 scrub starts 2024-03-30T14:06:58.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:58 smithi043 ceph-mon[29963]: 4.0 scrub ok 2024-03-30T14:06:58.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:58 smithi043 ceph-mon[42099]: 1.3 scrub starts 2024-03-30T14:06:58.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:58 smithi043 ceph-mon[42099]: 1.3 scrub ok 2024-03-30T14:06:58.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:58 smithi043 ceph-mon[42099]: 3.d scrub starts 2024-03-30T14:06:58.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:58 smithi043 ceph-mon[42099]: 3.d scrub ok 2024-03-30T14:06:58.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:58 smithi043 ceph-mon[42099]: from='client.26129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:06:58.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:58 smithi043 ceph-mon[42099]: 4.0 scrub starts 2024-03-30T14:06:58.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:58 smithi043 ceph-mon[42099]: 4.0 scrub ok 2024-03-30T14:06:58.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:58 smithi113 ceph-mon[34341]: 1.3 scrub starts 2024-03-30T14:06:58.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:58 smithi113 ceph-mon[34341]: 1.3 scrub ok 2024-03-30T14:06:58.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:58 smithi113 ceph-mon[34341]: 3.d scrub starts 2024-03-30T14:06:58.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:58 smithi113 ceph-mon[34341]: 3.d scrub ok 2024-03-30T14:06:58.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:58 smithi113 ceph-mon[34341]: from='client.26129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:06:58.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:58 smithi113 ceph-mon[34341]: 4.0 scrub starts 2024-03-30T14:06:58.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:58 smithi113 ceph-mon[34341]: 4.0 scrub ok 2024-03-30T14:06:59.325 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:59 smithi043 ceph-mon[29963]: 2.0 scrub starts 2024-03-30T14:06:59.325 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:59 smithi043 ceph-mon[29963]: 2.0 scrub ok 2024-03-30T14:06:59.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:06:59 smithi043 ceph-mon[29963]: pgmap v1152: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:06:59.326 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:59 smithi043 ceph-mon[42099]: 2.0 scrub starts 2024-03-30T14:06:59.326 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:59 smithi043 ceph-mon[42099]: 2.0 scrub ok 2024-03-30T14:06:59.326 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:06:59 smithi043 ceph-mon[42099]: pgmap v1152: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:06:59.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:59 smithi113 ceph-mon[34341]: 2.0 scrub starts 2024-03-30T14:06:59.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:59 smithi113 ceph-mon[34341]: 2.0 scrub ok 2024-03-30T14:06:59.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:06:59 smithi113 ceph-mon[34341]: pgmap v1152: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:06:59.908 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-30T14:06:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-30T14:07:00.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:06:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug there is no tcmu-runner data available 2024-03-30T14:07:00.305 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:00 smithi043 ceph-mon[29963]: 4.e scrub starts 2024-03-30T14:07:00.305 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:00 smithi043 ceph-mon[29963]: 4.e scrub ok 2024-03-30T14:07:00.305 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:00 smithi043 ceph-mon[29963]: 5.f scrub starts 2024-03-30T14:07:00.305 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:00 smithi043 ceph-mon[29963]: 5.f scrub ok 2024-03-30T14:07:00.305 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:00 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:07:00.305 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:00 smithi043 ceph-mon[42099]: 4.e scrub starts 2024-03-30T14:07:00.305 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:00 smithi043 ceph-mon[42099]: 4.e scrub ok 2024-03-30T14:07:00.305 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:00 smithi043 ceph-mon[42099]: 5.f scrub starts 2024-03-30T14:07:00.305 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:00 smithi043 ceph-mon[42099]: 5.f scrub ok 2024-03-30T14:07:00.306 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:00 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:07:00.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:00 smithi113 ceph-mon[34341]: 4.e scrub starts 2024-03-30T14:07:00.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:00 smithi113 ceph-mon[34341]: 4.e scrub ok 2024-03-30T14:07:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:00 smithi113 ceph-mon[34341]: 5.f scrub starts 2024-03-30T14:07:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:00 smithi113 ceph-mon[34341]: 5.f scrub ok 2024-03-30T14:07:00.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:00 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:07:00.799 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":1153,"stamp":"2024-03-30T14:06:59.873708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1666,"num_read_kb":1418,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":609,"num_bytes_recovered":1067973,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3896,"ondisk_log_size":3896,"up":456,"acting":456,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":456,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6296444,"kb_used_data":8444,"kb_used_omap":179,"kb_used_meta":6287756,"kb_avail":743664772,"statfs":{"total":767960285184,"available":761512726528,"internally_reserved":0,"allocated":8646656,"data_stored":5881554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":184005,"internal_metadata":6438662459},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003568"},"pg_stats":[{"pgid":"4.c","version":"421'30","reported_seq":1071,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777197+0000","last_change":"2024-03-30T14:05:53.414839+0000","last_active":"2024-03-30T14:06:32.777197+0000","last_peered":"2024-03-30T14:06:32.777197+0000","last_clean":"2024-03-30T14:06:32.777197+0000","last_became_active":"2024-03-30T14:05:02.368055+0000","last_became_peered":"2024-03-30T14:05:02.368055+0000","last_unstale":"2024-03-30T14:06:32.777197+0000","last_undegraded":"2024-03-30T14:06:32.777197+0000","last_fullsized":"2024-03-30T14:06:32.777197+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'30","last_scrub_stamp":"2024-03-30T14:05:53.414675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:05:53.414675+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:04.999571+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":100,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.590072+0000","last_change":"2024-03-30T14:06:22.564184+0000","last_active":"2024-03-30T14:06:32.590072+0000","last_peered":"2024-03-30T14:06:32.590072+0000","last_clean":"2024-03-30T14:06:32.590072+0000","last_became_active":"2024-03-30T14:05:02.374330+0000","last_became_peered":"2024-03-30T14:05:02.374330+0000","last_unstale":"2024-03-30T14:06:32.590072+0000","last_undegraded":"2024-03-30T14:06:32.590072+0000","last_fullsized":"2024-03-30T14:06:32.590072+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:22.564105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:06:22.564105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:43.927593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"521'23","reported_seq":384,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:50.657581+0000","last_change":"2024-03-30T14:06:12.471677+0000","last_active":"2024-03-30T14:06:50.657581+0000","last_peered":"2024-03-30T14:06:50.657581+0000","last_clean":"2024-03-30T14:06:50.657581+0000","last_became_active":"2024-03-30T14:06:12.471241+0000","last_became_peered":"2024-03-30T14:06:12.471241+0000","last_unstale":"2024-03-30T14:06:50.657581+0000","last_undegraded":"2024-03-30T14:06:50.657581+0000","last_fullsized":"2024-03-30T14:06:50.657581+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"510'22","last_scrub_stamp":"2024-03-30T14:06:06.883865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:06:06.883865+0000","objects_scrubbed":1,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:36.262077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":993,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.778214+0000","last_change":"2024-03-30T14:06:32.381500+0000","last_active":"2024-03-30T14:06:32.778214+0000","last_peered":"2024-03-30T14:06:32.778214+0000","last_clean":"2024-03-30T14:06:32.778214+0000","last_became_active":"2024-03-30T14:04:54.315300+0000","last_became_peered":"2024-03-30T14:04:54.315300+0000","last_unstale":"2024-03-30T14:06:32.778214+0000","last_undegraded":"2024-03-30T14:06:32.778214+0000","last_fullsized":"2024-03-30T14:06:32.778214+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:32.381393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:02:06.483964+0000","last_clean_scrub_stamp":"2024-03-30T14:06:32.381393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:44.606428+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":1053,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775728+0000","last_change":"2024-03-30T14:05:59.005586+0000","last_active":"2024-03-30T14:06:32.775728+0000","last_peered":"2024-03-30T14:06:32.775728+0000","last_clean":"2024-03-30T14:06:32.775728+0000","last_became_active":"2024-03-30T14:05:13.374989+0000","last_became_peered":"2024-03-30T14:05:13.374989+0000","last_unstale":"2024-03-30T14:06:32.775728+0000","last_undegraded":"2024-03-30T14:06:32.775728+0000","last_fullsized":"2024-03-30T14:06:32.775728+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:05:59.005483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:05:59.005483+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:28.678058+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"421'45","reported_seq":1110,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777522+0000","last_change":"2024-03-30T14:05:21.376681+0000","last_active":"2024-03-30T14:06:32.777522+0000","last_peered":"2024-03-30T14:06:32.777522+0000","last_clean":"2024-03-30T14:06:32.777522+0000","last_became_active":"2024-03-30T14:04:54.309600+0000","last_became_peered":"2024-03-30T14:04:54.309600+0000","last_unstale":"2024-03-30T14:06:32.777522+0000","last_undegraded":"2024-03-30T14:06:32.777522+0000","last_fullsized":"2024-03-30T14:06:32.777522+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'45","last_scrub_stamp":"2024-03-30T14:05:21.376569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:05:21.376569+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:50.606469+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1026,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776882+0000","last_change":"2024-03-30T14:06:06.136540+0000","last_active":"2024-03-30T14:06:32.776882+0000","last_peered":"2024-03-30T14:06:32.776882+0000","last_clean":"2024-03-30T14:06:32.776882+0000","last_became_active":"2024-03-30T13:42:45.315251+0000","last_became_peered":"2024-03-30T13:42:45.315251+0000","last_unstale":"2024-03-30T14:06:32.776882+0000","last_undegraded":"2024-03-30T14:06:32.776882+0000","last_fullsized":"2024-03-30T14:06:32.776882+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:06.136452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:57:12.187291+0000","last_clean_scrub_stamp":"2024-03-30T14:06:06.136452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:32.859925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1019,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:41.313423+0000","last_change":"2024-03-30T14:06:41.313446+0000","last_active":"2024-03-30T14:06:41.313423+0000","last_peered":"2024-03-30T14:06:41.313423+0000","last_clean":"2024-03-30T14:06:41.313423+0000","last_became_active":"2024-03-30T13:42:49.321360+0000","last_became_peered":"2024-03-30T13:42:49.321360+0000","last_unstale":"2024-03-30T14:06:41.313423+0000","last_undegraded":"2024-03-30T14:06:41.313423+0000","last_fullsized":"2024-03-30T14:06:41.313423+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:41.313355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:05:26.170570+0000","last_clean_scrub_stamp":"2024-03-30T14:06:41.313355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:10.451996+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":618,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589926+0000","last_change":"2024-03-30T14:06:18.063780+0000","last_active":"2024-03-30T14:06:32.589926+0000","last_peered":"2024-03-30T14:06:32.589926+0000","last_clean":"2024-03-30T14:06:32.589926+0000","last_became_active":"2024-03-30T14:06:12.472854+0000","last_became_peered":"2024-03-30T14:06:12.472854+0000","last_unstale":"2024-03-30T14:06:32.589926+0000","last_undegraded":"2024-03-30T14:06:32.589926+0000","last_fullsized":"2024-03-30T14:06:32.589926+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:18.063574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:18.063574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:37.746573+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":652,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.588699+0000","last_change":"2024-03-30T14:06:04.904808+0000","last_active":"2024-03-30T14:06:32.588699+0000","last_peered":"2024-03-30T14:06:32.588699+0000","last_clean":"2024-03-30T14:06:32.588699+0000","last_became_active":"2024-03-30T14:05:51.009740+0000","last_became_peered":"2024-03-30T14:05:51.009740+0000","last_unstale":"2024-03-30T14:06:32.588699+0000","last_undegraded":"2024-03-30T14:06:32.588699+0000","last_fullsized":"2024-03-30T14:06:32.588699+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:04.904721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:06:04.904721+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:07.858677+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1050752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"421'27","reported_seq":1049,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:58.222682+0000","last_change":"2024-03-30T14:06:58.222724+0000","last_active":"2024-03-30T14:06:58.222682+0000","last_peered":"2024-03-30T14:06:58.222682+0000","last_clean":"2024-03-30T14:06:58.222682+0000","last_became_active":"2024-03-30T14:04:32.238711+0000","last_became_peered":"2024-03-30T14:04:32.238711+0000","last_unstale":"2024-03-30T14:06:58.222682+0000","last_undegraded":"2024-03-30T14:06:58.222682+0000","last_fullsized":"2024-03-30T14:06:58.222682+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'27","last_scrub_stamp":"2024-03-30T14:06:58.222605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:06:58.222605+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:07.804761+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1024,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776281+0000","last_change":"2024-03-30T14:05:58.340520+0000","last_active":"2024-03-30T14:06:32.776281+0000","last_peered":"2024-03-30T14:06:32.776281+0000","last_clean":"2024-03-30T14:06:32.776281+0000","last_became_active":"2024-03-30T14:05:10.381761+0000","last_became_peered":"2024-03-30T14:05:10.381761+0000","last_unstale":"2024-03-30T14:06:32.776281+0000","last_undegraded":"2024-03-30T14:06:32.776281+0000","last_fullsized":"2024-03-30T14:06:32.776281+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:58.340420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:00:21.110376+0000","last_clean_scrub_stamp":"2024-03-30T14:05:58.340420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:27.438884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":781,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589696+0000","last_change":"2024-03-30T14:05:29.344185+0000","last_active":"2024-03-30T14:06:32.589696+0000","last_peered":"2024-03-30T14:06:32.589696+0000","last_clean":"2024-03-30T14:06:32.589696+0000","last_became_active":"2024-03-30T13:58:48.216542+0000","last_became_peered":"2024-03-30T13:58:48.216542+0000","last_unstale":"2024-03-30T14:06:32.589696+0000","last_undegraded":"2024-03-30T14:06:32.589696+0000","last_fullsized":"2024-03-30T14:06:32.589696+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:29.344072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:52.603772+0000","last_clean_scrub_stamp":"2024-03-30T14:05:29.344072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:44.065692+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":112,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.590585+0000","last_change":"2024-03-30T14:06:02.610467+0000","last_active":"2024-03-30T14:06:32.590585+0000","last_peered":"2024-03-30T14:06:32.590585+0000","last_clean":"2024-03-30T14:06:32.590585+0000","last_became_active":"2024-03-30T14:05:48.013542+0000","last_became_peered":"2024-03-30T14:05:48.013542+0000","last_unstale":"2024-03-30T14:06:32.590585+0000","last_undegraded":"2024-03-30T14:06:32.590585+0000","last_fullsized":"2024-03-30T14:06:32.590585+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:02.610385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:21.081938+0000","last_clean_scrub_stamp":"2024-03-30T14:06:02.610385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:21.547464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"82'128","reported_seq":1034,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.774925+0000","last_change":"2024-03-30T14:05:51.010923+0000","last_active":"2024-03-30T14:06:32.774925+0000","last_peered":"2024-03-30T14:06:32.774925+0000","last_clean":"2024-03-30T14:06:32.774925+0000","last_became_active":"2024-03-30T14:05:51.010364+0000","last_became_peered":"2024-03-30T14:05:51.010364+0000","last_unstale":"2024-03-30T14:06:32.774925+0000","last_undegraded":"2024-03-30T14:06:32.774925+0000","last_fullsized":"2024-03-30T14:06:32.774925+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:05:31.157117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:05:31.157117+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:57.606298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"421'46","reported_seq":1171,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776298+0000","last_change":"2024-03-30T14:06:31.465038+0000","last_active":"2024-03-30T14:06:32.776298+0000","last_peered":"2024-03-30T14:06:32.776298+0000","last_clean":"2024-03-30T14:06:32.776298+0000","last_became_active":"2024-03-30T14:05:02.381261+0000","last_became_peered":"2024-03-30T14:05:02.381261+0000","last_unstale":"2024-03-30T14:06:32.776298+0000","last_undegraded":"2024-03-30T14:06:32.776298+0000","last_fullsized":"2024-03-30T14:06:32.776298+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'46","last_scrub_stamp":"2024-03-30T14:06:31.464933+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:28.831527+0000","last_clean_scrub_stamp":"2024-03-30T14:06:31.464933+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:59.116350+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":114,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.588806+0000","last_change":"2024-03-30T14:06:25.534314+0000","last_active":"2024-03-30T14:06:32.588806+0000","last_peered":"2024-03-30T14:06:32.588806+0000","last_clean":"2024-03-30T14:06:32.588806+0000","last_became_active":"2024-03-30T14:05:10.391971+0000","last_became_peered":"2024-03-30T14:05:10.391971+0000","last_unstale":"2024-03-30T14:06:32.588806+0000","last_undegraded":"2024-03-30T14:06:32.588806+0000","last_fullsized":"2024-03-30T14:06:32.588806+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:25.534220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:06:25.534220+0000","last_clean_scrub_stamp":"2024-03-30T14:06:25.534220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:47.374607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"510'18","reported_seq":1335,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:53.319864+0000","last_change":"2024-03-30T14:06:53.319864+0000","last_active":"2024-03-30T14:06:53.319864+0000","last_peered":"2024-03-30T14:06:53.319864+0000","last_clean":"2024-03-30T14:06:53.319864+0000","last_became_active":"2024-03-30T14:05:48.018872+0000","last_became_peered":"2024-03-30T14:05:48.018872+0000","last_unstale":"2024-03-30T14:06:53.319864+0000","last_undegraded":"2024-03-30T14:06:53.319864+0000","last_fullsized":"2024-03-30T14:06:53.319864+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"510'18","last_scrub_stamp":"2024-03-30T14:06:53.319756+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-30T13:51:15.146255+0000","last_clean_scrub_stamp":"2024-03-30T14:06:53.319756+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:21.906843+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":53,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589313+0000","last_change":"2024-03-30T14:05:52.207952+0000","last_active":"2024-03-30T14:06:32.589313+0000","last_peered":"2024-03-30T14:06:32.589313+0000","last_clean":"2024-03-30T14:06:32.589313+0000","last_became_active":"2024-03-30T14:05:48.022206+0000","last_became_peered":"2024-03-30T14:05:48.022206+0000","last_unstale":"2024-03-30T14:06:32.589313+0000","last_undegraded":"2024-03-30T14:06:32.589313+0000","last_fullsized":"2024-03-30T14:06:32.589313+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:52.207826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:05:52.207826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:09.810738+0000","scrub_duration":140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":809,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589101+0000","last_change":"2024-03-30T14:06:21.907236+0000","last_active":"2024-03-30T14:06:32.589101+0000","last_peered":"2024-03-30T14:06:32.589101+0000","last_clean":"2024-03-30T14:06:32.589101+0000","last_became_active":"2024-03-30T14:04:57.309699+0000","last_became_peered":"2024-03-30T14:04:57.309699+0000","last_unstale":"2024-03-30T14:06:32.589101+0000","last_undegraded":"2024-03-30T14:06:32.589101+0000","last_fullsized":"2024-03-30T14:06:32.589101+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:21.907124+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T13:56:08.385059+0000","last_clean_scrub_stamp":"2024-03-30T14:06:21.907124+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:24.997459+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1023,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776116+0000","last_change":"2024-03-30T14:06:02.362759+0000","last_active":"2024-03-30T14:06:32.776116+0000","last_peered":"2024-03-30T14:06:32.776116+0000","last_clean":"2024-03-30T14:06:32.776116+0000","last_became_active":"2024-03-30T14:05:10.369990+0000","last_became_peered":"2024-03-30T14:05:10.369990+0000","last_unstale":"2024-03-30T14:06:32.776116+0000","last_undegraded":"2024-03-30T14:06:32.776116+0000","last_fullsized":"2024-03-30T14:06:32.776116+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:02.362655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:06:02.362655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:17.954023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"421'47","reported_seq":150,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:57.224274+0000","last_change":"2024-03-30T14:06:57.224319+0000","last_active":"2024-03-30T14:06:57.224274+0000","last_peered":"2024-03-30T14:06:57.224274+0000","last_clean":"2024-03-30T14:06:57.224274+0000","last_became_active":"2024-03-30T14:05:10.391828+0000","last_became_peered":"2024-03-30T14:05:10.391828+0000","last_unstale":"2024-03-30T14:06:57.224274+0000","last_undegraded":"2024-03-30T14:06:57.224274+0000","last_fullsized":"2024-03-30T14:06:57.224274+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'47","last_scrub_stamp":"2024-03-30T14:06:57.224211+0000","last_deep_scrub":"216'29","last_deep_scrub_stamp":"2024-03-30T13:52:52.973602+0000","last_clean_scrub_stamp":"2024-03-30T14:06:57.224211+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:16.995325+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":106,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.591059+0000","last_change":"2024-03-30T14:05:54.644373+0000","last_active":"2024-03-30T14:06:32.591059+0000","last_peered":"2024-03-30T14:06:32.591059+0000","last_clean":"2024-03-30T14:06:32.591059+0000","last_became_active":"2024-03-30T14:05:10.384945+0000","last_became_peered":"2024-03-30T14:05:10.384945+0000","last_unstale":"2024-03-30T14:06:32.591059+0000","last_undegraded":"2024-03-30T14:06:32.591059+0000","last_fullsized":"2024-03-30T14:06:32.591059+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:54.644196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:53:01.031216+0000","last_clean_scrub_stamp":"2024-03-30T14:05:54.644196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:18.604189+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1006,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776229+0000","last_change":"2024-03-30T14:06:10.198636+0000","last_active":"2024-03-30T14:06:32.776229+0000","last_peered":"2024-03-30T14:06:32.776229+0000","last_clean":"2024-03-30T14:06:32.776229+0000","last_became_active":"2024-03-30T14:05:02.355737+0000","last_became_peered":"2024-03-30T14:05:02.355737+0000","last_unstale":"2024-03-30T14:06:32.776229+0000","last_undegraded":"2024-03-30T14:06:32.776229+0000","last_fullsized":"2024-03-30T14:06:32.776229+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:10.198510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:10.198510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:18.919645+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":1009,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776786+0000","last_change":"2024-03-30T14:06:00.329645+0000","last_active":"2024-03-30T14:06:32.776786+0000","last_peered":"2024-03-30T14:06:32.776786+0000","last_clean":"2024-03-30T14:06:32.776786+0000","last_became_active":"2024-03-30T14:04:57.309024+0000","last_became_peered":"2024-03-30T14:04:57.309024+0000","last_unstale":"2024-03-30T14:06:32.776786+0000","last_undegraded":"2024-03-30T14:06:32.776786+0000","last_fullsized":"2024-03-30T14:06:32.776786+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:00.329540+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T14:04:20.952248+0000","last_clean_scrub_stamp":"2024-03-30T14:06:00.329540+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:09.159979+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"487'10","reported_seq":127,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:50.657516+0000","last_change":"2024-03-30T14:05:47.569855+0000","last_active":"2024-03-30T14:06:50.657516+0000","last_peered":"2024-03-30T14:06:50.657516+0000","last_clean":"2024-03-30T14:06:50.657516+0000","last_became_active":"2024-03-30T14:05:02.378061+0000","last_became_peered":"2024-03-30T14:05:02.378061+0000","last_unstale":"2024-03-30T14:06:50.657516+0000","last_undegraded":"2024-03-30T14:06:50.657516+0000","last_fullsized":"2024-03-30T14:06:50.657516+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"487'10","last_scrub_stamp":"2024-03-30T14:05:47.569760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:05:47.569760+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:14.100898+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"421'49","reported_seq":1157,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776379+0000","last_change":"2024-03-30T14:05:54.450308+0000","last_active":"2024-03-30T14:06:32.776379+0000","last_peered":"2024-03-30T14:06:32.776379+0000","last_clean":"2024-03-30T14:06:32.776379+0000","last_became_active":"2024-03-30T14:05:48.013215+0000","last_became_peered":"2024-03-30T14:05:48.013215+0000","last_unstale":"2024-03-30T14:06:32.776379+0000","last_undegraded":"2024-03-30T14:06:32.776379+0000","last_fullsized":"2024-03-30T14:06:32.776379+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'49","last_scrub_stamp":"2024-03-30T14:05:54.450160+0000","last_deep_scrub":"91'21","last_deep_scrub_stamp":"2024-03-30T13:44:05.917064+0000","last_clean_scrub_stamp":"2024-03-30T14:05:54.450160+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:19.068715+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"499'2","reported_seq":773,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775576+0000","last_change":"2024-03-30T14:05:30.754898+0000","last_active":"2024-03-30T14:06:32.775576+0000","last_peered":"2024-03-30T14:06:32.775576+0000","last_clean":"2024-03-30T14:06:32.775576+0000","last_became_active":"2024-03-30T14:05:23.027201+0000","last_became_peered":"2024-03-30T14:05:23.027201+0000","last_unstale":"2024-03-30T14:06:32.775576+0000","last_undegraded":"2024-03-30T14:06:32.775576+0000","last_fullsized":"2024-03-30T14:06:32.775576+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"499'2","last_scrub_stamp":"2024-03-30T14:05:30.754755+0000","last_deep_scrub":"499'2","last_deep_scrub_stamp":"2024-03-30T14:05:30.754755+0000","last_clean_scrub_stamp":"2024-03-30T14:05:30.754755+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:58.196997+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":90,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776338+0000","last_change":"2024-03-30T14:05:43.139037+0000","last_active":"2024-03-30T14:06:32.776338+0000","last_peered":"2024-03-30T14:06:32.776338+0000","last_clean":"2024-03-30T14:06:32.776338+0000","last_became_active":"2024-03-30T14:05:10.389836+0000","last_became_peered":"2024-03-30T14:05:10.389836+0000","last_unstale":"2024-03-30T14:06:32.776338+0000","last_undegraded":"2024-03-30T14:06:32.776338+0000","last_fullsized":"2024-03-30T14:06:32.776338+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:43.138904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:05:43.138904+0000","last_clean_scrub_stamp":"2024-03-30T14:05:43.138904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:48.643259+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":472,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.590482+0000","last_change":"2024-03-30T14:06:32.446539+0000","last_active":"2024-03-30T14:06:32.590482+0000","last_peered":"2024-03-30T14:06:32.590482+0000","last_clean":"2024-03-30T14:06:32.590482+0000","last_became_active":"2024-03-30T14:05:51.010139+0000","last_became_peered":"2024-03-30T14:05:51.010139+0000","last_unstale":"2024-03-30T14:06:32.590482+0000","last_undegraded":"2024-03-30T14:06:32.590482+0000","last_fullsized":"2024-03-30T14:06:32.590482+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:32.446450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:06:32.446450+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:48.178673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"510'18","reported_seq":692,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:50.657543+0000","last_change":"2024-03-30T14:06:39.464075+0000","last_active":"2024-03-30T14:06:50.657543+0000","last_peered":"2024-03-30T14:06:50.657543+0000","last_clean":"2024-03-30T14:06:50.657543+0000","last_became_active":"2024-03-30T14:05:48.024814+0000","last_became_peered":"2024-03-30T14:05:48.024814+0000","last_unstale":"2024-03-30T14:06:50.657543+0000","last_undegraded":"2024-03-30T14:06:50.657543+0000","last_fullsized":"2024-03-30T14:06:50.657543+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"510'18","last_scrub_stamp":"2024-03-30T14:06:39.463949+0000","last_deep_scrub":"218'9","last_deep_scrub_stamp":"2024-03-30T13:52:57.873554+0000","last_clean_scrub_stamp":"2024-03-30T14:06:39.463949+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:40.879509+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"421'35","reported_seq":1144,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:33.463936+0000","last_change":"2024-03-30T14:06:33.463969+0000","last_active":"2024-03-30T14:06:33.463936+0000","last_peered":"2024-03-30T14:06:33.463936+0000","last_clean":"2024-03-30T14:06:33.463936+0000","last_became_active":"2024-03-30T14:05:48.022376+0000","last_became_peered":"2024-03-30T14:05:48.022376+0000","last_unstale":"2024-03-30T14:06:33.463936+0000","last_undegraded":"2024-03-30T14:06:33.463936+0000","last_fullsized":"2024-03-30T14:06:33.463936+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'35","last_scrub_stamp":"2024-03-30T14:06:33.463859+0000","last_deep_scrub":"421'35","last_deep_scrub_stamp":"2024-03-30T14:06:33.463859+0000","last_clean_scrub_stamp":"2024-03-30T14:06:33.463859+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:39.881076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":53,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589391+0000","last_change":"2024-03-30T14:05:59.043577+0000","last_active":"2024-03-30T14:06:32.589391+0000","last_peered":"2024-03-30T14:06:32.589391+0000","last_clean":"2024-03-30T14:06:32.589391+0000","last_became_active":"2024-03-30T14:05:48.022990+0000","last_became_peered":"2024-03-30T14:05:48.022990+0000","last_unstale":"2024-03-30T14:06:32.589391+0000","last_undegraded":"2024-03-30T14:06:32.589391+0000","last_fullsized":"2024-03-30T14:06:32.589391+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:59.043489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:55.458779+0000","last_clean_scrub_stamp":"2024-03-30T14:05:59.043489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:28.883155+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":739,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.774999+0000","last_change":"2024-03-30T14:05:45.745085+0000","last_active":"2024-03-30T14:06:32.774999+0000","last_peered":"2024-03-30T14:06:32.774999+0000","last_clean":"2024-03-30T14:06:32.774999+0000","last_became_active":"2024-03-30T14:05:23.026736+0000","last_became_peered":"2024-03-30T14:05:23.026736+0000","last_unstale":"2024-03-30T14:06:32.774999+0000","last_undegraded":"2024-03-30T14:06:32.774999+0000","last_fullsized":"2024-03-30T14:06:32.774999+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:45.744982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:41.142941+0000","last_clean_scrub_stamp":"2024-03-30T14:05:45.744982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:07.935939+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":1072,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:35.991585+0000","last_change":"2024-03-30T14:06:35.991612+0000","last_active":"2024-03-30T14:06:35.991585+0000","last_peered":"2024-03-30T14:06:35.991585+0000","last_clean":"2024-03-30T14:06:35.991585+0000","last_became_active":"2024-03-30T14:05:13.376299+0000","last_became_peered":"2024-03-30T14:05:13.376299+0000","last_unstale":"2024-03-30T14:06:35.991585+0000","last_undegraded":"2024-03-30T14:06:35.991585+0000","last_fullsized":"2024-03-30T14:06:35.991585+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:35.991460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:06:35.991460+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:50.551910+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1037,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:51.083380+0000","last_change":"2024-03-30T14:06:51.083407+0000","last_active":"2024-03-30T14:06:51.083380+0000","last_peered":"2024-03-30T14:06:51.083380+0000","last_clean":"2024-03-30T14:06:51.083380+0000","last_became_active":"2024-03-30T13:58:47.204999+0000","last_became_peered":"2024-03-30T13:58:47.204999+0000","last_unstale":"2024-03-30T14:06:51.083380+0000","last_undegraded":"2024-03-30T14:06:51.083380+0000","last_fullsized":"2024-03-30T14:06:51.083380+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:51.083308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:52.887014+0000","last_clean_scrub_stamp":"2024-03-30T14:06:51.083308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:57.342668+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"421'47","reported_seq":888,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775756+0000","last_change":"2024-03-30T14:05:51.551210+0000","last_active":"2024-03-30T14:06:32.775756+0000","last_peered":"2024-03-30T14:06:32.775756+0000","last_clean":"2024-03-30T14:06:32.775756+0000","last_became_active":"2024-03-30T14:05:48.010044+0000","last_became_peered":"2024-03-30T14:05:48.010044+0000","last_unstale":"2024-03-30T14:06:32.775756+0000","last_undegraded":"2024-03-30T14:06:32.775756+0000","last_fullsized":"2024-03-30T14:06:32.775756+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'47","last_scrub_stamp":"2024-03-30T14:05:51.551027+0000","last_deep_scrub":"402'37","last_deep_scrub_stamp":"2024-03-30T14:02:06.552075+0000","last_clean_scrub_stamp":"2024-03-30T14:05:51.551027+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:52.768463+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1054,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776459+0000","last_change":"2024-03-30T14:06:15.252088+0000","last_active":"2024-03-30T14:06:32.776459+0000","last_peered":"2024-03-30T14:06:32.776459+0000","last_clean":"2024-03-30T14:06:32.776459+0000","last_became_active":"2024-03-30T14:05:48.009994+0000","last_became_peered":"2024-03-30T14:05:48.009994+0000","last_unstale":"2024-03-30T14:06:32.776459+0000","last_undegraded":"2024-03-30T14:06:32.776459+0000","last_fullsized":"2024-03-30T14:06:32.776459+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:15.252005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:45:45.975163+0000","last_clean_scrub_stamp":"2024-03-30T14:06:15.252005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:19.851201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":997,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775201+0000","last_change":"2024-03-30T14:05:55.395143+0000","last_active":"2024-03-30T14:06:32.775201+0000","last_peered":"2024-03-30T14:06:32.775201+0000","last_clean":"2024-03-30T14:06:32.775201+0000","last_became_active":"2024-03-30T14:04:54.319778+0000","last_became_peered":"2024-03-30T14:04:54.319778+0000","last_unstale":"2024-03-30T14:06:32.775201+0000","last_undegraded":"2024-03-30T14:06:32.775201+0000","last_fullsized":"2024-03-30T14:06:32.775201+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:55.395044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:05:55.395044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:13.300881+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":1031,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777280+0000","last_change":"2024-03-30T14:05:41.264840+0000","last_active":"2024-03-30T14:06:32.777280+0000","last_peered":"2024-03-30T14:06:32.777280+0000","last_clean":"2024-03-30T14:06:32.777280+0000","last_became_active":"2024-03-30T14:05:05.004102+0000","last_became_peered":"2024-03-30T14:05:05.004102+0000","last_unstale":"2024-03-30T14:06:32.777280+0000","last_undegraded":"2024-03-30T14:06:32.777280+0000","last_fullsized":"2024-03-30T14:06:32.777280+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:05:41.264730+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T14:02:06.183099+0000","last_clean_scrub_stamp":"2024-03-30T14:05:41.264730+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:08.239137+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":53,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589530+0000","last_change":"2024-03-30T14:06:02.061179+0000","last_active":"2024-03-30T14:06:32.589530+0000","last_peered":"2024-03-30T14:06:32.589530+0000","last_clean":"2024-03-30T14:06:32.589530+0000","last_became_active":"2024-03-30T14:05:48.021019+0000","last_became_peered":"2024-03-30T14:05:48.021019+0000","last_unstale":"2024-03-30T14:06:32.589530+0000","last_undegraded":"2024-03-30T14:06:32.589530+0000","last_fullsized":"2024-03-30T14:06:32.589530+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:02.061083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:06:02.061083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:30.280253+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"421'42","reported_seq":631,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:44.412412+0000","last_change":"2024-03-30T14:06:44.412480+0000","last_active":"2024-03-30T14:06:44.412412+0000","last_peered":"2024-03-30T14:06:44.412412+0000","last_clean":"2024-03-30T14:06:44.412412+0000","last_became_active":"2024-03-30T14:05:31.110871+0000","last_became_peered":"2024-03-30T14:05:31.110871+0000","last_unstale":"2024-03-30T14:06:44.412412+0000","last_undegraded":"2024-03-30T14:06:44.412412+0000","last_fullsized":"2024-03-30T14:06:44.412412+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'42","last_scrub_stamp":"2024-03-30T14:06:44.412318+0000","last_deep_scrub":"93'18","last_deep_scrub_stamp":"2024-03-30T13:47:28.872529+0000","last_clean_scrub_stamp":"2024-03-30T14:06:44.412318+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:47.329231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":211,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:57.811442+0000","last_change":"2024-03-30T14:06:57.811442+0000","last_active":"2024-03-30T14:06:57.811442+0000","last_peered":"2024-03-30T14:06:57.811442+0000","last_clean":"2024-03-30T14:06:57.811442+0000","last_became_active":"2024-03-30T14:05:10.390745+0000","last_became_peered":"2024-03-30T14:05:10.390745+0000","last_unstale":"2024-03-30T14:06:57.811442+0000","last_undegraded":"2024-03-30T14:06:57.811442+0000","last_fullsized":"2024-03-30T14:06:57.811442+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-30T14:06:57.811340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T14:06:57.811340+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:06.902223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00126391,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":115,"num_read_kb":110,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":771,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775200+0000","last_change":"2024-03-30T14:06:16.450625+0000","last_active":"2024-03-30T14:06:32.775200+0000","last_peered":"2024-03-30T14:06:32.775200+0000","last_clean":"2024-03-30T14:06:32.775200+0000","last_became_active":"2024-03-30T14:05:23.029617+0000","last_became_peered":"2024-03-30T14:05:23.029617+0000","last_unstale":"2024-03-30T14:06:32.775200+0000","last_undegraded":"2024-03-30T14:06:32.775200+0000","last_fullsized":"2024-03-30T14:06:32.775200+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:16.450479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:00:25.597831+0000","last_clean_scrub_stamp":"2024-03-30T14:06:16.450479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:40.513078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1032,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:43.301180+0000","last_change":"2024-03-30T14:06:43.301202+0000","last_active":"2024-03-30T14:06:43.301180+0000","last_peered":"2024-03-30T14:06:43.301180+0000","last_clean":"2024-03-30T14:06:43.301180+0000","last_became_active":"2024-03-30T14:05:02.375363+0000","last_became_peered":"2024-03-30T14:05:02.375363+0000","last_unstale":"2024-03-30T14:06:43.301180+0000","last_undegraded":"2024-03-30T14:06:43.301180+0000","last_fullsized":"2024-03-30T14:06:43.301180+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:43.301118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:43.301118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:53.234315+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":1008,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:56.317766+0000","last_change":"2024-03-30T14:06:56.317793+0000","last_active":"2024-03-30T14:06:56.317766+0000","last_peered":"2024-03-30T14:06:56.317766+0000","last_clean":"2024-03-30T14:06:56.317766+0000","last_became_active":"2024-03-30T14:00:56.844488+0000","last_became_peered":"2024-03-30T14:00:56.844488+0000","last_unstale":"2024-03-30T14:06:56.317766+0000","last_undegraded":"2024-03-30T14:06:56.317766+0000","last_fullsized":"2024-03-30T14:06:56.317766+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:56.317692+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T14:05:33.136504+0000","last_clean_scrub_stamp":"2024-03-30T14:06:56.317692+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:25.680443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":90,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775896+0000","last_change":"2024-03-30T14:06:14.741843+0000","last_active":"2024-03-30T14:06:32.775896+0000","last_peered":"2024-03-30T14:06:32.775896+0000","last_clean":"2024-03-30T14:06:32.775896+0000","last_became_active":"2024-03-30T14:05:10.385323+0000","last_became_peered":"2024-03-30T14:05:10.385323+0000","last_unstale":"2024-03-30T14:06:32.775896+0000","last_undegraded":"2024-03-30T14:06:32.775896+0000","last_fullsized":"2024-03-30T14:06:32.775896+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:14.741707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:06:14.741707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:28.894718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"500'43","reported_seq":1443,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:50.657833+0000","last_change":"2024-03-30T14:06:39.022189+0000","last_active":"2024-03-30T14:06:50.657833+0000","last_peered":"2024-03-30T14:06:50.657833+0000","last_clean":"2024-03-30T14:06:50.657833+0000","last_became_active":"2024-03-30T14:05:31.112587+0000","last_became_peered":"2024-03-30T14:05:31.112587+0000","last_unstale":"2024-03-30T14:06:50.657833+0000","last_undegraded":"2024-03-30T14:06:50.657833+0000","last_fullsized":"2024-03-30T14:06:50.657833+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'43","last_scrub_stamp":"2024-03-30T14:06:39.022084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:06:39.022084+0000","objects_scrubbed":8,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:59.190264+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":106,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589437+0000","last_change":"2024-03-30T14:05:50.650038+0000","last_active":"2024-03-30T14:06:32.589437+0000","last_peered":"2024-03-30T14:06:32.589437+0000","last_clean":"2024-03-30T14:06:32.589437+0000","last_became_active":"2024-03-30T14:05:48.017537+0000","last_became_peered":"2024-03-30T14:05:48.017537+0000","last_unstale":"2024-03-30T14:06:32.589437+0000","last_undegraded":"2024-03-30T14:06:32.589437+0000","last_fullsized":"2024-03-30T14:06:32.589437+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:50.649954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:12.975288+0000","last_clean_scrub_stamp":"2024-03-30T14:05:50.649954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:04.926746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":728,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775145+0000","last_change":"2024-03-30T14:06:22.775255+0000","last_active":"2024-03-30T14:06:32.775145+0000","last_peered":"2024-03-30T14:06:32.775145+0000","last_clean":"2024-03-30T14:06:32.775145+0000","last_became_active":"2024-03-30T14:05:13.377385+0000","last_became_peered":"2024-03-30T14:05:13.377385+0000","last_unstale":"2024-03-30T14:06:32.775145+0000","last_undegraded":"2024-03-30T14:06:32.775145+0000","last_fullsized":"2024-03-30T14:06:32.775145+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:22.775125+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T14:06:22.775125+0000","last_clean_scrub_stamp":"2024-03-30T14:06:22.775125+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:29.872812+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":448,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589624+0000","last_change":"2024-03-30T14:05:30.383963+0000","last_active":"2024-03-30T14:06:32.589624+0000","last_peered":"2024-03-30T14:06:32.589624+0000","last_clean":"2024-03-30T14:06:32.589624+0000","last_became_active":"2024-03-30T14:05:02.354378+0000","last_became_peered":"2024-03-30T14:05:02.354378+0000","last_unstale":"2024-03-30T14:06:32.589624+0000","last_undegraded":"2024-03-30T14:06:32.589624+0000","last_fullsized":"2024-03-30T14:06:32.589624+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:30.383863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:03:13.730670+0000","last_clean_scrub_stamp":"2024-03-30T14:05:30.383863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:48.038095+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":393,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.590054+0000","last_change":"2024-03-30T14:06:12.470388+0000","last_active":"2024-03-30T14:06:32.590054+0000","last_peered":"2024-03-30T14:06:32.590054+0000","last_clean":"2024-03-30T14:06:32.590054+0000","last_became_active":"2024-03-30T14:06:12.470051+0000","last_became_peered":"2024-03-30T14:06:12.470051+0000","last_unstale":"2024-03-30T14:06:32.590054+0000","last_undegraded":"2024-03-30T14:06:32.590054+0000","last_fullsized":"2024-03-30T14:06:32.590054+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:46.189428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:28.373277+0000","last_clean_scrub_stamp":"2024-03-30T14:05:46.189428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:09.871386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"421'21","reported_seq":272,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776201+0000","last_change":"2024-03-30T14:06:01.182687+0000","last_active":"2024-03-30T14:06:32.776201+0000","last_peered":"2024-03-30T14:06:32.776201+0000","last_clean":"2024-03-30T14:06:32.776201+0000","last_became_active":"2024-03-30T14:05:48.010976+0000","last_became_peered":"2024-03-30T14:05:48.010976+0000","last_unstale":"2024-03-30T14:06:32.776201+0000","last_undegraded":"2024-03-30T14:06:32.776201+0000","last_fullsized":"2024-03-30T14:06:32.776201+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-30T14:06:01.182588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:06:01.182588+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:11.440031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":152,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.588750+0000","last_change":"2024-03-30T14:06:12.469306+0000","last_active":"2024-03-30T14:06:32.588750+0000","last_peered":"2024-03-30T14:06:32.588750+0000","last_clean":"2024-03-30T14:06:32.588750+0000","last_became_active":"2024-03-30T14:06:12.468985+0000","last_became_peered":"2024-03-30T14:06:12.468985+0000","last_unstale":"2024-03-30T14:06:32.588750+0000","last_undegraded":"2024-03-30T14:06:32.588750+0000","last_fullsized":"2024-03-30T14:06:32.588750+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:45.547752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:51.919203+0000","last_clean_scrub_stamp":"2024-03-30T14:05:45.547752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:10.263518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1010,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.774895+0000","last_change":"2024-03-30T14:05:53.286502+0000","last_active":"2024-03-30T14:06:32.774895+0000","last_peered":"2024-03-30T14:06:32.774895+0000","last_clean":"2024-03-30T14:06:32.774895+0000","last_became_active":"2024-03-30T14:04:54.329763+0000","last_became_peered":"2024-03-30T14:04:54.329763+0000","last_unstale":"2024-03-30T14:06:32.774895+0000","last_undegraded":"2024-03-30T14:06:32.774895+0000","last_fullsized":"2024-03-30T14:06:32.774895+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:53.286374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:43:59.810189+0000","last_clean_scrub_stamp":"2024-03-30T14:05:53.286374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:18.716633+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":98,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776553+0000","last_change":"2024-03-30T14:06:29.796095+0000","last_active":"2024-03-30T14:06:32.776553+0000","last_peered":"2024-03-30T14:06:32.776553+0000","last_clean":"2024-03-30T14:06:32.776553+0000","last_became_active":"2024-03-30T14:05:10.387768+0000","last_became_peered":"2024-03-30T14:05:10.387768+0000","last_unstale":"2024-03-30T14:06:32.776553+0000","last_undegraded":"2024-03-30T14:06:32.776553+0000","last_fullsized":"2024-03-30T14:06:32.776553+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:29.795955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:29.795955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:56.768115+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":921,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:48.800826+0000","last_change":"2024-03-30T14:06:48.800878+0000","last_active":"2024-03-30T14:06:48.800826+0000","last_peered":"2024-03-30T14:06:48.800826+0000","last_clean":"2024-03-30T14:06:48.800826+0000","last_became_active":"2024-03-30T14:05:13.374887+0000","last_became_peered":"2024-03-30T14:05:13.374887+0000","last_unstale":"2024-03-30T14:06:48.800826+0000","last_undegraded":"2024-03-30T14:06:48.800826+0000","last_fullsized":"2024-03-30T14:06:48.800826+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:48.800735+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T14:06:48.800735+0000","last_clean_scrub_stamp":"2024-03-30T14:06:48.800735+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:05.300363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":771,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775605+0000","last_change":"2024-03-30T14:05:31.705349+0000","last_active":"2024-03-30T14:06:32.775605+0000","last_peered":"2024-03-30T14:06:32.775605+0000","last_clean":"2024-03-30T14:06:32.775605+0000","last_became_active":"2024-03-30T14:05:23.027144+0000","last_became_peered":"2024-03-30T14:05:23.027144+0000","last_unstale":"2024-03-30T14:06:32.775605+0000","last_undegraded":"2024-03-30T14:06:32.775605+0000","last_fullsized":"2024-03-30T14:06:32.775605+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:31.705232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:18.862784+0000","last_clean_scrub_stamp":"2024-03-30T14:05:31.705232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:55.636278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"487'60","reported_seq":393,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:53.821103+0000","last_change":"2024-03-30T14:06:30.830561+0000","last_active":"2024-03-30T14:06:53.821103+0000","last_peered":"2024-03-30T14:06:53.821103+0000","last_clean":"2024-03-30T14:06:53.821103+0000","last_became_active":"2024-03-30T14:05:02.351915+0000","last_became_peered":"2024-03-30T14:05:02.351915+0000","last_unstale":"2024-03-30T14:06:53.821103+0000","last_undegraded":"2024-03-30T14:06:53.821103+0000","last_fullsized":"2024-03-30T14:06:53.821103+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"487'60","last_scrub_stamp":"2024-03-30T14:06:30.830372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:06:30.830372+0000","objects_scrubbed":10,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:41.709820+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":1790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":995,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:37.252192+0000","last_change":"2024-03-30T14:06:37.252219+0000","last_active":"2024-03-30T14:06:37.252192+0000","last_peered":"2024-03-30T14:06:37.252192+0000","last_clean":"2024-03-30T14:06:37.252192+0000","last_became_active":"2024-03-30T14:04:32.225606+0000","last_became_peered":"2024-03-30T14:04:32.225606+0000","last_unstale":"2024-03-30T14:06:37.252192+0000","last_undegraded":"2024-03-30T14:06:37.252192+0000","last_fullsized":"2024-03-30T14:06:37.252192+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:37.252116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:06:37.252116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:00.804341+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":440,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.588910+0000","last_change":"2024-03-30T14:06:26.760069+0000","last_active":"2024-03-30T14:06:32.588910+0000","last_peered":"2024-03-30T14:06:32.588910+0000","last_clean":"2024-03-30T14:06:32.588910+0000","last_became_active":"2024-03-30T13:58:47.199657+0000","last_became_peered":"2024-03-30T13:58:47.199657+0000","last_unstale":"2024-03-30T14:06:32.588910+0000","last_undegraded":"2024-03-30T14:06:32.588910+0000","last_fullsized":"2024-03-30T14:06:32.588910+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-30T14:06:26.759968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:43.281063+0000","last_clean_scrub_stamp":"2024-03-30T14:06:26.759968+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:49.851304+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":106,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589836+0000","last_change":"2024-03-30T14:05:49.672004+0000","last_active":"2024-03-30T14:06:32.589836+0000","last_peered":"2024-03-30T14:06:32.589836+0000","last_clean":"2024-03-30T14:06:32.589836+0000","last_became_active":"2024-03-30T14:05:10.379010+0000","last_became_peered":"2024-03-30T14:05:10.379010+0000","last_unstale":"2024-03-30T14:06:32.589836+0000","last_undegraded":"2024-03-30T14:06:32.589836+0000","last_fullsized":"2024-03-30T14:06:32.589836+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:49.671876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:05:49.671876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:10.705751+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":622,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:38.976634+0000","last_change":"2024-03-30T14:06:38.976661+0000","last_active":"2024-03-30T14:06:38.976634+0000","last_peered":"2024-03-30T14:06:38.976634+0000","last_clean":"2024-03-30T14:06:38.976634+0000","last_became_active":"2024-03-30T14:05:51.010372+0000","last_became_peered":"2024-03-30T14:05:51.010372+0000","last_unstale":"2024-03-30T14:06:38.976634+0000","last_undegraded":"2024-03-30T14:06:38.976634+0000","last_fullsized":"2024-03-30T14:06:38.976634+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:38.976557+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T14:06:38.976557+0000","last_clean_scrub_stamp":"2024-03-30T14:06:38.976557+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:58.695201+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"421'21","reported_seq":124,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775925+0000","last_change":"2024-03-30T14:06:22.485148+0000","last_active":"2024-03-30T14:06:32.775925+0000","last_peered":"2024-03-30T14:06:32.775925+0000","last_clean":"2024-03-30T14:06:32.775925+0000","last_became_active":"2024-03-30T14:05:23.027240+0000","last_became_peered":"2024-03-30T14:05:23.027240+0000","last_unstale":"2024-03-30T14:06:32.775925+0000","last_undegraded":"2024-03-30T14:06:32.775925+0000","last_fullsized":"2024-03-30T14:06:32.775925+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-30T14:06:22.485044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:06:22.485044+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:33.258501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":448,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589440+0000","last_change":"2024-03-30T14:06:13.953901+0000","last_active":"2024-03-30T14:06:32.589440+0000","last_peered":"2024-03-30T14:06:32.589440+0000","last_clean":"2024-03-30T14:06:32.589440+0000","last_became_active":"2024-03-30T14:05:02.351288+0000","last_became_peered":"2024-03-30T14:05:02.351288+0000","last_unstale":"2024-03-30T14:06:32.589440+0000","last_undegraded":"2024-03-30T14:06:32.589440+0000","last_fullsized":"2024-03-30T14:06:32.589440+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:13.953768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:05:04.382303+0000","last_clean_scrub_stamp":"2024-03-30T14:06:13.953768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:28.332449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":707,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.978561+0000","last_change":"2024-03-30T14:06:32.978596+0000","last_active":"2024-03-30T14:06:32.978561+0000","last_peered":"2024-03-30T14:06:32.978561+0000","last_clean":"2024-03-30T14:06:32.978561+0000","last_became_active":"2024-03-30T14:06:12.472079+0000","last_became_peered":"2024-03-30T14:06:12.472079+0000","last_unstale":"2024-03-30T14:06:32.978561+0000","last_undegraded":"2024-03-30T14:06:32.978561+0000","last_fullsized":"2024-03-30T14:06:32.978561+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:32.978474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:06:32.978474+0000","last_clean_scrub_stamp":"2024-03-30T14:06:32.978474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:00.453347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":752,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:38.639044+0000","last_change":"2024-03-30T14:06:38.639069+0000","last_active":"2024-03-30T14:06:38.639044+0000","last_peered":"2024-03-30T14:06:38.639044+0000","last_clean":"2024-03-30T14:06:38.639044+0000","last_became_active":"2024-03-30T14:05:23.032172+0000","last_became_peered":"2024-03-30T14:05:23.032172+0000","last_unstale":"2024-03-30T14:06:38.639044+0000","last_undegraded":"2024-03-30T14:06:38.639044+0000","last_fullsized":"2024-03-30T14:06:38.639044+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:38.638957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:06:38.638957+0000","last_clean_scrub_stamp":"2024-03-30T14:06:38.638957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:08.440754+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"82'128","reported_seq":679,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777209+0000","last_change":"2024-03-30T14:06:01.321053+0000","last_active":"2024-03-30T14:06:32.777209+0000","last_peered":"2024-03-30T14:06:32.777209+0000","last_clean":"2024-03-30T14:06:32.777209+0000","last_became_active":"2024-03-30T14:05:13.375610+0000","last_became_peered":"2024-03-30T14:05:13.375610+0000","last_unstale":"2024-03-30T14:06:32.777209+0000","last_undegraded":"2024-03-30T14:06:32.777209+0000","last_fullsized":"2024-03-30T14:06:32.777209+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:01.320915+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-30T13:57:14.131256+0000","last_clean_scrub_stamp":"2024-03-30T14:06:01.320915+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:11.501513+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"421'60","reported_seq":1214,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777014+0000","last_change":"2024-03-30T14:06:00.211726+0000","last_active":"2024-03-30T14:06:32.777014+0000","last_peered":"2024-03-30T14:06:32.777014+0000","last_clean":"2024-03-30T14:06:32.777014+0000","last_became_active":"2024-03-30T14:05:31.112101+0000","last_became_peered":"2024-03-30T14:05:31.112101+0000","last_unstale":"2024-03-30T14:06:32.777014+0000","last_undegraded":"2024-03-30T14:06:32.777014+0000","last_fullsized":"2024-03-30T14:06:32.777014+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'60","last_scrub_stamp":"2024-03-30T14:06:00.211598+0000","last_deep_scrub":"108'28","last_deep_scrub_stamp":"2024-03-30T13:45:49.140418+0000","last_clean_scrub_stamp":"2024-03-30T14:06:00.211598+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:14.411804+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":104,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:56.776254+0000","last_change":"2024-03-30T14:06:56.776281+0000","last_active":"2024-03-30T14:06:56.776254+0000","last_peered":"2024-03-30T14:06:56.776254+0000","last_clean":"2024-03-30T14:06:56.776254+0000","last_became_active":"2024-03-30T14:05:48.013164+0000","last_became_peered":"2024-03-30T14:05:48.013164+0000","last_unstale":"2024-03-30T14:06:56.776254+0000","last_undegraded":"2024-03-30T14:06:56.776254+0000","last_fullsized":"2024-03-30T14:06:56.776254+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:56.776173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:17.366740+0000","last_clean_scrub_stamp":"2024-03-30T14:06:56.776173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:13.600189+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":587,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589249+0000","last_change":"2024-03-30T14:06:15.900394+0000","last_active":"2024-03-30T14:06:32.589249+0000","last_peered":"2024-03-30T14:06:32.589249+0000","last_clean":"2024-03-30T14:06:32.589249+0000","last_became_active":"2024-03-30T14:06:12.470995+0000","last_became_peered":"2024-03-30T14:06:12.470995+0000","last_unstale":"2024-03-30T14:06:32.589249+0000","last_undegraded":"2024-03-30T14:06:32.589249+0000","last_fullsized":"2024-03-30T14:06:32.589249+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:15.900273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:06:15.900273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:17.378314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":98,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:50.768371+0000","last_change":"2024-03-30T14:06:50.768397+0000","last_active":"2024-03-30T14:06:50.768371+0000","last_peered":"2024-03-30T14:06:50.768371+0000","last_clean":"2024-03-30T14:06:50.768371+0000","last_became_active":"2024-03-30T14:05:10.386648+0000","last_became_peered":"2024-03-30T14:05:10.386648+0000","last_unstale":"2024-03-30T14:06:50.768371+0000","last_undegraded":"2024-03-30T14:06:50.768371+0000","last_fullsized":"2024-03-30T14:06:50.768371+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:50.768296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:05:25.850051+0000","last_clean_scrub_stamp":"2024-03-30T14:06:50.768296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:14.329153+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"82'128","reported_seq":1026,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777169+0000","last_change":"2024-03-30T14:05:42.236006+0000","last_active":"2024-03-30T14:06:32.777169+0000","last_peered":"2024-03-30T14:06:32.777169+0000","last_clean":"2024-03-30T14:06:32.777169+0000","last_became_active":"2024-03-30T14:01:02.741907+0000","last_became_peered":"2024-03-30T14:01:02.741907+0000","last_unstale":"2024-03-30T14:06:32.777169+0000","last_undegraded":"2024-03-30T14:06:32.777169+0000","last_fullsized":"2024-03-30T14:06:32.777169+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:05:42.235898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:05:42.235898+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:56.772347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":473,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:55.762939+0000","last_change":"2024-03-30T14:06:55.762980+0000","last_active":"2024-03-30T14:06:55.762939+0000","last_peered":"2024-03-30T14:06:55.762939+0000","last_clean":"2024-03-30T14:06:55.762939+0000","last_became_active":"2024-03-30T14:06:12.469099+0000","last_became_peered":"2024-03-30T14:06:12.469099+0000","last_unstale":"2024-03-30T14:06:55.762939+0000","last_undegraded":"2024-03-30T14:06:55.762939+0000","last_fullsized":"2024-03-30T14:06:55.762939+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:55.762872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:06:55.762872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:07.050031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"421'24","reported_seq":1104,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776562+0000","last_change":"2024-03-30T14:05:36.395782+0000","last_active":"2024-03-30T14:06:32.776562+0000","last_peered":"2024-03-30T14:06:32.776562+0000","last_clean":"2024-03-30T14:06:32.776562+0000","last_became_active":"2024-03-30T14:05:02.367149+0000","last_became_peered":"2024-03-30T14:05:02.367149+0000","last_unstale":"2024-03-30T14:06:32.776562+0000","last_undegraded":"2024-03-30T14:06:32.776562+0000","last_fullsized":"2024-03-30T14:06:32.776562+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'24","last_scrub_stamp":"2024-03-30T14:05:36.395661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:05:36.395661+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:48.653139+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":90,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777283+0000","last_change":"2024-03-30T14:05:46.080898+0000","last_active":"2024-03-30T14:06:32.777283+0000","last_peered":"2024-03-30T14:06:32.777283+0000","last_clean":"2024-03-30T14:06:32.777283+0000","last_became_active":"2024-03-30T14:05:10.386580+0000","last_became_peered":"2024-03-30T14:05:10.386580+0000","last_unstale":"2024-03-30T14:06:32.777283+0000","last_undegraded":"2024-03-30T14:06:32.777283+0000","last_fullsized":"2024-03-30T14:06:32.777283+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:46.080797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:05:46.080797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:59.736296+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":106,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.590153+0000","last_change":"2024-03-30T14:06:03.763278+0000","last_active":"2024-03-30T14:06:32.590153+0000","last_peered":"2024-03-30T14:06:32.590153+0000","last_clean":"2024-03-30T14:06:32.590153+0000","last_became_active":"2024-03-30T14:05:10.392291+0000","last_became_peered":"2024-03-30T14:05:10.392291+0000","last_unstale":"2024-03-30T14:06:32.590153+0000","last_undegraded":"2024-03-30T14:06:32.590153+0000","last_fullsized":"2024-03-30T14:06:32.590153+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:03.763173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:06:03.763173+0000","last_clean_scrub_stamp":"2024-03-30T14:06:03.763173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:21.054140+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"451'129","reported_seq":540,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.590534+0000","last_change":"2024-03-30T14:06:16.562827+0000","last_active":"2024-03-30T14:06:32.590534+0000","last_peered":"2024-03-30T14:06:32.590534+0000","last_clean":"2024-03-30T14:06:32.590534+0000","last_became_active":"2024-03-30T14:05:05.006173+0000","last_became_peered":"2024-03-30T14:05:05.006173+0000","last_unstale":"2024-03-30T14:06:32.590534+0000","last_undegraded":"2024-03-30T14:06:32.590534+0000","last_fullsized":"2024-03-30T14:06:32.590534+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":4,"last_scrub":"451'129","last_scrub_stamp":"2024-03-30T14:06:16.562720+0000","last_deep_scrub":"451'129","last_deep_scrub_stamp":"2024-03-30T14:06:16.562720+0000","last_clean_scrub_stamp":"2024-03-30T14:06:16.562720+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:24.560038+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"489'11","reported_seq":116,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:50.657735+0000","last_change":"2024-03-30T14:06:04.917412+0000","last_active":"2024-03-30T14:06:50.657735+0000","last_peered":"2024-03-30T14:06:50.657735+0000","last_clean":"2024-03-30T14:06:50.657735+0000","last_became_active":"2024-03-30T14:05:10.373980+0000","last_became_peered":"2024-03-30T14:05:10.373980+0000","last_unstale":"2024-03-30T14:06:50.657735+0000","last_undegraded":"2024-03-30T14:06:50.657735+0000","last_fullsized":"2024-03-30T14:06:50.657735+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'11","last_scrub_stamp":"2024-03-30T14:06:04.917261+0000","last_deep_scrub":"376'10","last_deep_scrub_stamp":"2024-03-30T14:01:08.312068+0000","last_clean_scrub_stamp":"2024-03-30T14:06:04.917261+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:30.390464+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"421'39","reported_seq":61,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.588568+0000","last_change":"2024-03-30T14:05:48.033917+0000","last_active":"2024-03-30T14:06:32.588568+0000","last_peered":"2024-03-30T14:06:32.588568+0000","last_clean":"2024-03-30T14:06:32.588568+0000","last_became_active":"2024-03-30T14:05:48.016442+0000","last_became_peered":"2024-03-30T14:05:48.016442+0000","last_unstale":"2024-03-30T14:06:32.588568+0000","last_undegraded":"2024-03-30T14:06:32.588568+0000","last_fullsized":"2024-03-30T14:06:32.588568+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'39","last_scrub_stamp":"2024-03-30T14:05:33.969365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:05:33.969365+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:50.585237+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"512'4","reported_seq":98,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777884+0000","last_change":"2024-03-30T14:05:54.975769+0000","last_active":"2024-03-30T14:06:32.777884+0000","last_peered":"2024-03-30T14:06:32.777884+0000","last_clean":"2024-03-30T14:06:32.777884+0000","last_became_active":"2024-03-30T14:05:10.378426+0000","last_became_peered":"2024-03-30T14:05:10.378426+0000","last_unstale":"2024-03-30T14:06:32.777884+0000","last_undegraded":"2024-03-30T14:06:32.777884+0000","last_fullsized":"2024-03-30T14:06:32.777884+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"512'4","last_scrub_stamp":"2024-03-30T14:05:54.975662+0000","last_deep_scrub":"512'4","last_deep_scrub_stamp":"2024-03-30T14:05:54.975662+0000","last_clean_scrub_stamp":"2024-03-30T14:05:54.975662+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:19.490729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":45,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.588723+0000","last_change":"2024-03-30T14:05:48.019211+0000","last_active":"2024-03-30T14:06:32.588723+0000","last_peered":"2024-03-30T14:06:32.588723+0000","last_clean":"2024-03-30T14:06:32.588723+0000","last_became_active":"2024-03-30T14:05:48.018859+0000","last_became_peered":"2024-03-30T14:05:48.018859+0000","last_unstale":"2024-03-30T14:06:32.588723+0000","last_undegraded":"2024-03-30T14:06:32.588723+0000","last_fullsized":"2024-03-30T14:06:32.588723+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:27.350301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:05:27.350301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:47.723085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"82'128","reported_seq":773,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776110+0000","last_change":"2024-03-30T14:06:00.944621+0000","last_active":"2024-03-30T14:06:32.776110+0000","last_peered":"2024-03-30T14:06:32.776110+0000","last_clean":"2024-03-30T14:06:32.776110+0000","last_became_active":"2024-03-30T14:05:13.375492+0000","last_became_peered":"2024-03-30T14:05:13.375492+0000","last_unstale":"2024-03-30T14:06:32.776110+0000","last_undegraded":"2024-03-30T14:06:32.776110+0000","last_fullsized":"2024-03-30T14:06:32.776110+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:00.944500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:06:00.944500+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:24.213862+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"421'21","reported_seq":122,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.590867+0000","last_change":"2024-03-30T14:06:05.603664+0000","last_active":"2024-03-30T14:06:32.590867+0000","last_peered":"2024-03-30T14:06:32.590867+0000","last_clean":"2024-03-30T14:06:32.590867+0000","last_became_active":"2024-03-30T14:05:10.386775+0000","last_became_peered":"2024-03-30T14:05:10.386775+0000","last_unstale":"2024-03-30T14:06:32.590867+0000","last_undegraded":"2024-03-30T14:06:32.590867+0000","last_fullsized":"2024-03-30T14:06:32.590867+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-30T14:06:05.603574+0000","last_deep_scrub":"316'15","last_deep_scrub_stamp":"2024-03-30T13:58:13.210392+0000","last_clean_scrub_stamp":"2024-03-30T14:06:05.603574+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:22.216211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1069,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:53.981505+0000","last_change":"2024-03-30T14:06:53.981527+0000","last_active":"2024-03-30T14:06:53.981505+0000","last_peered":"2024-03-30T14:06:53.981505+0000","last_clean":"2024-03-30T14:06:53.981505+0000","last_became_active":"2024-03-30T14:05:48.018661+0000","last_became_peered":"2024-03-30T14:05:48.018661+0000","last_unstale":"2024-03-30T14:06:53.981505+0000","last_undegraded":"2024-03-30T14:06:53.981505+0000","last_fullsized":"2024-03-30T14:06:53.981505+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:53.981447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:05:39.330317+0000","last_clean_scrub_stamp":"2024-03-30T14:06:53.981447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:57.421291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":45,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589253+0000","last_change":"2024-03-30T14:05:48.022745+0000","last_active":"2024-03-30T14:06:32.589253+0000","last_peered":"2024-03-30T14:06:32.589253+0000","last_clean":"2024-03-30T14:06:32.589253+0000","last_became_active":"2024-03-30T14:05:48.022330+0000","last_became_peered":"2024-03-30T14:05:48.022330+0000","last_unstale":"2024-03-30T14:06:32.589253+0000","last_undegraded":"2024-03-30T14:06:32.589253+0000","last_fullsized":"2024-03-30T14:06:32.589253+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:41.511271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:57:09.967517+0000","last_clean_scrub_stamp":"2024-03-30T14:05:41.511271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:58.436772+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"494'2","reported_seq":103,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:53.837593+0000","last_change":"2024-03-30T14:06:53.837624+0000","last_active":"2024-03-30T14:06:53.837593+0000","last_peered":"2024-03-30T14:06:53.837593+0000","last_clean":"2024-03-30T14:06:53.837593+0000","last_became_active":"2024-03-30T14:05:10.393013+0000","last_became_peered":"2024-03-30T14:05:10.393013+0000","last_unstale":"2024-03-30T14:06:53.837593+0000","last_undegraded":"2024-03-30T14:06:53.837593+0000","last_fullsized":"2024-03-30T14:06:53.837593+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'2","last_scrub_stamp":"2024-03-30T14:06:53.837506+0000","last_deep_scrub":"494'2","last_deep_scrub_stamp":"2024-03-30T14:05:19.855273+0000","last_clean_scrub_stamp":"2024-03-30T14:06:53.837506+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:12.992049+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"82'128","reported_seq":760,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.588732+0000","last_change":"2024-03-30T14:06:09.972729+0000","last_active":"2024-03-30T14:06:32.588732+0000","last_peered":"2024-03-30T14:06:32.588732+0000","last_clean":"2024-03-30T14:06:32.588732+0000","last_became_active":"2024-03-30T14:05:51.005009+0000","last_became_peered":"2024-03-30T14:05:51.005009+0000","last_unstale":"2024-03-30T14:06:32.588732+0000","last_undegraded":"2024-03-30T14:06:32.588732+0000","last_fullsized":"2024-03-30T14:06:32.588732+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:09.972622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:06:09.972622+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:25.753999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"82'128","reported_seq":1025,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776635+0000","last_change":"2024-03-30T14:05:46.358651+0000","last_active":"2024-03-30T14:06:32.776635+0000","last_peered":"2024-03-30T14:06:32.776635+0000","last_clean":"2024-03-30T14:06:32.776635+0000","last_became_active":"2024-03-30T14:05:05.005735+0000","last_became_peered":"2024-03-30T14:05:05.005735+0000","last_unstale":"2024-03-30T14:06:32.776635+0000","last_undegraded":"2024-03-30T14:06:32.776635+0000","last_fullsized":"2024-03-30T14:06:32.776635+0000","mapping_epoch":486,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:05:46.358558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:05:46.358558+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:13.773414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"421'22","reported_seq":112,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775951+0000","last_change":"2024-03-30T14:06:15.495396+0000","last_active":"2024-03-30T14:06:32.775951+0000","last_peered":"2024-03-30T14:06:32.775951+0000","last_clean":"2024-03-30T14:06:32.775951+0000","last_became_active":"2024-03-30T14:05:23.031448+0000","last_became_peered":"2024-03-30T14:05:23.031448+0000","last_unstale":"2024-03-30T14:06:32.775951+0000","last_undegraded":"2024-03-30T14:06:32.775951+0000","last_fullsized":"2024-03-30T14:06:32.775951+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'22","last_scrub_stamp":"2024-03-30T14:06:15.495262+0000","last_deep_scrub":"218'16","last_deep_scrub_stamp":"2024-03-30T13:52:59.883027+0000","last_clean_scrub_stamp":"2024-03-30T14:06:15.495262+0000","objects_scrubbed":4,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:40.696296+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":90,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775969+0000","last_change":"2024-03-30T14:05:45.084438+0000","last_active":"2024-03-30T14:06:32.775969+0000","last_peered":"2024-03-30T14:06:32.775969+0000","last_clean":"2024-03-30T14:06:32.775969+0000","last_became_active":"2024-03-30T14:05:10.372067+0000","last_became_peered":"2024-03-30T14:05:10.372067+0000","last_unstale":"2024-03-30T14:06:32.775969+0000","last_undegraded":"2024-03-30T14:06:32.775969+0000","last_fullsized":"2024-03-30T14:06:32.775969+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:45.083761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:05:45.083761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:47.159548+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":106,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.591023+0000","last_change":"2024-03-30T14:05:59.603064+0000","last_active":"2024-03-30T14:06:32.591023+0000","last_peered":"2024-03-30T14:06:32.591023+0000","last_clean":"2024-03-30T14:06:32.591023+0000","last_became_active":"2024-03-30T14:05:10.385463+0000","last_became_peered":"2024-03-30T14:05:10.385463+0000","last_unstale":"2024-03-30T14:06:32.591023+0000","last_undegraded":"2024-03-30T14:06:32.591023+0000","last_fullsized":"2024-03-30T14:06:32.591023+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:59.602977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:05:59.602977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:07.981574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":731,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589690+0000","last_change":"2024-03-30T14:06:22.915739+0000","last_active":"2024-03-30T14:06:32.589690+0000","last_peered":"2024-03-30T14:06:32.589690+0000","last_clean":"2024-03-30T14:06:32.589690+0000","last_became_active":"2024-03-30T14:06:12.472120+0000","last_became_peered":"2024-03-30T14:06:12.472120+0000","last_unstale":"2024-03-30T14:06:32.589690+0000","last_undegraded":"2024-03-30T14:06:32.589690+0000","last_fullsized":"2024-03-30T14:06:32.589690+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:22.915639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:22.915639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:38.700852+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"421'21","reported_seq":55,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589246+0000","last_change":"2024-03-30T14:05:48.030377+0000","last_active":"2024-03-30T14:06:32.589246+0000","last_peered":"2024-03-30T14:06:32.589246+0000","last_clean":"2024-03-30T14:06:32.589246+0000","last_became_active":"2024-03-30T14:05:48.015202+0000","last_became_peered":"2024-03-30T14:05:48.015202+0000","last_unstale":"2024-03-30T14:06:32.589246+0000","last_undegraded":"2024-03-30T14:06:32.589246+0000","last_fullsized":"2024-03-30T14:06:32.589246+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-30T14:05:22.816090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:05:22.816090+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:51.413393+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":768,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775253+0000","last_change":"2024-03-30T14:05:55.478858+0000","last_active":"2024-03-30T14:06:32.775253+0000","last_peered":"2024-03-30T14:06:32.775253+0000","last_clean":"2024-03-30T14:06:32.775253+0000","last_became_active":"2024-03-30T14:05:23.027423+0000","last_became_peered":"2024-03-30T14:05:23.027423+0000","last_unstale":"2024-03-30T14:06:32.775253+0000","last_undegraded":"2024-03-30T14:06:32.775253+0000","last_fullsized":"2024-03-30T14:06:32.775253+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:55.478773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:05:55.478773+0000","last_clean_scrub_stamp":"2024-03-30T14:05:55.478773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:17.735701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"489'18","reported_seq":137,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:50.657653+0000","last_change":"2024-03-30T14:05:43.545571+0000","last_active":"2024-03-30T14:06:50.657653+0000","last_peered":"2024-03-30T14:06:50.657653+0000","last_clean":"2024-03-30T14:06:50.657653+0000","last_became_active":"2024-03-30T14:05:10.386593+0000","last_became_peered":"2024-03-30T14:05:10.386593+0000","last_unstale":"2024-03-30T14:06:50.657653+0000","last_undegraded":"2024-03-30T14:06:50.657653+0000","last_fullsized":"2024-03-30T14:06:50.657653+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'18","last_scrub_stamp":"2024-03-30T14:05:43.545472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:05:43.545472+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:03.127404+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1015,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:41.461300+0000","last_change":"2024-03-30T14:06:41.461323+0000","last_active":"2024-03-30T14:06:41.461300+0000","last_peered":"2024-03-30T14:06:41.461300+0000","last_clean":"2024-03-30T14:06:41.461300+0000","last_became_active":"2024-03-30T14:05:48.011917+0000","last_became_peered":"2024-03-30T14:05:48.011917+0000","last_unstale":"2024-03-30T14:06:41.461300+0000","last_undegraded":"2024-03-30T14:06:41.461300+0000","last_fullsized":"2024-03-30T14:06:41.461300+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:41.461213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:41.461213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:53.113563+0000","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"82'128","reported_seq":680,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.588570+0000","last_change":"2024-03-30T14:06:18.965794+0000","last_active":"2024-03-30T14:06:32.588570+0000","last_peered":"2024-03-30T14:06:32.588570+0000","last_clean":"2024-03-30T14:06:32.588570+0000","last_became_active":"2024-03-30T14:05:51.005660+0000","last_became_peered":"2024-03-30T14:05:51.005660+0000","last_unstale":"2024-03-30T14:06:32.588570+0000","last_undegraded":"2024-03-30T14:06:32.588570+0000","last_fullsized":"2024-03-30T14:06:32.588570+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:18.965695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:06:18.965695+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:47.725375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"421'30","reported_seq":635,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589213+0000","last_change":"2024-03-30T14:06:28.955171+0000","last_active":"2024-03-30T14:06:32.589213+0000","last_peered":"2024-03-30T14:06:32.589213+0000","last_clean":"2024-03-30T14:06:32.589213+0000","last_became_active":"2024-03-30T14:06:12.469999+0000","last_became_peered":"2024-03-30T14:06:12.469999+0000","last_unstale":"2024-03-30T14:06:32.589213+0000","last_undegraded":"2024-03-30T14:06:32.589213+0000","last_fullsized":"2024-03-30T14:06:32.589213+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'30","last_scrub_stamp":"2024-03-30T14:06:28.955038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:06:28.955038+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:51.311210+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":90,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776215+0000","last_change":"2024-03-30T14:06:11.830739+0000","last_active":"2024-03-30T14:06:32.776215+0000","last_peered":"2024-03-30T14:06:32.776215+0000","last_clean":"2024-03-30T14:06:32.776215+0000","last_became_active":"2024-03-30T14:05:10.380610+0000","last_became_peered":"2024-03-30T14:05:10.380610+0000","last_unstale":"2024-03-30T14:06:32.776215+0000","last_undegraded":"2024-03-30T14:06:32.776215+0000","last_fullsized":"2024-03-30T14:06:32.776215+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:11.830638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:06:11.830638+0000","last_clean_scrub_stamp":"2024-03-30T14:06:11.830638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:31.923697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"510'15","reported_seq":72,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:50.658019+0000","last_change":"2024-03-30T14:05:49.086863+0000","last_active":"2024-03-30T14:06:50.658019+0000","last_peered":"2024-03-30T14:06:50.658019+0000","last_clean":"2024-03-30T14:06:50.658019+0000","last_became_active":"2024-03-30T14:05:48.021934+0000","last_became_peered":"2024-03-30T14:05:48.021934+0000","last_unstale":"2024-03-30T14:06:50.658019+0000","last_undegraded":"2024-03-30T14:06:50.658019+0000","last_fullsized":"2024-03-30T14:06:50.658019+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'14","last_scrub_stamp":"2024-03-30T14:05:49.086752+0000","last_deep_scrub":"509'14","last_deep_scrub_stamp":"2024-03-30T14:05:49.086752+0000","last_clean_scrub_stamp":"2024-03-30T14:05:49.086752+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:03.467752+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":764,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776518+0000","last_change":"2024-03-30T14:06:13.569464+0000","last_active":"2024-03-30T14:06:32.776518+0000","last_peered":"2024-03-30T14:06:32.776518+0000","last_clean":"2024-03-30T14:06:32.776518+0000","last_became_active":"2024-03-30T14:05:23.029918+0000","last_became_peered":"2024-03-30T14:05:23.029918+0000","last_unstale":"2024-03-30T14:06:32.776518+0000","last_undegraded":"2024-03-30T14:06:32.776518+0000","last_fullsized":"2024-03-30T14:06:32.776518+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:13.569359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:03:15.101611+0000","last_clean_scrub_stamp":"2024-03-30T14:06:13.569359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:32.621038+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"82'128","reported_seq":656,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775076+0000","last_change":"2024-03-30T14:05:51.011444+0000","last_active":"2024-03-30T14:06:32.775076+0000","last_peered":"2024-03-30T14:06:32.775076+0000","last_clean":"2024-03-30T14:06:32.775076+0000","last_became_active":"2024-03-30T14:05:51.011025+0000","last_became_peered":"2024-03-30T14:05:51.011025+0000","last_unstale":"2024-03-30T14:06:32.775076+0000","last_undegraded":"2024-03-30T14:06:32.775076+0000","last_fullsized":"2024-03-30T14:06:32.775076+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:05:32.677209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:05:32.677209+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:52.215324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"421'27","reported_seq":1056,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777950+0000","last_change":"2024-03-30T14:05:59.364261+0000","last_active":"2024-03-30T14:06:32.777950+0000","last_peered":"2024-03-30T14:06:32.777950+0000","last_clean":"2024-03-30T14:06:32.777950+0000","last_became_active":"2024-03-30T14:04:32.226780+0000","last_became_peered":"2024-03-30T14:04:32.226780+0000","last_unstale":"2024-03-30T14:06:32.777950+0000","last_undegraded":"2024-03-30T14:06:32.777950+0000","last_fullsized":"2024-03-30T14:06:32.777950+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'27","last_scrub_stamp":"2024-03-30T14:05:59.364079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:05:59.364079+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:17.464204+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1022,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777036+0000","last_change":"2024-03-30T14:05:50.415539+0000","last_active":"2024-03-30T14:06:32.777036+0000","last_peered":"2024-03-30T14:06:32.777036+0000","last_clean":"2024-03-30T14:06:32.777036+0000","last_became_active":"2024-03-30T14:05:10.383734+0000","last_became_peered":"2024-03-30T14:05:10.383734+0000","last_unstale":"2024-03-30T14:06:32.777036+0000","last_undegraded":"2024-03-30T14:06:32.777036+0000","last_fullsized":"2024-03-30T14:06:32.777036+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:50.415461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:05:50.415461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:04.336645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1039,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775831+0000","last_change":"2024-03-30T14:06:28.345633+0000","last_active":"2024-03-30T14:06:32.775831+0000","last_peered":"2024-03-30T14:06:32.775831+0000","last_clean":"2024-03-30T14:06:32.775831+0000","last_became_active":"2024-03-30T14:05:02.377052+0000","last_became_peered":"2024-03-30T14:05:02.377052+0000","last_unstale":"2024-03-30T14:06:32.775831+0000","last_undegraded":"2024-03-30T14:06:32.775831+0000","last_fullsized":"2024-03-30T14:06:32.775831+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:28.345543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:34.815783+0000","last_clean_scrub_stamp":"2024-03-30T14:06:28.345543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:53.918070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":106,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589143+0000","last_change":"2024-03-30T14:05:48.017829+0000","last_active":"2024-03-30T14:06:32.589143+0000","last_peered":"2024-03-30T14:06:32.589143+0000","last_clean":"2024-03-30T14:06:32.589143+0000","last_became_active":"2024-03-30T14:05:48.017431+0000","last_became_peered":"2024-03-30T14:05:48.017431+0000","last_unstale":"2024-03-30T14:06:32.589143+0000","last_undegraded":"2024-03-30T14:06:32.589143+0000","last_fullsized":"2024-03-30T14:06:32.589143+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:39.545440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:24.598976+0000","last_clean_scrub_stamp":"2024-03-30T14:05:39.545440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:06.459494+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"421'21","reported_seq":1065,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775938+0000","last_change":"2024-03-30T14:06:00.204082+0000","last_active":"2024-03-30T14:06:32.775938+0000","last_peered":"2024-03-30T14:06:32.775938+0000","last_clean":"2024-03-30T14:06:32.775938+0000","last_became_active":"2024-03-30T14:04:54.327260+0000","last_became_peered":"2024-03-30T14:04:54.327260+0000","last_unstale":"2024-03-30T14:06:32.775938+0000","last_undegraded":"2024-03-30T14:06:32.775938+0000","last_fullsized":"2024-03-30T14:06:32.775938+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-30T14:06:00.203989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:06:00.203989+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:11.179397+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":98,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776000+0000","last_change":"2024-03-30T14:06:27.814638+0000","last_active":"2024-03-30T14:06:32.776000+0000","last_peered":"2024-03-30T14:06:32.776000+0000","last_clean":"2024-03-30T14:06:32.776000+0000","last_became_active":"2024-03-30T14:05:10.393274+0000","last_became_peered":"2024-03-30T14:05:10.393274+0000","last_unstale":"2024-03-30T14:06:32.776000+0000","last_undegraded":"2024-03-30T14:06:32.776000+0000","last_fullsized":"2024-03-30T14:06:32.776000+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:27.814502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:06:27.814502+0000","last_clean_scrub_stamp":"2024-03-30T14:06:27.814502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:35.697945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":100,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.590236+0000","last_change":"2024-03-30T14:06:06.583759+0000","last_active":"2024-03-30T14:06:32.590236+0000","last_peered":"2024-03-30T14:06:32.590236+0000","last_clean":"2024-03-30T14:06:32.590236+0000","last_became_active":"2024-03-30T14:05:02.373957+0000","last_became_peered":"2024-03-30T14:05:02.373957+0000","last_unstale":"2024-03-30T14:06:32.590236+0000","last_undegraded":"2024-03-30T14:06:32.590236+0000","last_fullsized":"2024-03-30T14:06:32.590236+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:06.583675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:06:06.583675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:18.792149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":784,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:46.502034+0000","last_change":"2024-03-30T14:06:46.502074+0000","last_active":"2024-03-30T14:06:46.502034+0000","last_peered":"2024-03-30T14:06:46.502034+0000","last_clean":"2024-03-30T14:06:46.502034+0000","last_became_active":"2024-03-30T14:05:48.023903+0000","last_became_peered":"2024-03-30T14:05:48.023903+0000","last_unstale":"2024-03-30T14:06:46.502034+0000","last_undegraded":"2024-03-30T14:06:46.502034+0000","last_fullsized":"2024-03-30T14:06:46.502034+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:46.501948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:28.314391+0000","last_clean_scrub_stamp":"2024-03-30T14:06:46.501948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:05.312856+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"82'128","reported_seq":961,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589834+0000","last_change":"2024-03-30T14:06:19.866986+0000","last_active":"2024-03-30T14:06:32.589834+0000","last_peered":"2024-03-30T14:06:32.589834+0000","last_clean":"2024-03-30T14:06:32.589834+0000","last_became_active":"2024-03-30T14:06:12.472272+0000","last_became_peered":"2024-03-30T14:06:12.472272+0000","last_unstale":"2024-03-30T14:06:32.589834+0000","last_undegraded":"2024-03-30T14:06:32.589834+0000","last_fullsized":"2024-03-30T14:06:32.589834+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":368,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-30T14:06:19.866782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:40:27.659073+0000","last_clean_scrub_stamp":"2024-03-30T14:06:19.866782+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:43.744359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"421'27","reported_seq":130,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589558+0000","last_change":"2024-03-30T14:06:31.491589+0000","last_active":"2024-03-30T14:06:32.589558+0000","last_peered":"2024-03-30T14:06:32.589558+0000","last_clean":"2024-03-30T14:06:32.589558+0000","last_became_active":"2024-03-30T14:05:10.385458+0000","last_became_peered":"2024-03-30T14:05:10.385458+0000","last_unstale":"2024-03-30T14:06:32.589558+0000","last_undegraded":"2024-03-30T14:06:32.589558+0000","last_fullsized":"2024-03-30T14:06:32.589558+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'27","last_scrub_stamp":"2024-03-30T14:06:31.491495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:06:31.491495+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:46.790429+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":53,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589131+0000","last_change":"2024-03-30T14:06:31.869847+0000","last_active":"2024-03-30T14:06:32.589131+0000","last_peered":"2024-03-30T14:06:32.589131+0000","last_clean":"2024-03-30T14:06:32.589131+0000","last_became_active":"2024-03-30T14:05:48.022381+0000","last_became_peered":"2024-03-30T14:05:48.022381+0000","last_unstale":"2024-03-30T14:06:32.589131+0000","last_undegraded":"2024-03-30T14:06:32.589131+0000","last_fullsized":"2024-03-30T14:06:32.589131+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:31.869751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:48:25.494903+0000","last_clean_scrub_stamp":"2024-03-30T14:06:31.869751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:58.784022+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":90,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775366+0000","last_change":"2024-03-30T14:05:51.054553+0000","last_active":"2024-03-30T14:06:32.775366+0000","last_peered":"2024-03-30T14:06:32.775366+0000","last_clean":"2024-03-30T14:06:32.775366+0000","last_became_active":"2024-03-30T14:05:10.380531+0000","last_became_peered":"2024-03-30T14:05:10.380531+0000","last_unstale":"2024-03-30T14:06:32.775366+0000","last_undegraded":"2024-03-30T14:06:32.775366+0000","last_fullsized":"2024-03-30T14:06:32.775366+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:51.054480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:42.616155+0000","last_clean_scrub_stamp":"2024-03-30T14:05:51.054480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:59.439677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":100,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589986+0000","last_change":"2024-03-30T14:06:10.742316+0000","last_active":"2024-03-30T14:06:32.589986+0000","last_peered":"2024-03-30T14:06:32.589986+0000","last_clean":"2024-03-30T14:06:32.589986+0000","last_became_active":"2024-03-30T14:05:02.374776+0000","last_became_peered":"2024-03-30T14:05:02.374776+0000","last_unstale":"2024-03-30T14:06:32.589986+0000","last_undegraded":"2024-03-30T14:06:32.589986+0000","last_fullsized":"2024-03-30T14:06:32.589986+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:10.742188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:10.742188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:12.829107+0000","scrub_duration":146,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"503'14","reported_seq":116,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:45.270770+0000","last_change":"2024-03-30T14:06:45.270798+0000","last_active":"2024-03-30T14:06:45.270770+0000","last_peered":"2024-03-30T14:06:45.270770+0000","last_clean":"2024-03-30T14:06:45.270770+0000","last_became_active":"2024-03-30T14:05:02.373527+0000","last_became_peered":"2024-03-30T14:05:02.373527+0000","last_unstale":"2024-03-30T14:06:45.270770+0000","last_undegraded":"2024-03-30T14:06:45.270770+0000","last_fullsized":"2024-03-30T14:06:45.270770+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'14","last_scrub_stamp":"2024-03-30T14:06:45.270708+0000","last_deep_scrub":"503'14","last_deep_scrub_stamp":"2024-03-30T14:05:38.484297+0000","last_clean_scrub_stamp":"2024-03-30T14:06:45.270708+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:45.854051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":788,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775672+0000","last_change":"2024-03-30T14:06:08.523691+0000","last_active":"2024-03-30T14:06:32.775672+0000","last_peered":"2024-03-30T14:06:32.775672+0000","last_clean":"2024-03-30T14:06:32.775672+0000","last_became_active":"2024-03-30T14:05:48.023811+0000","last_became_peered":"2024-03-30T14:05:48.023811+0000","last_unstale":"2024-03-30T14:06:32.775672+0000","last_undegraded":"2024-03-30T14:06:32.775672+0000","last_fullsized":"2024-03-30T14:06:32.775672+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:08.523576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:06:08.523576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:31.810652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1037,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775535+0000","last_change":"2024-03-30T14:06:25.993285+0000","last_active":"2024-03-30T14:06:32.775535+0000","last_peered":"2024-03-30T14:06:32.775535+0000","last_clean":"2024-03-30T14:06:32.775535+0000","last_became_active":"2024-03-30T14:05:10.382829+0000","last_became_peered":"2024-03-30T14:05:10.382829+0000","last_unstale":"2024-03-30T14:06:32.775535+0000","last_undegraded":"2024-03-30T14:06:32.775535+0000","last_fullsized":"2024-03-30T14:06:32.775535+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:25.993110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:06:25.993110+0000","last_clean_scrub_stamp":"2024-03-30T14:06:25.993110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:44.320090+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":90,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775267+0000","last_change":"2024-03-30T14:06:00.002874+0000","last_active":"2024-03-30T14:06:32.775267+0000","last_peered":"2024-03-30T14:06:32.775267+0000","last_clean":"2024-03-30T14:06:32.775267+0000","last_became_active":"2024-03-30T14:05:10.373968+0000","last_became_peered":"2024-03-30T14:05:10.373968+0000","last_unstale":"2024-03-30T14:06:32.775267+0000","last_undegraded":"2024-03-30T14:06:32.775267+0000","last_fullsized":"2024-03-30T14:06:32.775267+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:00.002758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:00.002758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:22.058916+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"421'27","reported_seq":1093,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777543+0000","last_change":"2024-03-30T14:06:27.029077+0000","last_active":"2024-03-30T14:06:32.777543+0000","last_peered":"2024-03-30T14:06:32.777543+0000","last_clean":"2024-03-30T14:06:32.777543+0000","last_became_active":"2024-03-30T13:58:47.204696+0000","last_became_peered":"2024-03-30T13:58:47.204696+0000","last_unstale":"2024-03-30T14:06:32.777543+0000","last_undegraded":"2024-03-30T14:06:32.777543+0000","last_fullsized":"2024-03-30T14:06:32.777543+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'27","last_scrub_stamp":"2024-03-30T14:06:27.028900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:06:27.028900+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:34.843559+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":100,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.588248+0000","last_change":"2024-03-30T14:05:58.604822+0000","last_active":"2024-03-30T14:06:32.588248+0000","last_peered":"2024-03-30T14:06:32.588248+0000","last_clean":"2024-03-30T14:06:32.588248+0000","last_became_active":"2024-03-30T14:05:02.378933+0000","last_became_peered":"2024-03-30T14:05:02.378933+0000","last_unstale":"2024-03-30T14:06:32.588248+0000","last_undegraded":"2024-03-30T14:06:32.588248+0000","last_fullsized":"2024-03-30T14:06:32.588248+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:58.604739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:05:58.604739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:27.526047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":765,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:37.465699+0000","last_change":"2024-03-30T14:06:37.465721+0000","last_active":"2024-03-30T14:06:37.465699+0000","last_peered":"2024-03-30T14:06:37.465699+0000","last_clean":"2024-03-30T14:06:37.465699+0000","last_became_active":"2024-03-30T14:05:23.028766+0000","last_became_peered":"2024-03-30T14:05:23.028766+0000","last_unstale":"2024-03-30T14:06:37.465699+0000","last_undegraded":"2024-03-30T14:06:37.465699+0000","last_fullsized":"2024-03-30T14:06:37.465699+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:37.465634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:06:37.465634+0000","last_clean_scrub_stamp":"2024-03-30T14:06:37.465634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:42.555255+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1016,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:46.497543+0000","last_change":"2024-03-30T14:06:46.497583+0000","last_active":"2024-03-30T14:06:46.497543+0000","last_peered":"2024-03-30T14:06:46.497543+0000","last_clean":"2024-03-30T14:06:46.497543+0000","last_became_active":"2024-03-30T13:58:48.180840+0000","last_became_peered":"2024-03-30T13:58:48.180840+0000","last_unstale":"2024-03-30T14:06:46.497543+0000","last_undegraded":"2024-03-30T14:06:46.497543+0000","last_fullsized":"2024-03-30T14:06:46.497543+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:46.497454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:05:19.446184+0000","last_clean_scrub_stamp":"2024-03-30T14:06:46.497454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:08.898646+0000","scrub_duration":209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"421'35","reported_seq":538,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:40.494189+0000","last_change":"2024-03-30T14:06:40.494221+0000","last_active":"2024-03-30T14:06:40.494189+0000","last_peered":"2024-03-30T14:06:40.494189+0000","last_clean":"2024-03-30T14:06:40.494189+0000","last_became_active":"2024-03-30T14:05:23.032100+0000","last_became_peered":"2024-03-30T14:05:23.032100+0000","last_unstale":"2024-03-30T14:06:40.494189+0000","last_undegraded":"2024-03-30T14:06:40.494189+0000","last_fullsized":"2024-03-30T14:06:40.494189+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'35","last_scrub_stamp":"2024-03-30T14:06:40.494130+0000","last_deep_scrub":"421'35","last_deep_scrub_stamp":"2024-03-30T14:05:15.761820+0000","last_clean_scrub_stamp":"2024-03-30T14:06:40.494130+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:55.727198+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"65'1","reported_seq":1039,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:34.429227+0000","last_change":"2024-03-30T14:06:34.429254+0000","last_active":"2024-03-30T14:06:34.429227+0000","last_peered":"2024-03-30T14:06:34.429227+0000","last_clean":"2024-03-30T14:06:34.429227+0000","last_became_active":"2024-03-30T14:05:02.375804+0000","last_became_peered":"2024-03-30T14:05:02.375804+0000","last_unstale":"2024-03-30T14:06:34.429227+0000","last_undegraded":"2024-03-30T14:06:34.429227+0000","last_fullsized":"2024-03-30T14:06:34.429227+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T14:06:34.429161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:06:34.429161+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:35.338490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"489'15","reported_seq":1366,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:50.657789+0000","last_change":"2024-03-30T14:06:33.980613+0000","last_active":"2024-03-30T14:06:50.657789+0000","last_peered":"2024-03-30T14:06:50.657789+0000","last_clean":"2024-03-30T14:06:50.657789+0000","last_became_active":"2024-03-30T14:05:10.388934+0000","last_became_peered":"2024-03-30T14:05:10.388934+0000","last_unstale":"2024-03-30T14:06:50.657789+0000","last_undegraded":"2024-03-30T14:06:50.657789+0000","last_fullsized":"2024-03-30T14:06:50.657789+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'15","last_scrub_stamp":"2024-03-30T14:06:33.980502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:06:33.980502+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:36.485976+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":94,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775775+0000","last_change":"2024-03-30T14:05:53.028418+0000","last_active":"2024-03-30T14:06:32.775775+0000","last_peered":"2024-03-30T14:06:32.775775+0000","last_clean":"2024-03-30T14:06:32.775775+0000","last_became_active":"2024-03-30T14:05:10.381039+0000","last_became_peered":"2024-03-30T14:05:10.381039+0000","last_unstale":"2024-03-30T14:06:32.775775+0000","last_undegraded":"2024-03-30T14:06:32.775775+0000","last_fullsized":"2024-03-30T14:06:32.775775+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-30T14:05:53.028321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:05:53.028321+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:02.843553+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1850,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"421'29","reported_seq":120,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.590298+0000","last_change":"2024-03-30T14:06:30.511782+0000","last_active":"2024-03-30T14:06:32.590298+0000","last_peered":"2024-03-30T14:06:32.590298+0000","last_clean":"2024-03-30T14:06:32.590298+0000","last_became_active":"2024-03-30T14:05:02.370465+0000","last_became_peered":"2024-03-30T14:05:02.370465+0000","last_unstale":"2024-03-30T14:06:32.590298+0000","last_undegraded":"2024-03-30T14:06:32.590298+0000","last_fullsized":"2024-03-30T14:06:32.590298+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'29","last_scrub_stamp":"2024-03-30T14:06:30.511684+0000","last_deep_scrub":"421'29","last_deep_scrub_stamp":"2024-03-30T14:06:30.511684+0000","last_clean_scrub_stamp":"2024-03-30T14:06:30.511684+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:56.873539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":998,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776641+0000","last_change":"2024-03-30T14:06:18.317210+0000","last_active":"2024-03-30T14:06:32.776641+0000","last_peered":"2024-03-30T14:06:32.776641+0000","last_clean":"2024-03-30T14:06:32.776641+0000","last_became_active":"2024-03-30T14:04:54.310132+0000","last_became_peered":"2024-03-30T14:04:54.310132+0000","last_unstale":"2024-03-30T14:06:32.776641+0000","last_undegraded":"2024-03-30T14:06:32.776641+0000","last_fullsized":"2024-03-30T14:06:32.776641+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:18.317123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:06:18.317123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:21.307984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1002,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:33.313795+0000","last_change":"2024-03-30T14:06:33.313843+0000","last_active":"2024-03-30T14:06:33.313795+0000","last_peered":"2024-03-30T14:06:33.313795+0000","last_clean":"2024-03-30T14:06:33.313795+0000","last_became_active":"2024-03-30T14:05:02.366337+0000","last_became_peered":"2024-03-30T14:05:02.366337+0000","last_unstale":"2024-03-30T14:06:33.313795+0000","last_undegraded":"2024-03-30T14:06:33.313795+0000","last_fullsized":"2024-03-30T14:06:33.313795+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:33.313697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:06:33.313697+0000","last_clean_scrub_stamp":"2024-03-30T14:06:33.313697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:59.275008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":106,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.590769+0000","last_change":"2024-03-30T14:06:12.823187+0000","last_active":"2024-03-30T14:06:32.590769+0000","last_peered":"2024-03-30T14:06:32.590769+0000","last_clean":"2024-03-30T14:06:32.590769+0000","last_became_active":"2024-03-30T14:05:48.013445+0000","last_became_peered":"2024-03-30T14:05:48.013445+0000","last_unstale":"2024-03-30T14:06:32.590769+0000","last_undegraded":"2024-03-30T14:06:32.590769+0000","last_fullsized":"2024-03-30T14:06:32.590769+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:12.823088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:12.823088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:22.224843+0000","scrub_duration":226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"507'46","reported_seq":926,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589316+0000","last_change":"2024-03-30T14:06:12.468173+0000","last_active":"2024-03-30T14:06:32.589316+0000","last_peered":"2024-03-30T14:06:32.589316+0000","last_clean":"2024-03-30T14:06:32.589316+0000","last_became_active":"2024-03-30T14:06:12.467794+0000","last_became_peered":"2024-03-30T14:06:12.467794+0000","last_unstale":"2024-03-30T14:06:32.589316+0000","last_undegraded":"2024-03-30T14:06:32.589316+0000","last_fullsized":"2024-03-30T14:06:32.589316+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'46","last_scrub_stamp":"2024-03-30T14:05:47.333753+0000","last_deep_scrub":"507'46","last_deep_scrub_stamp":"2024-03-30T14:05:47.333753+0000","last_clean_scrub_stamp":"2024-03-30T14:05:47.333753+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:12.743805+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":768,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775563+0000","last_change":"2024-03-30T14:05:54.490003+0000","last_active":"2024-03-30T14:06:32.775563+0000","last_peered":"2024-03-30T14:06:32.775563+0000","last_clean":"2024-03-30T14:06:32.775563+0000","last_became_active":"2024-03-30T14:05:23.028851+0000","last_became_peered":"2024-03-30T14:05:23.028851+0000","last_unstale":"2024-03-30T14:06:32.775563+0000","last_undegraded":"2024-03-30T14:06:32.775563+0000","last_fullsized":"2024-03-30T14:06:32.775563+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:54.489923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:46:43.088964+0000","last_clean_scrub_stamp":"2024-03-30T14:05:54.489923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:06.714551+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1010,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776990+0000","last_change":"2024-03-30T14:05:45.378091+0000","last_active":"2024-03-30T14:06:32.776990+0000","last_peered":"2024-03-30T14:06:32.776990+0000","last_clean":"2024-03-30T14:06:32.776990+0000","last_became_active":"2024-03-30T13:42:49.322665+0000","last_became_peered":"2024-03-30T13:42:49.322665+0000","last_unstale":"2024-03-30T14:06:32.776990+0000","last_undegraded":"2024-03-30T14:06:32.776990+0000","last_fullsized":"2024-03-30T14:06:32.776990+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:45.377972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:30.465974+0000","last_clean_scrub_stamp":"2024-03-30T14:05:45.377972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:03.313779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1014,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.778136+0000","last_change":"2024-03-30T14:05:48.017012+0000","last_active":"2024-03-30T14:06:32.778136+0000","last_peered":"2024-03-30T14:06:32.778136+0000","last_clean":"2024-03-30T14:06:32.778136+0000","last_became_active":"2024-03-30T14:05:48.016496+0000","last_became_peered":"2024-03-30T14:05:48.016496+0000","last_unstale":"2024-03-30T14:06:32.778136+0000","last_undegraded":"2024-03-30T14:06:32.778136+0000","last_fullsized":"2024-03-30T14:06:32.778136+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:46.468684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:05:46.468684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:11.981952+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"421'11","reported_seq":1075,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.777563+0000","last_change":"2024-03-30T14:05:47.411296+0000","last_active":"2024-03-30T14:06:32.777563+0000","last_peered":"2024-03-30T14:06:32.777563+0000","last_clean":"2024-03-30T14:06:32.777563+0000","last_became_active":"2024-03-30T14:05:02.355441+0000","last_became_peered":"2024-03-30T14:05:02.355441+0000","last_unstale":"2024-03-30T14:06:32.777563+0000","last_undegraded":"2024-03-30T14:06:32.777563+0000","last_fullsized":"2024-03-30T14:06:32.777563+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'11","last_scrub_stamp":"2024-03-30T14:05:47.411182+0000","last_deep_scrub":"88'7","last_deep_scrub_stamp":"2024-03-30T13:43:50.960980+0000","last_clean_scrub_stamp":"2024-03-30T14:05:47.411182+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:49.839380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"186'2","reported_seq":440,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589389+0000","last_change":"2024-03-30T14:06:06.924072+0000","last_active":"2024-03-30T14:06:32.589389+0000","last_peered":"2024-03-30T14:06:32.589389+0000","last_clean":"2024-03-30T14:06:32.589389+0000","last_became_active":"2024-03-30T13:58:47.193793+0000","last_became_peered":"2024-03-30T13:58:47.193793+0000","last_unstale":"2024-03-30T14:06:32.589389+0000","last_undegraded":"2024-03-30T14:06:32.589389+0000","last_fullsized":"2024-03-30T14:06:32.589389+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-30T14:06:06.923970+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-30T13:51:15.932076+0000","last_clean_scrub_stamp":"2024-03-30T14:06:06.923970+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:31.937671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":90,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775698+0000","last_change":"2024-03-30T14:05:56.004008+0000","last_active":"2024-03-30T14:06:32.775698+0000","last_peered":"2024-03-30T14:06:32.775698+0000","last_clean":"2024-03-30T14:06:32.775698+0000","last_became_active":"2024-03-30T14:05:10.378262+0000","last_became_peered":"2024-03-30T14:05:10.378262+0000","last_unstale":"2024-03-30T14:06:32.775698+0000","last_undegraded":"2024-03-30T14:06:32.775698+0000","last_fullsized":"2024-03-30T14:06:32.775698+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:56.003910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:44:41.633284+0000","last_clean_scrub_stamp":"2024-03-30T14:05:56.003910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:04.783154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":53,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589039+0000","last_change":"2024-03-30T14:06:06.031064+0000","last_active":"2024-03-30T14:06:32.589039+0000","last_peered":"2024-03-30T14:06:32.589039+0000","last_clean":"2024-03-30T14:06:32.589039+0000","last_became_active":"2024-03-30T14:05:48.020210+0000","last_became_peered":"2024-03-30T14:05:48.020210+0000","last_unstale":"2024-03-30T14:06:32.589039+0000","last_undegraded":"2024-03-30T14:06:32.589039+0000","last_fullsized":"2024-03-30T14:06:32.589039+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:06.030953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:06.030953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:19.208110+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"525'25","reported_seq":1115,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:49.265722+0000","last_change":"2024-03-30T14:06:49.265774+0000","last_active":"2024-03-30T14:06:49.265722+0000","last_peered":"2024-03-30T14:06:49.265722+0000","last_clean":"2024-03-30T14:06:49.265722+0000","last_became_active":"2024-03-30T14:05:31.110927+0000","last_became_peered":"2024-03-30T14:05:31.110927+0000","last_unstale":"2024-03-30T14:06:49.265722+0000","last_undegraded":"2024-03-30T14:06:49.265722+0000","last_fullsized":"2024-03-30T14:06:49.265722+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"525'25","last_scrub_stamp":"2024-03-30T14:06:49.265649+0000","last_deep_scrub":"525'25","last_deep_scrub_stamp":"2024-03-30T14:06:49.265649+0000","last_clean_scrub_stamp":"2024-03-30T14:06:49.265649+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:50.035575+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":450,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:44.968678+0000","last_change":"2024-03-30T14:06:44.968702+0000","last_active":"2024-03-30T14:06:44.968678+0000","last_peered":"2024-03-30T14:06:44.968678+0000","last_clean":"2024-03-30T14:06:44.968678+0000","last_became_active":"2024-03-30T14:05:31.110665+0000","last_became_peered":"2024-03-30T14:05:31.110665+0000","last_unstale":"2024-03-30T14:06:44.968678+0000","last_undegraded":"2024-03-30T14:06:44.968678+0000","last_fullsized":"2024-03-30T14:06:44.968678+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:44.968609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:47:37.513596+0000","last_clean_scrub_stamp":"2024-03-30T14:06:44.968609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:12.814206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":53,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:48.968294+0000","last_change":"2024-03-30T14:06:48.968318+0000","last_active":"2024-03-30T14:06:48.968294+0000","last_peered":"2024-03-30T14:06:48.968294+0000","last_clean":"2024-03-30T14:06:48.968294+0000","last_became_active":"2024-03-30T14:05:48.019011+0000","last_became_peered":"2024-03-30T14:05:48.019011+0000","last_unstale":"2024-03-30T14:06:48.968294+0000","last_undegraded":"2024-03-30T14:06:48.968294+0000","last_fullsized":"2024-03-30T14:06:48.968294+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:48.968218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:20.069864+0000","last_clean_scrub_stamp":"2024-03-30T14:06:48.968218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:08:10.088088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":756,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775310+0000","last_change":"2024-03-30T14:05:42.706598+0000","last_active":"2024-03-30T14:06:32.775310+0000","last_peered":"2024-03-30T14:06:32.775310+0000","last_clean":"2024-03-30T14:06:32.775310+0000","last_became_active":"2024-03-30T14:05:23.029117+0000","last_became_peered":"2024-03-30T14:05:23.029117+0000","last_unstale":"2024-03-30T14:06:32.775310+0000","last_undegraded":"2024-03-30T14:06:32.775310+0000","last_fullsized":"2024-03-30T14:06:32.775310+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:42.706464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:27.061281+0000","last_clean_scrub_stamp":"2024-03-30T14:05:42.706464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:11.966576+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"421'46","reported_seq":116,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589249+0000","last_change":"2024-03-30T14:05:44.527597+0000","last_active":"2024-03-30T14:06:32.589249+0000","last_peered":"2024-03-30T14:06:32.589249+0000","last_clean":"2024-03-30T14:06:32.589249+0000","last_became_active":"2024-03-30T14:05:02.387256+0000","last_became_peered":"2024-03-30T14:05:02.387256+0000","last_unstale":"2024-03-30T14:06:32.589249+0000","last_undegraded":"2024-03-30T14:06:32.589249+0000","last_fullsized":"2024-03-30T14:06:32.589249+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'46","last_scrub_stamp":"2024-03-30T14:05:44.527472+0000","last_deep_scrub":"95'22","last_deep_scrub_stamp":"2024-03-30T13:44:29.151791+0000","last_clean_scrub_stamp":"2024-03-30T14:05:44.527472+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:02.849183+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":108,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:39.360721+0000","last_change":"2024-03-30T14:06:39.360743+0000","last_active":"2024-03-30T14:06:39.360721+0000","last_peered":"2024-03-30T14:06:39.360721+0000","last_clean":"2024-03-30T14:06:39.360721+0000","last_became_active":"2024-03-30T14:05:02.385930+0000","last_became_peered":"2024-03-30T14:05:02.385930+0000","last_unstale":"2024-03-30T14:06:39.360721+0000","last_undegraded":"2024-03-30T14:06:39.360721+0000","last_fullsized":"2024-03-30T14:06:39.360721+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:39.360648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:49:20.499768+0000","last_clean_scrub_stamp":"2024-03-30T14:06:39.360648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:56.678299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":985,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775134+0000","last_change":"2024-03-30T14:06:07.278680+0000","last_active":"2024-03-30T14:06:32.775134+0000","last_peered":"2024-03-30T14:06:32.775134+0000","last_clean":"2024-03-30T14:06:32.775134+0000","last_became_active":"2024-03-30T14:04:32.203392+0000","last_became_peered":"2024-03-30T14:04:32.203392+0000","last_unstale":"2024-03-30T14:06:32.775134+0000","last_undegraded":"2024-03-30T14:06:32.775134+0000","last_fullsized":"2024-03-30T14:06:32.775134+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:07.278519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:48.304004+0000","last_clean_scrub_stamp":"2024-03-30T14:06:07.278519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:12.295102+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":106,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.589889+0000","last_change":"2024-03-30T14:06:04.777125+0000","last_active":"2024-03-30T14:06:32.589889+0000","last_peered":"2024-03-30T14:06:32.589889+0000","last_clean":"2024-03-30T14:06:32.589889+0000","last_became_active":"2024-03-30T14:05:10.385006+0000","last_became_peered":"2024-03-30T14:05:10.385006+0000","last_unstale":"2024-03-30T14:06:32.589889+0000","last_undegraded":"2024-03-30T14:06:32.589889+0000","last_fullsized":"2024-03-30T14:06:32.589889+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:04.776993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:06:04.776993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:17.047662+0000","scrub_duration":173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"421'21","reported_seq":1040,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776002+0000","last_change":"2024-03-30T14:05:51.406671+0000","last_active":"2024-03-30T14:06:32.776002+0000","last_peered":"2024-03-30T14:06:32.776002+0000","last_clean":"2024-03-30T14:06:32.776002+0000","last_became_active":"2024-03-30T14:05:31.111349+0000","last_became_peered":"2024-03-30T14:05:31.111349+0000","last_unstale":"2024-03-30T14:06:32.776002+0000","last_undegraded":"2024-03-30T14:06:32.776002+0000","last_fullsized":"2024-03-30T14:06:32.776002+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-30T14:05:51.406568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:46.298397+0000","last_clean_scrub_stamp":"2024-03-30T14:05:51.406568+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:20.524986+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"65'1","reported_seq":789,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775438+0000","last_change":"2024-03-30T14:05:56.474530+0000","last_active":"2024-03-30T14:06:32.775438+0000","last_peered":"2024-03-30T14:06:32.775438+0000","last_clean":"2024-03-30T14:06:32.775438+0000","last_became_active":"2024-03-30T14:05:48.023279+0000","last_became_peered":"2024-03-30T14:05:48.023279+0000","last_unstale":"2024-03-30T14:06:32.775438+0000","last_undegraded":"2024-03-30T14:06:32.775438+0000","last_fullsized":"2024-03-30T14:06:32.775438+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-30T14:05:56.474240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:44.287804+0000","last_clean_scrub_stamp":"2024-03-30T14:05:56.474240+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:10.291749+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1048,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.775107+0000","last_change":"2024-03-30T14:06:05.161540+0000","last_active":"2024-03-30T14:06:32.775107+0000","last_peered":"2024-03-30T14:06:32.775107+0000","last_clean":"2024-03-30T14:06:32.775107+0000","last_became_active":"2024-03-30T14:05:10.382228+0000","last_became_peered":"2024-03-30T14:05:10.382228+0000","last_unstale":"2024-03-30T14:06:32.775107+0000","last_undegraded":"2024-03-30T14:06:32.775107+0000","last_fullsized":"2024-03-30T14:06:32.775107+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:06:05.161455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:52:17.548567+0000","last_clean_scrub_stamp":"2024-03-30T14:06:05.161455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:07:05.767872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1017,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-30T14:06:32.776236+0000","last_change":"2024-03-30T14:05:56.506831+0000","last_active":"2024-03-30T14:06:32.776236+0000","last_peered":"2024-03-30T14:06:32.776236+0000","last_clean":"2024-03-30T14:06:32.776236+0000","last_became_active":"2024-03-30T14:05:48.009068+0000","last_became_peered":"2024-03-30T14:05:48.009068+0000","last_unstale":"2024-03-30T14:06:32.776236+0000","last_undegraded":"2024-03-30T14:06:32.776236+0000","last_fullsized":"2024-03-30T14:06:32.776236+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:05:56.506725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:42:50.307175+0000","last_clean_scrub_stamp":"2024-03-30T14:05:56.506725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:06:58.173557+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":110,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1050752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2689,"ondisk_log_size":2689,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2332,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1537,"num_read_kb":1294,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":565,"num_bytes_recovered":12592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1059,"ondisk_log_size":1059,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1889,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103465,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":347152,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93398000,"statfs":{"total":95995035648,"available":95639552000,"internally_reserved":0,"allocated":860160,"data_stored":513803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":354613706},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397805,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":924984,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92820168,"statfs":{"total":95995035648,"available":95047852032,"internally_reserved":0,"allocated":1163264,"data_stored":840321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":946010566},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575174,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1028824,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":1027966,"kb_avail":92716328,"statfs":{"total":95995035648,"available":94941519872,"internally_reserved":0,"allocated":868352,"data_stored":514013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1052637637},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738699,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1037464,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":1036286,"kb_avail":92707688,"statfs":{"total":95995035648,"available":94932672512,"internally_reserved":0,"allocated":1196032,"data_stored":842197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1061157317},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":388,"seq":1666447310916,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":208064,"kb_used_data":1080,"kb_used_omap":168,"kb_used_meta":206807,"kb_avail":93537088,"statfs":{"total":95995035648,"available":95781978112,"internally_reserved":0,"allocated":1105920,"data_stored":776570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":172848,"internal_metadata":211770576},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182717,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1043288,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":1041854,"kb_avail":92701864,"statfs":{"total":95995035648,"available":94926708736,"internally_reserved":0,"allocated":1458176,"data_stored":1103777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1066858951},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953783,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898324,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":92846828,"statfs":{"total":95995035648,"available":95075151872,"internally_reserved":0,"allocated":864256,"data_stored":513490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":919009733},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692146,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":808344,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92936808,"statfs":{"total":95995035648,"available":95167291392,"internally_reserved":0,"allocated":1130496,"data_stored":777383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T14:07:00.805 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-30T14:07:00.806 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-30T14:07:01.055 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:07:01.066 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:01 smithi043 ceph-mon[42099]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:01.066 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:01 smithi043 ceph-mon[42099]: pgmap v1153: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:07:01.066 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:01 smithi043 ceph-mon[42099]: from='client.16272 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:07:01.367 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:01 smithi043 ceph-mon[29963]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:01.367 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:01 smithi043 ceph-mon[29963]: pgmap v1153: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:07:01.367 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:01 smithi043 ceph-mon[29963]: from='client.16272 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:07:01.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:01 smithi113 ceph-mon[34341]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:01.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:01 smithi113 ceph-mon[34341]: pgmap v1153: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:07:01.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:01 smithi113 ceph-mon[34341]: from='client.16272 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:07:02.390 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:02 smithi043 ceph-mon[29963]: 4.9 scrub starts 2024-03-30T14:07:02.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:02 smithi043 ceph-mon[29963]: 4.9 scrub ok 2024-03-30T14:07:02.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:02 smithi043 ceph-mon[42099]: 4.9 scrub starts 2024-03-30T14:07:02.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:02 smithi043 ceph-mon[42099]: 4.9 scrub ok 2024-03-30T14:07:02.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:02 smithi113 ceph-mon[34341]: 4.9 scrub starts 2024-03-30T14:07:02.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:02 smithi113 ceph-mon[34341]: 4.9 scrub ok 2024-03-30T14:07:03.390 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-mon[29963]: 4.1b scrub starts 2024-03-30T14:07:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-mon[29963]: 4.1b scrub ok 2024-03-30T14:07:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-mon[29963]: 1.b scrub starts 2024-03-30T14:07:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-mon[29963]: 1.b scrub ok 2024-03-30T14:07:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-mon[29963]: 4.3 scrub starts 2024-03-30T14:07:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-mon[29963]: 4.3 scrub ok 2024-03-30T14:07:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-mon[29963]: pgmap v1154: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:07:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-mon[29963]: 6.a scrub starts 2024-03-30T14:07:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-mon[29963]: 6.a scrub ok 2024-03-30T14:07:03.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-mon[42099]: 4.1b scrub starts 2024-03-30T14:07:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-mon[42099]: 4.1b scrub ok 2024-03-30T14:07:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-mon[42099]: 1.b scrub starts 2024-03-30T14:07:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-mon[42099]: 1.b scrub ok 2024-03-30T14:07:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-mon[42099]: 4.3 scrub starts 2024-03-30T14:07:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-mon[42099]: 4.3 scrub ok 2024-03-30T14:07:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-mon[42099]: pgmap v1154: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:07:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-mon[42099]: 6.a scrub starts 2024-03-30T14:07:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-mon[42099]: 6.a scrub ok 2024-03-30T14:07:03.392 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:14:07:03] ENGINE Bus STOPPING 2024-03-30T14:07:03.392 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:14:07:03] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-30T14:07:03.392 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:14:07:03] ENGINE Bus STOPPED 2024-03-30T14:07:03.393 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:14:07:03] ENGINE Bus STARTING 2024-03-30T14:07:03.393 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:14:07:03] ENGINE Serving on http://:::9283 2024-03-30T14:07:03.393 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:07:03 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:14:07:03] ENGINE Bus STARTED 2024-03-30T14:07:03.463 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:03 smithi113 ceph-mon[34341]: 4.1b scrub starts 2024-03-30T14:07:03.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:03 smithi113 ceph-mon[34341]: 4.1b scrub ok 2024-03-30T14:07:03.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:03 smithi113 ceph-mon[34341]: 1.b scrub starts 2024-03-30T14:07:03.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:03 smithi113 ceph-mon[34341]: 1.b scrub ok 2024-03-30T14:07:03.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:03 smithi113 ceph-mon[34341]: 4.3 scrub starts 2024-03-30T14:07:03.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:03 smithi113 ceph-mon[34341]: 4.3 scrub ok 2024-03-30T14:07:03.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:03 smithi113 ceph-mon[34341]: pgmap v1154: 152 pgs: 152 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:07:03.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:03 smithi113 ceph-mon[34341]: 6.a scrub starts 2024-03-30T14:07:03.464 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:03 smithi113 ceph-mon[34341]: 6.a scrub ok 2024-03-30T14:07:03.817 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-30T14:07:03.829 INFO:tasks.cephadm:Teardown begin 2024-03-30T14:07:03.829 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T14:07:03.858 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T14:07:03.893 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-30T14:07:03.893 DEBUG:teuthology.orchestra.run.smithi043:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-30T14:07:03.914 DEBUG:teuthology.orchestra.run.smithi113:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-30T14:07:03.959 INFO:tasks.cephadm:Stopping all daemons... 2024-03-30T14:07:03.959 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-30T14:07:03.959 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mon.a 2024-03-30T14:07:04.072 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:04 smithi043 systemd[1]: Stopping Ceph mon.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:07:04.213 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:04 smithi113 ceph-mon[34341]: 5.5 scrub starts 2024-03-30T14:07:04.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:04 smithi113 ceph-mon[34341]: 5.5 scrub ok 2024-03-30T14:07:04.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:04 smithi113 ceph-mon[34341]: 1.f scrub starts 2024-03-30T14:07:04.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:04 smithi113 ceph-mon[34341]: 1.f scrub ok 2024-03-30T14:07:04.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:04 smithi113 ceph-mon[34341]: from='client.? 172.21.15.43:0/298897594' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-30T14:07:04.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:04 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-30T14:07:04.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:04 smithi113 ceph-mon[34341]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-30T14:07:04.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:04 smithi113 ceph-mon[34341]: 4.d scrub starts 2024-03-30T14:07:04.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:04 smithi113 ceph-mon[34341]: 4.d scrub ok 2024-03-30T14:07:04.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:04 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:07:04.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:04 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:07:04.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:04 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:07:04.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:04 smithi113 ceph-mon[34341]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:07:04.391 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:14:07:04] ENGINE Bus STOPPING 2024-03-30T14:07:04.391 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:14:07:04] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-30T14:07:04.391 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:14:07:04] ENGINE Bus STOPPED 2024-03-30T14:07:04.391 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:14:07:04] ENGINE Bus STARTING 2024-03-30T14:07:04.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[42099]: 5.5 scrub starts 2024-03-30T14:07:04.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[42099]: 5.5 scrub ok 2024-03-30T14:07:04.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[42099]: 1.f scrub starts 2024-03-30T14:07:04.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[42099]: 1.f scrub ok 2024-03-30T14:07:04.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[42099]: from='client.? 172.21.15.43:0/298897594' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-30T14:07:04.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-30T14:07:04.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[42099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-30T14:07:04.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[42099]: 4.d scrub starts 2024-03-30T14:07:04.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[42099]: 4.d scrub ok 2024-03-30T14:07:04.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:07:04.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:07:04.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:07:04.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[42099]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:07:04.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[29963]: 5.5 scrub starts 2024-03-30T14:07:04.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[29963]: 5.5 scrub ok 2024-03-30T14:07:04.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[29963]: 1.f scrub starts 2024-03-30T14:07:04.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[29963]: 1.f scrub ok 2024-03-30T14:07:04.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[29963]: from='client.? 172.21.15.43:0/298897594' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-30T14:07:04.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-30T14:07:04.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[29963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-30T14:07:04.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[29963]: 4.d scrub starts 2024-03-30T14:07:04.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[29963]: 4.d scrub ok 2024-03-30T14:07:04.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:07:04.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:07:04.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:07:04.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-mon[29963]: from='mgr.14586 172.21.15.43:0/4269323937' entity='mgr.y' 2024-03-30T14:07:04.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a[29940]: 2024-03-30T14:07:04.287+0000 7ff60bdde700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:07:04.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a[29940]: 2024-03-30T14:07:04.287+0000 7ff60bdde700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-30T14:07:04.738 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:04 smithi043 podman[230401]: 2024-03-30 14:07:04.543192543 +0000 UTC m=+0.385464321 container died bb7d4ff2de3d44adaf589a1b09983cc543e1dca2bec1f803869a3d99a48346d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:07:04.738 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:14:07:04] ENGINE Serving on http://:::9283 2024-03-30T14:07:04.738 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:07:04 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: [30/Mar/2024:14:07:04] ENGINE Bus STARTED 2024-03-30T14:07:05.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:04 smithi043 podman[230401]: 2024-03-30 14:07:04.90100105 +0000 UTC m=+0.743272821 container cleanup bb7d4ff2de3d44adaf589a1b09983cc543e1dca2bec1f803869a3d99a48346d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-30T14:07:05.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:04 smithi043 bash[230401]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a 2024-03-30T14:07:05.364 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 30 14:07:05 smithi043 podman[230431]: 2024-03-30 14:07:05.210992722 +0000 UTC m=+0.662767146 container remove bb7d4ff2de3d44adaf589a1b09983cc543e1dca2bec1f803869a3d99a48346d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-a, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:07:05.376 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mon.a.service' 2024-03-30T14:07:05.426 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:07:05.427 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-30T14:07:05.427 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-30T14:07:05.427 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mon.c 2024-03-30T14:07:05.640 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:05 smithi043 systemd[1]: Stopping Ceph mon.c for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:07:06.024 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:05 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-c[42076]: 2024-03-30T14:07:05.775+0000 7f56bfb42700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:07:06.024 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:05 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-c[42076]: 2024-03-30T14:07:05.775+0000 7f56bfb42700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-30T14:07:06.025 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:05 smithi043 podman[230554]: 2024-03-30 14:07:05.938668978 +0000 UTC m=+0.293740789 container died d117d57e32aba55f3c0fd3b50867b8c5c53b703dbe19a6d1ab3ee41a91039bf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-c, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240326) 2024-03-30T14:07:06.317 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:07:06 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y[30217]: ::ffff:172.21.15.113 - - [30/Mar/2024:14:07:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:07:06.640 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:06 smithi043 podman[230554]: 2024-03-30 14:07:06.515301001 +0000 UTC m=+0.870372819 container cleanup d117d57e32aba55f3c0fd3b50867b8c5c53b703dbe19a6d1ab3ee41a91039bf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326) 2024-03-30T14:07:06.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Mar 30 14:07:06 smithi043 bash[230554]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-c 2024-03-30T14:07:06.935 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mon.c.service' 2024-03-30T14:07:06.973 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:07:06.973 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-30T14:07:06.974 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-30T14:07:06.974 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mon.b 2024-03-30T14:07:07.285 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:07 smithi113 systemd[1]: Stopping Ceph mon.b for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:07:07.540 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:07 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-b[34318]: 2024-03-30T14:07:07.291+0000 7f10fd156700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:07:07.540 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:07 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-b[34318]: 2024-03-30T14:07:07.291+0000 7f10fd156700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-30T14:07:07.800 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:07 smithi113 podman[170827]: 2024-03-30 14:07:07.547642274 +0000 UTC m=+0.356711268 container died 90cc320c3d3242a1d38ceb76f70f123f4ca7b904538080266f3d9b5f0dbb0190 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-b, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:07:08.213 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:07 smithi113 podman[170827]: 2024-03-30 14:07:07.997935086 +0000 UTC m=+0.807004125 container cleanup 90cc320c3d3242a1d38ceb76f70f123f4ca7b904538080266f3d9b5f0dbb0190 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:07:08.214 INFO:journalctl@ceph.mon.b.smithi113.stdout:Mar 30 14:07:07 smithi113 bash[170827]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mon-b 2024-03-30T14:07:08.421 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mon.b.service' 2024-03-30T14:07:08.475 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:07:08.475 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-30T14:07:08.475 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-30T14:07:08.475 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mgr.y 2024-03-30T14:07:08.808 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:07:08 smithi043 systemd[1]: Stopping Ceph mgr.y for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:07:09.112 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:07:08 smithi043 podman[230687]: 2024-03-30 14:07:08.820690077 +0000 UTC m=+0.140416118 container died b99b930d553f0e7a5f35293971ace5fdd4c225bfbaaf3440112fb5fadfe67832 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-30T14:07:09.390 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:07:09 smithi043 podman[230687]: 2024-03-30 14:07:09.31178002 +0000 UTC m=+0.631506103 container cleanup b99b930d553f0e7a5f35293971ace5fdd4c225bfbaaf3440112fb5fadfe67832 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-30T14:07:09.390 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Mar 30 14:07:09 smithi043 bash[230687]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-y 2024-03-30T14:07:09.723 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mgr.y.service' 2024-03-30T14:07:09.761 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:07:09.761 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-30T14:07:09.761 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-30T14:07:09.762 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mgr.x 2024-03-30T14:07:10.111 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 14:07:09 smithi113 systemd[1]: Stopping Ceph mgr.x for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:07:10.388 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 14:07:10 smithi113 podman[170958]: 2024-03-30 14:07:10.118166821 +0000 UTC m=+0.147063756 container died e8327be872dc9e591a9d6af20a44a2eec47aee1d5470a056b0fe78c3ce0332a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-30T14:07:10.713 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 14:07:10 smithi113 podman[170958]: 2024-03-30 14:07:10.469335783 +0000 UTC m=+0.498232726 container cleanup e8327be872dc9e591a9d6af20a44a2eec47aee1d5470a056b0fe78c3ce0332a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:07:10.714 INFO:journalctl@ceph.mgr.x.smithi113.stdout:Mar 30 14:07:10 smithi113 bash[170958]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-mgr-x 2024-03-30T14:07:10.945 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@mgr.x.service' 2024-03-30T14:07:10.983 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:07:10.984 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-30T14:07:10.984 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-30T14:07:10.984 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.0 2024-03-30T14:07:11.313 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 14:07:11 smithi043 systemd[1]: Stopping Ceph osd.0 for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:07:11.640 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 14:07:11 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-0[46512]: 2024-03-30T14:07:11.319+0000 7f993e6d0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:07:11.641 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 14:07:11 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-0[46512]: 2024-03-30T14:07:11.319+0000 7f993e6d0700 -1 osd.0 525 *** Got signal Terminated *** 2024-03-30T14:07:11.641 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 14:07:11 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-0[46512]: 2024-03-30T14:07:11.319+0000 7f993e6d0700 -1 osd.0 525 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:07:17.102 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 14:07:16 smithi043 podman[230821]: 2024-03-30 14:07:16.807189346 +0000 UTC m=+5.607702837 container died 4ccd7a154b9b7e81f7f1cd7343ca452f3ea0579b63d9b7ffeb814202f5961802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:07:17.390 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 14:07:17 smithi043 podman[230821]: 2024-03-30 14:07:17.308886309 +0000 UTC m=+6.109399805 container cleanup 4ccd7a154b9b7e81f7f1cd7343ca452f3ea0579b63d9b7ffeb814202f5961802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, ceph=True) 2024-03-30T14:07:17.391 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 14:07:17 smithi043 bash[230821]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-0 2024-03-30T14:07:17.861 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 14:07:17 smithi043 podman[230843]: 2024-03-30 14:07:17.577031255 +0000 UTC m=+0.760508826 container remove 4ccd7a154b9b7e81f7f1cd7343ca452f3ea0579b63d9b7ffeb814202f5961802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:07:18.119 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 14:07:17 smithi043 podman[230942]: 2024-03-30 14:07:17.869390685 +0000 UTC m=+0.112442801 container create 9765a845a19c25671ee6e353db7144923f1b532000e67e20d338e328304acc3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-0-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-30T14:07:18.119 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 14:07:17 smithi043 podman[230942]: 2024-03-30 14:07:17.774935808 +0000 UTC m=+0.017987924 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:07:18.390 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 14:07:18 smithi043 podman[230942]: 2024-03-30 14:07:18.126533014 +0000 UTC m=+0.369585130 container init 9765a845a19c25671ee6e353db7144923f1b532000e67e20d338e328304acc3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-4e50bbc, GIT_CLEAN=True) 2024-03-30T14:07:18.390 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 14:07:18 smithi043 podman[230942]: 2024-03-30 14:07:18.13272935 +0000 UTC m=+0.375781467 container start 9765a845a19c25671ee6e353db7144923f1b532000e67e20d338e328304acc3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-03-30T14:07:18.391 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 14:07:18 smithi043 podman[230942]: 2024-03-30 14:07:18.238776085 +0000 UTC m=+0.481828208 container attach 9765a845a19c25671ee6e353db7144923f1b532000e67e20d338e328304acc3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:07:18.834 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 30 14:07:18 smithi043 podman[230942]: 2024-03-30 14:07:18.54633774 +0000 UTC m=+0.789389859 container died 9765a845a19c25671ee6e353db7144923f1b532000e67e20d338e328304acc3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-0-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_CLEAN=True, RELEASE=squid-4e50bbc) 2024-03-30T14:07:19.161 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.0.service' 2024-03-30T14:07:19.203 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:07:19.203 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-30T14:07:19.203 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-30T14:07:19.204 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.1 2024-03-30T14:07:19.390 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 14:07:19 smithi043 systemd[1]: Stopping Ceph osd.1 for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:07:19.890 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 14:07:19 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-1[52902]: 2024-03-30T14:07:19.564+0000 7f0bea605700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:07:19.890 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 14:07:19 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-1[52902]: 2024-03-30T14:07:19.564+0000 7f0bea605700 -1 osd.1 525 *** Got signal Terminated *** 2024-03-30T14:07:19.891 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 14:07:19 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-1[52902]: 2024-03-30T14:07:19.564+0000 7f0bea605700 -1 osd.1 525 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:07:25.338 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 14:07:25 smithi043 podman[231075]: 2024-03-30 14:07:25.061308645 +0000 UTC m=+5.628126733 container died 78081c39ca5511678d3e60d819a6019fea6c0d16d3fd2c16251b76bd618ab22c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-1, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326) 2024-03-30T14:07:25.640 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 14:07:25 smithi043 podman[231075]: 2024-03-30 14:07:25.494943593 +0000 UTC m=+6.061761676 container cleanup 78081c39ca5511678d3e60d819a6019fea6c0d16d3fd2c16251b76bd618ab22c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-1, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc) 2024-03-30T14:07:25.640 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 14:07:25 smithi043 bash[231075]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-1 2024-03-30T14:07:26.109 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 14:07:25 smithi043 podman[231098]: 2024-03-30 14:07:25.713034923 +0000 UTC m=+0.652964731 container remove 78081c39ca5511678d3e60d819a6019fea6c0d16d3fd2c16251b76bd618ab22c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-1, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-30T14:07:26.214 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:07:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T14:07:25.859Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.43:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.43:8765: connect: connection refused" 2024-03-30T14:07:26.214 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:07:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T14:07:25.859Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.43:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.43:8765: connect: connection refused" 2024-03-30T14:07:26.214 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:07:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T14:07:25.859Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.43:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.43:8765: connect: connection refused" 2024-03-30T14:07:26.214 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:07:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T14:07:25.861Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.43:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.43:8765: connect: connection refused" 2024-03-30T14:07:26.390 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 14:07:26 smithi043 podman[231195]: 2024-03-30 14:07:26.019651278 +0000 UTC m=+0.017525441 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:07:26.390 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 14:07:26 smithi043 podman[231195]: 2024-03-30 14:07:26.147459886 +0000 UTC m=+0.145334072 container create fffdb3217e30e0bda96cd32a2b221cadce11aa91e045c088524cd98ed84ba35f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-1-deactivate, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-30T14:07:26.704 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 14:07:26 smithi043 podman[231195]: 2024-03-30 14:07:26.408429212 +0000 UTC m=+0.406303357 container init fffdb3217e30e0bda96cd32a2b221cadce11aa91e045c088524cd98ed84ba35f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5) 2024-03-30T14:07:26.704 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 14:07:26 smithi043 podman[231195]: 2024-03-30 14:07:26.414429827 +0000 UTC m=+0.412303972 container start fffdb3217e30e0bda96cd32a2b221cadce11aa91e045c088524cd98ed84ba35f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:07:26.704 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 14:07:26 smithi043 podman[231195]: 2024-03-30 14:07:26.499926647 +0000 UTC m=+0.497800807 container attach fffdb3217e30e0bda96cd32a2b221cadce11aa91e045c088524cd98ed84ba35f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-30T14:07:26.994 INFO:journalctl@ceph.osd.1.smithi043.stdout:Mar 30 14:07:26 smithi043 podman[231195]: 2024-03-30 14:07:26.71215688 +0000 UTC m=+0.710031032 container died fffdb3217e30e0bda96cd32a2b221cadce11aa91e045c088524cd98ed84ba35f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-1-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:07:27.391 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.1.service' 2024-03-30T14:07:27.439 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:07:27.439 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-30T14:07:27.439 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-30T14:07:27.439 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.2 2024-03-30T14:07:27.640 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:07:27 smithi043 systemd[1]: Stopping Ceph osd.2 for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:07:28.140 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:07:27 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2[197592]: 2024-03-30T14:07:27.742+0000 7fe6dfa97700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:07:28.140 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:07:27 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2[197592]: 2024-03-30T14:07:27.742+0000 7fe6dfa97700 -1 osd.2 525 *** Got signal Terminated *** 2024-03-30T14:07:28.141 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:07:27 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2[197592]: 2024-03-30T14:07:27.742+0000 7fe6dfa97700 -1 osd.2 525 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:07:33.516 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:07:33 smithi043 podman[231324]: 2024-03-30 14:07:33.214527958 +0000 UTC m=+5.576976675 container died 3f9ef916ccfdd3dd69f8f7b9f2233256f72308e9afe53adddf0b2a7911506877 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True) 2024-03-30T14:07:33.890 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:07:33 smithi043 podman[231324]: 2024-03-30 14:07:33.714424035 +0000 UTC m=+6.076872768 container cleanup 3f9ef916ccfdd3dd69f8f7b9f2233256f72308e9afe53adddf0b2a7911506877 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326) 2024-03-30T14:07:33.890 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:07:33 smithi043 bash[231324]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2 2024-03-30T14:07:34.267 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:07:33 smithi043 podman[231346]: 2024-03-30 14:07:33.957553055 +0000 UTC m=+0.741064600 container remove 3f9ef916ccfdd3dd69f8f7b9f2233256f72308e9afe53adddf0b2a7911506877 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc) 2024-03-30T14:07:34.586 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:07:34 smithi043 podman[231438]: 2024-03-30 14:07:34.275046995 +0000 UTC m=+0.114650806 container create 601e6ec0e8212837bf7e46fbbae5aa6dec7f338819c644854bb57f79050a86a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, ceph=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:07:34.586 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:07:34 smithi043 podman[231438]: 2024-03-30 14:07:34.178209248 +0000 UTC m=+0.017813035 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:07:34.586 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:07:34 smithi043 podman[231438]: 2024-03-30 14:07:34.511038202 +0000 UTC m=+0.350641990 container init 601e6ec0e8212837bf7e46fbbae5aa6dec7f338819c644854bb57f79050a86a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:07:34.586 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:07:34 smithi043 podman[231438]: 2024-03-30 14:07:34.51691188 +0000 UTC m=+0.356515655 container start 601e6ec0e8212837bf7e46fbbae5aa6dec7f338819c644854bb57f79050a86a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-deactivate, ceph=True, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-30T14:07:34.587 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:07:34 smithi043 podman[231438]: 2024-03-30 14:07:34.593970344 +0000 UTC m=+0.433574128 container attach 601e6ec0e8212837bf7e46fbbae5aa6dec7f338819c644854bb57f79050a86a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-03-30T14:07:34.890 INFO:journalctl@ceph.osd.2.smithi043.stdout:Mar 30 14:07:34 smithi043 podman[231438]: 2024-03-30 14:07:34.822850064 +0000 UTC m=+0.662453841 container died 601e6ec0e8212837bf7e46fbbae5aa6dec7f338819c644854bb57f79050a86a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-2-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:07:35.455 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.2.service' 2024-03-30T14:07:35.501 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:07:35.501 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-30T14:07:35.501 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-30T14:07:35.501 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.3 2024-03-30T14:07:35.730 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 30 14:07:35 smithi043 systemd[1]: Stopping Ceph osd.3 for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:07:36.140 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 30 14:07:35 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-3[65770]: 2024-03-30T14:07:35.869+0000 7f0ae2e19700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:07:36.141 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 30 14:07:35 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-3[65770]: 2024-03-30T14:07:35.869+0000 7f0ae2e19700 -1 osd.3 525 *** Got signal Terminated *** 2024-03-30T14:07:36.141 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 30 14:07:35 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-3[65770]: 2024-03-30T14:07:35.869+0000 7f0ae2e19700 -1 osd.3 525 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:07:37.463 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:37 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:37.044+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:38.213 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:38 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:38.059+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:38.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:37 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:37.918+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:39.213 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:39 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:39.063+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:39.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:38 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:38.879+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:40.213 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:40 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:40.112+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:40.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:39 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:39.913+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:40.214 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:39 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4[39900]: 2024-03-30T14:07:39.913+0000 7eff1845b700 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:13.830940+0000 front 2024-03-30T14:07:13.831037+0000 (oldest deadline 2024-03-30T14:07:39.731061+0000) 2024-03-30T14:07:41.142 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:40 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:40.945+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:07:41.143 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:40 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:40.959+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:41.143 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:40 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4[39900]: 2024-03-30T14:07:40.866+0000 7eff1845b700 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:13.830940+0000 front 2024-03-30T14:07:13.831037+0000 (oldest deadline 2024-03-30T14:07:39.731061+0000) 2024-03-30T14:07:41.463 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:41 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:41.149+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:41.844 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 30 14:07:41 smithi043 podman[231570]: 2024-03-30 14:07:41.532768231 +0000 UTC m=+5.783649459 container died 5632ad38e3f5e7ff6c93beefe90da7d8386d6fd628dbf354063f6146ee49a538 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-3, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, ceph=True) 2024-03-30T14:07:42.141 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 30 14:07:42 smithi043 podman[231570]: 2024-03-30 14:07:42.016968594 +0000 UTC m=+6.267849861 container cleanup 5632ad38e3f5e7ff6c93beefe90da7d8386d6fd628dbf354063f6146ee49a538 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-3, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:07:42.141 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 30 14:07:42 smithi043 bash[231570]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-3 2024-03-30T14:07:42.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:41 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:41.983+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:42.214 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:42 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:42.105+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:42.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:41 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:41.949+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:07:42.214 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:41 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4[39900]: 2024-03-30T14:07:41.896+0000 7eff1845b700 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:13.830940+0000 front 2024-03-30T14:07:13.831037+0000 (oldest deadline 2024-03-30T14:07:39.731061+0000) 2024-03-30T14:07:42.620 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 30 14:07:42 smithi043 podman[231604]: 2024-03-30 14:07:42.207225134 +0000 UTC m=+0.667152266 container remove 5632ad38e3f5e7ff6c93beefe90da7d8386d6fd628dbf354063f6146ee49a538 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-3, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-30T14:07:42.890 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 30 14:07:42 smithi043 podman[231702]: 2024-03-30 14:07:42.530986397 +0000 UTC m=+0.017422387 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:07:42.890 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 30 14:07:42 smithi043 podman[231702]: 2024-03-30 14:07:42.635985314 +0000 UTC m=+0.122421291 container create b1613c55618930b6acd9b80753dc474055540a2cfbaed5cd02ad88e7e682e842 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-3-deactivate, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-30T14:07:43.204 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 30 14:07:42 smithi043 podman[231702]: 2024-03-30 14:07:42.905326126 +0000 UTC m=+0.391762145 container init b1613c55618930b6acd9b80753dc474055540a2cfbaed5cd02ad88e7e682e842 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-3-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:07:43.204 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 30 14:07:42 smithi043 podman[231702]: 2024-03-30 14:07:42.911838265 +0000 UTC m=+0.398274243 container start b1613c55618930b6acd9b80753dc474055540a2cfbaed5cd02ad88e7e682e842 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:07:43.204 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 30 14:07:42 smithi043 podman[231702]: 2024-03-30 14:07:42.996293963 +0000 UTC m=+0.482729955 container attach b1613c55618930b6acd9b80753dc474055540a2cfbaed5cd02ad88e7e682e842 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True) 2024-03-30T14:07:43.213 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:43 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:43.113+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:43.213 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:42 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:42.915+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:07:43.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:43 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:43.003+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:43.214 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:42 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4[39900]: 2024-03-30T14:07:42.938+0000 7eff1845b700 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:13.830940+0000 front 2024-03-30T14:07:13.831037+0000 (oldest deadline 2024-03-30T14:07:39.731061+0000) 2024-03-30T14:07:43.500 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 30 14:07:43 smithi043 podman[231702]: 2024-03-30 14:07:43.211758019 +0000 UTC m=+0.698194001 container died b1613c55618930b6acd9b80753dc474055540a2cfbaed5cd02ad88e7e682e842 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-30T14:07:43.757 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 30 14:07:43 smithi043 podman[231748]: 2024-03-30 14:07:43.749358652 +0000 UTC m=+0.532463737 container remove b1613c55618930b6acd9b80753dc474055540a2cfbaed5cd02ad88e7e682e842 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-3-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-30T14:07:43.763 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.3.service' 2024-03-30T14:07:43.805 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:07:43.806 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-30T14:07:43.806 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-30T14:07:43.806 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.4 2024-03-30T14:07:44.102 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:43 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:43.877+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:07:44.102 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:43 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:43.989+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:44.103 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:43 smithi113 systemd[1]: Stopping Ceph osd.4 for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:07:44.103 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:43 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4[39900]: 2024-03-30T14:07:43.893+0000 7eff1845b700 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:13.830940+0000 front 2024-03-30T14:07:13.831037+0000 (oldest deadline 2024-03-30T14:07:39.731061+0000) 2024-03-30T14:07:44.463 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:44 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:44.108+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:44.463 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:44 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4[39900]: 2024-03-30T14:07:44.122+0000 7eff1be67700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:07:44.464 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:44 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4[39900]: 2024-03-30T14:07:44.122+0000 7eff1be67700 -1 osd.4 525 *** Got signal Terminated *** 2024-03-30T14:07:44.464 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:44 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4[39900]: 2024-03-30T14:07:44.122+0000 7eff1be67700 -1 osd.4 525 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:07:45.141 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:44 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:44.887+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:07:45.142 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:44 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4[39900]: 2024-03-30T14:07:44.924+0000 7eff1845b700 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:13.830940+0000 front 2024-03-30T14:07:13.831037+0000 (oldest deadline 2024-03-30T14:07:39.731061+0000) 2024-03-30T14:07:45.142 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:44 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:44.990+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:45.463 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:45 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:45.148+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:46.169 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:45 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:45.881+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:07:46.170 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:45 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:45.881+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:20.555343+0000 front 2024-03-30T14:07:20.555316+0000 (oldest deadline 2024-03-30T14:07:45.855219+0000) 2024-03-30T14:07:46.170 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:46 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:46.033+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:46.170 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:45 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4[39900]: 2024-03-30T14:07:45.936+0000 7eff1845b700 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:13.830940+0000 front 2024-03-30T14:07:13.831037+0000 (oldest deadline 2024-03-30T14:07:39.731061+0000) 2024-03-30T14:07:46.463 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:46 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:46.176+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:47.195 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:46 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4[39900]: 2024-03-30T14:07:46.942+0000 7eff1845b700 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:13.830940+0000 front 2024-03-30T14:07:13.831037+0000 (oldest deadline 2024-03-30T14:07:39.731061+0000) 2024-03-30T14:07:47.196 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:46 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:46.874+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:07:47.196 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:46 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:46.874+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:20.555343+0000 front 2024-03-30T14:07:20.555316+0000 (oldest deadline 2024-03-30T14:07:45.855219+0000) 2024-03-30T14:07:47.196 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:47 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:47.042+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:47.463 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:47 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:47.201+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:48.157 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:47 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:47.867+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:07:48.157 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:47 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:47.867+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:20.555343+0000 front 2024-03-30T14:07:20.555316+0000 (oldest deadline 2024-03-30T14:07:45.855219+0000) 2024-03-30T14:07:48.158 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:48.022+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:48.158 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:47 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4[39900]: 2024-03-30T14:07:47.963+0000 7eff1845b700 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:13.830940+0000 front 2024-03-30T14:07:13.831037+0000 (oldest deadline 2024-03-30T14:07:39.731061+0000) 2024-03-30T14:07:48.158 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:47 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4[39900]: 2024-03-30T14:07:47.963+0000 7eff1845b700 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:24.232984+0000 front 2024-03-30T14:07:24.233146+0000 (oldest deadline 2024-03-30T14:07:47.732946+0000) 2024-03-30T14:07:48.463 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:48.163+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:49.194 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:48.854+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:07:49.194 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:48.854+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:20.555343+0000 front 2024-03-30T14:07:20.555316+0000 (oldest deadline 2024-03-30T14:07:45.855219+0000) 2024-03-30T14:07:49.194 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4[39900]: 2024-03-30T14:07:48.967+0000 7eff1845b700 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:13.830940+0000 front 2024-03-30T14:07:13.831037+0000 (oldest deadline 2024-03-30T14:07:39.731061+0000) 2024-03-30T14:07:49.194 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4[39900]: 2024-03-30T14:07:48.967+0000 7eff1845b700 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:24.232984+0000 front 2024-03-30T14:07:24.233146+0000 (oldest deadline 2024-03-30T14:07:47.732946+0000) 2024-03-30T14:07:49.194 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:49 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:49.050+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:49.195 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:49 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:49.050+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:07:49.463 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:49 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:49.200+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:49.464 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:49 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:49.200+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:24.020192+0000 front 2024-03-30T14:07:24.020084+0000 (oldest deadline 2024-03-30T14:07:48.720151+0000) 2024-03-30T14:07:49.830 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:49 smithi113 podman[171103]: 2024-03-30 14:07:49.675628588 +0000 UTC m=+5.663208285 container died 96caabd20fcdb047200c26c002948e37f3965ad1510be63c8432288c9b58e0b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-30T14:07:50.095 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:49 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:49.836+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:07:50.095 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:49 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:49.836+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:20.555343+0000 front 2024-03-30T14:07:20.555316+0000 (oldest deadline 2024-03-30T14:07:45.855219+0000) 2024-03-30T14:07:50.095 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:50 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:50.019+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:50.095 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:50 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:50.019+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:07:50.454 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:50 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:50.210+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:50.454 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:50 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:50.210+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:24.020192+0000 front 2024-03-30T14:07:24.020084+0000 (oldest deadline 2024-03-30T14:07:48.720151+0000) 2024-03-30T14:07:50.455 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:50 smithi113 podman[171103]: 2024-03-30 14:07:50.103102239 +0000 UTC m=+6.090681942 container cleanup 96caabd20fcdb047200c26c002948e37f3965ad1510be63c8432288c9b58e0b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-30T14:07:50.455 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:50 smithi113 bash[171103]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4 2024-03-30T14:07:50.455 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:50 smithi113 podman[171134]: 2024-03-30 14:07:50.338170109 +0000 UTC m=+0.665417394 container remove 96caabd20fcdb047200c26c002948e37f3965ad1510be63c8432288c9b58e0b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2) 2024-03-30T14:07:51.059 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:50 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:50.854+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:07:51.059 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:50 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:50.854+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:20.555343+0000 front 2024-03-30T14:07:20.555316+0000 (oldest deadline 2024-03-30T14:07:45.855219+0000) 2024-03-30T14:07:51.059 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:50 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:50.974+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:51.059 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:50 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:50.974+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:07:51.060 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:50 smithi113 podman[171237]: 2024-03-30 14:07:50.687087214 +0000 UTC m=+0.018491867 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:07:51.060 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:50 smithi113 podman[171237]: 2024-03-30 14:07:50.822727406 +0000 UTC m=+0.154132098 container create 7e4c31b452f721a36e1d13a1f4a5f38eb0fad28e8ee4a0c18948f05e08beea01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:07:51.463 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:51 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:51.220+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:51.463 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:51 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:51.220+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:24.020192+0000 front 2024-03-30T14:07:24.020084+0000 (oldest deadline 2024-03-30T14:07:48.720151+0000) 2024-03-30T14:07:51.463 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:51 smithi113 podman[171237]: 2024-03-30 14:07:51.067029103 +0000 UTC m=+0.398433749 container init 7e4c31b452f721a36e1d13a1f4a5f38eb0fad28e8ee4a0c18948f05e08beea01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-03-30T14:07:51.464 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:51 smithi113 podman[171237]: 2024-03-30 14:07:51.072817952 +0000 UTC m=+0.404222598 container start 7e4c31b452f721a36e1d13a1f4a5f38eb0fad28e8ee4a0c18948f05e08beea01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-30T14:07:51.464 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:51 smithi113 podman[171237]: 2024-03-30 14:07:51.175578069 +0000 UTC m=+0.506982736 container attach 7e4c31b452f721a36e1d13a1f4a5f38eb0fad28e8ee4a0c18948f05e08beea01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4-deactivate, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=) 2024-03-30T14:07:51.871 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:51 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:51.827+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:07:51.872 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:51 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:51.827+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:20.555343+0000 front 2024-03-30T14:07:20.555316+0000 (oldest deadline 2024-03-30T14:07:45.855219+0000) 2024-03-30T14:07:51.872 INFO:journalctl@ceph.osd.4.smithi113.stdout:Mar 30 14:07:51 smithi113 podman[171237]: 2024-03-30 14:07:51.582541157 +0000 UTC m=+0.913945807 container died 7e4c31b452f721a36e1d13a1f4a5f38eb0fad28e8ee4a0c18948f05e08beea01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-4e50bbc) 2024-03-30T14:07:52.163 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:51 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:51.930+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:52.163 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:51 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:51.930+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:07:52.180 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.4.service' 2024-03-30T14:07:52.222 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:07:52.223 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-30T14:07:52.223 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-30T14:07:52.223 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.5 2024-03-30T14:07:52.463 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:52 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:52.270+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:52.463 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:52 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:52.270+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:24.020192+0000 front 2024-03-30T14:07:24.020084+0000 (oldest deadline 2024-03-30T14:07:48.720151+0000) 2024-03-30T14:07:52.463 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:52 smithi113 systemd[1]: Stopping Ceph osd.5 for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:07:52.857 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:52 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:52.576+0000 7f6270513700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:07:52.858 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:52 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:52.576+0000 7f6270513700 -1 osd.5 525 *** Got signal Terminated *** 2024-03-30T14:07:52.858 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:52 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:52.576+0000 7f6270513700 -1 osd.5 525 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:07:53.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:52 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:52.961+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:53.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:52 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:52.961+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:07:53.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:52 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:52.864+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:07:53.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:52 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:52.864+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:20.555343+0000 front 2024-03-30T14:07:20.555316+0000 (oldest deadline 2024-03-30T14:07:45.855219+0000) 2024-03-30T14:07:53.713 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:53 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:53.314+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:53.713 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:53 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:53.314+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:24.020192+0000 front 2024-03-30T14:07:24.020084+0000 (oldest deadline 2024-03-30T14:07:48.720151+0000) 2024-03-30T14:07:54.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:54 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:54.011+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:54.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:54 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:54.011+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:07:54.213 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:53 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:53.914+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:07:54.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:53 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:53.914+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:20.555343+0000 front 2024-03-30T14:07:20.555316+0000 (oldest deadline 2024-03-30T14:07:45.855219+0000) 2024-03-30T14:07:54.713 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:54 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:54.313+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:54.713 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:54 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:54.313+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:24.020192+0000 front 2024-03-30T14:07:24.020084+0000 (oldest deadline 2024-03-30T14:07:48.720151+0000) 2024-03-30T14:07:55.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:55 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:55.005+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:55.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:55 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:55.005+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:07:55.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:54 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:54.888+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:07:55.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:54 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:54.888+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:20.555343+0000 front 2024-03-30T14:07:20.555316+0000 (oldest deadline 2024-03-30T14:07:45.855219+0000) 2024-03-30T14:07:55.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:54 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:54.888+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:29.956310+0000 front 2024-03-30T14:07:29.956355+0000 (oldest deadline 2024-03-30T14:07:54.656218+0000) 2024-03-30T14:07:55.713 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:55 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:55.287+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:55.713 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:55 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:55.287+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:24.020192+0000 front 2024-03-30T14:07:24.020084+0000 (oldest deadline 2024-03-30T14:07:48.720151+0000) 2024-03-30T14:07:56.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:55 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:55.972+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:56.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:55 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:55.972+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:07:56.213 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:55 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:55.888+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:07:56.213 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:55 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:55.888+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:20.555343+0000 front 2024-03-30T14:07:20.555316+0000 (oldest deadline 2024-03-30T14:07:45.855219+0000) 2024-03-30T14:07:56.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:55 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:55.888+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:29.956310+0000 front 2024-03-30T14:07:29.956355+0000 (oldest deadline 2024-03-30T14:07:54.656218+0000) 2024-03-30T14:07:56.713 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:56 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:56.258+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:56.713 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:56 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:56.258+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:24.020192+0000 front 2024-03-30T14:07:24.020084+0000 (oldest deadline 2024-03-30T14:07:48.720151+0000) 2024-03-30T14:07:57.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:57 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:57.014+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:57.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:57 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:57.014+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:07:57.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:57 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:57.014+0000 7ffb77d26700 -1 osd.7 525 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14535.0:1462 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e525) 2024-03-30T14:07:57.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:56 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:56.897+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:07:57.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:56 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:56.897+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:20.555343+0000 front 2024-03-30T14:07:20.555316+0000 (oldest deadline 2024-03-30T14:07:45.855219+0000) 2024-03-30T14:07:57.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:56 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:56.897+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:29.956310+0000 front 2024-03-30T14:07:29.956355+0000 (oldest deadline 2024-03-30T14:07:54.656218+0000) 2024-03-30T14:07:57.713 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:57 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:57.266+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:14.118906+0000 front 2024-03-30T14:07:14.118994+0000 (oldest deadline 2024-03-30T14:07:37.018871+0000) 2024-03-30T14:07:57.713 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:57 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:57.266+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:24.020192+0000 front 2024-03-30T14:07:24.020084+0000 (oldest deadline 2024-03-30T14:07:48.720151+0000) 2024-03-30T14:07:57.713 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:57 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5[46248]: 2024-03-30T14:07:57.266+0000 7f626cb07700 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:30.421414+0000 front 2024-03-30T14:07:30.421353+0000 (oldest deadline 2024-03-30T14:07:56.321163+0000) 2024-03-30T14:07:58.197 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:57 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:57.895+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:07:58.198 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:57 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:57.895+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:20.555343+0000 front 2024-03-30T14:07:20.555316+0000 (oldest deadline 2024-03-30T14:07:45.855219+0000) 2024-03-30T14:07:58.198 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:57 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:57.895+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:29.956310+0000 front 2024-03-30T14:07:29.956355+0000 (oldest deadline 2024-03-30T14:07:54.656218+0000) 2024-03-30T14:07:58.198 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:58.010+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:58.198 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:58.010+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:07:58.198 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:58.010+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:31.776936+0000 front 2024-03-30T14:07:31.776841+0000 (oldest deadline 2024-03-30T14:07:57.076742+0000) 2024-03-30T14:07:58.198 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:58.010+0000 7ffb77d26700 -1 osd.7 525 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14535.0:1462 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e525) 2024-03-30T14:07:58.451 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:58 smithi113 podman[171368]: 2024-03-30 14:07:58.205704378 +0000 UTC m=+5.753123592 container died 71841c4ee45ff0c4ce7891aa82d66f55f1a814ede72a417195c5346352b202ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:07:58.713 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:58 smithi113 podman[171368]: 2024-03-30 14:07:58.54087878 +0000 UTC m=+6.088298035 container cleanup 71841c4ee45ff0c4ce7891aa82d66f55f1a814ede72a417195c5346352b202ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:07:58.713 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:58 smithi113 bash[171368]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5 2024-03-30T14:07:58.997 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:58 smithi113 podman[171392]: 2024-03-30 14:07:58.725343539 +0000 UTC m=+0.523830836 container remove 71841c4ee45ff0c4ce7891aa82d66f55f1a814ede72a417195c5346352b202ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-30T14:07:58.997 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:58.866+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:07:58.997 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:58.866+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:20.555343+0000 front 2024-03-30T14:07:20.555316+0000 (oldest deadline 2024-03-30T14:07:45.855219+0000) 2024-03-30T14:07:58.997 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:58 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:58.866+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:29.956310+0000 front 2024-03-30T14:07:29.956355+0000 (oldest deadline 2024-03-30T14:07:54.656218+0000) 2024-03-30T14:07:59.288 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:59 smithi113 podman[171491]: 2024-03-30 14:07:59.046781578 +0000 UTC m=+0.018152785 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:07:59.289 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:59 smithi113 podman[171491]: 2024-03-30 14:07:59.184560632 +0000 UTC m=+0.155931826 container create 11169d48a18efb88d588647c4819229864e1694d09a85ea6cb132f52c8a23c25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:07:59.289 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:59.003+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:07:59.289 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:59.003+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:07:59.289 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:59.003+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:31.776936+0000 front 2024-03-30T14:07:31.776841+0000 (oldest deadline 2024-03-30T14:07:57.076742+0000) 2024-03-30T14:07:59.289 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:59.004+0000 7ffb77d26700 -1 osd.7 525 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14535.0:1462 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e525) 2024-03-30T14:07:59.706 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:59 smithi113 podman[171491]: 2024-03-30 14:07:59.420556312 +0000 UTC m=+0.391927518 container init 11169d48a18efb88d588647c4819229864e1694d09a85ea6cb132f52c8a23c25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5-deactivate, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-30T14:07:59.706 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:59 smithi113 podman[171491]: 2024-03-30 14:07:59.426038 +0000 UTC m=+0.397409195 container start 11169d48a18efb88d588647c4819229864e1694d09a85ea6cb132f52c8a23c25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-30T14:07:59.706 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:59 smithi113 podman[171491]: 2024-03-30 14:07:59.545563779 +0000 UTC m=+0.516934987 container attach 11169d48a18efb88d588647c4819229864e1694d09a85ea6cb132f52c8a23c25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-30T14:07:59.963 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 30 14:07:59 smithi113 podman[171491]: 2024-03-30 14:07:59.714403798 +0000 UTC m=+0.685774997 container died 11169d48a18efb88d588647c4819229864e1694d09a85ea6cb132f52c8a23c25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-30T14:07:59.963 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:59.845+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:07:59.963 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:59.845+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:20.555343+0000 front 2024-03-30T14:07:20.555316+0000 (oldest deadline 2024-03-30T14:07:45.855219+0000) 2024-03-30T14:07:59.963 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:07:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:07:59.845+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:29.956310+0000 front 2024-03-30T14:07:29.956355+0000 (oldest deadline 2024-03-30T14:07:54.656218+0000) 2024-03-30T14:08:00.341 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:59.994+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:08:00.342 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:59.994+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:08:00.342 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:59.994+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:31.776936+0000 front 2024-03-30T14:07:31.776841+0000 (oldest deadline 2024-03-30T14:07:57.076742+0000) 2024-03-30T14:08:00.342 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:07:59 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:07:59.994+0000 7ffb77d26700 -1 osd.7 525 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14535.0:1462 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e525) 2024-03-30T14:08:00.363 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.5.service' 2024-03-30T14:08:00.405 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:08:00.406 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-30T14:08:00.406 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-30T14:08:00.406 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.6 2024-03-30T14:08:00.714 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:00 smithi113 systemd[1]: Stopping Ceph osd.6 for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:08:01.213 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:00 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:08:00.780+0000 7f5888ff2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:08:01.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:00 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:08:00.780+0000 7f5888ff2700 -1 osd.6 525 *** Got signal Terminated *** 2024-03-30T14:08:01.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:00 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:08:00.780+0000 7f5888ff2700 -1 osd.6 525 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:08:01.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:00 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:08:00.871+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:08:01.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:00 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:08:00.871+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:20.555343+0000 front 2024-03-30T14:07:20.555316+0000 (oldest deadline 2024-03-30T14:07:45.855219+0000) 2024-03-30T14:08:01.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:00 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:08:00.871+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:29.956310+0000 front 2024-03-30T14:07:29.956355+0000 (oldest deadline 2024-03-30T14:07:54.656218+0000) 2024-03-30T14:08:01.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:00 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:00.969+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:08:01.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:00 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:00.969+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:08:01.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:00 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:00.969+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:31.776936+0000 front 2024-03-30T14:07:31.776841+0000 (oldest deadline 2024-03-30T14:07:57.076742+0000) 2024-03-30T14:08:02.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:01 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:01.981+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:08:02.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:01 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:01.981+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:08:02.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:01 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:01.981+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:31.776936+0000 front 2024-03-30T14:07:31.776841+0000 (oldest deadline 2024-03-30T14:07:57.076742+0000) 2024-03-30T14:08:02.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:01 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:08:01.898+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:08:02.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:01 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:08:01.898+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:20.555343+0000 front 2024-03-30T14:07:20.555316+0000 (oldest deadline 2024-03-30T14:07:45.855219+0000) 2024-03-30T14:08:02.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:01 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:08:01.898+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:29.956310+0000 front 2024-03-30T14:07:29.956355+0000 (oldest deadline 2024-03-30T14:07:54.656218+0000) 2024-03-30T14:08:03.213 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:02 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:08:02.935+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:08:03.213 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:02 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:08:02.935+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:20.555343+0000 front 2024-03-30T14:07:20.555316+0000 (oldest deadline 2024-03-30T14:07:45.855219+0000) 2024-03-30T14:08:03.213 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:02 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:08:02.935+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:29.956310+0000 front 2024-03-30T14:07:29.956355+0000 (oldest deadline 2024-03-30T14:07:54.656218+0000) 2024-03-30T14:08:03.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:02 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:02.975+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:08:03.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:02 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:02.975+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:08:03.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:02 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:02.975+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:31.776936+0000 front 2024-03-30T14:07:31.776841+0000 (oldest deadline 2024-03-30T14:07:57.076742+0000) 2024-03-30T14:08:04.213 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:03 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:08:03.907+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:08:04.213 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:03 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:08:03.907+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:20.555343+0000 front 2024-03-30T14:07:20.555316+0000 (oldest deadline 2024-03-30T14:07:45.855219+0000) 2024-03-30T14:08:04.213 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:03 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:08:03.907+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:29.956310+0000 front 2024-03-30T14:07:29.956355+0000 (oldest deadline 2024-03-30T14:07:54.656218+0000) 2024-03-30T14:08:04.213 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:03 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:08:03.907+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-03-30T14:07:38.757101+0000 front 2024-03-30T14:07:38.757112+0000 (oldest deadline 2024-03-30T14:08:03.457021+0000) 2024-03-30T14:08:04.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:03 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:03.994+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:08:04.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:03 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:03.994+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:08:04.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:03 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:03.994+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:31.776936+0000 front 2024-03-30T14:07:31.776841+0000 (oldest deadline 2024-03-30T14:07:57.076742+0000) 2024-03-30T14:08:05.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:04 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:04.959+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:08:05.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:04 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:04.959+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:08:05.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:04 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:04.959+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:31.776936+0000 front 2024-03-30T14:07:31.776841+0000 (oldest deadline 2024-03-30T14:07:57.076742+0000) 2024-03-30T14:08:05.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:04 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:08:04.951+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.254752+0000 front 2024-03-30T14:07:15.254871+0000 (oldest deadline 2024-03-30T14:07:40.554809+0000) 2024-03-30T14:08:05.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:04 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:08:04.951+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:20.555343+0000 front 2024-03-30T14:07:20.555316+0000 (oldest deadline 2024-03-30T14:07:45.855219+0000) 2024-03-30T14:08:05.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:04 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:08:04.951+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:29.956310+0000 front 2024-03-30T14:07:29.956355+0000 (oldest deadline 2024-03-30T14:07:54.656218+0000) 2024-03-30T14:08:05.214 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:04 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6[52627]: 2024-03-30T14:08:04.951+0000 7f58815de700 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-03-30T14:07:38.757101+0000 front 2024-03-30T14:07:38.757112+0000 (oldest deadline 2024-03-30T14:08:03.457021+0000) 2024-03-30T14:08:06.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:05 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:05.970+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:08:06.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:05 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:05.970+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:08:06.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:05 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:05.970+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:31.776936+0000 front 2024-03-30T14:07:31.776841+0000 (oldest deadline 2024-03-30T14:07:57.076742+0000) 2024-03-30T14:08:06.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:05 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:05.970+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-03-30T14:07:37.577923+0000 front 2024-03-30T14:07:37.577870+0000 (oldest deadline 2024-03-30T14:08:05.178201+0000) 2024-03-30T14:08:06.554 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:06 smithi113 podman[171622]: 2024-03-30 14:08:06.287571474 +0000 UTC m=+5.641636060 container died 01a1da69832ca72465e6a8fc7a7373b2ea2e60034baade590ea8df22da64aac5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS) 2024-03-30T14:08:06.937 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:06 smithi113 podman[171622]: 2024-03-30 14:08:06.652121181 +0000 UTC m=+6.006185758 container cleanup 01a1da69832ca72465e6a8fc7a7373b2ea2e60034baade590ea8df22da64aac5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-30T14:08:06.937 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:06 smithi113 bash[171622]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6 2024-03-30T14:08:07.213 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:06 smithi113 podman[171644]: 2024-03-30 14:08:06.944874734 +0000 UTC m=+0.652729537 container remove 01a1da69832ca72465e6a8fc7a7373b2ea2e60034baade590ea8df22da64aac5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc) 2024-03-30T14:08:07.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:06 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:06.952+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:08:07.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:06 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:06.952+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:08:07.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:06 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:06.952+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:31.776936+0000 front 2024-03-30T14:07:31.776841+0000 (oldest deadline 2024-03-30T14:07:57.076742+0000) 2024-03-30T14:08:07.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:06 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:06.952+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-03-30T14:07:37.577923+0000 front 2024-03-30T14:07:37.577870+0000 (oldest deadline 2024-03-30T14:08:05.178201+0000) 2024-03-30T14:08:07.599 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:07 smithi113 podman[171742]: 2024-03-30 14:08:07.174681625 +0000 UTC m=+0.018285028 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:08:07.599 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:07 smithi113 podman[171742]: 2024-03-30 14:08:07.297815031 +0000 UTC m=+0.141418481 container create 51d49f9f326dcbf562ac4fd3e4cec215f99f3a10013e7e0502ff79c4e453944b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:08:07.599 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:07 smithi113 podman[171742]: 2024-03-30 14:08:07.515840656 +0000 UTC m=+0.359444061 container init 51d49f9f326dcbf562ac4fd3e4cec215f99f3a10013e7e0502ff79c4e453944b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6-deactivate, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:08:07.599 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:07 smithi113 podman[171742]: 2024-03-30 14:08:07.521347384 +0000 UTC m=+0.364950793 container start 51d49f9f326dcbf562ac4fd3e4cec215f99f3a10013e7e0502ff79c4e453944b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-30T14:08:07.599 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:07 smithi113 podman[171742]: 2024-03-30 14:08:07.606694547 +0000 UTC m=+0.450297957 container attach 51d49f9f326dcbf562ac4fd3e4cec215f99f3a10013e7e0502ff79c4e453944b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, ceph=True, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-30T14:08:07.932 INFO:journalctl@ceph.osd.6.smithi113.stdout:Mar 30 14:08:07 smithi113 podman[171742]: 2024-03-30 14:08:07.805842443 +0000 UTC m=+0.649445851 container died 51d49f9f326dcbf562ac4fd3e4cec215f99f3a10013e7e0502ff79c4e453944b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-6-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-03-30T14:08:08.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:07 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:07.940+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:08:08.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:07 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:07.940+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:08:08.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:07 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:07.940+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:31.776936+0000 front 2024-03-30T14:07:31.776841+0000 (oldest deadline 2024-03-30T14:07:57.076742+0000) 2024-03-30T14:08:08.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:07 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:07.940+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-03-30T14:07:37.577923+0000 front 2024-03-30T14:07:37.577870+0000 (oldest deadline 2024-03-30T14:08:05.178201+0000) 2024-03-30T14:08:08.384 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.6.service' 2024-03-30T14:08:08.427 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:08:08.427 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-30T14:08:08.427 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-30T14:08:08.428 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.7 2024-03-30T14:08:08.713 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:08 smithi113 systemd[1]: Stopping Ceph osd.7 for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:08:09.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:08 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:08.773+0000 7ffb7f73a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:08:09.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:08 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:08.773+0000 7ffb7f73a700 -1 osd.7 525 *** Got signal Terminated *** 2024-03-30T14:08:09.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:08 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:08.773+0000 7ffb7f73a700 -1 osd.7 525 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:08:09.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:08 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:08.944+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:08:09.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:08 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:08.944+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:08:09.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:08 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:08.944+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:31.776936+0000 front 2024-03-30T14:07:31.776841+0000 (oldest deadline 2024-03-30T14:07:57.076742+0000) 2024-03-30T14:08:09.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:08 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:08.944+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-03-30T14:07:37.577923+0000 front 2024-03-30T14:07:37.577870+0000 (oldest deadline 2024-03-30T14:08:05.178201+0000) 2024-03-30T14:08:10.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:09 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:09.943+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:08:10.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:09 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:09.943+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:08:10.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:09 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:09.943+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:31.776936+0000 front 2024-03-30T14:07:31.776841+0000 (oldest deadline 2024-03-30T14:07:57.076742+0000) 2024-03-30T14:08:10.214 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:09 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:09.943+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-03-30T14:07:37.577923+0000 front 2024-03-30T14:07:37.577870+0000 (oldest deadline 2024-03-30T14:08:05.178201+0000) 2024-03-30T14:08:11.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:10 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:10.950+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:08:11.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:10 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:10.950+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:08:11.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:10 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:10.950+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:31.776936+0000 front 2024-03-30T14:07:31.776841+0000 (oldest deadline 2024-03-30T14:07:57.076742+0000) 2024-03-30T14:08:11.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:10 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:10.950+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-03-30T14:07:37.577923+0000 front 2024-03-30T14:07:37.577870+0000 (oldest deadline 2024-03-30T14:08:05.178201+0000) 2024-03-30T14:08:12.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:11 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:11.923+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:08:12.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:11 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:11.923+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:08:12.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:11 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:11.923+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:31.776936+0000 front 2024-03-30T14:07:31.776841+0000 (oldest deadline 2024-03-30T14:07:57.076742+0000) 2024-03-30T14:08:12.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:11 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:11.923+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-03-30T14:07:37.577923+0000 front 2024-03-30T14:07:37.577870+0000 (oldest deadline 2024-03-30T14:08:05.178201+0000) 2024-03-30T14:08:13.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:12 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:12.939+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-30T14:07:15.474346+0000 front 2024-03-30T14:07:15.474332+0000 (oldest deadline 2024-03-30T14:07:37.074819+0000) 2024-03-30T14:08:13.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:12 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:12.939+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-03-30T14:07:22.975843+0000 front 2024-03-30T14:07:22.975897+0000 (oldest deadline 2024-03-30T14:07:48.275748+0000) 2024-03-30T14:08:13.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:12 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:12.939+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-03-30T14:07:31.776936+0000 front 2024-03-30T14:07:31.776841+0000 (oldest deadline 2024-03-30T14:07:57.076742+0000) 2024-03-30T14:08:13.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:12 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:12.939+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-03-30T14:07:37.577923+0000 front 2024-03-30T14:07:37.577870+0000 (oldest deadline 2024-03-30T14:08:05.178201+0000) 2024-03-30T14:08:13.213 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:12 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7[59142]: 2024-03-30T14:08:12.939+0000 7ffb77d26700 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.113:6804 osd.4 since back 2024-03-30T14:07:45.178673+0000 front 2024-03-30T14:07:45.178693+0000 (oldest deadline 2024-03-30T14:08:12.179022+0000) 2024-03-30T14:08:14.589 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:14 smithi113 podman[171873]: 2024-03-30 14:08:14.34391711 +0000 UTC m=+5.681931928 container died 4a0def2c0f06ca5a9c99f96810b89290f060669e9559c39231f4a56a4e2d8634 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:08:14.963 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:14 smithi113 podman[171873]: 2024-03-30 14:08:14.695842373 +0000 UTC m=+6.033857181 container cleanup 4a0def2c0f06ca5a9c99f96810b89290f060669e9559c39231f4a56a4e2d8634 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-30T14:08:14.963 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:14 smithi113 bash[171873]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7 2024-03-30T14:08:15.298 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:14 smithi113 podman[171895]: 2024-03-30 14:08:14.996733473 +0000 UTC m=+0.650991321 container remove 4a0def2c0f06ca5a9c99f96810b89290f060669e9559c39231f4a56a4e2d8634 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-30T14:08:15.550 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:15 smithi113 podman[171991]: 2024-03-30 14:08:15.306397144 +0000 UTC m=+0.114348610 container create d841aea5ade9c81f1182a564f2f0bc0b576dfb7f053c596238bfaa222626c68c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7-deactivate, ceph=True, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-30T14:08:15.551 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:15 smithi113 podman[171991]: 2024-03-30 14:08:15.209978704 +0000 UTC m=+0.017930151 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:08:15.551 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:15 smithi113 podman[171991]: 2024-03-30 14:08:15.558767391 +0000 UTC m=+0.366718817 container init d841aea5ade9c81f1182a564f2f0bc0b576dfb7f053c596238bfaa222626c68c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:08:15.842 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:15 smithi113 podman[171991]: 2024-03-30 14:08:15.563998254 +0000 UTC m=+0.371949677 container start d841aea5ade9c81f1182a564f2f0bc0b576dfb7f053c596238bfaa222626c68c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7-deactivate, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-30T14:08:15.842 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:15 smithi113 podman[171991]: 2024-03-30 14:08:15.65019971 +0000 UTC m=+0.458151146 container attach d841aea5ade9c81f1182a564f2f0bc0b576dfb7f053c596238bfaa222626c68c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-30T14:08:16.145 INFO:journalctl@ceph.osd.7.smithi113.stdout:Mar 30 14:08:15 smithi113 podman[171991]: 2024-03-30 14:08:15.850287299 +0000 UTC m=+0.658238745 container died d841aea5ade9c81f1182a564f2f0bc0b576dfb7f053c596238bfaa222626c68c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-30T14:08:16.485 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@osd.7.service' 2024-03-30T14:08:16.526 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:08:16.526 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-30T14:08:16.526 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-30T14:08:16.526 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@rgw.foo.a 2024-03-30T14:08:16.826 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Mar 30 14:08:16 smithi043 systemd[1]: Stopping Ceph rgw.foo.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:08:17.091 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Mar 30 14:08:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-rgw-foo-a[71193]: 2024-03-30T14:08:16.833+0000 7f45a5eac700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:08:17.092 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Mar 30 14:08:16 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-rgw-foo-a[71193]: 2024-03-30T14:08:16.833+0000 7f45b472bac0 -1 shutting down 2024-03-30T14:08:26.213 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:08:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T14:08:25.859Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.43:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.43:8765: connect: connection refused" 2024-03-30T14:08:26.213 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:08:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T14:08:25.859Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.43:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.43:8765: connect: connection refused" 2024-03-30T14:08:26.214 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:08:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T14:08:25.859Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.43:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.43:8765: connect: connection refused" 2024-03-30T14:08:26.214 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:08:25 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T14:08:25.861Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.43:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.43:8765: connect: connection refused" 2024-03-30T14:08:27.140 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Mar 30 14:08:26 smithi043 bash[231835]: time="2024-03-30T14:08:26Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-30T14:08:27.141 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Mar 30 14:08:26 smithi043 podman[231835]: 2024-03-30 14:08:26.857049576 +0000 UTC m=+10.120592046 container died b9e161922958d2fa1dcc15021343f10f8044bcb349ba4cf4bd8f379685f27afa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:08:27.529 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Mar 30 14:08:27 smithi043 podman[231835]: 2024-03-30 14:08:27.313465653 +0000 UTC m=+10.577008104 container cleanup b9e161922958d2fa1dcc15021343f10f8044bcb349ba4cf4bd8f379685f27afa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-rgw-foo-a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux ) 2024-03-30T14:08:27.529 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Mar 30 14:08:27 smithi043 bash[231835]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-rgw-foo-a 2024-03-30T14:08:27.780 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Mar 30 14:08:27 smithi043 podman[231872]: 2024-03-30 14:08:27.631345989 +0000 UTC m=+0.765814036 container remove b9e161922958d2fa1dcc15021343f10f8044bcb349ba4cf4bd8f379685f27afa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-rgw-foo-a, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True) 2024-03-30T14:08:27.780 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Mar 30 14:08:27 smithi043 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-30T14:08:27.780 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Mar 30 14:08:27 smithi043 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@rgw.foo.a.service: Failed with result 'exit-code'. 2024-03-30T14:08:27.780 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Mar 30 14:08:27 smithi043 systemd[1]: Stopped Ceph rgw.foo.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T14:08:27.780 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Mar 30 14:08:27 smithi043 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@rgw.foo.a.service: Consumed 3.747s CPU time. 2024-03-30T14:08:27.781 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@rgw.foo.a.service' 2024-03-30T14:08:27.819 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:08:27.819 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-30T14:08:27.819 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-30T14:08:27.820 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@prometheus.a 2024-03-30T14:08:28.125 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:08:27 smithi113 systemd[1]: Stopping Ceph prometheus.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:08:28.424 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:08:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T14:08:28.133Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-30T14:08:28.424 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:08:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T14:08:28.133Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-30T14:08:28.424 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:08:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T14:08:28.133Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-30T14:08:28.424 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:08:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T14:08:28.133Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-30T14:08:28.424 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:08:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T14:08:28.133Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-30T14:08:28.424 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:08:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T14:08:28.133Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-30T14:08:28.424 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:08:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T14:08:28.133Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-30T14:08:28.425 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:08:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T14:08:28.133Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-30T14:08:28.425 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:08:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T14:08:28.143Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-30T14:08:28.425 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:08:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T14:08:28.144Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-30T14:08:28.425 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:08:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T14:08:28.144Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-30T14:08:28.425 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:08:28 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a[73470]: ts=2024-03-30T14:08:28.144Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-30T14:08:28.425 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:08:28 smithi113 podman[172135]: 2024-03-30 14:08:28.154284106 +0000 UTC m=+0.142719399 container died 396a2e4a311209eb5833acf6b53cc7510955e3cdc18c60eecc0b7bc6a5dbe488 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T14:08:28.713 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:08:28 smithi113 podman[172135]: 2024-03-30 14:08:28.538607533 +0000 UTC m=+0.527042829 container cleanup 396a2e4a311209eb5833acf6b53cc7510955e3cdc18c60eecc0b7bc6a5dbe488 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T14:08:28.713 INFO:journalctl@ceph.prometheus.a.smithi113.stdout:Mar 30 14:08:28 smithi113 bash[172135]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-prometheus-a 2024-03-30T14:08:28.938 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@prometheus.a.service' 2024-03-30T14:08:28.975 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:08:28.976 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-30T14:08:28.976 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-03-30T14:08:29.173 INFO:teuthology.orchestra.run.smithi043.stdout:Deleting cluster with fsid: 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T14:08:31.098 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 14:08:30 smithi043 systemd[1]: Stopping Ceph alertmanager.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:08:31.378 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 14:08:31 smithi043 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a[88910]: ts=2024-03-30T14:08:31.104Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-30T14:08:31.378 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 14:08:31 smithi043 podman[232356]: 2024-03-30 14:08:31.145965014 +0000 UTC m=+0.158846129 container died 165ce8ff7c932b48df9973805204b5a141e551da0f3cd187ae76ff18cf4f019b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T14:08:31.640 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 14:08:31 smithi043 podman[232356]: 2024-03-30 14:08:31.560465661 +0000 UTC m=+0.573346775 container cleanup 165ce8ff7c932b48df9973805204b5a141e551da0f3cd187ae76ff18cf4f019b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T14:08:31.640 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 14:08:31 smithi043 bash[232356]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a 2024-03-30T14:08:32.140 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 14:08:31 smithi043 podman[232376]: 2024-03-30 14:08:31.778307568 +0000 UTC m=+0.630649669 container remove 165ce8ff7c932b48df9973805204b5a141e551da0f3cd187ae76ff18cf4f019b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T14:08:32.140 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 14:08:31 smithi043 podman[232376]: 2024-03-30 14:08:31.862549981 +0000 UTC m=+0.714892037 volume remove 4126d65f6f44ee6af51dbff3faac7c1022b7e372929d7f9f1968fa9bdd9558c8 2024-03-30T14:08:32.140 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 14:08:31 smithi043 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@alertmanager.a.service: Deactivated successfully. 2024-03-30T14:08:32.140 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 14:08:31 smithi043 systemd[1]: Stopped Ceph alertmanager.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T14:08:32.140 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Mar 30 14:08:31 smithi043 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@alertmanager.a.service: Consumed 1.892s CPU time. 2024-03-30T14:08:32.640 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 14:08:32 smithi043 systemd[1]: Stopping Ceph node-exporter.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:08:32.640 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 14:08:32 smithi043 podman[232492]: 2024-03-30 14:08:32.437484696 +0000 UTC m=+0.104924134 container died 4ef41d68b90e4be012ea6f03bc40a8c8d27ce6f87a9ce58f173d40e9d255ad48 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-30T14:08:32.932 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 14:08:32 smithi043 podman[232492]: 2024-03-30 14:08:32.748092142 +0000 UTC m=+0.415531589 container cleanup 4ef41d68b90e4be012ea6f03bc40a8c8d27ce6f87a9ce58f173d40e9d255ad48 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-30T14:08:32.933 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 14:08:32 smithi043 bash[232492]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a 2024-03-30T14:08:33.264 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 14:08:32 smithi043 podman[232509]: 2024-03-30 14:08:32.940789409 +0000 UTC m=+0.507967993 container remove 4ef41d68b90e4be012ea6f03bc40a8c8d27ce6f87a9ce58f173d40e9d255ad48 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-30T14:08:33.265 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 14:08:32 smithi043 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-30T14:08:33.265 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 14:08:33 smithi043 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-30T14:08:33.265 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 14:08:33 smithi043 systemd[1]: Stopped Ceph node-exporter.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T14:08:33.265 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Mar 30 14:08:33 smithi043 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@node-exporter.a.service: Consumed 4.870s CPU time. 2024-03-30T14:08:33.731 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-03-30T14:08:33.923 INFO:teuthology.orchestra.run.smithi113.stdout:Deleting cluster with fsid: 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T14:08:35.463 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:08:35 smithi113 systemd[1]: Stopping Ceph iscsi.iscsi.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:08:35.963 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:08:35 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[64665]: debug Shutdown received 2024-03-30T14:08:45.851 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:08:45 smithi113 bash[172557]: time="2024-03-30T14:08:45Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-30T14:08:45.851 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:08:45 smithi113 podman[172557]: 2024-03-30 14:08:45.526336614 +0000 UTC m=+10.124036875 container died 2f903e8571e343740df1900336a26310e1f46a03d9b79056308c65d0f8266d53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, GIT_CLEAN=True) 2024-03-30T14:08:46.134 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:08:45 smithi113 podman[172557]: 2024-03-30 14:08:45.932917479 +0000 UTC m=+10.530617750 container cleanup 2f903e8571e343740df1900336a26310e1f46a03d9b79056308c65d0f8266d53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-30T14:08:46.135 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:08:45 smithi113 bash[172557]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a 2024-03-30T14:08:46.399 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:08:46 smithi113 podman[172583]: 2024-03-30 14:08:46.142602185 +0000 UTC m=+0.617578467 container remove 2f903e8571e343740df1900336a26310e1f46a03d9b79056308c65d0f8266d53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-iscsi-iscsi-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:08:46.399 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:08:46 smithi113 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-30T14:08:46.399 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:08:46 smithi113 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-30T14:08:46.399 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:08:46 smithi113 systemd[1]: Stopped Ceph iscsi.iscsi.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T14:08:46.399 INFO:journalctl@ceph.iscsi.iscsi.a.smithi113.stdout:Mar 30 14:08:46 smithi113 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@iscsi.iscsi.a.service: Consumed 3.617s CPU time. 2024-03-30T14:08:48.213 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 14:08:47 smithi113 systemd[1]: Stopping Ceph grafana.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:08:48.214 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 14:08:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=server t=2024-03-30T14:08:48.036051805Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-03-30T14:08:48.214 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 14:08:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=tracing t=2024-03-30T14:08:48.036442647Z level=info msg="Closing tracing" 2024-03-30T14:08:48.214 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 14:08:48 smithi113 ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a[68570]: logger=ticker t=2024-03-30T14:08:48.036478466Z level=info msg=stopped last_tick=2024-03-30T14:08:40Z 2024-03-30T14:08:48.214 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 14:08:48 smithi113 podman[172858]: 2024-03-30 14:08:48.056548983 +0000 UTC m=+0.107034082 container died d71d74017d3f4c67fd3101752e2225b11d8b0f5a3d4f7c47b9d50035e0f07b69 (image=quay.io/ceph/grafana:9.4.12, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a, maintainer=Grafana Labs ) 2024-03-30T14:08:48.693 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 14:08:48 smithi113 podman[172858]: 2024-03-30 14:08:48.416272681 +0000 UTC m=+0.466757774 container cleanup d71d74017d3f4c67fd3101752e2225b11d8b0f5a3d4f7c47b9d50035e0f07b69 (image=quay.io/ceph/grafana:9.4.12, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a, maintainer=Grafana Labs ) 2024-03-30T14:08:48.693 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 14:08:48 smithi113 bash[172858]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a 2024-03-30T14:08:48.963 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 14:08:48 smithi113 podman[172877]: 2024-03-30 14:08:48.700746682 +0000 UTC m=+0.637394285 container remove d71d74017d3f4c67fd3101752e2225b11d8b0f5a3d4f7c47b9d50035e0f07b69 (image=quay.io/ceph/grafana:9.4.12, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-grafana-a, maintainer=Grafana Labs ) 2024-03-30T14:08:48.963 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 14:08:48 smithi113 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@grafana.a.service: Deactivated successfully. 2024-03-30T14:08:48.963 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 14:08:48 smithi113 systemd[1]: Stopped Ceph grafana.a for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T14:08:48.963 INFO:journalctl@ceph.grafana.a.smithi113.stdout:Mar 30 14:08:48 smithi113 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@grafana.a.service: Consumed 15.973s CPU time. 2024-03-30T14:08:49.283 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 14:08:49 smithi113 systemd[1]: Stopping Ceph node-exporter.b for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4... 2024-03-30T14:08:49.537 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 14:08:49 smithi113 podman[172993]: 2024-03-30 14:08:49.291786419 +0000 UTC m=+0.131418572 container died 5e993f09406b6f5614f70bf613ff8cad6e6910c5f23f532fd6512a2867f78b2e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-30T14:08:49.904 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 14:08:49 smithi113 podman[172993]: 2024-03-30 14:08:49.636743764 +0000 UTC m=+0.476375917 container cleanup 5e993f09406b6f5614f70bf613ff8cad6e6910c5f23f532fd6512a2867f78b2e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-30T14:08:49.905 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 14:08:49 smithi113 bash[172993]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b 2024-03-30T14:08:50.185 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 14:08:49 smithi113 podman[173012]: 2024-03-30 14:08:49.91286906 +0000 UTC m=+0.625779793 container remove 5e993f09406b6f5614f70bf613ff8cad6e6910c5f23f532fd6512a2867f78b2e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-30T14:08:50.186 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 14:08:49 smithi113 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-30T14:08:50.186 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 14:08:50 smithi113 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-30T14:08:50.186 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 14:08:50 smithi113 systemd[1]: Stopped Ceph node-exporter.b for 7ee932fa-ee9a-11ee-b647-cb9ed24678a4. 2024-03-30T14:08:50.186 INFO:journalctl@ceph.node-exporter.b.smithi113.stdout:Mar 30 14:08:50 smithi113 systemd[1]: ceph-7ee932fa-ee9a-11ee-b647-cb9ed24678a4@node-exporter.b.service: Consumed 4.513s CPU time. 2024-03-30T14:08:50.721 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T14:08:50.756 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T14:08:50.786 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-30T14:08:50.788 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630102/remote/smithi043/crash 2024-03-30T14:08:50.789 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/crash -- . 2024-03-30T14:08:50.834 INFO:teuthology.orchestra.run.smithi043.stderr:tar: /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-03-30T14:08:50.835 INFO:teuthology.orchestra.run.smithi043.stderr:tar: Error is not recoverable: exiting now 2024-03-30T14:08:50.837 DEBUG:teuthology.misc:Transferring archived files from smithi113:/var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630102/remote/smithi113/crash 2024-03-30T14:08:50.838 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar c -f - -C /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/crash -- . 2024-03-30T14:08:50.882 INFO:teuthology.orchestra.run.smithi113.stderr:tar: /var/lib/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-03-30T14:08:50.882 INFO:teuthology.orchestra.run.smithi113.stderr:tar: Error is not recoverable: exiting now 2024-03-30T14:08:50.883 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-30T14:08:50.884 DEBUG:teuthology.orchestra.run.smithi043:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-30T14:08:50.977 INFO:teuthology.orchestra.run.smithi043.stdout:2024-03-30T13:50:00.000116+0000 mon.a (mon.0) 1104 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-30T14:08:50.978 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-30T14:08:50.978 DEBUG:teuthology.orchestra.run.smithi043:> sudo egrep '\[SEC\]' /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-30T14:08:51.013 DEBUG:teuthology.orchestra.run.smithi043:> sudo egrep '\[ERR\]' /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-30T14:08:51.085 DEBUG:teuthology.orchestra.run.smithi043:> sudo egrep '\[WRN\]' /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-30T14:08:51.155 INFO:teuthology.orchestra.run.smithi043.stdout:2024-03-30T13:50:00.000116+0000 mon.a (mon.0) 1104 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-30T14:08:51.155 INFO:tasks.cephadm:Compressing logs... 2024-03-30T14:08:51.155 DEBUG:teuthology.orchestra.run.smithi043:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-30T14:08:51.197 DEBUG:teuthology.orchestra.run.smithi113:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-30T14:08:51.246 INFO:teuthology.orchestra.run.smithi043.stderr:find: ‘/var/log/rbd-target-api’gzip: No such file or directory -5 2024-03-30T14:08:51.246 INFO:teuthology.orchestra.run.smithi043.stderr: --verbose -- /var/log/ceph/cephadm.log 2024-03-30T14:08:51.246 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-mon.a.log 2024-03-30T14:08:51.246 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-03-30T14:08:51.246 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-client.rgw.foo.a.log 2024-03-30T14:08:51.246 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-mon.a.log: /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.1.log 2024-03-30T14:08:51.246 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/objectstore_tool.7.log 2024-03-30T14:08:51.246 INFO:teuthology.orchestra.run.smithi043.stderr: 57.8% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-client.rgw.foo.a.log.gz 2024-03-30T14:08:51.247 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.3.log 2024-03-30T14:08:51.247 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-mgr.y.log 2024-03-30T14:08:51.247 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.0.log 2024-03-30T14:08:51.247 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.cephadm.log 2024-03-30T14:08:51.247 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.log 2024-03-30T14:08:51.247 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.2.log 2024-03-30T14:08:51.247 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-03-30T14:08:51.247 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/objectstore_tool.8.log 2024-03-30T14:08:51.247 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-mon.c.log 2024-03-30T14:08:51.253 INFO:teuthology.orchestra.run.smithi113.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-30T14:08:51.253 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-30T14:08:51.253 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.6.log 2024-03-30T14:08:51.253 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-03-30T14:08:51.253 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.4.log 2024-03-30T14:08:51.253 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.7.log 2024-03-30T14:08:51.253 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.5.log 2024-03-30T14:08:51.253 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.7.log: --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-mgr.x.log 2024-03-30T14:08:51.253 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/tcmu-runner.log 2024-03-30T14:08:51.253 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.cephadm.log 2024-03-30T14:08:51.253 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.log 2024-03-30T14:08:51.254 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-03-30T14:08:51.254 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.log: gzip -5 --verbose -- /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-mon.b.log 2024-03-30T14:08:51.257 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/objectstore_tool.8.log: /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-mon.c.log: 91.9% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-03-30T14:08:51.275 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-volume.log: /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-mon.b.log: 68.0% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/tcmu-runner.log.gz 2024-03-30T14:08:51.280 INFO:teuthology.orchestra.run.smithi043.stderr: 94.2% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/objectstore_tool.8.log.gz 2024-03-30T14:08:51.305 INFO:teuthology.orchestra.run.smithi043.stderr: 95.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-30T14:08:51.365 INFO:teuthology.orchestra.run.smithi113.stderr: 85.7% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-03-30T14:08:51.415 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-03-30T14:08:51.520 INFO:teuthology.orchestra.run.smithi113.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-30T14:08:51.542 INFO:teuthology.orchestra.run.smithi113.stderr: 91.1% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-03-30T14:08:51.618 INFO:teuthology.orchestra.run.smithi113.stderr: 92.4% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-mgr.x.log.gz 2024-03-30T14:08:51.627 INFO:teuthology.orchestra.run.smithi113.stderr: 90.0% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-03-30T14:08:51.694 INFO:teuthology.orchestra.run.smithi043.stderr: 94.2% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/objectstore_tool.7.log.gz 2024-03-30T14:08:51.731 INFO:teuthology.orchestra.run.smithi043.stderr: 91.2% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-03-30T14:08:51.823 INFO:teuthology.orchestra.run.smithi113.stderr: 81.5% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-03-30T14:08:52.280 INFO:teuthology.orchestra.run.smithi043.stderr: 94.0% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-03-30T14:08:53.104 INFO:teuthology.orchestra.run.smithi043.stderr: 89.8% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-mgr.y.log.gz 2024-03-30T14:08:55.526 INFO:teuthology.orchestra.run.smithi043.stderr: 91.1% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-mon.c.log.gz 2024-03-30T14:08:58.960 INFO:teuthology.orchestra.run.smithi043.stderr: 90.3% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-mon.a.log.gz 2024-03-30T14:09:05.432 INFO:teuthology.orchestra.run.smithi113.stderr: 91.2% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-mon.b.log.gz 2024-03-30T14:11:23.601 INFO:teuthology.orchestra.run.smithi113.stderr: 93.1% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.6.log.gz 2024-03-30T14:12:48.273 INFO:teuthology.orchestra.run.smithi113.stderr: 93.4% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.7.log.gz 2024-03-30T14:14:31.951 INFO:teuthology.orchestra.run.smithi113.stderr: 93.9% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.5.log.gz 2024-03-30T14:14:34.852 INFO:teuthology.orchestra.run.smithi043.stderr: 94.2% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.2.log.gz 2024-03-30T14:14:52.724 INFO:teuthology.orchestra.run.smithi113.stderr: 93.4% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.4.log.gz 2024-03-30T14:14:52.725 INFO:teuthology.orchestra.run.smithi113.stderr: 2024-03-30T14:14:52.725 INFO:teuthology.orchestra.run.smithi113.stderr:real 6m1.514s 2024-03-30T14:14:52.725 INFO:teuthology.orchestra.run.smithi113.stderr:user 4m53.995s 2024-03-30T14:14:52.725 INFO:teuthology.orchestra.run.smithi113.stderr:sys 0m12.956s 2024-03-30T14:18:52.794 INFO:teuthology.orchestra.run.smithi043.stderr: 92.6% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.1.log.gz 2024-03-30T14:25:20.381 INFO:teuthology.orchestra.run.smithi043.stderr: 92.7% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.0.log.gz 2024-03-30T14:25:23.366 INFO:teuthology.orchestra.run.smithi043.stderr: 92.3% -- replaced with /var/log/ceph/7ee932fa-ee9a-11ee-b647-cb9ed24678a4/ceph-osd.3.log.gz 2024-03-30T14:25:23.437 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-03-30T14:25:23.437 INFO:teuthology.orchestra.run.smithi043.stderr:real 16m32.229s 2024-03-30T14:25:23.437 INFO:teuthology.orchestra.run.smithi043.stderr:user 8m21.989s 2024-03-30T14:25:23.437 INFO:teuthology.orchestra.run.smithi043.stderr:sys 0m21.966s 2024-03-30T14:25:23.438 INFO:tasks.cephadm:Archiving logs... 2024-03-30T14:25:23.439 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630102/remote/smithi043/log 2024-03-30T14:25:23.440 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T14:26:41.421 DEBUG:teuthology.misc:Transferring archived files from smithi113:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630102/remote/smithi113/log 2024-03-30T14:26:41.423 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T14:27:27.150 INFO:tasks.cephadm:Removing cluster... 2024-03-30T14:27:27.150 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 --force 2024-03-30T14:27:28.145 INFO:teuthology.orchestra.run.smithi043.stdout:Deleting cluster with fsid: 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T14:27:29.025 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 --force 2024-03-30T14:27:29.277 INFO:teuthology.orchestra.run.smithi113.stdout:Deleting cluster with fsid: 7ee932fa-ee9a-11ee-b647-cb9ed24678a4 2024-03-30T14:27:29.931 INFO:tasks.cephadm:Removing cephadm ... 2024-03-30T14:27:29.932 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-30T14:27:29.951 DEBUG:teuthology.orchestra.run.smithi113:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-30T14:27:29.967 INFO:tasks.cephadm:Teardown complete 2024-03-30T14:27:29.967 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-30T14:27:29.980 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-30T14:27:29.980 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T14:27:29.994 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T14:27:30.093 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-30T14:27:30.105 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-30T14:27:30.105 DEBUG:teuthology.orchestra.run.smithi043:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T14:27:30.108 DEBUG:teuthology.orchestra.run.smithi113:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T14:27:30.121 INFO:teuthology.orchestra.run.smithi043.stderr:bash: line 1: ntpq: command not found 2024-03-30T14:27:30.122 INFO:teuthology.orchestra.run.smithi113.stderr:bash: line 1: ntpq: command not found 2024-03-30T14:27:30.147 INFO:teuthology.orchestra.run.smithi113.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T14:27:30.148 INFO:teuthology.orchestra.run.smithi113.stdout:=============================================================================== 2024-03-30T14:27:30.148 INFO:teuthology.orchestra.run.smithi113.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 +71us[ +71us] +/- 92ms 2024-03-30T14:27:30.148 INFO:teuthology.orchestra.run.smithi113.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 -261us[ -252us] +/- 71ms 2024-03-30T14:27:30.148 INFO:teuthology.orchestra.run.smithi113.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 +206us[ +215us] +/- 65ms 2024-03-30T14:27:30.148 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:27:30.213 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T14:27:30.213 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2024-03-30T14:27:30.213 INFO:teuthology.orchestra.run.smithi043.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 34 +135us[ +141us] +/- 92ms 2024-03-30T14:27:30.213 INFO:teuthology.orchestra.run.smithi043.stdout:^* hv02.front.sepia.ceph.com 3 6 377 31 -245us[ -239us] +/- 71ms 2024-03-30T14:27:30.213 INFO:teuthology.orchestra.run.smithi043.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +213us[ +219us] +/- 65ms 2024-03-30T14:27:30.213 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:27:30.226 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-30T14:27:30.239 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-30T14:27:30.240 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-30T14:27:30.268 DEBUG:teuthology.orchestra.run.smithi043:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-30T14:27:30.398 DEBUG:teuthology.orchestra.run.smithi113:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-30T14:27:30.474 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-30T14:27:30.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:27:30.508 DEBUG:teuthology.orchestra.run.smithi113:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-30T14:27:30.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:27:30.537 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-30T14:27:30.548 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-30T14:27:30.617 INFO:teuthology.task.internal:Duration was 3663.843064 seconds 2024-03-30T14:27:30.617 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-30T14:27:30.699 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-30T14:27:30.699 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T14:27:30.702 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T14:27:30.763 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T14:27:30.784 INFO:teuthology.orchestra.run.smithi113.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T14:27:31.257 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-30T14:27:31.257 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi043.front.sepia.ceph.com 2024-03-30T14:27:31.257 DEBUG:teuthology.orchestra.run.smithi043:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T14:27:31.280 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi113.front.sepia.ceph.com 2024-03-30T14:27:31.280 DEBUG:teuthology.orchestra.run.smithi113:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T14:27:31.303 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-30T14:27:31.303 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T14:27:31.323 DEBUG:teuthology.orchestra.run.smithi113:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T14:27:31.372 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-30T14:27:31.372 DEBUG:teuthology.orchestra.run.smithi043:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T14:27:31.374 DEBUG:teuthology.orchestra.run.smithi113:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T14:27:33.021 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-30T14:27:33.032 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-30T14:27:33.032 DEBUG:teuthology.orchestra.run.smithi043:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T14:27:33.084 DEBUG:teuthology.orchestra.run.smithi113:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T14:27:33.128 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-30T14:27:33.139 DEBUG:teuthology.orchestra.run.smithi043:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T14:27:33.141 DEBUG:teuthology.orchestra.run.smithi113:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T14:27:33.179 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = core 2024-03-30T14:27:33.206 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern = core 2024-03-30T14:27:33.248 DEBUG:teuthology.orchestra.run.smithi043:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T14:27:33.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:27:33.261 DEBUG:teuthology.orchestra.run.smithi113:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T14:27:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:27:33.304 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-30T14:27:33.319 INFO:teuthology.task.internal:Transferring archived files... 2024-03-30T14:27:33.320 DEBUG:teuthology.misc:Transferring archived files from smithi043:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630102/remote/smithi043 2024-03-30T14:27:33.320 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T14:27:33.383 DEBUG:teuthology.misc:Transferring archived files from smithi113:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630102/remote/smithi113 2024-03-30T14:27:33.384 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T14:27:33.429 INFO:teuthology.task.internal:Removing archive directory... 2024-03-30T14:27:33.429 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T14:27:33.432 DEBUG:teuthology.orchestra.run.smithi113:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T14:27:33.478 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-30T14:27:33.490 INFO:teuthology.task.internal:Not uploading archives. 2024-03-30T14:27:33.490 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-30T14:27:33.517 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-30T14:27:33.517 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T14:27:33.519 DEBUG:teuthology.orchestra.run.smithi113:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T14:27:33.531 INFO:teuthology.orchestra.run.smithi043.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 14:27 /home/ubuntu/cephtest 2024-03-30T14:27:33.535 INFO:teuthology.orchestra.run.smithi113.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 14:27 /home/ubuntu/cephtest 2024-03-30T14:27:33.536 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-30T14:27:33.547 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-30T14:27:33.619 INFO:teuthology.nuke:Checking targets against current locks 2024-03-30T14:27:33.647 DEBUG:teuthology.nuke:shortname: smithi043 2024-03-30T14:27:33.648 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T14:27:33.676 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630102', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 13:15:45.196358', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM6PsHyrzJ+N5gtxEXZtkL0DOB2Yc+1hk6UM158QukrtKXAgr1PH+4Y1Hhvn9HNMZfMFYP2GXk/TAmnZ4Q4miGk='} 2024-03-30T14:27:33.680 DEBUG:teuthology.nuke:shortname: smithi113 2024-03-30T14:27:33.680 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T14:27:33.701 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi113.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630102', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 13:15:45.194864', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCu38/1DtzijFJsFOMosEFKsw6EaV5ESDEi6yWWilL+EjrhFGkFcqWMJntOUWFnepOK844Bl4KbsA8br8yrqP48='} 2024-03-30T14:27:33.735 INFO:teuthology.orchestra.console.smithi043:Power off 2024-03-30T14:27:33.735 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T14:27:33.754 INFO:teuthology.orchestra.console.smithi113:Power off 2024-03-30T14:27:33.754 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T14:27:33.764 DEBUG:teuthology.orchestra.console.smithi043:power off output: Chassis Power Control: Down/Off 2024-03-30T14:27:33.764 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:27:33.781 DEBUG:teuthology.orchestra.console.smithi113:power off output: Chassis Power Control: Down/Off 2024-03-30T14:27:33.781 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:27:33.796 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is on 2024-03-30T14:27:33.805 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is on 2024-03-30T14:27:37.797 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:27:37.806 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:27:37.908 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is on 2024-03-30T14:27:37.915 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is on 2024-03-30T14:27:41.912 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:27:41.921 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:27:42.024 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is off 2024-03-30T14:27:42.031 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is on 2024-03-30T14:27:42.125 INFO:teuthology.orchestra.console.smithi043:Power off completed 2024-03-30T14:27:46.034 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:27:46.144 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is off 2024-03-30T14:27:46.245 INFO:teuthology.orchestra.console.smithi113:Power off completed 2024-03-30T14:27:46.349 INFO:teuthology.run:Summary data: description: orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} duration: 3663.8430638313293 failure_reason: '"2024-03-30T13:50:00.000116+0000 mon.a (mon.0) 1104 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-30T14:27:46.349 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T14:27:46.439 INFO:teuthology.run:FAIL